Thorsten Berger
email: thorsten.berger@chalmers.se

Jan-Philipp Stegh
email: jan-philipp.steghofer@chalmers.se

Tewfik Ziadi
email: tewfik.ziadi@lip6.fr

Jacques Robin
email: jacques.robin@univ-paris1.fr

Jabier Martinez
email: jabier.martinez@tecnalia.com

J.-P Stegh

The State of Adoption and the Challenges of Systematic Variability Management in Industry

Keywords: variability management, software product lines, multiple-case study, challenges

Handling large-scale software variability is still a challenge for many organizations. After decades of research on variability management concepts, many industrial organizations have introduced techniques known from research, but still lament that pure textbook approaches are not applicable or efficient. For instance, software product line engineering-an approach to systematically develop portfolios of products-is difficult to adopt given the high upfront investments; and even when adopted, organizations are challenged by evolving their complex product lines. Consequently, the research community now mainly focuses on reengineering and evolution techniques for product lines; yet, understanding the current state of adoption and the industrial challenges for organizations is necessary to conceive effective techniques. In this multiple-case study, we analyze the current adoption of variability management techniques in twelve medium-to largescale industrial cases in domains such as automotive,

Introduction

Companies often need to engineer a portfolio of software variants instead of one-size-fits-all solutions. Creating variants allows tailoring systems towards varying stakeholder requirements-different functionalities, but also non-functional requirements, such as performance or power consumption. Variant-rich systems are especially common in traditional engineering domains including automotive, industrial automation, telecommunication. In addition, recent trends such as the Internet of Things (IoT) [START_REF] Atzori | The internet of things: A survey[END_REF], cyber-physical systems [113,156], or robotics [START_REF] Garcia | Variability modeling of service robots: Experiences and challenges[END_REF], further increase the need for customization.

Software product line engineering (SPLE) aims at effectively engineering a variant-rich system-a software product line-in an application domain. SPLE advocates establishing an integrated software platform from which individual variants can be derived, typically in an automated, configuration-driven process. SPLE provides a range of dedicated concepts, including processes, modeling techniques or design patterns, supported by commercial and open-source tools, such as pure::variants [START_REF] Beuche | pure::variants Eclipse Plugin[END_REF], Gears [106] or FeatureIDE [101].

Recognizing the benefits, especially radically decreased time-to-markets for new variants, industry has adopted SPLE concepts [START_REF] Jepsen | Running a software product line: standing still is going backwards[END_REF][START_REF] Flores | Mega-Scale Product Line Engineering at General Motors[END_REF][START_REF] Berger | A survey of variability modeling in industrial practice[END_REF]132,[START_REF] Chen | Variability management in software product lines: an investigation of contemporary industrial challenges[END_REF][START_REF] Bastos | Software product lines adoption in small organizations[END_REF]184] at different levels of maturity [START_REF] Bosch | Maturity and evolution in software product lines: Approaches, artefacts and organization[END_REF]. Yet, many organizations still lament an adoption barrier. In fact, most organizations create variants in ad hoc ways [START_REF] Berger | A survey of variability modeling in industrial practice[END_REF], such as clone & own, which is simple and cheap [START_REF] Dubinsky | An exploratory study of cloning in industrial software product lines[END_REF][START_REF] Businge | Clone-based variability management in the android ecosystem[END_REF]177,178], but does not scale with the number of variants and then requires product-line migration efforts [START_REF] Assunc ¸ão | Reengineering legacy applications into software product lines: a systematic mapping[END_REF][START_REF] Faust | Software product line migration and deployment[END_REF][START_REF] Jepsen | Minimally invasive migration to software product lines[END_REF][START_REF] Debbiche | Migrating the java-based apo-games into a composition-based software product line[END_REF][START_REF] Akesson | Migrating the android apo-games into an annotation-based software product line[END_REF][START_REF] Fenske | A taxonomy of software product line reengineering[END_REF]. When adopted, a product line can also limit flexibility, since evolving the platform affects many variants [129]. To improve this situation, we need to improve our empirical understanding of the state of product-line adoption and the needs for improvement in industrial practice.

Such an updated empirical understanding of industrial needs helps steer the scope of research efforts. Consider the many SPLE concepts that have been conceived by researchers. Already until 2011, a survey identified 91 variability management approaches [START_REF] Chen | A systematic review of evaluation of variability management approaches in software product lines[END_REF], most of which rely on feature modeling to specify variability information [START_REF] Kang | Feature-oriented domain analysis (FODA) feasibility study[END_REF][START_REF] Czarnecki | Cool features and tough decisions: A comparison of variability modeling approaches[END_REF][START_REF] Acher | FAMILIAR: A domain-specific language for large scale management of feature models[END_REF]141]. Another trend was to build hundreds of dedicated analyses for product lines, typically by lifting single-system analyses (e.g., model checking) [130,117,162]. A recent survey [186] identified 123 analyses from the literature-including lifted type-checking, static-analysis, and model-checking techniques. Already eight years ago, a survey [START_REF] Benavides | Automated analysis of feature models 20 years later: A literature review[END_REF] identified 30 different feature-model analyses in 53 papers). Other analyses check the consistency between feature models and implementation artifacts or analyze code properties. Notably, more recent work [138] shows that industrial needs substantially deviate from the state of the art, while most analyses are not applicable in industrial contexts. Likewise, traceability of features across a product line's lifecycle is another challenge [189], where the state of the art and the state of the practice differ significantly, and the low industrial relevance of solutions proposed in the literature prevents wider adoption of traceability for product lines in industry.

We present a study on the state of adoption of variability-management concepts and remaining practical challenges in twelve industrial cases from different organizations engineering variant-rich systems. We used document analysis, semi-structured interviews, and focus groups on cases that cover a wide range of domains and development scales, from a rather small web-application engineering case to ultra-large software engineering for automotive or industrial component production. In addition, we conducted a lightweight literature review on relevant SPLE adoption case studies, experience reports, surveys, and meta-studies, supporting the formulation and synthesis of challenges we present. Our industrial cases primarily represent the development in a small part of a company, such as a single division or development team. We refer to these as use cases in the remainder. We also investigated cases provided by tool vendors who are looking to integrate product-line engineering concepts into their tools, referred to as tool cases. For our study, we combined eliciting structured case descriptions with focus-group interviews to identify the concepts that were adopted, as well as variability drivers and variable assets.

We report our cases' drivers of variability, the SPLE concepts they adopted, as well as remaining challenges to be addressed by the research community and tool vendors. We believe our results support practitioners and researchers. Practitioners can use our results as a baseline to compare their own organization's capabilities and understand which concepts are at their disposal for future development. Researchers obtain the current state of adoption of SPLE concepts in industry, as well as they learn about remaining challenges.

Background

We briefly discuss strategies and important concepts for engineering variant-rich systems, gradually from ad hoc strategies to more advanced variant management strategies and concepts, partly inspired by the levels proposed by Antkiewicz et al. [START_REF] Antkiewicz | Flexible product line engineering with a virtual platform[END_REF]. Clone & own. An organization creates variants by copying and adapting existing variants to new requirements. Assets are propagated in an ad hoc way among the variants. No platform or any kind of systematic variability management exists. Clone & own is a simple and readily available [START_REF] Dubinsky | An exploratory study of cloning in industrial software product lines[END_REF][START_REF] Duc | Forking and coordination in multi-platform development: A case study[END_REF][START_REF] Businge | Clone-based variability management in the android ecosystem[END_REF]177] strategy for developing variants, but does not scale with the number of variants and easily causes maintenance overheads. Clone Management. Enhancing the governance, an organization could adopt a clone-management framework. Such frameworks [159,160,161,148,[START_REF] Antkiewicz | Flexible product line engineering with a virtual platform[END_REF] have been proposed, but have not found any documented adoption. They allow managing clones by using features as the main entities of reuse (instead of code assets) and record meta-data about the clones.

Configuration. An organization can introduce configuration mechanisms to reduce redundancies. Such a mechanism is an implementation technique to realize calibration or variation. In the latter case, it is commonly referred to as a variability mechanism [190,[START_REF] Berger | Variability Mechanisms in Software Ecosystems[END_REF], ranging from simple conditional compilation (e.g., #ifdef) via control-flow conditional statements (e.g., IF), build systems [START_REF] Berger | Feature-to-code mapping in two large product lines[END_REF][START_REF] Dietrich | A robust approach for variability extraction from the linux build system[END_REF], and component frameworks, to so-called feature modules [START_REF] Apel | An overview of feature-oriented software development[END_REF] or delta modules [163]; or com-Fig. 1 Feature model example [START_REF] Berger | Variability Mechanisms in Software Ecosystems[END_REF] binations thereof [START_REF] Behringer | Peopl: Projectional editing of product lines[END_REF]137]. The configuration options (a.k.a., calibration parameters or just parameters) and their constraints are typically declared in a model, such as a feature model (explained shortly). Using such a model, an interactive configurator tool can support the configuration process, guiding users by propagating choices or resolving configuration conflicts. Platform. Further scaling the development, an organization can adopt an integrated platform. There, instead of cloning, all variants are integrated into one software platform. By exploiting the commonalities among the variants, redundancies are removed while variability among the variants is represented by variation points within the platform. Variability is typically described in terms of features declared in a feature model, as follows. Feature. Using the notion of feature is core to scaling the development, for instance, when adopting a platform. A feature represents end-to-end functionality of a system [START_REF] Berger | What is a feature? A qualitative study of features in industrial software product lines[END_REF]. Features are intuitive entities that can be understood by different roles, including marketing experts, project leads, and customers. Features abstractly represent assets and are tracked, for instance, in a database or, more formally, in a feature model. Many features also serve as a configuration option (a.k.a., optional feature). Feature Model. Keeping an overview understanding of features, organizations should create a feature modelan intuitive, tree-like representation of features and their constraints [START_REF] Kang | Feature-oriented domain analysis (FODA) feasibility study[END_REF][START_REF] Czarnecki | Cool features and tough decisions: A comparison of variability modeling approaches[END_REF][START_REF] Acher | FAMILIAR: A domain-specific language for large scale management of feature models[END_REF]141]. The typical graphical notation is shown in Fig. 1, representing a small excerpt of the Linux kernel's variability model. For brevity, we refer to related work [START_REF] Berger | A study of variability models and languages in the systems software domain[END_REF][START_REF] Berger | Variability modeling in the real: A perspective from the operating systems domain[END_REF][START_REF] Berger | Variability Mechanisms in Software Ecosystems[END_REF] for an introduction into feature models and this particular example. SPLE Process. For an organization to effectively engineer a platform, textbook SPLE methods [START_REF] Apel | Feature-Oriented Software Product Lines[END_REF]150,[START_REF] Czarnecki | Generative Programming: Methods, Tools, and Applications[END_REF] introduce two main processes: domain engineering, which aims at engineering the platform, and application engineering, which aims at deriving individual variants from the platform. Each process comprises typical software-engineering tasks, such as requirements engineering, design, implementation and quality assurance, with domain engineering also containing scoping (determining and prioritizing platform features). Product-Line Quality Assurance. Enhancing the quality assurance, organizations can adopt analyses of product lines [186, [START_REF] Benavides | Automated analysis of feature models 20 years later: A literature review[END_REF]138], which differ conceptually from single-system analyses. The latter, especially dynamic analyses such as testing, can only be used for individual variants. While analyses such as optimization for individual variants can be beneficial, single-system analyses are usually insufficient when a product-line platform has been adopted and errors related to all possible variants should be found. For instance, unwanted feature interactions can occur for certain variants based on the combination of features in the variant. Applying single-system analyses for finding such errors requires configuration sampling [START_REF] Cohen | Interaction testing of highly-configurable systems in the presence of constraints[END_REF]147].

Methodology

Our study aims to "collect and summarize evidence from a large representative sample of the overall population" [134] and can therefore be constructed as a survey. We follow the well-known guidelines by Kitchenham and Pfleeger [102] to structure this section, and we rely on the simplified 7-step process of Linåker et al. [119].

Aims and Objectives

As laid out in Sec. 1, the motivation for our study was to understand which variability management concepts are adopted in substantial industrial cases, and the challenges that industrial practitioners perceive in their daily work, determining the concepts that are still needed. Coming from an academic perspective, we also aimed to identify in how far the academic literature on SPLE takes industrial circumstances into account and to what extent the solutions offered by academia correspond to the practical challenges. Our discussions with practitioners, and our lightweight literature review, supported our view that there is a disconnect between SPLE practice and theory. We formulated three research questions:

RQ1: What are drivers for variability in our cases? RQ2: Which SPLE concepts are adopted in our cases? RQ3: What concepts are missing for our cases and for cases previously reported in the literature?

Planning, Scheduling, and Designing the Study

Based on the research questions, we planned a descriptive study using qualitative data. The study was planned to be conducted in several iterations where data was collected repeatedly, then analyzed, and finally followed up. The data collection itself ran during 2018. In addition, we conducted a lightweight literature review on industrial case studies and experience reports on adopting SPLE, to triangulate with the challenges from our twelve cases.

Our personal network gave us access to companies that engineer variant-rich systems and intend to invest into improving their engineering. We therefore employed purposive sampling, in particular expert sampling [START_REF] Etikan | Comparison of convenience sampling and purposive sampling[END_REF] in which we identified those experts that deal with the problems we set out to investigate. We created a list of potential candidates and narrowed it down by excluding those with insufficient resources to participate. We also selected companies that would provide a specific viewpoint and tried to achieve heterogeneity both in terms of company size, age, and size of the system, and implementation technology. We also achieved a good geographical distribution by including companies from five different countries in the European Union.

The final list consisted of twelve companies, nine of which create a variant-rich system and three of which are tool vendors selling software-engineering tools that want to integrate variability mechanisms into their existing tools. For assuring anonymity, we report the cases we studied and only provide high-level information about the different companies, and the meetings we had with them.

Data Collection

The primary data collection method was a case description provided by the companies that describes the case for product line engineering. It relied on a template that included the product and market context, the technology context, a description of existing processes and automation techniques, goals, and key performance indicators, the variability management practices currently in use, and the principal assets, their reuse and management. The template was piloted with one of the organizations and subsequently refined. The organizations involved in this study iteratively improved the case descriptions to homogenize them and to ensure that relevant information was included.

Based on the description, we conducted a preliminary analysis to identify open questions. This resulted in an interview guide for each of the cases that was used in semi-structured interviews or in small focus groups. While the interviews were conducted with experts from individual companies, the focus groups included representatives from several companies. We chose to use the latter format when cases were similar to each other and we needed to understand the differences and similarities better, and when resources allowed such a meeting. Each of these occasions included one or two researchers and at least two industrial participants. In all cases, the industrial participants were engineers working actively with software product lines. Each interview or focus group lasted between 30 and 60 minutes and was conducted by the researchers in person. Data was collected in the form of extensive notes and shared among all authors. The semi-structured format allowed us to explore additional aspects that were not covered in the original case descriptions. Specifically, we had the following number of small focus-group meetings or interviews: power electronics (3 meetings, 4 participants), traffic control (1 meeting, 1 participant), chip modeling (2 meetings, 3 participants), modeling platform (2 meetings, 2 participants), railway (8 meetings, 3 participants), aerospace (3 meetings, 4 participants), truck manufacturing (2 meetings, 3 participants), web application (1 meeting, 2 participants), automotive firmware (2 meetings, 3 participants), requirements engineering (1 meeting, 1 participant).

On two occasions, we held larger focus group meetings with a majority of the involved researchers and companies. This assured that we met representatives of all companies at least once. We discussed adopted concepts and needs, providing another opportunity to explore differences and similarities.

We also conducted a lightweight literature study where we collected and inspected meta-studies, surveys, exploratory studies, industrial case studies, and experience reports. Even though, we know from experience that the large majority of these publications focus on praising the benefits of SPLE (e.g., cost savings and shorter time to markets) while not providing sufficiently detailed data on the adopted concepts and on challenges (i.e., remaining challenges for SPLE research, as opposed to those that were solved in the case study), we decided to systematically collect those publications and triangulate our synthesized challenges with those from these cases. To collect publications, we used our own expertise as well as we consulted the SPLE community's "Hall of Fame,"1 a book with a collection of successful SPLE cases [121], and the Software Engineering Institute's catalog of case studies [175].

Data Analysis

We analyzed the data in several iterations. The first iteration based on the use case descriptions was mainly targeted at identifying which aspects of the current product line approach in the cases remained unclearto be able to follow-up with our industrial contacts. For this purpose, we transformed each use case into a narrative as presented in Sec. 6. This allowed us to structure the information and find aspects that needed clarification. We then derived questions for semi-structured interviews and small focus groups from this. The results also guided the discussion in the larger focus groups. We met regularly to discuss the open questions and to gain an overall understanding of the issues.

We then merged information from the interviews and the small and large focus groups in the common narrative. In addition, we performed coding on the available data. The codes focused on the current practices and the challenges stated by the companies. We pre-defined codes based on variability management concepts and known challenges from the literature, and complemented these with emergent codes that were based on the first analysis round and the information from the interviews and focus groups. Once a stable set of codes emerged, we conducted a coding workshop to harmonize and refine the codes. Again, all information available at this point was used to validate the codes, to join codes with a high degree of similarity, and to refine the codes, in particular with respect to their concrete formulation. If questions arose during the data analysis or the researchers disagreed on the data, we used personal contacts within the organizations to clarify issues quickly before misunderstandings could arise or bias could manifest.

Once the concepts were identified and the authors agreed on the adopted concepts, we forwarded the information to our contacts for member checking. We also asked clarifying questions about the overall study and the long-term perspective of the cases.

For the literature review, we analyzed the collected publications by reading through the paper, specifically searching for challenges related to variability management concepts that were not solved within the respective case. We mapped those challenges to our challenges, enriching the challenge descriptions. Of course, the validity of reporting challenges from other cases, most of which are at least one or two decades old, is lower than the challenges we extracted from our companies. Still, they substantially enhance the relevance and richness of our reported challenges.

Literature Review

In our lightweight literature review we identified the following related meta-studies, surveys, case studies, and experience reports.

Meta Studies

Marimuthu et al. [124] conduct a tertiary study of systematic studies on variability management. They provide detailed bibliometrics, but do not extract any challenges that might be reported in their identified studies. Bastos et al. [START_REF] Bastos | Software product lines adoption in small organizations[END_REF] investigate product-line adoption in small and medium-scale companies via a multi-method approach comprised of a mapping study, a case study, and a survey among experts. The mainly elicit success factors and practices, but no challenges. Chen et al. [START_REF] Chen | Variability management in software product lines: a systematic review[END_REF] conduct a literature review on variability management and, among other results, list the challenges evolution ("systematic approach to provide a comprehensive support for variability evolution is not available"), scalability of techniques, as well as testing and quality assurance in general. In another study, Chen et al. [START_REF] Chen | A systematic review of evaluation of variability management approaches in software product lines[END_REF] study the state of evaluation of variability management techniques, concluding the lack of proper evaluations as a challenge that researchers should address. Finally, Mohagheghi et al. [132] conduct a literature study on the benefits of software reuse (not limited to reuse via variability management), emphasizing that: "For industry, evaluating reuse of COTS or OSS components, integrating reuse activities in software processes, better data collection and evaluating return on investment are major challenges."

Exploratory Studies

Chen et al. [START_REF] Chen | Variability management in software product lines: an investigation of contemporary industrial challenges[END_REF] present an exploratory study relying on focus-group research investigating the perceived challenges of variability management using eleven participants from organizations that do consulting or in-house SPLE. With respect to variability modeling, the focus group reports, among others, the following challenges: visualization of features, evolution (maintenance) of models (in particular dependency management), and variability modeling being not very user-friendly. In general, the study questions academic techniques. It also points out challenges when migrating to an SPLE (cf. Sec. 7.3), specifically that clone detection techniques are not applicable to multiple systems. Furthermore, while structural variability is well supported, behavioral and timing aspects are not.

Surveys

A survey of variability modeling in industrial practice by Berger et al. [START_REF] Berger | A survey of variability modeling in industrial practice[END_REF] lists specific challenges for variability modeling, including visualization, model evolution, and traceability. Th örn et al. [184] survey variability management in small and medium-scale companies in Sweden; however, the reported challenges are rather general and not specific to variability-management concepts.

Experience Reports and Case Studies

Over the last decades, practitioners and researchers have published a large number of case studies and experience reports, the majority between the end of the 1990s and the beginning of the 2000s. We now list all cases we identified. When available, we provide all references that describe the case in detail. However, some cases were described in the respective source only, not as part of a publication on its own. All the cases in the book of van der Linden et al.

[121] describe successful adoptions of product lines, where usually the typical concepts are adopted (platform, feature modeling, automated product derivation, automated testing); for each, the obstacles and limitations in SPLE are also described. We inspected all the cases: AKVAsmart, Bosch [187,179,183], DNV Software, Mar-ketMaker [192], Nokia Mobile Phones, Nokia Networks, Philips Consumer Electronics Software for Televisions, Philips Medical Systems, Siemens Medical Solutions, and Telvent. Furthermore, the book also referenced the following cases, each of which we inspected as well: Salion [START_REF] Buhrdorf | Salion's experience with a reactive software product line approach[END_REF][START_REF] Clements | Salion, inc.: A software product line case study[END_REF], Testo [164], Axis and Ericsson [181], Axis and Securitas [START_REF] Bosch | Evolution and composition of reusable assets in product-line architectures: A case study[END_REF][START_REF] Bosch | Product-line architectures in industry: A case study[END_REF], and RPG Games [195] From the SPLE community's "Hall of Fame"2 we identified and inspected: Boeing [170], CelsiusTech Systems AB [START_REF] Bass | Software architecture in practice[END_REF][START_REF] Brownsword | A case study in successful product line development[END_REF], Cummins [START_REF] Clements | Software Product Lines: Practices and Patterns[END_REF], Ericsson Telecommunications Switches, Fiscan Security Inspection Systems [115], Hewlett Packard's printer firmwar Owen [188], Home-Away [107], Lockheed Martin, LSI Logic [START_REF] Hetrick | Incremental return on incremental investment: Engenio's transition to software product line practice[END_REF], Lucent, Siemens Healthcare [START_REF] Bartholdt | Re-engineering of a hierarchical product line[END_REF], Toshiba [128], U.S. Army [114], U.S. Naval Research Laboratory [START_REF] Bass | Software architecture in practice[END_REF], General Motors [START_REF] Flores | Mega-Scale Product Line Engineering at General Motors[END_REF], and Danfoss [START_REF] Jepsen | Running a software product line: standing still is going backwards[END_REF][START_REF] Jepsen | Minimally invasive migration to software product lines[END_REF][START_REF] Fogdal | Ten years of product line engineering at danfoss: lessons learned and way ahead[END_REF]. The cases Salion, Bosch, Market-Maker, Nokia, Philips (Medical Systems and Software for Television Sets) were already contained in the book of van der Linden et al. [121], explained above.

We inspected all cases of the Software Engineering Institute's catalog of case studies [175]: US Army's Common Avionics Architecture System (CAAS) [START_REF] Clements | The us army's common avionics architecture system (caas) product line: A case study[END_REF], CCT (Control Channel Toolkit) [START_REF] Clements | Control channel toolkit: a software product line case study[END_REF], Naval Underwater Warfare Center [START_REF] Cohen | Successful product line development and sustainment: A dod case study[END_REF][START_REF] Cohen | Acquisition pilot: Product line acquisition and measurement at nuwc[END_REF][START_REF] Cohen | Case study: A measurement program for product lines[END_REF], Argon [START_REF] Bergey | Software product lines: Experiences from the sixth dod software product line workshop[END_REF], ABB [START_REF] Ganz | Modular turbine control software: A control software architecture for the abb gas turbine family[END_REF]157,150,180], Deutsche Bank [START_REF] Faust | Software Product Line Migration and Deployment[END_REF], Dialect Solutions [178], E-COM [116], Ericsson [133,[START_REF] Andersson | Development and use of dynamic product-line architectures[END_REF], Enea [START_REF] Andersson | Development and use of dynamic product-line architectures[END_REF], Eurocopter [START_REF] Dordowsky | Adopting software product line principles to manage software variants in a complex avionics system[END_REF][START_REF] Hess | Rational clearcase migration to a complex avionics project -an experience report[END_REF], Hitachi [182], LG [150], Lufthansa [START_REF] Chastek | Engineering a production method for a software product line[END_REF], MSI [169], NASA [START_REF] Ganesan | Verifying architectural design rules of the flight software product line[END_REF], NASA JPL [START_REF] Gannod | Embedded software for a space interferometry system: automated analysis of a software product line architecture[END_REF], Nortel [START_REF] Dikel | Applying software product-line architecture[END_REF], ORisk Consulting [152], Overwatch Textron Systems [START_REF] Jensen | Experiences with product line development of multi-discipline analysis software at overwatch textron systems[END_REF], Ricoh [104],

Rockwell Collins [START_REF] Faulk | Product-line requirements specification (prs): An approach and case study[END_REF], Rolls-Royce [START_REF] Habli | Challenges of establishing a software product line for an aerospace engine monitoring system[END_REF], TomTom [174], and Wikon [146]. The cases CelsiusTech, Salion [START_REF] Clements | Salion, inc.: A software product line case study[END_REF], Axis [START_REF] Bosch | Design and Use of Software Architectures: Adopting and Evolving a Product-line Approach[END_REF], Boeing, Cummins, Danfoss, and DNV Software were already contained in one of the other sources above.

Finally, we also included some cases that we know from our experience are neither contained in the book of van der Linden et al., the SPLE community's "Hall of Fame" nor the Software Engineering Institute's catalog. These were: six German SMEs (including MarketMaker from above) [START_REF] John | Qualifikation von kleinen und mittleren unternehmen (kmu) im bereich software variantenbildung[END_REF], a telecommunication system known as Terrestrial Trunked Radio (TETRA) [149], Volvo Cars and Scania [START_REF] Eklund | Architecting automotive product lines: Industrial practice[END_REF][START_REF] Gustavsson | Architecting automotive product lines: Industrial practice[END_REF], Audi [START_REF] Hardung | Reuse of software in distributed embedded automotive systems[END_REF], and Daimler [START_REF] Dziobek | Functional variants handling in simulink models[END_REF][START_REF] Bayer | Process family engineering in automotive control systems: a case study[END_REF].

Summary

While we will report the identified challenges from the literature together with our challenges in Sec. 7, we learned that the majority of publications does not report challenges that pertain specifically to variability management or that have not been resolved in the course of the respective case. Most case studies report practices or lessons learned that contributed to the success, but not challenges. Especially, all are about successful adoption, and primarily report about the perceived benefits (some also quantified) that SPLE brought. Negative experiences, shortcomings of tooling, or actual challenges for the SPLE community are largely missing. For most of the case studies and experience reports, we conjecture that these are biased, since the case study authors primarily want to show success stories instead of problems and failed attempts. As such, most of these publications primarily report on the benefits achieved as well as success factors experienced as relevant for SPLE. When reporting about the specific product line, the predominant focus is on the product-line architecture, followed by organizational and process aspects. Interestingly, some publications even have "challenges" in the title, but those challenges are usually experiences and hindrances that the organization faced before adopting SPLE or that occurred during the case and that were addressed. Furthermore, it is apparent that some challenges mentioned in previous case studies have been addressed nowadays. For instance, for Bosch [187,179] and MarketMaker [192], the publications emphasize the lack of proper, industry-strength SPLE tooling (including feature modeling) as the main challenge, which is addressed with commercial and open-source tools nowadays. Also, Chen et al. [START_REF] Chen | Variability management in software product lines: an investigation of contemporary industrial challenges[END_REF] report that variability modeling is not very user-friendly, which can be seen as a solved challenge with the commercial and opensource feature-modeling tooling that exists nowadays. Finally, we also observed that the extent and level of details in which challenges relevant for SPLE researchers are reported is not sufficient. Some provide information about adopted concepts, for instance, as Linden et al. [121] point out for their collection of cases: "Most architectures are based on a platform, supporting the requirements of present and future products. Often there are several similar products that are combined in the product line to improve the benefit of reuse. The development of a common, variable platform is often considered as the basis for introducing the product line in the organisation. Plug-in mechanisms and the definition of the right interfaces seem to be crucial." The majority of publications does not provide a finer level of details.

Variability Drivers and Variable Artifacts in our Cases

We begin the report on our results by discussing the factors driving the variability in our cases in Sec. 5.1. These variability drivers affect various types of artifacts, which we discuss in Sec. 5.2. This data has been derived from the data we collected in the case descriptions, interviews, and focus groups.

Primary Variability Drivers

For our cases, a number of different variability drivers was reported, as shown in Table 1. The most prominent drivers are markets and hardware. Being able to place products on different markets with different regulations and to ensure that the products are able to adopt to new market needs is crucial. Hardware is a relevant driver, since many of our cases concern systems, and the software needs to be able to work with a variety of different target hardware. In many cases, it is the customer who can select certain hardware, and the software needs to be able to run on the hardware configuration chosen by the customer. This is one form of end-user customization, another important variability driver.

In times of Industry 4.0 and IoT the customers demand connected and smart drives. So, the variability of features within future drive generations will be quite high.

power electronics (use case)

An increase in variability through a number of forces was also reported. One case of firmware for power electronics, for instance, sees an increased need for variability driven by the more wide-spread use of different types of multi-core processors in their products (hardware) and increased industrial digitalization (markets, operating environments). The importance of the market and its growth as the prime driver of variability is also mentioned for the automotive firmware and traffic control cases. The latter also emphasizes that innovation is an important driver for variability: the company needs to be able to deliver innovative solutions while at the same time be able to maintain the existing products in the portfolio. For our modeling platform case, the organization explained that the product needs to compete with software-as-a-service (SaaS) offers, where customization is seen as an advantage. Furthermore, IoT is a new, core driver of variability, as prominently mentioned for the cases power electronics-more precisely, Industrial IoT-and chip modeling.

Another interesting driver is simulation. In our aerospace case, a simulator resembles the real aircraft, but has more variability through the use of models at different levels of fidelity. For instance, verifying a specific sub-system might require a detailed high-fidelity model, while for real-time simulation, the model needs to be replaced with a lower-fidelity model (which might use interpolation) due to limits in computation capability.

Variable Artifact Types

Not surprisingly, the most frequently mentioned variable artifact is source code as shown in Table 1. Many companies use conditional compilation with preprocessor directives to include variability information in the source code. Custom descriptors are also relatively common, e.g., as the foundation for code generation. These are often expressed using domain-specific languages. We found little evidence for variability in tests, and requirements. Only one company explicitly reports to use components as variable assets, but we expect that there are many companies that do this implicitly.

A use case that is a bit neglected in research is variability in models used for code generation. Five of our subjects write application logic in Simulink and then generate code. Apparently, common variabilitymanagement techniques on the code level are not applicable; instead, variability modeling concepts, especially variation-point support is needed in the models and needs to be supported by the modeling tools.

Adoption of Variability Management Concepts in our Cases

We now introduce our use and tool cases by describing core characteristics and the adopted variability management concepts. An overview can be found in Table 2, which also shows the near-term adoption goals. The particular challenges faced by the organizations will requirements components 6 operating system and base libraries 1 the need for simulations substantially increases variant space, due to models representing environment or hardware at different levels of fidelity 2 functional requirements such as processing power or connectivity and non-functional requirements such as power consumption and safety as well as intellectual property (IP) cores 3 XML files as source for code generator 4 proprietary DSML instances 5 documentation, language files, resources 6 OSGi bundles 7 meta-models and UML profiles 8 software, hardware, and simulation models 9 documentation 10 design variants as proprietary DSML instances (graphs) in XML files 11 proprietary DSML instances based on the IP-XACT standard (IEEE1685), with some in-house extensions for modeling registers and memory be presented thereafter, in Sec. 7. Each case description follows a common format: context, variability drivers, variability strategy, additional capabilities (e.g., traceability or testing), and product derivation.

Power Electronics Use Case

This case concerns the production of, among others, motor controllers (drives) for electric motors. Around 300 million drives are in industrial use worldwide and used in mining, ski lifts, big industry automation processes, and turbines (e.g., solar and wind turbines). The development is characterized as agile through clone & own.

The diversity in hardware and usage scenarios is the primary driver of variability. In addition, countryspecific regulations contribute to the number of required variants. The company expects a further increase in variability through trends such as multi-core processing and increased industrial digitalization, as well as adding more software features to the drives.

So far [the variants] are managed in a clone & own manner, but that will not be possible in the future [...] going to multi-core, digitalization or adding more features [...].

power electronics (use case)

This use case primarily relies on clone & own. Yet, configuration mechanisms in individual variants also exist. The drive software is written in C/C++ and a substantial part of the code is generated from XML files with an in-house generator. The source code contains preprocessor statements controlled by configuration options. Automatic testing of individual variants is in place through a continuous integration system using Jenkins with automated, nightly tests. The connection between customer adaptations and features is tracked in a database to ensure long-term maintainability. Likewise, rationales for variability decisions are recorded.

The specific variants are built through a Pythonbased build system that allows component selection. 6 declared over assets 10 a platform of models 11 tool vendor, applies to usage of tool 13 Eclipse features; very high level 14 also unified to some extent 15 for in-component/in-module configuration 16 constraints checking 18 Systems-On-Chip specific quality attributes such as heat dissipation or power consumption

Truck Manufacturing Use Case

This case comprises the software development of a large truck manufacturer. All of its products (80,000 trucks per year) come from the same platform. Almost every product shipped has a unique configuration. The main differentiator of the brand on the global market is full product customizability.

The truck manufacturing case relies on an integrated platform to manage variability and employs separate domain and application engineering. All configuration options are realised with configurable values (i.e., without #ifdef or similar constructs). Features are maintained in a feature database that contains traceability links to the code. Assets (different levels of specifications and source code) are stored in separate databases. Consistency is checked when a system is made ready for release.

Even a simple fuel level indication system in a truck has 24,000 variants! This means that only a very small portion of all possible variants can possibly be verified by testing.

truck manufacturing (use case)

On release, all relevant information about a system, including trace links and variability information (presence conditions) are released to the product data management (PDM) system. When deriving a product from the PDM, a specialized configurator selects the relevant components and derives source code parameters to generate the source code variant that implements the desired functionality using the possible values for configuration parameters and the constraints as input.

Aerospace Use Case

This case is related to the development of an aircraft simulator for a full, configurable aircraft. Both the simulator and the aircraft software can be seen as product lines.

Variability is driven by differences in equipment and software between aircrafts. Assets are primarily developed for the aircraft product line and then propagated to the simulator product line via clone & own of the entire product line. For the simulator, multiple variants target different scenarios ranging from simple computer-screen-based simulators to realistic simulator with actual cockpit hardware. We focus on the simulator.

Before the migration [to a product-line approach] it was estimated then by the product managers that we would save 6.8 million euros in three or four years. We have saved more than that.

aerospace (use case)

The simulator is an integrated platform with features that can be mandatory, optional, or a special type of optional features that need to be deletable without a trace to protect customer interests. Each of the latter is completely modularized, meaning that the complete module can be left out and that none of these features is crosscutting. Another type of feature, so called "role change equipment," are customer-configurable features that are placeholders for future development. This means that configurations can be partial and are selected with component/module selection at checkout time. The organization maintains an informal feature model as a spreadsheet with a hierarchy, but no explicitly modeled dependencies, as well as manifest files describing components. In addition, calibration parameters refine components.

To derive a product, these parameters are set at build time. Preprocessor directives are prohibited.

Automotive Firmware Use Case

This case concerns a complex product line for electronic control units (ECUs). Around 2,000 variants are delivered per year; each deliverable is a distinct configuration. Each variant can comprise over 100 function packages and up to 2,000 functional components; the latter are updated regularly (every three months).

Variability arises from the need to customize ECUs to different vehicle types and customers. An integrated platform was defined to reduce time-to-market of variants for customers. It is combined with a limited version of clone & own, since developers can choose to create a new branch for a new feature based on guidelines that take longevity and complexity of the feature into account. Packages are developed as part of the platform (domain engineering), but variant-and customer-specific packages can exist (application engineering). The functional components can be configured via static parameters, which are feature-like entities used within variation points relying on conditional compilation (e.g., with #ifdefs).

These parameters are arranged in a relatively flat hierarchy stored in a distributed feature database and allow feature-to-code traceability. Rationales are recorded by tracing feature information to requirements. An informal feature model maps the high-level features and the static parameters. Constraints are defined over the static parameters. Interestingly, these parameters represent both variations and versions, since some of them map to pre-processor macros, and the version-control system directly supports checking out variants. As such, variability and version management are to some extent unified (like in variation control systems [122,176,[START_REF] Berger | Software evolution in time and space: Unifying version and variability management (dagstuhl seminar 19191)[END_REF]).

We are [...] outstanding in managing product variants [...] Of course there is further potential for improvement and that is what we are seeking.

automotive firmware (use case) Also, dynamic (calibration) parameters for late binding exist, some of which can be defined by the customers after delivery. These parameters, including preconditions, are stored in separate databases, upon which experts configure customer adaptations using in-house tools. Configurations are recorded in a special database.

Various analyses are run on the distributed feature database, including internal feature consistency checking. An SPLE tool is increasingly used to complement the existing cross-database consistency analyzes with rules and predicates. Single-system performance analysis is done for the products shipped to customers.

Railway Use Case

This case concerns the development of signalling systems for urban transport networks for many cities in the world. Each city has specific needs for the signalling system, which is reflected in the variants.

The specific signalling system for each city is created from a different city's variant using clone & own by creating branches in the software repository. The variants are composed by component selection, where components represent modules for different sensors and functionality. In addition, each variant also contains a set of internal configuration options based on C preprocessor directives.

Developers tend to be specialized in one variant [...], as the architecture is different [...]. They have difficulties to switch from one variant to another. Due to these difficulties, merges are done by the most experienced developers, who we would want to use on more useful tasks.

railway (use case)

The software was refactored and broadly re-architected at least in one branch. Due to development constraints (e.g., time, budget, separate responsibilities), the company never had time to integrate these branches.

Customer adaptations are tracked, since each client is represented by a specific branch.

Web Application Use Case

This case focuses on the creation of web applications that, among others, allow companies to manage media campaigns. Each client receives a customized variant.

Different variants are created with clone & own. Each application consists of frontend and backend code and most variability is in the user-facing frontend part of the system, realized using JavaScript and AngularJS. Features are recorded in a highly informal feature model. Otherwise, the company relies on the knowledge of an expert engineer who knows the distribution of features across branches. The system is implemented using different services where a final product is a composition of different services from a core library, product-specific code, and third-party services. The core services can be adapted with calibration parameters in configuration files to achieve specific behavior for each product. Component selection is performed on the service level.

This architecture allows customer adaptations through the services and provides limited traceability between features and code, since features are mapped to services.

The most common adaptation in variants are UI customizations. Since the web UI is not standard, users usually want a special interface.

web application (use case)

Modeling Platform Use Case

This case concerns a commercial model-driven engineering tool for business architects, system architects, and developers. The tool is component-based and either comes pre-packaged for one of these target audiences or can be assembled based on customer wishes. It is also possible that specific features are developed for a certain customer. Additionally, different license schemes can be employed (e.g., commercial and open source versions), and six different operating systems versions are supported, further driving variability. There are twelve solutions, with two major releases per year, that include more than 50 modules and three meta-models. More than 30 modules with variations can be obtained from a store to fit specific needs. The store also contains scripts and model components for different versions of the platform.

To this end, an integrated software platform based on Eclipse plug-ins is used. All provided solutions share a common set of modules. Because features are localized in specific plug-ins, feature-to-code traceability is available. Currently, the variability in terms of packaging is not managed in a tool-supported way.

Variants are composed by component selection, relying on the Eclipse P2 packaging system, which considers constraints between features and plugins, and uses a limited feature model. The configurator defines which modules are part of a package and has limited support to set configuration options for the individual plug-ins.

No particular tool is used to manage the variability in terms of packaging. Technically the variability is ensured by the modular design.

modeling platform (use case)

Imaging Technology Use Case

This case comprises camera software that is packaged for individual customers based on customer requirements. This packaging includes configuring sensor parameters, prototyping and testing different sensor configurations, integrating onto the hardware for sensor validation and exporting sensor configuration parameters for use in a production software system.

An integrated platform along with a home-grown configurator tool is used to create these packages. Assets are currently managed via repositories (e.g., Git) and file-based storage. To create a product, module selection is used to package the correct firmware and driver software. In addition, calibration parameters are used to define the correct sensor and software configuration.

The key trend [we want] to follow [...] is mainly applying "continuous integration" to software infrastructure that supports the development and validation of hardware.

imaging technology (use case)

Traffic Control Use Case

This case concerns the development of several generations of integrated road traffic control and surveillance solutions, including custom sensors along with embedded software. Backend software is produced to process the data sent by these sensors. Different variants of the product are created for different customers and different regional markets, in particular if certification is necessary.

The challenges of the market are mostly technological. A relatively small group of players constantly improves the performance and versatility of the product. Regional presence is important in the market.

traffic control (use case)

Ad hoc reuse via clone & own is present, relying on branching in the version-control system, which the company considers bad practice and strives for an integrated platform. If a software needs certification it can become a permanent branch in the software repository. Calibration parameters are used to configure the software for specific sensors. Some of these are considered features exposed and sold to customers. Component selection happens at compile time where source code modules are statically linked or at runtime via dynamic linking.

Requirements Engineering Tool Case

This case is our first tool case, concerning a tool for improving the quality of requirements. It allows defining an ontology of the application domain. Based on the ontology, clients can write semi-natural language requirements. The company also develops pattern-based extractors to populate the ontology from semi-structured documents (conceptually similar to techniques that offer languages for defining patterns that can be used to extract requirements from semi-structured documents [154]). The clients of requirements engineering produce safetycritical software-intensive systems mostly in the transportation and defense industries.

One challenge faced by [our tool customers] is to take advantage of all the knowledge developed among different projects, produced in many times with no management in a disorganized way, and to reuse the existing requirements in new variations of the same systems.

requirements engineering (tool case) Currently, the tool customers apply clone & own of the requirements specification. The tool does not offer variability features out-of-the-box. However, one stated use-case is the extraction of variability informationincluding a vocabulary and ontology of variability and assets -from requirements about an existing productline platform. In other words, the requirements describe variation points and variants using dedicated terminology to be extracted by pattern-based extractors.

Hardware Modeling Tool Case

This case concerns the development of a tool that allows to assemble system models (e.g., about automotive suspensions) from pre-defined building blocks and simulate them. The tool interfaces with other tools such as the company's own product-lifecycle management tool. The variability in the considered models concerns the blocks that vary, e.g., among automotive suspensions.

The current tool uses an ad hoc representation of variability only visible in terms of component selection inside the architecture models. The modeling language allows realizing variability encoding [155], that is, using built-in conditionals relying on configuration options, which enables this component selection. The tool is already able to explore the design space by creating all possible architecture models that are supported by the selected components. Nevertheless, the current tool chain does not support product line concepts explicitly. In particular, the concept of feature is not supported and variability is only managed within the design space without any knowledge about the problem space.

One challenge faced by [our tool customers] is to design and compare a significant number of product [...] design variants.

hardware modeling (tool case)

Chip Modeling Tool Case

This case concerns a tool for designing Systems-on-Chips (SoC) and the reusable hardware component designs from which these SoCs are assembled. These designs are passed to machines that produce the corresponding integrated circuits. The company believes that emerging applications with huge growth potential, such as SoC for IoT will lead to a combinatorial explosion of variants, with features being related via complex trade-off constraints.

The variants differ by: their provided functions, execution performance of functions, packaging of SoC inside a circuit, power consumption, safety level (typically determined by standards), and life-cycle durations.

The modeling tool is based on the IP-XACT standard (IEEE 1685) and extensions for modeling registers and memory. The integrated software platform currently does not support any specific variability management facilities, nor is it seamlessly integrated with an SPLE tool. Consequently, the tool users are currently restricted to clone & own, specifically, using branching and merging facilities of the tool's built-in version-control facilities. Also, there is no centralized variability representation.

[There are] more and more applications with specificsfor example, the emerging IoT applications that require smaller platforms, with low or medium processing, and very hard constraints on reliability and power consumption.

chip modeling (tool case)

Variability Management Challenges

We now synthesize and discuss the challenges faced among our cases. We observed that our cases cover a wide range of maturity levels with respect to the adoption of variability management concepts. Consequently, we structure the challenges according to these maturity levels, providing the context in which the occurs and in which it should be addressed by researchers or tool builders.

We first present two general challenges that affect any maturity level. We then discuss those related to support for clone management, which are encountered in organizations that use clone & own as their main technique to derive new products. We then identify challenges that occur when migrating to an integrated platform, that is, when clone & own starts to be complemented by feature and asset management. Next, we discuss challenges that exist once migration is more or less complete and working with an integrated platform becomes the focus of work, followed by challenges that appear when the product line is modernized and evolved.

General Challenges

Challenge 1, Model-Driven Engineering. A common challenge we observed for any maturity level is modeldriven engineering (MDE) and code generation. While source code is still the most frequently mentioned variable artifact (cf. Sec. 5.2), we observed that often code is generated from domain-specific languages (DSLs). While the relation between DSLs and SPLE has been studied [193], the DSLs used in our cases do not support SPLE concepts. As seen in our power electronics, aerospace, and truck manufacturing cases, embedded systems organizations often rely on MDE using Simulink with code generation. Such a setup challenges not only handling cloned variants (e.g., when trying to trace features), but also adopting, working with, and evolving an integrated platform [103]. This observation is substantiated by our three tool cases striving to integrate variability mechanisms into the modeling tools.

Furthermore, the experience reports on Danfoss [START_REF] Fogdal | Ten years of product line engineering at danfoss: lessons learned and way ahead[END_REF], General Motors [START_REF] Flores | Mega-Scale Product Line Engineering at General Motors[END_REF], and CCT [START_REF] Clements | Control channel toolkit: a software product line case study[END_REF] also mention this challenge. They request a better integration of SPLE concepts, specifically variability mechanisms, with modeling tools. Notably, the experience report on Daimler [START_REF] Dziobek | Functional variants handling in simulink models[END_REF] focuses on handling variability in Simulink models, specifically, describes how to represent variation points in models. Nokia Networks [121] laments missing support for reusing systems engineering assets, in line with the experienced needs of customers of our our two tool cases chip modeling (Sec. 6.12) and hardware modeling (Sec. 6.11).

Challenge 2, Tool Integration. These elaborations from our cases and the literature illustrate a more general problem: tool integration. Since variability is a crosscutting concern, variability-related tooling usually needs to be integrated with other engineering tools as named, e.g., by our automotive firmware and aerospace cases that exhibit overall high maturity. This tool integration challenge was also expressed in the literature for Danfoss [START_REF] Fogdal | Ten years of product line engineering at danfoss: lessons learned and way ahead[END_REF], General Motors [START_REF] Flores | Mega-Scale Product Line Engineering at General Motors[END_REF], Siemens Medical Solutions [121], Fiscan [115], Argon [START_REF] Bergey | Software product lines: Experiences from the sixth dod software product line workshop[END_REF], and CCT [START_REF] Clements | Control channel toolkit: a software product line case study[END_REF], who not only request integrated tool chains, but more mature variability-related tooling in general. General Motors [START_REF] Flores | Mega-Scale Product Line Engineering at General Motors[END_REF] suggests to integrate SPLE concepts with product lifecycle management (PLM) concepts. Argon [START_REF] Bergey | Software product lines: Experiences from the sixth dod software product line workshop[END_REF] requests the integration of variability in version-control systems. In summary, this challenge is further supported by a recent study on product-line analyzes [138], which found that adopting such is, among others, a tool-integration problem.

Clone Management

As seen in Table 2, seven of our use cases exercise clone & own for managing variants-while for two of our tool cases, the customers' variant management also relies on clone & own. The other tool vendor's modeling tool (cf. Sec. 6.11) offers variability encoding [155] using built-in conditionals relying on configuration options. While clone & own is the most common strategy for engineering variants, we observed substantial awareness among our subjects of the problems connected to it.

Challenge 3, Visualize and Track Variability.

A need expressed by all cases is keeping an overview understanding of cloned variants, since understanding the content and purpose of individual variants is challenging without more abstract representations of the codebases.

Feature-Orientation. Adding the notion of features and feature locations to clone & own would enhance the practice. When features are present, then stakeholders can know what is in the branches, as opposed to trying to understand the difference between variants through assets differences, which is challenging. In addition to recording features, recording their location (e.g., through lightweight code annotations [START_REF] Ji | Maintaining Feature Traceability with Embedded Annotations[END_REF][START_REF] Andam | FLOrIDA: Feature LOcatIon DAshboard for Extracting and Visualizing Feature Traces[END_REF][START_REF] Abukwaik | Semiautomated feature traceability with embedded annotations[END_REF][START_REF] Entekhabi | Visualization of feature locations with the tool featuredashboard[END_REF]108]) also helps with maintenance. As such, this challenge is about keeping clone & own, but giving developers more control and overview understanding, which will help with maintenance (e.g., developers know the, potentially scattered [144,145], locations of a feature when cloning it to another variant).

In addition, better visualization and support for code propagation, ideally based on features, is needed. Especially for the power electronics case, better support for localization and propagation of features across cloned variants is requested. In summary, introducing some notion of feature-orientation to facilitate a more abstract understanding (abstracting over code-level structures) of variants, is an expressed need for our cases. The existing case studies on Axis and Securitas [START_REF] Bosch | Evolution and composition of reusable assets in product-line architectures: A case study[END_REF][START_REF] Bosch | Product-line architectures in industry: A case study[END_REF] emphasize the lack of architectural abstractions (features) in programming languages, as well as the exploratory study of Chen et al. [START_REF] Chen | Variability management in software product lines: an investigation of contemporary industrial challenges[END_REF] confirming this challenge.

Record and Analyze Variability Decisions. Several of our cases stated that they need a way to record and later analyze variability decisions (railway, imaging technology, and traffic control case), such as rationales for introducing variability. It was also stated that recording information about refactorings would be helpful (preferably using embedded annotations [START_REF] Ji | Maintaining Feature Traceability with Embedded Annotations[END_REF][START_REF] Andam | FLOrIDA: Feature LOcatIon DAshboard for Extracting and Visualizing Feature Traces[END_REF][START_REF] Abukwaik | Semiautomated feature traceability with embedded annotations[END_REF]168,109,110]). The recording should explicitly relate refactorings to either the introduction of new functionality (features) or to maintenance. This information otherwise needs to be recovered when later propagating code across variants or migrating towards a platform (e.g., when assessing architectural mismatches). Specifically, for the railway and traffic control case, there is a need to reconstruct the architectural and functional evolution of the product line. While architectural evolution is mainly represented by typical refactorings (for the railway case, eight typical refactorings from Fowler [START_REF] Fowler | Refactoring: Improving the Design of Existing Code[END_REF] are mentioned), functional evolution (i.e., the implementation of new functionality, bug fixes, and so on), needs to be distinguished.

Challenge 4, Cloning in Combination with Variability.

An interesting observation is that none of our subjects exercises pure clone & own, but that variants already use variation points. This challenge is also mentioned in existing experience reports about Danfoss [START_REF] Fogdal | Ten years of product line engineering at danfoss: lessons learned and way ahead[END_REF] [START_REF] Fogdal | Ten years of product line engineering at danfoss: lessons learned and way ahead[END_REF] for Danfoss, that using variability mechanisms also avoids merge conflicts during clone & own and allows more isolated feature development.

Limitations of Clone-Management Techniques. This challenge complicates using clone-management frameworks proposed in the literature [148,161,159,160,[START_REF] Antkiewicz | Flexible product line engineering with a virtual platform[END_REF] and techniques for integrating variants [START_REF] Fischer | Enhancing clone-and-own with systematic reuse for developing software variants[END_REF]126], since existing variability needs to be taken into account. In fact, as explained for the railway case (cf. Sec. 6.5), the integration is done by experts who should rather realize new functionality instead of recovering information about variability in cloned variants and re-engineering code. Furthermore, most of our use cases applying clone & own also have an integrated platform, that is, a project with common assets. We are only aware of one work in this direction [118], which should further be complemented with methodologies and tools.

Clone Management of Whole Product Lines. Approaches to enable product lines of product lines have been investigated in the literature [100, 158, 105]. However, what our cases require is clone management at the product-line level. A previous experience report on Philips Medical Systems [121] also emphasizes the reuse of components across product lines. Our aerospace case exercises clone & own for two highly complex product lines. The primary product line controls a real aircraft, and the cloned product line the simulator. The case strives to improve the-currently manual and laborious-synchronization between aircraft and the simulator product line. Specifically, it requests guidance for creating variability models (e.g., whether one or separate, but largely redundant models should be created; how they should be decomposed to reflect the architecture) and modeling constraints. This should help defining an aircraft or a simulator configuration, including matching an aircraft configuration to a suitable simulator configuration for a particular test activity.

Migration to an Integrated Platform

Our three use cases that want to establish an integrated platform, as well as the customers of our two tool cases that aim to establish a platform, practice clone & own. Recall the core motivation for our railway case to free experienced developers from performing migrations (cf. Sec. 6.5). As such, migrations should be semi-automated, supporting less experienced developers or enabling domain experts performing it. We observed the following challenges.

Challenge 5, Platform Migration Process and Tools. For two of our use cases (power electronics, railway), the need for a dedicated migration process was expressed. Such a process should be lightweight and should guide engineers through the whole migration. Almost all of our use cases expressed the need for commonality and variability analysis. Furthermore, such a process should guide through the identification and location of features (potentially integrated with manual featurelocation techniques [111]) or through creating the target architecture (potentially supported by feature-model composition techniques [START_REF] Acher | Composing feature models[END_REF]). According to our railway and aerospace cases, such a process should also support engineers in creating variation points with an appropriate variability mechanism (cf. Sec. 2), in other words, prescribing the introduction of variation points. This challenge is also mentioned by previous experience reports on Bosch [187], MSI [169], and Siemens Healthcare [START_REF] Bartholdt | Re-engineering of a hierarchical product line[END_REF], both requesting a process for incremental migration, the latter even during running projects, without disrupting the development.

Diffing of Cloned Variants. We observed the need for higher-level diffing support for cloned variants. Specifically, as pointed out for our railway case, there should be means to express historical additions, suppressions, and modifications at the highest level of abstraction, specifically distinguishing architectural and functional evolutions. As such, practically usable, dedicated diffing techniques, which support existing variation points (e.g., #ifdef), are needed, potentially building upon recent techniques such as intention-based clone integration [118]. Enhanced visualization capabilities should also support highly scattered features [145,144], as explained for the web application and power electronics case. Furthermore, the visualization should be guided by extracting the structure of the underlying configuration management base (including branching and revision structures), as pointed out for our railway case.

Notably, this challenge is also expressed for Ricoh [104], and the exploratory study of Chen et al. [START_REF] Chen | Variability management in software product lines: an investigation of contemporary industrial challenges[END_REF] reports that clone detection techniques are not applicable to software variants.

From a tool-vendor perspective, expressed for the chip modeling case, the organization strives to provide automated commonality and variability analyses to customers of the tool, so that existing cloned models can be migrated. As such, the challenge is to adopt automated analyses conceived by researchers, ideally adhering to standards, which are almost non-existent for variability management.

Asset Integration at Code and Model Level. Integrating assets into a platform is challenging. Such an integration differs from integrating assets during clone & own in the sense that platform integration needs to consider many more variants-those derived from the platform-which might make it more challenging. Understanding and characterizing both kinds of integration is subject to future work. For our cases, better mergerefactoring techniques taking existing variability (e.g., #ifdef in the cloned variants) into account are necessary. For instance, our railway case needs such techniques for automatically proposing integration strategies for features from variants, that is, focusing on identifying functional (feature-based) evolution from the branching history, ignoring refactorings and other non-functional evolutions. Finally, asset integration is challenged (Challenge 1) when MDE techniques or code generation are used (e.g., in the power electronics case). This requires focusing on the migration (i.e., integration) of models, while not ignoring customized code.

Training, Certification, and Budgeting. Other issues to be addressed by a process are to: (i) establish a common understanding among the stakeholders about product-line engineering concepts (requirements engineering case), calling for training support in a migration process, to (ii) establish certification support, as requested by an experience report on Rolls-Royce [START_REF] Habli | Challenges of establishing a software product line for an aerospace engine monitoring system[END_REF], and (iii) as expressed for our relatively small web application case, there is no dedicated budget to develop the platform, so assisting in budgeting is a challenge a platform migration process should support.

Definition of a Target Architecture. A core issue when adopting an integrated platform is architectural degradation, and therefore architectural mismatches among the cloned variants, which need to be resolved. Architectures of the cloned variants need to be compared and a target architecture [173, 3] defined, which is expressed as a core challenge (e.g., for our railway case). Specifically, expressed for the imaging technology case, the target architecture should be layered and modularized, to be maintainable. The goal is to enable fully automated variant derivation without programming effort (i.e., developing adaptations). Another challenge expressed is to create documentation (or generate such) about the architecture itself (chip modeling case), and about using such an architecture for variant derivation in parallel.

In our literature survey we found that most of the case studies focus on describing the architecture of the resulting product line. For instance, for RPG Games [195], the authors discuss the architecture devel-opment and necessary adaptations of cloned variants (e.g., changes of variable types and refactorings) to obtain a common product-line architecture. Recently, Kr üger et al. [START_REF] Debbiche | Migrating the java-based apo-games into a composition-based software product line[END_REF][START_REF] Akesson | Migrating the android apo-games into an annotation-based software product line[END_REF] provide datasets and experiences migrating cloned Java and Android game variants to such a common architecture.

Challenge 6, Migration Decision Support. A migration process should also provide decision support about the migration itself, based on the expected costs and benefits [START_REF] Ali | A comparative survey of economic models for software product lines[END_REF]112]. Specifically, for our power electronics case, the suggestion was made to enhance feature identification and location with indicators about the cost of the extraction and re-engineering of relevant assets for making them reusable or integrating into a platform.

Cost/Benefit Estimation. The need for effective cost/ benefit estimation is further supported by existing experience reports on Deutsche Bank [START_REF] Faust | Software Product Line Migration and Deployment[END_REF], on Philips Medical Systems [121], on Hitachi [182], on Ericsson [133, 10], on Axis and Ericsson [181], as well as on Axis and Securitas [START_REF] Bosch | Evolution and composition of reusable assets in product-line architectures: A case study[END_REF][START_REF] Bosch | Product-line architectures in industry: A case study[END_REF]. Specifically, for Philips Medical Systems [121], clone & own sometimes appeared to be beneficial (close to the break-even point), which led to tensions to develop outside the platform. Axis and Ericsson [181] also need cost estimation for decision making. They explain it for the case that when a new product is added, sometimes rewriting the components instead of adapting them is easier, depending on the extent of the changes. The challenge is also described in other case studies on Axis and Securitas [START_REF] Bosch | Evolution and composition of reusable assets in product-line architectures: A case study[END_REF][START_REF] Bosch | Product-line architectures in industry: A case study[END_REF], among others, demanding decision making support for: "Deciding to include or exclude a product in the product-line [...] Guidelines or methods for making more objective decisions would be valuable to technical managers."

Measurement. The pre-requisite for cost/benefit estimation are effective techniques to measure the costs and benefits of reuse. Previous experience reports on Bosch [183], Testo [164], Argon [START_REF] Bergey | Software product lines: Experiences from the sixth dod software product line workshop[END_REF], Overwatch Textron Systems [START_REF] Jensen | Experiences with product line development of multi-discipline analysis software at overwatch textron systems[END_REF], Wikon [146], and CCT [START_REF] Clements | Control channel toolkit: a software product line case study[END_REF] emphasize such a measurement technique as an important challenge. Generally, in a case study on Axis [START_REF] Bosch | Evolution and composition of reusable assets in product-line architectures: A case study[END_REF], the lack of economic models is lamented.

Challenge 7, Continuous Integration. Six of our cases (web application, traffic control, aerospace, automotive firmware, railway) explicitly point out the need for supporting continuous integration. Most of them have sets of automated unit tests (grown incrementally, typically with the discovery of bugs), UI tests, and integration tests. We found typical tools used for continuous integration, such as Jenkins, Maven, BuildBot, Git, and Jira. While some approaches to automate, e.g., interaction testing in continuous integration exists [START_REF] Johansen | A technique for agile and automatic interaction testing for product lines[END_REF], the larger challenge is to obtain a feature-oriented and configurable architecture that supports continuous integration, ideally with the tools mentioned.

This challenge is implicitly expressed in the cases for Philips Consumer Electronics Software for Televisions [121], for Overwatch Textron Systems [START_REF] Jensen | Experiences with product line development of multi-discipline analysis software at overwatch textron systems[END_REF], and for CelsiusTech [START_REF] Bass | Software architecture in practice[END_REF][START_REF] Brownsword | A case study in successful product line development[END_REF], where balancing between domain and application engineering is a core challenge, striving to bringing both closer together. In addition, CCT [START_REF] Clements | Control channel toolkit: a software product line case study[END_REF] demands a process for integration testing.

Working with an Integrated Platform

We observed the following challenges expressed for our use cases and tool cases when dealing with an integrated platform.

Challenge 8, Representation of Variability. A need expressed by all cases is a centralized (and ideally unified [START_REF] Berger | Software evolution in time and space: Unifying version and variability management (dagstuhl seminar 19191)[END_REF]) representation of variability. For instance, for the aerospace and automotive firmware cases, an overall view providing a unified description of the variability was demanded, ideally in the form of a feature model declared in an established SPLE tool. Specifically, in one of these two cases, the variability information is scattered across different representations, including a distributed feature database and configuration files, preventing a coherent view of all relevant product line information. Also recall that many different types of artifacts (e.g., requirements, architecture, design models, source code) are used in the cases (cf. Sec. 5.2), and that each artifact type has its own technical representation of variation points, which challenges obtaining a global view of variability. This challenge is closely related to Challenges 1 and 2, requiring variability support in modeling languages, tool integrations, and traceability support.

Missing Standards. Standardisation efforts in the direction to provide unified representations for variability and variation points exist, such as CVL [START_REF] Haugen | CVL: common variability language[END_REF] and VEL [165]. However, CVL was never adopted as a standard and VEL is explicitly aimed at providing an exchange language between tools rather than a fully integrated variability representation to be edited directly. Recently, a new initiative3 was launched, aiming again at establishing a common feature modeling language, supporting common, community-agreed usage scenarios for such a language [START_REF] Berger | Usage scenarios for a common feature modeling language[END_REF].

Representation of Behavioral Variability. Furthermore, the cases from the literature CCT [START_REF] Clements | Control channel toolkit: a software product line case study[END_REF], ENEA [START_REF] Andersson | Development and use of dynamic product-line architectures[END_REF], and the study of Chen et al. [START_REF] Chen | Variability management in software product lines: an investigation of contemporary industrial challenges[END_REF] request the support for quality properties when representing variability. Specifically, "structural variability is well supported, behavioral and timing aspects are not" [START_REF] Chen | Variability management in software product lines: an investigation of contemporary industrial challenges[END_REF].

Representation of Topological Variability. We observed the need for representing topological variability in the five cases that use design models (hardware modeling, modeling platform, chip modeling, aerospace, and power electronics). These are often XMLbased domain-specific modeling languages (DSMLs) using graphs as their underlying structure. As opposed to feature-oriented (switch on/off features) variability, topological variability requires dedicated modeling languages. Typically, organizations create their own domain-specific languages (DSLs) for this reason, given limitations of established variability modeling languages [START_REF] Berger | To connect or not to connect: Experiences from modeling topological variability[END_REF][START_REF] Fantechi | Topologically configurable systems as product families[END_REF][START_REF] Behjati | Architecture-level configuration of large-scale embedded software systems[END_REF]. A challenge is that topological variability can hardly be expressed using variability annotations and preprocessors, but typically requires more flexible code generators.

Representation of User-Interface Variability. An interesting was explained for the web application case, where the prime driver of variability is the customization of user interfaces. The current web frameworks do not offer any facilities for customization, so the company needs to rely on clone & own. UI variability has several peculiarities compared to traditional source code, as UIs encapsulate human computer interaction (HCI) assets (e.g., dialog models, context models, and aesthetic concerns). This challenge is related to Challenge 1 (MDE and code generation), given that abstract HCI models are sometimes used before the generation of concrete HCI implementations in target languages [125]. Also note that web applications as cases are rather rare among the existing literature. Exceptions are HomeAway [107] and MarketMaker [192]; however, none of the publications describes the realization of variability in the web-based user interfaces of these web applications.

Challenge 9, Feature Modeling. In the light of obtaining a unified representation of variability, almost all of our subjects (9 of 12) strive to adopt feature modeling. For the power electronics case, a feature model would aim at visualizing the variant space and keeping the variants manageable. For the automotive firmware case, a feature model would be the basis for a configurator with intelligent configuration facilities, which should support product derivation using defaults, choice propagation, and conflict resolution-easing the configuration (and avoiding having to decide all features) Likewise, for the modeling platform case, an improved product derivation through an intelligent configurator tool is needed. In the imaging technology case, a more intuitive representation of configuration knowledge is needed, where even configuration profiles (partial configurations) or a feature hierarchy is seen as beneficial. In fact, as explained for the automotive firmware case, the feature hierarchy is considered the most valuable information. Furthermore, the experience report by Audi [START_REF] Hardung | Reuse of software in distributed embedded automotive systems[END_REF] also requests modeling guidelines. A first step in this direction are probably the feature modeling principles of Nesic et al. [141].

Feature-Modeling Process. As such, a feature modeling process is needed, that supports obtaining a feature model from a diverse set of variability information sources [START_REF] Bécan | Breathing ontological knowledge into feature model synthesis: an empirical study[END_REF][START_REF] Davril | Feature model extraction from large collections of informal product descriptions[END_REF]110], integrated with feature-model management, merging, and synthesis techniques [START_REF] Acher | Composing feature models[END_REF][START_REF] Acher | FAMILIAR: A domain-specific language for large scale management of feature models[END_REF][START_REF] Acher | Support for reverse engineering and maintaining feature models[END_REF]171,172]. Our cases express that functional evolution is well reflected in commit messages (e.g., bug fix information, new feature implementations, new branches), which confirms some insights that commit messages are a good source for feature identification [110,109,196].

Variation-Point Methodology. Another issue pertaining to such a process is a variation-point methodology (expressed for the aerospace case). It should guide, upon the decision to introduce a feature (originating from some scoping process), the flow of variation points into requirements, code, models, and other affected artifact types.

Feature-Oriented Authorization. For the chip modeling, aerospace, and requirements engineering case, the need for user access control and authorization mechanisms was explicitly mentioned. Users have different access rights for different parts (features) of the product line. Especially the aerospace case needs to control visibility of variation points, where certain variation points or variants even need to be completely invisible (i.e., no trace should be visible that a variation point exists) for unauthorized personnel. Unfortunately, beyond work by Faegri et al. [START_REF] Faegri | A software product line reference architecture for security[END_REF], this challenge has not yet been recognized or received attention in the research community.

Feature-Oriented Views and Synchronization. For nine of our twelve cases, the need for better visualizations, especially establishing feature-oriented views [START_REF] Acher | Slicing feature models[END_REF][START_REF] Andam | FLOrIDA: Feature LOcatIon DAshboard for Extracting and Visualizing Feature Traces[END_REF]135,136,143,127], was expressed. In one of our cases it is planned to reverse-engineer feature constraints from the codebase and distributed feature databases, including identifying the model hierarchy [139,140], and to synthesize a feature model. This challenge is also reported by the exploratory study of Chen et al. [START_REF] Chen | Variability management in software product lines: an investigation of contemporary industrial challenges[END_REF], and by the survey of Berger et al. [START_REF] Berger | A survey of variability modeling in industrial practice[END_REF].

An additional issue arising (related to Challenge 11 below) is that materialized views need to be continuously synchronized with the information sources, including feature databases, which cannot be abandoned. In other words, feature models as views degrade and need to be re-synthesized or synchronized continuously, as expressed for the automotive firmware case.

Challenge 10, Quality Assurance. Our cases expressed various challenges with respect to assuring the quality of their variants or platforms. While most responses were rather vague (e.g., the power electronics case: combine testing and SPLE to reduce efforts), since the cases strive to adopt more systematic variability management in the first place, they expressed challenges with respect to assuring the consistency of assets, more effective validation techniques, and the verification of platforms or many variants. Also note that dedicated studies on product-line analyses in industrial practice exist [138].

Consistency Analyses. Recall that many cases strive to adopt feature models or have at least an informal, but structured feature model (cf. Table 2). When asked about analyses, they also confirmed that they would like to use feature-model analyses to assure their consistency, especially in relation to the other assets (including code and requirements with functional or quality properties), and under the continuous increase of concurrently maintained variants.

A further requested consistency-related analysis is dead-code analysis (automotive firmware case), and our requirements engineering case suggests a metricsbased approach [START_REF] El-Sharkawy | Metrics for analyzing variability and its implementation in software product lines: A systematic literature review[END_REF][START_REF] Berger | Towards system analysis with variability model metrics[END_REF] for completeness, correctness, and consistency of requirements.

Validation and Verification.

The following two cases explicitly expressed the need for assuring properties for all possible variants (a.k.a., variability-aware analysis [130,117]).

The truck manufacturing case requests safety verification for the widest possible number of variants with a scalable compositional verification approach. Such a compositional verification should verify each lowestlevel component individually, then reuse the results for verifying component combinations, showing that functional or quality properties are fulfilled for large sets of configurations that cannot be tested exhaustively. To this end, source code needs to be annotated with formal specifications, which in turn needs a clear picture of the product lines, especially configuration constraints. In the truck manufacturing case, this requires consolidating information scattered across various databases before.

For our chip modeling case, a challenge is to extend the modeling tool with verification capabilities for complex properties covering, for instance, market data to physical constraints concerning heat dissipation or power consumption. Then all variants should comply with the quality attribute specified for the platform (which is a model).

Given the static nature of variability-aware analyses, we can see future research direction in effectively combining validation and verification techniques, making both variability-aware for product lines.

Product Line Evolution and Modernization

Once an integrated platform has been established and a certain maturity has been achieved in working with it, the focus shifts towards its modernization and evolution. Our modeling platform case constantly aims at a more systematic management of existing variability. The automotive firmware case, on the other hand, seeks to eliminate redundancy in an already very advanced product line. In the automotive firmware, hardware modeling, and modeling platform case, the developers strive to constantly identify new and changed feature constraints to enhance configuration processes and, therefore, improve the platform's maturity.

In general, better support for evolving software product lines is already requested in the literature, specifically in the studies of Chen et al. [START_REF] Chen | Variability management in software product lines: a systematic review[END_REF][START_REF] Chen | A systematic review of evaluation of variability management approaches in software product lines[END_REF][START_REF] Chen | Variability management in software product lines: an investigation of contemporary industrial challenges[END_REF] (especially evolution of variability models, including dependency management), the survey Berger et al. [START_REF] Berger | A survey of variability modeling in industrial practice[END_REF] (model evolution), as well as for the cases Danfoss [START_REF] Fogdal | Ten years of product line engineering at danfoss: lessons learned and way ahead[END_REF] (evolution of architectures and interfaces, and evolving towards a software ecosystem, explained shortly), Axis and Securitas [START_REF] Bosch | Evolution and composition of reusable assets in product-line architectures: A case study[END_REF][START_REF] Bosch | Product-line architectures in industry: A case study[END_REF] (information distribution, asset version management, asset dependency management, and reuse of assets in different contexts), Nokia Networks [121] (backwards compatibility of assets, and quick changes in technology that need to be incorporated into the product line), Nokia Mobile Phones [121] (continuous architecture conformance checking), and Volvo Cars and Scania [START_REF] Eklund | Architecting automotive product lines: Industrial practice[END_REF][START_REF] Gustavsson | Architecting automotive product lines: Industrial practice[END_REF]. The latter even explain: "We have not found any literature describing how product lines are maintained; most of those we found described the transition to a software product line and those challenges." Valuable guidelines are provided by Svahnberg et al. [181], but thorough support is still needed. Our cases face the following challenges.

Challenge 11, Artifact Synchronization. The synchronization of different platform artifacts, including code and feature model, was emphasized for half of our cases (railway, aerospace, modeling platform, web application, automotive firmware, and chip modeling). A question that arises is how to keep the feature model in sync, for instance, whether incremental updates or a frequent re-synthesis should be done. Co-evolution scenarios in SPLE have been studied [START_REF] Hellebrand | Reverse engineering challenges of the feedback scenario in co-evolving product lines[END_REF]166], but have not yet been adopted in industry.

The synchronization challenge is, not surprisingly, further complicated by Challenge 1 (MDE), when models constitute the primary variable artifacts. A specific challenge expressed for the chip modeling and modeling platform case is co-evolution of the variability-aware language of the model and the product line (i.e., the feature model).

Finally, for our aerospace case, the organization expressed the need for a better coordination among teams developing new features on branches (i.e., project management and integration management).

Challenge 12, End-User and After-Market Traceability.

A challenge explicitly mentioned for two of the cases (aerospace, automotive firmware) is the need to trace customer adaptations on the after-market, as well as the general use of software assets. The experience reports by Danfoss [START_REF] Fogdal | Ten years of product line engineering at danfoss: lessons learned and way ahead[END_REF], Nokia Networks [121], Naval Underwater Warfare Center [START_REF] Cohen | Successful product line development and sustainment: A dod case study[END_REF][START_REF] Cohen | Acquisition pilot: Product line acquisition and measurement at nuwc[END_REF][START_REF] Cohen | Case study: A measurement program for product lines[END_REF], and PHILIPS Medical Systems [121], as well as the survey of Berger et al. [START_REF] Berger | A survey of variability modeling in industrial practice[END_REF], also mention this challenge. The former explains: "Frequency converters can collect a lot of data. [...] This information can be used for predictive maintenance purposes, not only for the frequency converter but also for the machine." [START_REF] Fogdal | Ten years of product line engineering at danfoss: lessons learned and way ahead[END_REF] Our cases explain that it needs to be clear which specific configurations the customers use in order to offer dedicated support and keep the software up-todate. As such, organizations have a need to keep track of product variations that have been delivered to customers in their own portfolio. Of course, the question arises how to efficiently store and efficiently keep such customer configuration databases updated.

Addressing this challenge would require continuous measurement, perhaps building upon monitoring support [153], which needs to be extended and integrated with variability management concepts.

Challenge 13, Dynamic Product-Line Platform. An improvement goal expressed for our aerospace case is to conceive its next-generation product-line architecture, which specifically aims at adopting dynamic variability. This challenge is also pointed out in existing experience reports on Enea [START_REF] Andersson | Development and use of dynamic product-line architectures[END_REF] and on Danfoss [START_REF] Fogdal | Ten years of product line engineering at danfoss: lessons learned and way ahead[END_REF].

The challenge is to adopt more modern variability mechanisms (supporting late dynamic binding) to support reconfiguration in the aircraft simulator environment. In fact, the organization expresses the need to adopt an ecosystem platform and strategy (to include contributions from suppliers as well as COTS components) with respective variability mechanisms [START_REF] Berger | Variability Mechanisms in Software Ecosystems[END_REF]167]. As such, the challenge is not only to migrate to an architecture supporting dynamic variability, but also foster inter-organizational reuse by allowing to extend variation points dynamically with third-party contributions.

Threats to Validity

We now consider threats on the construct, internal, and external validity of our multiple-case study, as well as its reliability. We also discuss our mitigation strategies using the categorization by Yin [194] as a guideline. Construct Validity. The initial data used in this paper is taken from semi-structured use case descriptions. Even though, the companies involved in the study refined these descriptions several times to harmonize them, these descriptions still show diversity in terms of level of detail and relevance. We addressed this threat by following up with the companies in semi-structured interviews and focus groups. This gives us high confidence in the data we collected and used for analysis and also helped us avoid any bias by triangulating the different data sources. To further avoid bias, we continuously discussed the material and ensured that each piece of information was interpreted by at least two of the authors. The additional member checking of the analyzed results confirms the correctness.

Not all subjects that participated in this study were familiar with the established terminology in the SPLE literature. That meant that we needed to map the terminology used in the use case description to the terminology used in this paper. To avoid any bias in this regard and limit the freedom of interpretation of the researchers, critical concepts were member checked before interacting with the companies. Internal Validity. To ensure that we could establish valid cause and effect relationships, we compared our findings across our cases and synthesized our results from the overall picture that emerged from a joint analysis, thus following the recommendations in [131]. The individuals we interviewed were highly skilled engineers that are currently working with variability and have reflected on their work with variants. We also made extensive use of cross-checking between the different cases and different investigators to ensure that findings are correct [194]. In addition, during the focus groups and interviews with the case companies, we started the data analysis by preliminary explanation-building [194] together with the experts from the companies, a process that was later completed by the researchers. The results were then submitted to the case companies for member checking and subsequent refinement. External Validity. It is the nature of a study that includes multiple cases that it offers better generalizability than studies of a single case. We followed the process of ana-lytical generalization to abstract beyond the specificities of the individual cases and identified underlying issues that are generic enough to affect a large class of cases. The fact that we found several instances for almost all of the challenges we identified provides confidence that our findings are applicable beyond the twelve cases in this study. In addition, our triangulation with the results of our lightweight literature review enhances the validity of the challenges we identified and formulated.

We used a purposive sample of companies that have expertise in software product lines when selecting our cases. We argue that this is an advantage: the cases we included understand the need to manage variability and have encountered the challenges we report on first hand. Thus, we gathered a more detailed and nuanced picture than if we had included cases with less maturity. Our sample supports the aims of the paper which is to show the level of adoption and the challenges for variability management in cases with a certain maturity. Since we selected mature organizations, our cases are, however, skewed towards rather large companies. Only large organizations can maintain relatively product lines and achieve a high level of sophistication in working with them. We would expect that smaller organizations with smaller product lines will not encounter all challenges mentioned here with the same severity, but that many of them will still encounter issues on a smaller scale.

In addition, the sample contains companies from five European countries. Geographic distribution is a relevant factor since organizational culture is determined by the country the organization resides in [123] and organizational culture in turn affects product line adoption [START_REF] Mu Ñoz | Adopting and institutionalizing a product line culture[END_REF]. However, we cannot control for cultural differences in other continents. In principle, companies in South America, Asia, or elsewhere could behave differently and thus exhibit different challenges. On the other hand, our challenges are related to very technical aspects, mainly architecture and process in the BAPO framework [120,142], as opposed to business and organization, which indicates that similar engineering companies face the same or very similar challenges. Replicating our study to confirm or refute individual challenges would be valuable future work.

We also aimed for diversity in terms of the domains in which the organizations are active. By combining nine cases that rely on variability to develop their products and three tool providers, we also integrate two different perspectives on product-line engineering. In addition, we also combine six companies that create physical systems that include hardware and software (e.g., aerospace and power electronics) and six companies that create only software (e.g., modeling platform and imag-ing technology) of which three are tool providers. Systems engineering companies like the ones we surveyed represent a large share of the value creation and the R&D investment within the European Union [151]. As such, we address a highly relevant market segment. Reliability. The research methodology has been jointly developed by the authors and refined over several iterations, often using feedback from the study participants. However, in order to use the resources made available to us by the case companies to maximum effect, we often used rather informal information channels with a fast response rate for clarifications, e.g., phone calls that were not recorded. That means that some of the findings are based on notes by an individual researcher and not on mechanical recordings. We have addressed this by triangulation with more formal techniques and by member checking. In addition, we have shared all collected information between all involved researchers. We have also recorded all steps we took in a case study protocol [194], an abbreviated version of which is presented in Sec. 3.

Conclusion

We presented a multiple-case study on the adoption and challenges of systematic variability management in industrial practice, organized in a common framework. Our twelve cases comprised nine use cases, where SPLE concepts are adopted for developing an organization's systems, and three tool cases, representing modeling tools for software and hardware, where the vendor sees a need for adopting SPLE concepts in the tools-enhancing variant-engineering practices among customers. Given that most experience reports [121] are more than a decade old and very diverse in terms of details reported, we provided an up-to-date snapshot on as many as twelve cases. With a specific focus on the adoption of concepts organized in a common framework, our work is also the first that systematically elicited this state of adoption from a range of companies-a surprisingly challenging task, given that we needed many iterations to understand the development and thereby also which concepts are used.

Variability Drivers. Not surprisingly, hardware is still one of the most significant drivers, directly followed by market pressure for customization. What appears to gain increasing relevance are Industry 4.0 and digitalization, challenging variability management even more.

Adoption of SPLE Concepts. Not surprisingly, our subjects from the automotive domain are most advanced in terms of adopting SPLE concepts. This highlights the importance of existing experience reports from this domain [START_REF] Flores | Mega-Scale Product Line Engineering at General Motors[END_REF]187,[START_REF] Gustavsson | Architecting automotive product lines: Industrial practice[END_REF][START_REF] Hardung | Reuse of software in distributed embedded automotive systems[END_REF][START_REF] Dziobek | Functional variants handling in simulink models[END_REF]183,[START_REF] Bayer | Process family engineering in automotive control systems: a case study[END_REF]. Yet, what this domain is still lacking is a well-unified management of variability, given a diversity of different asset types we observed, including features, configuration options, calibration parameters, components, and coarse-grained packages. All cases understand the benefit of features and strive to adopt them. It is a bit surprising that the concept of features is not well adopted. Perhaps, lifting programmers' work practices from low-level components to the domain-level is more challenging than expected. In fact, developers are not used to working on different abstraction levels [START_REF] Berger | Three cases of feature-based variability modeling in industry[END_REF]. Our automotive cases are a bit closer to adopting features, given the complexity of their variability. Yet, they do not use feature models or common SPLE tooling.

We observed a preliminary adoption of off-the-shelf SPLE tools in the domain, but since the adoption is still limited, the needs and requirements in this domain seem to not yet been fully understood and met.

While we omitted detailed information about the case companies for anonymity reasons, we can see an increased adoption of concepts in the large cases. Yet, our study again shows the need for adoption also in smaller companies, including those in the web application domain. Experience reports and surveys on adoption in such smaller companies [192,184,185] and strategies for incremental adoption of concepts [START_REF] Antkiewicz | Flexible product line engineering with a virtual platform[END_REF] are certainly relevant to consider in future research.

Challenges. We observe that the adoption of SPLE concepts is still a tool-integration problem, given all the different types of artifacts and existing tooling, which engineers are familiar with and that is core to the development. How this situation can be improved, and whether there are research questions (as opposed to it being just an engineering problem that tool integrators need to solve), are interesting questions to be answered in the future. Some steps in this direction were taken with the publicly available Common Variability Language proposal [START_REF] Haugen | Cvl: Common variability language[END_REF], the Variability Exchange Language [191], the recent initiative for a common feature-modeling language (cf. Challenge 8 in Sec. 7.4), or a recent Dagstuhl seminar on unifying versioning and variability concepts [START_REF] Berger | Software evolution in time and space: Unifying version and variability management (dagstuhl seminar 19191)[END_REF]. However, more agreement and adoption will be needed to consider it. Among the most requested needs by our practitioners are a better representation and visualization of variability, process support for conducting the migration, support for continuous integration, and traceability throughout the lifecycle including the aftermarket. Surprisingly, product-line analysis for validation or verification played a rather minor role in the results of our literature review and among the challenges faced by our cases, perhaps since more systematic variability management needs to be adopted in the first place. Future Work. Finally, beyond the scope of this study, it would be valuable future work to identify and justify research directions that are addressed in the research community, but that are not worth investigating based on empirical evidence. Such studies probably would need to focus on particular sub-directions in productline research, and might be hotly contested among the community, but would apparently be very useful to scope future research to focus on relevant challenges.

Table 1

 1 Variability drivers and variable artifacts as reported by the different cases

								use cases										tool cases
	power	electronics	truck	manufacturing	aerospace	automotive	firmware	railway	web	application	modeling	platform	imaging	technology	traffic control	requirements	engineering	hardware	modeling	chip modeling
	Variability Drivers
	hardware
	usage scenarios
	markets
	simulation				1
	end-user customization
	operating systems and middleware
	operating environments
	evolving requirements																			2
	design exploration
	Variable Artifact Types
	source code
	build system
	custom descriptors	3			4		3					7					8		10	11
	system models
	interfaces
	tests
	meta-data											5					9

Table 2

 2 Adopted product-line engineering concepts and near-term adoption goals

								use cases									tool cases
		power	electronics	truck	manufacturing	aerospace	automotive	firmware	railway	web	application	modeling	platform	imaging	technology	traffic control	requirements	engineering	hardware	modeling	chip modeling
	clone & own					1												11			11
	clone management					1
	configuration
	component/module selection
	conf. options / calib. parameters 15
	configurator tool
	features 2												13
	centralized variability representation 14
	feature model
	FODA-like model
	feature database
	informal, but structured feature model 4
	declared/managed feature constraints			6								6
	platform
	integrated software platform																			10
	product (variant) derivation
	integrated with SPLE tool
	separated domain and application engineering
	variability-aware analyses
	feature-model analyses
	visualizations (e.g., feature hierarchy)
	change-impact analyses 3
	consistency analyses																			16
	requirements completeness/correctness
	safety analyses
	performance analyses
	other quality attributes analyses																			18
	traceability
	feature-to-code
	customer adaptations
	variability decisions/rationales
	security
	feature-level authorization
	adopted concept	rudimentarily adopted		adoption goal	1 on product-line level	2 or feature-like entities
	3 on variant level	4 e.g., spreadsheet

  , AK-VAsmart [121], Philips Consumer Electronics Software for Televisions [121], Philips Medical Systems [121], and Dialect Solutions [178], as well as the case studies on Axis and Ericsson [181]. Interestingly, Staples et al. [178] point out for Dialect Solutions, and Fogdal et al.

http://www.splc.net/fame.html

http://www.splc.net/fame.html

https://modevar.github.io

Acknowledgment

We thank all practitioners who participated in the focus group discussions, as well as we thank the anonymous reviewers for very valuable comments to improve the paper. This work is supported by Vinnova Sweden, Fond Unique Interministériel (FUI) France, and the Swedish Research Council.