Mohsen Asadi

Bardia Mohabbati
email: mohabbati@sfu.ca

Dragan Gašević
email: dgasevic@acm

Ebrahim Bagheri
email: ebagheri@athabasca.ca

Developing Families of Method-Oriented Architecture

Keywords: Method engineering, Software Product Lines, SOA

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

The increase in the complexity of software-intensive systems has urged for the integration of seminal approaches such as Object-modeling Technique (OMT) and Objectory to form integrated (plan-driven) and unified frameworks such as the Rational Unified Process (RUP). Integrated approaches typically target development of a vast variety of software applications, which increase the size of methods and make them become "cook-book" approaches. Recent critical literature reviews and comprehensive case studies have shown that such cook-book approaches do not work successfully for all circumstances [START_REF] Rolland | Method engineering: towards methods as services Software Process: Improvement and Practice[END_REF]. Practitioners could potentially waste up to 35% of their effort by following the steps of standard development methods [START_REF] Harmsen | Situational Method Engineering[END_REF]. Moreover, the results of such studies reveal that the formal definition prescribed by a method in forms of stages and steps widely differ from the method actually being used [START_REF] Lings | Method-in-action and method-in-tool: some implications for case[END_REF]. These issues have motivated the software engineering research community to establish the Method Engineering (ME) [START_REF] Harmsen | Situational Method Engineering[END_REF] discipline. The ME community concentrates on the idea of providing an "adaptation framework whereby methods are developed to match specific organization situation" [START_REF] Rolland | Method engineering: towards methods as services Software Process: Improvement and Practice[END_REF]. The most prominent ME approach is the assembly-based method engineering that creates a new method by assembling existing method components [START_REF] Ralyté | Towards a generic model for situational method engineering[END_REF] [START_REF] Mirbel | Situational method engineering: combining assembly-based and roadmap-driven approaches[END_REF]. Despite the fact that ME has recently produced promising research results, there are still many open research challenges [START_REF] Rolland | Method engineering: towards methods as services Software Process: Improvement and Practice[END_REF]. In this paper, we focus on two key challenges, namely:

1. The lack of a standard model for describing method components limits the opportunities of method engineers, teams and organizations to share, discover, and retrieve distributed method components. When a method engineer wants to create a new method from scratch or by adapting (extending/constraining) an existing method, a common approach is to try to reuse existing method components from the method repository. Therefore, method components need to be discovered and composed with other method components. Due to the lack of standards, method engineers are forced to reuse method components from the local proprietary repositories, without effective capabilities for retrieving method components in repositories of their collaborators. Moreover, with this limitation of method component sharing, business opportunities of organizations are also limited. In fact, they cannot easily publicize and offer the methods that they are specialized in, as (for profit) services. 2. In essence, organizations initially adopt a method for software development.

Afterwards, components of the method may be subsequently added and gradually extended. Such extensions may be derived due to either the evolution of software development or various variations created for some specific method components. Some sources of these diversities may be differences among domains of systems under development (i.e., desktop application, web application, and real-time) or newly emerged software development approaches such as Model Driven Development, Component based Software Development as well as method types such as agile or plan-driven. Thus, there is the need for a systematic approach to manage variability of software methods and adapt software methods (families) that best suit the needs of a specific development context. The first challenge has been already recognized in the literature [START_REF] Deneckère | From Method Fragments to Method Services[END_REF][1] and some researchers have proposed to use of SOA and Web service standards and principles for dealing with the challenge [START_REF] Deneckère | From Method Fragments to Method Services[END_REF] [START_REF] Rolland | Method engineering: towards methods as services Software Process: Improvement and Practice[END_REF]. To this end, the concept of Method Services was coined in analogy of the concept of services in SOA. In order to address both of the above challenges at the same time, we propose combining principles (Sect. 4 and 5) of Software Product Line Engineering (SPLE, Sect. 3.1) and Service-oriented Architectures (SOAs, Sect. 3.2) [START_REF] Tsai | Service-oriented system engineering: a new paradigm. Service-Oriented System Engineering[END_REF]. We use the method service notion for defining method components. We also propose leveraging SPLE with the goal of addressing the second challenge. Our key idea is to introduce a concept of method families, which share a set of common method components, and yet have effective tools for variability management (e.g., feature models). With the use of SPLE principles, we can allow for a systematic modeling of method families and for an automated process of specialization of method families where each family specialization satisfies requirements of a specific situation.

Motivating Example

In order to illustrate the challenges that are tackled in this paper, let consider an organization, which develops software systems in two distinct domains, namely, information systems (both desktop and web-based systems) and real-time systems. We consider that the organization has adopted a base method (e.g., RUP) for the entire systems development process. The base method supposedly is a modular method and its method components are stored in a method repository. Moreover, the organization has employed different development approaches, including code centric development, component based development, and model driven development. Based on the scale and complexity of a project, the organization may follow different development policies such as agile or plan-driven. In addition, contingency factors such as time pressure, user involvement and project familiarity cause the source of diversity in method components. Furthermore, human factors (e.g., roles in the organization and their experience level) could be a source of variation points in the method activities. The organization might also intend to add more requirements for future variations of methods and integrate more project management method components in order to have a better support for project management and risk assessment tasks. As a response to the described circumstance, the organization requires to extend the base method using different method components. As a consequence, the complexity and variations of the base method are gradually increased in practice. This complexity leads to a limited sharing and management of lessons learned. Thus, there is a need to more effectively: 1) manage different variations of the base method that were observed and encountered in the previous projects and systematize the lessons learned; 2) anticipate further needs by considering all possible variations of the base method; and create a systematic method for adaptation of the base method considering the needs and requirements of the new development situations. Moreover, the organization, besides its own development projects, might also want to offer some consultancy services or partner with some other organizations based on expertise in method engineering. In such cases, the organization needs to have a standard method for publishing their competencies, so that other organizations can effectively discover and reuse such experience in similar development situations.

Background

Software Product Line Engineering

The SPLE paradigm aims at managing variability and commonality of core software assets of a given domain in order to facilitate the development of software-intensive products and to achieve high reusability [START_REF] Schmid | A comprehensive product line scoping approach and its validation[END_REF]. SPLE empowers the derivation of different product family applications (aka, family members) by reusing the realized product family assets such as common models, architecture, and components. In this context, software assets are characterized by a set of features shared by each individual product of a family. The set of all valid feature combinations defines a set of product line members of the family. A valid composition of features is called a configuration which in turn is a valid software product specialization. The development of a software family is performed by conducting the domain engineering lifecycle in which the common assets, family reference architecture and the variability models are developed. Afterwards, in the application engineering lifecycle, the common assets are reused and variability models are configured to produce a family. Feature modeling, as a popular technique for modeling variability, is employed to represent the variability and describe the allowed configurations of a software family. This technique is typically used in domain engineering to model an entire family based on the functional characteristics (aka features) that the family provides. Feature models formally and graphically define relations, constraints, and dependencies of software artifacts in a software product family. In essence, there are four types of relationships related to variability concepts in the feature model. They can be classified as: Mandatory (Required), Optional, Alternative feature group and Or feature group. Common features among various members of the family are modeled as mandatory features. In other words, mandatory features must be included in the description of their parent features and must be present in any final configuration. Optional features may or may not be included in a final configuration. Alternative features indicate that only one of the features from the feature groups can be opted. Once a feature model for an entire family is in place, a process of configuration follows. Configuration is a process of selecting features needed for specific applications. Recently, the research community has proposed effective methods for staged configuration where each stage addresses a specific set of requirements in the application development process [START_REF] Czarnecki | Staged Configuration through Specialization and Multi-level Configuration of Feature Models[END_REF].

Method Oriented Architecture

Service-oriented computing (SoC) is a computing paradigm that promises flexibility and agility in the development of collaborative software systems. Service-oriented Architecture (SOA) is the main architectural style for realizing the SoC vision. SOA provides an underlying structure enabling for interoperability and communications between services. Web service, reusable and loosely-coupled components, are the best known materialization of SOAs [START_REF] Tsai | Service-oriented system engineering: a new paradigm. Service-Oriented System Engineering[END_REF]. Web services are built on well-defined standards such as Web Services Description Language (WSDL). Furthermore, the widespread adoption of Web service technologies provides open standards which increase accessibility and interoperability of distributed software services in a networked environment.

On the other hand, ME approaches are hindered by the lack of standards for describing the interfaces of components of methods. Moreover, reusable method components are restrained to be adopted locally by their providers in proprietary repositories. Indeed, the discovery and retrieval of reusable method components can significantly enhance rapid method construction and reuse. The ME community has already proposed the notion of Method Oriented Architecture (MOA) [START_REF] Rolland | Method engineering: towards methods as services Software Process: Improvement and Practice[END_REF] [START_REF] Deneckère | From Method Fragments to Method Services[END_REF], which builds on and adopts SOA principles. Rolland proposed the MOA approach where Method as a Service (MaaS) is considered as an analogy to Software as a Services (SaaS) [START_REF] Rolland | Method engineering: towards methods as services Software Process: Improvement and Practice[END_REF]. MOA aims at developing an ME approach, which elevates the accessibility of method services and facilitates their automated composition. In MOA, method services are described by method providers through WSDL documents. On the other hand, clients search and retrieve the required method services and compose them in order to create their own more complicated method service.

Method Services and Feature Modeling

As mentioned in the introduction section, to address the challenge of variability in method engineering, we intend to apply SPL principles and techniques especially feature modeling. In SPL, functionalities of a set of similar software systems and their visibilities are presented in a feature model in terms of features and variability points, respectively. Likewise, a set of similar methods (we call a family of methods) may have commonalities and variabilities with respect to functionality (i.e. activities). Therefore, a family of methods provides the means for capturing the commonalities (core assets) of all possible methods of a given domain and also addresses variability by covering a comprehensive set of dissimilarities between the methods. In our proposal for family development, distinguishable characteristics of a method mostly including functionalities of the method (i.e. activities) are represented using features. For instance, in our motivating example, one feature of the family is Use-Case modeling. The methods commonality and variability, in terms of their features, are represented in feature model. The development of a family of methods is performed by conducting the domain engineering lifecycle (developing feature model and implementing features), which is followed by the application engineering lifecycle (developing target method with configuring feature model). We should note that the feature model is only the representation for family characteristics and the variability relations and we need to link them to corresponding method implementations (i.e. method fragment). We use method services as well as MOA techniques (i.e. Method service discovery and composition) to implement features of a family. Therefore, we refer to our approach as development of families of method oriented architectures. In order to clarify the difference between feature and method service, let us consider the process of method construction as a process of problem solving, in which the requirements model and the final method are considered as the problem space and the solution space, respectively. Since we intended to develop a range (i.e., family) of solutions (i.e. methods) which have common and variable parts, both the problem and solution spaces become more complex. By following SPLE principles, the family problem space (i.e., family requirements model) is decomposed and grouped into features which form a feature model. In other words, a feature intuitively represents sub-problems of the family problem space, and a feature model represents a hierarchical representation of the family space with variability. For instance, the problem space (feature model) of a described family method at the highest level is decomposed into (see Section 6) management, requirement engineering, development, and deployment sub-problems (features). On the other hand, method services form the solution space, in which one or more method services (sub-solutions), implement (solve) one or more features (sub-problems). From another point of view, features address what the properties of the solution are and method services represent the realization of those properties. Fig. 1 shows the use case modeling feature (one of the features of the feature model given in Section 6) and the corresponding use case modeling method service. As the figure shows, the method service represents how the modeling of use cases should be conducted. Also, a feature represents some functionality, which can be included in a method variant. One of the key concerns in method family engineering is the identification of method services for each feature and the binding of features onto method services. Then, in the application engineering lifecycle, method engineers select features from feature models corresponding to the requirements of the target method (i.e. feature configuration). Next, the method services bound to features in domain engineering are composed automatically and they form an initial method for application engineering. The initial method is adapted and improved until a suitable method is reached for the target problem and deployed.

Families of Method Oriented Architecture

Similar to developing software product lines, we propose two main lifecycles for method family engineering process, namely the Method Domain Engineering and Method Application Engineering lifecycles. Method Domain Engineering lifecycle is carried out one time for the whole family and develops the architecture of the method family, common assets, and variants. In this lifecycle, family features and their variability are modeled by a feature model and suitable method services corresponding to features (i.e. a feature implementation) are discovered and bound to the features. The method application engineering lifecycle develops a target method (i.e. a member of family) for a concrete application by configuring the feature model and assembling the method services related to the configuration. The method application engineering lifecycle is carried out every time a new method is required. The remainder of this section describes the main phases and activities of both lifecycles along with their associated product artifacts.

Method Domain Engineering

Method domain engineering aims at discovering, organizing, and implementing common assets of a method family. Moreover, determining the scope of a method family and describing the variability of the models is achieved during this lifecycle. The input of the lifecycle is domain knowledge relating to and describing the method family and the reusable assets, while variability models for the methods expressed using feature models are the output of this lifecycle. Fig. 2 illustrates the phases and stages of the method domain engineering lifecycle. Method domain engineering starts with the Method Family Scoping phase which is a key phase for achieving economic benefits of a product line [START_REF] Schmid | A comprehensive product line scoping approach and its validation[END_REF]. The Method Family Scoping phase aims at determining a set of products (Software Development Methods) which belong to the family. Scoping of the family is performed in the three stages [START_REF] Schmid | A comprehensive product line scoping approach and its validation[END_REF]. The Method portfolio scoping stage is a high level domain analysis process and uses the market inputs on existing methods, and expert knowledge to derive a standardized description of a method product line, technical domains that are relevant to it, and the range of methods that shall be supported with the method family. It systematizes the method product information, identifies the main features of the product line and checks the consistency. With regard to features of a method family, the development approaches used in methods (e.g., Model Driven Development-MDD, and SOA), final application domains (e.g., Information System, Embedded Systems, and Ubiquities Systems), and method types (e.g., agile or plan-driven methodologies) are determined through this phase using the project documents.

Later, the domain scoping stage uses the basis provided in the previous stage and the expert inputs to identify and group the major functional areas in terms of technical domains which belong to the current method family. Moreover, the benefits and risks pertaining to the various domains are explored and documented. For instance, benefits and risks of employing MDD are identified. Finally, in the asset analysis stage, based on the preconditions established in the previous two stages, precise functionality of the method components that should be supported by the method family are described. This stage determines which assets should be developed for reuse (commonality) and which ones as project-specific (variability). The method engineer indicates the variable features (project-specific) belonging to the family, the type of the variables (e.g. logic, workflow), set of variants for the variable, and status of variants (open or close) [START_REF] Kim | DREAM: A practical product line engineering using model driven architecture[END_REF]. The method product-line roadmap is produced as the output of this phase.

The Method Family Requirements Analysis phase aims at capturing requirements and developing a requirements model for the methods family. The family requirements model contains unique and unambiguous definitions for each requirement as well as the variability of the requirements. The phase receives the documents, stakeholders' viewpoints, and the product-line roadmap, and variability ranges. Similar to typical software engineering procedures, we define functional requirements and non-functional requirements for methods. The functional requirements show the properties that the method should provide, such as work products and required activities; and non-functional requirements include properties that the entire or a large part of methods in the family should have such as smoothness of transition between activities, robustness, and scalability. The method family requirements are elicited and documented. The method engineer gets an agreement of developers (i.e., stakeholders in this case) on the method family requirements. Next, family requirements are refined through decomposition, aggregation, and grouping. Afterwards, in the modeling family requirements stage, techniques such as the mapdriven technique [START_REF] Ralyte | Requirements definition for the situational method engineering[END_REF] are applied to develop the family requirements model. The family requirements model includes the functional requirements and is represented as family requirements map. The progression activity analyzes the family requirements model and defines the requirements filling the gaps in the family requirements model. Finally, the method engineer verifies the completeness and coherence of the family requirements models as well as the level of satisfaction of the stakeholders' needs by using Requirements verification and Validation activities [START_REF] Ralyte | Requirements definition for the situational method engineering[END_REF], respectively.

The goal of the next phase, Method Family Realization phase, is to identify common and variant features within the family and to model them with a feature model. Afterwards, the appropriate method services are discovered for each of the features. The feature model is developed by the Feature Model Development stage. That is, the common and variable functionalities of methods of the family are managed by representing them in a feature model. The method engineer starts from the requirements and analyzes the requirements, their granularity level and relationships, and then groups them into features. Moreover, the variability relations are identified between features. Additionally, nonfunctional requirements such as traceability and project management are analyzed and added to the feature model as features and their relations are also identified. Furthermore, the method engineer annotates the features with required information.

The requirements and feature family modeling phases produce the requirements model, requirements documents, and feature model of the method family. Feature family modeling, as described above, is followed by a Feature Driven Method for service discovery and selection. The stage of the feature-driven discovery is performed by considering each individual feature and their respective annotations. In essence, a feature annotation provides functional and non-functional keywords used to generate feature queries. The feature queries simply describe what the desired method services should be and how they should behave. In our current implementation, we adopted text-based approach to the discovery of method services. In evaluations of our current implementation of the feature-driven service discovery, we observed promising results in experimenting with the active service search engines while developing families of software services [START_REF] Mohabbati | ReCoIn: A Framework for Dynamic Integration of Remote Services in a Service-Oriented Component Model[END_REF]. Since, MOA uses SOA standards for defining and publishing method services, we may expect similar results for discovering of method services. Given that there are no publically available repositories of method services, we are now developing a test collection of method services. In this process, we can easily leverage existing service repositories (e.g., Seekda already used in our implementation) for storing method services. Other approaches can be leveraged in feature-driven service discovery such as logic-based approaches [START_REF] Klusch | Semantic Web Service Coordination[END_REF].

Method Application Engineering

Once the method domain model is created, then method engineers can take the method domain model and create different instances out of it based on target method requirements. We refer to this process as method application engineering. Therefore, method application engineering aims to develop a method for a target situation (e.g. a member of the method family) by utilizing the reusable assets created in the domain engineering lifecycle. The input of the lifecycle is the project documents for the concrete method and the output is the method satisfying the requirements. It captures the final method requirements, selects the corresponding features from the feature model, and finally assembles the method services bound to the selected features. The application engineering process is illustrated in Fig. 3.

The Application Method Requirement Analysis phase aims to define the requirements of the target method. The documents related to the required method are the inputs and its requirements model and the requirement documents are the outputs. The documents related to the target method should include definition (specify the type of the project at hand), domain (specify the application domain of the target method), and deliverable (specify the artifacts that should be produced) [START_REF] Coulin | A Lightweight Workshop-Centric Situational Approach for the Early Stages of Requirements Elicitation in Software Systems Developme[END_REF]. The family requirements model and documents are utilized through this phase to produce the method application requirements.

Capture Method Requirements

Fig. 3. Method Application Engineering Lifecycle

First, the method application requirements phase captures stakeholders' requirements and documents them. Then, method requirements are refined and clarified further and the agreement of stakeholders is achieved. Next, the method engineer develops the requirements model in the form of a requirements map. Moreover, non-functional requirements are utilized in the feature selection process. Finally, the method requirements are validated and verified to check the completeness and correctness of the method requirements. In all the activities of this phase, the family requirements model is used as a reference to facilitate the process of requirement analysis of the members of the method family. There is a possibility of capturing requirements which were not captured in method family requirement analysis. The activities of this phase concentrate on one method application, so they do not deal with variability in the family.

The Application Method Development phase creates the target method by configuring the method family and delivers the final method configuration to the developers. The method feature model configuration stage aims to develop the method by selecting the most appropriate set of features from the feature model through a stage configuration process. It receives the method requirements and produces the corresponding feature configuration. The stage configuration process [START_REF] Czarnecki | Staged Configuration through Specialization and Multi-level Configuration of Feature Models[END_REF] starts from the feature model and carries out successive specializations to create the final configuration. That is, the staged configuration process would limit the space of the method family to the space most relevant for the current method that is being built. Through the staged configuration, the method engineer produces the final configuration. Since in method domain engineering, the method engineer might want to bind a list of method services that have the same interfaces (i.e. situation and intention) but different nonfunctional properties defined in descriptors of method services, the final method service for each feature is selected from the list of alternative method services. The output of the stage is the set of the features (mandatory and optional) as well as their corresponding method services.

If the selected method services (features) do not cover all the requirements of the target method, the new method services for the remaining requirements are discovered in some other repositories or developed from scratch. After the method engineer makes sure that all required method services (features) have been gathered, he/she starts the composition of method services (features) via the Assemble Method Services stage. The selected features are divided into functional (e.g., requirement elicitation, use case modeling, and developing design model) and non-functional features (e.g. quality assurance, project monitoring, and traceability checking). First, the method services are orchestrated and the necessary adaptation and mediation are conducted. Then, a decision about the location of method services within a large scope (like quality assurance) is made. After creating the target method, the verification/validation task is done by the method engineer to check whether the method is free from defects and if the target method meets all requirements established in the requirements phase. Moreover, the completeness of the method is verified by a completeness task. Finally, the method is deployed to the stakeholder environment by preparing method documents, training developers, and supporting staff through the execution of the method.

Case Study

In this section, we represent our motivational example from Section 2 by following our proposed approach described in the previous section. Due to the space limitation, we only explain the domain method engineering lifecycle, which comprises Product Line Scoping, Family Requirement Analysis, Feature Modeling and Feature-based Method Service Discovery and Selection.

Product Line Scoping: By completing the activities of the product line scoping phase, we identified the criteria which specify the product line boundaries, the main functionality area, and core assets of the method family. Table 1 shows a part of the product line scoping results. One of the major functionality areas distinguished in the domain scoping by all variations of the method is the support for a generic development lifecycle. For instance, unit testing is a core asset in the method family.

Family Requirement Analysis: Functional and non-functional requirements with their commonalities and variability are captured and documented separately. Table 1 shows a part of requirements categorized based on their types. Functional requirements include activities and work products that should be supported with family method. The base method of the organization is explored to discover more detail requirements. The family requirements model is created first by using mapdriven approach [START_REF] Ralyté | Towards a generic model for situational method engineering[END_REF] and then verified and validated. Due to the space limitation for this paper, the requirements model is omitted from the paper.

Feature model Developmentbased on the family requirements model defined in the previous phase and the existing basis method in the organization, features and their corresponding relations are identified and modeled. The part of feature model designed for target organization is depicted in Fig. 4. Features show the method services required for the family and they can be considered as interfaces for representing method services in the family.

Feature-driven Method Service Discovery and Selection: the next step after feature model development is the discovery of method services. The aim is to find and select among available methods services, which can satisfy desired functional and nonfunctional requirements of the method for specific situations. As we described earlier, we consider each feature and their associated annotations as queries for method components stored in method repositories. In method service discovery, we assume that the method components, described by WSDL, are available and accessible through either the proprietary method repositories of the organization or public repositories provided by third-parties. Thereby, organizations can publish and share their method chunks as services. Although there are on-line repositories such as Open Process Framework (OPF) [20], available reusable method components are not accessible through standard interfaces. Moreover, there are no facilities to search and discover such available methods. Accordingly, in the process of discovery and selection, the proprietary method repository of the organization is initially used to method services. In case that some of the features are not associated with the organization's services, search queries are broadcasted to the public method repositories.

The Feature Model Plugin (http://gsd.uwaterloo.ca/projects/fmp-plugin/), available for Eclipse environment, is utilized and extended as tool support for modeling and configuring method family. It supports cardinality-based feature modelling, specialization of feature diagrams and configuration based on feature diagrams. Our method chunk service repository is based on the publicly-available Seekda (http://seekda.com) service repository. Our current implementation of feature-driven service discovery is described in [START_REF] Mohabbati | ReCoIn: A Framework for Dynamic Integration of Remote Services in a Service-Oriented Component Model[END_REF]. Table 1. Product line scoping and family requirements analysis outcomes after applying the proposed method on the motivational example. It is important to notice that the table does not give all items identified these phased, but some of the most notable examples.

Phase Identified work Product and domains

Product Line Scoping Portfolio

• Application properties -application domain (Information systems, Real-time), application type (intra-organization, Organization-customer, inter-organization), source system (it can either use legacy system or does not have system code). The most prominent sub-area of ME, Situational Method Engineering (SME), proposed by Welke et al [START_REF] Welke | Method Engineering: a proposal for situation-specific methodology construction[END_REF] is concerned with the creation of methods 'on-the-fly' (i.e. construct or adapt a method according to situation of the project at hand). The ME approaches are classified by Ralyte et al [START_REF] Ralyté | Towards a generic model for situational method engineering[END_REF] as: Ad-Hoc (i.e. Method created from scratch); Extension-Based (i.e. Method is created by extending an existing method [START_REF] Ralyté | Towards a generic model for situational method engineering[END_REF]); Paradigm-based (an existing meta-model is adapted, instantiated, or abstracted to create a new method [START_REF] Ralyté | Towards a generic model for situational method engineering[END_REF]); and Assembly-based (a method is created by reusing existing method components [START_REF] Ralyte | An assembly process model for method engineering[END_REF][16] [START_REF] Asadi | Patterns of Situational Method Engineering[END_REF]). These approaches mostly focus on reusability and modularity principles. Besides this classification, Karlsson et [START_REF] Karlsson | Method configuration: adapting to situational characteristics while creating reusable assets[END_REF] proposed the Method Configuration approach (more general than extension based) in which a target method is created by adding/removing elements and features. They concentrate on variability management and reusability. All mentioned approaches are based on one or more of the following principles -meta-modeling, reuse, modularity, and flexibility. Our proposed approach is similar to the assembly-based and method configuration by following of the modularity, reusability, and variability principles. However, our approach enables for a higher degree of reusability by leveraging SOA principles and for a more systematic variability management by employing SPLE principles (As shown in software engineering SPLE increases reusability [START_REF] Clements | Software product lines[END_REF]). Gonzalez-Perez [20] explained the benefits of ISO/IEC 24744 meta-model for both method specification and enactment and proposed a product-centric approach to developing a new methodology. Aharoni et al [START_REF] Aharoni | A Domain Engineering Approach for Situational Method Engineering[END_REF] enriched the ISO/IEC 24744 for creating and tailoring methods through an approach called Application-based Domain Modeling (ADOM). The approach is based on the layered framework including application, domain, and language. The domain (methodology) layer contains different method concepts as well as the specification of their exact usage situation. The application layer, called endeavor layer, includes specific method components and situational methods, which are created based on domain model terminology, rules, and constraints. The language layer defines any modeling language that can be used for describing meta-models and method components. Our approach differs from these approaches in using variability modeling language (i.e. feature modeling) and software product line principles. Moreover, we provide a reference architecture for a whole family which eases configuring and developing methods. Additionally, we use a new concept for method component (i.e., method service), which utilizes standards in SOA to improve discovering and reusing method components.

Recently MOA [START_REF] Deneckère | From Method Fragments to Method Services[END_REF][1] was proposed which empowered the assembly-based method engineering principles with a standard for describing method components (in terms of method service) and with service discovery principles for finding distributed method components. Our approach also utilizes MOA to describe and discover method services corresponding to the features of a method family.

Discussion

Two main issues regarding the proposed approach are validity and cost-benefit analysis of the approach. For both issues, it is required to conduct an empirical study. We did a case-study in which we explained the steps of the method. However, it cannot completely ensure the validity of approach, its benefits and limitations. In order to clarify these issues in our method, we make our argument based on analogy between software and methods as proposed by Osterweil [START_REF] Osterweil | Software processes are software too[END_REF]. Therefore, our assumption is "software processes are software too" [START_REF] Osterweil | Software processes are software too[END_REF]. Considering this analogy, we can adopt similar approaches and techniques used in software engineering for solving existing problems in method engineering. As we see, the method engineering community proposed MOA inspired from SOA to deal with the lack of standard for defining the method fragment interfaces [START_REF] Rolland | Method engineering: towards methods as services Software Process: Improvement and Practice[END_REF]. But, to use analogy as a viable strategy for solving a problem in method engineering, referred to as the target domain, we need to identify the corresponding construct in software engineering (source domain) and define a mapping schema. For example, in method engineering, the method fragment notion (called method service) is mapped to service notion in SOA and method notion is mapped to software service. Hence, we can use SOA principles and have benefits of SOA in the MOA domain. The other problem method engineers deal with is variability in the base method and configuring the method based on the target project for which some approaches have been proposed [START_REF] Ralyté | Towards a generic model for situational method engineering[END_REF] [START_REF] Karlsson | Method configuration: adapting to situational characteristics while creating reusable assets[END_REF]. On the other hand, software variability is a well-known problem in the software engineering community and many techniques and approaches have been proposed like feature modeling to manage the problem and various success stories in using product families and associated techniques have been reported. As an example, Clements and Northrop reported that Nokia was able to increase its production capacity for new cellular phone models from 5-10 to around 30 models per year, which alleviated Nokia's main challenge being the high pace of market demand and customer taste change [START_REF] Clements | Software product lines[END_REF]. These results ensure both validity and benefits of software families. Therefore, we tried to make an analogy between software family and method base and coined the notion of family of methods. We mapped the features to the method fragment interfaces and handled the variability in base method and configuration problem according to the target project requirements. As a result, we expect similar benefits to be reaped within the method engineering domain. We are also aware of the cost of creating family or reengineering current methods into method family (i.e., creating a method feature model), but for long term the benefits that will be achieved can recompensate these costs as happened in the broader software engineering practice.

Conclusion and Future Work

In this paper, we have presented an approach for developing families of software developments methods. We exploited the notion of method services to facilitate the discovery of distributed method components. Such discovered method components can be used as an implementation for both sets of common and variable method assets of a family of methods. The proposed approach makes use of feature modeling to manage variability of method families. Managing and modeling variability enables for a more effective method construction and for a more systematic method reuse. We believe that the described concept of families of method-oriented architectures may not be entirely feasible now, due to the lack of a complete method sharing ecosystem, but with the growing interest for services economy, more attention to such ecosystems can easily be envisioned to appear soon. Thus, our approach is a small step towards making this vision possible. The adoption of widely used SOA standards helps in publishing and sharing method components. Furthermore, organizations will be able to take the advantages of distributed architectures to design, implement, execute and reuse available method components. Last but not least, the long term goal is to enable different organizations and enterprises to publish, advertise, discover and reuse their methods components.

While the paper proposed a methodology for the combined use of SPLE and SOA principles in method engineering, the contribution of this paper deserves to be considered in a broader context of its implications. As already demonstrated in the previous research [START_REF] Montero | From Feature Models to Business Processes[END_REF], transforming configured feature models into workflow and service composition languages is possible. Thus, the combined use of SOA and SPLE enables for leveraging existing workflow engines (e.g., BPEL) in management and execution of software projects. Moreover, with such an executable representation of methods as workflows, one can also expect an increased compliance of projects with the steps defined by methods. As workflow management provides also best practices (i.e., workflow patterns), the combined use of workflows with software methodologies might lead to further benefits such as improved parallelization of some stages. With representation of method components as services, tracking of the project progress could also be improved, while the invocation of method services can explicitly be associated with the other tools used in different method stages.

As future work, we intend to provide a more comprehensive evaluation of the proposed approach by developing a collection of method service to be used for experimentation. We aim to extend our approach from different perspectives to reduce the manual intervention needed in the final method development. We plan to use ontology-based representation for feature models to automate consistency checking of features in method families as described in [START_REF] Montero | From Feature Models to Business Processes[END_REF]. Furthermore, we intend to extend the feature modeling language to allow method engineers to add concepts of domain ontologies for annotation of features. This will consequently be used for advanced ontology-based discovery and composition of method services [START_REF] Klusch | Semantic Web Service Coordination[END_REF]. Currently, we are developing an environment that supports our proposed process. The environment will include the modeling of method families, annotation of feature models, discovering of method services, stage configuration of feature models, and deployment to standard workflow management engines.

Fig. 1 .

 1 Fig. 1.The relation between use case modeling feature (on the left part) and its corresponding method services which define both process and product model for use-case (adapted from[START_REF] Mirbel | Situational method engineering: combining assembly-based and roadmap-driven approaches[END_REF]).

Fig. 4 .

 4 Fig. 4. A sample feature model of a family of software development methods

  • Development Approach -systems can be developed by following multiple approaches such as Component based Development, Model Driven Development, or Test Driven Development. • Human Factors such skill level includes beginner, medium, and expert (i.e., analyst, designer, developer, and, tester). • Contingency Factors -user involvement, project familiarity, project scale and complexity, innovation level, and project dependency.

						• Project Management -monitoring, risk management, configuration and change
						management, postmortem reviewing, metric management, human resource
			Domain		management. • Generic software development lifecycle (requirement engineering, analysis, design, development, deployment), reusability, management (risk, people), maintenance, test model, implementation models, design model, and Application Technology (Include
			Asset		Data-base, and GUI, is distributed). • Functional requirement engineering, non-functional requirement engineering, behavioral analysis, structural structure, functional analysis, feasibility study, architecture design, project planning, test case development, unit testing, and risk
						management. • Common -Specification in high level abstraction, covering generic software
						development lifecycle, manage and monitor the project, capture requirements, model
	Analysis	Family Requirement ts	Requirements Requiremen	Functional Functional	Non-	requirements, validate requirements, defining the infrastructure of system, and plan the project. • Variables-Goal-based requirement extraction, consider review sessions (product and plan review), having stand up meeting, having lightweight design process, formal verification on each abstraction level, concurrency, configuration of software and hardware, having platform independent models, having platform specific models, component identification, component specification, component interaction, component assembly, and PIM and PSM synchronization. • Common -iterative process, incrementally development, traceability to requirements, clear separation of concerns, smooth transition between activities, and method flexibility. • Variables -semi automatic refinement between abstraction level, method scalability, lightweight process, and formal checking.

Mohsen Asadi, Bardia Mohabbati, Dragan Gašević, Ebrahim Bagheri

Mohsen Asadi, Bardia Mohabbati, Dragan Gašević, Ebrahim Bagheri

Mohsen Asadi, Bardia Mohabbati, Dragan Gašević, Ebrahim Bagheri

Mohsen Asadi, Bardia Mohabbati, Dragan Gašević, Ebrahim Bagheri

Mohsen Asadi, Bardia Mohabbati, Dragan Gašević, Ebrahim Bagheri


Pascal Kuhn
email: pascal.kuhn@dlr.de

Stefan Wilbert

David Schüler

Christoph Prahl

Thomas Haase

Lourdes Ramirez

Luis Zarzalejo

Angela Meyer

Laurent Vuilleumier

Philippe Blanc

Jean Dubrana

Andreas Kazantzidis

Marion Schroedter-Homscheidt

Tobias Hirsch

Robert Pitz-Paal

Validation of Spatially Resolved All Sky Imager Derived DNI Nowcasts

Mainly due to clouds, Direct Normal Irradiance (DNI) displays short-term local variabilities affecting the efficiency of concentrating solar power (CSP) plants. To enable efficient plant operation, DNI nowcasts in high spatial and temporal resolutions for 15 to 30 minutes ahead are required. Ground-based All Sky Imagers (ASI) can be used to detect, track and predict 3D positions of clouds possibly shading the plant. The accuracy and reliability of these ASIderived DNI nowcasts must be known to allow its application in solar power plants. Within the framework of the European project DNICast, an ASI-based nowcasting system was developed and implemented at the Plataforma Solar de Almería (PSA). Its validation methodology and validation results are presented in this work. The nowcasting system outperforms persistence forecasts for volatile irradiance situations.

INTRODUCTION

Solar irradiance on earth's surface presents large temporal and spatial variabilities affecting the reliability of solar power plants. For timescales below 60 minutes, this variability is mainly caused by clouds. To enable profityielding operation and to avoid blackouts, irradiance forecasts with high spatial and temporal resolutions (e.g. 25 m 2 , 60 seconds) for the next 30 minutes are required [START_REF] Hirsch | Towards the Definition of Requirements for Direct Normal Irradiance Nowcasting Systems[END_REF]. These real-time and high-resolution forecasts with the specified forecasts horizons are called nowcasts. Due to technical limitations, neither numerical weather predictions nor satellite forecasts can provide this information, but only ground-based All Sky Imagers (ASI).

Within the framework of the European project "Direct Normal Irradiance Nowcasting methods for optimized operation of concentrating solar technologies", different approaches to achieve nowcasts are investigated at PSA [START_REF] Massip | DNICast: Report on algorithms for nowcasting methods based on sky imagers[END_REF][START_REF] Blanc | Short-term forecasting of high resolution local DNI maps with multiple fish-eye cameras in stereoscopic mode[END_REF]. Located in the Tabernas Desert in the south of Spain, the PSA is a research center focusing on solar thermal technologies. Scientists from CIEMAT (Centro de Investigaciones Energéticas, Medioambientales y Tecnológicas) and DLR (German Aerospace Center) operate several meteorological measurement stations including ASI cameras, radiometers and a shadow camera system. This paper introduces the validation methodology and the benchmarking framework, which can be used to benchmark and compare different nowcasting approaches. This way, the optimum nowcasting configuration can be determined. The validation is conducted for an example nowcasting data set provided by Patras University and MINES ParisTech, which is presented in [START_REF] Blanc | Short-term forecasting of high resolution local DNI maps with multiple fish-eye cameras in stereoscopic mode[END_REF].

BENCHMARKING FRAMEWORK AND METHODOLOGY

For the validation of the ASI-derived DNI maps, twenty pyranometers, seven pyrheliometers and a shadow camera system are available to derive reference data. Six shadow cameras take photos of the ground from the top of an 87 m high solar tower (CESA1). Six concurrent images are combined to one so-called orthoimage, which is a geometrically corrected image (Fig. 1a).

Using the orthoimage of the investigated timestamp and two reference orthoimages, taken when no shadow fell on the PSA and when the PSA was completely shaded, as well as corresponding DNI measurements, a spatially resolved DNI map is calculated. In addition to DNI values, the reference system is able to calculate global horizontal irradiance (GHI) and global tilted irradiance (GTI) reference irradiance maps. The benchmarking is conducted with nowcasts provided by the system presented in [START_REF] Blanc | Short-term forecasting of high resolution local DNI maps with multiple fish-eye cameras in stereoscopic mode[END_REF]. The configuration of this nowcasting system consists of two ASI cameras and one radiometer station (southernmost in Fig. 1a). A detailed validation of the reference system is given in [START_REF] Kuhn | Shadow camera system for the generation of solar irradiance maps[END_REF] and preparatory works regarding the reference system are presented in [START_REF] Schenk | Design and operation of an irradiance measurement network[END_REF]. In this paper, the focus lays on the methodology to validate and benchmark nowcasting systems. The results presented here are derived from selected days between September and November 2015. These months are especially challenging due to low sun elevation angles, resulting in large deviations of cloud shadows for small errors in cloud height estimations.

Days without transient clouds and timestamps with hardware failures are excluded. The validation is conducted for lead times of 0 min, 5 min, 10 min, 15 min, 20 min and 30 min ahead, exploring the limits of ASI-derived nowcasts.

The validation process consists of three steps:

1. The one minute averaged DNI values in each pixel of the nowcasted DNI map which contain radiometers are compared to their corresponding ground measurements. 2. The average DNI over differently sized areas typical for solar plants are compared to the figures from the reference, derived from shadow cameras and ground measurements. 3. Furthermore, ASI-derived nowcasts are compared to persistence forecasts.

In this paper, we will give examples of these three steps. Additionally, requirements for applications in CSP plants defined in [START_REF] Hirsch | Towards the Definition of Requirements for Direct Normal Irradiance Nowcasting Systems[END_REF] and [START_REF] Hirsch | DNICast: Technical report on the functional requirements for the nowcasting method[END_REF] are discussed.

Comparing Nowcasted Data with Radiometer Stations

Measurements from radiometers are compared to DNI values of the corresponding pixels in the nowcasts for different lead times. As an example, three pyrheliometers are compared to the corresponding pixels from the nowcasted data set. Bias, relative bias, mean absolute error (MAE), relative MAE, root mean square error (RMSE), relative RMSE, standard deviation (std) and relative std are calculated for each station and each day. The formulas are briefly explained in the following with o i as the DNI values from the observations in the reference data set and p i as the predicted DNI values from the nowcasted data set at timestamp i. N equals the total number of timestamps included in the evaluated validation interval. Relative values are only derived for single days and O m is set to the daily mean observed DNI. [START_REF] Gueymard | A review of validation methodologies and statistical performance indicators for modelled solar radiation data: Towards a better bankability of solar projects[END_REF] The bias is the arithmetic average of the values of the differences:

    N i i i o p N bias 1 1 (1)
The relative bias is given by:

     N i i i m o p O N bias relative 1 100 (2)
The Mean Absolute Error (MAE) is:

    N i i i o p N MAE 1 | | 1 (3)
The relative MAE is defined as follows:

     N i i i m o p O N MAE relative 1 | | 100 (4) The Root Mean Square Error (RMSE) is 2 / 1 1 2 / ) (           N i i i N o p RMSE (5)
The relative RMSE is defined as follows:

2 / 1 1 2 / ) ( 100           N i i i m N o p O RMSE relative (6)
The standard deviation (std) is:

2 / 1 1 2 1 2 ) ( ) ( 1                 N i N i i i i i o p o p N std (7)
The relative std is defined as follows:

2 / 1 1 2 1 2 ) ( ) ( 100                  N i N i i i i i m o p o p O N std relative (8)
Before calculating the error metrics for 9 days, we look at specific days to get an understanding of the system performance for different irradiance situations. These days are Sept. 18 th 2015, Sept. 19 th 2015 and Nov. 22 nd 2015.

In Fig. 2a, the DNI values of Sept. 18 th 2015 are displayed as measured by a reference station (green, solid), as predicted by the persistence forecast (red, dashed) and as predicted by the nowcasting system at the station's position (blue, dotted). The DNI values from both the nowcasts and the persistence forecasts are displayed at a lead time of 10 min. The reference DNI in this graph comes from the southernmost station in Fig. 1a and is only given for timestamps with available nowcasted data.

The persistence forecast uses the DNI from the southernmost station as input to derive the Linke turbidities following [START_REF] Ineichen | A new airmass independent formulation for the Linke turbidity coefficient[END_REF]. From these Linke turbidities, the future DNI is calculated.

In order to provide an understanding of the persistence forecast, its predictions are displayed for the whole day.

Figure 2b depicts the bias and standard deviation for the nowcasts ("ASI", solid) and the persistence forecast ("Persist", dashed) at different lead times. The values for three pyrheliometers are depicted as stars for the nowcasts and as circles for the persistence forecasts. The lines show the average of the results for the three pyrheliometers. Both the persistence forecasts and the ASI system use the DNI measured at the southernmost station at the time of the creation of the nowcast, resulting in little deviation at lead time 0 min for the persistence forecast. In Fig. 3a, RMSE and MAE for the nowcasts ("ASI", solid, stars) and the persistence forecast ("Persist", dashed, circles) at different lead times are shown in the same way as the bias and the std in Fig. 2b. Figure 3b shows the corresponding relative properties for the nowcasts.

For the often constant irradiance conditions present on Sept. 18 th 2015 (clear sky with few transient clouds), the nowcasting system does not beat the persistence forecast. The corresponding graphs for Sept. 19 th 2015 are depicted in Fig. 4 and Fig. 5. Unlike Sept. 18 th 2015 (Fig. 2a), this day displays a high volatility caused by transient clouds. In this situation, the nowcasting system surpasses the persistence forecasts for all lead times above 5 min. Figure 6 and Fig. 7 are linked to Nov. 22 nd 2015. This day displays less volatility than Sept. 19 th 2015, but more than Sept. 18 th 2015. Due to low sun elevation angles below 33 degrees (resulting in large errors for small deviations in cloud height detection) and fairly constant irradiance conditions, this day is a challenge for the nowcasting system, which does not outperform the persistence forecasts. For lead times below 20 min the deviations of the persistence forecasts and the ASI system are similar. The error metrics for all considered 9 days increase with increasing lead times and are depicted in Fig. 8a and Fig. 8b. Depicted are the mean values averaged both over the days and three pyrheliometers. Due to the often constant sky conditions in Southern Spain and autumnal elevation angles, persistence forecasts and ASI nowcasts display similar error metrics. For days and periods similar to Sept. 19 th 2015, the nowcasting system outperforms the persistence forecast. As nowcasting systems are most relevant for volatile irradiance conditions, this result is encouraging. The results are in alignment with publications on other ASI systems: RMSE values between 155 W/m 2 and 200 W/m 2 for 10 min GHI forecasts are reported in [START_REF] Bernecker | Continuous short-term irradiance forecasts using sky images[END_REF] and RMSE values also for GHI forecasts up to 250 W/m 2 for 23 min lead time in [START_REF] Schmidt | Evaluating the Spatio-Temporal Performance of Sky Imager Based Solar Irradiance Analysis and Forecasts[END_REF]. Note that these values hold for GHI.

A hybrid ASI/stochastic learning approach presented in [START_REF] Chu | Hybrid intra-hour DNI forecasts with sky image processing enhanced by stochastic learning[END_REF] achieved RMSE values between 55 W/m 2 and 140 W/m 2 for DNI forecasts and a lead time of 10 min. In [START_REF] Fu | Predicting solar irradiance with all-sky image features via regression[END_REF], RMSE values for DNI forecasts between 169 W/m 2 for 5 min lead time and 191 W/m 2 for 15 min lead time as well as MAE values between 139 W/m 2 (5 min) and 152 W/m 2 (15 min) are demonstrated. As the specifics of the selected days heavily influence the error metrics, a large variety of conditions or the same data set must be included for detailed comparisons of these error metrics.

Comparing Field Averages

For CSP plant operation, point like DNI forecasts are less relevant than forecasting average DNI values for a given area, e.g. the whole solar field of a plant or subfields. The DNI forecast errors of spatially averaged figures are generally smaller as discrepancies are reduced. Hence, the statistical properties derived above do not provide the final quality information that is required for the plant operator working with nowcasting data. Therefore, spatially aggregated nowcasted DNI figures are compared to the corresponding reference values from the shadow camera system. Three different spatial aggregations are considered (Fig. 9): The first field corresponds to a whole CSP power plant having a size of 2 km × 2 km ("Whole field"). The second field corresponds to a subfield of a CSP plant with 650 m × 650 m ("Big subfield"). The third field corresponds to 60 m × 300 m ("Small subfield"). Choosing differently sized fields provides an understanding of the aggregation effects reducing error margins.

Besides whole field averages, CSP plant operators are interested in subfield figures, as these subfields can be managed separately (e.g. regarding mass flow rates of heat transfer fluid). As Sept. 18 th 2015 is nearly a clear sky day with very low temporal and spatial irradiance variations, the persistence forecast outperforms the nowcasting system. For the more turbulent Sept. 19 th 2015, the persistence forecasts, benefiting from spatial averaging, also surpass the nowcasting system for more than 5 min lead time. Partially, this is caused by large deviations in the nowcasted data set visible for early hours in Fig. 2a and Fig. 4a.

For Sept. 19 th 2015, the error metrics of the persistence forecasts are lowest for the big subfield. At first glance, this is unexpected as the metrics should increase with decreasing field sizes due to spatial averaging. This is understandable as the pyrheliometer providing live DNI input to the ASI and to the persistence forecast systems is close to the big subfield but not to the small subfield. Furthermore, the dominant cloud direction on this day is westwards, from the reference station to the big subfield. Thus, the data for the big subfield benefits more from the station's position than the whole field and the small subfield. The behavior of field aggregated persistence forecasts is totally different from the comparisons based on singular radiometers (see e.g. Fig. 2b): The field aggregated persistence error metrics do not rise significantly with increasing lead times as the considered temporal and spatial variabilities are similar for most situations. This effect is further discussed in [START_REF] Kuhn | Shadow camera system for the generation of solar irradiance maps[END_REF].

CONCLUSION AND PERSPECTIVES

In this paper, the methodology to validate and benchmark nowcasting systems is presented using an exemplary data set. The nowcasts deviate from the reference system similarly to previous findings reported in the literature. For high variability in irradiance, the nowcasting system outperforms the persistence forecast. For periods with low irradiance variability, the persistence forecasts surpass the nowcasts. From a plant operator's point of view, nowcasting systems are most important for volatile periods of time, in which persistence forecasts will fail. This indicates the applicability of ASI camera systems for CSP plant operation.

To the best of our knowledge, this is the first time nowcasted two dimensional irradiance maps are compared with spatially resolved reference irradiance maps. The reference system is briefly described here and in more detail in [START_REF] Kuhn | Shadow camera system for the generation of solar irradiance maps[END_REF]. In this publication, a thorough validation of the reference system including additional improvements is presented.

The optimum ASI based nowcasting configuration has to be further investigated. Different algorithmic approaches, the optimal number of ASI cameras and the benefits of additional sensors such as ceilometers must be benchmarked. At PSA, a benchmarking framework for nowcasting systems is operational. With this reference system, large-scale comparisons for different nowcasting configurations can be performed. This way, the development of the optimum nowcasting system is supported.

FIGURE 1 .

 1 FIGURE 1. (a) Instrument setup at PSA. Black pixels are excluded from the evaluation. Red circles indicate the position of the ASIs. The red square is at the position of all six shadow cameras. The green stars represent two-axis trackers with pyrheliometers and pyranometers. The blue circles mark Si-pyranometers. (b) Reference DNI map generated from the image on the left, reference images and ground data. An example of an orthoimage for Sept. 19 th 2015, 12:57 UTC+1, is shown in Fig. 1a. Shadows are clearly visible. The positions of the radiometers and cameras are marked. The derived reference DNI map for this timestamp is shown in Fig. 1b.

FIGURE 2 .

 2 FIGURE 2. (a) DNI values of a reference station (green, solid), the persistence forecast (red, dashed) and the ASI-derived nowcasted values at the station's position (blue, dotted) on Sept. 18 th 2015. Persistence forecast and nowcasts have a lead time of 10 min. (b) Bias and standard deviation for the nowcasts ("ASI", solid, stars) and the persistence forecast ("Persist", dashed, circles) at different lead times. The values of three pyrheliometers are depicted as stars for the nowcasts and with circles for the persistence forecasts. The lines show the average of the results for the three pyrheliometers.

FIGURE 3 .

 3 FIGURE 3. Statistical properties of the nowcasted DNI map-pixels in comparison to three pyrheliometers on Sept. 18 th 2015. (a) RMSE and MAE for the nowcasts ("ASI", solid, stars) and the persistence forecast ("Persist", dashed, circles) at different lead times. The values of three pyrheliometers are depicted as stars for the nowcasts and with circles for the persistence forecasts. The lines show the average of the results for the three pyrheliometers. (b) Relative bias, relative MAE, relative RMSE and relative standard deviation of the nowcasts at different lead times.

FIGURE 4 .FIGURE 5 .

 45 FIGURE 4. (a) DNI values of a reference station (green, solid), the persistence forecast (red, dashed) and the ASI-derived nowcasted values at the station's position (blue, dotted) on Sept. 19 th 2015. Persistence forecast and nowcasts have a lead time of 10 min. (b) Bias and standard deviation for the nowcasts ("ASI", solid, stars) and the persistence forecast ("Persist", dashed, circles) at different lead times. The values of three pyrheliometers are depicted as stars for the nowcasts and with circles for the persistence forecasts. The lines show the average of the results for the three pyrheliometers.

FIGURE 6 .FIGURE 7 .

 67 FIGURE 6. (a) DNI values of a reference station (green, solid), the persistence forecast (red, dashed) and the ASI-derived nowcasted values at the station's position (blue, dotted) on Nov. 22 nd 2015. Persistence forecast and nowcasts have a lead time of 10 min. (b) Bias and standard deviation for the nowcasts ("ASI", solid, stars) and the persistence forecast ("Persist", dashed, circles) at different lead times. The values of three pyrheliometers are depicted as stars for the nowcasts and with circles for the persistence forecasts. The lines show the average of the results for the three pyrheliometers.

FIGURE 8 .

 8 FIGURE 8. Statistical properties of the nowcasted DNI maps in comparison to three pyrheliometers for 9 days in September, October and November 2015. Depicted are the mean values averaged both over the days and three pyrheliometers. (a) Bias and standard deviation for the nowcasts (NC, solid) and the persistence forecast (Fcast, dashed) at different lead times. (b) RMSE and MAE for the nowcasts (ASI, solid) and the persistence forecast (Persist, dashed) at different lead times.

FIGURE 9 .

 9 FIGURE 9. Three differently sized fields corresponding to solar power plants are chosen to compare spatially averaged DNI values of the nowcasts to the reference. The position of the reference station is marked with a green star. [Google Maps] Figure 10a presents the MAE calculated for spatially averaged DNI values for the specified fields on Sept. 18 th 2015. In Fig. 10b, the corresponding RMSE of Sept. 18 th 2015 is shown (see Fig. 2a for DNI measurements of this day).Figure 11a and Fig. 11b display spatial MAE and RMSE values for Sept. 19 th 2015 (see Fig. 4a for DNI measurements of this day).

  Figure 10a presents the MAE calculated for spatially averaged DNI values for the specified fields on Sept. 18 th 2015. In Fig. 10b, the corresponding RMSE of Sept. 18 th 2015 is shown (see Fig. 2a for DNI measurements of this day).Figure 11a and Fig. 11b display spatial MAE and RMSE values for Sept. 19 th 2015 (see Fig. 4a for DNI measurements of this day).

FIGURE 10 .

 10 FIGURE 10. Statistical properties of the nowcasted DNI maps for different field sizes in comparison to the shadow camera reference system on Sept. 18 th 2015. (a) Mean absolute error for the nowcasts ("ASI", solid) and the persistence forecast ("Persist", dashed) at different lead times and different field sizes. (b) RMSE error at different lead times and different field sizes.

FIGURE 11 .

 11 FIGURE 11. Statistical properties of the nowcasted DNI maps for different field sizes in comparison to the shadow camera reference system on Sept. 19 th 2015. (a) Mean absolute error for the nowcasts ("ASI", solid) and the persistence forecast ("Persist", dashed) at different lead times and different field sizes. (b) RMSE error at different lead times and different field sizes.

ACKNOWLEDGEMENT

The research leading to these results has received funding from the European Union's FP7 Programme under Grant Agreement no. 608623 (DNICast project).


Keywords: perceptron, overlapping set visualization, antibiotherapy, user study

come

Translating visually the reasoning of a perceptron: the weighted rainbow boxes technique and an application in antibiotherapy

Translating visually the reasoning of a perceptron: the weighted rainbow boxes technique and an application in antibiotherapy

Jean-Baptiste Lamy, Rosy Tsopra LIMICS, Université Paris 13 -Sorbonne Paris Cité, UPMC Université Paris 6, Sorbonne Universités, INSERM UMRS 1142, Bobigny, France Email: jean-baptiste.lamy@univ-paris13.fr rosytsopra@gmail.com Each box covers the columns corresponding to the antibiotics having the disadvantage, and antibiotics were ordered so as to limited "holes" in the boxes. Here, a single hole is present (in the "ceftriaxone" column). The height of a box is proportional to the importance of the corresponding disadvantage, allowing the computation of a score for each drug, by visually summing the height of the boxes stacked in the drug's column. Here, lower score means better antibiotics, thus nitrofurantoin is the best treatment.

I. INTRODUCTION

The artificial neuron, also called perceptron, is a simplified model of a biological neuron, and it is the base unit for artificial neural networks (ANN) [START_REF] Abraham | Handbook of Measuring System Design[END_REF]. A perceptron has n inputs I and produces one output O (see example at the top of Figure 3). A weight w i is associated with each input I i . As a processing unit, the perceptron computes the weighted sum of its inputs, by multiplying the value of each input by its weight and summing them, and then it applies on the result the activation function f. A typical activation function is a threshold function (returning 1 if x is above the threshold, and 0 otherwise) or a sigmoid function. Consequently, the output can be computed with the following formula: O = f (I 1 ×w 1 +I 2 ×w 2 +...+I n ×w n ).

A single perceptron can achieve simple reasoning and solve linearly separable problems. On the contrary, more complex problems require several perceptrons. Although it is limited in terms of reasoning power, a single perceptron is enough to deal with some real-life problems. An example is the application of a lexicographic order or the computation of a (linear) score, e.g. "if input I 1 is true, increase the score by w 1 , if input I 2 is true, increase the score by w 2 ,...". For example, in antibiotherapy, the physician has to choose an antibiotic drug to prescribe. R Tsopra et al. [START_REF] Tsopra | An algorithm using twelve properties of antibiotics to find the recommended antibiotics, as in CPGs[END_REF] have shown that the most appropriate antibiotic can be determined by considering several antibiotic properties and then ranking the antibiotics according to their properties, considered in a lexicographic order.

A problem of perceptrons is their "black-box" nature: they produce an output but they do not explain to a Human user why the result is the obtained one. The visualization of the reasoning of a perceptron, i.e. the representation of the input values, the output value and the relation between them, is a solution to this problem. If we consider a perceptron with Boolean inputs, the representation of several input vectors can be seen as an overlapping set visualization problem (in which the elements are the input vectors, and the sets are the sets of input vectors for which a given input value is true). In this work, our objective was to improve rainbow boxes, a visualization techniques for overlapping sets [START_REF] Lamy | Rainbow boxes: a technique for visualizing overlapping sets and an application to the comparison of drugs properties[END_REF], for the visualization of the input and output values of a perceptron, and for translating visually its reasoning. The paper describes the resulting techniques, called weighted rainbow boxes, and a medical applications in antibiotherapy.

The rest of the paper is organized as follows. Section II presents related works on the visualization of ANN and overlapping sets. Section III describes the extensions we propose to rainbow boxes. Section IV describes an application in antibiotherapy. Section V describes a small user study on this application. Section VI discusses the proposed technique and its application. Finally, section VII concludes the paper with perspectives.

II. RELATED WORKS

A. Visualization of artificial neural networks

Several aspects of an ANN can be visualized: the topology of the network, the weight associated with each connection, the input and output vectors, and the evolution of the learning (which is a time-related feature, considering, for example, the evolution of the weights or of the error surface). The visualization tools proposed in the literature address one or more of these four aspects.

The weights can be visualized with Hinton diagrams [START_REF] Hinton | Distributed representations[END_REF]. The topology of the network is usually represented as an oriented graph. In addition to the topology, the Bond diagram displays the weights using colored arrows on the arc [START_REF] Wejchert | Neural network visualization[END_REF]. Specific tools have also been developed for visualizing the topology of networks in the Human brain (i.e. connectomics) [START_REF] Xia | BrainNet Viewer: a network visualization tool for human brain connectomics[END_REF]. When considering a single neuron, the input vectors and the output are usually represented using a hyperplane. The (single) output is represented by a color ranging from black to white. Two or three inputs are considered, with one dimension for each. Thus, the hyperplane is a line (for two inputs) or a plane (three inputs). For more inputs, the visualization of the hyperplane is problematic. This approach has been extended to networks made of several neurons, using several hyperplanes organized in a matrix [START_REF] Duch | Issues in intelligent systems: Paradigms[END_REF].

Learning has led to more complex visualizations. Lascaux [START_REF] Craven | Visualizing learning and computation in artificial neural networks[END_REF] displays the topology of the network and represents each neuron by a square cell, showing its current state. It can show the activation of neurons, but also the backward-propagation of error signals during learning. Another approach consisted in applying principal component analysis to the weights [START_REF] Gallagher | Visualization of learning in multilayer perceptron networks using principal component analysis[END_REF].

In conclusion, only one of the proposed approaches, the hyperplane, allows reproducing visually the reasoning of an ANN, but it is limited to 2-3 inputs.

B. Visualization of overlapping sets

Overlapping set visualization considers several elements, as well as sets containing all or part of these elements. The sets are overlapping: a given element can belong to more than one set. A typical simple example is the "amino-acid properties" problem. There are 20 amino-acids, which share physical or chemical properties. Amino-acids can be considered as elements and properties as overlapping sets of these elements. B Alsakallah et al. recently reviewed the various methods proposed for overlapping set visualization [START_REF] Alsallakh | Visualizing Sets and Set-typed Data: Stateof-the-Art and Future Challenges[END_REF] and distinguished 6 categories: Euler / Venn diagrams, overlays on a map, graphs and node-link diagrams, matrix-based techniques (rainbow boxes fit in this category), aggregationbased techniques, and scatterplot-based techniques.

Last year, we proposed a new technique: rainbow boxes [START_REF] Lamy | Rainbow boxes: a technique for visualizing overlapping sets and an application to the comparison of drugs properties[END_REF]. It targets small structured datasets (Figure 2). It displays the elements to be compared in columns, and the sets in labeled rectangular boxes that cover all the columns corresponding to the elements in the set. Larger boxes are placed at the bottom and two boxes can be side-by-side as long as they do not cover the same columns. A box can have holes, if the elements in the set are not displayed in consecutive columns. The technique is based on two hypothesis: (1) it exists an optimal order of the columns with a small number of holes (or even none at all) and (2) it is possible to find this order. Hypothesis 1 is application domain-dependent but seems to be valid in several situations. For hypothesis 2, the problem complexity is factorial. We proposed a heuristic algorithm to find, in a satisfying time, a near-optimal column order minimizing the number of holes. An example of a perceptron (top) with 3 inputs, and its visual representation using weighted rainbow boxes, for the four following (I 1 , I 2 , I 3 ) input vectors: (0, 1, 1), (1, 1, 1), (0, 0, 1) and (0, 0, 0) (bottom).

III. WEIGHTED RAINBOW BOXES

Weighted rainbow boxes extend rainbow boxes by adding per-set weights, represented graphically by an additional visual variable: the vertical size of the boxes. The weight of a set is represented by the height of the corresponding box. Figure 3 shows an example of weighted rainbow boxes.

Let's consider a category of perceptron, with (a) Boolean inputs (the false and true values being represented by 0 and 1, respectively), (b) a Boolean or positive real output, (c) strictly positive weights, (d) no bias (if really needed, the bias could still be added as the weight of an additional input which is always true), and (e) one of the two following activation functions: the threshold function f thres if the output is Boolean, and the no-op function f noop if the output is a real, defined as follows (t being a fixed parameter):

f thres (x) = 1 if x > t, 0 otherwise f noop (x) = x
The visual representation of the input vectors of a perceptron can be seen as an overlapping set visualization problem: each input vector is an element (i.e. a column in the rainbow boxes), and for each input I i , we can define the set (i.e. the box) of input vectors for which I i is true. These sets are overlapping, because a given input vector can belong to more than one set, if the input vector contains several true values. Consequently, weighted rainbow boxes can be used to visualize the input vectors as overlapping sets, using the weight of the inputs of the perceptron. Standard rainbow boxes use arbitrary "rainbow" colors; for visualizing a perceptron, we propose to use a single hue and a saturation proportional to the weight (as in Figure 3).

The boxes are stacked vertically and their height is proportional to the weights of the perceptron, consequently the total height of the stacked boxes corresponds to the weighted sum of the input values of the perceptron. Thus, it is very easy to compute visually this sum. If the activation function is f noop , the sum corresponds directly to the output value of the perceptron for the represented input vector (see O x in Figure 3). If the activation function is f thres , the threshold can be represented visually by a horizontal line, whose Y coordinate corresponds to the value of the parameter t. It is easy to determine visually if the weighted sum of the input values is higher or lower than the threshold value, i.e. if the stacked boxes are higher than the threshold line or not. Therefore, weighted rainbow boxes can at the same time provide an overview of the dataset (i.e. the input vectors), display the weights of each input and allow the visual computation of the output value for each input vector.

Gaps and holes can complicate the visual computation of the output value, because in these cases the boxes are not tightly stacked. However, in practice we observed that, at least when a single hole or gap is involved, it does not prevent the visual comparison of the output values, although it can make it more difficult.

IV. APPLICATION IN ANTIBIOTHERAPY

Doctors frequently prescribe non-optimal antibiotics, despite the associated risks: complications for the patient and emergence of bacteria resistance [START_REF] Evans | A computer-assisted management program for antibiotics and other antiinfective agents[END_REF], [START_REF] Llor | Antimicrobial resistance: risk associated with antibiotic overuse and initiatives to reduce the problem[END_REF]. In primary care, the antibiotic choice should take into account 6 antibiotic properties we identified in previous studies [START_REF] Tsopra | An algorithm using twelve properties of antibiotics to find the recommended antibiotics, as in CPGs[END_REF], [START_REF] Tsopra | Towards evidence-based CDSSs implementing the medical reasoning contained in CPGs: application to antibiotic prescription[END_REF]: the level of efficacy, the complexity of the administration protocol, the risk of emergence of bacteria resistance, the risk of adverse effects, the extent of the bacteria spectrum and the preciosity of the antibiotic class. However, in clinical practice, it is difficult for GPs to choose the best antibiotic to prescribe in a specific indication because: (i) they are not aware of all the properties of all antibiotics, (ii) the value of these properties may evolve quickly [START_REF] Alper | How much effort is needed to keep up with the literature relevant for primary care?[END_REF], (iii) they have to take a decision in a few minutes [START_REF] Deveugele | Consultation length in general practice: cross sectional study in six European countries[END_REF] whereas they don't have any tool for comparing antibiotics. Helping GPs to compare antibiotics could improve antibiotic prescriptions.

A. Knowledge base

We considered the 6 properties and 7 indications in urinary infections: cystitis in adults with risk of complications, cystitis in children, cystitis in normal situation (i.e. adults without risk of complications), prostatitis, pyelonephritis in adults with risk of complications, pyelonephritis in children, pyelonephritis in normal situation. We designed a knowledge base containing the values of each property for each antibiotic in each indication (the values of some property is indication-dependent). Since the medical reasoning of experts is based on the progressive exclusion of the worst antibiotics, we considered properties in their "negative form", each property becoming a potential disadvantage: I 1 moderate efficacy, I 2 complex protocol, I 3 risk of emergence of resistance, I 4 risk of adverse effects, I 5 broad spectrum, I 6 precious class (the I x labels correspond to the associated input of the perceptron). Each disadvantage is Boolean: either it is present (true) or not (false). The knowledge base was formalized using OWL (Ontology Web Language); it belongs to the ALIF family of description logics. The OwlReady ontology-oriented programming tool [START_REF] Lamy | Ontology-Oriented Programming for Biomedical Informatics[END_REF] was used for accessing the ontology.

B. Reasoning

For determining the optimal antibiotics for a given indication, experts start with the list of available antibiotics (i.e. potentially efficient in the indication), and then exclude all antibiotics presenting the most important disadvantage (i.e. moderate efficacy). If several antibiotics remains, experts then proceed similarly with the second most important disadvantage, and so on, until the optimal antibiotics are found. Thus, the best antibiotics can be retrieved by ranking the available antibiotics according to disadvantages, in the following lexicographic order: consider I 1 (efficacy) first, then I 2 (protocol), then I 3 (resistance) and finally I 4 (adverse effect), I 5 (spectrum) and I 6 (precious class) with an equal weight for each.

This reasoning corresponds to a single perceptron with 6 inputs (one per disadvantage), one input vector for each antibiotic (i.e. a vector of Boolean values indicating the disadvantages of the antibiotic) and one output, the ranking score (lower score being better). The perceptron remains the same for the various indications, but with different input vectors (depending on the antibiotics available for the indication and the values of the properties for the antibiotics with regards to the indication). We translated the lexicographic order between the disadvantages into the following four domain constraints on the weights:

w 1 > w 2 + w 3 + w 4 + w 5 + w 6 w 2 > w 3 + w 4 + w 5 + w 6 w 3 > w 4 + w 5 + w 6 w 4 = w 5 = w 6
Additional visual constraints were added: (1) the minimum weight is 1.0 in order to display the label inside the boxes, (2) the total sum of the weights Σw i should be kept as minimal as possible to limit the height of the interface, (3) the various weights should be different enough to allow an easy visual distinction between them. Weights were defined manually. Figure 5 shows the resulting perceptron, able to compute a ranking score for antibiotics.

C. Visualization

We used weighted rainbow boxes for visualizing the perceptron and the datasets for the 7 indications. Figures 4 and1 show examples for pyelonephritis and for cystitis with risk of complications, with respectively 5 and 10 antibiotics. Each antibiotic is an element, displayed in a column, and each disadvantage is a set containing all antibiotics having the disadvantage, and displayed as a box. We colored the boxes in red colors, because they represent the disadvantages of the antibiotics and the red color is usually associated with negative opinion in occidental cultures. The height of each box is proportional to the importance of the corresponding disadvantage, e.g. the "moderate efficacy" box has the higher height. By summing visually the box heights in a given column, it is possible to compute a score; lower scores correspond to better antibiotics for the given indication. Thus it is easy to find visually the antibiotics with the fewest and less important disadvantages, i.e. the lowest scores. It corresponds to the column with the lowest total box height, e.g. ofloxacin in Figure 4 and nitrofurantoin in Figure 1. Only one hole is present (in Figure 1, at the intersection of the ceftriaxone column and the frequent/serious adverse effect box).

V. USER STUDY

For assessing the simplicity and the utility of the application, we recruited 11 GPs. Three were women. Eight were between 30 and 40 years old, two between 40 and 50, and one more than 50 years old.

GPs had to visualize two datasets using weighted rainbow boxes: one comparing 5 antibiotics (Figure 4), and another comparing 10 antibiotics (Figure 1). For each, they had to score their feeling about 4 affirmations using a 5-level Likert scale. Then, for each dataset, GPs had to mention the antibiotic they will choose for the prescription. Finally, we asked the opinion of the GPs about having this kind of interface in clinical practice, and GPs were also allowed giving free-text comments.

The results are shown in Table I. GPs found the interface simpler for the 5-drug dataset than for the more complex 10-drug dataset (affirmation #1). However, for both datasets, they globally found that the disadvantages were easy to read (affirmation #2). Thus, when the number of drug increases, the interface becomes more complex but still remains easy to read. Displaying the disadvantages of antibiotics was considered as interesting for about 90% of GPs, and useful in clinical practice for more than 90% of them. In both datasets, 8 GPs choose to prescribe the antibiotic with the lowest ranked score, i.e. the one recommended using the knowledge base and the exclusion reasoning we described. Finally, 3 GPs would not like to have this kind of interface in clinical practice whereas 5 would like (3 neutral).

VI. DISCUSSION

In this paper, we described weighted rainbow boxes, a visualization technique derived from rainbow boxes, and we have shown that they enable a visual reasoning with a computing power equivalent to a perceptron. Weighted rainbow boxes can present and compare the various properties of several items, as original rainbow boxes did, e.g. on amino-acids, but it can also translate visually some simple reasoning based on the properties, such as a score computation or a lexicographic order for ranking the items. We also described an application in antibiotherapy. A small user study showed that the weighted rainbow boxes were rather easy to read for GPs.

A. Weighted rainbow boxes

Weighted rainbow boxes allows the visualization of the inputs and the weights of a given perceptron, and the input vectors and the output values of a dataset for this perceptron. In the literature, we have seen that few techniques have been proposed for visualizing input and output values, the most noticeable one being the hyperplane (see section II-A). The hyperplane is limited to 2 or 3 inputs, and does not show the weights explicitly. On the contrary, weighted rainbow boxes can support dozens of inputs, but they are limited to Boolean inputs with positive weights.

Several methods exist for training a perceptron or a neural network made of several perceptrons, such as backpropagation. However, we have seen that visual constraints must be satisfied in addition to the domain constraints, in order to obtain an efficient visualization. The visual constraints were related to the minimum height of the boxes, the limited height of the computer screen, the difficulty of the Human vision to compare very similar heights, and the exclusion of negative weights. Thus, learning methods should be adapted if one wants to learn the weights of the perceptron before visualization. In addition, usual limitations of the perceptron also apply, notably the impossibility to perform exclusive or.

B. Application in antibiotherapy

We presented an application for decision support for the treatment of urinary infections, based on the properties of antibiotics. In medicine, decision support systems are usually not based on treatment properties, but rather implement clinical guidelines developed by health authorities. These guidelines provide recommendations, in particular for prescribing. They are implemented in clinical decision support systems; many such systems exist [START_REF] Sintchenko | Decision support systems for antibiotic prescribing[END_REF]. However, GPs are often resistant to follow guideline recommendations because they are voluminous texts difficult to read, and they don't always understand the argumentation of experts underlying the recommendation. This is why we proposed a different approach: a visual interface comparing the different antibiotics rather than giving explicit recommendations. The visual interface we proposed could be integrated in electronic prescribing software and possibly connected to electronic health record (EHR) e.g. WebChart.

The results of the user study are rather promising. GPs seem convinced that the visualization of antibiotic disadvantages is interesting and useful for their practice, but fewer GPs found the proposed interface simple and easy to read. However, it has been shown that usability evaluations often give poor results on radically new interfaces, for which users are not accustomed to [START_REF] Greenberg | Usability evaluation considered harmful (some of the time)[END_REF].

VII. CONCLUSION

In this paper, we presented weighted rainbow boxes. We showed that these techniques can translate visually the reasoning of a perceptron. We also showed that it can be applied in antibiotherapy, with interesting results. A perspective of this work consists in improving the weighted rainbow boxes technique for the visualization of more complex neural networks (e.g. involving several perceptron or hidden units).

Figure 1 .

 1 Figure 1. Weighted rainbow boxes showing the antibiotics available for treating cystitis in adults with risk of complication, and the disadvantages of each drug. The 10 available antibiotic drugs are displayed in columns, and the 6 disadvantages they may present are represented by rectangular red boxes.Each box covers the columns corresponding to the antibiotics having the disadvantage, and antibiotics were ordered so as to limited "holes" in the boxes. Here, a single hole is present (in the "ceftriaxone" column). The height of a box is proportional to the importance of the corresponding disadvantage, allowing the computation of a score for each drug, by visually summing the height of the boxes stacked in the drug's column. Here, lower score means better antibiotics, thus nitrofurantoin is the best treatment.

Figure 2 .

 2 Figure 2. Rainbow boxes displaying 8 properties of the 20 amino-acids.

Figure 3 .

 3 Figure 3.

Figure 4 .

 4 Figure 4. Weighted rainbow boxes showing the available antibiotics and their disadvantages in the pyelonephritis indication. The table on the right shows the inputs and their weights (determined in section IV.B), and for each drug, the values of the input vector and the output.

Figure 5 .

 5 Figure 5. Perceptron for the antibiotherapy application.

Table I

 I RESULTS OF THE EVALUATION. RED MEANS "DO NOT AGREE AT ALL", LIGHT RED "NOT AGREE", GRAY "NEUTRAL", LIGHT GREEN "AGREE" AND GREEN "FULLY AGREE". THE NUMBERS INDICATES THE NUMBER OF GPS THAT GAVE THIS RESPONSE.

ACKNOWLEDGEMENTS

This work was funded by the French drug agency (ANSM, Agence Nationale de Sécurité du Médicament et des produits de santé) through the RaMiPa project (AAP-2016).