Julius Roeder

Benjamin Rouxel

Sebastian Altmeyer

Clemens Grelck

Towards Energy-, Time-and Security-Aware Multi-core Coordination

Keywords: cyber-physical systems, non-functional properties, real-time, energy, security

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Cyber-physical systems (CPS) deeply intertwine software with physical components, such as sensors and actuators that impact the physical world. Broadly speaking the software controls the actuators of a physical system based on input from the sensors and specified policies. Our world is full of cyber-physical systems, ranging from washing machines to airplanes. Designing secure, safe and correct cyber-physical systems requires a tremendous amount of verification, validation and certification.

A common characteristic of cyber-physical systems is that non-functional properties of the software, such as time, energy and security, are as important for correct behaviour as purely functional correctness. Actuators must react on sensor input within a certain time limit, or the reaction might in the worst case become useless. In addition to general environmental concerns, energy consumption of computing devices becomes crucial in battery-powered cyber-physical systems. Security concerns are paramount in many cyber-physical systems due to their potentially harmful impact on the real world. However, more security typically requires more computing effort. More computing effort takes more time and consumes more energy. Thus, time, energy and security are connected in the triangle of non-functional properties.

The multi-core revolution has meanwhile also reached the domain of cyberphysical systems. A typical example is the ARM big.LITTLE CPU architecture that features four energy-efficient Cortex A7 cores and four energy-hungry, but computationally more powerful, Cortex A15 cores. Many platforms complement this heterogeneous CPU architecture with an on-chip GPU. Such architectures create previously unknown degrees of freedom regarding the internal organisation of an application: what to compute where and when. This induces a global optimisation problem, for instance minimising energy consumption, under budget constraints, for instance in terms of time and security.

We propose the domain-specific functional coordination language TeamPlay and the associated tool chain that consider the aforementioned non-functional properties as first-class citizens in the application design and development process. Our tool chain compiles coordination code to a final executable linked with separately compiled component implementations. We combine a range of analysis and scheduling techniques for the user to choose from like in a tool box. The generated code either implements a static (offline) schedule or a dynamic (online) schedule. With static/offline scheduling all placements and activation times are pre-computed; with dynamic/online scheduling certain decisions are postponed until runtime.

Both options are driven by application-specific global objectives. The most common objective is to minimise energy consumption while meeting both time and security constraints. A variation of the theme would be to maximise security while meeting time and energy constraints. Less popular, but possible in principle, would be the third combination: minimising time under energy and security constraints.

Both offline and online scheduling share the concept of making conscious and application-specific decisions as to what compute where and when. Our work distinguishes itself from, say, operating system level scheduling by the clear insight into both the inner workings of an application and into the available computing resources.

The specific contribution of this paper lies in the design of the energy-, timeand security-aware coordination language and the overall approach. Due to space limitations we can only sketch out the various elements of our tool chain and must refer the interested reader to future publications to some degree.

The remainder of the paper is organised as follows: In Section 2 we explain our view on coordination followed by a detailed account of our (domain-specific) coordination language in Section 3. In Section 4 we illustrate our approach by means of a real-world use-case, and in Section 5 we sketch out our tool chain implementation. We discuss related work in Section 6 and conclude in Section 7.

Coordination model

The term coordination goes back to the seminal work of Gelernter and Carriero [START_REF] Gelernter | Coordination languages and their significance[END_REF] and their coordination language Linda. Coordination languages can be classified as either endogenous or exogenous [START_REF] Arbab | Composition of interacting computations[END_REF]. Endogenous approaches provide coordination primitives within application code. The original work on Linda falls into this category. Exogenous approaches fully separate the concerns of coordination programming and application programming

We pursue an exogenous approach and foster the separation of concerns between intrinsic component behaviour and extrinsic component interaction. The notion of a component is the bridging point between low-level functionality implementation and high-level application design.

Components

We illustrate our component model in Fig. 1. Following the keyword component we have a unique component name that serves the dual purpose of identifying a certain application functionality and of locating the corresponding implementation in the object code. Source components and sink components form the quintessential interfaces between the physical world and the cyber-world representing sensors and actuators in the broadest sense. We adopt the firing rule of Petri-nets, i.e. a component is activated as soon as data (tokens) are available on each of its input ports.

Technically, a component implementation is a function adhering to the C calling and linking conventions [START_REF] Ritchie | The C programming language[END_REF]. Name and signature of this function can be derived from the component specification in a defined way. This function may call other functions using the regular C calling convention. The execution of a component (function), including execution of all subsidiary regular functions, must be free of side-effects. In other words, input tokens must map to output tokens in a purely functional way. Exceptions are source and sink components that are supposed to control sensors and actuators, respectively.

Stateful components

Our components are conceptually stateless. However, some sort of state is very common in cyber-physical systems. We model such state in a functionally transparent way as illustrated in Fig. 1, namely by so-called state ports that are short-circuited from output to input. In analogy to input ports and output ports, a component may well have no state ports. We call such a component a (practically) stateless component.

Our approach to state is not dissimilar from main-stream purely functional languages, such as Haskell or Clean. They are by no means free of state either, for the simple reason that many real-world problems and phenomena are stateful. However, purely functional languages apply suitable techniques to make any state fully explicit, be it monads [START_REF] Wadler | The Essence of Functional Programming[END_REF] in Haskell or uniqueness types [START_REF] Achten | The ins and outs of Clean I/O[END_REF] in Clean. Making state explicit is key to properly deal with state and state changes in a declarative way. In contrast, the quintessential problem of impure functional and even more so imperative languages is that state is potentially scattered all over the place. And even where this is not the case in practice, proving this property is hardly possible.

ETS-aware components

We are particularly interested in the non-functional properties of code execution. Hence, any component not only comes with functional contracts, as sketched out before, but additionally with non-functional contracts concerning energy, time and security (and potentially more in the future).

These three non-functional properties are inherently different in nature. Execution time and energy consumption can be measured, depend on a concrete execution machinery and vary between different hardware scenarios. In contrast, security, more precisely algorithmic security, depends on the concrete implementation of a component, for example using different levels of encryption, etc. However, different security levels almost inevitably incur different computational demands and, thus, are likely to expose different runtime behaviour in terms of time and energy consumption as well.

Knowledge about non-functional properties of components is at the heart of our approach. It is this information that drives our scheduling and mapping decisions to solve the given optimisation problem (e.g. minimising energy consumption) under constraints (e.g. execution deadlines and minimum security requirements).

Multi-version components

As illustrated in Fig. 2, a component may have multiple versions with identical functional behaviour, but with different implementations and, thus, different energy, time and (possibly) security contracts. Multi-version components add another degree of freedom to the scheduling and mapping problem that we address: selecting the best fitting variant of a component under given optimisation objectives and constraints. Take as an example our reconnaissance drone use case, that we will explore in more detail in Section 4. A drone could adapt its security protocol for communication with the base station in accordance with changing mission state: low security while taking off or landing, medium security while navigating to/from mission area, high security during mission. Continuous adaptation of security levels results in less computing and, thus, in energy savings that could be exploited for longer flight times. Our solution is to provide different versions of the same component (similar to [START_REF] Rusu | Multi-version scheduling in rechargeable energyaware real-time systems[END_REF]) and to select the best version regarding mission state and objectives based on the scheduling strategy. For the time being, we only support off-line version selection, but scenarios with online version control, as sketched out above, are on our agenda.

Component interplay

Components are connected via FIFO channels to exchange data, as illustrated in Fig. 3. Depending on application requirements, components may start computing at statically determined time slots (when all input data is guaranteed to be present) or may be activated dynamically by the presence of all required input data. Components may produce output data on all or on selected output ports.

Coordination Language

Our coordination language focuses on the design of arbitrary synchronous dataflow-oriented applications. It describes a graph structure where vertices are components (actors, tasks) while edges represent dependencies between components. A dependency/edge defines a data exchange between a source and a sink through a FIFO channel. Such a data item, called token, can have different types, from primitive types to more elaborate structures.

Similar to periodic task models [START_REF] Liu | Scheduling algorithms for multiprogramming in a hardreal-time environment[END_REF] a data-flow graph instance is called an iteration. A job is a component instance inside an iteration. As usual we require graphs to be acyclic (i.e. DAGs). The DAG iteratively executes until the end of time (or platform shutdown). The job execution order follows the (aforementioned) constraint that job i must finish before job i+1. However, iteration j+1 can start before the completion of iteration j as long as dependencies are satisfied. This allows us to exploit job parallelism, e.g. pipelining [START_REF] Tendulkar | Many-core scheduling of data parallel applications using SMT solvers[END_REF].

Fig. 4 presents the grammar of our coordination language written in pseudo-Xtext style. In the following we describe each production rule in more detail.

Program header

Rule Application (Fig. 4, line 1) describes the root element of our application. It is composed of the application name, a deadline and a period. All times refer to one iteration of the graph; they can be given in, for instance, hours, milliseconds, hertz or clock cycles.

Rule Datatype (Fig. 4, line 9) declares the data types used throughout the coordination specification. One data type declaration consists of the type's name, followed by a string representation of its implementation in user code (i.e. the actual C type like int or struct FooBar ) and, optionally, by the size in bytes. The size information allows for further analysis, e.g. regarding memory footprint. The string representation of the type's implementation is needed for code generation.

(Multi-version) Components

A component in our coordination DSL (Fig. 4, line 11) consists of a unique name, three sets of ports and a number of additional properties. Multi-version components (see Section 2.4) feature a number of versions, where each version consists of a unique name and the additional properties, now specific to each version. The simplified syntax for single-version components is motivated by their abundance in practice.

Ports represent the interface of a component. The inports specify the data items (or tokens) consumed by a component while the outports specify the data items (or tokens) that a component (potentially) produces. The third set of ports, introduced by the keyword state, are both input ports and output ports at the same time, where the output ports are short-circuited to the corresponding input ports, as explained in Section 2.2.

A port specification includes a unique name, the token multiplicity and a data type identifier. Token multiplicities are optional and default to one. They allow components to consume a fixed number of tokens on an input port at once, to produce a fixed number of tokens on an output port at once or to keep multiple tokens of the same type as internal (pseudo) state. The firing rule for components is amended accordingly and requires (at least) the right number of tokens on each input port. Typing ports is useful to perform static type checking and to guarantee that tokens produced by one component are expected by a subsequent component connected by an edge. To start with we require type equality, but we intend to introduce some form of subtyping at a later stage, Our three non-functional properties behave differently. While the security level is an algorithmic property of a component (version), energy and time critically depend on the execution platform. Therefore, we encode the (applicationspecific) security (level) as an integer number in the code, but not energy and time information. We keep the coordination code platform-independent and obtain energy and time information from a separate data base (to be elaborated on in Section 5).

Dependencies

Dependencies (or edges) represent the flow of tokens in the graph. Their specification is crucial for the overall expressiveness of the coordination language. We support a number of constructions to connect output ports to input ports (Fig. 4, line 27). In the following we illustrate each such construction with both a graphical sketch and the corresponding textual representation. Fig. 5a presents a simple edge between the output port x of component A and the input port y of component B. In our example the output port has a multiplicity of one token while the input port has a multiplicity of two tokens. We show token multiplicities in Fig. 5a for illustration only. In the coordination program token multiplicities are part of the port specification (Fig. 4, line 18), not the edge specification (line 30). Coming back to the example of Fig. 5a, component A produces one output token per activation, but component B only becomes activated once (at least) two tokens are available on its input port. Thus, component A must fire twice before component B becomes activated.

Fig. 5b shows an extension of the previous dependency construction where component A produces a total of four tokens: one on port x and three on port y. Component B expects two tokens on input port z while sink component C expects a total of six tokens on input port q. These examples can be extended to fairly complex dependency graphs.

Fig. 5c shows a so-called broadcast edge between a source component A producing one token and two sink components B and C consuming two tokens and one token, respectively (corresponding to Fig. 4, line 32). This form of component dependency duplicates the token produced on the output port of the source component and sends it to the corresponding input ports of all sink components. Token multiplicities work in the very same way as before: any tokens produced by a source component go to each sink component, but sink components only become activated as soon as the necessary number of tokens accumulate on their input ports. A broadcast edge does not copy the data associated with a token, only the token itself. Hence, components B and C in the above example will operate on the same data and, thus, are restricted to read access.

Components with a single input port or a single output port are very common. In these cases port names in edge specifications can be omitted, as they are not needed for disambiguation. token is present on port x then component B is fired; if a token is present on port y then component C is fired. If no tokens are present on either port at the end of the execution of A then neither B nor C are fired. This enables a powerful mechanism that can be used in control programs where the presence of a stimulus enables part of the application. For example, in a face recognition system an initial component in a processing pipeline could detect if there are any person on an image. If so, the image is forwarded to the subsequent face recognition sub-algorithms; otherwise, it is discarded.

Fig. 6b allows conditional dependencies driven by the scheduler (corresponding to Fig. 4, line 37. Similar to the previous case, component B and component C depend on component A, but only one is allowed to actually execute depending on a decision by the scheduler. For example, if the time budget requested by component B is lower than that requested by component C, the scheduler can choose to fire component B instead of C. Such a decision could be motivated by the need to avoid a deadline miss at the expense of some loss of accuracy.

Fig. 6c allows conditional dependencies driven by the user (corresponding to Fig. 4, line 39). In this case components B and C again depend on component A, but this time the dependency is guarded by a condition. If the condition evaluates to true then the token is sent to the corresponding route. There is no particular evaluation order for conditions, and tokens are simultaneously sent to all sink components whose guards evaluate to true. Like in the case of the broadcast edge all fired components receive the very same input data. If no guard returns true, the token is discarded.

The guards come in the form of strings as inline C code. The code generator will literally place this code into condition positions in the generated code. The user is responsible for the syntactic and semantic correctness of these C code snippets. This is not ideal with respect to static validation of coordination code, but similar to, for instance, the if-clause in OpenMP. On the positive side, this feature ensures maximum flexibility in application design without the need for a fully-fledged C compiler frontend, which would be far beyond our means.

For example, the Cexpr could contain a call to a function get_battery that enquires about the battery charge status. The coordination program may choose to fire all subsequent components as long as the battery is well charged, but only some as the battery power drains. Or, it may fire different components altogether, changing the system behaviour under different battery conditions.

Example use case reconnaissance drone

We illustrate our coordination approach by means of a use case that we develop jointly with our project partners University of Southern Denmark and Sky-Watch A/S [START_REF] Seewald | Component-based computation-energy modeling for embedded systems[END_REF]. Fixed-wing drones can stay several hours in the air, making them ideal equipment for surveillance and reconnaissance missions. In addition to the flight control system keeping the drone up in the air, our drone is equipped with a camera and a payload computing system. Since fixed-wing drones are highly energy-efficient, computing on the payload system does have a noticeable impact on overall energy consumption and, thus, on mission length. We illustrate our coordination approach in Fig. 7; the corresponding coordination code is shown in Fig. 8. We re-use the original application building blocks developed and used by Sky-Watch A/S. The drone's camera system takes pictures in predefined intervals. Our Im-ageCapture component represents this interface to the physical world. Global period and deadline specifications correspond to the capture frequency of the camera. The non-standard data types declared in the datatypes section of the coordination program are adopted from the original application code. We use the C types in string form for code generation and require that corresponding C type definitions are made available to the backend C compiler via header files.

Images are broadcast to three subsequent components. The VideoEncryption component encrypts the images of the video stream and forwards the encrypted images to follow-up component SaveVideo that stores the video in persistent memory for post-mission analysis and archival. Video encryption comes with three different security levels. For simplicity we just call them Encryption1, En-cryption2 and Encryption3. Different encryption levels could be used, for instance, for different mission environments, from friendly to hostile.

The drone also performs on-board analyses of the images taken. These are represented by our components ObjectDetector and GroundSpeed. Object detection can choose between three algorithms with different accuracy, time and energy properties: Darknet 3 , Tiny Darknet 4 , OpenCV. The ground speed estimator works by comparing two subsequent images from the video stream. This is the only stateful component in our model. The results of object detection and ground speed estimation are synchronised and fed into the follow-up component Decision that combines all information and decides whether or not to notify the base station about a potentially relevant object detected.

Transmission of the message is modelled by the sink component SendMessage, where the action returns to the physical world. To implement dynamic adaptation to dynamically changing mission phases, as sketched out in Section 2.4, we would need multiple versions of this component with different security levels as well. However, we leave dynamic adaptation to future work for now.

As Fig. 8 demonstrates, our coordination language allows users to specify non-trivial cyber-physical applications in a concise and comprehensible way. The entire wiring of components only takes a few lines of code. Our approach facilitates playing with implementation variations and, thus, enables system engineers to explore the consequences of design choices on non-functional properties at an early stage. Note that all ports in our example have a token multiplicity of one, and we consistently make use of default ports where components only feature a single input port or a single output port.

Coordination tool chain

Coordination file

Non-Functional Properties files

Syntactic & semantic analyses

Scheduling policy generator

Code generator

Components object files

Target compiler & linker

Binary file

Config file

Coordination Compiler

Fig. 9: Coordination workflow Figure 9 illustrates our coordination tool chain; its four main inputs are:

1. the coordination program, as described in Section 3; 2. timing and energy information per component: provided by timing/energy harvesting tools such as AbsInt aiT [START_REF] Ferdinand | aiT: worst-case execution time prediction by static program analysis[END_REF] for a specific architecture; 3. object files: provided by a C-compiler such as WCC [START_REF] Falk | Design of a wcet-aware C compiler[END_REF], containing binary code for each component (version). 4. a config file with configuration information, e.g. target hardware, security-level mission specifications, compiler passes to apply, etc.

For syntactic and semantic analysis, we use the parser generator ANTLR to derive a C++ parser from an Xtext grammar specification that is very similar to the one shown in Fig. 4. This implementation choice provides us with a graphical editor plug-in for the Eclipse IDE for free 5 ports refer to well defined data types; edges connect existing components; edges connect output ports with input ports; versions target available architectures.

Type checking entails validating that output and input ports connected by an edge use equivalent types. Using standard graph terminology this can be formalised as ∀src, sink ∈ E :

src type = sink type (1)
Deadlock checking in our context entails static detection of stable token consumption/production rates. Formally, the number of tokens produced by a component (vertex) must coincide with the sum of tokens expected by all successor components: ∀v ∈ V :

v prod = p∈Vsucc p cons (2)
Likewise, the number of tokens consumed by a component must match the sum of tokens produced by all predecessor components:

∀v ∈ V : v cons = p∈v pred p prod (3)
The second block of our coordination tool chain in Fig. 9 is the scheduling policy generator, which depends on configuration parameters provided by the user. In the case of static offline scheduling, the scheduling policy generator generates a schedule table with locations and release times for each component [START_REF] Roeder | Interdependent multi-version scheduling in heterogeneous energy-aware embedded systems[END_REF][START_REF] Rouxel | Hiding communication delays in contention-free execution for SPM-based multi-core architectures[END_REF]. In the case of dynamic online scheduling it performs a schedulability analysis for which we have adapted the techniques of Melani et al [START_REF] Melani | Response-time analysis of conditional dag tasks in multiprocessor systems[END_REF] or, alternatively, those of Casini et al [START_REF] Casini | Partitioned fixed-priority scheduling of parallel tasks without preemptions[END_REF].

Offline and online schedulers both have their specific benefits and drawbacks: offline schedulers are easy to implement (e.g. with alarms) and, as all release times are decided a-priori, scheduling overhead is minimal. However, offline schedulers are not work-conserving. Should a component finish quicker than suggested by its worst-case execution time, the corresponding core stays idle until the subsequent release time of some component. In contrast, online schedulers are work-conserving and, thus, more efficient in practice. However, this efficiency comes at the cost of higher runtime overhead and implementation difficulty since we need a mechanism that decides at runtime which component to execute next.

Whether to opt for offline or online scheduling depends on the application scenario at hand. Our tool chain merely facilitates users to make this choice. For offline scheduling we provide both an ILP-based solution [START_REF] Roeder | Interdependent multi-version scheduling in heterogeneous energy-aware embedded systems[END_REF] and a heuristic for larger use cases, where the solving an ILP proves to be too time-consuming.

Code generation is the final step in our tool flow. For the coordination part of an application, we generate C-code that manages components and their interaction through threads and processes according to the configured scheduling policy, including releasing, synchronisation, and communication of components. In a final step the generated C-code is compiled by a platform-specific C compiler and linked with the likewise compiled component implementations into an executable binary, ready to be deployed to the platform of choice.

We successfully applied our tool chain to the drone use-case presented in Section 4. At the time of writing we are able to generate a static schedule (both ILP-and heuristics-based) that optimises the overall energy consumption while meeting all time and security constraints. Our project partner Sky-Watch A/S successfully tested this code on an actually flying drone. We are still in the process of evaluating the outcome of these experiments compared to the original hand-coded software of Sky-Watch A/S. We envision in the very near future to have our code generator ready to produce dynamically scheduled applications.

Related work

Coordination is a well established computing paradigm with a plethora of languages, abstractions and approaches, surveyed in [START_REF] Ciatto | Twenty years of coordination technologies: State-of-the-art and perspectives[END_REF]. Yet, we are neither aware of any adoption of the principle in the broader domain of mission-critical cyberphysical systems, nor are we aware of energy-, time-or security-aware approaches to coordination similar to our approach.

In the area of exogenous coordination languages we mention the work on Reo [START_REF] Arbab | Reo: a channel-based coordination model for component composition[END_REF]. The objective of Reo is in the modelling and formal property verification of coordination protocols. Reo has a graphical syntax, in which every Reo program is a labeled directed hypergraph. Reo further has a (or rather many) formal semantics [START_REF] Jongmans | Overview of thirty semantic formalisms for Reo[END_REF]. Compared to our work, Reo is a much more theoretical approach to exogenous coordination, whereas our objective lies in the creation of a practical (and pragmatic) DSL to create executable energy-, time-and security-aware programs running on concrete machinery.

Another example of an exogenous coordination language is S-Net [START_REF] Grelck | Asynchronous stream processing with S-Net[END_REF], from which we draw inspiration and experience for our proposed design. However, S-Net merely addresses the functional aspects of coordination programming and has left out any non-functional requirements, not to mention energy, time and security, in particular.

A notable exception in the otherwise fairly uncharted territory of resourceaware (functional) languages is Hume [START_REF] Hammond | Hume: a domain-specific language for real-time embedded systems[END_REF]. Hume was specifically designed with real-time systems in mind, and, thus, guarantees on time (and space) consumption are key. However, the main motivation behind Hume was to explore how far high-level functional programming features, such as automatic memory management, higher-order functions, polymorphism, recursion, etc can be supported while still providing accurate real-time guarantees.

Bondavalli et al. [START_REF] Bondavalli | Dataflow-like languages for real-time systems: issues of computational models and notation[END_REF] present a simple in-the-large programming language to describe the structure of a graph-based application. However, they only model what we call components and simple edges, whereas their simple language neither accounts for multi-version components nor for complex communication structures, not to mention any notion of non-functional properties.

A term related to coordination is algorithmic skeletons. Merely as examples we mention FastFlow [START_REF] Aldinucci | Fastflow: high-level and efficient streaming on multicore[END_REF] and Musket [START_REF] Rieger | Musket: A domain-specific language for highlevel parallel programming with algorithmic skeletons[END_REF]. Again, all work in this area that we are aware of in one way or another focuses on the trade-off between programming efficiency and execution performance, whereas our focus is on energy, time and security as non-functional properties.

Lustre [START_REF] Halbwachs | The synchronous data flow programming language LUSTRE[END_REF][START_REF] Benveniste | The synchronous languages 12 years later[END_REF] was designed to program reactive system, such as automatic control and monitoring systems. In contrast to general-purpose programming language, Lustre models the flow of data. The idea is to represent actions done on data at each time tick, like in an electronic circuit. The tick can be extended to represent periods and release times for tasks, but still an action is required to describe outputs for each tick (like reusing the last produced data).

Lustre is synchronous which seems necessary for time-sensitive applications. However, Lustre does not decouple the program source code from its structure. The flow of data is extracted by the compiler through data dependencies of variables. We aim at expressing the flow of data with a much simpler and more explicit approach. We also act at a higher level by focusing on the interaction of components considered as black boxes.

In [START_REF] Alras | Model-based design of embedded control systems by means of a synchronous intermediate model[END_REF] Lustre is extended by meta-operators to integrate a complete modelbased design tool from a high-level Simulink model to a low-level implementation. Still, this extension, called Lustre++, does not separate the design of the program structure from actual feature implementation and remains at a too low level to only represent application structure as we intend to do.

The StreamIT [START_REF] Thies | StreamIt: A language for streaming applications[END_REF] language also describes graph-based streaming applications, but it is restricted to fork-join graphs while we need to support arbitrary graphs, possibly with multiple sources and/or sinks.

The Architecture Analysis & Design Language (AADL) [START_REF] Feiler | The architecture analysis and design language (AADL): An introduction[END_REF] targets realtime system design. It provides formal modeling concepts for the description and analysis of application architectures in terms of distinct components and their interactions. AADL supports early prediction and analysis with respect to performance, schedulability and reliability.

Conclusion

We propose the TeamPlay coordination language and component technology for the high-level design and development of cyber-physical systems. Our coordination DSL allows users to specify non-trivial streaming applications in a few lines of code while treating crucial non-functional properties such energy, time and security as first-class citizens throughout the application design process.

We describe a complete tool flow from syntactic and semantic validation of coordination programs to code generation for typical off-the-shelf heterogeneous multi-core hardware for cyber-physical systems. Our tool flow includes a variety of offline and online scheduling and mapping techniques that form a tool box, from which the user can choose the most appropriate combination with respect to application needs. We apply our approach to a real-world use case: a mission-critical reconnaissance drone. We demonstrate the merits of our approach in terms of specification conciseness. An initial version of our tool chain is functional, and we have run preliminary experiments on an actually flying drone. However, the outcome of these experiments is still under analysis and beyond the scope of this paper.

Our work continues in multiple directions. We currently work on a number of further application use cases, among others a car park monitoring system, a satellite communication system and a camera pill application from the medical domain. Further experience with these additional use cases will most likely motivate us to refine the design of our coordination DSL.

Implementation-wise we plan to extend and refine the various scheduling and mapping options. Our code generator currently expects a Linux-like environment with a certain level of operating system support. This is a realistic assumption for many cyber-physical systems, but others run in more bare-metal environments, e.g. where the form factor requires minimal computing hardware. Our more long-term vision is to adapt our coordination technology for safety-critical applications that must be secured against component failure or cyber attacks.

Fig. 1 :

 1 Fig. 1: Illustration of component model A component interacts with the outside world via component-specific numbers of typed and named input ports and output ports. As the Kleene star in Fig. 1 suggests, a component may have zero input ports or zero output ports. A component without input ports is called a source component; a component without output ports is called a sink component.Source components and sink components form the quintessential interfaces between the physical world and the cyber-world representing sensors and actuators in the broadest sense. We adopt the firing rule of Petri-nets, i.e. a component is activated as soon as data (tokens) are available on each of its input ports.Technically, a component implementation is a function adhering to the C calling and linking conventions[START_REF] Ritchie | The C programming language[END_REF]. Name and signature of this function can be derived from the component specification in a defined way. This function may call other functions using the regular C calling convention. The execution of a component (function), including execution of all subsidiary regular functions, must be free of side-effects. In other words, input tokens must map to output tokens in a purely functional way. Exceptions are source and sink components that are supposed to control sensors and actuators, respectively.

Fig. 2 :

 2 Fig. 2: Multi-version component with individual energy, time and security contracts

Fig. 3 :

 3 Fig. 3: Illustration of data-driven component interplay via FIFO channels

Fig. 5 :

 5 Fig. 5: Various edge construction examples

Fig. 6 :

 6 Fig. 6: Data-, scheduler-and environment-dependent edges

Fig. 7 :

 7 Fig. 7: Reconnaissance drone use case coordination model

Fig. 8 :

 8 Fig. 8: Coordination program for drone use case

  . The resulting parser validates the

	app drone {
	deadline 50 Hz
	period 50 Hz
	datatypes {
	( frame, " jpegFrame *")
	( num, " uint32_t ")
	( enc, " encryptedData *")
	( string, " char *")
	}
	components {
	ImageCapture { outports [ ( out, frame ) ] }
	Encryption {
	inports [ ( in, frame ) ]
	outports [ ( out, enc ) ]
	version Encryption1 { security 4}
	version Encryption2 { security 6}
	version Encryption3 { security 9}
	}
	ObjectDetector {
	inports [ ( in, frame ) ]
	outports [ ( obj, num ) ( frame, frame ) ]
	version TinyDarknet { arch " cpu / big "}
	version Darknet { arch " cpu gpu "}
	version OpenCV { arch " cpu gpu "}
	}
	GroundSpeed {
	inports [ ( in, frame ) ]
	outports [ ( speed, num ) ]
	state [ ( s, frame ) ]
	}
	Decision {
	inports [( obj, num ) ( frame, frame ) ( speed, num ) ]
	outports [ ( msg, string ) ]
	}
	SaveVideo { inports [ ( in, enc ) ] }
	SendMessage { inports [ ( msg, string ) ] }
	} edges {
	ImageCapture -> Encryption & ObjectDetector & GroundSpeed
	Encryption -> SaveVideo
	ObjectDetector . obj -> Decision . obj
	ObjectDetector . frame -> Decision . frame
	GroundSpeed -> Decision . speed
	Decision -> SendMessage
	} }

https://pjreddie.com/darknet/

https://pjreddie.com/darknet/tiny-darknet/

https://www.eclipse.org/Xtext/

This work is supported and partly funded by the European Union Horizon-2020 research and innovation programme under grant agreement No. 779882 (TeamPlay).

code <name> ETS-contracts code <name> ETS-contracts code <name> ETS-contracts code <name> ETS-contracts code <name> ETS-contracts code <name> ETS-contracts code <name> ETS-contracts code <name>


Espen Tønnessen Nordli
email: espentno@ifi.uio.no

Phu H Nguyen

Franck Chauvel

Hui Song

Event-based Customization of Multi-tenant SaaS Using Microservices

Keywords: Microservices, Architecture, Event-based, Cloud, SaaS, Customization, IoT, Edge, Security

Popular enterprise software such as ERP, CRM is now being made available on the Cloud in the multi-tenant Software as a Service (SaaS) model. The added values come from the ability of vendors to enable customer-specific business advantage for every different tenant who uses the same main enterprise software product. Software vendors need novel customization solutions for Cloud-based multi-tenant SaaS. In this paper, we present an event-based approach in a non-intrusive customization framework that can enable customization for multi-tenant SaaS and address the problem of too many API calls to the main software product. The experimental results on Microsoft's eShopOnContainers show that our approach can empower an event bus with the ability to customize the flow of processing events, and integrate with tenant-specific microservices for customization. We have shown how our approach makes sure of tenant-isolation, which is crucial in practice for SaaS vendors. This direction can also reduce the number of API calls to the main software product, even when every tenant has different customization services.

Introduction

Most businesses and public services rely on enterprise software such as enterprise resource planning (ERP) or customer relationship management (CRM), to name a few. Because every company has its unique organization, processes and culture, no off-the-shelf software directly fits. Companies eventually customize these software systems to meet their specific requirements. For simple scenarios, software vendors predict where and how their software products may

The research leading to these results has received funding from the European Commission's H2020 Programme under the grant agreement number 780351 (ENACT), and from the Research Council of Norway under the grant agreement numbers 296651 (ASAM) and 256594 (Cirrus).

be customized, and provide their customers with application programming interfaces (API), extension points or configuration choices. However, there are always customers whose requirements overstep the embedded customization capacity. These customers need the vendors to provide mechanisms for performing deep customization, that goes beyond the vendors' prediction.

Deep customization may affect any parts of a software product, including the user interface (UI), the business logic (BL), the database schemas (DB) or any combination thereof. When a software product used to be deployed on the customers' premises, each customer naturally ran its own customized version, in full isolation. Nowadays, software vendors are migrating their software products to the Cloud. In the Cloud-based multi-tenant software-as-a-service (SaaS) model, however, every customer must run the same code base (main product), which cannot be directly modified for one customer without affecting other customers. Software vendors desperately need novel deep customization solutions for the Cloud-based multi-tenant SaaS model.

More recently, leveraging the microservices architecture [START_REF] Thönes | [END_REF][START_REF] Newman | Building microservices: designing fine-grained systems[END_REF][START_REF] Dragoni | Microservices: Yesterday, Today, and Tomorrow[END_REF] for enabling deep customization of multi-tenant SaaS is a very promising direction as presented in [START_REF] Song | Deep customization of multi-tenant saas using intrusive microservices[END_REF][START_REF] Song | Customizing multi-tenant saas by microservices: A reference architecture[END_REF][START_REF] Song | Using Microservices to Customize Multitenant Software-as-a-Service[END_REF][START_REF] Nguyen | Using microservices for non-intrusive customization of multi-tenant saas[END_REF][START_REF] Song | Using Microservices to Customize Multi-Tenant SaaS: From Intrusive to Non-Intrusive[END_REF]. These microservices-based customization approaches vary in how they balance isolation and assimilation. Isolation guarantees tenantspecific customization only affects that one single tenant, whereas assimilation guarantees that customization capability can alter anything in the main software product. Intrusive microservices [START_REF] Song | Deep customization of multi-tenant saas using intrusive microservices[END_REF][START_REF] Song | Customizing multi-tenant saas by microservices: A reference architecture[END_REF][START_REF] Song | Using Microservices to Customize Multitenant Software-as-a-Service[END_REF] provide tight assimilation at the cost of security (tenant isolation), whereas the non-intrusive approach called MiSC-Cloud [START_REF] Nguyen | Towards customizing multi-tenant cloud applications using non-intrusive microservices[END_REF][START_REF] Nguyen | Using microservices for non-intrusive customization of multi-tenant saas[END_REF][START_REF] Song | Using Microservices to Customize Multi-Tenant SaaS: From Intrusive to Non-Intrusive[END_REF] trades assimilation for higher security. MiSC-Cloud orchestrates customization using microservices via API gateways.

In this paper, we present an event-based non-intrusive deep customization approach for multi-tenant SaaS using microservices as part of the MiSC-Cloud framework [START_REF] Nguyen | Using microservices for non-intrusive customization of multi-tenant saas[END_REF]. The event-based approach, in combination with the synchronous way of customization in [START_REF] Nguyen | Using microservices for non-intrusive customization of multi-tenant saas[END_REF], shows how the MiSC-Cloud framework can coordinate the execution of the BL components (microservices) of the main product as well as the customization microservices of tenants to obtain the desired customization effects in the multi-tenant context.

The remainder of this paper is structured as follows: Section 2 defines deep customization. Then, Section 3 presents the event-based customization approach with key techniques. In Section 4, we show a proof-of-concept for the proposed approach by applying it on a reference application for microservice architecture by Microsoft. Section 5 discusses related work. Finally, we provide in Section 6 our conclusions and possible future research directions.

Deep Customization

By contrast with other customization means such as settings, scripting languages or API, deep customization demands that one can possibly make any change to the system, as one can do with direct access to the source code. Changes can, therefore, affect the user interface (UI), the business logic (BL), the database schema (DB), or any combination thereof. Deep customization turns out difficult in multi-tenant SaaS environments, where all tenants originally run the same code (UI, BL and DB). Tenant-specific customization must affect only one single tenant. This work focuses on the customization of BL, especially based on events. In this way, customization microservices communicate with the main product, either in a synchronous way by requesting data and waiting for the response (RPC-like), or in an asynchronous way, by publishing and subscribing to events (pub/sub). The customization of UI and DB can be found in [START_REF] Song | Deep customization of multi-tenant saas using intrusive microservices[END_REF][START_REF] Song | Customizing multi-tenant saas by microservices: A reference architecture[END_REF][START_REF] Song | Using Microservices to Customize Multitenant Software-as-a-Service[END_REF][START_REF] Nguyen | Using microservices for non-intrusive customization of multi-tenant saas[END_REF][START_REF] Song | Using Microservices to Customize Multi-Tenant SaaS: From Intrusive to Non-Intrusive[END_REF].

Event-based Customization Approach

In this section, we first present the main components for enabling event-based customization of multi-tenant SaaS in Section 3.1. Then, Section 3.2 details how the event-based customization approach works. In Section 3.3, we discuss how the event-based customization fulfils the requirements of tenant isolation.

Main Components for Enabling Event-based Customization

Among the five main components of the MiSC-Cloud framework as presented in [START_REF] Nguyen | Towards customizing multi-tenant cloud applications using non-intrusive microservices[END_REF][START_REF] Nguyen | Using microservices for non-intrusive customization of multi-tenant saas[END_REF], we focus on presenting the Tenant Manager and the Event Bus as the key parts of the event-based customization approach. The API gateways, IAM Service, and WebMVC Customizer are the same as we described in [START_REF] Nguyen | Using microservices for non-intrusive customization of multi-tenant saas[END_REF].

The Tenant Manager is a service that manages the registration of customization microservices including the events registered for customization for different tenants. The service has a simple database that stores all the tenants that are using the application, all the different events that exist in the main product and finally all the customization microservices that exist for tenants and specific events. Additionally, it stores an endpoint for each customization that is used for halting the flow of events to be discussed further in the next section.

The Event Bus is key to enable event-based customization. Therefore, the prerequisite for enabling event-based customization is that the main product already has (part of) its logic flow orchestrated via events. If the main product already has an Event Bus, such an Event Bus can be extended to enable eventbased customization. If the main product does not have an Event Bus, a new one can be introduced as presented in [START_REF] Nguyen | Using microservices for non-intrusive customization of multi-tenant saas[END_REF]. It is important to note that a software product can be re-engineered to enable event-based logic orchestration at the back-end via an Event Bus. Different migration approaches from monolithic to microservices architecture already show some patterns and practices to migrate from synchronous calls into event-based communication between microservices [START_REF] Mazzara | Microservices: Migration of a mission critical system[END_REF][START_REF] Taibi | From monolithic systems to microservices: An assessment framework[END_REF]. Moreover, software vendors can also create user or system events within their software product to allow authorized event-based integration with external systems (of their customers). This event-based integration is similar to the traditional way of offering a rich REST API for synchronous integration, e.g., using traditional GET-PUT-POST statements.

Event-based Customization Flow

A customization microservice can subscribe to an event that is published to the Event Bus when something notable happens, such as when another microservice (of the main product or another tenant-specific customization) updates a business entity. When a microservice receives an event, it can update its business entities, which might lead to the publishing of more events. We design the event bus as a multi-tenant interface with the tenant-specific APIs needed to subscribe and unsubscribe to events and to publish events.

The flow of processing events in the original Event Bus implementation must be changed for customization purposes. Before publishing events to the consumers, it checks with the Tenant Manager for any customization that has been registered for any event and tenant (see Figure 1). If an event is not customized, then the event is processed in the standard fashion. In the case that an event is customized, the event is sent to the endpoint that is part of the response from the Tenant Manager. At this point, the tenant's microservice is responsible for storing the event until the required customization has been achieved. Then, the tenant's microservice can republish the event to the Event Bus, along with a flag that instructs the Event Bus to not check for customization again, to avoid an infinite loop. In some cases, customization microservices would require some execution context from the main product that does not exist in the events that they receive. To obtain such context, customization microservices can make authorized synchronous calls to the APIs of the main product as presented in [START_REF] Nguyen | Using microservices for non-intrusive customization of multi-tenant saas[END_REF]. In fact, events often contain enough execution context for customization microservices to execute customization scenarios. This means that only a few special customization scenarios would require such synchronous calls from customization microservices to the API of the main product. Combining the synchronous and asynchronous ways of customization can offer a more complete non-intrusive customization approach for multi-tenant SaaS. However, we recommend the use of event-based customization for as many customization scenarios as possible to reduce the traffic of API calls to the main product, which often leads to performance bottleneck when there are many customized tenants with unpredictable loads.

Tenant-isolation and Tenant-specific Event-Handlers

The Event Bus implementation and architecture in the main product must ensure that tenant isolation is still preserved. Instead of having one connection to a single event bus, there must be multiple connections, one per tenant. One example of such an event bus implementation is based on RabbitMQ that can make use of virtual hosts3 . This way allows us to have a logical separation per tenant, and the permission can easily be set so that each tenant is only allowed to interact with its own virtual host.

Proof-Of-Concept and Evaluation

In this section, we show a proof of concept of our approach for enabling deep customization of the eShopOnContainers by extending the Event Bus in the application. The .NET Microservices Sample Reference Application eShopOn-Containers4 has been chosen for a couple of reasons. First, eShopOnContainers has a clear separation between the user interface and the business logic of the application as a prerequisite of the MiSC-Cloud framework. Secondly, the application follows the microservices architecture, and as such, has loose coupling as compared to a monolithic application. Finally, the collaboration between the microservices that the application as a whole is made up of is done using events and a publish/subscribe system.

An Event Bus implementation must be associated with the authentication and authorization mechanisms of the IAM service for multi-tenant SaaS-based on Open ID Connect or OAuth 2.0. As an implementation of RabbitMQ already exists in the eShopOnContainers, we have extended it to enable event-based customization.

Let us consider the original eShopOnContainers in the GitHub repository as the main product being customized. We show how our event-based customization approach can enable different customization scenarios for two tenants as the representatives of multi-tenant context 5 . The first use case in Section 4.1 adds new logic to the main flow of the ordering process, without altering any of the existing functionality. The second use case in Section 4.2 requires a modification of the existing logic of the ordering process by halting the flow of the order.

Tenant A's Customization of the Ordering Process

The original ordering process is straightforward. After having logged in, a customer can add items in the shopping cart and then create an order with card payment and shipping address. What happens at the back-end is that the Basket service of the eShopOnContainers publishes a UserCheckoutAcceptedIntegra-tionEvent, which is consumed by the Ordering service to create and process the order, e.g., generating OrderSubmittedIntegrationEvent. Tenant A wants to change the original ordering process of eShopOnContainers to incorporate the shipping information from external (third-party) systems. This means that after the Basket service has published a UserCheckoutAcceptedIntegrationEvent, the Ordering service validates the order request before creating an order and an OrderSubmittedIntegrationEvent to trigger this customization. Here, we demonstrate the customization of Tenant A using the asynchronous way. The synchronous way of customization has been presented in [START_REF] Nguyen | Using microservices for non-intrusive customization of multi-tenant saas[END_REF].

The asynchronous way of customization has been used for the customization scenario in which the user has checked out (UserCheckoutAcceptedInte-grationEvent), and the corresponding order has been made (OrderSubmitted-IntegrationEvent). The customization microservice Shipping of Tenant A intercepts the OrderSubmittedIntegrationEvent and queries an external system for an estimated time for delivery. This information is then stored in the microservice's database, which can then be retrieved whenever the My Orders page is displayed. The customization result can be seen in Figure 2. The parts in red, e.g., SHIPPING DATE, are the customized content, which are only available for the users of Tenant A. What happens in the background is that we have added a new Event Handler that consumes the OrderSubmittedIntegrationEvent. Whenever this event is published by the main product to the event bus of Tenant A, the Event Handler consumes the event and calls the customization microservice Shipping, which is responsible for calculating the shipping information by integrating with an external system.

Tenant B's Customization of the Ordering Process

Tenant B wants to customize the ordering process with some additional steps to mark all the items with RFID. Before the order status is set to confirmed, all the order lines in the order should be scanned. Further, the order status should only be set to confirmed when all the items in the order have been scanned.

The second use case requires that the status of the order is not set to confirmed until all the items in the order have been scanned. To ensure this, we need to halt the flow of the application by capturing the OrderStatus-ChangedToAwaitingValidationIntegrationEvent. This is done by registering this event for the specific tenant in the Tenant Manager, as well as the endpoint that we want the event to be sent to. Figure 3 shows the customization flow triggered by the OrderStatusChangedToAwaitingValidationIntegrationEvent. This event is then stored in the database of the microservice for this customization until the RFIDTagScannedIntegrationEvent is published by the TenantARFIDService. The customization scenario depicted in Figure 3 starts when the Ordering service publishes the OrderStatusChangedToAwaitingValidationIntegration-Event. Next, the Event Bus implementation checks for any customization for this event by querying the Tenant Manager. As Tenant B has customized this event, the Event Bus sends the event to the endpoint specified in the response from the Tenant Manager rather than publishing to the RabbitMQ instance. At this point, the tenant has control of the event and can save it to the local database of Tenant B's Event Service before publishing OrderStatusChanged-ToAwaitingValidationEventSavedEvent to the Event Bus. The OrderStatus-ChangedToAwaitingValidationEventSavedEventHandler in Tenant B's RFID Service consumes this event, and stores the necessary data in its database. The next step of the use case is triggered whenever the endpoint in Tenant B's RFID Service is used to indicate that all the order lines have been scanned. The use of this endpoint also triggers RFIDTagScannedIntegration-Event, which is then consumed by the RFIDTagScannedIntegrationEvent-Handler in Tenant B's Event Service. At this point, the original OrderStatus-ChangedToAwaitingValidationIntegrationEvent is re-published to the Event Bus, and the handlers in the main product can perform their operations. Then, the event is re-published to the Event Bus, and it is processed normally by the main product. The result of the customization, after the RFID tags are scanned can be seen in Figure 4.

The asynchronous customization approach is based on the events in the application. Because all the events are isolated so that each tenant is only able to interact with their own events. This means that tenant isolation is still preserved.

Related Work

The notion of customizable SaaS applications with explicit support for variability management has been proposed and explored extensively [START_REF] Kabbedijk | Defining multi-tenancy: A systematic mapping study on the academic and the industrial perspective[END_REF]. There are many technical approaches addressing these complexities, such as design patterns, dependency injection (DI), software product lines (SPL), or API. While these approaches help predefining customization at design time, they do not have sufficient support for the complex and unanticipated behavioural coordination between the custom code and the main product at runtime.

The majority of SaaS customization approaches focus on a high-level modification of the service composition. Mietzner and Leymann [START_REF] Mietzner | Generation of BPEL customization processes for SaaS applications from variability descriptors[END_REF] present a customization approach based on the automatic transformation from a variability model to BPEL process. Here customization is a re-composition of services provided by vendors. Tsai and Sun [START_REF] Tsai | Saas multi-tenant application customization[END_REF] follow the same assumption but propose multiple layers of compositions. All the composite services are customizable until reaching atomic services, which are assumed to be provided by the vendors.

Middleware techniques can also support the customization of SaaS. Guo et al. [START_REF] Guo | A framework for native multitenancy application development and management[END_REF] discuss, in a high abstraction level, a middleware-based framework for the development and operation of customization, and highlighted the key challenges. Walraven et al. [START_REF] Walraven | A middleware layer for flexible and costefficient multi-tenant applications[END_REF] implemented such a customization, enabling middleware us-ing Dependency Injection. The dependency injection way for customization allows the custom code developers to introduce arbitrary coordination behaviour with the main product, and thus achieve a strong expression power. However, it also brings tight coupling between the custom code and the main product. Operating the custom code as an external microservice eases performance isolation, misbehaviour of the custom code only fails the underlying container, and the main product only perceives a network error, which does not affect other tenants. Besides, external microservices ease management: scaling independently resource-consuming customization and eventually billing tenants accordingly.

Conclusions

In this paper, we have presented an event-based customization approach that is part of our non-intrusive customization framework for multi-tenant SaaS. This asynchronous way of customization means that customization microservices can have event-based communication with the main product BL components for customization purposes. Using event-based communication between customization microservices and the main product BL components is important not only for the microservices architecture but also for non-intrusive deep customization capability. Enabling customization both synchronously and asynchronously provides a more flexible way of coordinating the customization logic between the BL components (microservices) of the main product and the customization microservices of tenants to obtain the desired customization effects in the multi-tenant context. Our event-based customization approach makes sure of tenant-isolation, which is crucial in practice for SaaS vendors. This approach can also help reducing the number of API calls that may lead to performance bottleneck when there are many customized tenants with unpredictable loads. We planned to collaborate with two SaaS vendors and their customer companies for an empirical study. Enabling event-based customization is also a way to prepare for offloading custom code to the Edge devices. The event bus could be open to events from microservices on Edge devices and maybe even to "things" in the IoT context.

Fig. 1 .

 1 Fig. 1. Event-based customization flow.

Fig. 2 .

 2 Fig. 2. Customization of Tenant A: An estimated time for delivery.

Fig. 3 .

 3 Fig. 3. Customization of Tenant B: The customization flow around the OrderStatus-ChangedToAwaitingValidationIntegrationEvent.

Fig. 4 .

 4 Fig. 4. Customization of Tenant B: After all the RFID tags have been scanned.

https://www.rabbitmq.com/vhosts.html

https://github.com/dotnet-architecture/eShopOnContainers

https://github.com/Espent1004/eShopOnContainersCustomised