Gayashan Amarasinghe
email: gamarasinghe@student.unimelb.edu.au

Marcos Dias De Assuncao
email: marcos.dias-de-assuncao@inria.fr

Aaron Harwood
email: aharwood@unimelb.edu.au

Shanika Karunasekera

Marcos D De Assunção

ECSNeT++ : A simulator for distributed stream processing on edge and cloud environments

Keywords: Simulation, Edge Computing, Distributed Stream Processing, Internet of Things

come   L'archive ouverte pluridisciplinaire

Introduction

The recent surge of Internet of Things (IoT) has led to widespread introduction and usage of smart apparatus in households and commercial environments. As a result the global IoT market is expected to hit $7.1 trillion in 2020 [START_REF] Hsu | An empirical examination of consumer adoption of Internet of Things services: Network externalities and concern for information privacy perspectives[END_REF]. These smart devices have demonstrated to hold great potential to improve many sectors such as healthcare, energy, manufacturing, building construction, maintenance, security, and transport [START_REF] Agostino | Combining Edge and Cloud Computing for Low-Power , Cost-Effective Metagenomics Analysis[END_REF][3][4] [START_REF] Stojkoska | A review of Internet of Things for smart home : Challenges and solutions[END_REF]. For example, Trenitalia, the main Italian train service operator is currently using an IoT based monitoring system to move from a reactive maintenance model to a passive maintenance model to deliver a more reliable service to travellers [START_REF] Gartner Inc | Leading the IoT -Gartner Insights on How to Lead in a Connected World[END_REF].

One common trait of many IoT deployments is that the devices monitor the environment they are deployed in, and produce events related to the current status of the said environment. This generates a time series of events that can be processed to extract vital information about the environment. Generally this event stream is unbounded and hence, Distributed Stream Processing (DSP) techniques can be utilised in order to process the events in real-time [START_REF] De Assunção | Distributed data stream processing and edge computing: A survey on resource elasticity and future directions[END_REF].

Apache Storm [START_REF] Toshniwal | Storm@twitter[END_REF], Twitter Heron [START_REF] Kulkarni | Twitter Heron[END_REF], Apache Flink [START_REF] Carbone | Apache Flink: Unified Stream and Batch Processing in a Single Engine[END_REF] are modern DSP frameworks widely used for distributed stream processing. Cloud computing has become the main infrastructure for deploying these frameworks, where they are generally hosted by large-scale clusters offering enormous processing capabilities and whose resources are accessed on demand under a pay-as-you-go business model.

As mentioned earlier, with the involvement of IoT, the sources where events are generated are located outside the cloud. Therefore, if we are to process these event streams using DSP frameworks, the current status quo is to generate data streams at the IoT devices and transmit them to the cloud for processing [START_REF] De Assunção | Distributed data stream processing and edge computing: A survey on resource elasticity and future directions[END_REF]. However, this arrangement is not ideal due to the higher latency caused by the communication overhead between these IoT devices and the cloud, which often becomes the bottleneck [START_REF] Beck | Mobile Edge Computing : A Taxonomy[END_REF] [START_REF] Shi | Edge Computing: Vision and Challenges[END_REF].

On the other hand, Edge computing is emerging as a potential computing paradigm that can be utilised for sharing the workloads with the cloud for suitable applications [START_REF] Amarasinghe | A Data Stream Processing Optimisation Framework for Edge Computing Applications[END_REF] [START_REF] Alam | Autonomic computation offloading in mobile edge for IoT applications[END_REF]. The goal of edge computing is to bring processing closer to the end users or in proximity to the data sources [START_REF] Shi | Edge Computing: Vision and Challenges[END_REF] [START_REF] Khan | Edge computing: A survey[END_REF]. As a result, concepts such as cloudlets, Mobile Edge Computing, and Fog Computing, are utilised where the processing is conducted at the base stations of mobile networks or servers that reside at the edge of the networking infrastructure, sometimes even referred as the Edge Cloud [START_REF] Khan | Edge computing: A survey[END_REF]. However, with the advancements in processing and networking technologies, processing should not be limited only to these processing servers that are at the edge of the networking infrastructure. Therefore, we utilise a broad definition for Edge Computing as computations executed on any processing element located logically outside the cloud layer which includes either edge servers and/or clients [START_REF] Shi | Edge Computing: Vision and Challenges[END_REF][15] [START_REF] Hu | Mobile edge computing: A key technology towards 5G, Whitepaper[END_REF][17] [START_REF]Edge tpu -run inference at the edge -edge tpu -google cloud[END_REF]. These devices are located closer to the data sources, sometimes acting as data sources or thick clients themselves. Wearable devices, mobile phones, personal computers, single board computers, many common smart peripherals with computational capabilities (such as many IoT devices), mobile base stations, and networking devices with processing capabilities belong to this category. With the increasing computational power of mobile processors, we believe these edge devices that are located closer to the end users, have potential that is not being fully utilised [START_REF] Morales | Symbiosis: Sharing mobile resources for stream processing[END_REF].

The commercial interest in the utilisation of IoT technologies for processing data at the edge, is shown with the invention of Google's implementation of artificial intelligence at the edge [START_REF]Edge tpu -run inference at the edge -edge tpu -google cloud[END_REF], in-device video analysis devices [START_REF]Google nest cam[END_REF][21] [START_REF]Provigil target tracking and analysis[END_REF], home automation devices [START_REF]Connode smart metering[END_REF], and industrial automation applications [START_REF]Dust networks applications: Industrial automation[END_REF]. However, these edge devices have limitations of their own as well -such as being powered by batteries and being connected to the cloud with limited bandwidth networks, which are often wireless.

The number of Edge computing devices (including IoT devices) is increasing exponentially. These devices have heterogeneous specifications and characteristics which cannot be generalised in a trivial manner. Many of these devices have potential that can be utilised in many ways. Therefore, finding efficient ways to utilise these heterogeneous resources is important and a timely requirement. However, creating a real test environment for running such experiments is challenging due to the amount of effort and investment required. As a result, simulation has become a suitable tool that can be used to run these experiments and to acquire realistic results [START_REF] Gupta | iFogSim: A toolkit for modeling and simulation of resource management techniques in the Internet of Things, Edge and Fog computing environments[END_REF] [START_REF] Sonmez | EdgeCloudSim: An environment for performance evaluation of Edge Computing systems[END_REF].

Currently, to the best of our knowledge, there is no complete simulation environment that provides both an accurate processing model and a networking model for edge computing applications -specifically for distributed stream processing application scenarios. The available simulation tools lack an accurate network communication model [START_REF] Gupta | iFogSim: A toolkit for modeling and simulation of resource management techniques in the Internet of Things, Edge and Fog computing environments[END_REF] [START_REF] Sonmez | EdgeCloudSim: An environment for performance evaluation of Edge Computing systems[END_REF]. However, on edge computing applications, the network communication plays a critical role mainly due to the bandwidth and power limitations [START_REF] Amarasinghe | A Data Stream Processing Optimisation Framework for Edge Computing Applications[END_REF] [START_REF] Sonmez | EdgeCloudSim: An environment for performance evaluation of Edge Computing systems[END_REF]. In addition, none of the available simulation tools have been verified against a real device deployment which we believe is equally important since simulation without a sense of accuracy and reliability of the acquired results is ineffective.

As a solution, we have implemented a simulation toolkit, ECSNeT++, with an accurate computational model for executing DSP applications, on top of the widely used accurate network modelling capabilities of OMNeT++. OMNeT++ is one of the widely used network simulators due to its in-detail network mod-elling and accuracy [START_REF] Sonmez | EdgeCloudSim: An environment for performance evaluation of Edge Computing systems[END_REF] [START_REF] Varga | An Overview of the OMNeT++ Simulation Environment[END_REF]. We have verified ECSNeT++ against a real device deployment to ensure that it can be properly calibrated to acquire realistic measurements compared to a real environment.

More specifically, we make the following contributions:

• We present an extensible simulation toolkit, which we call ECSNeT++, built on top of the widely used OMNeT++ simulator that is able to simulate the execution of DSP application topologies on various edge and cloud configurations to acquire the end to end delay, processing delay, network delay, and per-event energy consumption measurements.

• We demonstrate the effectiveness and adaptability of our simulation toolkit by simulating multiple configurations of two realistic DSP applications from the RIoTBench suite [START_REF] Shukla | RIoTBench: A Real-time IoT Benchmark for Distributed Stream Processing[END_REF] on edge-cloud environments.

• We verify our simulation toolkit against a real edge and cloud deployment with multiple configurations of the two DSP applications to show that the measurements acquired by simulations can closely represent real-world scenarios with proper calibration.

• We display the extensibility of our simulation toolkit by adopting the LTE network capability from the SimuLTE framework [START_REF] Nardini | Simulte-mec: Extending simulte for multi-access edge computing[END_REF] to simulate a realistic DSP application on an LTE network.

• We have made our simulation toolkit1 and the LTE plugin2 , open source and publicly available for use and further extension.

A high-level comparison of the features of ECSNeT++ against the state-ofthe-art simulators publicly available as shown in Table 1, indicates that EC-SNeT++ is the only simulation tool of its kind for simulating the execution of DSP applications on edge and cloud computing environments that has also been verified against a real experimental set-up to demonstrate its capabilities.

× × Comprehensive DSP Task Model × × × Limited DSP Task Placement × × × Verification against real set-up × × × ×
The rest of this paper is organised as follows. We provide a brief background about the characteristics of host networks and DSP applications in Section 2, whereas the modelling and simulation tool is described in Section 3. The evaluation scenario and performance results are presented in Section 4 followed by the in detail analysis of lessons learnt during our study in Section 5. Section 6 discusses related work and Section 7 concludes the paper.

Background

Characteristics of a Host Network

An execution environment (i.e. a host network) is a connected graph where vertices represent devices and the edges of the graph are the network connectivity. In our study we consider an environment that consists of cloud servers and edge computing devices, where the edge devices are connected to the cloud through a LAN and WAN infrastructure.

Each host device has data processing characteristics (with respect to the available CPU and the associated instruction set), network communication characteristics (via NICs and network protocols), and data storage characteristics (volatile and non-volatile storage). The network itself has a topology, different network components such as routers, switches and access points, and quantifiable characteristics such as the network latency, bandwidth and throughput.

The processing power and storage characteristics vary depending on the host device. A cloud node usually provides a virtual execution environment often with high processing power with multi-core multi-threaded CPUs, and high RAM capacity and speed. In contrast, an edge device has lower processing power and memory. For an example a Raspberry Pi 3 Model B device has a 4 core CPU with 1024MB memory, while a high-end mobile phone may have a quad/octa-core CPU with 6/8GB memory. In addition, these different devices may have their own instruction sets depending on the processor architecture. All of these processing characteristics and memory characteristics are generally available with the device specifications.

The most power consuming components of these host devices are the network subsystem and the processing unit. The power consumption characteristics of hosts for the most part depend on the underlying hardware. The cloud servers consume significant amount of power since they have unconstrained access to power. On the other hand, the edge devices are constrained by power availability. However, in most of these devices, it is possible to monitor and profile the power consumption characteristics empirically [START_REF] Kaup | Energy-efficiency and performance in communication networks: Analyzing energy-performance trade-offs in communication networks and their implications on future network structure and management[END_REF] or acquire them from the device specifications.

The characteristics of the underlying network depend on the network topology and the components of the network. A Local Area Network has higher bandwidth and throughput with lower latency but a smaller, less intricate topology. A Wide Area Network has a more complex topology but often with higher latency that increases with distance. Depending on the location of the edge devices and cloud servers, a combination of LAN and WAN may be utilised to build the host network.

Characteristics of a DSP Application

As illustrated in Figure 1, a DSP application consists of four components, source, operator, sink, and the dataflow. In any DSP application, the sinks, sources and operators, known as tasks, are placed and executed on a processing node. These different tasks in the application topology have their own characteristics depending on the application.

Source

Operator Sink Task Dataflow Event

Figure 1: Components and characteristics of a DSP application where a source generates events, which are then processed by the operators down stream, and reach a sink for final processing. Each source governs the event generation rate and the size of each event generated. Each operator manipulates either the properties of each event, the event processing rate or both.

A source generates events -therefore has an event generation rate and an event size associated with it. The event generation rate vary from one application to another. We can expect a uniform rate if the source is periodically reading a sensor and transmitting that value to the application. If the source is configured to read a sensor and report only if there is a change in reading, it can produce events at a rate that is difficult to predict. If the source is related to human behaviour, e.g.: activity data from a fitness tracker, the event generation rate can follow a bimodal distribution with local maxima, where people are most active in the morning and the evening [START_REF] Shukla | RIoTBench: A Real-time IoT Benchmark for Distributed Stream Processing[END_REF].

Each generated event has a fixed or variable size depending on the application. If the event is generated by reading a specific sensor, often an upper bound for the event size is known due to the known structure of an event. For example, a temperature reading from a thermometer may contain a timestamp, sensor ID, and the temperature reading. Therefore, the size of such an event can be trivially estimated. However, if each event is a complex data structure such as a twitter stream, which can contain textual data and other media, then the event size may vary accordingly.

For a given operator, two characteristics control the dataflow. For an operator T j , the selectivity ratio σ Tj is the ratio between the outgoing throughput λ out Tj and the ingress throughput λ in Tj [START_REF] Pietzuch | Network-aware operator placement for stream-processing systems[END_REF][28], i.e. σ Tj = λ out Tj /λ in Tj . We also introduce, for an operator T j , productivity ratio ρ Tj , which is the ratio between an outgoing message size β out Tj and the incoming message size β in Tj , i.e. ρ Tj = β out Tj /β in Tj . The distribution of the selectivity and productivity ratios depend on the incoming event size and the transformation conducted at the op-erator [START_REF] Shukla | RIoTBench: A Real-time IoT Benchmark for Distributed Stream Processing[END_REF]. For example, if the operator is calculating the average of incoming temperature readings for a given time window, an upper bound for the productivity ratio can be estimated since outgoing event structure is known. If the window length is known, the selectivity ratio can also be estimated. However, if the operator is tokenizing a sentence in a textual file, the productivity ratio and the selectivity ratio for that particular operator can vary depending on the input event. By profiling each operator, the selectivity and productivity ratio distribution can be estimated to a certain extent. But this is highly application dependent.

ECSNeT++ Simulation Toolkit

ECSNeT++ is a toolkit that we designed and implemented to simulate the execution of a DSP application on a distributed cloud and edge host environment. The toolkit was implemented on top of the OMNeT++ framework 3 [27] using the native network simulation capabilities of the INET framework 4 . OMNeT++ framework has matured since its inception in 1997 with frequent releases 5 . In addition to the maturity, and widespread use among the scientific community [START_REF] Cucinotta | CloudNetSim -Simulation of Real-Time Cloud Computing Applications[END_REF] [START_REF] Malik | CloudNetSim++: A GUI Based Framework for Modeling and Simulation of Data Centers in OMNeT++[END_REF], OMNeT++ enabled the development of a modular, component based toolkit to simulate the inherent characteristics of edge-cloud streaming applications and networks.

The high-level architecture of ECSNeT++ is illustrated in Figure 2. We implemented the host device model, and the power and energy model by extending the INET framework. The multi-core multi-threaded processing model is built on top of our host device model. The components of the DSP application model use all these features to simulate the execution of an application on an Edge-Cloud environment. The following subsections describe in detail how the different elements of a stream processing solution are modelled in the simulation toolkit. In Table 2 we summarize the symbols used throughout this paper.

Host Network and Processing Nodes

We have implemented a set of processing nodes of two categories: cloud nodes and edge nodes. As mentioned earlier, the edge computing nodes are the leaf nodes that are connected to the network via an either a IEEE 802.11 WLAN using access points for each separate network or via an LTE Radio Access Network with the use of ECSNeT++ LTE Plugin and eNodeB base stations. Depending on the scalability of the edge layer, the number of access points or base stations can be configured. They act as the bridge between the edge clients and the wired local area network (See Figure 4 and Figure 17). All the wired connections are simulated in this host network as full duplex Ethernet links. However, by utilising the INET framework with our host device implementations, it is possible to simulate a complex LAN and WAN configuration as required. In addition, the use of an LTE network for communication between the edge and the cloud by using the SimuLTE simulation tool 6 and the ECSNeT++ LTE Plugin, which provides a LTE network connectivity model, shows the extensibility of the networking model available in ECSNeT++.

We have implemented each type of edge device that use IEEE 802.11 connectivity by extending the WirelessHost module of INET framework which simulates IEEE 802.11 network interface controllers for communication. For LTE connected edge devices, each device is a standalone module with a LTE enabled network interface controller which implements the ILteNic interface from the SimuLTE tool. A cloud node is implemented by extending the Stand-ardHost module of the INET framework. This grants the cloud nodes the ability to connect to other modules (switches, routers, etc.) via a full-duplex Ethernet interface.

Each host has the capability to use either TCP or UDP transport for communication. The preferred protocol can be configured by enabling the <host>.hasTcp or <host>.hasUdp property and setting the <host>.tcpType or <host>.udpType property appropriately on the omnetpp.ini file (or the relevant simulation con- Idle power consumption Φ cpu (u)

CPU power consumption at utilisation u Φ wif i,idle

WiFi idle power consumption Φ wif i,up

WiFi transmitting power consumption Φ wif i,down WiFi receiving power consumption figuration file) to use the TCP or UDP protocol for communication.

DSP Application

Three separate task modules were developed to represent namely sources, operators, and sinks. A StreamingSource task module produces events depending on the configurations for event generation and event size. A StreamingOperator task module accepts incoming events, processes them using the processing model of the host and sends them to the next task or tasks according to the DSP topology. A StreamingSink task module is the final destination for all incoming events.

Each of these task modules should be placed on a host device with processing capabilities and can be connected according to the adjacency matrix of the DSP topology, which needs to be provided to the simulator, to simulate the dataflow of a DSP application. Instances of these modules should be configured, depending on the characteristics of each task in the DSP application topology. The processing of events in each of these tasks shall be done according to the processing model we have implemented in the simulation toolkit.

Placement Plan

The placement plan, which dictates the placement of one or more DSP application topology tasks (Sources, Operators and Sinks) on to the host networks, needs to be generated externally depending on the requirements of the application deployment and should be submitted to ECSNeT++. For example, the placement plan contains a mapping for each task in the DSP application topology to a host device(or devices) along with other characteristics relevant to the task if it is executed on the particular host device(or devices). Therefore, along with the placement of the topology, every attribute related to each task, e.g. the processing delay of each task, event generation rate and generated message size of a source, selectivity and productivity distribution of an operator, need to be configured in the placement plan itself. We have implemented an XML schema to define the placement plans. We have decoupled the placement plan generation from the ECSNeT++ to allow the users to use any external tool to generate placement plans and to produce an XML representation of the plan using our schema.

Source

StreamingSource module represents a streaming source in the topology. Responsibility of each StreamingSource module is to generate events as per the configuration and to send the generated events downstream to the rest of the application topology. Two attributes affect the event stream generated at any given source in the topology; the generated event size and the distribution of the event generation rate. These attributes can be configured in the Stream-ingSource module. Once configured, the StreamingSource module starts generating events as per the message size and event rate distributions and sends the generated events to the adjacent tasks.

The generated event size depends on the type of the application topology. A developer can simulate this characteristic by implementing the ISourceMes-sageSizeDistribution interface and configuring the placement plan by assigning the characteristic to the respective source such as a uniformly distributed message size distribution. Similarly the ISourceEventRateDistribution interface can be implemented, followed by an appropriate configuration in the placement plan, to incorporate different event generation rates for each source in the topology such as a bimodal event generation rate distribution.

Once these characteristics are configured for each source, they will produce events accordingly, that can be processed by operators connected to the data flow.

Operator

An operator in a topology can be simulated by configuring and placing a StreamingOperator module. The responsibility of each operator is to process any and all incoming streaming events and to produce one or more resulting events based on the characteristics that define its function; the selectivity ratio and the productivity ratio. By implementing the IOperatorSelectivity-Distribution interface, the selectivity ratio distribution of each operator can be simulated. Similarly the IOperatorProductivityDistribution interface should be implemented to simulate the productivity ratio distribution of a respective operator. Each of these implementations needs to be configured with appropriate parameters according to the application topology in the placement plan. The parameter values can be acquired by profiling the incoming and outgoing events of each operator or by analysing the topology.

When an event arrives at an operator, the operator changes the outgoing event size and the outgoing event rate based on the productivity distribution and the selectivity distribution of that operator, and emits a corresponding event (or events) which are sent to the next operator or sink according to the application topology (denoted by the adjacency matrix in the simulator).

Sink

We have implemented the StreamingSink module to simulate the functions of a sink in the topology. Each sink acts as the final destination for all the arriving events and emits the processing delay, network delay, and the total latency of each event.

Apart from the above configurations, for each of these sources, operators, and sinks, the attributes to calculate processing delay needs to be configured. The following subsection describes how the processing model has been implemented and how to configure this model appropriately for each task in the DSP application topology being simulated.

Processing Model

Each processing node in the host network has a processing module to simulate a multi-core, multi-threaded CPU. The processing of an event is simulated by holding the event in a queue for a time period δ TiVj .

Each processing node shall be configured with 3 parameters; number of CPU cores η Vj , frequency of a single CPU core C Vj , and the number of threads per core ψ Vj . The CPUCore module of ECSNeT++ is implemented to simulate a single CPU core instance. Each host device has CPUCore module instances equal to the number of CPU cores in that particular host as configured.

Figure 3 shows the processing of an event at a processing node. Since each processing node has tasks placed on it according to the placement plan as described earlier in Section 3.2.1, when an event arrives at a particular task, the task selects a CPU core for processing depending on the scheduling policy. We implemented a Round-robin scheduler for this study. However, it is possible to extend this behaviour to introduce other scheduling policies to the processing model by implementing the ICpuCoreScheduler interface. Once a CPU core is selected, the task sends the event to that core for processing. A CPU core maintains an exclusive non-priority FIFO queue for each operator, simulating the behaviour of a multi-threaded execution where the execution of each task is done in a single thread which depicts a thread per task architecture.

There exist two methods to acquire the per event processing delay for a task running on a particular device (δ TiVj ). Each task can be configured in the placement plan to directly provide per event processing time which can be measured by profiling the running time of a task per event. If this is not feasible, the number of CPU cycles required to process the particular event in

Processing Node T i T j T k CPU Core-0 Queue i Queue j Queue k CPU Core-1 Queue i Queue j Queue k Event n+1
Event n

Figure 3: A processing node (Va) with a dual core CPU and 3 DSP tasks (T i , T j , T k ) assigned to it, processes an incoming event n by sending the event to a selected CPU core in round-robin manner. Each CPU core maintains an exclusive non-priority queue for each task. Each queue holds an unprocessed event for a time period δ T i Va and releases the event back to the task that sent the event to that particular CPU core. This concludes processing of the particular event by the said task.

the particular CPU architecture can be determined empirically by profiling an operator [START_REF] Camesi | Continuous bytecode instruction counting for CPU consumption estimation[END_REF] or even by analysing the compiled assembly code, depending on the complexity of the application code, which in turn can be used to calculate the per event processing time.

In our experiments we use per event processing time directly since it can be measured from our real experiments.

After holding the event in the queue for the processing delay acquired by either of the above mentioned methods, the CPU core removes the event from the queue, and sends it back to the task that sent it, which completes the processing of the particular event on that task.

When a source has generated an event or an operator receives a processed event, it sends the event to the next operator or sink downstream. Each event is processed in this manner until it reaches a sink. This processing model is used by all the sources, operators, and sinks in the configured DSP application during the simulation.

Power and Energy Model

In ECSNeT++ we have implemented two power consumption characteristics for the edge devices connected via the IEEE 802.11 WLAN. The network power consumption measures the amount of power consumed while receiving and/or transmitting a processed event. The processing power consumption measures the amount of power consumed while processing a particular event at a CPU core. In addition, the idle power consumption measures the amount of power consumed by the device while idling.

We have implemented and configured the NetworkPowerConsumer class to simulate the power consumption of the network communication over IEEE 802.11 radio medium. Each processing node in the edge layer has an instance of a NetworkPowerConsumer assigned to it. When the state of the radio changes, the consumer calculates the amount of power that would be used by that state change and emits that value. The NetworkPowerConsumer class has to be configured with idle power consumption of the device without any load, idle power consumption of the network interface, transmitting power consumption of the network interface, and the receiving power consumption of the network interface. These values can be measured or estimated based on empirical data [START_REF] Kaup | Energy-efficiency and performance in communication networks: Analyzing energy-performance trade-offs in communication networks and their implications on future network structure and management[END_REF] [START_REF] Kaup | PowerPi: Measuring and modeling the power consumption of the Raspberry Pi[END_REF]. The power consumption is determined, depending on the radio mode, the state of the transmitter, and the state of the receiver.

We have implemented and configured the CPUPowerConsumer class on EC-SNeT++ to simulate the power consumed by the processor in each edge based processing node. With our processing model, a CPU core operates within two states, CPU idle state and the busy state. When the CPU core is processing an incoming event, as per Section 3.3, it operates in the busy state and otherwise in an idle state. When a CPU core changes its state, the CPUPowerConsumer calculates the amount of power associated with the relevant state change. The CPUPowerConsumer is configured using the power consumption rate with respect to its utilisation which can be measured or estimated [START_REF] Kaup | Energy-efficiency and performance in communication networks: Analyzing energy-performance trade-offs in communication networks and their implications on future network structure and management[END_REF] [START_REF] Kaup | PowerPi: Measuring and modeling the power consumption of the Raspberry Pi[END_REF].

When a power consumption change is broadcast by a consumer, the Ideal-NodeEnergyStorage module calculates the amount of energy consumed during the relevant time period by integrating the power consumption change over the time period and records that value. However, it is important to note that the IdealNodeEnergyStorage does not have properties such as memory effect, selfdischarge, and temperature dependence that can be observed on a real power storage.

The total power Φ total consumed at each edge device can be calculated by the following Equation (1) [START_REF] Kaup | Energy-efficiency and performance in communication networks: Analyzing energy-performance trade-offs in communication networks and their implications on future network structure and management[END_REF] [START_REF] Kaup | PowerPi: Measuring and modeling the power consumption of the Raspberry Pi[END_REF]. The total power consumption is estimated by summing the power consumption of individual components as configured. Here Φ idle is the idle power consumption without any load on the system, Φ cpu (u) is the power consumed by CPU when the utilisation is u%, and Φ wif i,up (r) and Φ wif i,dn (r) represent the power consumed by the network interface for transmitting and receiving packets over the IEEE 802.11 WLAN at the data rate of r.

Φ total =Φ idle + Φ cpu (u) + Φ wif i,idle + Φ wif i,up (r) + Φ wif i,dn (r) (1)
The components of this approximation function can be calibrated in the simulation configuration by either measuring the individual power consumption components of the device(s) or by referring to power characteristics specifications of the device(s). Both uplink and downlink energy consumption should be considered since as per our state based power model and the duplex nature of packet transmission, arrival or the departure of a packet at the relevant network interface, triggers the energy consumed for that state change at the network interface to be recorded. All such recorded state changes are accumulated to produce the total power and energy consumption for the simulation.

At this stage of our study, we are only interested in the power/energy consumption of the edge devices. Therefore, we have limited the power consumption simulation to only the edge devices, and the wireless network communication between the edge devices and the access points. However, due to the modular architecture of the ECSNeT++, the power model can be extended to other processing nodes and communication media such as the LTE RAN as well.

Experiments and Evaluation

We use two set-ups for experiments. To run the ECSNeT++ simulator in headless mode we use a m2.xlarge instance, which consists of 12 VCPUs, 48GB RAM, and a 30GB primary disk on the Nectar cloud 7 . We execute the simulation of ETL and STATS application topologies on this set-up with different operator allocation plans, and acquire the relevant measurements.

In order to verify the IEEE 802.11 WLAN based simulations of ECSNeT++ simulator against a real-world experimental set-up, we have established a simple experimental set-up using 2 Raspberry Pi 3 Model B (edge) devices which connect to two access points to create 2 separate edge sites. The access points are then connected to a router which connects to a single cloud node (cloud). The cloud node has the processing power of 24 core Intel(R) Xeon(R) CPU E5-2620 v2 at 2.60GHz and 64GB of memory. This selected host network for the real experiment set-up is shown in Figure 4 and the configurations are available in Table 3. The clocks of all these host devices are synchronized using the Network Time Protocol. All the communications between the edge and the cloud occur over the TCP protocol.

We have implemented the DSP applications using Apache Edgent8 framework which runs on Java version 1.8.0 65 on each edge device and cloud server in the experimental set-up. Apache Edgent is a programming model and a micro kernel which provides a lightweight stream processing framework that can be embedded in many platforms.

Application Topologies and Allocation Plans

We use a simplified version of the ETL application topology and the STATS application topology from the RIoTBench suite [START_REF] Shukla | RIoTBench: A Real-time IoT Benchmark for Distributed Stream Processing[END_REF] as the DSP applications for the experiments. The ETL application has a linear topology (See Figure 5) which extracts information from a given CSV file and converts the information to SenML format after preprocessing. The STATS application has a diamond topology (See Figure 6) which extracts information from a given CSV file and runs a statistical analysis of the information to produce a set of line plots. These topologies can be executed on either, edge, cloud or both. We have partitioned the ETL application topology into two partitions in 7 ways as illustrated in Figure 7 where one partition is placed on the edge device (purple nodes) and the other partition is placed on a cloud server (blue nodes). The two partitions communicate using the communication media as configured. Then we execute the application on both the real world set-up and the simulator to process a fixed number of events, and acquire delay and edge energy consumption measurements. Similarly, we have partitioned the STATS application topology into two partitions in 6 ways where one partition is placed on the edge and the other is placed on the cloud. Then we execute each placement plan on both the simulator and the real experiment set-up. For each plan, we also vary the source event generation rate while keeping other parameters constant.

σ = 5 ρ = 0.33 σ = 1 ρ = 1 σ = 1 ρ = 1 σ = 1 ρ = 1 σ = 0.2 ρ = 1.3 σ = 1 ρ = 1 σ = 1 ρ = 2

Measurements

In order to evaluate the applications, we are using four measurementsprocessing delay per event, network transmission delay per event, total delay per event, and average power consumption at each edge device. These measurements can be collected for each DSP application executed on each host network, and can be used to monitor the performance of each application on either the simulation or the real experiment.

Delay Measurements

Each individual operator in the DSP application topology appends the processing time taken for a particular event and the sum of these individual processing times is recorded as the per event processing time for each event. When Plan 7

Network between edge and cloud a network transmission occurs, the network transmission time for each particular event is recorded as well. The total delay per event is defined as the sum of individual processing delays and network transmission delays of the particular event. The same approach is used in both the simulation and the empirical set-up.

Power and Energy Measurements

To have a more clear estimate on the power consumed by edge equipment in the experiment set-up, we designed a power meter and a set of scripts to measure the power consumed by our Raspberry Pi device while the DSP application is being executed. Raspberry Pi's are devices that can be powered via USB. The power consumption is hence measured by inserting a shunt resistor in the 5V USB line as depicted in Figure 8. The resistor causes a voltage drop that is proportional to the current drawn by the Raspberry Pi. An Analog to Digital Converter (ADC) is used to measure the voltage across the resistor (i.e. ADC 1) in order to compute the amount of current drawn by the Pi, whereas ADC 2 measures the input voltage used to compute the power. ADC 1 and 2 are respectively an ADC Differential Pi and and ADC Pi Plus, both produced by AB Electronics 10 . Prior to each configuration (i.e. placement plan) of the ETL or STATS application is executed on the experimental set-up (Figure 4), the data collection scripts are executed in order to measure the power consumption of the edge device periodically. Measured power consumption values are used to calculate the average power consumption by the edge device in each experiment afterwards.

In ECSNeT++, we use the power and energy model as described in Section 3.4 to record the total energy consumption for each execution of the different configurations of the application and calculate the average power consumption for the respective time period.

Calibrating the Simulator

In this section we describe how the simulator has been calibrated in order to accurately simulate the real world experiments for the ETL application.

The WLAN between the edge node and the access point has a limited bandwidth of 11Mbps and operates in IEEE 802.11b mode. The rest of the Full duplex Ethernet network operates at 1Gbps bandwidth. The host network configurations as described in Table 5 are mirrored in the host configurations of the simulator and the experimental set-up. For the power consumption measurements, we have acquired the values illustrated in Table 6 on the experimental set-up and we have configured ECSNeT++ using the same values. For all the other parameters, we use the default configuration values in OMNeT++.

First, each task in each application is profiled to measure the per event processing time (in nano second precision) for each CPU architecture. These measured values are used as the per event processing time, δ TiVj value in each 0.132 placement plan. Then the network transmission delay between edge and cloud is measured for each case and these measured delay distributions are used for the configuration of delays in the simulator for each plan. While the event generation rate is varied, the initial event size of the source, and selectivity and productivity ratios of the operators are measured in the real application and the same values are used in the simulation set-up. Each application is executed until 10000 events are processed, and the experiments are concluded once the measurements are acquired for each of these processed events in both the simulator and the experimental set-up.

Evaluation of Delays

Figure 9 and Figure 10 show all the delay measurements for fixed source event generation rate of 10 events per second and 100 events per second respectively for all the considered placement plans of ETL application (Figure 7). Similarly, Figure 11, and Figure 12 shows the delay measurements of all the placement plans of STATS app running at 5 and 10 events per second. Each experiment is concluded when 10000 events have been processed. We conduct a qualitative analysis of the observed results for each scenario.

Network Delay

We first evaluate the network delay parameter for each scenario. In Figure 9 and Figure 10 the network latency values follow the same trend even though the real experimental set-up shows higher interquartile range with respect to the simulation set-up, where the variations in the simulated values are limited. In the experiments, we calculate the network delay as the time taken by an event to leave an edge device and arrive at the cloud server, which includes all the latencies caused by the management of packet delivery over the communication channel such as CSMA/CA, rate control, packet retransmission and packet ordering caused by failures, etc., which results in the variations we observe in the results. It is not possible to simulate an identical environment since it is not possible to measure these intricacies. Since the IEEE 802.11 implementation of OMNeT++/INET has the same characteristics and due to the close calibration of the simulator (Section 4.3) we can observe some variation in the network delay measurements of the simulation specifically when the amount of packets transmitted is high. The lack of variation in the Plan 5, 6 and 7 when compared with the Plan 1, 2, 3, and 4, is caused by the lack of the amount of events transmitted by Plan 5, 6 and 7. The "Join" Task which is the terminal task placed at the edge device at the Plan 5, has a selectivity ratio of 0.2 (see Figure 5) which results in ∼5 times reduction in the amount of events transmitted from the edge to the cloud which is also affecting the Plans 6 and 7. However, the slightly higher variation in Plan 7 is due to the "CSV to SenML" task which is placed on the edge nodes in this plan. This task has a productivity ratio of 2 which results in larger events being transmitted from the edge to the cloud over the network which in turn causes the slight variation that is visible.

In Figure 11, and Figure 12, Plan 1, 2, 3, 4 and 5 show higher variation in network delay which is caused by the structure of the topology where a large number of events are transmitted between the edge and the cloud due to multiple dataflows in the topology between the "SenML Parser" and the downstream operators.

By observing the trend in these figures we can conclude that for each case in each placement plan, the simulation can be used as an approximate estimation for the network delay measurements in a real experiment with similar parameters.

Processing Delay

As illustrated in Figure 9, Figure 10, Figure 11, and Figure 12 the processing delay measurements of the simulation, closely follow the same trend as the experiments. However, the processing delay in the experimental setup has a higher variation which is expected due the complex processing architecture present in real devices which involves context switching, thread and process management, IO handling and other overheads. We measure the per event processing delay in each task in each placement plan and use the mean per event processing delay for each task in the simulator. Therefore the lack of variation in the processing delay measurements is visible in the illustrations. In Figure 9, the higher variation in the Plan 4 is caused by the measured processing time difference between the two Raspberry Pi devices (danio-1 and danio-5) which is used as processing time delay per each task in each device configurations.

The processing model of ECSNeT++ is not as intricate as a real CPU architecture. We believe that making the processing model complex (and realistic) may have a larger impact in some simulation scenarios which would generate more accurate processing delay results.

Total Delay

Total delay is the sum of the processing delay and the network delay in each event. This is the most important measurement in terms of the quality of service provided to the end users by each DSP application. As illustrated in Figure 9, Figure 10, Figure 11, and Figure 12, the simulated total delay measurements closely follow the same trend as in the real experiments specially since the delay measurements are dominated by the processing time. With these observations it is also possible to conclude that the near-optimal placement plan for each DSP application scenario is the same in the real experiments and in the simulated experiments. Therefore, with these observations we can show that ECSNeT++ can be used to estimate the near optimal placement plans for Edge-Cloud environments with proper calibration where a real test environment is not available.

Evaluation of Power Consumption

As observed in Figure 13, Figure 14, Figure 15, and Figure 16, the calculated average power consumption values leave more detailed modelling and fine tuning to be desired on ECSNeT++. When the two event rates of each application are compared, processing lower number of events per second consumes less average power than processing higher number of events per second, because it takes more time to process 10000 events when event rate is low which results in decreasing the energy consumption per unit time i.e. power consumption. The idle power consumption is uniform in all the simulations since we have configured the simulator with the idle power consumption of 1.354W (See Table 6).

In Figure 15, and Figure 16, Plan 6 shows a large difference between the experiments and the simulations. In Plan 6, Multi Line Plot task is moved to the edge devices where it is writing the generated plots to the disk. However, unfortunately we have not modelled that in our simulator's processing power consumption model which we believe is the reason for the observed significant difference in processing power consumption comparison.

Extending the Network Model with LTE Connectivity

While previous experiments were conducted in an environment where edge devices were connected to the cloud through and IEEE 802.11 WLAN network, cellular networks are also being used widely for edge computing deployments [START_REF] Morales | Symbiosis: Sharing mobile resources for stream processing[END_REF] [START_REF] Nardini | Simulte-mec: Extending simulte for multi-access edge computing[END_REF]. Therefore, we have extended the network model of ECSNeT++ to use the SimuLTE [START_REF] Nardini | Simulte-mec: Extending simulte for multi-access edge computing[END_REF][37]11 simulation tool to introduce the LTE Radio Access Network (LTE RAN) connectivity. We use the LTE user plane simulation model of SimuLTE to develop a set of mobile devices, which we call LTEEnabledPhone, which acts as the User Equipments(UEs) that use LTE connectivity. In the simulation, these devices are then connected to the cloud through a base station known as eNodeB which allocates radio resources for each UE.

In addition, these UE devices move in a straight line, within a confined area of 90000 m 2 , with a constant speed of 1 ms -1 . This is modelled as a linear mobility model (LinearMobility) set via the mobilityType configuration that is inherited from the INET framework 12 .

The configuration parameters and the environment created for the LTE simulation is shown in Figure 17. The results of the simulation are illustrated in Figure 18. The simulation results illustrate that even with the speed and low latency of LTE connectivity, sharing the workload with edge devices is beneficial for stream processing applications in terms of lowering the total latency of events.

While we leave the comparison of these results against a real test environment for future work, due to the lack of physical resources to set up such an environment, we believe these experiments show the extensibility of ECSNeT++ to use other networking models with minimum configuration.

Lessons Learnt

Although we have shown that ECSNeT++ is capable of providing a useful estimation of delay and power measurements, there are some limitations that need to be addressed.

First of all, the lack of variation in simulated network delay measurements of some plans remains an issue that we wish to resolve in future work. The variation in the network delay values is caused by the variations in the transmission latency which in turn is largely caused by the intricacies of the network such as packet retransmission, collision avoidance and packet loss. A more detailed network modelling may be desired in scenarios where it is difficult to isolate the network between the edge and the cloud.

In the real experiments, the processing model shows higher inter-quartile range in each scenario as expected, due to the variations that can happen with the overheads in the underlying CPU architecture. Since we are using a fixed processing delay for each event, this higher variability is absent in the simulated results. In addition, the simplicity of the processing model, such as the lack of interruptions and context switching, also contributes to this factor. We believe these can be implemented in to the ECSNeT++ in future should the need arise.

It is also important to note the importance of proper calibration of the network parameters without which results were not accurate enough for consideration. We have identified that not only the overt parameters such as WLAN bit rate and Ethernet bit rate are critical, but also the parameters identified in Table 5, need to be explicitly configured in order to acquire realistic network latency measurements with respect to the experimental set-up.

When it comes to the power and energy consumption, we have found that in our simulation toolkit, the idle energy consumption to be the largest contributor for power consumption which is, as expected, the base power consumption. However, since it was not possible to measure the individual power consumption components for each event (processing power, transmission power, and idle power) in our real experimental set-up, we were unable to properly compare the distribution of components of the power consumption between the simulation and the real set-up. In addition, the state based power consumption model that we have implemented requires careful calibration of power consumption values associated with each state change, in order for it to be accurate. Therefore, while ECSNeT++ can be used to gather preliminary data on energy consumption, a more detailed power and energy consumption model would be a valuable extension to ECSNeT++ in future work.

Finally, we would like to emphasize that using proper calibration parameters for ECSNeT++ with respect to a real experimental set-up is the key step in acquiring reliable measurements for the conducted simulated experiments.

Related Work

While there are many tools to simulate cloud computing scenarios, only few simulation tools have been developed for edge computing scenarios. However, simulation is a valuable tool for conducting experiments for edge computing, mainly due to the lack of access to a large array of different edge computing devices. Also the ability to control the environment within bounds, provides a strong advantage to model a solution in a simulated environment first, to fine tune the associated parameters [25][26]. Here we look at the existing literature on such simulation tools that are either developed for edge computing scenarios, or provides us a platform to build our work upon.

EdgeCloudSim [START_REF] Sonmez | EdgeCloudSim: An environment for performance evaluation of Edge Computing systems[END_REF] is a recent simulation toolkit built upon widely popular CloudSim tool [START_REF] Calheiros | CloudSim: A toolkit for modeling and simulation of cloud computing[END_REF]. EdgeCloudSim is built to simulate edge specific modelling, network specific modelling, and computational modelling. While EdgeCloud-Sim provides an excellent computational model and a multi-tier network model, the toolkit lacks the ability to simulate characteristics of a distributed streaming application without significant effort. Our work exclusively provides the ability to simulate characteristics specific to DSP applications which is crucial for testing optimal strategies for task placement on both edge device and cloud server. Sonmez et.al., also acknowledge the importance of accurate network simulations, and that network simulation tools such as OMNeT++ provide a detailed network model. However, instead the authors have implemented their tool on CloudSim due to the high amount of effort required to build the computational model on a network focused simulator such as OMNeT++. In our work we have implemented the computational model with significant effort while benefiting from the detailed network simulation aspects of OMNeT++.

Gupta et al. [START_REF] Gupta | iFogSim: A toolkit for modeling and simulation of resource management techniques in the Internet of Things, Edge and Fog computing environments[END_REF] introduce iFogSim, which is also built on top of CloudSim tool. iFogSim provides the ability to evaluate resource management policies while monitoring the impact on latency, energy consumption, network usage and monetary costs. Their applications are modelled based on Sense-Process-Actuate model, which represents many IoT application scenarios. However, when compared to iFogSim, our simulation tool provides a fine grained control on placement of tasks of DSP applications based on a placement plan where the applications are executed using our multi-core, multi-threaded processing model. It is important to note that unlike iFogSim, in our work we simulate a realistic IEEE 802.11 WLAN network along with an Ethernet network (with the ability to build more intricate networks with less effort), and uses UDP or TCP protocol for communication, which provides a significant similarity to a realistic application. The effects of these simulations are represented in our measurements. We wish to incorporate valuable features such as SLA awareness available in iFogSim, in our future work.

In addition to these edge computing specific simulation tools, OMNeT++ [START_REF] Varga | An Overview of the OMNeT++ Simulation Environment[END_REF] is a discrete event simulation tool which is mainly built for network simulations. With the addition of INET framework, OMNeT++ presents a comprehensive, modular, extendable network simulation tool that is widely being used. In our simulation toolkit, we use these networking capabilities while building our computational model and power consumption model on top of it.

CloudSim [START_REF] Calheiros | CloudSim: A toolkit for modeling and simulation of cloud computing[END_REF] provides a simulation tool for comprehensive cloud computing scenarios such as modelling of large scale cloud environments, service brokers, resource provisioning, federated cloud environments, and scaling. CloudNetSim [START_REF] Cucinotta | CloudNetSim -Simulation of Real-Time Cloud Computing Applications[END_REF] is a cloud computing simulation tool built upon OMNEST 13 /OMNeT++ toolkit with a state based CPU module with multiple scheduling policies where applications can be scheduled to be executed on these modules. We gained inspiration from the CPU module and the scheduling policies introduced in this work, when developing our computation model. Similarly, CloudNetSim++ [START_REF] Malik | CloudNetSim++: A GUI Based Framework for Modeling and Simulation of Data Centers in OMNeT++[END_REF], which focuses on the aspects of cloud computing, is also built on top of OMNeT++.

Table 1 compares our proposed solution against other relevant state-of-theart modelling and simulation tools available in the literature. We have implemented our simulation toolkit to cover the limitations of the existing tools in assessing DSP applications running on edge-cloud environments. We have compared ECSNeT++ with a real experimental set-up to show the importance of calibration since none of the available simulation tools have conducted such a study. While there are few simulation tools for edge computing scenarios, they lack the ability to simulate the characteristics of a DSP application being executed on an edge-cloud environment, and lack a realistic network simulation which is critical. Our work is the first of its kind to implement these features on top of OMNeT++.

Conclusion

In this study we presented a unique simulation toolkit, ECSNeT++, built on top of the widely used OMNeT++ toolkit to simulate the execution of Distributed Stream Processing applications on Edge and Cloud computing environments. We used multiple configurations of the ETL and STATS application, tow realistic DSP application topologies from the RIoTBench suite, to conduct the experiments. In order to verify ECSNeT++ we used a realistic experimental set-up with a Raspberry Pi 3 Model B device and a cloud node with a WLAN and LAN network where the same experiments with identical network, host device, and application configurations were executed on both the simulator and the real experimental set-up. We measured network delay, processing delay, the total delay per each event processed, and average power consumption by the edge device to show that when properly calibrated, the delay measurements of ECSNeT++ toolkit follow the identical trends as a similar real environment. In addition, we demonstrated the extensibility of ECSNeT++ by adopting LTE Radio Access Network connectivity for the network model of the simulations. We also believe that ECSNeT++ is the only verified simulation toolkit of its kind that is implemented with characteristics specific to executing DSP applications on Edge-Cloud environments.

In our future work, we would like to investigate further on implementing a more complex computational model, and also to focus on implementing detailed power consumption models for ECSNeT++. We believe that with the open source availability and extensibility of ECSNeT++, researchers will be able to develop effective and practical utilisation strategies for Edge devices in Distributed Stream Processing applications.

Figure 2 :

 2 Figure 2: High level architecture of ECSNeT++ which is built on top of the OMNeT++ simulator and the INET framework

Figure 4 :

 4 Figure 4: Simple experimental set-up used for simulator calibration and verification

Figure 5 :

 5 Figure 5: Simplified ETL Application Topology from the RIoTBench suite

Figure 6 :

 6 Figure 6: Simplified STATS Application Topology from the RIoTBench suite

Figure 7 :

 7 Figure 7: Possible placement plans for the Simplified ETL Application Topology -purple nodes are placed on the edge while blue nodes are placed on the cloud

Figure 8 :

 8 Figure8: The power measurement set-up for the Raspberry Pi 3 device. ADC 1 and 2 are used to measure the voltage across the shunt resistor and the input voltage respectively. Data collection scripts record the power consumption periodically, using the voltage and current measurements.

  environments and evaluation of resource provisioning algorithms, Software -Practice and Experience 41 (1) (2011) 23-50. arXiv:0307245, doi:10.1002/spe.995.

Figure 10 :

 10 Figure10: Experiments vs Simulation for the ETL application where the source is generating 100 events per second until 10000 events are processed. The total delay measurement is composed by the sum of network delay and the processing delay. Median delay of each plan on experiment and simulation are shown by the horizontal blue and red line respectively.

Figure 11 :

 11 Figure11: Experiments vs Simulation for the STATS application where the source is generating 5 events per second until 10000 events are processed. The total delay measurement is composed by the sum of network delay and the processing delay. Median delay of each plan on experiment and simulation are shown by the horizontal blue and red line respectively.

Figure 13 :

 13 Figure13: Average power consumption on the edge device in the Experiments vs Simulation for the ETL application where the source is generating 10 events per second until 10000 events are processed. Each value in the plot shows the average power consumption of that component.

Figure 15 :

 15 Figure15: Average power consumption on the edge device in the Experiments vs Simulation for the STATS application where the source is generating 5 events per second until 10000 events are processed. Each value in the plot shows the average power consumption of that component.

Figure 16 :Figure 17 :

 1617 Figure16: Average power consumption on the edge device in the Experiments vs Simulation for the STATS application where the source is generating 10 events per second until 10000 events are processed. Each value in the plot shows the average power consumption of that component.

Table 1 :

 1 Comparison of ECSNeT++ simulation toolkit against state-of-the-art

	Feature	CloudSim EdgeCloudSim OMNeT++ iFogSim ECSNeT++
		[30]	[26]	[27]	[25]	toolkit
	Edge Devices	×		×
	Cloud Server			×
	Realistic Network Model	Limited			Limited
	Power Model		×
	Processing Model			×
	Device Mobility

Table 2 :

 2 Symbols used in the model

	Symbol	Description
	T i	Task i
	Op i	Operator i
	So i	Source i
	Si i	Sink i
	V j	Edge device or Cloud server j
	σ Ti	Selectivity ratio of T i
	ρ Ti	Productivity ration of T i
	r Soi	Event generation rate of So i
	m Soi	Generated event size of So i
	δ

TiVj Processing time per event, per-task T i , per-device V j ζ TiVj CPU cycles consumed per event, per-task, per-device C Vj Per core processing frequency of device V j η Vj Number of CPU cores on device V j ψ Vj Number of threads per CPU core on device V j Φ total Total power consumption Φ idle

Table 3 :

 3 Specification of host devices used in experimental set-up

		Edge	Cloud
	Proc. Architecture ARMv7	x86 64
	Processor Freq.	1.2GHz 2.6GHz
	CPU Cores	4	24

Table 4 :

 4 Characteristics of the operators of the ETL and STAS applications used in the experiments

	Application	Operator(s)
		Name	Selectivity Productivity
	ETL	SenML Parser	5	0.33
		Range Filter	1	1
		Bloom Filter	1	1
		Interpolation	1	1
		Join	0.2	1.3
		Annotate	1	1
		Csv To SenML	1	2
	STATS	SenML Parser	5	0.33
		Linear Regression	1	2
		Block Window Average	1	1
		Distinct Approx. Count	1	1
		Multi Line Plot	0.04	0.0025

Table 5 :

 5 Network configuration used in both the real device deployment and the simulation.

	Property	Value
	WLAN operation mode	IEEE 802.11b
	WLAN Bit rate	11 Mbps
	WLAN frequency	2.4 GHz
	WLAN radio bandwidth	20 MHz
	WLAN MTU	1500 B
	WLAN transmission queue length 1000
	WLAN RTS threshold	3000 B
	WLAN MAC retry limit	7
	WLAN Transmission power	31 dBm
	Ethernet Bit rate	1 Gbps
	UK 9 . Both ADCs are based on Microchip MCP3422/3/4 that enable a resolu-
	tion of up to 18 bits. Data collection scripts are executed by another Raspberry
	Pi device in the environment

Table 6 :

 6 Measured average power consumption values for Raspberry Pi 3 Model B used in the simulation. See Equation (1) for usage of these values.

	Property	Value (W)
	Φ idle	1.354
	Φ cpu (100%) 0.258
	Φ cpu (400%) 0.955
	Φ wif i,idle	0.118
	Φ wif i,up	0.175
	Φ wif i,dn

  Figure14: Average power consumption on the edge device in the Experiments vs Simulation for the ETL application where the source is generating 100 events per second until 10000 events are processed. Each value in the plot shows the average power consumption of that component.

		2.00												1.354	2.081
		1.75
	Averagte power consumption (W)	0.75 1.00 1.25 1.50	1.581	1.354	1.615	1.354	1.629	1.354	1.655	1.354		1.629	0.413	1.354 expr-total sim-network sim-cpu sim-idle
		0.50
		0.00 0.25	Plan1 0.026 0.044		Plan2 0.046 0.079	Plan3 Placement plan Plan4 0.046 0.075 0.080 0.121		Plan5 0.224	Plan6 0.026 0.133
			1.773		1.775	1.354	1.773	1.780	1.354	1.783	1.354	1.787	1.354	1.808	1.354
				1.354				1.354
														expr-total sim-network sim-cpu sim-idle
												0.386		0.387	0.401
						0.313			0.315
				0.160				0.160
				0.098		0.091	0.091		0.088		0.144		0.145	0.131

Available at https://sedgecloud.github.io/ECSNeTpp/

Available at https://github.com/sedgecloud/ECSNeT-LTE-Plugin

https://omnetpp.org/

https://inet.omnetpp.org/

OMNeT++ v5.3 was released on April, 2018.

http://simulte.com/index.html

https://nectar.org.au/research-cloud/

http://edgent.apache.org/

https://www.abelectronics.co.uk

https://github.com/assuncaomarcos/pi-powermeter

http://simulte.com/

https://inet.omnetpp.org/docs/showcases/mobility/basic/doc/index.html

OMNEST is the commercial variant of OMNeT++ tool.

Processing delay

Figure 18: Simulation for the ETL application running on the mobile devices over an LTE network, where the source is generating 10 events per second until 10000 events are processed. The total delay measurement is composed by the sum of network delay and the processing delay. Median delay of each plan is shown by the horizontal red line.