Olivier Aumage

Camille Coti

StarPU profiling interface

This document introduces the programming interface exposed by the StarPU task-based parallel runtime system for performance monitoring tools to collect performance-related metrics on the execution of StarPU applications. It defines the entry point, the callback methods, the performance counters available. It also presents the usage of the mechanisms.

Introduction to the StarPU callback interface

The StarPU callback interface can be used by third-party libraries for profile and trace data collection, to conduct performance analysis. It provides a set of callbacks that are triggered at specific points of the execution, as described in this document. Data is passed through these callbacks for these external libraries to collect information about the execution.

First, such a library must be loaded, as described in section 2. Second, the library initializes the callbacks by registering the function it provides onto StarPU's callback points, as described in section 3.

StarPU provides two categories of callbacks: the first one gives information on the execution of the StarPU tasks on the available resources, as described in section 4; the second one gives information on the internal state of the scheduler, as described in section 5.

Loading a tooling library

The library must be passed using the STARPU_PROF_TOOL environment variable or preloaded using LD_PRELOAD, upon the launch of the compiled StarPU program. For instance: STARPU_PROF_TOOL=./libstarpu_myprof.so ./myprogram

Library initialization

The library must implement a function starpu_prof_tool_library_register. This function registers callback routines using the register and unregister functions, whose pointers are passed as parameters. Its signature is the following one: typedef void (*starpu_prof_tool_entry_func)(starpu_prof_tool_entry_register_func reg, starpu_prof_tool_entry_register_func unreg);

For example: void starpu_prof_tool_library_register(starpu_prof_tool_entry_register_func reg, starpu_prof_tool_entry_register_func unreg) { enum starpu_prof_tool_command info = 0; reg(starpu_prof_tool_event_driver_init, &driver_init_cb, info); reg(starpu_prof_tool_event_driver_init_start, &driver_init_start_cb, info); /* ... */ }

The signature of these registration and unregistration functions is: typedef void (*starpu_prof_tool_entry_register_func)(enum starpu_prof_tool_event event_type, starpu_prof_tool_cb_func cb, enum starpu_prof_tool_command info);

Task performance callbacks

This section presents the task performance callbacks that can be used for performance analysis of the application. The performance counters used to monitor the performance of StarPU and, in particular, retrieve information from the scheduler, are defined in section 5.

Callbacks signature

Callback functions have the following prototype:

typedef void (*starpu_prof_tool_cb_func)(struct starpu_prof_tool_info*, union starpu_prof_tool_event_info*, struct starpu_prof_tool_api_info*);

For instance: void myfunction_cb( starpu_prof_tool_info* prof_info, starpu_event_tool_info* event_info, starpu_api_tool_info* api_info );

The arguments contain information that are passed to the callback (see section 4.2).

Data structures

Profiling information

This data structure provides the main information about the event concerned by the callback. • conf: a pointer to the current internal configuration;

• event_type: the type of the event that triggered this callback, see section 4.3;

• starpu_version: the current version of starpu (major, minor, release) and can be used to make a tool; compatible across versions taking into account API changes;

• thread_id: id of the current thread;

• worker_id: id of the current worker;

• device_number: number of the device related to the event that triggered this callback;

• device_type: type of device used by the event that triggered this callback, see section ;

• memnode: memory node, used in data transfers;

• bytes_to_transfer: how many bytes are being transferred by this memory movement, used in data transfers;

• bytes_transfered: how many bytes have been transferred by this memory movement, used in data transfers;

• fun_ptr: function pointer that tells us which function is about to be executed. It has to be resolved in the profiling tool.

Please note that not all the fields are relevant for all events. For instance, memnode is used for data transfers but not for executions. Likewise, fun_ptr is used for executions but not for data transfers.

For example, a callback function that would print the name of the event and the worker id would be:

void myfunction_cb( starpu_prof_tool_info* prof_info, starpu_event_tool_info* event_info, starpu_api_tool_info* api_info ) { if( prof_info != NULL ){ printf("%d on worker %d\n", prof_info->event_type, prof_info->worker_id); } }

Driver types

This data structure provides information on how the event concerned by the callback is being executed. enum starpu_prof_tool_driver_type { starpu_prof_tool_driver_cpu, starpu_prof_tool_driver_gpu };

For the moment, we define two types of events: on a GPU and on a CPU.

Event info

This data structure provides information on the event itself. It can be used to determine which event was triggered when the same function is registered for all the callbacks. union starpu_prof_tool_event_info { enum starpu_prof_tool_event event_type; };

API info

This data structure is empty for the moment. struct starpu_prof_tool_api_info { };

List of events

The list of events for which StarPU provides callbacks is given by the following enum:

enum starpu_prof_tool_event { starpu_prof_tool_event_none = 0, starpu_prof_tool_event_init , starpu_prof_tool_event_terminate, starpu_prof_tool_event_init_begin, starpu_prof_tool_event_init_end, starpu_prof_tool_event_driver_init, starpu_prof_tool_event_driver_deinit, starpu_prof_tool_event_driver_init_start, starpu_prof_tool_event_driver_init_end, starpu_prof_tool_event_start_cpu_exec, starpu_prof_tool_event_end_cpu_exec, starpu_prof_tool_event_start_gpu_exec, starpu_prof_tool_event_end_gpu_exec, starpu_prof_tool_event_start_transfer, starpu_prof_tool_event_end_transfer, starpu_prof_tool_event_user_start, starpu_prof_tool_event_user_end }
;

Initialization and shutdown

When the StarPU runtime environment is initialized, it triggers the callback corresponding to the event: starpu_prof_tool_event_init When it is shut down, it triggers the callback corresponding to the event: starpu_prof_tool_event_terminate The initialization itself begins and ends with the following events, respectively:

starpu_prof_tool_event_init_begin starpu_prof_tool_event_init_end
The driver, one for every computation resource, begins and ends with the following events, respectively: starpu_prof_tool_event_driver_init starpu_prof_tool_event_driver_deinit

Tasks

Tasks start and end on CPU resources and trigger the following callbacks: starpu_prof_tool_event_start_cpu_exec starpu_prof_tool_event_end_cpu_exec They start and end on GPU resources and trigger the following callbacks: starpu_prof_tool_event_start_gpu_exec starpu_prof_tool_event_end_gpu_exec

Memory transfers

The active part of memory transfers (i.e., when StarPU is in memory transfer-related routines) start and end triggering the following callbacks, respectively: starpu_prof_tool_event_start_transfer starpu_prof_tool_event_end_transfer

Other callbacks

Users can define their own callbacks: starpu_prof_tool_event_user_start starpu_prof_tool_event_user_end

In addition, the following callback is defined as a dummy event and always set to 0: starpu_prof_tool_event_none

Scheduler performance counters

This chapter presents the StarPU performance monitoring framework. It summarizes the objectives of the framework. It then introduces the entities involved in the framework. It presents the API of the framework, as well as some implementation details. It exposes the typical sequence of operations to plug an external tool to monitor a performance counter of StarPU.

Objectives

The objectives of this framework are to let external tools interface with StarPU to collect various performance metrics at runtime, in a generic, safe, extensible way. For that, it enables such tools to discover the available performance metrics in a particular StarPU build as well as the type of each performance counter value. It lets these tools build sets of performance counters to monitor, and then register listener callbacks to collect the measurement samples of these sets of performance counters at runtime.

Entities

The performance monitoring framework is built on a series of concepts and items, organized in a consistent way. The corresponding C language objects should be considered opaque by external tools, and should only be manipulated through proper function calls and accessors.

Performance Counter

The performance counter entity is the fundamental object of the framework, representing one piece of performance metrics, such as for instance the total number of tasks submitted so far, that is exported by StarPU and can be collected through the framework at runtime. A performance counter has a type and belongs to a scope. A performance counter is designated by a unique name and unique ID integer.

Performance Counter Type

A performance counter has a type. A type is designated by a unique name and unique ID number. Currently supported types include:

Type Name Type Counter is within the scope of a task codelet

Performance Counter Set

A performance counter set is a subset of the performance counters belonging to the same scope. Each counter of the scope can be in the enabled or disabled state in a performance counter set. A performance counter set enables a performance monitoring tool to indicate the set of counters to be collected for a particular listener callback.

Performance Counter Sample

A performance counter sample corresponds to one sample of collected measurement values of a performance counter set. Only the values corresponding to enabled counters in the sample's counter set should be observed by the listener callback. Whether the sample contains valid values for counters disabled in the set is unspecified.

Performance Counter Listener

A performance counter listener is a callback function registered by some external tool to monitor a set of performance counters in a particular scope. It is called each time a new performance counter sample is ready to be observed. The sample object should not be accessed outside of the callback.

Application Programming Interface

The API of the performance monitoring framework is defined in the starpu_perf_monitoring.h public header file of StarPU. This header file is automatically included with starpu.h. An example of use of the routines is given in Section 3.6.

Implementation Details

Performance Counter Registration

Each module of StarPU can export performance counters. In order to do so, modules that need to export some counters define a registration function that is called at StarPU initialization time. This function is responsible for calling the _starpu_perf_counter_register() function once for each counter it exports, to let the framework know about the list of counters managed by the module. It also registers performance sample updater callbacks for the module, one for each scope for which it exports counters.

Performance Sample Updaters

The updater callback for a module and scope combination is internally called every time a sample for a set of performance counter must be updated. Thus, the updated callback is responsible for filling the sample's selected counters with the counter values found at the time of the call. Global updaters are currently called at task submission time, as well as any blocking tasks management function of the StarPU API, such as the starpu_task_wait_for_all(), which waits for the completion of all tasks submitted up to this point. Per-worker updaters are currently called at the level of StarPU's drivers, that is, the modules in charge of task execution of hardware-specific worker threads. The actual calls occur in-between the execution of tasks. Per-codelet updaters are currently called both at task submission time, and at the level of StarPU's drivers together with the per-worker updaters. A performance sample object is locked during the sample collection. The locking prevents the following issues:

• The listener of sample being changed during sample collection;

• The set of counters enabled for a sample being changed;

• Conflicting concurrent updates;

• Updates while the sample is being read by the listener.

The location of the updaters' calls is chosen to minimize the sequentialization effect of the locking, in order to limit the level of interference of the monitoring process. For Global updaters, the calls are performed only on the application thread(s) in charge of submitting tasks. Since, in most cases, only a single application thread submits tasks, the sequentialization effect is moderate. Per-worker updates are local to their worker, thus here again the sample lock is un-contented, unless the external monitoring tool frequently changes the set of enabled counters in the sample.

Counter operations

In practice the sample updaters only take snapshots of the actual performance counters. The performance counters themselves are updated with ad-hoc procedures depending on each counter. Such procedures typically involve atomic operations. While operations such as atomic increments or decrements on integer values are readily available, this is not the case for more complex operations such as min/max for computing peak value counters (for instance in the global and per-codelet counters for peak number of submitted tasks and peak number of ready tasks waiting for execution), and this is also not the case for computations on floating point data (used for instance in computing cumulated execution time of tasks, either per worker or per codelet). The performance monitoring framework therefore supplies such missing routines, for the internal use of StarPU.

Runtime checks

The performance monitoring framework features a comprehensive set of runtime checks to verify that both StarPU and some external tool do not access a performance counter with the wrong typed routines, to quickly detect situations of mismatch that can result from the evolution of multiple pieces of software at distinct paces. Moreover, no StarPU data structure is accessed directly either by the external code making use of the performance monitoring framework. The use of the C enum constants is optional; referring to values through constant strings is available when more robustness is desired. These runtime checks enable the framework to be extensible. Moreover, while the framework's counters currently are permanently compiled in, they could be made optional at compile time, for instance to suppress any overhead once the analysis and optimization process has been completed by the programmer. Thanks to the runtime discovery of available counters, the applicative code, or an intermediate layer such as skeleton layer acting on its behalf, would then be able to adapt to performance analysis builds versus optimized builds. Step 0: Initialize the library First, the library needs to be initialized. StarPU opens it, looks for the function starpu_prof_tool_library_register, and calls it. This is the place where callbacks must be registered; a registration and an un-registration function are provided when this function returns.

Exported Counters

void starpu_prof_tool_library_register(starpu_prof_tool_entry_register_func reg, starpu_prof_tool_entry_register_func unreg) { /* ... */ }

Step 1: Register callbacks

We can either use a different function for each callback, or the same function for each callback. If we follow the latter option, the even type that triggered the callback can be identified using the parameters passed to the callback. void starpu_prof_tool_library_register(starpu_prof_tool_entry_register_func reg, starpu_prof_tool_entry_register_func unreg) { enum starpu_prof_tool_command info = 0; reg(starpu_prof_tool_event_driver_init, &driver_init_cb, info); reg(starpu_prof_tool_event_driver_init_start, &driver_init_start_cb, info); reg(starpu_prof_tool_event_driver_init_end &driver_init_end_cb, info); reg(starpu_prof_tool_event_driver_start_cpu_exec, &myFunction_cb, info); reg(starpu_prof_tool_event_driver_end_cpu_exec, &myFunction_cb, info); /* ... */ }

Step 2: Use the information passed to the callbacks The callback functions are passed some information about the context in which the event has happened. The fields of the data structures are described in this document. For instance, we can get the event type and the worked it happened on; for callbacks related to the execution of tasks, we can get a pointer to the function executed by the task (which can be resolved, for instance, using a tool like BFD).

void myfunction_cb(struct starpu_prof_tool_info* prof_info, union starpu_prof_tool_event_info* event_info, struct starpu_prof_tool_api_info* api_info ) { switch( prof_info->event_type ) { case starpu_prof_tool_event_start_cpu_exec: printf( "Start function %p on worker %d\n", prof_info->fun_ptr, prof_info->worker_id ); break; case starpu_prof_tool_event_end_cpu_exec, printf( "End function %p on worker %d\n", prof_info->fun_ptr, prof_info->worker_id ); break; } }

Step 3: Load the library at run-time

The library can be loaded by StarPU using either the LD_PRELOAD or the STARPU_PROF_TOOL environment variable: STARPU_PROF_TOOL=./libstarpu_myprof.so ./myprogram

In a StarPU application

This section presents a typical sequence of operations to interface an external tool with some StarPU performance counters. In this example, the counters monitored are the per-worker total number of executed tasks (starpu.task.w_total_executed) and the tasks' cumulated execution time (starpu.task. w_cumul_execution_time).

Step 0: Initialize StarPU StarPU must first be initialized, by a call to starpu_init(), for performance counters to become available, since each module of StarPU registers the performance counters it exports during that initialization phase. int ret = starpu_init(NULL);

Step 1: Allocate a counter set A counter set has to be allocated on the per-worker scope. The per-worker scope id can be obtained by name, or with the pre-defined enum value starpu_perf_counter_scope_per_worker. enum starpu_perf_counter_scope w_scope = starpu_perf_counter_scope_per_worker; struct starpu_perf_counter_set *w_set = starpu_perf_counter_set_alloc(w_scope);

Step 2: Get the counter IDs Each performance counter has a unique ID used to refer to it in subsequent calls to the performance monitoring framework.

int id_w_total_executed = starpu_perf_counter_name_to_id(w_scope, "starpu.task.w_total_executed"); int id_w_cumul_execution_time = starpu_perf_counter_name_to_id(w_scope, "starpu.task.w_cumul_execution_time");

Step 3: Enable the counters in the counter set

This step indicates which counters will be collected into performance monitoring samples for the listeners referring to this counter set.

starpu_perf_counter_set_enable_id(w_set, id_w_total_executed); starpu_perf_counter_set_enable_id(w_set, id_w_cumul_execution_time);

Step 4: Write a listener callback This callback will be triggered when a sample becomes available. Upon execution, it reads the values for the two counters from the sample and displays these values, for the sake of the example. This step allocates the listener structure and prepares it to listen to the selected set of per-worker counters. However, it is not actually active until Step 6, once it is attached to one or more worker. struct starpu_perf_counter_listener * w_listener = starpu_perf_counter_listener_init(w_set, w_listener_cb, NULL);

Step 6: Set the listener on all workers This step actually makes the listener active, in this case on every StarPU worker thread.

starpu_perf_counter_set_all_per_worker_listeners(w_listener);

After this step, any task assigned to a worker will be counted in that worker selected performance counters, and reported to the listener.

Step 5 :

 5 void w_listener_cb(struct starpu_perf_counter_listener *listener, struct starpu_perf_counter_sample *sample, void *context) { int32_t w_total_executed = starpu_perf_counter_sample_get_int32_value(sample, id_w_total_executed); double w_cumul_execution_time = starpu_perf_counter_sample_get_double_value(sample, id_w_cumul_execution_time); printf("worker[%d]: w_total_executed = %d, w_cumul_execution_time = %lf\n"Initialize the listener


Felicien Ihirwe

Davide Di Ruscio

Simone Gianfranceschi
email: simone.gianfranceschi@intecs.it1davide.diruscio

Alfonso Pierantonio
email: alfonso.pierantonio@univaq.it

Assessing the Quality of Low-Code and Model-driven Engineering Platforms for Engineering IoT Systems

Keywords: Software quality, Low-code Development platforms, Model-driven Engineering, Internet of Things

Over the last few years, industry and academia have proposed several Low-Code and Model-driven Engineering (MDE) platforms to ease the engineering process of the Internet of things (IoT) systems. However, deciding whether such engineering platforms meet the minimum required software quality standards is not straightforward. Software quality can be defined as the degree to which a software system achieves its intended goal. Various software quality standards have been established to aid in the software quality assessment process; however, due to the nature of engineering IoT platforms, such models may not entirely suit the IoT domain. This paper presents a model for assessing the software quality of Low-Code and MDE platforms for engineering IoT platforms. The proposed software quality model is based on and extends the ISO/IEC 25010:2011 software product quality model standard. It is intended to assist IoT practitioners in assessing and establishing quality requirements for engineering IoT platforms. To determine the effectiveness of the proposed model, we used it to evaluate the quality of 17 IoT engineering platforms, and the results obtained are promising.

I. INTRODUCTION

The Internet of Things (IoT) systems offer enormous benefits in our daily lives by enabling seamless communication with our surroundings. Such systems demand many development skills, from handling tiny microcontrollers to more extensive and complex cloud-based systems. In the IoT domain, systems often include safety-critical tasks that, if mishandled, might have disastrous consequences and even cost human lives [START_REF] Ciccozzi | Model-driven engineering for mission-critical iot systems[END_REF]. To guarantee robustness and safety during operation, it is critical to investigate the correctness and the quality of the platforms and the process by which systems are developed.

Over the last years, both academia and industry have proposed novel languages and tools to support the engineering of IoT systems. To this end, MDE and Low-Code paradigms are employed to conceive engineering platforms specific to the IoT domain. In the software engineering process, MDE promotes the use of models as first-class citizens in software development [START_REF] Ihirwe | Towards a modeling and analysis environment for industrial iot systems[END_REF]. Its goal is to improve productivity and reduce time to market by allowing the development of systems using models defined with concepts that are much less linked to the underlying implementation technology and much closer to the problem domain [START_REF] Di Ruscio | Model Transformations[END_REF]. Low-Code Development Platforms (LCDPs) are cloud-based software development platforms that leverage a Platform-as-a-Service paradigm to enable users with little or no programming skills to create fully functional apps with dynamic graphical user interfaces [4]- [START_REF] Khorram | Challenges & opportunities in low-code testing[END_REF]. For the sake of readability, we use IoT engineering platforms when we need to refer Low-Code and MDE platforms indistinguishably.

Deciding whether an IoT engineering platform meets the required standards for adoption in terms of quality is not a straightforward process as it involves considering and exploring various sources of information. MDE and low-code development technologies frequently rely on rigorous verification and validation processes conducted under predefined arbitrary constraints to guarantee the quality of generated systems. However, in most cases, the quality of the employed engineering is not taken into account seriously or is overlooked [START_REF] Kahraman | A framework for qualitative assessment of domain-specific languages[END_REF], [START_REF] Moreira | Evaluating the usability in domain-specific languages[END_REF].

Practitioners typically rely on well-established standards and practices to improve confidence in whether a system or a product fits the wanted quality requirements. The ISO/IEC 25000 to ISO/IEC 25099 series of International Standards, titled "Systems and software engineering -Systems and software Quality Requirements and Evaluation (SQUARE)", aims to address issues concerning software quality requirements specification and evaluation [START_REF] Estdale | Applying the iso/iec 25010 quality models to software product[END_REF]. The ISO/IEC 25010:2011 standard, in particular, [START_REF] Iso | :2011, Systems and software engineering -Systems and Software Quality Requirements and Evaluation (SQuaRE) -System and software quality models[END_REF], is intended to help people who are interested in developing or purchasing systems and software products to specify and evaluate their quality requirements.

This paper presents a quality model for assessing the quality of IoT engineering platforms based on the ISO/IEC 25010:2011 standard. In doing so, we enhanced the standard's product quality model by defining a product quality assessment model that is more suitable for the IoT domain. Initially, the ISO/IEC-25010:2011 standard, which replaced ISO/IEC-9126-1:2001 standard, defines two main sets of quality models, namely "quality in use model", and "product quality model". The former is related to the outcome of interaction when a product is used in a specific context. In contrast, the latter is related to software platforms' static and dynamic properties. These models are defined in terms of characteristics, with some of them further subdivided into sub-characteristics [START_REF] Iso | :2011, Systems and software engineering -Systems and Software Quality Requirements and Evaluation (SQuaRE) -System and software quality models[END_REF]. Both models are equally crucial for thoroughly assessing the quality of a given platform. However, the proposed model focuses on the product quality model since it emphasizes on platform's technical quality rather than usage quality which can be challenging to measure for platforms in their early development stages.

To evaluate the effectiveness of the proposed model, we employ it to assess the quality of 17 IoT platforms selected from our previous studies [START_REF] Ihirwe | Low-code engineering for internet of things: A state of research[END_REF], [START_REF] Ihirwe | Cloud-based modeling in iot domain: a survey, open challenges and opportunities[END_REF]. We present the methodology we used to choose such platforms, perform the quality assessment, and subsequently present and discuss the obtained results. We summarize this paper's contribution as follows:

• We propose a quality model based on the ISO/IEC 25010:2011 standard for evaluating the quality of IoT engineering platforms; • We assess the quality of 17 IoT engineering platforms by relying on the proposed model; • We present and discuss the findings as well as the limitation of the study in order to validate the effectiveness of the proposed model. The remaining of the paper is structured as follows: The background of the study is presented in Section II. The proposed product quality model is presented in Section III. In Section IV we go through the selected primary studies and present the followed evaluation process. In Section V, we present the results from the conducted assessment by reflecting specific research questions. Section VI discusses the results as well as its limitations. In Section VII, we present the related work and Section VIII concludes the paper and discusses perspective future work.

II. BACKGROUND

A. IoT Engineering Platforms

The main goal of MDE is to increase productivity and reduce time to market by enabling the development of systems using models defined with concepts that are much less tied to the underlying implementation technology, and much closer to the problem domain [START_REF] Di Ruscio | Model Transformations[END_REF]. Same as MDE, low-code software development processes aim to improve software development processes by raising abstraction and hiding implementationlevel details. Both approaches employ model-driven development (MDD) in their development stack; the important distinction seems to be how they enforce factors such as cloud-based deployment, target users, setup, and so on [START_REF] Di Ruscio | Low-code development and model-driven engineering: Two sides of the same coin?[END_REF]. In addition to that, not all MDE approaches seek to reduce the amount of code required to develop software solutions, and not all low-code approaches are model-driven [START_REF] Di Ruscio | Low-code development and model-driven engineering: Two sides of the same coin?[END_REF]. Although this is true, their difference in practice is not yet clear, and it is widely debated how much work done in MDE is directly transferable to LCDPs [START_REF] Cabot | Positioning of the low-code movement within the field of model-driven engineering[END_REF].

Nowadays, we witness a growing number of successful generic LCDPs on the market (e.g., Google App Maker and Microsoft Power Platform). Despite their success, LCDPs' development capabilities are still limited regarding how complex, intelligent, and sophisticated they may be [START_REF] Tisi | Lowcomote: Training the Next Generation of Experts in Scalable Low-Code Engineering Platforms[END_REF]. In the IoT domain, only a few LCDPs are available, and they provide limited functionalities given the inherent complexity and heterogeneity of typical IoT systems. Among others, IoT-specific platforms such as Node-RED [START_REF]Node-RED: Low-code development platform[END_REF] and Atmo-sphericIoT [START_REF] Atmosphereiot | Fast time to first data[END_REF] have demonstrated a significant push toward the development of fully-fledged multi-layer IoT platforms. However, their capacity to generate code and interact with lowlevel IoT devices is still limited. Unlike LCDPs, MDE presents a more significant number of platforms that can still generate low-level platform-specific code. Still, they often suffer from integration and interoperability issues because most of them are deployed and operated locally [START_REF] Ihirwe | Low-code engineering for internet of things: A state of research[END_REF].

B. Software Quality Models

The discipline of Software Quality Engineering [START_REF] Kan | Metrics and Models in Software Quality Engineering[END_REF] is concerned with improving the approach to software quality. However, the various perspectives present throughout the software life cycle show what constitutes software quality is frequently contested. In this context, relying on software quality models to support the quality management of software systems is widely accepted [START_REF] Ortega | Construction of a systemic quality model for evaluating a software product[END_REF]. A quality model is typically defined as a set of sub-characteristics and their interrelationships that serve as the foundation for specifying quality standards and evaluating quality [START_REF] Gordieiev | Evolution of software quality models in context of the standard iso 25010[END_REF]. Various standards have been defined in the literature to assist in the evaluation process. The ISO/IEC 9126:1991 standard aimed at defining a software quality paradigm and a set of guidelines for assessing the characteristics associated with it [START_REF] Côté | In search for a widely applicable and accepted software quality model for software quality engineering[END_REF]. This standard was later revised by ISO/IEC 25010:2011, which included a model for software system quality with well-formed specifications for quality characteristics of software products [START_REF] Kahraman | A framework for qualitative assessment of domain-specific languages[END_REF].

The ISO/IEC 25010:2011 standards help design, develop and acquire systems and software products with the specification for evaluating their quality requirements [START_REF] Estdale | Applying the iso/iec 25010 quality models to software product[END_REF]. The standards are made up of two quality models, each with its own set of characteristics, some of which are further subdivided into sub-characteristics. First, the "quality in use model" focuses on the outcome of interaction when a product is used under particular contexts [START_REF] Iso | :2011, Systems and software engineering -Systems and Software Quality Requirements and Evaluation (SQuaRE) -System and software quality models[END_REF]. This model is primarily intended to provide targets for promoting the development and verification efficiency, as well as to anticipate quality in use before delivery [START_REF] Estdale | Applying the iso/iec 25010 quality models to software product[END_REF]. This model includes five characteristics, namely effectiveness, efficiency, satisfaction, risk freedom, and context coverage where some of them are further subdivided into nine sub-characteristics. The "product quality model" refers to the static and dynamic qualities of a software platform. This model primarily focuses on providing assessment ground for people supplying software products and those acquirers who wish to get more involved in the process technically [START_REF] Estdale | Applying the iso/iec 25010 quality models to software product[END_REF]. The model is divided into eight characteristics, namely functional appropriateness, performance efficiency, compatibility, reliability, usability, security, maintainability, and portability which are further elaborated into 31 sub-characteristics.

In the past, the ISO/IEC 25010:2011 standard has been adopted to assess not only the product quality of IoT systems [START_REF] Hussain | An application of the iso/iec 25010 standard in the quality-in-use assessment of an online health awareness system[END_REF]- [START_REF] Tambotoh | Software quality model for internet of things governance[END_REF] but also in domains such as Big data [START_REF] Rahman | Systematic mapping study of non-functional requirements in big data system[END_REF], Machine Learning [START_REF] Siebert | Towards guidelines for assessing qualities of machine learning systems[END_REF], Software Product Lines (SPL) [START_REF] Luana | [END_REF], Customer Relationship Management (CRM) systems [START_REF] Boarim | CRM systems quality evaluation[END_REF] and mobile apps [START_REF] Koepp | The quality of mobile apps used for the identification of pressure ulcers in adults: Systematic survey and review of apps in app stores[END_REF], just to mention a few. In MDE domain, approaches such as [START_REF] Goulão | Quality in model-driven engineering: A tertiary study[END_REF] relied on it for assessing the performance of MDE quality studies, while [START_REF] Kahraman | A framework for qualitative assessment of domain-specific languages[END_REF], [START_REF] Moreira | Evaluating the usability in domain-specific languages[END_REF] adopted it in order to assess the quality of Domain Specific Languages (DSLs) whileand design architectures quality [START_REF] Darwish | Framework for engineering design systems architectures evaluation and selection: Case study[END_REF] adopted it to assess the quality of design architectures. Although ISO/IEC 25010:2011 cannot be considered a one-size-fits-all solution, it can be a beneficial approach for evaluating the quality of IoT engineering platforms. A detailed description of the mentioned quality characteristics is given in the next section.

III. THE PRODUCT QUALITY MODEL

In this section, we discuss the product quality model of the ISO/IEC 25010:2011 standard to enable the quality evaluation of IoT engineering platforms. To this end, by referring to Figure 1, in the following, we discuss the common model characteristics concerning the peculiarities of the IoT domain.

A. Functional suitability

This factor evaluates how well a platform meets specified and implied objectives when used in specific contexts. In the IoT engineering context, its corresponding sub-characteristics can be defined as follows.

Functional completeness is the extent to which a given platform supports the design of all the layers that comprise an IoT ecosystem. For instance, the platform's capacity to represent the complete IoT ecosystem from the edge, fog, to the cloud layers while seemingly handling all of the communication heterogeneity that might occur. Concerning Functional Correctness, this can be defined as the extent to which the platform applies specific correctness methodologies during the development phases, including but not limited to correct by construction, model-checking, model validation, and rulebased modeling. Finally, the Functional appropriateness refers to the amount to which the given functionalities facilitate the completion of defined activities and objectives. IoT engineering platforms have diverse functionalities; in this case, we can propose to analyze how the specified correctness methodology permits achieving the goal.

B. Performance efficiency

This characteristic determines how the platforms consume resources under specific conditions. This characteristic is mainly related to run-time attributes of the underlying system infrastructure, which in some situations cannot be predicted accurately. In the context of IoT engineering platforms, the corresponding sub-characteristics can be defined as follows: Time behavior refers to how well a platform function meets its criteria in terms of response and processing times, as well as throughput rates. The platform's responsiveness can primarily indicate this, and it is worth noting that it can only be evaluated on platforms that are accessible.

Resource utilization measures how much and what kind of resources are used to suit the platform's needs. Although this can somewhat be predicted when a platform runs on well-known underlying infrastructures, the actual utilization can only be evaluated accurately during the platform's usage. Finally, the Capacity is defined to assess the platform's ability to model and coordinate a large and complex system that spans multiple IoT sub-domains.

C. Compatibility

This characteristic implies the platform's capacity to interchange data with other platforms. The two sub-characteristics are defined as follows: Co-existence refers to the degree to which a platform continues to operate efficiently while sharing a shared environment and resources with other platforms without hammering the other platforms. Interoperability is referred to the degree to which a platform can exchange information with other platforms while in use. In our context, this can be assessed in terms of the platform's capability to support the data exchange during the development process by consuming or sending information to/from external services via dedicated means such as REST APIs.

D. Reliability

This characteristic reflects how a platform can complete a set of tasks in a given amount of time. Its sub-characteristics can be defined as follows: Maturity reflects how well the platform supports all of the basic functionalities of a typical engineering platform, such as design, code generation, and deployment. Availability is concerned with the extent to which a platform is operational when needed. Fault tolerance is the degree to which a platform performs as expected despite the presence of hardware or software faults. In our case, this can be evaluated as the platform's capability to support advanced mechanisms, including but not limited to self-adaption and self-healing. Finally, the Recoverability reflects the extent to which a platform, in case of interruptions or failures, can recover data directly affected and re-establish the desired state of the system. To that end, the platform should be able to handle the mechanisms including but not limited to selfrecovery or self-redeployment, etc.

E. Usability

This characteristic reflects how well specific users can use a platform to achieve particular goals with efficiency, effectiveness, and satisfaction. The sub-characteristics are as follows: Appropriateness recognizability is defined as the degree to which users can determine whether a platform is appropriate for their needs. This can be promoted in the IoT engineering world by the degree to which the platform is customized concerning the underlying environment. Learnability can be defined as the extent to which the platform aids developers in learning how to use it, e.g. with context-based modeling support, on-the-fly suggestions, and so on.

Operability represents the extent to which a platform has attributes that make the entire development process more accessible, for instance, using functionalities like auto-completion for textual languages, guide-through mechanisms, multi-view modeling, palette show/hide, and palette element search. User error protection reflects the extent to which the platform provides some protection to avoid errors, such as static analysis or on-the-fly error handling. User interface aesthetics is the extent to which the platform provides pleasant and satisfying user interfaces. Finally, Accessibility can be considered as to

F. Security

This characteristic refers to how successfully a platform protects information and data so that users, services, and external systems have appropriate access to data according to specific authorization levels. Its six sub-characteristics have been renamed as follows: Confidentiality refers to the extent to which a platform assures that data is exclusively available to those who have been granted access. Integrity can be defined as the extent to which an IoT platform prohibits unwanted access, modification of the platform, or data.

Non-repudiation can be measured by the extent to which the platform enables methods for recording all actions conducted during development and proves that they have been performed so that they cannot be contradicted later. Accountability can be measured as the extent to which a platform enables tracing attributes like versioning, historical action retrieval, and so on. Finally, Authenticity can be described as the ability of the plat-form to support different types of authentication mechanisms while accessing platform resources.

G. Maintainability

This characteristic refers to the degree of a platform to be improved, repaired, or adapted to changes. Its corresponding sub-characteristics are defined below: Modularity can be measured as the extent to which a platform is decoupled into discrete sub-systems that can be modified independently of other parts. Reusability reflects the extent to which a platform or a component of a platform can be used in more than one system. While evaluating this, we can focus on the platform's ability to be decomposed into small reusable sub-systems.

Analyzability assesses the efficacy and efficiency with which it is feasible to determine the impact of modifying parts of the platform by either removing it or injecting failures into it. In the IoT engineering context, whether the platform provides means supporting the analysis of the system under development and/or the platform itself is also looked at. In terms of Modifiability, the assessment can be performed on the extent to which a platform can be successfully and efficiently modified without introducing flaws or deteriorating the quality of existing products. Finally, Testability can be assessed in terms of the capability of the platform to provide testing supports for its components or the system under development.

H. Portability

This characteristic refers to how fast and successfully a platform can be moved from or to different hardware and software operational environments. Its corresponding subcharacteristics are defined below: Adaptability refers to the extent to which a platform can be effectively and efficiently adapted to different environments. Installability reflects the degree of effectiveness and efficiency with which a platform can be successfully installed and/or uninstalled in a given environment. Replaceability measures the extent to which a platform can be updated, replaced, and redeployed in the same environment and still performs as expected.

IV. QUALITY ASSESSMENT OF IOT ENGINEERING

PLATFORMS

This section shows the quality assessment process followed to analyze 17 IoT engineering platforms using the product quality model discussed in the previous section. The platforms of interest have been identified as presented in Section IV-A. The research questions that we answered through the performed evaluation are presented in Section IV-B. The quality assessment process that has been followed is presented in Section IV-C.

A. Selection of the evaluated IoT engineering platforms

In [START_REF] Ihirwe | Low-code engineering for internet of things: A state of research[END_REF], we examined 16 different platforms to gain a better understanding of the current state of the art in supporting the development of IoT systems, with a focus on languages and tools available in the MDE field and emerging LCDPs. In [START_REF] Ihirwe | Cloud-based modeling in iot domain: a survey, open challenges and opportunities[END_REF], we examined 22 IoT modeling environments, assessing their strengths and weaknesses in terms of cloud-based modeling capacity, accessibility, openness, and artifact generation. Combining the two data sources, we started with a total of 38 approaches, which have been filtered by considering the following exclusion criteria:

• Approaches that were published before 2012 have been discarded; • Duplicated approaches have been removed; • Approaches that do not permit developing fully functional IoT applications have not been considered. • Approaches that depend on already-included platforms have been filtered out; • Approaches that are generic and that do not explicitly target the IoT domain have not been considered in this study. By applying such criteria, 15 out of the initial 38 IoT development platforms were identified. In addition, we have added two more approaches that we believe are very promising and were published after our first study, bringing the total to 17. Figure 2 depicts the detailed paper selection procedure that we used. To better understand the selected basic studies, we have categorized them quantitatively based on publishing year, publisher type, article type, and the distribution among MDE and LCDPs approaches. The result is shown in Figure 3a, 3b, 3c and 3d respectively.

B. Research questions

The performed assessment aimed at answering the following research questions:

C. Assessment process

The quality assessment process of the considered IoT engineering platform has been done iteratively by going over all of the reference papers. We established a set of questions for each sub-characteristic that must be addressed to confirm the platform's competence with respect to what is included in the model described in Section III. By doing this, we made the evaluation process result easier and more reflective of the proposed model. Following that, we read the entire document and responded to the questions. Each question can be answered with "Yes" or "No". For instance, the following questions have been formulated to help in assessing the platform's Functional Suitability:

• Does the platform support the design and development of IoT systems? • Does it support all layers (Edge, Fog, Cloud)?

• Does it mention any support for dealing with different communication protocols? Consequently, given the paper under analysis, if the corresponding presented platform succeeds on at least 50% of the questions, it is marked as supporting the quality characteristic of interest. This procedure has been developed and implemented for all 31 sub-characteristics. The defined questions have been created purely to facilitate the review process and are entirely consistent with what is provided in the model. Unfortunately, due to space constraints, we could not present the extended table of questionnaire used to assess all of the quality characteristics. In this regards, a full set of evaluation questionnaire used in the evaluation has been published and can be accessed from an online database available at [START_REF] Ihirwe | Software product quality evaluation questionnaire for IoT LCDP&MDE[END_REF].

As mentioned in the previous section, some of the characteristics heavily depend on the dynamic properties of the underlying system infrastructures as well as during their usage. For instance, verifying sub-characteristics related to performance efficiency and usability might be difficult in general. Thus, during the performed analysis we relayed on information provided in the reference papers when available. We considered such kind of characteristics unsupported if the corresponding articles do not mention any mechanism addressing them. As shown in Table I, MDE approaches support an average of 12 of the 31 possible sub-characteristics, whereas LCDPs support an average of 18. Furthermore, different sub-characteristics are not supported at all. Figure 4 shows the overall characteristic-level performance of the analyzed platforms. Figure 4a and Figure 4b depict the overall performance of MDE platforms and LCDPs against the eight main characteristics, respectively. To this end, an aggregated performance sum from all sub-characteristics was produced for each characteristic. Both categories (MDE and LCDPs) perform well enough in quality characteristics related to Functional suitability, Portability, and Usability, with general supporting rates of 75.8%, 57.8%, and 50% for MDE and 83.3%, 66.7%, and 66.7% for LCDPs, respectively. It is important to note that the listed supporting/non-supporting rates reflect the aggregated characteristic performance calculations from its corresponding sub-characteristics. Although this provides us with an overall picture of characteristic performance, it does not allow us to make a final judgment on whether certain sub-characteristics are better supported than others. For example, in terms of Usability aspects, all 11 MDE platforms satisfy the User interface quality characteristic, although only two of them satisfy the accessibility sub-characteristic. MDE platforms have limited security and performance efficiency support, with overall support rates of 5.5% and 24.2%, respectively. For instance, in MDE, from the studies considered, only IoTML [START_REF] Conzon | BRAIN-IoT: Model-based framework for dependable sensing and actuation in intelligent decentralized iot systems[END_REF] promotes security by enhancing authentication, confidentiality, and integrity mechanisms while accessing the platform. On the other hand, LCDPs fall short concerning security and maintainability, with overall support rates of 36.7% and 30%, respectively. Each of the security aspects is implemented at least once by LCDPs, with the AtmosphereIoT [START_REF] Atmosphereiot | Fast time to first data[END_REF] platform supporting all. Consequently, as shown in Table I, MDE accounts for 135 points out of 341 possible support, representing approximately 39.6%, whereas LCDPs account for 95 points out of 186 possible, which would be about 51.1%.

Answer to RQ1: Overall, MDE quality is around 39.6%, while LCDPs account for 51.1%, resulting in an overall quality of the selected engineering platforms being approximately 45.5%.

2) Quality sub-characteristics support of IoT engineering platforms (RQ2): In the previous section, we focused on overall quality characteristic support; in this section, we focus on individual sub-characteristic by highlighting the most and least addressed ones. Even though the average supporting rate of studied quality sub-characteristics for both MDE and LCDPs will be equal to the average supporting rate for the As shown in Table I, IoT MDE and LCDPs cover all quality sub-characteristics such as user interface and installability. This is expected, given that the fundamental principle of MDE and LCDP technologies centers around enabling an easy-touse and executable environment for developing applications with less effort, which cannot be accomplished without providing a user interface.

Figure 5 depicts an overall performance of subcharacteristics among IoT MDE, and LCDPs engineering platforms. As indicated, none of the chosen MDE approaches address quality sub-characteristics such as non-repudiation, time-behavior, accountability, and co-existence. LCDPs, on the other hand, fall short on fault-tolerance, non-repudiation, reusability, analyzability, and testability, with a consistent minimal support rate of 16.7%, which implies a generic rate of at least 1 out of 6 LCDPs supports at least one of the subcharacteristics. On the other hand, Figure 6 depicts the average quality performance for both MDE and LCDP platforms. As can be seen, sub-characteristics such as user interface, installability, and functional completeness are most well supported by both categories, while testability, time-behavior, accountability, and non-repudiation are the least supported.

Answer to RQ2: The top three quality subcharacteristics addressed by both technologies are user interface, installability, and maturity. On the other hand, selected MDE approaches were unable to address quality sub-characteristics such as non-repudiation, accountability, co-existence, and time-behavior. In contrast, LCDPs fall short of addressing fault tolerance, non-repudiation, and reusability with a consistent rate of 1 out of 6 LCDPs.

VI. DISCUSSION

In this section, we discuss the proposed model's suitability and its limitation with respect to general software quality assessment of software systems.

A. Model suitability

The proposed model aims to assist practitioners who have to design and develop IoT systems by exploiting low-code or MDE platforms. According to the results presented in Section V, security-related characteristics are the least addressed. This is particularly pertinent given how IoT security concerns are dynamic and unstructured, leading to uncertainty among software developers in terms of concepts and terms [START_REF] Mozzaquatro | Towards a reference ontology for security in the internet of things[END_REF]. MDE approaches are most affected since most conventional MDE platforms are deployed locally and used offline, making incorporating any form of security capabilities less required (e.g., authentication). We can argue that the dominant Eclipse Development Environment1 , which hosts a lot of classical MDE-based platforms, has a significant impact on this problem. On the other hand, although LCDPs neither excel in such security-related aspects, it is critical and rational to be integrated since such platforms are deployed in cloudbased environments, which are more likely to be attacked by unwanted intruders.

Furthermore, besides security, the results show that LCDPs lack quality criteria for general maintainability. For instance, according to the Table I, the reusability of LCDP is shown to be less supported among others. This is generally true and can mainly be since more LCDPs are tailor-made, and most of their application developments and deployments are bound to a particular technology [START_REF] Di Ruscio | Low-code development and model-driven engineering: Two sides of the same coin?[END_REF], making them difficult to modify and reuse elsewhere. In our study, only Node-RED [START_REF]Node-RED: Low-code development platform[END_REF] showcased means for supporting the usability of aspects of its components through the Node-RED modules that can be composed, built, deployed and reused separately from one instance to another.

Due to the tight coupling between functions and their subfunctions found in different LCDPs and some MDE platforms, analysability becomes very hard to achieve. Our proposed model defines software analysability as the means to assess the efficacy and efficiency with which it is feasible to determine the impact of modifying parts of the platform by either removing it or injecting failures into it. According to the obtained results, only 27% of MDE platforms support such features, while for LCDPs, only one out of 6 support them. This Still, in the actual implementation of the system, such quality is often ignored [START_REF] Bouwers | Quantifying the analyzability of software architectures[END_REF]. Concerning software testability, as indicated above, only 2 out of 11 of the analysed platforms showed means for supporting it. The testing can either be done at the platform sub-functions level or at the system they develop. In general, developers tend to disregard this aspect. For instance, in the LCDP domain, low-code testing is still in its early stages, with no formal structure to the domain's ideas, concepts, and hypotheses which undoubtedly contributes to such lacking [START_REF] Khorram | Challenges & opportunities in low-code testing[END_REF].

Another interesting fact is that the overall performance of IoT engineering platforms (MDEs and LCDPs combined), regardless of characteristics and sub-characteristics, is about 45.5%, in which MDE accounts for 39.6%, whereas LCDPs have 51.1%. Although we acknowledge that these measures cannot be regarded as a definitive measure of the IoT engineering platforms' quality questions, we believe they can point researchers in the right direction regarding the present state of the art in IoT engineering quality evaluation support. Consequently, we can finally draw the line of how the proposed model satisfies the quality evaluation procedure as "promising"; it unveiled several concepts that reflect what is already available in the IoT engineering domain.

B. Limitations

In this paper, we have extended the ISO/IEC 25010:2011 standard model to propose a software product model for assessing the quality of IoT engineering platforms. To evaluate the effectiveness of the proposed model, we employed it to determine the quality of 17 IoT platforms selected from our previous studies [START_REF] Ihirwe | Low-code engineering for internet of things: A state of research[END_REF], [START_REF] Ihirwe | Cloud-based modeling in iot domain: a survey, open challenges and opportunities[END_REF]. In terms of limitations, we can state the followings:

• Although the presented model contributes positively to the software product quality assessment of IoT engineering platforms, it only performs very well when the evaluation is done at the platform's technical implementation level. However, the proposed model performed poorly regarding run-time software product quality assessment, such as quality linked to performance efficiency. This is primarily due to the implementation nature of the LCDP and MDE platforms. We believe that assessing such quality could be heavily influenced by the environment in which such software is deployed. • The evaluation methods employed in this study and the reported results are critical because they are exclusively based on what was identified in the selected tool's papers. However, in some instances, the published content of the paper may not correctly reflect the full capabilities of the platform under consideration. Furthermore, software platforms evolve, and new development is regularly contributed to the platforms. Therefore, we believe that integrating the results found in the papers and the actual inputs from the tool vendors can significantly increase the legitimacy of the findings. We intend to address this in future research. ISO/IEC 25010:2011 standard has been heavily used to assess the quality of complex software and systems. To name a few, [START_REF] Hussain | An application of the iso/iec 25010 standard in the quality-in-use assessment of an online health awareness system[END_REF] relied on it to assess the quality of online health awareness systems, [START_REF] Bertrand-Martinez | Classification and evaluation of iot brokers: A methodology[END_REF], [START_REF] Schipor | Euphoria: A scalable, event-driven architecture for designing interactions across heterogeneous devices in smart environments[END_REF] for IoT brokers, and [START_REF] Tambotoh | Software quality model for internet of things governance[END_REF] for IoT implementations. Although this quality model's scope is intended general for software and computer systems, it can also be applied to assess larger systems and services [START_REF] Iso | :2011, Systems and software engineering -Systems and Software Quality Requirements and Evaluation (SQuaRE) -System and software quality models[END_REF]. For instance, in [START_REF] Koepp | The quality of mobile apps used for the identification of pressure ulcers in adults: Systematic survey and review of apps in app stores[END_REF], the model has been adopted in order to evaluate the quality of mobile applications, as well as in [START_REF] Rahman | Systematic mapping study of non-functional requirements in big data system[END_REF], [START_REF] Siebert | Towards guidelines for assessing qualities of machine learning systems[END_REF] for Machine Learning and Big data systems.

There have been several approaches in the modeling domain for quality measurements. For example, in [START_REF] Kahraman | A framework for qualitative assessment of domain-specific languages[END_REF], a Framework for Qualitative Assessment of DSLs (FQAD) was presented. The FQAD framework is based on ISO/IEC 25010:2011 when determining the evaluation's perspective, the assessment's goal, and selecting relevant quality characteristics to guide the assessment process. In addition to that, the authors in [START_REF] Moreira | Evaluating the usability in domain-specific languages[END_REF] relied on the standard to assess the quality assurance of DSLs. In contrast, [START_REF] Darwish | Framework for engineering design systems architectures evaluation and selection: Case study[END_REF], the standard was employed to evaluate the quality of design architectures.

In [START_REF] Luana | [END_REF], the authors used this standard to perform a mapping study on the quality assessment of software product lines. The standard was used in [50] to assess the security quality of mobile cloud computing-based technologies. Finally, it was used in [START_REF] Boarim | CRM systems quality evaluation[END_REF] to evaluate the quality aspects of customer relationship management (CRM) systems. A large number of extensions and suggestions on the standard have been proposed. To name a few, authors in [START_REF] Estdale | Applying the iso/iec 25010 quality models to software product[END_REF] proposed the extension to meet the lifetime service-oriented quality aspect of software systems. In [START_REF] Rahman | Systematic mapping study of non-functional requirements in big data system[END_REF], the authors conducted a study to assess the product quality of Big data systems concerning non-functional requirements, while in [START_REF] González | Swet-qum: A quality in use extension model for semantic web exploration tools[END_REF], the authors extended the model to Semantic Web exploration tools.

Finally, the authors in [START_REF] Farshidi | Model-driven development platform selection: Four industry case studies[END_REF] presents a highly comprehensive effort to assist the quality evaluation of MDD platforms, as well as a mapping from the proposed model to the ISO/IEC 25010:2011 standard. In their paper, a multi-criteria decision-making (MCDM) model for MDD platforms is used to help in choosing an optimal quality sufficient platform for their requirements. Nevertheless, the approach as well as the results presented are too generic, whereas our approach focuses primarily on IoT specific platforms. Furthermore, the approach, in our opinion, focuses mostly on LCDPs, while other conventional MDE platforms, such as those based on the Eclipse Development Environment, are not taken into account at all.

Even though all of the approaches relied on the standard used in this paper, none specifically target IoT engineering platforms. Based on the previous discussions, we can conclude that this study is novel and unique compared to other studies. While there are many quality assessments of DSLs available, only a small number of them refer to an established standard in the evaluation [START_REF] Kahraman | A framework for qualitative assessment of domain-specific languages[END_REF], and we could not find any that addresses the IoT modeling domain in particular. As a result, we are confident that our study is the premier to use a wellestablished quality standard to evaluate the product quality of IoT engineering platforms.

VIII. CONCLUSION AND FUTURE WORK

In this paper, we presented an extension of the ISO/IEC 25010:2011 product quality model for assessing the quality aspects of IoT engineering platforms. We evaluated the software product quality of 17 IoT engineering platforms using the proposed model. The findings revealed that the overall performance of IoT engineering platforms is roughly 45.5%, with LCDPs doing slightly better than MDE platforms. Furthermore, security and maintainability aspects are found to be less addressed, whereas functional appropriateness, portability, and usability were found to be the most addressed. In the future, we plan to evaluate the quality in the use of the IoT engineering platform by extending the quality in use model of the ISO/IEC 25010:2011 standard in which we will be able to accommodate other quality aspects beyond the software product quality model.

Figure 1 :

 1 Figure 1: Software Product Quality Model

•:

  RQ1To what extent do the considered IoT engineering platforms meet the characteristics of the proposed quality model? • RQ2: What are the most and the least addressed quality sub-characteristics by the considered IoT engineering platforms?

Figure 2 :

 2 Figure 2: Primary studies selection process

Figure 3 :

 3 Figure 3: Overview of the selected basic studies

Figure 4 :

 4 Figure 4: Quality characteristics support

Figure 5 :

 5 Figure 5: Quality sub-characteristics performances

Figure 6 :

 6 Figure 6: Average quality sub-characteristics performances

TABLE I :

 I Assessment overview

	Characteristic	Sub-characteristic	MDE platforms	MDE(%)	LCDP platforms	LCDP(%)
	Functional			75.8		83.3
	suitability	Func. Completeness	[32] [33] [34] [35] [36] [37] [38] [39] [40]	81.82	[41] [15] [16] [42] [43]	83.3
		Func. Correctness	[44] [33] [32] [35] [36] [45] [38] [39] [40]	81.82	[15] [16] [42] [43] [46]	83.3
		Func. Appropriateness	[44] [34] [32] [36] [38] [39] [40]	63.6	[15] [16] [42] [41] [46]	83.3
	Performance			24.2		38.9
	efficiency	Time-behavior	Unsupported	0	[15] [16]	33.3
		Resource Utilization	[32]	9.09	[15] [16]	33.3
		Capacity	[33] [32] [36] [37] [38] [39] [40]	63.3	[15] [16] [41]	50
	Compatibility			27.3		58.3
		Co-existence	Unsupported	0	[15] [16] [42]	50
		Interoperability	[32] [35] [36] [45] [38] [39]	54.5	[15] [16] [42] [43]	66.6
	Reliability			40.9		41.7
		Maturity	[44] [34] [32] [35] [36] [45] [38] [39] [40]	81.82	[15] [16] [42] [41] [43]	83.3

https://www.eclipse.org/

Availability [START_REF] Harrand | Thingml: A language and code generation framework for heterogeneous targets[END_REF] [START_REF] Kirchhof | Montithings: Model-driven development and deployment of reliable iot applications[END_REF]