Marianne Pichon
email: marianne.pichon@gmail.com

Patrick Millot
email: patrick.millot@univ-valenciennes.fr

Simon Enjalbert
email: simon.enjalbert@univ-valenciennes.fr

Toward an on-line and non-obtrusive workload assessment method

Keywords: Workload assessment methods, multi-task context, sampling period, temporal load, functional load

Workload is an index introduced in the 70 th for ergonomic purposes, for evaluating the adequacy of tasks to the human operator abilities. Methods based on a self evaluation such as SWAT and TLX gave the best results, but mainly for assessing a total workload after the task has been performed. But in highly dynamical systems as transportation, the driver (or pilot) abilities can be enhanced by driving assistance tools on-line. Therefore new challenges appear which needs new methods for assessing Workload on-line, in real time and without disturbing the human operator. At LAMIH, an online assessment method has been developed and partially validated by Millot in the past. This method is based on one hand, on two workload generators: temporal demands (time pressure) and functional demands (task difficulties). On the other hand, to cope with a possibility of assessment on-line, the workload is defined through the analogy with the physical notions of "power" and "energy". The instantaneous workload Wl(t) is seen as the "power" the human operator invests on-line in the task in order to cope with the task demands. After a time available denoted TA, the human operator has spent a quantity of "energy" WL defined as the sum of the successive instantaneous Wl(t) along TA. These ideas have been validated in multitask situations for discrete as well as continuous tasks like driving tasks, but especially with temporal demands. This paper first compares several methods with the LAMIH's method. It then proposes an extension of this LAMIH's method in order to cope with the new dynamical constraints. Finally it proposes experimental protocols for validating the new LAMIH's method by comparing it with SWAT and TLX methods.

INTRODUCTION

The present work is a part of the European FP7 project ITERATE (Information Technology for Error Remediation And Trapping Emergency). A review of workload assessment methods was needed in order to build driver behavioural model for cars, ships, trains face to critical situations.

The base of this study is the workload definition given by Sperandio was chosen (1978): workload is the rate of activity, i.e. physical mental and sensori-motor, supplied by the operator in order to perform a task.

From this definition, several characteristics of a workload assessment method of workload can be highlighted. The workload measure must take into account the different kinds of activities performed by the operator. Particularly, Casali and Wierville introduced four generic classes of activities, i.e. perception, mental, communication and psychomotor [START_REF] Casali | A comparison of rating scale, secondary-task, physiological, and primary-task workload estimation techniques in a simulated flight task emphasizing communications load[END_REF]Wierville et al., 1985). Each activity generates a dedicated i.e. perceptual load, mental load, communication load and physical load, each contributing to the global workload.

Different workload assessment methods were then tested and compared according to several criteria, i.e. sensitivity, diagnosticity, selectivity, obtrusiveness, bandwidth and reliability (O'Donnel and Eggemeier, 1986). Particularly, obtrusiveness can be observed when the same human processing mechanisms are used for the evaluation as well as the task execution. One of the major results were that the self-rate assessment workload methods gave the best results according to the criteria given previously. Two selfrate methods will be analysed in this paper, SWAT (Subjective Workload Assessment Technique) [START_REF] Reid | Subjective Workload Assessment Technique (SWAT): A user's guide[END_REF] and TLX (NASA Task Load Index, [START_REF] Hart | Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research[END_REF].

In our present dynamical application contexts, two complementary constraints for selecting a relevant workload assessment method must be added to the criteria above. The assessment method must be used on-line during the task performing and does not disturb the operator (no obtrusiveness). A track to cope with the obtrusiveness issue can be inspired from the Sheridan's method developed by Sheridan and based on time measures [START_REF] Sheridan | Definitions, Models and Measures of human Workload[END_REF]. Another way of solution is the LAMIH's method introduced by [START_REF] Millot | Supervision des procédés automatisés et ergonomie[END_REF]. The advantages of this method are the assessment on-line and the low obtrusiveness.

An analysis of these four workload assessment methods and a comparison is then presented, in order to highlight the method which reaches the maximum of the criteria.

ANALYSIS OF FOUR WORKLOAD ASSESSMENT METHODS

Two self-rate assessment methods: SWAT and TLX

Several studies have shown that SWAT and TLX are sensitive [START_REF] Hill | Comparison of four subjective workload rating scales[END_REF], non-intrusive [START_REF] Rubio | Evaluation of Subjective Mental Workload: A Comparison of SWAT, NASA-TLX, and Workload Profile Methods[END_REF] and usable workload assessment methods [START_REF] Mitchell | Assessment of New Workload Tools for the Rail Industry[END_REF].

TLX is a multi-dimensional rating procedure that provides an overall workload score based on a weighted average of rating on six dimensions: mental demands (MD), physical demands (PD), temporal demands (TD), self satisfaction regarding the operator own performance (OP), effort (EF), and frustration (or stress, FR).

Each of the six dimensions is quoted R i , i [0, 6] and is subjectively evaluated by the human operator her/himself by pointing a mark on a continuous scale between 0 to 1; R i [0, 1].

Then quantifying workload on a unique scale, needs the aggregation of each R i in weighting each one regarding the others trough a weighting factor i , as shown in the expression (1).

(1)

The human operator is asked to evaluate each . For that purpose the analyst builds combination of two dimensions (pairs) and asks the operator to choose the most important one. Therefore with the 6 dimensions, we build 15 different pair-wise comparisons of dimensions. This operation is performed after the end of the task. Despite this method cannot be used on-line, it satisfies the criteria given by O'Donnel and Eggemeier. Then it gives a total workload value, i.e. the whole resources supplied by the operator during the task duration.

SWAT assumes that workload is made of 3 dimensions; time load (temporal pressure), mental load (functional demands) and psychological stress load. Each dimension is quantified with 3 levels; Low (L), Medium (M) and High (H). Therefore the workload measure is on a range of 27 values defined by the triplet (time load level, mental load level, psychological stress load level) and asks to the operator to classify these cards from the lowest to the highest workload value. This operation is performed before the experiment. This classification depends on the appreciation of the operator who evaluates her/his own workload in a general situation. But an important issue is that it remains constant along time [START_REF] Valot | A method of designing ergonomics for activity dynamics: an aeronautical example[END_REF].

Then after this first step of calibration, the evaluation can be done online in specific situations, by asking the operator to evaluate the level of time load, mental load and psychological stress load. The place of the corresponding card among the 27 cards allows estimation of operator's workload. Experiments on aircraft pilots allow validating the method with questions, asked on-line with a sampling period of 5 minutes.

SWAT takes into account the different generators of load and satisfies to the criteria given by O'Donnel and Eggemeier. Contrary to TLX, this method can be used online but with a quite long sampling period. Moreover the operator has to stop her/his task to evaluate her/his workload and this can be obtrusive for some being of tasks which need the same processing mechanisms as the estimation.

Sheridan's Workload assessment

For mainly temporal tasks Sheridan defined workload as the operators' occupation rate (2) [START_REF] Sheridan | Definitions, Models and Measures of human Workload[END_REF].

(

) 2
Where TA is the time available to perform the task and TR, the time required by the human operator to perform the task.

Therefore TA only depends on the task demands and TR takes into account the difficulties that the operator has to face with. More the operator is face to a difficult task, more her/his time required TR is long. The implementation can be simple as the analyst is able to detect the instants when the task begins and ends and the instants when the human starts and stops. It is also non obtrusive. Furthermore it gives a total workload.

LAMIH's workload assessment

LAMIH's method distinguishes the task demands on one hand which corresponds to the intrinsic task difficulties and workload on the other hand which results of the human activities for performing the tasks. These activities can be more or less difficult to perform according to the operators own capabilities for instance, skill, expertise, health, etc.

The method follows two steps: the first step deals with the formulation of the task demands, the second one consists in observing how the human takes these demands into account [START_REF] Millot | Supervision des procédés automatisés et ergonomie[END_REF].

For the task demands formulation, the time available is a function of time TA(t). TA(t) has a linear expression (3) Fig. 1. , for discrete tasks, i.e. tasks which has a beginning and a final instants.

- For a continuous task, TA(t) has another definition, for instance in the context of a car driving, the Time Available is given (4) and Fig. 2. [START_REF] Riera | Activity modelling and workload assessment of a car driver[END_REF].

(4) Fig. 2. The car driving context.

These expressions are related to pure temporal demands.

Millot introduced the functional part of the demands trough an artefact. In fact, the effect of an increase of a task difficulty can be seen as a decrease of the available time, i.e. the increase of the temporal demands. For that he introduced the gravity of the situation G(t) representing the functional demand of the task, felt by the operator. The resultant Time Available is called subjective time available TA S calculated by the expression (5).

(5)

For the second step aiming at formulating instantaneous workload Wl(t), Millot used an analogy with the Physic notions of power and energy. When performing a task the operator is supposed to invest a "power" sufficient to satisfy the task demands. Therefore instantaneous workload can be written by expression (6).

(

) 6
With the same analogy, he defines total workload WL the expression [START_REF] Berger | Real time assessment of human operator workload placed in a simulated multitask context of a monitoring and supervising process[END_REF], as the "energy" spent along the task duration TR. WL can be seen as a value of the fatigue due to the task i.e. the resources which have been consumed.

Generally, the gravity G(t) is not a continuous function as we can see Fig. 3. ; Therefore WL cannot be easily calculated with (8). Millot introduced Wl j called sampling workload and showed that the Wl j is a good estimator of Wl(t) if the sampling period T is very short [START_REF] Millot | Supervision des procédés automatisés et ergonomie[END_REF].

(

) 8
Where Wl j is the sampling workload value during the j th sample and T is the sampling period These formulations were tested by experiments in the context of supervision of discrete variables and compared with values of workload assessed by the MCH scale (Modified Cooper-Harper Scale) [START_REF] Millot | An implicit method for dynamic task allocation between man and computer in supervision posts of automated processes[END_REF][START_REF] Debernard | Dynamic task allocation between controller and AI systems in air-traffic control[END_REF]. Different levels of difficulty of this task were used during experiment, low, medium and high. Workload was assessed with the expression (8) for an empirical evolution of G, drawn Fig. 3. The obtained values were compared with a dedicated MCH [START_REF] Millot | Supervision des procédés automatisés et ergonomie[END_REF]. A good correlation between the method and the MCH, and the sensitivity of the LAMIH's method with the task difficulty has been shown.

Fig. 3. Example of a gravity function G(t).

Another issue introduced by the LAMIH's method is the context of the task operating, i.e mono-task or multi-task context. The method supposed that in multitask situations, the operator defines a new available time for each task which is reduced proportionally to the number of tasks ( 9).

(

) 9
Where i is the number of the task, n, the quantity of tasks and TA r i , the time available recalculated by the operator. This expression was tested with the supervision task of about 100 discrete variables and compared with values of workload assessed by the MCH Scale. The gravity of the situation used is drawn Fig. 3. Finally a new experimental campaign mixed continuous and discrete tasks in a car drivers context [START_REF] Berger | Real time assessment of human operator workload placed in a simulated multitask context of a monitoring and supervising process[END_REF][START_REF] Riera | Activity modelling and workload assessment of a car driver[END_REF]. The result of these experiments showed a good sensitivity of the LAMIH's method. This method can be used on-line and is not obtrusive. The workload evolution is sampled with a shorter sampling period than SWAT and is assimilated to instantaneous value of workload due to its short sampling period. Another practical issue is the possibility to take into account, the mono-task or multi-task context which is excluded from the other methods previously presented. However the time load is the only one load generator taken into account by the method. In theory the gravity function G(t) represents the functional demand of the task. But in practice this gravity function is difficult to determine a priori.

Comparison of the workload assessment method

Table 1 compares the four workload assessment methods, according to the different criteria given above. The LAMIH's method is the most relevant method according to the criteria presented in Table 1. and particularly, the unique method which allows a detailed task analysis in a multi task context. Unfortunately, this method takes into account temporal load only. Legend: S, sensitivity; O, obtrusiveness; U, usability.

The next section presents a way to improve the LAMIH's method, in order to develop a workload assessment method which reaches criteria presented in this paper.

PROPOSAL OF A WORKLOAD ASSESSMENT METHOD DEVELOPMENT

Our objective is to develop of a general workload assessment method which can be used on-line and in the multi-task context; and is sensitive, non obtrusive and easily usable (mathematical expressions).

The idea is to take different load generators into account, allowing assessment on-line with a sampling period around 1 second and without disturbing for the operator.

A selection of the generators of load which influence workload must be made. For example in TLX, the own performance of the operator is used as a generator of load. But this term, performance, may be ambiguous. According to the operator, the same performance can be observed but the resources supplied by operators can be different regarding their various expertise levels for instance. Therefore one first proposal is to reuse the same generator of charge as SWAT, i.e. temporal load, functional load related to the task and stress, as they match the workload definition given by Sperandio.

Temporal load can be easily assessed as it was described by the LAMIH's method. Stress is partly due to time load. More the time available decreases, more the stress of the operator increases. If the functional demand of the task is high due to a risk of danger for example, the operator is stressed also. A second proposal is to consider the time load and the functional load only and find an expression estimating the correlation of the two parameters with stress.

Furthermore a study must be performed in order to find an estimation of the functional load by a mathematical expression, as made in the LAMIH's method.

As proposed by Wierville and Casali, a task is defined by different natures of activities, perception, mental, communication and psycho-motor. These activities imply different resource supplies from the operator. For example, pull a glass of water on the grass is less resources demanding than pull the same glass of water on the dehydrated plant. The first task demands only psychomotor activity. The second one demands psychomotor activity, but also a mental activity in order to detect the plant which really needs water. Therefore it is possible to give a degree of the functional load, low, medium or high (by comparison with SWAT), by a detailed task analysis. This way of functional load estimation is moreover linked to the task, which allows a better adaptation of the method to different operators, such as car driver, train driver, for example.

The first step of future study is to find different combinations of activities (perception, mental, communication or psychomotor) for an operator, performing a specific task; in order to find the way to create low, medium or high functional load contexts. Secondly experiment will be performed, by using the previous study. in the task we study the temporal load remains constant and a variation of the functional load level will be simulated. Workload of the subjects will be assessed by three methods, SWAT, TLX and LAMIH's method taking into account initially only temporal load.

A comparison between the value obtained by SWAT and TLX with the proposed method, will allow obtaining discrepancies. This will be performed according to Fig. 4. Some details have to be given concerning the comparison of the values obtained by SWAT and by TLX, due to the different sampling period of these methods. Indeed the nature of the sampling period characterizes the nature of the estimated workload. Therefore the results given by LAMIH's method at a sampling period of 1 second cannot directly be compared neither with SWAT nor with TLX. We must calculate the sampling Wl j with the same period T (for instance 5 minutes) as used in SWAT. Finally according to expression (8), to be compared with TLX, the LAMIH's results must be integrated all over the task duration.

When these particularities will be taken into account, we expect the respective discrepancies between SWAT and TLX and the LAMIH's method will represent the functional load of the task due to the specific activities performed by the operator during the experiment. From the discrepancies collected during experiment a data analysis should allow finding a formulation representing the functional load of a task due to the activities which it implies. Where dt and T are the respective sampling period.

CONCLUSION

Four workload assessment methods were analysed according to different criteria: the generators of charge which are taken into account, a formulation of the sensitivity, the obtrusiveness, the usability and the sampling period of measurement. Among the four workload methods, two allow a short sampling period, SWAT and the LAMIH's method.

By analysing the LAMIH's method, the need to take into account the multi-task context is another criterion of workload assessment method which has been highlighted. The LAMIH's method is the only one which proposes a solution for evaluating in multi-task contexts. This method will be the base of the development of our new workload assessment method.

However the LAMIH's method, contrary to the SWAT and the NASA-TLX methods, evaluates the temporal load only. An improvement of the LAMIH's method is then suggested by including functional load and stress.

Stress is a double manifestation of temporal load and functional load. The interest is taken in the temporal load and functional load only.

The temporal demand is assessed on-line with a sampling period around 1 second and without disturbing for the operator, as shown by the Millot's work. The issue is the determination of the functional load expression which has the same characteristics.

Firstly, a list of different functional load level (low, medium and high) has to be classified as function of the activities performed in a specific task. Task analysis is the background of this study. The operator will be placed in different experimental context with the same temporal load but with increasing functional loads. A comparison will be made between SWAT and TLX and the LAMIH's workload estimation; calculated at the relevant sampling periods. A data analysis following the experiments expects to find a formulation of the functional load. The authors gratefully acknowledge the support of these institutions.

Fig. 1 .

 1 Fig. 1. Time Available variable TA.

Fig. 4 .

 4 Fig. 4. Experimental set-up for functional load assessment.

  work has been supported by CISIT (International Campus on Safety and Intermodality in Transportation), the Nord-Pas-de-Calais Region, the European Community, the Regional Delegation for Research Technology, the Ministry of Higher Education and Research, and the National Center for Scientific Research.

Table 1 . Comparison of four workload assessment methods.

 1

			TLX	SWAT		Sheridan	Millot
			Temporal	Time	load	Temporal	Temporal
			demand,	(temporal		demand	demand
			Mental	pressure),
	Load generators	demand, Physical, demand Own perfor-mance,	Mental (functional demands) Psychological load stress load
			Effort
			Frustration (or
			stress)
	S				
	O					
	U					
	On-line measure	Measure at the end of the task	Sampling period around 5 minute	Measure at the end of the task	Sampling period around 1 second
	Multitask	context				


Halima Bouzidi
email: fhbouzidi@esi.dz

Hamza Ouarnoughi

Smail Niar

Abdessamad Ait

El Cadi

Performance Prediction for Convolutional Neural Networks in Edge Devices

Keywords: CNN, GPU, Performance Modeling, Multiple Linear Regression, Multi-Layer Perceptrons, Support Vector Machine, Random Forest, eXtreme Gradient Boosting

Running Convolutional Neural Network (CNN) based applications on edge devices near the source of data can meet the latency and privacy challenges. However due to their reduced computing resources and their energy constraints, these edge devices can hardly satisfy CNN needs in processing and data storage. For these platforms, choosing the CNN with the best trade-off between accuracy and execution time while respecting Hardware constraints is crucial. In this paper, we present and compare five (5) of the widely used Machine Learning based methods for execution time prediction of CNNs on two (2) edge GPU platforms. For these 5 methods, we also explore the time needed for their training and tuning their corresponding hyperparameters. Finally, we compare times to run the prediction models on different platforms. The utilization of these methods will highly facilitate design space exploration by providing quickly the best CNN on a target edge GPU. Experimental results show that eXtreme Gradient Boosting (XGBoost) provides a less than 14.73% average prediction error even for unexplored and unseen CNN model architectures. Random Forest (RF) depicts comparable accuracy but needs more effort and time to be trained. The other 3 approaches (OLS, MLP and SVR) are less accurate for CNN performances estimation.

I. INTRODUCTION AND MOTIVATIONS

Machine Learning (ML) approaches have recently become very popular for several use cases. Their fields of applications, in health, agriculture, transport, etc., are growing constantly. This increasing interest in ML in general and Convolution Neural Networks (CNN) in particular, could be explained by two main reasons. The first one is the availability of very powerful Hardware (HW) platforms in Internet of Things (IoT), edge, fog, Cloud computing and High performance computing (HPC) platforms. The second reason is the availability and the diversity of very large datasets on the Internet. These datasets allow to improve the training of ML algorithms and therefore enhance their accuracy.

Hence, the use of CNN has made it possible to obtain significant improvements in terms of accuracy and flexibility. However, their development and the choice of the best CNN for a given problem is difficult. Each month, a large number of complex and accurate CNN models are proposed. Consequently, finding the CNN which gives the best trade-off between accuracy and execution time is a tedious task [START_REF] Chen | Deep learning with edge computing: A review[END_REF].

For edge devices with limited computing power and realtime constraints such IoT or autonomous cars, choosing the best CNN implementation is problematic. In [START_REF] Bianco | Benchmark analysis of representative deep neural network architectures[END_REF], authors have shown that CNN is not necessarily correlated with the number of FLoating-point OPerations (FLOPs) or the size of the CNN.

Having a design tool for rapid CNN performance estimation becomes crucial to reduce design time and to obtain a highperformance system. Early execution time estimation allows to quickly determine the best CNN implementation for a given application requirements and a given hardware (HW) constraints. Depending on the CNN to execute, the user may need different HW to minimize inference time. Such a tool can be used either to choose the best CNN for a given HW platform and/or to explore different HW platforms for a specific CNN. In this paper, we focus on proposing a methodology to help the designer in choosing the most efficient CNN for a given HW platform. The CNN execution time estimator must provide a high accuracy, a low design time and a high level of flexibility to be adapted to different CNNs and HW platforms. In addition, the number of estimator hyperparameters must be reduced and easy to tune.

In this work, we compare some of the most successful stateof-the-art prediction algorithms for estimating the execution time of the CNN inference phase. As these algorithms have a set of (hyper)parameters, in the rest of the paper, they are called Models. The following ML-based models have been considered: Multiple Linear Regression using the Ordinary Least Squares (OLS), Multi-Layer Perceptrons (MLP), Support Vector Regression (SVR), Random Forest (RF), and eXtreme Gradient Boosting (XGBoost). Our work is not intended to propose new ML-based models but to compare the 5 models in order to analyze their strengths and weaknesses. For each of the studied models, we analyze prediction accuracy, time to tune and train the model and finally, time to run the prediction models on different Hardware platforms.

The remainder of this paper is organized as follows. Section II gives a literature review of CNN performance estimation and design space exploration. In section III, we first analyze and discuss the CNN features that have to be considered in the execution time prediction, then a survey of the used ML-based approaches is presented. Section IV is devoted to experimental results. Finally, conclusion and future work will be given in section V.

II. RELATED WORKS

Motivated by speed and security purposes, there was recently a trend towards migrating ML applications from cloud and central computing to edge computing. To find a good matching between CNN requirements in terms of speed and energy consumption either in the training or in the inference phases, several projects have been conducted recently targeting edge platforms.

In [START_REF] Amarís | A comparison of gpu execution time prediction using machine learning and analytical modeling[END_REF], the authors compare: linear regression, support vector machines and random forests with a Bulk synchronous parallel (BSP) based analytical model. Machine learning approaches have been used to provide reasonable predictions without detailed knowledge of application code or hardware characteristics. The authors use profiling information from 9 benchmarks on 9 different GPUs. In opposite to our work, their method is not dedicated to ML applications. In addition, in our work we use a larger set of benchmarks and we obtain a better accuracy.

In [START_REF] Justus | Predicting the computational cost of deep learning models[END_REF], the authors propose a methodology to estimate training time for CNNs. In their approach, training time is seen as the product of the training time per epoch and the number of epochs which need to be performed to reach the desired level of accuracy. The deep learning network is decomposed in several parts and the execution time estimation operates on these parts. Timings for these individual parts are then combined to provide a prediction for the whole CNN execution time.

In [START_REF] Wang | Characterizing deep learning training workloads on alibaba-pai[END_REF], the authors propose an analytical framework to characterize deep learning training workloads in large AI clouds and clusters. Using different training architectures, the authors try to identify performance bottleneck for various DNN workloads. For the training phase, weight and gradient communication consumes almost 62% of the total execution time on average. Their simple analytical performance model is based on the key workload features and allows only to identify architectural bottlenecks.

PALEO [START_REF] Qi | Paleo: A performance model for deep neural networks[END_REF] is another framework for estimating training execution time. In PALEO the number of floating point operations required for an epoch is multiplied by a scaling factor to obtain execution time for the training phase. However, PALEO does not take into account numerous other operations which do not scale linearly with the number of floating point operations and which has a big impact on execution time. In the literature we may also find several tools dedicated to power and memory optimization for edge HW platforms.

In [START_REF] Rodrigues | Fine-grained energy profiling for deep convolutional neural networks on the jetson TX1, year=2017[END_REF], the authors proposed a multi-variable linear regression model to predict energy consumption of CNNs based on the number of SIMD instructions and main memory accesses. They used an Nvidia Jetson TX1 GPU and they obtained an average relative error of about 20%. Our profiling and analysis phases, detailed in Section III, have shown that these 2 parameters must be enhanced by others to obtain a more accurate estimation.

The idea behind tools in [START_REF] Cai | Neuralpower: Predict and deploy energy-efficient convolutional neural networks[END_REF], [START_REF] Stamoulis | Hyperpower: Power-and memory-constrained hyper-parameter optimization for neural networks[END_REF] is to explore the hyperparameter space for NNs running on a given hardware platform and to estimate power and memory usages. In most of the existing projects, a learning-based polynomial regression approach is used. In our work, we do not treat power consumption estimation. Only tools for execution time estimation are presented and compared. However, as we will see in the following sections, the same approaches can be developed for power and energy consumption.

III. PROPOSED APPROACH

Figure 1 gives an overview of our proposed modeling methodology. The approach includes three main steps. The benchmarking step aims to define the benchmarks characteristics and determine the most impacting CNN features. To make the model simple, only these features will be taken into account in the models. In the data collection step, we extract the data used in the elaboration of the CNN execution time prediction. Finally, the modeling step details the used MLmodels, their training process with hyperparameters tuning and their evaluation. As explained in the following section, each of the considered models includes a set of hyperparameters and a set of parameters. The hyperparameters are different from one model to another. They draw the model architecture. The parameters produce one instance of the obtained model and explain the manner by which the model is implemented for the prediction. As example for MLP, the hyperparameters correspond to the number of neurons, the number of hidden layers, the activation function, etc. The parameters for MLP correspond to the weights and the biases values used in the model. More details will be given in sub-Section III-C.

A. Problem Formulation

Predicting CNNs inference time on GPU platforms can be formulated as follows: As inputs, we have a CNN (noted CN N i ) characterized by a set of n features f 1 , f 2 , . . . , f n , and a GPU platform (noted GP U j ) represented by a set of m Hardware characteristics c 1 , c 2 , . . . , c m . The number of convolutional and fully-connected layers, the image size, and the number of neurons are examples of CNN features. Platform parameters may correspond to the number of cores in the GPU, the memory size, the clock frequency, etc.

An inference time prediction function (noted T ) is a mapping function from CN N i and GP U j to R +

T : CN N i , GP U j -→ ŷ ŷ = T (f 1 , f 2 , . . . , f n , c 1 , c 2 , . . . , c m ) ( 1
)
where ŷ is the predicted inference time of CN N i on the GP U j .

In this paper, due to the lack of space, we only present the case of two different edge GPUs, namely Nvidia Jetson AGX Xavier [START_REF]Jetson AGX xavier developer kit[END_REF] and Nvidia Jetson TX2 [START_REF]Harness AI at the edge with the jetson TX2 developer kit[END_REF].

The same modeling methodology has been used on each edge GPU platform and two sets of models have been obtained. The equation 1 becomes:

ŷ = T GP Uj (f 1 , f 2 , . . . , f n ) (2)
where GP U j ∈ {AGX, T X2}. In this context, our approach aims to answer the following questions:

1) What are the most important CNN features that impact inference time? Our answer is given in sub-section III-B. 2) What are the most relevant modeling algorithms that give the best accurate predictions of CNNs inference time? Our answer is given in sub-section III-C. Our modeling approach is based on finding the relationship between the input CNN features and observed inference time. For this purpose, we rely on regression models which are part of supervised ML algorithms.

B. CNN Features for our Prediction Models

CNNs inference time is mainly impacted by the following factors:

1) Computational complexity, which corresponds to GPU cores activities; 2) Memory requirements, which corresponds to read and write memory operations; 3) CNN internal properties, which corresponds to dependencies between computation operations and memory operations. These features are detailed below.

1) Computational complexity:

The total number of FLOPs is used to measure the computational complexity when performing CNNs inference. As reported in table I, this number is highly dominated by the number of operations performed in convolutional layers. Convolutional layers represent the bottlenecks of computations in CNNs. Our experimentation shown in figure 2 confirms that CNNs inference time is not linearly correlated to the number of FLOPs. We can then conclude that considering only the computational complexity is not enough to predict accurately the CNN inference time.

2) Memory requirements: Memory activities have a significant impact on the execution time on GPUs. However, it's hard to extract the information about memory activities and requirements without profiling the CNNs on the GPU platform. This solution must be avoided during the prediction as it's a time consuming task and will also add a significant overhead to the prediction latency. To overcome this problem, we rely on some specific characteristics of CNNs that are correlated to memory activities. Our experiments show that the major memory requirements of CNNs can be devoted mainly to 3 factors:

1) Reading CNNs parameters, i.e weights and biases, 2) Reading the input data, writing the output results and 3) Reading and writing the intermediate data of the hidden layers, i.e activations. During the CNN inference, convolutional filters and activations are constantly accessed which increases the inference time [START_REF] Siu | Memory requirements for convolutional neural network hardware accelerators[END_REF]. Moreover, this time increases when activations and weights can not be mapped entirely in cache memories. Cache misses considerably increase the CNN inference time. Given the above facts, we assume that both weights and activations are highly impacting memory activities when performing CNNs inference.

3) CNN internal properties:

In addition to computational complexity and memory requirements, other properties related to CNNs internal architecture impacting inference time have to be taken into account. We have also considered the Weighted sum of neurons in convolutional layers. This metric is calculated by multiplying the number of neurons in convolutional layers by the filter size: height×width×depth. The idea here is to give more importance to neurons with large filter sizes. The number of neurons in fully connected layers is taken as it is, because the neuron is associated to a single weight.

To select the most important CNN's features in the execution time prediction, we used the F-score metric of the XGBoost algorithm. This technique is used for all of the prediction models detailed in section III-C. The considered features and their corresponding impacts are depicted in figure 3.

C. Modeling Approaches

Five ML-based algorithms have been used to design the CNN inference time prediction models:

1) Multiple Linear Regression using the Ordinary Least Squares (OLS) 2) Multi-Layer Perceptrons (MLP) 3) Support Vector Regression (SVR) 4) Random Forest (RF) 5) eXtreme Gradient Boosting (XGBoost)

In order to tune each model's hyperparameters, we applied an exhaustive grid search. During this process, we perform a K-fold cross validation [START_REF] Rodriguez | Sensitivity analysis of kfold cross validation in prediction error estimation[END_REF] to select the best hyperparameters values and combination. Once the hyperparameters are tuned, the training of the chosen ML algorithms is performed on the parameters using the training dataset. Finally, the obtained prediction models are evaluated using the test dataset. We give an overview of the ML algorithms used in our approach in the following.

1) Multiple Linear Regression using the Ordinary Least Squares (OLS):

Multiple linear regression approaches use a linear function to model the relationship between a target variable (y) and n input features (X 1 , X 2 , . . . , X n ):

y = α n X n + α n-1 X n-1 ... + α 2 X 2 + α 1 X 1 + β + ǫ (3)
In this equation α1, α2, . . . , α n and β are the parameters of the MLR. They are fixed during the training phase in such a way to minimize the difference (ǫ) between estimated (ŷ) and real (y) values of the inference execution times.

Among the variety of Multiple Linear Regression algorithms, we have chosen the most frequently used Ordinary Least Squares regression (OLS) [START_REF] Matloff | Statistical regression and classification: from linear models to machine learning[END_REF]. In our study, we have applied a stepwise regression to choose the best combination of features with the highest impact on CNNs inference time. The results of the stepwise regression are given in table II. The features are added based on their importance showed in the figure 3.

We notice that the model has achieved a highest adjusted R 2 (98.8%) with only four input features, namely : FLOPs, sum of activations, weighted sum of neurons, and number of convolutional layers parameters. The reasons behind choosing Multiple Linear Regression for CNNs inference time prediction are: 1) their simplicity in the implementation, and 2) their short training time. However, this approach is not accurate when the linear relationship between input features and the target variable, execution time in our case, is not valid. To overcome this limitation, other algorithms which have the ability to model both linear and non-linear relationships have been implemented and compared. These algorithms are presented in the following sub-sections.

2) Multi-Layer Perceptrons: The Multi-Layer Perceptrons (MLP) [START_REF] Murtagh | Multilayer perceptrons for classification and regression[END_REF] is a part of Artificial Neural Networks (ANN). The MLP is a succession of hidden layers where each layer applies the following transformation on the input features:

ẑ = θ( n i=1 w i x i + b) (4)
where ẑ is the output value, w i is the weight of the feature x i , b is the bias and θ is the activation function. Choosing the good MLP architecture is done by fixing the hyperparameters, such as the number of layers, number of neurons, types of activation function, etc. We rely on the grid search method with the K-fold cross validation to choose the best hyperparameters values. Once these hyperparameters are fixed, parameters values of w i and b are fixed at the training phase.

3) Support Vector Regression: Support Vector Machine [START_REF] Hearst | Support vector machines[END_REF] is one of the most powerful data driven algorithms used for classification and regression problems. The SVR deals with non-linearity by using kernels which are functions that map the input data from the original space to a higher dimensional space where features can be linearly separable. The most important hyperparameters for the SVR are:

• Kernel type: linear, polynomial, sigmoid, and radial basis function (RBF).

• Gamma: the kernel coefficient for the polynomial, sigmoid and RBF kernels.

• Cost (C): regularization factor which controls the tradeoff between the training error and the generalization of the model.

• Epsilon, which defines the interval of errors within which no penalty is applied to training loss function. These hyperparameters have been tuned using the grid search method. 4) Random Forest: Unlike the above methods based on the best line or hyperplane fitting, Random Forest (RF) uses decision trees. RF is based on the bagging technique where predictions are made by multiple predictors (i.e. decision trees). Each of them is trained on a subset of training samples and a subset of features selected randomly. The final prediction is calculated by averaging the predictors outputs. We tune the following hyperparameters:

• Number of predictors needed to train the model, • Maximum depth of each decision tree,

• Minimum number of samples required to split nodes, • Minimum number of samples required for a leaf node,

• Maximum number of features to consider when splitting the nodes. We have also used the bootstrap method for data sampling.

5) eXtreme Gradient

Boosting: eXtreme Gradient Boosting (XGBoost) [START_REF] Chen | Xgboost: A scalable tree boosting system[END_REF] is a decision-tree based algorithm. Unlike Random Forest, XGBoost is based on the boosting technique.

The boosting technique makes predictions from weak predictors that are arranged sequentially: the first predictor is trained on the entire dataset, where the subsequent predictors are trained on the residuals of the prior predictors. This technique helps to focus on the mispredicted values. The algorithm uses the gradient descent algorithm to minimize prediction errors. As explained in Section III-B, we use first the feature importance calculated by XGBoost to select the most impacting features (see figure 3). These features are then used by the performance estimator. XGBoost has several hyperparameters to tune, which can be categorized into three groups:

• General hyperparameters, • Booster hyperparameters, • Learning hyperparameters. Once the optimal combination of hyperparameters is obtained, we perform a full training on the entire training dataset, with the early stopping technique.

IV. EXPERIMENTAL RESULTS

This section details the followed evaluation methodology and the obtained evaluation results. We used two Nvidia GPU platforms, namely the Jetson TX2 and the Jetson AGX Xavier. The hardware specifications of each platform is described in the table III. We have configured the platforms to profiling mode (maximum power mode) in order to minimize the host CPU interference. We have used the same underlying software

A. Evaluation Methodology

As shown in figure 1, our modeling process is subdivided into three main steps: 1) Benchmarking, 2) Data collection and 3) Modeling. such as: Inception, ResNet, MobileNet, etc., have been tested on the two GPU platforms. An FP-32 bits representation for the weights and tensors has been used. The weights of these CNNs have been set randomly as their values have no impact on the inference time. We varied three factors:

1) Input Image Sizes: Here the impact of the image size on the inference time is studied. We tested the frequently used image sizes in the literature from 32*32 to 2400*2400 pixels depending on the CNN. 2) CNN Variants: Different variants of the same CNN architecture are considered in order to extend our dataset. Some of the considered CNN, such as ResNet V1, have up to 12 different variants. 3) CNN Architectures: Finally, we consider different architectures of CNN to quantify their impact on the inference time.

In total, we obtained a dataset of 2056 and 1975 (respectively) on AGX and TX2 (respectively) as inputs for our performance estimation models. This difference is due to the fact that the TX2 GPU platform has a smaller memory and some of the benchmarks couldn't be executed. In the experiments, 70% of the input-data has been used for training and 30% for tests and accuracy measurements. In the experimental results section, we will evaluate the accuracy of each of the 5 models, when exploration is realized in 3 groups.

2) Data collection step:

The dataset used for CNNs inference time modeling is collected from two sources:

• CNNs implementation descriptions: we have developed a parser module that takes CNNs implementation as input and gives their important feature values (see figure 3).

• Performance measurements: CNNs inference times measurements have been collected using the Nvidia profiling tool Nvprof [START_REF]Nvprof overview[END_REF]. Each inference has been ran 100 times on 100 randomly chosen images in order to minimize the profiling overhead.

At this point the dataset is collected and ready to use in the modeling step.

3) Modeling step: Figure 4 details the modeling steps followed to obtain the 5 prediction models. As shown in the figure 4, our modeling process has two inputs:

1) ML prediction model name, which corresponds to one of the five ML algorithms described in section III-C. 2) Collected data, which is the dataset obtained in the data collection step as described in Table IV. The collected data is split into training data and test data. During the training phase, the search space for the hyperparameters is firstly initialized for each prediction model before being tuned using the training data. The tuning is realized using grid search and K-fold cross validation techniques in order to select the best values of hyperparameters. Finally, the prediction models are trained with the obtained optimal hyperparameters.

To evaluate the obtained models on different configurations we have also split the test data into three exploration spaces: considered. Based on an original CNN architecture, new CNN variants have been obtained by changing the features such as the number of layers and the used operators. If we consider ResNet V1 as example, we trained our predictors on ResNetV1-20 to ResnetV1-100 and we predict inference time for ResNetV1-200. This leads to new numbers of: FLOPs, activations, neurons, parameters and layers.

• Performance estimation of New CNN Architectures (NCA): In this group of experiments, we evaluate our predictors on new hand crafted synthetic CNN models. We randomly generate synthetic CNN architectures where the input size, number of layers, type of layers, number of filters, type and parameters of convolutions, parameters of fully-connected layers, and type of pooling and batch normalization layers, have been randomly set. This exploration space is the most difficult as we evaluate the capacity of the 5 models to predict the performances of completely new CNN architectures not included in the training dataset. The aim of this 3 rd group of experiments is to measure the capacity of the 5 models to estimate execution time of completely new and unseen CNN architectures only by characterizing them through their features. Inference execution time accuracy is measured using the Mean Absolute Percentage Error (MAPE).

B. Results and Discussion

After detailing the evaluation methodology, this section presents and discusses the obtained modeling results in three perspectives: First, we discuss the predicted CNN inference times using the 5 ML prediction models compared to the measured ones. Second, the Mean Absolute Percentage Error (MAPE) is presented to evaluate the accuracy of each ML model. We also discuss the hyperparameters configuration. In the third part of this section, we compare execution times for training and for tuning, the number of tested hyperparameters' configurations and finally the prediction latency which corresponds to the execution time of the 5 prediction approaches on 3 different Hardware platforms: on the Jeston AGX GPU, on the Jetson TX2 GPU and on a development station Intel Xeon. Implementing our predictors on edge GPU platforms will allow to adapt the executed CNN to application constraints at run time.

1) Predicted vs Measured CNN inference time: Figures 6, 7, 8, 9, and 10 present the comparison between the predicted (y axis) and the measured (x axis) CNN inference time on the AGX platform. The comparison has been realized for NIS, NCV (on the left of the figures) and NCA (on the right of the figures). The used dataset has been detailed in section IV-A and Table IV.

For NIS and NCV, the predicted execution times are very close to the measured values, in all of the prediction models except for OLS (see figure 6). As we have trained our prediction models on different CNN variants and different input image sizes, the prediction models have been able to capture the correlation between different CNN variants and input sizes. We can also notice that OLS overestimates the CNN inference time especially for high execution time values. This is due to the non-linearity between input features and the CNN inference time where the computation complexity and memory requirements are very high. This result confirms also that the inference times could not be accurately estimated by using a linear regression .

For NCA, the difference between the predicted and the measured execution times are higher than for NIS and NCV, in particular for MLP, OLS and SVR prediction models. As explained in the previous section, the reason behind this drawback is the fact that in NCA, new and randomly generated CNNs are explored. For these reasons, the predictions models are less accurate for these unexplored and unseen CNN architectures.

In the appendix of the paper, figures 11, 12, 13, 14, and 15 show the results for the NVIDIA Jetson TX2 platform. We observe the same conclusions as for the NVIDIA AGX.

2) Prediction accuracy and hyperparameters configuration analysis : To evaluate the accuracy of our prediction models, the Mean Absolute Percentage Error (MAPE) has been chosen. Figure 5 and Table V give the obtained MAPE for the 5 studied prediction models for NIS, NCV and NCA. The prediction models use data obtained by profiling the benchmarks (Table IV) on two edge GPU platforms (see section IV-A1).

From figure 5 and Table V, we can note that the MAPE average values are mostly between ∼ 7% in the best case and ∼ 26% in the worst case. We can also notice that the lowest MAPE values have been obtained for NIS and NCV.

In general we can say that for NIS, NCV and NCA, XGBoost outperforms the other prediction models and offers the lowest MAPE values. XGBoost is among the most powerful ML algorithms. The Boosting technique, to enhance the prediction accuracy through many estimators arranged sequentially, makes it very accurate. However, we noticed that XGBoost is sensitive to hyperparameters values. These values need to be set carefully in order to achieve the best performances.

RF is composed of different decision trees trained on random subsets of training data samples and features. This property helps to reduce the variance error. Nevertheless, prediction approaches based on ensemble learning, such as RF, need to be trained on different scales of data in order to obtain an accurate mapping of features and execution times. In addition, by increasing the number or the depth of the decision trees, the prediction accuracy converges according to our the experiments. These 2 factors increase the complexity of the training and the latency of the prediction in RF. For this reason, when a short interval of time for tuning and training is desired, XGBoost will be more efficient than RF.

MLP has generally good performances with a slight loss of generalization for NIS and NCA. This can be due to its nature that tends to overfit data. Furthermore, MLP is very sensitive to the variation of hyperparameters, which makes it very hard to tune. The size of the MLP network has an important impact on the accuracy. We noticed that large MLP networks are prone to overfitting compared to the small ones. SVR is less accurate for NIS and NCA compared to the aforementioned models. The variation in the MAPE values in the three exploration spaces is quite high which can be interpreted as overfitting. In terms of hyperparameters configuration, SVR is sensitive to the type of kernels and the cost (C). According to our results, linear kernels perform the best, whereas the very small values of C, lower than 1, lead to a huge loss of generalization.

As expected, OLS has the highest MAPE values because of its limited capacity to capture the different patterns in the training data. Moreover, the inclusion of non relevant features in OLS can drastically decrease the model's accuracy. In this case, techniques for regularization such as LASSO and Ridge are recommended to improve the model performances. Due of time limitation, this point is considered as a possible extension.

If we compare the results of the two GPU platforms, Nvidia AGX and TX2, we notice that the MAPE is very similar. This means that our modeling approach is easily adaptable to other platforms in order to obtain execution time prediction. V summarizes the obtained statistics of the 5 tested modeling approaches.

3) Comparison of the 5 models in terms of training time, tuning cost and latency: Table

In addition to MAPE average values, table V gives the training and tuning time of each prediction model for AGX and TX2 in Google Colaboratory [START_REF] Bisong | Google colaboratory[END_REF]. We can observe, that except for SVR and XGBoost, the prediction models require more time to be trained for AGX than for TX2. This is due to the fact that the higher resource capacity of AGX allows to run more CNNs which gives a largest training data compared to TX2. Table V shows also that tuning the model hyperparameters is time consuming task, especially for models with high number of hyperparameters.

Tested hyperparameters values in grid search for the AGX and for the TX2 are different. This is due to the fact that CNN execution times ranges are different. For the MLP and XGBoost, as both of them use the gradient descent algorithm, the training convergence for AGX and for TX2 is different.

To quantify the execution time of the 5 models, they have been executed on an AGX, TX2 and Intel Xeon based Processor development station. Results are given in the three rightmost columns in Table V. The comparison of the execu-tion latency is important to measure the performance of the 5 models to explore a large number of CNN architectures in a limited interval of time. In addition, when an online tuning of the CNN or the Hardware platform is needed, having a short execution time of the prediction model is interesting. We note that in terms of speed, OLS outperforms all of the other models. This is explained by the simplicity of OLS which is a simple linear equation. However OLS is the less accurate. XGBoost has a smaller execution time compared to MLP, SVR and RF. Which makes this technique very interesting due to the good trade-off in terms of accuracy and latency. The highest latency of RF is due to the complexity of exploring the decision trees included in the Random Forest.

V. CONCLUSION

Edge computing is one of the area targeted by CNNbased applications. Both CNNs and edge computing are highly growing and frequently changing domains. Finding the best matching between CNN model architecture and Hardware of the edge device under real time constraints is a very time consuming and complex task. In this paper, we compared five state of the art ML-based models for CNN inference time prediction on edge GPUs. These 5 models are useful for an early performance estimation of CNN-based applications. Using these models, rapid design space exploration will be possible to guide the designer to an efficient CNN model and/or to the most adequate Hardware platform.

We demonstrated that XGBoost and RF gave good execution time predictions, with an average accuracy close to 92% for NIS and NCV and 86% for NCA. In terms of trade-off between: accuracy, model tuning complexity and prediction latency, XGBoost is the best approach. Exploring new synthetic CNNs, i.e called NCA in our paper, is more complex and less accurate than the other two spaces: NIS and NCV.

As future work, we plan to extend our comparison to explore additional CNNs' characteristics that may improve our prediction models. We also plan to extend our ML-based models to predict not only CNNs inference execution time but also energy consumption and resource utilization. Finally, we will also integrate the hardware configuration as input in our models to perform cross platform predictions.

Fig. 1 :

 1 Fig. 1: Modeling methodology for CNNs execution time prediction models.

Fig. 2 :

 2 Fig.2: Execution time versus floating-point operations (FLOPs). This figure demonstrates that using only the numbers of FLOPS is not sufficient to estimate execution time. This figure gives the execution time of some of our CNN benchmarks detailed in Section IV ranked by their FLOPs.

Fig. 3 :

 3 Fig.3: The feature importance calculated by XGBoost. Features with a high F-score are the most impacting on the CNNs inference time.

Fig. 4 :

 4 Fig. 4: The process of constructing the prediction models.

Fig. 5 :Fig. 6 :Fig. 7 :Fig. 8 :Fig. 9 :

 56789 Fig. 5: Mean Absolute Percentage Error (MAPE) for the Nvidia AGX (a) and TX2 (b) GPU platforms with the corresponding 95% Confidence Interval. In this figure we compare the 5 prediction models for exploring: New Image Sizes (NIS), New CNN Variants (NCV) and New CNN Architectures (NCA).

Fig. 10 :

 10 Fig. 10: Predicted vs. Measured Inference Time for XGBoost (AGX).

TABLE I :

 I FLOPS ESTIMATED BY TENSORFLOW PROFILER OF SOME STATE OF THE ART CNNS. (B: BILLIONS, AND M: MILLIONS).

	CNN Model	Conv2D	Add	Mul	Pooling
	ResNet-50	7.71B	31.02M 25.58M	1.81M
	DenseNet-121	5.67B	7.89M	8.02M	1.98M
	DPN-98	23.34B	70.54M 61.63M	2.71M
	GoogleNet	3.00B	6.61M	6.64M	12.55M
	ResNet-101V2	14.38B	52.32M 44.59M	2.16M
	Inception-v3	5.67B	23.80M 23.85M 12.18M

TABLE II :

 II EVOLUTION OF THE ADJUSTED R 2 DURING THE STEPWISE LINEAR REGRESSION. FEATURES ARE ADDED TO THE OLS PREDICTION MODEL ONE BY ONE, FROM LEFT (MOST IMPORTANT FEATURE) TO RIGHT (LESS IMPORTANT FEATURE) OF THE TABLE

	Features	Total	Sum of	Weighted	Number	Total	Input im-	Number	Number	Number	Number	Number
		number	activa-	sum of	of con-	number	age size	of fully	of batch	of batch	of convo-	of fully
		of	tions	neurons	volution	of layers		con-	normal-	normal-	lutional	con-
		FLOPs			parame-			nected	ization	ization	layers	nected
					ters			parame-	parame-	layers		layers
								ters	ters
	Adjusted R 2	0.970	0.985	0.987	0.988	0.988	0.988	0.988	0.988	0.988	0.988	0.988

TABLE III :

 III HARDWARE PLATFORMS USED IN THE EXPERIMENTS

	Hardware feature	Jetson TX2	Jetson AGX Xavier
	CPU(ARM)	6-core Denver and A57 2 GHz	8-core Carmel 2.26 GHz
	Memory	8 GB 128-bit LPDDR4	16 GB 256-bit LPDDR4x
	Memory bandwidth	58.4 GB/s	137 GB/s
	GPU	256-core Pascal 1.3 GHz	512-core Volta 1.37 GHz + 64 tensor cores
	Power	7.5W/15W	10W/15W/30W
	configuration in the two edge GPU platforms. CNNs have
	been implemented on GPU using the Keras 2.3.1 API with
	TensorFlow 1.14 as backend [18]. This framework is running
	on top of Cuda version of 10.0 and a cuDNN version of 7.5.3.
	The host operating system in both platforms is Linux Ubuntu
	18.04.3 LTS with a kernel 4.9.149-tegra.

1 )

 1 Benchmarking step: The benchmarking step is mainly based on profiling CNNs inference on different GPUs. The experiments have been designed based on executing state-ofthe-art image classification CNNs by varying the parameters presented in table IV. As shown in table IV, we have implemented the state-of-theart CNNs dedicated to image classification. CNN architectures

TABLE IV :

 IV DETAILS OF THE BENCHMARKS USED IN THE EXPERIMENTS. IN TOTAL, WE HAVE 2056 AND 1975 (RESPECTIVELY) INPUT-DATA FOR AGX AND TX2 (RESPECTIVELY) FOR 5 PERFORMANCE ESTIMATION MODELS: 70% HAVE BEEN USED FOR TRAINING AND 30% FOR TESTS AND

		ACCURACY CALCULATIONS
	CNN Architectures	# CNN variants	Input Image Sizes (squared)
	GoogleNet	1	[224,240,256,299,320,331,448,480, 512,568,600,720,800,896,1024]
	Inception V3	1	[75,90,112,128,150,224,240,256,
	InceptionResNet V2	1	299,320,331,448,480,512,568,600, 720,800,896,1024]
	DPN DenseNet	4 5	[32,56,64,75,90,112,128,150,224, 240,256,299,320,331,448,480,512,
	Xception	1	568,600,720,800,896,1024]
	EfficientNet	4	[32,56,64,75,90,112,128,150,224,
	MNASNet	5	240,256,299,320,331,448,480,512, 568,600,720,800,896,1024,1200,1600]
	ResNet V1	12
	ResNet V2	7
	MobileNet V1 MobileNet V2 MobileNet V3 ResNext SENet	4 5 4 2 6	[32,56,64,75,90,112,128,150,224, 240,256,299,320,331,448,480,512, 568,600,720,800,896,1024, 1200,1600,1792,2400]
	ShuffleNet V1	4
	ShuffleNet V2	3

TABLE V :

 V PREDICTION MODELS ANALYSIS: ACCURACY, TRAINING, TUNING, AND LATENCY

			MAPE	Training time	Grid Search execution time	Total number of hyperparameters		Prediction latency
	Prediction Model	Test Data	AGX	TX2	AGX	TX2	AGX	TX2	AGX	TX2	AGX	TX2	Intel Xeon
		NIS	24.43%	26.32%
	OLS	NCV	20.87%	22.95%	8.14 us	7.3 us	0		0		319.6 ns	468.42 ns	113.93 ns
		NCA	19.30%	20.32%
		NIS	13.04%	15.78%
	MLP	NCV	9.23%	9.65%	1.53 s	722 ms	10.35 hr	11.82 hr	14400	15840	22.65 us	30.90 us	9.95 us
		NCA	13.34%	16.31%
		NIS	13.30%	13.84%
	SVR	NCV	7.67%	8.37%	127 ms	191 ms	23.61 hr	21.43 hr	18144	18144	41.79 us	52.14 us	20.76 us
		NCA	15.00%	16.72%
		NIS	11.19%	11.07%
	RF	NCV	10.55%	8.95%	4.93 s	2.01 s	4.69 hr	4.22 hr	71280	71280	1.03 ms	1.45 ms	393.3 us
		NCA	13.11%	15.22%
		NIS	8.75%	10.95%
	XGBoost	NCV	8.12%	9.11%	163 ms	914 ms	11.27 mn	15.17 mn	237	238	2.03 us	3.49 us	175.20 ns
		NCA	12.29%	14.73%

• Performance estimation of New Image Sizes (NIS):In this first group of experiments, we evaluate our 5 models on state-of-the-art CNN models with new input sizes. We varied the input size from 32*32 to 2048*2048 pixels. We obtained different values of FLOPs, sum of intermediate activations, sum of neurons and eventually different number of parameters. The number and the type of layers remain the same.• Performance estimation of New CNN Variants (NCV):In this second group of experiments, we measure performance estimation when new variants of the same CNN is