Mourad Dridi

Yasmina Abdeddaim

Chiara Daini

Work In Progress: A New Task Model for Real-Time DNNs over GPU

Recently, deep neural networks (DNNs) have been utilized in real-time systems such as autonomous vehicles, where meeting temporal constraints is essential. However, executing such systems on CPU-GPU architectures can make scheduling analysis challenging due to the added delays caused by computing and memory contention. In addition, classic task models are not directly able to model accurately such systems. In this article, we propose a new task model called DNN Task Model (DTM). This model considers both DNN properties and GPU architecture at the same time. It allows us to distinguish between CPU and GPU tasks, provides information about the DNN application and give more accurate execution time analysis through consideration of data quality. We compute DTM from a source CUDA file and a set of real-time specifications of the system. The proposed model is extensible enough to be adopted to various DNN type applications allowing designer to compare candidate software and GPU architectures. Furthermore, we propose a graph optimization inspired by Tensor-RT.

I. INTRODUCTION

As the use of Deep Neural Networks (DNNs) continue to raise in the field of real-time systems, they are becoming a popular choice for applications like autonomous driving. For the inference stage, DNNs are often executed on CPU-GPU architectures. This allows a balance of computational power and energy and a faster processing of the data [START_REF] Xiang | Pipelined Data-Parallel CPU/GPU Scheduling for Multi-DNN Real-Time Inference[END_REF] enabling the efficient execution of the DNNs. GPU systems provide high performance by offering greater computational power than most common CPUs. However, the use of GPUs also introduces contention, which can lead to additional variability in task execution time [START_REF] Liu | Understanding Time Variations of DNN Inference in Autonomous Driving[END_REF].

A. Problem Statement

Alongside the need for computational efficiency, real-time DNNs are commonly characterized by real-time requirements, which need to be satisfied to guarantee the safe and correct behavior of the system. However, current DNN applications deployed over CPU-GPU systems have poor performance in terms of execution time variation. [START_REF] Liu | Understanding Time Variations of DNN Inference in Autonomous Driving[END_REF] found a non-negligible variations in execution time for DNN inference, which significantly challenges the worst case execution time estimation and scheduling problem. For example, in autonomous driving applications, the majority of DNN models show variations larger than 100ms, which affects autonomous driving safety [START_REF] Liu | Understanding Time Variations of DNN Inference in Autonomous Driving[END_REF]. There are several sources of uncertainty that contribute to the execution time variation issue in real-time DNN applications, including the input quality such as image resolution, communication latency and contention of concurrent tasks for resources such as memory, CPU, and GPU [START_REF] Liu | Understanding Time Variations of DNN Inference in Autonomous Driving[END_REF]. In addition, classic task models are not immediately suitable with CPU-GPU architectures and real-time DNNs. They are not able to model simultaneously and accurately both CPU tasks and GPU tasks, streams and communication delays.

B. Contributions

In this article, we first propose a new task model that allows designers to perform an accurate schedulability analysis in order to assess the predictability of real-time DNN applications deployed over CPU-GPU architectures. We call the proposed model DNN Task Model (DTM). The DTM model describes both the details of the GPU architecture and the DNN application. It allows us to distinguish between CPU tasks and GPU tasks. The execution time for GPU tasks takes into account the targeted data quality and its impact on the execution time analysis. Additionally, the model specifies the type of operation for each GPU task, affording deeper understanding of the DNN network and enabling optimization of the task graph. From the source CUDA file and the real-time requirements of the system, the DTM task model is computed.

The second contribution of this article is to propose a graph optimization of the generated DTM. The optimization is inspired by Tensor-RT, which is a tool for optimizing DNN inference [START_REF] Jeong | TensorRT-Based Framework and Optimization Methodology for Deep Learning Inference on Jetson Boards[END_REF]. Graph optimization of Tensor-RT may not always align with real-time constraints. Our approach takes this into consideration and prioritizes the verification of realtime constraints compliance for every possible optimization.

The remainder of the paper is organized as follows. The next section introduces background elements about CPU-GPU architecture and Tensor-RT. Section III proposes our task model. Then, Section IV presents the graph optimization of DTM. An illustrative example is given in Section V. Section VI presents related works and Section VII concludes the article.

II. BACKGROUND

A. GPU Execution Model and CUDA file

The considered GPU execution model for a given GPU kernel involves the following steps: first, a CPU task is launched, which initiates the execution of the kernel. Then, data is transferred from the CPU memory to the GPU memory. The GPU executes the kernel. Finally, the result of the computation is then transferred back from the GPU memory to the CPU. GPU kernels are typically described in a file called a CUDA source file. This file includes different CPU and GPU kernels, which are functions that are executed both on the GPU and CPU [START_REF] Xiang | Pipelined Data-Parallel CPU/GPU Scheduling for Multi-DNN Real-Time Inference[END_REF]. It can also include data transfers between the CPU and GPU. Kernels are assigned into streams, which are sequences of kernels that are executed in order. Streams are used to synchronize the execution of different kernels and manage the data flow between them [START_REF] Zhou | S3DNN: Supervised Streaming and Scheduling for GPU-Accelerated Real-Time DNN Workloads[END_REF].

B. DNN Inference and Tensor-RT

In the context of neural networks, we typically go through two phases: training and inference. In this work, we focus only on the inference phase wich is the process of using the trained model to make predictions on new data [START_REF] Jeong | TensorRT-Based Framework and Optimization Methodology for Deep Learning Inference on Jetson Boards[END_REF]. In order to make this process more efficient, it is important to optimize the DNN for inference. One tool that can be used for this purpose is Tensor-RT, which is a library developed by NVIDIA for optimizing DNNs for deployment on NVIDIA GPUs. It can be used to perform tasks such as layer fusion and precision calibration to improve DNN inference [START_REF] Jeong | TensorRT-Based Framework and Optimization Methodology for Deep Learning Inference on Jetson Boards[END_REF]. In this work, we consider an architecture consisting of multiple CPUs and a single GPU with multiple Streaming Multiprocessors (SMs). We consider also dependent periodic real-time tasks.

III. DNN TASK MODEL (DTM)

As shown in Fig 1(a), from the CUDA source file and realtime specifications of the system, we compute the DTM.

• The CUDA source file: outlines the software architecture of a program by specifying which functions will be executed on CPU and GPU. Additionally, it describes how data are transferred between the CPU and GPU.

• The system specification: provides information regarding the real-time constraints of the system and the quality of the data to be processed. These considerations must be taken into account during the modeling step in order to achieve an accurate scheduling analysis.

A. Task Model

We model kernels specified in the CUDA source file as two sets of tasks Γ CP U and Γ GP U . Γ CP U describes tasks executed on the CPU, while Γ GP U describes tasks executed on the GPU.

The task set Γ CP U is composed of n periodic tasks Γ CP U ={τ 1 , τ 2 , . . . , τ n }. This task set is composed by two types of tasks, traditional tasks that are independent of the GPU, and tasks that are executed on the CPU to trigger the execution of a kernel on a GPU. Each task leads to an infinite sequence of jobs. Each task τ i of Γ CP U is defined as follows:

τ i = { O τi , T τi , D τi , Π τi , C τi , N ode τi , N ext} where:
• O τi is the first release time of the task τ i .

• T τi is the period of the task.

• D τi is the implicit deadline of the task (D τi = T τi ).

• Π τi is the fixed priority of the task. 1 denotes the highest priority level while a larger value is a lower priority. The scheduling algorithm is not considered in this paper. • C τi specifies the computation time needed by the task defined as its Worst Case Execution Time (WCET). We consider a measurement-based WCET estimation. • N ode τi identifies the CPU running the task. This parameter allows us to introduce the mapping configuration, i.e. to which CPU each task is assigned to. • N ext is the precedence constraints function with:

N ext : {Γ CP U , Γ GP U } → {Γ CP U , Γ GP U } τ i → N ext(τ i ) = {τ j . . . τ k } For a given CPU task τ i , N ext(τ i )
determines the set of tasks τ j that have precedence constraints with the task τ i where τ j can be a GPU task or CPU task.

The task set Γ GP U ={ϕ 1 , ϕ 2 , . . . , ϕ m } comprises m periodic tasks. Each task leads to a sequence of infinite jobs. Each task ϕ i of Γ GP U is defined as follows:

ϕ i = { O ϕi , T ϕi , D ϕi , Π ϕi , N ext, C ϕi , Stream ϕi , T ype ϕi , Data ϕi , Size ϕi } where:
• Parameters O ϕi , T ϕi , D ϕi , N ext and Π ϕi have the same definition as tasks of Γ CP U . • C ϕi specifies the computation time needed by the task defined as its WCET. The value of this parameter is dependent on the quality of data being handled C ϕi (data quality). It is possible to set different levels of data quality, such as "low (L)", "medium (M)" and "high (H)", and thus assign appropriate values to this parameter, with the understanding that C ϕi (H) > C ϕi (M) > C ϕi (L). • Stream ϕi specifies to which stream the task ϕ i is assigned. We can have multiple streams for a given system. Multiple GPU tasks can belong to the same stream. • T ype ϕi specifies the type of operation that the task performs. In the context of DNN, there are several types of tasks, such as convolution, pooling and activation. • Data ϕi specifies data that is used by the task ϕ i .

• Size ϕi specifies the number of sub tasks (blocks) of the task ϕ i . A GPU task can be composed of multiple subtasks. Each sub task models a thread block.

The C ϕi parameter allows us to increase the accuracy of our analysis of execution time by considering multiple levels of quality. Data ϕi and T ype ϕi provide more information about the considered task and allow to apply our graph optimization. Stream ϕi , N ext, and Size ϕi enable a more accurate scheduling analysis. The DNN processing component is not inherently periodic, but it can be used as part of a larger periodic system that repeats the same steps (e.g. acquiring data, processing data using the DNN) at regular intervals. For that, we assume in this work that tasks belonging to the same DNN network have the same periods and deadlines.

B. Guidelines for Generating DTM

Each function in the CUDA code is modeled as a task. By analyzing keywords and interpreting different syntax components of the code, we can identify the various characteristics and parameters of each task. For example, to identify CPU and GPU tasks, specific keywords such as "global" and "device" can be used, which indicate that the task is executed on the GPU and CPU respectively. Here are some of the most commonly used keywords in CUDA.

• global or host indicate that a function is is executed on either the GPU or CPU. It can be modeled as a GPU task or a CPU task, respectively. • cudaMemcpy is used to transfer data between the host (CPU) and device (GPU) or between different memory spaces on the device. It can be considered in the definition of dependencies between tasks in DTM. • cudaStreamCreate and cudaMemcpyAsync manage a stream. It can be considered in the definition of dependencies between GPU tasks. There are many other keywords and syntax components available for specific tasks and functionality. In the final version of this work, more detailed information will be provided on all used keywords and how they are modeled in DTM.

IV. OPTIMIZED DTM : GRAPH OPTIMIZATION

Figure 1(b) shows the optimization we propose in this article. Using the generated model, we apply graph optimization rules in order to produce an optimized model in terms of computation time while considering real-time constraints of the system. We apply transformations to the set of GPU tasks with the aim of reducing the data transfer between the CPU and GPU. This leads to a reduction in communication delays. These optimizations can reduce memory usage, increase computation throughput, and improve overall performance.

Our optimization rules are inspired by Tensor-RT which involve merging multiple tasks into a single task when they belong to the same neural network, use the same data, and have constraints of precedence. Furthermore, tasks are fused based on operation type, such as merging Convolution task and ReLU Activation task into one. However, not all Tensor-RT optimizations are suitable for real-time systems, unlike our approach, which is specifically designed to meet these constraints. In the final version of this work, more detailed rules of graph optimization applied to DTM will be provided.

V. EXAMPLE OF A REAL-TIME DNN OVER CPU-GPU ARCHITECTURE

In this section, we give an example of CUDA code for one DNN inference application and one CPU task. The DNN application is composed of 4 GPU kernels with precedence constraints, using the same stream. By applying DTM to this code example, we will generate the following task model. For DNN inference application, we will have 8 dependent tasks, 4 GPU tasks (ϕ A , ϕ B , ϕ C , ϕ D ) and 4 CPU tasks (τ 2 , τ 3 , τ 4 , τ 5 ). For the second application, we will have one independent CPU task (τ 1 ). In Fig 2 (a) and in Table I, we give more details about the generated DNN task model for this example. Task parameters are given as an illustration to showcase our approach. This example is based on a DNN (4 layers: Normalization, Convolution, ReLU, Fully Connected (FC)) which can be used for image classification. Now we apply graph optimization on the generated DNN task model. GPU tasks ϕ B and ϕ C utilize the same data (data1) and are part of the same DNN application. As such, we suggest merging these two GPU tasks into a unified task ϕ BC .

As a result of merging the two previous GPU tasks, there will be one attached CPU task (τ 34 ) instead of two separate tasks (τ 3 and τ 4 ). Task model: In [START_REF]Ai meets real-time: Addressing real world complexities in graph response-time analysis[END_REF] the authors propose a periodic graphbased task model and a method to reduce the graph response time bounds by merging graph nodes. [START_REF] Houssam-Eddine | The HPC-DAG Task Model for Heterogeneous Real-Time Systems[END_REF] presents the HPC-DAG Task Model, a workload-based task model, which allows specifying real-time tasks deployed over heterogeneous embedded platforms. [START_REF] Casini | Optimized partitioning and priority assignment of real-time applications on heterogeneous platforms with hardware acceleration[END_REF] defines Multi-segment suspension based task model. Suspension is used for the offload mechanism such as memory copy and migration on the GPU. These models represent significant efforts to support real-time systems on GPU. However, Scheduling framework: [3], [START_REF] Kang | LaLaRAND: Flexible Layer-by-Layer CPU/GPU Scheduling for Real-Time DNN Tasks[END_REF], [START_REF] Zhou | S3DNN: Supervised Streaming and Scheduling for GPU-Accelerated Real-Time DNN Workloads[END_REF] present scheduling frameworks that have been proposed to schedule DNNs over a CPU-GPU architecture. However, models used in these frameworks are not optimally adapted for DNN applications deployed over CPU-GPU architectures.

DNN modifications: [START_REF] Zhou | S3DNN: Supervised Streaming and Scheduling for GPU-Accelerated Real-Time DNN Workloads[END_REF], [START_REF] Yang | Re-thinking CNN frameworks for time-sensitive autonomousdriving applications: Addressing an industrial challenge[END_REF], [START_REF]Ai meets real-time: Addressing real world complexities in graph response-time analysis[END_REF] propose techniques for the modification of a neural network that can improve its usage and throughput. For example, using lower-resolution images in object detection context.

DTM is distinct from most previous models. The main difference lies in our approach, which aims to accurately model both DNN and CPU-GPU properties simultaneously. DTM introduces new parameters such as data, type, size, and stream, which improve the accuracy of modeling DNN applications over CPU-GPU. These parameters enable more precise estimation of execution time while considering the quality of the processed data and allow to apply graph optimizations. Furthermore, our proposed model is automatically generated from a NVIDIA CUDA file. Finally, DTM comes with an adapted graph optimization inspired by Tensor-RT.

VII. CONCLUSION

In this article, we introduce DTM, a novel task model designed to model real-time DNN deployed over a CPU-GPU architecture. DTM takes into account all relevant factors that can affect the execution time of tasks, allowing for more accurate assessments of task schedulability. Additionally, we propose a graph optimization technique to speed up the inference of DNN while satisfying real-time system constraints.

In the future, we plan to provide a detailed explanation of DTM's rules and the specifics of the graph optimizations implemented. Additionally, we aim to integrate DTM and its optimizer into the Cheddar real-time system simulation tool.

We will implement DTM into Cheddar which is a GPL framework that provides a scheduling simulator, schedulability tests and various features related to the design and the scheduling analysis of real-time systems [START_REF] Dridi | ECTM: A network-on-chip communication model to combine task and message schedulability analysis[END_REF]. CheddarADL allows the users to describe both the software and the hardware parts of the system they expect to analyze [START_REF] Dridi | ECTM: A network-on-chip communication model to combine task and message schedulability analysis[END_REF]. We will extend CheddarADL to implement DTM.

Fig. 1 .

 1 Fig. 1. General Approach: (a) DNN Task Model (b) DTM Graph Optimization

6 * 8 // GPU code here 9 *

 689 Kernel 1 named DNN Kernel 1 uses data1 and data2. kernel 2 named DNN kernel 2 uses data1. kernel 3 named DNN kernel 3 uses data1, and kernel 4 named DNN kernel 4 uses data1 and data2. 1 #include <cuda_runtime.h> 2 __global__ void DNN_Kernel_1(int * data1, data2 = * data2;} 7 __global__ void DNN_Kernel_2(int * data1) { data1 = * data1;} 10 __global__ void DNN_Kernel_3(int * data1) { 11 // GPU code here 12 * data1 = * data1;} 13 __global__ void DNN_Kernel_4(int * data1, , * data2, * data3; 19 cudaMalloc(&data1, sizeof(int) * 100); 20 cudaMalloc(&data2, sizeof(int) * 100); 21 cudaMalloc(&data3, sizeof(int) * 100); 22 cudaStream_t stream; 23 cudaStreamCreate(&stream); 24 dim3 block(10,1,1); 25 dim3 grid(3,1,1); 26 DNN_Kernel_1<<<grid, block, 0, stream>>>(

  38 void kernel5(int * data3) { // CPU code here 39 * data3 = * data3;}

Fig. 2 .

 2 Fig. 2. (a) Task graph of DTM -(b) Task graph of the optimizated DTM

  Fig 2 (b) illustrates the optimized graph.VI. RELATED WORKSThere are many previous research works on the execution of real-time DNNs inference over CPU-GPU at different levels.