Tian Xia

Ye Tian

Jean-Christophe Prévotet

Fabienne Nouvel

Ker-ONE: a New Hypervisor Managing FPGA Reconfigurable Accelerators

Keywords: Embedded Systems, FPGA, Partial Reconfiguration, Virtualization, Hypervisor

In the last decade, research on CPU-FPGA hybrid architectures has become a hot topic. One of the main challenges in this domain is to efficiently and safely manage Dynamic Partial Reconfiguration (DPR) resources. This paper focuses on the management of reconfiguration by a custom hypervisor named Ker-ONE, on an ARM-FPGA platform. Using a virtualization approach, virtual machines (VM) may access resources independently, being unaware of the existence of other VMs. Our custom hypervisor guarantees the independence and isolation of VM domains. The purpose of our work is to provide an abstract and transparent interface for virtual machines to access reconfigurable resources, while meeting real-time constraints. This means that software engineers do not need to focus on implementation details. In this paper, we present a complete architecture in which hardware accelerators are seen as virtual devices which are universally mapped in each VM space as ordinary peripherals. The hypervisor automatically detects VMs' requests for DPR resources and handles them dynamically according to a preemptive allocation mechanism. We also evaluate the efficiency of our framework by measuring the critical overhead during DPR management and allocations. The results demonstrate that our mechanisms are implemented with low overhead compared to other approaches and that they are compatible with real-time scheduling.

Introduction

Today, the concept of CPU-FPGA hybrid processors has become more and more popular in both academic and commercial worlds. Unlike in traditional FPGA devices in which CPU cores are synthesized in the FPGA fabric as soft processors, the hybrid approach provides System on Chip (SoC) architectures 5 with CPU and FPGA domains that are independently implemented. CPU-FPGA hybrid processors have several advantages. First, general purpose processors are able to implement complex and flexible computing systems, with a huge variety of applications. Second, FPGA accelerators offer a constant improvement in performance of intensive computations and act as a powerful 10 support for processors. Additionally, the dynamic partial reconfiguration (DPR) technology on FPGA has been playing an important role in high performance adaptive computing [START_REF] Becker | Dynamic and partial fpga exploitation[END_REF].

Meanwhile, in the embedded computing domain, virtualization has gained a lot of interest and achieved enormous progress. This technique allows to separate 15 tasks into isolated domains without extra porting efforts. It has been proven that it can provide users with increased energy efficiency, shortened development cycles and enhanced security [START_REF] Heiser | The role of virtualization in embedded systems[END_REF] [START_REF] Xu | The study and evaluation of arm-based mobile virtualization[END_REF]. Therefore, we made the assumption that the combination of both DPR and virtualization is an interesting idea to significantly accelerate applications and guarantee flexibility. [START_REF] Shin | Compositional real-time scheduling framework[END_REF] While considered as quite promising, the exploitation of DPR-enhanced virtualization also brings up new challenges. In virtualization, guest OSs usually run in strongly-isolated environments called virtual machines (VM). Each VM has its own software tasks and virtual resources which abstract physical resources. In this context, the use of hardware accelerators by VMs must be 25 dynamic and independent. Note that these accelerators could be shared by multiple VMs. This means that an abstract and transparent layer has to be provided so that the isolation of virtual machines will not be undermined.

Ideally, the actual allocation and management should be performed by an hypervisor, and should remain hidden from guest OSs. Furthermore, in addition 30 2 A c c e p t e d m a n u s c r i p t to the complex problem of real-time scheduling that is often met in embedded systems, the sharing of FPGA resources among multiple virtual machines may significantly increase the management complexity. This constitutes a real challenge for designers to guarantee real-time capability.

In this paper, we address these challenges by proposing a framework ex-35 tending virtualization with DPR management. This framework features a new resource mapping and management mechanisms to provide transparent virtual FPGA resources to the VMs. Ideally, the FPGA accelerators are designed to fit in the reconfigurable area and are implemented with dedicated preemption mechanisms and context save/restore methods. Then, virtual machine tasks 40 may be programmed to access these accelerators as native devices. No further details are required for the user to deploy tasks in virtual machines.

The major contributions of this paper are listed as follows:

• We propose a lightweight micro-kernel that is compatible with real-time constraints.

45

• We describe a new approach for FPGA resource virtualization which maps resources as native accelerators in the VM domains to ease the programming of user tasks.

• We present an original FPGA management framework and a new hardware accelerators model which enable preemptive scheduling of FPGA resources 50 among multiple VMs.

• We run extensive experiments on an ARM-FPGA platform [START_REF]Xilinx, Zynq-7000 All Programmable SoC Technical Reference Manual 1025[END_REF] to evaluate the performance of our proposed approaches. Analysis and proof of the real-time capability of our framework are also provided.

The remainder of the paper is organized as follows: section 2 presents the 55 related works. Section 3 describes the architecture of the proposed hypervisor.

In Section 4 the mechanisms of the DPR management in a virtual environment are presented. Section 5 shows the performance of the micro-kernel and performs some comparisons with existing architectures. In Section 6, we demonstrate the

Related Works

An hybrid CPU-FPGA architecture generally features CPUs that are dedicated to the embedded system domain. For example, Xilinx released the Zynq-7000 series which features an ARM-FPGA SoC. ARM processors have also been 65 introduced in Cyclone-V and Arria-V Altera families. Intel has proposed its Atom processor E600C Series, which consists of an Intel Atom processor SoC and an FPGA within the same chip. Recently, Intel has taken a further step by releasing a Xeon/FPGA platform dedicated to data centers.

In the academic domain, embedded CPU-FPGA based systems have also 70 been massively studied. Numerous works have tried to provide current reconfigurable FPGA devices with OS support ( [START_REF] Jozwik | Rainbow: An operating system for software-hardware multitasking on dynamically partially reconfigurable fpgas[END_REF], [START_REF] Göhringer | Cap-os: Operating system for runtime scheduling, task mapping and resource management on reconfigurable multiprocessor architectures[END_REF] [START_REF] Agron | Run-time services for hybrid cpu/fpga systems on chip[END_REF], [START_REF] Vu | On-demand reconfiguration for coprocessors in mixed criticality multicore systems[END_REF]). One successful approach in this domain is ReconOS [START_REF] Agne | Reconos: An operating system approach for reconfigurable computing[END_REF], which is based on an open-source RTOS (eCos) that supports multithreaded hardware/software tasks. ReconOS provides a classical solution for managing hardware accelerators in a hybrid system. However, 75 virtualization is not fully discussed in these works.

In [START_REF] Huang | Hardware resource virtualization for dynam[END_REF], reconfiguration management is implemented by providing the OS4RS framework in Linux. Virtual hardware allows the same devices and the same logic resources to be simultaneously shared between different applications. However, this approach is proposed for a single OS only, without considering vir-80 tualization. Another study is described in [START_REF] Wang | pvfpga: accessing an fpga-based hardware accelerator in a paravirtualized environment[END_REF]. This was one of the earliest researches in this domain. The authors tries to extend the Xen hypervisor to support FPGA accelerator sharing among several virtual machines. However, this research proposes an efficient CPU/FPGA data transfer method, with a relatively simple FPGA scheduler that provides a FCFS (first-come, first served ) 85 sharing of the accelerator, without including DPR technology.

DPR virtualization is much more popular on cloud servers and data centers, which generally have a higher demand for computing performance and flexibility.

4

A c c e p t e d m a n u s c r i p t For example, in [START_REF] Byma | Fpgas in the cloud: Booting virtualized hardware accelerators with openstack[END_REF], authors use partial reconfiguration to split a single FPGA into several reconfigurable regions that are managed as a single Virtual FPGA 90 Resource (VFR). Based on the same principle, the work of RC3E [START_REF] Knodel | Rc3e: Provision and management of recon-1060 figurable hardware accelerators in a cloud environment[END_REF] provides several vFPGA models, allowing users to access DPR resources as full FPGA, virtual FPGA or background accelerators. However, DPR virtualization on these platforms are inappropriate for embedded systems, in which available resources are drastically limited compared to those available in servers or data 95 centers.

Another interesting research [START_REF] Jain | Virtualized execution and management of hardware tasks on a hybrid arm-fpga platform[END_REF] proposed a framework dedicated to hardware task virtualization on a hybrid ARM-FPGA platform. In this work, the authors modified the CODEZERO hypervisor to manage reconfigurable accelerators. In this work, the classical DPR technology is not exploited for hardware 100 reconfiguration. Instead, reconfigurable computing components are quite simple and seem more appropriate to systems with light but frequently-switched computations.

In this paper, we propose an original approach for DPR virtualization on an embedded hypervisor named Ker-ONE, an updated version of a custom micro-105 kernel [START_REF] Xia | Mini-nova: A lightweight arm-based virtualization microkernel supporting dynamic partial reconfiguration[END_REF]. Efforts have been made to provide efficient DPR resource sharing among virtual machines, while meeting the applications' constraints.

The Ker-ONE Architecture

Overview

In this section, we describe the design and implementation of the Ker-ONE 110 micro-kernel, which lays the foundation of our framework by offering the mandatory virtualization capabilities. Ker-ONE outperforms other approaches since it is very small and fast. Moreover, it provides enhanced real-time support.

Currently, the design of Ker-ONE is based on few assumptions:

• In a first research step, we only have considered single-core architectures, 115 leaving multi-core systems to future prospects.

5

A c c e p t e d m a n u s c r i p t • We mainly deal with virtualization of simple guest OSs such as µC/OS or FreeRTOS, instead of complex systems such as Linux, since paravirtualizing these types of OS would be quite expensive and error prone.

• In order to provide strong protection to critical tasks, we made sure that all 120 critical real-time tasks execute in one specific guest real-time OS (RTOS).

The less critical tasks execute in general-purpose OSs (GPOSs). There- to pieces of software and hardware on top of which the system security is built.

Normally, a smaller TCB size corresponds to higher security since it reduces the system's attack surface. In our case the TCB is kept small, which leads to 135 improved security.

6

A c c e p t e d m a n u s c r i p t

The user environment runs in user mode and is composed of additional system services, such as device drivers, file systems, VM bootloaders, which run as server processes (see Figure 1). Note that this framework is designed to be tual resources to the virtual machines.

In the following, we briefly introduce our approach to virtualize some basic system resources. the vGIC, which will insert a virtual interrupt to the VM and forced it to jump to its local exception vector.

Note that the states of virtual interrupts are consistent and independent in each VM. For example, a virtual interrupt can be disabled or masked by one VM, while the corresponding physical interrupt can still be collected by other 180 VMs.

Ker-ONE Optimzation

One important issue that influences the real-time capability of an OS is the kernel critical path, i.e the kernel code that cannot response to any events or be preempted. Longer and uncertain kernel critical paths will make an OS 185 unsuitable for hard real-time tasks. For example, for a monolithic OS kernel such as Linux, the costly and unpredictable kernel path severely undermined its real-time capability [START_REF] Marchesotti | A measurement-based analysis of 1070 the responsiveness of the linux kernel[END_REF]. This can only be solved by patching the kernel to make it fully preemptive or to use the concept of micro-kernel [START_REF] Regnier | Evaluation of interrupt handling timeliness in real-time linux operating systems[END_REF].

In micro-kernels, this problem is naturally relieved due to the simplicity of 190 the kernel. In our research, by applying a series of new optimization methods, we improved our micro-kernel to obtain higher performance by shortening the kernel critical paths.

8

A c c e p t e d m a n u s c r i p t interrupt arrives, the VMM will check the shared memory region and handles the events according to the state of the virtual registers.

Optimized Inter-VM Communications

The issue of IVC can be interpreted as the classical inter-process communication (IPC) problem in micro-kernels. In Ker-ONE, we use simple and optimized asynchronous communication methods instead of classic synchronous IPC model to achieve lower complexity. An IRQ-based IVC mechanisms is implemented in 230 our system. Ker-ONE leverages the VMM/VM shared memory region to facilitate asynchronous IVC. For each VM, a shared memory page is created that can be accessed from both VMM and VM sides. The sending and receiving processes of IVC mechanisms are performed with only several lines of read/write instructions on the shared memory. Therefore, this approach is shorter and 235 lightweight compared to the simplified fast IPC model in L4 micro-kernels [START_REF] Heiser | The okl4 microvisor: convergence point of microkernels and hypervisors[END_REF].

Real-Time Capability

Ker-ONE has been designed to host one RTOS and several GPOSs. The RTOS tasks are considered as critical with real-time constraints. We assume here that users are responsible for defining a scheduling strategy for the real-240 time task set with a suitable scheduler (Rate Monotonic, EDF, server-based, etc). Ker-ONE is responsible for guaranteeing real-time constraints with no or at least minimal modification of the original RTOS scheduling settings. This requires several features: the scheduling accuracy for the RTOSs, the guarantee of efficient CPU bandwidth for these RTOSs and the compliance 245 with the RTOSs' original scheduler. These characteristics will be discussed in the following subsections.

Timer Virtualization

A RTOS scheduler relies on timer ticks to determine if a specific task is ready to execute. In classic virtualization, a physical timer is managed by a 250 VMM, and VMs are provided with virtual timers that may be be accessed by traps or hyper-calls. This method is generally problematic. First, trapping into the hypervisor at each timer operation may imply high performance overhead [START_REF] Dall | Kvm/arm: the design and implementation of the linux arm 1080 hypervisor[END_REF]. Second, the VM timer resolution is bounded by the timer period of the hypervisor. For example, with an hypervisor period of 10ms, a guest OS with 255 1ms timer accuracy may not work correctly. In Ker-ONE we propose a high accuracy timer virtualization approach to improve the RTOS schedulability.

First, three independent physical timers are provided: a system timer, a RTOS timer and a GPOS timer (see Figure 3). being trapped in the hypervisor. For each VM, only one timer interface is mapped in its memory space, so that it can only access the allocated timer. A 265 guest OS is free to configure its timer, e.g. the clocking period, the interval value and interrupts.

This timer pass-through mechanism is especially advantageous for the RTOS since it fully controls a native physical timer directly. Without virtualization overhead, the performance of the RTOS scheduler is maximized.

270

Moreover, the GPOS timer has to be virtualized to protect the timer state of each GPOS, which includes saving and restoring the timers' registers values.

Although this slightly increases the VM switch overhead, this mechanism is still preferred for GPOSs since it avoids frequent hyper-calls or traps and facilitates the VM timer emulation. 275

Real-time Scheduling

Several researches on real-time scheduling in virtualization systems have already been led. For example, VMM schedulers based on compositional real-time framework [START_REF] Shin | Compositional real-time scheduling framework[END_REF] and server-based scheduler [START_REF] Xi | Rt-xen: Towards real-time hypervisor scheduling in xen[END_REF] have been designed to be used in RT-XEN and other micro-kernels. However, they either require additional 280 model computation [START_REF] Shin | Compositional real-time scheduling framework[END_REF] or require modifications of the OS original scheduling interface, which is against our intention.

In our work, we assume that users have already designed a workable schedule for a given real-time tasks set executed on a native machine. The purpose of

12

A c c e p t e d m a n u s c r i p t the VMM scheduler is to host real-time tasks according to the original scheduling settings. This strategy minimizes the additional workload on users, and simplifies the micro-kernel.

The VMM scheduler follows the concept of background scheduling, which is quite simple and reliable. Low priority tasks are only allowed to execute when high priority tasks are idle. Ideally, low priority tasks have no influence on the 290 execution of high priority tasks, since only the idle time is donated.

In Ker-ONE, a priority-based preemptive round-robin strategy is applied (see Figure 4). GPOSs run at an identical low priority level, while the RTOS is assigned a higher priority. Within the same priority level, the CPU is shared according to a time-slice-based round-robin policy.

295

The RTOS can always preempt the GPOSs as long as it is ready to run.

The events evoking RTOS include timer ticks pre-set by the RTOS scheduler and sporadic interrupts for RTOS. In either case, RTOS will be immediately scheduled and start running.Note that, system service threads automatically inherit the priority of the caller VM, so that system services are also preemptable 300 and will not block the RTOS scheduling.

With the proposed scheduling policy, and the accurate pass-through timer introduced earlier, the influence on the original RTOS scheduler is minimized.

In section 5, we will demonstrate that the virtualization overhead on the RTOS scheduler is negligible, and that the original scheduling settings are maintained.

Dynamic Partial Reconfiguration Management

In this section, the CPU-FPGA architecture is studied, where CPU and FPGA are tightly integrated. FPGA resources are connected to a CPU with dedicated interfaces and can be mapped to its unified memory space. In this context, the role of Ker-ONE is to host several simple guest OSs with different 310 priorities.

In our architecture, we made the assumption that all critical tasks are hosted in a high-priority VM, with high performance. Non-critical tasks run in low- priority VMs, for which long latency and resource blocking may be tolerable.

To predict the behavior of critical tasks, we also assume that the FPGA 315 resources are always sufficient for the high-priority VM, whereas they can also be shared and re-used by low-priority VMs. This assumption seems reasonable in practice, since critical tasks are pre-determined in most embedded systems.

Accelerator Mapping

In our system, reconfigurable accelerators are hosted in different partial re-320 configuration regions (PRR), which can be seen as containers. These accelerators are denoted as hardware (HW) tasks.

Each HW task is an instance of an accelerator algorithm and can be implemented in different reconfigurable regions by downloading the corresponding bitstream into the targeted area via the PCAP interface [START_REF]Xilinx, Zynq-7000 All Programmable SoC Technical Reference Manual 1025[END_REF]. HW tasks are pre-325 sented as virtual devices (VD) in the VM domain, and completely abstract the implementation details.

Hardware Task Preemption

Considering that multiple VMs share FPGA resources, the RTOS tasks may be unexpectedly blocked when resources are occupied by GPOS tasks. To guarantee the timing constraints of real-time tasks, the HW tasks should be preemptible so that resources can be re-assigned to RTOS tasks when necessary.

405

We denote the VM corresponding to a HW task as a client.

HW tasks inherit the priorities of their VM clients, meaning that virtual devices in RTOS and GPOS have different priorities. In our policy, the execution of low-priority HW tasks can be preempted when RTOS virtual devices require more FPGA resources. Note that HW tasks with the same priority level cannot 410 be preempted.

The preemption mechanism requires to address several issues to make sure HW tasks can be safely stopped and resumed. First, to protect data integrity, accelerators may only be stopped when they reach some point in their execution, for example, the interval of data frames in communication processing. These 415 points are denoted as consistency points where the execution path is safe to be interrupted and can be resumed without a loss of data consistency. Designers of HW tasks have to identify the consistency points that allow the accelerators execution to be preempted and to save the interrupt state.

Additionally, the context of HW tasks must be properly handled. We define 420 the HW task context as the accelerator logic and the register states in the accelerator. The logic is stored in the bitstream file and is indexed in the HW Task Index table. On the other hand, the registers states depend on the design of accelerators.

As shown in Figure 6, in each IF, a 1KB buffer is implemented to store the 425 accelerator context when preempted, which can later be used to resume its execution. Since the format of the saved context depends on the accelerator design, it is the designer's work to implement the save/restore routine of an accelerator.

This routine is registered and called back by the VMM when preemption occurs. As a container, a PRR is allocated to HW tasks to provide FPGA resources and behaves as a state machine. The state determines if a PRR can be allocated to a specific HW task, and how it could be allocated.

Six states exist:

440

• Idle: The PRR is idle without any ongoing computation and is ready for allocation.

• Busy : The PRR is in the middle of a computation.

• Preempt: The PRR is running, but the computation will be stopped (preempted) once it reaches a consistency point.

445

• Switch: The PRR is in the middle of a context switch.

• Reconfig : The PRR is in the middle of reconfiguration.

• Hold : The PRR is allocated to a VM and is preserved for a certain amount of time.

The PRRs' behaviour can be described according to the flow chart given in We have also introduced the Hold intermediate state. PRRs that are allocated to a VM will first enter this state. This indicates that the PRR is reserved for a certain VM client. PRRs in the Hold state will block any re-assignment and will wait to be used by the VM. PRRs will be released and return to the Idle state when the preset waiting time Expire runs out.

460

A PRR holds the essential information in a PRR Descriptor data structure. This list indicates the PRR state (see Figure 7). It also includes the information of the currently-hosted HW task: the client VM ID, the virtual device ID (i.e.

accelerator ID) and the HW task priority, which are used to make allocation decisions. Note that, in our context, the bitstreams size is strictly pre-defined 465 by the size of the reconfigurable area. Therefore, the reconfiguration time of each PRR can be easily predicted. This factor is also included in the PRR Descriptor.

Management Mechanism

One major characteristic of virtualization is that VMs are totally indepen-470 dent from each other. In our case, however, VMs share reconfigurable resources. This can unfortunately lead to resource sharing issues that are well known in computing systems. In traditional OSs, such a problem can be solved by applying synchronization mechanisms like semaphores or spin-locks.

For Ker-ONE, such mechanisms are not suitable since they may undermine 475 the independence of VMs. Therefore, our system introduces additional management mechanisms to dynamically handle the VMs' request for PRR resources. Note that such requests may occur randomly and are unpredictable.

In Figure 8, the proposed management mechanism is described, which mainly involves two components: a Virtual Device Manager on the software side and a 480 PRR Monitor in the FPGA hardware.

The Virtual Device Manager is a particular software service implemented in an independent VM domain, which aims at detecting and handling the requests coming from VMs that want to use their virtual devices. This is performed through an Inter-VM Commnunication (IVC) mechanism.

485

The PRR Monitor is running in the static part of the FPGA and is in coop- The PPR Monitor on the FPGA side is responsible for searching appropriate allocation plans for such requests. This plan is referred as a solution. A complete 500 solution is formatted as:

Solution{vm, dev, M ethod(prr id), Reconf ig}, (1)
which includes the target VM, the required device, the actual allocation method and reconfiguration flag. The different methods include:

• Assign(prr id): this solution directly allocates the returned PRR (i.e.

prr id ), which is Idle, to the request VM. If the requested device dev id is 505 not implemented in this PRR, a Reconfig flag is also added.

• Preempt(prr id): all PRRs are Busy and none can be directly allocated, but the returned PRR (i.e. prr id ) can be preempted and re-allocated.

If the requested accelerator (dev id ) is not implemented in this PRR, a

Reconfig flag is also added.

510

• Unavailable: currently no PRR is available for Request(vm id, dev id, prio).

The PPR Monitor searches for the best solution by checking the PRR Descriptors (see Figure 7). For a given Request (vm id, dev id, prio), the PRR Monitor first obtains the list of compatible PRRs for the target device (dev id ) 515 by checking the HW Task Index table.

The states of these compatible PRRs are then checked for possible solutions.

If multiple solutions are found, the best one is chosen according to the selecting policy.

In our algorithm, assigning Idle PRRs are considered to be best solutions.

520

Preemption is chosen only when no Idle PRR exists. Besides, the selector always chooses the solution with a minimal PRR size since it causes the minimal reconfiguration overhead and power consumption. However, these policies can be easily modified and adapted.

22

A c c e p t e d m a n u s c r i p t The Virtual Device Manager is a special service provided by Ker-ONE, running in an independent VM. This service stores all the HW task bitstreams in its memory and is the only component that can launch PCAP reconfigurations.

The main tasks of this manager are: (1) to communicate with VMs and manage the virtual devices in their space; (2) to correctly allocate PRRs to VMs.

540

As already explained, if any VMs try to use an unavailable virtual device, this will automatically be detected by the VMM, and then forwarded to the Virtual Device Manager.

In Figure 10, the full flow to allocate an accelerator to a VM is depicted.

In this example, after a given Request(vm01, dev01, prio01), a solution {Assign the IPC PREEMPT signal, a GPOS is aware of its deprivation of FPGA resources. Advanced programming policies can be applied to improve the QoS of 615 its tasks. For example, GPOS tasks can move the computation workload from the accelerator to the CPU, to avoid the long blocking of specific computations.

Ker-ONE Virtualization Performance Evaluation

In this part, the performance of our micro-kernel is provided. Several experiments have been led to measure the impact of virtualization and make sure 620 that such a system can be used in very small real-time embedded systems.

The first experiment has focused on measuring the overhead of fundamental virtualization functions, such as VMM scheduling, hyper-calls, interrupt management, etc. Then the impact of virtualization on the RTOS execution has been quantified by measuring the overhead that is due to the VM schedul-625 ing. This study has been led using a standard RTOS benchmark. Finally, our platform has been used to implement specific applications taken from standard benchmarks to demonstrate its feasibility.

27

A c c e p t e d m a n u s c r i p t

Our experiments were performed on the ARM Cortex-A9 processor of Xilinx ZedBoard (i.e. the Zynq-7000 SoC), and the frequency has been set as 667 MHz.

630

In order to evaluate the performance of our platform, we have implemented multiple guest OSs (i.e. Mini-µC/OS-II) on top of Ker-ONE. These guest OS had to execute specific applications on a huge number of samples. Two main benchmarks have been considered, Thread-Metric [22] and MiBench [START_REF] Guthaus | Mibench: A free, commercially representative embedded benchmark suite[END_REF].

In all our tests, the VMM scheduling period was set to 33 ms. Guest OSs 635 used a 1 ms timer tick for their own schedule. These values are quite common timing configurations in this context and especially for µC/OS-II [START_REF] Yoo | Real-time scheduling for xen-arm virtual machines[END_REF]. Guest OSs were either configured as GPOS or RTOS according to the experimental requirements.

Basic Virtualization Functions Overhead 640

The different measurements that have been performed in the experiments allowed us to identify the most critical VMM functions. The platform has been configured to host four similar µC/OS-II at the same priority level. These were considered as GPOS and scheduled according to a round-robin strategy.

Software tasks were running in the guest OSs and making hyper-calls. The 645 overheads of the corresponding VMM services that were required to handle these hypercalls have then been recorded by a background monitor during several hours. Figure 12 depicts the experiments results, where minimal, average and maximum overheads are presented in microseconds.

The overhead latency that is required to generate an hyper-call, to process 650 this hyper-call in the VMM and to return back to a virtual machine has been evaluated. This corresponds to the VM entry/exit latency overhead. At this point, it is important to note that hyper-calls are generally performed by the guest OS and rarely by user tasks.

Since Ker-ONE is mapped to the VMs' address space, no switch between 655 VM is required. Hyper-calls entries and exits are relatively low cost processes since they only involve the save/restore of the CPU context. #$%&'%() 0123 456%&78(99 @AB&5C@DEB 0 FEAG §¡¨12H ($&G §¢ 12H F(DG §© 12H 2(F69%G©¥¤ ¥ 3 Another important metric is the virtual IRQ emulation latency that represents the cost of emulating a virtual interrupt for a VM. This functionality is critical for event-driven OSs and this latency has a huge impact on the events' 660 response time. This metric is also closely related to the guest OS' scheduling overhead since a guest OS is driven by a virtual timer tick to handle virtual time. This overhead is measured from the physical event's arrival time until the time at which the VM is forced to its local exception vector. This process involves the handling of physical IRQ and the emulation of the virtual GIC 665 interface registers.

I'56%&8(99 %AB&5I ¡ ¢ £ ¤ ¥ ¦ § ¨£ ¨£¥ ¨¤ ¨¤¥ ¨¥ ¨¥¥ ¨¦ ¨¦¥ ¨ § © !"#$%#&' ()01 23$4)&5 678 9@)5&43AB ( @3BC ¨£¤)0D &"$C ¨£ §)0D @&EC ¨¦F)0D 0&@G5#C¢ §£¢£¥ 1 H3$I #B4$PH ¡ ¢ £ ¤ ¥ ¥¡ ¦¥ ¦¡ ¦ § ¦¢ ¦¨ ¦£ ¦© ¦¤ ¦ ¥ ! " #$%&'%() 0123 45&61(7 8(9'5@% AB569' 0 C5@D ¦¥©12E ($&D ¦£¡12E C(FD ¦¥12E 2(CG7%D¡¡ §¤¥¨3 HBI&7) 2B6H
The virtual machine switch latency represents the cost of switching from one VM to another and may be relatively heavy. The overhead of the virtual machine switch is one of the key metric in most virtualization approaches, as it is usually quite cumbersome, and has a huge impact on the VMM efficiency.

670

In Ker-ONE, this switch is performed when a VM consumes its time quantum and moves to its successor, or when it suspends itself and the VMM schedules another VM or process. This switch includes several major procedures: ( Usually, the VMM uses these functions for management and emulation purposes and they are of great importance. Virtualization efficiency is closely related to the performance of these functions. In our case, we can note that these 680 functions exhibit low overheads. As shown in the results, frequently-called functions, i.e. hyper-calls and vIRQ emulation can be handled in less than 1 µs.

Furthermore, the virtual machine switch overhead, which constitutes the most expensive process, could be limited to 1 µs.

RTOS Virtualization Evaluation

685

In this section, we quantify the impact of virtualization mechanisms on the performance of guest RTOSs. This includes the OS kernel services as well as scheduling overhead. In order to estimate the impact of virtualization, a controlled experiment has been performed. The control group implements a native RTOS on an ARM Cortex-A9 processor, while in the experimental group, 690 the same RTOS is implemented on top of Ker-ONE. Differences in terms of performance have been measured.

In our experiments, Mini-µC/OS-II has been implemented as an RTOS in a VM. Three other VMs were used to host another instance of Mini-µC/OS-II, which plays the role of a GPOS. Benchmarks have run as applications in the 695 RTOS and a comparison between native execution and execution on a VM has been performed for each test.

30

A c c e p t e d m a n u s c r i p t

Benchmarking

We chose the Thread-Metric benchmark suite for the RTOS performance measurement [START_REF] Expresslogic | Measuring Real-Time Performance Of An RTOS[END_REF]. Thread-Metric has been developed by Express Logic in 2007 700 and has been applied in several works to measure and compare the performance of multiple RTOSs [START_REF] Best | Real-time operating system hardware extension core for systemon-chip designs[END_REF].

In our experiment one RTOS and three GPOSs (all Mini-µC/OS-II) run on top of Ker-ONE. The Thread-Metric suite is executed on the RTOS. In order to obtain the performance loss due to virtualization, the benchmarks results on 705 the native µC/OS-II are also collected and used as reference.

To provide an extensive evaluation, the XEN-ARM hypervisor has been evaluated to achieve a comparison with our micro-kernel. The XEN-ARM hypervisor Version-3.0 [START_REF] Hwang | Xen on arm: System virtualization using xen hypervisor for armbased secure mobile phones[END_REF] has been ported to our platform that is based on a Zyng-7000 device. A para-virtualized µC/OS-II (denoted as xeno-µC/OS), that 710 is available on the XEN website as been used as reference. The Thread-Metric benchmark has been executed on this kernel. Note that, since µC/OS-II runs on a single protection domain, no multiple page tables are necessary. Although XEN-ARM and Ker-ONE have different memory virtualization techniques, both virtualization contexts of µC/OS-II are similar in this case. The XEN's sup-715 port of user-level multiple protection-domains has not been used to provide fair comparison.

The role of the Thread-Metric benchmark is to provide a set of common kernel services to compare different RTOS in terms of performance. These services mainly deal with context switch, interrupts handling, message passing, 720 memory management, etc. For each OS service to be tested, the corresponding function as well as its dual function have been executed in pairs, e.g. allocating/ de-allocating memory, or sending/receiving messages. These functions were executed continually and the number of iterations has been evaluated. Finally, the number of iterations has been recorded every 30 seconds and denoted as • Message Processing: One task is created to repeatedly send and receive 735 message through the OS message queue (i.e. OSMessagePost, OSMes-sagePend).

• Memory Allocation: One task that allocates and releases memory through the OS memory block (i.e. OSMemGet, OSMemPut ).

• Synchronization Processing: One task that pends and posts semaphores 740 (i.e. OSSemPost, OSSemPend).

• Interrupt Handling: One task is created to generate software IRQ. The semaphore mechanism is used in the IRQ handler routine to guarantee the handling completion.

• Interrupt Preemption: Two different priorities tasks are created. The 745 lower priority task generates a software IRQ and while it is executing its IRQ handler routine, the other task is resumed and preempts the low priority one.

Based on the experiments above, the metric Performance Ratio has been defined and denoted as R P , which is computed as:

R P = S vm S native × 100%, (2)
where S vm is the benchmark score obtained by the guest OS, and S native concerns the native OS. R P measures the influence caused by virtualization. A 750 better virtualization technology means less performance loss and thus a higher

32

A c c e p t e d m a n u s c r i p t R P value. Table 3 presents the experimental results of the Thread-Metric benchmarks running on both Ker-ONE and native environments, and the corresponding performance ratios. As shown in Table 3, µC/OS-II has lower performance when virtualized.

755

This is predictable since the benchmark tests include intense executions of sensitive instructions and privilege operations on protected system resources. One typical operation is the context switch in a guest OS. Originally, this is performed with only a few lines of assembly code. However, in a para-virtualized implementation, this operation is normally re-directed to a bunch of assembly 760 lines of code and involves multiple hyper-calls and VMM handling. In our test, when a context switch is performed frequently, a noticeable extra overhead is caused compared to the original code.

Timer and interrupt virtualization also degrade the performance. The emulation of such mechanisms is particularly expensive if guest OS executes very 765 fast or/and require frequent interrupts. This may be noticed in the Interrupt

Handling and Interrupt Preemption benchmarks that are presented in Table 3.

In these benchmarks, a huge number of interrupts are generated and handled.

This results in a significant performance degradation and explains the relatively low performance that is obtained in these tests. In Table 3, we can also notice that our micro-kernel performs well when hosting RTOS. Regarding the services that are evaluated in the benchmarks, most losses are under 20%. For some functions such as task scheduling and message processing, the performance is even better and close to those obtained with native OS: only 10% of performance loss. This is due to the fact that, in the 775 Ker-ONE design, virtualization of resources have been optimized using a shared memory region (see Section 3.2.1), which reduces the number of hyper-calls and provides significant improvement in terms of performance.

Figure 13 provides a comparison between two different systems that are implemented in the same platform. The first is the Ker-ONE kernel. The 780 second is the Xen-ARM hypervisor. We may also note that Ker-ONE performs better than XEN-ARM when hosting the µC/OS-II guest OS.

At this point, it is important to notice that both kernels make use of a share memory region. This may be explained because of the virtual interrupts that are handled differently in XEN-ARM. In this hypervisor, these are manipulated as event channels that separate physical IRQs from VM event ports. This strategy is efficient to ensure isolation between virtual machines but is also more complex.In our ap-795 proach, Ker-ONE implements a simple virtual IRQ management that is oriented towards the GIC emulation. A simple function forwards the physical interrupts to the VMs. Moreover, the different VMs keep on using their own IRQ handlers, which simplifies the system.

RTOS Virtualization Overhead

800

Whereas the previous analysis has evaluated the performance of specific OS functions with the Thread-Metric benchmark suite, we also created our own custom benchmarks to estimate the scheduling and context switching overhead.

With these benchmarks, schedulability studies may be performed as described in [START_REF] Stewart | Measuring execution time and real-time performance[END_REF].During these tests, we carefully evaluated the worst-case RTOS task 805 response time. We have noticed that this occurs when the RTOS preempts the GPOS to get scheduled.

We define this response time as Response V M , which is composed of: delays caused by the VMM critical execution (∆V M M critical), by VMM scheduling (∆V M M sched) and by RTOS task release (relEv V M ). These three types of 810 overhead have an impact on the release delay of RTOS tasks as demonstrated in [START_REF] Aichouch | Evaluation of an rtos on top of a hosted virtual machine system[END_REF]. In our experiment, these types of overhead have been measured respectively and recorded during hours of execution.

A total number of 1,048,576 samples have been obtained during long time experiments. For each measurement, we have evaluated the minimal, average

36

A c c e p t e d m a n u s c r i p t

In Figure 14, the Task Release latency represents the time that is required to handle a virtual timer tick in an RTOS and to schedule a new task. Two latencies have been measured for native and VM corresponding to relEv N ative 835 and relEv V M , respectively. A loss of performance is to be expected in virtualization in the Task Release latency. This is mainly due to the emulation of sensitive instructions that are required to handle interrupts and to perform a context switch. It follows that the worst-case extra Release Event overhead can be estimated as:

840 ∆ relEv V M = relEv V M (WCET ) -relEv N ative (BCET ) , (3)
where relEv N ative (BCET ) is the best-case execution time of the native latency. In this equation, ∆ relEv V M has been estimated at 0.42 µs. Therefore, the equation that gives the total influence that virtualization causes on the RTOS response time ∆ Response V M may be written as:

∆ Response V M = ∆V M M critical + ∆V M M sched + ∆ relEv V M . (4)
According to the experiment results, ∆ Response V M

has been estimated at 3.08 845 µs. Considering that, the scheduling tick is usually set as 1ms or 10ms in RTOS, the virtualization overhead can be neglected in terms of the real-time task response time. Therefore its influence on real-time schedulability can be ignored.

Reconfigurable Accelerator Management Evaluation 850

In this section we evaluate how the tasks' execution time is influenced by the FPGA resources sharing and determine the tasks WCET. To this purpose, we define the allocation latency L alloc , which corresponds to the delay that is required before an accelerator (i.e. FPGA resources) is properly allocated and ready to start. This latency can be seen as the response time of a virtual device on Ker-ONE. Note that only RTOS tasks are considered here. In this configuration, virtual devices can always demand resources and cannot be preempted, meaning that the allocation latency can be determined via static analysis.

The allocation latency has two main sources: the allocation mechanism itself 860 and the Ker-ONE micro-kernel functions. Additional overhead is to be deplored if the allocated accelerator requires reconfiguration. Besides, the virtualization mechanism takes up extra time. For example, the page-table faults handling, IPCs and VM scheduling will noticeably contribute to the total allocation latency. The models of execution paths in different solutions can be calculated 865 according to the diagrams displayed in Figure 15.

In these models, the allocations consist of four different solution paths that can be decomposed into the following list of smaller atomic execution overheads:

• T trap : Time required by Ker-ONE to detect a page-table exception in VM domain and to invoke the Virtual Device Manager. • T preempt : Overhead due to the preemption of the current accelerator.

880

Based on this model,the worst-case allocation latency can be determined as follows:

L alloc(WCET) = max T P ath1 , T P ath2 , T P ath3 , T P ath4 .

(

) 5
In order to estimate and analyze the impact of L alloc , an experiment has been led and described in section 6.1. This experiment is taken from an OFDM receiver that is intended to be very flexible by considering several configurations of modulators and mappers according to the channel conditions. QAM blocks aim to take a complete frame at run-time. In order to respect the integrity of the OFDM process, both QAM and FFT modules may be preempted only when their data frame is completely processed.

Experimental Description

The experiment ran for several hours continuously. A custom monitor has been built to measure and record the various costs of allocation mechanisms on 925 the RTOS tasks.

Overhead Analysis

The measurement results of atomic execution overheads are provided in Table 4. According to this table, it may be seen that VM scheduling as well as virtual interrupt emulation are performed with a low overhead that is less 930 than 1µs. The highest overhead is obtained in T Solution [START_REF] Heiser | The role of virtualization in embedded systems[END_REF] , which occurs when a PRR is assigned with reconfiguration. In this case, this process requires a PCAP transfer which is time consuming since it consists of complex operations to organize the download of bitstream files.

According to the performed measurements, the allocation latency of different solution paths, as modeled in Figure 15, can be estimated as:

T P ath1 = 3.03µs, T P ath2 = 6.76µs + T RF CG , T P ath3 = 5.10µs + T preempt , T P ath4 = 9.96µs + T preempt + T RF CG .

We may notice that a 3 µs latency is obtained for a direct allocation. Other solutions have additional latencies due to preemption (T preempt ) or reconfiguration time. The costs of T preempt and T RCF G are mostly depending on the implementation and application of accelerators.

940

In Table 5, these costs are evaluated for all available accelerators. T RCF G is determined by the size of the bitstream, and therefore corresponds to three PRR areas. The preemption time T preempt is determined by the δ pre of the accelerator to be preempted. δ pre corresponds to the worst-case waiting time when preempted, and depends on the consistency points which are set as the 945 interval of data frames.

In terms of WCET analysis (i.e. T preempt(WCET) and T RCF G(WCET) ), it is important to note that they not only depend on the implementation, but also on the accelerators are being globally designed and used.

For example, considering the compatibility shown in

Therefore, for each accelerator, its worst-case allocation latency L alloc(WCET) can be calculated by obtaining T preempt(WCET) and T RCF G(WCET) according to the system design, and then following the equation 5.

955

Note that, the implementation of the PRRs and accelerators are set beforehand and the RTOS tasks' access to accelerators are also known. For each RTOS task, the impact of L alloc(WCET) can be predicted and be added to its WCET value for the RTOS schedulability analysis.

Discussions 960

From Table 5 we can notice that for the accelerators used in our experiment, T preempt is significantly lower than T RCF G . Therefore, from the RTOS point of view, preemption is always the best solution since it encourages to benefit from existing accelerators of low priority tasks, and reduces the need for reconfiguration. However, for GPOS tasks, being preempted will block their execution.

965

For a system in which preemptions may occur frequently, it is possible that a GPOS may never get access to hardware resources. Hence, a trade-off should be made regarding the allocation policy.

In our work, we made the assumption that allocating Idle PRRs is always better than preempting them. The reason is that we want to make sure that low prior-970 ity tasks will not be infinitely blocked by FPGA resources. The PRR Monitor has been designed accordingly. In a system that manages critical tasks of tight timing constraints, a new policy may be followed that gives more importance to preemption. 1 For purpose of simplicity, we present the resource usage portion on the total FPGA fabric, instead of the detailed amount of LUT, FF, etc.

In Table 6, we compare the HW acceleration approach with software. The re-975 sults show that the accelerator performance of heavy computation (i.e. FFT512/1024) significantly surpasses software implementation. Even though these accelerators suffer from allocation latency that may prolong the execution time, their benefit is still considerable. On the other hand, for relatively light computation(e.g QAM), although hardware accelerators are still faster, this advantage gets un-980 dermined when taking T RCF G into account. These results indicate that DPR technology is more suitable for large complex computation algorithms. Furthermore, in this example, the FPGA is capable of simultaneously providing total 8 virtual devices with only 3 PRR areas, whose total cost is around 23% of the available resources (2%, 8% and 13% respectively). More importantly,

985

from the above analysis it can be concluded that the real-time schedulability of RTOS VM is not undermined. Considering traditional FPGA design, to support both VMs, all 8 accelerators need to be implemented as static circuits, which may take up to 50% resources. Therefore, in our approach, the usage of FPGA is greatly reduced while the real-time safety can be preserved.

Conclusion

In this paper we have introduced an hypervisor which facilitates the DPR resource management in a system composed of several virtual machines. Our

3

 3

A c c e

  p t e d m a n u s c r i p t feasibility of the proposed system with practical hardware/software applications 60 and analyze the results.

  225

10

 10



  Independent physical timers with priority. Direct access to timers. Dedicated RTOS timer to facilitate real-time scheduling.

Figure 3 :

 3 Figure 3: Management of 3 independent physical timers by the VMM, RTOS and non-RTOSs respectively. For a single guest OS, only one timer interface is mapped in the corresponding memory space.

  305

13

 13

Figure 5 :

 5 Figure 5: Allocation of virtual devices to virtual machines

Figure 5 14 A c c e p t e d m a n u s c r i p t

 514 Figure 5 describes the way virtual devices are mapped to fixed addresses in all guest OSs. A unified interface, i.e. a standard structure of registers, is provided to users. Like any other peripherals in ARM systems, OSs access these devices 330

  400

17

 17

A c c e

  p t e d m a n u s c r i p t

18

 18

A c c e

  p t e d m a n u s c r i p t Preemption is performed by making a context switch on a PRR i.e. stopping 430 one HW task and reloading another. A complete context switch includes: (1) the reconfiguration of the accelerator logic by downloading a bitstream into the target PRR; (2) the saving and resuming of the corresponding register states, following user-designed routines. In the following we introduce how a PRR is designed to facilitate the preemption policy.

435 4 . 3 . 1 .

 431 PRR State Machine

Figure 7 :

 7 Figure 7: PRRs state machine

20

 20

Figure 8 :

 8 Figure 8: Overview of the DPR management framework in Ker-ONE

A c c e p t e d m a n u s c r i p t computation workloads.

545 23 A c c e p t e d m a n u s c r i p t

 23

Figure 11 : 25 A c c e p t e d m a n u s c r i p t

 1125 Figure 11: Virtual device manager handling solutions

28

 28

Figure 12 :

 12 Figure 12: Basic virtualization functions overhead in microseconds (µs) with minimum, average and maximum values.

29

 29

725

  test score. A high score means a low overhead in the OS kernel function and obviously better performance. The tests provided by Thread-Metric are: 31 A c c e p t e d m a n u s c r i p t • Calibration Test: A basic single-task rolling counter function to set up a performance baseline for comparisons. • Preemptive Context Switching: Five tasks of different priorities are cre-730 ated. Staring from the lowest priority task, each task resumes the next higher priority task and suspends itself. The sequence of OS scheduling (i.e. OSTaskSuspend, OSTaskResume, OSSched in µC/OS-II ) is evaluated.

Figure 13 :

 13 Figure 13: Comparison of Thread-Metric Performance Ratio (R P ) for para-virtualized µC/OS-II on Ker-ONE and XEN-ARM.

815 35 AG122 3 G12Figure 14 :

 35314 Figure 14: VM RTOS task response overhead in microseconds (µs) with minimum, average and maximum values

855 and can be used to represent the increase of a task execution time when running 37 AFigure 15 :

 3715 Figure 15: Execution paths of DPR resources allocation

870•

  T resume : Time required by Ker-ONE to schedule back to a VM. • T P L irq : Time required by Ker-ONE to receive IRQs from the PRR Monitor and to redirect them to the Virtual Device Manager. • T Search : Time required by the Virtual Device Manager to receive the VM requests and to search for solutions. 875 38 A c c e p t e d m a n u s c r i p t • T Solution(1)(2)(3) : Execution time to handle different solutions: (1) direct assignment, (2) assignment with reconfiguration, (3) preemption. • T irq pcap , T irq stop : Time required by the Virtual Device Manager to handle the following IRQs (i.e. IRQ PCAP Over, IRQ PRR Stop).

885

  As mentioned earlier, our experiments were performed on the Xilinx Zed-Board (Zynq-7000 SoC). This SoC consists of two parts: the processing system (PS) which provides a dual-core ARM Cortex-A9 processor, and the programmable logic (PL) which includes a partially reconfigurable FPGA fabric. The CPU operating frequency has been set 667 MHz and the FPGA logic was 890 driven by a 100 MHz clock. The proposed experiment is shown in Figure 16. The FPGA fabric on PL side has been initially implemented with three PRRs of different sizes. Four hardware accelerators, i.e. QAM16, QAM64, FFT512, FFT1024, have been implemented and stored into bitstream files. During the initialization stage of 895 Ker-ONE, these files have been loaded into the RAM memory and are only accessible by the Virtual Device Manager.

  43

A c c e p t e d m a n u s c r i p t

  990

  44

A c c e p t e d m a n u s c r i p t framework is based

  on Ker-ONE, a micro-kernel running on the ARMv7 architecture. This micro-kernel is able to host multiple OSs. In each virtual machine,995DPR accelerators are mapped as universally-addressed peripherals, which can be accessed as ordinary devices. Through dedicated memory management, our kernel automatically detects the request for DPR resources and allocates them dynamically. Dedicated management components are implemented on both software and hardware sides to handle allocations at run-time. We also propose an 1000 efficient preemptive allocation mechanism that emphasizes the sharing and enhances security for virtual machine systems. In this paper we have described implementation details and presented extensive experiments to evaluate the overhead of allocation in our framework. Through evaluations and analysis, we have demonstrated that the proposed framework is capable of virtual machine 1005 DPR allocation with low overhead and guaranteed real-time schedulability. As prospects, we would like to evaluate our framework more deeply by applying real-scenario implementations, e.g. complex communication systems with realtime tasks, to discuss the capability and schedulability of hosted guest OSs. We would also like to develop more sophisticated searching algorithms, so that the 1010 overall performance may be improved.

A c c e p t e d m a n u s c r i p t

Priority-based Round-Robin Scheduling

			Real-Time Event	Real-Time Event	Real-Time Event
	Priority	RTOS	Idle	Idle	Idle
	2 Priority 1 Priority 1	GPOS GPOS	m RTOS Suspend RTOS Preempt a n u s c r i p t Round-Robin Scheduling Figure 4: Priority-based Round-Robin Scheduling.
	The system timer is dedicated to the host and can only be accessed by the micro-kernel. The RTOS timer is exclusively used by the RTOS VM. The GPOS timer is shared by the other A c c e p t e d 260
	VMs.
	Second, we allow VMs to access and program the timer directly without
			11

  Note that the physical positions of these virtual devices are not determined since they can be implemented in different PRRs.An interface component (IF) has been implemented on the FPGA side. This interface can be seen as an intermediate layer between the logical virtual devices 335 and the actual accelerators. It is in charge of connecting the virtual machines with accelerators so that software can control their behavior. Each IF is exclusively associated to a specific virtual device in a specific VM. Therefore, mapping of reconfigurable accelerators is performed in two steps.First, the IF is statically mapped to the VM address space as a virtual device interface. Second, the IF

		is triggered. This mechanism guarantees the unique use of accelerators, and CPU FPGA
		interface automatically detects any VM's request on unavailable FPGA resources. IF PR accelerator Client VM
	365	See the example in Figure 5. In VM #1, an application is free to program VD
	370	m a n u s c r i p t and command Dev #1 (FFT-2048) as the associated IF is currently connected to PRR #1, where the HW task is implemented. Meanwhile, VM #1 cannot give orders to Dev #2 and #3 since these their IFs are currently read-only. Any writing on these IFs will cause a page-fault exception to the VMM. This type of exception will be handled as a VM's request for FPGA resources, which is automatically detected. m PR Accelerator Context context save/restore Buffer VMM DMA a n u s c r i p t Figure 6: Reconfigurable accelerator model
	We leverage the ARM paging mechanism to control the VMs access to IFs. When an IF is connected, its registers are mapped as read/write so that a VM can directly control the accelerator. On the other hand, for unavailable devices A c c e p t e d 360 A c c e p t e d
		(with an unconnected IF), the registers are set as read-only and whenever a
		VM configures or commands a virtual device by writing to its IF, a VM exit
		15 16

340

is dynamically connected to the target PRR that implements the corresponding device function. The second mapping is performed on the FPGA side.

As shown in Figure

5

, IFs are initiated on the FPGA side and are assigned to physical memory addresses on the processor side. Their physical addresses are configured to be aligned to 4KB memory pages. The VMM manipulates the 345 page table of each VM to map IFs to the VM's guest physical address space as independent device interfaces. IFs that offer the same virtual device function are mapped to identical address spaces in different VMs. For example, in Figure

5

, though the QAM accelerator is mapped to the same virtual address for all VMs, it has different IFs in the FPGA.

350

As the mapping between a particular IF and the VM space of a virtual device is fixed, an IF can be identified with two identifiers: vm id and dev id (i.e. referred to as IF(vm id, dev id )), which indicates the VM and the accelerator algorithm to which it is associated. An IF has two states: connected to a certain PRR or unconnected. When it is connected, the corresponding virtual device 355 is implemented in the PRR and is ready to be used. Being in the unconnected state means that the target accelerator is unavailable.

4.2. Hardware Task Model

HW tasks are accelerator instances running in PRRs containers. PRRs provide FPGA resources to implement their algorithms. A given PRR may not be compatible if its area (i.e. amount of resources) is insufficient to implement the 375 corresponding accelerator algorithm of virtual device function. Therefore, the compatibility information of HW tasks must be foreseen beforehand. An HW Task Index table is created to provide a quick look-up search for HW tasks.

In this table, the compatible PRRs for each virtual device are listed. For each compatible PRR, a HW Task Descriptor structure is provided, which stores 380 the information of the corresponding bitstream, including its identifier, memory address and size. This information is used to correctly launch PCAP transfers and perform reconfiguration. Figure 6 depicts the model of HW tasks and its interaction with VMs. As shown in this figure, VMs access HW tasks via IFs. We proposed a standard 385 register structure to facilitate the multiplexing of PRR resources, denoted as the partially reconfigurable (PR) accelerator interface. It is implemented in the IF, and conveys the register values from the IF to HW task. Once the IF is connected to an HW task, a VM can operate on the IF registers to control the HW task behavior.

390

In Table

1

, the structure of the PR accelerator interface is listed. VMs start the HW task workload by setting the START flag. When the required computation is over, the OVER flag is set and the result is returned in the RESULT register. Additionally, HW tasks can be programmed to perform a DMA transfer or to generate interrupts.

395

Note that, since a PR accelerator interface structure is implemented in the IF, its register values are persistent for the VM. When an IF is disconnected from a PRR, the state of the corresponding virtual device (e.g. results, status) is still stored in this IF and can be read by the VM. In this way, the consistency of the virtual device interface is guaranteed.

Table 1 :

 1 Ports Description in the PR accelerator interface

	Register	Width	Description
	STAT	32-bit	HW task status register
	START	8-bit	Start flag
	OVER	8-bit	Computation Over flag
	CMD	32-bit	Command register
	DATA ADDR	32-bit	Data buffer address register
	DATA SIZE	32-bit	Data buffer size register
	RESULT	64-bit	Computation result register
	INT CTRL	32-bit	Interrupt controller register
	Custom Ports 8*32-bit Provide 8 IP-defined ports

  It can only be directly allocated to VMs when it is in Idle state and requires no reconfiguration. In other situations, the PRR changes to the Preempt state to stop the current running task and to the Reconfig. state when performing PCAP reconfiguration.

	Idle	Start	Busy	Stop	Preempt	Allocable
		Over				Non-allocable
					Reach
	Timeout				Consistency points
	Hold PRR Descriptor Start	Switch Over STATE	m Switch Allocate PRR PCAP Begin PCAP Over a n u s c r i p t Reconfig. RCFG_TIME VM_ID DEV_ID PRIO
	A c c e p t e d
	455
			19

450 Figure 7. As depicted in this figure, a PRR can be allocated in two states: Idle and Busy.

  Request(vm id, dev id, prio), which is composed of the VM ID, the device ID and a request priority. The device ID identifies the accelerator functionality. The request priority is equal to the priority of the calling VM. Note that, when a

	495
	running HW task is preempted, the interrupted task is automatically composed
	as a request, indicating the corresponding virtual device still has unfinished
	21

eration with the Virtual Device Manager to dynamically monitor reconfigurable accelerators and search proper solutions to the VMs' requests.

4.4.1. PR Resource Requests and Solutions

As described earlier, every time a VM tries to use an unavailable virtual 490 device, a page-fault exception is triggered and then handled by the Virtual Device Manager as a partially reconfigurable (PR) resource request:

Table

  Figure 9: Solution searching in the PPR monitorFigure 9 depicts the interaction between the PPR Monitor and the Virtual

	Virtual Device Manager	Reschedule	Micro-kernel
		Request	Valid Solution	IRQ (New Solution)
	HW Task index	Request		Unavailable
	PRR Descriptor Table	Search Solution
				Search List
	PR Regions			PRR Monitor

525

Device Manager. Normally the selected solution is sent to the Virtual Device Manager for further handling. However, if there is no valid solution (i.e. Unavailable), this unsolved request is added to the Search List, which is a waiting queue of all unsolved requests.

The PPR Monitor keeps searching solutions for requests in this queue on 530 the FPGA side, and acknowledges the Virtual Device Manager whenever a new solution is found. The searching runs in parallel with VMs, following a prioritybased FIFO principle, so that when a requests conflict occurs, the PPR Monitor always chooses the highest priority request.

4.4.2. Virtual Device Manager

535

Table 2 :

 2 Communication signals for PRR Allocations, among Virtual Device Manager (VDM),

	PRR Monitor (PM) and VM.
	Signal	Type Send Recv. Message
	IPC WAIT	IPC	VDM VM	Device is currently unavailable
	IPC READY	IPC	VDM VM	Device is ready
	IPC PREEMPT	IPC	VDM VM	Device is preempted
	IRQ New Solution IRQ	PM	VDM	New solution is found
	IRQ PCAP Over	IRQ	PM	VDM	Reconfiguration is over
	IRQ PRR Stop	IRQ	PM	VDM	Preemption is complete

are sent from the PRR Monitor to the Virtual Device Manager, to acknowledge the events for unfinished solutions. These signals are listed in Table

2

.

4.5. User Programming Model

A major purpose of our framework is to considerably simplify the coding aspects of software applications by making the access to devices as transparent 585 as possible. Ideally, the manipulation of virtual devices is only performed by read/write operations from/into the interface registers, without knowing the resource management at lower level. This is called the native programming model. In this case, the IPC WAIT and IPC READY signals are just ignored.

Alternatively, guest OS may follow the guest programming model, meaning 590 the OS is aware of the true state of virtual devices. For example, when a virtual device is unavailable, the guest OS may be programmed to stop the waiting task, run other tasks and only resumes the suspended task when the IPC READY signal is received.

610

PRRs can always be blocked by RTOS tasks, which will cause unpredictable blocking time. Therefore, using the guest programming model is more appropriate for GPOS if FPGA resources are intensely shared. Additionally, with

Table 3 :

 3 Thread-Metric benchmarks results for both native and virtual µC/OS-II

	Test Object	Native µC/OS-II VM µC/OS-II Performance Ratio (%)
	Calibration Test	764458	753879	98.6
	Preemptive Context Switching	32113328	28927171	90.1
	Message Processing	18431136	16748720	90.9
	Memory Allocation	104601611	85091278	81.3
	Synchronization Processing	108589466	90893213	83.7
	Interrupt Handling	32541832	25768399	79.2
	Interrupt Preemption	19089282	16425610	86.0

  The difference in terms of performance is due to the fact that Ker-ONE provides a simpler virtualization interface. All virtual resources may also note that XEN-ARM performs obviously worse compared to Ker-ONE.

		m a n u s c r i p t
	785 A c c e p t e d are implemented with smaller structures of smaller size. Additionally, Ker-ONE provides a dedicated physical timer pass-through for guest RTOS (see Section 3.3.1), which largely simplifies the timer virtualization compared to
		XEN system.
	790	Regarding the Interrupt Handling and Interrupt Preemption benchmarks, we
		34

Table 4 :

 4 Overhead measurement during DPR allocation

	m a n u s c r i p t
	900 A c c e p t e d
	of incoming bits into account and generate 16-bit width I and Q symbols. FFT
	blocks work on the outcoming QAM I and Q symbols to perform demodulation.
	39

Table 5 :

 5 Reconfiguration and preemption delays

	Virtual Device	δ pre (µs) 1	T RCF G (µs) PRR#1 PRR#2 PRR#3
	QAM16	47.0	231	810	-
	QAM64	31.0	231	810	-
	FFT512	24.1	-	810	1,206
	FFT1024	33.6	-	-	1,206

1

The worst-case waiting time when a running accelerator is forcibly stopped.

Table 5

 5 , a QAM-16 accelerator cannot preempt a FFT1024 since there is no resource competition between them, so T preempt(WCET) (QAM 16) is calculated as:T preempt(WCET) (QAM 16) = max δ QAM 16Meanwhile, since QAM-16 can only be implemented in PRR #1 and #2, the value of T RCF G(WCET) (QAM16) is determined as:T RCF G(WCET) (QAM 16) = max T P RR1RCF G , T P RR2 RCF G .

	m a n u s c r i p t
	A c c e p t e d
	950
	42

Table 6 :

 6 Comparisons between SW and HW implementation

	Algorithm	T HW (µs)	T SW (µs)	FPGA Resource Usage 1
		(per frame)	(per frame)
	QAM-16	47.0	1,513	2%
	QAM-64	31.0	1,174	2%
	FFT-512	71.1	6,582	8%
	FFT-1024	90.6	12,784	13%


Yuting He

Guanyu Yang

Yang Chen

Youyong Kong

Jiasong Wu

Lijun Tang

Xiaomei Zhu

Jean-Louis Dillenseger

Pengfei Shao

Shaobo Zhang

Huazhong Shu1

Jean-Louis Coatrieux

Shuo Li

DPA-DenseBiasNet: Semi-supervised 3D Fine Renal Artery Segmentation with Dense Biased Network and Deep Priori Anatomy

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

3D fine renal artery segmentation on abdominal CTA image targets on achieving 3D renal artery tree that reaches the end of interlobar arteries. If successful, clinicians will locate the blood feeding region corresponding to each interlobar artery easily which is important for the diagnosis and pre-operative planning of kidney disease [7,[START_REF] Shao | Laparoscopic partial nephrectomy with segmental renal artery clamping: technique and clinical outcomes[END_REF][START_REF] Shao | Precise segmental renal artery clamping under the guidance of dual-source computed tomography angiography during laparoscopic partial nephrectomy[END_REF]. For example, it will show the tumor-feeding artery branches for segmental renal arteries clamping before laparoscopic partial nephrectomy [START_REF] Shao | Laparoscopic partial nephrectomy with segmental renal artery clamping: technique and clinical outcomes[END_REF][START_REF] Shao | Precise segmental renal artery clamping under the guidance of dual-source computed tomography angiography during laparoscopic partial nephrectomy[END_REF]. With the increasing probability of kidney disease [9], 3D fine renal artery segmentation will play an important role in its diagnosis and treatment.

However, no one has reported success in 3D fine renal artery segmentation because it is a challenging task [3]: (1) Large intra-scale changes. The thickest renal artery of a patient can up to 7.4 mm, which can be more than 5 times of the thinnest artery as is shown in Fig. 1(a). This makes the network have to sensitive to different scale features, which increases the difficulty in feature extraction. (2) Large inter-anatomy variation. 11 different renal artery structures were found just from 461 patients [3]. The number of ostia, branch and accessory renal artery are variable between patients as Fig. 1(b) shows. This makes it difficult for a small dataset to cover all variation and causes the network to overfit easily.

(3) Thin structure. The thinnest artery is less than 1.5 mm which is much smaller than other organs in the same region so that the network is prone to lose these structures. (4) Small volume ratio. Renal arteries just account for 0.27% of the kidney region which will cause serious class imbalance problem, so the network is difficult to train. ( 5) Limitation of labeled data. It is difficult to learn the feature representation of different renal artery anatomies on a small data set, which limits the network's generalization ability. Therefore, how to overcome these challenges and to achieve 3D fine renal artery segmentation is an urgent problem.

There is no success to achieve 3D fine renal artery segmentation automatically, although some rough segmentation methods [START_REF] Li | Segmentation of renal structures for image-guided surgery[END_REF][START_REF] Taha | Kid-Net: Convolution networks for kidney vessels segmentation from CT-volumes[END_REF] that only segmented up to segmental arteries have been proposed. Li et al. [START_REF] Li | Segmentation of renal structures for image-guided surgery[END_REF] only segmented the main and the thick segmental renal arteries using 400 images. Taha et al. [START_REF] Li | Segmentation of renal structures for image-guided surgery[END_REF][START_REF] Taha | Kid-Net: Convolution networks for kidney vessels segmentation from CT-volumes[END_REF] used 99 cases to achieve the main arteries out of the renal. These methods cannot be applied to our task directly for two reasons: (1) The rough results. Main and segmental arteries usually correspond to multi blood feeding regions so that these rough results cannot be used to locate the specific blood feeding region of each vessel. (2) Large labeled dataset requirements. These methods used supervised learning which relies on a large labeled dataset and made it difficult to achieve satisfactory results when the 3D fine renal artery labeled dataset is small.

Semi-supervised learning gives us a tool to solve the limitation of labeled data because it uses unlabeled data to improve the model's performance [START_REF] Zhu | Semi-supervised learning literature survey[END_REF]. Nie et al. [START_REF] Nie | ASDNet: Attention based semi-supervised deep networks for medical image segmentation[END_REF] designed an attention-based network trained by 35 labeled and 20 unlabeled data to achieve the segmentation on a pelvic dataset. Bai et al. [START_REF] Bai | Semi-supervised learning for networkbased cardiac MR image segmentation[END_REF], applied CRF to a full convolutional network, and achieved the cardiac MR image segmentation with 240 unlabeled and 80 labeled images. However, these methods cannot be used for our tasks directly because they use unlabeled data to get inaccurate labels which lose thin structures easily so that the model is optimized in wrong targets and its performance is weak in our task.

We propose a novel semi-supervised framework (DPA-DenseBiasNet) for 3D fine renal artery segmentation in this paper. Our work has the following contributions: (1) To the best of our knowledge, this work is the first achievement in 3D fine renal artery segmentation. (2) It presents a dense biased connection method which merges multi-receptive field feature maps to adapt to large intra-scale changes. Further, each layer has direct access to the gradients from the loss function, leading to deep supervision [START_REF] Lee | Deeply-supervised nets[END_REF], to simplify the training process. Dense biased network (DenseBiasNet) is designed based on this method.

(3) Deep priori anatomy (DPA) is proposed for semi-supervised learning of thin structures. It embeds priori anatomical features from the encoder trained by unlabeled data to the DenseBiasNet to avoid inaccurate results sensitive to thin structures as optimizing target like other semi-supervised methods. Therefore, DenseBiasNet trained by a small labeled dataset has a higher generalization of different anatomies and keeps the segmentation ability of thin structures. The experimental results show that our framework has enormous potential for clinical application.

Methodology

As is illustrated in Fig. 2, DPA-DenseBiasNet adopts a dense biased network (DenseBiasNet) and deep priori anatomy (DPA) for 3D fine renal artery segmentation. DenseBiasNet (Sect.2.1) is a segmentation network which uses dense biased connection method for multi-receptive field feature maps merging and implicit deep supervision to achieve the segmentation with large scale changes and simplify the training process. DPA (Sect.2.2) is a semi-supervised method of thin structures which uses a trained encoder to provide priori anatomical features for DenseBiasNet to guide the adaptation of variable anatomical structures.

Dense Biased Network (DenseBiasNet) for Fine Segmentation

DenseBiasNet is a 3D fine renal artery segmentation network which based on dense biased connection method. It connects a part of feature maps in each layer to every other forward layer to build dense connectivity pattern. These feature maps are optimized by richer gradients and play a more important role in the network. Hence, we call the method dense biased connection.

Advantages of Dense Biased Connection: (1) It allows the network to adapt to large intra-scale changes via merging multi-receptive field feature maps which have different sensitivities to different scale vessels in each convolutional layer. (2) It simplifies the training process because the gradients from the loss function optimize each layer along the dense biased connection directly.

Dense Biased Connection: As is illustrated in Fig. 2(d), each layer gets a part of feature maps from all preceding layers as additional inputs and transmits a part of its output feature maps to all forward layers. If the sizes of feature maps do not match, maxpooling or upsample method will be used. We denoted the output of the l th layer as F l . The l th layer receives a part of feature maps of previous layers, F 0 , . . . , F l-2 , and all the feature maps of F l-1 as input:

F l = H l (F l-1 • F l-2 [0 : k l-2 ] • . . . • F 0 [0 : k 0 ]).
Where H l (•) can be a composite function of operations such as group normalization (GN), rectified linear units(ReLU), pooling, upsampling, or convolution (Conv). The symbol • refers to the concatenation of the feature maps. k 0 , . . . , k l-2 is the number of the feature maps the l th layer can receive from 0, ... , l -2 layers. In our experiments, we set k 0 , ..., k l-2 = 1.

Structure and Loss Function of DenseBiasNet: Figure 2(a) illustrates DenseBiasNet's structure. It comprises of 14 3D convolution layers which followed a GN and a ReLU, 3 maxpooling layers, 3 3D deconvolution layers used to change scales and a 1×1×1 convolution layer followed a softmax as the output layer to reduce the number of channels to classes. The dense biased connection is used throughout the network to adapts to different scale arteries.

The network is trained by minimizing the loss function consisting of dice coefficient loss and cross-entropy loss. The dice coefficient loss L dice helps to establish a balance between artery and background from a global perspective, and the cross-entropy loss L ce is used for correct classification of each voxel at a local perspective:

L seg = λ 1 - 1 C C c 2 N n y n,c ŷn,c N n y 2 n,c + N n ŷ2 n,c L dice - 1 N C c N n y n,c log ŷn,c Lce ( 1
)
where C is the number of channels output from the network, and N is the size of each channel. ŷn,c is the predicted result and y n,c indicates the label. λ is used to balance these loss functions. In our experiments, we set λ = 0.1.

Deep Priori Anatomy (DPA) Based Semi-supervised Learning

DPA is a novel semi-supervised method that avoids inaccurate optimizing targets which sensitive to thin structures during training. It first trains an autoencoder [START_REF] Vincent | Extracting and composing robust features with denoising autoencoders[END_REF] with numerous unlabeled data, and then uses the encoder part to extract input image's priori anatomical features of different semantic levels in different depth in order to guide the anatomical adaptation of the segmentation network which trained with a small labeled dataset as shown in Fig. 2

(b)(c).

Advantages of DPA: (1) Deep priori anatomical feature representation learned from unlabeled data which adapts to more anatomical structures than manual priori features. (2) It focuses on both local and global anatomical information because different semantic levels features are extracted from different depths. (3) It improves the network's generalization ability utilizing these priori features. (4) It is suitable for thin structures compared with other semisupervised methods [START_REF] Bai | Semi-supervised learning for networkbased cardiac MR image segmentation[END_REF][START_REF] Nie | ASDNet: Attention based semi-supervised deep networks for medical image segmentation[END_REF] because embedding priori knowledge will not introduce inaccurate labels that are easy to lose thin structures.

DPA for Semi-supervised Anatomical Adaptation: Figure 2 shows the process of extracting and embedding deep prior anatomical features into the DenseBiasNet. Prior to this, a convolutional denoising autoencoder was trained with numerous unlabeled data, and then its encoder part was frozen and used to extract anatomical features (Fig. 2(b)). The image x is putted into the encoder to obtain priori anatomical features at different depths {F f 1 , F f 2 , F f 3 , F f 4 } = f (x; θ f ). These feature maps (Fig. 2(c)) together with input image x are putted into the DenseBiasNet's different convolution layers to predict fine renal artery

segmentation ŷ = d(x, F f 1 , F f 2 , F f 3 , F f 4 ; θ d ).
The learning process is minimizing the loss function L seg (y, ŷ) to optimize the DenseBiasNet. y is the renal artery mask, f ( • ) is the encoder operation and d( • ) is the DenseBiasNet operation.

Experiments and Results

Setting: Abdominal contrast-enhanced CT images of 170 patients who underwent LPN surgery were included in this study. The pixel size of these CT images is between 0.59 mm 2 to 0.74 mm 2 . The slice thickness and the spacing in zdirection were fixed at 0.75 mm and 0.5 mm respectively. 52 of these images have renal artery mask, half of them were used as the training sets and the other half as the test set. The remaining 118 unlabeled images are used to train the denoising autoencoder. On the training set, we mirrored each image on three axes to expand the training data to 104 images. The kidney region of interest which size was 152 × 152 × Z was extracted firstly. The denoising autoencoder and DenseBiasNet are all trained with Adam where the batch size was 1, the learning rate was 1 × 10 -4 and the decay rate was 1 × 10 -5. They all trained 200 epochs on their corresponding data sets.

To demonstrate the advantage of our framework, we compared our method with 2 supervised methods (3D U-Net and VNet) and 2 semi-supervised methods (SemiFCN and ASDNet). We adopt the following evaluation metrics to evaluate our proposed method: mean dice coefficient (Dice), mean centerline distance (MCD) and mean surface distance (MSD).

Visual Superiority: Figure 3 shows the visual superiority of our framework. Case 1 is a left kidney region whose artery has many singular anatomical structures which difficult to segment. Case 2 is a right kidney region whose artery has great scale changes at the branches. Compared with ground truth, DPADense-BiasNet achieves fine segmentation in these cases thanks to the dense biased connection which merges multi-receptive field feature maps and DPA which ensures the segmentation quality of thin structures and generalization of different anatomies. Without the help of DPA, DenseBiasNet loses some parts in case1. SemiFCN cannot realize our task, because CRF removes thin renal arteries so that the inaccurate optimize target weakens the network's performance. ASDNet has serious mis-segmentations due to the more serious class imbalance caused by the confidence map and the instability caused by adversarial learning.

Evaluation Metrics Advantages: The advantages of our DPA-DenseBiasNet on each metric are demonstrated in Table 1. DPA-DenseBiasNet achieves the best segmentation results compared with other methods. The dice coefficient, mean centerline distance and mean surface distance are 0.861, 1.976 and 1.472, and their corresponding standard deviations are 0.095, 1.394 and 1.738. Ablation experiments in the last two rows validate the importance of DPA which improves the segmentation accuracy obviously.

Training Process Improvement: Figure 4 illustrates the improvement of the training process due to dense biased connection. We compared DenseBias-Net, 3D U-Net (3 layers have skip connections) and the network without skip connections. DenseBiasNet has faster convergence speed and higher test accuracy because the gradients from the loss function optimize each convolution layer directly.

Conclusion

This paper proposed a novel semi-supervised framework which achieved 3D fine renal artery segmentation. The proposed framework used dense biased connection method to enable DenseBiasNet to adapt to large intra-scale changes and simplify its training process. Further, our developed semi-supervised method of thin structures, DAP, embedded priori anatomical features from an encoder network to DenseBiasNet to improve its generalization of different anatomies. The performance of our framework was compared with other methods. The results showed that our framework had great prospects in the diagnosis and treatment of kidney disease.

Fig. 1 .

 1 Fig. 1. The challenges of 3D fine renal artery segmentation. (a) The thickest artery of a patient is up to 7.4 mm which can be more than 5 times of the thinnest artery. (b) The number of ostia, branch and accessory renal artery are variable between patients. (c) Renal arteries account for 0.27% of the kidney region and are much thinner than other organs.

Fig. 2 .

 2 Fig. 2. The framework of DPA-DenseBiasNet: (a) and (d) are DenseBiasNet and dense biased connection method which introduced in Sect.2.1. (b) is the encoder network from denoising autoencoder trained by numerous unlabeled data and (c) is the priori anatomical features from the encoder. (b) and (c) constitutes DPA which introduced in Sect.2.2 .

Fig. 3 .

 3 Fig. 3. The visual superiority of our framework (DPA-DenseBiasNet).

Fig. 4 .

 4 Fig. 4. The improvement of the training process by the dense biased connection. Dense-BiasNet has faster convergence speed and higher accuracy than the other two networks.

Table 1 .

 1 The advantages of our method (DPA-DenseBiasNet) on each metrics. Net [4] 0.750 (0.162) 5.070 (4.949) 4.385 (4.208) (semi)SemiFCN [2] 0.388 (0.259) 8.772 (10.085) 7.921 (10.593) (semi)ASDNet [10] 0.555 (0.191) 8.557 (5.124) 7.484 (5.132)

	Network	Dice	MCD	MSD
	V-Net [8]	0.787 (0.113) 2.872 (2.196) 2.213 (2.155)
	3D U-

DenseBiasNet 0.851 (0.110) 2.478 (2.090) 1.920 (2.354) (semi)Proposed 0.861 (0.095) 1.976 (1.394) 1.472 (1.738)

Acknowledgements. This research was supported by the National Key Research and Development Program of China (2017YFC0107903), National Natural Science Foundation under grants (31571001, 61828101), the Short-Term Recruitment Program of Foreign Experts (WQ20163200398), Key Research and Development Project of Jiangsu Province (BE2018749) and Southeast University-Nanjing Medical University Cooperative Research Project (2242019K3DN08).