Fernando Fernandes

Angeliki Santos

Olivier Kritikakou

Paolo Sentieys

Rech

Characterizing Deep Neural Networks Neutrons-Induced Error Model

We characterize the fault models for Deep Neural Networks (DNNs) in GPUs exposed to neutron. We observe tolerable and critical errors, and show that ECC is not effective in reducing critical errors.

I. INTRODUCTION

Deep neural Networks (DNNs) are an efficient tool to accelerate several applications, such as object detection, image segmentation, classification, and prediction [START_REF] He | Identity mappings in deep residual networks[END_REF]- [START_REF] Gkioxari | Mesh R-CNN[END_REF]. As a result, DNNs have been deployed in safety-critical and mission-critical domains, such as robotics, aeronautics and space exploration, smart healthcare, and autonomous driving. To reduce DNNs cost in terms of processing time and energy consumption, dedicated architectures with specialized hardware or reduced precision, e.g., half-precision floating-point (16 bits) or even short integer (8 bits), have been proposed, with satisfactory classification, segmentation, and detection accuracy. GPUs even feature Tensor Cores, i.e., a dedicated hardware to perform 4x4 matrix multiplications in one cycle [START_REF] Nvidia | NVIDIA Tesla V100 GPU architecture[END_REF] while, for more specific domains, ASIC accelerators are employed for DNN, achieving significant performance and low power consumption [START_REF] Jouppi | In-datacenter performance analysis of a tensor processing unit[END_REF].

When DNNs are deployed in safety-critical domains, realtime execution and reliability need to be paramount. Thus, to leverage the DNNs benefits towards safety-critical systems, a DNN-based system must be compliant with standards, such as ISO26262 and ISO/PAS 21448. GPU reliability can be undermined by several sources, including environmental perturbations, software errors, and process/temperature/voltage variations [START_REF] Laprie | Dependable computing and fault tolerance : Concepts and terminology[END_REF], [START_REF] Nicolaidis | Time redundancy based soft-error tolerance to rescue nanometer technologies[END_REF]. Radiation-induced soft errors are particularly critical, as they have been found to dominate error rates in commercial devices [START_REF] Baumann | Soft errors in advanced computer systems[END_REF]. DNNs are susceptible to transient fault induced by radiation [START_REF] Santos | Analyzing and increasing the reliability of convolutional neural networks on GPUs[END_REF]- [START_REF] Rech | High energy and thermal neutrons sensitivity of google tensor processing units[END_REF]. Particularly, GPUs have a high fault rate because of the high amount of available resources [START_REF] Debardeleben | GPU behavior on a large HPC cluster[END_REF]- [START_REF] Sullivan | Characterizing and mitigating soft errors in gpu dram[END_REF] and the possibility to have multiple output elements corrupted, undermining DNN reliability [START_REF] Santos | Analyzing and increasing the reliability of convolutional neural networks on GPUs[END_REF], [START_REF] Ibrahim | Soft error resilience of deep residual networks for object recognition[END_REF].

The main goal of this work is to obtain a realistic DNNs' transient error model extracted from neutron beam experi- ments and perform a realistic GPU reliability evaluation. We report findings from beam testing campaigns that assess GPU reliability as a DNN accelerator and identify the causes of critical errors (i.e., errors that impact detection) and tolerable errors. The testing campaigns consider two different GPU architectures, i.e., Kepler (Tesla K40) and Volta (Tesla V100), and several configuration. Specifically, (1) we evaluate the reliability of the three DNNs, (2) we measure the efficacy of the ECC on protecting the DNNs against errors that impact the classification/detection, (3) we characterize the criticality of the errors that can modify the GEMM output when used as the core of the DNNs operations, (4) we consider different floating-point data and operation precisions. The obtained results show that a single particle can spread through the GPU microarchitecture, affecting several parallel threads and output elements, significantly impacting DNN reliability. For DNNs, Single Error Correction Double Error Detection (SECDED) ECC can be useful but not always effective. Although several errors are masked by ECC and thus, do not propagate through the GPU, ECC can reduce neither the number of critical nor multiple errors.

II. EVALUATION METHODOLOGY

This section describes the characterized GPU devices and DNNs, the metrics adopted for DNN reliability evaluation, and how the beam experiments are performed. Devices: We consider Kepler (Tesla K40) and Volta (Tesla V100) NVIDIA GPUs. The tested NVIDIA K40 (Kepler) is built with the Kepler ISA and fabricated in a 28nm TSMC standard CMOS technology [17]. Tesla V100 (Volta) are designed with the Volta micro-architecture and built with TSMC FinFET 12nm [START_REF] Nvidia | NVIDIA Tesla V100 GPU architecture[END_REF]. Volta GPUs feature hardware acceleration for three IEEE754 float point precisions: double (FP64), float (FP32), and half (FP16). We also evaluate the reliability of Volta GPUs' tensor cores, i.e., specific hardware that performs 4x4 Matrix Multiplication, for GEMM kernels with FP16 precision. Tensor cores can also perform matrix multiplication for FP32 precision, however, the data will be cast to FP16 in the low-level operation. Both GPUs architectures have available Single Error Correction Double Error Detection (SECDED) Error Correcting Code (ECC) to protect the register file, shared memory, and caches. Our evaluation considers errors occurring only in the GPU core, not in the main memory. For both devices, we chose a beam spot sufficiently small (2cm of diameter) not to hit the onboard DDR when ECC is disabled. DNNs: This work considers three modern frameworks: i.) You Only Look Once (YOLO v1 and v3) [START_REF] Redmon | You only look once: Unified, real-time object detection[END_REF], [START_REF] Redmon | YOLOv3: An incremental improvement[END_REF], ii.) 3) A Detected Unrecoverable Error (DUE) (i.e., a program crash or device reboot). Not all errors in the output of the DNNs will impact the classification or object detection. We classify the observed SDCs between Tolerable SDCs (i.e., that do not impact classification/detection) and Critical SDCs (i.e., that impact classification/detection). When radiation modifies the classification or detection, we check if all the objects are detected and classified correctly. That is, we measure if the error created a false positive (i.e., add unexisting objects), or the error makes the DNN miss detect objects. Beam Experiment Setup: Our experiments were performed at the ChipIR facility of the Rutherford Appleton Laboratory, UK, and at the LANSCE facility at Los Alamos National Laboratory, US. Figure 1 shows the setup mounted in the ChipIR facility. Both facilities deliver a beam of neutrons with a spectrum of energies similar to the atmospheric neutron one [START_REF] Cazzaniga | Progress of the scientific commissioning of a fast neutron beamline for chip irradiation[END_REF]. The available neutron flux was about 3.5 × 10 6 n/(cm 2 /s), ∼8 orders of magnitude higher than the terrestrial flux (13 neutrons/(cm 2 • h) at sea level [START_REF] Slayman | JEDEC Standards on Measurement and Reporting of Alpha Particle and Terrestrial Cosmic Ray Induced Soft Errors[END_REF]). The Failure In Time (FIT) rate is calculated by dividing the number of observed errors by the received particles fluence (neutrons/cm 2 ).

Since the terrestrial neutron flux is low, it is improbable to see more than a single corruption during program execution in a realistic application. We have carefully designed the experiments to maintain this property (observed error rates were lower than 1 error per 1,000 executions). Each DNN code was tested for at least 24 effective hours, not including the setup, result check, initialization, and recovery from the DUE time.

We added setup software and hardware watchdogs to monitor the experiments. The software watchdog controls the application under test, and if it stops responding in a predefined time interval, the kernel is killed and relaunched. This watchdog detects kernel crashes or software hangs, i.e., application crashes or control flow errors that prevent the GPU from completing assigned tasks (e.g., an infinite loop). The hardware watchdog is an Ethernet-controlled switch that performs a host computer's power cycle if the host computer itself does not acknowledge any ping requests in a predefined time interval. The hardware watchdog is necessary to detect when the operating system hangs.

III. DNNS RELIABILITY AND ERROR MODEL

This section presents the results from radiation experiments for the evaluated Deep Neural Networks (DNNs) with ECC ON and OFF. Note that YOLOv3 could not be tested on ECC OFF due to beam time limitations.

A. Deep Neural Networks FIT rate

Figure 2a shows the normalized SDC and DUE FIT rates of YOLOv1, Faster R-CNN, and ResNet on Kepler, and YOLOv3 on Volta GPUs. The reported results are normalized to not reveal business-sensitive information. Even if the data is normalized, it allows a direct comparison among configurations. The reported values are relative to YOLOv1 ECC ON SDC FIT for Kepler and YOLOv3 ECC ON SDC FIT for Volta. Experimental data is presented with a 95% confidence interval. DUE: DUEs are more probable than SDCs for all tested configurations. DNN kernels have a high level of reuse that requires several device-host (CPU-GPU) synchronizations. A transient fault during those synchronizations could potentially result in a GPU DUE. As a result, while a significant portion of SDCs could be masked, DUEs could still undermine the device's reliability. For the same reason, Faster R-CNN and Resnet, which require a much larger number of synchronizations, show up to 5× higher DUE rate than YOLO. ECC ON DUE rate increases up to 30% for Faster R-CNN, Resnet, and YOLOv1. ECC is able to correct one-bit flip in the memories, and when a double-bit flip is detected, it throws a system exception. As DNNs use a large memory to perform classification/detection, multiple errors on memories are expected to happen, resulting in a higher DUE rate when ECC is ON. Equivalently, on Volta GPU, the DUE rate grows as the data representation increases. Comparing FP16 and FP32 versions of YOLOv3, the size of memory grows 2x. As the FP32 version of YOLOv3 performs more memory transfer than the FP16 version, the DUE is expected to be higher for FP32. Additionally, with the ECC ON on Volta, and when caches and register usage increase to store the FP32 precision, the double bit flips in the memories will be more frequent. SDC: Figure 2a shows that the SDC rates are related to framework complexity and accuracy. Compared to YOLO, the complex structures used on Faster R-CNN and Resnet increase their SDC rate by more than 10x. Figure 2a shows that Resnet has a higher FIT rate than YOLO (i.e., v1 and v3), though the rate is similar when compared to Faster R-CNN. Although Faster R-CNN and ResNet have a high detection/classification accuracy, it is insufficient to compensate for the higher error rate associated with a complex framework used for both DNNs.

The SDC rate for YOLOv1, Faster R-CNN, and Resnet on the ECC ON is 21%, 13.6%, and 22% the SDC rate seen for ECC OFF, respectively. ECC is not as effective on these workloads as other codes, mainly because neural networks are intrinsically resilient to data errors. ECC can reduce the GEMM SDC rate by about one order of magnitude [START_REF] Gonc ¸alves De Oliveira | Evaluation and Mitigation of Radiation-Induced Soft Errors in Graphics Processing Units[END_REF] (details Section III-B). However, ECC is less efficient in protecting a DNN, as some of the SDCs, that ECC masks would not have affected the DNN execution. On Volta GPU, YOLOv3 FP16 has a lower FIT than YOLOv3 FP32. The amount of per-core resources required to perform the operations depends on the chosen data precision. Namely, fewer resources will be used for the lower float precisions.

Figure 2b shows the percentage of the Tolerable SDCs vs. the Critical SDCs for all configurations tested. For object detection, the percentage of critical SDCs is much lower for Faster R-CNN and YOLOv3 than for YOLOv1. For YOLOv1, the percentage of critical SDCs is 8% for ECC OFF and 61% for the Kepler with ECC ON. For YOLOv3 with ECC ON, the percentage of critical SDCs is 21% and 27% for FP16 and FP32, respectively. For Faster R-CNN, the critical SDCs are 5% for ECC OFF and 25% for the Kepler with ECC ON. The differences in the critical errors percentages between the DNNs come from the detection mechanism. YOLOv1 is the simplest DNN tested in our setup (only 31 layers). An error is expected to impact much more the detection of simpler DNNs than the more accurate ones.

ECC can reduce the error rate, but the proportion of critical errors is not reduced, which is a symptom of the poor resiliency provided by ECC. Based on the GPU-Qin analysis in [START_REF] Li | Understanding error propagation in gpgpu applications[END_REF], it is known that ECC does not mask all the faults as the error in computing elements could propagate to the output. ECC reduces the absolute number of SDCs but has the side effect of increasing the portion of multiple errors (Details Section III-B), which are more likely to propagate through DNN layers and affect detection. On Volta, the percentages of critical errors are similar comparing the two data types when ECC is ON. It is worth noting that critical SDCs are less dependent on data type as the two DNNs have similar detection accuracy regarding the data precision.

B. Complex error models for DNNs

To completely characterize the critical errors on DNNs and how they can be generated at the low level, we also evaluated how the errors affect the General Matrix Multiplication algorithm (GEMM) used as the core of DNNs execution on modern GPUs. In fact, 70% to 86% of the operations for the evaluated DNNs are GEMM related operations.

Figure 3a shows the SDC and DUE normalized FIT rates for GEMM. Kepler values are relative to the SDC rate with ECC enabled, while Volta values are relative to the SDC rate of FP16 without Tensor cores. We selected matrix sizes that saturate the resources of each device (2048 × 2048 for the Kepler and 16384 × 16384 for Tesla V100). As device resources are saturated, data in Figure 3a can be used to compare the GEMM reliability of GEMM across architectures. Smaller matrices will lower the FIT rate given the number of unused resources.

From Figure 3a, it is clear that GEMM reliability depends on the device and the precision. Following previous studies [START_REF] Gonc ¸alves De Oliveira | Evaluation and Mitigation of Radiation-Induced Soft Errors in Graphics Processing Units[END_REF], the DUE rate is lower than the SDC rate for ECC OFF on Kepler. ECC triggers an application DUE when there is a double-bit error. When ECC is OFF, the double bit flips may lead to an SDC at the output of GEMM.

From Figure 3a we can notice that, independently of precision, the use of tensor cores increases DUE FIT. The GEMM with tensor cores is optimized to use the tensor cores to the maximum performance by performing warp level synchronizations. Our data suggest that the corruption of internal resources, necessary to perform specific warp level synchronizations of the tensor core, is particularly prone to generate a DUE. When used to improve the performance of the DNNs, tensor cores may also increase the DNNs' DUE rate even more. Figure 3a shows that tensor cores GEMM has a lower FIT rate when executed in FP16 precision for Volta GPU. This attests that the tensor core circuit is slightly more reliable than the combination of ADD, MUL, and the loop control variables needed to implement matrix multiplication in software. When GEMM is executed in FP32 precision, the tensor core SDC FIT rate increases significantly, being 20% higher than the pure GEMM one. In fact, as stated in Section II, the tensor cores on Volta architecture execute physical operations only in FP16 precision. FP32 precision inputs require a hardware casting to FP16 precision, increasing the GPU's occupation and, thus, the SDCs (and DUEs) FIT rate. When executed on Volta without the tensor cores, the GEMM FIT rate increases as the precision increases. The increase from FP16 to FP32 is about 2 × . As FP32 occupies 2 × more GPU resources, it is expected that the FIT rate will also have a factor near to 2 × .

We also investigate the output of the GEMM to classify the errors based on their coordinates. That is, Figure 3b shows the percentage of corrupted executions that the output affected Single, Square (four or more elements distributed in a rectangle), Line (i.e., multiple corrupted elements on the same row/column of the output matrix), and Randomly distributed errors. In most cases, GPU corruption affects more than a single output element. It is worth noting that multiple corrupted elements are not caused by multiple impinging particles. Instead, the impact of a single particle is spread during computation to across multiple output elements.

Square errors are potentially the most severe for DNNs. Square errors in GEMM are caused by faults that impact the scheduling or the execution of multiple threads in a Streaming Multiprocessor (SM). GEMM is a highly optimized version of matrix multiplication since it can divide the input matrices into chunks that fit nicely in the cache of a SM, reducing memory latency. If a fault can cause a thread to be incorrectly assigned or scheduled to a SM, or if some threads fail to synchronize, the whole SM output matrix portion is likely to be corrupted, leading to a Square error. Errors in memory elements protected by ECC (e.g., registers and caches) have been reported to manifest into either single corrupted element or line errors [START_REF] Rech | Threads distribution effects on graphics processing units neutron sensitivity[END_REF]. When ECC is turned ON, single and line errors (which are less critical for CNNs) are corrected, but all the other errors (including rectangular errors) are not corrected. As a result, the percentage of rectangular errors increases when ECC is enabled.

Figure 3b shows that the geometry of the output errors changes slightly when comparing Kepler and Volta. For FP32 GEMM on both GPUs with ECC ON, the distribution of the errors are mainly of Square and Line errors, showing that the criticality of the GEMM is directly related to how the error is propagated from the low-level fault to the algorithm output.

It is worth noting that the criticality of the errors increases for the GEMM without the tensor compared to the GEMM optimized with the tensor core. The matrix multiplication performed in hardware generated more Single errors than the version only performed in software. This result is following past results [START_REF] Rech | High energy and thermal neutrons sensitivity of google tensor processing units[END_REF] where the reliability of DNNs hardware accelerators have been shown to have a less critical error model than the operations performed only on software.

IV. CONCLUSIONS

We have discussed the reliability of DNNs on beam experiments and evaluated the criticality of the errors in the output. Although the ECC is a powerful technique to reduce the SDC rate, it cannot reduce the Critical errors proportionally on DNNs. We then characterized the GEMM kernels as the core of DNNs, to investigate the error models that can impact the DNNs reliability.

As radiation experiments demonstrated, the single bit flip in the memories is not the leading cause of critical errors. Additionally, the criticality of the output of the GEMM indicates that a single error in a single element of a GEMM kernel is not realistic as an error model to simulate critical errors on DNNs. For the final paper, we will perform an assembly-level fault injection in a range of DNNs with different error models to investigate the leading cause of the critical errors.

Fig. 1 :

 1 Fig. 1: Beam experiments setup

Fig. 2 :

 2 Fig.2: Figure2ashows the normalized Failure In Time for YOLOv1, Faster R-CNN, ResNet, and YOLOv3. Figure2bshows the distribution of the Tolerable vs Critical SDCs for the tested DNNs.

Fig. 3 :

 3 Fig.3: Figure3ashows normalized Failure In Time for the General Matrix Multiplication (GEMM) on Kepler and Volta. Figure3bGEMM Error Geometry.

  Fernando Fernandes dos Santos is with Universidade Federal do Rio Grande do Sul (UFRGS), Porto Alegre, Brazil. e-mail: ffsantos@inf.ufrgs.br, and with Univ Rennes, INRIA, Rennes, France, e-mail: fernando.fernandes-dossantos@inria.fr. Angeliki Kritikakou and Olivier Sentieys are with Univ Rennes, INRIA, Rennes, France. Paolo Rech is with the Department of Industrial Engineering of the University of Trento, Italy.

This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie (MSCA) grant agreement No 886202 and No 899546, and from The Coordenac ¸ão de Aperfeic ¸oamento de Pessoal de Nível Superior -Brazil (CAPES) -Finance Code 001. Neutron beam time was provided by ChipIR (DOI: 10.5286/ISIS.E.RB2000161).