F Bruguier

T Delobelle

C Effiong

A Gamatié

P.-Y Péneau

G Sassatelli

S Senni

L Torres

S A S Cortus

Michael Chapman
email: michael.chapman@cortus.com

Erven Rohou
email: erven.rohou@inria.fr

Evaluation of selected memory and communication technologies and exploitation opportunities in compilation and runtime management

Keywords:

This deliverable presents a number of experiments where non-volatile memory and 3D interconnect technologies are evaluated. The performance and energy consumption are mainly considered as metrics for the evaluation. The goal is to get insights on their impact within the design of typical heterogeneous multicore architectures as expected in the compute node design studied within the CONTINUUM project. From the resulting observations, we discuss possible opportunities enabling us to better leverage the advantages of such technologies by using compilation techniques and runtime system management approaches.

From the surveyed emerging memory and communication technologies presented in deliverable D3.1 [START_REF]Survey on emerging memory and communication technologies[END_REF], we carry out an evaluation of selected technologies in the present report. The main objective is to assess the impact of their presence in typical heterogeneous multicore system designs on performance and energy consumption. From this assessment, we identify possible opportunities for better leveraging the advantages of such technologies via compilation techniques and runtime system management.

The evaluations presented in the sequel are conducted at a cycle-approximate/accurate level by combining popular tools, such as gem5 [START_REF] Binkert | The gem5 simulator[END_REF], McPAT [START_REF] Li | An integrated power, area, and timing modeling framework for multicore and manycore architectures[END_REF] and NVSim [START_REF] Dong | A circuit-level performance, energy, and area model for emerging nonvolatile memory[END_REF]. In addition, we adopt a cycle-accurate dedicated to NoC simulation [START_REF] Effiong | On the performance exploration of 3d nocs with resistive-open tsvs[END_REF]. Choosing adequate simulation supports is central for the current study. Though faster, analytical simulation [START_REF] Caliri | Introduction to analytical modeling[END_REF][START_REF] An | CLASSY: a clock analysis system for rapid prototyping of embedded applications on mpsocs[END_REF][START_REF] An | High-level design space exploration for adaptive applications on multiprocessor systems-on-chip[END_REF] and transaction-level modeling [START_REF] Schirner | Quantitative analysis of the speed/accuracy trade-off in transaction level modeling[END_REF][START_REF] Latif | An Integrated Framework for Model-Based Design and Analysis of Automotive Multi-Core Systems[END_REF][START_REF] Latif | Design space exploration for complex automotive applications: an engine control system case study[END_REF] are not accurate enough for providing us with detaile insights.

The rest of the document is organized around three main sections as follows:

• Section 3 first focuses on the evaluation of performance and power consumption when integrating the STT-RAM memory technology in the cache memory hierarchy. Among the candidate non-volatile memory technologies, STT-RAM currently shows the most promising features (reasonable read/write latency and energy consumption) w.r.t. mainstream volatile technologies such as SRAM. The Parsec benchmark suite is used for evaluating various system configurations. It includes applications and kernels covering both compute-intensive and memory-intensive algorithms. The MAGPIE [START_REF] Delobelle | System-level Evaluation of Manycore Systems with Emerging Memory Technologies[END_REF][START_REF] Delobelle | Flot automatique d'évaluation pour l'exploration d'architectures à base de mémoires non volatiles[END_REF][START_REF] Senni | Embedded systems to high performance computing using STT-MRAM[END_REF] design framework is used for evaluation. It relies on a transformation flow that leverages some of the aforementioned tools.

• Section 4 is devoted to some investigations on application mapping in three-dimensional (3D) NoC-based multicore systems. A cycle-accurate model is used for performance assessment. Through Silicon Via (TSV) provides shorter vertical interconnect which provides higher bandwidth and enhances performance in 3D integration. However, this integration suffers from process variation induced during the manufacturing. One of such defect is the open-resistive defect caused by impurities and/or defect during manufacturing process. It leads to significant signal propagation delay on the TSV links. The presence of a high number of such defective TSVs significantly degrades performance when data travel across TSVs. In this study, we investigated the impact of process variation on a 3D-NoC based on different architectural parameters and application mapping heuristics. This is carried out with the aim of exploring the system performance under process variation and to determine optimal architectural parameters that can mitigate the effects of such process variation.

• Section 5 discusses some opportunities regarding the exploitation of studied memory and communication technologies, by using techniques in compilation and runtime management. Some of these techniques are already under investigation in the project.

• Finally, Section 6 gives concluding remarks.

Page 2 Version 2.0 (2017) Confidentiality: Public Distribution

Evaluation of NVM in Cache Memory Hierarchy

We first introduce the MAGPIE (Manycore Architecture enerGy and Performance evaluatIon Environment) evaluation framework [START_REF] Delobelle | System-level Evaluation of Manycore Systems with Emerging Memory Technologies[END_REF][START_REF] Delobelle | Flot automatique d'évaluation pour l'exploration d'architectures à base de mémoires non volatiles[END_REF][START_REF] Senni | Embedded systems to high performance computing using STT-MRAM[END_REF] that we devised in order to facilitate system design evaluation. MAGPIE makes it possible to carry out the impact assessment of typical non-volatile memory technologies in the cache memory hierarchy, on performance and energy consumption within a heterogeneous multicore architecture design.

MAGPIE framework

We give an overview of the MAGPIE design evaluation flow. We describe each involved step and its implementation through the integration of existing tools. MAGPIE 1 framework relies on a generic evaluation flow depicted in Fig. 1. The inputs of the flow comprise information related to the software and hardware parts of the system. The software-related inputs include a gem5 execution script file for each workload/application to be executed, together with 1 This framework has been developed in collaboration with the GREAT European H2020 project. It is freely distributed via the following address: http://www.lirmm.fr/continuum-project/pages/magpie.html Version 2.0 (2017) Confidentiality: Public Distribution the underlying operating system supported by the considered full-system simulator. From the hardware perspective, a number of parameters of the target manycore architectures are required: types and number of cores, memory hierarchy and its technology-specific properties, and the interconnect type. In general, the candidate full-system simulators for MAGPIE provide an IP library that significantly facilitates the instantiation of target architectures.

Evaluation workflow

Provided the above input information, MAGPIE proceeds through four main steps as follows:

1. Platform components calibration: the basic parameters of all hardware components are set up in the full-system simulator. Typically, the operating frequency of cores, the memory size and access latencies at different levels of the memory hierarchy are defined. For NVMs, their corresponding read/write latencies usually vary according to their characteristics such as their type [START_REF] Mittal | A survey of architectural approaches for managing embedded dram and non-volatile on-chip caches[END_REF] (e.g., ReRAM, PCRAM, MRAM), technology node (e.g., 65nm, 45nm) and size.

2. Manycore system execution: the full system simulation of the user-customized system is performed in order to obtain detailed execution statistics. Note that since the inputs of the MAGPIE flow can specify at the same time different system design choices, several simulation instances can be launched in parallel. This contributes to accelerate the design space exploration with MAGPIE. Beyond the global performance metrics such as execution time, the activity events related to each hardware device are important information. These events are used for power and energy estimation.

3. Surface and energy estimation: based on detailed execution statistics generated by gem5, the surface, power and energy consumption of the simulated system are estimated. For instance, the dynamic energy of a cache memory is determined based on its collected read/write activity events and the energy consumption of elementary memory access obtained, e.g., with CACTI.

4. Post-processing for graphical renderings: as a major goal of MAGPIE is to assist the user in design space exploration, the final evaluation metrics can be reported in both textual and graphical formats. Then the user can quickly gather insights from each evaluated design.

The next section presents an implementation of the above flow within MAGPIE.

Implementation

We seamlessly combine the gem5 simulator for performance evaluation, and NVSim and McPAT for estimating the energy respectively related to NVMs and the rest of the architecture. These tools are briefly described below.

Considered simulation and estimation tools. The gem5 [START_REF] Binkert | The gem5 simulator[END_REF] provides an accurate evaluation of system performance [START_REF] Butko | Accuracy evaluation of gem5 simulator system[END_REF] thanks to its high configurability for a fine grained architecture modeling. Its full-system simulation mode runs unmodified operating systems. It includes several predefined architecture component models, e.g., CPU, memory and interconnect. This simulator produces detailed execution statistics (even at micro-architecture level) for power and footprint area estimation. McPAT [START_REF] Li | An integrated power, area, and timing modeling framework for multicore and manycore architectures[END_REF] is a power, area and timing modeling framework for multi-threaded, multicore, and manycore architectures. It works with a variety of performance and thermal simulators via an XML template-based interface. This interface describes the micro-architecture specification and is used to communicate activity events generated by simulators. McPAT covers three simulation levels for estimation: architectural, circuit and technology (from 90nm to 22nm). NVM technologies are not addressed by McPAT. So, we use NVSim [START_REF] Dong | A circuit-level performance, energy, and area model for emerging nonvolatile memory[END_REF], which is a circuitlevel estimator for NVM performance, energy and area estimations. It supports different NVM technologies including STT-MRAM, ReRAM, and PCRAM. It uses the same modeling principles as CACTI.

Integration within MAGPIE framework. The MAGPIE framework defines several Python script programs that automates the whole flow illustrated in Fig. 1. The inputs of the flow are first read and used for an automatic calibration of the hardware architecture components in gem5. For NVMs, the NVSim tool is invoked by a script in order to calculate the corresponding read/write latencies based on the desired memory type, memory size, associativity and technology node, as specified in the inputs. Then, gem5 is automatically configured with the computed NVM access latencies. For this purpose, we modified gem5 so as to enable the configuration of memories with asymmetric read and write latencies, such as NVMs. Afterwards, the specified system execution scenarios are run in parallel (according to the number of cores available on the host machine) by automatically triggering the corresponding number of gem5 simulation instances.

Each gem5 simulation instance produces the execution statistics file related its design scenario.

From these files, all data required by NVSim and McPAT (e.g., execution time of the system, number of read/write transactions for memory blocks) are automatically extracted by another script. As these files can be huge, the script has been defined in such a way that it optimizes the reading of generated gem5 files. Then, it invokes the two estimation tools on the extracted data in order to generate the area, power and energy consumption for each captured design scenario.

The results are stored in textual files.

Finally, the above textual files are post-processed by several scripts for generating various userfriendly renderings: CSV files and graphical plots that compare the performance, area, power and energy evaluation of the different scenarios. For instance, the energy breakdown of the main hardware components can be easily plotted for a fine-grained analysis.

Though, MAGPIE does not adopt a metamodeling-based transformations as promoted in existing design frameworks [START_REF] Quadri | Modeling of Configurations for Embedded System Implementations in MARTE[END_REF][START_REF] Gamatié | A model-driven design framework for massively parallel embedded systems[END_REF][START_REF] Dekeyser | Using the UML Profile for MARTE to MPSoC Co-Design[END_REF], it somehow follows a model-driven engineering approach, where XML is one prominent intermediate representation formalism.

Evaluation of system configurations 3.2.1 Experimental setup

As a starting point, let us consider the Exynos 5 Octa (5422) chip sketched in Figure 2. It features two quad-core clusters: "big" and "LITTLE" running at 1GHz. Its main parameters are displayed in Figure 2. Each core has its private instruction/data L1 caches (32kB each), while each cluster has Version 2.0 (2017) Confidentiality: Public Distribution a single shared L2 cache. The L2 caches are connected to the DRAM memory via a 64-bit cache coherent interconnect. The chip incorporates its own system memory in the form of 2GB LPDDR3 RAM integrated in a Package-on-Package (PoP) fashion. This architecture has been modeled in gem5 and models have been calibrated so as to provide sufficient accuracy against this SoC [START_REF] Butko | Design exploration for next generation high-performance manycore on-chip systems: Application to big.little architectures[END_REF][START_REF] Butko | Fullsystem simulation of big.little multicore architecture for performance and energy exploration[END_REF]. From this initial template, we vary the number of big and LITTLE cores such that the total number of cores is always equal to eight. In the sequel, each architecture configuration composed of X big Cortex-A15 cores is denoted by "XA15". For instance, "0A15" denotes a homogeneous configuration composed of eight LITTLE Cortex-A7 cores while "1A15" refers to a configuration with seven LITTLE Cortex-A7 cores and one big Cortex-A15 core.

The aim is to assess the behavior of such system configurations in presence of MRAM technology. In particular, we consider a 45nm STT-RAM technology, integrated at last-level cache, i.e., L2 cache in the modeled system. In the reported experiments, system designs in which all L2 caches are in SRAM (resp. STT-RAM) are denoted with a suffix indicating the corresponding memory technology, i.e., "-SRAM" (resp. "-MRAM"). Integrating STT-RAM at L1 cache without adequate optimization can be penalizing in case of frequent memory access due to the higher access latency of such a technology compared to SRAM. In [START_REF] Bouziane | How could compile-time program analysis help leveraging emerging nvm features?[END_REF][START_REF] Bouziane | LLVM-based silent stores optimization to reduce energy consumption on STT-RAM cache memory[END_REF][START_REF]Description of specific optimizations for low-power[END_REF], we advocated silent store elimination as a useful technique for mitigating this issue while considering non-volatile memory in L1 cache.

The Parsec 3.0 benchmark suite is used to evaluate the above architecture design. Figure 3 summarizes the whole Parsec benchmark suite. The considered kernels and applications are compiled for ARMv7 cores. A Linux 3.14 operating system is used. Kernels are executed with their small input sets.

Evaluation results

Figures 4, 5, 6 and 7 respectively report for each Parsec 3.0 kernels or applications the corresponding temporal behavior and on-chip energy-to-solution considering the above scenarios. The estimated energy comprises the following on-chip devices: eight CPUs with their associated L1-instruction and L1-data caches, the two L2 caches, the bus interconnect and main memory controller.

More generally, we observe in those figures that integrating STT-RAM at L2 cache level slightly incurs a performance penalty due to the well-known higher write latency compared to SRAM [START_REF] Senni | Potential applications based on nvm emerging technologies[END_REF][START_REF] Senni | Emerging non-volatile memory technologies exploration flow for processor architecture[END_REF][START_REF] Senni | Exploring MRAM technologies for energy efficient Systems-on-Chip[END_REF]. Regarding Energy-to-solution (EtoS), we observe that the major part of configurations integrating STT-RAM in L2 cache shows an improvement of the power consumption compared to SRAM. The energy related to an L2 cache in NVM is determined with NVSim, while McPAT is used to estimate the energy of the other on-chip devices. The obtained energy reduction appears important when the system comprises more LITTLE cores compared to big cores.

While the CONTINUUM project targets around 30% energy reduction in its investigated design solutions, the reported evaluation suggests that this can be reached via a careful integration of NVMs with adequate choice of heterogeneous cores. For instance, the asymmetric heterogeneous eightcore configuration composed of a single big Cortex-A15 core and seven LITLLE Cortex-A7 cores can provide such energy reduction as shown in Figures 4(b) and 4(d). At the same time, Figure 4(f) shows that the same configuration can lead to no gain for some specific application. This suggests that application nature must be also taken into account in order to adapt the system configuration for a better outcome in term of energy. For instance, in Figure 4(f), combining two Cortex-A15 and six Cortex-A7 cores gives better energy reduction. On the other hand, among the whole evaluated scenarios, we observe that the smallest energy reduction is obtained with homogeneous multicore configurations, and in particular with eight big Cortex-A15 cores.

The above evaluations confirm the relevance of NVM technologies such as STT-RAM for energy gain when considered in memory hierarchy. We have been carrying out some complementary studies aiming at leveraging advanced cache replacement techniques [START_REF] Péneau | Performance and energy assessment of last-level cache replacement policies[END_REF] in presence of NVMs, for further performance and energy improvements. Finally, some complementary directions, including software optimization techniques, multicore architecture designs and adaptive workload management, are discussed later in Section 5.

Evaluation of a 3D NoC Model

The CONTINUUM project initially aims at using NoCs as candidate interconnects in the considered multicore heterogeneous compute node. In particular, 2D NoCs were the main target, while some investigations were also planed on 3D NoCs in order to identify potential avenues for energy-efficiency improvement when such advanced technologies will become mainstream in manufactured SoCs.

However, our ongoing work shows that the Cortus technology, which has been adopted for implementing our compute node design, rather accommodates crossbar instead of NoC. Indeed, there is a trade-off between complexity (die area required, power consumption), transfer speed, latency and throughput. Very simple systems composed of a few cores can use a bus. As the complexity increases, a crossbar becomes more attractive, allowing multiple accesses between cores via high-speed paths.

Since the number of potential paths between cores increases the complexity of the crossbar increases to a point that a large portion of the die is reserved for the crossbar and timing closure becomes increasingly difficult. At this point a NoC becomes better. The cost in terms of die area and power consumption of the network and the increase in latency are comparatively lower. Nevertheless, the point at which a network-on-chip becomes necessary can be postponed by using a multi-level crossbar system proposed by Cortus, where the number of communicating cores per crossbar is reduced. This is therefore the solution adopted in the CONTINUUM project.

The rest of this section is devoted to another study about 3D interconnect as introduced before so as to identify possible exploitation opportunities in the future, beyond the current project.

3D NoC design challenge

Three-dimensional (3D) integration provides improved performance, increased package density, noise reduction, smaller footprint and reduced power consumption than conventional two dimensional 2D process [START_REF] Feero | Networks-on-chip in a three-dimensional environment: A performance evaluation[END_REF][START_REF] Kim | A study of through-silicon-via impact on the 3d stacked ic layout[END_REF]. Unlike the 2D process, 3D process exploits the Z-direction vertical to enhance system performance. This is achieved by stacking multiple dies in the Z-direction and interconnecting them using Through silicon via (TSV). TSV provide shorter wire length which corresponds to reduced Version 2.0 (2017) Confidentiality: Public Distribution power and increase interconnection density, thereby enhancing the overall system functionality and performance [START_REF] Kim | Through-silicon-via aware interconnect prediction and optimization for 3d stacked ics[END_REF][START_REF] Umemoto | High-performance vertical interconnection for high-density 3d chip stacking package[END_REF].

On the other hand, 3D manufacturing process can lead to process variation caused by additional processing and stacking steps [START_REF] Effiong | On the performance exploration of 3d nocs with resistive-open tsvs[END_REF]. Process variation reduces manufacturing yield and leads to significant performance degradation. As stated in [START_REF] Topol | Three-dimensional integrated circuits[END_REF], TSV delays can vary significantly due to defects and/or impurities that are introduced during the manufacturing process. Such defect is known as open-resistive defect. TSV links with open resistive defect maintain weak electrical connection between dies, which leads to significant signal propagation delays [START_REF] Metzler | Through-silicon-via resistive-open defect analysis[END_REF][START_REF] Kologeski | Performance exploration of partially connected 3d nocs under manufacturing variability[END_REF].

Fig. 8 shows a partially connected three-layer 3D-NoC with non-defective and open-resistive TSVs links between the layers. Such heterogeneous configuration leads to unbalanced data propagation delay, where a data travel across defective TSV is slower due to defect caused by the manufacturing process. The overall system performance is impaired because of accumulating delays incurred when traversing many of such defective TSV links. Therefore, from a performance evaluation point of view, it is imperative to investigate the performance of communication architectures under process variation, with a view to finding out the best and worse case system performance even under process variation. One possible approach to detecting defective TSVs is to do a post silicon validation test. This involves prototyping on actual silicon but before product release. This approach incurs cost and design time overhead. The use of redundant TSVs to replace failed TSVs have been proposed [START_REF] Hsieh | Tsv redundancy: Architecture and design issues in 3-d ic[END_REF][START_REF] Ye | Tsv open defects in 3d integrated circuits: Characterization, test, and optimal spare allocation[END_REF]. This method increases silicon area, cost and design complexity. A recent approach suggested the use of asynchronous delay-insensitive logic for inter-tier links [START_REF] Darve | Physical implementation of an asynchronous 3d-noc router using serial vertical links[END_REF]. This method makes it possible to exploit TSV links regardless of their delays. This approach however leads to an asymmetric NoC communication performance [START_REF] Effiong | On the performance exploration of 3d nocs with resistive-open tsvs[END_REF].

In the current study, we follow a different approach to address the problem by exploring the extent to which the communication architecture performance is affected by process variation. First, we carry out extensive analysis on the impact of process variation on communication performance depending on state-of-the-art mapping heuristics of real world applications and different architectural parameters

Experimental implementation of a 3D NoC architecture

The considered architecture is Globally Asynchronous Locally Synchronous (GALS) based, where synchronous router communicates asynchronously. Two different schemes were used to provide asynchronous communication between the routers. These are: i) bi-synchronous FIFOs for intra-layer communication between two communicating routers, ii) fully asynchronous serialized vertical links for inter-layer communication. Bi-synchronous FIFOs provide a low cost, scalable and area efficient interface for routers with different clock frequencies and phase [START_REF] Panades | Bi-synchronous fifo for synchronous circuit communication well suited for network-on-chip in gals architectures[END_REF]. The inter-layer communication scheme between two routers residing one hop away from each other at two consecutive layers is given in Figure 9. The vertical links were serialized using fully asynchronous Quasi Delay Insensitive (QDI) asynchronous logic presented in [START_REF] Darve | Physical implementation of an asynchronous 3d-noc router using serial vertical links[END_REF]. Data transfer on the vertical link is encoded using four-phase dual-rail asynchronous protocol. This protocol uses two wires to encode one bit of information to be transmitted. An additional wire is needed to send back an acknowledgement from the receiver to the sender. Therefore, a total of three TSVs is needed to transmit one data bit on the vertical link. This protocol is delay-insensitive, ensuring that the vertical links behave correctly and reliably regardless of link delay caused by voltage, temperature, or manufacturing process variations.

Figure 9 shows the up and down vertical links between two communicating routers, where each arrow represents a TSV. The serial channel consists of trees of autonomous multiplexers used for serializing, i.e., self-controlled multiplexers (SCM) and de-serializing, i.e., self-controlled-demultiplexers (SDM) [START_REF] Darve | Physical implementation of an asynchronous 3d-noc router using serial vertical links[END_REF] data traversing the vertical links. The number of TSVs between each pair of communicating router varies according to the serialization rate. For a 3D-NoC with N-bits communication data, 3N TSVs are need to transmit the data through a vertical link, i.e., no serialization. This is because, 3 TSVs (i.e., bit_Woi, bit_W1i and acki) are needed to transmit 1 parallel bit data through the link as Version 2.0 (2017) Confidentiality: Public Distribution shown in Fig. 9. Serialization is used to reduce the number of TSV links in the network. This is achieved by multiplexing bits in time according to the serialization rate (i.e. the number of parts a flit must be divided into).

The additional TSV link referred to as Transaction ack in Figure 9, is used by the handshake interface to connect the asynchronous channel with the router and to inform the router when a new transaction can be initiated. For this signal to be valid, the receiving router must be able to receive a data and data corresponding to the previous transaction must have been sampled.

Since the architecture uses purely asynchronous logic design, the serialization subsystem performance solely depends on propagation delays of both gates and TSVs. The latency of each TSV is therefore accounted for in the description of each asynchronous serializer of the simulation model. This approach makes it possible to accurately analyze various metrics such as bandwidth and communication latencies in the NoC.

Application mapping on NoC-based multicore architecture

Application mapping is crucial in multi/many core systems because of vast application requirements. Applications are normally decomposed into a set of tasks which can be executed in parallel on different cores. Mapping heuristics determines how application tasks are mapped on the cores. The choice of mapping heuristics for a given application determines if the application requirements will be met or not. Mapping application tasks on multi/many core systems is carried out with a view to optimizing criteria such as compute performance and energy consumption [START_REF] Singh | Mapping on multi/many-core systems: Survey of current and emerging trends[END_REF]. In order to carry out our performance exploration, we consider some of the well-known application mapping heuristics and investigate the performance of the network depending on the mappings. In this work, we consider only static application mapping for simplicity. We also assume that only one task is mapped on a node. The considered mappings are briefly explained in the subsequent subsections. Critical path mapping In critical path mapping (CP), the tasks on the longest path of the application task graph are mapped first before the other tasks. The goal of this mapping is to reduce network contention and packet latency on those paths while exploiting TSVs as much as possible. As shown in Fig. 10(c) tasks T 0 , T 1 , and T 2 , which are on the critical path are mapped beginning from bottom right of the first tier onward.

Least-Comm mapping

In Least-comm-middle (LCM) mapping, tasks with the lowest communication activity are mapped at the middle tier(s). This is done with a view to balancing the load on the network since more packets tend to traverse the middle tier(s).

Performance evaluation

In order to assess the performance of the NoC under process variation, we inject traces from realworld application benchmarks into the network. The applications include: video conference encoder (VCE), Wifi baseband receiver (WIFI), multimedia system (MMS) and E3S consumer benchmarks [START_REF] Tran | Noctweak: a highly parameterizable simulator for early exploration of performance and energy of networks on-chip[END_REF]. The applications characteristics are given in Table 1. We carry out some evaluations by considering a 3x3x3 mesh network topology. This network uses ZXY dimension ordered routing and a credit based flow control without virtual channels. In order to investigate the network performance under variability, we simulated the network while taking into account different design parameters, i.e., serialization rates, TSV sizes and application mappings.

Impact of application mapping on 3D-NoC

The goal of this experiment is to investigate how the mapping heuristics described previously impact the network performance particularly in terms of network link utilization. Links with extremely high utilization create hotspots in terms of temperature and power consumption. Links with 50% over-utilization Links with over 70% over-utiization Idle or under-utilized links Links with 25% over-utilization Fig. 11 shows the link utilization of the mappings for VCE application. We have considered links with the load greater than a threshold to be over-loaded. It can be observed that for 3D-MinComm mapping, most of the TSVs links are over-loaded, while most 2D-links are either idle or under-utilized.

The reason being that this mapping attempts to exploit the high bandwidth TSVs. Similar argument can be made for 2D-MinComm mapping except that in this case the 2D-links are overloaded. Also, from Fig. 11(b) the middle and top layers are overload, while the bottom layer remains under-utilized. This creates unbalanced network load corresponding to hotspots in the over-utilized layers. In similar Page 16 Version 2.0 (2017) Confidentiality: Public Distribution manner, CP mapping creates hotspots in the first two layers while, the topmost layer is not overutilized as seen in Fig. 11(c). Hotspots appear on each layer of the network for LCM mapping as seen in Fig. 11(d). Compared to the other mappings, a greater percentage of the network links are over-utilize for LCM mapping.

Impact of architectural parameters on 3D-NoC performance

In order to investigate the impact of architectural parameters on 3D-NoC performance, we mapped the applications introduced previously on 3D-NoC with three configurations : a) architecture without serialization and process variation [Arch_only], b) architecture with serialization but without process variation [Arch_SER], c) architecture with serialization and process variation [Arch_SER_PV]. We considered serialization rate of 2 for the second and third configurations. Serialization rate of 2 implies that 16 TSVs are used between each pair of communicating router in the vertical direction. For the third configuration, we considered small TSVs (SM) out of which 15% are defective i.e. they have large-open resistive defect. We considered 400 ps delay for the defective TSVs. 12 shows the performance of the network configurations based on the application mappings. It can be observed that for all the applications, the network packet latency increases significantly for the Arch_SER when compared to Arch_only. To illustrate this point, let us consider the MMS application. We observe an increase of over 123% in packet latency for Arch_SER compared to Arch_only configuration for 3D-MinComm mapping. The reason being that Arch_only has twice as many vertical links as Arch_SER configuration. Therefore, more flits can travel in parallel in the Arch_SER than in Arch_SER. This demonstrates that vertical link serialization can significantly impair network performance especially when more communications utilize the vertical links. The network packet latency is further degraded for Arch_SER_PV when compared to Arch_only configuration. For the MMS application, we observed that the packet latency increases by 145% for Arch_SER_PV when compared to Arch_only 3D-MinComm mapping. Similar argument can be made for the other applications with 3D-MinComm mapping. This indicates that open-resistive TSVs lead to network Version 2.0 (2017) Confidentiality: Public Distribution performance degradation. However, for the other mappings, TSV defects do not significantly degrade the network performance for Arch_SER_PV when compared to Arch_SER configuration. The reason being that these mappings do not mainly utilize the TSV links as opposed to 3D-MinComm mapping that seeks to exploit the TSV links, but suffer significant performance loss when the TSV links are defective.

Further analyses of the results reveal that the serialization rate and process variation do not significantly impair the network performance for VCE and E3S applications using the other mappings (i.e. CP, LCM, and 2D-MinComm). One possible reason for this is that for these applications and mappings, only few of the total number of available vertical links are utilized. Therefore, the network performance is not significantly impaired since serialization and process variation concern only the vertical links. As an example, the E3S application has only 131 tasks therefore, the application can be mapped using CP, LCM, and 2D-MinComm in such as way that mainly the 2D-links are utilized.

Impact of serialization on 3D-NoC performance

In order to obtain a general optimal serialization rate for a 3D-NoC, we have considered the configurations shown in Table 2 for our simulations. In this table, Def-TSVs refers to defective TSVs. Fig. 13 shows the resulting network packet latency for each configuration. It can be observed that the network packet latency increases by 16% for serialization rate of 4 when compared to a serialization rate of 2 for a network without defective TSVs and by 29% for a network with defective TSVs. This demonstrates that although serialization rate of 4 can reduce the network area overhead by 75% when compared to a network without serialization, significant performance degradation occur. Therefore, the trade-off between performance and area cost.

Impact of TSV size on 3D-NoC performance

In order to obtain a general optimal TSV size for a 3D-NoC, we have considered the configurations shown in Table 3 for our simulations. Here, D-TSVs represents defective TSVs. We have considered a delay of 48 ps, 190 ps, 400 ps for defective large (LG), medium (MD) and small (SM) TSVs respectively.

Fig. 14 shows the performance of the network with the different TSVs. It can be observed that network packet latency is increased by 4%, 13% for MD and SM TSVs respectively when compared to LG TSVs. The network packet latency is not significantly impaired for MD TSVs when compared to LG TSVs. The reason is that the vertical link scheme utilizes quasi delay-insensitive asynchronous logic LG MD SM which ensures that bandwidth of TSV links with resistive defect can still be exploited regardless of their propagation delay.

Impact of process variation on 3D-NoC performance

In order to explore the extent to which process variation affects a 3D-NoC performance, we have considered the configurations shown in Table 4 for our simulations. We compare a network without defective TSVs to networks with defective TSVs. The main directions we already identified are summarized through the following items:

Energy-efficiency improvement by addressing the asymmetric nature of NVM memory accesses. The evaluation of multicore systems presented in Section 3, while integrating nonvolatile memory (NVM) technology in the cache hierarchy showed that energy saving is possible thanks to the characteristics of such memory technologies, i.e., a negligible leakage current that favors a significant decrease in static power consumption compared to concurrent SRAM-based solutions. At the same time, we observed that the high latency (and energy consumption) of write operations on NVMs can be detrimental to system performance due to the resulting longer execution time.

Therefore, an opportunity that is currently under consideration in the CONTINUUM project is to exploit adequate code transformations (or optimization) so as to mitigate the negative effect of writes on performance when using NVMs [START_REF] Péneau | Loop optimization in presence of STT-MRAM caches: A study of performanceenergy tradeoffs[END_REF]. After some preliminary studies, we have been investigating the so-called silent store elimination in programs [START_REF] Bouziane | How could compile-time program analysis help leveraging emerging nvm features?[END_REF][START_REF] Bouziane | LLVM-based silent stores optimization to reduce energy consumption on STT-RAM cache memory[END_REF][START_REF]Description of specific optimizations for low-power[END_REF]. Intuitively, a store in a program is said to be silent if it writes a value to a memory address where the same value is already stored. Then, our approach consists in transforming a given source code such that silent stores can be avoided. This enables to reduce the total number of writes. Our silent store code optimization has been implemented in LLVM compiler.

Energy-efficiency optimization via retention time relaxation. Another opportunity that is worth mentioning is to exploit further characteristics of NVMs, particularly the trade-off between their power consumption requirement and their non volatility capacity [START_REF] Bouziane | How could compile-time program analysis help leveraging emerging nvm features?[END_REF]. Indeed, it is well-known that relaxing the retention time of NVMs through a reduction of the planar area of their cell, contributes to decreasing their write current [START_REF] Smullen | Relaxing nonvolatility for fast and energy-efficient stt-ram caches[END_REF]. Consequently, this reduces their high dynamic energy and write latencies. Now, let us consider that the multicore system under design includes multiple NVM memory banks, with various non volatility capacity. The intuition here is to allocate data across those memory banks, e.g., according to the liveness of the corresponding variables in a program. For instance, a variable that is alive for a short period during program execution could be mapped to memory banks with low retention time, i.e., requiring a low dynamic energy and shorter write latency. On the opposite, a variable that must be alive for a longer period would be mapped to memory banks with high retention capability such that it could be accessed whenever needed during execution.

Leveraging liveness analysis of variables in programs according to compilation techniques is therefore a relevant opportunity to improve the energy-efficiency in presence of relaxed NVM retention time.

Compiler-assisted Adaptive Code Placement in Heterogeneous Systems. On the other hand, the evaluations reported in Section 3 showed the impact of the heterogeneous nature of cores on Version 2.0 (2017) Confidentiality: Public Distribution both performance and energy consumption. Typically, big cores such as Cortex-A15 provide high performance at the expense of more dissipated power. LITTLE cores on the other hand are counter-parts of big cores. This trade-off calls for adequate core selection for a given program in order to reach a good compromise in terms of performance and power consumption.

Here, both runtime management and compilation techniques can play an important role. Thanks to the former, one can dynamically assign threads or application in order to reach the expected compromise [START_REF] Butko | Position Paper: OpenMP scheduling on ARM big.LITTLE architecture[END_REF][START_REF] Butko | Efficient Programming for Multicore Processor Heterogeneity: OpenMP versus OmpSs[END_REF][START_REF] Garibotti | Efficient embedded software migration towards clusterized distributed-memory architectures[END_REF][START_REF] Cohen | Processor Virtualization and Split Compilation for Heterogeneous Multicore Embedded Systems[END_REF]]. Beyond useful system information monitored at runtime (e.g., CPU usage, instruction per cycle, cache miss rate and power consumption), further interesting information resulting from compiler-based static code analysis can be also helpful. For instance, compute-intensive code fragments include many arithmetic operations will be expected to run on big cores while code fragments consisting mainly of read/write of some inputs/outputs will be rather executed on LITTLE cores. On the other hand, the difference between the features of cores (e.g., cache hierarchy, presence of FPU or not...) could be reflected through the applied compilation options according to the target core. Multiversioning is one relevant technique under consideration within the CONTINUUM project. Finally, machine learning techniques could be foreseen in the workload scheduling and mapping loop for efficient execution in heterogeneous systems [START_REF] Gamatié | Performance prediction of application mapping in manycore systems with artificial neural networks[END_REF].

Workload allocation for optimized interconnect traffic. Finally, as illustrated in Section 4 on the evaluation of 3D-NoC multicore architecture design, application mapping can benefit from the aforementioned runtime management decisions in order to optimize the communication traffic. Note that the same observation generally holds for 2D NoCs, such as typical mesh networks [START_REF] Schonwald | Fully adaptive faulttolerant routing algorithm for network-on-chip architectures[END_REF][START_REF] Xu | Research and analysis of routing algorithms for noc[END_REF][START_REF] Latif | Design space exploration for complex automotive applications: an engine control system case study[END_REF][START_REF] Effiong | Roundabout: A network-on-chip router with adaptive buffer sharing[END_REF][START_REF] Effiong | Scalable and power-efficient implementation of an asynchronous router with buffer sharing[END_REF]. Beyond the workload allocation issue itself, it can be interesting to consider interconnects that favors an adaptive data routing inside the network [START_REF] Effiong | Distributed and dynamic shared-buffer router for high-performance interconnect[END_REF] for high throughput.

Finally, code optimization techniques could be beneficial to the interconnect activity. Typically, Lepak et al. [START_REF] Lepak | Silent stores and store value locality[END_REF] previously showed that eliminating silent stores helps to reduce multiprocessor bus traffic. We can therefore take advantage of this feature while applying this technique.

Concluding remarks

In this deliverable, we described some evaluations where non volatile memory and 3D interconnect technologies are evaluated. On the one hand, energy saving opportunity was shown thanks to the low leakage current of non volatile memories. On the other hand, 3D interconnects provide an alternative design for improving system performance by accelerating the communication traffic. Nevertheless, both technologies present challenging aspects that can significantly reduce these benefits. For instance, the higher cost of write operations in non volatile memories and the process variability issue can affect TSV links in 3D interconnects.

Therefore, in order to mitigate these potential limitations, a number of opportunities have been discussed by advocating both compilation and runtime system management techniques. Some of these techniques are already under investigation within the CONTINUUM project. The preliminary results [START_REF]Description of specific optimizations for low-power[END_REF] show promising improvements of energy-efficiency.

Figure 1 :

 1 Figure 1: MAGPIE evaluation flow.

Figure 2 :

 2 Figure 2: Exynos 5 Octa big.LITTLE architecture

Figure 3 :

 3 Figure 3: Parsec benchmark suite

  (a) Execution time (bodytrack) (b) Energy-to-Solution (bodytrack) (c) Execution time (canneal) (d) Energy-to-Solution (canneal) (e) Execution time (facesim) (f) Energy-to-Solution (facesim)

Figure 4 :

 4 Figure 4: Execution time and energy-to-Solution evaluation for Parsec (part 1)

  (a) Execution time (ferret) (b) Energy-to-Solution (ferret) (c) Execution time (fluidanimate (d) Energy-to-Solution (fluidanimate) (e) Execution time (freqmine) (f) Energy-to-Solution (freqmine)

Figure 5 :

 5 Figure 5: Execution time and energy-to-Solution evaluation for Parsec (part 2)

  (a) Execution time (streamcluster) (b) Energy-to-Solution (streamcluster) (c) Execution time (swaptions) (d) Energy-to-Solution (swaptions) (e) Execution time (vips) (f) Energy-to-Solution (vips)

Figure 6 :

 6 Figure 6: Execution time and energy-to-Solution evaluation for Parsec (part 3)

Figure 7 :

 7 Figure 7: Execution time and energy-to-Solution evaluation for Parsec (part 4)

Figure 8 :

 8 Figure 8: Partially connected 3D-NoC with open resistive TSVs. δt represents delay to traverse a non-defective TSV, while δt + n represent n additional delay caused by open-resistive defect

  such as TSV size, number of TSVs between each pair of communicating nodes, etc. Finally, we explore how the different architectural parameters can be combined to alleviate the detrimental effects of defective TSVs.

Figure 9 :

 9 Figure 9: Interlayer communication

Figure 10 :

 10 Figure 10: Application mapping heuristic (a)3D-Mincomm mapping (b) 2D-Mincomm mapping (c) Critical path mapping

Figure 11 :

 11 Figure 11: Network links utilization of VCE application based on mappings: (a) 3D-MinComm, (b) 2D-MinComm, (c) CP, (d) LCM.

Figure 12 :

 12 Figure 12: Impact of architectural parameters on 3D-NoC performance

Figure 13 :

 13 Figure 13: Performance of 3D-NoC with serialization rates

Figure 14 :

 14 Figure 14: Performance of 3D-NoC configurations with TSV sizes

Figure 15 :

 15 Figure 15: Performance of 3D-NoC configurations with defective TSVs

Table 1 :

 1 Application benchmarks characteristics

	Application Number of tasks Communication volume (packets)
	VCE	24	52060
	MMS	25	644098
	E3S	12	131
	WIFI	25	2160798

Table 2 :

 2 Serialization rates network configuration

	App	Mapping	SR % of Def-TSVs TSV size
	VCE 3D-MinComm	2 4	-15 -15	SM

Table 3 :

 3 Network configurations for TSV size

	App	Mapping	SR % D-TSVs TSV size
	VCE 3D-MinComm 2	15

Table 4 :

 4 Network configuration for process variation impact

	App	Mapping	SR TSV size % of Def-TSVs
				-
	VCE 3D-MinComm 2	15	5 10
				15
				20
	Fig. 15 shows the performance of the network with process variation. It can be observed that network
	packet latency increases as the number of defective TSVs. The network latency is increased by 4%,
	9%, 13%, 15% for 5%, 10% and 15% and 20% defective TSVs when compared a network without any
			Version 2.0
			(2017)
		Confidentiality: Public Distribution