Pascal Vivet

Eric Guthmuller

Yvain Thonnart

Gaël Pillonnet

Cesar Fuguet

Ivan Miro-Panades

Guillaume Moritz

Jean Durupt

Christian Bernard

Didier Varreau

Gael Pillonnet

I Miro- Panades

S Thuries

D Coriat

M Harrand

D Dutoit

IntAct: A 96-Core Processor With Six Chiplets 3D-Stacked on an Active Interposer With Distributed Interconnects and Integrated Power Management

Keywords: SoC infrastructure Analog, IOs, PHY, DFT Clusters of Cores

published or not. The documents may come   L'archive ouverte pluridisciplinaire

Abstract-In the context of high performance computing, the integration of more computing capabilities with generic cores or dedicated accelerators for AI application is raising more and more challenges. Due to the increasing costs of advanced nodes and the difficulties of shrinking analog and circuit IOs, alternative architecture solutions to single die are becoming mainstream. Chiplet-based systems using 3D technologies enable modular and scalable architecture and technology partitioning. Nevertheless, there are still limitations due to chiplet integration on passive interposerssilicon or organic. In this paper we present the first CMOS active interposer, integrating i) power management without any external components, ii) distributed interconnects enabling any chiplet-to-chiplet communication, iii) system infrastructure, Design-for-Test, and circuit IOs. The INTACT circuit prototype integrates 6 chiplets in FDSOI 28nm technology, which are 3D-stacked onto this active interposer in 65nm process, offering a total of 96 computing cores. Full scalability of the computing system is achieved using an innovative scalable cache coherent memory hierarchy, enabled by distributed Network-on-Chips, with 3Tbit/s/mm 2 high bandwidth 3D-plug interfaces using 20µm pitch micro-bumps, 0.6ns/mm low latency asynchronous interconnects, while the 6 chiplets are locally power-supplied with 156mW/mm2@ 82%-peak-efficiency DC-DC converters through the active interposer. Thermal dissipation is studied showing the feasibility of such approach.

Index Terms-3D technology, Chiplet, Active Interposer, Power Management, Network-on-Chip, Thermal dissipation

I. INTRODUCTION

N the context of high performance computing (HPC) and bigdata applications, the quest for performance requires modular, scalable, energy-efficient, low-cost many-core systems. To address the demanding needs for computing power, system architects are continuously integrating more cores, more accelerators, more memory in a given power envelope [START_REF] Agosta | Challenges in Deeply Heterogeneous High Performance Systems[END_REF]. It

The outline of the paper is as follows: section I introduces the chiplet paradigm in more detail, with a state of the art on these technologies and the proposed concept of active interposer. Section II presents an overview of the INTACT demonstrator architecture and 3D technology, while the sub-sequent sections detail the various sub-elements of the circuit: computing chiplet, power management, distributed interconnects, and testability. Section IX addresses the thermal issues. Finally, section X and XI present the final circuit results and conclusion.

II. CHIPLET AND ACTIVE INTERPOSER PRINCIPLE

A. Chiplet partitioning: concept and challenges

Chiplet partitioning is raising new interest in the research community [START_REF] Gupta | Goodbye, motherboard. Bare chiplets bonded to silicon will make computers smaller and more powerful: Hello, silicon-interconnect fabric[END_REF], in large research programs [START_REF]CHIPS program[END_REF] and in the industry [9]. It is actually an idea with a long history in the 3D technology field [START_REF]Technology and Applications of 3D[END_REF]. The concept of chiplet is rather simple: divide circuits in modular sub-systems, in order to build a system as a LEGO ® -based approach, using advanced 3D technologies. The motivation for chiplet-based partitioning is as follows: -It is driven by cost. Due to increasing issues in advanced CMOS technologies (7nm and below), achieving high yield on large dies in acceptable costs is not possible anymore, while shrinking all the analog IPs (power management, Fast IO SerDes, etc) is becoming increasingly difficult. By dividing a system in various sub-modules, called chiplets, it is possible to yield larger systems at acceptable cost, thanks to Known Good Die (KGD) sorting [10]. -It is driven by modularity. By an elegant divide & conquer partitioning scheme, chipletization allows to build modular systems from various building blocks and circuits, focusing more on functional aspects than on technology constraints. Circuit designers can deeply optimize each function: generic CPUs, optimized GPUs, embedded FPGAs, dedicated accelerators for machine learning, dense memory solutions, IO & services, while the system designer is picking the best combination to build a differentiated and optimized system. -It is enabled by heterogeneous integration. For chiplets, the right technology is selected to implement the right function: advanced CMOS for computing, DRAM for memory cubes like High Bandwidth Memory (HBM) [START_REF] Sohn | A 1.2V 20nm 307GB/s HBM DRAM with at-speed wafer-level I/O test scheme and adaptive refresh considering temperature distribution[END_REF], Non Volatile Mémory (NVM) technology for data processing within AI accelerators [START_REF] Wu | 14.3 A 43pJ/Cycle Non-Volatile Microcontroller with 4.7μs Shutdown/Wake-up Integrating 2.3-bit/Cell Resistive RAM and Resilience Techniques[END_REF], mature technology for analog functions (IOs, clocking , power management, etc). Chiplet integration is then performed using advanced 3D technologies, which are getting more and more mature, with reduced pitches, using Through Silicon Via (TSV) and micro-bumps [START_REF] Lin | A 7nm 4GHz Arm®-core-based CoWoS® Chiplet Design for High Performance Computing[END_REF] or even more advanced die-to-wafer hybrid bonding technologies [3][13].

To benefit from all these advantages and possibilities, there are nevertheless clear challenges for chiplets. The ecosystem needs to change progressively from IP-reuse to chiplet-reuse; this requires fundamental changes in the responsibilities of the various providers. These constraints are economical rather than technical, but they are strongly driving the technical choices.

For system level design, the simple LEGO ® cartoon (Fig. 1) needs some adequate methodologies to address system modeling, cost modeling, to perform technology and architecture partitioning while achieving an optimized system. A strong movement is building momentum towards the standardization of chiplet interfaces to enable this modularity between various vendors [START_REF]Open Compute ODSA project[END_REF].

Finally, many circuit level design issues arise: design of energy efficient chiplet interfaces, testability, power management and power distribution, final system sign-off in terms of timing, power, layout, and reliability, thermal dissipation. To address these 3D design challenges, new CAD tools and associated design flows must be developed [START_REF] Reiter | Multi-Die IC Design Tutorial[END_REF].

In this paper, a partitioning using identical chiplets is proposed to scale-out a large distributed computing system offering 96-cores, by using heterogeneous technologies. Many circuit design aspects are addressed in terms of chiplet interfaces, distributed interconnects, power management, testability, and associated CAD flows.

B. State of the Art on Interposers

In order to assemble the chiplets together, various technologies have been developed and are currently available in the industry (Fig. 2).

Fig. 2. State-of-the-art on recent interposer and 3D technologies

Firstly, organic substrate is the lowest cost solution, while offering larger interconnect pitches (130µm). This technology has been adapted by AMD for their EPIC processor family, with a first version with up to 4 chiplets [START_REF] Beck | Zeppelin: An SoC for Multichip Architectures[END_REF] and a recent version with up to 8 chiplets using a central IO die to distribute the system level interconnects [START_REF] Naffziger | AMD Chiplet Architecture for High-Performance Server and Desktop Products[END_REF]. Passive interposers, also called 2.5D integration, as proposed for instance by TSMC CoWoS [START_REF] Lin | A 7nm 4GHz Arm®-core-based CoWoS® Chiplet Design for High Performance Computing[END_REF] enable more aggressive chip-to-chip interconnects and pitches (40µm) but are still limited to "wire only" connections. A tradeoff in terms of cost, pitches and performances can be achieved by using a silicon bridge embedded within the organic substrate as presented by INTEL and their EMIB bridge [START_REF] Greenhill | A 14nm 1GHz FPGA with 2.5D Transceiver Integration[END_REF]. Finally, regular 3D stacking (for vertical assembly) may also be used, which is also orthogonal and complementary of interposer approaches. INTEL has presented recent results with their Foveros technology and LakeField processor [START_REF] Gomes | Lakefield and Mobility Compute: A 3D Stacked 10nm and 22FFL Hybrid Processor System in 12×12mm2, 1mm Package-on-Package[END_REF].

All these solutions are promising and show clear benefits in terms of cost and performances. Nevertheless, various challenges still arise: -Inter-chiplet communication is mostly limited to side-by-side communication, due to wire-only interposers. Longer range communication should rebound in the chiplets themselves, which is not scalable to build larger systems with numerous chiplets. The recent solution from AMD with their IOD [START_REF] Naffziger | AMD Chiplet Architecture for High-Performance Server and Desktop Products[END_REF] is partially solving these issues, with better communication distribution and easier IO integration, but may still not scale further on the long term. -Current interposer solutions do not integrate themselves less scalable functions, such as IOs, analogs, power management, close to the chiplets. The recent solution from INTEL with digital on top of analog partitioning is solving this issue, but is still limited today to a single die [START_REF] Gomes | Lakefield and Mobility Compute: A 3D Stacked 10nm and 22FFL Hybrid Processor System in 12×12mm2, 1mm Package-on-Package[END_REF]. -Finally, it is currently complex to integrate chiplets from different sources, due to missing standards, even if strong initiative are on-going [START_REF]CHIPS program[END_REF][START_REF]Open Compute ODSA project[END_REF]. Wire-only interposers prevent the integration of chiplets using incompatible protocols, while active interposer enable to bridge them, as adapted by zGLUE Inc. [51].

In order to tackle all these issues, this paper presents the concept of Active Interposer, which integrates logic functions within the large interposer area. The concept has been already introduced before, either as a low cost and limited active-light solution for ESD integration [START_REF] Hellings | Active-Lite Interposer for 2.5 & 3D Integration[END_REF] or with system level architecture explorations showing the capability to scale larger systems [START_REF] Yin | Modular Routing Design for Chiplet-Based Systems[END_REF] [START_REF] Vasil Pano | 3D NoCs with active interposer for multi-die systems[END_REF]. The next section presents the active interposer concept, enabled by technology improvements [START_REF] Chéramy | The active-interposer concept for high-performance chip-to-chip connections[END_REF].

C. Active Interposer principle and partitioning

The proposed Active Interposer concept is detailed in Fig. 3. Chiplets can be either identical or different, for homogeneous functions as presented here, or differentiated functions, as presented in Fig. 1. Chiplets, implemented in an advanced CMOS technology, may themselves be composed of clusters of cores. Each chiplet can contain its own interconnects for intrachip communication, which are extended in 3D for chiplet-tochiplet communication. The CMOS interposer integrates a scalable and distributed Network-on-Chip (NoC), which offers the main capability of allowing any chiplet-to-chiplet traffic, without interfering with unrelated chiplets. As a conclusion, a hierarchical 3D NoC is obtained, with 2D NoC within the chiplet, 2D NoC within the active interposer, which can be further refined and differentiated according to traffic classes. Moreover, dense 3D interconnects enable high bandwidth density with parallel signaling. Such a communication scheme enables fully modular and scalable cache-coherent architecture, for offering large many-cores [START_REF] Vasil Pano | 3D NoCs with active interposer for multi-die systems[END_REF][28] [START_REF] Guthmuller | Architectural exploration of a fine-grained 3D cache for high performance in a manycore context[END_REF].

In order to provide efficient power supply to each chiplet, power management and associated power converters can be directly implemented within the active interposer, to bring power supply closer to the cores, for increased energy efficiency in the overall power distribution hierarchy, and allowing Dynamic Voltage and Frequency Scaling (DVFS) scheme at the chiplet level. Moreover, all the less-scalable functions, such as analog IPs, clock generators, and circuit IOs with SerDes and PHYs for off-chip communication, as well as the regular System-on-Chip infrastructure, such as low performance IOs, test, debug, etc., can also be implemented in the bottom die. Finally, additional features can be integrated in the active interposer, to specialize for a given application, enabling to differentiate the overall system. For instance, if incompatible chiplets are assembled, the active interposer can implement protocol bridges.

Due to additional power budget within the interposer, the thermal challenge of 3D might increase. Nevertheless, most of the power budget is within the chiplets, thermal dissipation issues are then limited, as presented section IX.

Regarding technology partitioning, the active interposer should be implemented using a mature technology, with a low logic density to achieve high yield. Large logic density within a large interposer would lead, even using a mature technology, to an un-yieldable and costly system. A difference of at least two technology nodes between the computing chiplets and the interposer should lead to an acceptable cost, while allowing enough performances in the bottom die for analog and PHYs to sustain the overall system performances, as done in [START_REF] Gomes | Lakefield and Mobility Compute: A 3D Stacked 10nm and 22FFL Hybrid Processor System in 12×12mm2, 1mm Package-on-Package[END_REF].

III. INTACT CIRCUIT ARCHITECTURE

The proposed Active Interposer concept is implemented within a large-scale circuit demonstrator, offering 96-cores, called INTACT for "Active Interposer".

A. INTACT Circuit Architecture

cores : 6 chiplets 3D-stacked on active CMOS interposer

INTACT is the first CMOS active interposer [START_REF] Vivet | A 220GOPS 96-Core Processor with 6 Chiplets 3D-Stacked on an Active Interposer Offering 0.6ns/mm Latency, 3Tb/s/mm2 Inter-Chiplet Interconnects and 156mW/mm2@ 82%-Peak-Efficiency DC-DC Converters[END_REF] integrating: i) Switched Capacitor Voltage Regulator (SCVR) for on-chip power management; ii) flexible system interconnect topologies between all chiplets for scalable cache coherency support; iii) energy efficient 3D-Plugs for dense inter-layer communication; iv) a memory-IO controller and PHY for socket communication. Fig. 4 presents an overview of the overall many-core circuit architecture, with the chiplets, the distributed interconnects, the integrated power management, and the system infrastructure, which are detailed hereinafter.

Each chiplet is a 16-core sub-system composed of 4 computing clusters of 4 cores, integrating their own distributed coherent memory caches, and their associated system level interconnects. The chiplet architecture and associated memory hierarchy is presented section V.

The chiplet interconnects are extended through the active interposer for chiplet-to-chiplet communication using distributed NoCs and various kinds of communication links, using so-called "3D Plug" communication interfaces. For offchip communication, the active interposer integrates a memory-IO controller and 4x32bits 600Mb/s bidirectional LVDS links offering a total of 19.2 GB/s off-chip bandwidth. The communication IPs and overall communication architecture are presented in section VII and VIII respectively.

The active interposer integrates a power management IP for supplying individually each chiplet and offering on-demand energy efficient power management, below each chiplet and surrounded by pipeline NoC links. The SCVR is presented in more detail section VI.

Finally, the active interposer integrates some regular Systemon-Chip infrastructure elements such as clock & reset generation, thermal sensors, stress sensors, low speed interfaces (UART, SPI) for debug and configuration, and Design-for-Test (DFT) logic for Known Good Die (KGD) sorting and final test. 3D testability challenges and associated DFT are presented in more detail section IX.

In conclusion, INTACT offers a large-scale cache-coherent many-core architecture, with a total of 96 cores in 6 chiplets (4 cores x 4 clusters x 6 chiplets), which are 3D-stacked onto the active interposer.

B. INTACT Circuit Technology Partitioning

The 22 mm 2 chiplets are implemented in a 28nm FDSOI CMOS node, while the 200 mm 2 active interposer is using a 65nm CMOS node, which is a more mature technology. As presented in section III, this technology partitioning exhibits two technology nodes differences between the computing die and the active interposer. This enables enough performance in the bottom die for the interconnects, the analog parts and the system IOs, while still allowing a yieldable large scale active interposer.

Even though complex functions are integrated, the yield of the active interposer is high thanks to this mature 65 nm node and a reduced complexity (0.08 transistor/µm 2 , see section II.C), with 30% interposer area devoted to SCVR variabilitytolerant capacitors scheme. This technology partitioning leads to a practical and reachable circuit and system in terms of silicon cost using advanced 3D technologies (more details in terms of yield analysis can be found in [START_REF] Gitlin | Generalized cost model for 3D systems[END_REF]).

C. INTACT physical design and 3D technology parameters

For enabling system integration, and allowing efficient chiplet-to-chiplet communication, an aggressive 3D technology has been developed and used. A summary of the respective chiplet, interposer, and 3D technologies is given in Table I.

As presented in Fig. 5 with the circuit 3D-cross section, the 6 chiplets are 3D-stacked in a face-to-face configuration using 20 µm-pitch micro-bumps (μ-bumps) onto the active interposer (2x smaller pitch compared to state of the art [START_REF] Coudrain | Active Interposer Technology for Chiplet-Based Advanced 3D System Architectures[END_REF]). These dense chip-to-chip interconnects enable a high bandwidth density, up to 3TBit/s/mm 2 as detailed in section VI.A, using parallel signaling through thousands of 3D signal interfaces. For bringing power supplies and allowing off-chip communication, the active interposer integrates TSV-middle with a pitch of 40 µm and an aspect ratio of 1:10 (10 µm diameter for a silicon height of 100 µm) and a keep-out zone of 10 µm. Finally, the overall system is assembled onto a package organic substrate (10 layers), using C4 bumps with a pitch of 200 µm. In terms of complexity: 150,000 3D connections are performed using µ-bumps between the chiplets and the active interposer, with 20,000 connections for system communication, using the various 3D-Plugs, and 120,000 connections for power supplies using the SCVRs; while 14,000 TSVs are implemented for power supplies and off chip communication. Due to the high level of complexity of the system, 3D assembly sign-off has been performed using the Mentor 3DStack CAD tool [50]. In Fig. 6, we present the respective floorplans of the chiplets and the active interposer. For the chiplet, one can see the 4 computing clusters and associated L1/L2/L3 caches, while for the active interposer one can see the different SCVR, which are supplying power to each individual chiplet, the distributed system interconnects, and the system IOs on the circuit periphery. Dense 3D connectivity is done in various locations of the circuit using the 3D-plug interfaces and associated µbumps. Finally, the overall circuit has been packaged in a Ball Grid Array (BGA) flip-chip package with 10 layers. In addition, one can see the 6 chiplets onto the package, before the final assembly with the cover lead and the package serigraphy.

More details on the various 3D technology element (µbumps, TSVs, RDL, etc.) and 3D assembly steps, with in-depth technology characterization can be found in [START_REF] Coudrain | Active Interposer Technology for Chiplet-Based Advanced 3D System Architectures[END_REF].

IV. COMPUTING CHIPLET ARCHITECTURE

A. Chiplet Overview

The focus of this architecture is its scalability, so we chose to design homogeneous chiplets embedding both processors and caches [START_REF] Guthmuller | A 29 Gops/W[END_REF]. With the current memory mapping, the architecture can be tiled up to 8x7 chiplets, last 2D-mesh row being reserved for off-chip IO accesses, achieving a maximum number of 56 chiplets, for a hypothetical total of 896 cores. The last-level cache is large enough with respect to computing power to release the pressure on the external memory access.

Each chiplet is composed of 4 clusters of 4 32-bit scalar cores (MIPS32v1® compatible ISA) as shown in Fig. 7. System interconnects are extended to 3D using synchronous and asynchronous so-called "3D Plugs". Chiplets form a single fully cache coherent architecture composed of the following elements: separate 16 KB L1 Instruction-cache (I-cache) and Data-cache (D-cache) per core with virtual memory support, a shared distributed L2-cache with 256 KB per cluster, and an adaptive distributed L3-cache, with 4 L3 tiles (4×1 Mbytes) per chiplet.

All clocks (Cluster, L3 tile and interconnect) are generated by 11 Frequency Locked Loop (FLL) clock generators. To mitigate PVT variation, particularly across the dies in a 3D stack, we implement a timing fault methodology for Fmax/Vmin tracking [START_REF] Miro-Panades | In-situ Fmax/Vmin tracking for energy efficiency and reliability optimization[END_REF]. Finally, chiplets are tested using IEEE1687 IJTAG, compressed full-scan, memory Built-In Self Test (BIST), and boundary scan for 3D IOs test, to allow for Known Good Die (KGD) assembly, as explained section VIII.

B. Chiplet Interconnects and their extension

Four different system level interconnects (N0 to N3) make up the system communication infrastructure as shown in Fig. 7, three of which are extendable in 3D: (N0) within cluster, a NoC crossbar allows the communication between the four cores through I&D caches and Network Interface; (N1) between L1 and L2 caches, a 5-channel 2D-mesh interconnect implements the coherency protocol and is routed in the interposer through passive links (two on each side); (N2) between L2 and L3 tiles, Fig. 6. Chiplet and Active Interposer floorplans, details of the 3D-plug µ-bumps, final 3D integration and package a 2-channel 2D/3D-mesh interconnect; (N3) between L3caches and off-chip DRAM memory, a 2-channel 2D/3D-mesh interconnect. (N1) 2D-mesh is fully extended to other chiplets for maximum throughput and lowest short-reach latency as shown in section VII. Peripherals are also connected to this network, which conveys IO requests. (N2) and (N3) networks implement a hierarchical routing scheme where a single router among the four of the chiplet 2D-mesh is used to reach the active interposer. This architecture reduces the 3D footprint for N2 and N3 networks, which are less bandwidth demanding. Using asynchronous logic for N2 3D plug allows for low latency L2 to L3 communications.

(N3) L3->DRAM 2D-Mesh Interconnect (N1) Coherent L1<->L2 2D-Mesh Interconnect

C. System memory mapping

The memory hierarchy is distributed and adaptive (Fig. 8): the 1 TB memory space is physically distributed among L2caches accessed through (N1) network. Cluster coordinates in the 2D-mesh are encoded in the 8 most significant bits of the address, forming a Non Uniform Memory Architecture (NUMA), as done in [START_REF] Olofsson | Kickstarting "Highperformance Energy-efficient Manycore Architectures with Epiphany[END_REF]. Due to the X-first routing scheme of (N1) network, access to IO controllers located in the external FPGA is done through the North port of the (X=3,Y=5) router found in upmost right (X=1,Y=2) chiplet. Thus these IOs are mapped at [0x3600000000:+40GB] memory segment. As mentioned previously, the processor implements a NUMA memory hierarchy, and the most significant bits of the physical address designates the coordinates of the L2 cache containing that data. To improve performance, the Operating System (OS) needs to map data and code close to the cores that use them. OS can do that through the virtual memory subsystem by the mapping of physical memory pages. To assist the OS in this task, our architecture implements two hint bits in the page table: the local (L) and remote (R) bits. They are automatically set by the MMUs and signal if a given physical memory page has been accessed by a core in the same (local) or different (remote) cluster than the L2 cache that hosts the cache lines in that page, also called the "home node". For instance, pages with the R bit set but the L bit unset are candidates for migration.

2) L2 caches

L2 caches are 16-way 256kB set associative write-back caches handling 64 bytes cache lines. The scalable cache coherence protocol exploits the fact that shared data are mostly either sparsely shared read-write or widely shared read-only [START_REF] Dumas | Dynamic Coherent Cluster: A Scalable Sharing Set Management Approach[END_REF] [START_REF] Fu | Coherence domain restriction on large scale systems[END_REF]. Thus, L2 caches maintain L1-caches, TLBs and IO coherence using a Directory-based Hybrid Cache Coherence Protocol (DHCCP) based on write-through L1 caches. L2cache lines have two states: a list mode where coherence is maintained by issuing multicast updates/invalidates; a counter mode where only broadcast invalidates are sent for this line.

In list mode, the sharers' set of this line is stored as a linked list: the first sharer ID is in the L2 directory and the following in another memory bank (heap). When a core writes to this line, the respective home L2 cache sends update messages to sharers, thus keeping their copy up to date.

When the number of sharers reaches a predefined threshold (4 in our implementation) or if the heap is full (4096 entries in our implementation), the cache line is put in counter mode where only the sharers' count is stored and broadcast invalidates are issued. The (N1) 2D-mesh and (N0) crossbar NoCs provide hardware support for broadcast and only L1 sharers of this line answer the broadcast, thus limiting the impact of broadcasts on scalability.

This hybrid sharing set representation is efficiently handling both main categories of shared data [START_REF] Dumas | Trace-driven exploration of sharing set management strategies for cache coherence in manycores[END_REF]. Write-through associated update messages also mitigate false sharing problems. The L2-cache coherence directory represents only 2% of die area with 15 bits core/cache IDs, showing the scalability of the cache coherence protocol. Section X.C shows scalability results for up to 512 cores.

3) L3 caches

L3-cache tiles are 16-way 1MB set associative write-back caches handling 128 bytes cache lines with one dirty bit per 64 bytes block. Tiles are dynamically allocated to L2-caches by software, forming a Non Uniform Cache Architecture (NUCA) as presented in Fig. 8. In case of L3 cache overlap, a shared tile behaves as a shared cache: more space is allocated to the most demanding memory segment. By overlapping L3 caches, the L3 cache controller located at output of each L2 cache offers a L3 fault tolerant adaptive repair. The controller uses a list of defective tiles to redirect traffic initially directed at these tiles to known working tiles. More detail on L3 micro-architecture and performance can be found in [START_REF] Guthmuller | Adaptive Stackable 3D Cache Architecture for Manycores[END_REF][29].

V. INTEGRATED VOLTAGE REGULATOR

A. Principle and 3D-staking

Granular power delivery is a key-feature to improve the overall energy efficiency of multi-core processors [START_REF] Meinerzhagen | An energy-efficient graphics processor featuring fine-grain DVFS with integrated voltage regulators, execution-unit turbo, and retentive sleep in 14nm tri-gate CMOS[END_REF]. To The typical input voltage range Vin is 1.5~2.5V to reduce the delivered current Iin from TSV, package and motherboard. Thus, the number of package's power IOs can be reduced compared to a direct Vcore distribution from external VR. The power distributed network loss is also reduced.

B. Circuit design

The switched capacitor voltage regulators (SCVR) have been chosen thanks to their fully-integration capability [START_REF] Meinerzhagen | An energy-efficient graphics processor featuring fine-grain DVFS with integrated voltage regulators, execution-unit turbo, and retentive sleep in 14nm tri-gate CMOS[END_REF][32] [START_REF] Meyvaert | A 1.65W fully integrated 90nm Bulk CMOS Intrinsic Charge Recycling capacitive DC-DC converter: Design amp; techniques for high power density[END_REF][34] [START_REF] Souvignet | A Fully Integrated Switched-Capacitor Regulator With Frequency Modulation Control in 28-nm FDSOI[END_REF]. The chosen topology is parallel-series 3stage gearbox scheme to cover a large Vout range while maintaining power efficiency (Fig. 10). Thus, the SCVR generates 7 lossless voltage-ratio from 4:1 to 4:3. From 1.8Vin, the SCVR provides from 0.35V to 1.35V, which cover the lowto-high chiplet's power modes. The gearbox scheme is interleaved into 10 phases to reduce the Vcore ripple and to increase the control bandwidth. The number of interleaved phases is also chosen to maintain power efficiency at lowvoltage level where required power for chiplet drops off. The feedback control is based on one-cycle hysteresis controller proposed in [START_REF] Andersen | A 10 W On-Chip Switched Capacitor Voltage Regulator With Feedforward Regulation Capability for Granular Microprocessor Power Delivery[END_REF]. The voltage controller is centralized and sequences the charge injection in the interleaved converters at each clock cycle. The clock generation and controller is integrated on-chip.

C. Physical design on interposer

As shown in Fig. 11, each SCVR occupies 50% of the chiplet footprint (11.4mm 2 ) and is composed of 270 regular Unit Cells (UC), with a 0,2mm pitch, in a checkerboard pattern. The I/O device transistor may operate on an up to 3.3V input voltage. A MOS-MOM-MIM capacitor stack maximizes the capacitance density (8.9 nF/mm 2 ) with 102nF flying capacitor per SCVR. To deal with potential process defaults on the large area of the interposer, fault-tolerant protocol is also included to mitigate the effect of defective unit cells on overall power efficiency.

D. Experimental results

As shown in Fig. 12.a, the SCVR achieves a measured 156 and 351mW/mm 2 power density at 82% peak efficiency and similar LDO's efficiency, respectively. SCVR maintains more than 60% from 0.45 to 1.35V covering the full supply voltage range of the chiplet (Fig. 12.c). The SVCR delivers up to 5A output current while maintaining higher efficiency than an LDO (Fig. power density is comparable to other wide range SCVRs. 3D integration of the SCVR on the interposer minimizes the system area and cost, with no impact on the chiplets.

E. Discussion

Since the power efficiency obtained by the integrated VR is lower than external DC-DC converters, the overall power efficiency of the computing system could improve by allowing fine-grain DVFS without increasing the Bill-of-Material (BoM) and IOs numbers. The power density is smaller than previously published results but the converters are fully integrated within the active interposer, not on the same die, thus reducing the cost impact of the proposed active power mesh. The interposer integration opens the opportunity for dedicated post-process high density capacitors (e.g. deep trench capacitors) connected through TSV. We also prove the up-scaling capability of SCVR by fabricating the largest die area SCVR with a built-in capacitor fault-mitigation scheme.

VI. 3D-PLUG COMMUNICATION INTERFACES

A. 3D-Plug features overview

As presented section V.B, the different chiplet system level interconnects are extended throughout the active interposer, by using generic chiplet-interposer interfaces, called 3D-Plugs.

Each 3D-Plug integrates both the logical and physical interfaces. As presented Fig. 14, it contains the micro-bump array, the micro-buffer cells (bi-dir driver with ESD protection, level shifter and pull-up), and boundary-scan logic for Designfor-Test. A bi-directional driver is used to allow testability of the interface before assembly (see section IX). This 3D interface is very similar to the 3D-Network-on-Chip interface, as presented earlier in [START_REF] Vivet | A 4x4x2 Homogeneous Scalable 3D Network-on-Chip Circuit with 326 MFlit/s 0.66 pJ/bit Robust and Fault Tolerant Asynchronous 3D links[END_REF]. Due to the 28nm/65nm technology partitioning, the micro-buffer cell also requires in that case a level shifter in order to bridge the voltage domains between the chiplet (typically 1.0V) to the active interposer (1.2V). In terms of physical design, the different 3D IOs of each 3Dplug have been created and assigned in an array fashion, while the micro-buffer cell has been designed as a standard cell and pre-placed within the pitch of the micro-bumps. All the other parts of the 3D-Plug (their logical interface and DFT) have been designed using automated place & route.

In order to build the system level interconnects of INTACT, different kind of 3D-plug have been designed, as presented in Table III below. Due to the different natures of the interconnects, in terms of traffic and distance/connectivity, two different kinds of 3D-Plugs have been designed, and compared in detail: one using synchronous design, as presented section VII.B, and one using asynchronous design, as presented section VII.C.

B. 3D-Plug Synchronous version

The microarchitecture of the source synchronous 3D-Plugs used for 2.5D passive (N1 NoC When attached to the 3D vertical active link, the 3D-Plug achieves 3 Tb/s/mm2 bandwidth density, 1.9x higher than [START_REF] Lin | A 7nm 4GHz Arm®-core-based CoWoS® Chiplet Design for High Performance Computing[END_REF]. 2.5D passive links reach a 12% higher bandwidth cross-section than [START_REF] Lin | A 7nm 4GHz Arm®-core-based CoWoS® Chiplet Design for High Performance Computing[END_REF] as shown in Fig 13 .b. The aggregate synchronous 3D/2.5D links bandwidth is 527 GB/s.

We performed a frequency, logic voltage and clock phase sweep on synchronous 2.5D/3D links. All 2.5D passive links were able to reach at least 1.25 Gb/s/pin in the [0.95V-1.2V] VDD range and the best link shown in Fig. 15. was able to reach this bandwidth at 1V, while reaching more than 1.6 Gb/s/pin at 1.2V. We obtained best results with a 180° CLK_TX_Φ1 phase and varying CLK_TX_Φ2 phase depending on frequency. While much shorter than passive links, 3D vertical links achieve slightly lower data rates of 1.21 Gb/s/pin upward and 1.23 Gb/s/pin downward as one side of these 3D-Plugs is implemented in the more mature and slower 65nm technology of the interposer.

C. 3D-Plug Asynchronous version

For its inherent robustness to any source of timing variations and low latency [START_REF] Thonnart | Latency Improvement of an Industrial SoC System Interconnect using an Asynchronous NoC Backbone[END_REF], asynchronous logic is well adapted for designing system level interconnects and Network-on-Chip architectures in a Globally Asynchronous Locally Synchronous (GALS) scheme. In the context of 3D architectures, asynchronous logic and its local handshakes enables interfacing two different dies without any clocking issues. By using robust Quasi Delay Insensitive (QDI) logic, an Asynchronous 3D Network-on-Chip has been earlier presented in [START_REF] Vivet | A 4x4x2 Homogeneous Scalable 3D Network-on-Chip Circuit with 326 MFlit/s 0.66 pJ/bit Robust and Fault Tolerant Asynchronous 3D links[END_REF] but presents some 3D throughput limitations due to the 4-phase handshake protocol.

For INTACT, an innovative 3D-Plug interface has been designed, to benefit from 2-phase handshake protocol at the 3D interface, which reduces the penalty of 3D interface delay within the interface cycle time, and thus increases the 3D interface throughput. As introduced in [START_REF] Hilgemberg Pontes | Two-phase Protocol Converters for 3D Asynchronous 1-of-n Data Links[END_REF], the principle is as follows (Fig. 16) : -Use asynchronous 2-phase protocol for 3D interface communication, to reduce 3D interface delay penalty, -Use asynchronous 4-phase protocol for on-chip communication, within the active interposer, for its inherent simplicity, low latency and performance [START_REF] Thonnart | Latency Improvement of an Industrial SoC System Interconnect using an Asynchronous NoC Backbone[END_REF], -Introduce a protocol converter, from 2-phase protocol to 4phase protocol and respectively, using an ad-hoc asynchronous logic encoding. A recent overview of asynchronous logic and signaling can be found in [START_REF] Nowick | Asynchronous Design-Part 1: Overview and Recent Advances[END_REF]. For implementing a low cost protocol converter, a 2-phase 1T-of-N multi-rail transition based signaling is used [START_REF] Hilgemberg Pontes | Two-phase Protocol Converters for 3D Asynchronous 1-of-n Data Links[END_REF], with N=4 (4-rail encoded, thus 4 wires for 2 bits). In this encoding and 2-phase protocol, one single transition on Raili indicates the i value, which is then acknowledged by a transition on the feedback path. This encoding is close to the 1-of-n on-chip protocol, which leads to the corresponding protocol converters, shown in Fig. 17 Similarly to the synchronous 3D-plug interface, the protocol converter also integrates all the 3D objects: micro-bumps, micro-buffers, and boundary scan (Fig. 16). Finally, since the same number of wires is used for both protocols, a bypass mode of the protocol converters is added, configuring the 3D-plug interface either in 2-phase mode or in 4-phase mode, for circuit measurements.

VII. ACTIVE INTERPOSER SYSTEM INTERCONNECTS

A. Overview

Different kinds of system interconnects have been implemented between the chiplets on the interposer, using the 3D plugs described in the previous section. These interconnects are used to transport the different levels of cache coherence in the memory hierarchy. As discussed earlier, a mix of synchronous and asynchronous implementations was used, depending on latency and power targets. The structure of the different interconnects is shown in Fig. 18, with clock-domain crossings, conversion interfaces, pipelining and routers. These three interconnects will be detailed in the next paragraphs. To assess their performance, on-chip traffic generators and probes were inserted in the chiplets networks on chip, for throughput and latency measurements.

B. L1-L2 cache interconnect

Fig. 18a presents the first level of cache interconnect between local L1 and distributed L2 caches (N1 NoC). As this first level of cache traffic is intended to be localized using an adequate application mapping, most of the traffic is expected to be exchanged between neighboring chiplets. Aside from clockdomain crossing between the two chiplets using synchronous 3D-Plugs, no other adaptation is required, and routing is entirely performed within the chiplets. Therefore, only passive metal wires are used on the interposer to connect the microbumps of neighboring chiplets.

Physical design of these interposer passive links was optimized to reduce delay and crosstalk between the nets. A dedicated routing scheme on two levels of metal was used (M3-M5 horizontal, M2-M4 vertical), with trace widths of 300nm and spacing of 1.1μm. Additional shielding was used for clock nets running at twice the datarate. Crossings with minimumwidth unrelated wires on the interposer showed very little impact on crosstalk or delays in the signal, and were therefore allowed on the other interposer metal levels.

Point-to-point connection between two adjacent 3D-Plugs was measured at 1.25 GHz, with a latency of 7.2 ns. Most of this latency is due to the clock-domain crossings in 3D-Plugs.

For large applications, nevertheless, L1-L2 cache coherence traffic needs to extend farther than between adjacent chiplets. In that case, pipelining and routing is handled by the intermediate chiplets. The main advantage in this case is that this is done using the advanced technology node in the chiplets, which has better performance and lower power consumption than the interposer does. However, the major drawback is the accumulation of pipeline and clock-domain crossings, which adds extra latency for distant L1-L2 traffic.

The 2D NoC frequency in the chiplet runs at 1 GHz, but the one-way latency from the source 3D-Plug to the destination 3D-Plug can be as high as 44 cycles on the longest path from chiplet 00 to chiplet 12, with two intermediate chiplets, five routers and 8 to 10 FIFO stages between routers. Nevertheless, this solution is very energy efficient with only 0.15pJ/bit/mm.

C. L2-L3 cache interconnect

Fig. 18b presents the second level of cache interconnect between distributed L2 and L3 caches (N2 NoC). The main performance target is in this case to offer low latency long reach communication. For this purpose, it was chosen to implement it in fully asynchronous logic on the interposer, using the ANoC quasi-delay-insensitive network-on-chip [START_REF] Thonnart | Latency Improvement of an Industrial SoC System Interconnect using an Asynchronous NoC Backbone[END_REF]. This allows for only two synchronous/asynchronous conversions on an end-toend path, to save on clock-domain-crossing latency. Deeppipelining on the ANoC allows to insert an asynchronous pipeline stage every 500μm to preserve throughput with almost no impact on the latency compared to inverter-based buffering.

The asynchronous 3D-Plug in two-phase mode allows an injection rate in the network for 72-bit data words up to 520 MHz, while the 2D NoC is able to sustain up to 0.97 GHz on every link, which limits the in-network contention of overlapping network paths. The efficient asynchronous pipeling allows an end-to-end latency on the synchronous interfaces of the 3D-Plugs of only 15.2 ns, with 4 clock cycles and 11.2 ns of asynchronous latency across 4 routers and 25mm of pipelined links.

D. L3-ExtMemory Interconnect

Fig. 18c presents the last interconnect between the distributed L3 caches and the external memory (N3 NoC). Considering the intrinsic contention of this last level of cache traffic, and the longer latency for paginated access to the external memory, the focus was put on energy efficiency, then on low latency. This interconnect is implemented as a global synchronous NoC, with clock-domain crossings at the source 3D-Plug and in the memory IO interface. Two-stage FIFOs are inserted every 1mm, and tight clock-tree balancing was performed to increase the throughput. This results in a 72-bit synchronous network running up to 750MHz, with a latency of 2ns/mm, for a good energy efficiency of 0.24pJ/bit/mm.

E. System Interconnect comparison and conclusions

Table IV summarizes the different figures of merit for the three interconnects, and provides a benchmark with respect to the 3D NoC in [START_REF] Vivet | A 4x4x2 Homogeneous Scalable 3D Network-on-Chip Circuit with 326 MFlit/s 0.66 pJ/bit Robust and Fault Tolerant Asynchronous 3D links[END_REF]. It shows that neighboring connections can be efficiently made using the synchronous 3D-Plug in an advanced technology node, with a high throughput and a low power consumption. For longer-range communication, limiting the number of clock-domain crossings is key for performance. The networks on chip in the active interposer can provide wide interconnects optimized for latency in the asynchronous version, with 0.6ns/mm, or for power consumption in the synchronous version, with 0.24pJ/bit/mm, with performance metrics twice as good as [START_REF] Vivet | A 4x4x2 Homogeneous Scalable 3D Network-on-Chip Circuit with 326 MFlit/s 0.66 pJ/bit Robust and Fault Tolerant Asynchronous 3D links[END_REF] in the same 65nm technology node as the active interposer.

The achieved low level interconnect performances could be used for a more systematic system level study, such as [START_REF] Yin | Modular Routing Design for Chiplet-Based Systems[END_REF], by trading off different traffic classes, latency, and energy, thanks to the extended active interposer traffic capabilities.

VIII. ACTIVE INTERPOSER TESTABILITY AND 3D DESIGN-FOR-TEST ARCHITECTURE

A. Testability Challenges

With such 3D active interposer, testability is raising various challenges. First, it is required to ensure Know Good Die (KGD) sorting to achieve high system yield [10]. This implies that the 3D test architecture must enable EWS test of the chiplet and the interposer (pre-bond test, before 3D assembly), and final test (post-bond, after 3D assembly in the circuit package). Moreover, due to fine pitch µ-bumps, reduced test access is observed, µ-bumps cannot be directly probed in test mode. This implies to include additional IO pads, which are only used for test purpose, and not in functional mode (see Fig. 19). Finally, with 3D technologies, additional defects may be encountered, such as µ-bumps misalignments, TSV pinhole, shorts, etc. which lead to specific care for testing the 3D objects and interfaces. Another concern is also regarding the Automatic Test Pattern Generation (ATPG) engineering effort, where easy re-targeting of test patterns from pre-bond test to post-bond test should be proposed to reduce test development efforts.

Numerous researchers have addressed specific test solutions for 3D defaults, see for instance [START_REF] Reddy | A Cost-Aware Framework for Lifetime Reliability of TSV based 3D-IC design[END_REF][41], for testing generic 3D architectures using die wrappers and elevators [START_REF] Papameletis | A DfT Architecture and Tool Flow for 3-D SICs with Test Data Compression, Embedded Cores, and Multiple Towers[END_REF], and for testing 2.5D passive interposers [START_REF] Goel | Test and debug strategy for TSMC CoWoS™ stacking process based heterogeneous 3D IC: A silicon case study[END_REF]. A standardization initiative on 3D testability has emerged with the recent IEEE 1838 standard [44]. Nevertheless, no work addressed initially the testability of active interposers.

B. 3D Design-for-Test Architecture

Within the INTACT architecture, the test of the 3D system must address the test of all the following elements: i) the regular standard-cell based logic, ii) all memories using BIST engines and Repair, iii) the distributed 3D interconnects and IOs: 3D connections of active links and passive links, which are implemented by micro-bumps, and finally iv) the regular package IO pads for off-chip communication through the TSVs.

In order to test the Active Interposer and its associated chiplets, the proposed 3D Design-for-Test architecture (Fig. 20) is based on the two following main Test Access Mechanisms (TAMs), as proposed earlier in [START_REF] Durupt | IJTAG supported 3D DFT using Chiplet-Footprints for testing Multi-Chips Active Interposer System[END_REF]: -A IJTAG IEEE1687 hierarchical and configurable chain, accessed by a primary JTAG TAP port, for testing all the interconnects and memories, based on the concept of "chiplet footprint", -A Full Scan logic network using compression logic, for reduction of test time and of number of test IOs. By using IJTAG IEEE 1687, the JTAG chain is hierarchical and fully configurable: the JTAG chain provides dynamic access to any embedded test engines. The active interposer JTAG chain is designed similarly to a chain of TAPs on a PCB board. It is composed of "chiplet footprints", which provide either access to the above 3D-stacked chiplet or to the next chiplet interface, and which are chained serially. The JTAG network is used to test and control the 3D active links, the 3D passive links, the off-chip interfaces, and the embedded test engines, such as the memory BISTs.

The Full Scan logic network offers efficient and parallel full scan test of the whole 3D system logic. In order to reduce the number of 3D parallel ports, compression logic is used in both the chiplets and the active interposer, with a classical tradeoff (shift time/pin count). Independent scan paths are used between the chiplets and the active interposer, to facilitate the test architecture integration.

C. Test CAD Flow and Test coverage

The proposed 3D Design-for-Test architecture has been designed and inserted using Tessent TM tools from Mentor, a Siemens Business. By using IJTAG and IEEE1687, high level languages such as "Instrument Connectivity Language" (ICL) and "Procedural Description Language" (PDL) are provided and enable to handle the complexity of such a system. In particular, it is possible to fully-automate the test pattern generation of Memory BIST engines, from ATPG at chiplet level to ATPG of the same patterns within the full 3D system, enabling so-called test pattern retargeting. As presented in Table V, full testability is achieved for all logic, 3D interconnects and regular package IOs, and memory BIST engines, before 3D assembly and after 3D assembly.

Using the proposed DFT architecture & test patterns, the full system was tested using an Automated Test Equipment (ATE): -The 28nm chiplet has been tested at wafer level using a dedicated probe card, with a binning strategy. -The active interposer has not been tested at wafer level, supposing the maturity of the 65nm technology and its high yield due to its low complexity (see section IV.B). Nevertheless, its standalone DFT and dedicated IOs were initially planned and designed as mentioned above. -The full INTACT circuit, after 3D assembly and packaging, has been tested within a dedicated package socket.

IX. THERMAL CHALLENGES AND STRATEGY

A. Thermal Challenges In 3D technology, thermal dissipation is a challenge that needs to be properly addressed. Due to more integration in a smaller volume, a larger power density is usually observed in 3D, while the thermal dissipation itself is getting more complex in the overall 3D stack of the circuit and package, overall leading to thermal hotspots or even thermal runaway [START_REF] Torregiani | Thermal analysis of hot spots in advanced 3D-stacked structures[END_REF]. In the generic context of logic-on-logic stacking, thermal dissipation is worse because multiple layers of compute dies need to dissipate their heat on top of themselves. On the contrary, in the case of interposer based systems, a single layer of chiplets is dissipating heat, while heat extraction can be performed from the top package face, similarly to a regular flipchip packaged circuit. Nevertheless, contrarily to passive interposers, in the case of an active interposer, the bottom layer is also part of the power budget, and dissipates heat as well. Since the power budget of the active interposer layer is rather limited, with most power budget within the chiplets, this should help the overall thermal dissipation.

Finally, due to the heterogeneous structure of such a 3D stack, many materials are composing the device, with silicon substrate, Back-End of Line (BEOL) in copper, underfill composite materials between the chiplets and interposer, microbumps (SnAg), TSVs (copper), etc. This assembly leads to strong anisotropic thermal effects, favoring and increasing thermal hotspots effects. Moreover, due to the thin layer effect of the interposer (100µm), the horizontal thermal dissipation is reduced in the interposer, while it remains mostly the vertical thermal dissipation through the chiplets. These various thermal effects have been widely studied in the literature [START_REF] Harris | Thermal effects of heterogeneous interconnects on InP/GaN/Si diverse integrated circuits[END_REF] [START_REF] Santos | Thermal performance of 3D ICs: Analysis and alternatives[END_REF], and need to be taken into account in the full system.

B. Thermal modeling strategy

With all the 3D thermal challenges: increased power density and design complexity on the design side, fine grain material effects on the technology side, coupled to the regular package and board thermal information, an accurate thermal exploration must be performed with the adequate thermal methodology. Various thermal tools are available: either circuit level tools able to cope with detailed circuit and technology description but with simple packaging condition, or package level tools able to cope with detailed packaging, but with reduced die and technology information. In order to achieve an accurate thermal exploration covering all modeling aspects, the Project Sahara solution, a thermal analysis prototype from Mentor Graphics a Siemens Business, was selected [START_REF]A Complete Guide to 3D Chip-Package Thermal Co-Design, 10 key considerations[END_REF].

As presented Fig. 21, an adequate thermal methodology has been setup to allow modelling of low level structures (TSV, micro-bumps, underfill), with a design entry at GDS level and with accurate static or dynamic power maps, all this in the context of the full system (package and fan). The methodology has been qualified on a previous 3D logic-on-logic design with silicon thermal measurements 16 [START_REF] Vivet | A 4x4x2 Homogeneous Scalable 3D Network-on-Chip Circuit with 326 MFlit/s 0.66 pJ/bit Robust and Fault Tolerant Asynchronous 3D links[END_REF]. More details of the thermal methodology can be found in [START_REF] Santos | Thermal Exploration and Sign-off analysis for advanced 3D integration[END_REF].

C. Thermal simulation results.

The INTACT circuit and package has been modelled, as presented Fig. 22 with a detailed cross-section of the 3D circuit. In terms of power budget, a scenario with a maximum static power budget of 28 Watts is simulated, corresponding to a worst-case situation of 3 Watts per chiplet (x6) and 10 Watts in the active interposer, while the nominal circuit power budget is 17 Watts as presented in section X.B. As a result, Fig. 23 shows the thermal exploration, without Heat Sink (max temperature 150°C), with a regular Heat Sink & Fan (max temperature 53°C), while no hotspots appear within the computing chiplet. Even for this worst-case scenario, due to a still limited power density of 0.14W/mm 2 , the thermal dissipation of the active interposer can be achieved using a regular heatsink and fan. As shown in Fig. 24, a complete development board has been designed for measurement and application evaluations including running Linux on the chip. The board features two FPGAs with a 2x16 GB 64-bit DDR4 memory and various peripherals: 8x PCIe Gen3, SATA, 1Gb Ethernet, 10Gb Ethernet, HDMI, USB, SD-Card and UART. The demonstration board also features a power infrastructure with voltage and current sensing. Each FPGA is connected to 2 of the 4 LVDS links of the chip.

A. Circuit Performances

The chiplet is functional in the 0.5V-1.3V range with Forward Body Biasing (FBB) [START_REF] Beigné | A 460 MHz at 397 mV, 2.6 GHz at 1[END_REF] up to ±2V. Fig. 25.a shows that a core frequency of 1.15 GHz is achieved @ 1.3V with 0/+1 (VDDS/GNDS) FBB. Single core performance is 2.47 Coremark/MHz and 1.23 DMIPS/MHz. At chip level, maximum energy efficiency is 9.6 GOPs/Watt on Coremark benchmarks (IPC=0.8/core) @ 0.6V taking into account voltage regulation losses in the interposer as shown in Fig. 25.c. As expected, FBB boosts performance: in typical @ 0.9V, a frequency increase of 24% is achieved with -1/+1 FBB, while in typical @ 680MHz, an energy efficiency increase of 15% is achieved with asymmetric 0/+1 FBB.

B. Circuit Power budget and energy efficiency

In Fig. 25.b and Fig. 25.c we show overall chip power and performance measurements with a 0/+1 FBB. Power consumption and energy efficiency while running Coremark benchmark is compared to a theoretical system using a digital LDO instead of the proposed fully integrated SCVR. Using an LDO at the same VIN = 2.5V would result in a 2x increase in power consumption, a lower VIN would be needed to limit losses at the expense of more power pins and voltage-drop issues. The power breakdown in Fig. 26 shows the low power budget of the active interposer with only 3% of total power consumed by the active interposer logic. The cores+L1$ represent over half the power consumption of the chiplets, themselves consuming the majority of the measured circuit power (17 Watts).

C. Circuit Scalability

Lastly, Fig. 27 shows the scalability of the cache-coherent architecture that is analyzed by running a 4 Mpixels image filtering application from 1 to 512 cores. The filter is composed of a 1D convolution, followed by a transposition of the image and ends with another 1D convolution. Software synchronization barriers separate these steps and the transposition, in particular, involves many memory transfers. Results for more than 96 cores were obtained by RTL simulation with additional chiplets. Software is executed on a single cluster up to 4 cores and on a single chiplet up to 16 cores. Compared to a single core execution, a 67x executiontime speedup is obtained with 96 cores and 340x with 512 cores. The slight uptick above 128 cores results from the threshold where the dataset fits in caches. This quasi-linear speedup, ignoring limitations of the external memory bandwidth, shows the scalability of network protocols and their 3D implementations.

D. Comparison to prior art

Compared to prior art (Table VI), the INTACT circuit is the first CMOS active interposer validated on silicon, which offers a chiplet-based many-core architecture for high performance computing. The active interposer solution allows for integrated voltage regulators without any external passives, using free die area available in the active interposer, offering DVFS-perchiplet and achieving 156 mW/mm 2 at 82% peak power efficiency, with 10-50% more efficiency with respect to LDO converters integrated in organic schemes. The SCVR is also fault tolerant to mitigate the effect of defective unit cells on the overall power efficiency.

Regarding interconnects, contrary to previous point-to-point solutions, the active interposer offers flexible and distributed NoC meshes enabling any chiplet-to-chiplet communication for scalable cache-coherency traffic, with 0.6 ns/mm inter-chiplet latency using asynchronous signaling within the interposer, and a 0.59 pJ/bit synchronous 3D-Plug energy efficiency with 3 Tb/s/mm2 bandwidth density, which is twice better than previous circuits.

The overall system integrates a total of 96-cores, in 6chiplets, offering a peak computing power of 220 GOPs (peak mult-acc), which is quite comparable to advanced state of the art processor systems. Finally, the overall distributed interconnects and cache coherency memory architecture are scalable up to 896 cores, showing the architecture partitioning capability to target larger computing scale. Séverine Chéramy holds an engineering degree having specialized in material science. She has spent over eight years at GEMALTO, a leading smart-card company developing technologies for secure solutions such as contactless smart cards & electronic passports. In 2008, she joined CEA-Leti as 3D project leader, and then as 3D Integration laboratory manager. This group develops technology and integration for 3DIC, in strong relationship with 3D design, model and simulation teams. Since January 2017, she's responsible for 3DIC integration strategy and related business development. She is also director of the 3D project of ITR (Institute of Technological Research) Nanoelec.

Fabien Clermidy received his Ph'D degree and Thesis Supervisor degree from INPG, respectively in 1999 and 2010. In 2000, he joined the CEA-LIST in Paris, where he was involved in the design of an application-specific parallel computer. In 2003, he joined the CEA-LETI in Grenoble, in the digital circuit laboratory, where he led the design of various large many-core circuits. His research interest covers wide scope of digital systems : many-core architecture, network-on-chip, energy efficient design, embedded systems, interaction with advanced technologies. He is currently at the head of Digital System and Circuit Division at CEA-LIST, a CEA institute. He has published more than 80 papers in international conferences and is author or co-author of 14 patents.

Fig. 1 .

 1 Fig. 1. Chiplet partitioning concept

Fig. 4 .

 4 Fig. 4. INTACT overall circuit architecture

Fig. 5 .

 5 Fig. 5. INTACT : from concept to 3D-cross section

Fig. 7 .

 7 Fig. 7. Chiplet architecture, offering a 16-core scalable coherent fabric

Fig. 8 .

 8 Fig. 8. Memory mapping and cache allocation D. Cache Coherency Scalability 1) L1 caches Each core has a private L1 cache that implements a Harvard's architecture with separate 4-way 16 KB cache memories for instruction (I) and data (D) with 64 bytes cache lines. L1 Dcaches are write-through and implement a fully associative write buffer composed of 8 128-byte entries flushed either on explicit synchronization or on expiration of a 8-cycle timer. L1 I/D-caches include a MMU (Memory Management Unit), which consists of two, per-core private, fully associative, 16entry TLBs (Translation Lookaside Buffers) for instruction and data. MMUs with coherent TLBs translate the 32-bit virtual address space (4 GB) in processor cores onto the 40-bit physical address space (1 TB) mapped as shown in Fig. 8. The hardware guarantees the coherency of both L1 I/D-caches and both I/D-TLBs (see next section).As mentioned previously, the processor implements a NUMA memory hierarchy, and the most significant bits of the physical address designates the coordinates of the L2 cache containing that data. To improve performance, the Operating System (OS) needs to map data and code close to the cores that use them. OS can do that through the virtual memory subsystem by the mapping of physical memory pages. To assist the OS in

  allow DVFS per-chiplet, fast transitions, and mitigate IR-drop effects, 6 integrated voltage regulators (VR) have been included in the interposer layer which individually supply each chiplet by Vcore from Vin as shown in Fig.9. The power is delivered through the µ-bump flip-chip matrix. The Vin voltage is delivered from the interposer back-face through a 40 µm-pitch TSV array. The VRs are fully integrated into the active interposer without needing any external component.

Fig. 9 .

 9 Fig. 9. Switched Capacitance Voltage Regulator (SCVR) cross-section.

Fig. 10 .

 10 Fig. 10. SCVR unit-cell schematics and hierarchy

Fig. 11 .

 11 Fig. 11. SCVR layout on the interposer

  12.b). The peak power efficiency is relatively constant against Vin typical range. As shown in Fig 12.d, the feedback control achieves less than 10ns step response for a middle-tozero load transient (0.8 A to 0 A), while the full load is defined at peak efficiency (1.2 A).

Fig. 12 .

 12 Fig. 12. SCVR experimental results: a) power efficiency vs voltage conversion ratio and gearbox configurations, b) efficiency over output current, c) efficiency vs input voltage at 2:1 ratio, d) load transientTable II compares the 3D stacked SCVR to some previously published SCVR in 2D-context. The proposed SCVR exhibits the highest number of lossless ratio and the highest delivered power with a commonly available capacitor to enable widely spread use. Even if the SCVR is affected by TSV grid, the

Fig. 14 .

 14 Fig. 14. 3D-Plug physical and logical interface overview

  Fig. 13. a) Synchronous 3D-Plug micro-architecture and b) comparison to state of the art

Fig. 16 .

 16 Fig.[START_REF] Gomes | Lakefield and Mobility Compute: A 3D Stacked 10nm and 22FFL Hybrid Processor System in 12×12mm2, 1mm Package-on-Package[END_REF]. 3D Plug asynchronous version overview, composed of protocol converters between the on-chip communication and the 3D interface.

  .

Fig. 17 .

 17 Fig. 17. 3D-Plug asynchronous version details, composed of a) 4-phase to 2phase protocol converter, and b) 2-phase to 4-phase protocol converter.

Fig. 15 .

 15 Fig. 15. Synchronous 3D-Plug max data rate for 2.5D passive links

Fig. 19 .

 19 Fig. 19. Chiplet layout (zoom), with 3D-Plug interface and additional test pads

Fig. 20 .

 20 Fig. 20. 3D Design-for-Test architecture for INTACT, overview and detailed

Fig. 21 .

 21 Fig. 21. 3D chip-package thermal flow, from early exploration to sign-off.

Fig. 22 .

 22 Fig. 22. INTACT circuit and package cross section used for thermal modelling.

Fig. 24 .

 24 Fig. 24. Development board fitting in a standard PC case

Fig. 25 .

 25 Fig. 25. a) Maximum core frequency, b) Power consumption at Fmax (FBB=(0,1)), c) Power efficiency at Vmin

Fig. 23 :Fig. 26 .

 2326 Fig. 23 : a) Package temperature (without Heat Sink) b) Package temperature (with Heat Sink & Fan) c) Chiplet thermal map Peak temperature = ~150°C Peak temperature = ~53°C Peak temperature = ~55°C

Fig. 27 .

 27 Fig. 27. Execution speedup up to 512 cores

companies and 10

 10 academic laboratories. He is the chief architect of the scalable, shared memory, manycore TSAR architecture, and is presently working on scalable Operating Systems for those kind of machines.QuentinL. Meunier received the diploma from the Ensimag school (Grenoble, France) in 2007 and a Ph.D. degree in Computer Science from Université de Grenoble (France) in 2010. Since 2011, he has been associate professor at Sorbonne Université, in the LIP6 laboratory (Paris, France). His research interests include manycore architectures and cache coherence, high-performance computing, and side-channel attacks and counter-measures. Alexis Farcy graduated in electronic engineering in 2000 from the "Institut des Sciences et Techniques de Grenoble", France, and was employed by STMicroelectronics (Crolles). From 2000 to 2007 he was among the Advanced Interconnects and Passive Components Module, focusing on interconnect performance analysis for advanced technology nodes, integration of advanced inductors and 3D capacitors in BEOL, and high-frequency characterizations of low-k and high-k dielectrics. He received the PhD degree in Electronic, Optronic and Systems from the University of Savoie, France, in 2009. Since 2007 he works in the field of 3D integration on innovative technologies, processes and materials for 3D integration and performance assessment for Photonics and image sensors. Alexandre Arriordaz is a Senior Product Engineering Manager for Calibre Design Solutions at Mentor, A Siemens Business. He is leading Product Management and Software Development teams located in Grenoble (France) focusing on Circuit Reliability & Verification product lines. In parallel to this activity, he is also a technical interface for various European projects dealing with R&D topics like 3D-IC or Silicon Photonics. Prior to joining Mentor, Alexandre was a fullcustom design engineer at Freescale Semiconductor (now NXP), working on advanced testchip/SRAM compiler developments. He holds a Masters degree in Electronics from the University de Nice-Sophia-Antipolis (France).

TABLE I :

 I INTACT MAIN CIRCUIT FEATURES AND 3D TECHNOLOGY DETAILS

Chiplet technology FDSOI 28nm, 10 metals, 0.5V-1.3V+adaptive biasing Chiplet area 4.0 mm x 5.6 mm = 22.4 mm 2 Chiplet complexity 395 Million transistors, 18 transistors/µm 2 density Interposer tech. CMOS 65nm bulk, 7 metals, MIM option, 1.2V Interposer area 13.05 mm x 15.16 mm = 197.8 mm 2 Interposer complexity 15 Million transistors, 0.08 transistors/µm 2 density 3D technology

		Face2Face, Die2Die assembly onto active interposer
	µ-bump technology Ø10µm, pitch 20µm
	#µ-bumps	150 000 (20k signals + 120k powers + 10k dummies)
	Inter-chiplet distance 800µm
	TSV technology	TSV middle, Ø10µm, height 100µm, pitch 40µm
	#TSV	14 000 TSV (2 000 signals + 12 000 power supply)
	Backside RDL	10µm width, 20µm pitch
	C4-bumps	Ø90µm, pitch 200µm, 4,600 bumps
	Flipchip package	BGA 39 x 39, 40mm x 40mm, 10 layers
	Balls	Ø500µm, pitch 1mm, 1 517 balls

TABLE II :

 II COMPARISON WITH COMPARABLE SCVR USING MOS OR MIM CAPACITOR TECHNOLOGY

	Reference	[32]	[33]	[35]	This work	Unit
	Integration context	2D	2D	2D	3D	-
	CMOS tech.	65	90	28	65	nm
	Capacitor tech.	MOS	MOS	MIM	MOS+MIM	-
	Ratio	2	1	4	7	-
	Interleaving	8	21	8	10	-
	Module area	1.11	2.14	0.46	11.4	mm 2
	Flying cap density	5.5	5.6	11.7	8.9	nF/mm 2
	Vin range	2.3	2.3~2.6	1.8	0.9~2.9	V
	Vout range	0.8~1.2	1.0	0.2~1.1	0.4~1.8	V
	Max power	0.67	0.12	-	2.6	W
	Step response	250	15,000	200	10	ns
	Droop voltage	150	95	100	20	mV
	Peak efficiency	71	69	73	81	%
	Power density	550	770	310	156	mW/mm 2

TABLE III :

 III 3D-PLUG TYPES AND USAGE IN INTACT

	System Level Interconnect	3D-Plug type	Active interposer link type
	L1L2 (N1 NoC)	Synchronous version Short Reach, passive link
	L2L3(N2 NoC)	Asynchronous version Long reach, active link
	L3ExtMem (N3 NoC) Synchronous version Long reach, active link

TABLE IV

 IV

	1.0V	Sync.	Sync.	Sync.	8 FIFOs	Sync.	Sync.			Sync.
	Chiplet 00	3DPlug	3DPlug	Router		Router	3DPlug	Chiplet 11	3DPlug	Chiplet 12
	Interposer
		1.5 mm		5 mm
	a) L1 -L2 cache interconnect with passive nearest-neighbor links
	Chiplet 00 1.0V	Sync Async	4-2ph Async.	QDI async. 10 stages/link pipeline	4 QDI routers on path		Async Sync Async. 3D-Plug 2-4ph Async.	Chiplet 12
	Active Interposer 1.2V	Async. 2-4ph		Async. Router	×4		4-2ph Async.	throughput for 3D increases local 2-phase QDI
					25 mm
	b) L2 -L3 cache interconnect with QDI asynchronous routing
	Chiplet 00 Active Interposer 1.0V 1.2V	Sync. 3DPlug Sync. 3DPlug	FIFO pipeline Router Sync. 8 stages/link	Sync. Router		Sync. Router		Sync. Router	MEM IO	EXT-MEM
						25 mm
	c) L3 -EXT -MEM interconnect with global synchronous routing
	clock-domain crossing	boundary scan + μ-buffer (ESD, level-shifter) 3D μ-bump array
	Fig. 18. INTACT system interconnect structure on longest path using different
	technologies for different traffic classes
							: COMPARATIVE PERFORMANCE OF SYSTEM INTERCONNECTS IN INTACT
			L1-L2 nearest L1-L2 farthest	L2-L3 4-phase	L2-L3 2-phase	L3-EXT-MEM	3DNOC [36]	Units
	Reach			1.5			15			25 (bottom left chiplet to upper right chiplet)	8	mm
	Word size				40				72	32	bits
	Interposer Chiplet	1 passive link -	3 passive links Active async. routing Active async. routing Active sync. routing Global sync. routing Local sync. routing Local sync. routing Local sync. routing	Active async. Routing	--
	3D Plug frequency	1.25		1.25			0.30	0.52	1.21	0.32	GHz
	2D NoC frequency	-			1.00			0.97	0.75	0.89	GHz
	End to end latency	2x4+[0-1] 7.2		44 44.0		4 + async. 15.2	4 + async. 15.2	37 49.5	4 + async 10	Cycles ns
	Propagation speed	4.8			2.9			0.6	0.6	2.0	1.2	ns/mm
	Energy / bit / mm	0.29		0.15			0.52	0.52	0.24	0.5	pJ/bit/mm

TABLE V :

 V INTACT DESIGN-FOR-TEST RESULTS** Limited test coverage is reported by the tool within the interposer, this is due to the asynchronous NoC that can be tested using a dedicated test solution not reported here

TABLE VI :

 VI STATE OF THE ART COMPARISON

			This work	[31] ISSCC'18 [4,15] ISSCC'18&20 INTEL AMD	[5] VLSI'19 TSMC	[6] ISSCC'17 INTEL	Units
		Chiplet Technology	FDSOI 28nm	FinFET 14nm	FinFET 14nm/7nm	FinFET 7nm	FinFET 14nm
	Technology	Interposer Technology Interposer extra features Total system yield Die-to-Die µbump pitch	Active CMOS 65nm yes high (mature tech. & low transistor count) 20	no N/A N/A N/A	MCM substrate no / IO die high > 100	Passive CoWoS ® no high 40	EMIB bridge no high 55	µm
		Voltage Regulator (VR) type 6 SCVR on interposer	on-chip distributed	LDO per core,	no	no
	Power	VR area Mgt VR peak efficiency	with MOS+MOM+MIM 34% of active interposer MIM>40% core area SCVR with MIM 82% 72%	with MIM -LDO limited	N/A N/A	N/A N/A
	Interconnect	Interconnect types 3D Plug power efficiency BW density Aggregate 3D bandwidth	Distributed scalable cache-coherent NoCs 0.59 3.0 527	N/A N/A N/A N/A	Scalable Data Fabric (SDF) 2.0 --	LIPINCON TM links 0.56 1.6 640	AIB interconnect 1.2 1.5 504	pJ/bit Tb/s/mm 2 GByte/s
		Number of chiplets	6	1	1 -4 / 1 -8	2	1 FPGA + 6 TxRx
		Number of cores	96	18	8 -32 / 8 -64	8	FPGA fabric
	CPU	Max Frequency Gops (32b-Integer)	1.15 220 (peak mult./acc.)	0.4 14.4	4.1 / 4.7 131.2 -1203	4 128	1 N/A	GHz Gop/s

ACKNOWLEDGMENT

The authors would like to thank STMicroelectronics and Didier Campos team for INTACT package design and assembly, PRESTO Engineering and Brice Grisollet for testing the INTACT circuit onto Automatic Test Equipement, Easii-IC and Jean-Paul Goglio and his team for designing the INTACT application demonstration board. Finally, the authors would like to thank many other contributors from Mentor Graphics on the CAD tool teams and from CEA-LETI on both the design team and technology teams for their dedication to make this concept and circuit a successful realization.

This work was in part funded by the French National Program Programme d'Investissements d'Avenir, IRT Nanoelec under Grant ANR-10-AIRT-05, and also by the following research projects: SHARP CA109 CATRENE project, MASTER3D CT312 CATRENE project, Hubeo+ CARNOT project.

XI. CONCLUSION

The presented Active interposer leverages the 3D integration benefits by offering a baseline of functionalities such as voltage delivery, chiplet-to-chiplet communications, IOs, shared by most of computing assemblies. The active interposer allows a flexible assembly with common functionalities while maintaining the yield management benefits. For this reduced power density and budget, thermal dissipation is not an issue within the active interposer, as for a regular passive interposer.

3D integration and Active Interposer open the way towards efficient integration of large-scale chiplet-based computing systems. Such scheme can be applied for integration of similar chiplets as presented in this paper, but also for smooth integration of heterogeneous computing chiplets [START_REF] Martinez | ExaNoDe: combined integration of chiplets on active interposer with bare dice in a multi-chip-module for heterogeneous and scalable high performance compute nodes[END_REF].

Pascal Vivet received his PhD from Grenoble INPG in 2001, designing an asynchronous microprocessor. After 4 years within STMicroelectronics, he joined CEA-Leti in 2003 in the digital design lab. His research interests covers wide aspects of circuit and system level design, ranging from system integration, multicore architecture, Network-on-Chip, energy efficient design, related CAD aspects, and in strong links with advanced technologies such as 3D, Non-Volatile-Memories, photonics. He was project leader on 3D circuit design from 2011 to 2018. He is currently Scientific Director of the Digital Systems and Integrated Circuits Division at CEA-LIST, a CEA institute. He has authored and co-authored more than 120 papers and holds several patents in the field of digital design. He is member of IEEE.

Eric