Thèse M Laurent Fesquet

Maître De

M Hayder Mra- Bet

M Matthieu Tuna

M Zied Marrakchi

M Christophe Alexandre

FARHAT M Ramsis

M Fathi

M Christian Layouni

pour avoir Masson

Mme Mariem Turki

M Alp Kilic

M Vinod Pangracious

Keywords: Integrated Circuit, Verification, Multi-FPGA, Prototyping, Custom, Cabling

Multi-FPGA based prototyping is no longer optional for hardware/software integration. We can classify multi-FPGA prototyping platforms in three categories: off-the-shelf, custom and cabling.

The off-the-shelf platform consists of a ready-made generic multi-FPGA board, where all the inter-FPGA connections are fixed and realized using PCB traces. The custom platform consists of a build-your-own multi-FPGA board tailored for a specific design, where all the inter-FPGA connections are realized using PCB traces as well. The cabling platform consists of multiple ready-made FPGA boards connected by cables and connectors. The cabling platform is semi off-the-shelf due to that it consists of multiple ready-made boards, and semi custom due to that its connections inter FPGAs as well as connections to external interfaces are user-defined and tailored for a specific design.

There are three existing problems regarding to multi-FPGA based prototyping: (1). FPGA I/Os are becoming a scarce resource, worsening the inter-FPGA bandwidth generation after generation.

Unfortunately, multi-FPGA platforms suffer from large timing delays in inter-FPGA communication compared to intra-FPGA net delays. Therefore, it becomes more and more difficult to prototype an SoC/ASIC design at proper performance. (2). 70% of multi-FPGA prototyping platforms are home-made custom platforms due to performance requirement, external interfaces, and cost. Nevertheless, crafting a home-made custom multi-FPGA platform is today a manual process, thus, time-consuming. The board exploration with different FPGA types, which helps the engineers to design an optimum prototyping platform, can not be done. As the ratio between the logic capacity and the number of FPGA I/Os is increasing at an exponential rate, it becomes more and more challenging to design a high-performance custom multi-FPGA platform. (3). The cabling platform benefits from the availability and the customization. The performance of the cabling platform depends on the distribution of the cables and the placement of the external interfaces.

Nevertheless, there is no tool to automatically have a solution for the cable distribution and external interface placement. A high-performance cables distribution becomes more and more difficult to be achieved due to the pin limitation. Compared to the off-the-shelf platform, the added value, in terms of performance, of cabling or custom platforms can be heavily impaired by an inefficient board design.

). An automatic design flow for creating a custom platform is proposed, thus increasing the productivity, enabling the board exploration, and optimizing cost and performance.

(3). The cabling platform is proposed where one board is composed of one FPGA and several connectors. The connections between FPGAs as well as the connections to external interfaces can be added or removed by only connecting or disconnecting the cables (resp. daughter boards) with or from the connectors. Then, an algorithm is proposed to automatically find a solution for the cable distribution. (4). Thanks to the developed automatic tools, the achieved performances for a set of designs mapped on the three different categories of multi-FPGA platforms are compared.

The performance gains between these platforms are quantified.

Résumé

Face à la difficulté grandissante de l'intégration matériel/logiciel, le prototypage à base de cartes multi-FPGA devient obligatoire dans l'arsenal des techniques de vérification pré-silicium. Les plateformes de prototypage multi-FPGA peuvent être classées en trois catégories: sur étagère, sur mesure et câblées. La plateforme sur étagère se compose d'une carte multi-FPGA prêt à l'emploi et générique. Toutes les connexions inter-FPGA sont fixées et réalisées à l'aide de pistes sur le PCB. La plateforme sur mesure se compose d'une carte multi-FPGA conçue spécifiquement pour un design donné. Les connexions inter-FPGA sont également réalisées à l'aide de pistes PCB.

Les plateformes câblées, qui se composent de plusieurs cartes mono-FPGA prêt à l'emploi, sont connectées grâce à des câbles peuvent être connectées/déconnectées à volonté. Elles peuvent être considérées comme "semi-sur étagère" par le fait qu'elles sont constituées de plusieurs cartes prêt Les contributions de ce manuscrit sont: [START_REF]Performance Comparison between Multi-FPGA Prototyping Platforms: Hardwired Off-the-Shelf, Cabling and Custom Qingshan Tang[END_REF]. Un nouvel algorithme de routage exploitant les pistes multi-points connectant plus de deux FPGA, permettant ainsi d'augmenter les performances.

(2). Un flot de conception automatique permettant de créer une plateforme sur mesure, augmentant ainsi la productivité et permettant l'exploration de cartes. [START_REF]Routing Algorithm for Multi-FPGA Based Systems Using Multi-Point Physical Tracks Qingshan Tang[END_REF]. Une architecture de plateforme câblée est proposée ainsi qu'un algorithme permettant automatiquement de trouver une solution pour la distribution des câbles. [START_REF]Design for prototyping of a parameterizable cluster-based Multi-Core System-on-Chip on a Multi-FPGA board Qingshan Tang[END_REF]. Finalement une comparaison entre ces différentes plateformes est réalisée d'un point de vue quantitatif et qualitatif.

Mots

Thesis Motivation

The continuous improvement of integrated circuit technologies leads to develop more complex and higher performance circuits. As a consequence, more and more efforts are required for their verification.

Design Verification

According to International Business Strategies, Inc. (IBS [START_REF] Ibs | International Business Strategies[END_REF]), the cost of verification is growing at an exponential rate and has already been the highest portion of the overall design cost (~70%) as shown in the Figure 1 Enterprise Simulator [Incisive, 2014], Mentor Graphics ModelSim/SE [ModelSim, 2014],

and Synopsys VCS [START_REF] Vcs | Functional Verification Solution[END_REF].

• Emulation: Emulators are deployed to simulate an entire SoC/ASIC (Full-chip Verification) and thereby address the limitations of simulators. In the emulation, the RTL is still unstable (and therefore requires detailed debug analysis and frequent overnight recompiles). This technique provides high-visibility into the RTL and has much faster execution speed than the simulation (~1M Hz). Nevertheless, it is the most expensive (about millions of dollars) and takes several days to several weeks for set-up. Emulators are hardware platforms as shown in Figure 1.3. The three major existing hardware emulator platforms are Cadence Palladium emulator [START_REF] Palladium | Cadence Palladium series[END_REF], Mentor Graphics Veloce Emulation Systems [Veloce, 2014],

and Synopsys EVE ZeBu Emulation [ZeBu, 2014].

• Virtual Prototyping: Virtual prototype is the earliest available Software Verification in the project, permitting co-development of hardware along with early software (i.e. If the virtual prototype shows that there is not enough processing bandwidth for concurrent applications, extra or different CPUs might be added). This technique represents fully functional software models of SoC/ASICs, boards, virtualized I/Os, external interfaces, all running on a host PC. But these software models are loosely-timed. Virtual prototyping offers good system visibility and control, which is especially useful for debug. Nevertheless, due to that it is not cycle-accurate, it can not ensure the functionality of the software in the hardware. As simulators, virtual prototypes are software tools on a host PC. The three major existing virtual prototype tools are Cadence Virtual System Platform [Virtual, 2014], Mentor Graphics

Vista [START_REF] Vista | Electronic System Level Design[END_REF], and Synopsys Virtualizer [Virtualizer, 2014].

• FPGA-Based Prototyping: FPGA-based prototypes permit to run the software at almost real-time speed in a cycle-accurate and bit-accurate model of the SoC/ASIC (Hardware/-Software Verification). In the FPGA-based prototyping, the SoC/ASIC RTL is relatively stable. This technique is the one that offers the best execution speed (~10M Hz) -Cadence Protium Rapid Prototyping Platform [START_REF] Protium | Protium Rapid Prototyping Platform[END_REF] affords both the hardware platform and the implementation tool.

-Synopsys supplies both the hardware platform HAPS [HAPS, 2014] and the implementation tool ProtoCompiler [ProtoCompiler, 2014].

-S2C [S2C, 2014] sells both the hardware platform and the implementation tool.

-Dini Group produces only the hardware platform [DINI, 2014].

-HyperSilicon offers only the hardware platform [START_REF] Hypersilicon | Fpga prototype, soc verification[END_REF].

-Auspy accommodates only the implementation tool [START_REF] Auspy | Auspy Development Inc[END_REF].

-Flexras Technologies provides only the implementation tool Wasga [START_REF] Flexras | Flexras Technologies[END_REF].

FPGA-Based Prototyping

As the complexity of System on Chip (SoC) circuits and the quantity of software to be developed are increasing, the software developers can no longer wait for the chip to be fabricated for the integration of the hardware/software phase in order to meet the ever-shrinking time-to-market window [START_REF] Huang | SoC HW/SW Verification and Validation[END_REF]. The resulting trend is that FPGA-based prototyping is no longer optional [START_REF] Amos | FPGA-Based Prototyping Methodology Manual[END_REF].

Advantages of FPGA-based prototyping

High performance and timing-accuracy

Only FPGA-based prototyping provides both the speed and timing-accuracy necessary to properly test many aspects of the SoC/ASIC design (such as: real-time dataflow, hardware/software integration).

• Real-time dataflow: One of the reason that verifying an SoC is hard is due to that its state depends on many variables, including its previous state, the sequence of inputs and the wider system effects (and possible feedback) of the SoC outputs. Running the SoC design at almost real-time speed connected into the rest of the system (with "real world" interfaces) allows us to see the immediate effect of real-time conditions, inputs and system feedback when they change.

• Hardware/Software integration: Due to that software has already come to dominate SoC development effort, it is increasingly common that the software effort is on the critical path of the project schedule. FPGA-based prototyping offers a hardware model, the closest to the future silicon for testing hardware/software integration.

High mobility

Due to the small form factor of FPGA-based prototyping and its ability to work standalone, it has high mobility. It is often used for pre-production demonstration of new product capabilities at trade shows.

1.1.3.2 Disadvantages of FPGA-based prototyping

Low visibility

Even though there are many ways to instrument an FPGA in order to gain some visibility into design's functionality, it is still only a fraction of the information that is readily available in a simulator or an emulator. Therefore, we should always wait until the SoC/ASIC designs' RTL is fairly mature in simulation or emulation before passing it over to the FPGA-based prototyping.

Long set-up time

Due to that SoC/ASIC designs are often FPGA-hostile, the RTL modifications for FPGA-based prototyping need to be done as follows:

• The top-level pads of SoC/ASICs need to be adapted for the FPGA tool flow.

• The gated-clock and the complex generated clocks in SoC/ASICs need to be converted in FPGAs.

• Deal with some SoC design elements that are not available in FPGAs such as analog circuitry, BIST, SoC primitive and third party IP.

• Memories in SoC/ASICs need to be handled with FPGA memory resources (i.e. block RAM and distributed RAM) or external memory resources.

Therefore, the set-up time is long for FPGA-based prototyping.

Even though FPGA-based prototyping is not ideal, it provides a unique pre-silicon model of target silicon, allowing the software to be introduced to a cycle-accurate and high performance model of the hardware as early as possible.

Multi-FPGA Platform

Due to that the silicon area overhead of FPGA versus ASIC technology has been measured to be about 40x [START_REF] Kuon | Quantifying and Exploring the Gap Between FPGAs and ASICs[END_REF], FPGA technology requires that an ASIC logic design should be partitioned across multiple FPGA devices to achieve the necessary logic capacity. The number of FPGAs depends on the size of the prototyped SoC/ASIC, ranging from a few [Krupnova, 2004] up to 60 FPGAs [START_REF] Asaad | A Cycle-accurate, Cycle-reproducible multi-FPGA System for Accelerating Multi-core Processor Simulation[END_REF].

According to their role and characteristics, we have classified the multi-FPGA prototyping platforms in three major categories:

• Hardwired Off-the-Shelf Platform [DINI, 2014] [GiDEL, 2014[START_REF] Polaris | Cables Vs. PCB Traces[END_REF] (abbreviation: off-the-shelf): consists of a ready-made generic multi-FPGA board, where all the inter-FPGA connections are fixed and realized using PCB traces. The connections to external interfaces are also fixed but can be realized using PCB traces or connectors.

• Hardwired Custom Platform [START_REF] Reflex | [END_REF] (abbreviation: custom): consists of a buildyour-own specific multi-FPGA board, where all the inter-FPGA connections are realized using PCB traces as well. Different from the off-the-shelf platform that has generic and balanced connections, the connections inter FPGAs as well as the connections to external interfaces of the custom platform are user-defined and tailored for a specific design.

• Cabling Platform [HAPS, 2014[START_REF] Prodesign | FPGA Prototyping[END_REF]: which is a relatively new notion compared to other two platforms, consists of multiple ready-made FPGA boards connected by cables and connectors. The cabling platform, which is in between the off-the-shelf and the custom platform as shown in Figure 1.4, is semi off-the-shelf due to that it consists of multiple ready-made boards, and semi custom due to that its connections inter FPGAs as well as connections to external interfaces are user-defined and tailored for a specific design. Nevertheless, all the inter-FPGA connections are realized using cables and connectors instead of PCB traces.

Pin Limitation Problem

Mapping multi-million gates SoCs into any kind of the multi-FPGA platform is very challenging.

The mapping process can be divided in two main steps. The first step is the partitioning of the design, meaning that the design is divided into several parts. Each part fits in the logic capacity of a single FPGA. The signals crossing design's parts located in different FPGAs are called cut nets.

The second step routes the cut nets, meaning that a cut net is allocated to an inter-FPGA track on the platform. One should note that even though the logic capacity and the number of FPGA I/Os are increasing generation after generation, the logic capacity increases at a much higher rate than the number of FPGA I/Os [Schirrmeister, 2014]. Figure 1.5 shows the ratio between the logic capacity and the number of I/Os for each FPGA generation from Virtex-4 to Virtex-7 in Xilinx

Virtex Family and from Stratix2 to Stratix5 in Altera Stratix Family [DINI, 2014]. About 2000 gates are trying to get through one FPGA I/O in Virtex-4 or Stratix2. In the case of the latest generation Virtex-7 (resp. Stratix5), this is almost 20000 (resp. 16000) gates. This means that FPGA I/Os are becoming a scarce resource, worsening the inter-FPGA bandwidth generation after generation. Unfortunately, multi-FPGA platforms suffer from large timing delays in inter-FPGA communication compared to intra-FPGA net delays [START_REF] Amos | FPGA-Based Prototyping Methodology Manual[END_REF]. Therefore, this trend has a direct impact on multi-FPGA system performance (in terms of the system clock frequency) and makes it more and more difficult to prototype an SoC/ASIC design at proper performance. 70% of multi-FPGA prototyping platforms are home-made custom platforms due to performance requirement, external interfaces, and cost. Indeed home-made custom platforms are tailored for a specific design and external interfaces, and the cost decreases as more platforms are needed. Nevertheless, crafting a home-made custom multi-FPGA platform is today a manual process. Therefore, three critical aspects exist:

• Productivity: Crafting a home-made custom multi-FPGA platform is a time-consuming process (about 9 months [Sekhar, 2014]). The performance and the cost of the platform lie on the FPGA expertise and SoC DUT knowledge of the prototyping team.

• Exploration: There are many different FPGA types (i.e. vendor: Xilinx [Xilinx, 2014] or Altera [START_REF] Altera | FPGA CPLD and ASIC from Altera[END_REF], family: Virtex-7 or Stratix5, device: 2000T or GXAB, and package: FLG1925 or F1932), and different FPGA types have different logic capacity and numbers of FPGA I/Os. For a given design, the FPGA type chosen by the engineers influences the achieved performance and cost. The tradeoff between the performance and the cost exists due to that the performance of multi-FPGA platforms is limited by the inter-FPGA communications [START_REF] Amos | FPGA-Based Prototyping Methodology Manual[END_REF]. The board exploration with different FPGA types, which helps the engineers to design an optimum prototyping platform, can not be done.

• Performance: As the ratio between the logic capacity and the number of FPGA I/Os is increasing at an exponential rate, it becomes more and more challenging to keep performance as it was in multi-FPGA platform.

Problem of Cabling Paradigm

The cabling platform benefits from the availability and the customization. The performance of the cabling platform depends on the distribution of the cables and the placement of the external interfaces. Nevertheless, there is no tool to automatically have a solution for the cable distribution.

Today, the cables (resp. the external interfaces) are distributed (resp. placed) according to the experience of board designers. A high-performance cables distribution becomes more and more difficult to be achieved due to the pin limitation.

Compared to the off-the-shelf platform, the added value, in terms of performance, of cabling or custom platforms can be heavily impaired by an inefficient board design.

Thesis Contributions

In order to solve the problems presented above, the major contributions of the manuscript include the following:

• A new routing algorithm is proposed to spare FPGA I/Os by exploiting multi-point tracks that connect more than two FPGAs, thus increasing the performance.

• An automatic design flow for creating a custom platform is proposed, thus increasing the productivity, enabling the board exploration, and optimizing cost and performance.

• The cabling platform is proposed where one board is composed of one FPGA and several connectors. The connections inter FPGAs as well as the connections to external interfaces can be added or removed by only connecting or disconnecting the cables (resp. daughter boards) with or from the connectors. Then, an algorithm is proposed to automatically find a solution for the cable distribution.

• Thanks to the developed automatic tools, the three different categories of multi-FPGA platforms (off-the-shelf, custom, and cabling) are compared. The performance gains between these platforms are quantified.

THESIS ORGANIZATION

Thesis Organization

The rest of this manuscript is organized as follows.

Chapter 2 studies the state of the art of multi-FPGA prototyping platforms and details the inter-FPGA communication architectures. As there are three different multi-FPGA prototyping platforms, the state of the art is classified into four parts: Hardwired off-the-shelf multi-FPGA platform, Hardwired custom multi-FPGA platform, Cabling multi-FPGA platform, and Comparison of different platforms. Then, the inter-FPGA communication architectures are discussed due to that they are the critical path of all the multi-FPGA prototyping platforms.

Chapter 3 focuses on the off-the-shelf platform. An overview of multi-FPGA off-the-shelf platforms and the implementation flow are presented. There are two types of inter-FPGA tracks:

2-point tracks that connect two FPGAs and multi-point tracks that connect more than two FPGAs.

A new routing algorithm of routing cut nets in 2-and multi-point tracks is proposed.

Chapter 4 focuses on the custom platform. First, an overview of the custom platform is presented. Then, the automatic design flow for creating a custom platform is proposed. Different steps of the proposed automatic design flow are explored in order to achieve lower cost and higher performance. With the proposed automatic design flow, board exploration (exploring different FPGA types for the given design) has been addressed. Plenty of feasible solutions can be generated in board exploration and board designers can make the tradeoff between cost and performance.

Chapter 5 focuses on the cabling platform. A cabling platform is proposed with an algorithm to automatically find a solution for the cable distribution.

Chapter 6 compares the achieved performances for a set of designs mapped on the three different categories of multi-FPGA platforms. The performance gains between these platforms are quantified.

Finally, Chapter 7 concludes the manuscript and suggests topics for future work.

Chapter 2 Background

Introduction

The previous Chapter has presented the different multi-FPGA prototyping platforms and the difficulties using them optimally. The purpose of this Chapter is to give to the user the necessary background to understand the rest of this manuscript. This Chapter is organized as follows:

• Section 2.2 discusses the state of the art of multi-FPGA prototyping platforms.

• Section 2.3 details the inter-FPGA communication architectures, which are the critical path of all the multi-FPGA prototyping platforms in performance (in terms of the system clock frequency).

State of the Art

Despite the fact that hardware/software integration is a more and more concerning problem, few papers have been published regarding multi-FPGA prototyping platforms. As there are three different multi-FPGA prototyping platforms, the study of the related work is classified into four parts:

• Hardwired off-the-shelf multi-FPGA platform

• Hardwired custom multi-FPGA platform

• Cabling multi-FPGA platform • Comparison of different platforms 2.2.1 Hardwired Off-the-Shelf Multi-FPGA Platform
The related work of the hardwired off-the-shelf multi-FPGA platform contains two parts:

• The platform overview that presents the definition of the off-the-shelf platform

• The state of the art in the implementation flows, more specifically the state of the art in the routing algorithms

Platform Overview

In this manuscript, the off-the-shelf platform consists of a ready-made generic multi-FPGA board, where all the inter-FPGA connections are fixed and realized using PCB traces. The connections to external interfaces are also fixed but can be realized using PCB traces or connectors (connected to daughter boards). An example of such platform is the commercial platform DNV7F4A as shown in [START_REF] Flexras | Flexras Technologies[END_REF], targeting at automatizing the implementation flow.

This manuscript will study the process of the routing and propose an improved routing algorithm. The state of the art of the routing algorithms is discussed in the following. Routing algorithms for multi-FPGA platforms heavily depend on the underlying hardware routing architectures. Thus, they can be classified according to the routing architectures.

Direct architecture

All the FPGAs are directly interconnected [Walters, 1991] as shown in After loading the partitioned design and the multi-FPGA platform architecture, the algorithm will calculate initial mux by obtaining the maximum ratio of cut nets and inter-FPGA tracks between each pair of FPGAs.

• Group cut nets to GNet

The cut nets, which have the same driver FPGA and the same receiver FPGAs, are grouped together. The group, which is called GNet, will be put in one track. Each GNet contains a maximum of mux cut nets.

• Routing in 2-point tracks

PathFinder [START_REF] Mcmurchie | PathFinder: A Negotiation-Based Performance-Driven Router for FPGAs[END_REF] was used primarily for routing intra-FPGA nets.

It is adapted to deal with GNets and routes GNets in 2-point tracks. An iterative negotiationbased approach is used in PathFinder. During the first routing iteration, GNets are freely routed without paying attention to track sharing. Individual GNets are routed using Dijkstra's shortest path algorithm [START_REF] Cormen | Introduction to Algorithms[END_REF]. At the end of the first iteration, tracks may be congested because multiple GNets try to use one track. During subsequent iterations, the cost of using a track is increased, based on the number of GNets that share the track, and the history of congestion on that track. Thus, GNets are forced to negotiate for tracks. If a track is highly congested, GNets which can use lower congestion alternatives are forced to do so. On the other hand, if the alternatives are more congested than the track, then a GNet may still use that track. Due to that one track can only contain one GNet, if there are two GNets in one track in the result of PathFinder, there is a conflict. A routing result is feasible only when the number of conflicts is 0.

• The proposed algorithm tries to minimize mux. If feasible, decrease the mux and iterate the previous steps beginning from grouping cut nets with the new mux. If not feasible, exit with the best mux.

Nevertheless, this algorithm does not exploit multi-point tracks, which may waste FPGA I/Os.

Indirect architecture

All the FPGAs are interconnected through partial crossbars [START_REF] Varghese | An efficient logic emulation system[END_REF] as shown in Multi-terminal nets routing, through partial crossbars, is discussed in [START_REF] Mak | Board-Level Multiterminal Net Routing for FPGA-Based Logic Emulation[END_REF] [ Song et al., 2003] [Ejnioui and[START_REF] Ejnioui | [END_REF]. Even though multi-terminal nets routing on indirect architectures spares FPGA I/Os, multi-FPGA platforms performance is tied to inter-FPGA delays, and this kind of architecture worsens this effect as nets are routed through at least 2 inter-FPGA tracks plus the crossbar component. Moreover, if most of cut nets are 2-terminal nets, this architecture will waste FPGA I/Os. Therefore, this architecture is not used for FPGA-based prototyping.

Hybrid architecture

A mixture of direct connections and partial crossbars are used in [Khalid, 1999] [Khalid and[START_REF] Khalid | [END_REF] as shown in Figure 2.6.

Figure 2.6: The hybrid architecture

The routing algorithm in [START_REF] Jain | Hybrid Multi-FPGA board evaluation by limiting multi-hop routing[END_REF] proposes the use of a crossbar to route multiterminal nets and direct connections to route 2-terminal nets. However, this kind of architecture suffers of the same performance degradation as the indirect architecture and is not used for FPGAbased prototyping.

Conclusion

Industrial multi-FPGA prototyping platforms (Cadence Protium Platform [Protium, 2014], Dini

Group Platform [DINI, 2014] and Reflex FPP Platform [ReFLEX, 2014]) have adopted direct architecture for performance motivation. They afford 2-and multi-point tracks. Nevertheless, the existing tools do not automatically route and multiplex cut nets in multi-point tracks, which waste FPGA I/Os that are already a scarce resource. In Chapter 3, which focuses on the offthe-shelf platform, we will target the direct architecture and we propose to spare FPGA I/Os by automatically routing and multiplexing multi-terminal nets over multi-point tracks.

Hardwired Custom Multi-FPGA Platform

The related work of the hardwired custom multi-FPGA platform contains two parts:

• The platform overview that presents the definition of the custom platform

• The state of the art of the design flows for creating a custom platform

Platform Overview

The custom platform consists of a build-your-own specific multi-FPGA board, where all the inter-FPGA connections are realized using PCB traces as well. The advantage of the custom platform is that it is tailored for a specific design. An example is shown in for the given design and no FPGA I/O is wasted. Therefore, the design partitioning is facilitated due to that the DDR is connected to the FPGA where the memory controller logic is placed in the custom platform, thus higher performance can be achieved. Nevertheless, 4-7 months are needed for the PCB layout and the PCB fabrication.

Design Flow for Creating a Custom Platform

Typical design flow

The typical design flow as shown in Figure 2.8 for creating a home-made custom multi-FPGA platform is detailed as follows: 2. Estimate the minimum number of FPGAs required in the platform (according to the logic capacity of the input design, the logic capacity of the chosen FPGA type, and the maximum FPGA logic capacity utilization).

3. Floorplan FPGAs and external interfaces on the Printed Circuit Board (PCB). 6. Map the SoC/ASIC design on this latter board, meaning that the design is partitioned and routed into each FPGA. As there are fewer available inter-FPGA tracks than the number of cut nets, several cut nets need to be multiplexed and sent together onto a single track on the platform.

7. Run the PnR Compiler with the design sub-netlists to generate the bitstreams for each FPGA.

Then, the generated bitstreams are downloaded into the platform to model the design.

The typical design flow is mainly manual and iterative. Therefore, this process is timeconsuming, and the optimization of the cost and the performance depends on the designers' experience.

Cadence design flow

In [START_REF] Cadence | ASIC Prototyping Simplified[END_REF], a design flow for creating a custom platform is proposed, but only the process of selecting FPGA user I/Os is automated (by the Cadence FSP tool [FSP, 2014]) as shown in Figure 2.9.

Figure 2.9: Cadence's design flow for creating a custom platform.

1. Estimate the minimum number of FPGAs required for partitioning the design RTL according to the chosen FPGA type.

2. Place the FPGAs on the Allegro FPGA System Planner canvas [FSP, 2014] and generate a Verilog board netlist with the FPGAs, 3. Use a partitioning tool to ensure that the RTL can be partitioned within the selected number of FPGAs.

4. If the selected number of FPGAs is not reasonable, increase or decrease the number of FPGAs and iterate the previous processes.

5.

Use the top level of the partitioned RTL to define the connections required among the different FPGAs.

6. Once the I/O synthesis is completed in Allegro FPGA System Planner, the Verilog board connectivity file can be generated. Some iterations with the I/O definitions may be needed to ensure that all the I/Os can be placed on the different FPGAs.

7. Use this connectivity file once again in the RTL partitioning tool. The RTL partitioning tool can now map the wires in the top level to ports/traces on the board 8. In some cases, it may be required to iterate a bit to get to successful results, analyze results of the partitioning tools and make the required changes to the FPGA board architecture.

9. Once all the requirements are met, the board layout can be generated.

Even though the proposed point tool Cadence FSP automatically selects FPGA user I/Os, the Cadence design flow for creating a custom platform is still mainly manual and iterative. Therefore, this process is time-consuming, and the optimization of the cost and the performance depends on the designers' experience.

Conclusion

The existing design flows are mainly manual and iterative, thus creating a custom platform is time-consuming and the optimization of the cost and the performance depends on the designers' experience.

In Chapter 4, which focuses on the custom platform, we propose an automatic design flow for creating a custom multi-FPGA platform. The proposed automatic design flow reduces the timeto-market of new products and lowers the entry barrier of board designers while optimizing the cost and the performance. With the proposed automatic design flow, board exploration (exploring different FPGA types for the given design) has been addressed. Plenty of feasible solutions can be generated in board exploration and board designers can make the tradeoff between cost and performance.

Cabling Multi-FPGA Platform

The related work of the cabling multi-FPGA platform contains two parts:

• The platform overview that presents the definition of the cabling platform

• The state of the art of the design flows for creating a cabling platform

Platform overview

The cabling platform, which is a relatively new notion compared to other two platforms, consists of multiple ready-made FPGA boards connected by cables and connectors. In between the offthe-shelf and the custom platform, the cabling platform is semi off-the-shelf as it is ready-made, and semi custom as the inter-FPGA connections can be changed by connecting or disconnecting the cables in order to be tailored for the given design.

The hybrid cabling platform

In [Hauck, 1995], a hybrid cabling platform as shown in Figure 2.10(a) is proposed. The inter-FPGA tracks are realized by a hybrid of the cables and the PCB traces. To allow a specific circuit to be implemented in this structure, the SpringBok system is composed of a baseplate with sites for cards. The cards in the platform as shown in Figure 2.10(b) are large enough to contain an arbitrary device on the top, as well as an FPGA on the bottom. Note that the device can be a chip, or external interfaces (such as PCI, DDR). The inter card routing structure is a mesh as shown in Figure 2. 10(c) and the connections are fixed. There is the potential that the simple connections will not be able to accommodate all the logic or routing assigned to a given location (Intermediate FPGAs are needed from the driver FPGA to the destination FPGA, which degrades the performance). However, as opposed to the off-the-shelf platform, "extender" cards can be inserted between a card and the baseplate to deal with these problems. For signals that must go long distances in the array, sets of "extender" cards with cables can be inserted to carry these long-distance wires. Nevertheless, the inter-FPGA tracks in the hybrid cabling platform is not fully customized for the given design due to that several inter-FPGA tracks are realized by fixed PCB traces. Therefore, the achieved performance is limited. The quasi cabling platform

In [START_REF] Kulmala | Evaluating Large System-on-Chip on Multi-FPGA Platform[END_REF] and IBM [START_REF] Asaad | A Cycle-accurate, Cycle-reproducible multi-FPGA System for Accelerating Multi-core Processor Simulation[END_REF], a cabling platform is used as shown in Figure 2.11. All the inter-FPGA tracks are realized by cables, but the connections to external interfaces are realized using PCB traces. The inter-FPGA tracks can be changed in order to be tailored for the given design. Due to that the cabling platform is ready-made, the connections to external interfaces may be not tailored for the given design. Thus, FPGA I/Os occupied by external interfaces may be wasted. Moreover, there is no discussion about the cable distribution. The cable distribution and the external interface placement, which are essential to the achieved performance in the cabling platform, depend on the experience of board designers. The cabling platform

The cabling platform consists of multiple ready-made FPGA boards connected by cables and connectors. Almost all the FPGA I/Os are used for FPGA-to-connector connections. The connections between FPGAs as well as the connections to external interfaces can be added or removed by only connecting or disconnecting the cables (resp. daughter boards) with or from the connectors in order to be tailored for the given design.

The proFPGA cabling platform as shown in Figure 2.12 is proposed in [START_REF] Prodesign | FPGA Prototyping[END_REF].

For each FPGA, there are only 4 connectors. The width of connectors is defined as the pairs of tracks passing through this connector. Among the four connectors, three of them have 148 pairs of tracks, and the rest one has 98 pairs of tracks. Nevertheless, the coarse-grained connector may waste FPGA I/Os when external interfaces do not occupy so many FPGA I/Os but need at least one connector. Moreover, there is no tool to automatically have a solution for the cable distribution.

The cable distribution and the external interface placement, which are essential to the achieved performance in the cabling platform, depend on the experience of board designers. Therefore, the cable distribution and the external interface placement, which are essential to get higher performance in the cabling platform, depend on the experience of board designers. The typical design flow as shown in Figure 2.14 for creating a cabling multi-FPGA platform is detailed as follows:

Figure 2.14: The typical design flow for creating a cabling platform 1. Choose the FPGA type (i.e. vendor: Xilinx [Xilinx, 2014] or Altera [START_REF] Altera | FPGA CPLD and ASIC from Altera[END_REF], family:

Virtex-7 or Stratix5, device: 2000T or GXAB, and package: FLG1925 or F1932).

2. Estimate the minimum number of FPGAs required in the platform (according to the logic capacity of the input design, the logic capacity of the chosen FPGA type, and the maximum FPGA logic capacity utilization.

3. Manually find a solution of the distribution of the cables and the external interface daughter boards. Then, the cables and external interface daughter boards are connected. Finally, a board netlist is generated.

4. Map the SoC/ASIC design on this latter board, meaning that the design is partitioned and routed into each FPGA. As there are fewer available inter-FPGA tracks than the number of cut nets, several cut nets need to be multiplexed and sent together onto a single track on the platform.

5. Evaluate the performance. If the performance is acceptable, pass to the next step. If not, iterate the previous step from re-connecting the cables and external interface daughter boards.

6. Run the PnR Compiler with the design sub-netlists to generate the bitstreams for each FPGA.

Then, the generated bitstreams are downloaded into the platform to model the design.

The typical design flow is mainly manual and iterative. Therefore, this process is timeconsuming, and the optimization of the performance depends on the designers' experience.

Conclusion

According to our knowledge, there is no tool to automatically have a solution for the cable distribution. In Chapter 5, which focuses on the cabling platform, we propose a cabling platform with an algorithm to automatically optimize the cable distribution and the external interface placement.

Nowadays, the two existing cabling platforms (proFPGA and Synopsys HAPS) have different width granularity of the connectors (resp. cables). With the proposed algorithm, the optimal width of connectors in terms of performance is explored.

Comparison of Different Multi-FPGA Platforms

Combining [START_REF] Amos | FPGA-Based Prototyping Methodology Manual[END_REF][START_REF] Cadence | ASIC Prototyping Simplified[END_REF], these three different platforms are compared qualitatively in terms of availability, performance, flexibility, and cost as shown in Table 2.1. The availability is the time to get access to the platform. The off-the-shelf platform is available instantly. For the cabling platform, 1-2 months are needed to find an optimal cable distribution.

For the custom platform, 4-7 months are needed to find a good solution due to that the PCB layout and the PCB fabrication is needed. The achieved performance of multi-FPGA platforms depends on the inter-FPGA data rate and inter-FPGA tracks distribution. For the off-the-shelf and the cabling platform, the data rate is high due to that they are ready-made by the FPGA experts in the commercial companies. For the custom platform done by the in-house team, it depends on designers' experience on how to design high-speed signaling, high performance multiplexing and so forth. For the off-the-shelf platform, the inter-FPGA tracks distribution is generic and balanced.

For the cabling and the custom done by the in-house team, it depends on the designers' experience on how to distribute inter-FPGA tracks in order to achieve higher performance. Naturally, the custom platforms should achieve highest performance because it is tailored for the given design.

Unfortunately, the performance of the cabling and the custom platform heavily depend on designers' experience because designing such platforms is still a manual process. Therefore, not so good performance platforms can be often seen. The flexibility of the platform is its ability to be tailored for different designs. The cabling platform has the highest flexibility due to that the same platform can be tailored for different designs by only re-connecting the cables. The off-the-shelf platform has medium flexibility due to its generic and balanced connections. The custom platform has the lowest flexibility due to that one custom platform tailored for one design may be not tailored for another design. The unit price for the custom platform is high as it integrates the development cost. The deployment cost corresponds to the quantity of platforms needed. This cost for custom platforms is low because it consists of the Bill-Of-Material (BOM) plus the production cost.

Conclusion of the State of the Art

Different kinds of platforms are widely used: off-the-shelf [START_REF] Hyder | Defect Tolerance in Multiple-FPGA Systems[END_REF], custom [Krupnova, 2004], and cabling [START_REF] Kulmala | Evaluating Large System-on-Chip on Multi-FPGA Platform[END_REF]] [Asaad et al., 2012]. The contributions of this manuscript are classified into four parts: Firstly, a new routing algorithm is proposed to spare FPGA I/Os by exploiting multi-point tracks. Secondly, an automatic design flow for creating a custom platform is proposed. Thirdly, the cabling platform and an algorithm to automatically optimize the cable distribution and the external interface placement, are proposed. Finally, the achieved performances are compared when a set of designs are mapped on the three platforms.

The performance gains between these platforms are quantified.

Even though the three platforms are different, they all have their performance limited by the inter-FPGA connections. Different inter-FPGA communication architectures, more or less complex, can be implemented leading to different achieved performances. Before exploiting different multi-FPGA platforms, we have to explain these different architectures.

Inter-FPGA Communication Architectures

According to [START_REF] Amos | FPGA-Based Prototyping Methodology Manual[END_REF][START_REF] Pinmux | Pin Multiplexing White Paper and IP[END_REF], inter-FPGA tracks are the critical path of all the multi-FPGA prototyping platforms in performance. Therefore, inter-FPGA communication architectures need to be detailed in order to understand the rest of this manuscript.

Time-Division-Multiplexing

As FPGA I/O pins are becoming a scarce resource, the number of cut nets outmatches the few available inter-FPGA physical tracks. Therefore, the cut nets are sent between FPGAs in a pipelined way [START_REF] Babb | Logic Emulation with Virtual Wires[END_REF]] [Inagi et al., 2010] using the Time-Division-Multiplexing (TDM) technique. This technique consists in multiplexing several cut nets onto a single track on the platform.

The maximum number of cut nets passing through one FPGA I/O is called the TDM ratio. Due to the pin limitation, the TDM ratio is increasing generation after generation. In [Krupnova, 2004],

four ST Microelectronics SoCs are prototyped on a platform made of Virtex-II FPGA, and the TDM ratio is 4. In [START_REF] Schelle | Intel nehalem processor core made FPGA synthesizable[END_REF], the Intel Nehalem processor core is mapped on a platform mixing Virtex-4 and Virtex-5, and the TDM ratio is 24. In [START_REF] Asaad | A Cycle-accurate, Cycle-reproducible multi-FPGA System for Accelerating Multi-core Processor Simulation[END_REF], the IBM's Bluegene/Q is prototyped on a Virtex-5 only platform, and the TDM ratio achieved is between 32 and 96.

The architecture of the basic Time-Division-Multiplexing is shown in Figure 2.15. The cut nets go through, a multiplexer (noted as M U X) connected to the FPGA I/O(s), the inter-FPGA track, and are de-multiplexed (noted as DM U X) in the receiving FPGA. After the partitioning, the parts located in different FPGAs need to work in the same system clock (noted as sys_clk). The multiplexer and the de-multiplexer use a fast clock (noted as if _clk) to propagate the multiplexed data over the inter-FPGA track. Each cut net has a time slot allocated corresponding to a fast clock cycle. All the multiplexed data should be sent in-between two system clock pulses. The latency is the number of the fast clock cycles from the first capture to the last update. In the multi-FPGA based prototyping, the system clock frequency is the fast clock frequency divided by the latency and need to be synchronous with the fast clock frequency [START_REF] Amos | FPGA-Based Prototyping Methodology Manual[END_REF]. Therefore, for a given inter-FPGA communication architecture and a given TDM ratio, the total latency need to be fixed. Ideally, when the latencies of added MUX and DMUX IPs are not taken into consideration, the latency equals the TDM ratio plus one. For example, in Figure 2.15, if the TDM ratio is 4, ideally the latency will be 5. For a given inter-FPGA communication architecture, the higher is the TDM ratio, the higher is the latency and the lower is the system clock frequency. Generation after generation the TDM ratio is increasing, implying that multi-FPGA system performance is reducing. • Logic Multiplexing: Logic blocks of the FPGA fabric are used to implement the multiplexing for inter-FPGA communication. The inter-FPGA data rate is ~125M bps, thus the system clock frequency is constrained. When the TDM ratio is 4, the system clock frequency is ~10M Hz [START_REF] Amos | FPGA-Based Prototyping Methodology Manual[END_REF].

• ISERDES/OSERDES: Dedicated output parallel-to-serial converters (OSERDES) and input serial-to-parallel converters (ISERDES) are used for inter-FPGA communication. The data rate is ~1Gbps [START_REF] Amos | FPGA-Based Prototyping Methodology Manual[END_REF]. This technique is widely used for multi-FPGA prototyping as it can achieve higher system clock frequency. When the TDM ratio is 4, the system clock frequency is ~30M Hz [START_REF] Pinmux | Pin Multiplexing White Paper and IP[END_REF].

• Multi-Gigabit Transceiver (MGT): Configurable hard-macros MGTs are implemented for inter-FPGA communication. The data rate can be as high as ~10Gbps [MGT, 2014]. Nevertheless, the MGT has a high latency (~30 fast clock cycles) that limits the system clock frequency and only a few is available. When the TDM ratio is 4, the system clock frequency is ~7M Hz [START_REF] Tang | Future Inter-FPGA Communication Architecture for Multi-FPGA Based Prototyping[END_REF]. In addition, the communication between MGTs is not errorfree. They come with a non-null bit error rate (BER). Therefore, at this moment, MGT is not used as inter-FPGA communication architecture in multi-FPGA prototyping.

Logic Multiplexing

In Logic Multiplexing, logic blocks of the FPGA fabric are used for inter-FPGA communication as shown in Figure 2.16. The inter-FPGA communication data rate is ~125M bps [START_REF] Amos | FPGA-Based Prototyping Methodology Manual[END_REF]. The synchronous method for Logic Multiplexing is often system-synchronous [START_REF] Amos | FPGA-Based Prototyping Methodology Manual[END_REF][START_REF] Serialio | High-Speed Serial I/O Made Simple[END_REF]. The system clocks for two FPGAs are generated by FPGA integrated PLL (Phase Locked-Loop) and have the same frequency with the clocks from the same Board Oscillator, thus are synchronous with each other. The fast clock is also generated by PLL in each FPGA, but is a frequency multiplier of the system clock. Therefore, the fast clock is synchronous with the system clock in each FPGA [Bilgic, 1982]. As the frequency of the fast clock is a frequency multiplier of the system clock, the fast clocks in two FPGAs are synchronous. According to [START_REF] Amos | FPGA-Based Prototyping Methodology Manual[END_REF][START_REF] Pinmux | Pin Multiplexing White Paper and IP[END_REF], the critical path of multi-FPGA platforms is inter-FPGA tracks due to the output delay T out (delay from flip-flop to pad, 5ns), the inter-FPGA board trace delay T board (1ns), the input delay T in (delay from pad to flip-flop, 1ns) and the tolerance delay T tolerance (safe margin for board clock distribution and etc., 1ns). The total delay T is the sum of all these delays as shown in Equation 2.1. Therefore, T = 8ns and the fast clock frequency if _clk = 1 T = 125M Hz (thus, the data rate is 125M bps due to the transmission is single data rate).

T = T out + T board + T in + T tolerance (2.1)
The maximum number of cut nets passing through one logic multiplexer is noted as mux_logic.

According to [START_REF] Amos | FPGA-Based Prototyping Methodology Manual[END_REF] In the Xilinx FPGA Virtex-7, ISERDES/OSERDES can have programmable widths of 2, 3, 4, 5, 6, 7, or 8 bits [SelectIO, 2014]. The recommended width is 4 for optimal performance [START_REF] Pinmux | Pin Multiplexing White Paper and IP[END_REF]. In this manuscript, a 4-bit wide SERDES is instantiated as depicted in Figure 2.17. ).

In [START_REF] Pinmux | Pin Multiplexing White Paper and IP[END_REF] The problem of complexity can be resolved by partitioning tools and keeping FPGA utilization low. Nevertheless, the bank-to-bank limitation inconveniences PCB designers. This limitation has already been taken into consideration in the off-the-shelf platform [DINI, 2014] and the cabling platform [HAPS, 2014[START_REF] Prodesign | FPGA Prototyping[END_REF], which consist of ready-made boards. When designing a custom platform, if ISERDES/OSERDES is chosen as the inter-FPGA communication architecture, the inter-FPGA physical tracks need to be bank-to-bank.

Even though ISERDES/OSERDES has a higher data rate than Logic Multiplexing, one inter-FPGA track in ISERDES/OSERDES consumes a pair of FPGA I/Os in mode LVDS and several inter-FPGA tracks in ISERDES/OSERDES need to be used to propagate the clock. Therefore, the achieved performance of Logic Multiplexing and ISERDES/OSERDES need to be compared in different TDM ratios. The DNV7F2A board as shown in Figure 2.18, which is afforded by the company Dini Group [DINI, 2014], is used as the platform.

There are two Xilinx Virtex-7 2000T FPGAs, which have the package FHG1761 and the speed grade -1. The design is partitioned into two parts and implemented into two FPGAs. There are 300 inter-FPGA tracks between these two FPGAs. According to [START_REF] Inagi | Globally Optimal Timemultiplexing of Inter-FPGA Connections for Multi-FPGA Prototyping Systems[END_REF], only the cut nets that have the same driver FPGA and the same receiver FPGAs, can be multiplexed together.

In this manuscript, all the cut nets are from one FPGA (i.e. FPGA A) to another FPGA (i.e. FPGA B) in order to facilitate the comparison. The result of the performance comparison is shown in Figure 2.19. The X-axis is the TDM ratio and the Y-axis is the system clock frequency. When the TDM ratio is 1, there is no need for multiplexing. The performance is compared when the TDM ratio is more than 1. If the TDM ratio is noted as ratio, there will be 300 * ratio cut nets due to that there are 300 inter-FPGA tracks between two FPGAs. When Logic Multiplexing is implemented, one inter-FPGA track is used to propagate one multiplexed data, the maximum number of cut nets passing through a logic multiplexer mux_logic = 300 * ratio 300 = ratio and the system clock frequency sys_clk is calculated according to Equation 2 The results show that as the TDM ratio is increasing, the achieved performance in Logic Multiplexing and ISERDES/OSERDES is reducing. In a duo-FPGA platform, ISERDES/OSERDES can always have higher performance than Logic Multiplexing. Nevertheless, the performance in ISERDES/OSERDES steps down instead of sliding down as in Logic Multiplexing, due to that the latency of SERDES is not changed when several OSERDES inputs (resp. ISERDES outputs) are not connected.

Conclusion of Inter-FPGA Communication Architectures

Multi-FPGA boards are widely used for rapid system prototyping. Nevertheless, these boards suffer from large timing delays in inter-FPGA communication compared to intra-FPGA net delays, as well as a limited bandwidth between FPGAs due to limited I/Os per FPGA. In order to solve the I/O pin limitation problem, cut nets are sent between FPGAs in a pipelined way using the Time-Division-Multiplexing technique. The maximum number of cut nets passing through one FPGA I/O is called the TDM ratio. There are three inter-FPGA communication architectures: Logic Multiplexing, ISERDES/OSERDES and Multi-Gigabit Transceiver (MGT). As we have seen, only Logic Multiplexing and ISERDES/OSERDES are today used for Time-Division-Multiplexing in the multi-FPGA based prototyping. In this Section, the achieved performances of these two multiplexing architectures are compared in different TDM ratios. Experiments are done in a duo-FPGA platform with the testbench LFSR to validate the achieved performance. The results show that as the TDM ratio is increasing, the multi-FPGA system performance is worsening and ISERDES/OSERDES can always have higher performance than Logic Multiplexing.

Conclusion

In this Chapter, the related work of the manuscript and the background of inter-FPGA communication are discussed.

Despite the fact that hardware/software integration is a more and more concerning problem, few papers have been published regarding multi-FPGA prototyping platforms. As there are three different multi-FPGA prototyping platforms, the conclusions of the related works are classified into four parts: First, the existing tools do not automatically route and multiplex cut nets in multipoint tracks, which waste FPGA I/Os that are already a scarce resource. In Chapter 3, which focuses on the off-the-shelf platform, we will target the direct architecture and we propose to spare FPGA I/Os by automatically routing and multiplexing multi-terminal nets over multi-point tracks. Second, the existing design flows for creating a custom platform are mainly manual and iterative (thus are time-consuming), and the optimization of the cost and the performance depends on the designers' experience. In Chapter 4, which focuses on the custom platform, we propose an automatic design flow for creating a custom multi-FPGA platform. Third, according to our knowledge, there is no tool to automatically have a solution for the cable distribution. In Chapter 5, which focuses on the cabling platform, we propose a cabling platform with an algorithm to automatically optimize the cable distribution and the external interface placement. Finally, different platforms are compared.

The performance of multi-FPGA platforms is limited by the inter-FPGA tracks. As there are fewer available inter-FPGA tracks than the number of cut nets, several cut nets need to be multiplexed and sent together onto a single track on the platform. There are two multiplexing techniques used for multi-FPGA based prototyping: Logic Multiplexing and ISERDES/OSERDES.

The achieved performance of these two architectures is compared in an off-the-shelf duo-FPGA platform. The results show that as the TDM ratio is increasing, the multi-FPGA system performance is worsening and ISERDES/OSERDES can always have higher performance than Logic Multiplexing.

Chapter 3

Hardwired Off-the-Shelf Multi-FPGA Platform

Introduction

The previous Chapter has discussed the background (the state of the art of different multi-FPGA platforms and inter-FPGA communication architectures). The purpose of this Chapter is to discuss the off-the-shelf platform and present a typical design implementation flow for multi-FPGA platforms. The main contribution is to propose an algorithm that routes and multiplexes multi-terminal nets in multi-point tracks to spare FPGA I/Os, in order to improve the performance.

The rest of this Chapter is organized as follows. Section 3.2 is an overview of the off-theshelf platform. Section 3.3 shows the typical implementation flow for multi-FPGA platforms.

From Section 3.4 to Section 3.6, each step of the flow is detailed. In Section 3.7, experiments are conducted using Gaisler Research Benchmarks [START_REF] Gaisler | Gaisler Research[END_REF] and OpenCores Benchmarks [START_REF] Opencores | [END_REF]. Finally, Section 3.8 concludes this Chapter.

Platform Overview

In this manuscript, the off-the-shelf platform consists of a ready-made generic multi-FPGA board, where all the inter-FPGA connections are fixed and realized using PCB traces. The connections to external interfaces are also fixed but can be realized using PCB traces or connectors (connected to daughter boards). An example of such platform is the commercial platform DNV7F4A as shown in

Logic Synthesis

The first step of the typical implementation flow is to synthesize the input design targeting FPGAs, which translates the design from RTL to netlist. After the logic synthesis, the logic capacity of the input design is represented by the number of logic elements contained. There are four kinds of logic elements which are taken into consideration: Look-Up [START_REF] Quartus | Design Entry and Synthesis[END_REF], or third party tools such as Synopsys Synplify tool [START_REF] Synplify | Logic Synthesis for FPGA Implementation[END_REF] and Mentor Graphics Precision tool [START_REF] Precision | Advanced FPGA Synthesis[END_REF].

Design Partitioning

Due to that the silicon area overhead of FPGA versus ASIC technology has been measured to be about 40x [START_REF] Kuon | Quantifying and Exploring the Gap Between FPGAs and ASICs[END_REF], FPGA technology requires that an ASIC logic design should be partitioned across multiple FPGA devices to achieve the necessary logic capacity. In the partitioning, the design is split into pieces small enough to fit into the individual FPGAs.

In the following of this Section, we will introduce the notion of design partitioning, which is not the objective of this manuscript but only gives an overview. The partitioning has been an area of active research from 1970s [START_REF] Kernighan | An Efficient Heuristic Procedure for Partitioning of Electrical Circuits[END_REF] to 2000s [START_REF] Selvakkumaran | Multi-Resource Aware Partitioning Algorithms for FPGAs with Heterogeneous Resources[END_REF].

There have been numerous approaches tried, and several promising approaches have emerged.

While many of these have been focused on bi-partitioning, or breaking into exactly two partitions [START_REF] Kernighan | An Efficient Heuristic Procedure for Partitioning of Electrical Circuits[END_REF]] [Fiduccia and Mattheyeses, 1982] [Krishnamurthy, 1984] [Hagen and Kahng, 1992] [Dutt, 1993] [Bui and Moon, 1994] [Riess et al., 1994] [Yang and Wong, 1994],

there has been work on extending them to k-way partitioning [Sanchis, 1989] [Chan et al., 1994] [ [START_REF] Woo | An Efficient Method of Partitioning Circuits for Multiple-FPGA Implementation[END_REF]] that splits into more than two partitions.

To obtain more than two partitions, there are two major approaches:

• Recursive bi-partitioning (also called multi-level bi-partitioning): Partitions are split recursively until the desired number of partitions is obtained.

• Simultaneous computation of all partitions (also called k-way partitioning).

In the study of [START_REF] Yarack | An Evaluation of Move-Based Multi-Way Partitioning Algorithms[END_REF], several move-based k-way approaches have been compared. All these approaches have in common that for k-way partitioning the amount of memory needed grows quadratic with k. Therefore, k-way approaches can not be applied, if k becomes large. Then, the approaches are restricted on recursive bi-partitioning.

There are two kinds of constraints in the partitioning: connection constraints (the distribution of inter-FPGA connections) and size constraints (the number of logic elements in one FPGA).

With these constraints, the problem of optimally partitioning a netlist (modeled as hypergraph) is known to be NP-hard [START_REF] Garey | Computers and Intractability; A Guide to the Theory of NP-Completeness[END_REF]. And the netlist partitioning algorithms (or called as the hypergraph partitioning algorithms) are heuristics.

However, previous presented bi-partitioning works have primarily focused on problems where there are no restrictions on how the partitions are interconnected (that is, there is no reason to prefer or avoid connections between any pair of partitions). Unfortunately, in many multi-FPGA platforms only a subset of the FPGAs are connected, and routing between FPGAs not directly connected (or less directly connected) will use many more resources than routing between connected (or highly connected) FPGAs. Works that have handled the limited connectivity problem take a significant amount of time [START_REF] Roy | A Timing Driven N-Way Chip and Multi-Chip Partitioner[END_REF], possibly even exponential in the number of partitions [Vijayan, 1990].

[ [START_REF] Kuznar | Cost Minimization of Partitions into Multiple Devices[END_REF] proposed an algorithm to partition a given netlist into multiple FPGA types to minimize total cost, where each FPGA type in a given library can have distinct price, size, and pin capacity. Their method recursively applies a variant of [START_REF] Fiduccia | A Linear-time Heuristic for Improving Network Partitions[END_REF] bi-partitioning which allows some uphill moves. In subsequent work, [START_REF] Kuznar | Multi-way Netlist Partitioning into Heterogeneous FPGAs and Minimization of Total Device Cost and Interconnect[END_REF] allow logic resources (logic elements) to be duplicated, i.e., they introduce functional replication to minimize the total cost of FPGAs and the total number of cut nets. Finally, [START_REF] Chou | Circuit Partitioning for Huge Logic Emulation Systems[END_REF] [ [START_REF] Huang | Multi-Way System Partitioning into a Single Type or Multiple Types of FPGAs[END_REF] have proposed an algorithm to partition a circuit into instances of a single FPGA type, such that the number of FPGAs is minimized. Their algorithms significantly improve over recursive [START_REF] Fiduccia | A Linear-time Heuristic for Improving Network Partitions[END_REF] on benchmarks.

The approach of [START_REF] Hauck | Logic Partition Orderings for Multi-FPGA Systems[END_REF] to the multi-FPGA partitioning problem is to harness the work on standard bi-partitioning for some restricted situations (i.e. connection constraints and size constraints on each partition). This is done by recursively applying the bi-partitioning algorithms to the circuit until it is cut into the required number of pieces. Then, the multi-level bi-partitioning algorithm has been improved to dedicate to larger number of partitions in [START_REF] Alpert | Multilevel Circuit Partitioning[END_REF]] [Karypis et al., 1997] [Drechsler et al., 2002].

However, modern FPGA architectures incorporate heterogeneous logic resources (e.g., LUT, REG, RAM, DSP and etc.). Thus, the partitioning algorithm must now ensure that the logic resources used in each partition can be accommodated by the logic resources provided at the different regions of the FPGA. For example, a partitioning solution that places most of the REG on one side of the bi-partitioning and most of the RAM on the other side of the bi-partitioning, even if it is balanced in terms of the total number of logic resources on either side of the cut, it is not very useful for FPGA placement as it may over-subscribe these two logic resource types.

As a result, previous presented partitioning algorithms [START_REF] Kuznar | Multi-way Netlist Partitioning into Heterogeneous FPGAs and Minimization of Total Device Cost and Interconnect[END_REF]] [Chou et al., 1994] [Huang and Kahng, 1995] [Hauck and Borriello, 1995] [Alpert et al., 1997] [Cong et al., 1997] [Wichlund and Aas, 1998] [Karypis et al., 1997] [Cong and Lim, 2000] can not be used to develop partitioning-based placement methods for FPGAs with heterogeneous logic resources, as they can lead to partitions that have highly unbalanced logic resource requirements. To illustrate this, a multi-level hypergraph partitioning algorithm is used (hMetis [START_REF] Karypis | hMetis 1.5: A Hypergraph Partitioning Package[END_REF])

to bi-partition twelve different circuits synthesized for the Xilinx Virtex-2 architecture, which contain cells that map to different logic resources. In [START_REF] Selvakkumaran | Multi-Resource Aware Partitioning Algorithms for FPGAs with Heterogeneous Resources[END_REF], a new class of multi-resource hypergraph bi-partitioning algorithms are presented to simultaneously balance the different logic resources assigned to each one of the partitions.

Nowadays, there are four commercial tools: Cadence Protium tool [START_REF] Protium | Protium Rapid Prototyping Platform[END_REF], Synopsys

Certify tool [Synopsys, 2014], Auspy tool [START_REF] Auspy | Auspy Development Inc[END_REF] and Flexras Wasga tool [START_REF] Flexras | Flexras Technologies[END_REF],

targeting at the partitioning problem. In this manuscript, the design partitioning is done by the Flexras Wasga tool provided by the company Flexras Technologies [START_REF] Flexras | Flexras Technologies[END_REF]. The partitioning tool integrates the multi-resource multi-level hypergraph partitioning algorithm taking the connection constraints and the size constraints into consideration. The objective of the design partitioning is to minimize the total number of cut nets, which contributes to increase the performance of multi-FPGA prototyping.

Design Routing

After the partitioning, the design has been divided into several parts. Then, the partitioned design will be routed in a multi-FPGA platform. In the partitioned design, there are two types of cut nets as shown in the example of is proposed by [START_REF] Turki | Iterative Routing Algorithm of Inter-FPGA Signals for Multi-FPGA Prototyping Platform[END_REF]. Nevertheless, this algorithm only uses 2-point tracks and routes both 2-terminal and multi-terminal nets through a sequence of 2-point tracks. In this manuscript, we enhance this algorithm to efficiently route multi-terminal nets in multi-point tracks in order to save FPGA I/Os.

Multi-Terminal Net Routing

In the manuscript, a multi-point track is assumed to connect all the FPGAs in a multi-FPGA board. Therefore, if there are m FPGAs in the board, multi-point tracks are m-point. The number of terminals for a cut net in the design, which is noted as n, varies from 2 to m. An example of routing a multi-terminal net in 2-and multi-point tracks will be studied in the following. • If there are tracks between FPGA 0 and FPGA 1 and between FPGA 0 and FPGA 2, the multi-terminal net will be routed as shown in Figure 3.4(a).

• If there is no track between FPGA 0 and FPGA 2, but there are tracks between FPGA 0 and FPGA 1 and between FPGA 1 and FPGA 2, the multi-terminal net will be routed as shown in Nevertheless, if m is much more than n, precisely if m is more than 2(n -1), using the multi-point track can be a waste of FPGA I/Os.

Routing Algorithm

The proposed routing algorithm is shown in Figure 3.6. The pink areas are the added functions.

The algorithm tries to minimize the maximum number of cut nets passing through one multiplexer (noted as mux). If Logic Multiplexing is chosen as the inter-FPGA communication architecture, mux = mux_logic. If ISERDES/OSERDES is chosen, mux = mux_serdes.

Compute initial mux

After loading the design and the board model, the algorithm will calculate initial mux by obtaining the maximum ratio of cut nets and inter-FPGA tracks between each pair of FPGAs.

Group cut nets to GNet

The cut nets, which have the same driver FPGA and the same receiver FPGAs, are grouped together. The group, which is called GNet, will be put in one track. Each GNet contains a maximum of mux cut nets.

Enable routing in multi-point tracks

An example of a board with both 2-and multi-point tracks is shown in Figure 3.7(a). There are a 2-point track between FPGA 1 and FPGA 2 and a multi-point track connecting all the FPGAs.

• The existing routing algorithm [START_REF] Turki | Iterative Routing Algorithm of Inter-FPGA Signals for Multi-FPGA Prototyping Platform[END_REF] does not enable routing in multi-point tracks. The algorithm considers a multi-point track as a 2-point track. Executing the algorithm leads to the solution presented in Figure 3.7(b). The driver is in FPGA 0, one receiver is in FPGA 1 and another receiver is in FPGA 2. The algorithm considers the multi-point track as the 2-point track between FPGA 0 and FPGA 1. Therefore, in order to route from the driver in FPGA 0 to the receiver in FPGA 2, 4 FPGA I/Os and a routing hop are needed. algorithm [START_REF] Cormen | Introduction to Algorithms[END_REF]. At the end of the first iteration, tracks may be congested because multiple GNets try to use one track. During subsequent iterations, the cost of using a track is increased, based on the number of GNets that share the track, and the history of congestion on that track. Thus, GNets are forced to negotiate for tracks. If a track is highly congested, GNets which can use lower congestion alternatives are forced to do so. On the other hand, if the alternatives are more congested than the track, then a GNet may still use that track. Due to that one track can only contain one GNet, if there are two GNets in one track in the result of PathFinder, there is a conflict. A routing result is feasible only when the number of conflicts is 0.

An example is shown in Figure 3.8. There are two GNets: one GNet N1 from FPGA 0 (driver)

to FPGA 3 (receiver) and another GNet N2 from FPGA 1 (driver) to FPGA 3 (receiver). There is one inter-FPGA track between FPGA 0 and FPGA 1, FPGA 0 and FPGA 2, FPGA 1 and FPGA 3, FPGA 2 and FPGA 3. N1 will be routed before N2. In the first iteration, there are two choices of paths for N1 according to Dijkstra's shortest path algorithm. One path is (FPGA 0 -FPGA 1 -FPGA 3). Another path is (FPGA 0 -FPGA 2 -FPGA 3). N1 is freely routed. In the example, N1 chooses the path (FPGA 0 -FPGA 1 -FPGA 3). The congestion (noted as C) of the tracks (between FPGA 0 and FPGA 1, FPGA 1 and FPGA 3) plus 1. N2 chooses its shortest path (FPGA 1 -FPGA 3). The congestion of the track (between FPGA 1 and FPGA 3) plus 1. Both N1 and N2 pass through the track between FPGA 1 and FPGA 3. Therefore, there is a conflict. In the second iteration, for N1, the path (FPGA 0 -FPGA 1 -FPGA 3) has higher congestion value than the path (FPGA 0 -FPGA 2 -FPGA 3). Therefore, N1 chooses the path (FPGA 0 -FPGA 2 -FPGA 3).

Then, N2 chooses its shortest path (FPGA 1 -FPGA 3). After that, a feasible solution has been found. The quality of the routing is independent from the routing order of GNets. The proposed algorithm tries to minimize mux as shown in Figure 3.6,

• If feasible, decrease the mux and iterate the previous steps beginning from grouping cut nets with the new mux.

• If not feasible, examine whether there is a feasible solution found before:

-If so, exit with the mux of the previous feasible solution.

-If not, increase the mux and iterate the previous steps with the new mux.

Results of the proposed routing algorithm

As discussed above, intermediate FPGAs may be necessary in routing 2-point tracks if there is no available track between the FPGA (the Driver) and the FPGA (the Receiver). Routing hop (noted as hop) is defined as the number of intermediate FPGAs needed. For example, the routing of GNet N1 in Figure 3.8 causes a routing hop. When the routing hop exists, the performance degrades.

At the end of the proposed routing algorithm, the results consist of a group of feasible solutions (mux, hop). The optimal (mux, hop) will be chosen to achieve the highest performance.

Performance Evaluation

The inter-FPGA communication architectures have been presented in Section 2.3 and the achieved performance has been evaluated. Nevertheless, the routing hops are not taken into consideration.

In this Section, the inter-FPGA communication architectures with routing hops will be proposed and the performance will be evaluated. As there are two inter-FPGA communication architectures (Logic Multiplexing and ISERDES/OSERDES), this Section is classified into two parts. if _clk_0 in ISERDES is generated and propagated from FPGA 0, but the fast clock frequency if _clk_1 in OSERDES is generated inside FPGA 1. Therefore, there is a phase shift between if _clk_0 and if _clk_1, and there is a difficulty to realize the routing hops of ISERDES/OSERDES in a pipelined way. In this manuscript, data starts to propagate from ISERDES of FPGA 1 to OS-ERDES of FPGA 1 only when all the data is received in ISERDES of FPGA 1. Thus, when considering routing hops, the achieved performance calculated in Equation 2.3 will be modified as in Equation 3.2. In the performance evaluation, the optimal (mux, hop) pair will be chosen to achieve the higher performance according to the chosen inter-FPGA communication architecture.

sys_clk = if _clk lat = 250 M Hz (7 + ceil( mux_serdes 4 )) * (1 + hop) (3.2)

Experiments

Targeted Platform

The proposed routing algorithm is independent of FPGA types.

Results

In the experiments, four testbenches (leon2, leon3_avnet, leon3mp, netcard) from Gaisler Research Benchmarks [START_REF] Gaisler | Gaisler Research[END_REF] and three testbenches (vga_lcd, des_perf, ethernet) from Open- The percentage of each ter for each testbench is shown in the Table. The number of total cut nets of most testbenches is largely more than the available FPGA I/Os (max. 1200) as shown in Figure 3.12(a). According to the previous discussion, routing an n-terminal net in a multi-point track can reduce the used FPGA I/Os when m ≤ 2(n -1) and can waste the FPGA I/Os when m > 2(n -1) compared to routing in a 2-point track. In the testbenches (netcard, vga_lcd, des_perf), there are more than 18% of cut nets can benefit from multi-point routing as shown in Figure 3.12(b). After the design partitioning, each testbench will be routed in the targeted off-the-shelf platform DN9000K10PCI. The results are shown in Table 3.2. The number of cut nets passing through one multiplexer is noted as mux (mux = mux_logic in case of Logic Multiplexing, mux = mux_serdes in case of ISERDES/OSERDES). The maximum number of intermediate FPGAs needed from the driving FPGA to the receiving FPGA (routing hops) is noted as hop.

There are four scenarios depending on the global routing algorithm and the inter-FPGA communication architecture.

• The Turki's algorithm [START_REF] Turki | Iterative Routing Algorithm of Inter-FPGA Signals for Multi-FPGA Prototyping Platform[END_REF], Logic Multiplexing (noted as LM)

• The Turki's algorithm, ISERDES/OSERDES (noted as SERDES)

• The proposed algorithm, Logic Multiplexing

• The proposed algorithm, ISERDES/OSERDES The results show that the system clock frequency of all the cases is about several to tens of MHz for all the testbenches as shown in Figure 3.13. There are two kinds of comparisons in Table 3.2. First, ISERDES/OSERDES will be compared with Logic Multiplexing in performance.

Then, the proposed algorithm will be compared with the Turki's algorithm in performance.

Figure 3.13: The achieved performance in the off-the-shelf platform

Logic Multiplexing VS ISERDES/OSERDES

According to Chapter 2, in a duo-FPGA platform, ISERDES/OSERDES can always have higher performance than Logic Multiplexing. Nevertheless, this is not the case in a multi-FPGA platform.

Table 3.2 shows that, in the testbenches (netcard, vga_lcd, des_perf), the achieved performance in ISERDES/OSERDES is lower than that in Logic Multiplexing (up to -47% gain).

This is due to that, in a multi-FPGA platform, the routing hops (hop) need to be considered because the performance is influenced by both mux and hop. In Logic Multiplexing, the routing hops are realized in a pipelined way according to Section 3.6.3. Therefore, one hop has the same impact on the performance as one mux according to Equation 3.1. The existence of routing hops may increase the achieved performance due to that the routing hops can reduce mux. As a conclusion, Logic Multiplexing is a hop-friend technique. In ISERDES/OSERDES, the routing hops are not realized in a pipelined way according to Section 3.6.3. The transmission of data in the intermediate FPGA can only start when all the data is received from the Driver FPGA. Therefore, one hop has more impact on the performance than one mux according to Equation 3.2. The existence of routing hops may degrade the performance. As a conclusion, the chosen ISERDES/OSERDES implementation is a hop-hostile technique. In the testbenches (leon2, leon3_avnet, leon3mp) that have not many multi-terminal nets, the mapping process in ISERDES/OSERDES can result no routing hop and achieve higher performance than Logic Multiplexing (up to 92% gain). Nevertheless, in the testbenches (netcard, vga_lcd, des_perf) that have many multi-terminal nets, the routing hops can not be avoided in the targeted off-the-shelf platform even with ISERDES/OSERDES (hophostile). Therefore, in the testbenches (netcard, vga_lcd, des_perf), the achieved performance in ISERDES/OSERDES is lower than that in Logic Multiplexing (up to -47% gain).

Turki's algorithm VS proposed algorithm

Table 3.2 shows that the proposed routing algorithm can reduce the mux and hop. If Logic Multiplexing (resp. ISERDES/OSERDES) is chosen, the proposed routing algorithm can increase the performance up to 25% (resp. 22%) gain compared to the Turki's algorithm in the off-the-shelf platform by routing multi-terminal nets in multi-point tracks.

Conclusion

In this Chapter, the off-the-shelf platform is discussed and the typical implementation flow for multi-FPGA platforms is presented. The main contribution is to propose an algorithm that routes and multiplexes multi-terminal nets in multi-point tracks to spare FPGA I/Os, in order to improve the system clock frequency. Experiments are conducted using four testbenches from Gaisler Chapter 4

Hardwired Custom Multi-FPGA Platform

Introduction

The previous Chapter has discussed the off-the-shelf platform and presented the typical implementation flow. The purpose of this Chapter is to discuss the custom platform and propose an automatic design flow for creating a custom platform.

The custom platform consists of a build-your-own specific multi-FPGA board, where all the inter-FPGA connections are realized using PCB traces. Different from the off-the-shelf platform that has generic and balanced connections, the connections inter FPGAs as well as the connections to external interfaces of the custom platform are user-defined and tailored for a specific design.

70% of multi-FPGA prototyping platforms are home-made custom platforms due to performance requirement (in terms of the system clock frequency), external interfaces, and cost. Indeed home-made custom platforms are tailored for a specific design and external interfaces, and the cost decreases as more platforms are needed. Nevertheless, crafting a home-made custom multi-FPGA platform is today a manual process. Only few point tools such as Cadence FSP tool [FSP, 2014] are available. Therefore, three critical aspects exist:

• Productivity: Crafting a home-made custom multi-FPGA platform is a time-consuming process (about 9 months [Sekhar, 2014]). The performance and the cost of the platform lie on the FPGA expertise and SoC DUT knowledge of the prototyping team.

• Exploration: There are many different FPGA types (i.e. vendor: Xilinx [Xilinx, 2014] or Altera [START_REF] Altera | FPGA CPLD and ASIC from Altera[END_REF], family: Virtex-7 or Stratix5, device: 2000T or GXAB, and package:

FLG1925 or F1932), and different FPGA types have different logic capacity and numbers of FPGA I/Os. For a given design, the FPGA type chosen by the engineers influences the achieved performance and cost. The tradeoff between the performance and the cost exists due to that the performance of multi-FPGA platforms is limited by the inter-FPGA communications [START_REF] Amos | FPGA-Based Prototyping Methodology Manual[END_REF]. As crafting a custom platform is a manual and timeconsuming process, the board exploration with different FPGA types, which helps the engineers to design an optimum prototyping platform (cost-optimal, performance-optimal or intermediate), can not be done.

• Performance: As the ratio between the logic capacity and the number of FPGA I/Os is increasing at an exponential rate, it becomes more and more challenging to design a highperformance multi-FPGA platform.

The rest of this Chapter is organized as follows. Section 4.2 presents an example of the custom platform. Section 4.3 shows the overview of the automatic design flow for creating a custom platform. From Section 4.4 to Section 4.8, each step of the flow is detailed. Then, Section 4.9 details the proposed automatic design flow. Section 4.10 depicts the board exploration. In Section 4.11, experiments are conducted using Gaisler Research Benchmarks [START_REF] Gaisler | Gaisler Research[END_REF] and OpenCores

Benchmarks [START_REF] Opencores | [END_REF]. Finally, Section 4.12 concludes this Chapter.

Platform Overview

The custom platform consists of a build-your-own specific multi-FPGA board, where all the inter-FPGA connections are realized using PCB traces. The advantage of the custom platform is that it is tailored for a specific design.

Specific Design

In Figure 4.1, a custom platform is created for a specific design. The design contains the following features:

• Processor

• Floating-point unit (FPU) and custom co-processor (CP)

• Separate instruction and data cache

• AHB and APB on-chip buses 5. Map the SoC/ASIC design on this latter board, meaning that the design is partitioned and routed into each FPGA.

The input design is synthesized targeting FPGAs (from RTL to design netlist). As the design is bigger than the FPGA, the design netlist is partitioned into several parts according to the number of FPGAs. Each part's capacity fits in a single FPGA. The signals crossing design's parts located in different FPGAs are called cut nets. Then, the cut nets are routed meaning that a cut net is allocated to an inter-FPGA track in the custom platform. As there are fewer available inter-FPGA tracks than the number of cut nets, several cut nets need to be multiplexed and sent together onto a single track on the platform.

6. Run the PnR Compiler with the design sub-netlists to generate the bitstreams for each FPGA.

Then, the generated bitstreams are downloaded into the platform to model the design.

The logic elements in each design sub-netlist are translated and placed in the corresponding logic elements in each FPGA. FPGA internal routing resources are chosen to connect the placed logic elements inside one FPGA. Then, the bitstreams are generated for each FPGA and are downloaded into the platform to model the design.

Problem

The custom platform is built, tailored to a specific design. As discussed in Section 2.2.2 of the state of the art, crafting a custom platform is today a manual and iterative process, thus is timeconsuming (about 9 months [Sekhar, 2014]). Only few point tools such as Cadence FSP tool [FSP, 2014] are available. The performance of multi-FPGA platforms depends on the inter-FPGA data rate and tracks distribution. The data rate for the off-the-shelf platform is high (~1 Gbps) due to that it is ready-made by FPGA experts in commercial companies [START_REF] Protium | Protium Rapid Prototyping Platform[END_REF]] [DINI, 2014].

For the custom platform done by the in-house team, the data rate depends on the designers' experience on how to design the high-speed signaling, high performance multiplexing and so forth. The tracks distribution for the off-the-shelf platform is generic and balanced as shown in Figure 3.1.

For the custom platform done by the in-house team, it depends on the designers' experience on how to distribute tracks in order to achieve higher performance. Naturally, the custom platform should achieve higher performance than the off-the-shelf platform because it is tailored for the given design. Unfortunately, we can see that the performance of the custom platform heavily depends on designs' experience because designing such platform is still a manual process. Therefore, we often see not so good performance platforms.

The Overview of the Automatic Design Flow

In 5. Interconnect Synthesis: Inter-FPGA tracks are distributed according to the distribution of cut nets and a board netlist is generated.

6. Design Routing / Multiplexing: The cut nets are routed meaning that a cut net is allocated to an inter-FPGA track in the custom platform. As there are fewer available inter-FPGA tracks than the number of cut nets, several cut nets need to be multiplexed and sent together onto a single track on the platform.

Before launching the automatic design flow, the FPGA Lib need to be established. The chosen FPGA for the automatic design flow need to be contained in the FPGA Lib.

FPGA Lib

In the FPGA Lib, there are hundreds of different FPGAs, which have different types depending on the vendor (i.e. Xilinx [Xilinx, 2014] or Altera [START_REF] Altera | FPGA CPLD and ASIC from Altera[END_REF]), the family (i. • Logic Capacity (represented by the number of logic elements contained in the FPGA). There are four kinds of logic elements which are taken into consideration: Look-Up Table (noted as LU T ), Register (REG), Block RAM (RAM ), and Embedded DSP (DSP ).

• I/O Capacity (represented by the number of I/Os, the number of I/O pairs in LVDS and the number of banks contained in the FPGA).

• Estimated Price (7 $ per ~1000 logic elements).

Logic Synthesis

The first step of the proposed automatic design flow is to synthesize the input design targeting FPGAs, which translates the design from RTL to netlist. After the logic synthesis, the logic capacity of the input design is represented by the number of logic elements (LU T , REG, RAM , and DSP ) contained. This step can be realized by Xilinx XST Synthesis tool [XST, 2014], Altera

Quartus Synthesis tool [START_REF] Quartus | Design Entry and Synthesis[END_REF], or third party tools such as Synopsys Synplify tool [Synplify, 2014] and Mentor Graphics Precision tool [START_REF] Precision | Advanced FPGA Synthesis[END_REF].

Estimate the number of FPGAs

After the logic synthesis, the next step is to estimate the minimum number of FPGAs required in the platform according to the logic capacity of the input design, the logic capacity of the chosen FPGA type and the maximum FPGA logic capacity utilization. The logic elements in the prototyped design (resp. in the chosen FPGA) are noted as: LU T design , REG design , RAM design and DSP design (resp. LU T F P GA , REG F P GA , RAM F P GA and DSP F P GA ). The logic capacity of one FPGA can not be fully used due to the intra-routability, possible product upgrades of the prototyped design and etc. The maximum FPGA logic capacity utilization is noted as F . The minimum number of FPGAs required in the platform (noted as N U M F P GA ) is shown in Equation 4.1, which is the ceiling value of the largest ratio of logic elements in the prototyped design to the available logic elements in the target FPGA. If the prototyped design is DSP-intensive, it may be more beneficial to choose an FPGA with more DSPs.

N U M F P GA = ⌈max( LU T design F * LU T F P GA , REG design F * REG F P GA , RAM design F * RAM F P GA , DSP design F * DSP F P GA )⌉ (4.1)
When the logic elements in the prototyped design netlist exceeds the logic capacity of the chosen FPGA multiplied by the maximum FPGA logic capacity utilization, the multi-FPGA platform is needed. In this case, the minimum number of FPGAs required in the platform need to be estimated. This step can be realized by the commercial tool waCore provided by the company Flexras

Technologies [START_REF] Flexras | Flexras Technologies[END_REF]. As a result, an XML file that contains the information of the number of FPGAs required in the platform is generated.

External Interface Placement

One of the advantages for FPGA-based prototyping is to enable "real world" testing, while the prototyped designs are put into actual hardware and real external interfaces (such as DDR, PCI, Ethernet and etc.) are used. After estimating the number of FPGAs in the platform, the next step is to connect external interfaces to FPGAs.

There are many possibilities to connect external interfaces to FPGAs. Different possibilities result different results in performance. In order to achieve higher performance, an algorithm (either exhaustive or heuristic) need to be proposed to optimize the placement of external interfaces.

The exhaustive method can find the best performance by testing all the possibilities. While the number of tests is enormous (i.e. NP-hard problems), heuristic methods need to be implemented to achieve the optimal performance. As the number of external interfaces in a prototyped design is usually about several to dozens, the number of possibilities is limited. In the Algorithm, different placements of external interfaces can be classified into three different types of cases: all the external interfaces connected to the same FPGA, each external interface connected to a different FPGA, and other cases where at least two FPGAs are used and at least one FPGA is connected with several external interfaces. If the placement is feasible, each case will be implemented respectively by the rest steps of the automatic design flow (presented in Section 4.3) to evaluate the achieved performance. A feasible placement means that the available FPGA I/Os after assigning the XIs are still more than 0 for each FPGA. Finally, the automatic design flow chooses the case that achieves the highest performance. The number of FPGAs in the multi-FPGA platform is noted as num_f pga. The number of external interfaces is noted as num_inter. The table XI[...] represents the placement of external interfaces (i.e. XI[0] = 0 means that the external interface numerated as 0 is connected to the FPGA 0). The maximum number of FPGAs connected to external interfaces is noted as num. If num_f pga > num_inter, num will equal num_inter. If not, num will equal num_f pga. The number of FPGAs connected to external interfaces is noted as i.

• If i = 1, it means that all the external interfaces will be placed in one FPGA, therefore the algorithm will go to the function Place_all_the_XIs_to_the_same_FPGA().

• If i = num_inter, it means that each external interface will be placed in a different FPGA, therefore the algorithm will go to the function Place_each_XI_to_a_different_FPGA().

• If 1 < i < num_inter, it means that at least two FPGAs are used and at least one FPGA is connected with several external interfaces, therefore the algorithm will go to the function Other_case().

An example, where num_f pga is 4 and num_inter is 3, will be studied in this Section. In the example, 4 FPGAs are numerated as FPGA 0, FPGA 1, FPGA 2 and FPGA 3.

Other cases

In the other cases, where at least two FPGAs are used and at least one FPGA is connected with several External Interfaces, the number of different cases depends on the number of FPGAs connected to external interfaces (num) and the number of external interfaces in the prototyped design (num_inter), thus is pow(num, num_inter) cases. There can be some identical cases (i.e. PCI in one FPGA, DDR and Ethernet in another FPGA as shown in external interfaces is noted as num, the external interfaces will only be connected to the FPGAs indexed between (0, num -1) to remove the symmetry. In the example, the number of FPGAs connected to external interfaces is 2. Therefore, the external interfaces will only be connected to the FPGAs indexed between (0, 1) to remove the symmetry. In the example, the case as shown at the middle of If the cases are not identical, they need to be executed to obtain the result. Finally, the selected placement is the one with the highest performance.

Conclusion

In this Section, we proposed an external interface placement algorithm (that is realized in python programming language). Different placements of external interfaces can be classified into three different types of cases: all the external interfaces connected to the same FPGA, each external interface connected to a different FPGA, and other cases where at least two FPGAs are used and at least one FPGA is connected with several external interfaces. Finally, the automatic design flow chooses the case that achieves the highest performance. As a result, the FPGA-Interface tracks file in XML format is generated.

Design Partitioning

After the external interface placement, the next step is the design partitioning. As discussed in Section 3.5, the design partitioning in the multi-FPGA prototyping is a NP-hard problem [START_REF] Garey | Computers and Intractability; A Guide to the Theory of NP-Completeness[END_REF]. The partitioning needs to take the connection constraints (the distribution of inter-FPGA connections) into consideration, in order to achieve a higher performance solution compared to the partitioning without any board information. In this manuscript, the design partitioning is done by the commercial tool waPart provided by the company Flexras Technologies [START_REF] Flexras | Flexras Technologies[END_REF]. Nevertheless, the custom platform with the distribution information of inter-FPGA connections is not available at this moment. Therefore, before partitioning the design, a temporary balanced platform need to be generated. Then, the partitioning tool takes the connection constraints and the size constraints (the number of logic elements in one FPGA) of the temporary balanced platform into consideration. The objective of the design partitioning is to minimize the total number of cut nets, which contributes to increase the performance of multi-FPGA prototyping.

The generation of the board netlist for the temporary balanced platform is detailed in

Reset Chain

According to [START_REF] Amos | FPGA-Based Prototyping Methodology Manual[END_REF], the FPGAs in the platform need to be reseted at the same time, which is called reset synchronization. The asynchronous global reset enters in each FPGA and is captured by a flip-flop (FF) in order to be synchronous to the system clock edge. As the delay of the Interface global reset to different FPGAs can be different (i.e. the delay of the Interface global reset to FPGA 2 is larger than the delay of the Interface to FPGA 0), there is a risk that the reset is captured in different system clock edges in different FPGAs. The reset chain architecture as depicted in Figure 4.10(a) is proposed in [START_REF] Amos | FPGA-Based Prototyping Methodology Manual[END_REF] to resolve this problem. In the architecture, the global reset enters in one FPGA and then propagated to other FPGAs by inter-FPGA tracks. The Pipe1 is not replicated because the synchronization of the incoming global reset has to be done in only one place. The output of Pipe1 in FPGA 0 will drive two Pipe2: one in FPGA 0 and another in FPGA 1. The reset will be captured by these two Pipe2 in the same system clock edge due to that the inter-FPGA communication architectures (Logic Multiplexing and ISERDES/OSERDES) presented in Chapter 2.3 assure that the transmission of the data will be finished in one system clock cycle. Nevertheless, there is still a possibility that the pipeline stage in each FPGA would introduce delay and reduce the performance because if there is one Flip-Flop (noted as FF) in each stage, then it might be placed near the input pad, the output pad or anywhere in between; this might also be different in each FPGA. Therefore, the reset chain pipeline stage using three FFs is shown in

Generation of the Temporary Balanced Platform

After reserving I/Os for global signals, the reset chain and external interfaces, a temporary balanced platform will be generated. The first step is to choose the FPGA that has the minimum number of available I/Os. If the chosen FPGA is X, the number of inter-FPGA tracks between FPGA X and the other FPGA is the same for all the FPGA pairs. According to Equation 4.2, this number equals the number of available FPGA I/Os divided by the number of remaining FPGAs in the platform. In the example, the chosen FPGA is FPGA 0 and the minimum number of available I/Os is 1045 in case of Logic Multiplexing (resp. 497 in case of ISERDES/OSERDES). Therefore, 522 in case of Logic Multiplexing (resp. 248 in case of ISERDES/OSERDES) will be distributed between FPGA 0 and FPGA 1 and between FPGA 0 and FPGA 2 as shown in Figure 4.13(a) (resp.

as shown in Figure 4.13(b)).

T racks (X, Y ) = Avail I/Os on X number of F P GAs in the platf orm -1

Conclusion

In this Section, the design partitioning that partitions the design netlist into several parts is discussed. The first step is the generation of the temporary balanced platform by the self-developed tool (that is realized in python programming language). Then, the design netlist will be partitioned with the temporary balanced platform. The partitioning process is done by the commercial tool provided by the company Flexras Technologies [START_REF] Flexras | Flexras Technologies[END_REF]. As a result, the partitioned design netlist file in Verilog format and the cut nets distribution file in XML format are generated.

Interconnect Synthesis

After the design partitioning, the next step is to distribute inter-FPGA tracks according to the distribution of cut nets in order to tailor the platform for the given design. We call this step the Interconnect Synthesis. In this manuscript, we propose an Interconnect Synthesis Algorithm, which is depicted in Figure 4.15. The proposed interconnect synthesis algorithm is realized in python programming language.

Platform Generation

The rest of the design model will be processed according to the previous discussion to generate 2-point tracks. Finally, the generated custom platform is shown in Figure 4.24(a) for Logic Multiplexing (resp. as shown in Figure 4.24(b) for ISERDES/OSERDES).

Conclusion

In this Section, we proposed the interconnect synthesis algorithm that distributes inter-FPGA tracks according to the distribution of cut nets in order to tailor the platform for the given design. The proposed interconnect synthesis algorithm is realized in python programming language.

The results show that the generated platform can achieve less maximum number of cut nets passing through one multiplexer than the generic platform, thus higher performance. There are two types of inter-FPGA tracks: 2-point track and multi-point track. Therefore, the generated platforms have two types: "Platform with only 2-point tracks" and "Platform with 2-and multi-point tracks". In the experiments of Section 4.11, these two types of platforms will be compared in performance.

Automatic Design Flow for Creating a Custom Platform

The previous Sections have detailed the point tools for the main steps when creating a custom platform. In this Section, we propose an automatic design flow as depicted in Figure 4.25, which automatically joins the point tools. The proposed automatic design flow that is realized in python programming language, consists of the platform generation flow and the implementation flow.

In the platform generation flow, a custom platform tailored for a given design is automatically generated by only choosing the FPGA type used. In the implementation flow, the partitioned design is routed into the generated multi-FPGA platform. As most of off-the-shelf platforms, there is only one FPGA type used in one custom platform. Therefore, different custom platforms can be identified by the input design and the FPGA type. 1. Logic Synthesis: The input design is synthesized targeting FPGAs (from RTL to design netlist). This step can be realized by Xilinx XST Synthesis tool [START_REF] Xst | XST[END_REF], Altera Quartus Synthesis tool [START_REF] Quartus | Design Entry and Synthesis[END_REF], or third party tools such as Synopsys Synplify tool [START_REF] Synplify | Logic Synthesis for FPGA Implementation[END_REF] and Mentor Graphics Precision tool [START_REF] Precision | Advanced FPGA Synthesis[END_REF].

2. Estimate the number of FPGAs: When the logic elements in the prototyped design netlist exceeds the logic capacity of the chosen FPGA multiplied by the maximum FPGA logic capacity utilization, the multi-FPGA platform is needed. In this case, the minimum number of FPGAs required in the platform need to be estimated. This step can be realized by the commercial tool waCore provided by the company Flexras Technologies [START_REF] Flexras | Flexras Technologies[END_REF].

As a result, an XML file that contains the information of the number of FPGAs required in the platform is generated.

3. Interface Placement: External interfaces to FPGAs are connected. We propose an External Interface Placement Algorithm in Section 4.6 to do this step. The proposed external interface placement algorithm is realized in python programming language. At the end of this step, the FPGA-to-Interface tracks file in XML format is generated.

4. Design Partitioning: As the SoC/ASIC design is bigger than the FPGA, the design netlist is partitioned into several parts according to the number of FPGAs. Each part's capacity fits in a single FPGA. This step is done by the commercial tool waPart provided by the company

Flexras Technologies [START_REF] Flexras | Flexras Technologies[END_REF]. The partitioning needs to take the connection constraints (the distribution of inter-FPGA connections) into consideration, in order to achieve a higher performance solution. We propose an algorithm to generate a temporary balanced platform in Section 4.7 (realized in python programming language). At the end of this step, the partitioned design netlist in Verilog format and the cut nets distribution file in XML format are generated. After finishing the platform generation flow, the automatic design flow passes to the implementation flow. The implementation flow takes the partitioned design netlist, the board netlist, the FPGA lib and the user definition file as inputs. Due to that the design has already been synthesized and partitioned in the platform generation flow, the implementation is to route and multiplex the partitioned design into the generated platform to verify the functionality and to evaluate the performance. We propose a routing algorithm (that enhances the Turki's algorithm [START_REF] Turki | Iterative Routing Algorithm of Inter-FPGA Signals for Multi-FPGA Prototyping Platform[END_REF]) in Chapter 3 to do the design routing. The proposed routing algorithm is realized in C++ programming language. It generates a TCL format file that contains the information of the cut net to inter-FPGA track mapping. Then, the commercial tool waRoute provided by the company Flexras Technologies [START_REF] Flexras | Flexras Technologies[END_REF] will read this TCL format file, in order to multiplex several cut nets onto a single track and add the multiplexing IPs in each part of the partitioned design.

Finally, the design sub-netlists in Verilog format are generated, which can be loaded by FPGA PnR tools to generate the bitstreams.

The proposed automatic design flow optimizes the cost and the performance while reducing the time of availability. In the following, the evaluations of the cost, the performance and the time of availability are discussed.

Cost Evaluation

According to [START_REF] Amos | FPGA-Based Prototyping Methodology Manual[END_REF], the cost of creating a custom platform consists of the personnel cost, the material cost, the assembly cost, and the opportunity cost. An example of a custom platform with 4 Virtex-6 FPGAs is studied in [START_REF] Amos | FPGA-Based Prototyping Methodology Manual[END_REF]. The costs of the example are depicted in Table 4.2 and detailed in the following. • Personnel cost: It consists of the costs of the FPGA circuit development (from design RTL to board netlist), the PCB layout designer, and the test engineer. The manufacture of PCB is assumed to be out-sourced.

• Material cost: After the custom platform has been designed, it has to be built and this requires Bill of Materials (BoM) management and component purchase effort. The approximate cost for the material includes the PCB manufacture, the board itself, the FPGAs, and other components. Nearly half of the material cost would be the cost of FPGAs [START_REF] Amos | FPGA-Based Prototyping Methodology Manual[END_REF].

• Assembly cost: Assembly can be done in-house or be out-sourced.

• Opportunity cost: It is expressed as a missed Return on Investment (ROI), meaning that benefit which has not been realized elsewhere because resources are allocated to the given project.

If more copies of the custom platform are needed for remote deployment, the personnel cost and the opportunity cost will not be changed and can be neglected (i.e. if more than 10 copies are needed, the personnel cost and the opportunity cost can be neglected). Therefore, the rest costs are the material cost and the assembly cost. The material cost is ten times more than the assembly cost. Thus, the assembly cost can be neglected. Nearly half of the material cost would be the cost of FPGAs [START_REF] Amos | FPGA-Based Prototyping Methodology Manual[END_REF]. Therefore, in this manuscript, the cost evaluation can be defined as Equation 4.5. The number of copies (boards) of the custom platform is noted as N U M Board .

The cost of FPGAs is the multiple of the price of one chosen FPGA (noted as P rice F P GA ) and the number of FPGAs required in the custom platform (noted as N U M F P GA ).

Cost = M aterial cost * N U M Board = 2 * P rice F P GA * N U M F P GA * N U M Board (4.5)
The prices of different FPGAs (P rice F P GA ) are stored in FPGA Lib detailed in Section 4.3.1, which is established before launching the automatic design flow. After choosing one FPGA and launching the automatic design flow, the flow estimates the number of FPGAs required in the custom platform (N U M F P GA ) according to Section 4.5. Finally, the cost of one custom platform copy will be obtained according to Equation 4.5.

Performance Evaluation

As discussed in Section 2.3, the achieved performance of multi-FPGA platforms is influenced by the inter-FPGA communication. There are two different inter-FPGA communication architecture:

Logic Multiplexing and ISERDES/OSERDES. The maximum number of cut nets passing through one multiplexer (logic multiplexer in Logic Multiplexing, serdes in ISERDES/OSERDES) is noted as mux. According to Chapter 3, intermediate FPGAs may be necessary if there is no available track between the FPGA (the Driver) and the FPGA (the Receiver) in Design Routing process.

The maximum number of intermediate FPGAs needed (routing hop) is noted as hop. When implementing Logic Multiplexing (resp. ISERDES/OSERDES), the performance (in terms of the system clock frequency) can be defined as Equation 4.6 (resp. as Equation 4.7) as detailed in Section 3.6.3.

sys_clk = if _clk lat = 125 mux + hop + 3 M Hz (4.6) sys_clk = if _clk lat = 250 M Hz (7 + ceil( mux 4 )) * (1 + hop) (4.7)

Time of Availability Evaluation

The time of availability is an essential element for FPGA-based prototyping. The earlier the prototyping platform is available, the earlier the developers can start the hardware/software integration.

Therefore, the time spent for creating a custom platform need to be taken into consideration. An example of a custom platform with 4 Virtex-6 FPGAs is studied in [START_REF] Amos | FPGA-Based Prototyping Methodology Manual[END_REF]. The time spent of the example is depicted in Table 4.3 and can be different according to different experts.

When creating the custom platform for a specific design, there are different tasks from design RTL to final PCB board. These tasks include: FPGA circuit development (from design RTL to board netlist), PCB layout, PCB manufacture, Assembly, Test and Rework. PCB manufacture and assembly are assumed to be out-sourced.

Conclusion

In this Section, we proposed the automatic design flow for creating a custom platform. The cost evaluation, the performance evaluation and the time of availability evaluation are discussed. The advantages of the proposed automatic design flow are to optimize the cost and the performance while reducing the time-to-market and lowering the barrier of the board designers.

Board Exploration

There are many different types of FPGAs (i.e. vendor: Xilinx [Xilinx, 2014] or Altera [START_REF] Altera | FPGA CPLD and ASIC from Altera[END_REF], family: Virtex-7 or Stratix5, device: 2000T or GXAB, and package: FLG1925 or F1932). tracks" and "2-and multi-point tracks"). First, custom platforms using Logic Multiplexing and ISERDES/OSERDES will be compared in performance. Second, custom platforms using ("only 2-point tracks" and "2-and multi-point tracks") will be compared in performance. Finally, the board exploration will be done.

Logic Multiplexing VS ISERDES/OSERDES

Custom platforms using Logic Multiplexing and ISERDES/OSERDES are compared as shown in Table 4.4. The chosen FPGA type is XC5VLX330FF1760 in order to compare with the targeted off-the-shelf platform used in Chapter 3. Therefore, the generated custom platforms have 6 FPGAs. The number of cut nets passing through one multiplexer is noted as mux (mux = mux_logic in Logic Multiplexing and mux = mux_serdes in ISERDES/OSERDES). The routing hop is noted as hop. The performance using Logic Multiplexing is calculated according to Equation 3.1 and using ISERDES/OSERDES is evaluated according to Equation 3.2. Logic Multiplexing can be noted as LM and the gain of ISERDES/OSERDES over Logic Multiplexing is calculated. The results show that ISERDES/OSERDES can always achieve higher performance than Logic

Multiplexing in the custom platform as depicted in Figure 4.28. The gain in performance is up to 100%. Nevertheless, according to Chapter 3, Logic Multiplexing can achieve higher performance than ISERDES/OSERDES with several testbenches in the off-the-shelf platform. This is due to that the proposed automatic design flow distributes the inter-FPGA tracks according to the distribution of cut nets permitting no routing hop in ISERDES/OSERDES, while the off-the-shelf platform has generic and balanced connections. In the following, the custom platform uses IS-ERDES/OSERDES as the inter-FPGA communication architecture. As the chosen FPGA type is XC5VLX330FF1760 in order to compare with the targeted off-the-shelf platform used in Chapter 3, the generated custom platforms have 6 FPGAs.

Therefore, the minimum value of ter is 4 in order to spare FPGA I/Os. The number of multi-point tracks in the generated custom platform is noted as num. For each parameter ter, the gain of "2and multi-point tracks" platform over "only 2-point tracks" platform is calculated.

multi-point tracks are more critical than 2-point tracks in terms of performance

As discussed in Chapter 3, multi-point tracks are more critical than 2-point tracks in terms of performance in the off-the-shelf platform. With taking this constraint into consideration, the results of the performance comparison is depicted in Table 4.5. Only the testbenches have a high ratio between multi-terminal nets and total cut nets (as shown in Table 3.1) are chosen in the experiments. The results show that the "2-and multi-point tracks" platform achieves lower performance than the "only 2-point tracks" platform as depicted in Figure 4.29. For each testbench, the less is ter, the more is the num and the lower is the performance. The gain in performance is negative and up to -8% (ter = 6), -18% (ter = 5) and -25% (ter = 4). This is due to that the maximum number of cut nets passing through a multi-point track should be a half of that through a 2-point track, in order to avoid that the critical path is multi-point tracks and increase the performance. The reason of that multi-point tracks are more critical than 2-point tracks in terms of performance in the off-the-shelf platform, is that multi-point tracks have a longer trace than 2-point tracks in PCB board, thus a larger delay. Assume that PCB designers can increase the performance of multi-point tracks without bothering the performance of 2-point tracks by efficiently designing the custom platform PCB layout. In this case, the maximum number of cut nets passing through a multi-point track can be the same with that through a 2-point track. Custom platforms using ("only 2-point tracks" and "2-and multi-point tracks") will be compared in performance as shown in Table 4.6. The results shows that the "2-and multi-point tracks" platform achieves higher performance than the "only 2-point tracks" platform as depicted in Figure 4.30. For each testbench, the less is the ter, the more is the num and the higher is the performance. The gain in performance is positive and up to 0 (ter = 6), 10% (ter = 5) and 12% (ter = 4). Nevertheless, the performance gain is very limited and multi-point tracks increase the difficulty of PCB designs. Therefore, the custom platform using "only 2-point tracks" as inter-FPGA tracks is preferred. In the following, the custom platform is the platform using "only 2-point tracks".

Board Exploration

Gaisler Research Benchmarks [START_REF] Gaisler | Gaisler Research[END_REF] are chosen in the experiments for the board exploration, due to that they are the largest available benches representative of industrial designs. One advantage of the proposed automatic design flow is reducing the time to craft a custom platform.

In order to study this advantage, the CPU time and the occupied internal memory size (noted as MEM size) are measured. Another advantage is to permit board designers to choose the optimum FPGA type according to their specification (i.e. the cost and the performance). Three different FPGA types, which are widely used in the off-the-shelf prototyping platforms, are used in the experiments to generate the custom platforms. FPGA type 1 is XC5VLX330FF1760, FPGA type 2 is XC6VLX550TFF1759 and FPGA type 3 is XC6VLX760FF1760. The information of their logic capacity, I/O capacity and price is shown in Table 4.1. As most off-the-shelf platforms, all the FPGAs for implementing the ASIC logic have the same FPGA type in one generated custom platform. Therefore, the custom platform can be represented by the chosen FPGA type. In the experiments, each testbench is implemented respectively in three different custom platforms as shown in Table 4.7. These results are obtained by changing the parameters in the automatic design flow and relaunching the flow to get the achieved performances. The used PC has 4 intel xeon cores and maximum 16 GB internal memory. The filling rate is the maximum logic capacity utilization rate which ranges from 40% to 65%. The cost of the generated platform is calculated according to the Equation 4.5. For each generated custom platform, ten copies (boards) are needed. The performance is calculated according to the Equation 4.7 as ISERDES/OSERDES is taken as the multiplexing architecture. The number of external interfaces is noted as # of XIs. For the same testbench, different custom multi-FPGA platforms have different costs and different performance. An optimal solution means that no one (the cost or the performance) can be made better off without making another worse off. There are three different optimal solutions: cost-optimal, performance-optimal and intermediate. Board designers can choose the most adapted multi-FPGA board according to their specifications in the optimal solutions for the given design. For example, when leon2 is chosen, the solutions with the FPGA type (type 1 and type 3) are optimal solutions as shown in Figure 4.31(a). The FPGA type 1 solution achieves the lowest cost and the FPGA type 3 solution achieves the highest performance. Nevertheless, the FPGA type 2 solution is neglected due to that another solution with the FPGA type 1 can achieve the higher performance with lower cost. When leon3_avnet is chosen, all the solutions are optimal solutions as shown in Figure 4.31(b). The solution with the FPGA type 2 is an intermediate solution due to that it can achieve higher performance but higher cost than the FPGA type 1 solution, and lower performance but lower cost than the FPGA type 3 solution.

The logic capacity of the chosen FPGAs is classified as follows: FPGA type 1 < FPGA type 2 < FPGA type 3. In most cases, a larger FPGA that has higher logic capacity and higher price can achieve higher performance than a smaller FPGA that has lower logic capacity and lower cost.

It is because there is less FPGAs in the multi-FPGA platform with a larger FPGA and less (mux and hop) can be achieved. Nevertheless, in several cases, a smaller FPGA that is less expensive could be used to reduce the overall cost of the custom platform, with higher or equal performance than a larger FPGA. For example, the solution with FPGA type 1 (resp. FPGA type 2) that is less expensive can achieve higher (resp. equal) performance than the solution with FPGA type 2 (resp.

FPGA type 3) in the testbenches leon2 and netcard (resp. leon3mp). form is customized for the design. Moreover, due to that the inter-FPGA delay and the tolerance delay in the cabling platform is higher than in the hardwired platform, the data rate is heavily limited in the cabling platform when implementing Logic Multiplexing that is system-synchronous.

Nevertheless, the data rate in ISERDES/OSERDES (that is source-synchronous and propagates the clock along with the data), is not influenced by these limitations. Therefore, we will focus on ISERDES/OSERDES and discuss the constraints of ISERDES/OSERDES in the inter-FPGA communication of the cabling platform.

As the ISERDES/OSERDES is implemented in the cabling platform, each inter-FPGA track consumes a pair of FPGA I/O pins in LVDS. The width of connectors (resp. the cables) is the number of inter-FPGA tracks in mode LVDS connected to this connector, and is assumed to be the same for all the connectors (resp. all the cables). For example, there are 12 inter-FPGA connections (6 tracks in mode LVDS) between one connector and one FPGA as shown in The cabling platform benefits from the availability and the customization. The performance of the cabling platform depends on the distribution of the cables and the placement of the external interfaces. Nevertheless, there is no tool to automatically have a solution for the cable distribution.

Today, the cables (resp. the external interfaces) are distributed (resp. placed) according to the experience of board designers. Compared to the off-the-shelf platform, the added value, in terms of performance, of cabling platforms can be heavily impaired by an inefficient cable distribution.

The two existing cabling platforms (proFPGA and HAPS) have different width granularity of the connectors (resp. cables). The connections inter FPGAs as well as connections to external interfaces of the cabling platform are distributed with the granularity of one connector (resp. one cable). The optimal width (granularity) of the connectors need to be explored to achieve the maximum performance in the cabling platform.

Automatic Design Flow for Creating a Cabling Platform

In the manuscript, we propose an automatic design flow for creating a cabling platform as shown in 1. Logic Synthesis: The input design is synthesized targeting FPGAs (from RTL to design netlist). This step can be realized by Xilinx XST Synthesis tool [START_REF] Xst | XST[END_REF], Altera Quartus Synthesis tool [START_REF] Quartus | Design Entry and Synthesis[END_REF], or third party tools such as Synopsys Synplify tool [START_REF] Synplify | Logic Synthesis for FPGA Implementation[END_REF] and Mentor Graphics Precision tool [START_REF] Precision | Advanced FPGA Synthesis[END_REF].

2. Estimate the number of FPGAs: The minimum number of FPGAs required in the platform need to be estimated. This step can be realized by the commercial tool waCore provided by the company Flexras Technologies [START_REF] Flexras | Flexras Technologies[END_REF]. As a result, an XML file that contains the information of the number of FPGAs required in the platform is generated. 4. Design Partitioning: This step is done by the commercial tool waPart provided by the company Flexras Technologies [START_REF] Flexras | Flexras Technologies[END_REF]. The partitioning needs to take the connection constraints (the distribution of inter-FPGA connections) into consideration, in order to achieve a higher performance solution. We propose an algorithm to generate a temporary balanced platform in Section 4.7 (realized in python programming language). At the end of this step, the partitioned design netlist in Verilog format and the cut nets distribution file in XML format are generated.

5. Interconnect Synthesis: Inter-FPGA tracks are distributed according to the distribution of cut nets with the granularity of one connector (resp. one cable). We enhance the Interconnect Synthesis Algorithm in Section 4.8 to do this step. The enhanced part in the Interconnect Synthesis Algorithm is called the cable distribution algorithm, which will be detailed in the following. The enhanced interconnect synthesis algorithm is realized in python programming language. Finally, the board netlist in Verilog format and the cable distribution file in text format are generated.

After finishing the platform generation flow, the automatic design flow passes to the implementation flow. The implementation flow takes the partitioned design netlist, the board netlist, the FPGA lib and the user definition file as inputs. Due to that the design has already been synthesized and partitioned in the platform generation flow, the implementation is to route and multiplex the partitioned design into the generated platform to verify the functionality and to evaluate the performance. We propose a routing algorithm (that enhances the Turki's algorithm [START_REF] Turki | Iterative Routing Algorithm of Inter-FPGA Signals for Multi-FPGA Prototyping Platform[END_REF]) in Chapter 3 to do the design routing. The proposed routing algorithm is realized in C++ programming language. It generates a TCL format file that contains the information of the cut net to inter-FPGA track mapping. Then, the commercial tool waRoute provided by the company Flexras Technologies [START_REF] Flexras | Flexras Technologies[END_REF] will read this TCL format file, in order to multiplex several cut nets onto a single track and add the multiplexing IPs in each part of the partitioned design.

Finally, the design sub-netlists in Verilog format are generated, which can be loaded by FPGA PnR tools to generate the bitstreams.

The proposed automatic design flow for creating a cabling platform in this Section is similar to the automatic design flow for creating a custom platform proposed in Section 4.9. The difference between these two automatic design flows is the interconnect synthesis algorithm. First, the granularity of distributing inter-FPGA connections is different. The granularity is one connector (resp. one cable) in the cabling platform instead of one inter-FPGA track in the custom platform.

Therefore, the available I/Os per FPGA is the number of connectors for the cabling system. Then, the generated results are different. In the custom platform, the interconnect synthesis generates the board netlist in Verilog format and the fsp format file for generating PCB layout. Nevertheless, in the cabling platform, the board netlist in Verilog format and the cable distribution file in text format (instead of a fsp format) are generated. The cable distribution file is a report, which tells the users how to connect the cables (resp. daughter boards) to the corresponding connectors.

Cable Distribution Algorithm

We have proposed the interconnect synthesis algorithm in Section 4.8 to automatically distribute inter-FPGA tracks according to the distribution of cut nets in order to tailor the custom platform for the given design. In this Section, we enhance this algorithm to automatically optimize the distribution of the cables in the cabling platform as shown in An example, which uses three FPGAs to create a cabling platform for an industrial design, is studied in this Section. The proposed cable distribution algorithm is independent of FPGA types.

The chosen FPGA in this example is XC7V2000TFLG1925. In the example, the maximum I/Os per FPGA are 24 (connectors) and each connector has 24 inter-FPGA tracks in mode LVDS (width of connectors).

There are two steps in the cable distribution algorithm. The first step is to reserve one connector for the Global Signals (i.e. global clock, global reset and etc., detailed in Section 4.7.1) and the reset chain in each FPGA, and reserve connectors for the external interfaces in the corresponding FPGAs. In the example, 1 connector is reserved for the Global Signals and the reset chain.

Therefore, available I/Os per FPGA are 23. The prototyped design in the example, has 1 DDR as the external interface. According to External Interface Placement Algorithm proposed in Section 4.6, the DDR will be connected to the FPGA 0. Then, after the Design Partitioning, the DDR is connected with the logic in Part 0 (contained in FPGA 0). Therefore, 3 connectors in FPGA 0 need to be reserved for DDR. After that, available I/Os per FPGA are shown in Figure 5.4(a).

According to Section 4.8, the design is modelled as Figure 5.4(b). In the design modelling, all the multi-terminal nets will be split in 2-terminal nets. Then, the direction of the cut net is neglected in the modelling. After having defined the inter-FPGA tracks of FPGA 2, cut nets of Part 2 will be removed from the design model. After that, there is no cut net in the design model and all the inter-FPGA tracks have been defined. The generated cabling platform is shown in Figure 5.6(b), One cable (resp. one connector) in the example means 24 inter-FPGA tracks in mode LVDS. Therefore, in the example, As discussed in this Section, one pair of FPGA I/Os per cable is used to propagate the clock when the width of connectors is smaller than the width of the bank. If not, one pair of FPGA I/Os per bank per cable is used to propagate the clock. In order that the inter-FPGA communication is full-duplex (one cut net can be propagated from FPGA A to FPGA B while another cut net from FPGA B to FPGA A), the number of FPGA I/Os used for propagating the clock in one cable need to be multiplied by 2.

Performance Evaluation

The inter-FPGA delay is 1ns in the hardwired platform (that includes both the off-the-shelf platform and the custom platform where inter-FPGA tracks are realized by PCB traces). The performance (in terms of the system clock frequency) can be defined as Equation 2.3 of Section 2.3 when implementing ISERDES/OSERDES as the inter-FPGA communication architecture in the hardwired platform. In Equation 2.3, the system clock frequency is the inter-FPGA fast clock frequency divided by the latency. Nevertheless, according to [Posner, 2013a], the inter-FPGA delay T inter is 4ns (resp. 5ns) when the maximum cables are 25cm (resp. 50cm) long in the cabling platform. The inter-FPGA delay is the sum of the FPGA-to-connector delay, the cable delay and the connector-to-FPGA delay as FPGAs are interconnected by cables. The increase in the inter-FPGA delay adds the latency. As the inter-FPGA data rate is 1Gbps for all the platforms in case of ISERDES/OSERDES, 3 (resp. 4) inter-FPGA fast clock cycles are added in the latency due to that the inter-FPGA delay increases from 1ns to 4ns in 25cm long cables (resp. 5ns in 50cm long cables). Therefore, the achieved performance calculated in Equation 2.3 will be modified as in Equation 5. data is received from the Driver FPGA. Therefore, when considering routing hops, the achieved performance calculated in Equation 5.3 (resp. Equation 5.4) will be modified as in Equation 5.5

(resp. as shown in Equation 5.6) if the maximum cables are 25cm (resp. 50cm) long in the cabling platform.

sys_clk = if _clk lat = 250 M Hz (10 + ceil( mux_serdes 4 )) * (1 + hop) (5.5) sys_clk = if _clk lat = 250 M Hz (11 + ceil( mux_serdes 4
)) * (1 + hop)

(5.6)

Time of Availability Evaluation

The time of availability is an essential element for FPGA-based prototyping. The earlier the prototyping platform is available, the earlier the developers can start the hardware/software integration.

Therefore, the time spent for creating a cabling platform need to be taken into consideration. According to [START_REF] Amos | FPGA-Based Prototyping Methodology Manual[END_REF], when creating the custom platform for a specific design, there are different tasks from design RTL to final PCB board. These tasks include: FPGA circuit development, PCB layout, PCB manufacture, Assembly, Test and Rework. Nevertheless, the cabling platform does not need PCB layout, PCB manufacture, Assembly, Test and Rework. This is because FPGA boards in the cabling platform are ready-made. The cable distribution (inter-FPGA distribution) in the cabling platform is user-defined and need to be tailored for the specific design.

This process is the FPGA circuit development (from design RTL to board netlist and cable distribution). According to [START_REF] Amos | FPGA-Based Prototyping Methodology Manual[END_REF], a manual FPGA circuit development (a manual cable distribution) needs 1-2 months. The proposed design flow in this Section for creating a cabling platform automatizes the FPGA circuit development. Therefore, with the help of the proposed automatic design flow, the time of availability for the cabling platform reduces from 1-2 months to several hours.

Optimal Width of Connectors

In the experiments, four testbenches (leon2, leon3_avnet, leon3mp, netcard) from Gaisler Research Benchmarks [START_REF] Gaisler | Gaisler Research[END_REF] will be used. According to our knowledge, these testbenches are the largest available benches representative of industrial designs. The proposed automatic design flow for creating a cabling platform is independent of FPGA types. The chosen FPGA is XC7V2000TFLG1925 due to that it is the latest FPGA. In order to be comparable with the latest off-the-shelf platform presented in Figure 3.1 that use the same FPGA type, the cabling platform has 4 FPGAs. As the XC7V2000TFLG1925 is the largest FPGA, the logic capacity utilization rate

OPTIMAL WIDTH OF CONNECTORS

113 had been reduced between 10% and 20% to force each testbench to fill in 4 FPGAs, while keeping the design internal structure (connections) unaltered.

Experiments explore the optimal width of connectors in the cabling platform to achieve higher performance. According to Section 5.2, the minimum width of connectors is 6 ( 1 4 bank) and the maximum width is 576. The number of the cables should be an integer, and the Equation 5. For all the testbenches, the results show that the optimal system clock frequency can be achieved when the width of connectors is 24 inter-FPGA tracks (1 bank) in the cabling platform.

When the width of connectors is less than 1 bank, more FPGA I/Os will be used for propagating the clock instead of the data, thus the system clock frequency may degrade. When the width of connectors is more than 1 bank, the interconnect synthesis has less adjustability to distribute inter-FPGA tracks due to the coarser granularity. In this case, the generated platform can be not adapted for the given design and the system clock frequency can degrade. As shown in Figure 5.7, when the width of connectors exceeds 96 (4 banks), the system clock frequency decreases rapidly. The system clock frequency attains 0 when the width of connectors is more than 288 (12 banks, only 2 cables per FPGA in this case) because the design is not routable in the generated platform. In mechanics, cables and connectors of 1-bank width are feasible [HAPS, 2014]. In the following, the cabling platform is established with the optimal width of connectors (1 bank).

Proposed Cabling Platform

In this manuscript, we propose a cabling platform where one board is composed of one FPGA and several connectors. All the connectors have the same width and the width of connectors is 1 bank (i.e. 24 inter-FPGA tracks in mode LVDS for Virtex-7 FPGA). All the FPGA I/Os are used for FPGA-to-connector connections. The connections between FPGAs as well as the connections to external interfaces can be added or removed by only connecting or disconnecting the cables (resp. daughter boards) with or from the connectors in order to be tailored for the given design.

An example of the proposed cabling platform is shown in Figure 5.8. One board can be connected with other FPGA boards by cables and connectors, and external interface daughter boards by connectors (connectors are capable to support the speed of external interfaces, i.e. DDR [START_REF] Posner ; Posner | Ddr3 memory and how useful a brick can be[END_REF]). If the prototyped design has external interfaces (i.e. DDR), the external interfaces will be connected with one FPGA by connectors. If there is no external interface, these connectors will serve as inter-FPGA tracks by connecting them with cables. Therefore, in this case, there is no FPGA I/O wasted in the cabling platform compared to the off-the-shelf platform. (i.e. T_B3, T_B4, T_B5 in the mother board B that are connected with the daughter board D can be served as inter-FPGA tracks by connecting the connector 0, 1, 2 in the Mother Board A with the connector 3, 4, 5 in the Mother Board B). Different from the off-the-shelf platform where specific external interfaces have been fixed to FPGAs, external interfaces in the cabling platform can be connected to any FPGAs by connectors. Therefore, the optimum placement of external interfaces can be explored in order to achieve higher performance (in terms of the system clock frequency) according to Section 4.6 and the partitioning process can be less constrained compared to the off-the-shelf platform. Nevertheless, when the external interface comes to be the external interfaces that contains few pins (i.e. UART only has 2 pins, meaning that 1 track in mode LVDS), one connector is occupied and some FPGA I/Os in the connector are wasted. (i.e. T_B2 has 24 tracks in mode LVDS, but T_U0 has only 1 track and 23 tracks in T_B2 are wasted)

The proposed cabling platform has a flexibility. Moving the cabling platform customized for a specific design to a cabling platform customized for another specific design can be easily done by using the same platform with only re-launching the automatic design flow and re-connecting the cables, if these two specific designs have the same logic capacity size. If the logic capacity sizes of these two specific designs are different, ready-made boards can be easily added to or removed

Conclusion

In this Chapter, the cabling platform is discussed. The cabling platform benefits from the availability and the customization. Nevertheless, there is no tool to automatically have a solution for the cable distribution. Today, the cables (resp. the external interfaces) are distributed (resp. placed) according to the experience of board designers. Compared to the off-the-shelf platform, the added value, in terms of performance, of cabling platforms can be heavily impaired by an inefficient cable distribution. The two existing cabling platforms (proFPGA and HAPS) have different width granularity of the connectors (resp. cables). The connections inter FPGAs as well as connections to external interfaces of the cabling platform are distributed with the granularity of one connector (resp. one cable). The optimal width (granularity) of the connectors need to be explored to achieve the maximum performance in the cabling platform.

The main contribution is threefold. First, we propose an automatic design flow for creating a cabling platform, more specifically an algorithm optimizing the distribution of the cables. Then, with the help of the proposed automatic design flow, the optimal width of connectors for a cabling platform is explored. Experiments are conducted using four testbenches from Gaisler Research Benchmarks. The results show that the optimal width of connectors is 1 bank in the cabling platform. Finally, we propose a cabling platform where one board is composed of one FPGA and several connectors. All the connectors have the same width and the width of connectors is 1 bank.

All the FPGA I/Os are used for FPGA-to-connector connections. The connections inter FPGAs as well as the connections to external interfaces can be added or removed by only connecting or disconnecting the cables (resp. daughter boards) with or from the connectors.

The next Chapter compares three different multi-FPGA platforms (off-the-shelf, custom, cabling). With the help of the proposed automatic tools, the performance gains between these platforms are quantified.

Chapter 6

Comparison of the Different Platforms

Introduction

The previous three Chapters have discussed the three different multi-FPGA platforms (off-theshelf, custom, and cabling), and have presented and proposed automatic flows for different platforms. In Section 2.2.4, these three multi-FPGA platforms have been compared qualitatively in terms of availability, performance, flexibility, and cost. With the developed automatic tools, this Chapter will re-compare these three platforms, and therefore quantify the performance gain between those platforms.

Performance Comparison of Different Multi-FPGA Platforms

The workflow of the performance comparison when implementing a specific design respectively into three different multi-FPGA prototyping platforms, is shown in Figure 6.1 and presented as follows. • When the design is implemented in the custom platform, the automatic design flow for creating a custom platform, detailed in Figure 4.25 of Chapter 4, is used.

• When the design is implemented in the cabling platform, the automatic design flow for creating a cabling platform, detailed in Figure 5.2 of Chapter 5, is used.

Experiments are conducted using four testbenches (leon2, leon3_avnet, leon3mp, netcard) from Gaisler Research Benchmarks [START_REF] Gaisler | Gaisler Research[END_REF]. The automatic flows are independent of FPGA types. For the comparison, all the multi-FPGA platforms use the same type of FPGAs (XC7V2000TFLG1925) and have 4 FPGAs. Each testbench is implemented respectively in three different multi-FPGA prototyping platforms to compare the performance as shown in Table 6.1.

The cabling platform is established with the optimal width of the cables (1 bank). There are two different types of cabling platforms depending on the length of the cables (25cm and 50cm). The performance (in terms of the system clock frequency) is noted as sys_clk. The unit for the system clock frequency is M Hz. The results show that the system clock frequency of the multi-FPGA platforms is about several to tens of MHz for all the testbenches as shown in Figure 6.2. The custom platform can achieve the highest performance compared to the other platforms. The performance gain of the custom platform is up to 124% compared to the off-the-shelf platform. This is due to that inter-FPGA tracks of the off-the-shelf are fixed and generic, and inter-FPGA tracks of the custom platform are distributed to be tailored for the given design. The performance gain of the custom platform is up to 22% (resp. 33%) over the cabling one in 25cm cable length (resp. 50cm cable length). This is because the inter-FPGA delay of the custom platforms (1ns) is less than the cabling platform delay (4ns for 25cm cable length and 5ns for 50cm cable length). In most of testbenches (leon2, leon3_avnet, netcard), the cabling platform can achieve higher performance compared to the offthe-shelf platform. The performance gain of the cabling platform is up to 111% in 25cm cable length (resp. 100% in 50cm cable length). This is due to that inter-FPGA tracks of the cabling platform are also distributed to be tailored for the given design. Nevertheless, in the testbench (leon3mp), the cabling platform achieves lower performance compared to the off-the-shelf platform. The performance gain of the cabling platform is -9% in 25cm cable length (resp. -16% in

QUANTITATIVE AND QUALITATIVE COMPARISON OF DIFFERENT PLATFORMS119

50cm cable length). This is because the cable distribution algorithm is not able to find a better distribution than the off-the-shelf (the worst case being the same distribution as the off-the-shelf platform due to the same constraints for both platforms explained in Section 2.3), and the inter-FPGA delay of the cabling platform is higher than the off-the-shelf platform inter-FPGA delay (1ns). With the proposed automatic tools, the time of availability for the cabling platform reduces from 1-2 months to several hours. Nevertheless, for the custom platform, the proposed design flow only automatizes the FPGA circuit development. Therefore, 3-5 months are still needed due to the PCB layout and the PCB fabrication. The achieved performance of multi-FPGA platforms depends on the inter-FPGA data rate, inter-FPGA tracks distribution, and inter-FPGA delay. The automatic design flow for creating a custom platform can generate high-performance board netlist with considering the constraints for high-speed signaling (i.e. LVDS), high performance multiplexing (i.e. ISERDES/OSERDES) and so forth. Therefore, the inter-FPGA data rate of the custom platform that before depended on the designers' experience, today is the same with the off-the-shelf and the cabling platform that are ready-made by the FPGA experts in the commercial companies. The proposed automatic design flows for creating a cabling and custom platform optimize the distribution of inter-FPGA tracks. Therefore, the inter-FPGA tracks distributions of the cabling and custom platform that before depended on the designers' experience, today are tailored for the given design. In the following, all these three multi-FPGA platforms are compared mutually in the quantified performance shown in Table 6.1 with considering the advantages and disadvantages of each platform shown in Table 6.2.

Cabling vs Off-the-Shelf

The inter-FPGA delay of the off-the-shelf platform (1ns) is much lower than the cabling platform (4ns or 5ns). This means that, if the cable distribution algorithm is not able to find a better distribution than the off-the-shelf (the worst case being the same distribution as the off-the-shelf platform due to the same constraints for both platforms explained in Section 2.3), only in this case, the off-the-shelf would give a better performance (i.e. leon3mp). In other cases, the cabling platform always outperforms the equivalent generic off-the-shelf platform in performance (25cm cable length: from 23% gain up to 111%; 50cm cable length: from 14% gain up to 100%). With the automatic tool, the time of availability for the cabling platform is only 2-12 hours. And for the other characteristics such as flexibility and cost, the cabling platform can outperform or equal the off-the-shelf platform as shown in Table 6.2. Therefore, the cabling platform is now an attractive alternative compared to the off-the-shelf platform.

Custom vs Off-the-Shelf

The custom platform can achieve higher performance than the off-the-shelf platform (up to 124% gain). Nevertheless, the proposed automatic design flow for creating the custom platform, is only automatizing the FPGA circuit development and generating the board netlist. According to Table 6.2, making a custom multi-FPGA PCB board needs about 3-5 months (time of availability).

Therefore, if only few pieces are needed, and the performance difference is not relevant enough for the user (e.g. 25MHz for leon3mp in the off-the-shelf and 27.78MHz for the custom), selecting the off-the-shelf scenario allows to run tests 3-5 months ahead at 25MHz. Nevertheless, the cost of the custom platform can be lower when a large quantity of platforms is needed (deployment cost).

The proposed automatic design flow helps the designers to predict the achieved performance in the custom platform and choose the optimum prototyping platform according to their specifications (board exploration). Therefore, choosing an off-the-shelf or a custom is a tradeoff depending on designers' specifications.

Custom vs Cabling

The performance achieved with the custom platform exceeds the cabling platform. However, the gain is limited (max. 22% gain for 25cm cable length and max. 33% gain for 50cm cable length).

Indeed, the availability of the custom platform is about 3-5 months which is much longer than the cabling platform, which is only 2-12 hours. On the flexibility side, the custom platform designed for leon2 may be not adapted for netcard. The flexibility criteria favors the cabling platform. For example, moving the cabling platform customized for a specific design (i.e. leon2) to a cabling platform customized for another specific design (i.e. netcard) can be easily done by using the same platform with only re-launching the automatic design flow and re-connecting the cables. If the logic capacity sizes of these two specific designs are different, ready-made boards can be easily added to or removed from the cabling platform. Then, the automatic design flow is re-launched and the cables are re-connected. Therefore, if the performance or the deployment cost are not stringent constraints, the cabling platform can be a better choice with the proposed automatic design flow.

Conclusion

In Section 2.2.4, these three multi-FPGA platforms (off-the-shelf, custom, and cabling) have been compared qualitatively in terms of availability, performance, flexibility, and cost. With the developed automatic tools, this Chapter has re-compared these three platforms, and therefore quantified the performance gain between those platforms. Experiments are conducted using four testbenches from Gaisler Research Benchmarks. The results show that choosing an off-the-shelf or a custom is still a tradeoff depending on designers' specifications. Nevertheless, apart from some stringent constraints (such as deployment cost or specific frequency needed), the relatively new cabling paradigm with the automatic, cable distribution tool, offers an attractive alternative compared to the two other platforms.

The next Chapter concludes this manuscript and lists the future lines of research.

Chapter 7

Conclusion and Future Work

Summary of the Thesis

The utility of PFGA-based prototyping is undisputed. It permits hardware designers to develop and test their systems, and gives software developers early access to a fully functioning hardware platform. As the complexity of System on Chip (SoC) circuits and the quantity of software to be developed are increasing, the software developers can no longer wait for the chip to be fabricated for the hardware/software integration in order to meet the ever-shrinking time-to-market window.

Thus, FPGA-based prototyping is no longer optional. As SoC/ASIC has a higher silicon area overhead than FPGA, multi-FPGA prototyping platforms are indispensable.

Even though the logic capacity and the number of FPGA I/Os are increasing generation after generation, the logic capacity grows at a much higher rate than the number of FPGA I/Os. Thus, FPGA I/Os are becoming a scarce resource, worsening the inter-FPGA bandwidth generation after generation. Unfortunately, multi-FPGA platforms suffer from large timing delays in inter-FPGA communication compared to intra-FPGA net delays. Therefore, it becomes more and more difficult to prototype an SoC/ASIC design at a proper performance.

We have classified multi-FPGA prototyping platforms in three categories: off-the-shelf, cus- and an acceptable internal memory resource [START_REF]Multi-FPGA Prototyping Board Issue : the FPGA I/O Bottleneck Qingshan Tang, Matthieu Tuna and Habib Mehrez[END_REF][START_REF]Routing Algorithm for Multi-FPGA Based Systems Using Multi-Point Physical Tracks Qingshan Tang[END_REF][START_REF]Design for prototyping of a parameterizable cluster-based Multi-Core System-on-Chip on a Multi-FPGA board Qingshan Tang[END_REF]. This means that the time spent for FPGA circuit development is reduced from 1-2 months to 2-12 hours. Therefore, the board exploration becomes feasible. Designers can choose the most adapted multi-FPGA board according to their specifications. The experiment results also show that the smaller FPGA that is less expensive could be used to reduce the overall cost of the platform with sometimes higher performance.

The cabling platform, which is a comparatively new notion compared to other two platforms, is in between the off-the-shelf and the custom platform. It is semi off-the-shelf due to that it consists of multiple ready-made boards, and semi custom due to that its connections inter FPGAs as well as connections to external interfaces are user-defined and tailored for a specific design.

Therefore, the cabling platform benefits from both the availability and the customization. The performance of the cabling platform depends on the distribution of the cables and the placement of the external interfaces. Nevertheless, there is no tool to automatically have a solution for the cable distribution. Today, the cables (resp. the external interfaces) are distributed (resp. placed) according to the experience of board designers. A high-performance cable distribution becomes more and more arduous because of the limited number of FPGA I/Os. Furthermore, the added value in performance of cabling platforms can be heavily weakened by an inefficient board design.

In this manuscript, we have proposed an automatic design flow for creating a cabling platform, more specifically an algorithm optimizing the distribution of the cables. Then, with the help of the proposed automatic design flow, the optimal width of connectors for a cabling platform is explored.

The experiment results show that the optimal width of connectors is 1 bank in the cabling platform.

Finally, we have proposed a cabling platform where one board is composed of one FPGA and several connectors. All the connectors have the same width and the width of connectors is 1 bank.

All the FPGA I/Os are used for FPGA-to-connector connections. The connections inter FPGAs as well as the connections to external interfaces can be added or removed by only connecting or disconnecting the cables (resp. daughter boards) with or from the connectors.

After finishing the discussions above on the three different categories of multi-FPGA platforms (off-the-shelf, custom, and cabling), board designers may have a question. Which category is the optimum solution according to designers' specification? Different categories of platforms had already been compared qualitatively in terms of availability, performance, flexibility, and cost.

With the developed automatic tools, these three platforms are re-compared. The experiment results show that choosing an off-the-shelf or a custom is still a tradeoff depending on designers' specifications. The proposed automatic design flow integrates the expertise for designing a highperformance board netlist. Thus, the performance of the custom platform that before depended on the designers' experience, can always be better than the off-the-shelf platform. Nevertheless, the proposed design flow only concerns FPGA circuit development. 3-5 months are still needed for creating a custom platform owing to PCB layout and PCB fabrication. When referring to the cabling platform, the proposed design flow reduces the time of availability from 1-2 months to 2-12 hours because the PCB of cabling platforms is ready-made. Furthermore, the experiment results

show that generally the cabling platform achieves higher performance than the off-the-shelf except the only case that the cable distribution algorithm is not able to find a better distribution (the worst case being the same distribution as the off-the-shelf platform). Therefore, the cabling platform is now more attractive than the off-the-shelf platform. Finally, we compare the cabling platform with the custom platform. The experiment results show that the custom platform achieves a limited performance gain with much longer time of availability. On the flexibility side, the custom platform tailored for one design may be not adapted for another design. Thus, another platform has to be created and at least 3-5 months need to be spent. Nevertheless, moving the cabling platform tailored for a specific design to a cabling platform customized for another specific design can be easily done by using the same platform with only re-launching the automatic design flow and re-connecting the cables. If the logic capacity sizes of these two specific designs are different, ready-made boards can be easily added to or removed from the cabling platform. Then, the automatic design flow is re-launched and the cables are re-connected. It takes only 2-12 hours.

Therefore, if the performance or the deployment cost are not rigorous constraints, the cabling platform can be a better choice with the proposed automatic design flow.

  -clés : Circuit Intégré, Vérification, Multi-FPGA, Prototypage, Sur Mesure, Câblée vii

  .1.

Figure 1 . 1 :

 11 Figure 1.1: Overall design cost

Figure 1 . 2 :

 12 Figure 1.2: Verification technology

  but the least visibility compared to other hardware-related verification techniques. It costs about ten thousands of dollars and takes several weeks to several months for set-up. Different from the Simulation/Emulation, FPGA-Based prototyping enables "real world" testing. The prototyped SoC/ASICs are put into actual hardware and real external interfaces (such as DDR, PCI, Ethernet and etc.) are used. Implementing a design into a FPGA-based platform is a challenging process. As emulators, FPGA-based prototypes are hardware platforms as shown in Figure 1.3. Nevertheless, it is smaller in form factor, which makes it useful for software developers. One or several software developers can have one in each desktop. The major existing FPGA-based prototype providers are:

Figure 1 . 3 :

 13 Figure 1.3: Emulator versus FPGA-based prototype in form factor

Figure 1 . 4 :

 14 Figure 1.4: Three different multi-FPGA prototyping platforms

Figure 1 . 5 :

 15 Figure 1.5: The ratio FPGA logic capacity over I/Os

Figure 2 . 1 ,

 21 which is the latest platform from Dini Group [DINI, 2014]. The platform uses four FPGAs of the latest Xilinx FPGA (family: Virtex-7, device: 2000T, and package: FLG1925, thus FPGA type: XC7V2000TFLG1925). Each FPGA is connected with a DDR by connectors. Only the FPGA D is connected with a PCIE by connectors. The connectors are fixed and specific. This means that the connectors to one type of the external interface (i.e. DDR) can not be transferred to the connectors to another type of the external interface (i.e. PCIE). If the prototyped design has less than 4 DDRs, several FPGA I/Os connected with DDRs are wasted. If the prototyped design has a PCIE, the PCIE part of the prototyped design is forced to be placed in FPGA D. These constraints add to the complexity of the design partitioning and may reduce the performance. The number of inter-FPGA tracks is shown in Figure 2.1(a). All the inter-FPGA tracks are fixed. There are two types of inter-FPGA tracks in the multi-FPGA prototyping platform: 2-point tracks and multi-point tracks. The number of multi-point tracks in the example is 120. The number of 2-point tracks between a pair of FPGAs are generic and balanced: 300 connections in all the horizontal and vertical pairs, and 150 connections in all the diagonal pairs. When implementing a specific design, these generic and balanced connections constraint the performance.

Figure 2 .Figure 2 . 1 :

 221 Figure 2.1: The off-the-shelf multi-FPGA platform Major existing commercial off-the-shelf platforms are Cadence Protium Rapid Prototyping Platform [Protium, 2014], Dini Group [DINI, 2014], S2C [S2C, 2014], and HyperSilicon [Hyper-Silicon, 2014].

Figure 2 . 2 :

 22 Figure 2.2: The typical implementation flow

Figure 2 . 3 .

 23 These direct inter-FPGA tracks can be divided into two categories: 2-point tracks directly connect one FPGA to another, and multi-point tracks connect several FPGAs together.

Figure 2 . 3 :

 23 Figure 2.3: The direct architecture

Figure 2 . 4 :

 24 Figure 2.4: The proposed routing algorithm

Figure 2

 2 Figure 2.5.

Figure 2 . 5 :

 25 Figure 2.5: The indirect architecture

Figure 2 . 7 .

 27 The design is bigger than the largest FPGA available and three of them are needed. There are many signals crossing from Part A to Part B but only few from Part A to Part S. Therefore, more inter-FPGA tracks are allocated between FPGA A and FPGA B than between FPGA A and FPGA S. In terms of external interfaces, there is only 1 DDR in the example. Therefore, 1 DDR is reserved only for FPGA B that contains the memory controller.

Figure 2 . 7 :

 27 Figure 2.7: Create a Custom Platform for a specific design

Figure 2 . 8 :

 28 Figure 2.8: The typical design flow for creating a custom platform

4 .

 4 Define the FPGA connectivity (FPGA-to-External-Interface and inter-FPGA tracks). 5. Select FPGA I/Os. The I/O DRC rules (such as clock capability, clock region and I/O standard voltage reference levels) are considered, when choosing I/Os for each track. Then, a board netlist and a PCB layout are generated.

Figure 2 . 10 :

 210 Figure 2.10: Multi-FPGA Platform: SpringBok

Figure 2 .

 2 Figure 2.11: A photo of Kulmala's quasi cabling platform

Figure 2 .

 2 Figure 2.12: proFPGA

Figure 2 .

 2 Figure 2.13: Synopsys HAPS

Figure 2 .

 2 Figure 2.15: The basic Time-Division-Multiplexing architecture

Figure 2 .

 2 Figure 2.16: Logic Multiplexing for Multi-FPGA Prototyping

Figure 2 .

 2 Figure 2.18: The DNV7F2A board

  .2. FPGA I/Os connected to inter-FPGA tracks in the DNV7F2A board are made bank-to-bank. Therefore ISERDES/OSERDES can be implemented in the DNV7F2A board (thus 150 pairs of inter-FPGA tracks in mode LVDS). In one FPGA, the I/Os connected to these 150 pairs of inter-FPGA tracks belong to 8 different banks. Therefore, 8 pairs of inter-FPGA tracks are reserved for propagating the clock and 142 pairs of inter-FPGA tracks are used for propagating the data. When ISERDES/OSERDES is implemented, the maximum number of cut nets passing through one ISERDES/OSERDES mux_serdes = 300 * ratio 142 and the system clock frequency sys_clk is calculated according to Equation 2.3.

Figure 2 .

 2 Figure 2.19: Logic Multiplexing VS ISERDES/OSERDES

Figure 3 . 1 ,

 31 which is the latest platform from Dini Group[DINI, 2014]. The platform uses four FPGAs of the latest Xilinx FPGA (family: Virtex-7, device: 2000T, and package: FLG1925, thus FPGA type: XC7V2000TFLG1925). Each FPGA is connected with a DDR by connectors.Only the FPGA D is connected with a PCIE by connectors. Nevertheless, the connectors are fixed and specific. This means that the connectors to one type of the external interface (i.e. DDR) can not be transferred to the connectors to another type of the external interface (i.e. PCIE).If the prototyped design has less than 4 DDRs, several FPGA I/Os connected with DDRs are wasted. If the prototyped design has a PCIE, the PCIE part of the prototyped design is forced to be placed in FPGA D. These constraints add to the complexity of the design partitioning and may reduce the performance. The number of inter-FPGA tracks is shown in Figure3.1(a). All the inter-FPGA tracks are fixed, bank-to-bank and LVDS compliant. There are two types of inter-FPGA tracks in the multi-FPGA prototyping platform: 2-point tracks and multi-point tracks. The number of multi-point tracks in the example is 120. The number of 2-point tracks between a pair of FPGAs are generic and balanced: 300 connections in all the horizontal and vertical pairs, and 150 connections in all the diagonal pairs. When implementing a specific design, these generic and balanced connections constraint the achieved performance.

  Figure 3.1(b) shows the picture of the platform and the PCB traces.

Figure 3 . 1 :

 31 Figure 3.1: The off-the-shelf multi-FPGA platform

Figure 3 . 2 :

 32 Figure 3.2: Typical implementation flow for multi-FPGA platforms

  Figure 3.3(a). One type of cut nets has one driver and one receiver (black arrow from one driver to one receiver), called 2-terminal nets. The second type of cut nets has one driver and multiple receivers (colored arrows from one driver to multiple receivers), called multi-terminal nets. Off-the-shelf multi-FPGA prototyping platforms come with two types of inter-FPGA tracks as shown inFigure 3.3(b). One type of tracks connects only two FPGAs (black line), called 2-point tracks. Another type of tracks connects more than two FPGAs (red line), called multi-point tracks. In Section 2.2.1.2, an iterative congestion-aware routing algorithm

3. 6 . 1 . 1

 611 Multi-terminal net routing in 2-point tracksRouting a multi-terminal net in 2-point tracks is shown in Figure3.4. The driver is in FPGA 0, one receiver is in FPGA 1 and another receiver is in FPGA 2. There are two ways to route this cut net. 4 FPGA I/Os are used in both cases.

  Figure 3.3: The partitioned design and the corresponding multi-FPGA platform

Figure 3 . 4 :

 34 Figure 3.4: Routing a multi-terminal net in 2-point tracks

Figure 3 . 5 :

 35 Figure 3.5: Routing a multi-terminal net in a multi-point track

Figure 3 . 6 :

 36 Figure 3.6: The proposed routing algorithm

Figure 3 . 7 :

 37 Figure 3.7: Routing a multi-terminal net in a multi-FPGA board

Figure 3 . 8 :

 38 Figure 3.8: Routing in 2-point tracks

3. 6 . 3 . 1

 631 Figure 3.9: routing hops in Logic Multiplexing

Figure 3 . 10 :

 310 Figure 3.10: routing hops in ISERDES/OSERDES

Figure 3 .

 3 Figure 3.11: The architecture of DN9000K10PCI

  Figure 3.12: The partitioning results

Research

  Benchmarks and three testbenches from OpenCores Benchmarks. The results in an off-the-shelf platform of six Virtex-5 show up to 25% (resp. 22%) gain in the system clock frequency compared to the existing 2-point routing algorithm in case of Logic Multiplexing (resp. ISERDES/OSERDES) used as inter-FPGA communication architecture. The next Chapter focuses on the custom platform. We propose an automatic design flow for creating a custom platform. The proposed automatic design flow reduces the time-to-market of new products and lowers the entry barrier of board designers while optimizing the cost and the performance.

• 8 /

 8 16/32-bits memory controller for DDR • On-chip peripherals such as UART, Timer, interrupt controller (IrqCtrl) and 16-bit I/O port • Advanced on-chip debug support unit and trace buffer • Local RAMs • Low complexity 32-bit PCI target-only interface • 10/100 Ethernet MAC • Memory management unit (MMU) The memory controller, UART, Timer, interrupt controller (IrqCtrl), 16-bit I/O port, debug serial link, Ethernet and PCI interact with peripherals. These peripherals, which are outside FPGAs but on the board, are called external interfaces.

Figure 4 . 1 : 1 . 2 .

 4112 Figure 4.1: Create a Custom Platform for a specific design

  this manuscript, we propose an automatic design flow of creating a custom multi-FPGA platform for a specific design by only choosing an FPGA type. The design flow overview is shown in Figure4.2. The flow takes the design RTL as inputs, through several steps, and generates a high performance board netlist with considering the constraints of high-speed signaling, highperformance multiplexing and so forth. After that, the flow implements the design into the generated board and generates bitstreams for each FPGA. The outputs of the flow is the board netlist and the design sub-netlists. The flow can be divided into 6 main steps and detailed as follows.

Figure 4 . 2 :

 42 Figure 4.2: The design flow overview for creating a custom platform

  3 external interfaces are numbered as (XI[START_REF]Multi-FPGA Prototyping Board Issue : the FPGA I/O Bottleneck Qingshan Tang, Matthieu Tuna and Habib Mehrez[END_REF], DDR), (XI[START_REF]Performance Comparison between Multi-FPGA Prototyping Platforms: Hardwired Off-the-Shelf, Cabling and Custom Qingshan Tang[END_REF], Ethernet) and (XI[0], PCI). The Ethernet is shortened as Eth in the following figures.

4. 6 . 1

 61 All the External Interfaces connected to the same FPGA When all the External Interfaces are connected to the same FPGA, the number of different cases depends on the number of FPGAs in the platform (num_f pga), thus is num_f pga cases. Figure 4.3 shows that the results of these cases are the same due to the symmetry. Therefore, only one case is executed to reduce the runtime. In the example, there are 4 different cases because there are 4 FPGAs in the platform. Nevertheless, there is only one case that will be executed.

Figure 4 . 3 :

 43 Figure 4.3: All the External Interfaces connected to the same FPGA

Figure 4 .

 4 5(a)), but not all the cases are the same (i.e. there may be other case such as Ethernet in one FPGA, PCI and DDR in another FPGA as shown in Figure 4.5(b)). In the other cases, all the possible placements are swept and the identical cases due to the symmetry are executed only one time. In the example, there will be pow(2, 3) = 8 cases in total due to that there are 3 external interfaces and 2 FPGAs connected to external interfaces.(a) symmetry: identical case (b) asymmetry: different case

Figure 4 . 5 :

 45 Figure 4.5: Other cases

Figure 4 .

 4 5(a) that DDR and Ethernet are placed in FPGA 2 and PCI is placed in FPGA 3 is the same with the case as shown at the left ofFigure 4.5(a) that DDR and Ethernet are placed in FPGA 0 and PCI is placed in FPGA 1.placed External Interface XI[k] will be connected to an FPGA with no less index number than the previously placed external interfaces XI[k-1] (i.e. XI[k] is placed to FPGA x, XI[k-1] is placed to FPGA y and x ≥ y). In the example, (XI[0], PCI), (XI[START_REF]Performance Comparison between Multi-FPGA Prototyping Platforms: Hardwired Off-the-Shelf, Cabling and Custom Qingshan Tang[END_REF], Ethernet), (XI[START_REF]Multi-FPGA Prototyping Board Issue : the FPGA I/O Bottleneck Qingshan Tang, Matthieu Tuna and Habib Mehrez[END_REF], DDR). If PCI is placed in one FPGA, DDR and Ethernet are placed in another FPGA, PCI will be placed in FPGA 0 and DDR and Ethernet will be placed in FPGA 1 as shown at the right ofFigure 4.5(a). In the example, the case as shown at the left of Figure4.5(a) that PCI is placed in FPGA 1 and DDR and Ethernet are placed in FPGA 0 is the same with the case as shown at the right ofFigure 4.5(a).

Figure 4 . 6 (

 46 realized in python programming language). It takes the XML file of FPGA-Interface tracks as input. This XML file is generated by the external interface placement algorithm and contains the information that which external interface will be connected to which FPGA. According to the chosen inter-FPGA communication architecture, the available I/Os per FPGA are different. In Chapter 2, we have detailed two existing inter-FPGA communication architectures. The first one is Logic Multiplexing that uses logic blocks of the FPGA fabric to implement the multiplexing for inter-FPGA communication. Another is ISERDES/OSERDES that uses dedicated output parallelto-serial converters (OSERDES) and input serial-to-parallel converters (ISERDES) with LVDS for inter-FPGA communication. If Logic Multiplexing (resp. ISERDES/OSERDES) is used as inter-FPGA communication architecture, the available I/Os are the number of FPGA I/Os (resp. the number of FPGA I/O pairs in mode LVDS). The first step of the algorithm is to reserve FPGA I/Os for global signals (i.e. global clocks, global reset), reset chain and external interfaces. Then, the temporary balanced platform netlist in Verilog format will be generated.

Figure 4 . 6 :

 46 Figure 4.6: The temporary balanced platform generation algorithm

4. 7 . 1

 71 Reservation of I/Os for Global Signals, Reset Chain and External Interfaces 4.7.1.1 Global Signals The Global Signals are peripherals that connect to all the FPGAs in the platform. The most common Global Signals are the PLL (affords global clocks), and the Reset Button (affords the global reset) as shown in Figure 4.7.

Figure 4 . 7 :

 47 Figure 4.7: Global signals (i.e. global clock and global reset)

Figure 4 . 8 :

 48 Figure 4.8: The platform after assigning global signals

Figure 4 . 9 :

 49 Figure 4.9: Global signal tracks VS multi-point tracks

Figure 4 .

 4 10(b).

  (a) The architecture of the reset chain (b) The pipeline using three FFs

Figure 4 . 10 :

 410 Figure 4.10: The reset chain

Figure 4 .

 4 Figure4.11(a). Nevertheless, in case of ISERDES/OSERDES, another pair of FPGA I/Os need to be reserved for sending the clock when sending the reset chain due to that ISERDES/OSERDES is source synchronous inter-FPGA communication architecture presented in Section 2.3.3. Therefore, 2 pairs of FPGA I/Os in mode LVDS in case of ISERDES/OSERDES are reserved for the reset chain in FPGA i and FPGA i+1 (0 <= i <= number of F P GAs -2) as shown in

Figure 4 .

 4 Figure 4.11(b). After assigning the FPGA I/Os for the reset chain, available I/Os per FPGA are

Figure 4 . 12 :

 412 Figure 4.12: The platform after assigning the external interfaces

  Figure 4.13: The generation of the balanced platform in the custom platform design flow: Step 1Once inter-FPGA tracks for an FPGA are defined, the number of available I/Os for corresponding FPGAs are updated. Then, the previous step is iterated to choose one FPGA and define the corresponding inter-FPGA tracks. In the example, after having defined the inter-FPGA tracks of FPGA 0, available FPGA I/Os are updated as shown in Figure4.14(a). In the next iteration, FPGA 1 has less available I/Os and inter-FPGA tracks of FPGA 1 will be defined. Therefore, 666 tracks in case of Logic Multiplexing (resp. 319 tracks in LVDS in case of ISERDES/OSERDES) will be defined between FPGA 1 and FPGA 2 as shown in Figure4.14(b) (resp. as shown in

Figure 4 .Figure 4 . 14 :

 4414 Figure4.14(c)). When all the inter-FPGA tracks are defined, the iteration will be finished and the temporary balanced platform will be generated.

Figure 4 .

 4 Figure 4.15: The interconnect synthesis algorithm

Figure 4 .

 4 Figure 4.16: An example of the partitioned design

Figure 4 .

 4 Figure 4.17: The modelling of the design

Figure 4 .

 4 Figure 4.18(d)).

Figure 4 . 19 :

 419 Figure 4.19: The generation of the custom platform: Step 2

Figure 4 . 20 :Figure 4 . 21 :

 420421 Figure 4.20: the custom platform in ISERDES/OSERDES

Figure 4 . 22 :

 422 Figure 4.22: The modelling of the design

Figure 4 . 23 :

 423 Figure 4.23: Generation of multi-point tracks

Figure 4 .

 4 Figure 4.24: result of the platform

  The automatic design flow starts by the platform generation flow. The inputs of the platform generation flow are the prototyped design RTL, the FPGA Lib and the user definition file. Through several steps, the flow generates the board netlist. The board netlist can be high-performance with considering the constraints for high-speed signaling (i.e. LVDS), high performance multiplexing (i.e. ISERDES/OSERDES) and so forth. The FPGA lib contains the logic capacity information and the user I/O information of different FPGA types. The user definition file contains the information of the chosen FPGA type, the maximum FPGA logic capacity utilization, the chosen inter-FPGA track type ("only 2 point tracks" or "2-and multi-point tracks"), and the chosen multi-

Figure 4 .

 4 Figure 4.25: The automatic design flow for creating a custom platform

5 .

 5 Interconnect Synthesis: Inter-FPGA tracks are distributed according to the distribution of cut nets. We propose an Interconnect Synthesis Algorithm in Section 4.8 to do this step. The proposed interconnect synthesis algorithm is realized in python programming language. Finally, the board netlist in Verilog format and the fsp format file (fsp: FPGA System Planner) are generated. The fsp format file can be loaded by Cadence Board Floorplan tool Allegro[FSP, 2014] to generate PCB layout. The screenshot of an example generated by the proposed automatic design flow is depicted in Figure4.26.

Figure 4 .

 4 Figure 4.26: The generated results loaded in the Floorplan tool Allegro

  Different FPGA types have different logic capacity, different numbers of FPGA I/Os and different prices. If different FPGAs are chosen, the generated multi-FPGA platforms have different costs and different performances. The board exploration is defined to generate different multi-FPGA platforms by choosing different FPGA types in the FPGA circuit development (from design RTL to board netlist). Then, an optimum prototyping platform (board netlist) will be chosen according to designers' specifications. According to Section 4.9, the board exploration is not possible when the FPGA circuit development is manual and one trial needs 1-2 months. The proposed design flow, which automatizes the FPGA circuit development, permits the board exploration due to that each launch of the flow spends only several hours. An example of the board exploration is shown in Figure 4.27. Three different custom platforms are generated by the automatic design flow with three different FPGA types. These FPGA types have the same number of FPGA I/Os and different logic capacities. According to their logic capacities, these FPGAs are classified as small FPGAs, middle FPGAs and large FPGAs. Different custom platforms have different costs and achieve different performance. The optimum prototyping platform (cost-optimal, performance-optimal, or intermediate) can be chosen.

Figure 4 .

 4 Figure 4.27: Board exploration

Figure 4 .

 4 Figure 4.28: Logic Multiplexing VS ISERDES/OSERDES

Figure 4 .

 4 Figure 4.29: Scene 1: 2-point tracks VS 2-and multi-point tracks

Figure 4 .

 4 Figure 4.30: Scene 2: 2-point tracks VS 2-and multi-point tracks

Figure 4 .

 4 Figure 4.31: Board exploration (optimal solutions)

Figure 5 . 1 ,

 51 therefore the width of connectors is 6. The multiple of the width of connectors and the maximum number of the connectors to one FPGA should be less than the maximum number of FPGA I/O pairs in mode LVDS as shown in Equation5.1. For example, if the chosen FPGA in the cabling platform is XC7V2000TFLG1925 (vendor: Xilinx, family: Virtex-7, device: 2000T, and package: FLG1925, thus FPGA type: XC7V2000TFLG1925), the maximum number of FPGA I/O pairs in mode LVDS is 576. Therefore, the maximum width of connectors is 576. width of connectors * number of connectors ≤ max. F P GA I/O pairs in LV DS (5.1)The minimum width of connectors is limited by the characteristics of one FPGA I/O bank.

Figure 5 . 1 :

 51 Figure 5.1: The width of connectors in the cabling platform

Figure 5 . 2 ,

 52 Figure 5.2, more specifically an algorithm optimizing the distribution of the cables. The proposed automatic design flow that is realized in python programming language, automatically joins the point tools. It consists of the platform generation flow and the implementation flow. In the platform generation flow, a cabling platform tailored for a given design is automatically generated by only choosing the FPGA type used. In the implementation flow, the partitioned design is routed into the generated multi-FPGA platform. As most of off-the-shelf platforms, there is only one FPGA type used in one cabling platform. Therefore, different cabling platforms can be identified by the input design and the FPGA type. The automatic design flow starts by the platform generation flow. The inputs of the platform generation flow are the prototyped design RTL, the FPGA Lib and the user definition file. Through several steps, the flow generates the cable distribution and the board netlist. The board netlist is high-performance with considering the constraints for high-speed signaling (i.e. LVDS), high performance multiplexing (i.e. ISERDES/OSERDES) and so forth. The FPGA lib contains the logic capacity information and the user I/O information of different FPGA types. The user definition file contains the information of the chosen FPGA type, the maximum FPGA logic capacity utilization, the inter-FPGA track type (In the cabling platform, only 2 point tracks exist), and the multiplexing architecture (In the cabling platform, only ISERDES/OSERDES is used). Different steps of the platform generation flow are presented as follows.

Figure 5 . 2 :

 52 Figure 5.2: The automatic design flow for creating a cabling platform

Figure 5 . 3 .

 53 The enhanced part is called the cable distribution algorithm, located in the left of the figure where "Cabling = Yes".Only 2-point tracks are supported in the cabling platform due to that only 2-point cables exist.

Figure 5 . 3 :

 53 Figure 5.3: The enhanced interconnect synthesis algorithm

Figure 5 . 4 :

 54 Figure 5.4: After assigning the global signals, the reset chain and the XIs

Figure 5 . 5 :

 55 Figure 5.5: The generation of the platform: Step 1

Figure 5 . 6 :

 56 Figure 5.6: The generation of the platform: Step 2

  1 and the Equation 5.2 should be fulfilled. Each testbench is implemented in different cabling platforms with different width of connectors as shown in Figure 5.7. These results are obtained by changing the parameter controlling the width of connectors in the automatic design flow and relaunching the flow to get the achieved performances. The runtime to run the whole flow is about several hours. (a) length of the cables = 25cm (b) length of the cables = 50cm

Figure 5 . 7 :

 57 Figure 5.7: The influence of the width of connectors in performance

Figure 5 . 8 :

 58 Figure 5.8: The cabling multi-FPGA platform

Figure 6 . 1 :

 61 Figure 6.1: The workflow of the performance comparison

Figure 6 . 2 :

 62 Figure 6.2: Comparison of the performance

  tom, and cabling. The off-the-shelf platform consists of a ready-made generic multi-FPGA board, where all the inter-FPGA connections are fixed and realized using PCB traces. The custom platform consists of a build-your-own specific multi-FPGA board, where all the inter-FPGA connections are realized using PCB traces as well. The cabling platform consists of multiple ready-made FPGA boards connected by cables and connectors, where all the inter-FPGA connections are realized using cables and connectors instead of PCB traces.Off-the-shelf platforms come with two types of inter-FPGA tracks: 2-point tracks or multipoint tracks. However, existing routing algorithms can not automatically employ multi-point tracks. In the manuscript, we have proposed a new routing algorithm to spare FPGA I/Os by automatically routing and multiplexing multi-terminal nets in multi-point tracks. The experiment results in an off-the-shelf platform of six Virtex-5 show up to 22% gain in performance, compared to the existing routing algorithm that routes multi-terminal nets of the design through a sequence of 2-point tracks and considers multi-point tracks as 2-point tracks only. 70% of multi-FPGA prototyping platforms are custom platforms due to performance requirement, external interfaces, and cost. Different from the off-the-shelf platform that has generic and balanced connections, the connections inter FPGAs as well as the connections to external interfaces of the custom platform are user-defined and tailored for a specific design. Naturally, the custom platforms should achieve the highest performance. Nevertheless, crafting a custom multi-FPGA platform is today a manual process. The performance and the cost of the platform depend on the FPGA expertise and SoC DUT knowledge of the prototyping team. Therefore, not so good performance platforms commonly occur. Crafting a custom platform is an iterative process, thus is time-consuming (4-7 months). The process involves FPGA circuit development (from design RTL to board netlist), PCB layout, PCB manufacture, Assembly, Test and Rework. The tradeoff between the performance and the cost exists due to that the performance of multi-FPGA platforms is limited by the inter-FPGA communications. The board exploration with different FPGA types in FPGA circuit development can not be done to design an optimum prototyping platform because one trial needs 1-2 months. As the ratio between the logic capacity and the number of FPGA I/Os is increasing at an exponential rate, it becomes tougher and tougher to keep performance as it was in multi-FPGA platforms. In this manuscript, we have proposed an automatic design flow for creating a custom platform, thus increasing the productivity, enabling board exploration, and optimizing cost and performance. The proposed automatic design flow automatically joins the commercial or self-developed point tools. We have developed several point tools such as: the external interface placement tool that automatically find a solution for the external interface placement, and the interconnect synthesis tool that automatically distribute inter-FPGA tracks according to the distribution of cut nets. The proposed automatic design flow drops the entry barrier of board designers, and can generate high-performance board netlist with taking the high-speed signaling, high-performance multiplexing into consideration. With the proposed automatic design flow, the board netlist of the custom platform can be generated with an acceptable CPU time(2-12 hours)

Figure 7 . 2 :Figure 7 . 3 :

 7273 Figure 7.2: Evolution of the maximum total user I/Os data rate

  2. Routing / Multiplexing: The cut nets are routed meaning that a cut net is allocated to an inter-FPGA track in the platform. As there are fewer available inter-FPGA tracks than the number of cut nets, several cut nets need to be multiplexed and sent together onto a single track on the platform.3. FPGA PnR Compiler: The bitstream for each FPGA is generated from the sub-design. Then, the generated bitstreams are downloaded into the platform to model the design.

	Nowadays, there are four commercial implementation flow tools: Cadence Protium tool [Pro-
	tium, 2014], Synopsys ProtoCompiler tool [ProtoCompiler, 2014], Auspy tool [Auspy, 2014], S2C
	tool [S2C, 2014], and Flexras Wasga tool

Table 2 .

 2

		1: Comparison of Different Multi-FPGA Prototyping Platforms
	Multi-FPGA		Performance		Cost
	Prototyping	Availability	Inter-FPGA	tracks	Flexibility	Unit Deployment
	Platforms		data rate	distribution		Price	Cost
	Hardwired Off-the-Shelf	instantaneous	high	generic & balanced	medium	low	high
	Cabling	1-2 months	high	designers' experience	high	low	high
	Hardwired Custom	4-7 months	designers' experience experience designers'	low	high	low

  , the latency in Logic Multiplexing is (mux_logic + 3) fast clock cycles as there are 3 flip-flops in the datapath of the fast clock domain. Therefore, the relationship between the system clock frequency sys_clk and the fast clock frequency if _clk is shown in Equation2.2. In Figure2.16, if _clk is 125M Hz, mux_logic is 4 and sys_clk is 17.86M Hz.

	sys_clk =	if _clk lat	=	125 mux_logic + 3	M Hz	(2.2)
	2.3.3 ISERDES/OSERDES
	In ISERDES/OSERDES, dedicated output parallel-to-serial converters (OSERDES) and input
	serial-to-parallel converters (ISERDES) are used for inter-FPGA communication as shown in Fig-
	ure 2.17. The Low-Voltage Differential Signaling (LVDS) is a signaling standard that provides
	high-speed data transfers by using a pair of FPGA I/O pins. Using ISERDES/OSERDES with
	LVDS, the inter-FPGA communication data rate for the targeted multi-FPGA board can be more
	than 1Gbps [Pinmux, 2014]. This architecture propagates the clock on a parallel path to the data
	for the synchronization and this synchronous method is source-synchronous [SerialIO, 2014]. The
	system clocks of two FPGAs have the same frequency due to that they are from the same Board
	Oscillator, thus are synchronous with each other. The fast clock of the source FPGA is generated
	by the PLL and is a frequency multiplier of the system clock, thus are synchronous with the sys-
	tem clock. Then, the fast clock is propagated from the source FPGA to the destination FPGA.
	Therefore, the fast clock of the destination FPGA is synchronous with that of the source FPGA.
	According to [SelectIO, 2014], a bank is a group of FPGA I/O pins that share a common resource
	such as one power supply or one output current reference. Several parallel ISERDES/OSERDES
	can be instantiated together with only propagating one clock, when the outputs of the source FPGA Figure 2.17: ISERDES/OSERDES for Multi-FPGA Prototyping
	(resp. the inputs of the destination FPGA) are the FPGA I/O pins from the same bank. If not, one
	pair of pins per bank between FPGAs is reserved to propagate the clock instead of the user data
	due to FPGA technology limitation. The pair of FPGA I/Os that receives the clock need to be
	clock capable. This kind of inter-FPGA communication is called bank-to-bank communication.

Table 2

 2 , the system clock frequency for ISERDES/OSERDES is measured. The

	SERDES need another clock (noted as if _clk_2x), which is a frequency multiplier of the fast
	clock [SelectIO, 2014]. When the data rate is 1Gbps, if _clk_2x = 500M Hz (due to that the
	transmission is double data rate) and the fast clock frequency is 250M Hz. The latency of IS-
	ERDES (resp. OSERDES) is 2 fast clock cycles [SelectIO, 2014] and 1 + w fast clock cycles
	are needed for the start pattern and w-bit wide logic Multiplexer/De-Multiplexer. The total delay
	T from OSERDES to ISERDES is the sum of the output delay T out (delay from OSERDES to
	pad, 0.5ns), the inter-FPGA board trace delay T board (1ns) and the input delay T in (from pad to
	ISERDES, 0.5ns), thus is 2ns. There is no need for the tolerance delay in ISERDES/OSERDES.
	Therefore, the total latency is ((5 + w) fast clock cycles + 2ns). As the data rate is 1Gbps, the for-
	mula of the total latency is changed to be (5+w) * 1ns+2ns 1(ns/f ast clock cycle) = (7+w) f ast clock cycles. There-
	fore, the relationship between the system clock frequency sys_clk and the fast clock frequency
	if _clk is shown in Equation 2.3. In Figure 2.17, if _clk is 250M Hz, if _clk_2x is 500M Hz,
	mux_serdes is 8 and sys_clk is 27.78M Hz.
	sys_clk =	if _clk lat	=	250 7 + ceil( mux_serdes 4	)	M Hz	(2.3)
	2.3.4 Logic Multiplexing VS ISERDES/OSERDES
	Logic Multiplexing and ISERDES/OSERDES are compared in the data rate, the complexity and
	the constraint as shown in

.2. ISERDES/OSERDES achieves higher data rate than Logic Multiplexing with better usage of FPGA resources (i.e. dedicated blocks) to implement the serialization. The added IPs in ISERDES/OSERDES has a high complexity while only a simple multiplexer and some control logic is required in Logic Multiplexing. ISERDES/OSERDES is only usable on the platforms which take the bank-to-bank limitation into consideration, while Logic Multiplexing has no constraint.

Table 2 .

 2

	2: Logic Multiplexing VS ISERDES/OSERDES
		Logic Multiplexing ISERDES/OSERDES
	Data Rate	LOW (~125Mbps)	HIGH (~1Gbps)
	Complexity	LOW	HIGH
	Constraint	NO	bank-to-bank

  The off-the-shelf multi-FPGA platform DN9000K10PCI[DINI, 2014] with both 2-and multi-point tracks is available and used for experiments. The architecture is shown in Figure3.11. There are six identical FPGAs. They are

	Xilinx FPGAs [Virtex-5, 2014] with the family Virtex-5, device LX330, and the package FF1760
	(noted as FPGA type: XC5VLX330FF1760). 1244160 Look-Up Table, 1244160 Register, 1152
	Block RAM and 768 Embedded DSP are contained in this board. The number of tracks for each
	line is shown in the figure.

Table 3 .

 3

			1: Partitioning results of testbenches
	testbench	Total Cut Nets 2-ter (%) 3-ter (%) 4-ter (%) 5-ter (%) 6-ter (%) m > 2(n -1) m ≤ 2(n -1)
	leon2	12430	42.84	56.85	0.31	0	0
	leon3_avnet	6072	98.39	1.61	0	0	0
	leon3mp	4460	98.57	1.41	0.02	0	0
	netcard	12929	27.34	10.3	20.53	41.51	0.32
	vga_lcd	2731	22.85	12.38	19.22	19.74	25.81
	des_perf	1764	56.29	25.34	12.87	4.76	0.74
	ethernet	1090	27.15	16.15	47.61	9	0.09

Table 3 .

 3

		2: The comparison of routing results with different algorithms
	Inter-FPGA		Turki's algorithm	proposed algorithm	gain	gain
	Communi-cation	testbench	mux hop	sys_clk mux hop (MHz)	sys_clk vs 2-(MHz) point LM vs
		leon2	19	2	5.21	18	2	5.43	4%
		leon3_avnet	8	1	10.42	8	1	10.42	0
	LM	leon3mp netcard	5 28	1 3	13.89 3.68	5 24	1 3	13.89 4.17	0 13%
		vga_lcd	6	3	10.42	5	3	11.36	9%
		des_perf	4	2	13.89	4	1	15.63	12%
		ethernet	6	1	12.5	4	1	15.63	25%
		leon2	68	0	10.42	68	0	10.42	0	92%
		leon3_avnet	32	0	16.67	32	0	16.67	0	60%
	SERDES	leon3mp netcard	18 71	0 3	20.83 2.5	18 134	0 1	20.83 3.05	0 22% -27% 50%
		vga_lcd	30	2	5.56	24	2	6.41	15% -44%
		des_perf	6	3	6.94	10	2	8.33	20% -47%
		ethernet	12	1	12.5	10	1	12.5	0	-20%

Table 4 .

 4 1: Logic capacity, I/O capacity and estimated price of different FPGA types

		FPGA type			Logic Capacity			I/O Capacity		Price
	Family	Device	Package	LUT	REG	RAM	DSP	I/Os	LVDS	bank	($)
	Virtex-5	LX330	FF1760	207,360	207,360	288	192	1,200	600	32	2,310
	Spartan-6	LX150T	FG900	92,152	184,304	134	180	540	270	6	1,050
	Virtex-6	LX550T	FF1759	343,680	687,360	632	864	840	420	21	3,850
	Virtex-6	LX760	FF1760	474,240	948,480	720	864	1,200	600	30	5,320
	Virtex-7	2000T	FLG1925	1,221,600	2,443,200	1,292	2,160	1,200	576	24	14,000

e. Virtex-7 or Stratix5), the device (i.e. 2000T or GXAB), and the package (i.e. FLG1925 or F1932). Different types of FPGAs have different logic capacity, different I/O capacity and different prices. An example of five different Xilinx FPGA types are shown in Table 4.1.

Table 4 .

 4

	2: Main costs associated with custom platform development
	Costs	Min	Max
	Personnel cost	$10,000	$15,500
	Material cost (per board)	$50,357	$58,333
	Assembly cost (per board)	$3,000	$4,333
	Opportunity cost	$37,500	$58,245
	Total cost	$47,500 + ($53,357/board) $73,745 + ($62,666/board)

Table 4 .

 4

	3: Time spent for typical tasks of custom platforms
	Task	Min	Max
	FPGA circuit development (from design RTL to board netlist)	1 month	2 month
	PCB layout	2 month	2.5 month
	PCB manufacture	0.5 month	1 month
	Assembly	0.15 month +	0.25 month +
	(per board)	(0.025 month/board) (0.025 month/board)
	Test (per board)	0.05 month/board	0.1 month/board
	Rework	0 month	1 month
	Total time	3.65 month +	6.75 month +
	spent	(0.075 month/board) (0.125 month/board)

Table 4 .

 4 [START_REF]Routing Algorithm for Multi-FPGA Based Systems Using Multi-Point Physical Tracks Qingshan Tang[END_REF] shows that more copies (boards) of the custom platform do not add many more over-

heads in time. The proposed design flow automatizes the FPGA circuit development. Therefore, with the help of the proposed automatic design flow, the time spent of the FPGA circuit development reduces from 1-2 months to several hours, which can be neglected compared to other tasks shown in

Table 4.3.

Table 4 .

 4

			4: Logic Multiplexing VS ISERDES/OSERDES
		Logic Multiplexing	ISERDES/OSERDES	gain vs
	testbench	mux hop	sys_clk	mux hop	sys_clk	Logic Multiplexing
	leon2	8	1	10.42 MHz	21	0	19.23 MHz	85%
	leon3_avnet	4	1	15.63 MHz	20	0	20.83 MHz	33%
	leon3mp	3	0	20.83 MHz	7	0	27.78 MHz	33%
	netcard	15	0	6.94 MHz	44	0	13.89 MHz	100%
	vga_lcd	5	0	15.63 MHz	16	0	22.73 MHz	45%
	des_perf	3	0	20.83 MHz	4	0	31.25 MHz	50%
	ethernet	3	0	20.83 MHz	6	0	27.78 MHz	50%

Table 4 .

 4

			5: Scene 1: 2-point track VS 2-and multi-point track
		2-				2-& multi-point
	testbench	point sys_clk num (MHz)	ter ≥ 6 sys_clk gain num (MHz) %	ter ≥ 5 sys_clk gain num (MHz) %	ter ≥ 4 sys_clk gain (MHz) %
	netcard	13.89	0	13.89	0	220	11.36	-18 330	10.42	-25
	vga_lcd	22.73	140	20.83	-8	240	20.83	-8	330	20.83	-8
	des_perf	31.25	10	31.25	0	30	31.25	0	100	27.78	-11
	ethernet	27.78	0	27.78	0	10	27.78	0	210	27.78	0

Table 4 .

 4

			6: Scene 2: 2-point track VS 2-and multi-point track
		2-				2-& multi-point
	testbench	point sys_clk num (MHz)	n ≥ 6 sys_clk gain num (MHz) %	n ≥ 5 sys_clk gain num (MHz) %	n ≥ 4 sys_clk gain (MHz) %
	netcard	13.89	0	13.89	0	220	14.71	6	330	14.71	6
	vga_lcd	22.73	140	22.73	0	240	25	10	330	25	10
	des_perf	31.25	10	31.25	0	30	31.25	0	100	31.25	0
	ethernet	27.78	0	27.78	0	10	27.78	0	210	31.25	12

Table 4 .

 4 Among these steps, the FPGA circuit development takes about 1-2 months. Therefore, this is impossible to do the board exploration due to the cost of the time. The proposed automatic design flow permits the board exploration by totally automatizing the FPGA circuit development process. The results show that one launch takes several hours (2-12 hours) with an acceptable internal memory occupied (about 2-4 GB). In case of the same FPGA type and the same number of FPGAs in the platform, the CPU time is longer when there are more external interfaces in the testbench. In case of the same testbench and the same filling rate, the CPU time is longer when there are more FPGAs in the platform. If the testbench has larger logic capacity (logic capacity of

					7: Board Exploration
	testbench	FPGA Filling # of CPU MEM Price type rate XIs time size	# of FPGA	Cost	sys_clk
			(%)		(h)	(GB)	($)		($)	(MHz)
		type 1	60	2	2.8	4.2	2,310	6	13,860	19.23
	leon2	type 2	60	2	2.5	4.3	3,850	4	15,400	16.67
		type 3	60	2	2.5	4.2	5,320	3	15,960	31.25
		type 1	65	10	12.3	4.7	2,310	6	13,860	20.83
	leon3_avnet	type 2	65	10	11.4	4.7	3,850	4	15,400	22.73
		type 3	65	10	10.9	4.7	5,320	3	15,960	31.25
		type 1	40	7	6.4	2.9	2,310	6	13,860	27.78
	leon3mp	type 2	40	7	6.3	2.8	3,850	4	15,400	31.25
		type 3	40	7	6.1	2.7	5,320	3	15,960	31.25
		type 1	55	3	5.5	3.5	2,310	6	13,860	13.89
	netcard	type 2	55	3	3.7	3.5	3,850	4	15,400	7.6
		type 3	55	3	3.4	3.5	5,320	3	15,960	20.83
	According to Table 4.3, crafting a custom platform for a given design is a time-consuming
	process. There are seven different tasks when crafting the custom platform: FPGA circuit devel-
	opment (from design RTL to board netlist), PCB layout, PCB manufacture, assembly, test and
	rework.

  According toChapter 3, intermediate FPGAs may be necessary if there is no available track between the FPGA (the Driver) and the FPGA (the Receiver) in Design Routing process. The maximum number of intermediate FPGAs needed (routing hop) is noted as hop. When implementing ISERDES/OSERDES, the routing hops are not realized in a pipelined way according to Section 3.6.3. The transmission of data in the intermediate FPGA can only start when all the

	3 (resp. Equation 5.4) if the maximum cables are 25cm (resp. 50cm) long in the
	cabling platform. The maximum number of cut nets passing through one ISERDES/OSERDES is
	noted as mux_serdes.
	sys_clk =	if _clk lat	=	250 M Hz 10 + ceil( mux_serdes 4	)	(5.3)
	sys_clk =	if _clk lat	=	250 M Hz 11 + ceil( mux_serdes 4	)	(5.4)

Table 6 .

 6

			1: Performance Comparison of Different Platforms
		Off-		Cabling
		the-	cables length	cables length		Custom
	testbench	Shelf sys_clk sys_clk gain vs sys_clk gain vs sys_clk gain vs = 50cm = 25cm	gain vs	gain vs
				Shelf		Shelf		Shelf	Cabling Cabling
		(MHz)	(MHz)		(MHz)		(MHz)		(50cm)	(25cm)
	leon2	11.36	16.67	47%	17.86	57%	20.83	83%	25%	17%
	leon3_avnet	15.63	17.86	14%	19.23	23%	22.73	45%	27%	18%
	leon3mp	25	20.83	-16%	22.73	-9%	27.78	11%	33%	22%
	netcard	6.58	13.16	100%	13.89	111%	14.71	124%	12%	6%

Table 6

 6 .2.

Table 6 .

 6 2: Updated Qualitative Comparison of Different Multi-FPGA Prototyping Platforms

	Multi-FPGA			Performance				Cost
	Prototyping	Availability	Inter-FPGA Inter-FPGA	tracks	Flexibility	Unit	Deployment
	Platforms		data rate	delay	distribution		Price	Cost
	Hardwired Off-the-Shelf	instantaneous	~1Gbps	1ns	generic & balanced	medium	low	high
	Cabling	2-12 hours	~1Gbps	4ns or 5ns	tailored for the design	high	low	high
	Hardwired Custom	3-5 months	~1Gbps	1ns	tailored for the design	low	high	low

à l'emploi, et "semi-sur mesure" par le fait que les connexions entre FPGAs sont définies par l'utilisateur et adaptées au design testé.Nous avons dégagé dans ce manuscrit trois problèmes majeurs en matière de prototypage à base de cartes multi-FPGA:[START_REF]Performance Comparison between Multi-FPGA Prototyping Platforms: Hardwired Off-the-Shelf, Cabling and Custom Qingshan Tang[END_REF]. L'évolution des FPGA tend à faire des entrées/sorties (E/S) une ressource rare, aggravant le problème de bande passante inter-FPGA génération après génération.En effet, les plateformes multi-FPGA souffrent des délais de communication importants inter-FPGA par rapport aux délais intra-FPGA. Par conséquent, il devient de plus en plus difficile de prototyper un design SoC/ASIC à des performances intéressantes. (2). 70% des plateformes de prototypage multi-FPGA sont des plateformes sur mesure en raison des besoins de performance et de coût. Néanmoins, la définition d'une plateforme sur mesure est aujourd'hui un processus majoritairement manuel et chronophage. Ainsi, l'exploration de cartes avec des types de FPGA différents, qui permettrait aux ingénieurs de concevoir une plateforme de prototypage optimale, ne peut pas être faite. Comme le rapport entre la capacité logique et le nombre d'E/S des FPGA est en augmentation à un rythme quasi-exponentiel, il devient de plus en plus difficile de concevoir une plateforme sur mesure performante.[START_REF]Routing Algorithm for Multi-FPGA Based Systems Using Multi-Point Physical Tracks Qingshan Tang[END_REF]. La plateforme câblée bénéficie de la disponibilité et de la personnalisation. Les performances d'une plateforme câblée dépendent de la distribution des câbles et du placement des interfaces externes. Néanmoins, il n'existe pas d'outil permettant d'obtenir automatiquement une solution pour la distribution des câbles. Une distribution de câbles permettant d'obtenir des fréquences de fonctionnement élevées devient de plus en plus difficile à atteindre en raison de la limitation des E/S. Par rapport aux plateformes sur étagère, la valeur ajoutée, en terme de performance, des plateformes câblées et sur mesure peuvent être fortement dégradée par une définition des interconnexions entre FPGA inefficace.

Remerciements

Remerciements ix

Algorithm 2 Function: Place_all_the_XIs_to_the_same_FPGA() Require:

XI[num_inter -1...0]; Ensure:

for k ∈ [0, num_inter) do XI[k] = 0 end for if f easible_placement() == 1 then

The rest steps of the automatic design flow() end if return The achieved performance 4.6.2 Each External Interface connected to a different FPGA When each External Interface is connected to a different FPGA, the number of different cases depends on the number of FPGAs in the platform (num_f pga) and the number of external interfaces in the prototyped design (num_inter), thus is A num_inter num_f pga cases. Figure 4.4 shows that the results of these cases are the same due to the symmetry. Therefore, only one case is executed to reduce the runtime. In the example, there are 24 different cases because there are 4 FPGAs in the platform and 3 XIs in the prototyped design. Nevertheless, there is only one case that will be executed. Research Benchmarks and three testbenches from OpenCores Benchmarks. The results show that even though Logic Multiplexing can achieve higher performance than ISERDES/OSERDES with several testbenches in the off-the-shelf platform, ISERDES/OSERDES can always achieve higher performance than Logic Multiplexing in the custom platform with up to 100% performance gain.

Even though many off-the-shelf platforms are the platforms using "2-and multi-point tracks" as inter-FPGA tracks, the custom platform using "only 2-point tracks" as inter-FPGA tracks is preferred (multi-point tracks only used for global signals as presented in Section 4.8). With the proposed automatic design flow, the custom platform can be generated with an acceptable CPU time (several hours) and an acceptable internal memory resource (several GB). The board exploration can be done to permit that designers can choose the most adapted multi-FPGA board according to their specifications. The board exploration also shows that the smaller FPGA that is less expensive could be used to reduce the overall cost of the platform with sometimes higher performance.

The next Chapter focuses on the cabling platform. We propose a cabling platform with an algorithm to automatically optimize the cable distribution and the external interface placement.

Chapter 5

Cabling Multi-FPGA Platform

Introduction

The previous Chapter has discussed the custom platform and proposed the automatic design flow for creating a custom platform. The purpose of this Chapter is to discuss the cabling platform and propose an automatic design flow for creating a cabling platform. The main contribution is threefold. First, an algorithm is proposed to optimize the distribution of the cables. Then, with the help of the automatic tool, the optimal width of connectors for a cabling platform is explored.

Finally, we propose a cabling platform where one board is composed of one FPGA and several connectors. All the FPGA I/Os are used for FPGA-to-connector connections. The connections inter FPGAs as well as the connections to external interfaces can be added or removed by only connecting or disconnecting the cables (resp. daughter boards) with or from the connectors.

The rest of this Chapter is organized as follows. Section 5.2 explains the overview of the cabling platform. Section 5.3 presents the proposed automatic design flow, more specifically an algorithm optimizing the distribution of the cables. In Section 5.4, the optimal width of connectors for a cabling platform is explored. Then, Section 5.5 proposes the cabling platform. Finally, Section 5.6 concludes this Chapter.

Platform Overview

The cabling platform, which is a relatively new notion compared to the off-the-shelf platform and the custom platform, consists of multiple ready-made FPGA boards connected by cables and connectors. In between the off-the-shelf and the custom platform, the cabling platform is semi off-the-shelf as it consists of multiple ready-made boards, and semi custom as its connections inter FPGAs as well as connections to external interfaces are user-defined and tailored for a specific design. Nevertheless, all the inter-FPGA connections are realized using cables and connectors instead of PCB traces.

Future Work

Before proposing different research axes, it is interesting to have a look at the FPGA's evolution especially the I/Os interface as it is the bottleneck for multi-FPGA prototyping.

Evolution of FPGAs and Analysis

In Figure 7.1, we can classify FPGA I/O pins into two types: user I/Os and MGT I/Os. In this manuscript, Xilinx Virtex Family is studied according to [Virtex-4, 2014] [Virtex-5, 2014] [Virtex-6, 2014] [Virtex-7, 2014]. The results show that the evolution of the data rate capability for user I/Os and MGT I/Os is different. In conclusion, the data rate of user I/Os that is used for multi-FPGA prototyping, is at a stop.

Conversely, the data rate of MGT I/Os that is not actually used for multi-FPGA prototyping, is increasing.

List of Publications International Conference Publications

Regular Presentations