Giovanni Agosta

Daniele Cattaneo

William Fornaciari

Andrea Galimberti

Giuseppe Massari

Federico Reghenzani

Federico Terraneo

Davide Zoni

Carlo Brandolese

Massimo Celino

Francesco Iannone

Paolo Palazzari

Giuseppe Zummo

Massimo Bernaschi

Pasqua D ' Ambra

Sergio Saponara

Marco Danelutto

Massimo Torquati

Marco Aldinucci

Yasir Arfat

Barbara Cantalupo

Iacopo Colonnelli

Roberto Esposito

Alberto R Martinelli

Gianluca Mittone

Olivier Beaumont

Berenger Bramas

Lionel Eyraud-Dubois

Brice Goglin

Abdou Guermouche

Raymond Namyst

Samuel Thibault

Antonio Filgueras

Miquel Vidal

Carlos Alvarez

Xavier Martorell

Ariel Oleksiak

Michal Kulczewski

Alessandro Lonardo

Piero Vicini

Francesca Lo Cicero

Francesco Simula

Andrea Biagioni

Paolo Cretaro

Ottorino Frezza

Pier Stanislao Paolucci

Matteo Turisini

Francesco Giacomini

Tommaso Boccali

Simone Montangero

Roberto Ammendola

TEXTAROSSA: Towards EXtreme scale Technologies and Accelerators for euROhpc hw/Sw Supercomputing Applications for exascale

  L'archive ouverte pluridisciplinaire

Introduction and long-term objectives

High Performance Computing (HPC) is a vital infrastructure for both industry and social actors in any country. In addition to traditional domains, such as oil & gas, finance, or weather forecasting, HPC is increasingly important for emerging domains such as bioinformatics, medicine, security and surveillance, which fall in the spectrum of High Performance Data Analytics (HPDA) and High Per- The emergence of heterogeneous HW architectures and the trends towards "Green HPC" have prompted Europe to align its research priorities in HPC along a Strategic Research Agenda (SRA 1 ) resulting from wide consultations within the European Technology Platform for HPC (ETP4HPC), the PRACE initiative 2 , and the PlanetHPC 3 initiative.

The SRA highlights significant technology challenges due to the need to achieve high efficiency while remaining within reasonable power and energy bounds. This challenging goal can only be addressed with an holistic approach that takes into account multiple factors across the HPC hardware/software stack, including the use of applicationspecific, extremely efficient hardware accelerators, efficient software management of resources, data and applications, and efficient cooling systems. Together, these components can provide the desired computational power while keeping under control the power consumption of the supercomputer.

To this end, the TEXTAROSSA project aims at providing key technological advances on all three aspects and validate them on new development platforms representative of future HPC systems, using a wide range of applications from different domains, both within traditional HPC and coming from emerging domains. In particular, TEXTAROSSA aims implementing the above-mentioned approach through the following technical goals.

1. https://www.etp4hpc.eu/sra.html (last accessed July 2021) 2. https://https://prace-ri.eu (last accessed July 2021) 3. https://cordis.europa.eu/project/id/248749 (last accessed July 2021) Technical goals. 1) energy efficiency and thermal control via innovative two-phase cooling technology at node and rack level, fully integrated in an optimized multi-level runtime resource management driven by power, energy, and thermal models fed by on-board sensor data; 2) sustained application performance through efficient exploitation of highly concurrent accelerators (GPUs and FPGAs) by focusing on data/stream locality, efficient algorithms and programming models, tuned libraries and innovative IPs; 3) seamless integration of reconfigurable accelerators by extending field-proven tools for the design and implementation such as Vitis and OmpSs@FPGA to support new IPs and methodologies such as mixed-precision computing and power monitoring and control. 4) development of new IPs for mixed-precision AI computing, data compression, security, power monitoring and control, and scheduling. 5) Integrated Development Platforms by developing two architecturally different, heterogeneous Integrated Development Vehicles (IDVs), one as a dedicated testbed for two-phase cooling technology, and one supporting the wider range of project technical goals.

Strategic goals.

Organization of the paper

The rest of this paper is organized as follows. In Section 2, we introduce the TEXTAROSSA co-design approach. In Section 3 we describe the key technological innovations provided by the TEXTAROSSA project, while in Section 4 we provide an overview of the application use cases. In Section 5 we draw some conclusions and highlight future research directions.

TEXTAROSSA Co-design approach

From a methodology point of view TEXTAROSSA adopts a co-design process as key strategy for Fast Forward and Exascale computing, considering the entire system stack from underlying technologies to applications. The co-design process concerns five layers covering the whole HPC stack: 1) User Application: representing a wide range of scenarios, from mathematical libraries, to miniApps and flagship codes for numerical modelling with massive parallelism in HPC/HPDA/AI applications. 2) Runtime Services: ensuring that application requirements are dynamically satisfied and mapped onto system resources, and including execution models with workload handling, fault tolerance and data management. 3) Programming Models: underlying the applications, they define the toolchains and SW development 4. https://textarossa.eu (last accessed July 2021) Figure 2. The TEXTAROSSA Co-Design Approach tools able to implement applications in parallel architectures. 4) System Architecture: including the processor core's microarchitecture, the arrangement of cores within a chip, memory hierarchy, system interconnect, and storage subsystems. 5) HW Platforms: concerning the HW platform at node and rack level able to achieve performance requirements in terms of computing power and energy consumption.

Figure 2 provides an overview of the co-design approach adopted in TEXTAROSSA, showing how the five layers of the HPC stack are addressed in each of the four main stages of the co-design process: 1) Gap Analysis: to compare the current state-of-art of the technological assets with the objectives of the project in order to identify the gap to be filled by developments or update in co-design process.

2) Requirements: to define specifications and requirements of the technological solutions for designing and developing.

3) Proof of Concept: to develop HW/SW prototype solutions able to achieve the KPIs (Key Performance Indicatoirs) of the project objectives. 4) Benchmarking: to provide performance results of the technological solutions by means of benchmark tools.

TEXTAROSSA technologies

TEXTAROSSA develops, starting from the results of previous European research activities mentioned in Figure 1, a set of technologies to deal with each of the four technology layers of the HPC stack, as well as applications, which will be covered in full in Section 4. Figure 3 provides an overview of the primary technology bricks adopted by TEXTAROSSA. In the rest of this section, we provide insights on specific technology bricks developed within the project

Programming Models & Toolchains

Vitis based HLS flow. Vitis [START_REF] Kathail | Xilinx vitis unified software platform[END_REF] is the HLS flow designed by Xilinx to cover all the steps required to translate an application, described through a C/C++ program, into a working bitstream running on an FPGA card and communicating with a program running on the host node. The Vitis environment allows emulating the whole system's behavior (host program and FPGA code) by running and debugging the C/C++ code in a standard IDE. Once the functional correctness has been achieved, i.e. the program produces the expected results, the HLS engine translates the pieces of C/C++ code mapped on the FPGA into equivalent, optimized, hardware implementation. The communication with the external world (host node, memory banks) is achieved through a presynthesized layer implementing the interfaces (PCIe, DDR and HBM memory banks) and the FPGA resources are accessed from the host node through the runtime and APIs developed by Xilinx. Vitis is largely customizable thanks to the open-source LLVM-based front-end 5 and many open-source accelerated libraries (e.g., math, video processing, AI, signal processing) 6 . In the TEXTAROSSA project, the Vitis environment will be extended with multi-precision arithmetic, allowing the usage of new data types based on the Posit format, and with a communication library used to perform inter-FPGA direct communications. The inter-node communication layer, namely MPI, will also be updated to transfer data without requiring memory copies between FPGAs and hosts. Furthermore, the APIs to access the 5. https://github.com/Xilinx/HLS (last accessed July 2021) 6. https://xilinx.github.io/Vitis Libraries (last accessed July 2021) FPGA will be used to build the TEXTAROSSA APIs that will be defined to access homogeneously the accelerators. Such extensions will require the integration into the Vitis flow of the new IPs defined in the project, thus we will develop the hardware modules, their C++ functional models and we will encode all the necessary info needed by the HLS to properly manage (i.e. schedule and connect) the new functionalities.

StarPU, OmpSs. Task-based programming models (like OmpSs 7 [START_REF] Duran | Ompss: a proposal for programming heterogeneous multi-core architectures[END_REF] and StarPU 8 [START_REF] Augonnet | Starpu: a unified platform for task scheduling on heterogeneous multicore architectures[END_REF]) address the challenges to program on heterogeneous computing nodes while achieving high productivity by providing higher-level abstractions that could help the programmer to generate high-performance code. For example: making the memory allocation and data copies automatic; providing the programmer with facilities to perform blocking from inside the accelerators; automating the code generation of the CPU and FPGA binaries, provided the C/C++ implementation, by transparently running open or vendor tools; allowing the use of parallelism based on tasking (instead of kernel invocations); providing support for data-dependent tasks, and managing the execution based on such data dependencies and/or providing FPGA execution trace generation support; choosing the computation resource that provides the best execution compromise for a given task and situation [START_REF] Bosch | Application acceleration on fpgas with ompss@fpga[END_REF]. This makes the programming environment completely hide the target architectures, providing a clean, high-level, abstract interface to the programmers, and incorporating all the intelligence on management and scheduling into the runtime system. We plan to move part of the runtime work to a fast hardware task scheduler [START_REF] Tan | A hardware runtime for task-based programming models[END_REF] to further enhance the performance of these programming models.

Training Deep Neural Network (DNN) is a memoryintensive operation. Indeed, the training algorithms of most DNNs require to store both the model weights and the forward activations in order to perform back-propagation. In practice, training is performed automatically and transparently to the user through autograd tools for backpropagation. Unfortunately, the memory limitation of current HW often prevents data scientists from considering larger models, larger image sizes or larger batch sizes, especially in recent NLP models [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF]. Our goal is to extend StarPU to enable inference and learning, taking advantage of both the heterogeneity of the architecture to place layers and the memory architecture to minimize transfers. In particular, we will rely on the RoToR framework 9 [7] which allows to control the memory consumption and to minimize the energy consumed by the data exchanges.

Several problems should be solved to use FPGAs more efficiently and control energy consumption. In the TEX-TAROSSA project, we will extend these two task-based runtime systems, OmpSs and StarPU, which use different approaches to support FPGA. This will allow the integrated development vehicle to benefit from their existing features, 7. https://github.com/bsc-pm-ompss-at-fpga (last accessed July 2021) 8. https://starpu.gitlabpages.inria.fr (last accessed July 2021) 9. https://gitlab.inria.fr/hiepacs/rotor (last accessed July 2021) but also to study their complementarity while validating the robustness of the new HW against different runtime systems.

FastFlow. Stream processing is gaining increasing industrial attention for real-time data analytics and datadriven applications [START_REF] Aldinucci | Porting decision tree algorithms to multicore using FastFlow[END_REF]. FastFlow [START_REF] Aldinucci | Fastflow: high-level and efficient streaming on multi-core[END_REF] is a C++ programming library targeting multi/many-cores. It offers both a set of high-level ready-to-use parallel pattern implementations and a set of mechanisms and composable components (called building blocks) to support low-latency and high-throughput data-flow streaming networks. FastFlow simplifies the development of parallel applications modeled as a structured, directed graph of processing nodes. The graph of concurrent nodes is constructed by the assembly of sequential and parallel building blocks and higher-level components (i.e., parallel patterns) modeling recurrent schemas of parallel computations (e.g., pipeline, task-farm, parallel-for, etc.). FastFlow efficiency stems from the optimized implementation of the base communication and synchronization mechanisms and its layered software design. Besides, stream processing is the natural paradigm for event-driven distributed applications that need to communicate with each other via message passing. Finally, some data streaming paradigms are naturally suited for implementation on reconfigurable platforms [START_REF] Neuendorffer | Streaming systems in fpgas[END_REF], e.g. the dataflow/actor paradigm. In TEXTAROSSA, we aim at exploiting reconfigurable platforms to accelerate HPDA tasks leveraging the FastFlow framework [START_REF] Aldinucci | Design patterns percolating to parallel programming framework implementation[END_REF].

Streamflow. The StreamFlow framework [START_REF] Colonnelli | Streamflow: cross-breeding cloud with HPC[END_REF], [START_REF] Colonnelli | HPC Application Cloudification: The Stream-Flow Toolkit[END_REF] is a container-native Workflow Management System (WMS) written in Python 3 and based on the Common Workflow Language (CWL) Standard [START_REF] Amstutz | Common workflow language[END_REF]. StreamFlow has been designed around two main principles: 1) Allowing the execution of tasks in multi-container environments, in order to support concurrent execution of multiple communicating tasks in a multi-agent ecosystem; 2) Relaxing the requirement of a single shared data space, in order to allow for hybrid workflow executions on top of multi-cloud or hybrid cloud/HPC infrastructures. StreamFlow source code is available on GitHub under the LGPLv3 license. A Python package is downloadable from PyPI and Docker containers can be found on Docker Hub. More details about the tool and its applications can be found in the StreamFlow website 10 .

Compiler Technology for Mixed-Precision Support. Error-tolerating applications are increasingly common in HPC. Proposals have been made at the HW level to take advantage of inherent perceptual limitations, redundant data, or reduced precision input [START_REF] Stanley-Marbell | Exploiting errors for efficiency: A survey from circuits to applications[END_REF], as well as to reduce system costs or improve power efficiency [START_REF] Venkataramani | Approximate computing and the quest for computing efficiency[END_REF]. At the same time, works on floating-point to fixed-point conversion tools [START_REF] Cherubin | Tools for reduced precision computation: a survey[END_REF], [START_REF] Cattaneo | Embedded operating system optimization through floating to fixed point compiler transformation[END_REF] allow us to trade-off the algorithm exactness for a more efficient implementation. Finally, new data types are emerging including BFloat16 [START_REF] Burgess | Bfloat16 processing for neural networks[END_REF] and Posit [START_REF] Gustafson | Beating floating point at its own game: Posit arithmetic[END_REF]. BFloat16 (Brain Floating Point) is used in upcoming Intel AI processors (NERVANA), XEON processors, Google Cloud TPU and ARMv8.6-A, as well as in RISC-V extensions [START_REF] Zoni | An fpu design template to optimize the accuracy-efficiency-area trade-off[END_REF]. Posit are a new compressed floating-point data format for which University of Pisa has developed a SW library called 10. https://streamflow.di.unito.it (last accessed July 2021) CppPosit [START_REF] Cococcioni | Vectorizing posit operations on risc-v for faster deep neural networks: experiments and comparison with arm sve[END_REF], [START_REF] Cococcioni | A fast approximation of the hyperbolic tangent when using posit numbers and its application to deep neural networks[END_REF]. From the first results of applying the CppPosit library to AI/DNN problems, Posit can lead to the same processing accuracy of float but with a data compression from a factor 2 to 4 [START_REF] Cococcioni | Vectorizing posit operations on risc-v for faster deep neural networks: experiments and comparison with arm sve[END_REF], [START_REF] Cococcioni | A fast approximation of the hyperbolic tangent when using posit numbers and its application to deep neural networks[END_REF]. This means that applying Posit to the application cases (HPC, HPDA and AI/CNN) has the potential to reduce data storage issues and allows for fast data movement. In TEXTAROSSA, we aim at exploiting and extending the tools for precision tuning developed as part of the H2020 FETHPC ANTAREX project [START_REF] Silvano | The antarex domain specific language for high performance computing[END_REF] to cover a wider range of target platforms, targeting FPGAs through integration with the TEXTAROSSA High Level Synthesis (HLS) toolchain. These tools, collected in the TAFFO framework [START_REF] Cherubin | TAFFO: Tuning assistant for floating to fixed point optimization[END_REF], [START_REF] Cherubin | Dynamic precision autotuning with taffo[END_REF] are implemented as a set of plugins for the LLVM compiler, and, based on programmer hints expressed as attributes, perform value range analysis, data type and code conversion, and static estimation of the performance impact. We aim at improving the performance estimation by exploiting recent analysis techniques [START_REF] Darulova | Towards a compiler for reals[END_REF] as well as deeper understanding of the target processor pipeline, by expanding the use of the tools to heterogeneous systems with reconfigurable components, and by considering emerging data types such as Posit and Bfloat16.

Runtime Services: Energy/Power Management

Power consumption represents one of the most remarkable cost items in the balance of the overall costs of an HPC center. With exascale computing platforms a linear increment of power consumption with computing power (due to the end of Dennard's scaling) becomes unsustainable. An energy-efficient exascale HPC infrastructure, much like current supercomputers, will rely on heterogeneous computing resources. Energy efficiency can then be guaranteed only if, on the SW side, we can rely on a suitable (hierarchical) resource management framework. Although the state-of-theart already includes some solutions, recent projects, like MANGO [START_REF] Flich | Exploring manycore architectures for next-generation HPC systems through the MANGO approach[END_REF] and RECIPE [START_REF] Agosta | The RECIPE Approach to Challenges in Deeply Heterogeneous High Performance Systems[END_REF], show that optimized solutions need to take into account the platform-specific characteristics and control knobs to profile the applications at design-time and monitor them at runtime [START_REF] Zoni | All-digital control-theoretic scheme to optimize energy budget and allocation in multi-cores[END_REF], [START_REF] Zoni | All-digital energyconstrained controller for general-purpose accelerators and cpus[END_REF], enabling more accurate resource mappings [START_REF] Massari | Predictive resource management for nextgeneration high-performance computing heterogeneous platforms[END_REF], [START_REF] Brandolese | Software energy estimation based on statistical characterization of intermediate compilation code[END_REF]. Furthermore, an integration of the resource manager with the programming model allows dynamically tuning the numerical accuracy (precision) of the tasks, with respect to the actual application requirements and power/energy constraints. Given the reference HW platform and the application use cases, the TEXTAROSSA project would represent an extremely interesting testbed for exploring novel power and energy management solutions, at the HW but also at the SW level. At the HW level, a specific support will be introduced to automatically instrument the computing platform with ad-hoc power monitors [START_REF] Zoni | Powertap: All-digital power meter modeling for runtime power monitoring[END_REF] and controllers [START_REF] Zoni | All-digital control-theoretic scheme to optimize energy budget and allocation in multi-cores[END_REF], [START_REF] Zoni | All-digital energyconstrained controller for general-purpose accelerators and cpus[END_REF], in order to reduce the response time of the power management dramatically, while increasing the effectiveness of the thermal management. On the software side, starting from an already existing resource management framework [START_REF] Bellasi | Effective runtime resource management using linux control groups with the barbequertrm framework[END_REF], we aim to extend it with the support for the new HW and, of course, new resource management policies, along with the integration of the precision tuning tool. Overall, this would allow us to explore all the possibilities offered by the platform, and the application-side integration, to increase the FLOPS-per-Watt ratio, with respect to state-of-the-art solutions in HPC.

Posit Hardware Accelerators

Within EPI, the Stencil and Tensor Accelerator (STX), based on RISC-V core, enables energy-efficiency for applications where the main computational kernels work on discretized grids and perform a series of operations like convolutions. It implements several floating point units able to work on fp32, fp16, fp8 IEEE and bfloat16 floating-point formats. To increase performance, one could try to reduce the number of bits in the floating point representation. But when reducing the precision of the used arithmetic, iterating over many timesteps, the derivations may become unstable and hence affect the final result of the simulations. The novel Posit binary arithmetic format can offer higher precision while using less bits than standard IEEE floatingpoint numbers. Recent literature shows [START_REF] Cococcioni | Vectorizing posit operations on risc-v for faster deep neural networks: experiments and comparison with arm sve[END_REF], [START_REF] Cococcioni | A fast approximation of the hyperbolic tangent when using posit numbers and its application to deep neural networks[END_REF] that 16bit Posit can leverage comparable results like fp32 and Posit with 8bit precision outperform in terms of accuracy fp16 (for CNN 8bit Posit can leverage comparable results like fp32). Calculations can be done even with simple bit manipulations on the Posit format without extraction, further decreasing the complexity of the operations. It is thus possible to enhance memory bandwidth, and lower level cache utilization, power footprint, and throughput of the arithmetic units.

Within TEXTAROSSA, a RISC-V unit will be extended with support for alternative data representations, including fine-grained reduced precision floating point [START_REF] Zoni | An fpu design template to optimize the accuracy-efficiency-area trade-off[END_REF] and Posit arithmetic. To complement the hardware IP developments, the LLVM compiler, also adopted in EPI, will be extended for Posit and data compression support and real-world HPC applications and CNN kernels will be ported to leverage the IP. Fast software co-design will be enabled through software implementation of Posit in the CppPosit library 11 . The novel IP will be ported to FPGAs for benchmarking. Both techniques have not been implemented on top of a completely co-designed accelerator so far and will provide a huge benefit for the European IP portfolio.

Hardware Platform Optimization

HPC systems have historically always been limited by thermal considerations and computing architecture needs both optimized heat dissipation solutions and run-time thermal control policies to operate reliably and efficiently. In-Quattro has developed an innovative thermal management solution (patent pending) based on two-phase mechanically pumped loops, which uses a flow boiling heat transfer for cooling electronics in a more efficient way. This would allow the use of the latent heat of vaporization so that flow rates would be significantly reduced, temperature gradients 11. https://github.com/eruffaldi/cppPosit (last accessed July 2021) Figure 4. Schematic of the two-phase cooling system (left) and its embedding in a server (right). would be small, and heat transfer coefficients could be large compared to liquid cooling and air cooling systems. Twophase cooling systems using evaporation and condensation are known to be the best way to meet demanding cooling requirements in terms of compactness, weight and energyconsumption. A possible configuration of the two-phase cooling system for a single node is shown in Figure 4.

The two-phase cooling system will be adapted to node(s) provided by E4 and ATOS and developed with the objective to serve an entire rack. In TEXTAROSSA, the innovative cooling system will be customized to fit the requirements of node and rack levels for exascale applications. It is foreseen to develop two solutions of the two-phase cooling system that could be patented during the project on the principal components of the cooling system (evaporator, condenser). This innovative technology is expected to improve the cooling efficiency up to 70% compared to traditional air cooling, and up to 30% compared to existing liquid cooling and will be tested on both ATOS and E4 infrastructures.

Both the design of cooling solutions and thermal control policies critically relies on thermal models, which in turn rely on experimental validation data. Thermal simulators for CPUs/MPSoCs have been proposed, but they can only represent a limited range of heat dissipation solutions, and are not easy to extend towards two-phase liquid cooling solutions or to encompass the simulation of an entire rack. In TEXTAROSSA, we will design and validate thermal models taking advantage of equation-based object-oriented modeling languages to increase the abstraction level of thermal models [START_REF] Terraneo | 3D-ICE 3.0: efficient nonlinear MPSoC thermal simulation with pluggable heat sink models[END_REF], thus overcoming the inflexibility of current simulators, and simplifying the design of nonlinear thermal models to account for two phase liquid cooling. For model validation, we will use a thermal test chip platform [START_REF] Terraneo | An Open-Hardware Platform for MPSoC Thermal Modeling[END_REF] to capture accurate thermal maps of chips connected to the proposed heat dissipation solution, considering spatial and temporal temperature variations and hot spots. The collected experimental data will be used to validate the thermal models and for the design of the policies. Through a multilevel thermal control strategy we aim to overcome the complexity of controlling an HPC platform from node to system level with minimal overhead. As the fastest temperature gradients occur at the silicon active layer, we will use fast event-based control loops [START_REF] Leva | Event-based power/performance-aware thermal management for high-density microprocessors[END_REF] acting on DVFS to limit the maximum operating temperature of compute elements. These inner control loop will in turn interact with higher level control loops operating the two phase cooling infrastructure of the node, which is comparatively slower and has higher overheads but has the capability to increase the heat transfer coefficient on-demand, thus allowing to relieve the need to reduce frequency using DVFS, in turn improving performance. A further supervisory control layer will allow to set the desired temperatures at the rack level based on reliability metrics. Multilevel control allows thus to partition the system level control problem into multiple interacting control loops, each optimized for the specific thermal dynamics to control.

Low-latency Communication between FPGAs

The usage of FPGAs as accelerators is getting so widespread that even big cloud providers are now installing reconfigurable devices in their instances (e.g. on Microsoft Azure and Amazon EC2). Interaction of hundreds to thousands of FPGAs require a scalable approach to hold them together, allowing a low latency connection among them, but a definitive approach has to be found to let users make the most of their flexibility and in the meanwhile easing the usage for software developers. As an example, the latest version of the Microsoft Catapult fabric, puts a Stratix 10 device between each NIC on the X86 servers and the ToR switch, enabling a fast path for accelerators to communicate among themselves with a few microseconds latency. The Brainwave project [START_REF] Chung | Serving dnns in real time at datacenter scale with project brainwave[END_REF] leverages this architecture to provide a deep learning platform for real-time AI inference on the cloud. While this framework offers a very friendly interface for users to deploy their models on top of this architecture, it loses the flexibility of delivering the cores as black boxes, and providing an implementation of only a few pre-trained models. Our approach, on the other hand, let users full control of the platform, allowing the implementation of custom processing tasks on FPGAs, still maintaining ease of usage by supplying a set of interfaces to integrate with the HLS tools developed in the project. Thus will allow developers to define a scalable application using a streaming programming model (Kahn Process Network [START_REF] Kahn | The semantics of a simple language for parallel programming[END_REF]) that can be efficiently deployed on a multi-FPGAs system. TEX-TAROSSA will develop a communication IP and its SW stack, providing the implementation of a direct network that allows low-latency communication between processing tasks deployed on FPGAs, even hosted in different computing nodes. The communication IP will be based on the ExaNet IPs (switch, router, high-speed channels [START_REF] Ammendola | Large scale low power computing system: Status of network design in exanest and euroexa projects[END_REF]) developed in ExaNeSt H2020 project [START_REF] Katevenis | Next generation of exascaleclass systems: Exanest project and the status of its interconnect and storage development[END_REF] and EuroEXA H2020 [START_REF] Biagioni | Euroexa custom switch: an innovative fpgabased system for extreme scale computing in europe[END_REF]. The direct communication between tasks deployed on FPGAs will avoid the involvement of the CPUs and system bus resources in the data transfers, improving the platform's energy efficiency and reducing communication latency.

TEXTAROSSA applications

In this section, we briefly provide an overview of the use case applications adopted in TEXTAROSSA. To address the variety of application domains of future Exascale systems, TEXTAROSSA applications include basic mathematical building blocks (MathLib), traditional HPC applications (UrbanAir, TNM, HEP, and RTM), and applications from emerging domains (RAIDER, DPSNN, Danger Detection).

MathLib

One of the basic guidelines in energy efficient computing is the optimization and the acceleration of algorithms and SW libraries that provide a reduction of the elapsed time of HPC applications and thus a significant cut in energy consumption. The new power-to-solution metrics requires a rethinking of many computational kernels of HPC applications looking for a trade-off between the reduction of the total energy and the minimization of the time-to-solution, promoting scalability. Within this context, extensions and improvements of high-performance algorithms and SW libraries for kernels in numerical linear algebra [START_REF] Agullo | Achieving high performance on supercomputers with a sequential task-based programming model[END_REF], [START_REF] Cojean | Resource aggregation for task-based cholesky factorization on top of modern architectures[END_REF] and graph computation, such as iterative [START_REF] Bernaschi | A factored sparse approximate inverse preconditioned conjugate gradient solver on graphics processing units[END_REF], [START_REF] Bernaschi | AMG based on compatible weighted matching for GPUs[END_REF], [START_REF] Bernaschi | BootCMatchG: An adaptive algebraic multigrid linear solver for GPUs[END_REF], [START_REF] Filippone | A parallel generalized relaxation method for high-performance image segmentation on GPUs[END_REF] and direct linear solvers, edge weighted graph matching, and fast multipole methods [START_REF] Agullo | Task-based fmm for heterogeneous architectures[END_REF] will be deployed.

HPDA and HPC-AI Applications

Real-time AI-based Data analytics on hetERogeneous distributed systems (RAIDER). Leveraging on the experience gained on the design of the GPU-RICH system for the NA62 experiment at CERN [START_REF] Ammendola | NaNet: a flexible and configurable lowlatency NIC for real-time trigger systems based on GPUs[END_REF], a proof-of concept of a real-time AI-based data analytics on heterogeneous distributed systems shall be designed. The application setup deploys a set of data streams from sensors as input to a Deep Neural Network, implemented over a pool of heterogeneous processing layers, that is in charge of performing the data analysis. Data from different streams are recombined through the processing layers through the network infrastructure.

Brain Simulation (DPSNN). The Distributed and Plastic Spiking Neural Network (DPSNN) application was developed by INFN to model brain cortex behaviour [START_REF] Pastorelli | Gaussian and exponential lateral connectivity on distributed spiking neural network simulation[END_REF], [START_REF] Ammendola | The brain on low power architectures-efficient simulation of cortical slow waves and asynchronous states[END_REF] and more recently to study sleep-related learning activities [START_REF] Capone | Sleep-like slow oscillations improve visual classification through synaptic homeostasis and memory association in a thalamo-cortical model[END_REF]. It is a scalable neural network simulation C++/MPI code for HPC platforms at extreme scales. It simulates the spiking dynamics of the brain cortex by slicing it into a grid of cortical columns populated with neurons and their interconnecting synapses.

Smart Cities (Danger Detection). In the context of smart cities the distributed video surveillance systems provide a huge amount of data for processing with AI techniques. A design activity shall be carried out for a real-time danger alarm system (e.g. smoke/fire detection in smart transportation or smart cities context). The system is composed of a network of smart cameras where an AI algorithm is implemented on an EDGE server implementing a preprocessing stage plus a CNN.

Traditional HPC Applications

Air Pollution (UrbanAir). The UrbanAir concerns the modelling and forecasting of the concentration and dispersion of pollutants. It is a 3D multiscale model that combines a numerical weather prediction (NWP) model, running at larger scale (e.g. mesoscale), with a city-scale geophysical flow solver (EULAG) for accurate prediction of contaminant (e.g. NO2, PM2.5, PM10) transportation through the street corridors, over buildings and obstacles. A design activity shall be carried out to use mixed-precision computing and energy-efficient accelerators for faster response while preserving results accuracy.

Quantum Simulation (TNM). The Tensor Network Method (TNM) is a class of powerful numerical methods developed to study the equilibrium and out-of-equilibrium properties of strongly correlated many-body quantum systems [START_REF] Omran | Generation and manipulation of schrödinger cat states in rydberg atom arrays[END_REF]. TNM are complementary to Monte Carlo methods as they do not suffer from the sign problem. A design activity shall be performed to overcome current TNM limitations studying how pushing the simulation boundaries towards high-dimensional systems with the support of HPC infrastructure. We aim to perform the first scalable 2D and 3D simulations of LGT in and out of equilibrium.

High Energy Physics (HEP). HEP community has built a collection of high-level SW frameworks largely used for simulation and data analysis of the LHC experiments. They include widespread libraries like Geant4 and Fluka for particle-matter simulation, the use of high-level analysis tools like those in ROOT, and simulation packages of high energy collisions. A design activity shall be focused to optimize these SW frameworks for the realization of code bases able to execute on multiple architectures, including the next generations of pre-and exascale EU HPCs.

Biomedical Application (HPC-Drugs). Ligand binding affinity predictions carried out with Molecular Dynamics simulation is one of the main research focus in computational chemistry today due its potential impact in industrial drug discovery. A design activity shall be carried out for HPC-backed pharmaceutical applications based on n-body kernel functions running in specialized cores of GPU and FPGA, relying on recently discovered nonequilibrium thermodynamics theorems and capable of delivering absolute binding free energies of drug-size molecules in a predictable wall-clock time with a credible confidence interval, hence bypassing the limitations of the traditional equilibrium-based Free Energy Perturbation (FEP) alchemical approaches [START_REF] Macchiagodena | Virtual double-system single-box: A nonequilibrium alchemical technique for absolute binding free energy calculations: Application to ligands of the sars-cov-2 main protease[END_REF].

Reverse Time Migration (RTM). The Reverse Time Migration application and mini-kernels are used within EPI to co-design the STX Accelerator and have been ported to FPGAs within the EuroEXA project. The respective kernels will be analysed to which extent they can leverage the new, energy-efficient, capabilities like Posit arithmetic and lossy compression to enhance performance and energy efficiency. The RTM kernels are stencil-based kernels. Hence, they provide conclusions on many stencil based applications. Reverse Time Migration by FHG for HPC applications to Oil & Gas and Geo-Services.

Conclusions

The TEXTAROSSA project aims to achieve a broad impact on the HPC field both in pre-exascale and exascale scenarios. The TEXTAROSSA consortium will develop new IPs, algorithms, methods and software components for HPC-AI, HPC and HPDA applications, mostly Open Source and able to be adopted as standalone building blocks or to interoperate with other Exascale-ready components. Through the participation of three supercomputing centers in the consortium, the proposed technologies will be tested by and known to the HPC community.

Figure 1 .

 1 Figure 1. The TEXTAROSSA Impact Strategy within the broader framework of European initiatives on HPC

Figure 1

 1 highlights the strategic goals of TEXTAROSSA, set against the EuroHPC initiative and the broader framework of European research in High Performance Computing. In particular, we highlight the following goals: 1) alignment with the European Processor Initiative (EPI) by testing, extending and boosting key technologies applicable to future EPI evolutions; 2) supporting the objectives of EuroHPC as reported in ETP4HPC's Strategic Research Agenda (SRA) for open HW and SW architecture. 3) building over European expertise gained through past research projects as well as through the Centers of Excellence in HPC. 4) opening of new usage domains, including High Performance Data Analytics (HPDA) and High Performance Artificial Intelligence (HPC-AI) applications, alongside support for traditional HPC domains. 1.1. TEXTAROSSA Consortium To achieve the above-mentioned goals, TEXTAROSSA, a three-year project co-funded by the European High Performance Computing (EuroHPC) Joint Undertaking, is led by ENEA (Italy) and aggregates 17 institutions and companies, including the linked third parties, located in 5 European countries: CINI, an Italian consortium grouping together three leading universities, Politecnico di Milano, Università degli studi di Torino, and Università di Pisa, Fraunhofer (Germany), INRIA (France), ATOS (France), E4 Computer Engineering (Italy), BSC (Spain), PSNC (Poland), INFN (Italy), CNR (Italy), In Quattro (Italy), Université de Bordeaux (France), CINECA (Italy) and Universitat Politècnica de Catalunya (UPC). The three Italian universities are part of the lab HPC: hpc-key-technologies-and-tools of CINI, created in 2021, that is grouping together the main academic and research entities working in the field of highperformance and exascale computing in Italy. More information on the activities carried out during the execution of TEXTAROSSA can be found in the project website 4 .

Figure 3 .

 3 Figure 3. The TEXTAROSSA key technologies

Acknowledgements

This work is supported by the TEXTAROSSA project G.A. n.956831, as part of the EuroHPC initiative.


Samet Gelincik

Michèle Wigger

Ligong Wang

DoF in sectored Cellular Systems with BS Cooperation Under a Complexity Constraint

The paper presents upper and lower bounds on the per-user degrees of freedom (DoF) of a sectored hexagonal cellular model where neighboring basestations (BS) can cooperate during at most κ interaction rounds over backhaul links of capacities µ = µ DoF • 1 2 log P , with P denoting the transmit power at each mobile user. The lower bound is based on practically implementable beamforming and adapts the way BSs cooperate to the sector structure of the cells. It improves over the naive approach that ignores this sector structure. The upper bound is information-theoretic and holds for all possible coding schemes, including for example ergodic interference alignment whose practical implementation currently seems out of reach. Lower and upper bounds show that the complexity constraint imposed by limiting the number of interaction rounds κ indeed limits the largest achievable DoF. In particular, irrespective of the backhaul capacity µ, the per-user DoF cannot exceed a threshold which depends on κ.

I. INTRODUCTION

This paper considers the uplink of a hexagonal cellular network with multiple-antenna mobile users and multipleantenna basestations (BS). BSs are equipped with directional antennas, which allow them to create intracell sectors whose communications do not interfere. Following current standards, each cell is divided into three sectors, and each BS has the same number of antennas M directed into any of the cell's sectors. The main quantity of interest in this paper is the degrees of freedom (DoF) per user. The DoF of any generic [START_REF] Jafar | Interference alignment a new look at signal dimensions in a communication network[END_REF] interference network with M -antenna transmitters and receivers is M/2 [START_REF] Jafar | Interference alignment a new look at signal dimensions in a communication network[END_REF]. Without cooperation, one needs interference alignment techniques, which are currently not implementable in practical systems, to achieve this DoF. When the BSs can cooperate over backhaul links, DoF M/2 can be achieved also with more practical coding techniques such as one-shot interference alignment [START_REF] Jafar | Interference alignment a new look at signal dimensions in a communication network[END_REF] combined with successive interference cancellation [START_REF] Ntranos | Cellular interference alignment[END_REF]. In fact, by means of backhaul cooperation, receiving BSs can pass their decoded messages to neighboring BSs according to a predefined decoding order of the BSs. The problem with this proposed successive interference cancellation scheme is its excessively long decoding delays stemming from the fact that BSs have to wait to decode their messages until all the preceding BSs have finished their decoding. To avoid such long delays, in this paper we restrict the number of interaction rounds between BSs.

Cooperation in communication networks was first considered in [START_REF] Willems | The discrete memoryless multiple access channel with partially cooperating encoders (corresp.)[END_REF], and subsequently in many other works such as [START_REF] Ntranos | Cellular interference alignment[END_REF]- [START_REF] Wigger | Conferencing in Wyner's asymmetric interference network: Effect of number of rounds[END_REF]. Previous results such as [START_REF] Willems | The discrete memoryless multiple access channel with partially cooperating encoders (corresp.)[END_REF] show that for small networks, it suffices that cooperation takes place over a small number of interaction rounds. This is not the case for large networks, where performance generally improves with increasing number of interaction rounds [START_REF] Wigger | Conferencing in Wyner's asymmetric interference network: Effect of number of rounds[END_REF].

The main interest of this paper is to understand the largest DoF that is achievable on the uplink when BS cooperation is limited in capacity and in the number of interaction rounds. For a lower bound, we propose a coding scheme that deactivates some of the mobile users in a way that the network decomposes into subnets, and then uses the backhaul links between BSs to collect quantized versions of the received signals within such a subnet at a single BS. This BS jointly decodes all the subnet's receive signals and distributes the decoded messages over the backhaul links to their intended BSs. The number of maximum allowed BS-interaction rounds κ limits the size of the subnets, because each receive antenna in the subnet needs to be reachable in κ/2 hops from one given BS to ensure that this BS can learn all the receive signals in the subnet and inform all the subnet's BSs about their decoded messages. The decoding in our scheme is thus reminiscent of clustered decoding as performed in [START_REF] Katz | On layered transmission in clustered cooperative cellular architectures[END_REF], [START_REF] Kim | Backhaul-constrained multi-cell cooperation using compressive sensing and spectral clustering[END_REF].

The new lower bound shows that DoF M/2 is achievable in the sectored hexagonal cellular model without interference alignment and with only two BS-interaction rounds and a backhaul DoF of M/6.

The upper bound presented in this paper is informationtheoretic and allows for any coding scheme. The bound shows that for small backhaul capacities, the DoF of the proposed scheme is close to optimal, and that, irrespective of the capacities of the backhaul links, for finite κ, the DoF is strictly smaller than M .

II. PROBLEM DEFINITION

A. Network Model

Consider the uplink communication of a cellular network consisting of N 1 hexagonal cells as depicted in Figure 1. Each cell contains a BS equipped with 3M directional receive antennas, with M antennas covering each of the three sectors of the cell. Usage of directional antennas, where side lobe radiation patterns are negligible, implies that communications in the three sectors of a cell do not interfere with each other. This leads to the interference graph in Figure 1, where each sector is depicted by a small circle, and interfering sectors in different cells are connected by solid black lines. (Noninterfering sectors in the same cell are unconnected.) Fig. 1: sectored cellular model. Magenta hexagonal regions depict the various cells, and the blue lines depict sectors. Each circle depicts a mobile user that is equipped with M transmit antennas. Each cell is associated to a BS such that each of the three sectors has a set of M receive antennas directed to that sector.

Mobile users in a sector perform orthogonal multiple-access as is typical for current 4G networks, and we thus restrict our model to a single mobile user per sector. For simplicity and symmetry, we assume that each mobile user is equipped with M transmit antennas. The uplink communication is then modeled by the following discrete-time input-output relation:

Y u,t = H u,u x u,t + υ : sector υ interferes sector u H u,υ x υ,t + Z u,t , t ∈ {1, . . . , n}, (1) where
• u ∈ {1, . . . , 3N } denotes the index of a given cell-sector; • x u,t denotes the real-valued 1 M -dimensional time-t signal sent by mobile user u; • Y u,t denotes the real-valued M -dimensional time-t signal received at the M BS receive antennas directed to sector u; • Z u,t denotes the real-valued M -dimensional i.i.d. standard Gaussian noise vector corrupting the time-t signal at user u; it is independent of all other noise vectors; • and H u ,u denotes a real-valued M -by-M random matrix with entries that are independently drawn according to a standard Gaussian distribution, which models the channel from mobile user u to the receive-antennas directed to sector u . 1 Setup and results are easily generalized to complex-valued inputs and outputs.

Channel matrices are randomly drawn but assumed constant over the n channel uses employed for the transmission of a message. In other words, the blocklength of a transmission is assumed shorter than the coherence time of the channel.

B. Communication Model with Backhaul Cooperation

Consider the uplink communication over the cellular network described in the previous section where each mobile user u ∈ {1, . . . , 3N } wishes to send an independent message W u , that is uniformly distributed over {1, . . . , 2 nRu }, to the BS in its cell. Communication takes place in two phases. In the first phase, each mobile user u applies an encoding function f (n) u : {1, . . . , 2 nRu } → R M ×n to its message W u and sends the n resulting vectors

X u := (X u,1 , . . . , X u,n ) = f (n) u (W u )
as its inputs over the network. Each encoding function f

(n) u
has to be chosen so that the input vectors satisfy an average input power constraint P :

1 n n t=1 X u,t 2 ≤ P with probability 1. (2)
The second communication phase takes place after each receiving BS j ∈ {1, . . . , N } has observed all outputs at the antennas in each of the three sectors of the cell. It is over the backhaul links connecting adjacent BSs. The backhaul links are assumed to be noise-free but rate-limited, and can be interactive. However, to limit complexity and latency of the backhaul communications, we restrict the interaction to only κ rounds, for a given positive integer κ.

For each j ∈ {1, . . . , N }, let Y j denote all the wireless signals observed at the 3M antennas of BS j:

Y j := Y uj,1,1 , . . . , Y uj,1,n , Y uj,2,1 , . . . , Y uj,2,n , Y uj,3,1 , . . . , Y uj,3,n , (3)
where u j,1 , u j,2 and u j,3 denote the three sectors in sector j.

In the kth cooperation round, k ∈ {1, . . . , κ}, each BS j ∈ {1, . . . , N } calculates the message V k j→i that it sends to a neighboring BS i as:

V k j→i = ψ k,(n) j→i Y j , V k-1 to j , (4)
for some cooperation functions ψ k,(n) j→i on appropriate domains. Here, V k-1 to j denotes the 6•(k-1) cooperation messages BS j has received from its 6 neighboring BSs during the first k -1 cooperation rounds:

V k-1 to j := V k i →j : BS i adjacent to BS j and k = 1, . . . , k -1 . ( 5
)
The cooperation functions have to be chosen in such a way that, for each pair of neighboring BSs (i, j) ∈ {1, . . . , N } 2 , the total information sent over the backhaul link from BS j to BS i is of rate not exceeding a given µ > 0:

1 n κ k=1 H V k j→i ≤ µ. (6)
Once the BS-cooperation phase is terminated, each BS j ∈ {1, . . . , N } proceeds to decode the three messages sent by the mobiles in the three sectors of its cell. To this end, for each sector u in its cell, it applies a decoding function h u on corresponding domains to produce the message estimates

Ŵu := h u Y j , V κ to j . (7)
An error occurs in the communication unless, for every sector u ∈ {1, . . . , 3N }:

Ŵu = W u . (8)

C. Capacity and Degrees of Freedom

A rate-tuple (R 1 , . . . , R 3N ) is said achievable, if for every > 0 and sufficiently large blocklengths n, there exist encoding, cooperation, and decoding functions {f

(n) u }, {ψ k,(n) j,i }, and {h (n) j }, such that Pr 3N u=1 Ŵu = W u ≤ . (9)
The capacity region C (P, µ, κ) is defined as the closure of the set of all achievable rate-tuples, and the sum capacity C Σ (P, µ, κ) as

C Σ (P, µ, κ) := sup (R1,...,R 3N ) ∈C(P,µ,κ) u∈{1,...,3N } R u .
In the high signal-to-noise ratio (SNR) regime, the quantity of interest is the degrees of freedom (DoF), which is defined as:

DoF (µ DoF , κ) := lim N →∞ lim P →∞ C Σ P, µ DoF • 1 2 log P, κ 3N • 1 2 log P .
Notice that here the backhaul cooperation rate µ scales with the power constraint P , and thus with the sum-capacity C Σ .

III. UPLINK SCHEME WITH COOPERATIVE BSS

The scheme has a parameter t, which is a positive integer. Choose a set of master BSs to form a regular grid of equilateral triangles where the three master cells forming a specific triangle lie 2t cell-hops away from each other. As we shall see, the proposed scheme takes place over 2t cooperation rounds, and thus one is allowed to choose any t ∈ 1, . . . , κ 2 .

For simplicity, we first present the scheme in the special case where t = 2. The general case is treated later.

A. Special case where t = 2

Consider the network in Figure 2, where the magenta cells indicate the master cells. We refer to cells that are adjacent to master cells as layer-1 cells, and the cells that are adjacent to layer-1 cells and that are not master cells as layer-2 cells. According to the way we choose the master cells, every cell is either a master cell, a layer-1 cell or a layer-2 cell.

Encoding: The mobile users in the network that are not shown in Figure 2 remain silent and do not transmit anything. Each of the remaining mobile users u transmits its message Fig. 2: Illustration of the network partition applied in our coding scheme for parameter t = 2. Magenta cells denote master cells and green hexagonal regions illustrate the decoding regions. All messages sent within a green hexagonal region are decoded by the BS in the magenta cell at the center of that region.

M u using an independent Gaussian codebook of power slightly less than P .

Cooperation between BSs: Cooperation takes place over 2t = 4 rounds as follows:

• Round 1: Each layer-2 BS applies an independent Gaussian vector quantizer to the M -dimensional receive signal of each of its sectors that contains an active mobile user.

The quantizers are chosen to be of rate

R q = M 2 log(P ). ( 10
)
Each layer-2 BS then sends the quantization message obtained for a given sector to one of the adjacent layer-1 BSs that lie in the same green hexagonal region as this sector, see Figure 2. Each layer-1 BS applies a rate-R q Gaussian vector quantizer to each of its own three M -dimensional receive signals, and sends the three quantization messages to its adjacent master BS. In Rounds 3 and 4, messages decoded at the master BS but intended for layer-1 or layer-2 BSs are sent over the backhaul links to the intended BSs. Specifically, messages for layer-1 BSs are directly passed from the master BSs to their intended BSs in Round 3. Messages for layer-2 BSs are first passed to adjacent layer-1 BSs in Round 3, and then to the intended layer-2 BSs in Round 4. Decoding: Each master BS declares its own message that it has produced during the decoding steps preceding Round 3. Each layer-1 BS declares the messages it has received in Round 3 and that were intended for it. Each layer-2 BS declares the messages that it has received in Round 4.

Analysis: Our interest is in the high-SNR asymptotics, i.e., in the DoF. Choosing the quantization rate R q as in (10) implies that each received signal is essentially quantized at the noise level, and thus allows us to achieve the same DoF as if the master BS had direct access to all the 30 receive signals in its decoding region. Since the channel matrix corresponding to the active sectors in a single green hexagonal region is of full rank with probability one, each of these 30 transmitted messages can be decoded with small probability of error at a DoF of M per active user. Notice now that for large networks (N → ∞) 5 6 of the mobile users are active, and thus the scheme achieves

DoF scheme = 5M 6 . (11)
We analyze the backhaul load for a single green hexagonal region. In Round 1, layer-2 BSs send in total 9 quantization messages, each of M DoF, to layer-1 BSs, which in turn relay these messages to their adjacent master BS in Round 2. Additionally, in Round 1, layer-1 BSs send in total 18 quantization messages, each of M DoF, to their adjacent master BSs. In Round 3, the master BS sends 27 decoded messages, each of DoF M , to the 6 adjacent layer-1 BSs. Of these 27 decoded messages, 9 are forwarded to adjacent layer-2 BSs in Round 4. Hence in total 54 cooperation messages of DoF M are sent in each green hexagonal region.

The backhaul load in our scheme is unevenly distributed among backhaul links. To balance this load and to achieve fairness among the rates achieved by the various mobile users, instances of the presented scheme are time-shared, such that in different instances different cells play the role of master cells. We are then interested in the average backhaul DoF used by our scheme. To calculate this average backhaul DoF, notice that each green hexagonal region is associated with a single master cell, and a cluster of 11 neighboring cells is associated with each of the master cells. The average backhaul load per cooperation link is then

µ DoF = 54M 6 • (11 + 1) = 3M 4 , ( 12
)
where the factor 6 accounts for the fact that each cell has 6 outgoing cooperation links.

B. General case

Choose the master cells so that they form a regular pattern of equilateral triangles where the master cells building these Fig. 3: Illustration of hexagon subnets for t = 3. Pink cells, red cells and blue cells represent master cells, corner cells and null cells, respectively. Green-filled circles refer to silenced users. triangles are 2t cell-hops away from each other. (See Figure 3 for a choice of magenta master cells when t = 3.) For = 1, . . . , t, we call layercells the cells that are at distance from the closest master cell. The distance between two cells here is defined as the minimum number of cell-hops needed to get from one cell to the other.

Encoding: Some of the layer-t mobile users are deactivated and do not transmit any message at all. All other mobile users are active, and send their messages using an independent Gaussian codebook of power P .

To describe the mobile users that are deactivated, we introduce the following denomination. We call the cells that are located t hops above the master cells the corner cells and the cells t hops below the master cells the null cells. We silence all users of the null cells and none of the corner cells. For all layer-t cells that are neither corner nor null cells, we deactivate the single mobile users that is closest to a corner cell. We conclude that among all the layer-t cells that surround a master cell, 3 of them have only active users, 3 have only deactivated users, and the remaining 6(t -1) cells have 2 active users. See Figure 3 for an example.

Cooperation between BSs: Cooperation takes place over 2t rounds.

• Rounds 1 to t: Each BS that is not a master BS applies a rate-R q Gaussian vector quantizer to each of its Mdimensional receive signals that corresponds to a sector with an active mobile user. For = 1, . . . , t -1, each layer-BS sends its 3 quantization messages to the closest master BS by means of multi-hop communication over rounds. Each layer-t BS sends each of its produced quantization messages to the master BS that is closest to the sector of each of the quantized signals. This transmission is again multi-hop over t rounds.

• Rounds t + 1 to 2t: Each master BS reconstructs the M -dimensional quantized signals corresponding to the quantization messages received in Rounds 1 to t. It then decodes the messages sent by the mobile users in the sectors corresponding to these quantization messages as well as the messages sent by the mobile users in its own cell. Finally, during cooperation rounds t + 1, . . . , 2t, it communicates each of the decoded messages using multihop communication to its intended BS. Decoding: Each master BS directly declares the messages guessed for its own cell before Round t + 1. The other BSs declare the messages that the master BSs have guessed for their cells and sent to them during Rounds t + 1, . . . , 2t.

Analysis: As before, the quantization rate R q is chosen at noise level, see (10). This allows us to achieve the same DoF as if the master BSs had direct access to all the receive signals for which they receive quantization information. Since there are 6 layer-cells around each master cell, the number of quantization messages received by each master BS is

3• t-1 =1
6 +(6t-3) = 9t(t-1)+6t-3 = 3(3t 2 -t-1). (13) Thus, including the 3 messages of the mobile users in its own cell, each master BS decodes 3t(3t -1) messages, each with a DoF equal to M .

For N 1, a cell partitioning can be obtained such that each master BS is associated with a cluster that includes all the layer-1, layer-2, . . ., layer-(t -1) cells that are closest to this master cell; among the layer-t cells, six are respectively shared between three clusters, while the rest are shared between two clusters, thus a total of (3t-1) layer-t cells should be counted for each cluster. Including the master cell itself, the number of cells counted in such a cluster is thus

1 + t-1 =1 6 + (3t -1) = 3t 2 . ( 14
)
The scheme thus achieves a DoF of

DoF scheme = M • 3t(3t -1) 3 • 3t 2 = (3t -1)M 3t . (15)
We next analyze the backhaul load that is associated with a single master cell. During Rounds 1 to t, a master BS receives a quantization message of DoF M from each sector that has an active user and that is closer to this master BS than to any other master BS. Since the quantization message for a sector in layer-is communicated over backhaul links, the total backhaul DoF used in these first t rounds is

3M t-1 =1 6 • + M (6t -3)t = 3M (t -1)t(2t -1) + (2t -1)t = 3M t 2 (2t -1). (16)
During Rounds t+1 to 2t, a decoded message of DoF M is communicated from the master BS to the corresponding BS for each quantization message that has been sent during Rounds 1 to t. Therefore, the total backhaul load during Rounds t + 1 to 2t is equal to that during Rounds 1 to t. The total backhaul load over all rounds is thus

6M t 2 (2t -1). (17)
To balance the backhaul load and to achieve fairness among the rates achieved by the various mobile users, the presented scheme is time-shared such that in different instances different cells play the role of the master cells. Under such a timesharing framework, the average backhaul DoF is the limiting quantity. The average backhaul load is obtained by dividing (17) by the number of outgoing (or incoming) backhaul links in a cluster, which by ( 14) is 6 • 3t 2 :

µ DoF = 6M t 2 (2t -1) 6 • 3t 2 = (2t -1)M 3 . (18)
C. Tradeoff Between DoF and Backhaul Load Time-sharing the described scheme for different values of parameter t yields the following result. Define µ DoF,0 = 0 and DoF 0 = M 3 , and for each t ∈ {1, . . . , κ 2 }:

µ DoF,t := (2t -1)M 3 and DoF t := (3t -1)M 3t . (19)
Theorem 1 (Achievability): Let f κ (•) be the mapping from µ DoF to DoF defined by the upper hull over the set

µ DoF,t , DoF t : t = 0, . . . , κ 2 , then DoF (µ DoF , κ) ≥ f κ (µ DoF ). (20)
Figure 4 illustrates this lower bound for different values of κ. Notice that even with only two conferencing rounds, a DoF of M/2 is achievable with our scheme with a backhaul DoF of only µ DoF = 0.5.

A naive alternative to our scheme would be to ignore the interference pattern caused by the sector structure of the cells and to simply deactivate all mobile users in layer-t cells. Compared to our scheme this naive approach reduces the DoF from M 1-1 3t to M 1-3t+2 3(t+1) 2 , and it reduces the number of required cooperation rounds from 2t to 2(t-1) because the first and the last cooperation rounds of our protocol become useless when all layer-t users are deactivated.

Define µ naive,0 = 0 and DoF naive,0 = M 3 , and for each t ∈ {1, . . . , κ 2 }:

µ naive,t := t(2t + 1)M 3(t + 1) (21) DoF naive,t := M 1 - (3t + 2) 3(t + 1) 2 . ( 22
)
Let g κ (•) be the mapping defined by the upper hull over the set µ naive,t , DoF naive,t : t = 0, . . . , κ 2 , then the DoF of this naive approach is given by

DoF naive = g κ (µ naive,t ). (23)
We now present an upper bound on the DoF. Theorem 2 (An information-theoretic converse): We have:

DoF (µ DoF , κ) ≤ min M 2 + 2 3 µ DoF , M 1 - 1 2(1 + κ + κ 2 ) . (24)
Proof: See the extended version [START_REF] Gelincik | Benefits of local cooperation in sectorized cellular models[END_REF]. Remark that for any finite κ, DoF (µ DoF , κ) < M , irrespective of the backhaul capacity µ DoF .

Figure 5 compares the new lower bound in Theorem 1 with the lower bound achieved by the naive approach. The figure also shows the information-theoretic upper bound in Theorem 2. Notice that the lower bounds on the DoF are based on practically implementable coding schemes. In particular, interference alignment is not used, even though it is known to achieve a DoF of M/2 over a generic network in the absence of BS cooperation. In contrast, the information-theoretic upper bound holds for any possible coding scheme. It is thus natural that there is a gap between the presented upper and lower bounds.

IV. CONCLUDING REMARKS

The paper presents a lower bound on the DoF of the uplink of a sectored cellular network with backhaul cooperation between BSs. This DoF is characterized as a function of the backhaul capacity, and also of the maximum number of allowed cooperation rounds. The presented results show that limiting the number of cooperation rounds also limits the DoF achievable over the network. In particular, to get a DoF of M , an infinite number of cooperation rounds is required as the number of cells in the network grows. If the number of cooperation rounds does not grow with the number of cells but is bounded, then also the DoF is bounded away from M , even in the idealistic case of unlimited backhaul cooperation capacity.

The obtained lower bound on the DoF is based on a practically implementable scheme. Obtaining dual results for the downlink is straightforward. Finite-SNR results and information-theoretic converses are derived in [START_REF] Gelincik | Benefits of local cooperation in sectorized cellular models[END_REF].

• Round 2 :

 2 The layer-1 BSs forward the Round-1 messages they received from layer-2 BSs to their adjacent master BSs.• Rounds 3 and 4: Each master BS uses the quantization messages received in Rounds 1 and 2 to reconstruct quantized versions of the 27 M -dimensional received signals observed in the sectors lying in the same green hexagonal region as the master BS itself. Using these 27 quantized M -dimensional signals and its own 3 Mdimensional receive signals, each master BS decodes all the messages sent by the 30 active mobile users in this green hexagonal region.

Fig. 4 :

 4 Fig. 4: Lower bound on the DoF in Theorem 1 as a function of µ DoF when M = 3 and for different values of κ.

Fig. 5 :

 5 Fig. 5: Lower and upper bounds on the DoF as a function of µ DoF for M = 3 and κ = 8.

ACKNOWLEDGEMENT

M. Wigger was supported by a Huawei Disruptive ICT Grant.