Yoann Desmouceaux

Mohsin Augustin

Victor Kazmi

Pierre Nguyen

Wenqin Pfister

Nathan Shao

Skrzypczak

Jordan Augé

Giovanna Carofiglio

Enzo Fenoglio

Alain Fiocco

Benoît Ganne

Guillaume Ladhuie

Malycia Ly

Luca Mus- Cariello

Guillaume Sauvage De Saint

Hassen Siad

André Surcouf

Axel Taldir

Éric Vyncke

Zacharie Brodard

Basile Bruneau

Clément Fischer

Clément Durand

Pierre-Jean Grenier

Maxime Larcher

Samuel Perez

Thomas Feltin

Benoît Pit--Claudel

Andi Sai

Arthur Toussaint

Zhiyuan Yao

Clemens Kunst

Alexandre Poirrier

Guillaume Solignac

Benedikt Tegethoff

Merci Aussi

Bernadette Charron-Bost

Thomas Clausen

Juan-Antonio Cordero-Fuertes

Marcel Enguehard

Victor Nguyen

Pierre Pfister

Wenqin Shao

Jérôme Tollet

Sonia Toubaline

Mark Townsley

Network-Layer Protocols for Data Center Scalability

Keywords: Data-center networking, Task Mobility, Multicast, Load Balancing, Segment Routing

With the development of demand for computing resources, data center architectures are growing both in scale and in complexity. In this context, this thesis takes a step back as compared to traditional network approaches, and shows that providing generic primitives directly within the network layer is a great way to improve efficiency of resource usage, and decrease network traffic and management overhead. Using two advanced network architectures, Segment Routing (SR) and Bit-Indexed Explicit Replication (BIER), network layer protocols are designed and analyzed to provide three high-level functions: (1) task mobility, (2) reliable content distribution and (3) load-balancing.

First, task mobility is achieved by using SR to provide a zero-loss virtual machine migration service. This then opens the opportunity for studying how to orchestrate task placement and migration while aiming at (i) maximizing the inter-task throughput, while (ii) maximizing the number of newly-placed tasks, but (iii) minimizing the number of tasks to be migrated. Second, reliable content distribution is achieved by using BIER to provide a reliable multicast protocol, in which retransmissions of lost packets are targeted towards the precise set of destinations having missed that packet, thus incurring a minimal traffic overhead. To decrease the load on the source link, this is then extended to enable retransmissions by local peers from the same group, with SR as a helper to find a suitable retransmission candidate. Third, load-balancing is achieved by way of using SR to distribute queries through several application candidates, each of which taking local decisions as to whether to accept those, thus achieving better fairness as compared to centralized approaches. The feasibility of hardware implementation of this approach is investigated, and a solution using covert channels to transparently convey information to the load-balancer is implemented for a state-of-the-art programmable network card. Finally, the possibility of providing autoscaling as a network service is investigated: by letting queries go through a fixed chain of applications using SR, autoscaling is triggered by the last instance, depending on its local state.

v Résumé Du fait de la croissance de la demande en ressources de calcul, les architectures de centres de données gagnent en taille et complexité. Dès lors, cette thèse prend du recul par rapport aux architectures réseaux traditionnelles, et montre que fournir des primitives génériques directement à la couche réseau permet d'améliorer l'utilisation des ressources, et de diminuer le trafic réseau et le surcoût administratif. Deux architectures réseaux avancées, Segment Routing (SR) et Bit-Indexed Explicit Replication (BIER), sont utilisées pour construire et analyser des protocoles de couche réseau, afin de fournir trois primitives : (1) mobilité des tâches, (2) distribution fiable de contenu, et (3) équilibre de charge.

Premièrement, pour la mobilité des tâches, SR est utilisé pour fournir un service de migration de machine virtuelles sans perte. Cela ouvre l'opportunité d'étudier comment orchestrer le placement et la migration de tâches afin de (i) maximiser le débit inter-tâches, tout en (ii) maximisant le nombre de nouvelles tâches placées, mais (iii) minimisant le nombre de tâches migrées. Deuxièmement, pour la distribution fiable de contenu, BIER est utilisé pour fournir un protocole de multicast fiable, dans lequel les retransmissions de paquets perdus sont cibleés vers l'ensemble précis de destinations n'ayant pas reçu ce paquet : ainsi, le surcoût de trafic est minimisé. Pour diminuer la charge sur la source, cette approche est étendue en rendant possibles des retransmissions par des pairs locaux, utilisant SR pour trouver un pair capable de retransmettre. Troisièmement, pour l'équilibre de charge, SR est utilisé pour distribuer des requêtes à travers plusieurs applications candidates, chacune prenant une décision locale pour accepter ou non ces requêtes, fournissant ainsi une meilleure équité de répartition comparé aux approches centralisées. La faisabilité d'une implémentation matérielle de cette approche est étudiée, et une solution (utilisant des canaux cachés pour transporter de façon invisible de l'information vers l'équilibreur) est implémentée pour une carte réseau programmable de dernière génération. Finalement, la possibilité de fournir de l'équilibrage automatique comme service réseau est étudiée : en faisant passer (avec SR) des requêtes à travers une chaîne fixée d'applications, l'équilibrage est initié par la dernière instance, selon son état local.

Mots-clefs -

Introduction

The development of virtualization, where computers are emulated and/or sharing an isolated portion of the hardware by way of Virtual Machines (VMs) [1], or run as isolated entities (containers [2]) within the same operating system kernel, has accelerated the development of microservices [3] and led to the commoditization of compute resources 1 . Thus, data centers are becoming "miniature" Internets, running on top of specialized network architectures, and possibly relying on overlay technologies to provide transparent services to the hosted applications [4]. Given this, architecting network paradigms suited for data center environments is a complex challenge, both from (i) an operational [5] and (ii) an energy consumption perspective [6] -due to (i) the complexity of orchestration and the richness of available architectures, and (ii) the need to minimize power consumption while maintaining quality of service.

In this context, the question explored in this thesis is whether, and how, it is possible to provide certain generic data-center primitives (task migration, reliable content distribution, load balancing, auto-scaling) as network services, thereby allowing for optimizing resources while incurring low operational complexity. This is accomplished by studying how certain standardized network frameworks [7,8] can be extended to express those primitives directly at the network layer, while operating transparently to the applications.

The remainder of this introductory chapter is structured as follows. Section 1.1 introduces and reviews traditional data center architectures, from network protocols through topologies, and to management frameworks. Then, section 1.2 introduces two network paradigms, Segment Routing [7] and Bit-Indexed Explicit Replication [8], which enrich network layers by allowing sources to include further expressiveness in emitted packets, than solely source and destinations. Finally, section 1.3 argues for the usage of these new paradigms to augment data center architectures, by providing network protocols that are transparent to the applications, while naturally providing (i) task mobility, (ii) reliable content distribution, and (iii) load-balancing and autoscaling. The chapter is concluded by section 1.4, which provides generic assumptions, definitions and notations that will be used throughout this thesis.

Background

This section introduces background on data centers: section 1.1.1 introduces traditional network protocols used in the Internet and in data centers, and section 1.1.2 reviews data center topologies and architectures.

Network Protocols

Network protocols underlying the Internet are traditionally presented as belonging to several layers. Each of these layers (layer n) provides service to the layer above (layer n + 1) and uses service from the layer below (layer n -1), and can transmit messages to and receive messages from those adjacent layers, making network processing effectively separated between independent Several protocols exist to provide the functions of each of these layers. As a typical protocol stack for traditional (and, to a certain extent, legacy) data center networks, Ethernet [11] is used as Layer-2 protocol, typically to connect physical machines belonging to a single rack2 . On top of this, the Internet Protocol (IP) [START_REF] Postel | Internet Protocol[END_REF][START_REF] Deering | Internet Protocol, Version 6 (IPv6) Specification[END_REF] is used as Layer-3 protocol, enabling end-to-end connections of hosts within the data center, and allowing interconnection to the Internet. In addition, routing protocols are usually run to build the tables of the routers connecting the different nodes of the data centers. For instance, the Border Gateway Protocol (BGP) [START_REF] Rekhter | A Border Gateway Protocol 4 (BGP-4)[END_REF], initially designed to enable routing between the different Autonomous Systems comprising the Internet, is one the possible routing protocols for data center environments [START_REF] Lapukhov | Use of BGP for Routing in Large-Scale Data Centers[END_REF]. Then, the Transmission Control Protocol (TCP) [START_REF] Postel | Transmission Control Protocol[END_REF] is usually used as Layer-4 protocol, allowing for reliable, ordered transmission of streams of data. Finally, one example of a Layer-7 protocol running on top of the data-center stack is the HyperText Transfer Protocol (HTTP) [START_REF] Nielsen | Hypertext Transfer Protocol -HTTP/1.1[END_REF] (or its secure version, HTTPS [START_REF] Rescorla | HTTP Over TLS[END_REF]). Initially designed for transfer of files for web browsing, the emergence of REpresentational State Transfer (REST) APIs [START_REF] Fielding | Architectural styles and the design of network-based software architectures[END_REF] makes HTTP also used as a data manipulation and retrieval protocol.

Although the above-mentioned protocol stack has been widely used across the Internet and in data center premises, it was designed for an "idealized" network, with the end-to-end principle [START_REF] Saltzer | End-to-end arguments in system design[END_REF] in mind. According to this principle, network functions (reachability, encryption, reliability, etc.) should, when possible, be fulfilled by the communicating endpoints of a logical transmission. In such an idealized network, each host is equally reachable and can equally perform "client" or "server" functions. However, this principle may not always be true, for multiple reasons. First, IPv4 address exhaustion3 makes the use of private addressing schemes and of Network Address Translation (NAT) devices to translate addresses at the ingress and egress of these private networks prevalent. Second, the progressive shift to a consumer-producer paradigm [START_REF] Zhang | Named data networking[END_REF], where end users often want to retrieve content, and data center facilities are often used to serve content, imposes an asymmetry between two types of hosts. Third, data centers (and especially public clouds) are often configured to run multi-tenancy scenarios [START_REF] Huang | A framework for native multitenancy application development and management[END_REF], wherein multiple virtual sub-facilities -each intended to belong to one tenant 4 -are sharing the same physical resources (physical machines and physical network).

For these reasons, data center networks usually rely on abstractions to provide multiple virtual flat networks, hiding the complexity of their physical network architecture to provide a consistent and isolated network to each of the multiple tenants [4]. Each tenant runs a set of VMs and is, for example, presented with a virtual Layer-2 network so that, in a way, the end-to-end principle is restored within this network. According to [4], the simplest way to implement these virtual networks in data center environments is by using Virtual Local Area Networks (VLANs) [START_REF]IEEE Standard for Local and metropolitan area networks -Bridges and Bridged Networks[END_REF]: each Ethernet packet carries a 12-bit VLAN identifier, and switches forward packets only between those physical ports which are configured to belong to the same VLAN. However, due to the restricted corresponding identifier space (only allowing for 2 12 = 4096 tenants), and the necessity for tenant networks to be included in a single physical Layer-2 network, other protocols have been developed that allow for more flexibility -at the cost of operational complexity. Among those, Virtual eXtensible Local Area Networks (VXLANs) [START_REF] Mahalingam | Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks[END_REF] allow for virtual Layer-2 network to span arbitrary topologies, by encapsulating Layer-2 frames within a UDP/IP packet containing a special header enabling a network to be divided among up to 2 24 ≈ 16.8 • 10 6 tenants. With VXLAN encapsulation, packets can be transmitted within one data center (or across several data centers) by using the underlying routing infrastructure. Another protocol which can be used for data center multitenancy is Network Virtualization using Generic Routing Encapsulation (NVGRE) [START_REF] Garg | NVGRE: Network Virtualization Using Generic Routing Encapsulation[END_REF], which encapsulates Layer-2 frames within GRE packets, and also allows for up to 2 24 tenants. Finally, inter-data-center multi-tenancy can be achieved by way of Virtual Private Network (VPN) protocols such as Virtual Private LAN Service (VPLS) [START_REF] Rekhter | Virtual Private LAN Service (VPLS) Using BGP for Auto-Discovery and Signaling[END_REF] or Ethernet VPN (EVPN) [START_REF] Drake | BGP MPLS-Based Ethernet VPN[END_REF]. With these protocols, Layer-2 frames belonging to a single tenant are transmitted across a Multi-Protocol Label Switching (MPLS) [START_REF] Viswanathan | Multiprotocol Label Switching Architecture[END_REF] capable network shared between two data center premises, using MPLS labels to distinguish between tenants.

Along with the virtualization of compute resources, network resources are also often virtualized in order to provide the encapsulation services mentioned above. Virtual routers, such as Open VSwitch (OVS) [START_REF] Pfaff | The design and implementation of Open VSwitch[END_REF] or the Vector Packet Processor (VPP) [START_REF] Barach | High-speed software data plane via vectorized packet processing[END_REF], are usually run as services in each physical machine, with virtual interfaces connected to each of the containers and/or VMs (possibly belonging to several tenants) hosted in that machine, and with physical interfaces connected to the physical infrastructure of the data center, as depicted in figure 1.1. These virtual routers then properly encapsulate or decapsulate traffic from and to the instances of the tenants, often using kernel-bypass techniques (e.g., DPDK [START_REF]Data Plane Development Kit (DPDK)[END_REF]) to provide high-performance packet processing. Other than connecting instances, such virtual routers can also be deployed (usually as virtual machines themselves) for more specific purposes, e.g., firewalling, intrusion detection, address translation, encryption, load-balancing, etc., thus providing Network Function Virtualization (NFV) [START_REF] Mijumbi | Network function virtualization: State-of-the-art and research challenges[END_REF].

As has been described above, these layers of virtualization and encapsulation allow to hide the complexity of the network infrastructure to tenants so that, within each sub-network, the endto-end principle can apply. However, this comes at the cost of operational complexity, as virtual routers must be deployed, and as the corresponding network configuration (identifier assignments, etc.) must be computed and distributed.

Data Center Architectures

According to [5,6,[START_REF] Al-Fares | A scalable, commodity data center network architecture[END_REF], one of the first data center architectures to have been used is the three-tiered architecture described in [START_REF]Cisco Data Center Infrastructure 2.5 Design Guide[END_REF] (figure 1.2). With this architecture, which is essentially a (potentially multi-rooted) tree, machines are grouped into racks connected to a Top-of-Rack (ToR) Figure 1.2 -Three-tiered data center network topology [START_REF]Cisco Data Center Infrastructure 2.5 Design Guide[END_REF] Figure 1.3 -Fat-tree data center network topology [START_REF] Al-Fares | A scalable, commodity data center network architecture[END_REF] with k = 4 router 5 . Each of these ToR routers is itself connected to one among several aggregation routersthus each aggregation router is the root of a subtree comprising several racks. Finally, aggregation routers are meshed to one or several core router(s), usually through links of higher capacity, thus allowing a host in any rack to reach any other host by crossing 1, 3 or 5 routers. These core routers can be connected to an egress router, which acts at the gateway of the data center. A drawback of this topology, as identified in [START_REF] Al-Fares | A scalable, commodity data center network architecture[END_REF], is that the links between core and aggregation routers act as bottlenecks, thus limiting the throughput achievable between two arbitrary hosts when many hosts communicate with other non-local hosts. Overcoming this limitations require increasing the number of high-throughput routers, and therefore incurs an often prohibitive hardware cost.

To address these drawbacks, an important amount of work has consisted of deriving topologies for data centers that are efficient in terms of throughput while minimizing hardware cost and environmental impact [5,6]. In [START_REF] Al-Fares | A scalable, commodity data center network architecture[END_REF], a data center architecture is presented, which aims at reducing the hardware cost of a throughput-efficient6 network, compared to three-tiered data centers. This is achieved by using a fat-tree topology [START_REF] Leiserson | Fat-trees: universal networks for hardware-efficient supercomputing[END_REF] (figure 1.3): with k being an even integer, k core routers are each connected to one router in each of k pods. Each of the k pods comprises two layers of k/2 routers: the upper (aggregation) layer is connected with the core layer, and the lower (ToR) layer is connected to hosts. Within a pod, the aggregation layer is meshed with the ToR layer. Finally, each ToR router is connected to k/2 hosts. This allows connecting k 3 /4 hosts with k(k + 1) routers while achieving maximum bissection bandwidth. However, the proposed architecture comes at an operational cost, as it imposes a custom addressing scheme, along with a hardware modification of the routing lookup mechanism to allow for prefix-and suffix-based routing. Furthermore, it requires a custom centralized flow scheduler in order to achieve full throughput.

VL2 [START_REF] Greenberg | VL2: a scalable and flexible data center network[END_REF] is another proposed hierarchical data center architecture, which uses a Clos topology [START_REF] Dally | Principles and practices of interconnection networks[END_REF] and provides separation of identifiers and locators through two separate address spaces and IP-in-IP encapsulation, while presenting a Layer-2 semantic to the whole overlay network. A centralized directory maintains a mapping between applications identifiers and locators of their ToR routers, and a shim layer running in each machine provides encapsulation services, while decapsulation is performed at the ToR routers. Source routing is used to balance traffic across (a) DCell [START_REF] Guo | DCell: a scalable and fault-tolerant network structure for data centers[END_REF] with n = 3 and ℓ = 1 (b) BCube [START_REF] Guo | BCube: a high performance, server-centric network architecture for modular data centers[END_REF] with n = 4 and ℓ = 1 Although the architectures in [START_REF] Al-Fares | A scalable, commodity data center network architecture[END_REF][START_REF] Greenberg | VL2: a scalable and flexible data center network[END_REF] possess good properties in terms of cost and achievable throughput, they remain hierarchically organized in three layers, which can pose scalability issues for very large data centers -for instance with [START_REF] Al-Fares | A scalable, commodity data center network architecture[END_REF], core routers need to have O(n 1/3 ) ports to build an n-machine data center. To address these scalability issues, recursive architectures have been introduced. Among those, DCell [START_REF] Guo | DCell: a scalable and fault-tolerant network structure for data centers[END_REF] is an architecture using routers with a fixed number of ports. Starting from a small building block of level 0 (a rack of n machines connected to an n-ports router), blocks of level ℓ + 1 are built by connecting s ℓ + 1 blocks of level ℓ, where s ℓ is the number of machines in a block of level ℓ, as depicted in figure 1.4a. This is achieved by realizing a full mesh between sub-blocks: two sub-blocks are connected by picking one machine into each of them, and connecting them to each other. Therefore, machines become parts of the routing infrastructure, with routers only providing rack connectivity. Thus, DCell uses a custom Layer-3 protocol, with a custom forwarding module in the physical machines: this has an operational cost and incurs additional resource (CPU) consumption. Another recursive topology is BCube [START_REF] Guo | BCube: a high performance, server-centric network architecture for modular data centers[END_REF], whose main difference to [START_REF] Guo | DCell: a scalable and fault-tolerant network structure for data centers[END_REF] is that it uses routers not only at the rack level, and provides multiple parallel paths between pairs of machines. With a building block (of level 0) consisting of n machines connected to an n-ports router, a block of level ℓ + 1 is built by connecting n blocks of level ℓ together, using n ℓ+1 routers, as depicted in figure 1.4b. Similarly to [START_REF] Guo | DCell: a scalable and fault-tolerant network structure for data centers[END_REF], machines are part of the forwarding infrastructure, and a custom source routing protocol is used to select a server-path between two nodes, with a custom module deployed in each machine.

Another research aspect related to data center architectures is that of traffic management. Indeed, it has been shown that data center traffic is bursty [START_REF] Benson | Network traffic characteristics of data centers in the wild[END_REF] -for instance, one of the data center studied in [START_REF] Benson | Network traffic characteristics of data centers in the wild[END_REF] exhibits a median flow inter-arrival time lower than 250 µs. Given this, architectures have been designed to properly balance traffic among data center links while handling asymmetry and reacting to congestion. For example, Hedera [START_REF] Al-Fares | Hedera: Dynamic flow scheduling for data center networks[END_REF] is a centralized architecture which aims at properly balance flows in order to provide near-optimal bissection bandwidth. It consists of a centralized scheduler, which periodically probes ToR routers and extracts characteristics of the larger flows. From this, it estimates the throughput demand of each flow (i.e., the throughput that the flow would be achieving if it was only limited by the interfaces of the sender and receiver), and uses simulate annealing to compute an almost-optimal placement of flows to satisfy these demands. Finally, corresponding rules are pushed to the routers using a Software Defined Network (SDN) controller [START_REF] Mckeown | OpenFlow: enabling innovation in campus networks[END_REF]. Similarly, inter-data-center centralized traffic management architectures have been proposed [START_REF] Jain | B4: Experience with a globally-deployed software defined WAN[END_REF][START_REF] Hong | Achieving high utilization with software-driven WAN[END_REF]. Another data-center traffic management framework is Conga [START_REF] Alizadeh | CONGA: Distributed congestion-aware load balancing for datacenters[END_REF], which acts at the "flowlet" level rather than making per-flow decisions. Flowlets are defined as bursts of packets sufficiently spaced in time so that they can be sent over different paths without causing packet reodering. With Conga, each intermediate router marks congestion information within a special VXLAN header, before reflecting this to the ToR router from which traffic originates. Each ToR router maintains a per-ToR-router congestion table, and makes traffic placement decisions accordingly. While providing reactive traffic balancing and global congestion knowledge, this architecture requires hardware modifications in the routers (to be able to mark congestion).

CHAPTER 1. INTRODUCTION

Next Header

Ext Hdr Length Routing Type = 4 Segments Left = 0, . . . , n Last Entry = n Flags Tag Segment 0 . . .

Segment n

Optional TLVs...

.5 -IPv6 Segment Routing Header [START_REF] Filsfils | IPv6 Segment Routing Header (SRH)[END_REF] Algorithm As described in sections 1.1.1 and 1.1.2, data center architectures have evolved to address scalability issues in terms of topology and traffic management. On the one hand, these evolutions often rely on complex overlays, specific routing schemes, and/or custom network protocol stacks. On the other hand, some protocols have been proposed in order to simplify network operation, by encoding instructions directly in the packets rather than provisioning ad-hoc rules in advance. Section 1.2 describes two such network architectures, and in section 1.3 will describe if and how they can be useful to solve data-center-specific issues.

Extending the Network Layer

This section introduces two network paradigms, Segment Routing (section 1.2.1) and Bit-Indexed Explicit Replication (section 1.2.2), which will be used throughout this thesis as foundational blocks to provide augmented data center services. A common point to these two architectures is that they rely on source-provided instructions embedded in a given network packet to influence the behavior of further devices (e.g., routers or hosts) that will process and/or forward that packet.

Segment Routing (SR)

Segment Routing (SR) [7] is a network architecture enabling source-routing capabilities in a controlled domain, and standardized by the Internet Engineering Task Force (IETF7 ) [START_REF] Previdi | Source Packet Routing in Networking (SPRING) Problem Statement and Requirements[END_REF]. SR is an architecture, for which there exists two data-plane instantiations or "flavors": (i) an MPLS flavor, where segments are encoded by way of MPLS labels [START_REF] Bashandy | Segment Routing with MPLS data plane[END_REF], and (ii) an IPv6 flavor [START_REF] Filsfils | IPv6 Segment Routing Header (SRH)[END_REF], where segments are encoded by way of IPv6 addresses. For the purpose of this manuscript, only the IPv6 flavour (IPv6 Segment Routing or SRv6) will be considered.

SRv6 allows a given packet to be forwarded along a set of "segments" encoded in the packet header, each of these segments being represented by an IPv6 address, and representing a specific function to be executed on the packet. This is done using an IPv6 extension header, defined in [START_REF] Filsfils | IPv6 Segment Routing Header (SRH)[END_REF] and depicted in figure 1.5. This header carries a list of segments, which the packet should traverse, with the last segment representing the ultimate destination address of the packet. Each segment is divided into a locator part (typically, the first 64 bits), and a function part (typically, the last 64 bits). The locator is a routable IPv6 prefix, meaning that the packet will be naturally directed to the next segment using the underlying routing infrastructure (possibly taking advantage of ECMP multipath routes), and that the packet can therefore naturally traverse non-SR-capable routers.

An SRv6 header is initialized with a list of segments (in reverse traversal order), with the last segment being the original destination address of the packet. Fields LastEntry and SegmentsLeft are initialized to the number of segments n, and the destination address of the packet rewritten to that of the segment at position n -i.e., the first segment in traversal order. As described in algorithm 1, upon receiving a packet, a simple SR endpoint will process the packet according to the function instructed in the segment identifier and, if applicable, will initiate forwarding the packet to the next segment. This is done by replacing the destination address of the packet by that of the next segment, and by decreasing the SegmentsLeft counter. Of course, if complex behavior is applied by the segment endpoint to the packet (e.g., adding a new header), the simple process described in algorithm 1 needs to be adapted accordingly -for instance, if inserting a new header, lines 6 → 8 of algorithm 1 might not be necessary. Some example functions that can be instantiated by segments are defined in [START_REF] Filsfils | SRv6 Network Programming[END_REF]. The simplest one is the "endpoint" function, denoted END, which essentially is a no-op and consists of simply forwarding the packet to the next segment. By simply using a sequence of END segments, source routing results, that is, a packet follows a source-specified path. Source routing was already defined as a component of IPv6 in its original specification [START_REF] Hinden | Internet Protocol, Version 6 (IPv6) Specification[END_REF] (later superseded by [START_REF] Deering | Internet Protocol, Version 6 (IPv6) Specification[END_REF]), but was later deprecated for security reasons [START_REF] Neville-Neil | Deprecation of Type 0 Routing Headers in IPv6[END_REF]. Indeed, unrestricted IPv6 source routing could lead to, among others, traffic amplification attacks and network topology exposures [START_REF] Biondi | IPv6 routing header security[END_REF].

To avoid reproducing these issues, Segment Routing introduces the concept of SR domain, i.e., a consistent and autonomous set of nodes controlled by the same administrative entity (e.g., a data center or an ISP network). Within a single domain, it is assumed that all nodes are trusted, in which case packets embedding an SRv6 header can be directly issued and received by hosts. For packets traversing a domain, SR defines the concept of ingress node. Upon entering an SR domain by reaching the ingress node, and depending on characteristics of the packets (source address, destination address, etc.), a packet will have an SRv6 header applied to it, containing a suitable segment list. The SRv6 header can either be inserted within the packet ("insert mode"), or a new outer IPv6 header can be prepended to the packet ("encap mode"), leaving the original packet intact. Upon reaching the last segment, the SRv6 header can be removed, so that it does not appear in the resulting packet, and so that the packet egresses the domain with its original form. As argued in [47, section 6], this does not have the security drawbacks of the original IPv6 source routing, provided that packets entering the domain and already containing segments pointing to inside the domain are discarded. In cases where SR packets are to traverse multiple domains, security can be achieved by the use of a Hash-based Message Authentication Code (HMAC), as described in [47, section 6.3.2].

Beyond the "endpoint" function, other modular functions are defined in [START_REF] Filsfils | SRv6 Network Programming[END_REF]. For instances, some functions allow for forwarding the packet according to non-default semantics -END.X allows forwarding along a specified interface, and END.T forwarding according to a lookup in a specific table. Other functions (END.DX2, END.DX2V, END.DT2U, END.DX, END.DT) allow decapsulating the SRv6 header and forwarding the resulting according to different lookup strategies (e.g., along a specified interface or according to a lookup in a specified table). Finally, some functions (END.B6, END.BM) allow insertion of new SR headers to a packet. The variety of these functions indicates how SR can be used for more complex applications than sole source routing.

Notational Note

The SRv6 header format as specified in [START_REF] Filsfils | IPv6 Segment Routing Header (SRH)[END_REF] and depicted in figure 1.5 contains segments in reverse traversal order: the first segment encoded in the header is the last traversed, the second encoded is the second-to-last traversed, etc. For ease of readability, however, throughout this CHAPTER 1. INTRODUCTION manuscript the inverse convention will be adopted. An SR list (s 1 , s 2 , . . . , s n ) will thus denote a segment list whose i th segment to be traversed is s i .

Related Work

SR has been used for different purposes, the first of which is traffic engineering. In [START_REF] Hartert | A declarative and expressive approach to control forwarding paths in carrier-grade networks[END_REF], traffic engineering goals (e.g., "the load on all links should not exceed 90% of their capacity) are considered and expressed in an almost natural (Scala-based) language. An algorithm is introduced to turn these requirements into a set of SR lists instantiating paths between endpoints. In [START_REF] Bhatia | Optimized network traffic engineering using segment routing[END_REF], a model of traffic engineering aiming at minimizing link utilization by directing flows through SR lists of length 2 is formulated, and offline and online resolution algorithms are proposed. In [START_REF] Gay | Expect the unexpected: Sub-second optimization for segment routing[END_REF], SR is used for sub-second traffic engineering. That is, links are continuously monitored, and when congestion (rather than link failure in traditional traffic engineering) is detected, new SR paths are computed to limit link usage. In [START_REF] Cianfrani | Incremental deployment of segment routing into an ISP network: a traffic engineering perspective[END_REF], an incremental deployment of SR within an ISP is considered, and an algorithm to minimize link utilization in a hybrid SR/IP network where the size of the SR domain gradually increases is proposed. Finally, in [START_REF] Lebrun | Software resolved networks: Rethinking enterprise networks with IPv6 segment routing[END_REF], an approach is proposed to provide traffic engineering within enterprise networks. A custom DNS resolver receives DNS queries augmented with traffic requirements, computes a suitable path and communicates it to the access router, before replying to the client with a corresponding "path ID". The client embeds this path ID as an END.B6 segment in outgoing packets; upon reaching the access router, this segment triggers insertion of the previously-provisioned path as a new SRv6 header.

Another interesting use case for SR is service function chaining. As introduced in [START_REF] Lebrun | Leveraging IPv6 segment routing for service function chaining[END_REF], it consists of specifying a list of Virtual Network Functions (VNFs) that should be applied to a packet as a set of SR segments. In [START_REF] Abdelsalam | Implementation of virtual network function chaining through segment routing in a Linux-based NFV infrastructure[END_REF], a Linux kernel module is presented, which allows forwarding SR packets belonging to a service chain, to VMs running SR-unaware VNFs. This is done by decapsulating and re-encapsulating packets, so that VNFs only are exposed to non-SR packets. In [START_REF] Abdelsalam | SERA: Segment routing aware firewall for service function chaining scenarios[END_REF], an SRaware VNF is presented, which allows native firewalling of SRv6 packets belonging to a service chain, by way of an extension of the Linux firewall iptables. In [START_REF] Duchêne | SRv6Pipes: enabling in-network bytestream functions[END_REF], the SRv6 service chaining abstraction is extended from packets to bytestreams. This is achieved by using a transparent proxy that terminates TCP connections and opens a new TCP stream between each pair of successive segments in the chain.

SR has other use cases which go beyond traffic engineering and service chaining. In [START_REF] Aubry | SCMon: Leveraging segment routing to improve network monitoring[END_REF], SR is used for network monitoring, by sending probe packets along a cycle of nodes, thus providing a synchronization-free method to detect link failures. Traffic duplication [START_REF] Aubry | Traffic duplication through segmentable disjoint paths[END_REF] is another use case, where SR is used to send TCP traffic across two parallel paths. When there is spurious packet loss or latency along one of the paths, this helps reducing the flow completion time, while only marginally increasing it when the paths are identical and non-faulty.

Finally, it is noteworthy that in addition to being under standardization at the IETF, SRv6 benefits from an open source implementation in the Linux kernel [START_REF] Lebrun | Implementing IPv6 segment routing in the linux kernel[END_REF] for both endpoints and routing stacks. This makes SRv6 an open and promising framework for innovative network applications.

Bit-Indexed Explicit Replication (BIER)

Bit-Indexed Explicit Replication (BIER) is a multicast architecture, where packets are sent to a source-specified set of destinations, and standardized by the IETF [8]. Traditional multicast protocols (e.g., Protocol Independent Multicast, PIM [START_REF] Nicholas | Protocol Independent Multicast -Dense Mode (PIM-DM): Protocol Specification (Revised)[END_REF]) rely on a client-based subscription model: a multicast group address is pre-provisioned, and clients wishing to be part of the group send "join" messages containing that address to their local router. Upon receiving such a message and before forwarding it to the next router leading to the source, routers record state regarding which interfaces lead to a client that has subscribed to the group (therefore collectively building a spanning tree). When receiving packets from the source addressed to the multicast address, routers know on which interface they should be duplicated to reach all destinations. This requirement of per-flow state in the routers as well as the complexity of group management have hindered the deployment of multicast in the Internet [START_REF] Diot | Deployment issues for the IP multicast service and architecture[END_REF].

BIER, on the other hand, alleviates this complexity by removing the need for per-flow state in intermediate routers. Rather, it lets multicast packets themselves carry information about the destinations that they need to be carried over to. While in traditional multicast protocols, routers solely duplicate packets to the correct interfaces, with BIER they also actively modify packets so as to update, in-path, this set of destinations. Similarly to SR, the set of destinations for a packet is specified by the source. Alternatively, and also similarly to SR, this set can be specified by the first BIER router traversed by the packet, if BIER is to be provided within a domain to which the source does not belong.

The key component to allow such capability lies in agreeing (in advance, and out-of-band) upon a mapping between each node (identified by its IP address) and an integer index. The second key component is the embedding, within each packet, of a bitstring representing the set of destinations for that packet. More precisely, bit i in the bitstring of a packet is set if and only if node i is part of the set of destinations for that packet. Upon sending a packet, a source will embed a bitstring with the set of destinations for that packet, therefore solely taking care of group management. Whereas with PIM, group management happens in-band (with "join" messages) and involves the routers recording state, group management with BIER is thus completely out-of-band.

BIER requires that, as an invariant, the bitstring always contains the current destination set for a given packet. That is, a packet starts with a bitstring containing the whole intended set for the packet, and upon reaching a destination only contains the bit corresponding to that destination. To achieve this, BIER routers use a simple mechanism, described in algorithm 2 and illustrated in figure 1.6. For each destination contained in a given bitstring, unicast routing tables are used to determine which interface can be used to reach (via the shortest path) that destination. Then, the packet is replicated to each of those interfaces leading to at least one destination. While sending the packet over a given interface, its bitstring is updated so that it contains only those destinations reachable via that interface.

In order to optimize the implementation, it is possible to build a special table containing a mapping between interfaces and bit-masks of destinations reachable via this interface. Then, an incoming bitstring is simply AND'd with each of these bit-masks to determine the outgoing bitstring that should be sent over the corresponding interface -when this results in an empty bitstring, the packet is not forwarded over that interface.

It is important to note that the way in which node IP addresses are mapped to BIER identifiers is not algorithmically significant, as long as the mapping is globally distributed and agreed upon between nodes. An interesting way of enforcing a mapping without having to distribute it is, for example, to use IPv6 destinations addresses in which one pre-defined byte carries the BIER identifier.

Similarly, the way the bitstring is encoded in packets does not influence the functioning of the protocol. Notably, the IETF has defined two simple encapsulation mechanisms [START_REF] Wijnands | Encapsulation for Bit Index Explicit Replication (BIER) in MPLS and Non-MPLS Networks[END_REF]: (i) an MPLS header and (ii) a UDP header. The UDP header is arguably more flexible, as it allows BIER packets to be carried over the Internet seamlessly, by establishing a point-to-point tunnel between two successive BIER routers. In addition, a proposal has been made to carry the bitstring within an IPv6 extension header [START_REF] Xie | Encapsulation for BIER in Non-MPLS IPv6 Networks[END_REF]. Finally, an interesting proposal [START_REF] Pfister | An IPv6 based BIER Encapsulation and Encoding[END_REF] consists of encoding bitstrings directly within IPv6 addresses. For simplicity, and for consistency with work carried on SR, the encapsulation mechanism that has been retained throughout this manuscript consists of a custom IPv6 hop-by-hop extension header carrying the bitstring of BIER packets, with the destination address being a fixed multicast address.

Related Work

In [START_REF] Braun | Performance comparison of resilience mechanisms for stateless multicast using BIER[END_REF], a proposal to increase the resiliency of multicast distribution with BIER or BIER-TE 8with fast re-route mechanisms is introduced. This is achieved by the encoding of an additional backup path in the BIER-TE header, and of a reset bitmask to be applied after taking the backup path in order to avoid duplicate packets. A similar mechanism is introduced in [72], where alternative paths are explored with BIER when links fail. In [START_REF] Giorgetti | First demonstration of SDN-based Bit Index Explicit Replication (BIER) multicasting[END_REF], an SDN controller is implemented and evaluated, which can configure an OVS BIER router through the OpenFlow protocol. Compared to a standard SDN multicast solution, this approach only requires configuring the ingress router of the BIER domain, by mapping destination multicast addresses to BIER bitstrings. The use of this implementation over an optical network testbed is demonstrated in [START_REF] Giorgetti | Bit Index Explicit Replication (BIER) multicasting in transport networks[END_REF]. In [START_REF] Brodard | Rover: Poor (but elegant) man's testbed[END_REF], use of BIER-TE over a time-slotted low-power wireless network is demonstrated. Inspection of the bitstring at the egress node facilitates monitoring transmission failures. Finally, a data-plane implementation of BIER in P4 for programmable hardware is presented in [START_REF] Braun | Scalable and reliable software-defined multicast with BIER and P4[END_REF], underlining the simplicity of packet processing allowed by BIER.

Thesis Statement: Augmenting Data Centers through the Network Layer

Section 1.1 has illustrated that data center architectures rely on complex architectures in order to address scalability issues. Notably, section 1.1.1 has demonstrated how complex tunneling architectures are often used as a resort to provide tenant isolation. Similarly, section 1.1.2 has shown that deployed data center topologies are often complex, with a need for custom addressing mechanisms and/or routing protocols, with machines sometimes involved in the routing plane. Then, section 1.2 has introduced two network architectures (SR and BIER) which add flexibility to the Layer-3. A shared property of these two architectures is that they provide source-routinginspired services, wherein paths to be followed by, set of destinations to be reached by, or set of instructions to be applied on a packet are instructed by the source of that packet -or, by the first node of an administrative domain.

Another property shared by both network architectures introduced in section 1.2 is that they can be implemented on top of IPv6 by using extension headers. The flexibility provided by those architectures, as well as the simplicity of running them atop an IPv6 network, make them interesting candidates for augmenting data centers (i.e., providing services at the network layer) while providing a simpler architectural model (i.e., avoiding complex overlays by using only a flat addressing space).

In itself, IPv6 possesses properties making it a good candidate for providing a large-scale data center infrastructure.

• First, and most obviously, the large addressing space provided by IPv6 (with 2 128 addresses as compared to 2 32 addresses with IPv4) is necessary to host large-scale data centers. For instance, a data center using public IPv4 addressing with a /16 IPv4 prefix could only host 2 16 = 65536 virtual machines. • Second, this large addressing space can be leveraged to provide a flat architectural model, without complex overlays. By using the principle of locator-identifier separation [START_REF] Saucez | Designing a deployable internet: the locator/identifier separation protocol[END_REF], it is indeed possible to provision only a physical addressing scheme, where the routing infrastructure only takes care of the reachability of physical machines. On top of that, tasks (VMs, containers, etc.) are assigned virtual identifiers, decoupled of their physical location. Note that such a similar architecture running atop IPv4, VL2 [START_REF] Greenberg | VL2: a scalable and flexible data center network[END_REF], had been discussed discussed in section 1.1.2. With IPv6, a possible instantiation of this principle is Identifer-Locator Addressing (ILA) [START_REF] Herbert | Identifier-locator addressing for IPv6[END_REF]. ILA consists of splitting the IPv6 space into two parts, and using the 64 low-order bits of addresses as identifiers, with the 64 low-order bits of addresses as locators. Naturally, the routing infrastructure will use longest-prefix-match routing and therefore route packets to the machine corresponding to the locator. However, applications only communicate with a common pseudo-locator, which is then translated by a shim layer to the real locator hosting the identifier. This way, tasks can be migrated without the applications noticing.

Those two reasons argue in favor of using IPv6 data centers. Nonetheless, I argue that it is only with the help of in-Layer-3 services, that one can make the case for flexible and efficient IPv6 data centers. This can be achieved by enriching the Layer-3 (through e.g., SR or BIER), adding custom behavior in the network stack by transparently connecting it to the applications, while letting these applications only treat native IPv6 packets.

To achieve such a transparent connection between the network stack and the applications, some assumptions must be made. Notably, as in [START_REF] Greenberg | VL2: a scalable and flexible data center network[END_REF], machines are assumed to be assigned fixed ("physical") addresses, with the routing infrastructure of the data center able to provide connectivity between all machines. Only upon topology changes (adding or removing of machines and/or routers) will the routing infrastructure be updated by the routing protocol running in the data center. Then, each machine hosts several VMs/containers, and a virtual router dispatches packets between the physical interfaces of the machine and the virtual interfaces bound to the hosted VMs/containers. Then, transparently connecting the Layer-3 stack to the applications is done by way of a shim layer running in that virtual router (figure 1.7), which will be able to pre-process packets before handling them to the application (or post-process packets after receiving them from the application). For certain scenarios, a crucial property of this shim layer is that, running in a privileged virtual router, it can inspect the state of the applications, without interacting with them. This way, unmodified application can benefit from additional services, transparently provided by the network layer.

Examples of pre-processing operations which can be performed within such a shim layer, and which are explored throughout this manuscript, are:

• inspecting whether a VM has completed migration, and deciding accordingly whether to forward locally or remotely -allowing transparent VM migration (part II);

• detecting whether a multicast packet is received out-of-order, and if so requesting suitable retransmissions, before forwarding packets in-order to the application -enabling reliable multicast (part III);

• detecting whether a host has a cached copy of a multicast packet and can perform a retransmission of it, or whether the retransmission request should be forwarded further -enabling peer-based multicast recoveries (part III);

• inspecting whether an application is "available" to serve a query, before forwarding the corresponding packet to that application or rather forward it to another one -enabling load-aware load-balancing (part IV);

• inspecting whether an application has Layer-4 state for a connection, and deciding whether to forward packet to that applications or rather to another one -enabling load-balancing consistency (part IV);

• inspecting whether an application is overloaded, and if so, requesting that another instance is launched -enabling monitor-less autoscaling (part IV).

In sum, such an augmented Layer-3 allows offering task mobility (part II), reliable multicast (part III), and load-balancing with autoscaling (part IV) as a network service, while using unmodified applications. As will be studied throughout this manuscript, this has benefits in terms of network traffic overhead, optimization of resource utilization, quality and fairness of service, and energy reduction.

Definitions and Working Assumptions

This section briefly introduces the working assumptions that will be used throughout this manuscript, and defines terms and notations which will be used recurrently.

• A router is a hardware device which performs Layer-2 or Layer-3 forwarding between interfaces according to manually or automatically configured tables. Note that the term router will be used to refer to any forwarding device, irrespective of whether it performs Layer-2 or Layer-3 forwarding (whereas devices capable of Layer-2 forwarding, and by extension devices capable of both Layer-2 and Layer-3 forwarding, are sometimes referred to as switches in the literature).

• A machine is a physical computer, running a single operating system and connected to a network via physical network cards (NICs). It can itself host virtual appliances (VMs or containers) by way of a hypervisor -and these virtual appliances themselves are connected to the host via virtual network interfaces. Throughout this manuscript, machines will often be represented by the letter m. Machines will sometimes be referred to as hosts (due to their ability to host these virtual appliances).

• An application instance is a piece of software running as a Virtual Machine or a container, providing a service to consumers that would want to interact with the application. This is usually achieved by running a server software, e.g., an HTTP server. Application instances can be replicated for scaling purposes, in which case it is assumed that each of the instances can provide identical services for a client -it is however necessary to ensure that a single network flow is always handled by the same application instance. By metonymy with the software running within them, application instances will occasionally be referred to as servers throughout this thesis. They will also be referred to as tasks or workloads, when this is more appropriate as per the context. Throughout this thesis, application instances will be represented by the letter i (in chapter 4), v (in chapter 3, where v is a mnemonic for VIP), or s (in part IV, where s is a mnemonic for server).

• A rack is a set of machines, located in a single physical pod, and connected to a Top-of-Rack (ToR) router.

• A virtual router is a softwarized router running in a physical machine, and whose purpose is to provide inter-connection between the physical interfaces of the machine, and the virtual interfaces bound to the application instances hosted by that machine. • A data center is defined to be a single entity, containing a set of routers and machines, and connected to the outside via a border gateway. Within the data center, an (IPv6) physical addressing plane is used by the machines, i.e., each machine is assigned a Physical IP address (PIP). Note that such PIPs do not necessarily have to be globally routable. A routing protocol is assumed to have converged, so that machines can communicate with one another transparently within the data center. Application instances are addressed by Virtual IP addresses (VIPs), and the VIPs are advertised by the border gateway of the data center, to the remainder of the Internet. The border gateway is in charge of routing packets coming from the outside of the data center and addressed to a VIP, to the correct corresponding machine(s). To do so, it will encapsulate these packets and route them to the PIP of that machine. The virtual router running there will then be in charge of decapsulation and handing over to the correct application instance, and of any extraneous behavior as defined in this thesis.

With these definitions, it is possible to provide a very simple model of a data center network. More precisely, let R be the set of routers in the data center and M be the set of physical machines. Then, the data center network can be modeled as a directed graph G = (V, A), where V = M ∪ R is the set of vertices, and A ⊆ (M ∪ R) 2 the set of arcs. Existence of an arc (u, v) ∈ A models the existence of a physical link between nodes u and v. Such links can exist between two routers, between a machine and a router, but also between a machine and itself (thereby modeling a loopback interface).

Then, for each ordered pair of machines (m, m ′ ) ∈ M 2 , a list A mm ′ ∈ 2 A represents the path from m to m ′ . For example, given the topology depicted in Figure 1.8 with three routers r 0 , r 1 , r 2 and four machines m 1 , m 2 , m 3 , m 4 , the arcs of the graph will be as follows: A = {(m 1 , m 1 ), (m 1 , r 1 ), (r 1 , m 1 ), (m 2 , m 2 ), (m 2 , r 1 ), (r 1 , m 2 ), (m 3 , m 3 ), (m 3 , r 2 ), (r 2 , m 3 ), (m 4 , m 4 ), (m 4 , r 2 ), (r 2 , m 4 ), (r 1 , r 0 ), (r 0 , r 1 ), (r 2 , r 0 ), (r 0 , r 2 )}. The path from e.g., m 1 to m 3 will be A m1m3 = {(m 1 , r 1 ), (r 1 , r 0 ), (r 0 , r 2 ), (r 2 , m 3 )}.

Finally, generic mathematical notation used throughout this thesis is introduced in table 1.1.

CHAPTER 1. INTRODUCTION

Notation Definition E 2 Cartesian product

E × E = {(x, y) : x ∈ E, y ∈ E} 2 E Set of parts of E {F : F ⊆ E} N Set of non-negative integers {0, 1, . . .} R Set of real numbers R + Set of non-negative real numbers {x ∈ R : x ≥ 0} C Set of complex numbers {x + iy : (x, y) ∈ R 2 }, i 2 = -1 ⌊x⌋ Floor function (⌊x⌋ ≤ x < ⌊x⌋ + 1) ⌈x⌉ Ceiling function (⌈x⌉ -1 < x ≤ ⌈x⌉) log x Natural logarithm (in base e) log 2 x Logarithm in base 2 P[X] Probability of event X P[X|Y ] Probability of event X conditioned to Y E[X] Expected value ("average") of random variable X E[X] = x x • P[X = x] if X is discrete R xf (x)dx if X has density f CDF of X Cumulative Distribution Function of random variable X CDFX (x) = P[X ≤ x] f (x) =x→a o(g(x)) f is dominated by g f (x) = ε(x)g(x) with limx→a ε(x) = 0 f (x) ∼x→a g(x) f is equivalent to g f (x) = (1 + o(1))g(x) f (x) =x→a O(g(x)) f is bounded above by g f (x) = ε(x)g(x) with ε(x) ≤ M, ∀x ∈ (x -δ, x + δ),
for some M > 0 and δ > 0

f (x) =x→a Θ(g(x))
f is bounded above and below by g f

(x) = ε(x)g(x) with m ≤ ε(x) ≤ M, ∀x ∈ (x -δ, x + δ),
for some m > 0, M > 0 and δ > 0

Thesis Contributions

This chapter concludes this introductory part by providing a summary of contributions in section 2.1 and a list of publications in section 2.2.

Thesis Summary and Outline

This thesis studies the usage of network-layer protocols to provide task mobility, reliable content distribution, and load-balancing in data center networks. It comprises 5 parts and 10 chapters, structured as follows.

Part I provides an introductory discussion. In chapter 1, background on data-center architectures and associated network protocols is introduced. Then, two interesting Layer-3 architectures are introduced, SRv6 and BIER, which make use of source-routing paradigms in order to extend the network layer. Finally, a discussion is made about how using such architectures can help augment data centers by providing scalability primitives directly at the network layer. Chapter 2 then summarizes how this concept was applied throughout this thesis, to provide task mobility, reliable content delivery, and load-balancing.

Part II studies task mobility in data centers. In chapter 3 (published in [START_REF] Desmouceaux | Zero-loss virtual machine migration in IPv6 data-centers with segment routing[END_REF]), the use of SRv6 to provide zero-loss VM migration is introduced. Traditional network protocols for VM migration rely on completely migrated VMs to signal their new location to a centralized directory, thus yielding a transient period during which packets destined to such VMs are lost. To alleviate this, chapter 3 proposes to pre-allocate, with SR, a loose "migration path" consisting of the old and new physical machines: this way, packets always reach the correct machine, whichever step the migration process is in. This path is provisioned by the control plane prior to initiating the actual migration, and until after the migration has completed. This is achieved by introducing two new SR functions: the first one checks (at the source host) whether the link to the VM is still up and forwards packets to the VM or to the second segment accordingly; the second one checks (at the destination host) whether the VM has fully restarted, and locally buffers or forwards packets to the VM as a result. Implementation and evaluation on a virtual router (VPP) show that it is indeed possible to migrate VMs without losing packets, thus decreasing latency and flow completion time for the hosted application.

Having introduced the possibility of task migration as a network service, chapter 4 (published in [START_REF] Desmouceaux | Flow-aware workload migration in data centers[END_REF]) studies how flow-aware workload migration can be achieved, that is, migrating communicating task closer to each other so as to optimize network traffic. Traditional data center management architectures have been introduced that consider inter-task network demands, or the cost of workload migration, but not both. In chapter 4, a multi-objective optimization program is introduced, aiming at maximizing the total inter-task throughput while minimizing the number of migrated tasks and maximizing the number of newly allocated tasks. A Mixed Integer Non-Linear Programming (MILNP) formulation of the problem is introduced to capture the different constraints and objectives, and is linearized into a Mixed Non-Linear Programming (MILP) formulation. To compute the set of Pareto-optimal solutions, the ε-constraint method is used as a baseline. Through simulations, the proposed approach is demonstrated to efficiently increase the total achievable throughput by migrating some of the communicating tasks close to one another. Due to the high computational cost of this approach, a heuristic solution method is 18 CHAPTER 2. THESIS CONTRIBUTIONS proposed, which incrementally computes solutions of increasing migration cost, so as to return an approximate Pareto front. Simulations show that the proposed heuristic is able to decrease the computational solving time by up to two orders of magnitude, while providing solutions close to those which are optimal.

Part III studies reliable content delivery in data centers. In chapter 5 (published in [START_REF] Desmouceaux | Reliable multicast with B.I.E.R[END_REF]), the use of BIER to provide efficient reliable multicast is introduced. Traditional reliable multicast protocols use a Negative-ACKnowledgement (NACK) scheme, whereby lost packets are signaled by corresponding destinations to the source. Then, the source performs either unicast retransmissions to each of the failing destinations, or a multicast retransmission to the whole group. In chapter 5, a NACK-based protocol aiming at minimizing retransmission traffic is proposed, taking advantage of BIER. Instead of using per-flow bitstrings, different bitstrings are used per-packet: retransmission traffic for a packet is sent only to the set of those destinations that have missed that packet. To achieve this, the source collects NACKs emitted by different destinations during a short time window, so as to construct a suitable retransmission bitstring. The protocol is implemented in a network simulator (ns3), and packet-level simulations are conducted in different topologies with different loss patterns. Especially in cases where a subset of the data-center exhibits localized losses, traffic is shown to be reduced as compared to multicast-based retransmissions (because the whole tree does not have to be flooded) and to unicast-based retransmissions (because the bottleneck links do not have to carry multiple copies of retransmitted packets). To complete the analysis, a mathematical model quantifies the footprint of retransmission traffic for unicast-, multicast-, and BIER-based retransmission, in arbitrary tree topologies. A first-order approximation when α → 0 (with α the link loss rate) is derived in theorem 5.1, showing that retransmission traffic scales as L log L with BIER retransmissions, as compared to L log 2 L with unicast retransmissions and L 2 with multicast retransmissions (where L is the number of links in the tree), theoretically confirming those benefits.

In chapter 6 (submitted as [START_REF] Desmouceaux | Reliable BIER with peer caching[END_REF]), the use of reliable BIER is extended to provide peer-assisted multicast recoveries. An extension to the BIER data-plane is proposed, which allows destinations to learn about topologically close peers that have subscribed to the same multicast flow. This is achieved by carrying and updating a peerstring in each packet, containing the set of destinations reached by the parent node for this packet. Then, upon missing a packet, SR is used by failing destinations to direct retransmission requests (NACKs) through a path comprising (i) one or several peers and (ii) the source, until finding one peer (or ultimately, the source) able to send a retransmission of the missed packet. Different policies for peer selection are proposed and mathematically evaluated. First, two simple static policies are analyzed, namely (i) selecting a random peer in a destination's subtree and (ii) selecting a designated peer in that subtree. Mathematical analysis reveals that the second policy generates less retransmission traffic (as retransmissions are sent by only one peer) but is less likely to succeed (since if that peer has not received the packet, no peer will be able to obtain a retransmission). Second, a dynamic policy is introduced, wherein each peer tries to dynamically learn from which peer it is more likely to obtain a retransmission. The proposed architecture is evaluated through packet-level network simulations, in different topologies and with different policies. Evaluation shows that it is able to reduce the load on core links (as compared to chapter 5) by increasing retransmission locality, thus decreasing the overall network overhead.

Part IV studies load-balancing in data centers. In chapter 7 (published in [START_REF] Desmouceaux | SRLB: The power of choices in load balancing with segment routing[END_REF][START_REF]6LB: Scalable and application-aware load balancing with segment routing[END_REF]), the use of SRv6 for application-aware load-balancing is studied. Usually, load-balancing devices in data centers are replicated, for resiliency and scalability. In doing so, consistent hashing is used to ensure that flows are assigned to the same server, whichever load-balancer they go through, and even in case of reconfiguration of the pool of application instances. However, these approaches assign queries to application instances regardless of their current load. To handle this, chapter 7 introduces the use of SR to direct the first packet (TCP SYN) of a query through a chain of two random candidate instances, each one successively deciding whether or not to accept it based on its local state. This provides a load-aware load-balancing mechanism running within the network layer, and without monitoring. To avoid unnecessary triangular traffic, further packets are directed directly towards the instance having accepted the connection. This is achieved by using an inband signaling protocol, where custom SR functions are used between the load-balancers and the servers to install and remove per-flow state. Resiliency is ensured by way of a consistent hashing algorithm assigning flows to lists (instead of singletons) of instances in a reliable way. A mathematical model of the client response time of the system for Poisson arrivals and exponential service times is derived. Using this model, it is possible to quantify the benefits of the proposed approach in terms of expected response time, tail response time, server load fairness, and energy reduction. The analysis shows that, under the reasonable assumption that the network delay is smaller than the mean service time, the performance is always increased as compared to random single-choice load-balancing (theorem 7.1). The proposed architecture is implemented as a plugin for a virtual router (VPP), able to extract information from a standard HTTP server (Apache) without disruption through a shared memory channel, and using this information to take connection acceptance decisions. Evaluation on a 48-instance testbed with synthetic and real traffic confirms these benefits, notably showing that it is possible to provide the same average quality of service (client response time) by reducing the number of VMs by 17% as compared to single-choice random load-balancing.

In chapter 8 (published in [START_REF] Pit-Claudel | Stateless loadaware load balancing in P4[END_REF]), the feasibility of implementing a stateless load-aware loadbalancing architecture in hardware is explored. Implementing load-balancing dispatchers in hardware is desirable for efficiency and scalability reasons. However, this is incompatible with maintenance of per-flow state, as required by load-aware architectures. To circumvent this, chapter 8 explores the use of covert channels to carry such state within the packet headers. The same mechanism is used as in chapter 7 to establish connections -i.e., connection establishment packets are sent through a chain of candidate instances until one of them accepts to handle the query, based on its local state. However, instead of installing flow state in the load-balancer, the instance having accepted the connection communicates its position in the SR header in a covert channel back to the client. This value is then automatically reflected by the client to the load-balancer, which can thus use this information to direct the packet to the correct server. Low-order bits of TCP timestamps are proposed as a covert channel usable by standard, unmodified TCP clients. In addition, the load-balancer ensures reliability in case of changes in the pool of instances by way of consistent hashing versioning. That is, packets are sent with an SR header containing instances that were previously the result of the consistent hashing operation. The feasibility of this architecture is demonstrated by implementation of a prototype on programmable hardware, using the P4 language and targetting the NetFPGA-SUME platform. Data-plane per-packet simulations of that implementation show that the incurred parsing latency is negligible (in the order of 10 µs). Simulations show that the proposed consistent hashing mechanism improves resiliency by one order of magnitude as compared to mechanisms that do not use versioning. In sum, this architecture allows to obtain the same benefits as those introduced in chapter 7, while providing a low-latency line-rate hardware implementation and increasing consistent hashing resiliency.

Chapter 9 (submitted as [START_REF] Desmouceaux | Joint monitorless load-balancing and autoscaling for zero-wait-time in data centers[END_REF]) introduces a monitor-less auto-scaling architecture using SRv6. In order to meet variations in traffic demands while ensuring a predefined level of service, data centers use auto-scalers to, in real time, adapt the number of replicated instances of a given service. Auto-scalers rely on centralized architectures to make these decisions, thus incurring a monitoring overhead. In chapter 9, the use of SRv6 to provide joint monitor-less load-balancing and auto-scaling is studied. Application instances are ordered into a fixed chain, and new queries are sent along this chain until one of them accepts to serve the connection, using the data-plane introduced in chapter 7. In addition, the last instance of the chain monitors its own usage state to trigger down-or up-scaling of the chain. Indeed, the last instance of the chain receiving too few queries indicates that the chain is over-provisioned, whereas its receiving too many queries indicates that the chain must be upscaled. A Markov chain model of the performance of the system is derived, allowing to express the number of queries held by each server. Using the Recursive Renewal Reward (RRR) technique, it is possible to derive the corresponding expected client response time, in the case of small chains, and therefore to deduce the possible energy savings offered by the system (i.e., the possible reduction in number of servers while achieving a similar service quality as with random load-balancing). Implementation of the architecture as a VPP plugin, and evaluation with real traffic traces confirms that these benefits hold in real environments: the same average client response time can be achieved with lesser VMs as with random load-balancing, while simultaneously diminishing the tail of the response time distribution.

Finally, part V concludes this manuscript, and a summary in French is provided in appendix A.

List of Publications

The following publications were published or submitted during the course of this PhD. Conference or Workshop Publications

Other Contributors

Although this thesis reports work mostly conducted by myself during these three years of PhD, some of the work reported in this thesis results from a collaboration with my co-authors.

My colleagues Pierre Pfister and Jérôme Tollet provided ideas and discussions on chapters 7, and Pierre ran the experiments reported in figure 7. [START_REF] Mahalingam | Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks[END_REF]. The experiments reported in chapter 8 are the result of a research internship conducted at Cisco by Benoît Pit--Claudel, and mentored by myself with input from Pierre Pfister.

Outside my direct work circle, Sonia Toubaline provided insightful input on the modeling part of chapter 4. Similarly, Juan-Antonio Cordero-Fuertes provided valuable input on the modeling part of chapters 5 and 6. Especially, the experiments reported in section 6.5.3 result from a collaboration with Juan-Antonio.

Part II

Task Mobility

Chapter 3

Zero-Loss Virtual Machine Migration with IPv6 Segment Routing

Virtual Machine (VM) live migration [START_REF] Clark | Live migration of virtual machines[END_REF] is a way to transfer a VM from a machine to another, by iteratively copying its memory. It is further suggested in [START_REF] Bolla | Seamless and transparent migration for TCP sessions[END_REF][START_REF] Nadgowda | Voyager: Complete container state migration[END_REF] that it is possible to perform live migration of individual processes or containers, thus attaining one further level of granularity of task placement flexibility. These techniques naturally find applications in the context of data centers, as these host heterogeneous workloads, whose lifetimes can vary greatly, and which can exhibit rapidly changing resource demands and inter-application dependencies. With live VM migration, applications need not be tied to a specific machine -and their relocation can become part of a "natural" mode of data center operation. In this context, architectures have been developed in which workload migration is used as a baseline to achieve different goals [START_REF] Ahmad | A survey on virtual machine migration and server consolidation frameworks for cloud data centers[END_REF]: energy minimization [START_REF] Ghribi | Energy efficient VM scheduling for cloud data centers: Exact allocation and migration algorithms[END_REF], network usage minimization [START_REF] Kakadia | Network-aware virtual machine consolidation for large data centers[END_REF], operational cost minimization [START_REF] Zeng | Cost minimization for big data processing in geo-distributed data centers[END_REF], maintenance [START_REF] Wang | Virtual routers on the move: live router migration as a network-management primitive[END_REF], etc. Thus, VM migration not only provides a way to accommodate hardware failures without service downtime, but more generally may be beneficial to the efficiency of the whole data center.

From a network perspective, a challenge raised by live migration lies in maintaining connectivity to a VM after it has been migrated. Indeed, hypervisors normally assume that VMs are migrated within a single LAN, using Reverse ARP (RARP) to advertise the new location of a VM after migration. Traditional techniques to overcome this issue rely on Layer 2 overlays, such as VXLAN [START_REF] Mahalingam | Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks[END_REF] or NVGRE [START_REF] Garg | NVGRE: Network Virtualization Using Generic Routing Encapsulation[END_REF]. Other approaches include making use of Mobile IP [START_REF] Johnson | Mobility Support in IPv6[END_REF] or of LISP [START_REF] Saucez | Designing a deployable internet: the locator/identifier separation protocol[END_REF]. The emergence of IPv6 [START_REF] Deering | Internet Protocol, Version 6 (IPv6) Specification[END_REF] data-centers introduces other opportunities, both for the addressing of workloads and for re-engineering the data-path. Among the proposed approaches for mobility within IPv6 data-centers, Identifier-Locator Addressing (ILA) [START_REF] Herbert | Identifier-locator addressing for IPv6[END_REF] has been proposed at the IETF, using high-order bytes of addresses to denote locators and low-order bytes of addresses to denote identifiers.

A drawback of all these approaches is that they incur a period of time, during which packets addressed to the migrating VMs are lost. This raises concerns for applications that are intolerant to packet losses (e.g., UDP-based delay-critical applications, virtual network functions, ...).

Statement of Purpose

The purpose of the chapter is to introduce a VM mobility solution, which provides "zero-loss" capabilities: assuming that the network is not lossy, any packet destined to a migrating VM will eventually be received by said VM. To that purpose, the Segment Routing (SR) architecture is leveraged.

As introduced in section 1.2.1, SR is an architecture which allows packets within a designated domain to be added an extraneous header, designating an ordered list of segments through which the packet is expected to go. Segments represent abstract functions to be performed on packets, and can be as simple as forward to next segment (enabling source routing and traffic engineering), but can also represent more complicate instructions (from custom encapsulation or routing behavior to complete virtual network functions). With IPv6 Segment Routing (SRv6), segments 24 CHAPTER 3. ZERO-LOSS VM MIGRATION WITH SRV6 are represented as IPv6 addresses embedded as a list in an IPv6 extension header [START_REF] Filsfils | IPv6 Segment Routing Header (SRH)[END_REF].

The idea underlying this chapter is to use SRv6 to solve the locator/identifier mapping synchronization problem during VM migration, by letting the gateway (proactively) route packets destined to a migrating VM through a path comprising the old and the new host machine. This way, the responsibility of the old host machine is reduced to (i) forwarding packets locally while the migration is not complete and (ii) forwarding packets to the next segment (the new host machine) once migration has completed. This way, no packets are lost during the migration, whereas traditional solutions would require a mapping to be updated once migration has completed, leading to potential packet losses during this period of time. Furthermore, the overhead on the old host machine is kept to a minimum, since no tunnel has to be established and no Layer-2 overlay is required.

Related Work

Numerous solutions have been proposed to address the issue of maintaining Layer-3 network connectivity during and after VM migration. The simplest involves using a Layer-2 overlay [START_REF] Mahalingam | Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks[END_REF][START_REF] Garg | NVGRE: Network Virtualization Using Generic Routing Encapsulation[END_REF], and relying on hypervisors sending gratuitous Reverse ARP messages after migration. In addition to the operational complexity incurred by such overlays, the VM is not reachable on the new host machine until the RARP has propagated, leading to potential packet losses.

Another simple solution consists of creating an IP tunnel between the source and the destination host machines. In [START_REF] Bradford | Live wide-area migration of virtual machines including local persistent state[END_REF], the Xen hypervisor is modified so that after migration, packets reaching the hypervisor at the source host machine are tunnelled towards the new host machine. After migration, the VM uses two different addresses (an "old" and a "new" address), and Dynamic DNS is used so that external clients can reach the VM via the new address. The drawback of this approach is that the hypervisor must co-operate with the destination host machine during an unpredictable amount of time by performing tunnelling. Furthermore, this is incompatible with para-virtualized interfaces such as virtio-net [START_REF] Russell | virtio: towards a de-facto standard for virtual I/O devices[END_REF], where packets destined to VMs are not handled by the hypervisor.

In [START_REF] Li | HyperMIP: Hypervisor controlled mobile IP for virtual machine live migration across networks[END_REF], Mobile IP is used to assist the migration process. Traffic from/to the VM is routed through a home agent, which tunnels it to the correct machine hosting the VM. It is the role of the hypervisor to update the home agent with the new location of the VM once it has migrated. Thus, after VM migration, packets can wrongfully reach the source host machine before registration of the new location is complete. This approach is improved in [START_REF] Watanabe | A performance improvement method for the global live migration of virtual machine with IP mobility[END_REF], by configuring, before migration, a dummy secondary interface for the destination network, and swapping primary and secondary interfaces after migration. This requires co-operation with the VM as two interfaces are used.

In [START_REF] Raad | Achieving subsecond downtimes in large-scale virtual machine migrations with LISP[END_REF], LISP is used to address the issue of Layer-3 connectivity during VM migration. Packets destined to a VM traverse a LISP router, which encapsulates them towards the current resource locator of the VM. The hypervisor is modified so that, after VM migration, the mapping system is updated to reflect the new resource locator of the VM. This avoids triangular routing, but once again the VM is not reachable during the period of time when the mapping is being updated.

Finally, other approaches orthogonal to the context of this chapter are worth mentioning. In [START_REF] Kalim | Seamless migration of virtual machines across networks[END_REF], TCP options are used to facilitate migration of TCP connections. In a Software Defined Networking (SDN) context, [START_REF] Mattos | XenFlow: Seamless migration primitive and quality of service for virtual networks[END_REF] proposes a framework for virtual router migration, in which control planes and data planes are migrated during two distinct phases.

Chapter Outline

The remainder of this chapter is organized as follows. Section 3.2 introduces the high-level assumptions and mechanisms used for SR-based migration, before a formal specification is given in section 3.3. The proposed mechanism is then evaluated on different workloads in section 3.4. Finally, section 3.5 concludes this chapter.

SR-based Migration

As described in section 1.4, this chapter assumes that a VM is located within an IPv6 datacenter, and accessible through a virtual IP address (VIP). Each machine within the data-center is accessible at a physical IP address (PIP), and can host several VMs. Located at the edge of the data-center, a gateway advertises (e.g., with BGP) the whole virtual address prefix. A location database maintains a mapping between each virtual address and the physical address of the machine hosting the corresponding VM. When receiving traffic for a given VIP, the router will apply an SR policy consisting of one segment (the PIP of the machine hosting the VM). Each machine is running a virtual router (in the implementation described in section 3.4, VPP1 ), which is connected to the physical interface(s) of the machine, and to each VM via a virtual interface (in the implementation described in section 3.4, a vhost-user interface2 ). In sum, under normal operation (when the VM is running on a single machine), the gateway will tunnel traffic for a VM towards the machine hosting it.

The mechanism introduced in this chapter assumes that an orchestrator is running in the data-center, which decides on the allocation of VMs to physical machines. When the orchestrator decides to move a VM from a host machine to another, it proactively modifies the routing tables of the gateway, so that the traffic for the corresponding VIP is applied an SR policy consisting of two segments, corresponding to the old and new machines hosting the VM. This way, when the VM is migrating, the gateway will direct traffic to the old host machine. As long as the VM has not completed migration, traffic will be intercepted by the old host machine (figure 3.1); as soon as migration is complete, the old host machine will simply forward traffic to the next segment (i.e., the new host machine). This way, no synchronization is required between the host machines and the networking infrastructure: rather, traffic is loosely sent to a logical path comprising both machines.

Detailed Specification

This section introduces a formal description of the SR functions necessary to perform zero-loss migration, as well as the behavior of the gateway.

Definitions

Forward to Local (f w)

The f w function simply forwards the packet to the next segment. In SR terminology [START_REF] Filsfils | SRv6 Network Programming[END_REF], this corresponding to the END behavior.

Forward to Local if Present (f wp)

The f wp function forwards the packet to the last segment (skipping intermediary segments), only if the corresponding VIP v is present locally; otherwise it acts as the f w function and forwards the packet to the next segment. The forwarding decision is made by the virtual router by inspecting its routing table, and seeing whether the entry for v corresponds to a virtual (local) interface, whose link-status is up. This corresponds to the END.S behavior [START_REF] Filsfils | SRv6 Network Programming[END_REF] with a custom policy for forwarding decisions.

Buffer and Forward to Local (bf w)

The bf w function inspects the last segment v. If v is not present locally (i.e., if the corresponding virtual interface is not ready), it will buffer the packets for further delivery. Otherwise, it flushes any buffered packet to the virtual interface corresponding to v, and forwards the current packet to the last segment v. Implementation-wise, packets are buffered in the local memory of the virtual router, per interface and in order. Such packet buffers must be provisioned with a size large enough to handle all potential packets coming during the VM downtime phase. If a VM is expected to receive traffic at rate of r packets/s and to be down during ∆t seconds, buffers must be provisioned with a size of r • ∆t packets.

Detailed Migration Process

For each VIP v, the controller maintains an entry L(v) in the gateway which is either R(m 1 ) ("running on m 1 ") or M (m 1 , m 2 ) ("migrating from m 1 to m 2 "), depending on the state of v. Conceptually, this forms a mapping v → L(v), where L(v) is the location of v. Practically, this is implemented by way of routing adjacencies in the FIB of the gateway. The gateway uses the transit behavior T.INSERT [START_REF] Filsfils | SRv6 Network Programming[END_REF] for v, that is, this routing adjacency triggers insertion of an SR header on packets destined to v. When the gateway receives a packet for v:

1. If the corresponding entry L(v) is R(m 1 ), then a SR header (m 1 ::f w, v) is inserted; 2. If the corresponding entry L(v) is M (m 1 , m 2 ),

Normal Operation

Under "normal operation", that is, when a VM v is running on a machine m 1 , the gateway is configured to map v to L(v) = R(m 1 ). Thus, packets destined to v are forwarded to m 1 by way of a single-hop SR tunnel.

Migration Operation

Figure 3.2 examplifies the operation that occurs when a VM is migrated. When the controller decides to move v from m 1 to m 2 , it updates the gateway so as to remap the entry for v to L(v) = M (m 1 , m 2 ). It then queries the hypervisors running on m 1 and m 2 to initiate the live migration process.

First, the VM is stopped on m 2 and running on m 1 , while the memory of v is iteratively copied by the hypervisor (out-of-band) from m 1 to m 2 . The gateway inserts an SR header (m 1 ::f wp, m 2 ::bf w, v) in the packets it receives for v. Since the virtual router on m 1 sees that the virtual interface of the VM is up, upon triggering of the f wp function, it will simply forward these packets to that interface (skipping the m 2 segment).

Second, the hypervisor has finished iteratively copying the memory of the VM to the new host machine, and needs to perform the copy of the last memory pages as well as the CPU state. At this point, the VM is stopped on both m 1 and m 2 . The gateway still inserts an SRH (m 1 ::f wp, m 2 ::bf w, v) in packets it receives. Now that the VM is stopped on m 1 (which the virtual router at m 1 detects by inspecting the status of the corresponding virtual interface, now in link-down state), the f wp function will simply forward packets to the next segment. Since m 2

is not yet running the VM (which the virtual router at m 2 detects by inspecting the status of the corresponding virtual interface, also in link-down state), the bf w function will temporarily buffer the packet.

Third, the VM is started back on machine m 2 . Before the controller (and thus the gateway) is notified of this, the gateway has a stale mapping L(v) = M (m 1 , m 2 ), and the same SRH is inserted in the packets destined to v. Thus, the first machine m 1 forwards these to m 2 . Now, the bf w function on m 2 will trigger release of the buffered packets to the virtual interface of the VM, and further packets are forwarded without buffering.

Finally, the controller is notified of the end of the migration, and can update the gateway to map v to L(v) = R(m 2 ): normal operation resumes, packets reach m 2 directly.

Evaluation

The SR functions described in section 3.3.2 have been implemented as VPP plugins. No collaboration between VPP and the hypervisor is needed; rather, forwarding decisions are made from within VPP by inspecting the state of the vhost-user interface corresponding to the VM of interest. Buffers for the bf w function are sized to 8192 packets (consuming 16 MB of RAM), allowing to sustain MTU-sized traffic at ≈ 1 Gbps for a typical VM downtime of 100 ms. A simple testbed is set up with three physical machines, as in figure 3.1: the first plays the role of a gateway gw, and the two other ones m 1 , m 2 represent compute nodes, which are candidates to host virtual machines. A client VM v 1 (representing the "outside" of the data-center) is attached to the gateway. A VM v 0 represents a server that will be migrated. That VM is first running on m 1 , then migrated from m 1 to m 2 . In order to reflect this, the gateway is configured in the M (m 1 , m 2 ) state, i.e., it inserts (m 1 ::f wp, m 2 ::bf w, v 0 ) in all packets destined to v 0 . Four mechanisms are compared:

1. no-migration, i.e., the VM is not migrated; The baseline scenario serves as an illustration of mechanisms such as LISP-based migration [START_REF] Raad | Achieving subsecond downtimes in large-scale virtual machine migrations with LISP[END_REF], wherein packet loss occurs due to the locator mapping being updated only after migration.

Ping (illustration)

In order to illustrate the behavior of these four mechanisms, a ping3 is run between the client VM v 1 and the server VM v 0 , while v 0 is migrated. One packet is sent every millisecond, and the RTT for each packet is recorded. Figure 3.3 shows the time at which an echo answer is received as a function of when the corresponding echo request was sent. Without migration, each answer is received approximately 0.1 ms after the corresponding query is sent. With non-SR migration, packets were lost during 722 ms (corresponding to the VM downtime, plus the network reconfiguration time), whereas with SR migration without buffer, packets were lost for only 174 ms (corresponding to the VM downtime). Finally, with SR migration with buffer, 175 packets were buffered while the VM was down, and replied to as soon as the VM went up again.

HTTP Workload

To understand the behavior of SR-migration when facing a delay-sensitive workload, a simple evaluation scenario is carried out. The server VM v 0 is set up with an Apache HTTP serverserving a default static file (whose size is 12 KB). As previously, the VM is first running on m 1 , then migrated from m 1 to m 2 . A traffic generator is attached to the gateway, sending a Poisson stream of 6000 queries with rate λ = 1500 s -1 , during which the VM is migrated. The experiment is The majority of queries are executed when the VM is not migrating, and exhibit small response times (≤ 3 ms). The other queries correspond to those that started while the VM was migrating. With non-SR migration, TCP SYN packets are lost when the VM is down, but some are also lost after the VM has restarted on m 2 , due to the reconfiguration delay. Due to the SYN retransmit delay of 1 s, those queries exhibit a response time ≥ 1 s (more than 19% of queries are concerned). With SR migration without buffer, TCP SYN packets are lost when the VM is down, but as soon as the VM is up they are successfully transmitted again. This explains why queries experiencing a SYN retransmit are less numerous than with the baseline scenario: less than 4% of queries have a response time greater than 1 s. Finally, with SR migration with buffer, TCP SYN packets are buffered while the VM is down. The corresponding response time is simply delayed by the VM downtime (≈ 100 ms in these experiments), rather by the SYN retransmit delay. Less than 0.7% of queries exhibit a response time greater than 1 s, whereas more than 99.2% of queries have a response time lower than 200 ms.

A similar experiment is performed on a more realistic workload, consisting of drawing a random number k (from an exponential distribution with mean E[k] = 4) and serving k copies of the previous static file. This allows to induce variability in the response times, as well as more network traffic. Again, the experiment is repeated 10 times, and client response times are recorded: results are depicted in figure 3.4b. With non-SR migration, more than 19% of queries are replied to within 1 s or more. In comparison, less than 4% of queries exhibit a response time greater than 1 s with SR migration without buffer, and this drops to 0.4% for SR migration with buffer. Furthermore, 99.5% of queries are served within less than 200 ms with the latter mechanism.

Iperf Workload

The previous set of experiments gave insight on the behavior of the different migration mechanisms in presence of short-lived flows. To understand the influence of SR-migration on longer connections, an experiment with throughput-intensive parallel TCP flows is performed. The same experimental platform as in the previous section is used -except that this time, a delay of 2.5 ms is added between the VM v 1 (representing clients) and the VM v 0 (representing the server to be migrated), so as to represent clients outside the data center. An instance of iperf with 50 parallel connections is started on the client VM, for 10 seconds (the rationale for using many parallel connections is to smooth the effects of TCP congestion control). Meanwhile, the server VM is migrated from m 1 to m 2 , using the same 4 mechanisms as before. The experiment is repeated 10 times.

Figure 3.5a depicts the instantaneous throughput (aggregated over the 50 subflows) for one of the 10 runs, collected every 0.5 s. Before and during VM migration, the throughput is rather stable, oscillating around 3.5 Gbps. When the migration ends, around t = 7 s, there is a drop in throughput, due to the VM downtime (≈ 150 ms in these experiments). While this drop is significant with non-SR migration (going to 1.8 Gbps in the run reported in figure 3.5a), it remains reasonably high with SR migration (2.7 Gbps in this experiment). The low performance of non-SR migration can be explained by TCP reducing its congestion window, due to the large amount of time when no packets are received by the VM. With SR migration, this downtime is lower, and thus the congestion window is less reduced. The impact of buffering, versus dropping packets while the VM is not yet up on the new host machine, seems to be negligible.

In order to quantify these behaviors, figure 3.5b depicts the distribution of the instantaneous aggregate throughput (collected every 0.5 s), over all the 10 runs. With the no migration sce- nario, the instantaneous throughput stays between 2.9 and 3.7 Gbps. SR migration is able to maintain the throughput above 2.3 Gbps at all times, whereas with non-SR migration the instantaneous throughput can drop to 0.4 Gbps during the migration phase. With non-SR migration, the throughput is lower than 2.2 Gbps for 5% of the time (compared to never with SR migration), and lower than 2.9 Gbps for 10% of the time (compared to 5% of the time with SR migration).

Iperf Sink Workload

The same set of experiments is repeated, except that in this case the iperf instance running in the VM is acting as a sink. This represents, e.g., use-cases where the VM receives a lot of data -for instance, if it is an HTTP proxy or a firewall. In such a scenario, packets lost during the migration process are actual data packets (rather than simple TCP acknowledgements packets, as in the previous set of experiments), which is expected to have a greater influence on the overall quality of service.

Figure 3.6a reports the instantaneous throughput (aggregated over the 50 subflows) for one of the 10 runs, collected every 0.5 s. This time, the drop in throughput when using non-SR migration is more critical, going to 0.2 Gbps. Figure 3.5b depicts the CDF of the instantaneous aggregate throughput as collected every 0.5 s, over the 10 runs. Non-SR migration still exhibits low performance, with the 5-th percentile for throughput being 0.6 Gbps, as compared to 1.9 Gbps for SR migration. Furthermore, in this case, there is a (small yet perceptible) benefit from buffering packets vs dropping them when using SR migration, as each x-th percentile for throughput is greater with SR migration with buffer than with SR migration with drop.

CHAPTER 3. ZERO-LOSS VM MIGRATION WITH SRV6

Summary of Results

This chapter has introduced a mechanism to perform zero-loss VM migration in IPv6 datacenters. Contrary to traditional approaches, which maintain connectivity to migrating VMs reactively by updating the location of VMs after they have migrated, this chapter introduces a proactive mechanism, which pre-provisions a logical path through which packets will flow shortly before, during, and shortly after, migration. This is enabled by the use of IPv6 Segment Routing, allowing to direct packets destined to a migrating VM through this logical path. This way, coupling between the data plane and the hypervisor is reduced to a minimum: the only operation that a virtual router performs upon receipt of a packet, is checking whether the corresponding interface is up, and forward the packet locally or to the next segment accordingly. Implementation on a real virtual router, VPP, and evaluation by way of diversified workloads, show that the proposed mechanism is indeed able to provide zero-loss VM migration, with benefits both in terms of TCP throughput and session opening latency.

Note that this chapter focuses on VMs processing traffic from/to the outside of the datacenter, i.e., egress-facing servers or virtual network functions. An interesting further question to be investigated is how to use similar techniques to provide seamless migration for VMs handling intra-data-center traffic.

Results from this chapter have been published in [START_REF] Desmouceaux | Zero-loss virtual machine migration in IPv6 data-centers with segment routing[END_REF].

Chapter 4

Flow-Aware Workload Migration in Data Centers

With the desirability of task (VM or container, also occasionally called "workload" in literature [START_REF] Cheng | Heterogeneity-aware workload placement and migration in distributed sustainable datacenters[END_REF]) mobility established in chapter 3, where it was also shown that it was possible to do so without losses, the question of an optimal task migration mechanism naturally arises, which will be investigated in this chapter. Task placement has been studied, and current approaches provide mathematical models focusing on optimal usage of resources and on flow minimization [START_REF] Zeng | Cost minimization for big data processing in geo-distributed data centers[END_REF], or models that also consider migration but do not take the detailed network topology into account [START_REF] Shrivastava | Applicationaware virtual machine migration in data centers[END_REF][START_REF] Huang | Multi-objective virtual machine migration in virtualized data center environments[END_REF].

This chapter introduces a multi-objective linear programming model of task migration that satisfies per-node resources constraints and inter-nodes communication requirements while minimizing the cost incurred by migrations. The fundamental assumptions are (i) that tasks have inter-communication demands that change with time, (ii) that tasks that communicate benefit from being topologically close, and therefore (iii) that changes in inter-communication demands may make a current task placement suboptimal, suggesting possible efficiency gains by relocation. Hence, the developed multi-objective linear programming model of task migration is intended to run iteratively, at regular time intervals. It uses task inter-communication demands to decide if and where to re-allocate tasks so as to provide the best possible task placement, while minimizing a migration cost. Network constraints are taken into account by modeling the traffic as a multicommodity flow problem, where a commodity represents an inter-task communication demand.

Statement of Purpose

For some distributed computing applications, where tasks frequently communicate with each other, the limiting factor for the completion of a running task is not necessarily its actual location (assuming that the task is placed on a machine with the full physical resources it needs), but can be the throughput at which it exchanges data with other tasks. Furthermore, if those tasks have changing traffic requirements, migrating them within the data center can be an efficient way to re-organize the load across all available machines so as to optimally satisfy these requirements.

With these two assumptions as a baseline -tasks need to be placed according to their communication requirements, and those requirements can change over time -the multi-objective mathematical problem introduced in this chapter consists of determining a placement which satisfies machine, network and applications constraints, while (i) maximizing the number of new tasks to be assigned, (ii) minimizing the migration cost and (iii) maximizing the total inter-task throughput.

The contribution of this chapter is therefore threefold: (i) it formulates a model of task migration which uses an accurate description of network demands and of the network topology; (ii) it proposes a multi-objective approach which generates a set of Pareto-optimal solutions, from which the operator can choose according to operational tradeoffs; and (iii) it proposes a heuristic method that can generate an approximate Pareto front while greatly reducing solving time as compared to obtaining the exact solution.

Evaluations show that the sets of solutions generated by this approximation method remain close to their optimal counterparts, regardless of the topology and the task-to-machine ratio.

CHAPTER 4. FLOW-AWARE WORKLOAD MIGRATION

Related Work

Data center traffic has been extensively studied: it has been shown that the amount of intrarack traffic is as voluminous as is extra-rack traffic [START_REF] Benson | Network traffic characteristics of data centers in the wild[END_REF], and that patterns emerge when considering the traffic matrix between interdependent tasks [START_REF] Kandula | The nature of data center traffic: measurements & analysis[END_REF][START_REF] Meng | Improving the scalability of data center networks with traffic-aware virtual machine placement[END_REF]. Some task pairs communicate more than others, in which case it makes sense to locate those on either the same machine, or on machines close to each other in the network topology. Therefore, data center architectures have been studied which take communication dependencies between tasks into account, from a network systems perspective [START_REF] Al-Fares | Hedera: Dynamic flow scheduling for data center networks[END_REF][START_REF] Lacurts | Choreo: Network-aware task placement for cloud applications[END_REF]. This motivates the need for models that can place tasks according to their pairwise communication demands.

The task placement problem consists of finding an efficient assignment of tasks to machines within a data center, satisfying a set of constraints (resources, possibly traffic demand) with respect to one or more objective functions such as energy consumption minimization [START_REF] Ghribi | Energy efficient VM scheduling for cloud data centers: Exact allocation and migration algorithms[END_REF][START_REF] Cheng | Heterogeneity-aware workload placement and migration in distributed sustainable datacenters[END_REF][START_REF] Ferreto | Server consolidation with migration control for virtualized data centers[END_REF][START_REF] Jin | Joint host-network optimization for energy-efficient data center networking[END_REF][START_REF] Liu | Task and server assignment for reduction of energy consumption in datacenters[END_REF], communication cost minimization [START_REF] Kakadia | Network-aware virtual machine consolidation for large data centers[END_REF][START_REF] Zeng | Cost minimization for big data processing in geo-distributed data centers[END_REF], network traffic minimization or resource utilization maximization. Different optimization approaches have been proposed for the task placement problem: solution techniques include deterministic algorithms, heuristics, meta-heuristics and approximation algorithms. Surveys of these methods can be found in [START_REF] Ahmad | A survey on virtual machine migration and server consolidation frameworks for cloud data centers[END_REF][START_REF] Pires | Virtual machine placement literature review[END_REF][START_REF] Ferdaus | Network-aware virtual machine placement and migration in cloud data centers[END_REF][START_REF] Usmani | A survey of virtual machine placement techniques in cloud data center[END_REF]. Many approaches have been proposed that aim at optimizing a single objective. Multi-objectives formulations have also been introduced, but they are often reduced to a mono-objective problem, failing to provide a set of Pareto-optimal solutions.

Single-objective approaches: In [START_REF] Shrivastava | Applicationaware virtual machine migration in data centers[END_REF], a model that considers resources constraints as well as dependencies between applications is proposed, which then re-allocates tasks while minimizing the migration impact. Contrary to the approach presented in this chapter, [START_REF] Shrivastava | Applicationaware virtual machine migration in data centers[END_REF] does not consider changing communication demands between applications, but only migrate tasks located on overloaded machines. In [START_REF] Huang | Multi-objective virtual machine migration in virtualized data center environments[END_REF], this model is refined by incorporating finer-grained server-side constraints, as well as network capacities of the machines. In [START_REF] Fang | VMPlanner: Optimizing virtual machine placement and traffic flow routing to reduce network power costs in cloud data centers[END_REF], the problem of optimizing both task placement and traffic flow routing so as to save energy is considered. A model is formulated that minimizes the power consumption of switches and links, while satisfying capacity and traffic demands constraints. In [START_REF] Kakadia | Network-aware virtual machine consolidation for large data centers[END_REF], a network-aware model is considered that places tasks with the aim of minimizing communication cost. A greedy consolidation algorithm is proposed, consisting of identifying task clusters based on network traffic (using a cost matrix). In [START_REF] Meng | Improving the scalability of data center networks with traffic-aware virtual machine placement[END_REF], a traffic-aware task placement model is introduced, where the objective is to minimize the aggregate traffic rates perceived by every switch. A two-tier approach given by a cluster-and-cut heuristic is proposed.

Multi-objective reduced to single-objective: Some VM placement problems are formulated as multi-objectives models, which are then reduced to single-objective models. In [START_REF] Chen | Optimized virtual machine placement with traffic-aware balancing in data ceter networks[END_REF], the authors consider the placement of VMs on machines by minimizing the size of clusters of VM serving the same jobs, and minimizing the maximum traffic on uplinks of top-of-rack switches. An iterative least-loaded-first based placement algorithm is proposed, which first gives the priority to locality, and then to channel occupancy. In [START_REF] Fang | Power-efficient virtual machine placement and migration in data centers[END_REF], a power-efficient VM placement and migration model is introduced, aiming at minimizing the number of machines used, the network energy consumption and the network end-to-end delay. The model is solved using a weighted sum approach. In [START_REF] Xu | Multi-objective virtual machine placement in virtualized data center environments[END_REF], VM placement is considered under three objectives to be minimized: total resource "waste", power consumption, and thermal dissipation costs. The solution approach is a combination of a generic grouping algorithm and a fuzz multi-objective evaluation. In [START_REF]A multi-objective approach to virtual machine management in datacenters[END_REF], a VM management framework considering initial VM placement and VM migration is introduced, with three objectives: elimination of thermal hotspots, minimization of power consumption and application performance satisfaction. The problem is solved using a weighted sum approach.

Pure multi-objective approaches: In [START_REF] Gao | A multi-objective ant colony system algorithm for virtual machine placement in cloud computing[END_REF], a bi-objective mathematical formulation for VM placement is proposed, aiming at minimizing resource wastage and power consumption. A multi-objective ant colony system algorithm is proposed to generate the set of non dominated solutions. In [START_REF] Pires | Multi-objective virtual machine placement with service level agreement[END_REF] (extending [START_REF]Virtual machine placement. A multi-objective approach[END_REF]), a multi-objective formulation of the VM placement problem is introduced, aiming at minimizing energy consumption and network traffic, and maximizing economical revenue while satisfying a service level agreement (SLA). A memetic algorithm given by an evolutionary process is proposed, and a Pareto set approximation is returned.

Chapter Outline

The remainder of this chapter is organized as follows. Section 4.2 introduces the framework needed to represent the state of the data center. Section 4.3 presents the mathematical formulation proposed for the flow-aware workload placement and migration problem, given by a multi-objective Mixed Integer Linear Program (MILP) [START_REF] Ehrgott | Multicriteria optimization[END_REF]. Section 4.4 describes the algorithm used to generate the set of Pareto-optimal solutions to this problem, and provides a computational example. An approximation method is proposed in section 4.5; its performance in terms of solving time and quality of the generated approximate solutions (as compared to the optimal ones) are then evaluated. Finally, section 4.6 concludes this chapter.

4.2. DATA CENTER REPRESENTATION 35 Notation Description i, i ′ , • • • ∈ I t Tasks to be run at time t m, m ′ , • • • ∈ M Machines r, r ′ , • • • ∈ R Routers A ⊆ (M ∪ R) 2 Network edges A mm ′ ∈ 2 A Path m → m ′ cuv > 0 Capacity of link (u, v) κm > 0 Machine CPU capacity ρi > 0 Task CPU requirement σi > 0 Task size C t ⊆ I t × I t Communicating tasks at time t d t ii ′ > 0 Throughput demand for i → i ′ at time t x t im ∈ {0, 1} Task i is placed on machine m at time t f t ii ′ (u, v) ∈ R + Flow for i → i ′ along (u, v) at time t

Data Center Representation

Before detailing the problem formulation (section 4.3), resolution (section 4.4) and approximation (section 4.5), this section introduces the framework used to describe the state (including the topology) of a data center. This is achieved by considering (i) a set of machines, (ii) the network topology connecting them, and (iii) a set of tasks to place on these machines. As this model considers task migration, for which it is necessary to be aware of the evolution of the state of the data center, the model makes the assumption that it is running at a given time t (where t is an abstract index without unit), and that the previous state of the data center corresponds to time (t -1). Thus as a convention, a variable superscripted with t represents the current state of the data center (to be optimized), and a variable superscripted with (t -1) is a known input representing the previous state of the data center. A summary of the notations used throughout this chapter is provided in table 4.1.

Tasks and Machines

Tasks to be run at time t are represented by a set I t . At time t, new tasks can arrive or existing tasks can finish executing. Arriving tasks correspond to those in I t \ I t-1 . Already existing tasks correspond to those in I t ∩ I t-1 , from among which those that were successfully assigned to a machine at time (t -1) have priority over new tasks and must be placed at some machine at time t (they can continue execution on the same machine, or be migrated to another, but cannot be stopped). Terminated tasks correspond to those in I t-1 \ I t , and thus are not of interest to the model.

Machines are represented by a set M. Each machine m ∈ M has a CPU capacity κ m > 0 which represents the amount of work it can accommodate. Conversely, each task i ∈ I t has a CPU requirement ρ i > 0, representing the amount of resources it needs in order to run. Finally, each task i ∈ I t has a size 1 σ i > 0, which will be used to model the cost of migrating the task from one machine to another.

Network Model

In order to take application dependencies into account, the physical network topology existing between the machines must be known. To that end, and as described in section 1.4, a network is modeled as a directed graph G = (V, A), where V = M ∪ R is the set of vertices, with R the set of routers and A the set of arcs. An arc (u, v) ∈ A can exist between two routers or between a machine and a router, but also between a machine and itself to model a loopback interface (so that two tasks on the same machine can communicate). Each of those arcs represents a link in the network, and has a capacity of c uv > 0. For each ordered pair of machines (m, m ′ ) ∈ M 2 , a list A mm ′ ∈ 2 A represents the path from m to m ′ (as illustrated in section 1.4).

Finally, at a given time t, an ordered pair of tasks (i, i ′ ) ∈ I t × I t can communicate with a throughput demand d t ii ′ > 0, representing the throughput at which i would like to send data to i ′ . Let G t d = (I t , C t ) be a weighted directed graph representing these communication demands, where each arc (i, i ′ ) ∈ C t is weighted by d t ii ′ . The graph G t d will be referred to as the throughput demand graph.

Mathematical Modeling

Based on the data center framework described in section 4.2, this section presents a multiobjective Mixed Integer Non Linear Program (MINLP) aiming at optimizing task placement and migration while satisfying inter-application network demands. A linearization as a multi-objective MILP is then derived, allowing for an easier resolution.

Variables

Two sets of variables are introduced, representing (i) task placement and (ii) the network flow for a particular placement.

Task placement

The aim of the model is to provide a placement of each task i ∈ I t on a machine m ∈ M, at a given timestep t. The binary variable x t im reflects this placement: x t im = 1 if i is placed on m, and x t im = 0 otherwise.

Network flow

In order to determine the best throughput that can be achieved between each pair of communicating tasks, a variant of the multi-commodity flow problem [124, p. 58] is used, where a commodity is defined by the existence of (i, i ′ ) ∈ C t . A commodity thus represents an inter-task communication.

For each link (u, v) ∈ A, f t ii ′ (u, v) is a variable representing the throughput for communication from i to i ′ along the link (u, v).

Constraints

Two sets of constraints are used to model this flow-aware workload migration problem: placement constraints (equations (4.1-4.3)) represent assignment of tasks to machines, whereas flow constraints (equations (4.4-4.8)) focus on network flow computation.

The placement constraints represent the relationship between tasks and machines. First, each task i ∈ I t must be placed on at most one machine:

m∈M x t im ≤ 1, ∀i ∈ I t (4.1)
Forcefully terminating a task is not desirable: if a task was running at time t -1, and is still part of the set of tasks at time t, it must not be forcefully terminated:

m∈M x t-1 im ≤ m∈M x t im , ∀i ∈ I t-1 ∩ I t (4.2)
where x t-1 im is a known input given by the state of the system at time t -1. If the task was already successfully placed at time t -1 (i.e., i ∈ I t-1 and m∈M x t-1 im = 1) and is still to be run at time t (i.e., i ∈ I t ), the left-hand-side of the equation will be 1, thus forcing placement of the task at time t.

Finally, the tasks on a machine cannot use more CPU resources than the capacity of that machine:

i∈I t ρ i x t im ≤ κ m , ∀m ∈ M (4.
3)

The flow constraints allow computing the throughput for each commodity (i.e., ordered pair of communicating task). For each link (u, v) ∈ A in the network, the total flow along the link must not exceed its capacity:

(i,i ′ )∈C t f t ii ′ (u, v) ≤ c uv , ∀(u, v) ∈ A (4.4)
For a commodity (i, i ′ ) ∈ C t , the flow going out of a machine m must not exceed the throughput demand for the communication from i to i ′ . Also, the flow must be zero if task i is not hosted by machine m:

v:(m,v)∈A f t ii ′ (m, v) ≤ d t ii ′ x t im , ∀m ∈ M, ∀(i, i ′ ) ∈ C t (4.5)
Conversely, for a commodity (i, i ′ ) ∈ C t , the flow entering a machine m ′ must not exceed the throughput demand for the communication from i to i ′ , and must be set to zero if task i ′ is not on

m ′ : v:(v,m ′ )∈A f t ii ′ (v, m ′ ) ≤ d t ii ′ x t i ′ m ′ , ∀m ′ ∈ M, ∀(i, i ′ ) ∈ C t (4.6)
Each router r ∈ R must forward the flow for each commodity -that is, the ingress flow must be equal to the egress flow:

v:(u,v)∈A f t ii ′ (u, v) = v:(v,u)∈A f t ii ′ (v, u), ∀u ∈ R, ∀(i, i ′ ) ∈ C t (4.7)
Finally, if a task i is placed on machine m and a task i ′ on machine m ′ , the corresponding flow must go through the path specified by A mm ′ . Otherwise, the flow computed by the model could go through a non-optimal path or take multiple parallel paths -and for simplicity of interpretation, the latter is considered out-of-scope for this chapter. Hence, the flow needs to be set to zero for all edges that do not belong to the path from m to m ′ :

f t ii ′ (u, v) ≤ c uv (1 -x t im x t i ′ m ′ ), ∀(i, i ′ ) ∈ C t , ∀m, m ′ ∈ M, ∀(u, v) ∈ A \ A mm ′ (4.8)
This constraint has no side effect if task i is not on m or task i ′ is not on m ′ , since in this case it reduces to f t ii ′ (u, v) ≤ c uv , already covered by equation (4.4).

Objective Functions

The migration model as presented in this chapter introduces three different objective functions, modeling (i) the placement of tasks, (ii) the overall throughput achieved in the network and (iii) the cost incurred by task migration, respectively. These functions depend on a placement, i.e., on an assignment of all variables x t im and f t ii ′ (u, v). Let x t (respectively f t ) be the characteristic vectors of the variables x t im (respectively f t ii ′ (u, v)). The placement objective is simple and expresses that a maximal number of tasks should be successfully placed on a machine. When removing inter-task communication, this degenerates to a standard assignment problem wherein each task should be placed on a machine satisfying its CPU requirement, while also not exceeding the machine capacity. The placement objective function is simply the number of tasks which are successfully placed on a machine:

P (x t ) = i∈I t m∈M x t im (4.9)
The throughput objective expresses the need to satisfy applications throughput demands. Modeling network dependencies between tasks in a data center is usually (e.g., in [START_REF] Shrivastava | Applicationaware virtual machine migration in data centers[END_REF]) done through the use of a cost function depending on the network distance between two machines. Having introduced a representation of the physical network and of applications dependencies in section 4.2.2, it is possible to represent the overall throughput reached in the data center. To compute the throughput of the communication from a task i to a task i ′ , it suffices to identify the machine m on which i is running and take the flow going out of this machine for this commodity:

m∈M x t im v:(m,v)∈A f t ii ′ (m, v).
This expression is quadratic in variables x t im and f t ii ′ (u, v), but, owing to the fact that (4.5) constrains the flow to be zero for machines to which i is not assigned, can be simplified to m∈M v:(m,v)∈A f t ii ′ (m, v). Therefore, the overall throughput in the data center can be expressed as:

T (f t ) = (i,i ′ )∈C t m∈M v:(m,v)∈A f t ii ′ (m, v) (4.10)
Since each machine has a loopback link in the graph G (i.e., that (m, m) ∈ A, ∀m ∈ M ), this formulation also covers the case where i and i ′ are hosted on the same machine.

Finally, the migration cost reflects the cost of moving tasks from one machine to another. The fundamental assumption is that tasks in a data center have communication demands that can evolve over time (modeled by d t ii ′ ). This means that migrating a task to a machine topologically closer to those machines hosting other tasks with which it communicates, can be a simple way to achieve overall better performance. The migration cost incurred between two successive times (t -1) and t is modeled as the sum of the sizes of tasks that have migrated. Since at time t, the assignment x t-1 is known, it is possible to know if a task i ∈ I t ∩ I t-1 placed on a machine m ∈ M has moved, by comparing x t im to (1x t-1 im ). Also, if a task is to be shut off at time t (i.e., i ∈ I t-1 \ I t ), it must not be part of the computation of the number of migrated tasks. The total migration cost from time t -1 to time t can thus be expressed as:

M (x t ) = i∈I t ∩I t-1 m∈M σ i x t-1 im (1 -x t im ) (4.11)
Using these three objectives, it is possible to express the flow-aware placement and migration model as a multi-objective MILNP:

(P)                    max T (f t ) max P (x t ) min M (x t ) subject to      (4.1-4.8) x t ∈ {0, 1} I t ×M f t ∈ (R + ) C t ×A (4.12)
These objectives tend to compete with each other. If a task starts communicating with another task, migrating it to the same machine (or to a machine closer in the topology) can increase the throughput at which they communicate, thus increasing the throughput objective function -but this will incur an increase of the migration cost. Likewise, if a new task arrives at t and needs important CPU resources, placing it right away can consume CPU capacity that could have been used for co-locating tasks: increasing the placement objective can in some cases decrease the throughput objective.

As an illustration, consider two machines m 1 and m 2 with CPU capacities κ 1 = 1 and κ 2 = 3, with a bottleneck link (of capacity 1) in between. Assume that at (t -1) there were two tasks i 1 , i 2 with CPU requirements ρ i1 = 1 and ρ i2 = 1, and that i 1 was placed on m 1 and i 2 on m 2 . At time t, i 1 starts to communicate with i 2 (with throughput demand d t i1i2 = 2), and a third task i 3 arrives with CPU requirement ρ i3 = 2. Then, if this new task is to be placed on some machine, m 2 is the only machine with enough capacity to host it. But placing i 3 on m 2 would prevent a migration of i 1 from m 1 to m 2 , which would have increased the throughput of communication i 1 → i 2 . This situation is summarized in figure 4.1.

m 1 i 1 m 2 i 2 capa=1Gbps m 1 i 1 m 2 i 2 i 3 1Gbps capa=1Gbps m 1 m 2 i 2 i 1 i 3 2Gbps capa=1Gbps throughput=2,placed=2,migrated=1 throughput=1,placed=3,migrated=0 i 1 i 2 demand =2Gbps t-1 t i 1 i 2 demand =0Gbps

Linearization

All constraints expressed in section 4.3.2 are linear with respect to the variables x t im and f t ii ′ (u, v), except for the path enforcement constraint in equation (4.8). Exploiting the fact that x t im ∈ {0, 1}, this constraint can be linearized:

f t ii ′ (u, v) ≤ c uv (2 -x t im -x t i ′ m ′ ), ∀(i, i ′ ) ∈ C t , ∀m, m ′ ∈ M, ∀(u, v) ∈ A \ A mm ′ (4.13) If x t im × x t i ′ m ′ = 1, the equation will be f t ii ′ (u, v) ≤ c uv or f t ii ′ (u, v) ≤ 2 c
uv , and will therefore be superseded by equation (4.4).

The set of constraints can be further compressed by writing only one equation per machine m ∈ M instead of one per tuple m, m ′ ∈ M. This does not alter the model but makes the formulation more compact:

f t ii ′ (u, v) ≤ c uv   2 -x t im - m ′ ∈M:(u,v) / ∈A mm ′ x t i ′ m ′   , ∀(i, i ′ ) ∈ C t , ∀m ∈ M, ∀(u, v) ∈ A (4.14)
In this way, the flow-aware workload migration problem can be expressed as the following multi-objective Mixed Integer Liner Program (MILP):

(P ′ )                    max T (f t ) max P (x t ) min M (x t ) subject to      (4.1-4.7), (4.14) x t ∈ {0, 1} I t ×M f t ∈ (R + ) C t ×A (4.15)

Flow-Aware Workload Migration

This section describes an algorithm (Algorithm 3) solving the previously introduced multiobjective MILP (4.15). This algorithm runs at a given time t representing the "present". It takes the current inter-application communication requirements and the "past" (from time t -1) task placement as inputs, and returning a new placement as a solution.

Resolution Algorithm

A Pareto-optimal approach [123, p. 24] is used: a set of optimal solutions is generated, from which a final solution z t = (x t , f t ) can be extracted based on an operator-defined policy 40

CHAPTER 4. FLOW-AWARE WORKLOAD MIGRATION Algorithm 3 Multi-Objective Migration Algorithm z t-1 ← input ⊲ initial state C t , d t ii ′ , B, εp, εp ← input ⊲ parameters S ← ∅ for εp ∈ [εp, εp] with step δp do for εm ∈ [B, 0] with step -δm do if ∃z ′ ∈ S, P (z ′ ) ≥ εp ∧ M (z ′ ) ≤ εm then continue ⊲ dominant solution already existing end if z ← solution of (4.18) ⊲ if z is not dominated, it is a candidate solution if ∃z ′ ∈ S, z ′ ≻ z then
⊲ remove any existing solution dominated by z for z ′ ∈ S do if z ≻ z ′ then S ← S \ {z} end if end for S ← S ∪ {z} ⊲ add z to candidate solutions end if end for end for ⊲ select the best solution according to the considered policy z t ← getSolutionFromPolicy(S) getSolutionFromPolicy(). In the context of this migration model, a placement z = (x, f ) dominates a placement z ′ = (x ′ , f ′ ) if all the objectives functions of the former have better values than the ones of the latter (with at least of one having a strictly better value). That is denoted z ≻ z ′ :

z ≻ z ′ ⇔          P (z) ≥ P (z ′ ) T (z) ≥ T (z ′ ) M (z) ≤ M (z ′ ) (P (z), T (z), M (z)) = (P (z ′ ), T (z ′ ), M (z ′ )) (4.16)
A set S = {z 1 , . . . , z n } of solutions is Pareto-optimal if and only if:

∀(z, z ′ ) ∈ S 2 , z = z ′ ⇒ (z ≻ z ′ ∧ z ′ ≻ z) (4.17)
To generate the set of Pareto-optimal solutions for the multi-objective problem, the ε-constraint method [123, p. 98] is used. It consists of reducing a multi-objective linear programming model to a single-objective model, by relaxing all objective functions but one and considering them as constraints. These constraints are obtained by bounding the relaxed objective function by a number ε that varies in a certain interval. Although its initial formulation only considers two-objectives problems, it can be extended to problems with a greater number of objective functions [START_REF] Laumanns | An adaptive scheme to generate the pareto front based on the epsilon-constraint method[END_REF], as it is the case here.

The two functions that are relaxed are the placement objective and the migration cost, because it is simple to grid the corresponding space. The single-objective MILP that results is given by:

(P ε )                    max T (f t ) subject to                (4.1-4.7), (4.14) P (x t ) ≥ ε p M (x t ) ≤ ε m x t ∈ {0, 1} I t ×M f t ∈ (R + ) C t ×A (4.18)
where ε p , ε m are bounds varying in [ε p , ε p ] and [0, B] respectively.

For the placement objective, a simple upper bound for ε p corresponds to the number of tasks: Similarly, a lower bound for ε p can be easily derived from tasks that were running at the previous iteration and that are still to be run. As per equation (4.2), they cannot be forcefully terminated:

ε p = i∈I t 1 = |I t | (4.19)
u = r0, v ∈ {r1, r2} cuv = cvu = 40 u ∈ {r1, r2}, v ∈ M cuv = cvu = 40 u ∈ M cuu = 200
ε p = i∈I t-1 ∩I t m∈M x t-1 im (4.20)
For the migration cost, correctly choosing the upper bound B for ε m has a large impact on the running time of the algorithm. This bound B will be referred to as the migration budget.

Computational Example

In large-scale deployments, the amount of CPU resources is likely sufficient for all tasks to be deployed. However, complex situations can exist, for which it is beneficial to have an understanding of the implications that different placement choices can have. For instance, when new tasks arrive that have a high CPU requirement, choosing to place them upfront can consume CPU capacity which could have been used to migrate a smaller task and satisfy its throughput demand. This section illustrates the model response in situations where placement and migration are competing with each other, by way of an environment in which the machines are nearly overloaded in the initial state. The algorithm has been implemented in Python using the Gurobi MILP solver [START_REF] Inc | Gurobi optimizer reference manual[END_REF], and all the simulations have been run on a virtual machine with 4 GB of RAM and an 2.6 GHz CPU.

Simulation Parameters

A simple two-tier tree topology is used, consisting of one root router r 0 linked to two ToR routers r 1 , r 2 , each linked to 10 machines (Figure 4.2). In order to represent the fact that tasks located on the same machine can achieve a higher throughput, each machine has a link to itself, with a higher capacity than other links in the data center, as depicted in Table 4.2.

The CPU capacity is set to 3 for each machine -which gives a total capacity of 60. An initial state with 40 tasks with CPU requirement ρ i = 1 and a throughput demand graph G t-1 d with 80 arcs of intensity d t-1

ii ′ = 10 is considered, with an initial placement z t-1 that is optimal with respect to throughput demands2 .

Given this initial placement, this study explores the model behavior when communication demands of existing tasks change, while new bigger tasks arrive in the system. Communication demand changes are modeled by adding 10 random arcs (of intensity d t ii ′ = 10) between the already running tasks. Arrival of new tasks is modeled by creating 10 new tasks with CPU requirement ρ i = 2, and by adding 10 random arcs (of intensity d t ii ′ = 5) going out of these tasks.

Results

In order to obtain the set of Pareto-optimal solutions S at time t, one iteration of Algorithm 3 was run. Figure 4.3 shows a three-dimensional representation of the solutions computed for one of those runs: Pareto-optimal solutions can be visually identified as those for which no other solutions have better coordinates in all dimensions. Results show that non-trivial behavior occurs when the data center is almost fully loaded: while placing a few new tasks improves the overall throughput, allocating all of them has a detrimental effect on the overall throughput because it prevents migration of other tasks.

This shows the interest of using a multi-objective model in cases where the objectives are conflicting with each other.

Pareto Front Approximation

Section 4.4 introduced a generic solution approach, which allows to compute the exact Pareto front for the three competing objectives (placing the highest number of tasks, minimizing the cost incurred by migration, achieving the best inter-application throughput) by way of an ε-constraint method. Although this approach generates an exact set of efficient solutions, and thus can be useful for small and highly-constrained deployments, its computational complexity makes it unsuitable to model larger infrastructures.

In order to overcome this issue, this section derives and experimentally analyses a heuristic for approximating the Pareto front. Section 4.5.1 introduces the approximation method, and section 4.5.2 provides the results of computational experiments in order to validate the quality of the proposed approach.

Heuristic Formulation

In large deployments, it is assumed that there are enough resources to accommodate all tasks to be run, thus the remainder of this section assumes that all tasks can be placed (i.e., that there exists at least one solution to (4.18) such that P (x) = |I t |). In order to focus on migration, it is also assumed that no new task arrives (i.e., I t = I t-1 := I).

Given this assumption, a heuristic allowing to approximate the Pareto front in the throughputmigration plane, i.e., for (P (z), T (z), M (z)) ∈ {|I|}×[0, +∞)×[0, B], is introduced. The issue with the ε-constraint approach introduced in section 4.4 is that, when considering the migration of B tasks, there are O(|I| B |M| B ) possible recombinations, which can be computationally expensive. In order to limit the combinatorics of the recombination, the introduced heuristic splits the migration process in smaller chunks, considering fewer migrations at a time, and proceeding in a greedy way until the total budget is exhausted, as described in Algorithm 4.

Algorithm 4 Heuristic Multi-Objective Resolution

z t-1 ← input ⊲ initial state C t , d t ii ′ , B ← input ⊲ parameters µ ← 0 ⊲ total migration cost ∆ ← input ⊲ incremental budget Ŝ ← ∅ ẑ0 ← z t-1 τ ← 1 ⊲ step index while µ < B do ⊲ reduce ∆ at last iteration so that µ + ∆ ≤ B ∆ ← min{∆, B -µ} ⊲ find a placement within ∆ of the previous one ẑτ ← solution of                    max T ( f τ ) subject to                (4.1-4.7), (4.14) with t → τ P (x τ ) = |I| M (x τ , xτ-1 ) ≤ ∆ xτ ∈ {0, 1} I×M f τ ∈ (R + ) C t ×A if M (x τ , xτ-1 ) = 0 then break ⊲ stop if no task could be migrated end if Ŝ ← Ŝ ∪ {ẑ τ } ⊲ compute the migration cost w.r.t. initial state µ ← M (x τ , x0 ) τ ← τ + 1 end while z t ← getSolutionFromPolicy( Ŝ) return z t
For clarity, let M (x t , x t ′ ) represent the migration cost when going from state x t to x t ′ :

M (x t , x t ′ ) = i∈I∩I t ′ m∈M σ i x t ′ im (1 -x t im ) (4.21)
Let ∆ > 0 be a parameter representing an incremental budget. Starting from an initial configuration z t-1 , and given an upper-bound B on the migration cost, intermediary states (ẑ 0 = z t-1 , ẑ1 , . . . , ẑτ , . . . , ẑτ ) are created. At each step τ ≥ 1, a new placement is generated by maximizing the throughput objective, with the constraint that no more than ∆ is incurred as a migration cost as compared to step (τ -1). This way, no more than O(|I| ∆ |M| ∆ ) recombinations are considered at each step, allowing for a smaller expected computational complexity. This process ends when no further tasks can be migrated, or when a total migration cost of B (as compared to step 0) has incurred. The heuristic tracks the solutions generated while generating the successive solutions, so as to build an approximate Pareto front Ŝ.

Computational Experiments

This section experimentally illustrates the benefits of using the Pareto front approximation heuristic, both in terms of computational complexity and quality of the approximation. The sensitivity of the heuristic to the incremental budget parameter ∆ is also experimentally explored. The experimental platform is the same as in section 4.4.2.

Simulation Parameters

In a common data center, all machines in a rack are linked to a top-of-rack (ToR) router, and those ToR routers are linked to each other via an aggregation router. In data centers, several such configurations can be linked together by way of a third routing layer -in all cases, resulting in a tree-like topology, as described in section 1.1.2.

For the purposes of these simulations, a k-rack topology is defined as a tree where a core router r 0 is attached to k ToR routers r 1 , r 2 , . . . , r k . Each ToR router 10 machines m h,1 , m h,2 , . . . , m h,10 . This topology thus comprises 10 × k machines, and Figure 4.2 depicts a 2-rack topology. Links capacities are set as per Table 4.3, i.e., links between the core router and the ToR routers, and between the ToR routers and the machines have a capacity of 40. Loopback interfaces on the machines have a capacity of 1000 -so as to favor placement of tasks which communicate with each other on the same machine.

r h (1 ≤ h ≤ k) is attached to 44 CHAPTER 4. FLOW-AWARE WORKLOAD MIGRATION u = r0, v ∈ {r1, . . . , r k } cuv = cvu = 40 u ∈ {r1, . . . , r k }, v ∈ M cuv = cvu = 40 u ∈ M cuu = 1000
Five k-rack topologies are considered, for k ∈ {2, 3, 4, 5, 10} (for a corresponding number of machines |M| ∈ {20, 30, 40, 50, 100}). For each of these topologies, four scenarios are considered with different tasks-per-machine ratios, such that |I| ∈ {|M|, 2|M|, 5|M|, 10|M|} -as a way to compare between different load factors for the machines. This makes for a total of 5 × 4 = 20 scenarios.

CPU requirements of tasks are set to ρ i = 1, and CPU capacities of the machines to κ m = 20. This allows for up to 20 tasks per machine, thus ensuring that all tasks can always be placed. To simplify understanding the migration cost, tasks size (for computing the migration cost) are set to σ i = 1, thus the migration cost simply becomes the number of migrated tasks. Tasks are initially uniformly randomly assigned to the machines. The communication matrix is generated by: (i) for each task i ∈ I, uniformly choosing a random number D i of destinations in {1, 2, 3}, (ii) for each of these D i destinations, uniformly choosing a task i ′ ∈ I \ {i} and (iii) setting a throughput demand

d ii ′ for the communication i → i ′ uniformly in {1, 2, . . . , 10}.
For each of these 20 scenarios, the exact Pareto front is computed using Algorithm 3, with a migration budget (maximum migration cost) B = 30. The approximate Pareto front as described in Algorithm 4 is computed for five different values of the incremental budget ∆ ∈ {1, 2, 3, 4, 5}.

Comparative Results

In order to illustrate the results, the cases of the 2-rack topology (20 machines) with 40 and 100 tasks are first considered as examples. For these two cases, Figures 4.4a and 4.4b depict the generated exact and approximate Pareto fronts. Those figures show the different solutions from among which one can be chosen, after running either algorithm. As depicted in the figures, migrating more tasks obviously increases the migration cost -while enabling communicating tasks to be co-located, hence increasing the total achievable throughput. Depending on requirements, a solution can be chosen from the Pareto-optimal set that offers a tradeoff between improving the overall throughput and incurring more migrations. In these two examples, it can be observed that the approximate Pareto fronts generated by algorithm 4 for ∆ ≥ 3 closely follow the exact front. It can also be observed that for ∆ = 1, results are of lower quality, and that in the example of Figure 4.4b, the algorithm was unable to find solutions for more than 24 migrations.

In order to quantify the impact of these algorithms on the quality of the obtained solution set, let θ 20 (respectively θ 0 , θ 30 ) represent the throughput corresponding to a migration cost of 20 (respectively 0, 30) in the Pareto front. More formally, this means that θ 20 is the coordinate such that (|I|, θ 20 , 20) ∈ S; for instance, in Figure 4.4b, θ 20 = 876 for the exact Pareto front. Figure 4.5a (respectively 4.5b) shows the throughput improvement θ 20 /θ 0 (respectively θ 30 /θ 0 ) for the 20 tested scenarios, for the exact Pareto front as well as the approximate Pareto fronts. This represents the relative improvement in terms of overall throughput if the operator chooses to migrate 20 (respectively 30) tasks after inspecting the set of Pareto-optimal solutions. It appears that, for ∆ ≥ 2, the achievable throughput improvement when using the heuristic method introduced in section 4.5.1 is close to the exact solution. Overall, choosing ∆ = 5 yields the better results: for 20 migrations, the results are better in all the twenty tested instances, and for 30 migrations they are outperformed by ∆ = 4 in only three instances. For ∆ = 5, the throughput improvement remains within a 3% error as compared to the exact solution after 20 migrations, and within a 5% error after 30 migrations. Using ∆ = 1 yields results of lower quality, because the algorithm is not able to explore reconfigurations with two migrations at the same time (i.e., swapping tasks), and because it can terminate prematurely if the throughput cannot be improved by migrating only one task. For 20 migrations, ∆ = 3 also yields results of lower quality, but this is more of an artificial nature: the number of migrations considered is 18 instead of 20 (since 20 is not a multiple of 3).

Finally, Figure 4.6 depicts the time needed to generate the exact and approximate Pareto fronts. While computing the exact Pareto front is very time-consuming, the approximation heuristics introduced reduces the required computation time. Overall, the lowest computation times are achieved for ∆ ∈ {3, 4, 5}, whereas ∆ ∈ {1, 2} exhibit higher running times. The choice ∆ = 5, which can be observed to exhibit the best results, in terms of throughput improvement, in the majority of cases -with a computation time constantly better than the reference exact approach, performing approximately one order of magnitude faster when B = 30 (from 5.1× faster for 100 machines and 200 tasks, to 1800× faster for 30 machines and 60 tasks).

To conclude, the approximation method presented in section 4.5.1 can greatly reduce the time needed to compute a set of approximate Pareto optimal solutions, while generating solutions close enough to the exact ones. It is realistic to note that, although an improvement in solving time is observed regardless of the topology size, Figure 4.6 suggests that using a MILP approach (exact or approximate) might be unfit for too large topologies.

Summary of Results

In data centers, applications may exhibit time-varying communications dependencies, and migrating tasks in such environments can be a means to re-distribute the load over the whole infrastructure, while optimizing inter-task communications. In this chapter, a linear programming framework is introduced, which represents the data center topology, and communications dependencies between tasks. Using this framework, a multi-objective model is developed, which, when solved, can propose a reorganization of tasks so as to (i) place the highest number of new tasks, (ii) maximize the overall achievable throughput, and (iii) minimize the migration cost incurred by this reorganization.

The approach developed is generic with respect to the network topology and the inter-application communication demands, and is novel in that, when compared to other workload migration frameworks, it generates a set of Pareto-optimal solutions. From among this solution set, a placement can be chosen, which yields the best tradeoff in terms of throughput improvement versus the cost incurred by task migration. A solution algorithm using the ε-constraint method is proposed as a baseline. Then, a heuristic is presented, which generates an approximate Pareto front. Computational experiments on different topologies show that the proposed approximation builds solutions close to the optimal Pareto front, while greatly reducing the computation time.

Having shown the desirability of considering inter-application dependencies in migration models, an interesting question that arises is how the proposed model can be extended to take into account more fine-grained parameters. These could include the distance between source and destination hosts when migrating, the energy cost induced by machines which are on or off, and the possibility to load-balance flows across multiple paths.

Results from this chapter have been published in [START_REF] Desmouceaux | Flow-aware workload migration in data centers[END_REF].

Part III

Reliable Content Distribution

Chapter 5

Reliable Multicast with BIER Developed alongside their unicast counterparts, multicast protocols were never offered as universally available network services in the Internet [START_REF] Diot | Deployment issues for the IP multicast service and architecture[END_REF] -in part, as the operational complexity of multicast was perceived as exceeding the potential benefits from efficient one-to-many distribution of content. "Native" multicast was therefore, when available, confined to within single networks, or even to within single links -and multicast between sites ("inter-network multicast") was established by way of overlays (e.g., MBONE [START_REF] Eriksson | Mbone: The multicast backbone[END_REF]). The complexity of multicast protocols is in part due to their group-based nature: schematically, when a client wishes to receive messages sent to a multicast group, it will explicitly and periodically send join messages to its "local multicast router" (using IGMP). This router will forward these join message upwards in the multicast tree (using e.g., PIM [START_REF] Nicholas | Protocol Independent Multicast -Dense Mode (PIM-DM): Protocol Specification (Revised)[END_REF]), until reaching the multicast source. Intermediate routers are expected to build, and maintain, flow state (a minima, a multicast tree) for as long as join messages are regularly received, in order to provide connectivity to all members of the multicast group.

Bit-Indexed Explicit Replication (BIER) [8] was designed to eliminate this complexity, and to enable lightweight inter-network multicast -with the ambition being that intermediate routers maintain no flow state, other than that of an existing unicast routing table, and that intermediate routers are not involved in group management. The key idea in BIER, as described in section 1.2.2, is that the source of a multicast data packet encodes the set of destinations (i.e., the members of the group) as a bitstring, and includes this bitstring in the header of each multicast data packet. Intermediate routers only need to be able to interpret that bitstring -leaving group management a matter for only the clients and the source.

Statement of Purpose

The original BIER specification [8] emphasizes the flexibility provided by BIER at the flow level, allowing for adding to and removing from the set of destinations of a flow without the need of building multicast trees and maintaining flow state in the routers. It is possible to take this one step further to modifying the set of destinations on a per-packet basis, and to do so without incurring any additional overhead.

This flexibility can be used to develop efficient reliable multicast: if the source of a multicast data packet is informed as to the set of destinations to which a retransmission is required, it can use BIER for minimizing the traffic footprint of this retransmission: by setting the bitstring so as to contain only the destinations affected by a multicast data packet loss, the retransmission will be forwarded only along the shortest path tree covering the source and these destinations.

This chapter studies this use of BIER for reliable multicast -and compares this "reliable BIER" to two reliable multicast references: (i) the mechanisms known from e.g., NORM [START_REF] Adamson | Reliable messaging for tactical group communication[END_REF], in which the source, when informed about a destination in the multicast group being affected by a multicast data packet loss, will retransmit the multicast data packet to all destinations, and (ii) retransmission by way of unicast(s) to those destinations affected by a multicast data packet loss: this is for instance the case in RMTP [START_REF] Paul | Reliable Multicast Transport Protocol (RMTP)[END_REF], if the number of affected destinations is below a given threshold.

Figure 5.1 illustrates the intuition that when faced with a multicast data packet loss (figure 5.1a), a NORM-style retransmission from the source and to the entire multicast group (naturally) will impose a load on all links in the multicast tree, regardless of if they lead to destinations affected by a multicast data packet loss (figure 5.1b). Unicast retransmissions (figure 5.1c), while traversing only the shortest-path tree between the source and the destinations affected by a multicast data packet loss, may cause the same multicast data packet to be retransmitted across the same link multiple times -whereas BIER utilizes only the shortest path tree between the source and the destinations affected by a multicast data packet loss, with each packet only retransmitted across the same link once (figure 5.1d). This chapter formalizes a simple, reliable, multicast mechanism using BIER, and examines if the suggested intuition holds -and in which conditions. To that purpose, network simulations are conducted, and an analytical model is developed.

(Semi-)Reliable Multicast

While the term "reliable" is used throughout this chapter, and is generally used in the literature, it is perhaps more realistic to describe the attained multicast network services as semireliable. For example, maintaining a retransmission buffer (regardless if centralized at the source or distributed/peer-based) indefinitely is hardly feasible -nor will excessive retransmissions necessarily increase the overall success rate across heavily congested paths. Therefore, this chapter will not consider mechanisms whereby a source adapts its sending rate to the worst destination; rather, it will assume that the source sends a stream at a fixed rate (e.g., a live broadcast media stream), and that destination applications might decide to give up on certain packets -if they are behind heavily congested links, and/or if they no longer would need the packet after retransmission.

Related Work

While never widely deployed as an inter-networking service, several reliable multicast protocols have been developed [START_REF] Popescu | A survey of reliable multicast communication[END_REF]. Reliable multicast protocols assume the existence of a multicast tree, to which are added (i) detection, (ii) reporting, and finally (iii) repairing of packet losses -the latter, through successful retransmission of lost packets. These mechanisms differ, depending on how reliability is understood, on the intended set of receivers, on transmission requirements, and on which trade-offs are acceptable for a given usage [START_REF] Atwood | A classification of reliable multicast protocols[END_REF].

Loss estimation and recovery can be handled exclusively at the source, such as with the "xPress Transfer Protocol" (XTP) [START_REF] Strayer | XTP: the Xpress Transfer Protocol[END_REF], a sender-reliable protocol (using acknowledgements and retransmission timers) designed for small sets of receivers. For larger multicast groups, the "Reliable Multicast Protocol" (RMP) [START_REF] Whetten | A high performance totally ordered multicast protocol[END_REF] pushes responsibility for detecting losses and requesting retransmissions to the receivers, through NACKs. "Log-Based Receiver-reliable Multicast" (LBRM) [START_REF] Holbrook | Log based receiver reliable multicast for distributed interactive simulation[END_REF] uses a log server for caching packets sent by the source, which also reacts to requests for retransmissions. A hierarchical architecture is suggested: a destination that has not received a packet will first solicit retransmission from a local log server -and only if that fails, will solicit a primary log server. With the "Reliable Multicast Transport Protocol" (RMTP), [START_REF] Paul | Reliable Multicast Transport Protocol (RMTP)[END_REF] proposes a hierarchical architecture, wherein intermediate routers in the multicast tree will contribute retransmissions in case of isolated losses, but with global recovery handled by the source. [START_REF] Floyd | A reliable multicast framework for light-weight sessions and application level framing[END_REF] introduces "Scalable Reliable Multicast" (SRM), which uses receiver-based reliability (receivers detect losses and request retransmissions) combined with low-rate multicast by every member to report the highest sequence number received. The "Tree-based Multicast Transport Protocol" (TMTP) [START_REF] Yavatkar | A reliable dissemination protocol for interactive collaborative applications[END_REF] is another instance of a reliable multicast protocol, wherein destinations are grouped in a tree of different domains, within which local recovery can be performed.

The IETF standardized "Negative-acknowledgment Oriented Reliable Multicast" (NORM) [START_REF] Bormann | NACK-Oriented Reliable Multicast (NORM) Transport Protocol[END_REF], using NACKs and source-based retransmissions to attain reliability. NORM also proposes redundancy and recovery by way of Forward Error Coding (FEC) -either proactively, or in response to negative acknowledgements. A TCP-friendly congestion-control mechanism has been proposed in [START_REF] Macker | A tcp friendly, rate-based mechanism for nack-oriented reliable multicast congestion control[END_REF]. The IETF has also examined the "Pragmatic General Multicast" (PGM) protocol [START_REF] Gemmell | The PGM reliable multicast protocol[END_REF], a hierarchical receiver reliable protocol, targeting large multicast groups. Losses are reported via unicast NACKs and recovery can be performed by retransmission from a Designated Local Repairers (i.e., not necessarily from the source), also via multicast. This is similar to RMTP, in which retransmissions are performed by Designated Receivers -though RMTP supports both unicast and multicast retransmissions, depending on the number of reported losses of a given packet.

In the context of data-centers, [START_REF] Li | Reliable multicast in data center networks[END_REF] proposes the end-host based protocol "Reliable Data Center Multicast" (RDCM): through a central controller, RDCM explicitly builds a multicast tree, and a multicast-tree-aware backup overlay, for data dissemination. Retransmissions are performed on a peer-to-peer (unicast) basis: every receiver is responsible for providing, if needed, retransmissions for up to two of its peers.

The performance of reliable multicast protocols has been studied both analytically and through network simulations. For example, [START_REF] Bhagwat | Effect of topology in performance of reliable multicast communication[END_REF] develops an analytical model and carries simulations to study the performance of a generic reliable block-based multicast protocol using stop-and-wait, positive acknowledgements, and selective retransmissions. This model quantifies the number of transmission attempts until full reception, assuming independent losses in different links. [START_REF] Nonnenmacher | Reliable multicast: Where to use FEC[END_REF] investigates the optimal placement of FEC in reliable multicast trees, by way of studying generic models of such trees (i.e., a single path common to all receivers, a set of completely separate paths to each receiver) and a refinement of the model developed in [START_REF] Bhagwat | Effect of topology in performance of reliable multicast communication[END_REF]. The number of successful receptions for different types of trees is studied by way of analysis and simulation in [START_REF]Performance modelling of reliable multicast transmission[END_REF], which also derives a generic approximation for the expected number of transmissions for reliable delivery.

Finally, models relying on TCP overlays for multicast have also been developed: [START_REF] Baccelli | The one-to-manv TCP overlay: A scalable and reliable multicast architecture[END_REF] introduces the One-to-Many TCP Overlay for reliable multicast services, as an application-level multicast alternative to IP reliable multicast; [START_REF] Basin | Source of instability in data center multicast[END_REF] studies the performance of TCP-based reliable multicast trees, built as a set of reliable point-to-point links, in data-centers.

Chapter Outline

The remainder of this chapter is organized as follows: section 5.2 details the use of BIER for light-weight, reliable multicast. Section 5.3 and section 5.4 evaluate the performance of this reliable multicast mechanism (denoted reliable BIER) in different topologies and with different losses, by way of network simulations -and compare the performance with other reliable multicast mechanisms. Generalizing the observations from the network simulations, section 5.5 provides an analytical study of reliable BIER performance. Finally, section 5.6 concludes this chapter.

Reliable BIER -Specification

To provide reliable multicast as a network service transparent to applications, reliable BIER is designed to operate as a shim-layer above the network layer, as depicted in figure 5.2. This reliable BIER shim layer is upper-layer agnostic, and as such supports both standard transport layers (such as UDP) and encapsulation mechanisms such as IP-in-IP.

The reliable BIER shim layer assumes a unique flow ID from the upper layer, and maintains for each flow ID a sequence number, monotonically increased for each new packet being handed by the upper layer. The tuple (flow ID, sequence number) allows uniquely identifying each original multicast data packet in the network, identifying when a multicast data packet is received out of order, etc. For many transport layers, the flow ID would be a hash of the tuple (protocol number, source IP address, source port), whereas for more specific transport layers a transport-layer-specific identifier could be used (e.g., the session ID for QUIC [START_REF] Hamilton | QUIC: A UDP-Based Secure and Reliable Transport for HTTP/2[END_REF]).

For each outgoing multicast data packet, the reliable BIER shim layer takes a destination set, rather than a single destination IP address, from the upper layer -and compresses this into the destination-bitstring. The destination-bitstring and the sequence number comprise the reliable BIER header, to be included in each multicast data packet. The precise form of the reliable BIER header has no algorithmic significance -but in an IPv6 context can, for instance, take the form of a destination options extension header.

For each incoming multicast data packet, the reliable BIER shim layer at each destination will inspect the sequence number to detect losses, and signal losses by way of sending negativeacknowledgements (NACKs) from destinations towards the source. Having this signaling only involve the end-points (source and destinations) serves, in part to run the protocol over any "standard" set of BIER routers doing best-effort forwarding1 , and in part to facilitate deployment (only the end-points need to agree on parameters, for example). As in other reliable multicast protocols, e.g., [START_REF] Adamson | Reliable messaging for tactical group communication[END_REF][START_REF] Gemmell | The PGM reliable multicast protocol[END_REF], NACKs are used in order to avoid an "ACK storm". Of course, for very large error rates or errors affecting a wide range of destinations, this may lead to a "NACK storm". Such a storm of control traffic could be avoided e.g., by allowing intermediate routers to aggregate NACKs before forwarding them upwards, without changing the end-to-end behavior specified in this chapter.

For the purpose of this chapter, a slightly modified socket API is used -specifically, allowing the sender to provide the set of destinations (rather than a single multicast group address) to which a multicast data packet is to be forwarded.

Source Operation

A reliable BIER source operates as detailed in algorithm 5. On sending a packet through a socket, the reliable BIER shim layer caches a copy of the packet, with which it associates a BIER retransmit bitstring with all bits cleared, and a timer of duration ∆t agg , where ∆t agg represents a window of time between the first NACK is received and a retransmission is made. Within this window, for each NACK received, the corresponding bit in the BIER retransmit bitstring is set; at the end of this window, the cached multicast data packet is retransmitted with the destination-bitstring set to the associated BIER retransmit bitstring. This permits aggregation of retransmissions to multiple destinations in a single BIER packet, thus potentially reducing the number of transmissions of this packet. If after retransmission, a subsequent NACK for the same packet is received, a new ∆t agg window is opened and the aggregation mechanism is restarted.

p C[S] ← p, R[S] ← 0, T [S] ← ∞ delete C[S -w], R[S -w], T [S -w] ⊲ garbage collection S ← S + 1 for s ∈ C with T [s] ≤ Tnow do ⊲ perform retransmits p ← C[s] ⊲ retrieve cached packet insert reliable BIER header with flow F , seq s insert BIER header with bistring R[s] transmit p R[s] ← 0, T [s] ← ∞ end for end for for each received NACK packet with flow F , seq s, bit b do if s ∈ C then R[s] = R[s] OR 2 b ⊲ add b to the retransmit bitstring T [s] ← min{T [s], Tnow + ∆tagg} ⊲ schedule retransmit end if end for

Destination Operation

A reliable BIER destination operates as described in algorithm 6. In short, a destination will send a NACK to the source when it detects that a packet was lost -a packet being deemed lost when one of its successors is received2 . If necessary, NACKs for a lost packet are then retransmitted regularly by way of a timer, until a retransmission is received.

More precisely, for each incoming multicast data packet, the reliable BIER shim layer parses the reliable BIER header and either hands it off to the upper layer, or (if received out-of-order) records it in a buffer, C. For each multicast data packet that (1) is received out-of-order, and (2) creates a "hole" (i.e., a set of missing packets between two consecutively received packets) in C, the reliable BIER shim layer adds the element(s) corresponding to this "hole" in the list of lost packets, L. Each element in L is identified by the sequence number s of the corresponding lost packet, and is associated with a timer T [s], and a NACK count N [s].

For each element of L, a NACK is sent towards the source. The NACK contains a reliable BIER header wherein the included bitstring indicates the bit of the client sending the NACK3 , and the sequence number corresponding to the lost packet. Then, the NACK count is incremented, and a new timer for this packet is set to expire after ∆t retry (a configurable retry delay). Upon timer expiration, if no retransmission has been received, and if the NACK count is below a configurable limit ℓ, another NACK is sent and the process is restarted. When the retransmission count reaches ℓ, it is assumed that the source is not able to offer timely retransmission (for instance, due to congestion on the path), and the destination gives up trying to request. This achieves a "poorman's congestion control", by limiting the number of possible retransmission of a multicast data packet.

S = n then ⊲ received in-order packet L ← L \ {S}, delete T [S], N [S] ⊲ un-schedule NACK transmit p to application n ← n + 1 else if S > n then ⊲ received out-of-order packet if S / ∈ C then ⊲ cache packet and un-schedule NACK C[S] ← p L ← L \ {S}, delete T [S], N [S] end if ⊲ schedule NACK

Parameter Discussion

The value of ∆t agg directly influences the global behavior of reliable BIER:

• when ∆t agg = 0, no aggregation is performed, and the protocol degenerates to individual, unicast-based, retransmissions;

• when ∆t agg > 0, aggregation is enabled: the greater ∆t agg , the greater the probability of aggregating retransmissions, but at the cost of a greater delay for recovery;

• when ∆t agg = RT T max -RT T min , maximum aggregation is enabled; higher values of ∆t agg will not provide further benefit.

Determination of the value of ∆t agg thus requires taking into account RTT variations and error probability along the different paths, as well as the sensitivity of the application to delay: it thus corresponds to a policy decision. Furthermore, to make sure that each destination sends at most one NACK for each multicast data packet (re)transmission failure, ∆t retry should be greater than ∆t agg + RT T max .

Data-Center Simulations

Regardless of the underlying network topology, content delivery with BIER from a given source will follow (but not construct) a shortest-path tree. Thus for this first set of simulations, reliable BIER is tested on a simple tree-topology modeling a data-center, depicted in figure 5.3: a core router, connected to two aggregation routers -each of which is connected to two Top-of-Rack routers (ToR), and with each rack hosting 10 machines.

The purpose of the set of tests in this section is to examine if the intuition introduced in section 5.1, and depicted in figure 5.1, holds: that using BIER (rather than multicast or unicast) for retransmissions can yield a measurable and significant diminution of the traffic footprint.

To this end, three different scenarios are constructed around the same physical topology depicted in figure 5.3. These scenarios serve to explore how reliable BIER performs both when losses are spatially located and when they are not, specifically: Uncorrelated localized losses, where background traffic is present inside the leftmost rack, i.e., where both the source and destination of the background traffic are members of the leftmost rack, saturating individual links between the ToR switch and the machines in the rack, and thus affecting these machines individually -but with the rest of the data-center unaffected. This scenario is studied in section 5.3.2.

Correlated localized losses, where background traffic is present inside the two leftmost racks, i.e., where source and destination are members of the two leftmost racks. This saturates the incoming links to the two leftmost ToR switches, and thus will affect destinations on all machines within a rack together -again, with the rest of the data-center unaffected. This scenario is studied in section 5.3.3.

Bursty, non-localized losses, where losses are not related to localized background traffic, but are produced by a loss model in each individual link. This scenario is studied in section 5.3.4.

Simulation Parameters and Setup

The BIER shim layer described in section 5.2 has been implemented in ns3 [147], with the bit number → IP-address mapping assumed a priori available in all routers, as discussed in section 1.2.2. UDP is used as transport protocol, and BIER and reliable BIER headers are implemented as IPv6 extension headers. The reliable BIER parameters from section 5.2 are chosen as follows: ∆t agg = 7 ms (NACK aggregation delay), ∆t retry = 15 ms (NACK retransmission timer), and ℓ = 3 (retransmission limit). All links are homogeneous, point-to-point, with 1 Gbps capacity, with an MTU of 1500 octets and a propagation delay of 1 µs. Network interfaces all have taildrop queues of size 512 packets. The links themselves are lossless -except for the simulations in section 5.3.4, which considers link-losses according to a Gilbert-Elliot loss model [START_REF] Elliott | Estimates of error rates for codes on burst-noise channels[END_REF]. For these simulations, the multicast data packet source is attached to the core router, which generates a constant bit-rate reliable BIER flow of 500 Mbps. All 40 machines are destinations for this flow.

This scenario can be considered to represent e.g., broadcasting of a live media, where a constant transmission bitrate has to be sustained, and where a retransmitted multicast data packet is of no value if received "too late". Thus, some packets may not be received by all destinations, and the ratio of packets successfully received after retransmissions (the delivery ratio) will be a metric of interest -as will the network load of the different links in the network, as well as the sum of traffic over all links in the network (the traffic footprint).

When unicast background traffic is introduced in the network (for the simulations in section 5.3.2 and 5.3.3), it takes the form of 19 UDP flows of a constant bit-rate of 500 Mbps. Each flow has a randomly selected source and destination. These flows are injected into the network in a staggered fashion, starting every 200 ms, and each lasting until the end of the simulation. The simulations in section 5.3.2 and 5.3.3 differ in the domain from which the (source, destination) pairs are randomly chosen.

As a reference, reliable BIER (i.e., using BIER for retransmissions, as per this chapter) is compared with multicast (as in e.g., [START_REF] Adamson | Reliable messaging for tactical group communication[END_REF]) and unicast (as in e.g., [START_REF] Paul | Reliable Multicast Transport Protocol (RMTP)[END_REF]) retransmissions of NACKed multicast data packets.

Uncorrelated, Localized Losses

For this set of simulations, UDP background flows are introduced with sources and destinations both within the leftmost rack (figure 5.3) as described in section 5.3.1. This will saturate some of the links between the ToR router and the individual machines, leading to packet losses in the "downwards" interfaces of the leftmost ToR router. BIER aggregation will thus only happen when, by chance, two or more clients detect a packet loss (and thus generate NACKs) at the same time.

Figure 5.4 depicts the results of a 4-second simulation run, specifically the usage of the two core links, the delivery ratio, and the distribution of the number of clients in BIER retransmissions. A first observation from figure 5.4b is that with multicast retransmissions, the rightmost aggregation link carries unnecessary traffic, unlike unicast and BIER retransmissions. It can also be observed that multicast retransmissions saturate the core links faster than the two other mechanisms: this is because excess retransmissions produce additional congestion, leading to additional losses of original transmissions, in turn leading to additional retransmissions.

Comparing with unicast retransmissions, the use of BIER retransmissions further minimizes the traffic footprint: even with uncorrelated losses, retransmissions are aggregated when several clients do not receive the same packet when using BIER. This is illustrated in figure 5.4d, which depicts the Cumulative Distribution Function (CDF) of the number of simultaneous clients to which a BIER retransmission is performed: ≈ 10% of BIER retransmissions are destined for multiple (≥ 2) destinations, and thus benefit from aggregation. This allows a further reduction of link usage, as depicted in figure 5.4b.

A conclusion to draw from these simulations is that in case of localized losses, BIER and unicast retransmissions are preferable to multicast retransmissions -due to the latter incurring unnecessary traffic on links in paths unaffected by losses. Another conclusion is that when multiple destinations do not receive a given multicast data packet, BIER retransmissions allow aggregation -an advantage over unicast retransmissions.

Correlated, Localized Losses

For this set of simulations, UDP background flows are introduced with sources and destinations within the two leftmost racks (figure 5.3) as described in section 5.3.1. This will, again, saturate some of the links -this time, in addition to between the individual machines and the ToR routers, also between the ToR routers and aggregation routers. A loss on one of these links will affect all the destinations within a rack.

Figure 5.5 depicts the results of a 4-second simulation. For the same reasons as in section 5.3.2, multicast retransmissions cause unnecessary traffic on the right "half" of the data-center (see figure 5.3) -and cause earlier saturation on the left core link.

A single lost multicast data packet will, in this scenario, typically fail to be received by several destinations, therefore unicast retransmissions will generate a larger traffic footprint as compared to BIER retransmissions. With a link capacity of 1 Gbps and a multicast flow of 0.5 Gbps, the link between the source and the core router can sustain the unicast retransmission load only when each multicast data packet is, on average, retransmitted no more than once. Beyond that, the link becomes saturated with retransmissions, thus preventing "legitimate" original transmissions to succeed. This explains why unicast recovery incurs a lower delivery ratio, as depicted in figure 5.5c. This is also why the link usage on the first core link is lower for unicast retransmissions than for BIER retransmissions, as depicted in figure 5.5a.

The distribution of the number of simultaneous clients to which a multicast data packet is retransmitted is depicted in figure 5.5d: ≈ 46% of BIER retransmissions are destined for multiple destinations, and thus benefit from aggregation.

Unlocalized, Bursty Losses

The simulations in sections 5.3.2 and 5.3.3 illustrate the benefits of BIER retransmissions when losses are spatially localized. Instead of creating background UDP flows, this section assumes uncontrolled, exogenous congestion in the data center -modelled by a Gilbert-Elliott loss model [START_REF] Elliott | Estimates of error rates for codes on burst-noise channels[END_REF] on all links. This model is used to model bursty transmissions, and [START_REF] Hasslinger | The Gilbert-Elliott model for packet loss in real time services on the internet[END_REF] shows that it accurately describes packet losses in the Internet. In sum, the Gilbert-Elliott loss model prescribes that a link can be in either a good or bad state. In good state, the link is ideal (no losses), whereas in bad state, the probability of a transmission to be successful (i.e., to not be lost) is h. For each packet to be transmitted over a link, the state of the link may change: from bad to good with a probability of r, and from good to bad with a probability p.

For the purpose of the simulations in this section, the success probability in the bad state is set to h = 0.5, and the transition probability from bad to good to r = 0.01 (i.e., the expected loss burst duration is 100 packets). The transition probability from good to bad, p, is set so that the average packet loss ratio is α, according to equation ( 4) in [START_REF] Hasslinger | The Gilbert-Elliott model for packet loss in real time services on the internet[END_REF]: p = rα 1-h-α = 0.01α 0.5-α . In order to quantify the sensitivity of the system to different congestion levels, 19 simulations are run, for α ∈ {0.001, 0.002, . . . , 0.01, 0.02, . . . , 0.1}.

Figure 5.6 depicts the simulation results. A stability limit (α = 6% for BIER and multicast, α = 0.9% for unicast) can be observed in figure 5.6a. Above this limit, retransmissions compete with original transmissions over the link between the source and the core router, causing some of these original transmissions to be lost, thus requiring additional retransmissions. This then causes the delivery ratio to deteriorate, as depicted in figure 5.6c. Thus for this stability metric, reliable BIER shows superior results as compared to unicast retransmissions, while behaving similarly as compared to multicast retransmissions. Figure 5.6b shows the aggregate traffic (the sum of traffic induced by transmissions and retransmissions over all links) for each of the values of α. While unicast retransmissions (in the stability zone) behave slightly worse than BIER retransmissions, multicast retransmissions incur a substantial traffic footprint (of approximately 1.7× that of BIER retransmissions, for α = 6%).

A conclusion to draw from these simulations is that, when losses are unlocalized and bursty, BIER retransmissions are vastly preferable to multicast retransmissions, in terms of global traffic footprint and also vastly preferable to unicast retransmissions, in terms of avoiding saturation of individual links.

Finally, the CDF of the number of simultaneous clients to which a multicast data packet is retransmitted is depicted in figure 5.6d, which shows that when α ≥ 6%, more than 50% of BIER retransmissions are destined for multiple destinations, and thus benefit from aggregation.

Influence of the Aggregation Timer

As described in section 5.2.3, increasing the aggregation timer ∆t agg (i.e., the amount of time during which the source collects NACKs for a given packet before retransmitting) directly reduces the induced network traffic (since aggregating more NACKs means that the corresponding retransmission is sent to more clients), at the cost of packets being delivered later to the application. In order to quantify this phenomenon, an experiment using the scenario of section 5.3.4 is conducted. For two target loss probabilities α = 1% and α = 0.8% (slightly above and below the stability limit observed in figure 5.6a, respectively), different values for the parameter ∆t agg are used, ranging from 25 µs to 10 ms. Figure 5.7a depicts the usage of the link between the source and the core router, averaged above the duration of the simulation, with α = 1%. It is interesting to observe that enabling NACK aggregation (i.e., making ∆t agg = 0) causes a substantial improvement in performance. In particular, the use of any non-zero NACK aggregation delay leads to a significant benefit in terms of link usage: the smallest non-zero explored value (∆t agg = 25 µs) makes the link usage drop from 1 Gbps (with ∆t agg = 0) to 645 Mbps. Further increases of the aggregation delay lead to minor reductions of link usage (e.g., from 645 Mbps for ∆t agg = 25 µs, to 615 Mbps for ∆t agg = 10 ms), at the cost of linearly increasing the delay of retransmitted packets by ∆t agg . Figure 5.7b depicts the results for α = 0.8%: a similar pattern can be observed, with a lower amount of traffic for unicast retransmissions -due to α being below the stability limit.

ISP Topology Simulations

Sections 5.3.2, 5.3.3, and 5.3.4 illustrated the benefits of reliable BIER for reducing the traffic footprint in strict tree topologies such as those from data centers -begging the question of if these benefits are dependent on these topologies. In order to answer that question, this section presents simulations of a real topology, specifically that of BT Europe (Aug. 2010) from [START_REF]Internet Topology Zoo[END_REF] (see figure 5.9). Note that while the topology used comes from a real deployment, this simulation does not claim to reproduce realistic Internet traffic: the goal is to explore the behavior of reliable BIER.

This topology consists of 24 routers, connected by 1 Gbps links. It is assumed that the unicast routing protocol has converged and each router has perfect shortest paths to all other routers. For the purpose of this simulation study, the source node is attached to router 17 (in London), and a destination is attached to each of the other 23 routers. The simulation parameters and multicast traffic flow parameters are as per section 5.3.1. Background traffic flows are also as per section 5.3.1, noting that router 17 is never chosen as source or destination for a background flow.

Figure 5.8a depicts the link usage between the source and router 17 -revealing that unicast retransmissions rapidly saturate the link, and that multicast retransmissions saturate the link faster than BIER retransmissions. Figure 5.8b depicts the link usage of the link between router 17 (to which the source is attached) and router 5 (one of its directly adjacent peer routers), noting that it is up to twice as high for multicast retransmissions as it is for both BIER retransmissions and for unicast retransmissions. The link load for unicast retransmissions on this link is lower than for reliable BIER. The reason for this is that, as unicast retransmissions already saturated the link between the source and router 17 (figure 5.8a), fewer unicast retransmissions make it onto the link between router 17 and router 5 (figure 5.8b), causing an overall lower data delivery ratio when using unicast retransmissions. This is depicted in figure 5.8c, which also indicates a higher data delivery ratio for BIER retransmissions than for multicast retransmissions.

The distribution of the number of simultaneous clients to which a multicast data packet is retransmitted is depicted in figure 5.8d: ≈ 71% of BIER retransmissions are destined for multiple destinations, and thus benefit from aggregation.

The conclusion to draw from these simulations is that the results obtained with a data-center topology (section 5.3) also can be valid for other topologies.

Reliable BIER Performance Analysis

The simulations in sections 5.3 and 5.4 illustrate the performance benefits of BIER-retransmissions for reliable multicast, both when faced with (i) rare, isolated losses, and with (ii) correlated, frequent losses in traffic-intensive environments. Specifically, reliable BIER was observed to result in a substantially lower traffic footprint in the simulated scenarios, with equivalent or better multicast data packet delivery ratios than when using multicast and unicast retransmissions.

This section aims at generalizing these observations by way of formulating an analytical model of arbitrary tree topologies, and of using this model to analytically quantify the number of successful and failed transmissions4 necessary for a reliable multicast operation to succeed (i.e., for all destinations to have received a copy of a multicast data packet). Section 5.5.2 derives an exact expression of this as M B

[i] , for reliable BIER -and for comparison, M m [i] and M u [i] for when using multicast and unicast retransmissions, respectively.

These exact expressions, however, become mathematically intractable for large trees, thus section 5.5.3 develops a first-order approximation of the average traffic footprints of reliable multicast using BIER, multicast, and unicast retransmissions, respectively.

Model, Assumptions and Definitions

Network links have an associated packet loss probability5 α ∈ [0, 1]. As illustrated in section 5.1, multicast transmissions and retransmissions (regardless of if BIER, multicast, or unicast retransmissions) span a tree, rooted in the source. For describing these trees, the following notation is introduced: routers and destinations are indiscriminately termed node, and each node is uniquely labeled with the path from the root of the tree to it6 ; [[i], j] denotes the j-th child of node [i], and the term "the subtree [i]" refers to the subtree which is rooted in node

[i]. Finally, the set of children of [i] is denoted c([i]): c([i]) = {[[i], 1], [[i], 2], . . .}
This analysis assumes retransmissions by the source until all destinations have received a copy of the multicast data packet, and that the source collects all generated NACKs before retransmitting a packet (i.e., ℓ = ∞, no NACKs are lost, ∆t agg ≥ RT T max -RT T min , ∆t retry ≥ ∆t agg + RT T max ). It quantifies, under these assumptions, (1) the number of retransmissions of a multicast data packet that are made by the source, and (2) the total number of transmissions in the network, until all clients have received (at least) one copy of the multicast data packet.

[i]

[[i], k] [[i], j]
transmission attempts

X [i] j T [i] X [i] k M [i] [[i], k] M [[i], j] M Parent of [i] successful transmissions

• M ⋆

[i] : is the number of packets transmitted inside a subtree [i] to ensure that all destinations receive a copy, where ⋆ indicates the considered variant (B for BIER, m for multicast, u for unicast). If [i] is a leaf, then by convention M ⋆

[i] = 0.

The number of attempts by [i] is the worst of the number of transmissions on all links ([i], [[i], j]):

T [i] = max [[i],j]∈c([i]) X [i]→j (5.1)

Computation of T

[i] , X [i]→j , and M ⋆ [i] Each node [[i], j] needs to receive T [[i]
,j] copies of the multicast data packet from its parent, [i]. For each of these, the number of transmissions over the link ([i], [[i], j]) until the copy is successfully received at [[i], j] is geometrically distributed with success probability (1α), which leads to the following proposition: Proposition 5.1. The total number of transmissions over the link

([i], [[i], j]) follows a negative binomial distribution with (random) parameter T [[i],j] . For x ≥ k ≥ 1: P[X [i]→j = x|T [[i],j] = k] = x -1 k -1 α x-k (1 -α) k (5.2)
and the number of attempts from node [i], T [i] is:

P[T [i] = k] = [[i],j]∈c([i]) P[X [i]→j ≤ k] - [[i],j]∈c([i]) P[X [i]→j ≤ k -1] (5.3)
Equations (5.2) and (5.3) allow computing the probability density function (PDF) for X [i]→j and T [i] recursively from the leaves towards the root, using the convention that T [i] = 1 for a leaf.

X [i]→j ) is M B [i] = 11.
BIER retransmissions:

M B [i]
When using BIER retransmissions, M B [i] is the sum of the transmissions on each link ([i], [[i], j]) and the packets transmitted in each subtree [[i], j], i.e., :

M B [i] = [[i],j]∈c([i]) (X [i]→j + M B [[i],j] ) (5.4)
Figure 5.11 provides a detailed example of a BIER reliable transmission, with the corresponding values for T

[i] , X [i]→j and M B [i] displayed in Fig. 5.11d. Multicast retransmissions: M m [i]
The number of multicast data packets sent over a network with multicast retransmissions can be obtained by adapting the previously presented model (section 5.

U [i]→[[i],j] = T [i] -X [i]→j f =1 (1 + Y f [i]→j F f [[i],j] ) (5.5)
T [i] at each node [i], total transmissions X [i]→j at each link [i] → [[i], j],
and the total number of unnecessary packets

U [i]→[[i],j] flooded by each node [i] in the subtree [[i], j].
The total number of packets sent in the tree (the sum of all

X [i]→j and U [i]→[[i],j] ) is M m [1] = 16.
where the mean number of packets sent in a multicast flood from

[i], E[F [j] ]
, can be recursively computed as follows (with E[F [j] ] = 0 for every leaf [j]):

E[F [i] ] = [[i],j]∈c([i]) (1 + (1 -α)E[F [[i],j] ]) (5.6)
From this, the total number of packets sent in a subtree [i] until all of its destinations receive a copy, M m

[i] , can be computed recursively. It corresponds, for each child [[i], j], to the number of transmissions over the link [i] → [[i], j] required by [[i], j], plus the number of packets sent inside [[i], j] so as to cover all destinations, plus the unnecessary multicast packets originating from [i]:

M m [i] = [[i],j]∈c([i]) X [i]→j + M m [[i],j] + U [i]→[[i],j] (5.7)
Proposition 5.2 indicates a simple way to compute the average traffic footprint for multicast retransmissions, using

T [i] and F [i] .
Proposition 5.2. Let [i] be a node in the tree. With multicast retransmissions, the mean number of packets sent until all destinations in [i] obtain a copy can be computed as such:

E[M m [i] ] = E[T [i] ]E[F [i] ] (5.8)
Proof. by induction. If [i] is a leaf, M m [i] = 0 and F [i] = 0, hence the result holds. Otherwise, let [i] be a node that is not a leaf, and assume that the result holds for all children of [i]. Then, using equation (5.7), and Wald's equation to expand (5.5), it follows that:

E[U [i]→[[i],j] ] from equation
E[M m [i] ] = [[i],j]∈c([i]) E[X [i]→j ] + E[M m [[i],j] ] + E[T [i] -X [i]→j ](1 + (1 -α)E[F [[i],j] ]) = [[i],j]∈c([i]) E[X [i]→j ] + E[T [[i],j] ]E[F [[i],j] ] + E[T [i] -X [i]→j ](1 + (1 -α)E[F [[i],j] ]) = [[i],j]∈c([i]) E[X [i]→j ] + (1 -α)E[X [i]→j ]E[F [[i],j] ] + E[T [i] -X [i]→j ](1 + (1 -α)E[F [[i],j] ]) = [[i],j]∈c([i]) E[T [i] ](1 + (1 -α)E[F [[i],j] ]) = E[T [i] ]E[F [i] ]
where the equality

E[X [i]→j ] = E[T [[i],j] ] 1-α
and equation (5.6) were used.

Figure 5.12 provides a detailed example of multicast reliable transmission, with corresponding values for the variables T

[i] , X [i]→j , U [i]→[[i],j] , and M m [i] .
Unicast retransmissions:

M u [i]
With unicast retransmissions, losses experienced by each destination are treated individually by the source. Given the loss of a multicast data packet (sent by the source [1]) at a destination

E[R [1]→[c] ] = 1 -(1 -α) d([c]) α(1 -α) d([c]) (5
P[Z [1]→[c] = k] = (1 -α) d [1 -(1 -α) d ] k , ∀k ≥ 0 E[Z [1]→[c] ] = 1 -(1 -α) d (1 -α) d
For each of these first

Z [1]→[c] (unsuccessful) attempts, N [1]→[c]
unicast packets will be sent through the chain of links from [1] to [c], where

N [1]→[c]
is distributed as:

P[N [1]→[c] = k] = (1 -α) k-1 α 1 -(1 -α) d , ∀1 ≤ k ≤ d E[N [1]→[c] ] = 1 -αd(1 -α) d -(1 -α) d α(1 -(1 -α) d )
The last (successful) unicast attempt will generate d packets (one per link). Hence, the total number of unicast packets sent until the destination [c] receives a copy,

R [1]→[c]
, is:

E[R [1]→[c] ] = E[Z [1]→[c] ]E[N [1]→[c] ] + d = 1 -(1 -α) d α(1 -α) d
as desired. (unicast retransmissions) transmitted in the binary tree of figure 5.1 until all destinations receive a copy.

The previous proposition allows to compute the total number of multicast data packets sent from the source with the unicast reliability mechanism, M u [1] . This variable corresponds to the multicast data packets sent in the first multicast flood, plus, for each destination that did not receive a copy of the multicast data packet, the number of unicast retransmissions needed until the copy is successfully received. Proposition 5.4 expresses M u

[1] and provides a closed expression for its mean, E[M u [1] ].

Proposition 5.4. Let C be the set of destinations, and let F [1] be the (random) set of destinations that have successfully received a copy after the first multicast flood by [1]. Then, the number of multicast data packets sent from the source, under unicast retransmissions, until each destination has received a copy is:

M u [1] = F [1] + [c]∈C 1 {[c] / ∈F [1] } R [1]→[c] (5.10)
and its mean is:

E[M u [1] ] = E[F [1] ] + [c]∈C (1 -(1 -α) d([c]) ) 2 α(1 -α) d([c]) (5.

11)

Proof. From the definition of F [1] , equation (5.10) holds. The mean number of multicast data packets sent in the network is therefore:

E[M u [1] ] = E[F [1] ] + [c]∈C P[[c] / ∈ F [1] ]E[R [1]→[c] ]
And since

P[[c] / ∈ F [1] ] = 1 -(1 -α) d([c]
) , the result in (5.11) is obtained by using equation (5.9).

Computational Example

For trees of small height, it is possible to recursively derive the average number of transmissions needed in the network in order to deliver a multicast data packet to all destinations, in an exact manner, for BIER reliability (proposition 5.1 and equation (5.4)), multicast reliability (proposition 5.2) or unicast reliability (proposition 5.4). As an example, figure 5.13 reports the results of this computation, for a binary tree of height 2 (as in figure 5.1): as expected, reliable BIER incurs the lowest overhead.

Total Traffic Approximation

Directly computing the traffic footprint is intractable when the depth of the tree is important. Therefore, this section provides a first-order approximation of the expected number of packets

E[M ⋆ [i]
] transmitted in the network before all destinations receive a copy, under the assumption that losses are rare (i.e., α → 0, as in [START_REF]Performance modelling of reliable multicast transmission[END_REF]), for arbitrary trees and for each retransmission mechanism (reliable BIER, unicast retransmissions, multicast retransmissions). This generalizes the model provided in [START_REF]Performance modelling of reliable multicast transmission[END_REF], which provides a first-order approximation of the expected number of source-transmissions E[T [i] ], for multicast retransmissions.

For a link l of the tree, d(l) is the depth of l (d(l) = 1 for a link rooted at the source); L is the total number of links in the tree, and D is the average depth of a links in the tree: D = 1 L l d(l). C is the number of destinations, and ∆ is the average squared depth of a destination: ∆ = 1 C c d(c) 2 . Given these parameters, Theorem 5.1 describes first-order (for α) approximations of the number of transmissions in the network:

Theorem 5.1. The average number of multicast data packets that need to be sent in the tree until all destinations have received a copy are, for reliable BIER (M B

[1] ), for multicast retransmissions (M m [1] ) and for unicast retransmissions (M u [1] ), given by the following approximations when α → 0:

     E[M B [1] ] = L + LDα + O(α 2 ) E[M m [1] ] = L + [L 2 -L(D -1)]α + O(α 2 ) E[M u [1] ] = L + [C∆ -L(D -1)]α + O(α 2 )
(5.12)

Proof. The proof is deferred to section 5.5.5.

When there are no losses (α = 0), the transmission of one multicast data packet yields L packets in the tree (one per link), whichever retransmission mechanism (BIER, multicast or unicast) is used. In addition to these L packets, with BIER retransmissions, the traffic is approximately LDα packets, as compared to approximately L 2 α packets for multicast retransmissions and approximately C∆α for unicast retransmissions (Theorem 5.1). The traffic due to unicast or multicast retransmissions can thus be orders of magnitudes bigger than the corresponding BIER traffic, if the number of links and/or the depth of destinations is important.

When the tree is regular (i.e., each node has the same number of children), it is possible to further simplify these expressions. Let c be the "arity" of the tree, defined as the number of children per node (for instance, a binary tree has c = 2), and h be the height of the tree (where a tree of height 0 is defined to have a single node). With these notations, it is possible to derive the following approximations (when the arity c is fixed) for L → ∞:

     lim α→0 1 α E[M B [1] -L] = Θ(L log L) lim α→0 1 α E[M u [1] -L] = Θ(L log 2 L) lim α→0 1 α E[M m [1] -L] = Θ(L 2 ) (5.13)
This provides a hierarchy between the three studied mechanisms, and shows that BIER retransmissions reduce the traffic footprint from a quadratic behavior in the number of links (with multicast retransmissions) to a log-linear behavior.

Proof. The number of links in the tree can be expressed as

L = h k=1 c k = c h+1 -c c-1
. This allows expressing h as a function of L:

h = 1 log c log 1 + L c -1 c = log L log c + O(1)
Then, the average depths of links in the tree is:

D = 1 L h k=1 kc k = c -1 c h+1 -c × hc h+2 -(h + 1)c h+1 + c (c -1) 2 = h - 1 c -1 + h c h -1 = h - 1 c -1 + c c -1 h L O(1) = log L log c + O(1)
which yields LD = Θ(L log L), as desired to obtain the result for BIER retransmissions. (unicast retransmissions) transmitted in the tree of figure 5 For multicast retransmissions, since LD = Θ(L log L), this yields:

L 2 -L(D -1) = Θ(L 2 )
as desired.

Finally, for unicast retransmissions, the sum of the squared depths of clients is:

C∆ = h 2 c h = log L log c + O(1) 2 1 + L c -1 c = c -1 c log 2 c L log 2 L + O(L log L) = Θ(L log 2 L)
and since LD = Θ(L log L), this finally yields:

C∆ -L(D -1) = Θ(L log 2 L)
as desired, thus concluding the proof.

Discussion

The accuracy and relevance of approximations from theorem 5.1 can be assessed against simulations in realistic tree topologies. Two examples are examined in this section: (1) reliable multicast over a datacenter-like topology as depicted in figure 5.3, and (2) multicast flows over fat-tree-like topologies (introduced in figure 1.3).

The datacenter-like topology of figure 5.3 yields the parameters L = 47, C = 40, D = 177 47 , and ∆ = 16. When α → 0, retransmissions will incur a footprint of 177α packets for BIER, 510α packets for unicast, and 2079α packets for multicast. In order to quantify the quality of the approximation for this example, the means for M B

[1] , M m [1] and M u [1] have been computed over 10 6 random samples, for different values of α with 0 ≤ α ≤ 1%. Figure 5.14 depicts the results of these simulations, as well as the linear approximation from theorem 5.1. For BIER and unicast retransmissions, the approximation accurately fits the computed mean. For multicast retransmissions, the approximation is within a 6% error margin of the computed value.

For reliable multicast flows over fat-tree-like topologies, the root has k children, each having k/2 children, each also having k/2 children. For these trees, the parameters become:

L = k + k 2 2 + k 3 4 , L × D = k + k 2 + 3k 3 4 , C = k 3 4 , ∆ = 9
, allowing calculating the approximation of theorem 5.1. Table 5.1 depicts the approximate retransmission footprint for BIER, multicast and unicast retransmissions. It can be observed that unicast retransmissions exhibit a footprint approximately twice as high as BIER retransmissions. The footprint for multicast retransmissions is at least one order of magnitude higher, and clearly does not scale with the number of clients.

Proof of Theorem 5.1

This section provides a proof of theorem 5.1 and can be skipped at first reading. For this, three lemmas will be needed. Lemma 5.1 first gives an approximation at order 1 in α of the probabilities of having one or two transmissions. Then, lemma 5.2 gives a bound on the corresponding probability distributions, which will be used in lemma 5.3 to show that terms corresponding to three or more transmissions do not contribute to the terms of order 1 in α.

For an arbitrary node [i] in the tree, let l([i]) be the number of links in the subtree rooted at

[i], with l([i]) = 0 if [i] is a leaf.
Lemma 5.1. Let [i] be a node in the tree, and (if

[i] is not a leaf ) [[i], j] be an arbitrary child of [i].
The following approximations hold when α → 0:

P[X [i]→j = 1] = 1 -(1 + l([[i], j]))α + O(α 2 ) P[X [i]→j = 2] = (1 + l([[i], j]))α + O(α 2 ) P[T [i] = 1] = 1 -l([i])α + O(α 2 ) P[T [i] = 2] = l([i])α + O(α 2 )
Proof. by induction over the structure of the tree. If [i] is a leaf, then T [i] = 1 by definition (a client needs one copy of the packet). Otherwise, let [i] be a node that is not a leaf, and assume that the result holds for all children of [i]. Let [[i], j] be an arbitrary child of [i]. Equation (5.2) yields:

P[X [i]→j = 1] = P[T [[i],j] = 1](1 -α) = (1 -l([[i], j])α + O(α 2 ))(1 -α) = 1 -(1 + l([[i], j]))α + O(α 2 ) P[X [i]→j = 2] = P[T [[i],j] = 1]α(1 -α) + P[T [[i],j] = 2](1 -α) 2 = (1 -l([[i], j])α + O(α 2 ))α(1 -α) + (l([[i], j])α + O(α 2 ))(1 -α) 2 = (1 + l([[i], j]))α + O(α 2 )
Then, the definition of T [i] gives:

P[T [i] = 1] = [[i],j]∈c([i]) P[X [i]→j = 1] = [[i],j]∈c([i]) 1 -(1 + l([[i], j]))α + O(α 2 ) = 1 - [[i],j]∈c([i]) [1 + l([[i], j])] α + O(α 2 ) = 1 -l([i])α + O(α 2 ) P[T [i] = 2] = [[i],j]∈c([i]) P[X [i]→j ≤ 2] - [[i],j]∈c([i]) P[X [i]→j ≤ 1] = [[i],j]∈c([i]) (1 + O(α 2 )) - [[i],j]∈c([i]) 1 -(1 + l([[i], j]))α + O(α 2 ) = [[i],j]∈c([i]) [1 + l([[i], j])] α + O(α 2 ) = l([i])α + O(α 2 )
The following lemma provides a geometric bound on the distribution of X [i]→j and T [i] variables, and will be useful to then bound their expectations. Lemma 5.2. Let [i] be a node in the tree, and (if

[i] is not a leaf ) [[i], j] be an arbitrary child of [i]. There exist positive constants A [[i],j] , B [[i],j] , C [i] , D [i]
such that, for all α ∈ [0, 1):

P[X [i]→j = x] ≤ A [[i],j] (B [[i],j] α) x-1 , ∀x ≥ 1 P[T [i] = k] ≤ C [i] (D [i] α) k-1 , ∀k ≥ 1
Proof. by induction over the structure of the tree. If [i] is a leaf, then T [i] = 1 and the result holds with

C [i] = 1, D [i] = 1.
Otherwise, assume that [i] is not a leaf, and that the result holds for all children of [i]. Let [[i], j] be an arbitrary child of [i], and x ≥ 1. Using the induction hypothesis, and the fact that α ≤ 1:

P[X [i]→j = x] = x k=1 P[T [[i],j] = k] x -1 k -1 α x-k (1 -α) k ≤ x k=1 C [[i],j] (D [[i],j] α) k-1 x -1 k -1 α x-k (1 -α) k = C [[i],j] α x-1 x k=1 (D [[i],j] ) k-1 x -1 k -1 (1 -α) k = C [[i],j] α x-1 (1 -α)[1 + D [[i],j] (1 -α)] x-1 ≤ C [[i],j] α x-1 [1 + D [[i],j] ] x-1
The result for

X [i]→j follows, with A [[i],j] = C [[i],j] and B [[i],j] = 1 + D [[i],j] .
The result for T [i] remains to be proven. For t ≥ 1:

P[T [i] = k] = [[i],j]∈c([i]) P[X [i]→j ≤ k] - [[i],j]∈c([i]) P[X [i]→j < k] = [[i],j]∈c([i]) P[X [i]→j < k] + P[X [i]→j = k] - [[i],j]∈c([i]) P[X [i]→j < k]
When developing the first product, a term

[[i],j]∈c([i]) P[X [i]→j < k]
appears, which cancels out with the second product. Remaining terms in the first product are indexed with σ. These terms contain one or more factors of the form P

[X [i]→j = k] where [[i], j] is a child of [i],
P[T [i] = t] ≤ σ P[X [[i],j(σ)] = k] × 1 ≤ σ A [[i],j(σ)] (B [[i],j(σ)] α) k-1
The result for

T [i] follows, with C [i] = σ A [[i],j(σ)] and D [i] = maxσ B [[i],j(σ)] .
Lemma 5.3 provides an approximation of the expectations of X [i]→j and T [i] variables, at order 1 in α, using lemmas 5.1 and 5.2. Lemma 5.3. Let [i] be a node in the tree, and (if [i] is not a leaf ) [[i], j] be an arbitrary child of [i]. The following approximations hold when α → 0:

E[X [i]→j ] = 1 + (1 + l([[i], j]))α + O(α 2 ) E[T [i] ] = 1 + l([i])α + O(α 2 ) Proof. First, it will be shown that +∞ x=3 xP[X [i]→j = x] = O(α 2 )
. By summing the inequalities in lemma 5.2, and provided that α is small enough (α < 1/B [[i],j] ), it is possible to write:

+∞ x=3 xP[X [i]→j = x] ≤ A [[i],j] B 2 [[i],j] +∞ x=0 (x+3)(B [[i],j] α) x α 2 = A [[i],j] B 2 [[i],j] 3 -2B [[i],j] α (1 -B [[i],j] α) 2 α 2 = O(α 2 )
Then, using lemma 5.1, E[X [i]→j ] can be approximated as:

E[X [i]→j ] = P[X [i]→j = 1] + 2P[X [i]→j = 2] + +∞ x=3 xP[X [i]→j = x] = 1 -(1 + l([[i], j]))α + 2(1 + l([[i], j]))α + O(α 2 ) = (1 + l([[i], j]))α + O(α 2 ) which concludes the proof for E[X [i]→j ]. The proof for E[T [i] ] is similar.
This allows proving theorem 5.1 for BIER reliability. In the following, D([i]) denotes the sum of depth of links in the tree rooted at [i]:

D([i]) = l∈[i] d(l).
Theorem (Traffic footprint for BIER). Let [i] be a node in the tree. The following approximation holds when α → 0:

E[M B [i] ] = l([i]) + D([i])α + O(α 2 )
Proof. by induction over the structure of the tree. The results holds for leaves, because

M B [i] = l([i]) = D([i]) = 0 by definition. Otherwise, let [i]
be a node that is not a leaf, and assume that the result holds for the children of [i]. Then:

E[M B [i] ] = [[i],j]∈c([i]) E[X [i]→j ] + E[M B [[i],j] ] = [[i],j]∈c([i]) [1 + (1 + l([[i], j]))α + l([[i], j]) + D([[i], j])α + O(α 2 )] =   [[i],j]∈c([i]) 1 + l([[i], j])   +   [[i],j]∈c([i]) 1 + (D([[i], j]) + l([[i], j]))   α + O(α 2 ) = l([i]) + D([i])α + O(α 2 )
In the last sum, the first term corresponds to the link from [i] to a child [[i], j], and the second term corresponds to the sum of depths of all links in the subtree rooted at [[i], j] incremented by 1, i.e., the depth as counted from the root [i].

A proof of theorem 5.1 for multicast reliability can now be expressed.

Theorem (Traffic footprint for multicast). The following approximation holds when α → 0:

E[M m [1] ] = L + L 2 -L(D -1) α + O(α 2 )
Proof. Let d(l) be the depth of a link l as seen by the root (a link between the root and one of its children having depth 1). Using equation (5.6), it is possible to write:

E[F [1] ] = l (1 -α) d(l)-1 = l 1 -(d(l) -1)α + O(α 2 ) = L -L(D -1)α + O(α 2 )
Combining proposition 5.2 and lemma 5.3 yields:

E[M m [1] ] = E[F [1] ]E[T [1] ] = (L -L(D -1)α + O(α 2 ))(1 + Lα + O(α 2 )) = L + [L 2 -L(D -1)]α + O(α 2 )
which concludes the proof.

Finally, the following proves theorem 5.1 for unicast reliability.

Theorem (Traffic footprint for unicast). The following approximation holds when α → 0:

E[M u [1] ] = L + [C∆ -L(D -1)] α + O(α 2 )
Proof. As in the proof for multicast reliability, the first term in equation (5.11) can be approximated as:

E[F [1] ] = L -L(D -1)α + O(α 2 ).
Hence, the whole expectation can be approximated as:

E[M u [1] ] = E[F [1] ] + [c]∈C (1 -(1 -α) d([c]) ) 2 α(1 -α) d([c]) = E[F [1] ] + [c]∈C α 2 d([c]) 2 + O(α 3 ) α = L -L(D-1)α + O(α 2 ) + [c]∈C d([c]) 2 α + O(α 2 ) = L + [C∆ -L(D -1)] + O(α 2 )
concluding the proof.

Summary of Results

This chapter has proposed a scalable network service offering efficient and reliable multicast. NACK-based, this network service uses BIER (Bit-Indexed Explicit Replication) for ensuring that traffic (both original transmissions and retransmissions) are forwarded over a minimal shortest path tree, requiring maintenance of neither per-flow nor per-group state by intermediate routers: the source will encode, for each (re)transmission of a multicast data packet, the precise destination set -be that every member of a given group, or those members having issued a NACK to request retransmission.

The performance of this network service is compared with "classic" reliable multicast mechanisms, where retransmissions are either unicast (to all destinations having sent a NACK) or multicast to all destinations in a given group (when a NACK for a multicast data packet was received from any destination).

Simulation studies in both data-center-like and in Internet-like topologies, and when faced with different loss models, show that the proposed BIER-based reliable multicast network service is able to achieve reliability, while overcoming the two main shortcomings of these reference mechanisms:

(i) contrary to multicast reliability, links not concerned by losses are not affected by retransmissions and (ii) contrary to unicast reliability, links concerned by losses do not unnecessarily carry multiple copies of the same packets.

Generalizing from the simulation studies, an analytical model is presented, which quantifies the retransmission footprint incurred by the three mechanisms in any topology -and which shows that the BIER-based reliable multicast network service incurs a consistently lower overhead.

Results from this chapter have been published in [START_REF] Desmouceaux | Reliable multicast with B.I.E.R[END_REF].

Chapter 6

Reliable BIER with Peer Caching

As the size and complexity of Data-Center Networks (DCNs) [START_REF] Bilal | Quantitative comparisons of the state-of-the-art data center architectures[END_REF] and Content Distribution Networks (CDNs) [START_REF] Pathan | A taxonomy of CDNs[END_REF] grow, efficient multicast distribution of content becomes increasingly desirable [START_REF] Li | Reliable multicast in data center networks[END_REF][START_REF] Kim | Efficient multicast schemes using in-network caching for optimal content delivery[END_REF][START_REF] Ni | Large-scale cooperative caching and application-level multicast in multimedia content delivery networks[END_REF]. Several protocols have been proposed, which offer reliable multicast delivery services. NORM (Negative-acknowledgement-Oriented Reliable Multicast) [START_REF] Adamson | Reliable messaging for tactical group communication[END_REF], among others, uses sequence numbers in data packets to detect packet losses, and negative acknowledgements (NACKs) which trigger a multicast transmission of the missing packet to the multicast group. Other protocols [START_REF] Paul | Reliable Multicast Transport Protocol (RMTP)[END_REF][START_REF] Li | Reliable multicast in data center networks[END_REF] use unicast retransmissions to those destinations having missed a packet.

As introduced in section 1.2.2, BIER is a multicast protocol which removes the need for flowstate in intermediate routers by allowing compact, explicit, source-based specification of the set of destinations for each multicast packet. In chapter 5, BIER has been extended to offer reliable multicast delivery of content: using sequence numbers and NACKs for detecting and requesting retransmissions of missing packets, retransmissions themselves are made, using BIER, to the exact set of destinations having sent a NACK. This both avoids flooding the whole original multicast tree (figure 6.1a), and prevents duplicate retransmissions over the same link (as would be the case, for unicast retransmissions to each destination, figure 6.1b).

As with NORM, retransmissions in reliable BIER are done by the source (figure 6.1c). However, "local recovery", where retransmissions are performed by neighbors of destinations having missed a packet (figure 6.1d), may be preferable to retransmissions from the source [START_REF] Yavatkar | A reliable dissemination protocol for interactive collaborative applications[END_REF][START_REF] Li | Reliable multicast in data center networks[END_REF][START_REF] Levine | Improving internet multicast with routing labels[END_REF]. For instance, this might be desirable for Quality-of-Experience reasons (e.g., when the source and the destination having lost a packet experience a substantial round-trip delay), or when there are links towards the source whose usage should be limited (e.g., so as to not overload the source, or for traffic engineering or economic reasons).

Statement of Purpose

The purpose of this chapter is to extend the reliable BIER mechanism introduced in chapter 5 to allow a destination having lost a multicast packet to request local retransmission thereof from local peers (i.e., which are topologically close, and part of the destination set for the multicast flow) which may have successfully received a copy thereof -before requesting a retransmission from the source. This is achieved by (i) each destination caching successfully received packets for a small amount of time, and (ii) destinations detecting a packet loss sending a NACK through an ordered set of peers (candidates for retransmission) followed by the source, using Segment Routing.

The advantages of this approach are threefold: (i) the use of peer-based recovery reduces the number of retransmissions from the source, (ii) the use of SR allows a destination to generate a single NACK for a lost packet, which ultimately and automatically will be forwarded to the source if no local retransmissions are possible, and (iii) BIER-based retransmissions (from the source or from any peer) reduce the overall traffic by avoiding both unnecessary duplicate unicast retransmissions across a link close to the source, and multicast floods across the entire multicast tree. With "standard" multicast retransmissions [START_REF] Adamson | Reliable messaging for tactical group communication[END_REF], the source re-floods the whole tree as a result (a). With unicast retransmissions [START_REF] Paul | Reliable Multicast Transport Protocol (RMTP)[END_REF], the source will send multiple copies of the same packet over the same link (b). With BIER retransmissions [START_REF] Desmouceaux | Reliable multicast with B.I.E.R[END_REF], the source re-floods only the subset of failing destinations (c). With peer-based BIER retransmissions introduced in this chapter, the NACKs are sent to a peer that had cached the packet, which then floods the failing destinations, spanning a smaller tree (d).

Related Work

Work related to BIER and Segment Routing has already been reviewed in section 1.2.1 and in section 1.2.2. Different approaches to reliable multicast exist, most of which are not based on BIER, and have been reviewed in section 5.1.1.

Among those approaches, it is noteworthy that some of them use some form of caching, in intermediary nodes or in destinations. In LBRM [START_REF] Holbrook | Log based receiver reliable multicast for distributed interactive simulation[END_REF], a specific node (the log server ) is designated to cache packets from the source. With TMTP [START_REF] Yavatkar | A reliable dissemination protocol for interactive collaborative applications[END_REF], destinations are grouped into different domains, in each of which local recovery can be performed. With RMTP [START_REF] Paul | Reliable Multicast Transport Protocol (RMTP)[END_REF], intermediate routers (designated receivers) can participate in retransmissions; similarly, PGM [START_REF] Gemmell | The PGM reliable multicast protocol[END_REF] allows recovery from designated local repairers. Finally, RDCM [START_REF] Li | Reliable multicast in data center networks[END_REF] uses unicast retransmissions in a backup overlay in which each peer is responsible for recoveries to at most two destinations.

The differences between these approaches and the mechanism introduced in this chapter are threefold: (i) routers are not involved in the caching mechanism, leaving this only to the destinations and therefore not requiring expanding the memory capability of the routing hardware; (ii) retransmissions conducted by peers use BIER and therefore will exclusively target those destinations having missed a packet; and (iii) the use of SR to probe peers for availability of cached packets allows for flexible and pluggable probing policies.

Information-Centric Networking (ICN)

The proposal in this chapter is built on edge-caching and cooperative content management, which has some notions in common with Information Centric Networking (ICN). ICN [START_REF] Zhang | Named data networking[END_REF] is a paradigm where content is stored as named data packets, and where users send interest packets requesting named data packets. As data packets are only identified by names, caching by intermediate routers is possible, and a packet needs only be delivered once per interface, corresponding to a previously-emitted interest. ICN thus shares properties with reliable multicast protocols, by enabling recovery of data from nearby routers.

What is proposed in this chapter differs from ICN in that it (i) assumes push-based multicast applications and (ii) does not rely on routers performing caching, by offloading this task exclusively to the set of destinations -i.e., requires no extensive modifications to routers nor to applications.

Chapter Outline

The remainder of this chapter is organized as follows. Section 6.2 gives a birds-eye view of the proposed extension to reliable BIER, and section 6.3 a detailed specification of how BIER is used to construct a reliable multicast framework which can accommodate diverse policies for selecting peers for local retransmissions. Section 6.4 introduces basic taxonomy and example peer selection policies. Section 6.5 provides a theoretical analysis and discussion of performance and cost tradeoffs for each of these policies, which are experimentally evaluated by way of network simulations in sections 6.6-6.8: section 6.6 introduces the characteristics of the simulation environment, and sections 6.7 and 6.8 describe and discuss the main performance results over both a data center topology and a real ISP topology. Finally, section 6.9 concludes this chapter.

OVERVIEW: RELIABLE BIER WITH PEER CACHING

Overview: Reliable BIER with Peer Caching

In this chapter, the reliable BIER mechanism introduced in chapter 5 is extended to support recovery from peers. Rather than sending a NACK directly to the source to request retransmission of a lost packet, this chapter proposes that a NACK be first sent through an ordered set of peer(s), each of which might be able to provide a retransmission if they have a cached copy of the lost packet (figure 6.1d). Retransmission from the source is solicited as a "last resort". As with reliable BIER, peers can aggregate NACKs before performing a BIER-based retransmission. This locality in retransmissions is expected to reduce delays, as well as to reduce the load on the source, and on its egress links [START_REF] Atwood | A classification of reliable multicast protocols[END_REF][START_REF] Levine | Improving internet multicast with routing labels[END_REF].

Segment Routing Recovery

Requesting retransmission from an ordered list of peers, followed by the source, is done by sending a NACK using Segment Routing (an "SR-NACK"). Each segment will trigger an action which is: (i) if the peer is unable to perform the retransmission, forward the packet to the next segment; (ii) if the peer is able to perform the retransmission, stop forwarding the segment and perform a BIER retransmission (figure 6.1d). This is illustrated in figure 6.2: in (a) an SR-NACK is received by a peer which is able to satisfy the retransmission request; in (b) an SR-NACK is received by a peer which is not able to satisfy the retransmission request, and the next segment is another peer which, then, is able to satisfy the retransmission request; in (c) neither of the two peers receiving the SR-NACK is able to satisfy the retransmission request, and the SR-NACK is therefore forwarded to the source of the multicast packet.

The combined approach thus consists of the source performing an initial BIER transmission of a multicast packet. Destinations receiving the packet may (in addition to processing it) cache it for a short amount of time for possible peer-retransmission. A destination detecting a packet loss (e.g., by receiving a subsequent packet belonging to the same multicast flow) will construct an SR-NACK, containing a number of peers followed by the source. A peer receiving an SR-NACK for which it is able to offer a retransmission will behave as if it was the source in reliable BIER: collect NACKs for this packet for a small amount of time, record the destinations requesting retransmission, and use BIER for retransmitting the packet to exactly the set of destinations from which an SR-NACK was received, when the timer expires.

Peer Caching with Peerstrings

This recovery mechanism is agnostic to the manner in which the set of candidate peers is chosen. If the network operator has instrumented the network in such a way that some peers are "better" candidates for retransmissions (e.g., they are more likely to have cached packets, they are behind less costly or less lossy links, etc.), destinations can be administratively configured to send NACKs to those -with the drawback of requiring instrumentation and configuration. Thus, this chapter introduces a modification to the BIER forwarding plane, allowing destinations to learn about candidate peers.

To this end, an additional bitstring is introduced in BIER headers, henceforth denoted a peerstring. This peerstring is set so as to allow a destination, detecting a packet loss, to identify potential peers from which a retransmission can be requested. A minima, the peerstring is empty, which defaults to requesting retransmission from the source, as in reliable BIER. A maxima, the peerstring contains all destinations (i.e., is a copy of the bitstring as inserted by the source)and any peerstring in between these two extremes is valid.

With the goal of encouraging locality in retransmissions, one simple policy is that, for a given destination, the peerstring contains the set of destinations that share the same parent as itself. This is illustrated in figure 6.3a: when destination 0 receives a packet, the peerstring has a bit set for all destinations which have the same parent as itself (i.e., destination 2). When a router forwards a reliable BIER packet over an interface i, it must, in addition to updating the bitstring for that interface, update the peerstring -essentially setting the peerstring to the union of the bitstrings for all other outgoing interfaces.

An extension to this principle is to include two peerstrings in a data packet received by a destination d: one peerstring indicating destinations with the same parent as d, and a second peerstring indicating destinations with the same grandparent (but not the same parent) as d -as illustrated in figure 6.3b. Thus, this affords more flexibility, but at the expense of more per-packet overhead. This mode of operation will be referred to as the two-peerstrings mode.

Specification

In the proposed BIER extension, the BIER header defined in [START_REF] Wijnands | Encapsulation for Bit Index Explicit Replication (BIER) in MPLS and Non-MPLS Networks[END_REF] is extended to include an additional bitstring, the peerstring, denoted P S1 and described in section 6.2.2. When BIER operates in two-peerstrings mode, the header will also include a second peerstring, P S2. This header is included in all multicast data packets, and is processed by each intermediate router. Each multicast data packet also includes a reliable BIER header, defined in chapter 5, which 6.3. SPECIFICATION 79 conveys flow identifiers and sequence numbers, and which allows detecting lost packets in a flow. This header only carries end-to-end semantics, and is not processed by intermediate BIER routers.

Source Operation

Packet transmission:

The source adds a reliable BIER header to each multicast data packet, containing a flow identifier and a sequence number, as well as a standard BIER header [START_REF] Wijnands | Encapsulation for Bit Index Explicit Replication (BIER) in MPLS and Non-MPLS Networks[END_REF] extended with peerstrings, as described above. The bitstring BS in the BIER header contains the set of destinations receiving packets from within the flow. The peerstring P S1 is set to BS and, if included, P S2 is also set to BS, as illustrated in figures 6.3a and 6.3b. The multicast source also caches a copy of each sent packet during a time interval ∆t s cache . Packet retransmission: When receiving an SR-NACK for a given packet, a source starts a timer ∆t s agg , during which it collects potential subsequent SR-NACKs for the same packet from other destinations. Upon expiration of this timer, a retransmission of the packet is performed, with the set of destinations that have sent a NACK as the BIER bitstring.

Intermediate Router Operation

BIER bitstring processing: BIER packets are processed according to the BIER specification [8]. Only bits corresponding to destinations for which the shortest-path is via interface i are preserved in the bitstring contained in multicast data packets transmitted over that interface i.

Peerstring processing: Upon receipt of a reliable BIER packet with a bitstring BS in , and before forwarding it over interface i (with outgoing bitstring BS out i ), a router must update P S1 and, if included, also P S2. In two-peerstrings mode, first P S2 out i is set to P S1 in . Then, the peerstring P S1 out is set to the OR of the bitstrings sent over all other interfaces (formally, P S1 out i ← ∪ j =i BS out j ) 1 . This way, the P S1 sent over each interface contains the set of those other destinations, to which this router has sent a copy of the packet. Note that the use of bitwise operators to compute peerstrings makes it a simple operation to be implemented in hardware.

Destination Operation

Packet reception: Upon receipt of a packet by a destination, the packet is cached for a duration of ∆t p cache for potential retransmission to a peer. The peerstring(s) of the packet are inspected. The included P S1 is used for updating the set P 1 of "local" peers. If included, P S2 is used for updating the set P 2 , of second-most local peers.

Packet loss: Upon detection of loss of a packet in a given flow (by receiving a packet in the same flow, whose reliable BIER header indicates a higher sequence number), a destination builds an SR-NACK packet. The SR-NACK contains a reliable BIER header with the flow identifier and the sequence number of the requested packet, and its segment list is set to (p 1 , . . . , p n , s), where the p i 's are peers selected from P 1 and P 2 , and where s is the source. Different policies can be used for deciding which peers to include from P 1 and P 2 , and in which order, as described in section 6.4.

While sending the SR-NACK, the destination starts a ∆t d retry timer. When this timer expires, if no retransmission is received, the same SR-NACK is retransmitted -until either the missing packet is received, or a retransmission request limit R lim is reached -after which recovery is aborted. (If semi-reliability is unacceptable, R d lim must be set to ∞.) Packet retransmission: Upon receipt of an SR-NACK, a peer inspects the reliable BIER header of the SR-NACK and extracts the flow identifier and sequence number. If a cached copy of the requested packet is available, it is scheduled for retransmission; otherwise, the SR-NACK is forwarded to the next entry in the segment list (i.e., to the next peer or, ultimately, to the source).

Retransmissions from peers use the same mechanism as those from the source: a timer ∆t p agg is used to collect other NACKs before sending the copy to the set of destinations which have NACKed the packet.

1 P S1 out i ← BS in \ BS out i
is an equivalent way of proceeding, as ∪ j =i BS out j = BS in \ BS out i holds as an invariant.

Peer Selection Policies

As introduced in section 6.3.3, upon missing a packet, a destination will build an SR-NACK with peer(s) extracted from P 1 and P 2 . The framework introduced in this chapter is agnostic to the policy used for selecting those peers. To illustrate this, the remainder of this section suggests examples of simple policies for selection of peers to be included in SR-NACKs: (i) random selection of peers, (ii) clustered selection of peers, and (iii) a simple adaptive (statistically-driven) policy.

Random Peer Selection

This policy builds an SR segment list (p, s) where s is the source, and p is a peer randomly selected from P 1 . Randomly selecting peers from P 1 may increase locality of retransmissions, but rarely allows aggregation of multiple retransmissions into a single BIER packet. As an example, if ten destinations d 1 , . . . , d 10 have the same parent (thus, for each d i , P 1 = {d 1 , . . . , d 10 } \ {d i }), and d 1 , d 2 both detect loss of the same packet, the probability that d 1 and d 2 send an SR-NACK for this packet to the same peer in {d 3 , . . . , d 10 } is 1/8.

Deterministically Clustered Peer Selection

This policy builds an SR segment list (p, s) such that all destinations with the same parent router (i.e., all destinations with the same P 1 ) select the same p. As a convenient convention, for this chapter, all d will select as p the element in P 1 \ {d} with the highest index.

This policy generalizes for two-peerstring mode by building a SR segment list (p 1 , p 2 , s). All d will select as p 1 the element in P 1 \ {d} with the highest index, and as p 2 the element in P 2 \ {d} with the highest index.

While this policy favours aggregation of local retransmissions into a single BIER packet, it does not guard against selecting an unsuitable peer, e.g., a peer located behind a particularly lossy link.

Adaptive Statistically-driven Peer Selection

As long as the constraint that the last segment in the SR segment list for a SR-NACK must be the source is satisfied, any adaptive policy -allowing a destination to observe and "learn" which peers are good candidates from whom to request retransmissions -can be be used for selecting additional peers for inclusion.

Formally, this is an instance of the Multi-Armed Bandit problem: an agent (a destination needing a retransmission) repeatedly activates one of several casino arms (in this case: sends an SR-NACK to a peer) and collects a reward (here: obtains a retransmission or not). The goal is a policy which maximizes the expected reward [START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF][START_REF] Kuleshov | Algorithms for multi-armed bandit problems[END_REF][START_REF] Velmorel | Multi-armed bandit algorithms and empirical evaluation[END_REF].

As an illustration, a simple ε-greedy peer selection policy is employed: with probability ε (ε ≪ 1), a destination detecting a packet loss sends an SR-NACK to a random peer among the set of available peers. With probability (1ε), it sends an SR-NACK to the peer from which it has received the highest number of successful retransmissions so far. The value of ε reflects the trade-off between exploration (contacting random peers and gathering statistics) and exploitation (requesting retransmission from the best known candidate) -between reactivity to changes and performance after convergence.

Policy Analysis

The impact of using peer caching and peer retransmissions, and of each of the policies introduced in section 6.4, can be quantified analytically. Section 6.5.1 provides a basic analysis of the benefits of local retransmissions, and section 6.5.2 derives an analytical model for the random and clustered policies. Section 6.5.3 then explores the benefits from a simple, adaptive policy.

Recovery Locality

The assumption in this chapter is that sending local recovery requests to a local peer is likely to be both "cheaper" than sending the request to the multicast source, and successful -i.e., a "close" neighbour is likely to have successfully cached the requested packet. This section explores the latter of these two assumptions, by quantifying the probability distribution of the distance from a destination having not received a given packet, to the closest peer that has (and is therefore able to perform a retransmission).

A regular tree topology is assumed, wherein inner nodes are intermediate BIER routers, and leaves are destinations. Nodes at a given depth are assumed to have the same number of children, and links at a given depth have the same loss probability. As corresponds to the operation of BIER, the root node of the tree is assumed to be the multicast source.

The tree is of height h, with node ranks indexed by their depth in the tree, from 0 (the source) to h. Similarly, links ranks are indexed from 0 (links from source to first descendants) to h -1. Each node at rank i ∈ {0, . . . , h} has c i children (with c h = 0) and α i is the loss probability of links at rank i ∈ {0, . . . , h -1}. Multicast transmission of a single packet to all leaves is considered. Lemma 6.1 gives the probability that no nodes in a subtree rooted at a given node receives the multicast transmission. Lemma 6.1. The probability b i that a multicast transmission from a node at rank i ∈ {0, . . . , h} does not reach any destination (within its subtree), is:

b i = [α i + (1 -α i )b i+1 ] ci (6.1)
Proposition 6.1. The distribution of D, defined as the shortest distance from an arbitrary destination to a destination having successfully received the multicast transmission can for k ∈ {1, . . . , h} be derived from (6.1):

f D (2k) = h-k-1 i=0 (1 -α i ) (α h-k + (1 -α h-k )b h-k+1 ) × [1 -[(1 -α h-k )b h-k+1 + α h-k ] c h-k -1 ] (6.2)
with:

f D (0) = h-1 i=0 (1 -α i ) f D (∞) = b 0
In proposition 6.1, f D (0) corresponds to the probability that a destination has successfully received the multicast transmission.

Let D denote the distance towards the closest successful destination, as seen from a destination having not received the packet. The probability distribution of D can be computed as the conditional distribution of D (see proposition 6.1) given that the packet is not received (which has probability 1f D (0)), as shown in corollary 6.1. Corollary 6.1. The distribution of D, defined as the minimum distance from a destination that missed a packet to a destination which successfully received the packet is, for k ∈ {1, . . . , h}:

f D (2k) = f D (2k) 1 -f D (0) = h-k-1 i=0 (1 -α i ) 1 - h-1 i=0 (1 -α i ) (α h-k + (1 -α h-k )b h-k+1 ) × [1 -[(1 -α h-k )b h-k+1 + α h-k ] c h-k -1 ] (6.3) with: f D (0) = 0 f D (∞) = b0 1-f D (0) = b0 1-h-1 i=0 (1-αi)
The probabilities α i of loss at each link appear in equations (6.2) and (6.3), allowing to reflect different topology assumptions. In some topologies, loss probabilities may, e.g., be assumed negligible for links close to the source, and more significant for links close to destinations (i.e., α i < α j if i < j). Two models for rank-dependent link loss probabilities are considered:

• Linear increase (lin):

α i = α max i h-1 .
• Exponential increase (exp): It can be observed that the expected shortest distance between a failing destination and a successful peer is lower for the exponential loss model (in which the loss probability for top links is lower) than for the linear loss model. In other words, as expected, if top links are less lossy, it is more likely for a failing destination to be able to recover from a close destination.

α i = α max e i -1 e h-1 -1 .
This confirms the intuition that in networks with this type of loss distribution (such as can be envisioned in data-centers, or in networks where the "last hop" is for instance a wireless link or a consumer grade residential xDSL), selection of local recovery peers (e.g., peers that are in the same subtree or in the immediately upper subtree) should be preferred to selection of peers farther away.

Clustered and Random Policies

The properties and performance of the two simplest (and most "extreme") static peer policies of section 6.4 -random and clustered peer selection -is studied by way of two metrics: the number of recovery successes (performance) and the number of incurred recovery retransmissions (cost). A recovery is successful if a destination detecting the loss of a packet sends an SR-NACK to a peer that previously has received and cached a copy thereof. The number of recovery retransmissions incurring is the number of unique peers, which are selected for retransmission by the set of destinations which have lost that packet. This is because each selected peer will send only one BIER transmission as a response to receiving a set of SR-NACKs for the same multicast packet.

For the remainder of this section, a subtree with n destinations is considered, and recovery of one packet within this subtree is examined.

The following variables are introduced (where X denotes a particular selection policy: R for the random policy and D for the clustered policy):

• K, the number of destinations which did not receive the packet by way of the original multicast transmission from the source;

• S X , the number of recovery successes, i.e., destinations which did not receive the packet by way of the original multicast transmission from the source, but which successfully received a retransmission from a peer, in response to an SR-NACK.

• T X , the number of recovery retransmissions, i.e., of unique peers which received an SR-NACK for a given multicast packet. Figure 6.5 shows an example with n = 10 destinations, K = 4 destinations missing a packet, S X = 3 successful requests (out of 4) and T X = 2 retransmissions. Lemmas 6.2 and 6.3 describe the probability density function (PDF) of the number of recovery successes (S R , S D ) and of the number of retransmissions (T R , T D ), for the random and clustered policies, respectively. For the number of retransmissions with the clustered policy, it is assumed that a peer, from receiving the first SR-NACK and until retransmission of the packet, waits a sufficiently large amount time so as to maximize the ability of aggregating retransmissions into a single BIER-transmission. Lemma 6.2 (Random selection policy). Given K = k destinations (0 ≤ k ≤ n) that did not receive the multicast transmission from the source, the probability that, from among these k destinations, s (0 ≤ s ≤ k) will choose a peer which did receive the multicast transmission from the source,

P[S R = s|K = k] ≡ f S R ,k (
s), is, under the random selection policy:

f S R ,k (s) = k s k -1 n -1 k-s n -k n -1 s (6.4)
Given s recovery successes, the probability of t retransmissions (0

≤ t ≤ s), P[T R = t|S R = s, K = k] ≡ f T R ,k,s (t), is: f T R ,k,s (t) = n -k t t i=0 (-1) t-i t i i n -k s (6.5)
Proof. Consider the transmission of a multicast packet to a cluster of n destinations, from among which k do not receive the packet. Each of the k non-successful destinations then selects a peer at random among (n -1) peers (all peers but itself), and there are (nk) successful peers, thus this selection is successful with probability n-k n-1 . Therefore, the probability that s of the k non-successful destinations sends a NACK to a successful peer follows a binomial law with parameter n-k n-1 , yielding equation (6.4).

To derive equation (6.5), assume that there are s successful recoveries. The number of retransmissions t incurred by these s recoveries corresponds to the number of unique elements sampled when drawing with replacement s elements from a set of (nk) elements. From [START_REF] Mendelson | What is the distribution of the number of unique original items in a bootstrap sample?[END_REF], this is:

1 (n -k) s × (n -k)! (n -k -t)! × s t
where s t (Stirling number of second kind) represents the number of ways to partition s elements into t non-empty subsets, and can be expressed as:

s t = 1 t! t i=0 (-1) t-i t i i s
Combining these two expressions yields equation (6.5), which concludes the proof.

Lemma 6.3 (Clustered selection policy). The probability that s destinations (out of k ≥ 2 destinations that did not receive the multicast transmission from the source) will choose a peer which did receive the multicast transmission from the source,

P[S D = s|K = k] ≡ f S D ,k (
s), is, under the clustered selection policy:

f S D ,k (s) =            n-k n if s = k k n 1 -k-1 n-1 if s = 1 k n k-1 n-1 if s = 0 0 otherwise (6.6)
The probability of having t retransmissions,

P[T D = t|K = k] ≡ f T D ,k (t), is: f T D ,k (t) =      1 -k n k-1 n-1 if t = 1 k n k-1 n-1 if t = 0 0 otherwise (6.7)
As edge cases, when k = 1, there is one recovery/retransmission: S D = T D = 1; and when k = 0, there are no recoveries/retransmissions: S D = T D = 0.

Proof. Consider the clustered policy, whereby a failing destination sends a NACK to a designated peer p, and p itself if failing sends NACKs to another designated peer p * . Consider the transmission of a multicast packet to a cluster of n destinations, from among which k do not receive the packet. Each of the k non-successful destinations then sends a recovery request (NACK) to p (or p * if the non-successful destination is p itself). If the designated peer p is not amongst the k failing destinations (which happens with probability n-k n ), all the recoveries are successful, and SD = k. If the designated peer p is among the failing destinations, but not its retransmitter p * (which happens with probability k n 1 -k-1 n-1 ), then only one recovery request is successful (the one from the designated peer p), and SD = 1. Finally, if both the designated peer p and its retransmitter p * are among the k failing destinations (which happens with probability k n k-1 n-1 ), no retransmission request is successful, and SD = 0. Combining these three cases yields equation (6.6).

To derive equation (6.7), it suffices to note that both cases SD = k and SD = 1 yield one retransmission, and that SD = 0 yields zero retransmissions. From lemmas 6.2 and 6.3 follow two key results, describing the average number of recovery successes (proposition 6.2) and the average number of retransmissions (proposition 6.3 and corollary 6.2). Proposition 6.2. Assuming K = k destinations that did not receive the multicast transmission from the source, the expected number of recovery successes for the random and clustered approaches is the same, and has the value:

E[S R |K = k] = E[S D |K = k] = k(n -k) n -1 (6.8)
Proof. Assuming that k destinations have missed the multicast packet, SR has a binomial distribution with k samples and with success probability n-k n-1 . Thus, E[SR|K = k] = k × n-k n-1 by definition, which proves equation (6.8) for the random policy.

From equation (6.6), the expected value of SD can be computed as:

E[SD|K = k] = k × n -k n + 1 × k n 1 - k -1 n -1 = k n -1 (n -k) which proves that E[SR|K = k] = E[SD|K = k].
Proposition 6.3. Assuming K = k destinations that did not receive the multicast transmission from the source, the expected number of retransmissions for random (S R ) and clustered (S D ) policies have the following expressions:

E[T R |K = k] = (n -k) -(n -k) n -2 n -1 k (6.9) E[T D |K = k] = 1 -k(k-1) n(n-1) if k ≥ 1 0 if k = 0 (6.10)
Proof. Assuming that k destinations have missed the multicast packet, and that s recoveries are successful, the expected number of incurred recovery retransmissions for the random policy is computed by averaging equation (6.5):

E[TR|SR = s, K = k] = s t=0 t × P[TR = t|SR = s, K = k] = (n -k) 1 - n -k -1 n -k s
Then, the average number of retransmissions with no assumption on s can be obtained by combining this result with equation (6.4), yielding (with p = k-1 n-1 to ease notation):

E[TR|K = k] = k s=0 E[TR|SR = s, K = k] × P[SR = s, K = k] = (n -k) -(n -k) k s=0 k s p k-s (1 -p) s n -k -1 n -k s = (n -k) -(n -k) n -2 n -1 k
For the clustered policy, deriving the expected number of retransmissions is done by averaging equation (6.7), and by noting that since the number of retransmissions is either 0 or 1, the expected value is simply the probability that there is one retransmission:

E[TD|K = k] = P[TD = 1|K = k] = 1 -k(k-1) n(n-1) k ≥ 1 0 k = 0 concluding the proof.
The following corollary compares the behavior of both policies when considering that each destination has, independently, the same probability of having not received the multicast transmission from the source. Corollary 6.2. Assuming that each destination has (independently of the others) a probability β ∈ [0, 1] of not having received the multicast transmission from the source (i.e., the probability of having k destinations which did not receive the multicast transmission from the source is binomial with parameter β, and thus

P[K = k] = n k β k (1 -β) n-k
), then the expected number of retransmissions with the random policy grows linearly with the number of destinations, whereas the expected number of transmissions with the clustered policy is bounded:

E[T R ] ∼ n→∞ n(1 -β)(1 -e -β ) = Θ(n) E[T D ] ∼ n→∞ 1 -β 2 = Θ(1) (6.11)
Proof. Assuming that each destination has missed the multicast packet independently with probability β, the number K of failing destinations follows a binomial distribution:

P[K = k] = n k β k (1 -β) n-k (6.12)
For the random policy, it is possible to deduce the expected number of retransmissions (averaged over the failing destinations k) by combining equations (6.9) and (6.12): which is, if n → ∞ and β = 0, β = 1, using Taylor's 1st-term approximation of 1

E[TR] = n k=0 E[TR|K = k] × P[K = k] = (n -nβ) - n k=0 n k β k (1 -β) n-k (n -k) n -2 n -1 k = n(1 -β) 1 -1 - β n -1
-β n-1 n : E[TR] = n(1 -β)(1 -e -β ) + O(1)
The same reasoning can be used for the clustered policy. The expected number of retransmissions becomes, by using (6.10) and (6.12):

E[TD] = n k=0 E[TD|K = k] × P[K = k] = n k=1 n k β k (1 -β) n-k 1 - k n k -1 n -1 = 1 -(1 -β) n -β 2 which is, if n → ∞ and β = 0: E[TD] = 1 -β 2 + o(1)
concluding the proof.

From proposition 6.2, both policies yield the same performance on average, i.e., the expectation of the number of recovery successes is the same for both policies. The clustered policy achieves the same reliability as the random policy by concentrating recovery requests on a single peer, which allows aggregation of retransmissions into a single BIER retransmission. Since aggregation occurs less often in the random policy, in terms of retransmission cost, the random policy is more expensive (linear in the number of destinations vs constant), as shown in corollary 6.2. This difference is only possible because random and clustered policies achieve their (equal) average performance through different probability density distributions, as shown in figure 6.6. In this example, it is assumed that k = 35 destinations out of n = 100 have not received the initial multicast transmission. While with the random policy there is negligible chance that no less than 15 and no more than 30 destinations are able to obtain a retransmission from a peer, the clustered policy operates on an all-or-nothing fashion: all 35 destinations will obtain a retransmission from a peer with high probability -but recovery may be mostly unsuccessful with non-negligible probability (∼ 12% for 0 successes, 23% for only one success over 35).

In other terms, given a Service Level Agreement (SLA) commitment specifying a minimum fraction of destinations (1-δ) (with δ ≪ 1) being served without the need for source retransmission (either because destinations receive the packet in the first BIER transmission from the source or because the first peer recovery request is successful), there is a higher probability that systems are SLA-compliant when using the random policy, than systems using the clustered policy, for small tolerance values (δ > 0.05 in figure 6.7) -and the gap between random and clustered policies is substantial for a non-negligible interval (δ ∈ [0.05, 0.25] in figure 6.7). The probability that a (1δ) fraction of destinations is served without resorting to source retransmissions, under policy X, corresponds to function g n,β,X (δ) described in equation (6.13), and is illustrated in figure 6.7 for n = 100, β = 0.2, and random and clustered policies. It corresponds to:

g n,β,X (δ) = P[S X > K -nδ] = n k=0 P[S X > K -nδ|K = k]P[K = k] (6.13)
where P[S X > x|K = k] can be computed from equations (6.4) and (6.6). When source retransmissions are significantly less preferable (e.g., due to a substantially higher delay incurred) than peer retransmissions, this may translate into a substantial quality degradation of the clustered policy with respect to the random policy. While the gap amplitude is dependent on the value of β (i.e., lower loss probabilities at destination lead to shorter gaps, as it can be expected), the trend shown in figure 6.7 is invariant for values of n and β.

Going Adaptive: the ε-Greedy Policy

For the purpose of this analysis, a binary (c = 2) multicast BIER transmission tree with height h = 7 is considered, so as to illustrate the ability of the simple ε-greedy adaptive policy to learn and adapt to changes in networking conditions, under two different scenarios:

• A static scenario, where all links are lossy (α = 0.1), except for links between the source and destination 0 -i.e., with a static "best" (ideal) peer from which to request retransmission.

• A dynamic scenario which reflects a situation with failure of a "good" peer. Specifically destination 0 is, as in the static scenario and for the same reasons, the "best" (ideal) destination (α = 0 for links from destination 0 and to the source) up until multicast packet number 8000, after which it becomes a "bad" destination behind highly lossy links (α = 0.4). Results of these simulations are reported in figure 6.8, which depicts the ratio of failing destinations choosing a successful peer as retransmission candidate, and figure 6.9, which depicts the number of failing destinations choosing destination 0 as a retransmission candidate2 . It is worth observing that, in these simulations, recoveries are idealized: retransmissions from contacted peers are always successful if the contacted peer holds a copy of the requested BIER packet.

Unsurprisingly, ε = 0 (i.e., choosing deterministically the destination with highest success record) achieves steadier performance than does ε = 0.2 (i.e., choosing random destinations for recovery 20% of the time) in static conditions, as depicted in figure 6.8a. Using ε = 0.2, however, performance is less impacted by the failure of destination 0, and the system adapts faster to the new conditions, as depicted in figure 6.8b. As shown in figure 6.9b, failing destinations when using ε = 0.2 switch to destination 32 quicker, after the failure of former ideal destination 0.

Simulation Environment

The reliable multicast mechanism described in this chapterhas been implemented in ns3 [147] as four components: (i) a BIER forwarding plane (as described in section 6.3.2), (ii) a Segment Routing forwarding plane, (iii) a reliable BIER layer for a source (section 6.3.1), and (iv) a reliable BIER layer for destinations (section 6.3.3).

The reliable BIER layer at the source interfaces with the UDP socket API, to transform UDP multicast packets into BIER packets, while also caching a copy of sent packets so as to be able to retransmit them on receipt of SR-NACKs. The reliable BIER layer at a destination also interfaces with the UDP socket API, collecting received BIER packets before handing them (in-order) over to the UDP socket. This layer also caches a copy of received packets, so as to be able to retransmit them on receipt of a SR-NACK from a peer, and generate SR-NACKs when a packet loss is detected.

The parameters as defined in section 6.3 are: ∆t s agg = ∆t p agg = 7 ms, ∆t p cache = 50 ms, ∆t s cache = 100 ms, ∆t d retry = 15 ms, R d lim = 3. Notably, the value of the SR-NACK retry delay is set significantly greater than the NACK aggregation delay, ∆t d retry ≫ ∆t agg , so that a second NACK is only sent if the source (or peer) has had the chance to send a retransmission and has failed.

Links and Link Loss Model

All links are point-to-point, with 1 Gbps throughput and 1 µs propagation delay, have an MTU of 1500 bytes, and are attached to interfaces with drop-tail queues of size 512 packets. The link loss model used for all links in the simulations is a clock-based Gilbert-Elliott model [START_REF] Elliott | Estimates of error rates for codes on burst-noise channels[END_REF][START_REF] Gilbert | Capacity of a burst-noise channel[END_REF], where the probability of a successful transmission is k = 1 in good state and h = 0.5 in bad state. Transitions from bad to good, and from good to bad, are triggered with exponential clocks, of mean 1/r = 2.5 ms and 1/p = (h-α) αr = (0.5-α) α × 2.5 ms, respectively, where α ∈ [0, 1] is a parameter representing "packet loss probability".

According to [START_REF] Hasslinger | The Gilbert-Elliott model for packet loss in real time services on the internet[END_REF], the probability π B of being in bad state is

π B = p p+r = 1 1+(h-α)/α = α 0.5 , yielding an expected link loss rate of π B (1 -h) + (1 -π B )(1 -k) = 0.5π B = α.
This justifies using α as a parameter to tune the average packet loss probability. This loss model is only applied to multicast data packets -SR-NACKs are not subject to losses, as the path from destinations to the source is supposedly less lossy3 .

Data-Center Simulations

For the purpose of evaluating the mechanism introduced in this chapter, a data-center-like topology is used. This reflects, e.g., distributed storage, distribution of software upgrades, preplacement of content, etc.

Network Topology

The topology used in this set of simulations is as follows, illustrated in figure 6.10:

• a source, "outside" the data-center -reachable across a capacity constrained connection (e.g., a connection incurring higher delays, which has limited throughput, or subject to a higher, congestion-induced, loss probability) is attached to a core router;

• the core router is attached to 2 aggregation routers;

• each aggregation router is attached to 2 Top-of-Rack (ToR) routers;

• each ToR router is attached to 10 machines in a rack (destinations).

Evaluation Objectives

The objective of using SR-NACKs for requesting retransmissions from peers, rather than directly from the source, is specifically not to minimize the number of global transmissions -but rather to maximize the number of retransmissions that can be satisfied locally, i.e., within the data-center. Consequently, one key metric is the number of retransmissions performed by the source -corresponding to the load of the link between the source and the core router (bold link in figure 6.10).

Static Peer Policy -One-Peerstring Mode

To baseline the benefits of locality, the two static peer selection policies introduced in section 6.4.1 (random) and in section 6.4.2 (clustered) are tested in one-peerstring mode: a destination, which detects that a packet has been lost, sends an SR-NACK towards first a "local" peer4 according to the policy, then to the source. These are compared with simple reliable BIER without peer recovery (i.e., wherein NACKs are sent directly to the source). Figure 6.11 depicts the results of 19 four-second long simulations using a 500 Mbps multicast flow (i.e., 166673 BIER-packets generated by the source), for different values of α (see section 6.6) applying uniformly to all links. Figure 6.11a shows the link usage of the ingress link to the core router (averaged over the duration of the simulation). Using simple reliable BIER, the source performs all retransmissions, and this link saturates for α ≥ 7%. With peer-based retransmissions, even with α = 10%, this link remains well below saturation, with a link usage below 680 Mbps. This is detailed in figure 6.11a.

On receiving an SR-NACK, a peer waits for ∆t p agg , to allow receiving SR-NACKs from other peers, before a single aggregate BIER retransmission is made. Thus, the number of destinations in the bitstring is an indicator of the ability to reduce the number of retransmissions. This is depicted in figures 6.11c and 6.11d, for random and clustered peer selection, and for retransmissions made by the source or by a peer.

Retransmissions by the source are required when both the originator of an SR-NACK and the selected peer have not received a given packet -e.g., when the packet was lost over the link between an aggregation router and a ToR router (see figure 6.10), in which case all the destinations below that ToR router would need to receive a retransmission, explaining the "step" at 10 destinations in source bitstrings (figures 6.11c and 6.11d). The similar step around 20 destinations in the source bitstrings is due to either a loss over the link between an aggregation router and the core router, or a packet being lost over two distinct links (between any aggregation router and ToR router). Figures 6.11c and 6.11d also show that random peer selection (expectedly) does not facilitate aggregation, whereas clustered peer selection allows for some -since peers are only selected within the same subtree as that of the originator of the SR-NACK. Thus (i) a peer retransmission bitstring has ≤ 9 destinations, and (ii) if a loss is over a link between the ToR router and an aggregation router, only retransmissions from the source are possible.

Static Peer Policy -Two-Peerstrings Mode

A variation of the clustered peer policy (section 6.4.2) is possible when using two peerstrings. A destination detecting that a packet has not been received will send an SR-NACK towards first the leftmost "local" peer, then the leftmost second-most local5 peer, and finally the source. The objective is, again, to reduce the number of retransmissions needed from the source, at the cost of potentially more total (but local) traffic.

The baseline for this approach is the clustered peer selection policy of section 6.7.3, as well as standard reliable BIER, therefore the same topology, traffic patterns, and link loss model are used. The simulation results for different values of α are depicted in figure 6.12, where figure 6.12a shows that the two-peerstring approach yields a further reduction in the link usage on the ingress link to the core router, due to fewer retransmissions being required by the source. In figure 6.12b, this reduction appears to be from 2.2× less (when α = 0.05) to 3.3× less (when α = 0.02), when comparing to the one-peerstring approach.

Using the two-peerstring mode, retransmissions by the source are required only when neither the originator of an SR-NACK, nor its selected local peer, nor its selected second-most local peer, have received a given packet -e.g., when the packet was either (i) lost over the link between an aggregation router and the core router, or (ii) lost over all of the links from an aggregation router and to the connected ToR routers (figure 6.10). In this case all the destinations below that aggregation router would need to receive a retransmission, which explains the "step" at 20 destinations in source bitstrings in figure 6.12c. The similar, but smaller, step around 10 destinations in the source bitstrings indicates the loss of a packet over only one of the links between an aggregation router and a ToR router. This confirms a greater degree of aggregation (among 20 destinations, rather than 10) of source retransmissions.

Adaptive Peer Policy

To simulate the existence of some local peers being "more suitable" than others, the link loss model is modified such that each ToR router has exactly one destination with a non-lossy ToRto-destination link. An adaptive policy should then enable all other destinations connected to that ToR router to "learn" that this peer is the "most suitable" peer for retransmissions, and therefore, when detecting that a packet has not been received, send an SR-NACK towards first this "most suitable" peer, and only then to the source. To exemplify this, and to examine the intuitions from section 6.5.3, the ε-greedy policy has been implemented and tested against the random peer selection policy6 (section 6.7.3) and standard reliable BIER. In order to allow for sufficient exploration, ε = 0.2 is used.

Simulations are run for different values of α, with results depicted in figure 6.13. As ε-greedy allows directing an SR-NACK towards peers that have a greater chance of being able to retransmit a packet, these are less often forwarded to the source, reducing the number of source retransmissions (figure 6.13b) and the link usage of the ingress link to the core router (figure 6.13a). Finally, when a SR-NACK is received by a "more suitable" peer, its retransmissions are more often successful (as it is connected to its ToR router over a non-lossy link), reducing subsequent retransmissions of that packet. Figure 6.13c visualizes the behavior of the ε-greedy policy through plotting the number of times each peer is chosen by a destination as a candidate in an SR-NACK7 . It can be observed that peers 0, 10, 20, 30 (i.e., the leftmost node of each rack, which is behind a non-lossy link) indeed contribute to most of the recoveries.

ISP Simulations

To validate the genericness of the proposed approach with respect to the network topology, another set of simulations has been conducted, this time in a real ISP topology. As compared to section 6.7, the resulting spanning tree is non-regular. Use cases for reliable multicast in such topologies include video streaming of live events.

Network Topology

As in section 5.4, the topology used for the purpose of this set of simulations is the "BT Europe, August 2010" topology extracted from the Internet Topology Zoo [START_REF]Internet Topology Zoo[END_REF]. The topology is depicted in figure 5.9, and consists of 24 nodes. All nodes are located in Europe, except for nodes 11 (New York) and 12 (Washington DC), which are not visible on the figure, and have each a link to node 17 (London). As this topology features remote (transatlantic) nodes, it is suited to evaluate scenarios wherein it is costly to traverse certain links. As such, the scenario studied in this section will be the transmission of a multicast stream from node 11 (New York) to all other nodes.

Another point of interest of using such a non-regular topology is, that it provides a more natural framework to test adaptive peer selection policies. Indeed, some destination(s) will automatically be both close to the source (thus, more prone to have received packets because there are less links in the path to the source) and close to an important number of other peers. As an illustration, figure 6.15 shows the closeness centrality of each node in the graph, defined as the inverse of the mean distance to the other nodes. It can be observed that nodes 17 (London), 21 (Amsterdam) and 5 (Frankfurt) exhibit the highest centrality, meaning that they are 'local' to a large part of destinations, and would therefore be natural candidates to act as retransmitting peers.

Peer Selection Policies Evaluation

For the purpose of the simulations, a client is attached to each of the nodes, except for node 11 (NYC), to which a source is attached; the Dijkstra algorithm is first run offline to construct routing tables. As in section 6.7, links have 1 Gbps capacity, and the same simulation parameters are used. A multicast flow of 500 Mbps is sent from the source to each of the clients during four seconds, in 19 different simulations, for different values of the average link loss probability α. The random selection policy with one peerstring is used to evaluate the core properties of the mechanism introduced in this chapter. Additionally, the ε-greedy policy is evaluated, as a way to examine the convergence of destinations under adaptive policies towards highly-central nodes as retransmitters. Finally, as a baseline, simple reliable BIER as in chapter 5 is used.

Results are reported in figure 6.14. Figure 6.14a depicts the average usage of the link between the source and the router to which it is attached. Whereas with standard reliable BIER the link is saturated for high values of α (α ≥ 6%), usage of this link is reduced with peer-based BIER retransmissions (the worst case being 720 Mbps for α = 10%), showing that the proposed mechanism allows protecting the source from having to retransmit too many packets. This can be also observed on figure 6.14b, depicting the number of individual packet retransmissions performed by the source: the number of source retransmissions falls by a factor of at least 2.2× when using peer-based retransmissions.

With respect to the comparison between static and adaptive policies, figure 6.14b shows that the number of source-based retransmissions is further reduced when using the adaptive policy. This can be explained by observing, for each peer, the number of packets successfully recovered through this peer, for the static policy and the adaptive policy (figure 6.14c) -the figures display the distribution for the lossiest tested scenario (corresponding to α = 0.1), so as to better visualize the differences. Whereas with the static policy the distribution of retransmitters is relatively uniform, with the adaptive policy it can be clearly observed that peers 17 and 5 (and, to a lesser extend, peer 21) contributes to most of the recoveries. This confirms that retransmissions are handled by peers with high centrality (see figure 6.15), providing some aggregation of the retransmissions (as shown in figure 6.14d).

Summary of Results

This chapter extends the reliable multicast service based on BIER introduced in chapter 5. The proposed extension uses BIER for ensuring that data traffic is forwarded over minimal shortest path trees, and SR-based NACKs for reporting losses and requesting retransmissions. SR-NACKs allow failing destinations to contact an ordered set of peers for local recovery, before requesting retransmission from the source. All retransmissions are performed through BIER-enabled shortest paths.

The proposed protocol is compatible with standard BIER operation (RFC 8279 [8]): no caching is made by intermediate routers, retransmissions are regular BIER packets, and the retransmission logic is handled exclusively by sources and destinations. In addition, a lightweight extension to the BIER forwarding plane is proposed (the processing of a peerstring in the BIER header), allowing destinations to automatically learn about potential candidates from which to ask retransmissions. In absence of this extension, peer-based recoveries can still be requested if destinations are manually configured to do so. By allowing for local repair of multicast failures, the proposed mechanism limits the amount of source retransmissions, and thus their impact in terms of network traffic and delay.

The proposed framework is generic enough to accommodate a broad spectrum of policies for selection of recovery peers, including static, adaptive, and operator-defined. Example of such policies are introduced and analyzed, both analytically and by way of network simulations. Evaluation suggests that substantial benefits in terms of increasing locality of recoveries and reducing usage of costly links can be achieved with relatively simple policies.

Part IV

Load-Balancing

Chapter 7

6LB: Scalable and Application-Aware Load Balancing with Segment Routing

Virtualization and containerization has enabled scaling of application performance by way of (i) running multiple instances of the same application within a data center, and (ii) employing a load-balancer for dispatching queries between these instances.

For the purpose of this chapter, it is useful to distinguish between two categories of loadbalancers:

1. Network-level load-balancers, which operate below the application layer -a simple approach being to rely on Equal Cost Multi-Path (ECMP) [START_REF] Thaler | Multipath issues in unicast and multicast next-hop selection[END_REF] to homogeneously distribute network flows between application instances. This type of load-balancer typically does not take application state into account, which can lead to suboptimal resource utilization.

2. Application-level load-balancers, which are bound to a specific type of application or application-layer protocol, and make informed decisions on how to assign servers to incoming requests. This type of load-balancer typically incurs a cost from monitoring the state of each application instance, and sometimes also terminates network connections (such as is the case for an HTTP proxy).

A desirable load-balancer combines the best of these categories: (i) be application or applicationlayer protocol agnostic (i.e., operate below the application layer) and (ii) incur no monitoring overhead -yet (iii) make informed dispatching decisions depending on the state of the applications.

Furthermore, data centers are more and more utilized to run virtualized network functions alongside traditional applications. In light of this, network load-balancers are more and more running as virtual functions (running for instance in virtual machines). This allows for the loadbalancers themselves to take full advantage of the flexibility and redundancy of a virtualized data center: for resiliency, to allow a faulty load-balancer instance to be safely removed and replaced without incurring a service outage, or for scalability, by growing (and shrinking) the number of load-balancers to be able to accommodate different daily traffic demands and/or unexpected traffic peaks. The resulting architecture will thus distribute incoming flows between an edge router and several load-balancer instances, each of which will then redistribute the flows to application instances [START_REF] Eisenbud | Maglev: A fast and reliable software network load balancer[END_REF][START_REF] Rahman | Load balancer as a service in cloud computing[END_REF]. A challenge arising from this architecture is to provide a consistent service when traffic for a given flow is directed by the edge router to a different load-balancer instance. This can happen e.g., when a load-balancer instance is added or removed, causing the corresponding ECMP mapping between the edge router and the load-balancers to be updated correspondingly.

Thus in addition to the three desired properties exposed above can be added: (iv) be able to be fully distributed, providing the same service regardless of whether traffic is directed to different load-balancer instances within the lifetime of a flow.

These four objectives may appear irreconcilable: operating below the application layer makes it hard to take application state into account, and balancing by state rather than deterministically ties a flow to a given load-balancer instance. This chapter aims at providing a solution satisfying these four objectives, by challenging the traditional network paradigm wherein a packet is deterministically assigned only one destination. 3) allows the flow to re-browse the same set of candidate instances, then to be re-pinned to the one that had accepted the connection (section 7.4).

Statement of Purpose

The purpose of this chapter is to propose 6LB, a load-balancing approach that is applicationinstance-load-aware, yet is both application and application-layer protocol independent and does not rely on centralized monitoring or transmission of application state.

A key argument behind this design goal is that an application instance itself is best positioned to know if it should be accepting an incoming query, or if doing so would degrade performance. Thus, 6LB disregards any design by which queries are unconditionally assigned to an application instance by the load-balancer. Rather, 6LB offers a received query to several candidate application instances, ensuring that exactly one instance accepts and processes the query.

The architecture behind 6LB is as follows, and as illustrated in figure 7.1: the edge router receives and uses ECMP to assign each incoming flow to a load-balancer. Each load-balancer selects a list of candidate application instances, to which it forwards the flow, using Segment Routing (introduced in section 1.2.1). In this way, 6LB enables that flow acceptance decisions are made strictly locally by an application instance, based on its real-time state information about itself, only -i.e., without any centralized monitoring.

Once a flow is accepted by an application instance, it is "pinned" to it by the load-balancer: subsequent packets in that flow are all forwarded directly to that application instance. As depicted in figure 7.3, this is accomplished by the load-balancer inspecting the TCP handshake and establishing a mapping between a flow and the application instance serving it. A mechanism is provided to permit a load-balancer to recover this mapping, if for some reason it is lost1 . The final mechanism in 6LB is consistent hashing, as illustrated in figure 7.2: a given flow will be assigned to the same set of candidate application instances, regardless of by which 6LB instance (past, current, or future, in case the pool of instances changes) it is assigned.

While the algorithms developed in this chapter are generally applicable for any number of candidate application instances, the concept of power of two choices [START_REF] Mitzenmacher | The power of two choices in randomized load balancing[END_REF] applies: selecting two candidate application instances ensures a low network footprint while providing a significant loadbalancing improvement -with diminishing returns beyond that.

In addition to introducing the concept of directing queries through a chain of random candidate application instances with SR and letting them perform local connection acceptance decisions, the contributions of this chapter are: (i) a consistent hashing algorithm, (ii) an in-band stickiness protocol (the union of which allows to scale the number of load-balancer instances for reliability and performance), (iii) an analytic model analyzing the performance of SR-based load-balancing, (iv) experiments with the 6LB load-balancing architecture conducted on a large testbed, and finally (v) kernel-bypass implementations of 6LB and a state-of-the-art load-balancer (Maglev [START_REF] Eisenbud | Maglev: A fast and reliable software network load balancer[END_REF]) with performance comparison of the two implementations.

Related Work

Among existing load-balancing approaches below the application layer, Maglev [START_REF] Eisenbud | Maglev: A fast and reliable software network load balancer[END_REF] and Ananta [START_REF] Patel | Ananta: cloud scale load balancing[END_REF] aim at providing a software load-balancer instance that can be scaled at will, and make use of ECMP to distribute flows between those instances. In addition to a flow stickiness table, they also make use of consistent hashing [START_REF] Karger | Consistent hashing and random trees: Distributed caching protocols for relieving hot spots on the World Wide Web[END_REF][START_REF] Karger | Web caching with consistent hashing[END_REF][START_REF] Thaler | Using name-based mappings to increase hit rates[END_REF], for ensuring that data packets within a given flow are directed to the same application instance -regardless of the selected load-balancer instance forwarding a data packet, and with minimal disruption when the set of application instances changes. However, flows are distributed to application instances regardless of their current load. This is taken one step further in Duet [START_REF] Gandhi | Duet: Cloud scale load balancing with hardware and software[END_REF] and Rubik [START_REF] Gandhi | Rubik: Unlocking the power of locality and end-point flexibility in cloud scale load balancing[END_REF], by moving the load-balancing function to hardware instances, while handing traffic over to software instances in case of failure. Conversely, [START_REF] Handigol | Plug-n-serve: Loadbalancing web traffic using openflow[END_REF][START_REF] Wang | Openflow-based server load balancing gone wild[END_REF] use Software Defined Networking (SDN), with a controller monitoring the application instance load and network load, and then installing network rules to direct flows to these application instances. Spotlight [START_REF] Aghdai | Spotlight: Scalable transport layer load balancing for data center networks[END_REF] allows for weighted ECMP, with weights periodically updated by a centralized monitoring mechanism in order to reflect the current load of applications. Other frameworks, such as Trumpet [START_REF] Moshref | Trumpet: Timely and precise triggers in data centers[END_REF], can be used to gather precise monitoring information on the network load.

Simple application-aware load-balancing policies (random, shortest queue, threshold) have been introduced in [START_REF] Eager | Adaptive load sharing in homogeneous distributed systems[END_REF]; in [START_REF] Mitzenmacher | The power of two choices in randomized load balancing[END_REF][START_REF] Azar | Balanced allocations[END_REF], it is shown that performing a load-balancing decision based on two random servers is sufficient to exponentially decrease the response time as compared to a random strategy. This concept has been used by [START_REF] Shen | Locality-aware and churn-resilient load-balancing algorithms in structured peer-to-peer networks[END_REF] for peer-to-peer applications. Another similar idea, proposed in [START_REF] Dean | The tail at scale[END_REF][START_REF] Ananthanarayanan | Effective straggler mitigation: Attack of the clones[END_REF], consists of duplicating queries among several replicas, and serving the quickest reply to the client. Somewhat similarly, in [START_REF] Aubry | Traffic duplication through segmentable disjoint paths[END_REF], SR is used to duplicate traffic through two different disjoint paths, so as to decrease latency and packet loss.

In [START_REF] Cardellini | Dynamic load balancing on web-server systems[END_REF], three load-balancing techniques are listed, which can be used for dispatching queries among Web servers: DNS round-robin, dispatchers that perform NAT or destination IP rewrite, and redirect-based approaches. Application-aware load-balancing mechanisms for static Web content include [START_REF] Zhang | Flexsplit: A workload-aware, adaptive load balancing strategy for media clusters[END_REF][START_REF] Ciardo | EquiLoad: a load balancing policy for clustered web servers[END_REF][START_REF] Zhang | Workload-aware load balancing for clustered web servers[END_REF], which assign queries as a function of their estimated size so that each application instance becomes equally loaded. In [START_REF] Sharifian | A content-based load balancing algorithm with admission control for cluster web servers[END_REF], a feedback approach is used to estimate the parameters of a queuing model representing the system, before making a load-balancing decision.

Application-layer load-balancers, e.g., HAProxy [START_REF]HAProxy: the reliable, high-performance TCP/HTTP load balancer[END_REF], also propose application-awareness by estimating the load on each application instance and assigning new queries accordingly. Load estimates are obtained either by tracking open connections through the load-balancer to the application instances (and thus do not consider other loads), or by periodically probing the backends for load information (and thus suffer from polling delay and incur network overhead). Another issue with application-level load-balancers is that network connections are reset when a failure causes a flow to be migrated from one load-balancer instance to another. The load-balancer introduced in [START_REF] Gandhi | Yoda: A highly available layer-7 load balancer[END_REF] aims at solving this by keeping per-flow TCP state information in a distributed store.

Chapter Outline

The remainder of this chapter is organized as follows. Section 7.2 describes the use of Segment Routing for performing load-balancing. Section 7.3 describes a consistent hashing algorithm, which allows to distribute the load-balancing function into different instances, for scalability. Section 7.4 describes how "stickiness" can be established and recovered between load-balancer instances and application instances, using an in-band channel. An analytical performance model of 6LB is derived in section 7.5. 6LB is then experimentally evaluated in section 7.6, by way of a synthetic workload and a realistic workload consisting of a Wikipedia replica, and the performance of the implementation in terms of packet forwarding capabilities is evaluated. Finally, section 7.7 concludes this chapter.

Service Hunting with Segment Routing

This section describes how application-load-aware load-balancing can be performed, by way of a concept which will be referred to as Service Hunting.

Description

Service Hunting uses SR to direct network packets from a new flow through a set of candidate application instances until one accepts the connection. It assumes that applications are identified by virtual IP addresses (VIPs), and can be replicated among several instances, identified by their physical addresses. As introduced in section 1.4, machines run a virtual router (e.g., VPP [START_REF] Barach | High-speed software data plane via vectorized packet processing[END_REF]), which dispatches packets between physical NICs and application-bound virtual interfaces. Finally, a load-balancer within the data center advertises routes for the VIPs.

When a new flow 2 for a VIP arrives at the load-balancer, the latter will select a set of candidate application instances from a pool, and insert an SR header identifying this set into the IPv6 data packet. The SR header will contain a list of segments, each indicating that the query can be processed by either of these application instances, and with the VIP as the last segment. Different policies can be used to select the list of candidate application instances to include in the SR header. It has been shown in [START_REF] Mitzenmacher | The power of two choices in randomized load balancing[END_REF] that selecting two random candidate application instances is enough to greatly improve load-balancing fairness, with a decreasing marginal benefit when using more than two instances. Thus, 6LB assigns each new flow to two pseudo-randomly chosen application instances -by way of a consistent hashing scheme, described in section 7.3.

When the flow reaches a candidate application instance, the corresponding segment in the SR header indicates that the virtual router may either forward the packet to the next segment, or may directly deliver it to the virtual interface corresponding to the application instance. This purely local decision of whether to accept a query is based on a policy shared only between the virtual router and the application instance, running on the same machine. To guarantee satisfiability, 2 Typically, a TCP SYN packet as part of a connection request.

SERVICE HUNTING WITH SEGMENT ROUTING

103

SC

Single-Choice policy (baseline)

SRc

Static acceptance policy (Algorithm 7) with threshold c e.g., SR4 is the policy of Algorithm 7 with c = 4 SR dyn Dynamic acceptance policy (Algorithm 8) however, the penultimate3 segment indicates that the application instance must not refuse the query. It can be noted that this mechanism is not without similarities with the one introduced in chapter 3 for achieving zero-loss VM migration, in that the virtual router can decide to "skip" subsequent segments in the list (in chapter 3, a potential locator for a VM; here, a potential host for a workload), based on local decisions.

Connection Acceptance Policies

An application agent running within the virtual router decides whether the local application instance should to accept a flow. The application agent may make this decision based on whatever information it has locally available -from the operating system, or from real-time application metrics, if exposed. If information is exchanged with the server application software through shared memory, this incurs no system calls or synchronization, thereby imposing a negligible runtime cost.

This section describes two simple policies for deciding whether or not to accept new flows. They assume that the application uses a standard master-slave thread architecture. Section 7.6.1 will illustrate the application of these policies, in case of an HTTP server such as Apache. Table 7.1 summarizes the notation used throughout this chapter to designate the different policies.

Static (SR c )

With n worker threads in the application instance, and a threshold parameter c ∈ {0, . . . , n+1}, algorithm 7 describes a policy, SR c , where an application instance accepts the flow if and only if strictly less than c worker threads are busy (except for the last in the SR list, which must always accept). Thus, an application instance which is "too busy" will be assigned a connection only if all previous application instances in the list are also "too busy". The choice of the parameter c directly influences the global system behavior: small values of c yield better results under light loads, and high values yield better results under heavy loads. As extreme examples, when c = 0, all requests are satisfied by the last application instance of their SR lists; when c = n + 1, all requests are satisfied by the first: both cases reduce to a random load-balancing scheme. If the chosen value of c is too small as compared to the load, almost all connections are treated by the last application instance of their SR lists, and vice-versa.

If the typical load is known, the value of c can be configured statically (by using the results of the analysis carried out in section 7.5) -otherwise, a dynamic policy can be employed.

Dynamic (SR dyn )

When the typical load is unknown, the policy SR dyn adapts c to maintain a rejection ratio of each application instance of 1 2 , as detailed in algorithm 8. Previous acceptance decisions are

Protocol Overhead

Inserting an IPv6 SR header to direct a connection through multiple application instances has an impact in terms of packet size overhead. To quantify this, table 7.2 depicts the number of extra bytes needed to direct a packet through two (new flow ) or one (pinned flow ) application instances, for different protocols. As compared to other equivalent solutions allowing to direct a request through a set of instances (by sticking several successive tunneling headers), the proposed approach has the lowest overhead. After flow pinning, the overhead incurred by using SR as a steering mechanism is of 12 bytes as compared to GRE.

Reliability

The solution described in this section (and more generally in this chapter) focuses on the dataplane: it is assumed that a controller takes care of installing the mapping between a VIP and the set of addresses of machines hosting a corresponding application instance. Notably, as in other distributed load-balancing approaches [START_REF] Eisenbud | Maglev: A fast and reliable software network load balancer[END_REF][START_REF] Patel | Ananta: cloud scale load balancing[END_REF], the controller should take care of health-checking the application instances, and removing them from the set of available instances when they are found unresponsive.

Since connection establishment packets go through a chain of instances rather than a single one, the properties of 6LB when facing failures need to be considered. Two scenarios can be distinguished. First, if a whole machine goes down (critical failure), new flows whose first candidate application instance is hosted on this machine will fail to be established, and new flows whose second candidate application instance is hosted on this machine will fail only if the first instance rejected them. This incurs a p R % failure overhead as compared to single-choice load-balancing approaches, where p R is the percentage of connections being rejected by a first instance (e.g., 50% with SR dyn ). However, this happens only during the short amount of time before the controller detects that the machine is down and updates the backend pool on load-balancers accordingly 4 . Second, if an application instance goes down (crashes or becomes unresponsive) but the machine hosting it still remains up (non-critical failure), the virtual router on that machine will be able to forward connection establishment packets to the next instance in the SR list, for new flows whose first candidate instance is failing. Thus, for non-critical failures, 6LB increases the reliability of the system as compared to single-choice approaches, with a (100p R )% failure reduction.

Horizontal Scaling with Consistent Hashing

Elastic scaling of the number of load-balancer instances is required, in order to accomodate dynamic data center loads and configurations [START_REF] Patel | Ananta: cloud scale load balancing[END_REF]. When a load-balancer instance is added or removed, the ECMP function (see figure 7.1) performed by the edge router(s) may rebalance existing flows between remaining load-balancer instances. Thus, it is necessary to ensure that the mapping from flows to lists of candidate application instances is consistent across all load-balancers. This is achieved by the use of consistent hashing, depicted in figure 7.2 -which must also be resilient to modifications to the set of applications instances: adding or removing an application instance must have minimal impact on the mapping of existing flows.

Consistent hashing for load balancing is used for instance in Maglev [START_REF] Eisenbud | Maglev: A fast and reliable software network load balancer[END_REF], which proposes an algorithm mapping an incoming flow to one application instance. This section introduces a new consistent hashing algorithm (generalizing the one from Maglev) to allow each flow to map to an ordered list of application instances. Although 6LB uses 2 choices, the mechanism presented in this section is agnostic to this value, and is therefore presented for lists of C instances.

Generating Lookup Tables

With M buckets and N application instances, and where N ≪ M , a pseudo-random permutation p[i] of {0, . . . , M -1} is generated for each application instance i ∈ {0, . . . , N -1} -e.g., by listing the multiples of the i-th generator5 of the group (Z M , +). These permutations are then used to generate a lookup table t : {0, . . . , M -1} → {0, . . . , N -1} C , mapping each bucket to a list of C application instances, following the procedure described in Algorithm 9. This table t is then used to assign SR lists of application instances to flows: each network flow will be assigned an SR list by hashing its network 5-tuple into a bucket j and taking the corresponding list t[j].

Generating the lookup table t is done by browsing through the set of application instances in a circular fashion, making them successively "skip" buckets in their permutation until finding one that has not yet been assigned C application instances. Once each bucket has been assigned C application instances, the algorithm terminates. This process is illustrated in figure 7.4a, for C = 2 choices, with N = 4 application instances and M = 7 buckets. For each application instance i, the corresponding permutation table p[i] is shown, where a circled number j n means that bucket j has been assigned to that application instance at step n. For each bucket j, the lookup table t[j] returned by the algorithm is also shown. For instance, bucket 3 is assigned to instance 1 (at step 5) and instance 2 (at step 6), thus the lookup table for bucket 3 is (1, 2). The "skipping" behavior occurs e.g., at step 9, where bucket 5 is skipped in p [1][j] because it was already assigned two application instances.

Analysis

Resiliency

t ← [(-1, -1), . . . , (-1, -1)] n ← 0 while true do for i ∈ {0, . . . , N -1} do if nextIndex[i] = M then ⊲ permutation exhausted continue end if c ← p[i][nextIndex[i]] ⊲ advance in i's permutation ⊲ skip buckets for which the SR list is already filled while t[c][C -1] ≥ 0 do nextIndex[i] ← nextIndex[i] + 1 if nextIndex[i] = M then ⊲ permutation exhausted continue 2 ⊲ continue the upper loop end if c ← p[i][nextIndex[i]] end while ⊲ c is now the first bucket with SR list not filled choice ← 0 while t[c][choice] ≥ 0 do choice ← choice + 1 end while ⊲ choice is now the first available position in the SR list t[c][choice] ← i nextIndex[i] ← nextIndex[i] + 1 n ← n + 1 if n = M × C then return t end if
end for end while figure 7.4a. Assuming that flows are assigned to the first or second application instance in their SR lists with equal probability (as with the SR dyn policy), the question is how flows mapped to a non-removed application instance (1, 2, 3 in this example) are affected by the table recomputation. For each bucket, one failure is counted for each non-removed application instance appearing in the lookup table before recomputation, but not after. In the example of figure 7.4, the only failure is induced by bucket 4, as the second entry of its lookup table, 1, does not appear in its newly computed lookup table, (3,2). With 10 non-removed flows, the failure rate in this example is thus 10%.

Intuitively, mapping flows to two application instances, instead of just to one, increases resiliency: it is less likely that the SR lists of a bucket before and after recomputation have empty intersection -for this to happen, a single bucket would need to be re-assigned twice.

The resiliency of algorithm 9 is studied by way of a simulation. An initial lookup table was computed. Then, k application instances were removed and the lookup table was recomputedwhich allowed computing the previously introduced failure rate. The parameters were N = 1000 application instances, M = 65537 buckets, and 20 experiments were performed, for each value of k from 0 to 30.

Figure 7.5 reports the failure rate as a function of the number of removed instances. First, with C = 1 (i.e., mapping each flow onto a single application instance), results identical to in [162, figure 12] are obtained, confirming that the algorithm reduces to the algorithm from [START_REF] Eisenbud | Maglev: A fast and reliable software network load balancer[END_REF] in this case. Using algorithm 9 for mapping each flow to two application instances (C = 2) shows up to 44% fewer failures (when k = 8) -or, to put it differently, 44% fewer TCP connections being reset.

Fairness

Each application instance picks the same number of buckets (as first or second entry), except potentially one in the last round. Assuming a probability of acceptance of 1 2 (as with SR dyn ), this guarantees that traffic is equally spread between application instances. Note that a given application instance is not assigned the same number of first-choice buckets and second-choice buckets; this is nonetheless compensated by the fact that application instances do balance the load between themselves -this is evaluated in section 7.6.2.

Bucket j 0 1 2 3 4 5 6 SR list t[j] (3,1) (1,2) (3,1) (1,2) (3,2) (2,1) (3,2) (b) After removal of Application Instance 0

Complexity

If permutations p[i] are randomly distributed, this algorithm is a variant of the coupon collector's problem, and is expected to terminate in M log M + O(M ) steps for C = 1 [START_REF] William | An introduction to probability theory and its applications[END_REF], and in M log M + M log log M + O(M ) steps for C = 2 [START_REF] Newman | The double dixie cup problem[END_REF]. Hence, choosing two (rather than one) application instances requires only 1 + log log M log M ≤ 1.368 times more steps.

In comparison to the naïve algorithm consisting in building two uncorrelated lookup tables for the first and second application instances in the SR list, the benefit of using Algorithm 9 is twofold: the generation time is smaller, and jointly building the two entries make the scheme more resilient to changes as shown in figure 7

In-band Stickiness Protocol

A load-balancer instance should, for each flow it handles, have knowledge of the application instance which has accepted the flow. First, this allows packets to be directly directed to the handling instance, without hopping through the chain of candidates (thereby reducing triangular traffic). Second, this ensures that, when the consistent hashing table is recomputed (e.g., due to changes in the pool of applications), existing connections are protected against potential changes in the lookup table. Thus, a signaling mechanism is required between the load-balancer and the application instances. Four properties should be satisfied: (i) no external control traffic should be generated, (ii) deep packet inspection should be minimized, (iii) incoming packets should go directly to the application instance handling the flow, and (iv) outgoing packets should not transit through the load-balancer. The latter property, called Direct Server Return (DSR) and introduced in [START_REF] Eisenbud | Maglev: A fast and reliable software network load balancer[END_REF][START_REF] Patel | Ananta: cloud scale load balancing[END_REF], is crucial to the scaling of the load-balancer software: it enables it to treat only client-incoming packets, which are often more lightweight than server-emitted packets (acknowledgements vs data packets).

To satisfy (i) and (ii), SR headers are inserted into packets part of the accepted flow -i.e., a set of SR functions are used for communicating between the load-balancer and the application instance. Objective (iii) is accomplished by having the appliation instance signal to the loadbalancer when it has accepted a flow (by adding an SR header to, typically, the TCP SYN+ACK), and (iv) by making other traffic (packets other than, typically, the TCP SYN+ACK) bypass the load-balancer and be sent directly from the application instance to the client.

SR Functions

SR functions are used to encode actions to be taken by a segment endpoint, directly in the SR header. This is closely linked to how IPv6 addresses are assigned: since each machine is assigned a (typically, 64 bit [START_REF] Deering | IP Version 6 Addressing Architecture[END_REF]) IPv6 prefix, it is possible to use the lower-order bytes in this prefix to designate different functions, as recommended by the SR draft specification [START_REF] Filsfils | SRv6 Network Programming[END_REF]. These functions will also depend on the address of the first segment in the SR list (the "sender" of the function). In practice, when a machine whose physical prefix is m receives a packet with SR header (x, . . . , m::f, . . . ), it will trigger a function f with argument x, which will be denoted by m.f(x). In terms of a state machine, each SR function will thus (i) move the node from one state to another and (ii) trigger an action on the packet containing the SR function.

IN-BAND STICKINESS PROTOCOL

Handshake Protocol

When a (TCP) flow is initiated, SR functions are added to the TCP handshake, so as to inform the load-balancer which candidate application instance has accepted the flow -thus establishing a handshake protocol between the load-balancer and the application instance handling a flow. This handshake protocol is formally described as state machines in tables 7.3 and 7.4, and detailed below:

1. Upon receipt of a flow (typically, a TCP SYN packet) from a client c for an application whose VIP is v, the load-balancer ℓ 1 will insert an SR header (ℓ 1 , m 1 ::ca, m 2 ::cf, v) comprising the physical addresses m 1 , m 2 of the machines hosting the two candidate application instances as given by the hashing function, and the original VIP. The suffix ca in the addresses indicates a function connectAvail, whereas cf represents a function connectForce. The first application instance in the list will make a local decision on whether to accept the flow. In case of refusal, the packet will be forwarded to the second application instance, which will have to forcefully accept the flow.

2. The (virtual router running in the) machine m i (i ∈ {1, 2}) corresponding to the instance that has accepted the connection enters a waiting state for this flow. While in this state, it will temporarily direct traffic from the application towards the load-balancer, so that the latter can learn which application instance has accepted the connection. To do so, it inserts an SR header (m i , ℓ::cs, c) in packets coming from the application, where cs is a createStickiness function.

3. Upon receipt of such a packet, the load-balancer enters a steering state, during which traffic from the client to the application is sent using (ℓ, m i ::as, v) as an SR header, as standing for a function ackStickiness. This permits both steering the traffic directly to the correct application instance, and acknowledging the creation of a stickiness entry.

4. Then, when (the virtual router running in) m i receives such a packet, it enters a direct return state for this flow. As m i has acknowledged the creation of the stickiness entry on the load-balancer, it thus does not need to send traffic through it anymore. Subsequent traffic of this flow will therefore be sent by m i directly towards the client c, without using SR -thus allowing DSR.

5. Finally, when (the virtual router running in) m i receives a connection termination packet from the application (typically, a TCP FIN or RST), it will insert an SR header (m i , ℓ::rms, c), where rms designates a removeStickiness function. This allows explicitly signaling connection termination to the load-balancer. When receiving this packet, the load-balancer will start a small timer, at the expiration of which it will remove the corresponding stickiness entry -using a small timer ensures that packets are correctly directed to the rightful application instance while the transport layer connection teardown is happening. In addition to this explicit connection termination process, periodic garbage collection is used to remove stale entries from the load-balancer.

Failure Recovery

When adding or removing a load-balancer instance, traffic corresponding to a given flow might be redirected to a different load-balancer instance from the one over which it was initiated.

In order to recover state, when a new load-balancer instance receives a flow for which it does not have any state, incoming data packets corresponding to an unknown flow are added an SR header (ℓ, m 1 ::rs, m 2 ::rs, d), where rs is an SR function recoverStickiness. Consistent hashing ensures that {m 1 , m 2 } is the same set as the one used by the previous load-balancer, with high probability 6 . When receiving a packet for this SR function, the application instance that, in the past, had accepted the flow, will re-enter the steering state, so as to notify the load-balancer. Conversely, an application instance that had previously not accepted the flow will simply forward the packet to the next application instance in the SR list.

Performance Analysis

In this section, an analytic model describing the performance of 6LB with the SR c policy (algorithm 7) is derived. By way of this model, 6LB is compared to a Single-Choice random flow assignment approach (SC), which reflects the behavior of standard consistent-hashing approaches (such as Maglev).

System Model

It is assumed that the system contains N application instances, with N → +∞, and that consistent hashing uses enough buckets such that the SR list associated with a flow is uniformly chosen amongst the N 2 possible lists.

In this section, the expected response time for the static acceptance policy SR c described in algorithm 7 is derived, for c an integer threshold parameter. While a similar model has been considered in [190, section 4.4.4], the contribution in this chapter is that metrics of interest to 6LB (namely, the expected response time, the response time with forwarding delay, the fairness index, the probability of wrongful rejection, the response time distribution, and the reduction in number of servers) are derived, numerically computed, and validated against a real deployment experiment.

Incoming flows are assumed distributed according to a Poisson process of rate Λ = N λ, and each application instance offers an exponentially-distributed response time 7 , with a processing rate µ normalized to µ = 1. For stability, the arrival rate must verify λ < 1. For i ≥ 0, s i is the fraction of application instances for which there are i or more pending flows (with s 0 = 1). This allows writing (as also found in [START_REF] Mitzenmacher | The power of two choices in randomized load balancing[END_REF]):

dsi dt = λ(1 + s c )(s i-1 -s i ) -(s i -s i+1 ), ∀1 ≤ i ≤ c dsi dt = λs c (s i-1 -s i ) -(s i -s i+1 ), ∀i > c (7.1)
When i ≤ c, the probability of a flow being sent to an application instance which already handles (i -1) other flows (i) directly is (s i-1s i ) and (ii) after having being rejected by an application instance which already handles c or more flows is s c (s i-1s i ). This yields a total probability of (1 + s c )(s i-1s i ). The same reasoning applies for i > c, where the probability of a flow being sent to an application instance which already handles (i -1) flows (i ≥ 1) is the probability of having been rejected by a first application instance already handling c or more flows, before having been sent to an accepting application instance, yielding s c (s i-1s i ). The probability of a flow leaving an application instance already handling i flows is (s is i+1 ). Since the per-application-instance arrival rate is λ and the processing rate is µ = 1, this yields equation (7.1).

To study the behavior of the system once in equilibrium, it is necessary to find a fixed point to the differential system (7.1). Setting dsi dt = 0 yields the following system of equations (where

s 0 = 1): 0 = λ(1 + s c )(s i-1 -s i ) -(s i -s i+1 ), ∀1 ≤ i ≤ c 0 = λs c (s i-1 -s i ) -(s i -s i+1 ), ∀i > c (7.2)
For 1 ≤ n ≤ c, summing equation (7.2) over i = n to +∞ gives

s n = λ(1 + s c )(s n-1 -s c ) + λs 2 c , or: s n = λ[s n-1 (1 + s c ) -s c ].
Since s 0 = 1, this gives s 1 = λ. More generally, solving this recursion yields:

s n = (1 -λ)(λ(1 + s c )) n -λs c 1 -λ(1 + s c ) , ∀1 ≤ n ≤ c (7.3)
Then, for n > c, summing equation (7.2) over i = n to +∞ gives s n = λs c s n-1 , i.e.:

s n = (λs c ) n-c s c , ∀n ≥ c (7.4)
Plugging n = c into equation (7.3) allows formulating an implicit equation for s c :

s c (1 -λs c ) = (1 -λ)[λ(1 + s c )] c (7.5)
This polynomial equation can be solved explicitly for c ≤ 5, and numerically otherwise. Using this solution in equations (7.3) and (7.4) allows obtaining the value of s n for any n, i.e., the distribution of the number of flows awaiting being handled by an arbitrary application instance.

Expected Response Time

The expected number X of flows in the system can be computed, given that the probability that an application instance handles n flows is (s ns n+1 ), as:

E[X] = N +∞ n=0 n(s n -s n+1 ) = N +∞ n=1 s n .
According to Little's law [START_REF] Little | A proof for the queuing formula: L = λW[END_REF], the expected time T spent in the system by a flow is:

E[T ] = E[X] Λ = 1 λ +∞ n=1 s n . Summing s n ,
as obtained in equations (7.3) and (7.4), and using equation (7.5) gives:

c-1 n=1 s n = λ -λs c (c -1) -s c 1 -λ(1 + s c ) , +∞ n=c s n = s c 1 -λs c
Hence: ). As a reference, this value is compared to 1 1-λ , the expected response time for SC, when clients are randomly assigned to one application instance. It can be observed that the SR c policies uniformly yield an improvement over SC. When c is small, lower values of λ yield the highest gain; when c is important, higher values of λ yield the highest gain. For example, choosing SR 8 offers an improvement over SR 4 only when λ ≥ 0.983, and thus might rarely be suitable.

E[T ] = 1 -cs c (1 -λs c ) -λs c (1 + s c ) (1 -λs c )(1 -λ(1 + s c )) (7.6)

Additional Forwarding Delay

In cases where the intra-data-center (server-to-server) forwarding delay is significant as compared to the job duration, forwarding a query to the second application instance in an SR list incurs an additional cost. If δ > 0 denotes the network delay, multiplying this by the probability of being rejected by a first application instance gives the expected additional delay. Thus, the response time including delay, T , verifies:

E[ T ] = E[T ] + δ × s c (7.7)
The following theorem states the conditions under which SR c does not degrade performance as compared to SC: Theorem 7.1. As long as the network delay δ is smaller than the average job duration 1/µ = 1, the response time including delay with SR c is better than with SC:

E[ T ] ≤ 1 1 -λ , ∀δ ≤ 1
Proof. Let c ≥ 1 a threshold parameter, and λ ∈ [0, 1). First, it will be shown that sn ≤ λ n for all n ≥ 0. For 1 ≤ n ≤ c, sn = λ[sn-1sc(1sn-1)] ≤ λsn-1; for n > c, sn = λscsn-1 ≤ λsn-1. Thus sn ≤ λsn-1 for all n ≥ 1, and since s0 = 1, it follows by induction that sn ≤ λ n . It remains to show that

E[ T ] ≤ 1 1-λ . Let δ ∈ [0, 1], then: E[ T ] = 1 λ +∞ n=1 sn + δsc = 1 λ c-1 n=1 sn + 1 λ sc 1-λsc + δsc. Using sn ≤ λ n , sc ≤ λ c and δ ≤ 1 gives: E[ T ] ≤ 1 λ c-1 n=1 λ n + 1 λ λ c 1-λ c+1 + 1 • λ c = 1-λ c-1 1-λ + λ c-1 1-λ c+1 + λ c = 1 1-λ + λ c-1 ( 1 1-λ c+1 -1 1-λ + λ). Since c ≥ 1, λ c+1 ≤ λ 2 , which yields: E[ T ] ≤ 1 1-λ + λ c-1 ( 1 1-λ 2 -1 1-λ + λ) = 1 1-λ -λ c+2 1-λ 2 ≤ 1 1-λ ,
index F = E[X] 2 E[X 2 ]
. SC vs SR2, SR4 and SR8.

Fairness Index

Jain's fairness index [START_REF] Jain | A quantitative measure of fairness and discrimination for resource allocation in shared computer system[END_REF], defined as

F = E[X] 2 E[X 2 ] ∈ [0, 1]
, is a measure for how even a load is distributed in a system: the closer it is to one, the more evenly is the load distributed.

Computing F requires computing E[X 2 ], the second moment of the number of flows in the system:

E[X 2 ] = N +∞ n=0 n 2 (s n -s n+1 ) = N +∞ n=1 (2n -1)s n
Using equations (7.3), (7.4) and (7.5), this yields:

c-1 n=1 (2n -1)s n = 1 (1-λ(1+s c )) 2 λ 2 (s c + 1)((c-1) 2 s c + 1) + λs c (2cs c -3s c + 2c -c 2 -2) -2cs c + λ + s c , +∞ n=c (2n -1)s n = s c (c(2 -2λs c ) + 3λs c -1) (1 -λs c ) 2
Combining those expressions with the expression for E[X] from section 7.5.2 allows to compute F . Figure 7.8 depicts F , for different SR c policies, and for λ ∈ [0, 1) -and compares with λ 1+λ , the fairness index for the SC policy. It can be observed that SR c policies provide a better fairness than the reference SC policy, and that low values of c are more suitable for a low rate of new flow arrivals whereas high values of c are preferable for higher rates of new flow arrivals.

Wrongful Rejections

As has been shown in section 7.5.3, using the SR c policy yields better performance than SC, because an overloaded application instance can offload a query to another random instance. However, the proposed mechanism is not fully equivalent to the canonical "power-of-two-choices" scheme [START_REF] Mitzenmacher | The power of two choices in randomized load balancing[END_REF], wherein the least loaded of two random instances is chosen. Suboptimal decisions happen when a first instance handling n ≥ c flows rejects the connection to a second instance with strictly more than n flows. It is possible to estimate the quantity of such wrongful rejections: the probability of hitting such a pair of instances is (s ns n+1 )s n+1 . Using equation (7.4), the probability p w of wrongful rejection can be expressed as:

p w = +∞ n=c (s n -s n+1 )s n+1 = λs 3 c 1 + λs c (7.8)
In order to quantify this, figure 7.9 shows the probability of wrongful rejection for different SR c policies. For example, with SR 4 , wrongful rejections happen with probability lower than 4.5% when λ ≤ 0.9.

Response Time Distribution

The model also allows deriving the distribution of the time that a flow exists in the system. Knowing this distribution allows, for example, characterizing the performance of 6LB for Service Level Agreement (SLA) metrics, of the form "No more than x% of clients should experience a response time ≥ y". The distribution of the time T a flow waits, will be derived by computing its characteristic function ϕ T (θ) = E[e iθT ] (for θ ∈ R). Assume that the system is at its equilibrium given by equation (7.2), and that application instances use a FIFO policy with exponential response times. When a flow is being directed to an application instance that is already handling (k -1) flows (k ≥ 1), its waiting time will be distributed as the sum of k independent and identically distributed (i.i.d.) exponential random variables (E 1 , . . . , E k ) of parameter µ = 1. The characteristic function of one such variable is

E[e iθE1 ] = 1 1-iθ , hence E[e iθT |k -1 clients] = 1 1-iθ k .
When a flow arrives at the system, it will, with probability (

1 k≤c +s c )(s k-1 -s k ) = 1 λ (s k -s k+1
), be directed to an application instance which is already handling (k -1) other flows. Based on this, it is possible to express the characteristic function of the waiting time of an arbitrary flow:

E[e iθT ] = +∞ k=1 1 λ (s k -s k+1 ) 1 1 -iθ k
Using equations (7.3), (7.4) and (7.5), this can be expressed as:

E[e iθT ] = (1 -λ)(1 + s c ) 1 -λ(1 + s c ) -iθ - s c (1 -λs c ) (1 -λs c -iθ)(1 -λ(1 + s c ) -iθ)(1 -iθ) c-1 (7.9)
which can be inverted to find p T , the probability density of T , using p T (t) = 1 2π +∞ -∞ e -iθt E[e iθT ]dθ. Figure 7.10 depicts the CDF of this probability distribution, for λ = 0.88, and for various SR c policies. At this high load, the distribution for the SR c policies exhibit lower response times and less variance as compared to SC.

Integrating this probability density allows finding π x , the x-th percentile of response time, defined as the number satisfying:

P[T ≤ π x ] = πx 0 p T (t)dt =
x 100 (7.10) Figure 7.11 depicts π 90 , the 90-th percentile of response time, for various SR c policies and for λ ∈ [0, 1), and is compared to ln (10) 1-λ , the same metric for the SC policy. Similarly as in figure 7.6, the response time with SR c is lower than with SC, and small values of c are more suitable for low request rates.

Reducing the Number of Application Instances

The developed model allows estimating the gain, in terms of how many fewer application instances are required to attain a certain SLA, when using 6LB as compared to when using "plain" SC. Assume that a system faces a daily request rate profile with a peek rate Λ 0 , and that the goal is to provide a given SLA µ 0 on the 90-th percentile of response time: no more than 10% of clients should receive a target response time greater than µ 0 (i.e., π 90 = µ 0 ). With a simple SC load-balancer, the system faces a normalized request rate of λ 0 = Λ 0 /N , and the 90-th percentile of response time is π 90 = ln (10) 1-Λ0/N -thus, requiring deploying N = Λ0 1-ln(10)/µ0 application instances to meet the SLA.

As per equation (7.10), let π(λ) be the function giving the 90-th percentile of response time π 90 as a function of λ, when using 6LB with the SR c policy. In order to meet the SLA, i.e., to ensure that π(Λ 0 /N ) = µ 0 , N ′ = Λ0 π -1 (µ0) application instances must be deployed. Comparing SC and 6LB with SR c yields:

N ′ N = 1 -ln(10)/µ 0 π -1 (µ 0 ) (7.11)
Figure 7.12 depicts this reduction in number of application instances, as a function of the target SLA µ 0 , between SC and SR 4 . If the SLA requires that no more than 10% of clients experience a response time greater than e.g., µ 0 = 6, then if that is met by a deployment of e.g., N = 100 application instances when using SC, only N ′ = 71 application instances will be required if using 6LB with the SR 4 policy.

Evaluation

This section describes an evaluation of 6LB on real software. Section 7.6.1 introduces the implementation and experimental perform; then, section 7.6.2 reports results of tests conducted on a synthetic workload, and section 7.6.3 on a realistic workload.

Experimental Platform

The experimental platform used for evaluating 6LB is composed of a load-balancer and a server agent for the Apache HTTP server.

Load-Balancer

The load-balancer performing consistent hashing, SR header insertion and flow steering is implemented as a VPP plugin [START_REF] Barach | High-speed software data plane via vectorized packet processing[END_REF]. Having kernel-bypass capabilities and embedding an IPv6 Segment Routing stack, VPP is a suitable choice to build a performing implementation. As a reference, Maglev [START_REF] Eisenbud | Maglev: A fast and reliable software network load balancer[END_REF] was also implemented to evaluate the single-choice consistent hashing flow assignment policy SC.

Apache HTTP Server Agent

A server agent for the Apache HTTP server [START_REF]The Apache HTTP server project[END_REF] has been implemented as a VPP plugin, accessing Apache's scoreboard shared memory8 to allow the virtual router to access the state of the application instance. Apache uses a worker thread model: a pool of worker threads is started in advance, and received queries are dispatched to those threads. Thus, a simple exposed metric is the state of each worker thread, allowing to count the number of busy/idle threads, and use this to decide on connection acceptance, using one of the policies described in section 7.2.2.

System platform

The experiments described in sections 7.6.2 and 7.6.3 are conducted on a common platform. An edge router and two load-balancer instances are deployed as 2-core VMs residing in one physical machine. N = 48 application instances of an Apache HTTP server reside each in a 2-core VM, all of which are hosted across 4 physical machines (distinct from the one hosting the edge router/loadbalancers). The edge router is configured to split traffic for the application across the two loadbalancer instances, by way of ECMP, as in figure 7.1. VPP instances running in the edge router VM, in the load-balancer VMs, and in each of the VMs of the application instances, are on the same Layer-2 link, with routing tables statically configured. Each physical machine has a 24-core Intel Xeon E5-2690 CPU. The size of the consistent hashing table of the load-balancer instances was set to M = 65536 (except for the experiments of figures 7.17 and 7.18), and the Apache servers were configured to use the mpm prefork module, each with 32 worker threads and with a TCP backlog of 128.

The tcp abort on overflow parameter of the Linux kernel was enabled, triggering a TCP RST when the backlog of TCP connections exceeds queue capacity, rather than silently dropping the packet and waiting for a SYN retransmit. Thus under heavy load, it is application response delays that are measured, and not possible TCP SYN retransmit delays.

Poisson Traffic

Traffic and Workload Patterns

To evaluate the efficiency of the connection acceptance policies from section 7.2.2 under different loads, 6LB was tested against a simple CPU-intensive web application, consisting of a PHP script running a for loop with an exponentially distributed number of iterations, and whose duration is 190 ms in average. Using such a distribution ensures that job durations exhibit reasonable variance (the standard deviation of the exponential distribution is equal to its mean). A traffic generator sends a Poisson stream of queries (HTTP requests), with rate λ. A bootstrap step consisted of identifying λ 0 , the maximum rate sustainable by the 48-instances farm, i.e., the smallest value of λ for which some TCP connections were dropped.

Connection Acceptance Policies Evaluation

With ρ = λ/λ 0 as the normalized request rate, for 20 values of ρ in the range (0, 1), a Poisson stream of 80000 queries with rate ρ was injected in the load-balancers, using the policies SR 4 , SR 8 , SR 16 , and SR dyn . As baseline, the same tests were run with a policy SC where queries are pseudo-randomly assigned to one application instance, without Service Hunting, using the single-choice consistent hashing algorithm of Maglev [START_REF] Eisenbud | Maglev: A fast and reliable software network load balancer[END_REF].

Figure 7.13 depicts mean response times for each tested request rate and for each policy, and show that, among those, SR 4 yields the best response time profile, up to 2.3× better than SC for ρ = 0.87. SR 8 and SR 16 likewise perform better than SC for all loads, but with a lesser impact. SR dyn offers results close to the best tested static policy. In order to validate the analytical model introduced in section 7.5, the response time as obtained from equation (7.6) is displayed in dotted lines alongside the experimental results9 : it can be observed that the model accurately fits the data, as long as ρ < 0.9. After that, the assumptions (steady state, infinite number of servers) do not hold anymore. Figure 7.15 shows the CDF of the page response time for the 80000 queries batch with ρ = 0.89, for each policy. SC exhibits a very dispersed distribution of response times, whereas the different SR c policies yield lower, and less dispersed, response times. This can be explained by inspecting the evolution of the mean instantaneous load (the number of busy worker threads) over all application instances, as well as the corresponding fairness index:

( 48 i=1 xi(t)) 2 48 48
i=1 xi(t) 2 (where x i (t) is the load of server i at time t), depicted in figure 7.14 10 . As SR 4 better spreads queries between all servers (the fairness index is closer to 1), and servers are individually less loaded, better response times result.

For lighter loads, a similar behavior can be observed, except that SR c policies for high values of c exhibit no benefits as compared to SC. Figure 7.15 shows the CDF of the page load time for an experiment where ρ = 0.71: SR 16 yields no improvement over SC, and SR 8 yields a relatively small improvement, however the SR 4 policy provides a substantial improvement in response times -and SR dyn remains able to successfully match SR 4 , the best tested static policy.

Reducing the Number of Application Instances

Previous experiments have shown how 6LB is able to yield a reduced page response time, for a given request rate. Conversely, if an SLA on the target response time is to be satisfied, 6LB can be used to decrease the number of application instances needed to reach that SLA. In order to quantify this, a simple experiment has been conducted to find out how many VMs can be shut off while achieving a pre-defined SLA. Assume that the 48 VMs were deployed with SC to attain an average response time of 0.58 s, i.e., that the application faces a total request rate of ρ = 0.71 (values taken from figure 7.13). Using the same request rate, a batch of 80000 requests was ran against less and less VMs with the SR 4 policy, until the same average response time was reached. Figure 7. [START_REF] Lapukhov | Use of BGP for Routing in Large-Scale Data Centers[END_REF] shows the average response time as a function of the number of VMs: with 6LB, 40 VMs are needed to meet the same SLA as compared to 48 VMs with SC -a reduction of 17%.

Influence of the Consistent Hashing Table Size

Using a smaller hash table can be beneficial in environments with tight resources, but at the cost of evenness in the distribution of the application instances within first segments of the SR lists (as explained in section 7.3). In order to quantify this, a Poisson stream of 80000 requests with request rate ρ = 0.71 was sent to the load-balancers against the SR 4 policy, using different hash table sizes. Figure 7.17 shows the average response time as a function of the table size used (tables have sizes 2 k for performance reasons). The response times are almost identical for high table sizes, with a noticeable influence when M ≤ 1024. Also, except when M ≤ 128, the average response time stays lower than when using the SC policy with the same rate.

Consistent Hashing Resiliency

The resiliency of the consistent hashing mechanism introduced in Algorithm 9 in real conditions is evaluated through a simple experiment, where a simultaneous change in the application instances pool and in the load-balancer pool is introduced. With M = 4096 buckets in the consistent hashing tables, 1000 long-lived flows are injected in the system and handled by the first 6LB instance. Then, x application instances are removed, while at the same time the ECMP router is reconfigured to use the second 6LB instance. The number of connection resets is recorded for SR 8 and SC, and depicted in figure 7.18 for several values of x (averaged over 10 experiments). 6LB increases the resiliency over SC (as described in section 7.3.2): apart from "unavoidable" resets corresponding to connections that were pinned to a removed instance, no more than 2% of extra connections were reset by 6LB, as compared to 4% with SC.

SYN → SYN-ACK Latency

In order to quantify the additional forwarding latency induced by 6LB, figure 7.19 depicts the SYN → SYN-ACK latency as seen by the client for SR 4 and SC, for the experiment where ρ = 0.71. As compared to SC, with 6LB, the SYN packet can be forwarded to an extra candidate instance, and the SYN-ACK packet must be forwarded through the load-balancer. Overall, this increases the median latency by 69 µs. Restricted to those connections that are accepted by the second candidate in the SR list (corresponding to 38% of the 80000 queries in this experiment), the median latency is increased by 57 µs. For connections accepted by the first instance, the median latency is increased by 32 µs.

Comparison against Centralized Policies

Centralized load-balancing policies do not offer the resiliency of consistent hashing approaches, but in exchange provide more fairness. In order to position 6LB as compared to this class of load-balancers, two centralized policies are evaluated: (i) Round-Robin and (ii) weighted Round-Robin with feedback [START_REF] Aghdai | Spotlight: Scalable transport layer load balancing for data center networks[END_REF]. With the latter policy, feedback is obtained by polling the load of each application instance every 200 ms (over an out-of-band TCP channel), before adjusting the weight of the instance in the Round-Robin algorithm accordingly 11 . Figure 7.20 depicts the average page load time as a function of the request rate ρ. Results show that Round-Robin provides more fairness than single-choice consistent hash, but is outperformed by SR dyn (with equivalent results for light loads ρ ≤ 0.7). For heavier loads, the feedback policy slightly improves performance over Round-Robin, but remains outperformed by SR dyn : this shows the benefit of using instantaneous information rather than relying on periodic feedback.

Influence of the Variance of Service Times

To understand the influence of the variability of job service times, an experiment is conducted, where job CPU times distributions have different variances. To that purpose, the previously used exponential distribution is replaced with several log-normal distributions 12 . The response times are set to have the same median as previously, but different variance parameters, allowing to evaluate from constant to very skewed response times. Figure 7.21 depicts the mean response time as a function of the coefficient of variation 13 of service times, for a Poisson stream of 80000 queries at rate ρ = 0.71, against SR dyn , SC, and Round-Robin.

In the extreme case where response times are constant, Round-Robin performs the best (as instances will have totally processed a query before being assigned a new one) and 6LB performs better than SC. Indeed, with single-choice consistent-hashing queries can be placed on a server that is already busy (if "unlucky once"), whereas 6LB needs to be "unlucky twice" for this to happen. 11 The weight w is adjusted with w = 0.1 + 0.9 exp(-8 × (b/32) 2 ), where b is the current number of busy worker threads.

12 Log-normal distributions are a simple class of positive-valued distributions with a parameter influencing the variance. A log-normal distribution with parameters µ, σ has density

1 xσ √ 2π exp[-(log x -µ) 2 /(2σ 2 )]
, median e µ and variance (exp(σ 2 ) -1) exp(2µ + σ 2 ). 13 The coefficient of variation of X is the standard deviation of X divided by its average. When the skewness of job service times increases, 6LB's use of local information becomes a greater and greater advantage, and it eventually shows the best performance among all approaches.

Wikipedia Replay

To evaluate the efficiency of 6LB when exposed to a realistic workload, an experiment has been constructed to reproduce a typical (and, popular) Web-service. Thus an instance of MediaWiki14 (version 1.28), as well as a MySQL server and the memcached cache daemon, were installed on each of the 48 application instances. The wikiloader tool from [START_REF] Van Baaren | Wikibench: A distributed, Wikipedia based web application benchmark[END_REF], and a dump of the database of the English version of Wikipedia from [START_REF] Urdaneta | Wikipedia workload analysis for decentralized hosting[END_REF], were used to populate the MySQL databases, resulting in each instance containing an individual replica of the English Wikipedia.

Traffic and Workload Patterns

A traffic generator, able to replay a MediaWiki access trace and to record response times was developed, and experiments were run using 24 hours of traces from [START_REF] Urdaneta | Wikipedia workload analysis for decentralized hosting[END_REF]. These traces correspond to 10% of all queries received by Wikipedia during this timeframe, from among which only traffic to the English Wikipedia was extracted and used for the experiment.

A first experiment was to size the server farm, i.e., to identify the smallest number of application instances necessary to be able to serve queries while exhibiting reasonable response times. With 28 instances, the median response time during peak hours is smaller than 400 ms: the remainder of this section will assume this size for the server farm.

Connection Acceptance Policies Tested

Given the superior performance of SR 4 and SR dyn in the experiments from section 7.6.2, the 24-hour trace was replayed against both SR 4 and SR dyn , and client-side response times were collected. As a baseline, the trace was also replayed against the reference SC policy.

The experiment allowed classifying queries into two groups: (i) requests for static pages, which are not CPU-intensive, and for which response times were of the order of a millisecond, and (ii) requests for wiki pages, that trigger memcached or MySQL and thus are more CPU-intensive. 6LB was found to offer only a small improvement over SC for static page response time (figure 7.24, top). However, the load times of wiki pages15 exhibited interesting differences.

Figure 7.22 depicts the wiki page request rate and the median wiki page load time for the three tested policies during the 24h replay (data has been binned in 10 minutes slots). It can be observed that at the off-peak period around 8:00 UTC, when the system was lightly loaded and subject to a

Experimental Results

To understand the response time variability over 24 hours, figure 7.23 depicts deciles 1-9 of the wiki page load time distribution, for each 10 minutes bin. Again, SR 4 and SR dyn show less variability under higher loads than does SC. Among SR 4 and SR dyn , the latter has the lower variability under lighter loads, but is outperformed under higher loads.

Finally, as an indicator of "global good behavior", figure 7.24 (bottom) depicts the CDF of the wiki page load times over the whole day. Overall, the median response time went from 0.22 s with SC to 0.18 s with SR 4 and 0.16 s with SR dyn . Furthermore, the tail of the distribution is steeper when using 6LB, with the 90-th percentile going from 0.67 s with SC to 0.32 s with SR 4 and 0.31 s with SR dyn .

Throughput Evaluation

The advantages provided by 6LB in load-balancing fairness come at the cost of some overhead as compared to single-choice load-balancing approaches, notably due to maintaining flow state and performing IPv6 SR header insertion. To understand the impact of 6LB in terms of CPU overhead, the packet-forwarding performance of the VPP implementation introduced in this chapter is evaluated. Maglev [START_REF] Eisenbud | Maglev: A fast and reliable software network load balancer[END_REF] with GRE encapsulation has also been implemented as a VPP plugin, and serves as a reference point. Evaluation was conducted on a single core of a machine running an Intel E5-2667 CPU at 3.2 GHz, with an Intel X710 10 Gbps NIC. The load-balancer was manually initialized to install a pre-determined number of flow entries, and a packet generator (sitting on another machine on the same Ethernet link) was set to send TCP ACK packets corresponding to these flows, at line rate. Packets were set to return to the packet generator, and the number of packets effectively forwarded by 6LB was recorded -allowing to determine the maximum forwarding capability of the implementation, for upstream traffic. ACK packets were used rather than SYN, as they are expected to represent the majority of the upstream traffic.

Figure 7.25 depicts the achievable forwarding rate (in millions of packets per second, Mpps), as a function of the number of flow entries installed. Two main results are to be noted. First, the kernel bypass and vectorization capabilities of VPP make it very efficient for load-balancing (be it singlechoice or 6LB), with a raw forwarding capability of around 8 Mpps with one CPU core -whereas [START_REF] Eisenbud | Maglev: A fast and reliable software network load balancer[END_REF] reports 2.7 Mpps for the kernel bypass implementation of Maglev, and 0.5 Mpps without without kernel bypass. When the number of flows reaches approximatively 10 5 , the performance of both implementations degrades, as the flow table cannot reside entirely in the CPU cache.

Second, it can be observed that 6LB incurs only 8% CPU overhead as compared to the loadbalancer reference implementation. This overhead can be explained by the greater complexity of the per-packet operations, and the fact that the hash-table for flow state needs to handle collisions -whereas the one from the reference load-balancing plugin does not. Yet, this CPU overhead remains relatively negligible, and the additional 8% resources that might need to be deployed to use 6LB should be largely compensated by the fact that less application instances need be deployed, due to the greater fairness induced (as shown in section 7.6.2).

Summary of Results

This chapter has introduced 6LB, an innovative network service offering flexible, scalable, reliable, distributed, application-aware, but at the same time application-agnostic and applicationprotocol-agnostic, load balancing. This is accomplished by an architecture in which (i) load-balancers use an extended consistent hashing algorithm to map incoming flows onto a set of candidate application instances, (ii) to offer -not impose -these network flows to the candidate application instances, leaving them the decision of whether or not to accept a flow. Once an application instance has accepted a flow (iii) data packets of no interest to the load-balancer are sent directly from the application instance to the client. When a network flow is reassigned to another load balancer (e.g., if a load balancer is added to or removed from the system), this will be detected, and in-band signaling will reestablish the necessary state in this new load-balancer for continued operation, ensuring (iv) that a traffic flow between a client and an application instance becomes pinned to that application instance, regardless of changes to the load balancing infrastructure. The use of Segment Routing, specifically SR Functions, allow defining and implementing this as a network service, i.e., entirely below the application layer.

This chapter has also introduced a simple two-choice random assignment policy (motivated by the concept of power of two choices), combined with a static or dynamic query acceptance policy. These policies were compared to a naïve one-choice random query dispatch policy, by way of an analytical model, as well as an evaluation on a 48-servers deployment. Evaluation of those policies, conducted using a simulated Poisson workload as well as on a Wikipedia replica, shows that 6LB is able to better spread the load between all application instances than single-choice consistenthashing load-balancers. Evaluation of the packet-forwarding performance of the implementation shows that these benefits are attained at a negligible cost in terms of CPU overhead.

Results from this chapter have been published in [START_REF] Desmouceaux | SRLB: The power of choices in load balancing with segment routing[END_REF][START_REF]6LB: Scalable and application-aware load balancing with segment routing[END_REF]. A subsequent study on how to use the data-plane of 6LB to improve cache hit-rates in Content Distribution Networks (CDNs), by performing popularity estimation in the data-plane, has been published in [START_REF] Desmouceaux | A contentaware data-plane for efficient and scalable video delivery[END_REF].

Chapter 8

Stateless Load-Aware Load Balancing in P4

In data-center and cloud architectures, as was detailed in chapter 1, workload are virtualized, with applications replicated among multiple application instances, each capable of independently serving incoming queries [2,3,[START_REF] Hindman | Mesos: A platform for fine-grained resource sharing in the data center[END_REF]. An important functional part in these architectures is the loadbalancer (LB), dispatching incoming queries amongst application instances. To make the loadbalancer "invisible", a Virtual IP Address (VIP), shared by all application instances providing the same service, is advertised to the Internet in place of the address of the load-balancer, requiring the load-balancer to provide per-connection consistency (PCC), i.e., ensuring that traffic from a connection (typically identified by its network 5-tuple: source & destination addresses, L4 protocol, source & destination port) is always directed to the same application instance. Naïve load balancing use Equal Cost Multi-Path (ECMP) [START_REF] Thaler | Multipath issues in unicast and multicast next-hop selection[END_REF] to map connections to application instances, using a hash function and a modulo operation.

Statement of Purpose

A drawback of using ECMP for load-balancing is the lack of resiliency to changes to the application instance set, which causes the modulus in the ECMP operation to change and most connections to be redistributed across application instances, thus breaking PCC and causing connection resets. Consistent hashing [START_REF] Karger | Consistent hashing and random trees: Distributed caching protocols for relieving hot spots on the World Wide Web[END_REF][START_REF] Karger | Web caching with consistent hashing[END_REF][START_REF] Thaler | Using name-based mappings to increase hit rates[END_REF] attempts to address this, by providing a more resilient mapping of connections across application instances [START_REF] Eisenbud | Maglev: A fast and reliable software network load balancer[END_REF][START_REF] Patel | Ananta: cloud scale load balancing[END_REF] through maintaining an intermediate table which, with high probability, yields a persistent mapping of the 5-tuple space to the set of application instances, even when faced with changes in the application instance set. Maglev [START_REF] Eisenbud | Maglev: A fast and reliable software network load balancer[END_REF] uses consistent hashing with per-connection state to maximize the probability of PCC: a connection breaks only when both (i) per-connection state is removed (if memory is exhausted, for instance due to a denial-of-service attack -or if traffic is rebalanced to a new LB instance) and (ii) consistent-hashing changes the mapping of the connection to a different application instance (if there is a change in the set of application instances, which should affect only a small number of connections).

The pseudo-random nature of consistent hashing assigns queries to application instances regardless of their actual load state [START_REF] Eisenbud | Maglev: A fast and reliable software network load balancer[END_REF]. While this does not pose any problem for non-CPU-intensive applications (e.g., serving static Web pages), performance may degrade for CPU-intensive applications (e.g., data processing), for which the number of concurrently served queries per application instance must be minimized. Assigning queries to the least loaded from among two randomly chosen application instances (rather than to one randomly chosen application instance) was shown in [START_REF] Mitzenmacher | The power of two choices in randomized load balancing[END_REF] to improve load-balancing fairness. Based on this, 6LB, introduced in chapter 7, combines a Maglev-like consistent hashing with assigning connections to set of two application instances, which decide amongst themselves which will accept the connection. This is achieved by forwarding connection request (SYN) packets using Segment Routing (introduced in section 1.2.1) and then by maintaining state in the LB as to which instance has accepted the connection.

The need to maintain per-flow connection-state make Maglev and 6LB difficult to implement on programmable hardware devices, whereas it is known that hardware-based load-balancers offer 128

CHAPTER 8. STATELESS LOAD-AWARE LOAD BALANCING IN P4

a potential performance benefit [START_REF] Gandhi | Duet: Cloud scale load balancing with hardware and software[END_REF][START_REF] Miao | SilkRoad: Making stateful layer-4 load balancing fast and cheap using switching ASICs[END_REF][START_REF] Olteanu | Stateless datacenter load-balancing with Beamer[END_REF]. Beamer [START_REF] Olteanu | Stateless datacenter load-balancing with Beamer[END_REF] circumvents this state requirement, by calling on assistance from the network stack of application instances to maintain PCC. When the set of application instances changes, and consistent hashing maps a flow to a different application instance, Beamer directs packets from that flow to that new application instance, while also embedding the address of the previously used application instance in the packet header. This allows the new application instance, in case it does not have connection state for a received packet, to forward it to the previous instance -somewhat similarly to the recovery mechanism of 6LB introduced in section 7.4.3.

A shared requirement of [START_REF]6LB: Scalable and application-aware load balancing with segment routing[END_REF][START_REF] Olteanu | Stateless datacenter load-balancing with Beamer[END_REF] is that an application instance is able to direct packets to a second application instance, when needed. For 6LB, only connection establishment packets (or first connection recovery packets) are proposed to two application instances, allowing load sharing -at the expense of state in the LB for forwarding subsequent packets directly to the correct application instance, using the stickiness mechanism of section 7.4. Beamer offers packets to two application instances only when there is a change in consistent hashing -but then, does so for all packets in a flow, to avoid keeping state in the LB.

Thus the question: is it possible to provide a stateless load-balancer that dispatches queries according to the state of the applications? This requires the LB to be able to (i) send connection requests through a chain of "candidate" application instances for local connection acceptance decisions, and (ii) statelessly direct packets in an established flow to the one application instance which accepted the connection request.

As a possible solution, this chapter introduces SHELL, an application-agnostic, applicationload-aware, stateless load-balancer, which (i) proposes new connections to a set of pseudo-randomlychosen application instances, each making a local acceptance decision, and (ii) "marks" subsequent packets in a flow so as to allow the load-balancer to direct them to the appropriate application instance, without requiring the load-balancer to maintain per-connection-state.

The statelessness of the load-balancer makes it a candidate for an implementation in a hardware platform. Thus, this chapter proposes a prototype P4 [START_REF] Bosshart | P4: Programming protocol-independent packet processors[END_REF] implementation of SHELL targeting the NetFPGA SUME [START_REF] Zilberman | NetFPGA SUME: Toward 100 Gbps as research commodity[END_REF] platform using the P4-NetFPGA [START_REF] Ibanez | The P4 → NetFPGA workflow for line-rate packet processing[END_REF] framework, as well as an extensive performance evaluation of the P4 implementation and of the stable hashing algorithm.

Chapter Outline

The remainder of this chapter is organized as follows. Section 8.2 provides an overview of SHELL, with section 8.3 detailing key design aspects. The P4-NetFPGA implementation of the load-balancer is detailed in section 8.4, followed by a performance evaluation in section 8.5. Resiliency of consistent hashing is evaluated in section 8.6, before section 8.7 concludes this chapter.

Overview

SHELL consists of 3 main components: a control plane, a P4-based load-balancing data-plane, and a server agent.

The control-plane constructs two tables (see section 8.3.1): (i) a consistent hashing table, used to direct new connection request packets (e.g., TCP SYNs) to a set of candidate application instances, which improves fairness, and (ii) a choice history table, for directing subsequent packets in a flow (e.g., TCP ACKs), which improves resiliency.

The P4 load-balancer uses 5-tuple hashing to map each new connection request to a list of candidate application instances from the consistent hashing table provided by the control-plane. Segment Routing is then used to direct such packets through the selected list of application instances, until one accepts the connection (note that the last application instance in the set must always accept), as in 6LB (chapter 7). Then, C i , the position in the list of the application instance which accepted the connection, is communicated back to the client (see section 8.3.2), which in turn includes it in all further packets from the client to the load-balancer. This enables the P4 load-balancer to send these packets directly to the application instance handling the connection. When a change in the set of application instances causes modifications to some of the consistent hashing buckets, changes are saved in a history. The P4 load-balancer then directs (using SR) subsequent packets to the current and previous application instances associated with the value C i received in the packets. Finally, the server agent (i) accepts new connection requests or forwards them to the next candidate application instance, and (ii) forwards further packets until reaching the application instance that accepted the connection. This server agent is implemented as a Linux kernel module; the details hereof are out-of-scope for this chapter.

An exemple execution is illustrated in figure 8.1. (a) A SYN packet is directed through c = 2 candidate instances (c 1 , c 2 ). In this example, the two instances are hosted on machines m 2 , m 1 , and the second candidate (hosted on m 1 ) accepts the connection. This choice is reported in the covert channel of the SYN-ACK packet. More precisely, the covert channel encodes a label C 2 indicating that the second candidate accepted the connection (rather than encoding the full value m 1 of the address corresponding to this choice), thus carrying only one bit of information. (b) Subsequent packets from the client are directed by the LB to the correct machine m 1 , by looking up the machine corresponding to the covert channel value C 2 in a consistent hashing table. (c) Upon reconfiguration of the pool of instances, with high probability the assigned instance is not modified in the consistent hashing table, due to the resiliency property of consistent hashing. (d) A reconfiguration modifies the candidate list to (c t 1 , c t 2 ) = (m 3 , m 2 ). SHELL then uses the history matrix to go through (the machines hosting) the previous instances that used to be the c 2 for this bucket -in this example, (c t 2 , c t-1 2 ) = (m 2 , m 1 ).

Description

In this section, c is the number of candidate application instances through which connection requests are directed, B the number of buckets used in consistent hashing, and h the "depth" of the consistent hashing "history matrix" maintained for a bucket t[b] (an example of t[b] for a bucket b is shown in table 8.1). Finally, c covert channel labels C 1 , . . . , C c are defined, where label C i indicates that "at the time of connection establishment, this connection was accepted by the i-th application instance in the SR list embedded in the packet".

The behavior of the P4 load-balancer and of the application instance depends on which packet connection is handled. However, if the history matrix has been updated and the connection had been opened before that update, it is possible that the flow is not handled by m 3 . When this is the case, since m 2 , . . . , m 13 are also included in the SR header, the packet is directed through the previous instances corresponding to that bucket and that choice index C 1 , until reaching the correct one.

6.

All subsequent (i.e., TCP non-SYN) packets received at the application instance will be examined, and if corresponding connection-state is found, will be processed locally.

Otherwise, such packets are forwarded to the next segment, allowing to try an "older" instance which used to map to the same bucket and label C i . In the unlikely event that there is no next segment, the packet is dropped -this corresponds to the case where the history was not long enough to include the application instance that had accept the connection in the first place.

History Matrix Computation

First, the same algorithm as in 6LB (algorithm 9) is used to generate, for each bucket b, a candidate list of application instances ℓ = (m 1 , . . . , m c ). This list is recorded as the first row of the history matrix for the bucket: t[b][0][:] = ℓ. When the set of application instances is modified, the history matrix needs to be modified. Algorithm 10 describes the mechanism used to generate the history matrix when this occurs:

• First, all entries in the history matrix are offset by one, i.e., ∀b, t

[b][i+1][:] ← t[b][i][:].
• Then, the new mapping of choice index to candidate application instances is calculated for each bucket b, again using algorithm 9.

• By design, and as depicted in figure 7.5 and [162, figure 12], consistent hashing will leave most bucket entries unmodified after reconfigurations of the application pool, i.e., t

[b][0][:] = t[b][1][:] for most b.
Therefore, duplicate history entries are removed (last step of algorithm 10), so as to avoid repetitions in the created SR headers, and to span a potentially longer history of application instances having occupied a bucket.

Possible covert channels

The server-state-aware and stateless load-balancing approach described in this chapter relies on the application instance being able to instruct clients to include C i (the position of the application instance in the segment list at time of connection establishment) in all packets subsequent to the connection request. This label C i is then used by the load balancer to direct these packets to the appropriate application instance.

If it is possible to modify the client networking stack so as to cooperate with the load-balancing architecture, a simple option would be to embed the full identifier (rather than a label C i ) of the instance having accepted the connection in packets sent to the client, then reflect this identifier in the packets sent by the client. This can be achieved with transport protocols such as QUIC [START_REF] Hamilton | QUIC: A UDP-Based Secure and Reliable Transport for HTTP/2[END_REF], which embed a connection identifier chosen by servers and reflected by clients.

However, for TCP connections, it is necessary to convey C i from the application instance to the client, and make the client relay C i in subsequent packets, without client-side modification1 . This requires using a covert channel from the application instance, through the client, and to the load-balancer, which the client neither inspects nor interferes with. Approaches accomplishing this include:

• TCP sequence numbers, which are reflected by endpoints in the acknowledgement field.

Since the initial sequence number is chosen by the server, this allows implementing the covert channel through the high-order ⌈log 2 c⌉ bits of the sequence number. Note that this method fails for connections whose length outrun 1/c of the sequence number space (i.e., 2 32 /c bytes), because in this case the high-order bits will be modified in the middle of the connection. In this case, it would be necessary to implement a mechanism to detect when the high-order bits are about to wrap, as well as a flow table to record flow state when this happens. Alternatively to recording flow state, the LB could insert an SR list comprising all c × h segments of the history matrix to perform a recovery through all candidates, but this would potentially have a high network overhead.

• TCP timestamps, which are also reflected by TCP endpoints. The covert channel can be carried in the ⌈log 2 c⌉ low-order bits of the TCP timestamp, as is done in [START_REF] Duchene | Making multipath TCP friendlier to load balancers and anycast[END_REF] -and according to [START_REF] Medina | Measuring interactions between transport protocols and middleboxes[END_REF][START_REF] Honda | Is it still possible to extend TCP?[END_REF] this is resilient to middlebox processing. Modifying the timestamp by at most c-1 units (c = 2 usually [START_REF] Mitzenmacher | The power of two choices in randomized load balancing[END_REF]) will have a negligible effect on RTT estimation.

To avoid using a flow table, SHELL uses TCP timestamps as covert channel for TCP connections: the server agent of the accepting application instance encodes its index C i in the low-order bits of the TCP timestamp -and the LB inspects TCP timestamp sent by clients.

P4 Load-Balancer Implementation

This section describes the P4 implementation of the data-plane introduced in section 8.3.

P4-LB IMPLEMENTATION PERFORMANCE

Data Plane

The workflow of the P4 implementation of the LB data-plane of SHELL is illustrated in figure 8.2. It uses three match-action tables, corresponding to (i) segment lists to be inserted into SYN packets, as given by the first row of the history matrix, (ii) segment lists to be inserted in non-SYN packets, as given by columns of the history matrix, and (iii) a Layer-2 lookup for output packets.

Processing a packet starts with parsing its headers. If they do not match the expected format (e.g., the packet uses UDP), the packet is dropped; otherwise, they are processed by the matchaction pipeline. Both (i) the 5-tuple hash of the packet, computed by way of an external function from the P4-NetFPGA framework, and, (ii) in case of a non-SYN packet, the covert channel value as found in the TCP timestamp field (whose parsing is described below), are used as keys to the match tables, in order to access the corresponding segment list. The segment list is then fed to an action, which builds the SR header. To complete the match-action pipeline, a match is performed on the newly-added destination address of the packet (i.e., the first SR segment), which calls an action making the packet egress through the correct interface. Finally, the de-parsing stage emits the packet with the newly-built headers.

The TCP timestamp is embedded in an option of the TCP header, and TCP options are formatted as Type-Length-Values (TLVs) [START_REF] Postel | Transmission Control Protocol[END_REF] -which makes it impossible to know in advance where to peek into the packet so as to retrieve its TCP timestamp. For want of a lookahead function and of variable-length header support in the P4 compiler, only a set of "reasonable options" is parsed (specifically, SACKs and timestamps, i.e., those that can be found in non-SYN packets according to [START_REF] Medina | Measuring interactions between transport protocols and middleboxes[END_REF]). Depending on the length of the TCP header found in the "data offset" (denoted by d off ) field, an option header with the identified possible combinations of TLVs is parsed, corresponding to lengths d off ∈ {8, 11, 13, 15}. Figure 8.3 depicts an example of such a parsing, when d off = 11. The bits of the covert channel are then stored into a meta-data field2 . A parameter of interest to the performance of the data-plane is the maximum size of the parsed TCP header d max off : its influence on the performance is evaluated in section 8.5.

P4-LB Implementation Performance

A key element to the performance of SHELL is the per-packet latency incurred in the P4dataplane of the LB. It depends primarily on two factors: (i) the latency incurring when receiving a packet over an ingress interface, inserting an SR header, and transmitting this (now larger) packet over an egress interface (section 8.4.1), and (ii) the latency incurring from extracting C i from within the TCP timestamp (section 8.3.2). These factors are evaluated using the P4-NetFPGA framework and the Xilinx Vivado software suite, simulating packets going through one interface of a 10G NetFPGA-SUME, and are documented in this section.

Effect of SR Header Insertion on Latency

For the purpose of the simulations, c = 2 choices were used for SYN packets, and a history of h = 2 was used for ACK packets, thus SR headers with 3 segments were inserted, increasing the packet size by 56 bytes between ingress and egress. Feeding ingress packets at maximum line-speed fills the egress interface queue over time, thus progressively increasing packet forwarding latencyas depicted in figure 8.4, showing stable results from above 1500 packets. When the egress buffer is empty, the latency is 2.1 µs, and when the buffer is full the latency oscillates between 8.2 µs and 9.0 µs. Thus, for subsequent simulations, batches of 4800 packets are injected at line-rate on the ingress interface.

The batch of packets consists of a mixture of SYN packets, ACK packets with odd TCP timestamps and ACK packets with even TCP timestamps. Table 8.2 reports the throughput and worst-case latency obtained: SHELL, comparable to Beamer [START_REF] Olteanu | Stateless datacenter load-balancing with Beamer[END_REF], can sustain 60 Mpps, i.e., 22× as much as what is reported in [START_REF] Eisenbud | Maglev: A fast and reliable software network load balancer[END_REF] for the single-core software implementations of Maglev, while also providing application instance load-awareness. This allows noting that, in a controlled environment (e.g., a data-center, where SHELL typically would be deployed), where a strict set of TCP options can be enforced, it is possible to trade off parsing safety against reduced latency. Conversely, parsing more potential options increases the amount of logic consumed on the FPGA. As reported in [START_REF] Miao | SilkRoad: Making stateful layer-4 load balancing fast and cheap using switching ASICs[END_REF][START_REF] Roy | Inside the social network's (datacenter) network[END_REF].

on the FGPA for different values of d max off , this can increase LUT (Look-Up Tables, implementing the FPGA logic) usage by up to one third.

Consistent Hashing Resiliency

The resiliency of the consistent hashing algorithm detailed in 8.3.1 is evaluated by way of simulations. Long-lived connections are particularly vulnerable to application instance reconfigurations -thus, as a worst-case scenario, simulations using infinite-length connections, and subject to successive application instance removal/insertions, are performed; each simulation is repeated 5 times.

For a random connection, assuming that it was uniformly drawn from among the B × c buckets and choices possible, the probability that it is reset3 after n application instance reconfigurations is depicted in figure 8.6a (top), which shows the impact of the history depth h, and figure 8.6b (top), which shows the impact of the number of buckets B.

In realistic scenarios, application instance reconfigurations are expected to be rare, and connections are unlikely to last for more than a few application instance reconfigurations -at most. The bottom graphs of figures 8.6a and 8.6b depict the connection reset probability using the connection duration distribution model from [206, Figure 7.a] (where connections have a median duration of 365 ms), and with the application instance reconfiguration rate distribution taken as the highest 1% rates over a month for 100 clusters from [START_REF] Miao | SilkRoad: Making stateful layer-4 load balancing fast and cheap using switching ASICs[END_REF]Figure 2] (in this worst-case distribution, reconfigurations have a median rate of 13.5 min -1 ).

In these conditions, less than 1% of the connections were lost when using SHELL with a nonvoid history (h > 1). In comparison, without history (i.e., h = 1, as would be the case in Maglev and 6LB), more than 5% of the connections were lost.

Summary of Results

This chapter has introduced SHELL, an application-agnostic load-balancing architecture which combines application load awareness (by using a power-of-choices scheme upon connection establishment), statelessness (by using a covert channel to indicate which of the candidates had accepted the connection), and resiliency (by using consistent hashing and versioning). Being stateless makes SHELL suitable for a hardware implementation, as demonstrated in this chapter through prototype development using the P4-NetFPGA framework. An evaluation of throughput and latency of this prototype implementation shows that the attainable performance is equal to that of other hardware implementations, while providing application-awareness and therefore improving loadbalancing fairness. Further, simulation of the consistent hashing resiliency shows that the number of long-lived connections dropped, even in worst-case scenarios, is negligible.

Results from this chapter have been published in [START_REF] Pit-Claudel | Stateless loadaware load balancing in P4[END_REF].

Chapter 9

Joint Auto-Scaling and Load-Balancing with Segment Routing

Virtualization and cloud architectures, wherein different tenants share computing resources to deploy their workloads, reduce the possible granularity of task allocation in data centers [3]. To optimize cost and energy, applications can be (i) replicated among multiple instances running in containers or virtual machines (VMs) [1,2], and (ii) automatically scaled up or down in order to meet a given Service Level Agreement (SLA) [START_REF] Al-Dhuraibi | Elasticity in cloud computing: state of the art and research challenges[END_REF]; this, to trade off cost and user quality of experience. Two functions serve this purpose: (i) a load-balancer, which dispatches queries onto identical instances, and (ii) an autoscaler, which monitors application instances to scale up or down the number of application replicas.

A main challenge for network load-balancers is to provide performance and resiliency, while taking application state into account. Some architectures, like Equal Cost Multi-Path (ECMP) [START_REF] Thaler | Multipath issues in unicast and multicast next-hop selection[END_REF] or Maglev [START_REF] Eisenbud | Maglev: A fast and reliable software network load balancer[END_REF], distribute flows among instances pseudo-randomly, allowing forwarding packets without terminating Layer-4 connections, and thus providing a high throughput. The use of consistent hashing (as in [START_REF] Eisenbud | Maglev: A fast and reliable software network load balancer[END_REF]) also allows for resiliency, in case an existing flow is handed over to another load-balancer. Such architectures, nonetheless, assign flows to instances regardless of their load state -but it has been demonstrated [START_REF] Mitzenmacher | The power of two choices in randomized load balancing[END_REF] that considering application load can greatly improve overall performance. Other load-balancing architectures do take application state into account, by terminating Layer-4 connections [START_REF]HAProxy: the reliable, high-performance TCP/HTTP load balancer[END_REF], and/or using centralized monitoring [START_REF] Wang | Openflow-based server load balancing gone wild[END_REF] -thus incurring a performance overhead and degrading resiliency.

Similarly, autoscalers use centralized monitoring, with an external agent gathering load metrics from all application instances and collapsing these into a decision [START_REF] Al-Dhuraibi | Elasticity in cloud computing: state of the art and research challenges[END_REF]. This can therefore sometimes cause decisions to be made on out-of-date information. Furthermore, such agents typically collect external metrics (e.g., CPU load of a VM as seen by the hypervisor), ignoring applicationprovided metrics that could possibly be more suitable to make scaling decisions.

Statement of Purpose

The purpose of this chapter is to introduce a unified architecture for load-balancing and autoscaling, and to analyze and quantify its characteristics. The proposed architecture supports application-load-aware load-balancing and autoscaling decisions, thus eliminating the need for centralized monitoring and decision-taking. To decrease operational overhead, the proposed architecture operates entirely within the network layer (Layer-3), removing the need from terminating or proxying network connections. Using the dataplane of 6LB (introduced in chapter 7), queries are directed with SR through several application instances, each able to make a local decision to accept or refuse the query, and the last one able to make autoscaling decisions. In sum, and as described in section 9.2, the local state of the applications is used to make both load-balancing decisions and autoscaling decisions. The contributions of this chapter are twofold:

1. An analytical model of the behavior of the proposed load-balancing policy for a chain of n servers is formulated. The model is solved by extending the Recursive Renewal Reward (RRR) 138 CHAPTER 9. JOINT AUTO-SCALING AND LOAD-BALANCING WITH SR technique [START_REF] Gandhi | Exact analysis of the M/M/k/setup class of Markov chains via recursive renewal reward[END_REF] to n-dimensional Markov chains, and numerical results are obtained when the state space is reasonably small.

2. The proposed load-balancing and autoscaling architecture is implemented within a virtual router (VPP [START_REF] Barach | High-speed software data plane via vectorized packet processing[END_REF]), and tested against traces of a real workload, allowing to confirm that the properties of the system hold in real environments.

Related Work

This section reviews the literature on load-balancing and autoscaling, before presenting a mathematical framework (the Recursive Renewal Reward technique) used throughout this chapter. Segment Routing has been introduced in section 1.2.1, and work related to load balancing has been reviewed in section 7.1.1.

Autoscaling

Methods to provide autoscaling have been classified by [START_REF] Al-Dhuraibi | Elasticity in cloud computing: state of the art and research challenges[END_REF] as reactive and proactive. Reactive methods consist of regularly gathering measurements, and taking actions accordingly when thresholds are violated. For instance, in [START_REF] Hasan | Integrated and autonomic cloud resource scaling[END_REF] up/downscaling is triggered when bounds on some observed metrics are violated; [START_REF] Beloglazov | Adaptive threshold-based approach for energy-efficient consolidation of virtual machines in cloud data centers[END_REF] has a similar approach with dynamic threshold adjustment. Such approaches incur an overhead due to the gathering of statistics, and a time gap between detection of violations and appropriate reaction.

Conversely, proactive approaches consist of anticipating changes and acting correspondingly. Machine learning techniques are used in [START_REF] Vasić | Dejavu: accelerating resource allocation in virtualized environments[END_REF] to classify workloads by their resource allocation preferences, and [START_REF] Kalyvianaki | Self-adaptive and self-configured CPU resource provisioning for virtualized servers using Kalman filters[END_REF] uses control theory to track CPU usage and to allocate resources accordingly. While solving the issue of timeliness, proactive approaches suffer the need to collect statistics and perform centralized computations. This chapter avoids this by enabling applications instances to make local autoscaling decisions.

Recursive Renewal Reward

The Recursive Renewal Reward technique (RRR) [START_REF] Gandhi | Exact analysis of the M/M/k/setup class of Markov chains via recursive renewal reward[END_REF] has been introduced as a means to extract metrics from a class of Markov chains. It consists of identifying a home state S 0 and a metric of interest M (t) (the reward ). Then, this allows evaluating the expected value of M by computing its average earning rate over a cycle from S 0 to itself, divided by the average duration of the cycle:

E[M ] = E[ S0→S0 M (t)dt] E[ S0→S0 1dt] (9.1)
RRR applies to bi-dimensional Markov chains with a one-dimensional repeating pattern, in which case computing the average earning rate of the reward reduces to solving a finite number of equations, by restricting to the border row that generates the pattern. RRR has been applied to various problems, from vehicular networks [START_REF] Lioris | Platoons of connected vehicles can double throughput in urban roads[END_REF] to query duplication in data centers [START_REF] Lee | On scheduling redundant requests with cancellation overheads[END_REF].

Chapter Outline

The remainder of this chapter is organized as follows. Section 9.2 gives an overview of the architecture introduced in this chapter. Using RRR, an analytical model for the response time of the system is introduced for n = 2 servers in section 9.3, and extended for n ≥ 3 servers in section 9.4. Numerical results are given in section 9.5, then results of experimentation on a realistic testbed are presented in section 9.6. Finally, section 9.7 concludes this chapter.

Joint Load-Balancing and Autoscaling

This section presents an overview of the architecture introduced in this chapter. Application instances are ordered into a chain, and an SLA on response times is decided by the data-center operator. A load-balancer dispatches requests through this chain with SR, using the dataplane of 6LB introduced in section 7.4. Each of the instances in the chain decides to accept the connection if this yields to locally satisfying the SLA, otherwise forwards it to the next instance in the chain, as illustrated in figure 7.3. This way, queries are served by the first instance of the chain able home state is chosen as (0, 0). To compute the mean reward of M (t) over a cycle from the home state to itself, the reward is decomposed into two parts: (i) the mean reward when going one level down in the Markov chain (i.e., until s 2 has one less client), and (ii) the mean reward when going from a state (i, 0) to the home state (i.e., until all clients have left). Table 9.1 summarizes the notation used in this chapter.

Reward when going one level down

For j ≥ 1, let R D (i,j)→(⋆,j-1) be the mean reward earned between entering state (i, j) and reaching row j -1 (i.e., reaching any state (k, j -1)). Due to the repeating structure of the chain, that reward only needs to be computed for the border row {0, . . . , c} × {1} (identified in gray in figure 9.3). This is formalized by the following lemma: Lemma 9.1. For i ∈ {0, . . . , c} and j ≥ 1, R D (i,j+1)→(⋆,j) = zR D (i,j)→(⋆,j-1) . Therefore, only R D (i,1)→(⋆,0) needs to be computed, which will be denoted by R D i .

Proof. Since the restriction of the chain to {0, . . . , c} × {j, j + 1, . . .} (including transitions to the (j -1) th row) is isomorphic to its restriction to {0, . . . , c} × {j + 1, j + 2, . . .} (including transitions to the j th row), and since there is exactly one more client in the second instance in the latter restriction, the mean reward of z N 2 (t) earned when going from (i, j + 1) to row j is the same as the mean reward of z N 2 (t)+1 earned when going from (i, j) to row j -1. Hence, R D

(i,j+1)→(⋆,j) = E[ (i,j+1)→(⋆,j) z N 2 (t) dt] = E[ (i,j)→(⋆,j-1) z N 2 (t)+1 dt] = zR D (i,j)→(⋆,j-1) .
Let p D (i,j)→(k,j-1) be the probability that, starting from state (i, j), the first state visited when reaching row j-1 is (k, j-1). Since the chain has a recursive structure along the vertical dimension, the following lemma holds: Lemma 9.2. For j ≥ 1 and k ∈ {0, . . . , c}, p D (i,j)→(k,j-1) is independent from j, and will therefore be denoted by p D i→k .

Proof. The proof is similar to that of lemma 9.1.

It is therefore possible to formulate a system of equations 1 for the p D i→k , for k ∈ {0, . . . , c}:

p D 0→k = µ λ + µ δ 0k + λ λ + µ p D 1→k (9.2) p D i→k = µ λ + 2µ δ ik + µ λ + 2µ p D i-1→k + λ λ + 2µ p D i+1→k ∀1 ≤ i ≤ c -1 (9.3) p D c→k = µ λ + 2µ δ ck + µ λ + 2µ p D c-1→k + λ λ + 2µ c l=0 p D c→l p D l→k (9.4)
In these equations, the leftmost terms denote the probability of reaching (k, 0) from (i, 1) directly, while the other terms express reaching (k, 0) from (i, 1) by transitioning to an adjacent state first. The rightmost term in equation (9.4) comes from the possibility of reaching (k, 0) from (i, 1) by going to row 2 first, and uses lemma 9.2 to compute the transition from row 2 back to row 0. Having computed the p D i→k , it is possible to state a system of equations for the R D i :

R D 0 = z λ + µ + λ λ + µ R D 1 (9.5) R D i = z λ + 2µ + µ λ + 2µ R D i-1 + λ λ + 2µ R D i+1 ∀1 ≤ i ≤ c -1 (9.6) R D c = z λ + 2µ + µ λ + 2µ R D c-1 + λ λ + 2µ zR D c + c k=0 p D c→k R D k (9.7)
1 δnm denotes the Kronecker symbol: δnm = 1 if n = m and 0 otherwise.

The first terms in these equations denote the mean reward of z N2 earned when in state (i, 1), that is, z times the expected time spent in (i, 1), since in these states N 2 = 1 and z N2 = z. The other terms come from the reward earned if going to an adjacent state instead of directly going to row 0. The bracketed term in equation (9.7) uses lemma 9.1 to express the reward when going from (c, 2) to row 1, and then lemma 9.2 for going from a non-deterministic state in row 1 to row 0.

Reward when going from row 0 to the home state

Having computed R D i , it is possible to compute R L i , the mean reward earned when going from a state (i, 0) with i ≥ 1 to the home state (0, 0) -with R L 0 = 0, since reaching (0, 0) means the end of the cycle. The system of equations for the R L i is:

R L 0 = 0 (9.8) R L i = 1 λ + µ + µ λ + µ R L i-1 + λ λ + µ R L i+1 ∀1 ≤ i ≤ c -1 (9.9) R L c = 1 λ + µ + µ λ + µ R L c-1 + λ λ + µ   R D c + c j=0 p D c→j R L j   (9.10)
This system is obtained in a similar fashion to the R D i . Again, the bracketed term in equation (9.10) uses lemma 9.1 and expresses the possibility to go from (c, 0) to (c, 1), back to one of the possible states in row 0, and finally to (0, 0).

Mean reward over a cycle

Finally, it is possible to express R C , the mean reward over a cycle from the home state to itself:

R C = 1 λ + R L 1 (9.11)
This allows deriving the total expected value of z N2 , using equation (9.1):

E[z N2 ] = E[ (0,0)→(0,0) z N2(t) dt] E[ (0,0)→(0,0) 1dt] = R C (z) R C (
1) (9.12)

Closed-form Solution for c = 1

When the threshold is c = 1 (i.e., s 1 accepts queries only when empty), solving equations (9.2)-(9.12) allows deriving the expected value of N 2 in closed form:

E[N 2 ] = ρ 2 (1 + ρ)( √ 1 + ρ -ρ) (9.13)
Proof. The system of equations for the p D i→j is the following: (9.14). This yields, after some algebra:

p D 0→0 = µ λ + µ + λ λ + µ p D 1→0 (9.14) p D 0→1 = λ λ + µ p D 1→1 (9.15) p D 1→0 = µ λ + 2µ p D 0→0 + λ λ + 2µ (p D 1→0 p D 0→0 + p D 1→1 p D 1→0 ) (9.16) p D 1→1 = µ λ + 2µ + µ λ + 2µ p D 0→1 + λ λ + 2µ (p D 1→1 ) 2 + p D 1→0 p D 0→1 ( 9
p D 0→0 = 1 √ 1 + ρ (9.18) p D 0→1 = 1 - 1 √ 1 + ρ (9.19) p D 1→0 = 1 ρ 1 + ρ - 1 ρ (9.20) p D 1→1 = 1 + 1 ρ - 1 ρ 1 + ρ (9.21)
The system of equations for the R D i is:

R D 0 = z λ + µ + λ λ + µ R D 1 (9.22) R D 1 = z λ + 2µ + µ λ + 2µ R D 0 + λ λ + 2µ (zR D 1 + p D 1→0 R D 0 + p D 1→1 R D 1 ) (9.23)
Substituting R D 0 , p D 1→0 and p D 1→1 in (9.23) and rearranging gives, after some algebra:

R D 1 = z µ 1 + ρ + √ 1 + ρ 1 + ρ + √ 1 + ρ -ρ(1 + ρ)z (9.24)
The equation for R L 1 is:

R L 1 = 1 λ + µ + λ λ + µ R D 1 + p D 1→1 R L 1 (9.25)
Substituting p D 1→1 and R D 1 , and solving for R L 1 yields:

R L 1 = 1 µ 1 + ρ + (1 + ρz) √ 1 + ρ √ 1 + ρ(1 + ρ + √ 1 + ρ -zρ(1 + ρ)) (9.26)
From this, one can express the total reward over a cycle, using equation (5.9):

R C = 1 λ 1 + 2ρ + ρ 2 + (1 + 2ρ -ρz) √ 1 + ρ √ 1 + ρ(1 + ρ + √ 1 + ρ -zρ(1 + ρ)) (9.27)
Finally, this allows us to formulate the probability generating function of the number of clients in the second server, using equation (9.12):

E[z N 2 ] = (1 + √ ρ + 1 -ρ 2 )(ρ + √ 1 + ρ + z(1 - √ 1 + ρ)) (1 + ρ)(1 + ρ + √ 1 + ρ -zρ(1 + ρ)) (9.28)
It is finally possible to retrieve the expected number of clients in the second server by taking the derivative of the probability generating function at z = 1:

E[N2] = ∂E[z N 2 ] ∂z z=1 = ρ 2 (1 + ρ)( √ 1 + ρ -ρ)

Solving for c ≥ 2

When the threshold is c ≥ 2, it is possible to express the general form for E[z N2 ], and compute numerical2 values for its coefficients for any value of ρ, as described below.

First, the values of p D i→k can be computed numerically by solving the system of equations (9.2),(9.3), (9.4). Plugging these in equations (9.5),(9.6),(9.7) allows computing the R D i , yielding the following result: Lemma 9.3. For all i ∈ {0, . . . , c}, R D i = Aiz+Biz 2 C-Dz , where A i , B i , C, D are real constants, and B c = 0. chain currently serving < c clients, or by the last one otherwise. Hence, the state of the system can be modeled by a vector in S = {0, . . . , c} n-1 × N, where the i-th coordinate represents the number of clients in the i-th server's queue.

In each state, the system can transition upwards with rate λ to the state in which the first available instance has gained one client, and downwards with rate µ to any of those states where one client has left. Formally, let e i be the vectors of the canonical basis of R n : e i = (δ i1 , . . . , δ in ). For each state x = (x 1 , . . . , x n ), let the upward direction of x be u(x) = x + e min{i∈{1,...,n}|xi<c} , with the convention min ∅ = n. A set of downward directions is also defined as d(x) = i:xi>0 {xe i }. With this notation, from each state x there is a transition to u(x) with rate λ, and a transition to each state in d(x) with rate µ.

Let N i be the number of clients in the i-th instance. For the first n-1 instances, this metric can be evaluated without RRR by solving the balance equations, since the underlying Markov chain has finite state space {0, . . . , c} n-1 . To compute N n , the number of clients in the last instance, the RRR technique is extended to n-dimensional Markov chains that repeat in one dimension: this new method will be referred to as RRR n . Let 0 = (0, . . . , 0) be the home state. The metric of interest is chosen to be M (t) = z Nn(t) , the probability generating function of the number of clients in the last instance.

Whereas with RRR 2 a border row {0, . . . , c} × {1} was identified, from which the reward could be recursively computed, RRR n uses a border hypercube C 1 = {0, . . . , c} n-1 × {1}. The approach is similar to RRR 2 : computing the reward from any state in the border hypercube C 1 down to any state in the base hypercube C 0 = {0, . . . , c} n-1 × {0}, and from any state of the base hypercube to the home state 0. To ease notation, let C = {0, . . . , c} n-1 and c = (c, . . . , c) ∈ C.

First, it is possible to compute p D i→k , the probability that y = (k, 0) ∈ C 0 is the first state visited in the base hypercube when coming from a state x = (i, 1) ∈ C 1 of the border hypercube, by solving the following system:

p D i→k = µ λ + (1 + |d(i)|)µ δ ik + µ λ + (1 + |d(i)|)µ j∈d(i) p D j→k + λ λ + (1 + |d(i)|)µ p D u(i)→k , ∀i ∈ C \ {c}, ∀k ∈ C (9.29) p D c→k = µ λ + nµ δ ck + µ λ + nµ j∈d(c) p D j→k + λ λ + nµ j∈C p D c→j p D j→k , ∀k ∈ C (9.30)
Then, R D i , the mean reward earned when going from a state x = (i, 1) ∈ C 1 down to C 0 , can be found by solving:

R D i = z λ + (1 + |d(i)|)µ + µ λ + (1 + |d(i)|)µ k∈d(i) R D k + λ λ + (1 + |d(i)|)µ R D u(i) , ∀i ∈ C \ {c} (9.31) R D c = z λ + nµ + µ λ + nµ k∈d(c) R D k + λ λ + nµ zR D c + k∈C p D c→k R D k (9.32)
This allows computing R L i , the mean reward earned when going from a state x = (i, 0) ∈ C 0 to the home state: R L 0 = 0 (9.33) Finally, the mean reward of M (t) = z Nn(t) over a cycle, R C , can be expressed as:

R L i = 1 λ + |d(i)|µ + µ λ + |d(i)|µ k∈d(i) R L k + λ λ + |d(i)|µ R L u(i) , ∀i ∈ C \ {c, 0} (9.34)
R D c = 1 λ + (n -1)µ + µ λ + (n -1)µ k∈d(c) R D k + λ λ + (n -1)µ R D c + k∈C p D c→k R L k ( 9
R C = 1 λ + R L

Numerical Results

To evaluate the performance of the load-balancing and autoscaling strategy proposed in this chapter, the expected number of clients in the system is computed as described in section 9.4, allowing to derive the average response time using Little's law [START_REF] Little | A proof for the queuing formula: L = λW[END_REF]: E[T ] = E[N ]/λ. Note that this computation can only be carried for reasonably small chains, as the non-recursive part of the state space has a size which scales as Θ((c + 1) n ).

As a baseline, random "single-choice" load-balancing is used, for which the response time is:

E[T ] = 1 1 -ρ/n 1 µ
(9.37) Figure 9.5 depicts the average response time for first-available-instance load-balancing (denoted by SR 2 ), with an SLA S = 2 and a threshold c = 2. In other words, to avoid a response time greater than S = 2, all instances but the last serve no more than c = 2 clients. (Note that response times and SLAs in this section are described in 1/µ units.) Each of the thin lines shows the response time as a function of the request rate ρ when using n instances, for n ∈ {1, . . . , 5}. The thick line shows the response time when applying autoscaling, assuming a perfect satisfaction of the SLA. As a baseline, figure 9.5 also depicts this response time for single-choice load-balancing (SC), with the same SLA. As expected, the proposed mechanism allows to use fewer VMs to meet the same SLA as compared to random load-balancing. For instance, the expected response time (in 1/µ units) with SR 2 is 1.83 when ρ = 1.8 with n = 3 VMs, as compared to 2.5 with n = 3 VMs or 1.81 with n = 4 VMs for single-choice LB. In this example, one more VM is thus needed with single-choice to reach the same service quality. Figure 9.5 (bottom) depicts the number of VMs necessary to meet the SLA, for both policies, confirming that the proposed approach reduces the number of necessary VMs as compared to single-choice LB.

To understand the impact of the SLA on the behavior of the system, figure 9.6 depicts the average response time as a function of the request rate, with a different SLA S = 3. The firstavailable-instance policy is used with c = 3 as a threshold (SR3), with single-choice load-balancing (SC) as baseline. The fairer allocation of queries achieved with SR induces a reduction in the number of VMs needed to achieve a given SLA, as depicted in figure 9.6 (bottom). Due to the less strict SLA, less VMs are required overall as compared the previous example of figure 9.5. Finally, figure 9.7 depicts the response time and number of VMs needed when the request rate ρ(t) exhibits a diurnal pattern, for S = 2 and c = 2. With this specific example, never more than 4 VMs are needed to satisfy the SLA with SR 2 , whereas SC requires up to 6 VMs. The number of VM-hours needed decreases from 83.4 to 66.3, yielding a 21% energy cost reduction.

Wikipedia Replay

While the results presented in section 9.5 provide analytical insights on the benefits of the proposed architecture, analysis is limited to small chains (due to the increasing size of the state space when n increases). To support the belief that these benefits also hold for larger chains and in real environments, this section presents an implementation of the architecture, and its effects when subject to a real workload.

To that purpose, the load-balancer performing insertion of SR headers and flow steering as described in figure 7.3, and the server agent performing algorithms 7 and 11, are implemented as VPP plugins. As workload, a typical Web environment (Apache, MySQL, PHP, memcached) is used to host the MediaWiki software3 , with a dump of the English Wikipedia -similarly as was done in section 7.6.3. This environment is replicated across 24 VMs (each with 2 vCPUs) running on two Intel Xeon E5-2690 machines. The experiment consists of replaying traces of 24 hours of queries destined to dynamic pages from the English Wikipedia4 [START_REF] Urdaneta | Wikipedia workload analysis for decentralized hosting[END_REF], from a traffic generator hosted on a third machine. This trace is interesting not just because it represents real traffic, but also because it exhibits a diurnal pattern, for which autoscaling is desirable. As a reasonable SLA, c = 4 is taken as threshold for algorithm 7 (SR 4 ). That is, a connection request is forwarded along the chain of application instances, until reaching one with < 4 busy threads in Apache (i.e., twice the number of cores of the VM) -or reaching the last instance. As baseline, Single-Choice randomized load-balancing (SC) with a simple feedback autoscaling algorithm is used: response times are collected every two minutes, and if the response time as estimated by equation (9.37) with n+1 (respectively n-1) VMs is found to be smaller (respectively greater) than a given SLA, upscaling (respectively downscaling) is initiated. For the fairest possible comparison, the 24-hours experiment is first run with SR 4 , then the resulting average response time over the day is used as the SLA for the SC experiment, ensuring that both policies yield similar average response times.

Results are depicted in table 9.2: while the average response time for all queries is similar with both mechanisms (by design of the experiment), the running cost (in VM-hours) is reduced by 12% when using the proposed mechanism. This is further detailed in figure 9.8, which depicts the request rate as a function of the time of day, alongside the number of running VMs and the instantaneous response time for both policies. The number of VMs automatically follows the request rate, as expected, and with SR 4 the number of running VMs is constantly less than or equal to its baseline counterpart. The gain is greater at peak hours, where SR uses around 10 VMs, vs 12 or more for the baseline. Finally, figure 9.9 depicts the distribution of response times over all the queries. In addition to be able to serve queries with the same average response time with fewer VMs, the proposed architecture reduces the tail of the response time distribution (a property highly desirable in data-centers, especially when services are chained and/or parallelized [START_REF] Dean | The tail at scale[END_REF]). Notably, the 99 th percentile goes from 1.17 s with the baseline to 0.762 s with SR, and the 99.9 th percentile from 17.2 s to 1.58 s, an improvement of one order of magnitude.

Summary of Results

This chapter has introduced a unified load-balancing and autoscaling framework, which allows for local, decentralized decisions. Using Segment Routing, queries are dispatched through a chain of application instances, each of which taking local acceptance decisions, and the last of which taking autoscaling decisions. The proposed architecture yields operational benefits (due to the absence of need for monitoring) as well as service level benefits (as locally-taken decisions provide better load-balancing fairness and autoscaling reactivity). For small chains, benefits in terms of response time and number of instances are demonstrated analytically, by using the Recursive Renewal Reward technique (extended to n-dimensional state spaces). In addition, the proposed architecture has been implemented within a virtual router and evaluated with realistic traffic and workloads, confirming its applicability. Evaluation shows that it is possible to provide the same average response time as with random load-balancing, while decreasing the number of required VMs and reducing the tail latency by an order of magnitude, but also removing the need for centralized monitoring to perform autoscaling.

Conclusion

The growing demand for computing resources has led to the development of complex data center architectures -however, these architectures often provide sub-optimal resource utilization and/or high management overhead, because they fail to provide a tight integration between the applications and the network. This thesis has shown that encoding rich behavior directly at the network layer, and allowing it to transparently interact with applications, is a way to greatly improve scalability of data center architectures. Adopting such an approach has been shown to allow for providing (i) transparent task mobility, (ii) low-footprint reliable multicast, and (iii) in-band load-balancing and autoscaling -therefore allowing for greater scalability and resource usage. This is achieved by architecting network protocols on top of recently-introduced network frameworks (Segment Routing and Bit-Indexed Explicit Replication) which provide source-routing capabilities. Benefits of this approach have been demonstrated by a combination of theoretical analysis, network simulations, and evaluation of real implementations on large-scale testbeds.

In part II, the possibility of performing seamless task mobility in data centers has been investigated. Chapter 3 introduced a framework that enables zero-loss migration of Virtual Machines, using Segment Routing. This is achieved by opening a logical path comprising the source and destination host machines involved in a VM migration, and using SR with special functions to browse this path. This sets the ground for an architecture wherein task can be freely moved according to their communication patterns, rather than placed on one machine once and for all. In chapter 4, this is used as a baseline assumption, to introduce an multi-objective optimization framework aiming at (i) placing the greatest number of tasks, while (ii) migrating already running ones according to their pairwise communication patterns, so as to (iii) maximize overall achievable throughput. A resolution algorithm using the ε-constraint method is used as a baseline, and a heuristic algorithm is proposed, which allows reducing the solving time while providing solutions close to the optimal ones. Then, part III has investigated how BIER and SR could be used to provide efficient reliable multicast in data centers. Chapter 5 introduced a baseline BIER-based reliable multicast protocol: using Negative Acknowledgements, destinations having missed a copy of a multicast data packet are gathered into a set, and retransmissions are sent to those destinations only, by using a suitable BIER packet. This allows for a lightweight usage of the network resources, as shown by simulations and proven by theoretical analysis. This framework has been further extended in chapter 6, which further diminishes the pressure on costly links, by allowing retransmissions to be sent by peers close to the failing destinations, rather than the source. This is achieved by way of using Segment Routing to send NACKs through a path of local "potential retransmitting peers", each of them deciding on whether they can retransmit a copy of the missed packet, or need to send the NACK to the next candidate.

Finally, part IV has explored how Segment Routing could be used to provide efficient loadbalancing services. Chapter 7 set the basis for the part, by introducing 6LB, a load-balancing framework that allows queries to be sent (with SR) through a chain of candidate servers, each of those taking local decisions as to whether to accept the query. This enables load-balancing decisions to be taken by the application themselves, rather than by a central entity which would have to monitor the state of all application instances. This provides non-negligible benefits in terms of client quality of experience (or conversely, in terms of power needed to satisfy a predetermined 154 CHAPTER 10. CONCLUSION quality of experience), as shown by large-scale evaluation on real traces as well as theoretical analysis. Feasibility of a hardware implementation of 6LB has then been explored in chapter 8. Through the use of covert channels, the requirement of per-flow-state inherent to 6LB can be removed, allowing for a stateless implementation. This is validated by the implementation of a prototype for the NetFPGA-SUME platform, demonstrating that it is possible to perform the loadbalancing dispatching function with high throughput and negligible latency. Finally, the possibility of expanding the data-plane of 6LB to provide autoscaling has been explored in chapter 9. By sorting application instances into a fixed chain, and sending queries with SR through that chain, the last instance will be naturally under-or over-loaded if the chain itself is under-or over-dimensioned. By observing its local state and reacting accordingly, the last instance can therefore trigger upor down-scaling, which allows for monitor-less, decentralized auto-scaling. The validity of this approach is confirmed both by mathematical analysis and large-scale evaluation on real traces.

In sum, the work carried in this thesis demonstrates that it is possible to provide generic datacenter primitives where decisions are taken directly in the data-plane, rather than by a central entity. This therefore goes in complement of traditional Software Defined Networking architectures, wherein the data-plane would be constantly updated to reflect the last decisions taken by the controller, by going one level farther in granularity and realtime-ness. This has been shown to provide better resource usage and lower network usage, while requiring less monitoring and centralization.

To conclude, such an approach provide greater scalability in data center premises, both (i) by unifying the network layer and decreasing the need for out-of-band monitoring, and (ii) by providing better resource utilization as a result of this ability to take local decisions. Such architectures are therefore desirable, for reasons concerning both operational complexity and reduction of energy consumption. la méthode proposée augmente efficacement le débit total réalisable, et ceci grâce à la migration de tâches communicant ensemble vers des noeuds plus proches dans la topologie. Du fait du coût élevé en calcul de cette approche, une méthode heuristique est proposée pour calculer de manière incrémentale des solutions au coût de migration croissant, ce qui permet de retourner une approximation de la frontière de Pareto. Des simulations montrent que l'heuristique proposée permet de réduire le temps de calcul des solutions d'un à deux ordres de grandeurs, tout en fournissant des solutions proches de l'optimum.

La partie III étudie la diffusion fiable de contenus dans les centres de données. Le chapitre 5 (publié dans [START_REF] Desmouceaux | Reliable multicast with B.I.E.R[END_REF]) introduit l'utilisation de BIER pour fournir de la diffusion multipoint fiable et efficace ("BIER fiable"). Les protocoles de diffusion multipoint fiable utilisent traditionnellement un schéma reposant sur des acquittements négatifs (Negative-ACKnowledgements, NACKs), par lequel les destinations perdant des paquets le signalent à la source. La source utilise ensuite ces signaux pour produire ou bien des retransmissions point-à-point vers chacune des destinations ayant perdu le paquet, ou bien des retransmissions multipoint vers l'ensemble du groupe. Le chapitre 5 propose un protocole reposant sur des NACKs qui vise à minimiser l'impact du trafic de retransmission, en utilisant BIER. Au lieu d'utiliser des chaînes de bits (bitstrings) différentes pour chaque flux, des chaînes de bits différentes sont utilisées pour chaque paquet : une retransmission pour un paquet perdu est transmise uniquement à l'ensemble des destinations qui n'ont pas reçu ce paquet. Pour accomplir cela, la source rassemble les NACKs envoyés par les différentes destinations pendant un court intervalle de temps, et ce afin de construire une chaîne de bits appropriée pour la retransmission. Ce protocole a été implémenté dans un simulateur de trafic réseau (ns3 [147]), ce qui a permis de conduire une campagne de simulations dans différentes topologies, avec différents scénarios de perte de paquets. Les simulations montrent que, surtout dans le cas où un sous-ensemble du centre de données est caractérisé par des pertes localisées, le trafic total émis par le protocole est réduit (i) comparé aux retransmissions multipoint, car l'arbre tout entier n'a pas à être inondé en cas de perte, et (ii) comparé aux retransmissions point-à-point, car le trafic n'est pas dupliqué sur les liens limitants. Un modèle mathématique vient ensuite compléter l'analyse, permettant ainsi de quantifier l'empreinte du traffic de retransmission pour les trois mécanismes (retransmissions point-à-point, multipoint, et BIER), dans des topologies arborescentes arbitraires. Une approximation au premier ordre en α → 0 (où α est le taux de perte de paquets sur les liens) est calculée dans le théorème 5.1. Cette approximation montre que le trafic dû aux retransmissions se comporte en L log L avec les retransmissions BIER, comparé à L log 2 L pour les retransmissions point-à-point, et L 2 pour les retransmissions multipoint (où L est le nombre de liens de l'arbre) : ceci confirme théoriquement les bénéfices précédemment mentionnés.

Le chapitre 6 (soumis en tant que [START_REF] Desmouceaux | Reliable BIER with peer caching[END_REF]) étend l'utilisation du BIER fiable pour fournir des retransmissions multipoint assistées par des pairs. Une extension au plan de données de BIER est proposée, afin de permettre aux destinations d'apprendre quels pairs (i.e., des destinations ayant souscrit au même flux multipoint) sont proches topologiquement. Ceci est accompli grâce à une chaîne de bits des pairs (peerstring) incluse dans chaque paquet, qui est mise à jour par chaque noeud afin qu'elle contienne invariablement l'ensemble des destinations atteintes par ce noeud et pour ce paquet. Ensuite, lorsqu'un paquet est perdu, SRv6 est utilisé par les destinations n'ayant pas reçu le paquet afin de diriger les NACKs à travers un chemin comprenant un ou plusieurs pairs et la source, jusqu'à trouver un pair (ou en dernier recours, la source) capable de transmettre une retransmission du paquet perdu. Différentes politiques de sélection des pairs sont proposées et évaluées mathématiquement. En premier lieu, deux politiques statiques simples sont analysées : la sélection aléatoire d'un pair dans le sous-arbre d'une destination, et la sélection d'un pair pré-désigné dans ce même sous-arbre. L'analyse mathématique révèle que la seconde politique génère moins de trafic de retransmission (car les retransmissions sont envoyées uniquement par un pair), mais a moins de chance de réussir (car si le pair pré-désigné n'a pas reçu le paquet non plus, aucun pair ne recevra de retransmission). En second lieu, une politique dynamique est introduite, selon laquelle chaque pair essaie d'apprendre dynamiquement de quel autre pair il a le plus de chance d'obtenir une retransmission. L'architecture proposée est évaluée grâce à des simulations réseaux paquet par paquet, dans différentes topologies et avec différentes politiques de sélection des pairs. L'évaluation montre qu'en utilisant cette architecture, il est possible de réduire la charge sur les liens centraux (comparé au chapitre 5) en augmentant la localité des retransmissions, ce qui permet ainsi de réduire l'empreinte totale du trafic réseau. La partie IV étudie l'équilibrage de charge dans les centres de données. Le chapitre 7 (publié dans [START_REF] Desmouceaux | SRLB: The power of choices in load balancing with segment routing[END_REF][START_REF]6LB: Scalable and application-aware load balancing with segment routing[END_REF]) introduit l'utilisation de SRv6 pour l'équilibrage de charge utilisant les charges instantanées des applications. Traditionnellement, les équilibreurs de charges des centres de données sont répliqués, à des fins de fiabilité et de passage à l'échelle. Lorsque c'est le cas, le hachage cohérent (consistent hashing) est utilisé, afin de s'assurer qu'un flux est toujours assigné au même serveur le long de son existence, et ce quel que soit l'équilibreur de charge par lequel il est traité, et même en cas de reconfiguration de l'ensemble des instances d'application. Cependant, une telle approche fait que les requêtes sont assignées aux instances d'application sans prendre en compte leur charge instantanée. Pour remédier à cela, le chapitre 7 introduit l'utilisation de SRv6 pour diriger le premier paquet (TCP SYN) d'une requête au travers d'une chaîne de deux applications candidates aléatoires, chacune d'entre elles décidant successivement si elle peut ou non accepter la requête, selon son état local. Ceci permet d'avoir un mécanisme d'équilibrage de charge qui prend en compte les états courants des applications, directement au sein de la couche réseau, et sans recourir à une observation centralisée de l'état des applications. Pour éviter un trafic triangulaire non nécessaire, les paquets suivants sont envoyés directement à l'instance ayant accepté la connection. Ceci est accompli grâce à un protocole de signalisation intra-bande, en utilisant des fonctions SRv6 dédiées entre les équilibreurs de charges et les serveurs afin d'installer et de retirer l'état nécessaire pour chaque connexion. La fiabilité du mécanisme au regard des changements de l'ensemble des équilibreurs de charge et/ou des applications est assurée grâce à un hachage cohérent qui assigne les flux à des listes (et non pas des singletons) d'instances d'applications. Un modèle mathématique du temps de réponse observé par les clients est ensuite introduit, pour des arrivées Poisson et des services suivant une loi exponentielle. Grâce à ce modèle, il est possible de quantifier les bénéfices de l'approche proposée en termes d'espérance du temps de réponse (ainsi que de la queue de la distribution correspondante), d'équité de charge des serveurs, et de réduction d'énergie. L'analyse démontre que, sous l'hypothèse raisonnable que le délai réseau unidirectionnel entre deux machines est inférieur au temps de service moyen, les performances en temps de réponse sont toujours améliorées, comparé à de l'équilibrage de charge aléatoire (théorème 7.1). L'architecture proposée a été implémentée comme module pour un routeur virtuel (VPP), qui extrait de manière transparente des informations d'un serveur HTTP standard (Apache) grâce à une mémoire partagée, et utilise ces informations pour prendre des décisions d'acceptation de connexion. Une évaluation conduite sur une plateforme d'essai comprenant 48 instances, et utilisant à la fois du trafic synthétique et des traces réelles, confirme ces bénéfices. L'évaluation montre notamment qu'il est possible de fournir la même qualité de service moyenne aux clients (i.e., le même temps de réponse moyen), tout en réduisant le nombre de machines virtuelles nécessaires de 17% comparé à l'équilibrage de charge aléatoire.

Le chapitre 8 (publié dans [START_REF] Pit-Claudel | Stateless loadaware load balancing in P4[END_REF]) explore ensuite la faisabilité d'implémenter, dans des périphériques matériels, une architecture d'équilibrage de charge sans état et utilisant les charges instantanées des applications. En effet, implémenter des équilibreurs de charge dans des périphériques matériels est désirable pour des raisons d'efficacité et de passage à l'échelle. Cependant, ceci est incompatible avec le maintien d'état par connexion, ce qui est notamment requis par les architectures utilisant les charges instantanées des applications. Pour contourner ce problème, le chapitre 8 explore l'utilisation de canaux cachés (covert channels) pour transporter cet état directement dans les en-têtes des paquets. Les connexions sont établies grâce au même mécanisme que dans le chapitre 7 : les paquets d'établissement de connexion sont envoyés à travers une chaîne d'applications candidates, jusqu'à ce que l'une d'entre elles accepte (selon son état instantané) de servir la requête. Cependant, au lieu d'installer de l'état dans l'équilibreur de charge, l'instance qui a accepté la connexion communique au client sa position dans la liste SRv6, via un canal caché. Cette valeur est alors automatiquement reflétée par le client jusqu'à atteindre l'équilibreur de charge, qui peut ensuite utiliser cette information pour diriger le paquet vers le serveur ayant initialement accepté la connexion. Pour réaliser un tel canal caché en pratique et permettre son utilisation par des clients standards et non-modifiés, le chapitre 8 propose d'utiliser les bits de poids faible des estampille de temps (timestamps) du protocole TCP. De plus, l'équilibreur de charge s'assure de la fiabilité du système en cas de changement de l'ensemble des applications, grâce à un versionnage des tables de hachage cohérent -ainsi, les paquets se voient insérer un en-tête SRv6 comprenant les instances qui étaient précédemment retournées par la fonction de hachage cohérent. Un prototype a été réalisé sur une carte réseau reprogrammable (utilisant le langage P4 [START_REF] Bosshart | P4: Programming protocol-independent packet processors[END_REF] et ciblant la plateforme NetFPGA-SUME [START_REF] Zilberman | NetFPGA SUME: Toward 100 Gbps as research commodity[END_REF]), prourant la faisabilité d'implémentation de cette architecture. Des simulations paquet par paquet de cette implémentation montrent que la latence due à l'analyse des en-têtes des paquets est négligeable (de l'ordre de 10 µs). Des simulations du mécanisme de hachage cohérent proposé montrent que la fiabilité est améliorée d'un ordre de grandeur, comparé à des mécanismes n'utilisant pas le versionnage. En résumé, cette architecture permet d'obtenir les mêmes bénéfices que ceux introduits au chapitre 7, tout en bénéficiant d'une implémentation matérielle à faible latence et haut débit, et en améliorant la fiabilité du hachage cohérent.

Le chapitre 9 (soumis en tant que [START_REF] Desmouceaux | Joint monitorless load-balancing and autoscaling for zero-wait-time in data centers[END_REF]) introduit une architecture d'auto-équilibrage sans observation centralisée, utilisant SRv6. Afin de satisfaire les variations journalières en demande de trafic tout en satisfaisant un niveau de service prédéfini, les centres de données utilisent traditionnellement l'auto-équilibrage (autoscaling) afin d'adapter en temps réel le nombre d'instances répliquées d'un service donné. Les auto-équilibreurs utilisent des architectures centralisées pour prendre ces décisions, ce qui induit un surcoût d'observation. Dès lors, le chapitre 9 étudie l'utilisation de SRv6 pour fournir un service commun de distribution de charge et d'auto-équilibrage, sans observation centralisée. Les instances de l'application répliquée sont ordonnées le long d'une chaîne fixe, et les nouvelles requêtes sont envoyées le long de cette chaîne, jusqu'à ce que l'une d'entre elles accepte de servir la connexion -et ce en utilisant le plan de données introduit au chapitre 7. De plus, la dernière instance de la chaîne garde trace de l'historique de son propre état de charge afin de déclencher l'ajout ou le retrait d'un serveur en fin de chaîne. En effet, le fait que la dernière instance de la chaîne reçoive trop peu (ou trop) de requêtes est un indicateur que la chaîne est sur-provisionnée (ou sous-provisionnée). Un modèle de Markov de la performance du système est introduit, et permet d'exprimer stochastiquement le nombres de requêtes servies par chaque serveur. Ce modèle est résolu grâce à la technique RRR (Recursive Renewal Reward [START_REF] Gandhi | Exact analysis of the M/M/k/setup class of Markov chains via recursive renewal reward[END_REF]), ce qui permet d'exprimer, pour de petites chaînes, l'espérance du temps de réponse perçu par les clients -et ainsi d'en déduire les gains possibles d'énergie offerts par le système (i.e., la réduction possible du nombre de serveurs requis pour assurer la même qualité de service qu'avec de l'équilibrage de charge aléatoire). L'architecture a été implémentée en tant que module pour VPP, et une évaluation avec des traces de trafic confirme que ces bénéfices peuvent également être observés dans des environnements réels. Ainsi, comparé à l'équilibrage de charge aléatoire, le même temps de réponse moyen peut être offert aux clients, tout en réduisant le nombre de machines virtuelles nécessaires, et en diminuant simultanément la queue de la distribution de temps de service.

Enfin, le manuscrit de thèse se termine par une conclusion donnée en partie V.

List of Figures

This thesis comprises Premi èrement, pour la mobilit é des t âches, SR est utilis é pour fournir un service de migration de machine virtuelles sans perte. Cela ouvre l'opportunit é d' étudier comment orchestrer le placement et la migration de t âches afin de (i) maximiser le d ébit intert âches, tout en (ii) maximisant le nombre de nouvelles t âches plac ées, mais (iii) minimisant le nombre de t âches migr ées. Deuxi èmement, pour la distribution fiable de contenu, BIER est utilis é pour fournir un pro-tocole de multicast fiable, dans lequel les retransmissions de paquets perdus sont cible és vers l'ensemble pr écis de destinations n'ayant pas rec ¸u ce paquet : ainsi, le surco ût de trafic est minimis é. Pour diminuer la charge sur la source, cette approche est étendue en rendant possibles des retransmissions par des pairs locaux, utilisant SR pour trouver un pair capable de retransmettre. Troisi èmement, pour l' équilibre de charge, SR est utilis é pour distribuer des requ êtes à travers plusieurs applications candidates, chacune prenant une d écision locale pour accepter ou non ces requ êtes, fournissant ainsi une meilleure équit é de r épartition compar é aux approches centralis ées. La faisabilit é d'une impl émentation mat érielle de cette approche est étudi ée, et une solution (utilisant des canaux cach és pour transporter de fac ¸on invisible de l'information vers l' équilibreur) est impl ément ée pour une carte r éseau programmable de derni ère g én ération. Finalement, la possibilit é de fournir de l' équilibrage automatique comme service r éseau est étudi ée : en faisant passer (avec SR) des requ êtes à travers une chaîne fix ée d'applications, l' équilibrage est initi é par la derni ère instance, selon son état local. Title: Network-Layer Protocols for Data Center Scalability Keywords: Data-center networking, Task Mobility, Multicast, Load Balancing, Segment Routing Abstract: With the development of demand for computing resources, data center architectures are growing both in scale and in complexity. In this context, this thesis takes a step back as compared to traditional network approaches, and shows that providing generic primitives directly within the network layer is a great way to improve efficiency of resource usage, and decrease network traffic and management overhead. Using two advanced network architectures, Segment Routing (SR) and Bit-Indexed Explicit Replication (BIER), network layer protocols are designed and analyzed to provide three high-level functions: (1) task mobility, (2) reliable content distribution and (3) load-balancing. First, task mobility is achieved by using SR to provide a zero-loss virtual machine migration service. This then opens the opportunity for studying how to orchestrate task placement and migration while aiming at (i) maximizing the inter-task throughput, while (ii) maximizing the number of newly-placed tasks, but (iii) minimizing the number of tasks to be migrated. Second, reliable content distribution is achieved by using BIER to provide a reliable multicast protocol, in which retransmissions of lost packets are targeted towards the precise set of destinations having missed that packet, thus incurring a minimal traffic overhead. To decrease the load on the source link, this is then extended to enable retransmissions by local peers from the same group, with SR as a helper to find a suitable retransmission candidate. Third, load-balancing is achieved by way of using SR to distribute queries through several application candidates, each of which taking local decisions as to whether to accept those, thus achieving better fairness as compared to centralized approaches. The feasibility of hardware implementation of this approach is investigated, and a solution using covert channels to transparently convey information to the load-balancer is implemented for a state-of-the-art programmable network card. Finally, the possibility of providing autoscaling as a network service is investigated: by letting queries go through a fixed chain of applications using SR, autoscaling is triggered by the last instance, depending on its local state.

Universit é Paris-Saclay

Espace Technologique / Immeuble Discovery Route de l'Orme aux Merisiers RD 128 / 91190 Saint-Aubin, France

  Réseaux de centres de données, Mobilité, Multicast, Équilibre de charge, Seg-. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.1 Network Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.2 Data Center Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Extending the Network Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Segment Routing (SR) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.2 Bit-Indexed Explicit Replication (BIER) . . . . . . . . . . . . . . . . . . . . 1.3 Thesis Statement: Augmenting Data Centers through the Network Layer . . . . . 1.4 Definitions and Working Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . 2 Thesis Contributions 2.1 Thesis Summary and Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 List of Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . II Task Mobility 3 Zero-Loss Virtual Machine Migration with IPv6 Segment Routing 3.1 Statement of Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.2 Chapter Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 SR-based Migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Detailed Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 Detailed Migration Process . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1 Ping (illustration) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.2 HTTP Workload . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.3 Iperf Workload . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.4 Iperf Sink Workload . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Summary of Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Flow-Aware Workload Migration in Data Centers 4.1 Statement of Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.2 Chapter Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Data Center Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix x Chapter 1

Figure 1 . 1 -

 11 Figure 1.1 -Virtual router operation

Figure 1 . 4 -

 14 Figure 1.4 -Advanced data center network toplogies

Figure 1 . 6 -

 16 Figure 1.6 -Example of BIER bitstring processing

Figure 1 . 8 -

 18 Figure 1.8 -Toy network example

Figure 3 . 1 -

 31 Figure 3.1 -SR VM Migration: migrating VM.

Figure 3 . 2 -

 32 Figure 3.2 -SR Migration: detailed example.

Figure 3 . 3 -

 33 Figure 3.3 -Illustration of SR migration: ping experiment for the different mechanisms.

CHAPTER 3 .Figure 3 . 5 -

 335 Figure 3.5 -Evaluation of SR migration: iperf experiments with 50 clients

Figure 3 . 6 -

 36 Figure 3.6 -Evaluation of SR migration: iperf sink experiments with 50 clients

Figure 4 . 1 -

 41 Figure 4.1 -Competition between objective functions

Figure 4 . 2 -

 42 Figure 4.2 -Network topology used in the example run

Figure 4 . 3 -

 43 Figure 4.3 -Multi-objective model example run: solutions and Pareto-optimal solutions.

  (a) Tree topology, 20 machines, 40 tasks. Tree topology, 20 machines, 100 tasks.

Figure 4 . 4 -

 44 Figure 4.4 -Pareto front approximation: Exact Pareto front and approximate Pareto fronts from Algorithm 4 for ∆ ∈ {1, 3, 5}.

Figure 4 . 5 -

 45 Figure 4.5 -Pareto front approximation: throughput improvement for 20 and 30 migrations, θ20/θ0 and θ30/θ0. Exact Pareto front vs approximate Pareto fronts from Algorithm 4 for ∆ ∈ {1, 2, 3, 4, 5}. 20 different scenarios.

Figure 4 . 6 -

 46 Figure 4.6 -Pareto front approximation: time to generate the approximate Pareto front, for up to B = 30 migrations. Exact Pareto front (Algorithm 3) vs approximate Pareto fronts (Algorithm 4) for ∆ ∈ {1, 2, 3, 4, 5}. 20 different scenarios.

(a)

  Initial transmission (b) Multicast retransmission (c) Unicast retransmission (d) BIER retransmission

Figure 5 . 1 -

 51 Figure 5.1 -Comparison of different reliable multicast mechanisms. In this example, two clients do not receive a packet and send a NACK to the source (a). Multicast retransmissions (b) will vainly incur traffic towards the clients that had successfully received the packet. With unicast retransmissions (c), identical packets will be transmitted on the same link. BIER retransmissions (d) ensure that the traffic footprint is minimal.

Figure 5 . 2 -

 52 Figure 5.2 -BIER shim layer

Algorithm 6

 6 Reliable BIER Destination Operation ∆tretry ← NACK retransmission delay ℓ ← NACK retransmission limit F ← unique flow identifier L ← {} ⊲ seqnum of lost packets T ← {} ⊲ NACK transmit timers for lost packets N ← {} ⊲ number of times a NACK has been sent C ← {} ⊲ recovered packets pending for app n ← 0 ⊲ next expected seqnum for each incoming packet p with flow F , seq S do if

  retransmissions, core link #1 Unicast retransmissions, core link #1 Multicast retransmissions, core link #1 (a) Link usage, between core router and first aggregation router. retransmissions, core link #2Unicast retransmissions, core link #2 Multicast retransmissions, core link #2 (b) Link usage, between core router and second aggregation router. CDF of the number of clients in BIER retransmit bitstrings.

Figure 5 . 4 -

 54 Figure 5.4 -Uncorrelated localized losses experiment. BIER retransmits vs unicast and multicast retransmits.

  retransmissions, core link #1 Unicast retransmissions, core link #1 Multicast retransmissions, core link #1 (a) Link usage, between core router and first aggregation router. retransmissions, core link #2Unicast retransmissions, core link #2 Multicast retransmissions, core link #2 (b) Link usage, between core router and second aggregation router. Delivery ratio for the multicast flow. CDF of the number of clients in BIER retransmit bitstrings.

Figure 5 . 5 -

 55 Figure 5.5 -Correlated localized losses experiment. BIER retransmits vs unicast and multicast retransmits.

Figure 5 . 6 -

 56 Figure 5.6 -Unlocalized bursty losses experiments. 19 simulations with different loss probabilities.

60 CHAPTER 5 Figure 5 . 7 -

 60557 Figure 5.7 -Influence of ∆tagg: link usage between the source and the core router.

  source to router 17 link Unicast retransmissions, source to router 17 link Multicast retransmissions, source to router 17 link(a) Link usage, between the source and router 17 (London).

  retransmissions, example 1st hop link Unicast retransmissions, example 1st hop link Multicast retransmissions, example 1st hop link (b) Link usage, between router 17 (London) and router 5. Delivery ratio for the multicast flow. Node 17 is the source. CDF of the number of clients in BIER retransmit bitstrings.

Figure 5 . 8 -

 58 Figure 5.8 -ISP topology experiment. BIER retransmits vs unicast and multicast retransmits.

Figure 5 . 9 -

 59 Figure5.9 -Network topology (picture from[START_REF]Internet Topology Zoo[END_REF]).

Figure 5 .

 5 Figure 5.10 -Notation

Figure 5 . 11 -

 511 Figure 5.11 -Reliable BIER examplified -solid blue arrows represent successful transmissions, dashed blue arrows represent unsuccessful transmissions, and dashed black arrows represent NACKs. A NACK is sent by two nodes (a) upon receipt of a subsequent packet in the stream. After retransmission (b), the last node still has not received a copy of the packet, and sends a second NACK upon timeout of ∆tretry (Algorithm 6). The third attempt (c) ensures that delivery is successful. Figure (d) shows the total number of attempts T [i] at each node [i], and total transmissions X [i]→j at each link [i] → [[i], j]. The total number of packets sent in the tree (the sum of all X [i]→j ) is M B [i] = 11.

  5.2). Consider the transmission from a node [i] to a node [[i], j]: transmitted packets can be classified into two categories: (i) packets sent until the subtree [[i], j] is covered, and (ii) packets flooded by [i] inside the subtree [[i], j] after it has been covered. The latter packets come from retransmissions from the source [i] that are due to other subtrees [[i], k] having not yet been covered. Let U [i]→[[i],j] be the number of packets that fall into the second category. The number of floods is T [i] -X [i]→j (i.e., the number of times [i] transmits after [[i], j] has already received enough packets): index these floods with f ∈ [1, T [i] -X [i]→j ]. For each of these floods, let Y f [i]→j be a Bernoulli variable of parameter (1α) representing the success of transmission on the link [i] → [[i], j], and F f [[i],j] be a variable representing the number of packets flooded in the subtree [[i], j]. The number of unnecessary packets is then, for each of these floods, one packet (from [i] to [[i], j]) plus, if the transmission succeeded (i.e., if Y f [i]→j = 1), the number of packets F f [[i],j] resulting from a multicast flood sourced at [[i], j]:

Figure 5 . 12 -

 512 Figure 5.12 -Example of multicast reliable transmission. In addition to the conventions of figure 5.11, green arrows represent unnecessary retransmissions. In the second attempt (b), node [[1], 1] floods one unnecessary packet. In the third attempt (c), the root floods three unnecessary packets, and node [[1], 2] floods one unnecessary packet. Figure (d) shows the total number of attempts T [i] at each node [i], total transmissions X [i]→j at each link [i] → [[i], j], and the total number of unnecessary packets U [i]→[[i],j] flooded by each node [i] in the subtree [[i], j]. The total number of packets sent in the tree (the sum of all X [i]→j and U [i]→[[i],j] ) is M m [1] = 16.

3 . 5 . 3 .

 353 [c], connected to the source in d([c]) hops, the number of retransmissions from the source before successful delivery of the packet to [c] is a random variable, R [1]→[c] , whose mean is described in proposition 5.Proposition The mean value of R [1]→[c] is:

. 9 )

 9 Proof. Let [c] be a destination at depth d([c]); for simplicity, write d = d([c]). The unicast retransmission will succeed if the packet successfully traverses d successive links: the probability of a unicast success from the source to [c] is therefore (1α) d . Let Z [1]→[c] represent the number of trials before (and not counting) the unicast success. Z [1]→[c] is geometrically distributed with parameter (1α) d :

66 CHAPTER 5 Figure 5 . 13 -

 665513 Figure 5.13 -Number of packets M B [1] (BIER retransmissions), M m [1] (multicast retransmissions) and M u [1]

68 CHAPTER 5 Figure 5 . 14 -

 685514 Figure 5.14 -Number of packets M B [1] (BIER retransmissions), M m [1] (multicast retransmissions) and M u [1]

  and other factors of the form P[X [[i],j ′ ] < k]. Let j(σ) be one of the j such that P[X [[i],j(σ)] = k] appears in the term. An upper-bound for the other factors is 1, effectively keeping only the contribution of P[X [[i],j(σ)] = k]:

76 CHAPTER 6 .

 766 RELIABLE BIER WITH PEER CACHINGSource

Figure 6 . 1 -

 61 Figure 6.1 -Comparison of different reliability mechanisms. In this example, red clients are assumed to have missed reception of a packet and sent a NACK.With "standard" multicast retransmissions[START_REF] Adamson | Reliable messaging for tactical group communication[END_REF], the source re-floods the whole tree as a result (a). With unicast retransmissions[START_REF] Paul | Reliable Multicast Transport Protocol (RMTP)[END_REF], the source will send multiple copies of the same packet over the same link (b). With BIER retransmissions[START_REF] Desmouceaux | Reliable multicast with B.I.E.R[END_REF], the source re-floods only the subset of failing destinations (c). With peer-based BIER retransmissions introduced in this chapter, the NACKs are sent to a peer that had cached the packet, which then floods the failing destinations, spanning a smaller tree (d).

  (A, B, S) Has copy -> don't forward SR-NACK (a) SR-NACK to peer+source, peer retransmission.

  (A, B, C, S) Has copy -> don't forward SR-NACK (b) SR-NACK to two peers+source, second peer retransmission. (A, B, C, S) No copy -> forward SR-NACK SR-NACK (A, B, C, S) (c) SR-NACK to two peers+source, source retransmission.

Figure 6 . 2 -

 62 Figure 6.2 -SR-based recovery scenarios: SR-NACK sent by A. A red destination indicates that it is not able to satisfy the retransmission request, whereas a green destination indicates that it is.

Figure 6 . 3 -

 63 Figure 6.3 -Example of single-and double-peerstring operation

82 CHAPTER 6 Figure 6 . 4 -

 82664 Figure 6.4 -PDF of the recovery at 2k hops (conditioning to a loss).

Figure 6 . 5 -

 65 Figure 6.5 -Example with a policy X over n = 10 destinations, with K = 4 destinations (2, 5, 6 and 7) having lost a packet. Arrows indicate recovery requests (SR-NACKs): destination 2 requests recovery from peer 1; destination 5 from peer 2; and destinations 6 and 7 from peer 10; SX = 3 recovery requests are successful (i.e., those from 2, 6 and 7), as peers 1 and 10 have correctly received the original packet; TX = 2 retransmissions are performed (from 1 and 10).

Figure 6 .

 6 Figure 6.4 depicts the distribution of the minimum distance from a destination that missed a multicast transmission, and to a destination successfully receiving a multicast transmission, f D (2k). Two tree topologies with 256 destinations are considered: (i) h = 5, (c 0 , . . . , c 5 ) = (1, 2, 2, 2, 32); and (ii) h = 7, (c 0 , . . . , c 7 ) = (1, 2, 2, 2, 2, 2, 8).It can be observed that the expected shortest distance between a failing destination and a successful peer is lower for the exponential loss model (in which the loss probability for top links is lower) than for the linear loss model. In other words, as expected, if top links are less lossy, it is more likely for a failing destination to be able to recover from a close destination.This confirms the intuition that in networks with this type of loss distribution (such as can be envisioned in data-centers, or in networks where the "last hop" is for instance a wireless link or a consumer grade residential xDSL), selection of local recovery peers (e.g., peers that are in the same subtree or in the immediately upper subtree) should be preferred to selection of peers farther away.

  Thus P[TD = 1|K = k] = P[SD = k|K = k] + P[SD = 1|K = k] and P[TD = 0|K = k] = P[SD = 0|K = k], yielding equation (6.7).

Figure 6 . 6 -

 66 Figure 6.6 -Number of destinations able to obtain a retransmission of a packet from a peer, when k = 35 out of n = 100 have not received the initial multicast transmission.

Figure 6 . 7 -

 67 Figure 6.7 -Probability that a fraction (1δ) of destinations successfully receive the packet directly from the source (in the first transmission) or from a contacted peeri.e., excluding source retransmissions, with n = 100, β = 0.2.

Figure 6 . 8 -

 68 Figure 6.8 -Success ratio (after 1st recovery) for the ε-greedy policy, for static and dynamic scenarios.

Figure 6 . 9 -

 69 Figure 6.9 -Number of destinations that did not receive the multicast transmission from the source, and which are selecting peers 0 and 32 under the ε-greedy policy, in static and dynamic scenarios

Figure 6 . 10 -

 610 Figure 6.10 -Datacenter topology for simulations of section 6.7

Figure 6 . 11 -

 611 Figure 6.11 -Data-center topology: clustered vs random peer selection within same subtree, and for different values of the Gilbert-Elliott link loss probability α.

Figure 6 . 12 -

 612 Figure 6.12 -Data-center topology: clustered peer selection in two subtrees vs in one subtree, for different values of the Gilbert-Elliott link loss probability α.

  ε-greedy (c) Adaptive (ε-greedy) strategy in subtree: histogram of peer choices when in non-exploratory mode (α = 0.10).

Figure 6 . 13 -

 613 Figure 6.13 -Data-center topology with one ideal peer per rack: ε-greedy peer selection in subtree vs random, for different values of the Gilbert-Elliott link loss probability α.

  link usage (b/s) Average loss probability (⍺) BIER-PEER ε-greedy, source to core link BIER-PEER random, source to core link BIER, source to core link (a) Link usage between node 11 (New York) and its router Histogram of peer contributions to recoveries (α = 0.10), for random and ε-greedy strategies. Node 11 (NYC) is the source. Adaptive (ε-greedy) strategy in subtree: aggregation from source and peers.

Figure 6 . 14 -Figure 6 . 15 -

 614615 Figure 6.14 -ISP topology: ε-greedy vs. random peer selection in subtree, for different values of the Gilbert-Elliott link loss probability α.

100 CHAPTER 7 .Figure 7 . 1 -

 100771 Figure 7.1 -6LB architecture: load-balancers assign a flow to a set of candidate instances, through which the connection is passed until one accepts the connection (section 7.2). The flow is then pinned to the instance having accepted the connection (section 7.4).

Figure 7 . 2 -

 72 Figure 7.2 -6LB consistent hashing: when a flow is rebalanced to another load-balancer instance by the edge router, consistent hashing (section 7.3) allows the flow to re-browse the same set of candidate instances, then to be re-pinned to the one that had accepted the connection (section 7.4).

Figure 7 . 3 -

 73 Figure 7.3 -TCP connection pinning from client c to application v, with 3 instances running on m1, m2, m3. The path (source, segments, destination) is indicated between curly braces. The active segment is underlined.

Figure 7 .Algorithm 9

 79 Figure 7.4b illustrates how this scheme is resilient to changes to the pool of application instances, by showing how removing application instance 0 modifies the tables t[j] from the example of

Figure 7 . 4 -Figure 7 . 5 -

 7475 Figure 7.4 -Example of permutation tables p[i] and lookup table t (C = 2, M = 7, N = 4), before and after removal of application instance 0

Figure 7 . 6 -

 76 Figure 7.6 -Performance analysis of 6LB: mean response time E[T ]. SC vs SR2, SR4 and SR8.

Figure 7 . 7 -

 77 Figure 7.7 -Performance analysis of 6LB: worst-case response time with delay E[ T ]. SC vs SR2, SR4 and SR8.

Figure 7 .

 7 Figure 7.6 depicts the value of E[T ] for different SR c policies, and for λ ∈ [0, 1). As a reference, this value is compared to1 1-λ , the expected response time for SC, when clients are randomly assigned to one application instance. It can be observed that the SR c policies uniformly yield an improvement over SC. When c is small, lower values of λ yield the highest gain; when c is important, higher values of λ yield the highest gain. For example, choosing SR 8 offers an improvement over SR 4 only when λ ≥ 0.983, and thus might rarely be suitable.

  which completes the proof.

Figure 7 .Figure 7 . 8 -

 778 Figure 7.7 gives the expected response time including delay, in the "worst-case" in which the network delay equals the job duration (δ = 1).

Figure 7 . 9 -

 79 Figure 7.9 -Performance analysis of 6LB: probability of wrongful rejection for different SRc policies.

Figure 7 . 10 -Figure 7 . 11 -

 710711 Figure 7.10 -Performance analysis of 6LB: CDF of response time for λ = 0.88. SC vs SR2, SR4 and SR8.

4 Figure 7 . 12 -

 4712 Figure 7.12 -Performance analysis of 6LB: reduction in number of instances when using SR4 vs SC, for different 90-th percentile SLAs

Figure 7 . 13 -

 713 Figure 7.13 -Connection acceptance policies evaluation: average page load time. SC vs SR4, SR8, SR16, SR dyn .

Figure 7 . 14 -

 714 Figure 7.14 -Connection acceptance policies evaluation: instantaneous server load, ρ = 0.89. SC vs SR4.

Figure 7 . 15 -Figure 7 . 16 -

 715716 Figure 7.15 -Connection acceptance policies evaluation: CDF of page load time: ρ = 0.71 (top) and ρ = 0.89 (bottom). SC vs SR4, SR8, SR16, SR dyn .

Figure 7 . 17 -

 717 Figure7.17 -Influence of the consistent hashing table size M : SR4, ρ = 0.89.

Figure 7 . 18 -

 718 Figure 7.18 -Connection resets when removing x application instances and simultaneously switching to another 6LB instance

Figure 7 . 19 -

 719 Figure 7.19 -SYN → SYN-ACK latency. SR4, ρ = 0.71.

Figure 7 . 20 -

 720 Figure 7.20 -Average page load time for centralized policies

Figure 7 . 21 -

 721 Figure 7.21 -Average page load time for different variances of job times, ρ = 0.71.

Figure 7 . 24 -Figure 7 . 25 -

 724725 Figure 7.24 -6LB Wikipedia replay: CDF of page load time over the 24 hours

Figure 8 . 1 -

 81 Figure 8.1 -SHELL overview

Figure 8 . 3 -

 83 Figure 8.3 -Example TCP TLV parsing in the P4 LB, for d off = 11.

Figure 8 . 4 -

 84 Figure 8.4 -SHELL P4 dataplane evaluation: per-packet latency for a burst of 4800 packets (c = 2, h = 2, d max off = 11)

8. 5 . 2

 52 Effect of TCP Parsing on Latency and FPGA Resources As explained in section 8.4, the TCP timestamp option is parsed by matching a predefined set of option headers, thus the greater d max off (maximum admissible size of parsed TCP headers), the more program branches -and the greater the latency. This is evaluated by testing different values of d max off ∈ {8, 11, 13, 15}, and depicted in figure 8.5. The latency varies from 1.8 µs (d max off = 8) to 2.1 µs (d max off = 15) for an empty egress queue, and its average after the egress queue has filled up goes from 7.7 µs (d max off = 11) to 8.5 µs (d max off = 15).

  (b) h = 3, different number of buckets B. Top: infinite connections. Bottom: connections according to model.

Figure 8 . 6 -

 86 Figure 8.6 -SHELL consistent hashing evaluation: probability for a connection started at time t = 0 to be reset after n backend reconfigurations. 500 application instances, c = 2 choices. Backend reconfiguration rate model from[START_REF] Miao | SilkRoad: Making stateful layer-4 load balancing fast and cheap using switching ASICs[END_REF][START_REF] Roy | Inside the social network's (datacenter) network[END_REF].

Figure 9 . 5 -

 95 Figure 9.5 -Expected response time E[T ] and number of VMs needed to satisfy SLA, as a function of the request rate. SLA = 2. SR with threshold c = 2 (SR2) vs single-choice (SC).

36 ) 2 : 9 . 2 .

 36292 The similar form between equations (9.29)-(9.36) and equations (9.2)-(9.11) allows computing E[z Nn ] similarly as for n = Theorem The probability generating function of N n has the form E[z Nn ] = En+Fnz Cn-Dnz , where C n , D n , E n , F n are real constants, which depend on c and n.

Figure 9 . 6 -

 96 Figure 9.6 -Expected response time E[T ] and number of VMs needed to satisfy SLA, as a function of the request rate. SLA = 3. SR with threshold c = 3 (SR3) vs single-choice (SC).

Figure 9 . 7 -

 97 Figure 9.7 -Expected response time E[T ] and number of VMs needed to meet SLA = 2, for diurnal request rate pattern ρ(t) = 2.7(1 + sin(2π(t/24 -1/4)). SR with threshold c = 2 (SR2) vs single-choice (SC).

Figure 9 . 8 -Figure 9 . 9 -

 9899 Figure 9.8 -Wikipedia replay: mean response time (moving average with 10 min bins) and number of needed VMs, over 24 hours. SR with c = 4, vs SC with SLA = 0.29 s. For an identical average response time, SR4 consistently reduces the required number of VMs.

  89 figures, listed below. 1.1 Virtual router operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Three-tiered data center network topology [34] . . . . . . . . . . . . . . . . . . . . 1.3 Fat-tree data center network topology [35] with k = 4 . . . . . . . . . . . . . . . . 1.4 Advanced data center network toplogies . . . . . . . . . . . . . . . . . . . . . . . . 1.5 IPv6 Segment Routing Header [47] . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.6 Example of BIER bitstring processing . . . . . . . . . . . . . . . . . . . . . . . . . 1.7 Example of shim-layer running in a virtual router . . . . . . . . . . . . . . . . . . . 1.8 Toy network example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Titre : Protocoles de couche r éseau pour l'extensibilit é des centres de donn ées Mots cl és : R éseaux de centres de donn ées, Mobilit é, Multicast, Équilibre de charge, Segment Routing R ésum é : Du fait de la croissance de la demande en ressources de calcul, les architectures de centres de donn ées gagnent en taille et complexit é. D ès lors, cette th èse prend du recul par rapport aux architectures r éseaux traditionnelles, et montre que fournir des primitives g én ériques directement à la couche r éseau permet d'am éliorer l'utilisation des ressources, et de diminuer le trafic r éseau et le surco ût administratif. Deux architectures r éseaux avanc ées, Segment Routing (SR) et Bit-Indexed Explicit Replication (BIER), sont utilis ées pour construire et analyser des protocoles de couche r éseau, afin de fournir trois primitives : (1) mobilit é des t âches, (2) distribution fiable de contenu, et (3) équilibre de charge.

  Example of shim-layer running in a virtual router. In that example, workloads are containers, and a shared memory (shm) is used so that the shim layer can transparently retrieve state from the application (as in part IV).

	1.3. THESIS STATEMENT: AUGMENTING DATA CENTERS THROUGH THE NETWORK
	LAYER							13
	physical machine
								management interface
			controller			NIC
	virtual router					kernel bypass
								NIC
								NIC
								NIC
								NIC
	shim		shim		shim		shim
	vNIC	shm	vNIC	shm	vNIC	shm	vNIC	shm
	container	container	container	container
	Figure 1.7 -

Table 1 .

 1

1 -Generic mathematical notation

Chapter 2

•

  Yoann Desmouceaux, Sonia Toubaline, Thomas Clausen, Flow-Aware Workload Migration in Data Centers, Springer Journal of Network and Systems Management, vol. 26, no. 4, pp. 1034-1057, October 2018 (chapter 4). • Yoann Desmouceaux, Thomas Clausen, Juan-Antonio Cordero-Fuertes, Mark Townsley, Reliable Multicast with B.I.E.R., IEEE/KICS Journal of Communications and Networks, vol. 20, no. 2, pp. 182-197, April 2018 (chapter 5).

	20	CHAPTER 2. THESIS CONTRIBUTIONS
	Journal Publications
	• Yoann Desmouceaux, Pierre Pfister, Jérôme Tollet, Mark Townsley, Thomas Clausen, 6LB: Scalable and Application-Aware Load Balancing with Segment Routing, IEEE/ACM Trans-
	actions on Networking, vol. 26, no. 2, pp. 819-834, April 2018 (chapter 7).
	• Yoann Desmouceaux, Thomas Clausen, Juan-Antonio Cordero-Fuertes, Mark Townsley, Re-liable BIER with Peer Caching, submitted to IEEE Transactions on Network and Service
	Management (chapter 6).

  then a SR header (m 1 ::f wp, m 2 ::bf w, v) is inserted.

	Experiment	Variant	Downtime Lost/buffered packets
	HTTP static	Non-SR migration	101 ms	1160
	HTTP static	SR migration (drop)	107 ms	197
	HTTP static	SR migration (buffer)	104 ms	216
	HTTP dynamic	Non-SR migration	100 ms	1100
	HTTP dynamic	SR migration (drop)	102 ms	190
	HTTP dynamic SR migration (buffer)	103 ms	199
	iperf	Non-SR migration	192 ms	1090
	iperf	SR migration (drop)	192 ms	859
	iperf	SR migration (buffer)	185 ms	789
	iperf sink	Non-SR migration	167 ms	3390
	iperf sink	SR migration (drop)	163 ms	2810
	iperf sink	SR migration (buffer)	163 ms	3050
	Table 3.1 -Evaluation of SR migration: average VM downtimes and number of lost packets (or buffered
	packets for the corresponding mechanism) for the different evaluated workloads and migration mechanisms.

Table 4 . 1 -

 41 Inputs and variables to the model

Table 4 . 2

 42

-Link capacities for the example topology

Table 4 . 3 -

 43 Link capacities for the k-rack topology

	Total throughput
							Δ=1
							Δ=3
							Δ=5
							Pareto front
	0	2	4	6	8	10	12	14
				Migration cost

  Algorithm 5 Reliable BIER Source Operation

	∆tagg ← NACK aggregation delay w ← packet cache window size F ← unique flow identifier B ← destination bitstring S ← 0 ⊲ sequence number C ← {} ⊲ packet cache R ← {} ⊲ retransmit bitstrings T ← {} ⊲ retransmit timers for each outgoing packet p do
	insert reliable BIER header with flow F , seq S
	insert BIER header with bitstring B
	transmit

  Link usage, between the source and the core router (averaged over all the duration of a simulation). Aggregate multicast traffic (over all links and for all the duration of a simulation).

	5.4. ISP TOPOLOGY SIMULATIONS		59
	0 2x10 8 4x10 8 6x10 8 8x10 8 1x10 9 9 1.2x10 (a) 0 0 Average link usage (b/s) 5x10 9 1x10 10 1.5x10 10 2x10 10 2.5x10 10 0 Aggregate multicast traffic (bytes) (b) 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1 Average loss probability (⍺) BIER retransmissions, source to core link Unicast retransmissions, source to core link Multicast retransmissions, source to core link 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1 Average loss probability (⍺) BIER retransmissions Unicast retransmissions Multicast retransmissions 0.2 0.4 0.6 0.8 1 1.2 1.4 Average delivery ratio 0.2 0.4 0.6 0.8 1 CDF 0	0	0.001 (c) Average delivery ratio for the multicast flow. 0.002 0.003 0.004 0.005 0.006 0.007 0.008 0.009 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 BIER retransmissions Multicast retransmissions Unicast retransmissions Average loss probability (⍺) 5 10 15 # of destinations ⍺=0.01 ⍺=0.02 ⍺=0.03 ⍺=0.04 ⍺=0.05 ⍺=0.06 ⍺=0.07 ⍺=0.08 ⍺=0.09 ⍺=0.10	0.09	0.10	20

(d) CDF of the number of clients in BIER retransmit bitstrings.

Table 5 .

 5 .3 until all destinations receive a copy. Solid lines represent exact values (obtained by simulation), dotted lines represent the low-loss approximation given in theorem 5.1.

	k 4	L 28	L × D 68	C 16	BIER unicast multicast 68α 104α 744α
	6	78	204	54	204α	360α	5958α
	8	168	456	128	456α	864α	27936α
	10	310	860	250	860α	1700α	95550α
	12	516	1452	432	1452α	2952α	265320α
	14	798	2268	686	2268α	4704α	635334α
	16 1168	3344	1024	3344α	7040α	1362048α

1 -Average number of retransmissions per multicast data packet for (k, k/2, k/2) tree topologies (approximation as per theorem 5.1)

Table 7 .

 7

	1 -Notation

Table 7 .

 7 .5. 3 -Handshake protocol state machine for a given flow, at a load-balancer ℓ1

	State	Incoming SR function SR functions added	Next state
		SYN from client
	LB LISTEN	m1.connectAvail(ℓ1)	HUNTING
		m2.connectForce(ℓ1)
		data from client
	LB LISTEN	m1.recoverStickiness(ℓ1)	HUNTING
		m2.recoverStickiness(ℓ1)
		SYN from client
	HUNTING	m1.connectAvail(ℓ1)	HUNTING
		m2.connectForce(ℓ1)
	HUNTING	createStickiness(m) remove SR header	STEER(m)
	STEER(m)	data from client s.ackStickiness(ℓ1)	STEER(m)
	STEER(m)	removeStickiness(m) remove SR header	LB LISTEN after 10 sec

Table 7 . 4 -

 74 Handshake protocol state machine for a given flow, at a machine m

	109

  6LB Wikipedia replay: decile 1, . . . , 9 of wikipage load time request rate of around 110 pages per second, SC and SR 4 yielded similar performance, and SR dyn exhibited even lower response times. As the request rate increases, using the application-unaware SC policy yielded notably increased page load times -whereas when using SR 4 or SR dyn , a comparably much smaller increase in page load times incurred.

									123
	Median response time (s)		0.1 0.15 0.2 0.25 0.3 0.35 0.4	00:00	04:00	08:00	12:00 SC wiki pages 16:00 SR 4 wiki pages SR dyn wiki pages	20:00
					Time of day (UTC)
	Queries per second		120 140 160 180 200 220 240					wiki pages
			100
				00:00	04:00	08:00	12:00	16:00	20:00
					Time of day (UTC)
	Figure 7.22 -6LB Wikipedia replay: query rate and median wikipage load time
	Decile 1-9	response time (s)	0 0.1 0.2 0.3 0.4 0.5 0.6	00:00	04:00	08:00	12:00 SC wiki pages 16:00	20:00
	Decile 1-9	response time (s)	0 0.1 0.2 0.3 0.4 0.5 0.6	00:00	04:00	Time of day (UTC) 08:00 12:00 SR 4 wiki pages 16:00	20:00
	Decile 1-9	response time (s)	0 0.1 0.2 0.3 0.4 0.5 0.6	00:00	04:00	Time of day (UTC) 08:00 12:00 SR dyn wiki pages 16:00	20:00
						Time of day (UTC)
	Figure 7.23 -

Table 8 . 2 -

 82 SHELL P4-NetFPGA dataplane performance

Table 8 . 3

 83

	134	CHAPTER 8. STATELESS LOAD-AWARE LOAD BALANCING IN P4
	Cumulative Distribution Function	0.2 0.4 0.6 0.8 1	doff_max = 8 doff_max = 11 doff_max = 13 doff_max = 15		buffer filling steady state
		0
		0		1	2	3	4	5	6	7	8	9
						Latency (μs)
	Figure 8.5 -SHELL P4 dataplane evaluation: distribution of per-packet latency for a burst of 4800
	packets (c = 2, h = 2, different d max off )
		Max TCP size		LUT	LUTRAM	FF		BRAM
		d max off d max off d max off d max off	= 8 = 11 = 13 = 15		36.9% 40.1% 43.8% 48.7%		19.4% 22.0% 24.9% 28.6%	33.3% 36.4% 40.2% 45.8%	59.3% 63.2% 67.7% 74.1%

-SHELL P4-NetFPGA dataplane resource usage on a NetFPGA-SUME. (LUT: Look-Up Tables; LUTRAM: Look-Up Tables used as RAM; FF: registers (Flip-Flops); BRAM: Block RAM.)

table 8 .

 8

	3, which depicts the resource usage

Table 9 .

 9 2 -SR Autoscaling Wikipedia replay: energy cost and response time

Although only wired data centers will be studied in this thesis, wireless data center architectures are an active research area[START_REF] Halperin | Augmenting data center networks with multi-gigabit wireless links[END_REF].

https://www.icann.org/en/system/files/press-materials/release-03feb11-en.pdf

For the purpose of this section, a tenant is defined as any entity using some of the resources of a data center for a single purpose, be it an individual, a company, or a logical entity such as a project to which resources are assigned, etc.

In this section, the word "router" is used to designate devices that can perform Layer-2 and/or Layer-3 forwarding, indifferently.

 6 This is assessed by computing the bissection bandwidth. The bissection width of a network is defined as the minimal width (number) of links that need to be cut so as to separate the network into two equal parts. The bissection bandwidth then corresponds to the total throughput that these cut links can sustain.

https://www.ietf.org

BIER-Traffic Engineering, encoding not destinations but link adjacencies to be traversed by a packet.

https://github.com/yoannd/phdthesis-data

https://gerrit.fd.io/r/vpp

https://github.com/qemu/qemu/blob/master/docs/interop/vhost-user.txt

In its standard implementation, the ping6 utility adapts its sending rate if it sees that probes are not replied to. For the purpose of these experiments, ping6 was recompiled so that packets are sent with the same rate, no matter how many of them are not answered.

For instance, the size of RAM plus storage for a virtual machine.

z t-1 is obtained by starting from a random state and solving(4.15) where only the throughput objective is considered, with the additional constraint that all tasks must be allocated.

It is to be noted, however, that if intermediate routers provides caching capabilities (as for instance in Information Centric Networks[START_REF] Zhang | Named data networking[END_REF]), they could be extended to intercept NACKs and perform retransmissions in place of the source. This would not change the end-to-end behavior as specified in this chapter.

The successor of the last data packet is a special end-of-connection packet. To ease readability, Algorithms 5 and 6 assume an infinite stream.

One reason for including a bitstring is, that this allows the originator to create the and BIER retransmit bitstring by a simple OR operation of received NACKs.

Colloquially speaking, to count the blue arrows in figure

5.1.

For lossless links, operating below capacity and with finite buffers, packet losses are due to buffer overflowthus while a link may be lossless, an interface may still experience packet losses

.6 The root is labelled[1]; the first child of the root is labeled [[1], 1], its second child [[1], 2]; the first child of [[1], 1] is [[[1], 1], 1]; etc.

Results have been smoothed with an Exponential Window Moving Average filter with parameter α = 0.01.

Whereas traffic from the source to the destination is bursty, and therefore likely to cause interface buffers to run full and drop packets, traffic from destinations to the source is expected to be sparse (essentially, SR-NACKs).

A peer is considered "local" if it has the same parent -in which case, it is indicated in the received P S1 in .

A peer is considered "second-most local" if it has the same grandparent, but not the same parent -in which case, it is indicated in the received P S in 2 .

The comparison is made to the random peer selection policy, rather than to any of the clustered peer selection policies. Indeed, selecting the destination with the non-lossy link for any of the clustering policies would amount to biasing in favour of that policy -selecting any other would amount to biasing in its disadvantage.

Only those times corresponding to the non-exploratory mode (where the destination actually selects its best peer) are retained. This allows subtracting the uniform noise due to the ε fraction of NACKs whose destinations are selected randomly.

For example, if a load-balancer is removed, and another load-balancer takes over the active flows it was serving.

The ultimate segment is the VIP.

A simple way to improve the reliability of the system during this short amount of time would be to monitor, in-band, the responsiveness of the servers (e.g., by gathering information about packet retransmissions or return traffic), and to rotate the order of SR lists for packets whose first instance is deemed unresponsive. This would increase reliability while ensuring that the browsed set of instances remains the one returned by consistent hashing.

Defined as the (i + 1)-st integer in {1, . . . , M -1} which is coprime to M .

This is not the case only when there is a simultaneous change in the pool of load-balancer instances and in the pool of application instances, and then only concerns those flows which correspond to consistent hashing failures and which, according to figure

7.5, amount to a few percents

This shared memory, internal by default, can be exposed as a named file by specifying the ScoreBoardFile directive in the server configuration.

A fit is performed on SC to rescale the units. The obtained scaling coefficients are then used for all policies.

These values have been smoothed through an Exponential Window Moving Average filter, of parameter α = 1 -exp(-δt) where δt is the interval of time in seconds between two successive data points.

https://www.mediawiki.org/wiki/Download

Those pages were identifiable by the string /wiki/index.php/ in their URL.

The rationale for embedding a label C i (rather than the full application identifier) in the covert channel, is that it is a more compact representation, requiring only ⌈log

c⌉ bits. In practice, if c = 2 (as is usual from the "power of two choices"[START_REF] Mitzenmacher | The power of two choices in randomized load balancing[END_REF]), only 1 bit is required. If one were to encode the full instance identifier in the covert channel, this would require ⌈log 2 |M|⌉, where M is the set of machines capable of hosting those instances.

To ensure compatibility with clients not using TCP timestamps, if a SYN packet is missing the TCP timestamp option (detected by d off ≤ 8), an SR header with only one candidate will be inserted. Lack of a TCP timestamp in non-SYN packets is then interpreted as a covert channel value of C 1 .

A connection assigned to a bucket/choice pair is deemed reset after n reconfigurations if the application instance to which it was initially assigned no longer appears in the corresponding history table.

Unfortunately, computing a closed-form as a function of ρ would involve solving polynomials of degree c + 1 in ρ.

https://www.mediawiki.org/wiki/Download

Dynamic pages are found by matching http://en.wikipedia.org/wiki/index.php/ in the URLs. Due to the capacity of the testbed (which could not sustain replaying every query), every second query was replayed.

Remerciements

La partie II étudie la mobilité des tâches dans les centres de données. Le chapitre 3 (publié dans [79]) introduit l'utilisation de SRv6 afin de permettre la migration de machines virtuelles sans perte de paquets. Les protocoles réseaux traditionnellement utilisés pour la migration de machines virtuelles (virtual machines, VMs) reposent sur la signalisation, par une VM venant de terminer une migration, de sa nouvelle localisation à un répertoire centralisé. Cela crée une période de temps transitoire durant laquelle les paquets destinés à de telles VMs sont perdus. Pour résoudre ce problème, le chapitre 3 propose de pré-allouer de manière conservative, grâce à SRv6, un chemin de migration comprenant les machines hôtes de laquelle et vers laquelle la migration est initiée. Ainsi, les paquets atteignent toujours la machine correcte, quelle que soit l'étape au sein du processus de migration. Ce chemin est alloué par le plan de contrôle avant que la migration en elle-même ne soit lancée, et jusqu'après son accomplissement. Ceci est accompli par l'introduction de deux nouvelles fonctions SRv6 : la première vérifie (au sein de la machine hôte source) si le lien vers la VM est toujours établi, et selon le cas, transfère les paquets ou bien vers la VM, ou bien vers le second segment ; la seconde vérifie (au sein de la machine hôte destination) si la VM a repris complètement son exécution, et en fonction, met en tampon localement les paquets, ou bien les transfère vers la VM. Ce mécanisme a été implémenté au sein d'un routeur virtuel (VPP [31]), et l'évaluation montre qu'il est en effet possible de migrer des VMs sans perte de paquets, ce qui permet de réduire la latence et la durée de service des flux servis par l'application hébergée sur la VM.

Une fois la possibilité de migrer des tâches sans perte introduite, le chapitre 4 (publié dans [80]) étudie comment il est possible de fournir de la migration de tâches utilisant les caractéristiques des flux, c'est-à-dire de migrer des tâches communicant les unes avec les autres de façon à les rapprocher dans la topologie, et ce pour optimiser le trafic réseau. Les architectures de gestion des centres de données introduites traditionnellement dans la littérature scientifique considèrent ou bien les demandes réseau tâche-à-tâche, ou bien le coût des migrations de tâches, mais pas les deux. Le chapitre 4 introduit un programme d'optimisation multi-objectif visant à maximiser le débit inter-tâche total, tout en minimisation le coût induit par la migration des tâches, et en maximisant le nombre de tâches nouvellement allouées. Un programme non-linéaire en variables mixtes entières (Mixed Integer Non-Linear Programming, MILNP) est introduit afin de capturer les contraintes et objectives permettant de modéliser ce problème, et ce programme est ensuite linéarisé en une formulation comme programme linéaire en variables mixtes entières (Mixed Integer Linear Programming, MILP). La méthode de l'ε-contrainte est utilisée pour calculer l'ensemble de solutions Pareto-optimales, et sert de référence. À l'aide de simulations, il est démontré que

//www.gartner.com/newsroom/id/3871416), the public cloud market revenue grew from $154

Data Availability

All the data and scripts necessary to reproduce the graphs included in this thesis can be publicly accessed, and can be reused by anyone interested in doing so 1 .

recorded over a fixed window of queries. When the end of the window is reached, if the number of accepted queries is significantly below (or above) 1 2 , the value of c is incremented (or decremented).

Algorithm 10 Consistent hashing history table construction

⊲ update version numbers for b ∈ {0, . . . , B -1}, i ∈ {h-1, h-2, . . . , 1}, j ∈ {0, . . . , c-1} do t

2.

A connection request (TCP SYN) received at an application instance is processed by the corresponding server agent. The agent examines the local state of the application (e.g., its CPU load), and determines to either accept the request, or to forward the packet to the next segment in the SR header, using e.g., algorithm 7 or algorithm 8 from chapter 7. Note that the last candidate in the list must always accept the connection.

A server agent having accepted a connection will record, for the connection lifetime, its own index C i ∈ {C 1 , . . . , C c } in the segment list received with the connection request, so as to be able to signal this to the client, through the covert channel. In the example from table 8.1 where the inserted SR header is (m 3 , . . . , m 4 , v), if the first candidate (m 3 ) accepts the connection, the corresponding server agent will record C 1 as a covert channel label.

3. When an application instance emits a packet, the corresponding server agent will embed the recorded label C i for that connection in the TCP SYN-ACK (and in all future outgoing packets for that connection), using a covert channel -how this is accomplished is discussed in section 8.3.2.

4.

All subsequent (i.e., TCP non-SYN) packets emitted by the client will automatically encode C i , since the covert channel is automatically reflected by the network stack of the client.

5.

All subsequent (i.e., TCP non-SYN) packets received at the LB are hashed, on their 5-tuple, identifying a bucket b. The LB also extracts the label C i inserted in the covert channel by the client. The i-th colum of t[b] is then used to generate an SRv6 header with segment list (t

In the example from to meet the SLA (this will be called first-available-server load-balancing). This allows for local, application-load-aware load-balancing decisions, without incurring any monitoring. A difference with respect to the mechanism introduced in chapter 7 is that the chain of application instances is fixed -whereas in chapter 7 lists of two pseudo-random candidates are chosen. Since queries are served by the first available candidate, last instances tend to be less loaded. If too few queries reach the last instance, this indicates that the chain is over-dimensioned; conversely, too many queries reaching the last instance indicates under-dimensioning. Therefore, autoscaling is achieved by the last instance in the chain, rather than by a central entity. By inspecting the quantity of received queries, that last instance will indicate to the load-balancer that another instance must be added to the chain, or that the last one can be removed. This allows for distributed, locally-driven autoscaling decisions, without centralized monitoring.

First-available-instance Load-Balancing

This chapter introduces a load-balancing policy using the framework described in chapter 7, that allows for monitorless up-or down-scaling of a set of n replicated application instances, while ensuring that an SLA is met. A replicated application whose mean service time is 1/µ is considered, for which the expected response time (as seen by clients) T must be lower than an agreed-upon SLA of S (in units of mean service time), i.e., E[T ] ≤ S × 1 µ . Then, a simple load-balancing policy consists of ordering the n instances into a chain (s 1 , . . . , s n ), setting a threshold c = ⌊S⌋, and making sure that each never handles more than c clients at a given time (see figure 9.1). Formally, each query is assigned (s 1 , . . . , s n ) as a candidate SR list, and each application instance in the list either accepts the query if it currently serves < c clients -otherwise, it forwards the query to the next one in the list (see algorithm 7). To ensure that all queries are served, the last instance s n must always accept connections. Thus, each of the first (n -1) instances never serve more than c clients, ensuring an expected response time lower than c/µ for queries served by those (assuming either a FIFO or Processor Sharing policy). The last instance accepts the other queries, and, provided that the chain is well-dimensioned, will also serve them within a sufficiently small amount of time -how to dimension the chain is discussed in section 9.2.2.

Autoscaling

A key goal of this load-balancing policy is to make it simple to automatically dimension the size of the chain, while maintaining the SLA: except when the last instance serves too many queries, the SLA will be satisfied by the chain. This can be exploited to perform autoscaling: when the last instance serves too many (or too few) queries, which can be detected locally, this is a signal that the chain must be scaled up (or down) -allowing for monitorless autoscaling, as illustrated in figure 9.2.

As formalized in algorithm 11, the virtual router hosting the last instance in the chain keeps statistics about the size of its queue upon receipt of queries. The fraction of SLA violations (more than c clients in the last instance) is maintained over a fixed time window, and when it goes above a preconfigured threshold ε up , the virtual router hosting the instance requests upscaling of the chain. The virtual router hosting the last instance also maintains the fraction of times it was empty upon query reception, and periodically checks whether this fraction is below a predetermined threshold ε down , in which case downscaling is requested.

First-Available-Instance LB with 2 Instances

The first-available-instance load-balancing scheme introduced in section 9.2.1 is analytically studied in this chapter. For ease of presentation and modeling, the case of n = 2 instances (for which the RRR technique applies) is analyzed first. The general case of n ≥ 3 instances (for which the RRR technique needs to be generalized) will be studied in section 9.4.

Markov Model

Arrivals are assumed to follow to a Poisson process of intensity λ > 0. Each server has a processing capacity of µ > 0, with exponentially-distributed service times (i.e., the probability of a service lasting less than t is 1e -µt ). To ease notation, let ρ = λ/µ be the normalized request rate. As described in section 9.2.1, queries are directed to the first instance, which will serve them if the number of pending connections is < c, or otherwise pass them to the second instance, which will serve them in any case. Probability that first state visited on row j -1 when starting from (i, j) is (k, j -1)

Probability that first state visited on row 0 when starting from (i,

Mean reward of z N 2 (t) when going from (i, j) down to row j -1

Mean reward of z N 2 (t) when going from (i, 1) down to row 0

Mean reward of z N 2 (t) when going from (i, 0) to (0, 0)

Mean reward of z N 2 (t) over a cycle from (0, 0) to (0, 0) Table 9.1 -Notation used when applying the RRR technique

Therefore, the whole system can be modeled as a Markov chain, with state space S = {0, . . . , c}× N. In state (i, j), i clients are in the first server's queue and j in the second server's queue. Transitions that increase j can only happen when i = c, whereas transitions that increase i can happen from any state with i < c. Figure 9.3 shows a graphical representation of this Markov chain.

Applying RRR to the Markov Model

To obtain the client response time, the expected number of clients will be derived [START_REF] Little | A proof for the queuing formula: L = λW[END_REF]. As the behavior of the first server is described by an M/M/1/c law, the probability distribution of N 1 , the number of clients served by the first instance, can be derived as:

and the expected number of clients in the first instance is:

.

To obtain the probability distribution of N 2 , the number of clients served by the second instance, the RRR technique is applied with metric of interest M (t) = z N2(t) (where z ∈ C), as in equation (9.1). This allows deriving the probability generating function of The values of R D i and p D i→k can then be inserted into equations (9.9),(9.10) to compute the R L i , yielding: Lemma 9.4. For all i ∈ {1, . . . , c}, R L i = Ei+Fiz C-Dz , where E i , F i are real constants, and C, D the same as in lemma 9.3.

Proof. By induction, we can express the R L i as a function of R L 1 , using equation (9.9): R L i = γiR L 1 +εi, where γi, εi are real constants. On the one hand, this gives R L c = γcR L 1 + εc. On the other hand, (9.10) gives R

This finally allows computing the probability generating function of N 2 , as stated in theorem 9.1:

Proof. This follows from applying lemma 9.4 to i = 1, and using equations (9.11) and (9.12).

This means that N 2 follows a geometric distribution when conditioned to N 2 ≥ 1. Combining the expressions for E[N 1 ] and E[N 2 ] finally yields the expected response time, as per Little's law [START_REF] Little | A proof for the queuing formula: L = λW[END_REF]:

First-Available-Instance LB with n ≥ 3 Instances

In this section, the model presented in section 9.3 is extended to the case of chains of length n ≥ 3. With first-available-instance load-balancing, a query is accepted by the first instance of the Appendix A

Résumé en français

Cette thèse étudie l'utilisation de protocoles de couche réseau pour fournir, au sein de réseaux de centres de données, des primitives de mobilité des tâches, de distribution fiable de contenu, et d'équilibrage de charge. Elle comprend 5 parties et 10 chapitres, structurés comme suit.

La partie I est une mise en contexte introductive. Le chapitre 1 parcourt l'état de l'art sur les architectures de centres de données (data centers) et sur les protocoles réseaux associés. Il introduit ensuite deux architectures réseau intéressantes, IPv6 Segment Routing (SRv6 [7]) et Bit-Indexed Explicit Replication (BIER [8]), qui utilisent le paradigme de routage à la source afin d'ajouter des fonctions à la couche réseau. Le chapitre 2 résume ensuite comment ce concept a été appliqué le long de cette thèse, afin de fournir des primitives de mobilité des tâches, de distribution fiable de contenu, et d'équilibrage de charge.

List of Tables

This thesis comprises 15 tables, listed below.

List of Algorithms