Mohammad Noormohammadpour

Cauligi S Raghavendra

Ming Hsieh

DATACENTER TRAFFIC CONTROL TECHNIQUES AND TRADE-OFFS 1 Datacenter Traffic Control: Understanding Techniques and Trade-offs

Keywords: Datacenters, traffic control, trade-offs, management schemes, transmission control, traffic shaping, load balancing, flow prioritization, multipath routing, traffic scheduling

Datacenters provide cost-effective and flexible access to scalable compute and storage resources necessary for today's cloud computing needs. A typical datacenter is made up of thousands of servers connected with a large network and usually managed by one operator. To provide quality access to the variety of applications and services hosted on datacenters and maximize performance, it deems necessary to use datacenter networks effectively and efficiently. Datacenter traffic is often a mix of several classes with different priorities and requirements. This includes user-generated interactive traffic, traffic with deadlines, and long-running traffic. To this end, custom transport protocols and traffic management techniques have been developed to improve datacenter network performance. In this tutorial paper, we review the general architecture of datacenter networks, various topologies proposed for them, their traffic properties, general traffic control challenges in datacenters and general traffic control objectives. The purpose of this paper is to bring out the important characteristics of traffic control in datacenters and not to survey all existing solutions (as it is virtually impossible due to massive body of existing research). We hope to provide readers with a wide range of options and factors while considering a variety of traffic control mechanisms. We discuss various characteristics of datacenter traffic control including management schemes, transmission control, traffic shaping, prioritization, load balancing, multipathing, and traffic scheduling. Next, we point to several open challenges as well as new and interesting networking paradigms. At the end of this paper, we briefly review inter-datacenter networks that connect geographically dispersed datacenters which have been receiving increasing attention recently and pose interesting and novel research problems. To measure the performance of datacenter networks, different performance metrics have been used such as flow completion times, deadline miss rate, throughput and fairness. Depending on the application and user requirements, some metrics may need more attention. While investigating different traffic control techniques, we point out the trade-offs involved in terms of costs, complexity and performance. We find that a combination of different traffic control techniques may be necessary at particular entities and layers in the network to improve the variety of performance metrics. We also find that despite significant research efforts, there are still open problems that demand further attention from the research community.

I. INTRODUCTION

Datacenters provide an infrastructure for many online services such as on-demand video delivery, storage and file

CLUSTER INTERCONNECTION ToR Switch ToR Switch Rack 1

Rack N Host/Server Fig. 1. A typical datacenter cluster sharing, web search, social networks, cloud computing, financial services, recommendation systems, and interactive online tools. Such services may dynamically scale across a datacenter according to demands enabling cost-savings for service providers. Moreover, considering some degree of statistical multiplexing, better resource utilization can be achieved by allowing many services and applications to share datacenter infrastructure. To reduce costs of building and maintaining datacenters, numerous customers rely on infrastructure provided by large cloud companies [1]- [3] with datacenters consisting of hundreds of thousands of servers.

Figure 1 shows the structure of a typical datacenter cluster network with many racks. A datacenter often hosts multiple such clusters with thousands of machines per cluster. A cluster is usually made up of up to hundreds of racks [START_REF] Singh | Jupiter Rising: A decade of Clos topologies and centralized control in Google's datacenter network[END_REF]- [START_REF] Subramanian | Network in Hyper-scale data centers -Facebook[END_REF]. A rack is essentially a group of machines which can communicate at line rate with minimum number of hops. All the machines in a rack are connected to a Top of Rack (ToR) switch which provides non-blocking connectivity among them. Rack size is typically limited by maximum number of ports that ToR switches provide and the ratio of downlink to uplink bandwidth. There is usually about tens of machines per rack [START_REF] Singh | Jupiter Rising: A decade of Clos topologies and centralized control in Google's datacenter network[END_REF]- [START_REF] Subramanian | Network in Hyper-scale data centers -Facebook[END_REF]. ToR switches are then connected via a large interconnection allowing machines to communicate across racks. An ideal network should act as a huge non-blocking switch to which all servers are directly connected allowing them to simultaneously communicate with maximum rate.

Datacenter network topology plays a significant role in determining the level of failure resiliency, ease of incremental expansion, communication bandwidth and latency. The aim is to build a robust network that provides low latency, typically up to hundreds of microseconds [START_REF] Judd | Attaining the Promise and Avoiding the Pitfalls of TCP in the Datacenter[END_REF]- [START_REF] Alizadeh | Less is more: trading a little bandwidth for ultra-low latency in the data center[END_REF], and high bandwidth across servers. Many network designs have been proposed for datacenters [START_REF] Greenberg | VL2: A Scalable and Flexible Data Center Network[END_REF], [START_REF] Al-Fares | A Scalable, Commodity Data Center Network Architecture[END_REF]- [START_REF] Singla | Jellyfish: Networking data centers randomly[END_REF]. These networks often come with a large degree of path redundancy which allows for increased fault tolerance. Also, to reduce deployment costs, some topologies scale into large networks by connecting many inexpensive switches to achieve the desired aggregate capacity and number of machines [START_REF] Singh | Jupiter Rising: A decade of Clos topologies and centralized control in Google's datacenter network[END_REF], [START_REF] Clos | A Study of Non-Blocking Switching Networks[END_REF]. Although the majority of these topologies are symmetrical, in practice, datacenter networks turn out to be often asymmetrical due to frequent failures of network elements (switches, links, ports, etc.) [START_REF] Zhou | WCMP: Weighted cost multipathing for improved fairness in data centers[END_REF]- [START_REF] Govindan | Evolve or Die: High-Availability Design Principles Drawn from Google's Network Infrastructure[END_REF]. In contrast to fixed networks, reconfigurable topologies involve optical circuit switching, wireless or a combination of both to adapt to traffic demands [START_REF] Porter | Integrating Microsecond Circuit Switching into the Data Center[END_REF]- [START_REF] Ghobadi | Projector: Agile reconfigurable data center interconnect[END_REF]. These topologies rely on fast algorithms that take into account the reconfiguration latency.

Many applications may need to span over multiple racks to access required volume of resources (storage, compute, etc.). This increases the overall volume of traffic across racks. A datacenter network with full bisection bandwidth allows for flexible operation and placement of applications across clusters and improves overall resource utilization and on-demand scale out for applications [START_REF] Singh | Jupiter Rising: A decade of Clos topologies and centralized control in Google's datacenter network[END_REF], [START_REF] Greenberg | VL2: A Scalable and Flexible Data Center Network[END_REF], [START_REF] Al-Fares | A Scalable, Commodity Data Center Network Architecture[END_REF], [START_REF]Introducing data center fabric, the next-generation Facebook data center network[END_REF]. This allows resources of any machine to be used by any application which is essential for hyper-scale cloud providers [1]- [3].

Designing networks for full bisection bandwidth is costly and unnecessary for smaller companies or enterprises. As a result, many datacenters may be over-subscribed, meaning the total inter-rack network capacity may be less than sum of intrarack capacities across all racks. The underlying assumption is that applications are mostly rack local. Increasing the oversubscription ratio affects performance of different topologies differently. For instance, over-subscribed Fat-Trees provide less flexible communication across machines compared to over-subscribed Jellyfish networks [START_REF] Singla | Fat-FREE Topologies[END_REF].

There is growing demand for datacenter network bandwidth. This is driven by faster storage devices, rising volume of user and application data, reduced cost of cloud services and ease of access to cloud services. Google reports 100% increase in their datacenter networking demands every 12 to 15 months [START_REF] Singh | Jupiter Rising: A decade of Clos topologies and centralized control in Google's datacenter network[END_REF]. Cisco forecasts a 400% increase in global datacenter IP traffic and 2.6 times growth in global datacenter workloads from 2015 to 2020 [START_REF]Cisco Global Cloud Index: Forecast and Methodology[END_REF].

Applications running on datacenters determine their traffic characteristics and the communication patterns across machines. Some popular applications include web services, cache followers, file stores, key-value stores, data mining, search indexing and web search. Many datacenters, especially cloud providers, run a variety of applications that results in a spectrum of workloads. Some applications generate lots of internal datacenter traffic, such as scatter-gather (also known as partition-aggregate) [START_REF] Kandula | The nature of data center traffic: measurements & analysis[END_REF]- [START_REF] Zats | DeTail: reducing the flow completion time tail in datacenter networks[END_REF] and batch computing tasks [START_REF] Dean | MapReduce: simplified data processing on large clusters[END_REF], [START_REF] Isard | Dryad: Distributed Data-parallel Programs from Sequential Building Blocks[END_REF]. As a result, the total traffic volume within a datacenter is often much more than that of entering or leaving it. Cisco reports this ratio to be greater than 3 which is expected to increase further by 2020 [START_REF]Cisco Global Cloud Index: Forecast and Methodology[END_REF]. Traffic control is necessary to highly utilize network bandwidth, keep latency low, offer quality of service, and fairly share resources among many users and applications by managing flow of traffic across the datacenter. There is a significant body of work on traffic control for datacenters. In this tutorial paper, we aim to review concepts in design and operation of traffic control schemes.

The rest of this paper is organized as follows. In §II, we present some related work. In §III, we review a variety of datacenter topologies, provide an overview of datacenter traffic patterns, set forth the challenges of traffic control for datacenter networks, and the objectives of datacenter traffic control. In §IV, we review management schemes that can be applied to datacenters and point to some research work that use different management schemes. Next, in §V, we present a variety of traffic control techniques, discuss them in detail and explore the benefits and drawbacks associated with them. In §VI, we discuss some general open traffic control problems. In §VII, we point to rising paradigms related to datacenters. In §VIII, we introduce a new research area that is a result of global distribution of datacenters. Finally, in §IX, we conclude the paper.

II. RELATED WORK

In this section, we briefly present some survey articles related to datacenter traffic control. In [START_REF] Liu | Low latency datacenter networking: A short survey[END_REF], authors provide a short survey of low latency datacenter networking by reviewing approaches taken to achieve low latency, namely reducing queuing delay, accelerating retransmissions, prioritizing mice flows and utilizing multi-path. In [START_REF] Ren | A survey on TCP Incast in data center networks[END_REF], authors survey the methods used to address the transport control protocol (TCP) incast problem (please see §III-C6). In [START_REF] Chen | Allocating bandwidth in datacenter networks: A survey[END_REF], authors survey bandwidth sharing in multi-tenant datacenters using techniques of static reservation, minimum guarantees and no guarantees (resources obtained in a best effort fashion). In [START_REF] Zhang | Survey on transport control in data center networks[END_REF], authors point out datacenter transport problems namely TCP incast, latency, challenges in virtualized environments, bandwidth sharing in multi-tenant environments, under-utilization of bandwidth, and TCP in lossless Ethernet environments also known as Converged Enhanced Ethernet (CEE). In [START_REF] Yadav | TCP ISSUES IN DATA CENTER SYSTEM-SURVEY[END_REF], authors discuss TCP issues in datacenters pointing to TCP incast, queue buildup and buffer pressure. In [START_REF] Wang | A survey on data center networking for cloud computing[END_REF], authors provide a comprehensive overview of datacenter networking for cloud computing discussing cloud computing network architectures, communication technologies used, topologies, routing and virtualization approaches. In [START_REF] Joglekar | Managing Congestion in Data Center Network using Congestion Notification Algorithms[END_REF], authors discuss various congestion notifications for datacenter networks. Finally, in [START_REF] Sreekumari | Transport protocols for data center networks: a survey of issues, solutions and challenges[END_REF], authors survey transport protocols proposed for datacenter networks and briefly explain each one of the variety of research efforts made in addressing the incast problem, outcast problem and in reducing latency for datacenter networks.

In this tutorial paper, we merely focus on traffic control concepts that can be applied to a variety of transport protocols including TCP. We also point to research efforts that use different techniques as examples so that readers can elevate their knowledge in case they are interested in further details.

We try to uncover the (not obvious) trade-offs in terms of complexity, performance and costs. This paper is different from prior work in that it covers a variety of aspects in traffic control in a conceptual way while not focusing on any specific transport, network or data link layer protocol. In addition, we provide an overview of datacenter traffic properties, topologies as well as traffic control challenges, objectives and techniques and their relationships which has not been done in prior work to our knowledge. Finally, at the end of this paper, we point to a recent research direction that involves inter-datacenter networks and offer three areas that demand further attention from the research community.

III. DATACENTER NETWORKS

In this section, we dive deeper into specifics of datacenter networks. We review a variety of topologies proposed and general traffic properties of datacenter networks, point to traffic control challenges in datacenters and explain several traffic control objectives sought by various parties (i.e., operators, tenants, etc.) involved in using datacenter networks.

A. Topologies

We shortly review popular physical datacenter topologies proposed and used in the literature either in testbed or simulation. Figure 2 shows examples of datacenter network topologies reviewed in the following (notice the network connecting ToR switches).

1) Fat-Tree: Fat-Tree [START_REF] Al-Fares | A Scalable, Commodity Data Center Network Architecture[END_REF], shown in Figure 2(a), is a multirooted tree topology where every root is called a core switch. It can be considered as a folded Clos [START_REF] Clos | A Study of Non-Blocking Switching Networks[END_REF] network [START_REF] Solnushkin | Fat-Tree Design[END_REF]. By increasing the number of roots, one can reduce the over subscription ratio (considering fixed capacity per link). This topology allows for high bisection bandwidth using a large number of less expensive switches allowing support for a large number of hosts at much less cost. There is an aggregate layer between the core and edge (ToR switches). The number of hops between any two servers attached to the same ToR switch is 2, to the same aggregate switch is 4 and otherwise is 6. A Fat-Tree topology built with k-port switches supports up to k 3 4 physical servers (assuming one physical port per server) and k 2 4 paths between any source and destination pair. As a result, it is possible to scale to huge clusters by interconnecting many inexpensive switches.

To effectively use a Fat-Tree, complex routing configurations may need to be done on switches to avoid creation of loops while using available paths for load balancing. For example, Portland [START_REF] Niranjan Mysore | PortLand: A Scalable Fault-tolerant Layer 2 Data Center Network Fabric[END_REF] is a custom routing and forwarding protocol which works out of Layer 2 and improves on fault tolerance (link failures), scalability, and ease of management (e.g. moving VMs across physical servers). Portland uses a Fabric Manager that holds information on address mappings and a fault matrix that maintains per link health status.

2) Leaf-Spine: Leaf-Spine (or Spine-and-Leaf) [START_REF] Alizadeh | On the data path performance of leafspine datacenter fabrics[END_REF], [START_REF]Cisco Data Center Spine-and-Leaf Architecture: Design Overview White Paper[END_REF], shown in Figure 2(g), is a two tier network topology where leaf (i.e., ToR) switches are attached to servers and every spine switch is directly connected to all leaf switches similar to a bipartite graph. The links connected between the servers and leaf switches may have a different capacity from the ones connecting leaf switches to the spine. Leaf-Spine makes it easier to expand on capacity and ports (by adding more spine or leaf switches) and also allows straight-forward usage of Layer 3 routing with load balancing support without creation of loops. As a downside, in case high bisection bandwidth is intended, scaling to more than hundreds of servers in a cluster can lead to increased costs due to need for spine switches with many high capacity ports.

3) VL2: VL2 [START_REF] Greenberg | VL2: A Scalable and Flexible Data Center Network[END_REF], shown in Figure 2(c), implements a complete routing and forwarding suite on top of 3-tier folded Clos networks (a multi-rooted tree) but differs from Fat-Tree in that switch-to-switch links are assigned much higher capacity than server-to-switch links. This requires less number of cables for connecting the aggregation and core layer. This topology has an intermediate tier that is connected to the aggregation trier in a bipartite graph topology. Each edge (ToR) switch is connected to two aggregation switches in a way that each aggregation switch gets connected to equal number of edge switches.

4) JellyFish: JellyFish [START_REF] Singla | Jellyfish: Networking data centers randomly[END_REF], shown in Figure 2(e), is a topology where ToR switches are connected in a random setting and according to some rules: first, ports are randomly connected between switches until no more links can be added; next, for any switch S with two or more free ports, an existing link A-B is removed and two links are added between two of S's free ports and two ends of the removed network link (i.e., A -S and B -S), until no switch has more than one free port. Since ToR switches are connected directly, the average path length (number of hops) is considerably smaller compared to 3-tier folded Clos. In addition, this topology is much easier to expand gradually. Authors show that with full bisection bandwidth, JellyFish supports more servers at the same cost compared to Fat-Tree. Also, with the same number of failed links, JellyFish offers a higher average throughput per server than Fat-Tree. One major problem with this topology is that the existing routing schemes cannot effectively use all paths since the number of hops across parallel paths changes by a large degree. Authors propose usage of k-shortest path routing for flows along with multipath TCP (MPTCP) [START_REF] Ford | TCP extensions for multipath operation with multiple addresses[END_REF].

5) DCell: DCell [START_REF] Guo | Dcell: A Scalable and Fault-tolerant Network Structure for Data Centers[END_REF], shown in Figure 2(d), is a hierarchical datacenter topology where a higher level DCell is built by putting together multiple lower level DCell structures. It can be incrementally expanded and does not have a single point of failure. DCell uses a custom routing algorithm that takes into account failures (DCell Fault-tolerant Routing) while aiming for near shortest path routing. Authors show that DCell offers higher network capacity compared to conventional tree topologies; however, it can perform much worse than multi-rooted trees such as Fat-Tree [START_REF] Guo | BCube: a high performance, server-centric network architecture for modular data centers[END_REF]. Implementing DCell requires changes to the server networking protocol stack.

6) BCube: BCube [START_REF] Guo | BCube: a high performance, server-centric network architecture for modular data centers[END_REF], shown in Figure 2(b), is a leveled structure where a higher level is built by recursively attaching lower levels. Servers require multiple network ports to connect to switches and they also act as forwarding elements for other servers. BCube uses source routing and requires changes to the server networking protocol stack which can be done either in hardware (network card) or software. If forwarding is : Host/Machine : ToR switch :Aggregation switch : Intermediate switch (c) VL2 [START_REF] Greenberg | VL2: A Scalable and Flexible Data Center Network[END_REF] : Host/Machine : ToR switch (d) DCell [START_REF] Guo | Dcell: A Scalable and Fault-tolerant Network Structure for Data Centers[END_REF] : Host/Machine : ToR switch (e) JellyFish [START_REF] Singla | Jellyfish: Networking data centers randomly[END_REF] : Host/Machine : ToR switch (f) Xpander [START_REF] Valadarsky | Xpander: Towards Optimal-Performance Datacenters[END_REF] : Host/Machine : ToR/Leaf switch : Spine Switch (g) Leaf-Spine [START_REF] Alizadeh | On the data path performance of leafspine datacenter fabrics[END_REF] Fig. 2. Several examples of datacenter topologies with two machines per rack implemented in software, it can add CPU overhead especially at high rates. Authors show that BCube is more resilient to switch failures compared to Fat-Tree and almost as resilient to server failures. 7) Xpander: Xpander [START_REF] Valadarsky | Xpander: Towards Optimal-Performance Datacenters[END_REF], shown in Figure 2(f), is a datacenter topology based on expander graphs [START_REF] Hoory | Expander graphs and their applications[END_REF] which offers all the performance improvements of JellyFish over Fat-Tree topology with higher throughput as network is incrementally expanded. This topology is made by connecting multiple meta nodes based on the following rules: first, each meta node is made up of equal number of ToR switches; second, no two ToRs are connected within the same meta node; third, same number of links is used to connect every pair of meta nodes. Compared to JellyFish, Xpander offers the benefit of being structured rather than random which improves implementation predictability. Authors test Xpander topology with similar routing and forwarding techniques used in JellyFish: k-shortest paths routing and MPTCP for multi-path load distribution.

B. Traffic Properties

Traffic characteristics of datacenters is highly dependant on applications and determines distribution of flow arrivals, flow sizes, and flow durations. For example, flows generated by web search queries are usually much smaller and shorter than that of batch computing jobs. This variety of applications could cause creation of long lived connections as well as short microbursts on the same network [START_REF] Hashemi | The Infrastructure Behind Twitter: Scale[END_REF]. There are limited publications on datacenter traffic characteristics. We review these works briefly focusing on applications and flow properties.

In [START_REF] Kandula | The nature of data center traffic: measurements & analysis[END_REF], authors collect a traffic dataset from a cluster running query processing applications on large volumes of data that run MapReduce like workloads under the hood to respond to queries. Authors find that more than 50% of flows last less than 100 ms, 80% of flows last less than 10 seconds, and almost all flows last less than 100 seconds. They also find that more than 50% of traffic is carried by flows that last less than 25 seconds, less than 10% by flows that last less than 10 seconds, and almost 1% of traffic by flows that last less than 1 second. In terms of flow arrivals, authors find periodic short-term bursts of flows and periods of long silence.

In another work [START_REF] Benson | Understanding data center traffic characteristics[END_REF], authors collect and process datasets from 19 datacenters with a wide range of workloads including web services, instant messaging and MapReduce. This work is focused on packet level traffic analysis and does not offer statistics on a per-flow basis. Authors observed an ON/OFF pattern for packet arrivals at ToR switches they monitored, meaning there was varying periods with no packet arrivals between bursts of packets.

In [START_REF] Benson | Network traffic characteristics of data centers in the wild[END_REF], authors study 10 datacenters categorized as educational (storage, email, web, video), enterprise (custom applications in addition to storage, web and email) and cloud (instant messaging, web, search, indexing, video). They report the number of active flows less than 10000 per second per rack across all datacenters. More than 40% of flows were less than 1 KB, more than 80% of were less than 10 KB, and almost all flows were less than 10 MB. According to their results, durations are more dependant on the specific datacenter: the smallest median flow duration was about 500 µs in one datacenter while the largest median flow duration was 1 second in a different datacenter. The largest flow duration was between 50 seconds and 200 seconds across all datacenters. This work also confirms the ON/OFF behavior of packet arrivals.

In a recent paper [START_REF] Roy | Inside the Social Network's (Datacenter) Network[END_REF], Facebook shares statistics on their traffic characteristics. They report flow size distributions on a per application basis for three major applications they run. Median and tail flow sizes for Hadoop, Web Server and Cache applications are reported to be between about 100 KB and 100 MB, 3 KB and 10 MB, 1 KB and 10 KB within racks while 1 KB and 1 MB, 5 KB and 500 KB, 30 KB and 3 MB between racks, respectively. Regarding flow durations, Hadoop flows had a median of about 300 ms and tail of less than 1000 seconds, Web Server flows had a median of about 900 ms and a tail of about 200 seconds, and Cache flows had a median of almost 400 seconds and a tail of almost 800 seconds, respectively. Per server, the median inter arrival time of various flow types was between 1000 µs and 10000 µs and the tail was between 10000 µs and 100000 µs. Finally, authors did not observe an ON/OFF packet arrival pattern at the switches which is suggested to be due to a large number of concurrent destinations, since ON/OFF pattern was observed on a per destination basis.

In addition to per-flow properties, since racks constitute a main unit in datacenter networks, one may be interested in how much traffic stays within racks. This could be helpful in deciding the over subscription ratio of cluster interconnection. The ratio of rack local to inter-rack traffic is dependent on applications that run within the datacenter and how instances of such applications are deployed on the servers. As a result, some prior work report a highly rack local traffic pattern [START_REF] Benson | Network traffic characteristics of data centers in the wild[END_REF] while some find traffic neither rack local nor all to all [START_REF] Roy | Inside the Social Network's (Datacenter) Network[END_REF],i.e., for some applications (e.g. Hadoop) traffic is mainly rack local while for others (e.g. Web Server and Cache) traffic is not at all rack local.

In summary, traffic characteristics, such as packet sizes, flow size distributions and flow inter-arrival times are highly correlated with applications. In addition, locality of traffic to racks is highly dependant on the way applications are deployed across the network and how such applications communicate with one another. For example, in [START_REF] Roy | Inside the Social Network's (Datacenter) Network[END_REF], authors report that servers of different types are deployed in separate racks and since most communication occurs between Web Servers and Cache Followers (different server types), there is less rack locality for these clusters. Given such strong dependency on applications, it is relatively hard to draw a general conclusion about datacenter traffic. Some common findings include the following. There is usually several orders of magnitude difference between median and maximum flow sizes (the actual difference varies according to applications). In addition, there can be a large number of flow arrivals per server (as many as thousands per second) with many concurrent flows. Finally, distributions of flow sizes and durations may be considerably different due to possibly uneven distribution of traffic across the network (flows may have to compete for bandwidth) and application of techniques like connection pooling which leads to long-lived connections that are not always transmitting.

C. Traffic Control Challenges

We present some datacenter traffic control challenges frequently pointed to in the literature. Table I provides a summary of these challenges.

1) Unpredictable Traffic Matrix: A variety of applications usually run on a datacenter creating many flows with different properties. Most datacenter flows are short1 (a few packets at most) [START_REF] Greenberg | VL2: A Scalable and Flexible Data Center Network[END_REF], [START_REF] Judd | Attaining the Promise and Avoiding the Pitfalls of TCP in the Datacenter[END_REF], [START_REF] Kandula | The nature of data center traffic: measurements & analysis[END_REF], [START_REF] Alizadeh | Data Center TCP (DCTCP)[END_REF], [START_REF] Benson | Understanding data center traffic characteristics[END_REF]- [START_REF] Roy | Inside the Social Network's (Datacenter) Network[END_REF], [START_REF] Alizadeh | pFabric: Minimal Near-optimal Datacenter Transport[END_REF] and many short flows may be only one packet long [START_REF] Alizadeh | pFabric: Minimal Near-optimal Datacenter Transport[END_REF]. However, most bytes are delivered by large flows [START_REF] Greenberg | VL2: A Scalable and Flexible Data Center Network[END_REF], [START_REF] Judd | Attaining the Promise and Avoiding the Pitfalls of TCP in the Datacenter[END_REF], [START_REF] Alizadeh | Data Center TCP (DCTCP)[END_REF]. [START_REF] Greenberg | VL2: A Scalable and Flexible Data Center Network[END_REF] reports 80% of flows to be less than 10 KB and 99% of flows to be less than 100 MB.

Datacenter applications also determine the flow arrival rates and distributions. The median flow inter-arrival time for a single machine was reported between 2 ms to 10 ms (100 to 500 flows per second) for different servers in Facebook datacenters [START_REF] Roy | Inside the Social Network's (Datacenter) Network[END_REF] (this median is calculated by measuring number of arriving flows per machine per second as samples averaged over number of machines). [START_REF] Benson | Network traffic characteristics of data centers in the wild[END_REF] reports between 100 to 10000 flow arrivals per switch in a one second bin in different educational and private datacenters. Finally, [START_REF] Kandula | The nature of data center traffic: measurements & analysis[END_REF] finds the median arrival rate of flows in a cluster to be 100 flows per millisecond.

High flow arrival rate majority of which being short can create an unpredictable and fluctuating traffic matrix which makes it hard to perform traffic engineering or capacity planning in longer time scales to improve performance [START_REF] Greenberg | VL2: A Scalable and Flexible Data Center Network[END_REF], [START_REF] Kandula | The nature of data center traffic: measurements & analysis[END_REF], [START_REF] Cao | Joint static and dynamic traffic scheduling in data center networks[END_REF], [START_REF] Hu | Coarse-grained traffic matrix estimation for data center networks[END_REF].

2) Mix of various flow types/sizes: Traffic in datacenters is a mix of various flow types and sizes [START_REF] Greenberg | VL2: A Scalable and Flexible Data Center Network[END_REF], [START_REF] Alizadeh | Data Center TCP (DCTCP)[END_REF], [START_REF] Benson | Understanding data center traffic characteristics[END_REF], [START_REF] Benson | Network traffic characteristics of data centers in the wild[END_REF], [START_REF] Wilson | Better Never Than Late: Meeting Deadlines in Datacenter Networks[END_REF], [START_REF] Chen | Scheduling Mix-flows in Commodity Datacenters with Karuna[END_REF]. Knowledge of various flow requirements and characteristics can help us design and tune transport protocols to more efficiently use network resources. Size and priority of a flow are usually determined by the application that initiates it. For some applications, flow sizes maybe unknown upon initiation.

Interactive flows which are created as a result of user interactions (for example generated by soft real time applications such as web search) can generate latency-sensitive flows that are usually short and should be delivered with high priority. Examples include queries (2 to 20 KB) and short messages (100 KB to 1 MB) [START_REF] Alizadeh | Data Center TCP (DCTCP)[END_REF]. Size of these flows is usually known apriori [START_REF] Wilson | Better Never Than Late: Meeting Deadlines in Datacenter Networks[END_REF]. Responsiveness of online services depends on how interactive flows are handled which can impact the number of users for an online service in the long run. In a study by Google, 400 ms increase in delay reduced the number of searches by 0.6% per user per day [START_REF] Brutlag | Speed matters for Google web search[END_REF]. Also, 100 ms added latency could reduce Amazon sales by 1% [START_REF] Linden | Make Data Useful[END_REF].

Throughput-oriented flows are not as sensitive to delay, but need consistent bandwidth [START_REF] Alizadeh | Data Center TCP (DCTCP)[END_REF]. They may range from moderate transfers (1 MB to 100 MB) such as ones created by data parallel computation jobs (e.g. MapReduce), to background long-running flows that deliver large volumes of data such as transfers that move data across datacenter sites for data warehousing and geo-replication [START_REF] Kandula | Calendaring for wide area networks[END_REF]. For these flows, it is still preferred to minimize the transfer time.

Deadline flows have to be completed prior to some deadlines. Their size is either known [START_REF] Chen | Scheduling Mix-flows in Commodity Datacenters with Karuna[END_REF] or a good estimate can typically be drawn [START_REF] Wilson | Better Never Than Late: Meeting Deadlines in Datacenter Networks[END_REF]. Both latency sensitive and throughput oriented flows might have deadlines. Deadlines can be either soft or hard which implies how value of its completion drops as time passes [START_REF] Zhang | Guaranteeing deadlines for inter-datacenter transfers[END_REF]. A soft deadline implies that it is still profitable to complete the flow and that the value decreases according to a utility function as we move past and away from the deadline. A hard deadline means zero value once the deadline has passed. For example, in interactive scattergather applications, if a query is not replied by its deadline (usually less than 300 ms [START_REF] Alizadeh | Data Center TCP (DCTCP)[END_REF], [START_REF] Decandia | Dynamo: Amazon's highly available key-value store[END_REF]), the final answer has to be computed without it [START_REF] Jalaparti | Speeding Up Distributed Request-response Workflows[END_REF] (i.e., zero value for that flow), while if a backup process is not completed in time, it is still valuable to finish it, although it might increase the risk of user data loss due to failures.

3) Traffic Burstiness: Several studies find datacenter traffic bursty [START_REF] Alizadeh | Less is more: trading a little bandwidth for ultra-low latency in the data center[END_REF], [START_REF] Alizadeh | Data Center TCP (DCTCP)[END_REF], [START_REF] Benson | Understanding data center traffic characteristics[END_REF]- [START_REF] Roy | Inside the Social Network's (Datacenter) Network[END_REF], [START_REF] Kapoor | Bullet trains: a study of NIC burst behavior at microsecond timescales[END_REF]. Theoretically, bursty traffic has been shown to increase packet loss and queuing delay while deceasing throughput [START_REF] Kleinrock | Queueing theory[END_REF]. In bursty environments, packet losses have been found more frequent at the network edge due to higher burstiness [START_REF] Benson | Understanding data center traffic characteristics[END_REF]. Burstiness can also lead to higher average queue occupancy in the network leading to increased flow completion times (FCT) for many flows [START_REF] Alizadeh | Less is more: trading a little bandwidth for ultra-low latency in the data center[END_REF] and increased packet drops due to temporary creation of full buffers in the network [START_REF] Alizadeh | Data Center TCP (DCTCP)[END_REF], [START_REF] Wu | ICTCP: Incast congestion control for TCP in data-center networks[END_REF]. In addition, highly bursty traffic can cause buffer space unfairness in shared memory switches if a switch port exhausts shared memory due to receiving long bursts [START_REF] Alizadeh | Data Center TCP (DCTCP)[END_REF].

Several causes can lead to creation of bursty traffic. Hardware offloading features, such as Large Send Offload (LSO), that reduce CPU load, can lead to higher burstiness. Interrupt Moderation (Coalescing) [START_REF]Interrupt Moderation[END_REF], which reduces CPU load and increases throughput by processing packets in batches, can lead to bursty traffic at the sender. Transport control features in software can create bursty traffic by scheduling a large window of packets to be sent together such as TCP slow start. Applications can increase burstiness by sending large pieces of data at once [START_REF] Kapoor | Bullet trains: a study of NIC burst behavior at microsecond timescales[END_REF].

4) Packet Reordering: Out of order arrival of packets can increase memory and CPU utilization and latency at the receiver especially at high rates [START_REF] Geng | Juggler: a practical reordering resilient network stack for datacenters[END_REF], [START_REF] Raiciu | How hard can it be? designing and implementing a deployable multipath TCP[END_REF]. Some transport protocol features, such as fast retransmit [START_REF] Stevens | RFC 2001: TCP Slow Start, Congestion Avoidance, Fast Retransmit, and Fast Recovery Algorithms[END_REF], may mistake reordering with packet loss. Different hardware and software features are involved in putting packets back in order including Large Receive Offloading (LRO) [START_REF]Large receive offload[END_REF], Receive Side Scaling (RSS) [START_REF]Receive Side Scaling[END_REF] and Generic Receive Offloading (GRO) [72]. LRO and GRO are usually implemented as part of driver software. Some NICs provide hardware support for these features. LRO focuses mostly on TCP/IPv4 stack while GRO is general purpose.

To understand the extend to which reordering can affect performance, we point to a prior work on improving the performance of handling out of order packet arrivals [START_REF] Geng | Juggler: a practical reordering resilient network stack for datacenters[END_REF]. Authors performed experiments with Vanilla Linux kernel and realized that at high rates (e.g. gigabits), significant reordering can increase CPU utilization to 100% and limit server interface link utilization to 70%. Even after applying optimizations at

Challenge

Description Implications

Unpredictable Traffic Matrix §III-C1

A traffic matrix represents the communication volume between pairs of end-points in a computer network. In datacenters, traffic matrix is varying and unpredictable.

Complicates traffic engineering and capacity planning.

Mix of Flow Types and Sizes §III-C2

Due to variety of applications that share the datacenter infrastructure, a mix of various flow types and sizes are generated. Flows may have deadline constraints or not and may be short latency-sensitive or large throughput-oriented. Also, for specific applications, flow sizes may be unknown.

Complicates flow scheduling to meet requirements of different flows over a shared medium.

Traffic Burstiness §III-C3 Burstiness has two different aspects. Traffic per flow could be highly bursty and flow arrival itself could be bursty as well. Burstiness intensity may change according to where traffic is measured, i.e., at end-point interfaces, at ToR switch ports, and so on.

Large buffer space at the switches to absorb bursts, careful and responsive traffic control to minimize average buffer space usage and react to bursts quickly. the driver and network stack level, CPU load increased by about 10% with server interface link utilization at 100%. 5) Performance Isolation: Performance isolation is necessary in cloud environments where multiple tenants use shared network resources [START_REF] Jeyakumar | EyeQ: Practical network performance isolation for the multi-tenant cloud[END_REF]- [START_REF] Popa | ElasticSwitch: practical work-conserving bandwidth guarantees for cloud computing[END_REF]. Isolation prevents selfish or malicious behavior that aims to either unfairly obtain more resources, such as by creating many flows or using custom aggressive transport protocols [START_REF] He | AC/DC TCP: Virtual congestion control enforcement for datacenter networks[END_REF], [START_REF] Cronkite-Ratcliff | Virtualized congestion control[END_REF], or to cause disruptions.

Enforcing performance isolation over a shared infrastructure is hard. To effectively isolate the effect of tenants and users, mechanisms need to be put in place in various layers and parts of the network. For example, a queue management scheme will need to divide buffer space according to users and tenants, bandwidth needs to be fairly divided, computational and memory overheads due to network communication needs to be controlled on a per user or per tenant basis, and all of this need to be done according to service level agreements between the operator, users and tenants.

6) The Incast Problem: When many end-hosts send traffic to one destination concurrently, the link attached to the destination turns into a bottleneck resulting in queue buildups, large queuing delays, and dropped packets [START_REF] Alizadeh | Data Center TCP (DCTCP)[END_REF], [START_REF] Wu | ICTCP: Incast congestion control for TCP in data-center networks[END_REF], [START_REF] Hwang | IA-TCP: a rate based incast-avoidance algorithm for TCP in data center networks[END_REF]- [START_REF] Yang | A Scalable Approach to Avoid Incast Problem from Application Layer[END_REF]. This problem becomes more serious in high bandwidth low latency networks [START_REF] Vasudevan | Safe and Effective Fine-grained TCP Retransmissions for Datacenter Communication[END_REF] and with shallow buffer switches [START_REF] Zarifis | DIBS: Just-in-time congestion mitigation for data centers[END_REF].

For example, the incast problem could appear in clusters running applications such as search and batch computing jobs like MapReduce that follow the partition-aggregate processing model. In search applications, a server may query a large number of other servers the results to which may be returned to that server at the same time creating sudden increase in incoming traffic. In a MapReduce job, a Reduce instance may download outputs of many Map instances for reduction which can cause a similar situation. Both scenarios follow the manyto-one communication pattern.

7)

The Outcast Problem: This problem occurs due to synchronized drops of packets from an input port of a switch which is referred to as port blackout [START_REF] Prakash | The TCP outcast problem: exposing unfairness in data center networks[END_REF]. This eventually leads to unfairness. For such synchronous drops to happen, two predicates have to be present. First, there needs to be contention between a large group of flows coming from one input port and a small group of flows coming from a different input port for access to the same output port. Second, the output port uses queues that follow TailDrop policy (if queue is full, the arriving packet is discarded). Port blackout occurs for the small group of flows and is observed temporarily over short periods of time. When the output queue is full, any arriving packet (during this window) is dropped which leads to consecutive drops for incoming flows. Such consecutive drops affect the smaller set of flows more than they affect the larger set (a smaller number of flows in the larger set are affected). The intensity of this problem increases as the ratio of flows in the larger over smaller group increases. This problem is called the "outcast" problem because some flows are cast aside (they cannot obtain bandwidth).

This could simply occur in tree-based topologies such as Fat-Tree and in partition-aggregate communication scenarios where many flows from different servers could be returning results to one server (many-to-one communication pattern). A disproportionate number of flows from incoming servers may end up on different input ports of the ToR switch attached to the receiver which could lead to flows on some port receiving less average throughput.

D. Traffic Control Objectives

Datacenter environments involve operators, tenants, and end-users with different objectives. Operators would like to use available resources as much as possible to provide higher volume of services, accommodate more tenants and eventually increase revenues. In addition, datacenter infrastructure may be shared by several tenants running various types of applications to offer different services. Tenants would like to receive a fair share of resources according to their service level agreement (SLA) which determines the cost of services. Finally, many end-users may rely upon services offered by datacenter tenants. They would like such services to be responsive and highly available. These possibly conflicting objectives are presented in the following. Table II provides a summary of traffic control objectives in datacenters.

1) Minimizing Flow Completion Times: Flow Completion Time (FCT) is the time from the initiation of a flow to its completion. Depending on applications, FCT can directly or indirectly affect the quality of experience and service offered to end users [START_REF] Alizadeh | Data Center TCP (DCTCP)[END_REF], [START_REF] Jalaparti | Speeding Up Distributed Request-response Workflows[END_REF], [START_REF] Dukkipati | Why flow-completion time is the right metric for congestion control[END_REF]. Major factors that determine FCT in datacenters include queuing and packet loss [START_REF] Judd | Attaining the Promise and Avoiding the Pitfalls of TCP in the Datacenter[END_REF], [START_REF] Alizadeh | Data Center TCP (DCTCP)[END_REF]. Occasionally, retransmission of lost segments can significantly increase latency due to the time it takes to identify and retransmit the lost data. For some applications, it may be more helpful to focus on minimizing mean or median latency (e.g. static web applications) [START_REF] Alizadeh | Data Center TCP (DCTCP)[END_REF], [START_REF] Gao | pHost: Distributed near-optimal datacenter transport over commodity network fabric[END_REF], [START_REF] Dukkipati | Why flow-completion time is the right metric for congestion control[END_REF]- [START_REF] Munir | On achieving low latency in data centers[END_REF], while for other applications tail latency may be more important (e.g. partitionaggregate) [START_REF] Zats | DeTail: reducing the flow completion time tail in datacenter networks[END_REF], [START_REF] Jalaparti | Speeding Up Distributed Request-response Workflows[END_REF].

2) Minimizing Deadline Miss Rate or Lateness: Many applications require timely delivery of data that can be viewed as flows with deadlines. In such applications, as long as pieces of data are delivered prior to the deadlines, customer SLAs are satisfied. The quality of services decrease as the fraction of missed deadlines increases. For some applications, delivery after the deadlines is still valuable and necessary. As a result, sometimes minimizing the amount by which we miss deadlines is more important which is referred to as "lateness" (e.g. synchronization of search index files).

3) Maximizing Utilization: Increasing resource utilization can reduce provisioning costs and increase revenues. By better utilizing network bandwidth, operators can accommodate more tenants or improve the quality of service for existing ones. Effectively utilizing the network depends partly on network topology and design parameters, and partly on network traffic control scheme.

4) Fairness: Many flows share datacenter resources such as link bandwidth and buffer space. In multi-tenant datacenters, it is necessary to make sure tenants receive fair share of network resources according to their service level agreement (SLA). Enforcing fairness also mitigates the starvation problem and prevents malicious behavior.

There are several definitions of fairness in networking context including max-min fairness, proportional fairness, and balanced fairness [START_REF] Bonald | A queueing analysis of max-min fairness, proportional fairness and balanced fairness[END_REF], [START_REF] Lan | An Axiomatic Theory of Fairness in Network Resource Allocation[END_REF]. Fairness criteria determine how link bandwidth or buffer space is divided among flows. Max-Min Fairness (MMF) [START_REF] Nace | Max-min fairness and its applications to routing and load-balancing in communication networks: a tutorial[END_REF], which aims at maximizing the minimum share, is the most widely used. In general, fairness can be considered over multiple dimensions each representing a different kind of resource (e.g., bandwidth, CPU cycles, and memory) [START_REF] Ghodsi | Dominant Resource Fairness: Fair Allocation of Multiple Resource Types[END_REF], [START_REF] Joe-Wong | Multiresource Allocation: Fairness-efficiency Tradeoffs in a Unifying Framework[END_REF]. We however focus on network bandwidth fairness in this paper.

Fairness should be considered among proper entities as defined by the fairness policy. For example, fairness may be across groups of flows as opposed to individual flows to prevent tenants from obtaining more resources by creating many flows. Strict fairness across all flows can also lead to increased number of missed deadlines [START_REF] Wilson | Better Never Than Late: Meeting Deadlines in Datacenter Networks[END_REF] and sub-optimal FCTs [START_REF] Hong | Finishing flows quickly with preemptive scheduling[END_REF]. One approach is to first apply fairness across tenants according to their classes of service and then across flows of each tenant considering flow priorities.

In addition to the traffic control objectives we mentioned, there are other objectives followed by many datacenter operators. An important objective is reducing energy costs by increasing energy efficiency. Since datacenter networks usually connect a huge number of servers, they are made of a large number of network equipment including fiber optics cables and switches. Due to varying amount of computational and storage load, average network load may be considerably less than its peak. As a result, operators may reduce energy costs by turning off a fraction of networking equipment at off-peak times [START_REF] Heller | ElasticTree: Saving Energy in Data Center Networks[END_REF]- [START_REF] Wang | Energy aware routing with link disjoint backup paths[END_REF] or by dynamically reducing the link bandwidths across certain links according to link utilizations [START_REF] Abts | Energy Proportional Datacenter Networks[END_REF]. There are however several challenges doing so. First, the resulting system should be fast enough to increase network capacity as computational or storage load increases to prevent additional communication latency. Second, it may be unclear where to reduce network capacity either by turning equipment off or by reducing link bandwidths (correlation between network load and placement of computation/storage can be considered for additional improvement [START_REF] Wang | GreenDCN: A General Framework for Achieving Energy Efficiency in Data Center Networks[END_REF]). Third, the effectiveness of such systems depends on load/utilization prediction accuracy. Further discussion on reducing datacenter power consumption is beyond the scope of this paper.

IV. DATACENTER TRAFFIC CONTROL MANAGEMENT

To enforce traffic control, some level of coordination is needed across the network elements. In general, traffic control can range from fully distributed to completely centralized. Here we review the three main approaches used in the literature namely distributed, centralized or hybrid. Table III provides an overview of traffic control management schemes.

A. Distributed

Most congestion management schemes coordinate in a distributed way as it is more reliable and scalable. A distributed

Objective Description

Minimizing Flow Completion Times (FCT) §III-D1 Faster completion times reduces the communication delay for distributed applications and improves their end-to-end responsiveness.

Minimizing Deadline Miss Rate or Lateness §III-D2 For time constrained applications, it is important to meet specific deadline requirements. For some applications, only transactions that complete prior to their deadlines are valuable in which case deadline miss rate is the right performance metric. For some applications transactions completed past the deadlines are still valuable or even necessary in which case lateness (i.e., by how much we miss deadlines), is the right metric.

Maximizing Utilization §III-D3

To maximize performance, it is desired to use available resources as much as possible.

Fairness §III-D4 Resources should be fairly divided across tenants and users while paying attention to their class of service and service level agreements (SLAs).

TABLE III SUMMARY OF DATACENTER TRAFFIC CONTROL MANAGEMENT

Scheme Benefits Drawbacks

Distributed §IV-A Higher scalability and reliability. Solutions can be completely end-host based or use explicit network feedback. More information in §IV-A1.

Access limited to local view of network status and flow properties. Limited coordination complicates enforcement of network-wide policies. More information in §IV-A2.

Centralized §IV-B Access to global view of network status and flow properties. Central control and management can improve flexibility and ease of enforcing network-wide policies. More information in §IV-B1.

A central controller can become a single point of failure or a network hotspot. Latency overhead of communicating with a central entity and control plane inconsistencies are other concerns. More information in §IV-B2.

Hybrid §IV-C Potentially higher reliability, scalability, and performance. More information in §IV-C1.

Higher complexity. Also, the final solution may not be as good as a fully centralized system. More information in §IV-C2.

scheme may be implemented as part of the end-hosts, switches, or both. Some recent distributed traffic control schemes include those presented in [START_REF] Zhu | Congestion Control for Large-Scale RDMA Deployments[END_REF], [START_REF] Hwang | IA-TCP: a rate based incast-avoidance algorithm for TCP in data center networks[END_REF], [START_REF] Hong | Finishing flows quickly with preemptive scheduling[END_REF], [START_REF] He | Presto: Edge-based load balancing for fast datacenter networks[END_REF], [START_REF] Mittal | TIMELY: RTT-based Congestion Control for the Datacenter[END_REF]. Designs that can be fully realized using end-hosts are usually preferred over ones that need changes in the default network functions or demand additional features at the switches such as custom priority queues [START_REF] Alizadeh | pFabric: Minimal Near-optimal Datacenter Transport[END_REF], in-network rate negotiation and allocation [START_REF] Hong | Finishing flows quickly with preemptive scheduling[END_REF], complex calculations in switches [START_REF] Nichols | Controlling queue delay[END_REF], or per flow state information [START_REF] Pan | Approximate fairness through differential dropping[END_REF]. End-host implementations are usually more scalable since every server handles its own traffic. Therefore, popular transport protocols rely on this type of implementation such as [START_REF] Alizadeh | Data Center TCP (DCTCP)[END_REF], [START_REF] Padhye | Modeling TCP Reno performance: a simple model and its empirical validation[END_REF].

As an example, the incast problem §III-C6, which is a common traffic control challenge, can be effectively addressed using end-host based approaches considering that incast congestion occurs at receiving ends. Some approaches are Serverbased Flow Scheduling (SFS) [START_REF] Zhang | Minimizing datacenter flow completion times with server-based flow scheduling[END_REF], pHost [START_REF] Gao | pHost: Distributed near-optimal datacenter transport over commodity network fabric[END_REF], NDP [START_REF] Handley | Re-architecting Datacenter Networks and Stacks for Low Latency and High Performance[END_REF] and ExpressPass [START_REF] Cho | Credit-Scheduled Delay-Bounded Congestion Control for Datacenters[END_REF]. SFS uses the generation of ACKs to control the flow of data towards receivers and avoid congestion. The sender looks at the flows it is sending and schedules the higher priority flows first, while the receiver controls the reception by deciding on when to generate ACKs. pHost uses a pull-based approach in which the sender decides on reception schedule based on some policy (preemptive or non-preemptive, fair sharing, etc). A source dispatches a Request To Send (RTS) to a receiver. The receiver then knows all the hosts that want to transmit to it and can issue tokens to allow them to send. NDP limits the aggregate transmission rate of all incast senders by maintaining a PULL queue at the receiver that is loaded with additional PULL requests when new packets arrive from a sender (a PULL request contains a counter which determines number of packets its associated sender is allowed to send). PULL requests are then sent to senders in a paced manner to make sure the overall incoming transmission rate at the receiver is not larger than per interface line rate. ExpressPass manages congestion across the network by controlling the flow of credit packets at the switches and end-hosts according to network bottlenecks (a sender is allowed to send a new data packet when it receives a credit packet).

Shifting more control to the network may allow for better resource management due to ability to control flows from more than a single server and availability of information about flows from multiple end-hosts. For example, flows from many hosts may pass through a ToR switch giving it further knowledge to perform scheduling and allocation optimizations. Some examples of this approach include RCP [START_REF] Dukkipati | Why flow-completion time is the right metric for congestion control[END_REF], PDQ [START_REF] Hong | Finishing flows quickly with preemptive scheduling[END_REF], CONGA [START_REF] Alizadeh | CONGA: Distributed Congestion-aware Load Balancing for Datacenters[END_REF], Expeditus [START_REF] Wang | Expeditus: Congestionaware Load Balancing in Clos Data Center Networks[END_REF], and RackCC [START_REF] Zhuo | Racklevel Congestion Control[END_REF]. RCP and PDQ perform in-network rate allocation and assignment by allowing switches and end-hosts to communicate using custom headers, CONGA gets help from switches to perform flowlet based load balancing in leaf-spine topologies, Expeditus performs flow based load balancing by implementing custom Layer 2 headers and localized monitoring of congestion at the switches, and RackCC uses ToR switches as means to share congestion information among many flows between the same source and destination racks to help them converge faster to proper transmission rates. To implement advanced in-network features, changes to the network elements might be necessary and switches may need to do additional computations or support new features.

1) Benefits: Higher scalability and reliability. Can be completely implemented using end-hosts. To further improve performance, network (i.e., switches, routers, etc.) can be involved as well. Completely end-host based approaches can operate simply by controlling the transmission rate according to implicit feedbacks from network (e.g. loss, latency). Network can offer explicit feedbacks (e.g. network queues' occupancy) to improve transmission rate management by allowing senders to make more accurate control decisions. For example, Explicit Congestion Notification (ECN) allows network to communicate high queue occupancy to end-hosts [START_REF] Floyd | RFC 3168: The Addition of Explicit Congestion Notification (ECN) to IP[END_REF], [START_REF] Bai | Enabling ECN over Generic Packet Scheduling[END_REF] or trimming packet payloads in case of highly occupied network queues (instead of fully discarding them) can help receivers get a complete picture of transmitted packets [START_REF] Handley | Re-architecting Datacenter Networks and Stacks for Low Latency and High Performance[END_REF], [START_REF] Cheng | Catch the whole lot in an action: Rapid precise packet loss notification in data center[END_REF].

2) Drawbacks: Usually just access to local view of network status and flow properties which allows for only locally optimal solutions. For example, while every end-point may strive to achieve maximum throughput for its flows by default (locally optimal), it may lead to a higher network wide utility if some end-points reduce their transmission rate in favor of other end-points with more critical traffic (globally optimal). Using distributed management, it may be harder to enforce new network wide policies (e.g. rate limits in a multi-tenant cloud environment) due to lack of coordination and limited view of network condition and properties.

B. Centralized

In centralized schemes a central unit coordinates transmissions in the network to avoid congestion. The central unit has access to a global view of network topology and resources, state information of switches, and end-host demands. These include flow sizes, deadlines and priorities as well as queuing status of switches and link capacities. Scheduler can proactively allocate resources temporally and spatially (several slots of time and different links) and plan transmissions in a way that optimizes the performance and minimizes contentions. To further increase performance, this entity can translate the scheduling problem into an optimization problem with resource constraints the solution to which can be approximated using fast heuristics. For large networks, scheduler effectiveness depends on its computational capacity and communication latency to end-hosts.

TDMA [START_REF] Vattikonda | Practical TDMA for datacenter Ethernet[END_REF], FastPass [START_REF] Perry | Fastpass: A Centralized "Zero-queue" Datacenter Network[END_REF] and FlowTune [START_REF] Perry | Flowtune: Flowlet control for datacenter networks[END_REF] are examples of a centrally coordinated network. TDMA divides timeline into rounds during which it collects end-host demands. Each round is divided into fixed sized slots during which hosts can communicate in a contention-less manner. All demands are processed at the end of a round and schedules are generated and given to end-hosts. FastPass achieves high utilization and low queuing by carefully scheduling traffic packet by packet considering variation in packet sizes. FlowTune improves on scalability of FastPass using centralized rate-assignment and end-host rate-control.

There are several challenges using a fully centralized approach. The central controller could be a single point of failure since all network transmissions depend on it. In case of a failure, end-hosts may have to fall back to a basic distributed scheme temporarily [START_REF] Perry | Fastpass: A Centralized "Zero-queue" Datacenter Network[END_REF]. There will be scheduling overhead upon flow initiation, that is the time it takes for the scheduler to receive, process the request, and allocate a transmission slot. Since majority of flows are short in datacenters, the scheduling overhead has to be tiny. In addition, this approach may only scale to moderate datacenters due to processing burden of requests and creation of a congestive hot-spot around the controller due to large number of flow arrivals. Bursts in flow arrivals [START_REF] Kandula | The nature of data center traffic: measurements & analysis[END_REF] can congest the controller temporarily and create scheduling delays. It may be possible to apply general techniques of improving scalability for central management of larger networks such as using a hierarchical design.

1) Benefits: Can provide higher performance with a global view of network status and flow properties. Such information may include utilization of different network edges, their health status as well as flows' size, deadline and priority. With this information, one can potentially direct traffic carefully according to network capacity across a variety of paths while allowing flows to transmit according to their priorities to maximize utility. Central management can improve flexibility and ease of managing network policies. For example, new routing/scheduling techniques can be rolled out much faster by only upgrading the central fabric. Centralized schemes also increase ease of admission control in case strict resource management is necessary for guaranteed SLAs.

2) Drawbacks: A central controller or management fabric can be a single point of failure or it may become a network hotspot in case of large networks. There is also latency and computational overhead of collecting network status and flow properties from many end-hosts (controller will have to process and understand incoming messages at high speed and act accordingly). Overhead of network resource allocation and scheduling (if central rate allocation is used). Finally, consistency of network updates can be an issue in large networks. For example, some updates my not be applied correctly at network elements (e.g. software bugs [START_REF] Govindan | Evolve or Die: High-Availability Design Principles Drawn from Google's Network Infrastructure[END_REF]) or different updates may be applied with varying latency that can lead to transient congestion or packet losses which may hurt performance of sensitive services.

C. Hybrid

Using a hybrid system could provide the reliability and scalability of distributed control and performance gains obtained from global network management. A general hybrid approach is to have distributed control that is assisted by centrally calculated parameters.

Examples of this approach include OTCP [START_REF] Jouet | OTCP: SDN-managed congestion control for data center networks[END_REF], Fibbing [START_REF] Vissicchio | Central Control Over Distributed Routing[END_REF], Hedera [START_REF] Al-Fares | Hedera: Dynamic Flow Scheduling for Data Center Networks[END_REF] and Mahout [START_REF] Curtis | Mahout: Low-overhead datacenter traffic management using end-host-based elephant detection[END_REF]. OTCP uses a central controller to monitor and collect measurements on link latencies and their queuing extent using methods provided by Software Defined Networking (SDN) [START_REF] Mckeown | OpenFlow: enabling innovation in campus networks[END_REF] which we will discuss further in §VII-A. For every new flow, the controller calculates parameters such as initial and minimum retransmission timeout and initial and maximum congestion window size considering which path is taken by flows which allows for fast convergence to steady state. Fibbing relies on a central controller to manipulate the costs associated with routes in the network or insert fake nodes into the routing tables of routers to force them to use or avoid some paths to control the distribution of load across network. Hedera and Mahout operate by initially allowing network to route all incoming flows using a distributed approach, then monitoring and detecting large flows that can be moved dynamically to different routes using a central controller with a global view of network status which allows for a more balanced distribution of load. While Hedera uses readily available switch capabilities (flow counters) to detect large flows, Mahout engineers and inserts a shim layer at the end-hosts to monitor and detect large flows to improve scalability.

1) Benefits: Reliability, scalability, and higher performance. A central controller can offer coarse grained solutions while fine-grained control is applied using a distributed scheme. Division of work between central and distributed components can be done in a way that minimizes the effects of centralized and distributed schemes' drawbacks. For example, in multi-tenant cloud datacenters, a hierarchical approach can be used where aggregate bandwidth given per tenant is calculated centrally while transmission control across flows per tenant is performed in a distributed fashion reducing central management overhead and increasing per tenant management scalability.

2) Drawbacks: Complexity may increase. Central controller now has limited control; therefore, the final solution may not be as good as a fully centralized system. Due to presence of centralized control, there still exists a single point of failure however with less impact in case of a failure compared to a fully centralized scheme. Also, the distributed component still operates on a locally optimal basis. For example, in multi-tenant cloud datacenters, if bandwidth per tenant is managed in a distributed fashion, due to limited local information per network element, it may be challenging to apply routing/scheduling policies that maximize utility according to flow properties.

V. DATACENTER TRAFFIC CONTROL TECHNIQUES

Many design parameters and approaches can be considered in development of any traffic control scheme. Figure 3 provides a high level breakdown of major datacenter traffic control techniques. In the following, we provide a list of these techniques and discuss some research efforts made regarding each. Figure 4 provides an overview of the relationships between challenges, objectives and techniques. Details of each relationship is further explained in Tables V and IV. More detailed information is provided in the following sections.

A. Transmission Control

Transmission control is the mechanism that controls the flow of data sent to the network. There is typically a window of outstanding bytes for each flow at the sender determining the volume of data that can be transmitted before data reception is acknowledged. This allows for implicit rate control. A larger window may increase the average transmission rate since it allows for more bytes in flight towards receiver. A significant body of work, including recent works, employ window-based rate control. Table VI provides a summary of this section. Some recent window-based approaches include DCTCP [START_REF] Alizadeh | Data Center TCP (DCTCP)[END_REF], D2TCP [START_REF] Vamanan | Deadline-aware Datacenter TCP (D2TCP)[END_REF], L2DCT [START_REF] Munir | Minimizing flow completion times in data centers[END_REF], MCP [START_REF] Chen | Towards minimaldelay deadline-driven data center TCP[END_REF], DAQ [START_REF] Ding | DAQ: deadline-aware queue scheme for scheduling service flows in data centers[END_REF], and PASE [START_REF] Munir | Friends, Not Foes: Synthesizing Existing Transport Strategies for Data Center Networks[END_REF]. DCTCP uses explicit congestion signals from switches that is piggybacked on ACKs to change the window size according to the extent of congestion. Packets are marked to convey congestion signal according to instantaneous queue length upon their arrival at the queue. D2TCP modulates window size based on flow deadlines: flows with closer deadlines reduce window size less than others in response to network congestion signals. from a flow up to current time. MCP changes window size to approximate the solution to an stochastic optimization problem that minimizes long term mean packet delay using flow information and network state, such as queue length at the switches. DAQ allows ToR switches to calculate initial sender window for end-hosts to converge to proper transmission rates faster. PASE also gets help from network elements to decide on its initial coarse-grained window size, and then performs fine-grained adjustments to the window size using congestion signals received similar to DCTCP.

Although fairly simple, window-based approaches only allow for coarse-grained control of rate which can result in considerable variations and create highly bursty traffic. For example, a sender can release a full window of packets before it has to wait for the receiver to acknowledge. This may be followed by a batch of acknowledgements that move the window forward and allow for another full window of packets to be sent.

To decrease the transmission rate variations at the senders, rate-based mechanisms that employ explicit rate control can be used. Packets in the outstanding window are carefully scheduled to achieve the proper bit rate which implies reduced burstiness of the traffic. Rate-control is necessary when using reservation based bandwidth control over shared links such as in [START_REF] Perry | Flowtune: Flowlet control for datacenter networks[END_REF].

Examples of earlier rate-based protocols include TCP Friendly Rate Control (TFRC) [START_REF] Floyd | Equation-based congestion control for unicast applications[END_REF] and Rate Control Protocol [START_REF] Dukkipati | Why flow-completion time is the right metric for congestion control[END_REF]. Several recent work also use rate-based methods in datacenters including PDQ [START_REF] Hong | Finishing flows quickly with preemptive scheduling[END_REF], D3 [START_REF] Wilson | Better Never Than Late: Meeting Deadlines in Datacenter Networks[END_REF], TIMELY [START_REF] Mittal | TIMELY: RTT-based Congestion Control for the Datacenter[END_REF], and RACKS [START_REF] Munir | On achieving low latency in data centers[END_REF]. TFRC calculates the allowed sending rate to compete fairly with TCP using an equation which captures various network parameters such as round trip time (RTT), loss rate, retransmission timeout, and segment size. RCP uses an equation based rate control to minimize average FCT with the help of network switches that divide bandwidth according to processor sharing policy. D3 uses flow deadlines as a factor in the rate calculation equation to reduce deadline miss rate (a closer deadline allows for a higher rate). RACS uses a similar approach to RCP but assigns weights to flows according to their priority, to emulate different scheduling algorithms at the switches according to weight assignment. PDQ improves on both RCP and D3 by rate allocation using switches and adding support for preemption. TIMELY calculates the sending rate as a function of network latency.

Explicit rate control can be applied both in hardware (e.g. NIC) and software (e.g. OS Kernel). While the former provides more precise rates, the latter is more flexible and allows for a larger number of flows to be rate controlled as well as more complex policies to be considered. Rate-control in software can result in creation of occasional bursts due to challenges of precise packet scheduling in OS kernel [START_REF] Floyd | Equation-based congestion control for unicast applications[END_REF].

Complementary to these two approaches for rate control at the sender, one can also apply token-based transmission control which adds an extra control layer to make sure senders only transmit according to some quota assigned to them (this is sometimes referred to as pull-based or credit-based approach). For example, such quota could be assigned according to congestion status at receivers (to address the incast problem discussed earlier in §III-C6) or to implement a form of receiver based flow scheduling policy (more information in §V-F).

B. Traffic Shaping

We can improve network performance by making sure that it conforms to required profile and policy rules. This can reduce contention while using network resources. For example, traffic shaping can prevent some flows from hogging others. Shaping can be used to provide some level of resource isolation and guarantees in cloud environments with many users. Finally, 4 Relationship Description Some Related Works

1 2
Transmission control at senders determines number of packets given to network driver for transmission. If a bunch of packets are sent to network interface for transmission, bursts may be created.

[124], [START_REF] Ghobadi | TCP pacing in data center networks[END_REF] 1 4

If transmission is performed over multiple interfaces for multipath delivery, careful transmission control is required to minimize packet reordering. This shall be done according to latencies of different paths to the receiver.

[126], [START_REF] Cao | Per-packet load-balanced, low-latency routing for Clos-based data center networks[END_REF], [START_REF] Xu | TinyFlow: Breaking elephants down into mice in data center networks[END_REF],

[98]

2 2
Rate limiting can reduce burstiness by adding a protective layer after transmission control in case a large number of packets are given to the network driver for transmission.

[129], [START_REF] Saeed | Carousel: Scalable Traffic Shaping at End Hosts[END_REF], [START_REF] He | Low Latency Software Rate Limiters for Cloud Networks[END_REF] 2 5 Rate limiting can improve performance isolation across tenants and users by preventing any tenant/flow from taking too much bandwidth starving other tenants/flows (a tenant may initiate many flows).

[73], [START_REF] Rodrigues | Gatekeeper: Supporting Bandwidth Guarantees for Multi-tenant Datacenter Networks[END_REF] 3 2 Packet pacing reduces burstiness by adding time spacing between consecutive packets prior to transmission. [START_REF] Aggarwal | Understanding the performance of TCP pacing[END_REF], [START_REF] Ghobadi | TCP pacing in data center networks[END_REF] 4 3 Prioritization helps allocate resources to flows accordingly and can improve the performance while scheduling a mix of flow types/sizes. For example, one could prioritize highly critical deadline flows over other flows.

[133], [START_REF] Bai | PIAS: Practical information-agnostic flow scheduling for data center networks[END_REF], [START_REF] Alizadeh | pFabric: Minimal Near-optimal Datacenter Transport[END_REF], [START_REF] Munir | Friends, Not Foes: Synthesizing Existing Transport Strategies for Data Center Networks[END_REF], [START_REF] Grosvenor | Queues don't matter when you can JUMP them![END_REF] 4 5 Prioritization can shield a tenant/application with high service guarantees from other tenants/applications with lower service guarantees. Priorities shall be determined according to tenant/application service level agreements (SLAs) or Quality of Service requirements (QoS).

[136], [START_REF] Hu | Providing bandwidth guarantees, work conservation and low latency simultaneously in the cloud[END_REF] 5 1 Datacenter topologies usually come with a large degree of path redundancy to provide low over-subscription ratio for better inter-rack communication. With load balancing, this capacity can be used to a higher extent giving the operators/tenants a better chance to handle any traffic matrix.

[5], [10]

6 4
Multipathing can increase packet reordering. To reduce reordering while using multiple paths, one can perform careful packet scheduling according to path latencies at transmission control level so that packets arrive in the right order at the receiver (it is nearly impossible to eliminate reordering). Assignment of data segments to sub-flows is also an important factor in how well receivers can handle reordering.

[68], [START_REF] Sinha | Harnessing TCP's burstiness using flowlet switching[END_REF] 7 2 Transmission control, rate limiting, and packet pacing all depend on careful scheduling of packets at senders which can mitigate burstiness.

[123], [START_REF] Radhakrishnan | NicPic: Scalable and Accurate End-Host Rate Limiting[END_REF], [START_REF] Saeed | Carousel: Scalable Traffic Shaping at End Hosts[END_REF],

[138] 7 3 When a mix of flow types is present, scheduling of packets according to flow priorities, deadlines, sizes and arrival order can help us better meet traffic control objectives.

[86], [START_REF] Bai | PIAS: Practical information-agnostic flow scheduling for data center networks[END_REF], [START_REF] Alizadeh | pFabric: Minimal Near-optimal Datacenter Transport[END_REF] 7 4

If multiple paths are used for a flow, scheduling can reduce reordering by determining when packets should be sent at the sender over each path to arrive at the receiver with minimal reordering.

[126]

7 6
Scheduling can mitigate the incast problem by preventing all the incoming data (from many flows) from arriving at a receiver at the same time. Various techniques can be used such as adding random delays while initiating requests (jittering) and receiver based scheduling using either ACKs or receiver window size.

[79], [81]

7 7
Effective queuing disciplines which determine how packets in a switch queue are scheduled for transmission, such as Stochastic Fair Queuing (SFQ), can mitigate port blackout §III-C7. [START_REF] Prakash | The TCP outcast problem: exposing unfairness in data center networks[END_REF] traffic shaping can be useful in resource scheduling where senders follow rates specified in the schedule. A summary of traffic shaping techniques has been provided in Table VII.

1) Rate Limiting: Rate limiting is usually applied by passing traffic through a Token Bucket filter. This ensures that average transmission rate does not exceed the token generation rate. Tokens are generated at a specific rate and an arriving packet can only be sent if there is a token available. Tokens are accumulated if there is no packet to be sent, but there is usually a cap on how many. This cap is to limit traffic burstiness in case the token bucket is idle for a while. Examples of works employing rate-limiting to provide resource isolation and guarantees include [START_REF] Wilson | Better Never Than Late: Meeting Deadlines in Datacenter Networks[END_REF], [START_REF] Jeyakumar | EyeQ: Practical network performance isolation for the multi-tenant cloud[END_REF], [START_REF] Perry | Flowtune: Flowlet control for datacenter networks[END_REF], [START_REF] Rodrigues | Gatekeeper: Supporting Bandwidth Guarantees for Multi-tenant Datacenter Networks[END_REF], [START_REF] Grosvenor | Queues don't matter when you can JUMP them![END_REF], [START_REF] Alizadeh | Stability analysis of QCN: the averaging principle[END_REF], [START_REF] Tian | Multi-tenant multi-objective bandwidth allocation in datacenters using stacked congestion control[END_REF].

Rate-limiting can be done in OS Kernel, as part of the hypervisor in a virtualized environment, or via NIC. Scheduling of packets in software (Kernel or Hypervisor) is generally less precise and can be computationally intensive in high bandwidths and with a large number of flows. To reduce CPU utilization, OSes usually send packets to NIC in batches which can further reduce the scheduling precision. For example, Classful Queuing Disciplines (Qdisc) offered by Linux allows for coarse grained rate-control by determining the time and count of packets that NIC receives from RAM, however, the actual schedule of packets on the wire is determined by NIC.

To improve software rate-limiting performance one can use userspace packet processing tools some of which we point to in §VII-D. For example, Carousel [START_REF] Saeed | Carousel: Scalable Traffic Shaping at End Hosts[END_REF] is a rate-limiter that is implemented as part of a software NIC in userspace. Carousel uses a variety of techniques to reduce CPU and memory usage and improve scalability including deferred completion signalling (rate-limiter only signals completion to applications when data is actually transmitted to offer backpressure and minimize buffer space usage) and single queue shaping (using a timing wheel and by assigning timestamps to packets over the time horizon).

Using NIC to schedule and send packets given rate limits reduces CPU load and traffic burstiness. Effective rate-limiting in hardware demands support for multiple queues and classes of traffic with hardware rate-limiters attached to them.

Hybrid rate-limiting approaches can be used to support a large number of priority classes while reducing hardware com- 4 Relationship Description Some Related Works 1 Transmission control can impact overall end-to-end latency by affecting queue occupancies at the network switches.

[124], [START_REF] Ghobadi | TCP pacing in data center networks[END_REF] 1 Transmission control determines how many packets are sent by each flow which can be done according to flow deadlines to reduce deadline miss rate and/or lateness.

[86], [START_REF] Vamanan | Deadline-aware Datacenter TCP (D2TCP)[END_REF], [START_REF] Wilson | Better Never Than Late: Meeting Deadlines in Datacenter Networks[END_REF] 1 Proper transmission control can maximize network bandwidth utilization while avoiding congestion (which usually leads to dropped packets and wasted bandwidth).

[112], [START_REF] Perry | Flowtune: Flowlet control for datacenter networks[END_REF], [START_REF] Gao | pHost: Distributed near-optimal datacenter transport over commodity network fabric[END_REF] 1 Transmission control plays significant role in how fairly flows share the network. For example, if one of two equally important flows is given higher transmission quota over longer periods of time, this can lead to unfairness.

[85]

2 Rate limiting can prevent congestion and reduce dropped packets. As a result, it helps maximize utilization and minimize wasted bandwidth.

[113]

2 Rate limiting can improve fairness by preventing selfish behavior of bandwidth hungry flows/tenants. [START_REF] Jeyakumar | EyeQ: Practical network performance isolation for the multi-tenant cloud[END_REF], [START_REF] Rodrigues | Gatekeeper: Supporting Bandwidth Guarantees for Multi-tenant Datacenter Networks[END_REF] 3 Packet pacing can reduce average queue occupancy in the network (switches/routers) and therefore reduces end-to-end latency.

[124], [START_REF] Ghobadi | TCP pacing in data center networks[END_REF], [START_REF] Alizadeh | Less is more: trading a little bandwidth for ultra-low latency in the data center[END_REF] 3 Packet pacing can increase effective utilization by preventing bursty behavior of flows which can lead to higher buffer occupancy and dropped packets. It stabilizes transmission rates and reduces transmission spikes.

[125], [START_REF] Alizadeh | Less is more: trading a little bandwidth for ultra-low latency in the data center[END_REF] 4 Prioritization can reduce average latency and flow completion times by giving higher priority to shorter flows.

[53], [START_REF] Bai | PIAS: Practical information-agnostic flow scheduling for data center networks[END_REF] 4

Prioritization according to flow deadlines can improve the overall deadline miss rate. For example, search queries with critical deadlines (e.g. 300 ms after arrival) can be given high priority and can be addressed before long running backup operations in a shared network environment.

[121]

5

Load balancing allows us to make best use of large path diversity in datacenters and maximize utilization over all available paths.

[116], [START_REF] Alizadeh | CONGA: Distributed Congestion-aware Load Balancing for Datacenters[END_REF], [START_REF] He | Presto: Edge-based load balancing for fast datacenter networks[END_REF], [START_REF] Zats | DeTail: reducing the flow completion time tail in datacenter networks[END_REF] 6

Multipathing can increase utilization by allowing long running flows to use bandwidth of several available paths.

[45], [START_REF] Raiciu | How hard can it be? designing and implementing a deployable multipath TCP[END_REF], [139]

7 Scheduling can reduce latency by applying scheduling disciplines that mimic Shortest Remaining Processing Time (SRPT).

[87]

7

Deadline miss rate or lateness can be reduced by scheduling flows according to their deadlines such as by allotting more capacity for flows with closer deadlines.

[28]

7 Scheduling can improve utilization by reducing contention within the network for using available bandwidth. For example, by carefully deciding on when packets should be sent by end-hosts, we can avoid sudden arrival of many packets at the switches which can lead to dropped packets.

[111], [START_REF] Perry | Fastpass: A Centralized "Zero-queue" Datacenter Network[END_REF] 7 Scheduling can improve fairness by giving bandwidth to flows based on a fair sharing policy such as Max-Min Fairness (MMF) [START_REF] Nace | Max-min fairness and its applications to routing and load-balancing in communication networks: a tutorial[END_REF].

[85], [START_REF] Munir | On achieving low latency in data centers[END_REF] A sender needs to first coordinate with the receiver before it embarks on transmission which can add some latency overhead.

plexity and keeping scheduling precision high. NicPic [START_REF] Radhakrishnan | NicPic: Scalable and Accurate End-Host Rate Limiting[END_REF] classifies and stores packets in queues located in RAM and labels them with proper rate limits by host CPU. Packets are then fetched by NIC via DMA and scheduled using hardware rate-limiters. NIC first decides on which flow's packets to send and then pulls them from RAM.

As the last option, rate-limits can also be applied at the application layer. Applications can do this by limiting the volume of data handed off to transport layer over periods of time. This approach is simple but requires changes to applications. Also, rate-limiting precision will be limited. Finally, this may lead to bursty traffic as incoming application data to transport layer may get buffered prior to transmission on the wire, i.e., applications have no control over how data is eventually sent.

Scheme Description Limitations

Rate Limiting §V-B1 Limits the transmission rate of outgoing packets from the rate limiter. A token bucket is usually used to limit maximum persistent packet transmission rate to token arrival rate.

Rate limiting has limited accuracy depending on how it is enforced, in software (usually less accurate but cheaper) or hardware (usually more accurate but needs hardware support). Also, various implementation techniques in software lead to different accuracy such as in the operating system kernel or using kernel bypass modules (see §VII-D).

Packet Pacing §V-B2 Inserts time spacing between consecutive packs to spread them uniformly across a window of round trip time (RTT). This reduces traffic burstiness and average network buffer occupancy, therefore improving end-to-end latency.

Packet pacing has limited accuracy and similar to ratelimiting, its accuracy depends on the implementation technique used.

2) Packet Pacing: Packet Pacing is the process of adding space between consecutive packets so that they do not arrive at the network back to back which reduces traffic burstiness. Burstiness can degrade network performance in several ways. Long bursts can overload switch buffer and create consecutive packet drops. Average latency of all packets then increases since they have to wait in longer queues. In addition, it creates transmission rate oscillations making it hard to do careful bandwidth allocation [START_REF] Alizadeh | Less is more: trading a little bandwidth for ultra-low latency in the data center[END_REF], [START_REF] Alizadeh | Data Center TCP (DCTCP)[END_REF].

Earlier work [START_REF] Aggarwal | Understanding the performance of TCP pacing[END_REF] experimenting with pacing in a general network setting has shown that it can considerably reduce queuing delay. Combined with other types of congestion signals, pacing can improve the performance by evenly distributing traffic across the timeline [START_REF] Alizadeh | Less is more: trading a little bandwidth for ultra-low latency in the data center[END_REF]. In addition, pacing should only be applied to long-running and throughput-oriented flows to reduce their impact on short latency-sensitive flows [START_REF] Alizadeh | Less is more: trading a little bandwidth for ultra-low latency in the data center[END_REF]. The benefit of pacing depends on the network bandwidth-delay product, buffer size, and the number of flows. Having so many flows that share a buffer can reduce the effectiveness of pacing due to inter-flow burstiness and creation of synchronized drops [START_REF] Ghobadi | TCP pacing in data center networks[END_REF].

Pacing can be done in both hardware and software, but hardware pacing can be more effective due to higher scheduling precision, especially at high rates where spacing between packets is tiny [START_REF] Alizadeh | Less is more: trading a little bandwidth for ultra-low latency in the data center[END_REF]. Software pacing may be performed by end-hosts as part of a driver or a kernel module. In cloud environments, due to widespread use of virtualization, packets may be paced at the virtual interfaces in the hypervisor software. Pacing in software may be overridden by NIC offloading features such as LSO, if enabled, since NIC is the actual entity that sends packets out on the wire. In hardware pacing, packets are buffered at the NIC, each assigned a timer, and scheduled to be sent when their timer goes off.

Pacing can also be done at the network edges (e.g. ToR switches) as opposed to end-hosts. For example, Queue Length Based Pacing (QLBP) [START_REF] Cai | Practical packet pacing in smallbuffer networks[END_REF] uses a pacing controller attached to edge queues to determine when the next packet in the queue is supposed to be sent as a function of queue length.

C. Prioritization

Table VIII provides an overview of this section. Classifying flows based on their priorities and treating them accordingly can improve performance. Such prioritization can be done innetwork by using multiple queues at the switches and allowing higher priority traffic to go over lower priority traffic [START_REF] Alizadeh | pFabric: Minimal Near-optimal Datacenter Transport[END_REF], [START_REF] Munir | Friends, Not Foes: Synthesizing Existing Transport Strategies for Data Center Networks[END_REF], [START_REF] Bai | PIAS: Practical information-agnostic flow scheduling for data center networks[END_REF], [START_REF] Grosvenor | Queues don't matter when you can JUMP them![END_REF], and at the senders by performing ratecontrol according to priorities [START_REF] Vamanan | Deadline-aware Datacenter TCP (D2TCP)[END_REF], [START_REF] Wilson | Better Never Than Late: Meeting Deadlines in Datacenter Networks[END_REF], [START_REF] Hong | Finishing flows quickly with preemptive scheduling[END_REF], [START_REF] Munir | Minimizing flow completion times in data centers[END_REF].

Priorities are usually assigned either based on flow size to minimize mean latency (by mimicking SRPT scheduling policy) [START_REF] Alizadeh | pFabric: Minimal Near-optimal Datacenter Transport[END_REF], [START_REF] Bai | PIAS: Practical information-agnostic flow scheduling for data center networks[END_REF] or based on deadlines to minimize the number of deadline missing flows [START_REF] Gao | pHost: Distributed near-optimal datacenter transport over commodity network fabric[END_REF], [START_REF] Munir | Friends, Not Foes: Synthesizing Existing Transport Strategies for Data Center Networks[END_REF]. Control traffic is naturally prioritized to improve the feedback timeliness and quality (e.g. ACKs in TIMELY [START_REF] Mittal | TIMELY: RTT-based Congestion Control for the Datacenter[END_REF] and Trimmed Packets in NDP [START_REF] Handley | Re-architecting Datacenter Networks and Stacks for Low Latency and High Performance[END_REF]) or decrease control loop delay (e.g. RTS in pHost [START_REF] Gao | pHost: Distributed near-optimal datacenter transport over commodity network fabric[END_REF]).

For many applications, flow sizes are either known or can be roughly estimated upon initiation [START_REF] Wilson | Better Never Than Late: Meeting Deadlines in Datacenter Networks[END_REF], [START_REF] Alizadeh | pFabric: Minimal Near-optimal Datacenter Transport[END_REF], [START_REF] Hong | Finishing flows quickly with preemptive scheduling[END_REF], [START_REF] Munir | Friends, Not Foes: Synthesizing Existing Transport Strategies for Data Center Networks[END_REF] making it easy to assign priorities by size to reduce mean latency. In case flow sizes are unknown apriori, dynamic prioritization can be used where packets of a flow first receive the highest priority, but get demoted to lower priorities as more of them is seen.

For example, dynamic Packet Prioritization (DPP) [START_REF]Cisco Application Centric Infrastructure Fundamentals[END_REF] uses two queues, an express queue which has higher priority and a normal queue. It reduces the priority of long running flows by counting their packets against a threshold. Having multiple queues allows for more precise classification of flows [START_REF] Munir | Friends, Not Foes: Synthesizing Existing Transport Strategies for Data Center Networks[END_REF], [START_REF] Bai | PIAS: Practical information-agnostic flow scheduling for data center networks[END_REF]. However, recent work shows that most benefit in reducing mean FCT can be obtained using up to 8 queues [START_REF] Alizadeh | pFabric: Minimal Near-optimal Datacenter Transport[END_REF]. Finding proper threshold values based on which flows are demoted to lower priorities may require solving an optimization problem that takes into account the flow arrival rates and flow size distribution. In datacenter environments with known traffic characteristics and workloads, such thresholds may be determined offline and applied in real-time [START_REF] Bai | PIAS: Practical information-agnostic flow scheduling for data center networks[END_REF]. It is also possible to virtually emulate the behavior of having infinite number of queues using only two actual queues per switch port, a high and a low priority queue [START_REF] Lu | One more queue is enough: Minimizing flow completion time with explicit priority notification[END_REF]. This can be achieved by assigning flows with highest priority to high priority queue while the rest of flows to low priority queue and dynamically changing flows assigned to high priority queue when other flows with a higher priority complete.

Prioritization can be performed fully at switches by keeping state on flows passing through and using some priority assignment criteria such as the total number of packets sent. This simplifies end-hosts at the expense of higher computational and memory burden on the switches.

Another approach is for the end-hosts to tag flows with

Custom approaches

Can be used by adding support to switches and/or end-hosts, may require changes in software/hardware to switches and/or end-hosts.

Enforcement Strict

A lower priority flow is only sent when there are no packets available from any of the higher priority flows. This minimizes the effect of lower priority flows on higher priority ones but can lead to starvation of lower priority flows.

Non-strict A lower priority flow can be sent even if there are packets available from higher priority flows. This occurs when a required volume of higher priority flows' demand is satisfied (e.g. one low priority packet is sent for every K ≥ 1 high priority packets sent) and mitigates the starvation problem of lower priority flows.

priority tags while switches just process tags and put packets in proper queues [START_REF] Bai | PIAS: Practical information-agnostic flow scheduling for data center networks[END_REF], [START_REF] Grosvenor | Queues don't matter when you can JUMP them![END_REF]. End-host priority tagging can be done at the NIC, OS kernel, hypervisor, or even by applications before packets are sent to the network. In case of virtualization, if end-host VMs cannot be trusted to assign priorities properly, middleboxes can be used (e.g. at the hypervisor) that monitor and tag packets (e.g. using OpenVSwitch [START_REF] Pfaff | The design and implementation of Open vSwitch[END_REF]) which applies to both static [START_REF] Alizadeh | pFabric: Minimal Near-optimal Datacenter Transport[END_REF] and dynamic [START_REF] Munir | Friends, Not Foes: Synthesizing Existing Transport Strategies for Data Center Networks[END_REF], [START_REF] Bai | PIAS: Practical information-agnostic flow scheduling for data center networks[END_REF] prioritization.

Priorities can also be assigned in different ways. Ethernet standard IEEE 802.1Q priority based forwarding [START_REF] Ek | [END_REF] that provides 8 levels is supported by many switch vendors and can also be used in datacenters [START_REF] Grosvenor | Queues don't matter when you can JUMP them![END_REF]. At the IP layer, Differentiated services (DiffServ) [START_REF] Blake | An architecture for differentiated services[END_REF] can be used [START_REF] Guo | RDMA over commodity Ethernet at scale[END_REF]. Custom queuing techniques and headers can also be used which may require changes to both switches and end-hosts [START_REF] Alizadeh | pFabric: Minimal Near-optimal Datacenter Transport[END_REF].

Strictly prioritizing flows can lead to starvation where lower priority flows cannot make progress due to large volume of higher priority traffic. A simple solution is to use weighted queuing instead of strict prioritization. For instance, DAQ [START_REF] Ding | DAQ: deadline-aware queue scheme for scheduling service flows in data centers[END_REF] uses a weighted round-robin between long and short flows to make sure that throughput-oriented flows keep making progress. Aging can also be used to address starvation while minimally affecting critical flows. An aging-rate can be used to increase the priority of low priority flows as a function of their waiting time [START_REF] Hong | Finishing flows quickly with preemptive scheduling[END_REF].

D. Load Balancing

Datacenter topologies typically provide a large degree of path redundancy. Properly distributing load across these paths reduces contention among flows while increasing overall resource utilization. Without effective load balancing many links may not be utilized while some experiencing congestion [START_REF] Benson | Understanding data center traffic characteristics[END_REF], [START_REF] Kandula | Flyways To De-Congest Data Center Networks[END_REF]. Table IX provides an overview of general load balancing concepts and their trade-offs.

Load balancing can be static or dynamic (adaptive). Static approaches use a fixed criteria to assign traffic to available paths such as by hashing specific fields from packet headers. For example, ECMP [150] is a popular static load balancing technique that only distributes load across equal cost paths. Adaptive load balancing dynamically selects paths for traffic according to distribution of load to minimize hot-spots. Various criteria can be used for path assignment such as per-hop or per-path queue occupancies [START_REF] Mahapatra | Load balancing mechanisms in data center networks[END_REF]. After choosing the initial path according to current load, some adaptive approaches keep monitoring the network status and distribution of traffic. They change direction of traffic to eliminate or reduce hot-spots. These approaches are referred to as reactive. If not applied with care, reactive load balancing might lead to oscillations.

Examples of reactive dynamic load balancing techniques include Planck [START_REF] Rasley | Planck: Millisecond-scale monitoring and control for commodity networks[END_REF], Hedera [START_REF] Al-Fares | Hedera: Dynamic Flow Scheduling for Data Center Networks[END_REF], MPTCP [START_REF] Ford | TCP extensions for multipath operation with multiple addresses[END_REF], DIBS [START_REF] Zarifis | DIBS: Just-in-time congestion mitigation for data centers[END_REF] and CONGA [START_REF] Alizadeh | CONGA: Distributed Congestion-aware Load Balancing for Datacenters[END_REF]. Planck uses a controller that monitors traffic and generates congestion events that include the trans- A new flow is assigned to any of the available paths using some fixed criteria such as by hashing parts of its packets' header. This approach is simple but inflexible. For example, in case two throughput oriented flows are assigned to the same path, they cannot be moved to other less utilized paths later.

Dynamic (reactive)

Flows can be moved across any of the available paths according to available bandwidth. Offers a better performance in general but adds the complexity of measuring link utilizations, accounting for flows, and calculating best flow assignments accordingly.

Dynamic (proactive)

After a flow is assigned to one of the available paths according to some criteria, its assignment will remain fixed. The initial assignment is performed according to network conditions such as available bandwidth. This approach is somewhat between the previous two assignments above in terms of implementation overhead, flexibility and performance.

Granularity per packet

Finest load balancing but can lead to high reordering.

per flow Coarse load balancing but achieves minimal reordering.

per flowlet A flowlet's size dynamically changes according to differences of latencies of candidate paths. At high rates and/or high latency difference between available paths, a flowlet can become significantly large. As a result, this can result in both fine and coarse grained load balancing (it is always somewhere between per packet and per flow). Flowlets have been found effective for load balancing over asymmetric (i.e., with different available bandwidth) paths [START_REF] Vanini | Let It Flow: Resilient Asymmetric Load Balancing with Flowlet Switching[END_REF]. As a drawback, flowlets may cause reordering of small flows and hurt their completion times.

per flowcell A flowcell has a fixed size that is usually about tens of packets. Using flowcells simplifies load balancing compared to flowlets (no need to carefully measure path latencies and schedule accordingly) and reduces possible reordering of small flows. It can however significantly increase reordering for larger flows that will be broken into many flowcells.

mission rate of flows passing through the congested link. It then routes traffic away from congested spots. Hedera initially places flows via hashing, and then uses a central controller to monitor the network, detect long running flows and reschedule such flows on a lightly loaded path to balance the load. MPTCP establishes multiple sub-flows from the beginning and then shifts load between them according to the level of congestion observed across each sub-flow. DIBS forwards packets that arrive at a full queue to one of the nearby switches instead of dropping them which will be forwarded towards the destination through another path. CONGA proposes a technique for leaf-spine topologies [START_REF] Alizadeh | On the data path performance of leafspine datacenter fabrics[END_REF] based on lazy evaluation.

A leaf switch has a table which holds the load seen along its outgoing paths. Such load information is collected by receiving switches and then piggybacked on traffic. Some proactive adaptive approaches include DeTail [START_REF] Zats | DeTail: reducing the flow completion time tail in datacenter networks[END_REF], Presto [START_REF] He | Presto: Edge-based load balancing for fast datacenter networks[END_REF] and Expeditus [START_REF] Wang | Expeditus: Congestionaware Load Balancing in Clos Data Center Networks[END_REF]. DeTail uses a per-hop adaptive method and operates in lossless environments with layer 2 flow control [START_REF]Benefits Of SAN/LAN Convergence[END_REF]- [START_REF]Priority Flow Control: Build Reliable Layer 2 Infrastructure[END_REF]. At every hop, packets are forwarded to the egress port with minimal queuing. Presto breaks flows into small units called cells and sends them across all available paths. This implies that small flows are kept intact as long as they fit into one cell. Expeditus dynamically assigns flows to paths in 3-tier Clos topologies. It uses dedicated packet tags to communicate load information across switches for path selection upon arrival of a new flow. For path election, the upstream switch sends a message to its downstream peer expressing congestion at its egress ports. The receiving switch compares the upstream congestion metrics with the ones for its ingress ports choosing the ports that minimize the maximum congestion along the path.

Load balancing can be done per-packet, per-group of packets that belong to the same flow, and per flow. While con-sidering these options, two important performance metrics are packet reordering and distribution of load across the network. Reordering is known to waste server resources and increase latencies [START_REF] Geng | Juggler: a practical reordering resilient network stack for datacenters[END_REF].

per flow load balancing minimizes packet re-ordering. Congested spots might be created in case multiple large flows are assigned to the same links. Even when carefully placing flows, per flow load balancing provides limited benefit if there is a large number of throughput-oriented flows [START_REF] Mahapatra | Load balancing mechanisms in data center networks[END_REF]. To improve performance, one might reroute flows from their initially assigned paths according to network conditions. However, moving flows from their initial paths might still result in re-ordering. For example, Hermes [START_REF] Zhang | Resilient Datacenter Load Balancing in the Wild[END_REF] performs per flow load balancing added that it can perform fast rerouting of flows according to network conditions. It can potentially reroute flows per packet in case conditions change rapidly and continuously. One however needs to consider the stability of such schemes, especially as load increases, since many flows may be interacting in the network (in case of instability, such schemes may keep changing routes, which increases reordering, while not improving load balancing). Flier [START_REF] Kabbani | Flier: Flow-level congestion-aware routing for direct-connect data centers[END_REF] is another flow-level load balancing scheme that reroutes flows according to level of congestion observed (via checking ECN markings) and failures (by paying attention to timeouts).

Per-packet load balancing provides the finest balancing degree but leads to packet re-ordering. For every new packet, one of the available paths can be chosen either randomly, according to usage history, or adaptively based on the distribution of load. Valiant Load Balancing (VLB) [START_REF] Zhang-Shen | Designing a fault-tolerant network using valiant load-balancing[END_REF] can uniformly distribute packets across paths. Packet Spraying [START_REF] Dixit | On the impact of packet spraying in data center networks[END_REF] (a.k.a. packet scatter) uses a Round Robin approach to multiplex packets of a flow across all possible next hops. Packet scatter can greatly balance load at the network core [START_REF] Gao | pHost: Distributed near-optimal datacenter transport over commodity network fabric[END_REF], [139] and reduce latency [START_REF] Alizadeh | pFabric: Minimal Near-optimal Datacenter Transport[END_REF], [START_REF] Zats | FastLane: Agile Drop Notification for Datacenter Networks[END_REF]. Another per-packet approach, DRB [START_REF] Cao | Per-packet load-balanced, low-latency routing for Clos-based data center networks[END_REF], reduces network queuing and buffering required at the receiver by choosing the forwarding paths that avoid clustering of packets between same source-destination pairs. DRILL [START_REF] Ghorbani | DRILL: Micro Load Balancing for Low-latency Data Center Networks[END_REF] determines the forwarding path of every packet of a flow independently by considering per port local queuing at the switches sending an arriving packet to the least occupied queue. It is argued that DRILL's packet reordering is minimal due to similar latencies across all the paths between every source and destination pair due to well-balanced switch queue occupancies.

Another option is to group several packets of a flow and perform load balancing per-group of packets. Packets can be grouped according to their inter-packet delay or their accumulative size. In the former case, all packets of a flow whose inter-packet delay is less than some timeout threshold form a flowlet [START_REF] Sinha | Harnessing TCP's burstiness using flowlet switching[END_REF]. Flowlet scheduling essentially makes use of natural traffic burstiness for load balancing §III-C3. In the latter case, packets are grouped with a limit on total group volume to form flowcells [START_REF] He | Presto: Edge-based load balancing for fast datacenter networks[END_REF], [START_REF] Xu | TinyFlow: Breaking elephants down into mice in data center networks[END_REF]. Each flowlet or flowcell can be sent over a different path to distribute load.

In flowlet scheduling, smaller timeout values allow for finer load balancing while larger values reduce reordering.

To minimize reordering, one can choose the timeout value to be greater than the difference between latencies of paths with minimum and maximum latencies. Flowlets have been found to effectively balance load while incurring minimal reordering in datacenters [START_REF] Alizadeh | CONGA: Distributed Congestion-aware Load Balancing for Datacenters[END_REF], [START_REF] Perry | Flowtune: Flowlet control for datacenter networks[END_REF]. However, dependence of flowlet switching on inter-packet intervals could lead to creation of arbitrarily large flowlets at high rates, which could lead to congestion in case they collide. Another drawback is the possibility that small flows are split into several flowlets which could lead to reordering and increased latency.

In flowcell scheduling, smaller flowcells balance load better while larger ones reduce reordering. Flows shorter than the cell size are guaranteed to be sent on a single path minimizing their latency and reordering. Previous work has used grouping thresholds of tens of kilobytes (10 KB at ToR switches [START_REF] Xu | TinyFlow: Breaking elephants down into mice in data center networks[END_REF] and 64 KB at the hypervisor layer [START_REF] He | Presto: Edge-based load balancing for fast datacenter networks[END_REF]) to effectively spread the load across the network and reduce creation of random hot-spots. As a drawback, this approach may lead to higher reordering for long flows compared to flowlets.

1) Data and Task Placement: Many datacenter applications need to access data stored on multiple servers to perform computations or respond to queries. Therefore, placement of data determines what options are available to access them. Such data could be a value for a key in a distributed keyvalue store or an object in a replicated or erasure coded store. For example, any of the replicas in a replicated store can be accessed or any of the k pieces out of n pieces of data would allow data recovery in an (n, k) erasure coded storage system.

Pieces of data can be distributed across racks and servers (depending on topology) to allow wider load balancing options. For example, Ceph [START_REF] Weil | Ceph: A scalable, high-performance distributed file system[END_REF] uses Selective Replication that distributes copies of the original data across the cluster according to their popularity. Ceph also distributes contents of large directories and ones with lots of writes across many servers to reduce hot-spots. HDFS [START_REF] Shvachko | The hadoop distributed file system[END_REF] allows for similar features but also considers the network topology while distributing replicas. For example, there is better connectivity and usually higher available bandwidth within a rack.

Placement of tasks (execution) could be as important as placement of data. Task schedulers and resource managers can place computation in accordance with placement of data to reduce network usage, contention for network access and queuing [START_REF] Jalaparti | Network-Aware Scheduling for Data-Parallel Jobs: Plan When You Can[END_REF]- [START_REF] Yekkehkhany | GB-PANDAS: Throughput and heavy-traffic optimality analysis for affinity scheduling[END_REF]. A task scheduler may consider the flow scheduling policy of the network (FCFS, SRPT, Fair Sharing, etc.) in addition to placement of data to improve overall task completion times [START_REF] Munir | Network Scheduling Aware Task Placement in Datacenters[END_REF].

2) Routing and Forwarding: Switches forward packets according to Forwarding Information Base (FIB) which contains a set of rules that determine outgoing port(s) for incoming packets. FIB rules can be installed proactively or reactively. Proactive installation may result in a larger number of rules as not all of them may be used at all times while reactive installation of rules may incur setup time overhead. Such rules can be installed either directly or by a routing protocol that calculates the rules and installs them such as BGP, IS-IS or OSPF. In case of routers, FIB usually reflects a subset of Routing Information Base (RIB) which is a table of routes learned or calculated by a router. For load balancing, various forwarding techniques can be used to direct traffic across several paths.

Standard distributed protocols can be used for load balancing. As a Layer 2 solution, VLAN based load balancing puts same machines on several virtual networks allowing traffic to be spread across paths via using different VLAN tags. It however provides limited scalability due to creation of large broadcast domains. Layer 3 routing for large networks with support for load balancing can be used in case multiple next hops are available for a destination. For example, Equal Cost Multipathing (ECMP) statically selects the next hop by hashing packet header fields. For fast convergence, IGP routing protocols can be used such as IS-IS or OSPF. Load balancing using these protocols is challenging since path costs need to be exactly equal and the number of supported equal paths is limited. BGP provides higher flexibility for this purpose [START_REF] Lahiri | Routing design for large scale data centers: BGP is the better IGP[END_REF] and can be customized to converge fast [START_REF] Filsfils | BGP Prefix Independent Convergence (PIC)[END_REF].

Load balancing can be performed using centralized techniques. In Layer 2, scalable forwarding can be built by replacing the default MAC address broadcast and discovery approach with a centralized one [START_REF] Agarwal | Shadow MACs: Scalable label-switching for commodity Ethernet[END_REF], [START_REF] Stephens | PAST: Scalable Ethernet for data centers[END_REF]. A controller can then setup Layer 2 forwarding that accounts for path diversity [START_REF] Stephens | PAST: Scalable Ethernet for data centers[END_REF]. Centrally implemented Layer 3 approaches allow FIBs to be calculated centrally using free routing software stacks such as Quagga [START_REF] Nascimento | QuagFlow: Partnering Quagga with OpenFlow[END_REF], and installed on switches to implement various protocol stacks such as OSPF and IS-IS with ECMP support which provides much higher flexibility [START_REF] Greenberg | VL2: A Scalable and Flexible Data Center Network[END_REF], [START_REF] Niranjan Mysore | PortLand: A Scalable Fault-tolerant Layer 2 Data Center Network Fabric[END_REF], [START_REF] Greenberg | Towards a next generation data center architecture: scalability and commoditization[END_REF], [START_REF] Jain | B4: Experience with a globally-deployed software defined WAN[END_REF]. For example, to help BGP converge faster, a central controller can calculate and update BGP tables in routers to achieve desired forwarding behavior [START_REF] Singh | Jupiter Rising: A decade of Clos topologies and centralized control in Google's datacenter network[END_REF]. Simpler centralized approaches in which new FIB rules are installed directly by a controller upon arrival of a new flow can also be used. Forwarding can be done in a way that distributes load, either by hashing or adaptively selecting least loaded paths. Using this approach, careful consideration of scalability is necessary.

Previous approaches relied mostly on network for routing. An end-host based approach is Source Routing which simplifies the network by moving the forwarding information to packets, and eliminating the need to disseminate updates of forwarding state [START_REF] Jyothi | Towards a flexible data center fabric with source routing[END_REF]. To properly encode paths in packets, end-hosts need to be aware of network topology and paths. In addition, a mechanism is needed to detect and disseminate network status and failure information to the end-hosts. BCube [START_REF] Guo | BCube: a high performance, server-centric network architecture for modular data centers[END_REF] uses probe packets to measure available bandwidth across paths and assign new flows to least loaded ones. PSSR [START_REF] Guo | Secondnet: a data center network virtualization architecture with bandwidth guarantees[END_REF] encodes a list of outgoing port numbers instead of addresses for next hops to decouple addressing from forwarding.

3) Effect of Failures: The scale of datacenter networks has made failures an important concept. Even using high quality and expensive equipment, the probability that some network element fails (e.g. a switch, a port or a link, etc.) can be considerably large at any moment [START_REF] Liu | F10: A Fault-Tolerant Engineered Network[END_REF]. When some network equipment fails, capacity in the network decreases accordingly; however, since datacenter networks are usually connected with large degrees of redundancy, network failures rarely lead to complete inability to reach parts of the network. However, the capacity loss due to failures in some parts of network can lead to complications in load balancing by affecting the effective capacity of different paths, i.e., creating capacity asymmetries. This may not have a significant impact on topologies that are inherently asymmetrical (e.g. JellyFish, Xpander, etc.); however, more basic load balancing techniques, such as ECMP, which are used in symmetric topologies (e.g. Fat-Tree, Leaf-Spine, VL2, etc.), will have a hard time effectively distributing load in case of failures [START_REF] Zhou | WCMP: Weighted cost multipathing for improved fairness in data centers[END_REF], [START_REF] Alizadeh | CONGA: Distributed Congestion-aware Load Balancing for Datacenters[END_REF], [START_REF] Vanini | Let It Flow: Resilient Asymmetric Load Balancing with Flowlet Switching[END_REF]. This is noteworthy considering that many industry datacenters are based on symmetric topologies.

A variety of solutions have been proposed to perform better load balancing in case of capacity asymmetries across paths examples of which include WCMP [START_REF] Zhou | WCMP: Weighted cost multipathing for improved fairness in data centers[END_REF], CONGA [START_REF] Alizadeh | CONGA: Distributed Congestion-aware Load Balancing for Datacenters[END_REF], HULA [START_REF] Katta | HULA: Scalable Load Balancing Using Programmable Data Planes[END_REF], Presto [START_REF] He | Presto: Edge-based load balancing for fast datacenter networks[END_REF], LetFlow [START_REF] Vanini | Let It Flow: Resilient Asymmetric Load Balancing with Flowlet Switching[END_REF], DRILL [START_REF] Ghorbani | DRILL: Micro Load Balancing for Low-latency Data Center Networks[END_REF] and Hermes [START_REF] Zhang | Resilient Datacenter Load Balancing in the Wild[END_REF]. WCMP mitigates asymmetries by extending ECMP and assigning weights to different paths proportional to their capacity referred to as weighted traffic hashing. Using WCMP, weights determine the number of hash entries per outgoing port at the switch which are selected proportional to capacity. CONGA and HULA operate by performing pathwise congestion monitoring and shifting traffic accordingly. As a result, if capacity of a path is reduced due to failures, its congestion metric will increase faster and it will automatically be assigned less traffic. Presto applies a weighted forwarding mechanism in a way similar to WCMP where weights are pushed to the end-host virtual switches. LetFlow is a simple approach where flowlets are used as means to dynamically adapt to varying path capacities. LetFlow relies on natural property of flowlets which allows them to shrink or expand (in size) according to available capacity over paths. DRILL performs load balancing over asymmetric topologies by first breaking them into groups of symmetric components and then performing load balancing on them per switch. Symmetric paths should have equal number of hops and their queues should be shared by same flows at every hop from source to destination. Hermes uses comprehensive sensing at the end-hosts to monitor path conditions and reroute flows affected by failures or congestion caused by asymmetries. Hermes considers monitoring of latency and ECN markings for detection of congestion while looking at frequent timeouts and retransmissions as signs of failures. In addition, to improve visibility into path conditions, Hermes uses active probing by sending small probe packets between end-host pairs periodically.

E. Multipathing

To improve the overall throughput for a single flow and provide better reliability in case of failures, multi-path techniques can be used [START_REF] Ford | TCP extensions for multipath operation with multiple addresses[END_REF], [START_REF] Raiciu | How hard can it be? designing and implementing a deployable multipath TCP[END_REF], [139], [START_REF] Wischik | Design, Implementation and Evaluation of Congestion Control for Multipath TCP[END_REF]. A flow is split into multiple sub-flows each sent over a different path. To receive traffic on multiple paths, the receiver needs to buffer data received from each sub-flow and put them in order. Buffering is proportional to sum of throughput across paths times the latency of the longest path [START_REF] Raiciu | How hard can it be? designing and implementing a deployable multipath TCP[END_REF], [START_REF] Li | The Delayed ACK evolution in MPTCP[END_REF]. Although latencies are quite small in datacenters, link bandwidths can be significantly high. Additional sub-flows can generally increase both memory and CPU utilization [START_REF] Raiciu | How hard can it be? designing and implementing a deployable multipath TCP[END_REF].

Depending on flow sizes, the applications may decide whether to use multiple sub-flows. Overhead of setup and teardown for multiple sub-flows may be considerable for short flows. For long-running background flows, using every bit of bandwidth available through multipathing may improve their total average throughput.

Several examples of multipath transports include MPTCP [139], XMP [START_REF] Cao | Explicit multipath congestion control for data center networks[END_REF] and MMPTCP [START_REF] Kheirkhah | MMPTCP: A multipath transport protocol for data centers[END_REF]. MPTCP leverages ECMP to route sub-flows over various paths and increase total throughput while balancing load across paths by moving load across sub-flows. XMP approximates the solution to an optimization problem that maximizes utilization. It uses RED [START_REF] Floyd | Random early detection gateways for congestion avoidance[END_REF], [START_REF] Jacobson | RED in a Different Light[END_REF] with ECN marking to keep the queue occupancies low. XMP achieves most benefit by using two sub-flows. MMPTCP aims to improve FCT for short flows by employing a two phase approach. First phase uses packet scatter by randomizing source port numbers and routing via ECMP to minimize FCT for short flows and second phase uses MPTCP to maximize throughput for long flows.

Although multipathing is generally helpful in increasing utilization, the benefit it offers is limited under various conditions. Under heavy workloads, multipathing may not improve throughput if most paths are already highly utilized [START_REF] Hong | Finishing flows quickly with preemptive scheduling[END_REF]. In addition, if paths have different characteristics, such as latency or capacity, multipathing may offer marginal increase (or even decrease) in throughput [START_REF] Wischik | Design, Implementation and Evaluation of Congestion Control for Multipath TCP[END_REF]. This may occur in datacenters in case different communication technologies are used across machines, such as a combination of wireless and wired networks or if available paths have varying number of hops or different capacities. Another complication is caused by overlapping paths where a multihomed sender's paths to the receiver actually have common edges [START_REF] Wallace | A Review of Multihoming Issues Using the Stream Control Transmission Protocol[END_REF]. This can occur in datacenters as well in case of link failures which reduce available paths or if sub-flows are mistakenly hashed to the the same links. Finally, multipathing may lead to unfairness if multipath flows share a bottleneck with regular flows [START_REF] Honda | Multipath congestion control for shared bottleneck[END_REF]. Unfairness may also arise when multipath flows with different number of sub-flows compete for bandwidth over a bottleneck.

F. Scheduling

Given a list of flows with their priorities and demands, the scheduling problem aims to optimize an utility function of several performance metrics such as utilization, fairness or latency. The objective is usually to maximize throughput for bandwidth-hungry flows and minimize FCT for latencysensitive flows considering fairness among flows in each class of service. Different scheduling techniques can be used to reduce FCT, provide better bandwidth guarantees to longrunning flows, and help deadline flows meet their deadlines. In general, scheduling a mix of flow types requires formulation of a complex optimization problem that is generally computationally expensive to solve. Table X offers an overview of scheduling techniques presented here.

1) Reservation: To provide better bandwidth guarantees and prevent creation of congestion spots resources may be first checked for availability and then allocated before an endhost can start transmitting a flow. Requesting resources can be done in different units and such requests might be processed centrally or in a distributed fashion.

In a fully centralized approach, end-hosts can report their demands to a central scheduler and ask for transmission slots or rates. Some examples are TDMA [START_REF] Vattikonda | Practical TDMA for datacenter Ethernet[END_REF], FastPass [START_REF] Perry | Fastpass: A Centralized "Zero-queue" Datacenter Network[END_REF], FlowTune [START_REF] Perry | Flowtune: Flowlet control for datacenter networks[END_REF] and TAPS [START_REF] Liu | TAPS: Software Defined Task-Level Deadline-Aware Preemptive Flow Scheduling in Data Centers[END_REF]. TDMA uses a coarse-grained centralized approach in which end-hosts send their demands to a fabric manager which then allocates them contention-less transmission slots. The scheduling is performed in a round by round basis where each round is several slots during which different hosts can communicate over the fabrics. FastPass allocates slices of time on a perpacket basis to improve utilization and considers variations in packet size. FlowTune performs centralized rate allocation on a per-flowlet basis. TAPS uses a central SDN controller (please refer to §VII-A) to receive and process flow requests with known demands, verify whether deadlines can be met on a per-task basis, allocate contention-less slices of time for senders to transmit and install forwarding rules.

Distributed reservation can be done upon connection setup. A sender can request the rate at which it would like to transmit. This rate along with the available bandwidth is considered by network fabrics to determine the rate and path of the flow. Allocated rate can be piggybacked on ACKs. RCP [START_REF] Dukkipati | Why flow-completion time is the right metric for congestion control[END_REF] and PDQ [START_REF] Hong | Finishing flows quickly with preemptive scheduling[END_REF] perform the rate allocation at switches with this approach. Receiver-based reservation can also be used. Receivers view multiple incoming flows and determine their transmission schedule. Senders can also communicate their demands to the receiver which calculates the transmission schedule [START_REF] Tseng | A Cross-Layer Flow Schedule with Dynamical Grouping for Avoiding TCP Incast Problem in Data Center Networks[END_REF]. In addition, token-based techniques can be used where a receiver sends back tokens to allow senders to transmit a unit of data (e.g. packet) per token. The token-based approach has been found very effective in addressing the incast problem which is achieved by controlling the rate at which tokens are sent back to senders from the receiver to ensure that data arriving at the receiver conforms with available capacity [START_REF] Gao | pHost: Distributed near-optimal datacenter transport over commodity network fabric[END_REF], [START_REF] Handley | Re-architecting Datacenter Networks and Stacks for Low Latency and High Performance[END_REF], [START_REF] Cho | Credit-Scheduled Delay-Bounded Congestion Control for Datacenters[END_REF].

2) Redundancy: Tail latency is an important quality metric for many latency-sensitive applications, such as search, as it determines quality of user experience. Late flows can take much longer than median latency to finish [START_REF] Alizadeh | Data Center TCP (DCTCP)[END_REF], [START_REF] Jalaparti | Speeding Up Distributed Request-response Workflows[END_REF]. An effective approach is to replicate flows, use the fastest responding replica and terminate the rest. For simplicity, creation of replicates may be performed completely at the application layer. The probability of more than one replica being late is usually significantly small. Replicated flows can be scheduled on different paths and can even target different servers to reduce correlation.

One approach is to replicate every flow and then take the one whose handshaking is finished earlier and terminate the rest [START_REF] Liu | Repflow on node.js: Cutting tail latency in data center networks at the applications layer[END_REF]. Another approach is to only replicate slow requests by reissuing them [START_REF] Jalaparti | Speeding Up Distributed Request-response Workflows[END_REF]. It is necessary to judiciously decide on the number of redundant requests to balance resource usage and response time. Since only a tiny portion of all flows are usually laggards, additional resources needed to allow large improvements may be small [START_REF] Jalaparti | Speeding Up Distributed Request-response Workflows[END_REF].

3) Deadline-Awareness: For many applications, criticality of flows can be captured as deadlines. Scheduling techniques are expected to minimize deadline miss rate. In case of hard deadlines, in which case delivery after deadline is pointless, flows can be terminated early if their deadlines cannot be met. D2TCP [START_REF] Vamanan | Deadline-aware Datacenter TCP (D2TCP)[END_REF], D3 [START_REF] Wilson | Better Never Than Late: Meeting Deadlines in Datacenter Networks[END_REF], PDQ [START_REF] Hong | Finishing flows quickly with preemptive scheduling[END_REF], and MCP [START_REF] Chen | Towards minimaldelay deadline-driven data center TCP[END_REF] are examples of deadline-aware scheduling schemes that reduce deadline miss rate by performing rate control according to flow deadlines, either by explicit rate allocation (D3, PDQ) or implicitly by adapting sender's outstanding window size (D2TCP, MCP). Tempus [START_REF] Kandula | Calendaring for wide area networks[END_REF] formulates a complex optimization problem to maximize the fraction of flows completed prior to their deadlines while considering fairness among flows. Amoeba [START_REF] Zhang | Guaranteeing deadlines for inter-datacenter transfers[END_REF] aims to guarantee deadlines by performing initial admission control via formulating an optimization problem. RCD [START_REF] Noormohammadpour | RCD: Rapid Close to Deadline Scheduling for datacenter networks[END_REF] and DCRoute [START_REF] Noormohammadpour | DCRoute: Speeding up Inter-Datacenter Traffic Allocation while Guaranteeing Deadlines[END_REF] aim to quickly determine whether deadlines can be met by performing close to deadline scheduling and guarantee deadlines via bandwidth reservation. In addition, considering task dependencies in meeting deadlines can help reduce deadline miss rate of tasks [START_REF] Liu | TAPS: Software Defined Task-Level Deadline-Aware Preemptive Flow Scheduling in Data Centers[END_REF], [START_REF] Chowdhury | Efficient Coflow Scheduling with Varys[END_REF].

While it is important to meet deadlines, finishing deadline flows earlier than necessary can hurt the FCT of latencysensitive traffic [START_REF] Chen | Scheduling Mix-flows in Commodity Datacenters with Karuna[END_REF]. In general, we can form an optimization problem for scheduling flows [START_REF] Chen | Towards minimaldelay deadline-driven data center TCP[END_REF]. A study of how wellknown scheduling policies perform under mix flow scenarios with varying fraction of deadline traffic can be found in [START_REF] Noormohammadpour | Comparison of Flow Scheduling Policies for Mix of Regular and Deadline Traffic in Datacenter Environments[END_REF].

4) Disciplines:

The following are some well-known scheduling disciplines. First Come First Serve (FCFS) is a simple policy where a task has to be completed before the next task can begin. FCFS provides bounded lateness when scheduling flows with deadlines [START_REF] Leontyev | Tardiness Bounds for FIFO Scheduling on Multiprocessors[END_REF] and is close to optimal for minimizing tail completion times given lighttailed flow size distributions [START_REF] Wierman | Is Tail-Optimal Scheduling Possible?[END_REF]. Processor Sharing (PS) divides available resources equally among flows by giving them access to resources in tiny time scales. Fair Queuing (FQ) approximates PS within transmission time of a packet and can be used to enforce max-min fairness. Earliest Deadline First (EDF) minimizes deadline miss rate for deadline flows. Shortest Job First (SJF) minimizes mean flow completion time in offline systems where all flows and their demands are known apriori. For online systems where requests can be submitted at any time, Shortest Remaining Processing Time (SRPT)

Scheme Description Limitations

Reservation §V-F1 Reserving bandwidth prior to a sender's transmission minimizes congestion and queuing latency.

Reservation adds the latency overhead of calculating a transmission schedule before a flow is allowed to transmit. In addition, it is challenging to enforce a transmission schedule network wide. Inaccuracies in transmission rates may necessitate continues updates to the schedule which is costly.

Redundancy §V-F2 A flow can be replicated multiple times to reduce the effect of high tail latency. The fastest reply is obtained and then replicas are terminated.

Effective if there is a considerable gap between tail and median latency. In addition, it is less effective when network is heavily loaded.

Deadline-Awareness §V-F3

Scheduling can be done according to deadlines to minimize deadline miss rate and lateness.

It may not be possible to meet all the deadlines in which case it should be determined whether deadline miss rate is more important (hard deadlines) or lateness (soft deadlines).

In presence of deadlines, it is unclear how to effectively schedule traffic if flow sizes are not known apriori or cannot be estimated.

Disciplines §V-F4

A variety of scheduling disciplines can be applied according to desired traffic control objectives. For example, SRPT minimizes mean latency while Fair Queuing maximizes fairness.

Disciplines can usually optimize for only one performance metric. A mix of scheduling policies can hierarchically optimize for more than one objective. If a utility of objectives is desired, well-known policies may provide solutions far from optimal.

Preemption §V-F5 Allows the network to update current schedule (along with already scheduled flows) according to new flow arrivals.

Preemption may offer limited benefit if all flows have similar properties (i.e., size, deadline, etc.).

Jittering §V-F6 Prevents a sender from initiating many flows together. For example, this helps mitigate the incast problem §III-C6.

This approach only offers very coarse grained control over incoming traffic.

ACK Control §V-F7 By carefully controlling when ACKs are sent to senders, a receiver can control the incoming flow of traffic.

This approach only offers coarse grained control over incoming traffic.

which is preemptive, minimizes mean FCT [START_REF] Schrage | A proof of the optimality of the shortest remaining processing time discipline[END_REF]. SRPT also offers close to optimal tail completion times while scheduling flows with heavy-tailed size distributions [START_REF] Wierman | Is Tail-Optimal Scheduling Possible?[END_REF]. Least Attained Service (LAS) [START_REF] Rai | Performance Analysis of LAS-based Scheduling Disciplines in a Packet Switched Network[END_REF], which prioritizes less demanding flows, can be used to approximate SJF without apriori knowledge of flow demands [START_REF] Kalyanasundaram | Minimizing flow time nonclairvoyantly[END_REF].

Many works use policies that approximate or implement well-known scheduling disciplines. RCP [START_REF] Dukkipati | Why flow-completion time is the right metric for congestion control[END_REF] performs explicit rate control to enforce processor sharing across flows sharing the same links. PDQ [START_REF] Hong | Finishing flows quickly with preemptive scheduling[END_REF] combines EDF and SJF giving higher priority to EDF to first minimize deadline miss rate and then minimize FCT as much as possible. FastPass [START_REF] Perry | Fastpass: A Centralized "Zero-queue" Datacenter Network[END_REF], implements least recently allocated first giving each user at least as much as their fair share to reach global user-level maxmin fairness. Also, across flows of a user, fewest remaining MTUs first is used to minimize FCT by emulating SJF. pFabric [START_REF] Alizadeh | pFabric: Minimal Near-optimal Datacenter Transport[END_REF] and SFS [START_REF] Zhang | Minimizing datacenter flow completion times with server-based flow scheduling[END_REF] follow the shortest remaining flow first which is essentially SRPT. PIAS [START_REF] Bai | PIAS: Practical information-agnostic flow scheduling for data center networks[END_REF] uses a dynamic priority assignment approach which approximates LAS by counting the number of packets sent by flows so far. RACS [START_REF] Munir | On achieving low latency in data centers[END_REF] uses weighted processor sharing with configurable weights to approximate a spectrum of scheduling disciplines such as SRPT and LAS.

Aside from research, one may be interested in what disciplines can be enforced using industry solutions. Switches by default provide support for FIFO queues per outgoing port which enforce FCFS scheduling policy. Some switches provide support for multiple levels of priority at the outgoing ports (multiple FIFO queues with different priorities). Using these queues, it is possible to mimic the SRPT scheduling policy by putting smaller flows into higher priority queues. For example, Cisco offers switches that support this feature using dynamic packet prioritization (DPP) [START_REF]Cisco Application Centric Infrastructure Fundamentals[END_REF] which tracks flows as their packets arrive (estimating a flow's size according to LAS policy) and assigns them to priority queues according to their sizes. Weighted Fair Queuing (WFQ) is also supported by many switch vendors per "class of service" or per flow where weights determine the proportional importance of flows, i.e., a flow is assigned bandwidth proportional to its weight.

5) Preemption: Many practical systems have to address arrival of requests in an online manner where requests can arrive at any time and have to be addressed upon arrival. Order of arrivals can impact the performance of scheduling algorithms due to race conditions which can lead to priority inversion. For example, in an online scenario with the objective of minimizing mean FCT, SJF might perform poorly if many short flows arrive shortly after a large flow. Preemptive scheduling policies (SRPT in this case) can be used to address this problem [START_REF] Hong | Finishing flows quickly with preemptive scheduling[END_REF].

6) Jittering: A server issuing many fetch requests can use jittering to desynchronize arrival of traffic from various flows and reduce peaks in traffic volume. Such peaks can lead to temporary congestion, dropped packets and increased latency. Jittering can be applied by adding random delays at the application layer when initiating multiple requests at the same time [START_REF] Alizadeh | Data Center TCP (DCTCP)[END_REF].

7) ACK Control: ACKs may be used as part of the network traffic scheduling process since they determine how a sender advances its outstanding window of bytes. They can be thought of as permits for the sender to transmit more data. A receiver can stall a sender by intentionally holding back on ACKs or limit sender's rate by delaying them. For example, a receiver can pause ACKs for low priority flows upon arrival of higher priority traffic and can generate ACKs to control which flows are assigned more bandwidth [START_REF] Zhang | Minimizing datacenter flow completion times with server-based flow scheduling[END_REF]. In addition, by reporting a small receiver window in ACKs, a receiver may limit the transmission rate of a sender [START_REF] Wu | ICTCP: Incast congestion control for TCP in data-center networks[END_REF], [START_REF] Hwang | IA-TCP: a rate based incast-avoidance algorithm for TCP in data center networks[END_REF]. This approach has considerable similarities with the token-based transmission control applied in schemes such as pHost [START_REF] Gao | pHost: Distributed near-optimal datacenter transport over commodity network fabric[END_REF], NDP [START_REF] Handley | Re-architecting Datacenter Networks and Stacks for Low Latency and High Performance[END_REF] and ExpressPass [START_REF] Cho | Credit-Scheduled Delay-Bounded Congestion Control for Datacenters[END_REF].

VI. OPEN CHALLENGES

In this section, we point to a few open challenges with regards to traffic control. To find an optimal solution, these problems may be modeled as complex optimization scenarios that are computationally expensive to solve (large number of variables and constraints, presence of integer variables and/or non-linear constraints, complex objective functions) and practically hard to enforce (lack of hardware support, slow response time of software implementations, presence of failures and errors). Current approaches apply a variety of heuristics and simplifying assumptions to come up with solutions that are practical and attractive to industry. In addition, such optimization scenarios may be infeasible due to presence of contradictory constraints meaning it may not be possible to optimize for all objectives given available resources. Therefore, it becomes necessary to relax some requirements. In the following, we do not provide any optimization models, rather we point to cases where such complex models may appear. Table XI offers an overview of open challenges in this section.

Handling Mix Workloads: Datacenter environments are usually shared by a variety of applications that generate different network workloads with different performance metrics. For example, two applications of search and backup may be sharing the same network; while search network traffic requires low communication latency, backup network traffic demands high throughput. Some network workloads may have deadlines, either soft or hard, which should be considered along with non-deadline network traffic. Effectively scheduling such mix of network workload is an open problem. As a solution, one can formulate a complex optimization problem for such scheduling that considers maximizing some utility function. This utility could be a function of performance metric variables such as deadline miss rate, average lateness, link utilizations, and latency which represents the value achieved by tenants and operators. For example, one can examine how the objective presented in [START_REF] Chen | Scheduling Mix-flows in Commodity Datacenters with Karuna[END_REF], i.e., to reduce per packet latency, translates into utility.

Load Balancing vs. Packet Reordering: In datacenters, there is usually a large number of paths between any two end-hosts which according to topology, could have equal or unequal lengths (number of hops and latency). To use all available paths, one could apply load balancing techniques. Per packet load balancing can provide the finest level of balancing but in general leads to significant packet reordering. In contrast, although per flow load balancing does not lead to packet reordering, it may not be effective in using available bandwidth over many parallel paths. In general, effective load balancing necessitates scheduling of packets over available paths according to available bandwidth. One could consider an additional constraint that allows sending packets on a path only if such scheduling does not lead to out of order arrival of packets at the receiver. But this might reduce utilization by decreasing packet transmission opportunities. As a result, effectively utilizing available bandwidth over all paths could be at odds with minimizing packet reordering. One could relax the no-reordering constraints by allowing reordering to some extent. This relationship can be formulated as a utility of reordering and bandwidth utilization maximizing which in general is an open problem. For example, the works discussed in §V-D and §V-E offer several possible approaches.

Achieving High Throughput and Low Latency: There is a strong connection between traffic control and the control theory. A traffic control scheme depends on concepts from control theory in managing flow of traffic across the network: transmission rate of senders is a function of feedback received from the network, and the time it takes from when a sender changes its rate until it receives a feedback of that change constitutes the loop delay. Due to existence of this loop delay, a network has limited responsiveness due to processing and propagation latency as well as queuing delay. The former two factors are usually much smaller in datacenters and queuing delay determines responsiveness. When transmitting at high rates, it is easy for senders to overload the network (which increases queue occupancy and queuing delay) before a feedback is provided to senders to reduce their rate (which takes at least as much as response time of the network) [START_REF] Jose | High Speed Networks Need Proactive Congestion Control[END_REF]. As a result, in distributed traffic control, achieving maximum throughput with minimal latency is an open problem. Using centralized schemes that perform bandwidth reservation prior to allowing senders to transmit is a candidate solution [START_REF] Perry | Fastpass: A Centralized "Zero-queue" Datacenter Network[END_REF]. There is however several trade-offs as discussed in §III.

Objective Mismatch: In §II, we presented a few objectives for traffic control. In practice, even for a single flow type, it

Open Challenge Description

Handling Mix Workloads Datacenter environments house a variety of applications that generate flows with different properties and requirements.

Effectively handling the mix of traffic workload requires clearly defining a utility function of performance metric variables (delay, utilization, deadline miss rate, lateness, fairness) and formulating an optimization problem.

Load Balancing vs. Packet Reordering

To minimize packet reordering, a sender needs to carefully schedule packets over all available paths while paying attention to other traffic workload. Enforcing a no reordering constraint can result in lower network utilization. As a result, to increase utilization while imposing acceptable level of reordering, one can consider a utility function of these factors and formulate an optimization problem solving which provides a desirable transmission schedule.

Achieving High Throughput and Low Latency

In distributed traffic control approaches, a feedback from network is provided to senders to adapt their transmission rate.

While transmitting at high rates, the network may not provide feedbacks fast enough leading to network overload and congestion. Therefore, limited network responsiveness may increase average queuing delay and latency at high throughput.

Objective Mismatch Due to resource constraints, it may not be possible to come up with a transmission schedule where all performance objectives are optimal. It then becomes necessary to define a utility of performance metric variables and aim to maximize utility by formulating an optimization scenario. may not be possible to optimize for all performance objectives since improving one may negatively impact the others. For example, maximizing fairness may be at odds with minimizing average latency as shown in Figure 5: the first approach is unfair but offers minimal FCT compared to second approach which is fair. Next, as shown in Figure 6, flows may have different priorities which determine their relative importance. The first approach only focuses on maximizing average throughput increasing completion time of high priority flow while the second approach minimizes FCT of high priority flow although it reduces average throughput. As another example, as stated in [START_REF] Hong | Finishing flows quickly with preemptive scheduling[END_REF], fairness could be at odds with minimizing deadline miss rate. As a result, it is desirable to define a utility of these objectives and aim to maximize utility. In general, maximizing a utility of objective variables such as latency, utilization, deadline miss rate, lateness, and fairness is an open problem. Current research efforts mostly focus on maximizing performance with regards to one or two of these metrics. For example, the approach presented in [START_REF] Kandula | Calendaring for wide area networks[END_REF] formulates an optimization scenario that considers both fairness and meeting deadlines.

VII. RELATED PARADIGMS

We review a few networking paradigms that have affected the design and operation of datacenter networks. Essentially, networks have become more flexible and controllable giving operators more room for performance optimizations.

A. Programmable Forwarding Planes

Programmable forwarding planes offer significant room for efficient control and management of datacenter networks. They make it possible to develop and implement custom policies and algorithms for network control and management. Forwarding plane can be programmed centrally using a controller that takes into account policies and resource constraints through some interface provided by forwarding elements. This is considered as part of Software Defined Networking (SDN) [START_REF]Software-Defined Networking (SDN) Definition[END_REF]. A comprehensive survey of SDN architecture and applications can be found in [202].

The dominant framework in this realm is OpenFlow [START_REF] Mckeown | OpenFlow: enabling innovation in campus networks[END_REF], [203] where forwarding elements, such as switches, can be managed via an open interface. An important benefit of an open interface is that switches built by different vendors can be operated in the same way allowing for cost effective expansion. Rolling out new updates to the network also becomes much easier as only controllers need to be patched. In general, there could be multiple controllers each managing a part of the network while coordinating together which is most applicable to large networks. There is two-way communication between switches and controllers: a controller can register for specific events at the switches and perform changes to the switches' forwarding table by adding new rules, modifying existing rules or removing them.

The forwarding process begins when a packet enters a switch from any port. The switch tries to match the packet to a forwarding rule according to its header fields and will forward it to the correct outgoing port. It is also possible to modify the packet contents (packet rewrite) before forwarding it. If the packet cannot be matched to any rule, it can be sent to the controller (if forwarding table is configured with the right table-miss entry) for further inspection and if necessary the controller will update the forwarding plane accordingly for forwarding of this packet and the rest of packets from the same flow. If there are multiple matches, the highest priority rule will be executed. More complex operations can be executed using Group Tables which allow for forwarding to multiple outgoing ports, selection of the outgoing port according to some hash of the packet (e.g. for load balancing), and switching to a connected output port for failover. Group Table features have been added since version 1.1 of OpenFlow [204] and currently version 1.5 has been released [START_REF]OpenFlow Switch Specification Version 1.5.0[END_REF].

By centrally managing forwarding rules, one can implement different routing protocol stacks in a centralized manner. For example, BGP protocol stack can be deployed on top of SDN [206]. In addition, one can implement a more sophisticated control plane protocol that understands and communicates with a variety of other protocols, such as legacy BGP routers, while running a custom protocol stack itself [START_REF] Singh | Jupiter Rising: A decade of Clos topologies and centralized control in Google's datacenter network[END_REF].

B. Programmable Data Planes

Generally, data plane operations are implemented using Application-Specific Integrated Circuits (ASICs) at the hardware layer allowing for forwarding at maximum rate but only offering a set of fixed switch functions that can only be changed by replacing the ASICs. This introduces a few issues namely being prone to bugs as well as long and unpredictable time to implement new functions [207], [208]. Programmable data planes (PDPs) allow the packet processing functions to be changed at the forwarding devices, i.e., switches can apply new forwarding functions at the line rate. This is orthogonal to programmable forwarding planes (e.g., SDN) where different forwarding rules can be selectively applied to packets. PDPs make it easier to deploy traffic control schemes that depend on custom in-network processing or feedback. For example, [START_REF] Alizadeh | Less is more: trading a little bandwidth for ultra-low latency in the data center[END_REF], [START_REF] Alizadeh | pFabric: Minimal Near-optimal Datacenter Transport[END_REF], [START_REF] Hong | Finishing flows quickly with preemptive scheduling[END_REF] rely on custom packet headers and feedback from switches.

PDPs can be realized using Protocol-Independent Switch Architecture (PISA) using which new features can be introduced to switches or bug fixes can be applied in a short time [START_REF]Barefoot Worlds Fastest and Most Programmable Networks[END_REF]. There are emerging proposals for hardware designs that allow for PISA [START_REF] Bosshart | Forwarding Metamorphosis: Fast Programmable Match-action Processing in Hardware for SDN[END_REF], [START_REF] Sivaraman | Packet Transactions: High-Level Programming for Line-Rate Switches[END_REF]. Hardware prototypes (switch chips) have also been built that have made this possible [START_REF]Barefoot Worlds Fastest and Most Programmable Networks[END_REF]. P4 [START_REF] Bosshart | P4: Programming protocol-independent packet processors[END_REF], [213] is a high level language to program PISA switches which is vendor independent, and protocol independent (i.e., operates directly on header bits and can be configured to work with any higher layer protocol). P4 compiler can also compile P4 code to run on a general purpose processor as software switches.

C. Advanced NICs

NICs have been providing basic offloading features to OSes, such as segmentation offloading, for many years. Several vendors have been developing NICs with advanced offloading features that perform complex transport tasks and deliver the results without minimal involvement from OS and CPU. These features allow complex operations at high line rates of datacenters (40 Gbps and more) doing which at the OS may incur significant CPU overhead and additional communication latency.

Examples of offloading features include cryptography, quality of service, encapsulation, congestion control, storage acceleration, erasure coding, and network policy enforcement. Examples of such NICs include Mellanox ConnectX [214] and Microsoft SmartNIC [START_REF] Firestone | SmartNIC: FPGA Innovation in OCS Servers for Microsoft Azure[END_REF], [START_REF] Greenberg | SDN for the Cloud[END_REF] developed as part of Open Compute Project (OCP) [217]. SmartNIC relies on FPGA accelerators and can be used to apply SDN/Networking policies. It also makes low latency transport possible using Lightweight Transport Layer (LTL) that creates end-to-end transport connection between NICs (FPGA does all processing of segmentation, ordering, and ACKs).

D. Userspace Packet Processing

By default, packets pass through the Operating System networking stack before they are received by applications. When packets arrive at the NIC, interrupts are generated to invoke the Operating System routines that read and process them. Processing of packets is usually done in batches to reduce CPU utilization at high rates, for example by enabling Interrupt Moderation [START_REF]Interrupt Moderation[END_REF].

To improve the packet processing performance (decrease packet processing latency and increase throughput), a different approach would be to bypass Operating System's networking stack and use polling instead of interrupts. This can be realized using kernel bypass modules, such as Netmap [START_REF] Rizzo | Netmap: Memory mapped access to network devices[END_REF], [219], Vector Packet Processing (VPP) [220], [221] and Data Plane Development Kit (DPDK) [222], which have been shown to reduce the number of required cycles to process a packet by up to 20× on average [START_REF] Rizzo | Netmap: Memory mapped access to network devices[END_REF]. These modules allow userspace programs to directly access NIC buffers to read incoming packets or write packets for transmission.

Userspace networking stacks have been developed on top of kernel bypass modules. Sandstorm [223] and mTCP [START_REF] Jeong | mTCP: a highly scalable user-level TCP stack for multicore systems[END_REF], implement TCP in userspace and rely on Netmap and VPP, respectively. SoftNIC [START_REF] Han | SoftNIC: A Software NIC to Augment Hardware[END_REF] is built on top of DPDK and allows developers to program custom NIC features in software. RAMCloud [START_REF] Ousterhout | The RAMCloud storage system[END_REF] distributed key-value store and FastPass [START_REF] Perry | Fastpass: A Centralized "Zero-queue" Datacenter Network[END_REF] make use of kernel bypass and polling to speed up Remote Procedure Calls (RPC). NDP [START_REF] Handley | Re-architecting Datacenter Networks and Stacks for Low Latency and High Performance[END_REF] is a datacenter transport protocol that is designed for ultra-low latency and operates on top of DPDK.

E. Lossless Ethernet and RDMA

TCP has been the dominant transport protocol across datacenters for the majority of applications. Since implemented as part of OS protocol stack, using TCP at high transmission rates can exhaust considerable CPU resources and impose notable amount of communication latency. Remote Direct Memory Access (RDMA) is a transport protocol that allows delivery of data from one machine to another machine without involving the OS networking protocol stack. RDMA operates on the NICs of machines communicating. Compared to TCP, RDMA offers higher bandwidth and lower latency at lower CPU utilization [START_REF] Zhu | Congestion Control for Large-Scale RDMA Deployments[END_REF], [START_REF] Mittal | TIMELY: RTT-based Congestion Control for the Datacenter[END_REF], [START_REF] Dragojevic | FaRM: fast remote memory[END_REF]. RDMA can also be used for seamless offloading of large datasets to nearby machines (as opposed to using pagefiles) [START_REF] Gu | Efficient Memory Disaggregation with Infiniswap[END_REF].

To use RDMA, the underlying network has to be lossless since RDMA does not support recovery from lost data by default. Ethernet which is the favorite choice of transport in datacenters, however, does not support reliability by default. Lossless Ethernet, also known as Converged Enhanced Ethernet (CEE), supports per-hop flow control at Layer 2. Backpressure is used as the mechanism to stop senders in case of a full buffer in the network. PAUSE messages are sent to previous hops, pausing the output ports that are connected to inputs with full buffers, until the ultimate senders receive a PAUSE message. The flow control mechanism used by lossless Ethernet is referred to as Priority Flow Control (PFC) and offers 8 priority levels for various classes of traffic [154], [START_REF]Priority Flow Control: Build Reliable Layer 2 Infrastructure[END_REF].

For Layer 2 networks, RDMA can be deployed using RDMA over Converged Ethernet (RoCE) [229] which is based on lossless Ethernet and works across a single Layer 2 domain. For larger networks that span across Layer 3, RDMA can be deployed on top of IP and UDP using version 2 of RoCE (RoCEv2) [230]. It can also be deployed on top of IP and TCP using iWARP [START_REF] Sharp | RFC 7306: Remote Direct Memory Access (RDMA) Protocol Extensions[END_REF] which implements a full TCP stack on end-host NIC to provide a lossless end to end transport. iWARP does not require a lossless infrastructure and can work on top of usual Ethernet, but is less performant and has limited capabilities compared to RoCE [232].

Using lossless Ethernet can lead to a few performance issues in general. Some hindering issues include Layer 2 Head of Line (HOL) Blocking, unfairness (because Layer 2 has no understanding of upper layer notions such as flows), and deadlocks (due to per-port/class PAUSE feature and possible circular dependency of routes). HOL blocking might occur since pausing happens on a per-port/class basis. Therefore, a flow can overflow a port causing it to be blocked stopping other flows going through that port as well. As a result, it is necessary to prevent formation of full buffers. Furthermore, loss-based congestion control approaches are rendered useless since there is no packet loss in case of full buffers.

Quantized Congestion Notification (QCN) [START_REF] Pan | QCN: Quantized congestion notification an overview[END_REF], which is fully implemented in Layer 2, can be used to reduce PAUSE messages by signaling senders before buffers are full. It sends notifications back to the sender's NIC from switches. Packets need to be tagged with a flow ID at the senders which will be used at the switches when notifications are generated to determine which flows should be slowed down. QCN is limited to boundaries of a single Layer 2 domain and therefore is insufficient for datacenters with large networks.

TCP Bolt [START_REF] Stephens | Practical DCB for improved data center networks[END_REF] and DCQCN [START_REF] Zhu | Congestion Control for Large-Scale RDMA Deployments[END_REF] operate across Layer 3 using RoCEv2. Both of these schemes use DCTCP [START_REF] Alizadeh | Data Center TCP (DCTCP)[END_REF] like ECN marking to reduce buffer occupancy and minimize PAUSE signals. To prevent deadlocks, TCP Bolt creates edge disjoint spanning trees (EDSTs) across the network with different PFC classes to prevent cyclic dependencies as flows are routed in the network. TIMELY [START_REF] Mittal | TIMELY: RTT-based Congestion Control for the Datacenter[END_REF] can also be used on lossless networks which uses a delay-based approach to detect increased buffer occupancy and manages its rate accordingly to reduce it.

VIII. BROADER PERSPECTIVE

Cloud companies and content providers, such as Google [1], Microsoft [START_REF]Azure Cloud Services by Location or Region[END_REF], Facebook [235] and Amazon [3], have built multiple datacenters in different continents and countries. Multiple datacenters offer a variety of benefits for distributed applications with geographically wide range of users such as email, multimedia (e.g., YouTube), social networks (e.g., Facebook, Instagram, and Google Plus) and online storage. These benefits include increased availability and fault-tolerance, global or regional load balancing, reduced latency to customers and reduced global bandwidth usage via caching. For example, to minimize user access latency, data can be placed on local datacenters close to users via replication.

To further improve reliability, load balancing and data availability, large datacenter operators (such as Amazon and Microsoft) operate in two hierarchies of zones and discrete datacenters. Each availability zone is usually made up of a few discrete datacenters that are close enough to communicate with negligible latency (e.g. less than 2 ms for Microsoft Azure, i.e., few tens of miles), while far enough to allow them to operate as distinct failure domains. Datacenters usually have rich connectivity within zones which themselves are connected using long haul fiber optics (usually hundreds of miles) [START_REF] Russinovich | Inside Microsoft Azure datacenter hardware and software architecture[END_REF]. These links are either owned by datacenter operators or leased from a provider with existing backbone infrastructure. These links that connect multiple datacenters within regions and across them are referred to as inter-datacenter networks. Maintaining and operating inter-datacenter networks requires significant capital investment from datacenter companies which makes it imperative to efficiently use them [START_REF] Kandula | Calendaring for wide area networks[END_REF], [START_REF] Jain | B4: Experience with a globally-deployed software defined WAN[END_REF], [START_REF] Hong | Achieving high utilization with software-driven WAN[END_REF].

In general, we can categorize traffic that goes within and across datacenters into traffic that is a result of direct interaction with users and the business internal traffic that is a result of backend data processing or migration. Recent work points to significant increase in the overall business internal traffic (which includes both intra and inter-datacenter traffic) that is growing at a much faster pace than user generated traffic [START_REF] Singh | Jupiter Rising: A decade of Clos topologies and centralized control in Google's datacenter network[END_REF], [START_REF] Jain | B4: Experience with a globally-deployed software defined WAN[END_REF], [START_REF]Building Express Backbone: Facebook's new long-haul network[END_REF]. Such increase not only demands higher interconnection bandwidth across servers within datacenters, but also higher network capacity across datacenters. Over the past decade, significant attention has been given to intra-datacenter networks to improve their performance and efficiency which was the topic of discussion in previous sections. However, similar attention has not been paid to increasing efficiency and performance of connectivity across datacenters.

To communicate across datacenters, many companies purchase bandwidth from ISP networks with present WAN infrastructure and are billed according to some usage criteria. A widely used pricing scheme calculates traffic costs by looking at 95 percentile of network bandwidth usage over some period of time [239]. A variety of research efforts focus on minimizing inter-datacenter traffic transit costs considering a similar pricing scheme by aiming to not increase the peak bandwidth usage or minimally increase it if necessary [240]- [START_REF] Jalaparti | Dynamic Pricing and Traffic Engineering for Timely Inter-Datacenter Transfers[END_REF].

Large datacenter operators take advantage of dedicated inter-datacenter connections over long haul optical networks. Google operates their own backbone network referred to as B4 [START_REF] Govindan | Evolve or Die: High-Availability Design Principles Drawn from Google's Network Infrastructure[END_REF], [START_REF] Jain | B4: Experience with a globally-deployed software defined WAN[END_REF]. Microsoft Global WAN [START_REF]How Microsoft builds its fast and reliable global network[END_REF] connects Microsoft datacenters over their private (dark) fiber network. Facebook has also developed their cross datacenter backbone network referred to as Express Backbone [START_REF]Building Express Backbone: Facebook's new long-haul network[END_REF]. These private dedicated networks offer a unique opportunity for further optimization of network resource management. Considering that all end-points of such networks are managed by one organization, we can improve efficiency by coordinating transmission across such end-points. In addition, despite their huge geographical scale, these networks are usually made up of tens to hundreds of links (that connect distant locations) making such coordination practically feasible. For example, B4 is currently managed centrally by a system called Bandwidth Enforcer [250], Microsoft Global WAN is managed by SWAN [START_REF] Hong | Achieving high utilization with software-driven WAN[END_REF] and Express Backbone is also managed centrally by a traffic engineering controller [START_REF]Building Express Backbone: Facebook's new long-haul network[END_REF].

A. Private Dedicated Inter-datacenter Networks

Bandwidth allocation is an effective approach for global traffic engineering over private dedicated networks [START_REF] Hong | Achieving high utilization with software-driven WAN[END_REF], [250]. To take into account latency overhead of bandwidth allocation, a hybrid approach is usually taken where an aggregate bandwidth is set aside for short latency-sensitive flows (mostly user generated) per link or such flows are assigned strictly higher traffic priority. Large long-running flows which we refer to as transfers will then become the focus of per flow bandwidth allocation. In addition, since the network environment is continuously changing as new transfers arrive or due to varying volume of higher priority traffic (from latency-sensitive flows), a slotted timeline is considered where transmission rates can be updated on a per timeslot basis. Figure 7 shows an example architecture for this purpose which is based on similar concepts as [START_REF] Hong | Achieving high utilization with software-driven WAN[END_REF], [250]. One way ro realize this architecture is using SDN. An inter-datacenter network that can be operated using SDN is sometimes referred to as Software Defined WAN (SDWAN). This allows data transmission and routing to be managed centrally according to transfer properties and requirements as well as network status and topology [START_REF] Zhao | The prospect of inter-data-center optical networks[END_REF].

A central Traffic Engineering Server (TES) calculates transmission rates and routes for submitted transfers as they arrive at the network. Rates are then dispatched to local agents that keep track of local transfers (initiated within the same datacenter) called site brokers. Senders first communicate with local site broker which in turn forwards transfer requests to TES. The local site broker then receives TES's response with the transmission rates and forwards it to the sender. Local site brokers add a level of indirection between senders and TES which can help in several ways. It reduces the request response overhead for TES by maintaining a persistent connection with brokers and possibly aggregating several transfer requests before relaying them to the server, which can be useful for hierarchical bandwidth allocation that is done by locally grouping many transfers and presenting them to TES as one (trading some traffic engineering accuracy for scalability). Site broker may also inform TES of network conditions within datacenters for more careful rate allocation and routing. Finally, site broker may modify TES's response according to varying local network conditions or allow senders to switch to a standby backup TES in case it goes offline. Other tasks in this system setup include the following.

Rate-limiting: We earlier discussed a variety of techniques for rate-limiting in §V-B1. TES can calculate transmission rates on a per transfer basis in which case end-hosts (e.g. virtual machines initiating the transfers) should comply with such rates by limiting rates before traffic is transmitted on the wire. In addition, applications that initiate the transfers can themselves apply rate-limiting by carefully controlling the amount of data handed off to the transport layer. Although simple, this approach demands changes to the applications. TES can also compute rates per groups of transfers which can improve scalability [250]. Rate-limiting can then be applied at some intermediate network element via traffic shaping/policing [START_REF]Comparing Traffic Policing and Traffic Shaping for Bandwidth Limiting[END_REF]. Transfers in a group will then have to use a bandwidth sharing policy among themselves such as Fair Sharing.

Routing: Upon arrival of a transfer, forwarding routes are calculated for it by TES. Such routes are then installed in the network by adding proper forwarding rules to network's switching elements. To reduce setup overhead or save network forwarding state, TES can also reuse existing forwarding rules or aggregate several of them as one if possible. Per transfer, senders may have to attach proper forwarding labels to their packets so that they are correctly forwarded (like a Virtual LAN ID). Such labeling may also be applied transparent to the sender at an intermediate network entity (e.g. hypervisor virtual switches, backbone edge, etc).

B. Research Directions

In this section, we provide an overview of several research directions considering the scenario mentioned above.

1) Inter-datacenter Global Rate-allocation and Routing: The majority of inter-datacenter related research is on global optimization of inter-datacenter networks. Metrics similar to ones we discussed in §II can be considered as objectives. For example, B4 [START_REF] Jain | B4: Experience with a globally-deployed software defined WAN[END_REF] and SWAN [START_REF] Hong | Achieving high utilization with software-driven WAN[END_REF] focus on maximizing utilization, Tempus [START_REF] Kandula | Calendaring for wide area networks[END_REF] aims to meet transfer deadlines and maximizes minimal fraction of transfers that complete prior to deadlines in case there is not enough capacity, Amoeba [START_REF] Zhang | Guaranteeing deadlines for inter-datacenter transfers[END_REF] performs admission control for incoming traffic and only admits new transfers if their deadlines can be guaranteed and DCRoute [START_REF] Noormohammadpour | DCRoute: Speeding up Inter-Datacenter Traffic Allocation while Guaranteeing Deadlines[END_REF] performs fast admission control to guarantee deadlines while minimizing packet reordering. The majority of prior work related to this problem either focus on delay tolerant transfers or aim at meeting deadlines while not considering completion times as a metric. For many long-running data transfer operations, completion times are important in increasing overall utility. For example, faster completion of backup operations may reduce the chance of data loss due to failures or speedy replication of data objects can improve average user's quality of experience. In addition, most prior work formulate complex optimization problems that are computationally expensive and slow especially if need be solved as transfers arrive and can increase scheduling latency. Further research is then necessary in developing global rate computation and routing algorithms that lead to solutions quickly and also consider completion times of transfers.

2) Inter-datacenter One-to-Many Transfers: Many services run across several datacenters close to regional users to offer a better quality of experience by minimizing customer access latency (e.g. CDNs cache objects for local viewers [START_REF] Roy | Inside the Social Network's (Datacenter) Network[END_REF], [240], [START_REF] Lin | EcoFlow: An Economical and Deadline-Driven Inter-Datacenter Video Flow Scheduling System[END_REF], [START_REF] Adhikari | Vivisecting youtube: An active measurement study[END_REF], [START_REF] Meshenberg | Active-Active for Multi-Regional Resiliency[END_REF]). This approach also reduces overall interdatacenter bandwidth consumption by keeping a copy of popular objects close to users. There is also need for propagating application state to multiple locations for synchronization (e.g. search index databases [START_REF] Jain | B4: Experience with a globally-deployed software defined WAN[END_REF]) or making multiple distant data copies for higher reliability and availability [START_REF] Zhao | The prospect of inter-data-center optical networks[END_REF]. All of these lead to data delivery from one datacenter to multiple datacenters referred to as Point to Multipoint (P2MP) transfers [START_REF] Noormohammadpour | DCCast: Efficient Point to Multipoint Transfers Across Datacenters[END_REF]. P2MP data delivery over private dedicated inter-datacenter networks can be considered as a special case of multicasting for which a significant body of work is available including innetwork multicasting [START_REF] Cotton | RFC 5771: IANA guidelines for IPv4 multicast address assignments[END_REF], [START_REF] Quinn | RFC 3170: IP multicast applications: Challenges and solutions[END_REF] and using overlay networks [START_REF] Banerjee | Scalable Application Layer Multicast[END_REF], [START_REF] Chu | A case for end system multicast[END_REF]. However, there is need for coordinated schemes that improve performance by carefully selecting multicast forwarding trees and assigning transmission slots to transfers. One should also note that centralized multicasting solutions proposed for intra-datacenter networks, such as [START_REF] Iyer | Avalanche: Data center multicast using software defined networking[END_REF], [START_REF] Cao | Datacast: A Scalable and Efficient Reliable Group Data Delivery Service for Data Centers[END_REF], may not work for inter-datacenter networks due to significant differences in topology (former is usually structured and regular while latter is not). New objectives can be considered in addition to ones proposed in §II for P2MP transfers, such as maximizing number of receivers per transfer that complete reception in a given period of time.

3) Inter-datacenter Failure-aware Routing: Given their scale, inter-datacenter networks may be exposed to a variety of physical conditions and natural environments. As a result, different inter-datacenter links may have significantly different link failure probabilities (possibly by as much as three orders of magnitude [START_REF] Ghobadi | Optical Layer Failures in a Large Backbone[END_REF]). In addition, inter-datacenter traffic in general is made up of a variety of classes with different quality of service requirements and priorities [START_REF] Govindan | Evolve or Die: High-Availability Design Principles Drawn from Google's Network Infrastructure[END_REF], [START_REF] Roy | Inside the Social Network's (Datacenter) Network[END_REF]. In general, transfers can be rerouted and rescheduled in reaction to failures. This however leads to possible service disruptions which can more seriously affect services that are sensitive to loss or latency. Given unique properties of private dedicated inter-datacenter networks, steps can be taken proactively as new transfers arrive to mitigate the effect of failures on overall utility. Several objectives can be considered per failure event including minimizing number of affected transfers upon failures considering their class of service, minimizing the amount by which latency increases across a variety of services, or maximizing utilization while leaving off spare capacity to perform quick in-network failover rerouting [START_REF] Liu | Traffic Engineering with Forward Fault Correction[END_REF]. The latter could be applied only to a fraction of inter-datacenter traffic (i.e., more important traffic). In addition, one could study these objectives for both one-to-one and one-to-many transfer scenarios (or a mix of them).

IX. CONCLUSIONS

Many online services today depend on datacenter infrastructures to provide high availability and reliability along with scalable compute at minimal costs. Datacenter environments may be shared by many tenants and applications offering different services to their consumers (end-users). Traffic control is a necessary task in datacenter networks to efficiently use the resources and fairly share them.

In this tutorial paper, we reviewed several challenges operators, tenants and applications are faced with in datacenter traffic control. We discussed different elements of traffic control explaining the problems, challenges, proposed solutions and their trade-offs. Despite significant research efforts, the majority of traffic control proposals are in the state of research and far from adoption in the industry considering metrics of complexity, performance and cost altogether. Further research efforts are required to reach solutions that offer higher performance at same or lower costs while maintaining low complexity.

More broadly, at the end of this paper, we also pointed to inter-datacenter communication as an evolving research area that requires further attention. We proposed a centralized architecture that captures the essence of existing solutions and discussed how efficient traffic control can be realized using it. We also pointed to three major active research areas regarding inter-datacenter communication that need further attention from the research community.

Fig. 3 .Fig. 4 .

 34 Fig. 3. High level breakdown of traffic control techniques

Fig. 5 .

 5 Fig. 5. It may not be possible to optimize all performance metrics together (mismatch between fairness and mean FCT)

Fig. 6 .

 6 Fig.6. It may not be possible to optimize all performance metrics together (mismatch between average throughput of transfers and FCT of high priority traffic)

Fig. 7 .

 7 Fig. 7. Central management of private dedicated inter-datacenter networks

TABLE II SUMMARY

 II OF DATACENTER TRAFFIC CONTROL OBJECTIVES

TABLE IV DESCRIPTION

 IV OF RELATIONSHIPS BETWEEN TECHNIQUES AND CHALLENGES IN FIGURE

TABLE V DESCRIPTION

 V OF RELATIONSHIPS BETWEEN TECHNIQUES AND OBJECTIVES IN FIGURE

Short flows are usually considered to be less than 1 MB[START_REF] Alizadeh | Data Center TCP (DCTCP)[END_REF],[START_REF] Wilson | Better Never Than Late: Meeting Deadlines in Datacenter Networks[END_REF],[START_REF] Joy | Improving flow completion time for short flows in datacenter networks[END_REF]

ACKNOWLEDGEMENT

We would like to thank the anonymous reviewers of IEEE Communications Surveys and Tutorials for their valuable comments and suggestions that helped us significantly improve the quality of this paper. The first author would also like to especially thank David Oran, Joshua Gahm, Narayan Venkat and Atif Faheem who provided valuable intellectual support during an internship at Cisco Research Center located in Cambridge, Massachusetts, USA.