David Espinel Sarmiento

Adrien Lebre
email: adrien.lebre@inria.fr

Lucas Nussbaum
email: lucas.nussbaum@loria.fr

Abdelhadi Chari
email: abdelhadi.chari@orange.com

Decentralized SDN Control Plane for a Distributed Cloud-Edge Infrastructure: A Survey

Keywords: IaaS, SDN, virtualization, networking, automation

Today's emerging needs (Internet of Things applications, Network Function Virtualization services, Mobile Edge computing, etc.) are challenging the classic approach of deploying a few large data centers to provide cloud services. A massively distributed Cloud-Edge architecture could better fit these new trends' requirements and constraints by deploying on-demand infrastructure services in Point-of-Presences within backbone networks. In this context, a key feature is establishing connectivity among several resource managers in charge of operating, each one a subset of the infrastructure. After explaining the networking management challenges related to distributed Cloud-Edge infrastructures, this article surveys and analyzes the characteristics and limitations of existing technologies in the Software Defined Network field that could be used to provide the intersite connectivity feature. We also introduce Kubernetes, the new de facto container orchestrator platform, and analyze its use in the proposed context. This survey is concluded by providing a discussion about some research directions in the field of SDN applied to distributed Cloud-Edge infrastructures' management.

I. INTRODUCTION

Internet of Things (IoT) applications, Network Function Virtualization (NFV) services, and Mobile Edge Computing (MEC) [START_REF] Bousselmi | Towards a Massively Distributed IaaS Operating System: Composition and Evaluation of OpenStack[END_REF] require to deploy IaaS services closer to the endusers in order to respect operational requirements. One way to deploy such a distributed cloud infrastructure (DCI) is to extend network points of presence (PoPs) with dedicated servers and to operate them through a cloud-like resource management system [START_REF] Lebre | Revising OpenStack to Operate Fog/Edge Computing Infrastructures[END_REF].

Since building such a DCI resource management system from scratch would be too expensive technically speaking, a few initiatives proposed to build solutions on top of OpenStack (RedHat DCN [START_REF]Deploying Distributed Compute Nodes to Edge Sites[END_REF] or StarlingX [START_REF]StarlingX, a complete cloud infrastructure software stack for the edge[END_REF] to name a few). These proposals are based either on a centralized approach or a federation of independent Virtual Infrastructure Managers (VIMs) 1 . The former lies in operating a DCI as a traditional single data center environment, the key difference being the widearea network (WAN) found between the control and compute nodes. The latter consists in deploying one VIM on each DCI site and federate them through a brokering approach to give the illusion of a single coherent system as promoted by ETSI NFV Management and Orchestration (MANO) framework [START_REF] Sabella | Mobile-Edge Computing architecture: The role of MEC in the Internet of Things[END_REF].

Due to frequent isolation risks of one site from the rest of the infrastructure [START_REF]Cloud Edge Computing: Beyond the Data Center (White Paper)[END_REF], the federated approach presents a significant advantage (each site can continue to operate locally). However, the downside relates to the fact that resource management systems code does not provide any mechanism to deliver inter-site services. In other words, VIMs have not been designed to peer with other instances to establish intersite services but rather in a pretty stand-alone way in order to manage a single deployment.

While major public cloud actors such as Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure already provide a globally deployed infrastructure to provide cloud services (i.e., In 2020, more than 170 global PoPs for Azure [START_REF]Azure Global Network[END_REF], more than 90 global PoPs for Google [START_REF]Google Cloud Locations[END_REF], more than 210 global PoPs for AWS [START_REF] Aws | AWS global infrastructure[END_REF]), they are limited in terms of the actions users can do concerning the management of networking resources. For instance, in the AWS environment, a virtual private cloud (VPC) can only exist at a region scope. Although a VPC spans all the availability zones in that region, subnetworks belonging to that VPC must reside entirely within a single Availability Zone.

Several academic studies investigated how this global vision can be delivered either through a bottom-up or top-down approaches. A bottom-up collaboration aims at revising lowlevel VIM mechanisms to make them collaborative, using, for instance, a shared database between all VIM instances [START_REF] Bousselmi | Towards a Massively Distributed IaaS Operating System: Composition and Evaluation of OpenStack[END_REF], [START_REF] Lebre | Revising OpenStack to Operate Fog/Edge Computing Infrastructures[END_REF], [START_REF] Cherrueau | A POC of OpenStack Keystone over CockroachDB[END_REF]. A top-down design implements the collaboration by interacting only with the VIMs' API leveraging, for instance, a P2P broker [START_REF] Soares | Re-designing Cloud Platforms for Massive Scale using a P2P Architecture[END_REF].

In addition to underlining the inter-site service challenges, these studies enabled us to identify key elements a resource management for DCIs should take into account:

• Scalability: A DCI should not be restricted by design to a certain amount of VIMs. • Resiliency: All parts of a DCI should be able to survive network partitioning issues. In other words, cloud service capabilities should be operational locally when a site is isolated from the rest of the infrastructure. • Abstraction and automation: Configuration and instantiation of inter-site services should be kept as simple as possible to allow the deployment and operation of complex scenarios. The management of the involved implementations must be fully automatic and transparent for the users.

In this article, we propose to extend these studies by focusing on the inter-site networking service, i.e., the capacity to interconnect virtual networking constructions belonging to several independent VIMs. For instance, in an OpenStackbased DCI (i.e., one OpenStack instance by POP), the networking module, Neutron [START_REF] Openstack | Neutron -Openstack Networking Service[END_REF], should be extended to enable the control of both intra-PoP and inter-PoP connectivity, taking into account the aforementioned key elements.

We consider, in particular, the following inter-site networking services [START_REF] Chari | Approaches for on-demand multi-VIM infrastructure services interconnection[END_REF]:

• Layer 2 network extension: being able to have a Layer 2 Virtual Network (VN) that spans several VIMs. This is the ability to plug into the same VN, virtual machines (VMs) deployed in different VIMs. • Routing function: being able to route traffic between a VN A on VIM 1 and a VN B on VIM 2. • Traffic filtering, policy, and QoS : being able to enforce traffic policies and Quality of Service (QoS) rules for traffic between several VIMs. • Service Chaining: Service Function Chaining (SFC) is the ability to specify a different path for traffic in replacement of the one provided by the shortest path first (SPF) routing decisions. A user needs to be able to deploy a service chaining spanning several VIMs, having the possibility to have parts of the service VMs placed in different VIMs.

Technologies such as Software Defined Networking (SDN), which proposes a decoupling among control and data plane [START_REF] Subramanian | Software-Defined Networking (SDN) with OpenStack[END_REF], can be leveraged to provide such operations among VIMs [START_REF]Network Functions Virtualisation (NFV) Ecosystem, Report on SDN Usage in NFV Architectural Framework[END_REF]. This paradigm has been applied, for instance, to provide centralized control of lower infrastructure layers of WANs (e.g., physical links or fabric routers and switches) in several proposals [START_REF] Mechtri | SDN for Inter Cloud Networking[END_REF]- [START_REF] Sanhaji | Congestion-based API for cloud and WAN resource optimization[END_REF], including the well-known Google B4 controller [START_REF] Jain | B4: Experience with a Globally-deployed Software Defined Wan[END_REF] and Espresso [START_REF] Yap | Taking the edge off with espresso: Scale, reliability and programmability for global internet peering[END_REF], and commonly referred nowadays as Software Defined WAN [START_REF] Yang | Software-Defined Wide Area Network (SD-WAN): Architecture, Advances and Opportunities[END_REF]. Similarly, VIMs specialized in the management of WANs, usually called WAN infrastructure managers (WIMs) [START_REF] Etsi | Network Functions Virtualisation (NFV); Management and Orchestration[END_REF], have been proposed as SDN stacks capable of controlling all the links and connectivity among multiple PoPs [START_REF] Marsch | 5G System Design: Architectural and Functional Considerations and Long Term Research[END_REF]- [START_REF] Soenen | Insights from SONATA: Implementing and integrating a microservicebased NFV service platform with a DevOps methodology[END_REF]. Moreover, the literature is rich on the study of SDN aspects such as the controller placement problem (CPP) [START_REF] Das | A Survey on Controller Placement in SDN[END_REF] or the optimization of computational paths [START_REF] Togou | A Hierarchical Distributed Control Plane for Path Computation Scalability in Large Scale Software-Defined Networks[END_REF].

The approach in which this survey is interested is called SDN-based cloud computing [START_REF] Azodolmolky | SDN-based Cloud Computing Networking[END_REF]. As such, SDN-based cloud computing has been applied to implement specific applications at a cloud-scale, such as load balancing [START_REF] Patel | Ananta: Cloud Scale Load Balancing[END_REF], [START_REF] Gandhi | Duet: Cloud scale load balancing with hardware and software[END_REF] or policy enforcement [START_REF] Leng | A lightweight policy enforcement system for resource protection and management in the SDN-based cloud[END_REF] functionalities. Our objective is to study how IaaS networking services can be delivered in a DCI context leveraging as much as possible existing solutions. To the best of our knowledge, this is the first study that addresses this question. Concretely, our contributions are (i) an analysis of the requirements and challenges raised by the connectivity management in a DCI operated by several VIMs taking OpenStack as IaaS management example, (ii) a survey of decentralized SDN solutions (analyzing their design principles and limitations in the context of DCIs), and (iii) a discussion of recent activities around the Kubernetes [START_REF]Kubernetes[END_REF] ecosystem, in particular three projects that target the management of multiple sites. Finally, we conclude this survey by (iv) discussing the research directions in this field.

The rest of this paper is organized as follows. Section II defines SDN technologies in general and introduces the Open-Stack SDN-based cloud computing solution. A review of previous SDN surveys is provided in Section III. Challenges related to DCI are presented and discussed in Section IV. Definitions of selected design principles used for the survey are explained in Section V. Section VI presents properties of the studied SDN solutions. Section VII introduces Kubernetes and its ecosystem. The discussion and future work related to the distribution of the SDN control plane for DCIs are presented in Section VIII. Finally, Section IX concludes and discusses future works.

II. SDN AND VIMS NETWORKING: BACKGROUND

In this section, we present background elements for readers who are not familiar with the context. First, we remind the main concepts around Software-Defined-Network as they are a strong basis for most solutions studied in this survey. Second, we give an overview of how the network is virtualized in OpenStack, the de facto open-source solution to use cloud computing infrastructures.

A. Software-Defined-Network

The Software-Defined-Network paradigm offers the opportunity to program the control of the network and abstract the underlying infrastructure for applications and network services [START_REF] Subramanian | Software-Defined Networking (SDN) with OpenStack[END_REF]. It relies on the control and the data plane abstractions. The former corresponds to the programming and managing of the network (i.e., it controls how the routing logic should work) The latter corresponds to the virtual or physical network infrastructure composed of switches, routers, and other network equipment that are interconnected. These equipment use the rules that have been defined by the control plane to determine how a packet should be processed once it arrives at the device. While the idea of control and data plane separation is present in IETF ForCES Working Group works [START_REF] Yang | Forwarding and Control Element Separation (ForCES) Framework[END_REF] and even earlier with the concept of programmable and active networks [START_REF] Van Der Merwe | The tempest-a practical framework for network programmability[END_REF], [START_REF] Tennenhouse | Toward an active network architecture[END_REF], the work achieved in 2008 around OpenFlow [START_REF] Mckeown | OpenFlow: Enabling innovation in campus networks[END_REF] is considered as the first appearance of Software Defined Networks in modern literature [START_REF] Feamster | The road to sdn: an intellectual history of programmable networks[END_REF]. In this initial proposal, the control plane is managed through a centralized software entity called the SDN controller. To communicate with every forwarding device or lower-level components, the controller uses standardized application program interfaces (APIs) called southbound interfaces. In addition to OpenFlow, the most popular southbound APIs are Cisco's OpFlex ones [START_REF]OpFlex: An Open Policy Protocol White Paper[END_REF].

Controllers also expose a northbound API, allowing communication among the controller and the higher-level components like management solutions for automation and orchestration. A generic SDN architecture with the aforementioned elements is presented in Figure 1. Overall, an SDN controller abstracts the low-level operations for controlling the hardware, allowing easy interaction with the control plane, as developers and users can control and monitor the underlying network infrastructure [START_REF] Medeiros | Applying Software-defined Networks to Cloud Computing[END_REF]. The authors refer to [START_REF] Kreutz | Software-defined networking: A comprehensive survey[END_REF] for fundamentals in a more detailed SDN history. Moreover, SDN principles have been successfully applied at the industry level with controllers such as VMware NSX [START_REF] Vmware | VMware NSX Data Center[END_REF], Juniper Contrail [START_REF]Contrail SDN controller[END_REF], or Nuage [START_REF]Nuage SDN controller[END_REF].

Fig. 1: SDN general architecture One of the possible application domains of the SDN technology lies in the cloud computing paradigm, which exposes to end-users software, applications, or virtualized infrastructures in a simple and easy-to-use way [START_REF] Armbrust | A view of cloud computing[END_REF].

By delivering network programmability, SDN abstractions provide the control and management necessary for cloud computing solutions to expose network resources to end-users. Referred to as SDN-based cloud networking (SDN-clouds or SDN-enabled clouds), this approach enables the configuration and provisioning of virtualized network entities using programmable interfaces in cloud computing infrastructures. It is used, for instance, to assure the multi-tenancy needs and the sharing of network resources among end-users [START_REF] Mechtri | SDN for Inter Cloud Networking[END_REF], [START_REF] Azodolmolky | SDN-based Cloud Computing Networking[END_REF], [START_REF] Azodolmolky | Cloud Computing Networking: Challenges and Opportunities for Innovations[END_REF], [START_REF] Son | A Taxonomy of SDN-enabled Cloud Computing[END_REF]. The Neutron OpenStack service described in the next section is a concrete example of such an SDN-based cloud networking service.

B. SDN-based Cloud networking: The Neutron example

To illustrate how the network resources are managed in a cloud computing infrastructure using SDN technologies, we discuss in this section Neutron, the "Network connectivity as a Service" of OpenStack, the de facto open-source solution to manage and expose Infrastructure-as-a-Service platforms [START_REF] Openstack | OpenStack[END_REF]. Neutron provides on-demand, scalable, and technology-agnostic network services. Neutron is generally used with Nova [START_REF]OpenStack Nova Project[END_REF], the OpenStack project to manage virtualized computational resources, in order to provide VMs with networking capabilities.

1) Neutron architecture:

Neutron is a modular and pluggable platform. The reference Neutron architecture, as shown in Figure 2 is composed of the following elements:

• Representational state transfer (REST) API: Neutron's REST API service exposes the OpenStack Networking API to create and manage network objects and passes tenant's requests to a suite of plug-ins for additional processing. It requires indirect access to a persistent database where related information to network objects is stored. [START_REF]OpenvSwitch[END_REF], LinuxBridges [START_REF]Linux Bridges[END_REF], etc.). Since 2020, Neutron's main ML2 mechanism driver changed from OVS to Open Virtual Network (OVN) [START_REF]Open Virtual Network[END_REF]. OVN is a sub-project in OpenvSwitch that provides native inkernel support for virtual network abstraction and a better control plane separation than an OpenvSwitch. Each abstraction has the same functionality as its physical counterpart: Network is an isolated L2 segment, Subnetwork is a block of IPv4 or IPv6 addresses contained inside a Network, Port is a virtual switch connection point used to attach elements like VMs to a virtual network. More extension network objects can be defined and exposed, extending the Neutron API. Some examples are Router, floatingIPs, L2GWs [START_REF] Openstack | Neutron Networking-L2GW[END_REF], BGP-VPNs [START_REF] Openstack | Neutron BGPVPN Interconnection[END_REF] or Virtual Private Network as a Service (VPNaaS) [START_REF]Neutron VPNaaS[END_REF]. A more complete list of OpenStack networking objects can be found in [START_REF]Networking API v2.0[END_REF]. Agents configure the mechanisms allowing these constructions to communicate among them (i.e., a tunnel between two OpenvSwitches to communicate VMs belonging to the same Subnetwork) or with the Internet (i.e., Network Address Translation (NAT) capabilities normally implemented by a Router to route traffic externally).

3) Multi-instance Neutron: Conceptually speaking, a single instance of the Neutron module could manage all network resources of a DCI. As long as it is possible to communicate at the IP level between all equipment, Neutron can configure the different network devices and deliver the expected network abstractions. Technically speaking, leveraging a single centralized module cannot satisfy DCI properties (i.e., Single Point of Failures (SPOF), network bottleneck, intermittent connectivity between network equipment, etc.) and the deployment of several collaborative instances (in the worst case, one per locations) is necessary. Unfortunately, the software architecture of Neutron does not allow collaborations between multiple instances. For instance, the Neutron database belongs to a single Neutron entity; that Neutron can only manage the resources created at one site. It is not possible for a distant Neutron to have knowledge nor access to the objects present in a database of another instance. Because information of resources is not shared among Neutrons, the notion of a virtual network spanning different VIMs does not exist today in the Neutron database. Further, operations like identifiers assignation, IP, and MAC address generation and allocations, DHCP services, or security group management are also handled internally at each Neutron instance. Such kind of constraints takes out the possibility to manage Neutron resources in a distributed way.

However, some projects propose Neutron's management in a more distributed way, such as the Tricircle project [START_REF]Tricircle Project[END_REF]. Tricircle achieves network automation in a multi-instance deployment of OpenStack using a Central Neutron instance managing a series of Local Neutrons with modified Neutron Core plug-ins. To allow communication with the Central Neutron when querying for an element that is not locally present, Tricircle uses the Tricircle Local Core plug-in, a modified version of the ML2 Core plug-in. Once this Local Core plug-in receives a request, it will internally send the same request to the Central Neutron that will answer using standard REST API calls. With the answer, the resource will be locally created using the same information present in the Central Neutron database, and then, the initial request will be treated by the real Core plug-in. For the Central Neutron, which gathers and maintains data consistency across the infrastructure, Tricircle provides a Tricircle Central plugin that allows the Tricircle to effectively manage and map central created resources and their respective locally created copies, storing this information in the Tricircle Database. Tricircle's major flaw lies in its hierarchical architecture as the Central Neutron becomes a SPOF and a bottleneck.

4) Summary:

Because VIMs such as OpenStack have not been designed to peer with other instances, several projects have investigated how it might be possible to deliver inter-site services. These projects aim at exposing multiple Openstacks as a single entity. Unfortunately, these solutions have been designed around a centralized architecture and face important limitations (scalability, network partitions, etc.).

Thus, how to decentralize the management of multiple clusters is an important question that our community must deal with. We propose to initiate the debate by focusing on the OpenStack Neutron service in the following. However, we underline that our study is valuable more generally since it provides an abstract enough analysis of DCI management's different challenges.

Before analyzing in detail those challenges, we discuss previous works and surveys on SDN technologies. In addition to underlining major studies that might be relevant to better understand SDN concepts, it enables us to position our current work with respect to previous surveys.

III. REVIEWS ON WORK & SURVEYS ON SDN AND SDN-BASED CLOUD COMPUTING

SDN Technology has been a hot topic for the last few years. In that sense, several studies have already been published. In this section, we underline the major ones. First, we discuss papers that discuss SDN technologies in general. Second, we review SDN-based cloud activities. By introducing these studies, we aim to underline that the use of multi-instance SDN technologies in a DCI context has not been analyzed in the literature yet.

In [START_REF] Bannour | Distributed SDN Control: Survey, Taxonomy and Challenges[END_REF] the authors presented a general survey on SDN technologies presenting a taxonomy based on two classifications: physical classification and logical classification. For every classifications, multiple subcategories were presented and explained, and the surveyed SDN solutions were placed according to their architectural analysis. The work finished by presenting a list of open questions such as scalability, reliability, consistency, interoperability, and other challenges such as statistics collection and monitoring.

In [START_REF] Karakus | A survey: Control plane scalability issues and approaches in Software-Defined Networking (SDN)[END_REF], and [START_REF] Yang | Scalability Comparison of SDN Control Plane Architectures Based on Simulations[END_REF], the authors focus on the scalability criteria. More precisely, the work done by Karakus et al. [START_REF] Karakus | A survey: Control plane scalability issues and approaches in Software-Defined Networking (SDN)[END_REF] provided an analysis of the scalability issues of the SDN control plane. The paper surveyed and summarized the SDN control plane scalability's characteristics and taxonomy through two different viewpoints: topology-related and mechanismsrelated. The topology-related analysis presents the relation between the topology of architectures and some scalability issues related to them. The mechanism-related viewpoint describes the relation between different mechanisms (e.g., parallelization optimization) and scalability issues. This work's limitation is that the analysis is done by only considering the throughput measured in established flows per second and the flow setup latency.

In [START_REF] Yang | Scalability Comparison of SDN Control Plane Architectures Based on Simulations[END_REF], Yang et al. provided a scalability comparison among several different types of SDN control plane architectures by doing simulations. To assign a degree of scalability, the authors proposed to measure the flow setup capabilities and the statistics collection. Although comparisons among controller architectures are made in these two articles, there is no analysis nor mention of the DCI context and related challenges.

Among other available studies in traditional SDN technologies, the work presented in [START_REF] Blial | An Overview on SDN Architectures with Multiple Controllers[END_REF], and [START_REF] Oktian | Distributed SDN controller system: A survey on design choice[END_REF] are probably the most interesting ones concerning DCI objectives. In their article [START_REF] Blial | An Overview on SDN Architectures with Multiple Controllers[END_REF], Blial et al. give an overview of SDN architectures composed of multiple controllers. The study focuses on the distribution methods and the communication systems used by several solutions to design and implement SDN solutions able to manage traditional networks. Similarly, the survey [START_REF] Oktian | Distributed SDN controller system: A survey on design choice[END_REF] discusses some design choices of distributed SDN control planes. It delivers an interesting analysis of the fundamental issues found when trying to decentralize an SDN control plane. These cornerstone problems are scalability, failure, consistency, and privacy. The paper analyses pros and cons of several design choices based on the aforementioned issues. While these two studies provide important information for our analysis, they do not address the cloud computing viewpoint as well as the DCI challenges.

In the field of SDN applied specifically to cloud computing, the works of Azodolmolky et al. [START_REF] Azodolmolky | SDN-based Cloud Computing Networking[END_REF], [START_REF] Azodolmolky | Cloud Computing Networking: Challenges and Opportunities for Innovations[END_REF] provide information about the benefits, and potential contributions of SDN technologies applied for the management of cloud computing networking. While these works represent an interesting entry point to analyze SDN-based cloud networking evolution, they mostly analyzed the networking protocols and implementations (e.g., VLAN, VXLAN, etc) that may be used in order to provide networking federation among a few data centers. More recently, Son et al. [START_REF] Son | A Taxonomy of SDN-enabled Cloud Computing[END_REF] presented a taxonomy of SDN-enabled cloud computing works as well as a classification based on their objective (e.g., energy efficiency, performance, virtualization, and security), the method scope (e.g., network-only, and joint network and host), the targeted architecture (e.g., Intradatacenter network (DCN), and Inter-DCN), the application model (e.g., web application, map-reduce, and batch pro-cessing), the resource configuration (e.g., homogeneous, and heterogeneous), and the evaluation method (e.g., simulation, and empirical). Additional metrics, such as data center power optimization, traffic engineering, network virtualization, and security, are also used to distinguish the studied solutions. Finally, the paper provides a gap analysis of several aspects of SDN technologies in cloud computing that have not been investigated yet. Among them, we can cite the question related to the extension of cloud computing concepts to the edge of the network (i.e the DCI we envisioned).

To summarize, prior surveys and works neither analyze the challenges of decentralized virtualized networking management in the context of DCIs nor the characteristics (pros/cons) of SDN solutions that could be used to execute this management between multiple instances of the same VIM. The present survey aims to deliver such a contribution.

IV. DISTRIBUTED NETWORK CONTROL MANAGEMENT CHALLENGES

As discussed in Section II, the control of the network elements of a DCI infrastructure should be performed in a distributed fashion (i.e., with multiple VIM networking services that collaborate together to deliver the same network capabilities across multiple sites). Obviously, decentralizing a controller such as Neutron brings forth new challenges and questions. We choose to divide them into two categories: the ones related to the organization of network information and the ones related to the implementation of the inter-site networking services. These challenges are summarized in Table I. The key words column is used to introduce the name of the challenges that will be used in the rest of the document. Finally, the term VIM refers to the VIM network service in the following.

A. Network information's challenges

Giving the illusion that multiple VIMs behave like a global SDN-based Cloud networking service requires information exchange. However, mitigating as much as possible data communications while being as robust as possible (w.r.t network disconnection or partitioning issues) requires to consider several dimensions as discussed in the following.

1) Identifying how information should be shared (information granularity): The first dimension to consider is the organization of the information related to the network elements. As an example, the provisioning of an IP network between two VIMs will require to share information related to the IPs that have been allocated on each VIM. A first approach may consist of sharing the information between the two VIMs each time an IP is allocated to one resource. This way will prevent possible conflict, but with an overhead in terms of communications (the global knowledge base is updated each time there is a modification). A second approach would be to split the range of IP addresses with the same Classless Inter-Domain Routing (CIDR or network prefix) between the two VIMs at the creation of the network (i.e., each VIM has a subset of the IPs and can allocate them without communicating with other controllers). This way prevents facing IP conflicts even in the case of network partitioning without exchanging Understanding the different structures that are manipulated will enable the definition of different information sharding strategies between multiple VIMs and identify the pros and cons of each of them.

Additionally, other elements related to local domain networking management that may be attached to a virtual network as local router gateways, external gateways, DHCP ports, DNS servers, fixed host routes, or floating IPs may not be likely to be shared with remote sites. In consequence, depending on the inter-site service, the granularity of the shared objects' information needs to be well specified to avoid conflicts among the networking management entities. If, in any case, the joint management of a networking construction is strictly required, the management entities should have the necessary mechanisms to do management coordination in order to provide some kind of data consistency.

2) Sharing networking information on-demand and in an efficient manner (information scope): The second dimension to consider is related to the scope of a request. Networking information should stay as local as possible. For instance, network information, like MAC/IP addresses of ports and identifiers of a network related to one VIM, does not need to be shared with the other VIMs that composed the DCI. Similarly, information related to a Layer 2 network shared between two VIMs as depicted in Figure 3 does not need to be shared with the 3rd VIM. The extension of this Layer 2 network could be done later. That is, only when it will be relevant to extend this network to VIM 3.

Taking into account the scope for each request is critical since sharing information across all VIMs should be avoided due to the heavy synchronization and communication needs. In other words, contacting only the relevant sites for a request will mitigate the network communication overhead and the limitations regarding scalability as well as network disconnections.

Obviously, the information-sharing protocol needs to be fast and reliable to avoid performance penalties that could affect 3) Facing network disconnections (information availability): Each VIM should be able to deliver network services even in case of network partitioning issues. Two situations must be considered: (i) the inter-site network resource (for instance, a Layer 2 network) has been deployed before the network disconnection and (ii) the provisioning of a new intersite network resource. In the first case, the isolation of a VIM (for instance VIM 2 in Figure 4) should not impact the intersite network elements: VIM 2 should still be able to assign IPs to VMs using the "local" part of the inter-site Layer 2 network. Meanwhile, VIM 1 and VIM 3 should continue to manage inter-site traffic from/to the VMs deployed on this same shared Layer 2 network.

In the second case, because the VIM cannot reach other VIMs due to the network partitioning issue, it is impossible to get information that is mandatory to finalize the provisioning process. The first way to address such an issue is to simply revoke such a request. In this case, the information availability challenge is only partially addressed. The second approach is to provide appropriate mechanisms in charge of finalizing the provisioning request only locally (e.g., creating temporary resources). However, such an approach implies integrating mechanisms to recover from a network disconnection. In the aforementioned scenario, the VIM may do the provisioning of a new VM in VIM 2 using an IP address already granted to a VM in VIM 1 or that belongs to another CIDR. Once the network failure is restored, VIM 1 will face issues to forward traffic to VIM 2 either because of the overlapping addresses or because there are two different CIDRs.

To satisfy the availability property, the inter-site connectivity management should be able to address such corner cases.

B. Technological challenges regarding inter-site networking services

Technological challenges are related to the technical issues that could be presented when trying to implement DCI networking services.

1) Standard automatized and distributed interfaces (automatized interfaces): A first challenge is related to the definition of the vertical and horizontal APIs to allow the provisioning of inter-site services from the end-users viewpoint but also to make the communication/collaboration between the different VIMs possible. This means that the interface which faces the user (user-side or north-side as traffic flows in a vertical way) and the interface which faces other networking services (VIMs-side or east-west-side as traffic flows in a horizontal way) have to be smoothly bridged among them. This integration needs to be done in order to provide the necessary user abstraction and the automation of the VIMs communication process. Consequently, this necessitates the specification and development of well-defined north-and eastwest-bound interfaces presenting to the user and to remote instances an abstract enough API with the available networking services and constructions. Thus, how to design efficient APIs for both northbound and east-west-bound communication is another problem to address in the case of inter-site connectivity management tools.

Within the framework of OpenStack, Neutron (see Section II-B) only presents a user-oriented interface to provide local services due to its centralized nature.

The Tricircle project [START_REF]Tricircle Project[END_REF] partially address this interface automation leveraging a hierarchical architecture where an API gateway node is used as an entry point to a geo-distributed set of OpenStack deployments. Neutron deployments are not aware of the existence of other local Neutrons but instead always communicate with the Neutron gateway, which is also the only interface exposed to the user.

2) Support and adaptation of networking technologies (networking technologies): Along with the initial networking information exchanges among VIMs to provide inter-site connectivity (MAC/IP addresses, network identifiers, etc.), the identification of the mechanism to actually do the implementation will be needed. Although there are many existing networking protocols to rely on to do the implementation (VLANs on an interconnection box, BGP-EVPN/IPVPN, VXLAN ids, GRE tunnels, etc.), they will need adaptation in the DCI case. Since the configuration of the networking mechanisms needs to be known by all the participant VIMs in a requested inter-site service, the exchange of additional implementation information will be required among the sites in an automatized way. This automation is required due to the fact that the user should not be aware of how these networking constructions are configured at the low-level implementation. Since a Cloud-Edge infrastructure could scale up to hundreds of sites, manual networking stitch techniques like [START_REF] Openstack | Neutron Networking-L2GW[END_REF] [53] will be simply not enough.

Depending on the implementation, the solution needs to be able to do the reconfiguration of networking services at two different levels:

• At the overlay level which implies the ability to configure virtual forwarding elements like GoBGP instances [START_REF] Osrg | GoBGP[END_REF], OpenvSwitch switches or Linux bridges.

• At the underlay level, which implies the ability to talk or communicate with some physical equipment like the Edge site gateway. As not all physical equipment is OpenFlow-enabled, the possibility to use other protocols may be an advantage when it is necessary to configure heterogeneous components or when internal routes should be exposed to allow traffic forwarding at the data plane level. Additionally, in the context of the challenge described in IV-A3, the mechanisms used for the implementation need to be capable of reconfiguring themselves in order to re-establish the inter-site traffic forwarding.

V. DISTRIBUTED SDN DESIGN PRINCIPLES

In addition to highlight the main challenges that should be addressed by a DCI network service management, our study aims at describing the state of the art of major SDN solutions and the analysis on how each controller may answer the different DCI proposed challenges. This section presents an overview of the major differences we identified between the solutions we have studied.

A. Architecture

The first point that distinguishes one solution from another is the way controllers are interconnected with each other [START_REF] Bannour | Distributed SDN Control: Survey, Taxonomy and Challenges[END_REF]- [START_REF] Oktian | Distributed SDN controller system: A survey on design choice[END_REF]. Figure 5 presents the connection topologies we identified during our analysis. We discuss in the following the pros and cons of each approach.

Centralized: Architecture presenting a single centralized controller with a global view of the system. It is the simplest and easiest architecture to manage, but at the same time, the less scalable/robust one due to the well-known problems of centralized architectures (SPOF, bottlenecks, network partitioning, etc.). Hierarchical: Tree-type architecture composed of several layers of controllers. Most solutions present a two-level tree consisting of local controllers and a "root" controller. As the names indicate, local controllers handle local operations such as intra-site routing. On the opposite, the "root" controller deals with all inter-site operations. While local controllers only have their own local view and are not aware of other local controllers' existence, the root controller should maintain a global knowledge of the infrastructure to communicate with local controllers each time it is mandatory. While this approach tackles the scalability challenge w.r.t. the centralized approach, it only increases the robustness partially as the root controller is still a centralized point. Distributed but logically centralized: Architecture where there is one controller per site, managing both intra and inter-site operations. Each time a controller creates or updates a network resource, it broadcasts all other controllers' modifications. This way enables controllers to maintain an up-to-date copy of the global knowledge, thus acting as a single logical entity. This design stands close to the initial SDN proposition [START_REF] Subramanian | Software-Defined Networking (SDN) with OpenStack[END_REF] as several controllers share global network information to present themselves as one single controller. Fully distributed: Architecture similar to the previous one but without communicating all creations/modifications to other controllers. In this approach, locally-created data remains in the instance where it has been created and shared with other instances only when needed. In such a case, explicit communications between controllers are instantiated in order to exchange technical information to establish, for instance, inter-site services. This way of interconnecting controllers increases the robustness w.r.t network disconnections as a network disconnection, or a node failure only impacts a subpart of the infrastructure. Hybrid: Two-layer architecture mixing the distributed and the hierarchical architectures. The control plane consists of several root controllers at the top layer. Each one of the roots manages multiple local controllers who are in charge of their respective sites. These root controllers are organized in a distributed fashion, gathering global network state information among them. When implementing a DCI network service, it is important to consider two kinds of operations: leaderless vs. leaderbased. Leaderless operations such as creating an only-local network and its sub-networks are "simple" operations that should not lead to network information inconsistencies [START_REF] Oktian | Distributed SDN controller system: A survey on design choice[END_REF] and thus do not require leadership mechanisms. On the opposite, leader-based operations, such as the assignment of an IP in an inter-site network, require a dedicated approach to avoid issues such as IP collisions. For those operations, there should be a leader to take consistent decisions among all controllers. Leaderships can be either given in two ways [START_REF] Li | Designing Fully Distributed Consensus Protocols for Linear Multi-agent Systems with Directed Graphs[END_REF]: in a static manner to a controller (i.e., the root node in a hierarchical approach) or by using consensus protocols. Consensus can be divided in two general types: leaderless consensus (i.e., such as EPAXOS [START_REF] Moraru | There is more consensus in egalitarian parliaments[END_REF] or Alvin [START_REF] Turcu | Be general and don't give up consistency in geo-replicated transactional systems[END_REF]), and leader-based consensus (i.e., such as PAXOS [START_REF] Lamport | The Part-Time Parliament[END_REF] or RAFT [START_REF] Ongaro | In Search of an Understandable Consensus Algorithm[END_REF]).

Leader-based consensus protocols such as the aforementioned ones are used for several tasks such as leader election, group membership, cluster management, service discovery, resource/access management, consistent replication of the master nodes in services, among others [START_REF] Ailijiang | Consensus in the Cloud: Paxos Systems Demystified[END_REF]. Consensus typically involves multiple instances agreeing on values. Moreover, consensus can be reached when any majority of the instances is available; for instance, a cluster of 5 instances can continue to operate even if two nodes fail. However, applying consensus protocols to a distributed SDN controller may present some problems. In RAFT, for instance, network failures can seriously impact the performance of the protocol: in the best case, the partitioning may reduce the normal operation time of the protocol; in the worst case, they render RAFT unable to reach consensus by failing to elect a consistent leader [START_REF] Zhang | When Raft Meets SDN: How to Elect a Leader and Reach Consensus in an Unruly Network[END_REF].

To avoid the limitation imposed by a single leader election, leaderless consensus protocols allow multiple nodes to operate as a leader at-a-time [START_REF] Palmieri | Leaderless consensus: The state of the art[END_REF]. This is achieved by dividing conflicting and non-conflicting operations. Non-conflicting operations can be executed without synchronization, while for the conflicting ones, the nodes proposing the operation assume the leadership. The per-operation-leader then collects the dependencies from remote nodes to compute the order among conflicting operations. However, as the system size gets bigger, leaderless protocols may present scalability problems. In EPAXOS, for instance, as the system size increases, more nodes could propose transactions generating more conflicting operations. As a consequence of this possibility, there is a higher probability of different nodes viewing different dependencies, which can fail to deliver fast decisions.

C. Internal communication protocols

Depending on the selected topology, the communication between controllers occurs either vertically (centralized and hierarchical) or horizontally (distributed). Those communications can be handled through different manners like polling information from other controllers periodically, using a publish/subscribe approach to send notifications automatically, or through explicit communication protocols between controllers.

D. Database management system

As largely discussed in Section IV, storing and sharing the state of the DCI network service would be an important challenge. Surveyed SDN solutions rely either on relational (SQL) or NoSQL databases.

1) SQL Databases: SQL databases are based on the relational data model; they are also known as Relational Database Managing System (RDBMS). In most cases, they use Structured Query Language (SQL) for designing and manipulating data and are normally deployed in a centralized node [START_REF] Binani | Sql vs. nosql vs. newsql-a comparative study[END_REF]. Relational databases can generally be vertically scalable, which means that their performance could be increased using more CPU or RAM. Some SQL databases such as MySQL Cluster [START_REF] Ronstrom | Mysql cluster architecture overview[END_REF] proposes to scale horizontally, generally sharding data over multiple database servers (a.k.a. "shared nothing" architecture).

2) NoSQL Databases: NoSQL database is a general term that gathers several kinds of databases that do not use the relational model. NoSQL databases can be gathered in four main types [START_REF] Khasawneh | Sql, newsql, and nosql databases: A comparative survey[END_REF]: document-based (e.g., MongoDB [START_REF] Banker | MongoDB in action[END_REF]), key-value pairs (e.g., Redis [START_REF] Paksula | Persisting objects in redis key-value database, white paper[END_REF]), graph databases (e.g., Neo4j [START_REF] Webber | A programmatic introduction to neo4j[END_REF]) or wide-column stores (e.g., Apache Cassandra [START_REF] Lakshman | Cassandra -a decentralized structured storage system[END_REF]). This kind of databases are by nature horizontal scalable as the unstructured data scheme allows information to be sharded in different sites, thus allowing different entities to access it simultaneously in a geographically distributed way [START_REF] Davoudian | A survey on nosql stores[END_REF], [START_REF] Stonebraker | Sql databases v. nosql databases[END_REF].

More generally, the database management system would be a key element of a DCI network service. It could be used as the means to share information between controllers, thus eliminating the need for a dedicated communication protocol as discussed in the previous paragraph.

E. SDN interoperability and maturity

The studied SDN controllers should be capable of achieving a good performance in heterogeneous and dynamic network environments. For this reason, the capacity to configure a different kind of equipment and the maturity of the solution will be explained in this section.

1) Network types targeted: The popularity of virtualization technologies leads to the abstraction of the physical network (a.k.a. the underlay network) into multiple virtual ones (a.k.a. overlay networks).

• Underlay network : Is a physical infrastructure that can be deployed in one or several geographical sites. It is composed of a series of active equipment like switches or routers connected among them using Ethernet switching, VLANs, routing functions, among other protocols. Due to the heterogeneity of equipment and protocols, the Underlay network becomes complex and hard to manage, thus affecting the different requirements that must be addressed like scalability, robustness, and high bandwidth. Because of these two levels of complexity, SDN controllers could be designed to deal with both levels or just one. Obviously, the richer the operations offered by controllers, the more difficult it would be to distribute the DCI network service.

2) Supported Southbound protocols: The reference SDN architecture exposes two kinds of interfaces: Northbound and Southbound. Northbound interfaces reference the protocol communication between the SDN controller and applications or higher layer control programs that may be automation or orchestration tools. Southbound interfaces are used to allow the SDN controller to communicate with the network's physical/virtual equipment. OpenFlow [START_REF]OpenFlow Switch Specifications[END_REF] is an industry-standard considered as the de facto southbound interface protocol. It allows entries to be added and removed to the switches and potentially routers' internal flow-table, so forwarding decisions are based on these flows. In addition to OpenFlow, SDN controllers may use other protocols to configure network components like NETCONF, LISP, XMPP, SNMP, OVSDB, BGP, among others [START_REF] Subramanian | Software-Defined Networking (SDN) with OpenStack[END_REF]. The Border Gateway Protocol (BGP), for example, allows different Autonomous Systems (ASes) to exchange routing and reachability information between edge routers.

More generally, as not all physical equipment is OpenFlowenabled, the possibility to use other protocols may be an advantage when it is necessary to configure heterogeneous components or when internal routes should be exposed to allow communication at the data plane level.

3) Readiness Level: The Technological Readiness Level (TRL) scale is an indicator of a particular technology's maturity level. Due to the complexity of the mechanisms we are dealing with in this survey, it is important to consider the TRL of technology in order to mitigate as much as possible development efforts. This measurement provides a common understanding of technology status and allows us to establish the surveyed SDN solutions' status, as not all solutions have the same maturity degree. To this end, the TRL proposed by the European Commission presented in [START_REF]Horizon work programme[END_REF] has been used to classify the different solutions we studied.

4) Additional considerations: OpenStack compatibility: As we also propose to take the example of an OpenStack-based system to explain how SDN solutions could be used in a multi-VIM deployment, the capability to integrate with Neutron is introduced as an illustrative example. Indeed, some SDN controllers may be able to integrate with Neutron to implement networking services or add additional functionalities, consuming the Neutron core API or its extensions. Therefore, having a driver to pass network information to the controller. c

VI. MULTI-CONTROLLER SDN SOLUTIONS

Although it would be valuable, delivering a survey of all SDN controller solutions that have been proposed [START_REF] Sdxcentral | SDN Controller Comparison Part 1: SDN Controller Vendors (SDN Controller Companies)[END_REF], [START_REF] Sdxcentral | SDN Controller Comparison Part 2: Open Source SDN Controllers[END_REF] is beyond the scope of this article. We limited our study to literature major solutions' state of the art and selected the best candidates that may fit the Distributed Cloud Infrastructure we are investigating. For the sake of clarity, we present the solutions we studied into two categories:

• Network-oriented SDN: Solutions designed to provide network programmability to traditional or virtualized network backbones. The controllers gathered in this category have not been designed to provide SDN capabilities for cloud computing networking environments. • Cloud-oriented SDN: Solutions that proposed an SDN way to manage the networking services of cloud computing infrastructures (as explained in Section II-A). While some of the controllers gathered in this category have been initially designed to manage traditional networks, they propose extensions to provide SDN features within the cloud networking services. For each selected proposal, we present a qualitative analysis and summarize their characteristics and whether they address the DCI challenges, respectively in Table II andTable III.

A. Network-oriented (controllers for SDN domains)

In this first part, we give an overview of the seven SDN solutions we investigated, namely DISCO [START_REF] Phemius | DISCO: Distributed Multidomain SDN Controllers[END_REF], D-SDN [START_REF] Santos | Decentralizing SDN's Control Plane[END_REF], Elasticon [START_REF] Dixit | Towards an Elastic Distributed SDN Controller[END_REF], FlowBroker [START_REF] Marconett | FlowBroker: A Software-Defined Network Controller Architecture for Multi-Domain Brokering and Reputation[END_REF], HyperFlow [START_REF] Tootoonchian | HyperFlow: A Distributed Control Plane for OpenFlow[END_REF], Kandoo [START_REF] Yeganeh | Kandoo: A Framework for Efficient and Scalable Offloading of Control Applications[END_REF], and Orion [START_REF] Fu | Orion: A Hybrid Hierarchical Control Plane of Software-Defined Networking for Large-Scale Networks[END_REF]. DISCO DISCO (Distributed SDN Control Plane) relies on the segregation of the infrastructure into distinct groups of elements where each controller is in charge of one group using OpenFlow as Control plane protocol. Each controller has an intra-domain (or intra-group) part that provides local operations like managing virtual switches, and an interdomain (or inter-group) part that manages communication with other DISCO controllers to make reservations, topology state modifications, or monitoring tasks. For the communication between controllers, DISCO relies on an Advanced Message Queuing Protocol (AMQP) message-oriented communication bus where every controller has at the same time an AMQP server and a client. The central component of every controller is the database where all intra-and inter-domain information is stored. We underline that there is no specific information on how the database actually works in the article that presents the DISCO solution.

DISCO can be considered to have a fully distributed design because every local controller stores information of its own SDN domain only and establish inter-domain communication with other controllers to provide end-to-end services only if needed. DISCO controllers do not act as a centralized entity and instead work as independent entities peering among them. It has a leader-less coordination because of its logically distributed condition (each controller is in charge of a subgroup, so there is no possible conflict). DISCO's evaluations have been performed on a proof of concept. For this reason, we assigned a TRL of 3 to DISCO.

Addressing the challenges:

• Information granularity: Addressed -due to the segregation of the infrastructure into distinct groups. • Information scope: Addressed -thanks to its per-group segregation. When an inter-domain forwarding path is requested, DISCO controllers use the communication channel to only contact the relevant sites for the request. Thus, avoiding global information sharing. • Information availability: Addressed -in case of network disconnections, each controller would be able to provide intra-domain forwarding. Besides, controllers that can contact each other could continue to deliver intersite forwarding. Finally, a recovery mode is partially provided, given that disconnected sites only need to contact the remote communication channels to retake the inter-domain forwarding service when the connectivity is reestablished. As aforementioned, we underline that due to its implementation and the information that is manipulated, DISCO is conflict-less. This makes the recovery process rather simple.

D-SDN

D-SDN (Decentralized-SDN) distributes the SDN control into a hierarchy of controllers, i.e., Main Controllers (MCs) and Secondary Controllers (SCs), using OpenFlow as control plane protocol. Similar to DISCO, SDN devices are organized by groups and assigned to one MC. One group is then divided into subgroups managed by one SC (each SC requests one MC to control a subgroup of SDN devices). We underline that the current article does not give sufficient details regarding how states are stored within Main and Secondary Controllers. The paper mainly discusses two protocols. The first one is related to communications between SCs and MCs using D-SDN's MC-SC protocol for control delegation. The second one, entitled D-SDN's SC-SC, has been developed to deal with fail-over 1 Challenge completely addressed. 2 ~Challenge partially addressed. 3 Challenge not addressed. 4 ? Undefined. As stated in the proposition, D-SDN has a hierarchical design: the MC could be seen as a root controller and SCs as local controllers. It has a leader-based coordination, with MC being the natural leader in the hierarchy. As D-SDN is presented as a proof of concept, we defined a TRL of 3.

• Information granularity: Addressed -due to the segregation of the infrastructure elements into distinct groups.

ElastiCon

Elasticon (i.e., elastic controller) is an SDN controller composed of a pool of controllers. The pool can be expanded or shrunk according to the size of the infrastructure to operate. Each controller within the pool is in charge of a subset of the SDN domain using OpenFlow as control plane protocol. The elasticity of the pool varies according to a load window that evolves over time. A centralized module triggers reconfigurations of the pool like migrating switches among controllers or adding/removing a controller based on the computed value.

While decisions are made centrally, it is noteworthy to mention that actions are performed by the controllers. To do so, each controller maintains a TCP channel with every other one creating a full mesh. This protocol enables controllers to coordinate themselves if need be. The states of Elasticon are shared through the Hazelcast distributed data store [START_REF]Hazelcast Project[END_REF], which can be accessed by all controllers. The use of a shared backend by the pool gives, as a result, a physically distributed but logically centralized design. As stated in Elasticon's work, the solution has been implemented as a prototype, and thus a TRL of 3 has been assigned to it.

• Information granularity: Partially addressed -Elasticon has been designed to distribute the control of infrastructure over several controllers. If the Hazelcast data store can be deployed across several sites, it is possible to envision distributing the pool of controllers between the different sites. By accessing the same database, controllers will be able to add information to the database and fetch the others' information to establish inter-site services. However, the consistency of the data store might be another issue to deal with. • Information scope: Undefined -it is linked to the database capabilities (in this case, to the way the Hazelcast data store shards the information across the different sites of the infrastructure). However, it is noteworthy to mention that most advanced database systems such as CockroachDB only favor data-locality across several geodistributed sites partially.

FlowBroker

FlowBroker is a two layers architecture using OpenFlow as a control plane protocol. It is composed of a series of broker agents and semi-autonomous controllers. The broker agents are located at the higher layer. They are in charge of maintaining a global view of the network by collecting SDN domain-specific network state information from the semiautonomous controllers deployed at the bottom layer. Semiautonomous controllers do not communicate among them, so they are not aware of other controllers' existence in the network. These controllers are only aware of interfaces in the controlled switches, thus, providing local-domain forwarding. By maintaining a global view, the broker agents can define how semi-autonomous controllers should establish flows to enable inter-domain path forwarding.

FlowBroker presents a hierarchical design clearly, with broker agents acting as root controllers and semi-autonomous domain controllers as local controllers. Although semiautonomous controllers can establish forwarding paths inside their own domain, communication with the broker agents is mandatory for inter-domain forwarding. Because of this reason, FlowBroker presents a leader-based coordination, where brokers act as leaders. However, we underline that there is not any information describing how the information is shared between the different brokers.

Regarding maturity, we assigned a TRL of 3 to FlowBroker because only a proof-of-concept has been implemented.

Addressing the challenges:

• Information granularity: Not addressed -the segregation into semi-autonomous controllers enables the efficient sharing of the information per site. However, the global view of the information that is maintained by the brokers does not enable the validation of this property. • Information scope: Not addressed -although the global view maintained by each broker allows them to contact only the semi-autonomous controllers that are involved in the inter-service creation, the result of each operation is forwarded to each broker in order to maintain the global view up-to-date. • Information availability: Addressed -as aforementioned, semi-autonomous controllers can continue to provide local-domain forwarding without the need of the brokers.

In the hypothetical case of a network disconnection and the subsequent reconnection, interconnected controllers can still forward the traffic among them. Actually, they only need to contact brokers in order to request the intersite forwarding configuration. Once the configuration of network equipment has been done, controllers do not need to communicate with brokers. Regarding the loss of connectivity with brokers, the recovery process is quite simple because the information shared between all brokers and semi-autonomous controllers is conflict-less.

• Automatized interfaces: Not addressed -because semiautonomous controllers do not have an east-west interface to communicate among them, but only communicate with brokers. Moreover, the way brokers exchange network knowledge to gather global network view is not discussed. • Networking technologies: Not addressed -since its use is only intended with OpenFlow protocol.

HyperFlow

Hyperflow is an SDN NOX-based [START_REF]NOX Network Control Platform[END_REF] multi-controller using OpenFlow as control plane protocol. The publish/subscribe message paradigm is used to allow controllers to share global network state information and is implemented using WheelFS [START_REF] Stribling | Flexible, Wide-Area Storage for Distributed Systems with WheelFS[END_REF]. Each controller subscribes to three channels: data channel, control channel, and its own channel. Events of local network domains that may be of general interest are published in the data channel. In this way, information propagates to all controllers allowing them to build the global view. Controller to controller communication is possible by publishing into the target's channel. Every controller publishes a heartbeat in the control channel to notify about its presence on the network.

As global networking information is shared by all participant controllers, the controller topology presents a physically distributed but logically centralized design. Every controller manages its own domain. In the case of network partitions, traffic forwarding can continue inside each controller domain and between the controllers that can contact each other. However, the dependency with respect to WheelFS is not discussed. In other words, the behavior of a controller that cannot contact WheelFS is undefined. More generally, the publish/subscribe paradigm enables Hyperflow to be leaderless. As this proposition has been implemented as a proof-ofconcept, a TRL of 3 has been assigned to HyperFlow.

Addressing the challenges:

• Information granularity: Addressed -thanks to WheelFS, it is possible to deploy one controller per site. Each one uses WheelsFS to share networking information in order to create inter-domain forwarding paths. • Information scope: Partially addressed -HyperFlow presents both a general information data channel and the possibility to communicate directly to specific controllers using their respective channel. Unfortunately, the paper does not clarify whether the establishment of inter-site forwarding is done by contacting the relevant controllers or if, instead, the general channel is used. In the former case, the exchange is efficient; in the latter, the information will be shared through all controllers.

Kandoo

Kandoo is a muti-controller SDN solution built around a hierarchy of controllers and using OpenFlow as control plane protocol. At the low level, local-domain controllers are in charge of managing a set of SDN switches and processing local traffic demands. At the high-level, the single root controller gathers network state information to deal with inter-domain traffic among the local domains. The Kandoo proposal authors claim that there are only a few inter-domain forwarding requests and that a single root controller is large enough to deal with. Regarding the local controllers, they do not know about the others' existence, thus only communicating with the root controller using a simple message channel to request the establishment of inter-domain flows. Unfortunately, the authors did not give sufficient information to understand how this channel works and how the inter-domain flows are set up.

By its two-level hierarchical design, Kandoo presents a leader-based coordination (the root controller being the architecture's natural leader). As the solution had been implemented as a proof-of-concept, a TRL of 3 has been assigned to Kandoo.

Addressing the challenges:

• Information granularity: Not addressed -the root controller is used to get information to do inter-domain traffic forwarding, thus gathering the global network view. • Information scope: Not addressed -similarly to the previous challenge, there is no direct communication between controllers: the single root controller is aware of all interdomain requests. • Information availability: Addressed -similarly to Flow-Broker solution, the root controller is only required to configure the inter-domain traffic. Once network equipment has been set up, there is no need to communicate with the root controller. The recovery process between local controllers and the root is simple: it consists of just recontacting the root once the network connectivity reappears (similarly to FlowBroker is conflict-less). • Automatized interfaces: Not addressed -there is not an east-west interface to communicate among local controllers.

• Networking technologies: Not addressed -since Kandoo does not implement other protocols besides OpenFlow.

Orion

Orion is presented as a hybrid SDN proposition using Open-Flow as a control plane protocol. The infrastructure is divided into domains that are then divided into areas. Orion leverages area controllers and domain controllers. Area controllers are in charge of managing a subset of SDN switches and establish intra-area routing. Domain controllers, at the top layer, are in charge of synchronizing global abstracted network information among all domain controllers and to establish inter-area routing paths for their managed area controllers. Synchronization of network states between domain controllers is done using a scalable NoSQL database. Moreover, a publish/subscribe mechanism is used to allow domain controllers to demand the establishment of inter-area flows among them. Finally, it is noteworthy to mention that area controllers are not aware of other area controllers' existence and only communicate with their respective domain controller. This communication is done via a simple TCP channel.

Orion is the only solution that presents a hybrid design: each domain follows a two-level hierarchy, and all domain controllers are arranged in a P2P way, using a NoSQL database to share information between each other. Unfortunately, the paper does not give details regarding the NoSQL database nor the coordination protocol among the domain controllers. Hence, it is not clear whether Orion uses a leader-based coordination in its P2P model. As the solution had been implemented as a proof-of-concept, a TRL of 3 has been assigned to Orion.

Addressing the challenges:

• Information granularity: Not addressed -although the infrastructure is divided into domains (each domain controller maintains its own view of the information), each area controller should notify its domain controller about all changes that occur at the low level. • Information scope: Not addressed -first, area controllers cannot contact directly other controllers to set up intersite forwarding services, and second, we do not know how information is shared between domain controllers (i.e., it is related to the database system, see Elasticon for instance). • Information availability: Undefined -in case of network disconnections, area controllers can continue to forward intra-domain traffic and inter-domain traffic on paths that have been previously established. In other words, domain controllers are used only for inter-domain path forwarding establishments. In the case of network disconnection, the area controller only needs to reconnect to its domain controller when needed and when the connection reappears.

There is no need for a specific recovery protocol because the information shared between area controllers and their respective domain controller is conflict-less. Only the recovery mechanism related to the DB that is used to share information among domain controllers is unclear.

B. Cloud-Oriented

In this second part, we present the five solutions we selected from the cloud computing area, namely DragonFlow [START_REF] Openstack | DragonFlow : Distributed implementation of Neutron within a large DC[END_REF], Onix [START_REF] Koponen | Onix: A Distributed Control Platform for Large-scale Production Networks[END_REF], ONOS [START_REF] Berde | ONOS: Towards an Open, Distributed SDN OS[END_REF], ODL [START_REF] Medved | OpenDaylight: Towards a Model-Driven SDN Controller Architecture[END_REF], and Tungsten [START_REF] Juniper | Contrail Architecture[END_REF].

DragonFlow

DragonFlow is an SDN controller for the OpenStack ecosystem, i.e., it implements the Neutron API and thus can replace the default Neutron implementation (see Section II-B). Drag-onFlow relies on a centralized server (i.e., the Neutron server) and local controllers deployed on each compute node of the infrastructure from the software architecture. Each local controller manages a virtual switch, providing switching, routing, and DHCP capabilities using entirely OpenFlow. A Dragon-Flow ML2 mechanism driver and a DragonFlow service plugin are activated in Neutron Server in order to provide system network information to all local controllers. Communication between the plug-ins at the Neutron server side and local controllers is done via a pluggable distributed database (currently supporting OVSDB [START_REF]The Open vSwitch Database[END_REF], RAMCloud [100], Cassandra [START_REF]Apache Cassandra[END_REF], and using etcd [START_REF]Cloud Native Computing Foundation[END_REF] as default back-end).

Local controllers periodically fetch all information of network state through this database and update virtual switches, routes, etc., accordingly.

By maintaining a global knowledge of the network elements through its distributed database, DragonFlow can be considered as a distributed but logically centralized controller (see Section V-A) at first sight. However, the fact that there is a root controller (i.e., the Neutron server-side) in charge of the management layer (i.e., updating configuration states in the distributed database) and local controllers that implement the control plane makes DragonFlow more a hierarchical solution than a distributed one. In other words, for all leader-based operations, the Neutron plug-in deployed at the server-side acts as the leader. In conclusion, although DragonFlow is presented as a distributed SDN controller, its design does not allow the management of a geo-distributed infrastructure (i.e., composed of multiple SDN controllers).

From the maturity viewpoint and according to its activity, we believe DragonFlow has reached a TLR 6. Initially supported by Huawei, the project is rather inactive right now.

Addressing the challenges:

• Information granularity: Partially addressed -similarly to Elasticon, if the distributed database service can be deployed across several sites, we can envision an infrastructure composed of several DragonFlow Neutron plugin. Each one will add information to the database, and all local controllers will be capable of fetching the necessary information to provide end-to-end services. • Information scope: Undefined -it is linked to the way the distributed database system shards the information across the different sites of the infrastructure. • Information availability: Undefined -similarly to the previous challenge and to the Elasticon solution, it is linked to the way the distributed database services deals with network partitioning issues. • Automatized interfaces: Partially addressed -Dragon-Flow controllers do not present an east-west interface to communicate with remote sites. Instead, the distributed database is used as a communication tool. • Networking technologies: Partially addressed -the controller incorporates the adaptation and reconfiguration of networking services, but it lacks the heterogeneity of networking protocols. Currently, DragonFlow does not support BGP dynamic routing [START_REF] Openstack | DragonFlow : BGP dynamic routing[END_REF].

Onix

Onix is a multi-controller SDN platform. In other words, Onix presents several building blocks to develop network services in charge of operating either overlay (using OpenFlow by default) or underlay (using BGP if needed) networks.

Onix's architecture consists of multiple controller instances that share information through a data store called Network Information Base (NIB). The infrastructure is divided into domains, each domain being managed by one instance. Depending on durability and consistency, a network service may use a specific database to implement the NIB module. If durability and strong consistency are required, a replicated transactional SQL database should be used among the instances. Otherwise, it is possible to use any kind of NoSQL system.

Regarding coordination aspects, the system leverages ZooKeeper [START_REF]ZooKeeper: A Distributed Coordination Service for Distributed Applications[END_REF] to deal with instance failures (using the Zookeeper Atomic Broadcast protocol for leader election). The responsibility of the SDN equipment is then determined among the controllers.

By using multiple controllers, and a global network database, the Onix architecture corresponds to a physically distributed but logically centralized one.

As Onix was built as a basis for Nicira's SDN products but was not really a commercial product, a TRL of 7 was assigned.

Finally, the Onix platform integrates some applications, including the management of multi-tenant virtualized DCs. This service allows the creation of tenant-specific Layer 2 networks establishing tunnels among the hypervisors hosting VMs in one single deployment. However, this module works in a stand-alone mode and does not interact with the OpenStack Neutron service.

Addressing the challenges:

• Information granularity: Partially addressed -similarly to solutions such as Elasticon or DragonFlow, it is related to the database used to share the information between the instances. • Information scope: Undefined -similarly to the previous challenge, it is related to the database. In case of strong consistency, the information should be synchronized across all instances. In the case of a NoSQL system, it depends on how the DB shards the information across different instances. ONOS ONOS (Open Network Operating System) is a modular and distributed SDN framework consisting of several network applications build on top of Apache Karaf OSGi container [START_REF]Apache Karaf[END_REF]. It supports the use of multiple control plane protocols like OpenFlow, NetConf, and others. ONOS has been created for overlay and underlay networks of service providers. Network states' information is stored using the Atomix database [START_REF]Open Networking Foundation[END_REF], a NoSQL framework developed for ONOS, which is also used for coordination tasks among controllers.

Similar to other proposals, the infrastructure is divided into domains with one controller per domain. Considering the shared back-end and the multiple controller instances, ONOS presents a physically distributed but logically centralized design. As aforementioned, ONOS has a leader-based coordination approach, leveraging the Atomix DB (more precisely, it uses the RAFT algorithm). Considering that ONOS is one of the most popular SDN open-source controllers and is used by several key actors in telecommunications [START_REF] Onos | ONOS -Community[END_REF], a TRL of 9 has been assigned to ONOS.

Finally, the modular design of ONOS allows the implementation of the Neutron API. Concretely, there are three applications, which consume Neutron API and provide ML2 drivers and Services plug-ins: SONA (Simplified Overlay Network Architecture), VTN (Virtual Tenant Network), and CORD (Central Office Re-architected as a Datacenter) VTN. Each application has been designed with different targets [START_REF] Onos | SONA architecture ONOS[END_REF], [START_REF] Onos | CORD VTN ONOS[END_REF]. SONA provides an ML2 driver and an L3 service plugin implementation. VTN provides service function chaining capabilities. CORD VTN extends VTN with its own interfaces for switching and routing configuration [START_REF] Onos | ONOS -OpenStack (Neutron) Integration[END_REF].

Addressing the challenges:

• Information granularity: Partially addressed -similarly to previous solutions that are composed around several instances and a global shared database. • Information scope: Undefined -it is linked to the way the Atomix database system shards the information across the different instances. • Information availability: Undefined -similarly to the previous challenge, it is linked to the Atomix system. • Automatized interfaces: Partially addressed -ONOS controllers use the Atomix framework for coordination tasks among controllers and to communicate among them.

• Networking technologies: Addressed -ONOS includes several networking technologies.

OpenDayLight

OpenDayLight (ODL) is a modular SDN platform supporting a wide range of protocols such as OpenFlow, OVSDB, NETCONF, BGP, among others. Originally, OpenDayLight was developed as a centralized controller to merge legacy networks with SDN in data centers, but its modularity allows users to build their own SDN controller to fit specific needs [START_REF] The | OpenDaylight is One of the Best Controllers for OpenStack[END_REF]. The internal controller architecture is composed of three layers: The southbound interface, which enables communication with network devices. The Service Adaptation Layer adapts the southbound plug-in's functions to higherlevel application/service functions. Finally, the northbound interface provides the controller's API to applications or orchestration tools. Network states are stored through a tree structure using a dedicated in-memory data store (i.e., developed for ODL). While the default implementation of ODL can be used in cluster mode for redundancy and high availability, its modularity allows the introduction of features aiming to allow different instances of the controller to peer among them like the SDNi [START_REF] Opendaylight | OpenDaylight SDNi Application[END_REF] or the more recent Federation [START_REF] Opendaylight | OpenDaylight Federation Application[END_REF] projects. ODL Federation service facilitates the exchange of state information between multiple OpenDayLight instances. It relies on AMQP to send and receive messages to/from other instances. A controller could be at the same time producer and consumer.

The Federation project of ODL corresponds to a physical and logical distributed design (each instance maintains its own view of the system). Moreover, it is a leader-less coordination approach because there is a flat on-demand communication between controllers, and no leader is needed for these exchanges.

The modularity of the controller allows multiple projects to implement the Neutron API. For instance, ODL comes with the OpenStack Neutron API application. This application provides the abstractions that are mandatory for the implementation of the Neutron API inside the controller. Among those implementations, we found: Virtual Tenant Network Manager (VTN), Group-Based Policy (GBP), and OVSDBbased Network Virtualization Services (NetVirt) [START_REF] Opendaylight | OpenDaylight NetVirt Application[END_REF].

By leveraging the Federation and NetVirt projects, it is possible to create virtual network resources spreading across several OpenStack instances. When the Federation manager receives a request to create an inter-site service between two OpenStack instances, it realizes the interconnection at the ODL level (i.e., creating shadow elements, etc.) and performs the matching with the OpenStack Neutron resources on the different sites. Although this enables to interconnect multiple instances of OpenStack, it is noteworthy to mention that Neutron instances remain unconscious of the information shared at the ODL level. In other words, there is no coordination mechanism that will prevent overlapping information at the Neutron level. This is rather critical as it may lead to consistency issues where an IP, for instance, can be allocated on each site without triggering any notification.

Since ODL is a community leader and industry-supported framework presented in several industrial deployment and continuous development, a TRL of 9 has been assigned to ODL [115].

Addressing the challenges:

• Information granularity: Addressed -through the Federation project, it is possible to leverage several controllers to operate an infrastructure (each controller maintains its own view). • Information scope: Addressed -each controller can interact with another one by using AMQP. In other words, there is not any information that is shared between controllers unless needed. • Information availability: Partially addressed -in case of network disconnection, ODL instances can satisfy local networking services (including the VIM ones). At the same time, the non-disconnected controllers can continue to provide the inter-site services. Since the intersite services are proposed outside the VIM networking module's knowledge, ODL assumes that there are no conflicts between networking objects when establishing the service. Actually, ODL cannot provide a recovery method in case of incoherence since it is not the entity in charge of the networking information management. This is an important flaw for the controller when it needs to recover from networking disconnections. • Automatized interfaces: Addressed -thanks to the use of AMQP as east-west interface among the controllers. • Networking technologies: Addressed -ODL implements several networking technologies allowing to reconfigure each controller's networking services.

Tungsten (Open-Contrail)

Tungsten Fabric (previously known as Juniper's Open-Contrail) is the open-source version of Juniper's Contrail SDN controller, an industry leader for commercial SDN solutions targeting overlay and underlay Networks. Tungsten has two main components: an SDN controller and a virtual Router (vRouter). The SDN controller is composed of three types of nodes:

• Configuration nodes that are responsible for the management layer. They provide a REST API [START_REF] Fielding | Chapter 5: Representational State Transfer (REST). Architectural Styles and the Design of Network-based Software Architectures[END_REF] that can be used to configure the system or extract operational status. Multiple nodes of this type can be deployed for HA purposes. Note that configuration states are stored in a Cassandra database (NoSQL). • Control nodes are in charge of implementing decisions made by the configuration nodes. They receive configuration states from the configuration nodes using the IF-MAP protocol and use IBGP to exchange routing information with other control nodes. They are also capable of exchanging routes with gateway nodes using BGP. • Analytic nodes are responsible for collecting, collating, and presenting analytic information.

The vRouter is a forwarding plane of a distributed router that runs in a virtualized server's hypervisor. It is responsible for installing the forwarding state into the forwarding plane. It exchanges control states such as routes and receives low-level configuration states from control nodes using XMPP.

Although there is no constraint on how the different nodes should be deployed, Tungsten architecture can be considered a two-level hierarchical design. Configuration nodes could be seen as root controllers and control nodes as local controllers (hence the configuration nodes can be considered as the leaders). Given the fact that the solution is used by several of the most important actors in the industry and that anyone can test the code, a TRL of 9 has been assigned to Tungsten. Tungsten integrates closely with Neutron consuming its API. Since Tungsten supports a large set of networking services, it is configured as a Core plug-in in Neutron.

Addressing the challenges:

• Information granularity: Not addressed -although multiple configuration nodes can share the network information through Cassandra, the internal design of Tungsten prevents the deployment of different configuration nodes across different sites. An extension has been proposed to handle multi-region scenarios [START_REF] Juniper | Contrail Global Controller[END_REF]. However, the extension exposes a centralized entity to orchestrate remote controllers.

C. Summary

This section described twelve solutions that propose leveraging on several controllers to manage virtualized networking infrastructure. Solutions such as FlowBroker, D-SDN, Tungsten, and Kandoo use a hierarchy of controllers to gather networking states and maintain a global view of the infrastructure. To avoid the SPOF issue of the root controller (see Section V-A, most of these systems propose to deploy multiple instances. By deploying as many root controllers as local ones, it is possible to transform such a hierarchical architecture into a distributed one and envision direct communication between each root controller when needed. The pending issue is related to the global view of the system that needs to be maintained by continuously exchanging messages among the root controllers (i.e., distributed but logically centralized architecture).

To deal with such an issue, solutions such as Elasticon, HyperFlow, Orion, DragonFlow, Onix, and ONOS, use a distributed database, which enables controllers to easily maintain and share global networking information. While it is one more step to fulfill the system's requirements, the efficiency of these systems depends on the capabilities offered by the database system. Even if dedicated systems have been designed for some of them (e.g., ONOS), they do not cope with the requirements we defined in terms of data locality awareness or network partitioning issues.

The two remaining systems, i.e., DISCO and ODL, propose a fully distributed architecture (i.e., without the need for a global view). More precisely, DISCO respects the principle of locality awareness and independence of every group composing the infrastructure: Each controller manages its respective group and peers with another only when traffic needs to be routed to it, thus sharing only service-concerning data and not necessarily global network information. This way of orchestrating network devices is also well fitted in cases of network partitions as an isolated DISCO controller will be capable of providing local domain services. The flaw of DISCO is to provide networking services without the scope of the VIM (i.e., it delivers mainly domain-forwarding operations, which includes only conflict-less exchanges). Offering the VIM expected operations (such as dynamic IP assignment) is prone to conflict and thus might be harder to implement in such an architecture. We discussed this point for ODL, which has a lot of similarities with DISCO (data locality awareness, AMQP to communicate among controllers, etc.). Through the Federation and Netvirt projects, ODL offers premises of a DCI networking service but at a level that does not enable it to solve conflicts. Leveraging the DISCO or ODL architecture and investigating how conflicts can be avoided is a future direction of this survey, as underlined in the following section.

VII. New cloud ecosystems: the Kubernetes case

Although OpenStack still remains the de facto opensource solution to operate private cloud platforms, it is noteworthy that the popularity of VMs as the main unit to execute workloads has been decreasing in favor of lighter technologies such as Docker-based containers [START_REF]The ideal versus the real: Revisiting the history of virtual machines and containers[END_REF]. By promising lowoverhead virtualization and improved performance, containers have become the new center of interest of DevOps [START_REF] Singh | Containers & Docker: Emerging roles & future of Cloud technology[END_REF]. Consequently, a couple of new frameworks in charge of managing the life cycle of container-based applications have been developed [START_REF] Sharma | Containers and Virtual Machines at Scale: A Comparative Study[END_REF]. Released by Google in 2016, Kubernetes [START_REF]Kubernetes[END_REF] (a.k.a. K8s) has become the default solution to manage containers on top of a distributed infrastructure. In addition to help DevOps create, deploy, and destroy containers, K8s proposes several abstractions that hide the distributed infrastructure's complexity. The way it manages network resources, for instance, differs from the OpenStack solution. K8s does not propose means to virtualize and control multiple network segments but rather expose services that relieve DevOps with the burden of managing network low-level aspects (e.g., IP assignments, L2 and L3 management, load balancing, etc.). While Kubernetes significantly from its network abstractions, the challenges at the low level remain close to the ones discussed previously in this article. After summarizing the Kubernetes architecture and networking capabilities, we discuss three projects that aim to deliver inter-site connectivity between multiple K8s instances. By introducing these proposals, we aim to underline that the management of multiple K8s clusters in a DCI context is still an ongoing study field that can benefit from a further discussion around distributed management.

A. Kubernetes overview

According to the ETSI standards, K8s can be considered as a Container Infrastructure Service (CIS), a service that provides a runtime environment for one or more container technologies [START_REF]Network Functions Virtualisation (NFV) Release 3; Architecture; Report on the Enhancements of the NFV architecture towards "Cloud-native" and "PaaS[END_REF]. 2 .

Concretely, K8s divides a cluster into two parts: a set of worker machines called Nodes, where containerized applications are executed, and a set of control plane machines called the Master nodes, in charge of managing the aforementioned Nodes. Figure 6 depicts the K8s default architecture. Each Node has an agent called kubelet that is in charge of creating and configuring containers according to the Master orders, an agent called kube-proxy that is used to define networking rules. Finally, a container runtime such as Docker [START_REF] Docker | Docker[END_REF], Linux Containers [START_REF]Linux containers[END_REF], or any other implementation of Kubernetes Container Runtime Interface (CRI) [125] to effectively start and execute containers. The Master is composed of the API server, the scheduler that assigns workloads to Nodes, the controller managers that maintain the cluster's expected state using control loops, and etcd, a key-value store used as Kubernetes backend. K8s does not directly deal with containers but works at the granularity of Pods. A Pod is a group of one or more containers with shared networking and storage resources and a specification defining how to run the workload (number of replicas, etc.).

In addition to basic operation on Pods (creation, deployment, etc.), K8s proposes several abstractions (objects or resources in the K8s terminology) to hide the architecture's distribution. In other words, DevOps do not have to deal with low-level aspects of the infrastructure but use K8s predefined objects. For instance, Volumes are storage units accessible in a Pod wherever they are deployed. Similarly, Services are used to logically abstract a Pods group with a DNS name and a virtual IP. Finally, a Namespace enables DevOps to isolate Pods within a cluster. Additional objects have been built on top of these abstractions in order to offer supplementary functionalities. For instance, ReplicaSet enables DevOps to define many replicas for a Pod and let the K8s controller maintain this number. Because of the modularity of K8s, more objects can be exposed by the API (for an up-to-date list, please refer to [START_REF]Kubernetes API overview[END_REF]). This philosophy of using predefined abstractions is a major change concerning the OpenStack solution where DevOps should deal with many infrastructure details.

B. Kubernetes networking

From the network point of view, there are four types of communications in K8s: (i) Highly coupled containerto-container, (ii) Pod-to-Pod, (iii) Pod-to-Service, and (iv) External-to-Service.

Rather than imposing a single implementation and in order to leverage modularity, K8s supports its networking functionality through the Container Network Interface (CNI) [START_REF]Container Network Interface specification[END_REF], an open-source project proposing a specification and libraries for writing plug-ins to configure network interfaces on Linux containers. A CNI plug-in is responsible for inserting a network interface into the container network, making necessary changes on the host, assigning an IP address to the interface, and configuring routes for the traffic exchange. A second plugin called the IP Address Management plug-in (IPAM) is used for the IP and routes configurations. Several CNI as well as IPAM plug-in implementations have been proposed to abstract the network within a K8s cluster [START_REF]Kubernetes Network Plugins[END_REF]- [START_REF]OpenStack kuryr[END_REF]. It is noteworthy to mention that the split between the CNI plug-in and the IPAM module provides more flexibility as it is possible to use a combination of two different solutions.

By default, implementing a plug-in should deliver the four types of communications with the following properties. Regarding Pod-to-Pod communications, a Pod on a Node can communicate with all Pods on all Nodes without NAT communications. Regarding Pod-to-Service communications, the virtual IP associated with a Service needs to be correctly translated and load-balanced to Pods registered to this Service (using technologies such as iptables or IP virtual servers (IPVS) [136]). Finally, for the External-to-Service exchanges, the configuration of the different routes exposing Services should be achieved (implementing the logic of the K8s NodePort, Load or Ingress controller objects).

C. Multi-Cluster Kubernetes

Like OpenStack, Kubernetes has been designed in a standalone manner: it exposes the resources of a single cluster.

Hence, being able to execute container-based applications across multiple sites (i.e., a DCI as we defined in this article) raises different questions in the Kubernetes ecosystem also [START_REF] Jenkins | To Multicluster, or Not to Multicluster: Inter-Cluster Communication[END_REF]. From the network viewpoint, a K8s Multi-Cluster architecture should deliver means to provide the aforementioned communications. More precisely, since Container-to-Container communication is limited at Pod's scope, solutions for the three latest communications are required. We discuss in the following three projects, namely Kubernetes Federation [START_REF] Sig Multicluster | Kubernetes Cluster Federation[END_REF], Istio Multi-Cluster service mesh [START_REF] Istio | Multicluster Deployments[END_REF] and Cilium MultiCluster CNI implementation [140]. From our point of view, these projects are the most advanced among a large number of proposals [START_REF] Admiralty | Multi Cluster Scheduler[END_REF]- [START_REF]Submariner, connected Kubernetes overlay networks[END_REF]. In addition to a brief description, we present whether they address the DCI challenges introduce in Section IV.

1) Kubernetes Federation: Kubernetes Federation (a.k.a. Kubefed) is a project developed by the multi-cluster working group of K8s, providing an easy-to-use way to manage multiple K8s clusters. Kubefed does not target the intersite networking connectivity but rather a global approach for deploying container-based applications across multi-sites through a common API. In other words, it does not leverage nor integrate SDN technologies but rather proposes a brokerlike approach to partially deal with the DCI challenges.

In detail, Kubefed relies on a two-layer architecture where a central cluster called host cluster will propagate its application configuration to a series of independent clusters called member clusters. To make this possible, the host leverages a federation API using custom resource definitions (CRDs), an object provided by the vanilla Kubernetes that allows DevOps to define their own data types. These new federated objects are then used to wrap basic objects. For example, FederatedService and FederatedDeployment objects are abstractions to wrap the vanilla Service and Deployment objects.

When a FederatedService is created, Kubefed creates matching K8s Services in the selected member clusters. To propagate each cluster's DNS records, Kubefed gathers all the locally generated DNS records in the host cluster and then pushes the records to each of the concerning clusters. This implies that Services must be exposed using a publicly available IP address. From the network point of view, Kubefed can only provide cross-cluster Pod-to-Service and External-to-Service communications, relying on the public routable IP addresses in both cases. Since it proposes management at the API level, no coordination is possible at the low-level networking implementation. In consequence, cross-cluster Pod-to-Pod communication is not proposed. More generally, Kubefed presents important flaws for the DCI context. In addition to the host cluster's limitation that is the only entry point for federated resources (SPOF), there is no collaboration among the different K8s instances. In other words, there is no mechanism to propagate modifications done on one particular K8s object to the other sites, even if this object has been created through a federated abstraction.

Addressing the challenges:

• Information granularity: Not addressed -the segregation of information into each cluster enables the efficient sharding of the information per site. However, the host cluster gathers global information about federated resources. • Information scope: Partially addressed -while Kubefed only contacts the relevant sites when deploying a federated resource, this is only done by the host cluster. • Information availability: Addressed -in case of network disconnection, each cluster is fully independent of the others, with the worst-case scenario being the isolation of the host cluster. Since a federated resource is deployed on the concerned clusters, the local resources' information remains locally available. Istio is an open-source implementation of a service mesh that provides traffic management, security, and monitoring. In the microservices context, a service mesh is a complementary layer to the application, and it is responsible for traffic management, policies, certificates, and service security [146]. To provide this, a service mesh introduces a set of network proxies that will be used to route requests among services. The idea is to inject a special sidecar container in the Pod of every microservice and route traffic through these sidecars instead of doing it through the Service. A central authority will then exerts control over the proxies to route traffic at the application layer (L7 of the OSI model). Hence, a service mesh follows a design pattern familiar to the SDN principles [START_REF] Antichi | Full-Stack SDN: The Next Big Challenge?[END_REF].

From the architecture viewpoint, an Istio deployment is logically composed by a control plane, which manages and configures the proxies and elements such as gateways to route traffic, and a data plane, which is composed of a set of intelligent proxies (Envoy [START_REF]Envoy[END_REF]) deployed as sidecars. Istio also proposes an Ingress Gateway object that acts as a load balancer at the mesh's edge receiving incoming or outgoing traffic.

This concept of service meshes can be extended to take account of multiple clusters. The idea is then to have a logical service mesh composed of several clusters [START_REF] Istio | Istio Replicated Control Planes[END_REF]. For this to be done, Services from remote clusters are created locally via ServiceEntries, an Istio resource that is not proposed by vanilla Kubernetes. The Istio Ingress Gateway is then used as an entry point for requests to the cluster's Services.

The Istio service mesh operates at the application layer. Offering its functionalities at this level implies that Istio specific resource definitions need to be used in deployments. Besides, considering all hops, a request must go through from containers to sidecars in a DCI context along with all the ServiceEntry rules treatment and processing requests on remote gateways could greatly add latency and potentially add a performance overhead [START_REF] Srinivasan | Connecting multiple Kubernetes Clusters on vSphere with Istio Service Mesh[END_REF], [START_REF] Istio | Istio Performance and Scalability[END_REF].

Regarding the three communication types, since Istio is a service mesh-oriented solution, it can only provide cross-cluster Pod-to-Service and External-to-Service communications using the replication of ServiceEntries. In the case of Pod-to-Service communication, Services with private virtual IPs are reachable through the Istio Ingress gateways.

Addressing the challenges:

• Information granularity: Addressed -due to the segregation of the infrastructure in independent clusters while proposing strategies to share the information related to external Services using the replication of remote Services as ServiceEntries. 3) Cilium Multi-Cluster: Cilium is a CNI plug-in that implements the Kubernetes networking API by leveraging the extended Berkeley Packet Filter (eBPF). EBPF is used to forward data within the Linux kernel. It is an alternative of IP Tables, which delivers better performance [START_REF] Mccanne | The bsd packet filter: A new architecture for user-level packet capture[END_REF], [153].

In addition to the Cilium plug-in, each K8s node executes a Cilium agent. (This agent is in charge of interacting with the CRI to setup networking and security for containers running on the node. Finally, a key-value store, global to the cluster, is used to share data between Cilium Agents deployed on different nodes.

Cilium proposes a multi-cluster implementation called Clus-terMesh. It provides Pod IP routing among multiple Kubernetes clusters thanks to tunneling techniques through eBPF (i.e., without dedicated proxies or gateways). Concretely, the Cilium key-value store of each cluster must be exposed as a public Service. This enables Cilium agents to collect information from all stores in order to create tunnels with all other clusters. Moreover, Cilium allows the creation of a Global Service by creating at each cluster a Service with an identical name and namespace. An additional Cilium annotation defining the Service as global is mandatory on each cluster. Cilium uses this annotation to automatically loadbalance requests throughout all Pods exposing the Service in the different clusters.

Thanks to its relation with K8S at the CNI level, Cilium effectively provides the three aforementioned communications types. However, such a model's scalability is questionable as ClusterMesh initiates a tunnel (e.g., VXLAN or Geneve) between each worker nodes pair.

Addressing the challenges:

• Information granularity: Addressed -Cilium ClusterMesh leverages several independent clusters operating the infrastructure, each one only managing its own deployment. • Information scope: Not addressed -because Cilium Clus-terMesh requires to connect all clusters before deploying applications, agents create tunnels to all remote Nodes at the cluster setup. In this sense, information is exchanged even before a user requires an inter-site resource to be deployed. • Information availability: Addressed -due to clusters being independent among them. In the case of networking partitioning, the isolated sites continue to propose their services locally, and sites still connected can continue to provide the ClusterMesh capabilities.

D. Summary

Building a system capable of providing a native distributed DCI management from scratch is not an easy task. As we have demonstrated for OpenStack, Kubernetes also needs to be extended.

While some ongoing works try to propose ways to make independent Kubernetes collaboration, we consider that there is still plenty of innovation and improvement opportunities. Kubernetes Federation proposes to leverage a "master" cluster exposing a federated API that will translate the application deployment to a series of worker clusters. The problem with this approach is that worker clusters are not aware of each other presence and the "master" cluster is the only entry point for federated resources. Other projects such as Istio and Cilium require the user to deploy applications independently in each cluster, which can be costly when needed in hundreds or thousand clusters [START_REF] Srinivasan | Connecting multiple Kubernetes Clusters on vSphere with Istio Service Mesh[END_REF], [START_REF] Srinivasan | Connecting multiple Kubernetes clusters on vSphere with the Cilium Cluster Mesh[END_REF].

VIII. DIRECTIONS FOR DCI NETWORKING RESEARCH

In the previous sections, we have studied major decentralized SDN technologies in the DCI context. While we identified that DISCO and ODL are good candidates, several open questions and challenges need further investigation. We discuss the most important ones in the following.

A. East-West SDN-based interfaces

While the East-West interface has been considered as the principal element to provide inter-controller communications [START_REF] Haleplidis | Software-Defined Networking (SDN): Layers and Architecture Terminology[END_REF], there is still no standard to communicate between controllers, and some proposals co-exist [START_REF] Yin | SDNi: A Message Exchange Protocol for Software Defined Networks (SDNS) across Multiple Domains[END_REF]- [START_REF] Benamrane | An East-West interface for distributed SDN control plane: Implementation and evaluation[END_REF]. This is an issue as such a standard is critical for delivering collaborations between networking modules of multiple VIM instances of a DCI. Moreover, this lack of standardization impacts the communication and automation between North and East-West interfaces. This leads to multiple ad-hoc solutions [START_REF] Kreutz | Software-defined networking: A comprehensive survey[END_REF], [START_REF] Lin | East-West bridge for SDN network peering[END_REF].

As we outlined in the last section, the East-West interface proposed by DISCO and ODL provides some references to design an efficient horizontal interface for inter-VIMs networking modules communications. Although the analyzed solutions leveraged AMQP as technology to do the East-West interface implementation, other technologies such as REST APIs could be used to provide synchronization and information exchanges among VIMs.

If we consider the model proposed by DISCO and ODL, the use of independent and local databases implies managing consistency at the application level (i.e., between the different controllers). This entails that the East-West interface should deliver additional calls to resolve conflicts depending on the controllers' inter-site service logic. Since neither DISCO nor ODL proposes a way to manage conflicts at the East-West interface level, this remains an open question, as already highlighted. Another solution could consist of leveraging distributed databases.

B. Leveraging new kinds of databases

As we highlighted in the summary of Section VI, solutions such as Elasticon, HyperFlow, Orion, DragonFlow, Onix, and ONOS, use a distributed database to share global networking information among controllers. While this approach is useful as it is intended to avoid a single point of failures and bottlenecks by logically splitting the database, it does not respect the principles of data locality and is not resilient enough in case of network partitions.

To illustrate this point, one can consider the Cassandra database [START_REF]Apache Cassandra[END_REF]. Cassandra is based on a distributed hash table (DHT) principles to uniformly distribute states across multiple locations based on the computed hash. This means that one specific controller's states are not necessarily stored in the same geographical location as the controller (e.g., a controller in site A will have states stored in remote sites B, C, and so forth). Thus, the principle of locality awareness is not respected as information belonging to site A will be spread to other sites with no possibility to control the location.

Likewise, an SDN-based cloud architecture that leverages Cassandra will not be resilient to network isolation. It will be impossible for the local controller to retrieve its states, which may have been spread over non-reachable sites.

However, data-related approaches different from traditional SQL and NoSQL engines can be relevant. In the last years, the concept of NewSQL has been gaining popularity and notoriety as an approach mixing the advantages of both traditional SQL and NoSQL systems. These kinds of engines search to propose the same scalability of NoSQL systems while keeping the relational model of traditional SQL (i.e., maintaining the ACID guarantees) engines [START_REF] Aslett | How will the database incumbents respond to NoSQL and NewSQL?[END_REF]. NewSQL engines generally try to leverage different memory storage modes, modes of partitioning, and concurrency control to provide the aforementioned properties.

While historically, database engines have used disk-oriented storage, today, NewSQL engines can profit from memory cost reduction and leverage memory-oriented approaches. Using this approach, new engines can get better performance because slow reads and writes to disk can be avoided [START_REF] Harizopoulos | Oltp through the looking glass, and what we found there[END_REF]. Moreover, almost all engines used to scale out splitting a database up into subsets, called partitions or shards.

NewSQL engines can be gathered in three main classes [START_REF] Pavlo | What's really new with newsql?[END_REF]: new architectures, transparent sharding middlewares, and Databases-as-a-Service.

• New architectures: This group gathers engines build from scratch and that mostly use a distributed shared-nothing architecture [START_REF] Stonebraker | The case for shared nothing[END_REF]. Most of them are also capable of sending intra-query data directly between nodes rather than having to route them to a central location. In this group we find solutions such as Clustrix [START_REF] Mariadb | A NEW APPROACH TO SCALE-OUT RDBMS[END_REF], Cock-RoachDB [165], Google Spanner [START_REF] Corbett | Spanner: Google's globally distributed database[END_REF] and its related solutions such as Google F1 [START_REF] Shute | F1: A distributed sql database that scales[END_REF], VoltDB [START_REF] Voltdb | Voltdb technical overview[END_REF], or HyPer [START_REF] Kemper | Hyper: A hybrid oltp olap main memory database system based on virtual memory snapshots[END_REF]. • Transparent sharding middleware: This group gathers engines that split a database into multiple shards that are stored across a cluster of single node instances. In this sharding, each node runs the same database management system. Each one has only a portion of the overall database, and data does not mean being accessed and updated independently by separate applications. Then, a centralized middleware component routes queries, coordinates transactions, and manages data placement, replication, and partitioning across the nodes. In this group we find solutions such as AgilData Scalable Cluster [START_REF] Agildata | AgilData Scalable Cluster for MySQL[END_REF] or MariaDB MaxScale [START_REF] Mariadb | MariaDB MaxScale[END_REF]. • Database-as-a-Service (DBaaS): While there are already DBaaS propositions, there are only a few NeWSQL DBaaS engines available. In this group we find solutions such as Amazon Aurora [172] or ClearDB [173]. For instance, Amazon Aurora does a decoupling between the engine storage and compute. In this sense, the network becomes the constraint because all input and outputs (I/O) will be written over it. In order to do this operation, Aurora relies upon a log-structured storage manager capable of improving I/O parallelism [START_REF] Verbitski | Amazon aurora: Design considerations for high throughput cloud-native relational databases[END_REF]. In order to measure the value of these new engines in the DCI context, an SDN-based application needs to be built on top of the selected solution to analyze the pros and cons of each one and verify if they can satisfy the DCI requirements highlighted in Section I.

On the other hand, solutions such AntidoteDB [START_REF]AntidoteDB[END_REF] or Riak [176] that have been designed on top of conflict-free replicated data type (CRDT) [START_REF] Shapiro | Conflict-Free Replicated Data Types[END_REF] could be leveraged by SDN controllers in order to address the aforementioned DCI challenges while respecting the principal characteristics of DCI architectures such as data locality awareness. A CRDT is a data structure that can be replicated across multiple nodes in a network. Each replica can be updated independently and concurrently. This means that each replica will be locally accessible and ready to use in the case of network partitions. The richness of the CRDT structure is that it is possible to eventually resolve the inconsistency between multiple replicas. However, CRDTs come with two important limitations. First, it requires to replicate the state of every site of the DCI infrastructure. Second, only elementary values can be structured as CRDT right now. For instance, it is not sure that a CIDR can be modeled as a CRTD. If solutions for these two problems might be found, CRDT may represent a step forward to provide a distributed solution while respecting the DCI properties.

C. Data plane technologies

The ecosystem to deliver traffic forwarding between virtual instances is old and extremely rich, with multiple proposals since the initial OpenFlow protocol [START_REF] Mckeown | OpenFlow: Enabling innovation in campus networks[END_REF] (BGP [START_REF] Rekhter | A Border Gateway Protocol 4 (BGP-4)[END_REF], SoftRouter [START_REF] Lakshman | The SoftRouter Architecture[END_REF], RCP [START_REF] Caesar | Design and Implementation of a Routing Control Platform[END_REF], as well as RouteFlow [START_REF] Nascimento | Virtual Routers as a Service: The RouteFlow Approach Leveraging Software-Defined Networks[END_REF] to name a few). This eco-system continues to grow with more recent solutions [START_REF] Thimmaraju | Taking Control of SDN-based Cloud Systems via the Data Plane[END_REF]- [START_REF] Wang | Enabling Policy Innovation in Interdomain Routing:A Software-Defined Approach[END_REF]. Since heterogeneity in networking protocols may be present in a DCI, the possibility to agree on which mechanisms to use when establishing an inter-site networking service needs to be considered in future works. While being only a theoretical proposition, the works presented in [START_REF] Openstack | Neutron-Neutron Interconnections[END_REF] for Neutron to Neutron Interconnections proposes such kind of mechanism agreement. In this sense, two Neutron instances will identify the mechanism to use and the corresponding parameters that will be exchanged (e.g., VLANs IDs, BGPVPN RT identifiers, VXLAN IDs, or L2GWs addresses).

D. Performance analysis

This survey focuses on defining a general distributed model for DCI networking management based on SDN technologies. While it gives valuable information, it would be relevant to evaluate the selected solutions under the performance perspective. Depending on the analyzed element (e.g., East-West interface, database, or networking protocols) of new proposals, the metrics and analyzed characteristics may vary:

• East-West interfaces: The use of a horizontal interface implies that besides the time expended by the system to answer a user request locally (e.g., CPU consumption, memory consumption, threads utilization), the time needed to synchronize with remote sites and provide a final result needs to be taken into account [START_REF] Benamrane | An East-West interface for distributed SDN control plane: Implementation and evaluation[END_REF]. Such delay will impact the total inter-site service creation time and may depend on the technical implementation of the solution (e.g., protocol used). In the architectural design, the quantity of messages needed to establish a service, needs to be optimized to minimize the overhead. Thus, future works should analyze the impact on the performance of their inter-site communication model and the technology used to implement it. • New kinds of databases: Like the East-West interface, the use of new kinds of database engines will add an extra delay to communicate with remote sites. NewSQL benchmarking studies such as the ones in [START_REF] Kaur | Performance evaluation of NewSQL databases[END_REF], [START_REF] Schreiner | Newsql through the looking glass[END_REF] could be extended to take into account the DCI case. Additionally, a comparison can be made w.r.t. traditional SQL database replication systems and distributed databases in two different aspects: the local execution time spent by the database executing CRUD actions (e.g., CPU consumption, memory consumption,...), and the time needed to communicate with remote sites in order to replicate data or synchronize them. Moreover, an important analysis should be done to clarify how the database will deal with conflicts or inconsistencies in network partitions. Although new database engines provide models based on theoretical assumptions, there are side effects that can be identified only by conducting experimental activities The aforementioned studies [START_REF] Bousselmi | Towards a Massively Distributed IaaS Operating System: Composition and Evaluation of OpenStack[END_REF], [START_REF] Cherrueau | A POC of OpenStack Keystone over CockroachDB[END_REF] can be taken as a good example of such kind of experimental-driven research needed in this context. • Networking protocols: Since the networking route exchange and traffic forwarding and routing will also impact the time needed for an inter-site service to be effectively deployed, the performance of the data plane technologies needs to be taken into account. For instance, in the case of BGP VPN routes exchanges, prior works analyzed the benefits and disadvantages of their use in several contexts [START_REF] Palmieri | VPN scalability over high performance backbones evaluating MPLS VPN against traditional approaches[END_REF], [START_REF] Mai | BGP performance analysis for large scale VPN[END_REF]. Although the implementation of several different networking protocols is a very challenging and complex task, solutions supported by large communities such as OpenStack, ODL, or ONOS could promote such development to perform further tests and analysis. Work such as [START_REF] Azodolmolky | SDN-based Cloud Computing Networking[END_REF], [START_REF] Risdianto | Deployment and evaluation of software-defined inter-connections for multi-domain federated sdncloud[END_REF] proposing the comparison of virtual networking implementations in SDN-based scenarios can be used as guidelines to identify major indicators (i.e., throughput, CPU utilization, latency, packet loss, etc.). • Consensus protocols application: If a consensus protocol is mandatory, further investigations will be needed in order to quantify the overall performance of the protocol. While works such as [START_REF] Sakic | Response time and availability study of raft consensus in distributed sdn control plane[END_REF]- [START_REF] Ho | A fast consensus algorithm for multiple controllers in software-defined networks[END_REF] already analyzed the use of consensus protocols in SDN, they mostly targeted traditional SDN networks and not DCI architectures. Indeed, the round-trip times in a geo-distributed infrastructure, composed by hundreds or thousands of nodes, could affect the protocol performance by adding latency for which the consensus protocol may not be designed [START_REF] Ailijiang | Consensus in the Cloud: Paxos Systems Demystified[END_REF]. Moreover, tasks such as leader election may create traffic overload due to the number of messages exchanged. For this reason, compared with a static master election or leaderless protocols will be needed to understand the different trade-offs that may be involved.

E. Security

Some of the open questions in SDN-based cloud computing solutions rely on security issues. While security in cloud data centers has been explored in the last decade [START_REF] Rittinghouse | Cloud Computing Implementation[END_REF], [START_REF] Almorsy | An Analysis of the Cloud Computing Security Problem[END_REF], more research in security for SDN and SDN-based solutions is needed [START_REF] Son | A Taxonomy of SDN-enabled Cloud Computing[END_REF]. Obviously, the decentralization reduces the impact of having a single point of failure into an architecture (e.g., DoS attacks), but some other components need to be revised. For instance, the inter-site communication needs to be secure enough to avoid uncontrolled or non-desired intrusions. The protocols allowing the database distribution may also be deeply studied in order to evaluate the impact of encryption technologies in the overall performance of future solutions. Finally, as a great number of tenants may share the same network medium to access DCI networking services, the isolation and security provided by the networking protocols used will need further studies.

IX. CONCLUSIONS

Leveraging each Point of Presence (PoPs) in Telcos' architectures as part of a Distributed Cloud Infrastructure (DCI) is an approach that will be mandatory soon to address the requirements posed by trends like NFV, IoT, and MEC. Although some resource management systems for this kind of infrastructure have been proposed, providing inter-site networking services in a distributed fashion is still a challenge to be addressed.

This paper explained the problems of centralized networking management in VIMs and how decentralized SDN approaches could represent an opportunity to tackle the inter-site connectivity management challenges. We then presented a survey of several multi-controller SDN solutions and their possible use and limitations as a DCI network management tool.

Solutions such as FlowBroker, D-SDN, Tungsten, or Kandoo propose maintaining a global view of the system through a hierarchy of controllers that does not enable to address the identified challenges.

The use of a distributed database to share global information among the controller proposed by Elasticon, HyperFlow, Orion, DragonFlow, Onix or ONOS does not entirely address the proposed challenges as the use of the database in a DCI context and under network partitioning is unclear and do not respect the general requirements. We, however, do not eliminate such an approach as new database engines have been recently proposed [START_REF]AntidoteDB[END_REF], [START_REF] Kaur | Performance evaluation of NewSQL databases[END_REF]. In particular, we should better understand how these systems behave according to the DCI challenges (data granularity, scope, and network partitioning issues) while respecting the DCI general requirements (such as the locality awareness).

Solutions following fully distributed architectures address several challenges while guaranteeing the general requirements. In particular, we would like to underline the potential of a solution like DISCO or ODL to distribute the connectivity management in a scalable, locality-aware, resilient, and easy manner. The work we should address in the short term is to investigate how VIM-related operations can be developed. In particular, a proposal should include creating on-demand Layer 2 extensions and Layer 3 to span overlay networks among the requested sites at the VIM-level, avoiding or solving the possible management conflicts.

We concluded our survey by analyzing the Kubernetes ecosystem. Kubernetes has gained important popularity in the DevOps community, in particular, due to its way of hiding low-level technical details thanks to high-level abstractions (e.g., DevOps no longer have to deal with IP assignments).

Although significantly differs from IaaS solutions, it has been designed to operate a single cluster. Similarly to OpenStack, the development of inter-site networking services at a low level is mandatory to enable K8s abstractions to be visible and consistent between multiple sites. Like DISCO or ODL, some proposals have been proposed. For instance, the Cilium ClusterMesh proposes such inter-site capabilities for the network in a decentralized manner. However, the requirement of interconnecting all worker nodes affects the scalability.

Whether we consider a resource management system for IaaS or CaaS solutions, the implementation of a distributed tool in which network devices are automatically configured, provisioned, and managed may represent a huge contribution to favor the advent of DCIs such as envisioned in Fog and Edge Computing platforms. Through this survey, our goal was to identify key elements that can be used to guide the design and implementation of such a tool.

Fig. 2 :

 2 Fig. 2: Neutron deployment components.

Fig. 3 :

 3 Fig. 3: Layer 2 extension Request

Fig. 4 :

 4 Fig. 4: Operate in a local any mode

Fig. 5 :

 5 Fig. 5: SDN topologies

Fig. 6 :

 6 Fig. 6: Kubernetes cluster components.

•

  Network plug-ins: While Neutron API exposes the virtual network service interface to users and other services, the actual implementation of these network services resides in plug-ins. The Neutron pluggable architecture com-

prises Core plug-ins (which implements the core API) and Service plug-ins (to implement the API extensions). Core plug-ins primarily deal with L2 connectivity and IP address management, while Service plug-ins support services such as routing. The Modular Layer 2 (ML2) is the main OpenStack Core plug-in. It supports type drivers to maintain type-specific network state (i.e., VLAN, VXLAN, GRE, etc.) and mechanism drivers for applying configuration to specific networking mechanisms (i.e., OpenvSwitch

  Used by most Neutron installations to route information between the Neutron-server and various Agents.2) Neutron networking:Neutron divides the networking constructions that it can manage as Core and extension resources. Port, Network, and Subnetwork are the basic Core object abstractions offered by Neutron.

• Network Agents: Agents implement the actual networking functionality closely associated with specific technologies and the corresponding plug-ins. Agents receive messages and instructions from the Neutron server on the message bus. The most common Agents are L3 (Routing functions), L2 (Layer 2 functions), and Dynamic Host Configuration Protocol (DHCP).

• Messaging queue:

  The user is only allowed to use this central API to manage or control networking resources. Since Local Neutrons always need to communicate with Central Neutron in order to answer local requests, network partitioning cases become the principal blocking point as isolated Local Neutrons will remain useless as long as the disconnection remains. Finally, Tricircle does not leverage real collaboration among Local Neutron. Indeed, there is neither communication nor knowledge of each other's presence since all the management is done at the Central Neutron.

TABLE I

 I

	: DCI Challenges summary
	information each time a new IP is allocated to a particular
	resource.

•

  Overlay network : Virtual network built on top of another network, normally the underlying physical network, and connected by virtual or logical links. Overlay networks help administrators tackle the scalability challenge of the underlay network. For instance, overlay networks leverage the use of encapsulation protocols like VXLAN because of its scalability (e.g., VXLAN provides up to 16 million identifiers while VLAN provides 4096 tags).

TABLE II :

 II Classification of SDN solutions.

	Extra consideration	OpenStack	compatibility
		Readiness Level
	Interoperability & maturity	Southbound Network types	Protocols targeted			Underlay Overlay
		Database	manage-	ment	system
				Among	root	nodes
		Internal	Communication	Protocols With	higher	layers
				Among	local	nodes
	Implementation	Coordination	Strategy		Leader-Leader-	based less
		Distributed Designs		(Flat)	Logically Hierarchical Hybrid	distributed
				(Flat)	Logically	centralized
	Model	Centralized	(Single)	Controller	Designs
		Proposals			Network-oriented solutions

TABLE III :

 III Summary of the analyzed SDN solutions. scenarios. The main idea is to have replicas of SCs in order to cope with network or node failures.

  Not addressed -D-SDN proposes an interface for SC-SC communication only for fault tolerance issues. Moreover, there is no information regarding MC-MC communication patterns. Not addressed -since it does not integrate any other networking technologies nor the capacity to provide inter-group service deployment.

• Information scope: Not addressed -the MC gathers global knowledge, and the communication between SC appear just for fault tolerance aspects.

• Information availability: Undefined -in case of disconnection, SCs controllers can continue to provide forwarding within its local domain at first sight. However, the article does not specify how the MC deals with such a disconnection. Besides, the controller does not provide any type of recovery method as D-SDN does not consider network partitioning issues. • Automatized interfaces: • Networking technologies:

  Not addressed -since it only operates in OpenFlow-based scenarios.

• Information availability: Undefined -similarly to the previous challenge, it is linked to the way the database services deals with network partitioning issues. In other words, intra/inter-domain forwarding paths that have been previously established should go on theoretically (network equipment has been already configured). Only the recovery mechanism to the DB is unclear.

• Automatized interfaces: Partially addressed -because each controller already has a TCP channel to commu-nicate with the other controllers. However, this communication channel is only used for coordination purposes. • Networking technologies:

  Partially addressed -since WheelFS is used as both communication and storage utility among controllers. Thus, it is used as the eastwest interface. However, HyperFlow's authors underlined that the main disadvantage of the solution is the use of

	WheelsFS: WheelsFS can only deal with a small number
	of events, leading to performance penalties in cases where
	it is used as a general communication publish/subscribe
	tool among controllers.
	• Networking technologies: Not addressed -since it does
	not integrate other networking technologies besides
	OpenFlow.
	• Information availability: Partially addressed -in case of
	a network partitioning, every controller can continue to
	serve their local forwarding requests. Regarding inter-
	forwarding, the dependency w.r.t. to WheelFS is unclear.

Theoretically speaking, inter-forwarding channels should survive disconnections (at least among the controllers that can interact). Moreover, WheelFS provides a recovery method to deal with network partitioning issues. Such a feature should enable controllers to request new interforwarding paths after disconnections without implementing specific recovery mechanisms. Similar to previous solutions, this is possible because the information shared through WheelFS is conflict-less.

• Automatized interfaces:

  Partially addressed -similarly to DragonFlow, the use of distributed DB to share information among instances can be seen as an east-west interface allowing communication among controllers.

	• Networking technologies: Partially addressed -the so-
	lution has been designed to use several networking
	technologies and protocols. Although the initial Onix
	proposition only supported OpenFlow, Onix design does
	not impose a particular southbound protocol but rather
	the use of the NIB as an abstraction entity for network
	elements.

• Information availability: Undefined -established services can go on, and disconnected sites can continue to operate in isolated mode. The main issue is related to the NIB that should provide the necessary consistency algorithms to allow recovery in case of network disconnection.

• Automatized interfaces:

•

  Information scope: Addressed -since Istio proposes the creation of ServiceEntries to reference remote Services, only the relevant clusters are taking into account to do the information exchange. • Information availability: Partially addressed -in case of network disconnections clusters remain locally operative. However, considering that ServiceEntries are replicated on demand, Istio does not provide mechanisms to ensure the consistency between K8S Services and the information related to the ServiceEntries. • Automatized interfaces: Not addressed -Istio does not implement an east-west interface allowing cluster collaboration. Instead, the user is in charge of mirroring the Istio configuration between the different clusters in order to deliver a Multi-Cluster service mesh. Not addressed -although the network routing logic is implemented at the Envoy proxy and at the Istio Ingress Gateway, Istio is independent of the low-level network technology used by the cluster.

• Networking technologies:

  Addressed -Cilium proposes to expose the local database to remote clusters to exchange information. The way remote clusters consume this information could inspire more propositions leveraging their databases' exposition as an east-west interface among clusters. It is noteworthy to mention that the user still needs to deploy applications and Services in each cluster to provide a Multi-Cluster Service.

• Automatized interfaces: • Networking technologies: Partially addressed -While the solution leverages eBPF as networking technology incorporating its adaptation and reconfiguration, it lacks the heterogeneity of networking protocols.

Unless specified, we used the term VIM in the rest of the article for infrastructure management systems of any kind such as OpenStack.

This interpretation is open to debate since it can also be defined as a Container Infrastructure Service Management (CISM), a function that manages one or more CISs[START_REF]Define the place of Kubernetes in the ETSI NFV MANO stack and document the result[END_REF]