Kiranpreet Kaur

Fabrice Guillemin

Francoise Sailhan

Container placement and migration strategies for Cloud, Fog and Edge data centers: A survey

Keywords: Container migration, migration techniques, placement strategies

Container placement and migration strategies for Cloud,

Fog and Edge data centers: A survey

Introduction

A notable trend in current networks is the network softwarization that promotes the adoption of virtualization technologies to support the rapid development of new services that readily adapt to the evolving customer needs. Network softwarization leads to the gradual replacement of hardware network function operating on purpose-built & proprietary network equipment by Virtualized Network Functions (VNFs) that are consolidated on commodity hardware. In practice, a network function (NF) CNFs may offer a wide range of networking capabilities that operate on the Universal Customer Premise Equipement (uCPE), up to the core network supporting e.g. tunneling, firewalling or application-level functions. Microservices have become instrumental in the design of complex NFVs that necessitate a decomposition into many of services -e.g. several hundreds services for core network functions. In such case, microservices are small services implementing a limited amount of functionalities that can be executed independently (even if they are logically dispersed at the edge, fog or in the cloud) -each microservice executes its own processes/functionalities and communicates via lightweight protocols.

Overall, cloud-native design offers a different approach to the development of softwarized networks, an approach that is suited to the agility and that supports an efficient scaling up and orchestration of the distributed network functions. Container-oriented approach is also increasingly privileged as a containerized microservice can be rapidly instantiated as required and also can be scaled-out independently, to support the increasing demand for more processing or storage, without unnecessarily scaling the overall network function.

While expectations are high for the wide applicability of network softwarization, putting the deployment of VNFs into practice is far from being a trivial task especially. With NFV, a network service is composed of series of network functions (a.k.a microservices) characterised by a predefined order, which is known as service chaining. By design, VNF supports a dedicated specific functionality and often remains state-dependant, i.e., in practice, states are stored and updated locally with the associated VNFs. Following the chain, traffic goes through the series of ordered network functions such that traffic may flow back and forth among distant VNFs as VNFs reside on distinct physical servers or data centers. During the operation of the VNF, traffic, network bandwidth, available storage and computational resources typically fluctuate over time, which results in imbalanced links/ resource usage. Thus, the efficient allocation and the continuous management of NFVs become more complex, considering the heterogeneity and the dynamics of the physical resources as well as the ephemeral nature of the services. To overcome this issue, a growing number of research effort has been devoted to support the migration of network services possibly to other physical server(s)/data center(s), which is key to preserve the Quality of Service (QoS) and meet the expectation of the user in terms of performance. While related topic including VM/container migration in cloud data centers has matured, the decoupling and elastic re-allocation of small networking functions across data centers spanning the edge to the core remains challenging.

Related surveys

With the virtualization gaining attention, many surveys on VM (e.g. [START_REF] Choudhary | A critical survey of live virtual machine migration techniques[END_REF][START_REF] Venkatesha | Survey of virtual machine migration techniques[END_REF][START_REF] Zhang | A survey on virtual machine migration: Challenges, techniques, and open issues[END_REF][START_REF] Leelipushpam | Live vm migration techniques in cloud environment-a survey[END_REF][START_REF] Patel | Live virtual machine migration techniques in cloud computing: A survey[END_REF][START_REF] Kapil | Live virtual machine migration techniques: Survey and research challenges[END_REF][START_REF] Strunk | Costs of virtual machine live migration: A survey[END_REF]) and container (e.g., [START_REF] Wang | A survey on service migration in mobile edge computing[END_REF][START_REF] Rejiba | A survey on mobility-induced service migration in the fog, edge, and related computing paradigms[END_REF][START_REF] Singh | A taxonomy and survey on container migration techniques in cloud computing[END_REF]) attempt summarizing the different Virtual Machine (VM) and container migration techniques and their challenges, from a general perspective, without special emphasis on their usage to support network softwarization. In addition, some surveys [START_REF] Herrera | Resource allocation in nfv:a comprehensive survey[END_REF][START_REF] Fei | Paving the way for nfv acceleration: A taxonomy, survey and future directions[END_REF][START_REF] Barakabitzea | 5g network slicing using sdn and nfv: A survey of taxonomy, architectures and future challenges[END_REF] provide a comprehensive state of the art on NFV solutions, introducing the basic concept and principles associated with the network slicing and virtualization considering in majority VM-enabled virtualization.

Among general purpose surveys, most surveys specifically deal with VMs for edge or fog [START_REF] Rejiba | A survey on mobility-induced service migration in the fog, edge and related computing paradigms[END_REF][START_REF] Taleb | On multi-access edge computing: A survey of the emerging 5g network edge cloud architecture and orchestration[END_REF][START_REF] Mao | A survey on mobile edge computing: The communication perspective[END_REF] computing. Only few recent surveys [START_REF] Wang | A survey on service migration in mobile edge computing[END_REF][START_REF] Rejiba | A survey on mobility-induced service migration in the fog, edge, and related computing paradigms[END_REF][START_REF] Singh | A taxonomy and survey on container migration techniques in cloud computing[END_REF] specifically concern the container-enabled NFV migration and each focuses either on edge [START_REF] Wang | A survey on service migration in mobile edge computing[END_REF], fog [START_REF] Rejiba | A survey on mobility-induced service migration in the fog, edge, and related computing paradigms[END_REF] or cloud migration [START_REF] Singh | A taxonomy and survey on container migration techniques in cloud computing[END_REF], With mobile edge computing [START_REF] Wang | A survey on service migration in mobile edge computing[END_REF], the user mobility is the key aspect that is considered. In [START_REF] Rejiba | A survey on mobility-induced service migration in the fog, edge, and related computing paradigms[END_REF], a detailed analysis outlines the trade-off that is made between the benefit associated with migrating a service (e.g., QoS improvement) and the related cost, which also brings out architectural design and implementation. In [START_REF] Singh | A taxonomy and survey on container migration techniques in cloud computing[END_REF], authors classify various techniques for cloud and container-based migration on the basis of the following taxonomy : (i) Architecture, (ii) Tools, (iii) Purpose, (iv) Scope, (v) Migration technique, and (vi) Evaluation.

Overall, the aforementioned surveys depict the migration strategies to adopt within a specific area of the network infrastructure (cloud versus edge, versus fog) with regards to some specific use cases (such as Connected & Autonomous Vehicles, Cloud video, Online gaming and Augmented reality). While it is essential to accurately characterize the migration strategy by taking into account the network infrastructure as a whole, that is, to identify the design rationale that needs to be made in order to migrate a containerized service on top of a shared infrastructure.

Contribution

The survey presented in this paper focuses on container migration spanning the Cloud, Fog and Edge computing levels. The main contributions are as follows: We provide for an extensive survey and classification of the placement strategies, whose goal is to identify the appropriate target server(s) to allocate the migrated service. A classification of the existing container migration techniques is proposed and reflects the way the service components hosted in a container are moved from one (or several) physical server(s) to another one(s). Subsequently, we perform a holistic review of the strategies for container migration over a geographically spanned networks (edge, fog, core and cloud levels) and we describe the frameworks and algorithms that have been used to migrate container-based services.

The organization of this paper is as follows: In Section 2, we present the motivation for container migration and the associated benefits. Placement strategies are reviewed in Section 3. A classification of container techniques is proposed in Section 4. In the subsequent Section 5, we review the container migration strategies for networks geographically spanned. Concluding remarks and research perspective are presented in Section 7.

Motivation for migration

The rise of microservice architecture amplifies the usage of containers that offer an ideal host for the small and self-contained microservices. Nowadays, network operators, cloud providers (e.g., AWS, Google) and content providers (e.g., Netflix, BBC) are adopting the microservice architectural style [START_REF] Balalaie | Microservices architecture enables devops: Migration to a cloud-native architecture[END_REF][START_REF] Sampaio | Supporting microservice evolution[END_REF] and deal with applications that may comprise even hundred or thousands of containers. Even though containers come up with the benefit of packing all the dependencies of a Network Function (NF) into a single unit, managing, deploying and migrating these containers in a large and multi-cloud infrastructure using self-made tools or scripts becomes increasingly complex and difficult to manage. In this regards, various container orchestration frameworks such as, Docker Swarm, Kubernetes and Apache Mesos provide additional support for deploying and managing a multi-tiered application as a set of containers on a cluster of nodes. In this regards, Kubernetes is characterised1 by a strong industry adoption -interested reader may refer to [START_REF] Al Jawarneh | Container orchestration engines: A thorough functional and performance comparison[END_REF][START_REF] Truyen | A comprehensive feature comparison study of open-source container orchestration frameworks[END_REF] for comparative studies.

With Kubernetes, the simplest and most straight-forward strategy to adopt for migrating stateless containers running in a pod (i.e. in a group of containers utilizing shared resources) is the following (Figure 1): a pod is recreated at the destination node; then, the controller/scheduler shifts all the requests from the old pod to the new one when this latter is in full-active stage; finally, the source pod is deleted. The container migration is facilitated because containers come Figure 1: General layout of migration in Kubernetes: container is migrated from cluster A to cluster B up with the benefit of packing all the dependencies of a NF into a single unit which is conveniently moved. Nonetheless, the orchestration of the container migration is a key challenge with large and multi-cloud infrastructure:

1. Inter-node balancing: Depending on the service dynamics, some nodes/clouds may become overloaded, provided that nodes/clouds capacity is heterogeneous. As illustration, edge and fog data centers are small comparing to cloud data centers. In order to balance the load between nodes and data centers, it is necessary to support migration.

2. Node failure and system maintenance: When a node encounters unexpected/planned failures/shutdown, the service continuity needs to be guaranteed through service replication or migration.

3. Attain the optimality of placement: Services join and leave the system of data centers while 5G users are expected to move. Containers share one set of resources (such as, CPU, RAM, disk on a physical server) that may vary over time. Depending on the placement of microservices and users, it may be relevant to move some containerized (micro)services e.g., close to the new location of the end user or far away to free resource for the incoming services.

As illustrated in Figure 2, orchestration technology is needed to continuously monitor the virtualized network (e.g., resource usage, failures and the arrival or departure of services); if necessary find the best place for the containerized micro-services (Section 3); perform the migration of the containerized (micro)services accordingly (Section 4). For these reasons, the study and encapsulation of numerous container-based approaches for placement and most importantly for migration that could fulfill these principle requirements is a major demand by business units offering services to customers.

Container placement strategies

Migration of a set of Cloud-Native Network Function (CNF)s is known to effectively bring more elasticity and scalability to (mission/latency-critical) applications. On the other hand, migration may entail service disruption and may come at the cost of intensive use of computing and communication resources, even though there is a strong practical need for migration. The service migration entail taking a decision concerning the service placement, which consists in determining whether, when and where to migrate. The service placement problem is an optimization problem that involves a balanced trade-off between the cost associated with the migration and the expected benefits.

The service placement problem (Table 1) is usually framed as a mathematical optimization (Integer Linear Programming and Mixed Integer Linear Programming), which is further solved by an optimization solver, heuristic methods or Machine Learning (ML) approaches. ILP and MILP solvers typically find nearly optimal solutions but are quite time-consuming and hence are not practically viable for large and complex problem instances.

Instead, heuristics (e.g., greedy algorithms) and meta-heuristics produce comparatively faster but sub-optimal results that usually achieve less objectives (e.g., low response time or reduced communication delay or load balancing or limited energy consumption). On the other hand, MLbased approaches (e.g., genetic algorithm, ant colony) are known to be more accurate solutions [START_REF] Alves Esteves | A heuristically assisted deep reinforcement learning approach for network slice placement[END_REF] thanks to their interactive learning and decision making abilities. [START_REF] Kaur | Latency and network aware placement for cloud-native 5g/6g services[END_REF][START_REF] Sun | Forecast-assisted nfv service chain deployment based on affiliation-aware vnf placement[END_REF][START_REF] Bhamare | Optimal virtual network function placement in multi-cloud service function chaining architecture[END_REF][START_REF] Hmaity | Virtual network function placement for resilient service chain provisioning[END_REF][START_REF] Luizelli | Piecing together the nfv provisioning puzzle: Efficient placement and chaining of virtual network functions[END_REF][START_REF] Li | Virtual network function placement considering resource optimization and sfc requests in cloud datacenter[END_REF] Mixed Interger Linear Programming (MILP) [START_REF] Hawilo | Orchestrating network function virtualization platform: Migration or re-instantiation?[END_REF][START_REF] Li | Cost-aware dynamic sfc mapping and scheduling in sdn/nfv-enabled space-air-ground integrated networks for internet of vehicles[END_REF][START_REF] Tang | Dynamic network function instance scaling based on traffic forecasting and vnf placement in operator data centers[END_REF][START_REF] Khoshkholghi | Service function chain placement for joint cost and latency optimization[END_REF][START_REF] Leivadeas | Vnf placement optimization at the edge and cloud[END_REF][START_REF] Hawilo | Network function virtualization-aware orchestrator for service function chaining placement in the cloud[END_REF] Heuristic Method [START_REF] Sun | Forecast-assisted nfv service chain deployment based on affiliation-aware vnf placement[END_REF][START_REF] Bhamare | Optimal virtual network function placement in multi-cloud service function chaining architecture[END_REF][START_REF] Luizelli | Piecing together the nfv provisioning puzzle: Efficient placement and chaining of virtual network functions[END_REF][START_REF] Li | Virtual network function placement considering resource optimization and sfc requests in cloud datacenter[END_REF][START_REF] Li | Cost-aware dynamic sfc mapping and scheduling in sdn/nfv-enabled space-air-ground integrated networks for internet of vehicles[END_REF][START_REF] Khoshkholghi | Service function chain placement for joint cost and latency optimization[END_REF][START_REF] Leivadeas | Vnf placement optimization at the edge and cloud[END_REF][START_REF] Hawilo | Network function virtualization-aware orchestrator for service function chaining placement in the cloud[END_REF]] Machine Learning (ML) [START_REF] Alves Esteves | A heuristically assisted deep reinforcement learning approach for network slice placement[END_REF][START_REF]DRL-based slice placement under realistic network load conditions[END_REF][START_REF] Pei | Optimal vnf placement via deep reinforcement learning in sdn/nfv-enabled networks[END_REF][START_REF] Quang | A deep reinforcement learning approach for vnf forwarding graph embedding[END_REF][START_REF] Manias | Machine learning for performance-aware virtual network function placement[END_REF] As detailed in Table 2, the above container placement strategies can be further categorized based on target architecture (cloud, fog, edge), type of placement (static versus dynamic), key objectives, algorithm to solve and evaluation method. In the case of static placement, an initial placement is typically proposed only once (at start). Instead, dynamic placement involves multiple reallocation decisions that are made over time. A new placement is proposed e.g., in case of overuse/under-use of computing or network resources, inflow/outflow of service instances [START_REF] Zhang | Toward online virtual network function placement in software defined networks[END_REF][START_REF] Laghrissi | A survey on the placement of virtual resources and virtual network functions[END_REF]. Contrary to the static placement which is based on initial constraints (e.g., expected delay/latency, initial bandwidth usage and initial resource availability), the dynamic placement involves a continuous monitoring of the physical resources and network to support the selection of the appropriate hosting server(s) and/or data center(s) despite changing resources/requirements. Static and dynamic placement strategies attempts to enhance the service quality and/or reduce the operational cost by means of various strategies, particularly: 1) Resource-aware placement: to avoid unwanted overuse/under-use of resources and decrease operational cost by balancing the load among distributed data-centers and hosts; 2) Latency-aware placement: to facilitate the fast inter-communication considering processing and migration delay, transmission and queuing delay; 3) Security-aware placement: to avoid the allocation on container owned by an adversary user, identifying the unexpected threat or failure and cross-container attacks. Once the placement decision has been made, migration must be carried out.

Classification of container migration

Container migration refers to the process of transferring or moving the components of a network function hosted within a container from one physical server (source node) to another one (destination node), possibly interrupting the network function operation. Network functions that are migrated are either stateless (i.e., no past data nor state needs to be persistent or stored) or stateful (i.e., the application state lasts and is stored, e.g., on disk). With stateless network function, migration is quite straightforward [START_REF] Puliafito | Container migration in the fog: A performance evaluation[END_REF] because the container operates in an isolated manner and is hence portable: the stateless container is simply re-allocated and restarted from scratch without conserving the existing state.

As depicted in Figure 3, there exists several techniques for moving the container from the source to the destination. They subdivide into cold and live migration depending on whether the containerized service should remain active and network-accessible during the whole migration.

Cold and Live Migration

There exists two ways of migrating a container: during the migration the containerized application is inactive (cold migration) or remains active (live migration).

Cold migration

This is the trivial form of migration in which the container is simply suspended and migrated between hosts. As illustrated in Figure 4, cold migration involves the freeze-transfer-resume steps: First, the container is freezed to ensure its associated state is not modifiable. Second, the dump state is transferred while the container is stopped. After the reception of the state at the destination node, the container is finally re-started and its state is resumed. Overall, cold migration involves a service downtime and thereby should be used in specific cases only, for instance when users are not using the service for a given time period or when the downtime is planned and users are informed. Live migration consists of migrating a running container without service interruption, i.e., container migrates from one node to another while it is running. The main portion of the state is transferred while the container is running; the container is stopped only during the transmission of the execution state. Therefore, service downtime is quite negligible for the end-user. Both cold and live migration entail the transfer of the original container. In practice, migrating an inactive service (cold migration) involves shutting down the running instance and thereby eliminating the need to handle the memory state. Instead, moving an active service (live migration) necessitates maintaining state consistency during the migration. In particular, in-memory state (including both kernel-internal and application-level state) should be moved in a consistent and efficient fashion. With live migration, the main concern lies in maintaining state consistency (as will be shown) while keeping to a minimum downtime (i.e. time between the container stops and resume) and total migration time (duration between when migration is initiated and when the container may be finally discarded at the source.

Handling State Consistency with Live Migration

Live migration can be approached in several ways: memory state can be sent ahead of time before the container is transferred (pre-copy) or later, i.e., after the container is transferred (post-copy) or combining the pre-copy and post-copy migration techniques (hybrid).

Pre-copy Live Migration

As shown in Figure 5, the container at source continues to run while pre-dump states are transmitted from source node to destination node. Therefore the service stays responsive during the transmission phase. At that time of copying and transferring of the pre-dump state, memory is kept modifiable at the source node. Then, the container is stopped and restarted at the destination node. The dump state and the memory content (memory pages) that have been modified are transferred. The service downtime (i.e., time between when the container is halted and resumed) is minimized because the container is stopped after the transmission of its state while the memory is also changeable.

Post-copy Live Migration

As Figure 6 depicts, the process is initiated by first halting the container at source node, the (minimal subset of) execution state is transmitted to the destination node and the container is resumed as soon as possible based on its latest execution state. Later on, the remaining state (including memory pages) is transferred to the destination node before deleting the container at source node. At the destination, if the restarted container attempts to access a memory page that is not yet available, fault page is demanded to the source node, hence causing an additional delay.

Hybrid Live Migration

As shown in Figure 7, hybrid approach advents by combining the pre-copy and post-copy migration techniques. Following the pre-copy approach, the pre-dump state is transmitted while the container is still alive at the source node. After halting the container, the full dump state (modified and execution state together) is transmitted. Then, the container is restarted using the full dump state. Final step proceeds to transfer the memory contents (faulted pages) that were caused In practice, pre-copy, post-copy or hybrid migration is performed using a snapshot/restore tool such as CRIU 2 , which has become a de facto standard to handle migration of linux container with OpenVZ, LXC, and Docker. CRIU is an open source tool that dumps the state of processes/containers into a collection of image files on disk and makes it possible to further resume an app (i.e., to restore an app) from exactly where it was suspended. Nonetheless, CRIU has some limitations. CRIU focuses on the internal state of the containerized application, which includes the states of the CPU, registers, signals and memory that are associated with the container. CRIU does not transfer any file/state across physical nodes. To this aim, complementary techniques shall be used to dispose of the files/information necessary for recovery at the destination node. In practice, files are transferred using the rsync primitive or a shared and possibly distributed file-system such as NFS, GlusterFS, or Virtuozzo3 that are used to store files and avoid transferring them.

Storage Migration

Typically, the state of a network function is local (i.e., accessed by the container by a virtual local disk) if the state is frequently accessed. For example, per-flow state (such as state for individual TCP connections) is local, as long as the traffic is distributed on a flow basis. In the container, the internal state is stored with the network function instance and thereby achieves good performance (e.g. fast read, write). Early work, e.g., [START_REF] Palkar | E2: A framework for nfv applications[END_REF], on NFV management assume that the state is internal; this assumption permits easy migration and elastic scaling of network functions. In practice, the state is then migrated as part of the container image.

Nevertheless, the transfer of the whole container file system results in a high network overload. In order to optimize and reduce the size of the container file system that is transferred from the source to the destination node, a number of works [START_REF] Ma | Second ACM/IEEE Symposium on Edge Computing (SEC)[END_REF][START_REF] Junior | Stateful container migration in geo-distributed environments[END_REF] take advantage of the layered structure of Docker. Docker storage is formed of several layers: base image layers are read-only while upper layer is read-write. Read-write layer encapsulates all the file system updates issued by the container since its creation, which encompasses (i) the files created by the containerized application as well as (ii) the files corresponding to the updated versions of the read-only layers. Thus, read-only layers can be fetched before the migration from a Docker repository (e.g., public cloud repository such as Docker Hub4 or self-hosted image hubs) while the thin top writable layer is transferred from the source to the destination node. Following, [START_REF] Junior | Stateful container migration in geo-distributed environments[END_REF] goes one step further and also checkpoint the current state of the read-write container layer, which further reduces the container's migration downtime.

Another line of research breaks the tight coupling between the NF state from the processing that network functions need to perform by externalizing the storage leveraging a resilient data store that is either central [START_REF] Kablan | Stateless network functions: Breaking the tight coupling of state and processing[END_REF][START_REF] Gember-Jacobson | Opennf: Enabling innovation in network function control[END_REF][START_REF] Ajagopalan | A. split/merge: System support for elastic execution in virtual middleboxes[END_REF] or distributed [START_REF] Woo | Elastic scaling of stateful network functions[END_REF] and that can be accessed by any NF. Nonetheless, any access (read, write, delete) to the externalized datastore involves a significant communication overhead. To reduce the communication overhead, in-memory data store is privileged in [START_REF] Woo | Elastic scaling of stateful network functions[END_REF][START_REF] Kulkarni | Reinforce: Achieving efficient failure resiliency for network function virtualization-based services[END_REF]: the state is stored in DRAM leveraging RAMCloud [START_REF] Ongaro | Fast crash recovery in ramcloud[END_REF] which corresponds to a key-value inmemory datastore with low latency access or Redis5 ).

Another approach introduced a variant of CRIU named VAS-CRIU that avoids costly file system operations that dominate the runtime costs and impact the potential benefits of manipulating in-memory process state. Contrary to CRIU that suffers from expensive filesystem write/read operations on image files containing memory pages, VAS-CRIU saves the checkpointed state in memory (as a separate snapshot address space in DRAM) rather than disk. This accelerates the snapshot/restore of address spaces by two orders of magnitude, and restore time by up to 9 times.

Applicability and Performance Evaluation

Few empirical studies evaluate the performance of container migration such as [START_REF] Puliafito | Container migration in the fog: A performance evaluation[END_REF][START_REF] Ramanathan | A comprehensive study of virtual machine and container based core network components migration in openroadm sdn-enabled network[END_REF][START_REF] Kotikalapudi | Comparing live migration between linux containers and kernel virtual machine: investigation study in terms of parameters[END_REF]. They compare the performance of various container migration techniques (e.g. cold, live migration) to that of VM migration and consider multiple virtualization platforms. First, the referred work [START_REF] Puliafito | Container migration in the fog: A performance evaluation[END_REF] analyzes the performance of cold and live -pre-copy, post-copy and hybrid -migration to identify the best techniques while transmitting stateful containers from one node to another. The comparison between cold and live migration indicates that, as expected, cold migration has the lowest total migration time and highest downtime in comparison to various live migration techniques because cold migration transmits the whole state at once after the suspension of the container at source node.

The delay associated to post-copy migration is high migration compared to that of cold migration as it passes on the faulted pages served on request from source node after resuming the container at destination node. Likewise, pre-copy migration depicts better results than post-copy migration when the network has sufficient throughput to convey changed pages quickly, which is the case if network throughput is greater than or close to the page change rate. Otherwise, precopy migration is less efficient compared to post-copy. On the other hand, the hybrid migration always involves higher migration time as it results from the combination of pre-copy and post-copy techniques.

Significantly, live migration keeps the container active during the migration process to reduce downtime and maintain responsiveness of the containerized service throughout the communication exchange. The evaluation of the downtime shows that the downtime is lower for the post-copy technique compared to the pre-copy technique and remains comparable to the hybrid technique. The evaluations concerning the amount of transferred data is also showing better results for postcopy, wherein the quantity of transferred data is always lower than for pre-copy and hybrid, but remains competitive to cold migration.

In [START_REF] Ramanathan | A comprehensive study of virtual machine and container based core network components migration in openroadm sdn-enabled network[END_REF], authors provide a detailed comparison of the performances associated with a VMenabled and container-enabled live migration supporting the functions of core network functions, including the Home Subscriber Server (HSS), Mobility Management Entity (MME), and Serving and Packet Gateway (SPGW).

First, the analysis of the migration time associated with the HSS VM is comparatively twice that of the HSS container. It takes a modest amount of additional time to complete the VM and container migration process while using a longer path. On the other hand, containers incur a higher downtime than VMs because the containerized HSS is stopped on the source host when checkpointing is initiated and is resumed only once after the complete restoration at the destination host.

Second, the MME VM has a migration time six/seven times higher than the MME container, as the network load and metadata size of the container is comparatively smaller than the VM. Therefore, the large image size and longer path clearly have an impact on the migration time of the VM. Conversely, the analysis of the container downtime shows double that of the VM because the migration process has to be stopped at the checkpoint stage and restarted only after restoration.

Finally, the SPGW VM also implies much higher migration time than the container due to the large size of the metadata for the VM. However, an interesting result can be observed: the downtime improves for the SPGW VM compared to the container migration, which was not the case with HSS and MME. During the SPGW migration, the UE recovery time is affected by the new UE connection that has to be successfully re-established by updating the sockets after the temporary failure occurred.

The work [START_REF] Kotikalapudi | Comparing live migration between linux containers and kernel virtual machine: investigation study in terms of parameters[END_REF] analyse the real-time behaviour of containers in the cloud environment, under two distinct workloads (100% and 66%) to . With regard to total migration time, downtime and disk utilization, Linux Containers (LXC) exhibits better outcomes compared to Kernel-based Virtual Machine (KVM) except for the CPU utilization which is better with KVM. In particular, the downtime of KVM is increased by 1.6 and resp. 1.75 times with the workload of 66% and resp. 100% in comparison to LXC. Similarly, the migration time of KVM is 1.35 and 1.45 times higher compared to LXC at the workload of 66% and 100% respectively. Similarly, the live migration with KVM and LXC which has an impact on on their disk utilization. The highest disk utilization of KVM is 455,555 writes/sec and 482,672 writes/sec at the workload of 66% and 100% respectively. Whereas, LXC has a maximum disk utilisation of 301,192 writes/sec and 330,528 writes/sec for a workload of 66% and 100% respectively. Moreover, the evaluations related to CPU utilization shows that LXC has a maximum CPU usage of 78.12% and 86.24% for a workload of 66% and 100% consecutively. However, KVM on the other hand performs better outcomes by lowering upto 73.09% and 74.07% for 66% and 100% workload respectively.

Strategies for container migration techniques

As shown in Figure 8, container migration schemes can be classified into three computing layers, which form the underlying virtualization infrastructure.

The topmost cloud layer constitutes the largest centralized storage and computing resource Moreover, the cloud-native principle enables network services to be implemented as a bundle of microservices interconnected to each other and deployed on distributed and container-based infrastructures (e.g., Kubernetes) [START_REF] Jay | Why use containers and cloud-native functions anyway?[END_REF] in the cloud. Nonetheless, there exists an inherent limitation associated with cloud computing: the long communication distance results in excessively long delay and the security factors in public cloud models risk the users privacy and unauthorized access to databases [START_REF] Liang | Toward migration of sgx-enabled containers[END_REF].

Fog computing provides a promising solution by decreasing the distance between end user's devices and cloud data centers. Cloud functions can be moved towards the end user device in the event of low-latency interactivity. In practice, containerized microservices migrate from centralized cloud to geo-distributed fog nodes [START_REF] Puliafito | Container migration in the fog: A performance evaluation[END_REF][START_REF] Ahmed | Fog computing applications: Taxonomy and requirements[END_REF], which share the workload and lessen the network traffic. Therefore, strategies under fog perform the migration among geo-distributed and heterogeneous data centers. In such case, careful migration of data volumes plays a significant role especially for live and stateful containers. Nonetheless, microservice requesting more computing/storage resources can be offloaded from fog nodes to cloud data centers.

Further, the edge nodes located near the end users provide comparatively lower latency at a cost of limited resource capacity in comparison to cloud and fog servers. Edge clouds enable the deployment of servers near to the user to fulfill the demand of latency-critical applications.In particular, migration techniques map/migrate the containers from one location to another depending on the user moves. That, later on optimize the quality-of-experience (QoE) and network-related requirements by dynamically mapping the containerized services on container-based virtualized environment [START_REF] Maheshwari | Traffic-aware dynamic container migration for real-time support in mobile edge clouds[END_REF].

While a cloud-fog-edge architecture has the potential to unlock tangible opportunities for industry, it remains pivotal to rely on a mature container migration strategy. In the following, we consider the migration techniques that can be followed to support the migration at any layer of the virtualization infrastructure. Table 3 compares the proposed approaches based on their migration type, architecture, scope and considered factors to be handled during migration and the detailed explanation is also provided in the proceeding section. Compared to VM Migration that has attracted considerable interest, there are not so much works that address container migration within the cloud ( § 5.1), the fog ( § 5.2) or the edge ( § 5.3).

Container Migration on Cloud

CloudHopper [START_REF] Benjaponpitak | Enabling live migration of containerized applications across clouds[END_REF] supports live migration of multiple interdependent containerized applications across multiple clouds over a wide network. The automated solution (relying on Ansible [START_REF]Ansible[END_REF]) offers multi-cloud support for three commercial clouds providers (namely, Amazon Web Services, Google Cloud Platform, and Microsoft Azure). The migration of multiple interdependent containers necessitates a network migration to (i) easily locate the other containers and (ii) hold the incoming traffic during the effective migration and eventually redirect when the service gets restored and ready. For this purpose, an IPsec VPN is set up between the source and target and a TCP/HTTP load balancer (HAProxy [START_REF]HAProxy -The Reliable, High Performance TCP/HTTP Load Balancer[END_REF]) is used and tuned to redirect the http traffic and to return unavailability message (HTTP 503 Service Unavailable Response) if timeout occurs during the migration. To support memory pre-copy, the CRIU's iterative migration capability is leveraged. Rather than supporting a parallel transfer of the multiple containers, migration is scheduled: containers are ordered by size and large-size containers are migrated first. The next container starts its migration when the previous container has a remaining transfer size that is equal to its transfer size. This scheduling approach uses more efficiently the network bandwidth and enables to start all containers almost immediately upon arrival at the target.

Further, the work [START_REF] Xu | Sledge: Towards efficient live migration of docker containers[END_REF] adopts the pre-copy algorithm for docker migration accross data centers of a cloud network. Different from VM, Docker has a layered image and Docker containers share the same OS kernel, which make live migration of a Docker container more complex as image, runtime state and context should be migrated. The migration starts by transferring the base layers of the docker image that are read-only by disconnecting the storage volume at the source and re-attaching it at the target node. Then, CRIU performs incremental memory checkpoint and supports the iterative migration of the upper layer which is read-write and thereby possibly updated during the whole migration process. The experimental results show 57% lessened total migration time, 55% lower image migration time, and 70% of downtime on average in comparison to mentioned state-of-the-art.

The work presented in [START_REF] Nadgowda | Voyager: Complete container state migration[END_REF] proposed a solution for live container migration named Voyager, which follows the design principle specified by Open Container Initiative (OCI) [START_REF]Open Container Initiative[END_REF] which is a consortium initiated by industry leaders (e.g. Docker, CoreOS) and encourages the common and open specifications of container technology. Voyager provides stateful container migration by using the CRIU-based memory migration and union mounts so as to retrieve source container data on the target node without copying container data in advance in order to minimize migration downtime. Thus, voyager support the so-called just-in-time zero-copy migration where container can restart before transmission of whole states at destination node. This allows Voyager containers to instantly restart at destination host during disk state transmission by means of on-demand copy-on-write and lazy replication.

The live migration model ESCAPE [START_REF] Azab | Toward smart moving target defense for linux container resiliency[END_REF] focuses on defense mechanisms for cloud containers by modeling the interaction between the attackers and respective victim hosts as a prey game. The container acts as a prey whose aim is to evade attacks/predator. For the checkpointing of a running containerized application while migration, the model employs an experimental version of Docker that includes the CRIU checkpoint tool. ESCAPE detects and circumvents attacks by either preventing any migration during an attack or migrating the container(s) far away from the potential attacker(s).

In [START_REF] Huang | Relocate: a container based moving target defense approach[END_REF], authors propose the frequent relocation of docker containers to reduce the impact of data leakage. Inspired by Moving target defense (MTD) technology, the approach promotes the container migration to shorten the container's life cycle and thereby guarantee the security of large-sized multi-tenant service deployment. Similarly, the defense framework introduced in [START_REF] Azab | Migrate: Towards a lightweight moving-target defense against cloud side-channels[END_REF] offers fast and high frequency migration of VMs/containers so as to obscure the migration process for the attackers. In particular, the destination hosts are chosen randomly which may degrade the performance by means of load and latency.

MigrOS [START_REF] Planeta | Migros: Transparent operating systems live migration support for containerised rdma-applications[END_REF] enables the transparent live migration of RDMA-enabled containers which require specialised circuitry of the network interface controllers (NICs) and thereby are not transparently supported so far. The OS-level migration strategy requires a modification to the RDMA protocol but still supports full backwards-compatibility and interoperability with the existing RDMA protocol, without compromising RDMA network performance. In order to evaluate the solution, the modified RDMA communication protocol has been integrated with SoftRoCE, a Linux kernel-level open-source implementation of the RoCEv2 protocol. In addition, the solution has implemented in NIC hardware.

In [START_REF] Liang | Toward migration of sgx-enabled containers[END_REF], the first migration framework of Intel Software Guard Extensions (SGX)-enabled containers is presented. SGX provides a trusted execution environment named enclave for containers. An enclave [START_REF]Intel SGX: Enclave[END_REF] corresponds to a secure separate encrypted area used by a process to store code or data. The key challenge behind migrating SGX-enabled containers relates to the SGX security model that prevents the states of the enclaves, which is encrypted, to be accessed during the migration process. In order to support the migration of the enclave, the solution encrypts the persistent data stored of the enclave using a symmetric key that is shared by the source and destination node. An empirical evaluation show that the migration of SGX-enabled containers introduce about 15% overhead. In [START_REF] Mavuş | Secure model for efficient live migration of containers[END_REF], author secure the live migration of container for both stateful and stateless applications. Application server acts as a control manager that orchestrates the migration process. Also, a secure migration path is established using SSH/SFTP that both support authentication, communication confidentiality and integrity.

Container Migration on Fog

The container migration strategy [START_REF] Junior | Stateful container migration in geo-distributed environments[END_REF] within Kubernetes for stateful services in geo-distributed fog computing environments attempts to minimize the downtime. In case of stateful migration, it is required to migrate the disk state along with the container, which is a time consuming process in large-sized and distributed migration. To address this issue, the layered structure provided by the OverlayFS file system [START_REF] Mizusawa | Performance evaluation of file operations on overlayfs[END_REF] is used to transparently snapshot the pod volumes and transfer the snapshot content prior to the actual container migration. At the source server, the snapshot content becomes read-only and a new empty read/write layer is added on top. Overall, the approach supports the check-pointing of the current state of the container layer. If needed, several snapshot transfers may be performed, which led to minimizing the container's migration downtime: experiments on a real fog computing test-bed show up to factor 4 downtime reduction during migration in comparison to a baseline with no volume checkpoint.

In [START_REF] Dupont | Edge computing in iot context: Horizontal and vertical linux container migration[END_REF], the migration framework supports both horizontal migration where containerized IoT functions are migrated from one gateway to another gateway and vertical migration in which IoT function containers are migrated from the gateway located at the edge to the Cloud. The strategy is quite straightforward: the stateless container is re-created at the target node and then deleted from the source node.

The formerly known Heptio Ark project, currently stands out as Velero [START_REF]Velero[END_REF] to leverage the migration of Kubernetes applications and their persistent volumes. Compared to existing tools, it utilizes the Kubernetes API instead of Kubernetes etcd to extract and restore the states. Which can be advantageous when users do not have access to etcd databases. The resources exposed by API servers are simple to backup and restore even for several etcd databases. Further, additional functionality of backing up and restoring of any type of Kubernetes volume is provided by activating the restic [START_REF]Restic[END_REF]. The release of Velero is available on GitHub [START_REF]Velero GitHub[END_REF].

Container Migration on Edge

The work presented in [START_REF] Ma | Efficient live migration of edge services leveraging container layered storage[END_REF] designs a third party tool to perform a live migration of services on edge infrastructure. The goal is to reduce the migration time by minimizing the transferred file size by leveraging the layer structure of the docker container storage system. As docker image is composed of layers usually emerged from Dockerfile represents a set of instructions in the image. During the container's whole life cycle, only the top storage layer is changeable. The layers underlying the top layers remain unchanged. Therefore, the proposed strategy transmits only the top layer during the migration process, rest underlying layers have been transmitted before commencing the process.

Moreover, authors consider the migration of the service to the end server located near the actively moving end-user: when a user shifts at a new location, then the offloading computation service also passes on to the edge server which is closer to the end user's new location. In order to attain the fast migration and lessen network traffic, the proposed framework already starts preparing the target edge node before the commencement of the migration process and parallelize & pipeline the following steps:

1. Parallelize the downloading of the images from a centralized registry at the target nearest edge node and pre-dump/send base memory images from source to target node while starting the container.

2. Reload the docker daemon on the target host (after halting the container at source node).

The reload can also be parallelized with the dirty memory transmission from source to target host or could be trigger just after the transmission of latest container layer. Note that container layer can be compressed before to transmit. Also, container layer compression and transmission can be pipelined. Similarly, the process of acquiring the memory difference at the target server could be pipelined.

The work [START_REF] Maheshwari | Traffic-aware dynamic container migration for real-time support in mobile edge clouds[END_REF] supports live migration based on Linux Container Hypervisor (LXD) and CRIU and introduces a novel heuristic scheme. The proposed heuristic follows these steps: First, a source node shortlists the containers that are characterised by high latency. For each high-latency container, the source node finds the neighbor node that is geographically closed and that is characterised by good resources availability (e.g. load, CPU, RAM, bandwidth) to migrate the container.

In order to perform the live migration of containers for latency critical industrial applications, the work [START_REF] Govindaraj | Container live migration for latency critical industrial applications on edge computing[END_REF] demonstrates the redundancy migration approach for edge computing. The approach skipped the stop-and-copy phase of traditional live migration that followed the snapshot and checkpointing, transmission and restoration of state image at the target node. Therefore, the key four composed phases are -1) Buffer and routing initialization phase, 2) Copy and restore phase, 3) Replay phase and 4) switch phase. This significantly minimizes the downtime by a factor of 1.8 in comparison to LXD (Linux containers Daemon) stock live migration as per the evaluation.

In [START_REF] Machen | Migrating running applications across mobile edge clouds: poster[END_REF], authors present the migration framework that follows the three-layered architecture -Base layer, Application layer and Instance layer to relocate containers or VMs across MECs. Aiming to enhance the performance by placing the service near to the user, the paper considers the stateful migration of applications and induces to minimize the overall migration time and service downtime. The following procedure is: First, the primary system configuration (guest OS, kernel, etc.) except application included base layer is transmitted on each MEC in order to avoid the transmission on each migration request. Second, the idle application and its data-included application layer is passed on when migration is triggered while keeping the service running. Then finally, the running states included instance layer is transmitted after suspending the service. Therefore, only the transmission time of the instance layer accumulated as service downtime. However, the detailed experimental explanation is not provided in the paper due to lack of space.

Another, an open-source multi-cloud and edge orchestration platform -Cloudify [START_REF][END_REF] affirms to support the pod migration without interrupting the containerized service from one node to another within the Kubernetes cluster.

The service-oriented architecture based KubeVirt [START_REF]KubeVirt[END_REF] project introduced by Red Hat enables the additional functionality to Kubernetes. It allows live migration of VM instances (acted as a pod) from one host to another host. Therefore, it could be profitable to relocate the containerized applications (running inside the VM) from one one node to another within a cluster. It's release is hosted on GitHub [START_REF]KubeVirt GitHub[END_REF].

Another prototypical implementation is also available on GitHub [START_REF]Podmigration-operator[END_REF][START_REF]Podmigration-operator[END_REF] to include the additional commands of <kubectl migrate> and <kubectl checkpoint> with the help of modified kubelet and customized container/cri. In this way, running pods can be checkpointed and migrated within a single or multi-clusters. Despite the fact that the work is considered to be a rough prototype, it is quite appreciable that contributed to the pod migration feature of stateful containers in Kubernetes.

Key Research Perspectives

Even though extensive research work has been actively proposed and improved during last decades on VMs live migration techniques. The same interest shifts towards applying these techniques on containers due to their unavoidable advantages. The aforementioned studies tried to solve some of the issues faced by container migration (OS-based virtualization) that are not concerned with VM migration (hardware-aware virtualization). Different approaches were also proposed to handle stateful & stateless container migration while reducing the migration time, downtime and size of transferred data. Still, there remain the unresolved challenges to address specific to dis-aggregated data centers, deploying and managing the chain of containerized microservices while migration along with avoiding the service disruption, drop in the QOS & disturbance of the ongoing exchange.

Current approaches consist in offloading the entire -or a portion of Docker image, preparing in advance the target node or using compression technique to tackle various migration-related KPIs. Stateful container migration requires transmitted memory states to resume the container from its suspension point at target nodes and storage data also required to transmit in case servers are located at different geographical locations as remote disk access could lead to increase the latency and transmission delay.

Existing orchestrators such as Kubernetes (K8s) [START_REF]Kubernetes[END_REF] are mostly cloud-oriented, it contains the features of horizontal scaling -means creating a set of instances of microservice based on workload or other criteria; failure recovery and service continuity. However, it is lacking the deployment of a chain of microservice across a multi-cluster network which is initiated by Edge Multi Cloud Orchestrator (EMCO) [START_REF]Edge Multi-Cluster Orchestrator (EMCO)[END_REF][START_REF]emco-base[END_REF]. Currently, it is a key demand of an orchestrator that can handle multiple clusters along with different types of clouds (such as edge/fog/core) to deploy applications for 5G and MEC services. Notably, as per the study done by Gartner [START_REF] Van Der Meulen | What edge computing means for infrastructure and operations leaders[END_REF], 75% of generated data is expected to be processed outside centralized cloud by 2025. Thus, the execution of applications on edge/fog/cloud data centers carefully placed within the network infrastructure, implies completely different settings.

EMCO is a central orchestrator that facilitates the management and deployment of geodistributed services across multiple distributed K8s clusters. It automates the life cycle (such as instantiation and termination) of composed service which is quite complex to handle for a large set of services supposed to deploy on distinct data centers across a multi-cluster network. It also supports multiple placement constraints based on affinity, anti-affinity or cost.

Nevertheless, there are still some challenges that require focus on. The design tool must be able to stick the microservices together. Also, it is hard to manage the connection alive during termination and re-establishment as a chain of microservices not only communicating with endusers but also with respective microservice that could possibly be placed on the server in different clusters.

Migration of containers is not only concerned with memory and storage migration but also needs to tackle the selection of an appropriate target host as complexity gets enlarged for multiple migrations. Containerization underlying on shared OS & some libraries and at the target-end there could be other containers running. Therefore, preparing the set of required libraries and docker image of the migrated container at the target host is a significant issue to disclose. Along with migrating the workload near to end users to meet various requirements (e.g., latency and service continuity) for the dis-aggregated fog and edge data centers that distribute and scale the workload.

Further, variant size of containerized NFs also required to examine service type while mapping -where first, the service composed of a chain of microservices must objectify the characteristics that could affect the model based on time-sensitivity, latency or load efficiency. Second, related to multiple migrations to transmit multiple container's states and memory simultaneously.

In this regard, a decentralized and multi-cloud orchestration of the migration across multiple technological domains constitutes a missing building block. The design of the model must be able to distinguish the services in order to place the particular set of microservices on distributed edge centers and others on centralized clouds. Aiming to save the resources at edge as these are the critical one. Moreover, approximating towards online strategy -where services are continuously arriving or departing the system and raising the issue of resource imbalance with uncertainty of arrival/departure time, the research must consider the problem of when to trigger the migration and selection of container to be migrated in a way to attain the lower migration rate. As migration rate directly influences the system's energy consumption. Also, it is difficult to predict an optimal placement for service in exchange with moving users. At the edge layer, the movement of a user from one geographical area to another needs to be handled by the controller that in turn takes a decision that may result in triggering a migration. Therefore, specific strategies should allow for an efficient decision which is more complex with the rise of inter-dependencies in large sets of microservices deployed on different data centers.

Conclusion

Majority of companies and open-source communities are adapting the cloud-native approaches as of their performance efficiency which further lays together on technologies -container, orchestration and microservices that are capable of providing the highly scalable, light-weighted, portable and flexible solutions. Through this work, we aimed to evaluate study analysis on the techniques focused on container migration starting from centralized followed to geo-distributed infrastructure.

The proposed taxonomy states the importance of re-allocating the containerized services in larger-cloud data centers in case of more resource requirements or placing them on latency-efficient fog/edge data centers in the event of latency-critical highly communicable application. Therefore, the development of a real-time migration model considering the telco infrastructure as a whole induces some challenges to address concerning the application downtime and migration time.

Figure 2 :

 2 Figure 2: Representation of Migration strategy with Placement

Figure 3 :

 3 Figure 3: Container migration Techniques

Figure 4 :

 4 Figure 4: Cold migration

Figure 5 :

 5 Figure 5: Pre-copy live migration

Figure 6 :

 6 Figure 6: Post-copy live migration

Figure 7 :

 7 Figure 7: Hybrid live migration

Figure 8 :

 8 Figure 8: Three-layered Cloud-Fog-Edge Infrastructure

Table 1 :

 1 Classification of placement methods

	Methods	Reference
	Interger Linear Program-
	ming (ILP)

Table 2 :

 2 Comparison of existing work related to placement of containers/instances

	Ref.	Archi.	Placement	Objectives			Algorithm	Evaluation
			Type
	[40]	Fog	Static	Response	time,	Greedy & Genetic	Comparison with 3
				Inter-container		Algorithm	approaches
				network communi-
				cation
	[41]	Fog	Static	Response time of	Ant colony opti-	Simulation
				task			mization
	[42]	Edge	Dynamic	Scheduling			Reviewed heuristic-	Case study
							based algorithms
	[43]	Cloud	Dynamic	Rebalancing, Load	Scheduling & Re-	Simulation	(with
				balancing			balancing process	real-time load)
	[44]	Cloud	Dynamic	Resource	utiliza-	Best Fit (BF), Max	Simulation	with
				tion, Number of	Fit (MF) & Ant	real-time workload
				instances			Colony Optimiza-	(compare	three
							tion based on Best	algorithms)
							Fit (ACO-BF)
	[45]	Three-	Dynamic	Resource	utiliza-	Best-fit		Use-case
		tier		tion
		(Container-
		VM-PM)
	[46]	Cloud	Dynamic	Traffic flow, Place-	One-shot, Round-	Theoretical Analy-
				ment cost,	Re-	ing and heuristic al-	sis and trace-driven
				sources			gorithm	simulations
	[47]	Cloud	Dynamic	Communication		Communication	Extensive	evalu-
				cost, Load balanc-	Aware		Worst	ation on Baidu's
				ing			Fit		Decreas-	data centers (Com-
							ing	(CA-WFD),	parison with exit-
							Sweep&Search	ing SOA strategies)
	[48]	Edge	Static	Container images'	KCBP (k-Center-	Trace-driven simu-
				retrieval time		Based		Place-	lations (Compared
							ment), KCBP-WC	with Best-Fit and
							(KCBP-Without-	Random)
							Conflict)
	[49]	Edge-	Dynamic	Service delay, Re-	Particle-swarm-	Use-case	bench-
		Fog-		source management	optimization	marking	(com-
		Cloud)					(PSO)-based meta-	parison of 4 ap-
							heuristic, Greedy	proaches)
							heuristic
	[50]	Containers	Static	Energy consump-	Improved genetic	Compared	with
		as a ser-		tion			algorithm	other 6 algorithms
		vice
		(CaaS)
	[51]	Edge-	Dynamic	Automate database	Markov	Decision	Testbed
		Fog-		container	place-	Processes (MDP)
		Cloud		ment decision
	[21]	Edge-	Static	End-to-end service	Greedy & Genetic	Evaluation of pro-
		Fog-		Latency			Algorithm	posed	strategy
		Cloud							solved	using	2
									algorithms

as pointed in the openstack annual user surveys: https://www.openstack.org/analytics

https://www.criu.org/

https://wiki.openvz.org/Virtuozzo Storage

https://hub.docker.com

https://redis.io/