Ali Akoglu
email: akoglu@arizona.edu

Genoveva Vargas

Genoveva Vargas-Solar
email: genoveva.vargas-solar@liris.cnrs.fr

Putting Data Science Pipelines on the Edge

Keywords: Disaggregated Data Centers, Data Science Pipelines, Edge Computing

published or not. The documents may come

Introduction

Data infrastructures such as Google, Amazon, eBay, and E-Trade are powered by data centers (DCs) with tens to hundreds of thousands of computers and storage devices running complex software applications. Since 2018, the word-wide spending in big data solutions and public cloud infrastructure has increased by 12% and 24%, respectively [START_REF]IDC: Forecasts Revenues for Big Data and Business Analytics Solutions[END_REF]. Existing IT architectures are not designed to provide an agile infrastructure to keep up with the rapidly evolving next-generation mobile, big data, and data science pipelines demands. These applications are distinct from the "traditional" enterprise ones because of their size, dynamic behavior, and nonlinear scaling and relatively unpredictable growth as inputs being processed. Thus, they require continuous provisioning and re-provisioning of DC resources [START_REF] Chen | Energyqare: Qosaware data center participation in smart grid regulation service reserve provision[END_REF][START_REF] Kannan | Grandslam: Guaranteeing slas for jobs in microservices execution frameworks[END_REF][START_REF] Xu | EnReal: An energy-aware resource allocation method for scientific workflow executions in cloud environment[END_REF] given their dynamic and unpredictable changes in the Service Level Objectives (SLOs) (e.g., availability response time, reliability, energy). Computer manufacturers, software developers, and service providers cannot cope with the dynamic and continuous changes in applications and workload types. Consequently, their offered services become unstable, fragile, and cannot guarantee the required SLOs for other application classes.

Authors list is given in alphabetical order.

This paper targets the execution of data science (DS) pipelines 3 supported by data processing, transmission and sharing across several resources executing greedy processes. Current data science pipelines environments promote highperformance cloud platforms as backend support for completely externalising their execution. These platforms provide various infrastructure services with compute resources such as general purpose processors (GPP), Graphics Processing Units (GPUs), Field Programmable Gate Arrays (FPGAs) and Tensor Processing Unit (TPU) coupled with platform and software services to design, run and maintain DS pipelines. These one-fits-all solutions impose the complete externalisation of data pipeline tasks that assume (i) reliable and completely available network connection; (ii) can be energy and economically consuming, allocating large scale resources for executing pipelines tasks. However, some tasks can be executed in the edge, and the backend can provide just in time resources to ensure ad-hoc and elastic execution environments.

Our research investigates architectural support, system performance metrics, resource management algorithms, and modeling techniques to enable the design of composable (disaggregated) DCs. The goal is to design an innovative composable "Just in Time Architecture" for configuring DCs for Data Science Pipelines (JITA-4DS) and associated resource management techniques. DCs utilize a set of flexible building blocks that can be dynamically and automatically assembled and re-assembled to meet the dynamic changes in workload's Service Level Objectives (SLO) of current and future DC applications. DCs under our approach are composable based on vertical integration of the application, middleware/operating system, and hardware layers customized dynamically to meet application SLO (application-aware management). Thus, DCs configured using JITA-4DS provide ad-hoc environments efficiently and effectively meeting the continuous changes in requirements of data-driven applications or workloads (e.g., data science pipelines). To assess disaggregated DC's, we study how to model and validate their performance in large-scale settings. We rely on novel model-driven resource management heuristics based on metrics that measure a service's value for achieving a balance between competing goals (e.g., completion time and energy consumption). Initially, we propose a hierarchical modeling approach that integrates simulation tools and models.

The remainder of the paper is organised as follows. Section 2 discusses related work identifying certain drawbacks and issues we believe still remain open. Section 3 JITA-4DS the just in time edge based data science pipeline execution environment proposed in this paper. Section 4 describes preliminary results regarding JITA-4DS. Finally Section 5 concludes the paper and discusses future work.

Related Work

The work introduced in this paper is related to two types of approaches: (i) disaggregated data centers willing to propose alternatives to one fits all architectures; and (ii) data science pipelines' execution platforms relying on cloud services for running greedy data analytics tasks.

Disaggregated data centers Disaggregation of IT resources has been proposed as an alternative configuration for data centers. Compared to the monolithic server approach, in a disaggregated data center, CPU, memory and storage are separate resource blades interconnected via a network. The critical enabler for the disaggregated data center is the network and management software to create the logical connection of the resources needed by an application [START_REF] Papaioannou | The benefits of a disaggregated data centre: A resource allocation approach[END_REF]. The industry has started to introduce systems that support a limited disaggregation capability. For example, the Synergy system by Hewlett Packard Enterprise (HPE) [1], and the Unified Computing System (UCS) [START_REF]Unified Computing[END_REF] M series servers by Cisco are two commercial examples of composable infrastructures. [START_REF] Papaioannou | The benefits of a disaggregated data centre: A resource allocation approach[END_REF] proposes a disaggregated data center network architecture, with a scheduling algorithm designed for disaggregated computing.

Data Science Environments Data analytics stacks environments provide the underlying infrastructure for managing data, implementing data processing workflows to transform them, and executing data analytics operations (statistics, data mining, knowledge discovery, computational science processes). For example, the Berkeley Data Analytics Stack (BDAS) from the AMPLAb project is a multi-layered architecture that provides tools for virtualizing resources, addressing storage, data processing and querying as underlying tools for big dataaware applications. AsterixDB from the Asterix project is a full-fledged big data stack designed as a scalable, open-source Big Data Management System (BDMS https://asterixdb.apache.org).

Cloud based Data Science Environments provide tools to explore, engineer and analyse data collections. They are notebook oriented environments 4 externalised on the cloud. They provide data labs and environments with libraries for defining and executing notebooks. Examples of existing data labs are Kaggle and CoLab from Google, and Azure Notebooks from Microsoft Azure. Platforms for custom modelling provide a suite of machine learning tools allowing developers with little experience to train high quality models. Tools are provided as services by commercial cloud providers that include storage, computing support and environments for training and enacting greedy artificial intelligence (AI) models. The main vendors providing this kind of platforms are Amazon Sage Maker, Azure ML Services, Google ML Engine and IBM Watson ML Studio. Machine Learning and Artificial Intelligence Studios give an interactive, visual workspace to build, test, and iterate on analytics models and develop experi-ments 5 . An experiment has at least associated one data set and one module. Data sets may be connected only to modules, and modules may be connected to either data sets or other modules. All input ports for modules must have some connection to the data flow. All required parameters for each module must be set. Machine learning runtime environments provide the tools needed for executing machine learning workflows, including data stores, interpreters and runtime services like Spark, Tensorflow and Caffe for executing analytics operations and models. The most prominent studios are, for example, Amazon Machine Learning, Microsoft Artificial Intelligence and Machine Learning Studio, Cloud Auto ML, Data Bricks ML Flow and IBM Watson ML Builder.

Discussion Data Analytics Stacks remain general solutions provided as "onefits-all" systems with which, of course, big data can be managed and queried through built-in or user-defined operations integrated into imperative or SQL like solutions. In contrast, data labs' objective is to provide tools for managing and curating data collections, automatically generating qualitative and quantitative meta-data. Curated data collections associated with a search engine can be shared and used in target data science projects. Data labs offer storage space often provided by a cloud vendor (e.g., users of CoLab use their google drive storage space for data collections, notebooks and results). Execution environments associate computing resources for executing notebooks that use curated data collections. Machine learning studios address the analytics and data management divide with integrated backends for dealing with efficient execution of analytics activities pipelines allocating the necessary infrastructure (CPU, FPGA, GPU, TPU) and platform (Spark, Tensorflow) services.

These environments provide resources (CPU, storage and main memory) for executing data science tasks. These tasks are repetitive, process different amounts of data and require storage and computing support. Data science projects have life cycle phases that imply in-house small scale execution environments, and they can evolve into deployment phases where they can touch the cloud and the edge resources. Therefore, they require underlying elastic architectures that can provide resources at different scales. Disaggregated data centers solutions seem promising for them. Our work addresses the challenges implied when coupling disaggregated solutions with data science projects.

JITA-4DS: Just in time Edge Based Data Science Pipelines Execution

The Just in Time Architecture for Data Science Pipelines (JITA-4DS), illustrated in Figure 1, is a cross-layer management system that is aware of both the application characteristics and the underlying infrastructures to break the barriers between applications, middleware/operating system, and hardware layers. Vertical integration of these layers is needed for building a customizable Virtual Data Center (VDC) to meet the dynamically changing data science pipelines' requirements such as performance, availability, and energy consumption. JITA-4DS fully exploits the virtualization from the virtual machine (VM) level into the VDC level (e.g., fine-grain resource monitoring and control capabilities). For the execution of data science pipelines, JITA-4DS can build a VDC that can meet the application SLO, such as execution performance and energy consumption. The selected VDC, then, is mapped to a set of heterogeneous computing nodes such as GPPs, GPUs, TPUs, special-purpose units (SPUs) such as ASICs and FPGAs, along with memory and storage units.

DS pipelines running on top of JITA-4DS VDC's apply sets of big data processing operators to stored data and streams produced by the Internet of Things (IoT) farms (see the upper part of Figure 1). In the JITA-4DS approach, the tasks composing a data science pipeline are executed by services that implement big data operators. The objective is to execute as just in time edge-based processes (similar to lambda functions), and they interact with the VDC underlying services only when the process to execute needs more resources. This means that services are running on the edge, on processing entities with different computing and storage capacities. They can totally or partially execute their tasks on the edge and/or on the VDC. This in turn, creates the need for novel resource management approaches in streaming-based data science pipelines. These approaches should support and satisfy the data management strategy and stream exchange model between producers and consumers, invoke tasks with the underlying exchange model constraints on the compute and storage resources in the suitable form and modality and meet multi-objective competing performance goals. Next we describe the architecture of big data operators and we show how Fig. 2. Architecture of a Big data/stream processing service they interact with the VDC. Later we will introduce our resource management approach for JITA-4DS.

Big Data/Stream producing and processing services We assume that services that run on the edge produce and process data in batch or as streams. Data and stream processing services implement operators for supporting the analysis (machine learning, statistics, aggregation, AI) and visualization of big data/streams produced in IoT environments. As shown in Figure 1, data and stream producing services residing on edge rely on underlying message-based communication layers for transmitting them to processing and storage services. These services can reside on edge or a VDC. A data/stream service implements simple or complex analytics big data operations (e.g., fetch, sliding window, average, etc.). Figure 2 shows the general architecture of a stream service.

The service logic is based on a scheduler that ensures the recurrence rate in which the analytics operation implemented by the service is executed. Stream/data processing is based on unlimited consumption of data ensured by the component Fetch that works if streams are notified by a producer. This specification is contained in the logic of the components OperatorLogic and Fetch. As data is produced, the service fetches and copies the data to an internal buffer. Then, depending on its logic, it applies a processing algorithm and sends the data to the services connected to it. The general architecture of a service is specialized in concrete services implementing the most popular aggregation operations. These services can process data and streams on edge or on a VDC.

Since RAM assigned to a service might be limited, and in consequence its buffer, every service implements a data management strategy by collaborating with the communication middleware and with the VDC storage services to exploit buffer space, avoiding losing data, and processing and generating results on time. Big stream/data operators combine stream processing and storage techniques tuned depending on the number of things producing streams, the pace at which they produce them, and the physical computing resources available for processing them on-line (on edge and VDC) and delivering them to consumers (other services). Stores are distributively installed on edge and on the VDC. Edge based Data Science (DS) Pipelines are expressed by a series of data processing operations applied to streams/data stemming from things, stores or services. A DS pipeline is implemented by mashing up services implementing operators based on a composition operation that connects them by expressing a data flow (IN/OUT data). Aggregation (min, max, mean) and analytics (k-means, linear regression, CNN) services can be composed with temporal windowing services (landmark, sliding) that receive input data from storage support or a continuous data producer, for instance, a thing. The connectors are Fetch, and Sink services that determine the way services exchange data from/to things, storage systems, or other services (on-demand or continuous). Services can be hybrid (edge and VDC) services depending on the number of underlying services (computing, memory, storage) required. To illustrate the use of a JITA-4DS, we introduce next a use case that makes full use of edge and VDC services configured ad-hoc for the analysis requirements. Use Case: Analysing the connectivity of a connected society The experiment scenario aims at analyzing the connectivity of the connected society. The data set used was produced in the context of the Neubot project 6 . It consists of network tests (e.g., download/upload speed over HTTP) realized by different users in different locations using and application that measures the network service quality delivered by different Internet connection types 7 . The type of queries implemented as data science pipelines were the following:

EVERY 60 seconds compute the max value of download_speed of the last 3 minutes FROM cassandra database neubot series speedtests and streaming RabbitMQ queue neubotspeed EVERY 5 minutes compute the mean of the download_speed of the last 120 days FROM cassandra database neubot series speedtests and streaming rabbitmq queue neubotspeed For deploying our experiment, we built an IoT farm and implemented a distributed version of the IoT environment on a clustered version of RabbitMQ. This setting enabled to address a scaleup setting in terms of several data producers (things) deployed on edge. We installed aggregation operators as services distributed on the things and an edged execution environment deployed on servers deployed in different devices. The challenge is to consume streams and create a history of connectivity information and then combine these voluminous histories with new streams for answering the queries. Depending on the observation window size, the services access the observations stored as post-mortem data sets from stores at the VDC level and connect to on-line producers that are currently observing their connections (on edge). For example, the second query observes a window of 10 days size. Our services could deal with histories produced in windows of size 10 days or even 120 days. Such massive histories could be combined with recent streams and produce results at reasonable response times (order of seconds).

Preliminary experimental results

Value of Service based Scheduling and Resource Management

JITA-4DS encourages a novel resource management methodology that is based on the time-dependent Value of Service (VoS) metric [START_REF] Kumbhare | Value based scheduling for oversubscribed power-constrained homogeneous hpc systems[END_REF] to guide the assignment of resources to each VDC and achieve a balance between goals that usually compete with each other (e.g., completion time and energy consumption). VoS allows considering the relative importance of the competing goals, the submission time of the task (e.g., peak vs non-peak period), or the task's nature as a function of task completion time. A primary difference of our VoS metric from earlier studies on "utility functions" (e.g., [START_REF] Khemka | Utility maximizing dynamic resource management in an oversubscribed energy-constrained heterogeneous computing system[END_REF][START_REF] Briceno | Time utility functions for modeling and evaluating resource allocations in a heterogeneous computing system[END_REF][START_REF] Kargahi | Performance optimization based on analytical modeling in a real-time system with constrained time/utility functions[END_REF]) is the fact that we combine multiple competing objectives and we consider the temporal value of performing resource management at a given instant of time. This ability is crucial for meeting the SLO of edge-based data science pipeline execution, where the nature and amount of the data change dynamically among streams of data arriving from heterogeneous and numerous sets of edge devices. In our earlier work [START_REF] Machovec | Utility-based resource management in an oversubscribed energy-constrained heterogeneous environment executing parallel applications[END_REF], we defined the value for a task as a monotonically-decreasing function of an objective (i.e., completion time, energy), illustrated in Figure 3. The soft threshold parameter specifies the limit on an objective (T h sof t ) until which the value earned by a task is maximum (v max ). Beyond the soft threshold, the value starts decreasing until the objective reaches the hard threshold. The hard threshold (T h hard ) specifies a limit on a given objective (v min ), beyond which zero value is earned. The linear rate of decay between v max and v min can be replaced by other functions if shown to provide an accurate representation.

In Equation 1, Task value (V (T ask j , t)) represents the total value earned by completing task j during a given period t. It is the weighted sum of earned performance and energy values based on Figure 3. The w p and w e coefficients are used for adjusting the weight given to the performance and energy values. The importance factor γ (T askj ) expresses the relative importance among tasks. If either the performance function or energy function is 0, then the VoS is 0.

V (T ask j , t) = (γ (T askj ) )(w p * v p (T ask j , t) + w e * v e (T ask j , t))

(

The VoS function, defined by Equation 2, is the total value gained by n tasks in the workload that are completed during a given time period t.

V oS(t) = n j=1 V (T ask j , t) (2)
The design of resource management heuristics for the JITA-4DS is a challenging problem. Resources must be interconnected and assigned to VDCs in a way that it will maximize the overall system VoS as defined in Equation 2. Towards this goal, we designed a simulation environment and evaluated various heuristics by experiments for a homogeneous environment much simpler than the JITA-4DS design illustrated in Figure 1. In the simplified environment, we studied the allocation of only homogeneous cores and memories from a fixed set of available resources to VMs, where each VM was for a single dynamically arriving task. Each task was associated with a task type, which has estimated execution time and energy consumption characteristics (through historical information or experiments) for a given number of assigned cores and assigned memory. To predict each application type's execution time and energy consumption, we use statistical and data mining techniques [START_REF] Kumbhare | Value based scheduling for oversubscribed power-constrained homogeneous hpc systems[END_REF][START_REF] Kumbhare | Dynamic power management for value-oriented schedulers in power-constrained hpc system[END_REF][START_REF] Kumbhare | A value-oriented job scheduling approach for power-constrained and oversubscribed hpc systems[END_REF], which represent the execution time and energy consumption as a function of the VDC resources. As an example, one of the heuristics was Maximum Value-per-Total Resources (Maximum VPTR). Its objective function is "task value earned / total amount of resources allocated," where the total amount of resources (TaR) for a task depends on the task execution time duration (TeD), the percentage of the total number of system cores (Cores) used and the percentage of the total system RAM used:

T aR = T eD × [(%Cores) + (%RAM )] (3)
We compare the VPTR (Value Per Total Resources) algorithm with a simple scheduling algorithm when applied to a workload that starts during peak usage time. For 80 cores, VPTR is able to have an improvement of almost 50% in energy value and 40% in performance value as shown in Figures 4 (a) and 4 (b), respectively. Figure 4 (c) shows the VoS when we combine both performance and energy values. Because the workload involves a peak system period, the Simple heuristic cannot efficiently utilise the resources, resulting in VPTR having up to 71% increase in normalized VoS.

In general, each of these percentages can be weighted for a given system, depending on factors such as the relative cost of cores and memory or relative demand for cores versus memory among applications in typical workloads. Each time the heuristic is invoked, it considers tasks that have arrived in the system but have not been assigned resources. For each task, independently, for each of its allowable resource configurations that will provide non-zero task value, we find the cores/memory that maximizes the task's VPTR. Then, among all mappable tasks, we select the task with the maximum VPTR, call this task m, and make that assignment of resources to task m, or if the allocation is for a future time, we create a place-holder for task m. We then update the system state information based on this assignment or place-holder and remove task m from the list of mappable tasks.

JITA-4DS Simulation

Because a composable system is a tightly integrated system, it will create new challenges in modeling application performance and designing resource allocation policies to maximize productivity. This creates the necessity for creating a simulation environment to experiment and evaluate composable systems and use them to design new resource allocation policies. We validated our simulation framework's functionality against emulation-based experiments based on a system composed of 64 nodes. Each compute node is a dual-socket with 125 GB of memory and InfiniBand QDR for the network interface. Each socket on the node has a CPU (Ivy Bridge-EP) with twelve cores and a maximum frequency of 2.40 GHz. The TDP of each CPU is 115 watts in our system. We access the powerspecific registers on each CPU to monitor and control the CPU power consumption. On this system, we collect empirical power-performance profiles for our test applications and create models. We use the publicly available benchmarks from the NAS parallel benchmark suite (NPB) [START_REF]NAS-NPB[END_REF]. In our study, a synthetic workload trace is a list of jobs in the order of arrival time. We use the benchmarks listed in Table 1 to form a job in the workload trace. Each job entry in the workload trace consists of job arrival time, job name, maximum job-value, job's input problem size, iteration count, node configuration range, soft threshold, and hard threshold. We experimentally select the sampling range for these parameters to ensure our unconstrained HPC system is oversubscribed. We create offline models for each hybrid benchmark by using the modeling technique discussed in [START_REF] Kumbhare | Dynamic power management for value-oriented schedulers in power-constrained hpc system[END_REF]. Similar to our emulation HPC prototype, we simulate a system composed of 64 nodes. We use a similar set of hybrid benchmarks and their models to create workload traces. For the simulation study, we create 50 workload traces, and each trace is composed of 1000 jobs in the order of their arrival time. Each trace simulates a workload of approximately 50 hours at three different systemwide power constraints (55%, 70%, and 85%). While simulating a workload trace under a given power constraint, we assume our model prediction error is zero for each benchmark. In Figures 5(a) and 5(b), we compare the system-value earning distribution from our emulation and simulation studies respectively using a set of value-based heuristics covering Value-Per-Time (baseline-VPT) and its variations [START_REF] Kumbhare | Dynamic power management for value-oriented schedulers in power-constrained hpc system[END_REF][START_REF] Kumbhare | A value-oriented job scheduling approach for power-constrained and oversubscribed hpc systems[END_REF] Common Power Capping (VPT-CPC), Job-Specific Power Capping (VPT-JSPC), and hybrid that combines VPT-CPC and VPT-JSPC. Even though our power-aware algorithms' normalised system-value earnings is higher in the simulations compared to the emulations, we observe a similarity in the pattern of the system-value earnings of the algorithms as the power constraint is relaxed from 55% to 85%. The reason for differences in the magnitudes of system-value earnings can be attributed to our simulation study, assuming all the system's CPUs are identical.

Discussion

In the JITA-4DS environment, the resource management problem is much more complex and requires the design of new heuristics. The computing resources allocated to the VDC for a given class of applications are a heterogeneous mixture of different types of processing devices (different CPUs, different GPUs, different accelerators, etc.) with various execution performance and energy consumption characteristics. They depend on each of the specific applications being executed by that VDC. For example, several aspects remain open, like the adhoc design of the JITA-4DS resource management system for a VDC built from a fixed set of components. The design of a JITA-4DS instance is determined by the execution time and energy consumption cost, and resources requirements of a data science pipeline. Therefore, it is necessary to identify the configuration choices for a given pipeline and define VDC resources' effective resource allocation strategies dynamically. In general, for determining the dynamic resources requirements of data science pipelines at runtime, it is necessary to consider two challenges. First, calculate a VDC-wide VoS for a given interval of time, weigh individual values of various instances of pipelines. Second, propose objective functions that can guide heuristics to operate in the large search space of resource configurations. The objective is to derive possible adequate allocations of the shared and fixed set of VDC resources for several instances of data science pipelines.

We have observed that regarding the resource management system for JITA-4DS, decisions must be made to divide the shared, fixed resource pool across different VDCs to maximize the overall system-wide VoS. All of the above single VDC challenges still apply and interact across VDCs. Additional problems, such as determining when resources should be reallocated across VDCs and do so in an on-line fashion, must be addressed. This includes the methodologies for reassigning resources that do not interfere with currently executing applications on different VDCs affected by the changes and measuring and accounting for the overhead of the establishment of new VDC configurations.

Conclusion and Future Work

This paper introduced JITA-4DS, a virtualised architecture that provides a disaggregated data center solution ad-hoc for executing DS pipelines requiring elastic access to resources. DS pipelines process big streams and data coordinating operators implemented by services deployed on edge. Given that operators can implement greedy tasks with computing and storage requirement beyond those residing on edge, they interact with VDC services. We have set the first simulation setting to study resources delivery in JITA-4DS.

We are currently addressing challenges of VDCs management on simpler environments, on cloud resource management heuristics, big data analysis, and data mining for performance prediction. To simulate, evaluate, analyze, and compare different heuristics we will build simulators for simpler environments and combine open source simulators for different levels of the JITA-4DS hierarchy.

Fig. 1 .

 1 Fig. 1. Just in Time Architecture for Data Science Pipelines -JITA-4DS

Fig. 3 .

 3 Fig. 3. General formulation for value vs. objective and thresholds.

Fig. 4 .

 4 Fig. 4. Value gain comparison of the VPTR over the simple heuristicTable1. NAS benchmarks used in this study.

Fig. 5 .

 5 Fig. 5. Comparing the simulation and emulation results.

Table 1 .

 1 NAS benchmarks used in this study.

	Benchmark	Description	MPI MPI+OpenMP
	CG	conjugate gradient
	EP	embarrassingly parallel
	F T	Fourier transform
	IS	integer Sort
	M G	multi-grid
	LU	lower-upper Gauss-Seidel solver
	BT	block tri-diagonal solver
	SP	scalar penta-diagonal solver

A Data Science Pipeline consists of a set of data processing tasks organised as a data flow defining the data dependencies among the tasks and a control flow defining the order in which tasks are executed.

A notebook is a JSON document, following a versioned schema, and containing an ordered list of input/output cells which can contain code, text (using Markdown mathematics, plots and rich media.

An experiment consists of data sets that provide data to analytical modules, connected together to construct an analysis model.

Neubot is a project on measuring the Internet from the edges by the Nexa Center for Internet and Society at Politecnico di Torino (https://www.neubot.org/)

The Neubot data collection was previously used in the context of the FP7 project S2EUNET.


Zihao Jian
email: zjian3@ualberta.ca

Minshan Xie
email: minshan1@ualberta.ca

Realistic face animation generation from videos

Keywords: 3D Face Reconstruction, Face Alignment, Face Animation Generation, Deep Learning

3D face reconstruction and face alignment are two fundamental and highly related topics in computer vision. Recently, some works start to use deep learning models to estimate the 3DMM coefficients to reconstruct 3D face geometry. However, the performance is restricted due to the limitation of the predefined face templates. To address this problem, some end-toend methods, which can completely bypass the calculation of 3DMM coefficients, are proposed and attract much attention. In this report, we introduce and analyse three state-of-the-art methods in 3D face reconstruction and face alignment. Some potential improvement on PRN are proposed to further enhance its accuracy and speed.

I. INTRODUCTION

3D face reconstruction has attracted much attention in the academic research field. Compared with 2D face images, 3D faces can provide much more space information from different angels. Therefore, using 3D face models as dataset in face recognition typically contributes to higher accuracy and better robustness than using 2D images. Besides, 3D face models are also widely utilized in many different fields such as animation [START_REF] Cao | 3d shape regression for real-time facial animation[END_REF], image restoration [START_REF] Yang | Structured face hallucination[END_REF], anti-spoofing [START_REF] Wang | Deep spatial gradient and temporal depth learning for face antispoofing[END_REF], and tracking [START_REF] Xiong | Global supervised descent method[END_REF]. However, it is still an open problem to apply 3D face models in these fields due to the scarce data of 3D faces with high resolution. Consequently, how to better reconstruct 3D face models from 2D face images has become an extremely important and popular research topic.

The existing methods for 3D face reconstruction can be generally divided into 3 categories: 3D modeling methods based on images processing, pre-defined model based methods and end-to-end deep learning methods. The traditional 3D face reconstruction are mostly based on images features like brightness and edges. In comparison, model based 3D face reconstruction is currently getting more popular. In this method, 3D models are mainly represented by triangular meshes or point clouds. For example, the 3D Morphable Model [START_REF] Blanz | A morphable model for the synthesis of 3d faces[END_REF] and CANDIDE-3 [START_REF] Ahlberg | Candide-3 -an updated parameterised face[END_REF] are the one of the most common used 3D face models in research field.

However, the above-mentioned methods are faced with some challenges. For instance, facial landmarks may become invisible due to self-occlusion when faces deviate from the frontal view. Additionally, traditional nonlinear models or cascaded linear regression are not complicated enough to cover all the facial features.

In recent years, some works started to apply deep learning network to estimate the 3DMM parameters of face models. However, the performance is restricted due to the limitation of the pre-defined face model. To avoid the limitation from certain face models, end-to-end 3D face reconstruction methods [START_REF] Jourabloo | Large-pose face alignment via cnn-based dense 3d model fitting[END_REF]- [START_REF] Tuan Tran | Regressing robust and discriminative 3d morphable models with a very deep neural network[END_REF] are proposed to get their own face models representation in the process without using pre-defined models.

II. RELATED WORK

3D Morphable Model (3DMM) is a widely used parameterized face model in 3D face reconstruction. In 3DMM, model features like facial shape, texture, camera position and light intensity are all determined by its coefficients. By calculating these coefficients, new face model can be automatically registered to a pre-defined face template [START_REF] Blanz | A morphable model for the synthesis of 3d faces[END_REF]. Early methods tend to first establish the correspondences of the special points (feature points [START_REF] Huber | Fitting 3d morphable face models using local features[END_REF] and landmarks [START_REF] Huber | A multiresolution 3d morphable face model and fitting framework[END_REF]) between input images and the output 3D face model. Then, the non-linear optimization function is solved to regress the 3DMM coefficients. However, these methods highly rely on feature or landmark detectors which are often unstable. In recent years, some literature start to use Convolutional Neural Networks (CNNs) to estimate the 3DMM coefficients [START_REF] Jourabloo | Large-pose face alignment via cnn-based dense 3d model fitting[END_REF]- [START_REF] Tuan Tran | Regressing robust and discriminative 3d morphable models with a very deep neural network[END_REF], which simplifies the process of calculation. However, the performance of 3DMM-based methods is restricted due to the limitation of the pre-defined face models [START_REF] Feng | Joint 3d face reconstruction and dense alignment with position map regression network[END_REF].

To address the aforementioned issues, some literature start to build end-to-end models that take still 2D image as input and output the 3D face model directly. In [START_REF] Jackson | Large pose 3d face reconstruction from a single image via direct volumetric cnn regression[END_REF], the authors designed the volumetric representation of the 3D facial geometry, based on which the Volumetric Regression Network (VRN) is proposed to make spatial predictions at a voxel level. This network accepts an RGB image as input and directly regresses a 3D volume which can converted to a 3D face model. It has been proven that the proposed VRN outperforms all of the 3DMM-based methods and is able to deal with facial expressions as well as occlusions. But the VRN outputs a volume with the dimensions of 192x192x200, which severely limits its computational efficiency. In [START_REF] Feng | Joint 3d face reconstruction and dense alignment with position map regression network[END_REF], a novel UV position map is designed to records the 3D shape of human face in UV space. Additionally, in order to keep the semantic meaning of points within UV position map, 3DMM-based UV coordinates are created. Afterwards, Position map Regression Network (PRN) is proposed to regress the 3D full structure. This network takes 2D facial image as well as the ground-truth 3D shapes as input and directly outputs the corresponding 3D face model.

Although both VRN and PRN achieve state-of-the-art performance in 3D face reconstruction, the resolution of reconstructed model largely depends on the size of the feature map. Therefore, in order to obtain a high-resolution face model, a large feature map has to be used, resulting in high memory cost and low speed.

Compared with the aforementioned dense vertices methods which are typically based on fully connected network, the regression of 3DMM parameters with low dimension and low redundancy usually consumes far less time and memory. Starting from this point, Guo, X. Zhu, Y. Yang, F. Yang, Z. Lei, and S. Z. Li proposed 3DDFA-V2 [START_REF] Guo | Towards fast, accurate and stable 3d dense face alignment[END_REF] to regress the parameters of 3DMM with a lightweight network. In the 3DDFA-V2 architecture, a meta-joint optimization strategy is utilized to dynamically regress a small set of 3DMM parameters which greatly enhances both speed and accuracy. More importantly, the core work of the novel 3DDFA-V2 method is the 3D aided short video synthesis which can simulate the face movement in and out of the plane and convert a still image into short video to improve the stability. Additionally, the proposed 3DDFA-V2 also impose an extra landmark-regression regularization to facilitate the parameters regression. With these three newly designed components, the 3DDFA-V2 has achieved high accuracy, fast speed and stability simultaneously, which is a milestone in the field of 3D face reconstruction and face alignment.

III. LITERATURE REVIEW

A. Eye tracking and animation for MPEG-4 coding

In 1998, N. Rossol, I. Cheng, W. F. Bischof, and A. Basu proposed an improved facial feature detection and tracking algorithms based on simple heuristics, and used the algorithm to build a model that presented the captured eye movement. [START_REF] Bernogger | Eye tracking and animation for mpeg-4 coding[END_REF]. The proposed approach first uses Hough transformation and deformable template matching, as well as color information to extract the eyes from a set of images sequence. Then, these extracted eye features are used to reconstruct the eye motions on a 3D facial model. From the experiment results, the proposed algorithms successfully synthesized highquality eye movements, but the time complexity also increased significantly. In the future, the extensions of rebuilding lip movements and network strategies for real-time modelling will be discussed.

B. QoE-Based Multi-Exposure Fusion in Hierarchical Multivariate Gaussian CRF

In the paper QoE-Based Multi-Exposure Fusion in Hierarchical Multivariate Gaussian CRF [START_REF] Shen | Qoe-based multi-exposure fusion in hierarchical multivariate gaussian crf[END_REF], R. Shen, I. Cheng, and A. Basu proposed a novel image fusion algorithm based on perceptual quality measures. To specify, perceived local contrast and color saturation are the measures used in this paper. Generally, perceived local contrast stands for the perception of local luminance variations of the surrounding luminance. However, there are many methods to measure local contrast. In order to deal with the noise in under-exposed regions, a threshold is introduced so that the under-exposed area will be brighter and preserves more details while the over-exposed area will be less vivid. Nevertheless, local contrast measure only works in the luminance channel so we need to use color saturation to supply it, especially when objects are captured at proper exposures and they have already shown saturated colors.

According to the experiment result, the fusion algorithm based on perceptual quality measures outperforms other methods, but the exploration of potential perceptual quality measures remains an open issue. In the future, we can try other measures that can be used in multi-exposure fusion.

C. A framework for adaptive training and games in virtual reality rehabilitation environments

In 2011, N. Rossol, I. Cheng, W. F. Bischof, and A. Basu proposed a framework for adaptive training and games in virtual reality rehabilitation environments [START_REF] Rossol | A framework for adaptive training and games in virtual reality rehabilitation environments[END_REF]. They built a customizable VR rehabilitation environment where clinicians are able to design, build, and customize their indoor training environments for the patients. Moreover, since Bayesian network can determine patient's skill level, the environment can dynamically respond to user actions and change itself correspondingly. According to the experiment result, participants who used the proposed VR system accomplished the real world obstacle course quicklier, with a mean time of 81.5 seconds while it is 104.5 seconds for the control group.

However, the proposed system was tested on non-disabled individuals, so further experiments need to be done to obtain a conclusive result. Besides, some research can be done on larger patient group to identify the potential problem of the system.

D. Perceptually Guided Fast Compression of 3-D Motion Capture Data

In the same year, A. Firouzmanesh, I. Cheng, and A. Basu proposed a fast compression of motion capture data technique [START_REF] Firouzmanesh | Perceptually guided fast compression of 3-d motion capture data[END_REF]. In this method, wavelet coding is selected as the core algorithm due to its low complexity, high compression ratios and prone to adjust the perceptual quality of motion. Experiments show that the proposed method is much faster than other comparable algorithms, and is suitable to be used in mobile devices. However, in its wavelet algorithm, only bone lengths and variation in rotation are picked as factors to optimize the coefficient selection algorithm, while there are many other factors for us to explore.

E. Panoramic video with predictive windows for telepresence applications

Telepresence applications are widely used in various fields, such as explosives disposal telemedicine and so on [START_REF] Baldwin | Panoramic video with predictive windows for telepresence applications[END_REF]. To create an effective telepresence application, it is important to transmit the panoramic video effectively. The paper proposed a predictive Kalman filter which can be applied to predict viewing direction for displaying panoramic images. From the experiment, the prediction method and panoramic video are helpful to enhance the operator interface for the telepresence system.

F. Gaussian and Laplacian of Gaussian weighting functions for robust feature based tracking

The performance of object tracking algorithms is mainly limited by noise in environment, shape characteristic of objects an so on. Weighting functions are used to assign weights to different pixels in the images so that the object tracking system will can emphasize the important features in images.

The paper [START_REF] Singh | Gaussian and laplacian of gaussian weighting functions for robust feature based tracking[END_REF] proposed Gaussian and Laplacian of Gaussian weighting functions to improve the performance of the KLT tracking algorithm.

From the result of experiment, the proposed Gaussian weighting function and LOG weighting function both outperform the averaging weighting function of KLT when tracking objects in random noise environment, although the computation consumption increase by 10%. Besides, it s also found that LOG weighting function works better at images with clear and sharp corners while Gaussian weighting function is better at dealing with real images which typically do not have sharp edges.

G. Nose shape estimation and tracking for model-based coding

In the traditional facial recognition system, eyes and mouth are always considered to be the most important features to detect. But actually the feature of nose shape is of the same importance to track and detect when analyzing or synthesising human facial expression.

The paper [START_REF] Yin | Nose shape estimation and tracking for modelbased coding[END_REF] proposes a facial feature detection method which first put facial features into corresponding windows by using the global region growing and local region growing method. Then is to extract the accurate shapes of facial features and finally a tracking model can match with individual faces to track facial expression.

The experiments on real video sequences prove the facial feature detection method for low bit rate video coding is applicable in practice.

H. Hough transform for feature detection in panoramic images

The omni-directional sensor can be utilized to obtain a 360 degrees field of view, which can be achieved simply by a camera through a radially symmetrical mirror and a traditional lens system. However, most existing mirror profiles violate the Single View Point (SVP) standard. Therefore, many existing methods cannot guarantee that the function of the lens system is equivalent to the standard perspective projection, which is a huge challenge. Such an imaging system with a non-SVP optical system does not benefit from the affine quality of straight line features represented as collinear points in the image plane. In order to use these non-SVP images, a new method is needed to detect these features. This paper [START_REF] Fiala | Hough transform for feature detection in panoramic images[END_REF]proposed a panoramic non-SVP image feature detection method based on improved Hough transform. The mathematical model of the feature extraction process is given.

Experiments and results show that the panoramic Hough transform can effectively detect distorted horizontal lines to supplement the function of vertical line detection in panoramic images. This transformation greatly improves the detection characteristics of the catadioptric panoramic sensor with a non-SVP mirror profile, and has good robustness.

I. Generating Realistic Facial Expressions with Wrinkles for Model-Based Coding

3D face reconstruction has always been a hot topic in academic research. However, the current face reconstruction technology can only roughly imitate the real face. It is very difficult to generate real facial texture . To solve this problem, this paper [START_REF] Yin | Generating realistic facial expressions with wrinkles for model-based coding[END_REF] proposes a facial expression synthesis method based on local texture update. Firstly, a color based deformable template matching method is used to estimate the reference points on the face. In addition, an extended dynamic mesh matching face tracking algorithm is proposed. Compared with the whole face texture compression scheme, this method can significantly reduce the computational complexity and update quality of the whole face texture.

Experimental results show that, compared with the overall texture updating method, the new strategy based on local wrinkle texture updating has some advantages, which greatly reduces the number of coefficients to be transmitted. In addition, the proposed method can also maintain the high fidelity of image quality.

J. Stereo Matching Using Random Walks

Binocular stereo matching has always been a research hotspot of binocular vision. Binocular camera takes left and right view images of the same scene, uses stereo matching algorithm to obtain disparity map, and then obtains depth map. The range of application of depth map is very wide, because it can record the distance between the object in the scene and the camera, it can be used for measurement, 3D reconstruction, and virtual view synthesis.

This paper [START_REF] Shen | Stereo matching using random walks[END_REF] proposes a two-phase stereo matching algorithm based on random walk frame. Random walks has been widely used in many fields. The main idea of random walk algorithm is to modeled the real-world problem as a mutidimension graphic problem. From calculating the probability of randomly walking from one node to another, we can get the weights from considering some local quality globally.

In this proposed method, a set of reliable pixel matching algorithms based on Laplacian matrix and neighborhood information. Then, using reliable set as seed, the difference of unreliable regions is determined by solving Dirichlet problem. In order to improve the accuracy of parallax map, the change of illumination between different images is considered when building prior matrix and Laplacian matrix.

K. Motion detection using background constraints

Motion detection has become a hot research topic and can be applied in many fields. For example, autonomous driving requires the function of motion detection, which combines the research of motion planning and motion detection. In order to avoid collisions, vision-based navigation systems need to recognize the presence of stationary and moving objects. This paper [START_REF] Elnagar | Motion detection using background constraints[END_REF] proposes a plane motion detection method based on background constraints. The camera is mounted on a platform that can rotate and translate. Using the information about the camera movement, the corresponding mapping of the pixels in the continuous image can be obtained to detect the movement.

The basic idea of this method is that if any point in the image is static, the point should satisfy the background constraint, while the point on an independent moving object is unlikely to satisfy the constraint. Therefore, it becomes easy to detect moving objects and can be used for translation and rotation.

Experiments and results show that the proposed motion detection method based on background constraints has strong robustness.

L. Modeling Fish-eye Lenses

A fish-eye lens is generally composed of a dozen different lenses. During the imaging process, the incident light is refracted to different degrees and projected onto an imaging plane with a limited size, making the fish-eye lens compared with ordinary lenses.

Compared with ordinary pinhole camera, fish eye camera can observe a wider range, which can enhance the visual robustness to a certain extent. This paper [START_REF] Basu | Modeling fish-eye lenses[END_REF] proposed a simple transformation that can be used to model variable resolution images. The experimental results show that our model works very closely with the real fish-eye lens.

M. Event Dynamics Based Temporal Registration

The video consists of a series of static images. However, when the video shakes severely, the image information in the video will become blurred. At this time, time registration can be used to make the video image more stable, which has been widely used in military, medical and other fields. In order to improve the effect of traditional Temporal Registration, the paper [START_REF] Singh | Event dynamics based temporal registration[END_REF] proposed the Event Dynamics Based Temporal Registration method, and verified its effectiveness with experiments.

N. Pose Recognition using the Radon Transform

Human posture recognition technology is to recognize posture actions by collecting all kinds of physical information generated by human movement, including body posture, finger movement, facial movement and so on.

The research on the analysis and recognition of human motion and posture has become a hot direction, and has been applied in various fields such as medical rehabilitation, safe driving and robot technology. The existing human pose recognition methods can be roughly divided into model-based and non model-based.

This paper [START_REF] Singh | Pose recognition using the radon transform[END_REF] proposed a method to recognize human postures and gestures by using radon transformation. Radon transform is an integral transform, which projects the image space into the parameter space of the line in the form of line integral.

O. Subjective and Objective Visual Quality Assessment of Textured 3D Meshes

With the development of 3D printing and true 3D display technology, 3D grid related technology has become a research hot spot again. With the rapid development of 3D modeling technology, 3D models and their applications have received extensive attention. As the main form of threedimensional model, three-dimensional grid model is widely used in computer-aided design, medical imaging, digital entertainment and other fields. However, the 3D mesh model is often distorted to different degrees after processing, which makes the visual quality of the 3D mesh model change. Therefore, it is very important to find an effective perceptual metric to evaluate the quality of these texture model artifacts.

This paper [START_REF] Guo | Subjective and objective visual quality assessment of textured 3d meshes[END_REF] designed and established a new subjective evaluation database texture 3D grid. In addition, the paper [START_REF] Guo | Subjective and objective visual quality assessment of textured 3d meshes[END_REF] also proposed a new objective index for the evaluation of the visual quality of texture grids as an optimized linear combination of grid quality and texture quality.

IV. CONCLUSION 3D face reconstruction and face alignment are the basis of real-time face animation generation. After our research on the related work, we identified that PRN provides us with an endto-end method to solve these two problems simultaneously. In this project, we will try to add data augmentation as well as the post-processing SFS method to improve the robustness and the precision of PRN. Besides, we will also explore other potential CNN architectures to further enhance the performance of our model.

V. REFERENCES


Blind-Spot Network for Self-supervsied InSAR phase filtering-Literature Review Zihao Wang * , Mingzhi Zhu † * University of Alberta. zihao17@ualberta.ca † University of Alberta. mingzhi4@ualberta.ca Abstract-Over the past few years, significant efforts have been made in-ground displacement detection domain using Interferometric Synthetic Aperture Radar (InSAR) technology. However, due to several factors, such as atmospheric change and undesirable material reflectance, the signals reflected by the ground and received by satellites are usually corrupted by noises. This paper will go through different serval existing solutions -Noise2Clean, Noise2Noise, Noise2Void and some relative existing methodologies. As a literature review paper, we will also review some papers that can potentially inspire our mind and improve our ongoing solution. In this way, we move one step further to optimize the InSAR project.

Index Terms-Convolutional Neural Networks , Image De-noising, Self-Surprised Learning, InSAR Image I. INTRODUCTION Image denoising is one of the most significant applications in the computer vision domain. In recent years, with deep neural networks flourish, several architectures have been proposed for image reconstruction and restoration tasks. Generally speaking, these models are based on supervised learning, which requires ground truth (target images) and noisy images (source images) for the purpose of model training.

In general, data acquisition could be complex and only corrupted data available. For InSAR images, This means we need to figure out some novel methodologies to let neural network learns only from the noisy data itself. This motivates the investigation of self-supervised training.

Our objective is to realize an InSAR image denoising end-to-end pip-line by only input noisy images and generate de-noised images as outputs.

We will firstly implement a blind-spot network on synthetic images to fine-tune and evaluate the model. Then we will use a simulator proposed by this paper [START_REF] Sun | Deepinsar-a deep learning framework for sar interferometric phase restoration and coherence estimation[END_REF] to generate InSAR data to train our model and optimize it using other algorithms like MDN if time permits.

Additionally, we are seeking the possibility of enhancement for our project result through literature review session. By going through the literature review, we will compare existing methods and conduct project enhancement. Our goal is to design a fast and reliable image denoising system at the end. Here is our summarized goal:

1) To develop an efficient image denoised model, A clear and denoised image can be generated with the trained model. 2) By doing literature review, any improvement would be acceptable. E.g. imagepreprocessing, model improvement, and more. 3) Compare with existing solutions, list our In-SAR project trained model advantages and possible future improvements.

II. MOTIVATION

The combination of computer vision and deep learning is the mainstream of our future work. As mentioned previously, image restoration and reconstruction tasks are widely applied in academic and industrial domains. As one of the critical branches, image denoising has a wide range of applications. This project has two creative points worth to be explored: Firstly, our project aimed at a particular and unique domain. In the last semester, we focused on the reconstruction of ultrasound images and generated promising results. The following articles each give our group plenty of tips on effectively developing high-definition, noise-reduced satellite images. And these articles are:

A. Image Noise2 Clean

Learning from PB-level noise InSAR by unsupervised learning. Increasingly, data acquired by SAR satellites and new interferograms can be generated.

B. Image Noise2 Noise

Based on the deep convolutional model, This points us to a path to eliminate noise from images and acquire clean data.

C. Image Noise2 Void

A new method that only requires a single nosy acquisition to train denoising CNNS, this powerful technique allows us to denoise InSAR images.

These projects are trying to figure out methodologies to fit in unique data sets and expand different fields' skill sets. Secondly and most importantly, this self-supervised denoising algorithm only requires noisy data and can be applied to a wide variety of research fields.

III. ITERATURE REVIEW

A. Noise2Noise Learning Image Restoration without Clean Data

Our thoughts mainly come from the following papers: Lehtinen, Jaakko, et al." Noise2noise: Learning image restoration without clean data." [START_REF] Lehtinen | Noise2noise: Learning image restoration without clean data[END_REF]. In this paper, they In this article -Noise2Noise Learning Image Restoration without Clean Data, the authors only used noisy data as input to train their model. While removing the need for clear images, they still need a pair of training images with different noise types, which helped them reconstruct the original signal. It is important to note that the authors do not use truth ground images to restore images but only corrupted images. At the beginning of the paper, the authors first illustrate that their method is feasible with the following formula.

argmin θ i L (f θ (x i ) , y i ) (1)
Once executability is proven. The authors start performing various image recovery and denoise tests. In the experimental stage, The authors apply Gaussian Noise and different Synthetic Noise to images. Later, the authors use Monte Carlo Rendering and Magnetic Resonance Imaging to perform further noise reduction tests. Through these experimental, the authors demonstrated that their Noise2Noise approach is feasible and that the authors can repair the complex corruption without clean data. This also gives us an extraordinary hint that we can reproduce the Noise2Noise method to restore InSAR images.

B. Noise2Void -Learning Denoising from Single Noisy Images Krull, Alexander, Tim-Oliver Buchholz, and Florian Jug. "Noise2void-learning denoising from single noisy images." [START_REF] Krull | Noise2void-learning denoising from single noisy images[END_REF]. The authors of this paper were inspired by the previous noise2noise method. The paper -Noise2Void inherits the Noise2Noise concept and goes a step further by eliminating the noise from images without clean target images. The principle of Noise2void is to train directly on the body of data that needs be denoised. Compare with Noise2Noise, Noise2Void does not require a pair of noised image or clear image, which was mainly up to the concept of "blind-spot". More specifically, when we use a single noisy image as input and target in normal convolution neural networks, the network will degenerate and learn as identity. Whereas in the blind-spot network, each pixel's receptive field excludes the pixel itself and predicts the pixel value from its surrounding values, preventing it from learning the identity, details can be seen in Fig. ??. It is a pity that they applied this method while training instead of realizing the blind-spot neural network. In this InSAR project, we are impossible to acquire a clear target image. Consequently, Noise2Void would be our one of the best option to denoise InSAR images.

C. Blind-Spot -High-Quality Self-Supervised Deep Image Denoising

The paper "High-Quality Self-Supervised Deep Image Denoising" proposed by Laine et al [START_REF] Laine | Highquality self-supervised deep image denoising[END_REF] raised a novel self-surprised methodologies to achieve image de-noising, similar to the noise2void algorithm, it did not require pairs of clean and noisy images and moreover, even did not require pairs of noisy data while training. This presented great advantages in the situation while lacking of data or pairs of images are unacceptable expensive. The authors were inspired by Noise2Void, which assumed that the noise was zero-mean and independent between pixels and thus predicted noise pixel value using surrounding context. What worth mentioning was that compared with masking scheme used by Noise2Void, the algorithm raised by this article truly implemented blind-spot network structure.

In their initial design, the blind-spot network structure combined four branches. Each has half of the input image(central pixels were excluded) and finally combined these branches with a 1x1 convolutional layer to obtain a receptive field in for direction while excluding the central pixel. While in practice, to save the computational resources, they rotated input images in 0, 90, 180, 270 degrees respectively, proposing a simpler architecture that generates similar result than the un-optimized version while avoiding the four-fold increase in training weight, which shown in Fig. 1 D. DnCNN -Beyond a gaussian denoiser: Residual learning of deep CNN for image denoising This paper, "Beyond a gaussian denoiser: Residual learning of deep CNN for image denoising" proposed by Zhang et al [START_REF] Zhang | Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising[END_REF], took one step forward by raising a feed-forward denoising convolutional neural networks (DnCNNs) for image de-noising based on the concept of discriminative model learning. What worth mentioning was that they implemented residual learning and batch normalization to speed up training process also improved performance. Comparing with other end-to-end de-noising pip-line which outputs the de-noised image, their proposed network architecture was designed to predict the residual images, more precisely, the noisy observation and the latent clean image, and the latent images was eliminated by the operations in hidden layers. What was more, batch normalization was applied to improve the performance of model.

E. An Unsupervised Generative Neural Approach for InSAR Phase Filtering and Coherence Estimation

In this paper "An Unsupervised Generative Neural Approach for InSAR Phase Filtering and Coherence Estimatio" [START_REF] Mukherjee | An unsupervised generative neural approach for insar phase filtering and coherence estimation[END_REF], the authors propose an InSAR phase filter and coherence estimator. Learning from PB-level noise InSAR by unsupervised learning, the authors manage to generate new interferograms to improve the InSAR machine's performance. This highly scalable GPUbased filter can help users monitor a wide area of the Earth's surface to detect any potential disasters, such as forest fires, tsunamis, etc.

F. Potential Improvement Images Pre-process

In order to capture large area of images, pictures are taken with the fish lens which caused barrel distortion [START_REF] Basu | Modeling fish-eye lenses[END_REF]. Since InSAR images are susceptible with input image quality, a lousy calibration camera output could cause terrible quality and noise images. The low resolution and distortion images seriously affect the final image output. In 1995, A. Basu presented a way to actively calibrate cameras to improve overall image quality [START_REF] Basu | Active calibration of cameras: theory and implementation[END_REF]. To achieve this goal, the camera can be mounted on an active platform. Only a scene with strong and stable edges is needed. The outcome is excellent with an active calibration camera. This technique cannot be applied in this InSAR project since the proposer already provides the noise images. However, it would be nice to have a similar approach when images were taken. After the photo is taken and needs to be transmitted back to the headquarter, image compression is required for image data transmission. In the papers, Enhancing Videoconferencing [START_REF] Shen | Qoe-based multi-exposure fusion in hierarchical multivariate gaussian crf[END_REF] and Variable Resolution Teleconferencing [START_REF] Basu | Variable resolution teleconferencing[END_REF] present a way to transmit data with a high compression ratio (98.56%) while keeping SNR only to 47.23 [START_REF] Basu | Enhancing videoconferencing using spatially varying sensing[END_REF]. With the Variable resolution technique, users can receive high-quality images while they have limited computing resource [START_REF] Basu | Videoconferencing using spatially varying sensing with multiple and moving foveae[END_REF]. Once the photos are received, the image's quality becomes substantial. Quality of Experience(QOE) Based MAP-HMGCRF fusion method can apply to the images to improve the overall image quality. As Satellite images, the images are captured by the Fig. 1: Blind-spot architecture and optimized version [START_REF] Laine | Highquality self-supervised deep image denoising[END_REF] devices for displaying large areas. Although the fish lens can increase the Coverage of the site, that is far from enough. Panoramic image can be applied and reassemble image into Panoramic photo. In computer vision, Panoramic images are generally Capturing a panoramic (360°) field-of-view with optical systems [START_REF] Fiala | Hough transform for feature detection in panoramic images[END_REF]. InSAR images can use this panoramic technique to merge multiple images as one.

G. Review Of Image Processing Method In Industrial Application

In this section, I will introduce some traditional image processing-based methods for de-noising in industrial applications. This paper, "NOSE SHAPE ESTIMATION AND TRACKING FOR MODEL-BASED CODING" [START_REF] Yin | Nose shape estimation and tracking for model-based coding[END_REF], to made facial feature detection less influenced by noise, they proposed a two-stage region-growing algorithm, more precisely, global region growing, which was used to localize the most significant features while ignoring the size information. And local region growing, where the region information was estimated. Similar to our project, tackling noise issue for soecial images, more precisely, ultrasound images in this paper "Segmentation of Arterial Walls in Intravascular Ultrasound Cross-Sectional Images Using Extremal Region Selection" [START_REF] Faraji | Segmentation of arterial walls in intravascular ultrasound cross-sectional images using extremal region selection[END_REF], normally, those IVUS images were corrupted by speckle noise, before features being extracted, non-linear median operator was applied to filter the IVUS images, and similarly, paper "Stereo Matching Using Random Walks" [START_REF] Shen | Stereo matching using random walks[END_REF] also applied this algorithm to smooth images. In this paper, "Enhancement of low visibility aerial images using histogram truncation and an explicit Retinex representation for balancing contrast and color consistency" [START_REF] Liu | Enhancement of low visibility aerial images using histogram truncation and an explicit retinex representation for balancing contrast and color consistency[END_REF] for foggy images enhancement and recovery, to get rid of the degeneration caused by noise, the authors invert and convert the two parameters into MSRCR domain. In this paper, "Generalized Random Walks for Fusion of Multi-Exposure Images" [START_REF] Shen | Generalized random walks for fusion of multi-exposure images[END_REF], experimental images were corrupted by white Gaussian, they firstly using the steerable pyramid transform to decompose into subbands and then employed Bayesian least square estimation and Gaussian scale mixture model to denoised each subband to get better performance. Instead of using a filter, this paper [START_REF] Shen | Qoe-based multi-exposure fusion in hierarchical multivariate gaussian crf[END_REF], weight pixels' contrast by the background luminance to suppress noise. There are also several image evaluation function summarized and mentioned by this paper [START_REF] Guo | Subjective and objective visual quality assessment of textured 3d meshes[END_REF].

IV. CONCLUSION In the current stage, our team managed to reproduce Self-Supervised Deep Image Denoising, and the result output is obvious and fabulous. As discussed above, there are many methodologies and improvement could be made to our ongoing project. The majority challenge is to improve the PSNR of images and results. In this paper, a number of literature are reviewed and discussed. However, an industrial-strength InSAR image denoise system is complicated to achieve. We are greedy to look for better methodologies to improve our InSAR project. With more literature review and research, our team thinks some improvements can be made. We will continuously investigate in our future research to continue to fulfill project expectations and goals.