Renan Souza

Vítor Silva

G A Coutinho

Patrick Valduriez

Marta Mattoso

Online Input Data Reduction in Scientific Workflows

Keywords: Scientific Workflows, Human in the Loop, Online Data Reduction, Provenance Data, Dynamic Workflows

Many scientific workflows are data-intensive and need be iteratively executed for large input sets of data elements. Reducing input data is a powerful way to reduce overall execution time in such workflows. When this is accomplished online (i.e., without requiring users to stop execution to reduce the data and resume execution), it can save much time and user interactions can integrate within workflow execution. Then, a major problem is to determine which subset of the input data should be removed. Other related problems include guaranteeing that the workflow system will maintain execution and data consistent after reduction, and keeping track of how users interacted with execution. In this paper, we adopt the approach "human-in-theloop" for scientific workflows by enabling users to steer the workflow execution and reduce input elements from datasets at runtime. We propose an adaptive monitoring approach that combines workflow provenance monitoring and computational steering to support users in analyzing the evolution of key parameters and determining which subset of the data should be removed. We also extend a provenance data model to keep track of user interactions when users reduce data at runtime. In our experimental validation, we develop a test case from the oil and gas industry, using a 936-cores cluster. The results on our parameter sweep test case show that the user interactions for online data reduction yield a 37% reduction of execution time.

CCS Concepts

• Massively parallel and high-performance simulations.

INTRODUCTION

Scientific Workflow Management Systems (SWMS) with parallel capabilities have been designed for executing data-intensive scientific workflows, scientific workflows for short hereafter, in High Performance Computing (HPC) environments. A typical execution may involve thousands of parallel tasks with large input datasets. When the workflow is iterative, it is repeatedly executed for each element of an input dataset. The more the data to be processed, the longer the workflow may take, which may be days depending on the problem and HPC environment [START_REF] Dias | Data-centric iteration in dynamic workflows[END_REF]. Configuring a scientific workflow with parameters and data to be processed is hard. Typically, the user needs to try several input data or parameter combinations in different workflow executions. These trials make the scientific experiment take even longer. It is well known that optimizing performance of the parallel system is a way to improve overall workflow execution time, but reducing the input data that was initially planned to be processed is another effective approach to reduce workflow execution time [START_REF] Deelman | Workflows and e-Science: an overview of workflow system features and capabilities[END_REF].

In scientific workflows, the total amount of data is very large, but not necessarily the entire input dataset has relevant data for achieving the goal of the workflow execution. This is particularly the case when a large parameter space needs to be processed in parameter sweep workflows. There may be slices of the parameter space that will not influence relevant results and thus, as with a "branch and bound" optimization strategy, can be bounded. A similar scenario occurs when the workflow involves a large input dataset. When domain-specialist users can actively participate in the computational process, practice frequently referred to as "human-in-the-loop", they may analyze partial result data and tell which part of the data is relevant or not for the final result [START_REF] Mattoso | Dynamic steering of HPC scientific workflows: A survey[END_REF]. Then, based on their domain knowledge, users can identify which subset of the data is not interesting and thus should be removed from the execution by the SWMS, thereby reducing execution time.

Data reduction can be accomplished in at least three different forms. First, users can do it before the execution starts. However, in most complex scenarios, the high number of possibilities makes it impossible to know beforehand the uninteresting subsets, without any prior execution. Furthermore, not only the initial dataset can be reduced, but also the data generated by the workflow, since the activities composing scientific workflows continuously produce significant amounts of partial data that are consumed by other activities. A second form of data reduction is to do it online. When the SWMS allows for partial result data analysis, the user may interact with the generated partial data, find which slice of the dataset is not interesting, and reduce the dataset online. We use the term online for the interactions where users are able to inspect workflow execution, analyze partial and performance data, and dynamically adapt (i.e., steer) workflow settings while the workflow is running (i.e., at runtime). The third form of data reduction is by stopping execution, reducing the data offline, and then resuming execution with the reduced dataset. Because of the difficulty in defining the exploratory input dataset and the long execution time of such iterative workflows, users frequently adopt the third form. However, in the offline form, the SWMS is not aware of the changes, and the results with one workflow configuration are not related to the others. Therefore, this is generally more time-consuming, there is no control or registration of user interactions, and the execution may become inconsistent [START_REF] Dias | Data-centric iteration in dynamic workflows[END_REF].

Online data reduction has obvious advantages but introduces several problems related to computational steering in HPC environments [START_REF] Mattoso | Dynamic steering of HPC scientific workflows: A survey[END_REF]. First, because of the complexity of their scientific question to address and the huge amounts of data, users do not exactly know beforehand which subset of dataset should be kept or removed. Also, if users cannot actively follow the result data evolution online, in particular domain data associated to execution and provenance data (history of data derivation), they can be driven to misleading conclusions when trying to identify the uninteresting subset of the data. Indeed, this is the main related challenge. Second, if they can find which subset to remove and actually try to remove, the SWMS must guarantee that the operation will be done consistently. Otherwise, it can introduce anomalous data, yielding to no control of data elimination, data redundancy, or even execution crash. Third, in a long run, there may be more than one user interaction, each removing more subsets, at different times. If the SWMS does not keep track of user actions, it may negatively impact the results' reproducibility and reliability. Although data reduction is not new in SWMS [START_REF] Deelman | Workflows and e-Science: an overview of workflow system features and capabilities[END_REF],

to the best of our knowledge, these problems related to online user-steered data reduction while maintaining data provenance have not been addressed by related works.

Our approach is to represent workflow input datasets as database relations. Each input element from the scientific domain dataset is represented as a tuple of the input relations. When the elements of the input dataset are files, we insert paths to these files. The approach is implemented in Chiron, a parallel SWMS that adopts a tuple-oriented algebraic approach [START_REF] Ogasawara | An algebraic approach for data-centric scientific workflows[END_REF]. Chiron has been used to manage workflow applications with user steering in domains, such as bioinformatics [START_REF] Costa | Capturing and querying workflow runtime provenance with PROV: a practical approach[END_REF], computational fluid dynamics [START_REF] Dias | Data-centric iteration in dynamic workflows[END_REF], astronomy [START_REF] Silva | Analyzing related raw data files through dataflows[END_REF], etc. Chiron continuously populates a relational database at runtime to store domain-specific data, workflow execution data, and, more importantly, provenance data, all integrated in the same database available for online queries. In this paper, we use the term workflow Database (wf-Database) to refer to this database. In addition to the traditional advantages of managing provenance data in scientific workflows (i.e., reproducibility, reliability, and quality of result data) [START_REF] Davidson | Provenance and scientific workflows: challenges and opportunities[END_REF], online provenance data management eases interactive domain data analysis [START_REF] Costa | Capturing and querying workflow runtime provenance with PROV: a practical approach[END_REF] [START_REF] Silva | Analyzing related raw data files through dataflows[END_REF]. Such interactive flexible data analysis through provenance helps finding which subset of a dataset to be removed. Moreover, execution monitoring is another very desirable feature in any data-intensive system, including SWMS, and can also be used to assist users in addressing the subset to be removed. However, Chiron does not control changes in input datasets, including removing a subset. In this work, we extend Chiron's wf-Database to maintain the provenance of the subsets of the dataset that are removed. Furthermore, we take advantage of a distributed in-memory database system coupled to Chiron, in a version called d-Chiron that is significantly more scalable [START_REF] Souza | Parallel execution of workflows driven by a distributed database management system[END_REF], to address consistency issues with respect to data reduction. We make the following contributions:

• A mechanism coupled to d-Chiron for online input data reduction, which allows users to remove subsets of the dataset to be processed at runtime. It guarantees that both execution and data remain consistent after reduction.

• An extension to a provenance data model (which is W3C

PROV compliant) to maintain the history of user interactions when users decide to reduce a dataset during workflow execution. • An adaptive monitoring approach that combines monitoring and computational steering. It helps users to follow the evolution of interesting parameters and result data to find which subsets of the dataset can be removed during execution. Also, since what users find interesting may change over time, this approach allows the user to steer the monitoring definitions, such as which data should be monitored and how.

Although existing solutions enable workflow execution monitoring [START_REF] Mandal | Toward an end-to-end framework for modeling, monitoring and anomaly detection for scientific workflows[END_REF][16] [START_REF] Reuillon | a workflow engine specifically tailored for the distributed exploration of simulation models[END_REF], there is no approach to enable users to run monitoring queries that integrate execution, provenance, and domain data, and dynamically adapt these queries online, to the best of our knowledge.

Paper organization: Section 2 gives a motivating example. Section 3 gives the background for this work. We present our online data reduction approach in Section 4 and our adaptive monitoring approach in in Section 5. Section 6 gives the experimental validation. Section 7 discusses related work. Section 8 concludes.

MOTIVATING EXAMPLE IN OIL AND GAS INDUSTRY

In ultra-deep water oil production systems, a major application is to perform risers' analyses. Risers are fluid conduits between subsea equipment and the offshore oil floating production unit. They are susceptible to a wide variation of environmental conditions (e.g., sea currents, wind speed, ocean waves, temperature), which may damage their structure. The fatigue analysis workflow adopts a cumulative damage approach as part of the riser's risk assessment procedure considering a wide combination of possible conditions. The result is the estimate of riser's fatigue life, which is the length of time that it will safely operate. The Design Fatigue Factor (DFF) may range from 3 to 10, meaning that the riser's fatigue life must be at least DFF times higher than the service life [START_REF]Recommended practice: riser fatigue[END_REF].

Sensors located at the offshore platform collect external conditions and floating unit data, which are stored in multiple raw files. Offshore engineers use specialized programs (mostly complex simulation solvers) to consume the files, evaluate the impact on the risers in the near future (e.g., risk of a fracture occurrence), and estimate the risers' fatigue life. Figure 1 presents a scientific workflow composed of seven piped specialized programs (represented by workflow activities) with a dataflow in between.

Figure 1. Risers Fatigue Analysis Workflow.

Each task of Data Gathering (Activity 1) decompresses one large file into many files containing important input data, reads the decompressed files, and gathers specific values (environmental conditions, floating unit's, and other data), which are used by the following activities. Preprocessing (Activity 2) performs pre-calculations and data cleansing over some other finite element mesh files that will be processed in the following activities. Stress Analysis (Activity 3) runs a computational structural mechanics program to calculate the stresses applied to the riser. Each task consumes pre-processed meshes and other important input data values (gathered from first activity) and generates result data files, such as histograms of stresses applied throughout the riser (this is an output file), and stress intensity factors in the riser and principal stress tensor components. It also calculates the current curvature of the riser. Then, Stress Critical Case Selection (Activity 4) and Curvature Critical Case Selection (Activity 5) calculate the fatigue life of the riser based on the stresses and curvature, respectively. These two activities filter out results corresponding to risers that certainly in a good state (no critical stress or curvature values were identified), which are of no interest to the analysis. Calculate Fatigue Life (Activity 6) uses previously calculated values to execute a standard methodology [START_REF]Recommended practice: riser fatigue[END_REF] and calculate the final fatigue life value of a riser. Compress Results (Activity 7) compresses output files by riser.

Most of these activities generate result data (both raw data files and some other domain-specific data values), which are consumed by the subsequent activities. These intermediary data need to be analyzed during workflow execution. More importantly, depending on a specific range of data values for an output result data (e.g., fatigue life value), there may be a specific combination of input data (e.g., environmental conditions) that are more or less important during an interval of time within the workflow execution. The specific range is frequently hard to determine and requires a domain expert to analyze partial data during execution.

For example, an input data element for Activity 2 is a file that contains a large matrix of data values, composed of thousands of rows and dozens of columns. Each column contains data for an environmental condition and each row has data collected for a given instant in time. Each row can be processed in parallel and the domain application needs to consume and produce other data files (on average, about 14 MB consumed and 6 MB produced per processed input data element). After many analyses online, the user finds that, for waves > 38m with frequency > 1Hz, a riser fatigue will never happen. Thus, within the entire matrix, any input data element that contains this specific uninteresting range does not need to be processed. Therefore, by reducing the input dataset, the overall data processed and generated are reduced and, more importantly, the overall execution time is reduced. In this paper, we use this workflow in our examples.

USER-STEERED WORKFLOWS

Mattoso et al. [START_REF] Mattoso | Dynamic steering of HPC scientific workflows: A survey[END_REF] analyze six aspects of computational steering in scientific workflows: interactive analysis, monitoring, human adaptation, notification, interface for interaction, and computing model. Despite their importance, the first three are essential and we mostly focus on those in this work. In fact, human adaptation is definitely the core of computational steering. However, users will only know how to fine-tune parameters or which subset needs further focus if they can explore partial result data during a longterm execution. For this, interactive analysis and monitoring play an important role to put the human in the loop.

Online provenance data management in SWMS is an essential asset to support all six aspects of computational steering in scientific workflows. In this section, we explain the three computational steering aspects explored in this paper.

Interactive analysis

We address two aspects of workflow data that should be interactively analyzed: (A) domain dataflow and (B) workflow execution information [START_REF] Mattoso | Dynamic steering of HPC scientific workflows: A survey[END_REF].

A. Domain dataflow. Workflows are composed of activities (scientific programs, scripts, or services) linked as a dataflow. Each activity invocation, or task, may consume input datasets and input raw data files and produce output datasets and files. These flows form the domain dataflow. To support domain dataflow interactive analysis, Chiron stores dataflow provenance data in the wf-Database during execution and makes them available for online user queries. Users can then query the wf-Database using a query interface or SQL following PROV-Wf [START_REF] Costa | Capturing and querying workflow runtime provenance with PROV: a practical approach[END_REF], a W3C PROVcompliant data model [START_REF] Moreau | the PROV data model[END_REF] that specializes PROV entities into domain-data entities to allow for domain dataflow analysis at a finer grain than PROV.

Chiron's tuple-oriented engine first stores input dataset as tuples in the wf-Database. In parameter sweep, tuples are data values from the Cartesian product of input parameters. Then, each task consumes input tuples retrieved from this database, executes them, and then stores the generated output tuples in the wf-Database immediately after task execution, adequately maintaining the data relationships to the input tuples. The workflow activities that generated the tuples are also stored in the wf-Database and linked accordingly. Large raw data files consumed or produced by each task are not stored in the wf-Database, but are rather linked to them, for file flow management. Thus, Chiron enables online fine-grained domain dataflow analysis [START_REF] Costa | Capturing and querying workflow runtime provenance with PROV: a practical approach[END_REF] as well as the analysis of related domain raw data files through file flow relationships [START_REF] Silva | Analyzing related raw data files through dataflows[END_REF]. Table 1 shows some useful queries for the riser fatigue analysis workflow involving domain data and provenance dataflow analysis. The corresponding generated SQL, as well as the relational schema, are in http://github.com/hpcdb/d-chiron. These queries reflect typical user interactions. When these workflows are executed as scripts, without Chiron's support, users look for files in their directories, open files, and try to do this analysis in an ad-hoc way, frequently writing programs to "query" these result files. Often they interrupt the execution to fine tune input data and save execution time. For Queries 𝑄1-𝑄4, the SWMS needs to store the history of the tuples generated in Activities 4 and 5 since the beginning of the flow, adequately linking each tuple flow in between. For example, environmental conditions (𝑄1) and hull conditions (𝑄2) are obtained in Activity 1, and stress-and curvature-related values are obtained in Activities 4 and 5, respectively. To correlate output tuples from Activity 4 or 5 to tuples from Activity 1, provenance data relationships are required.

B. Workflow execution information.

Lower level execution engine information, such as physical location (i.e., virtual machine or cluster node) where a task is being executed, can highly benefit data analysis and debugging in HPC execution. Users may want to interactively investigate how many parallel tasks each node is running. Moreover, tasks can run domain applications that result in errors. If there were thousands of tasks in a large execution, how to determine which tasks resulted in domain application errors and what the errors were? This also eases debugging, an important feature to be provided in large parallel executions. Furthermore, performance data analysis is very useful. Users are frequently interested in knowing how long tasks are taking. All this workflow execution information is important to be analyzed and can deliver much more interesting insights when linked to domain dataflow data. When execution data is stored separately from domain and provenance data, these steering queries are not possible or demand combining different tools and writing specific analysis programs [START_REF] Silva | Analyzing related raw data files through dataflows[END_REF].

To support all this, Chiron's wf-Database registers parallel workflow execution data. This means that all necessary execution information for the parallel engine to work are linked to domain dataflow data and managed in the same database. Table 2 shows some provenance queries for the Risers Analysis workflow that link workflow execution data to domain dataflow.

Table 2. Domain data linked to performance data.

𝑸𝟓

Determine the average of each environmental conditions (output of Data Gathering -Activity 1) associated to the tasks that are taking execution time more than 2 standard deviations of Curvature Critical Case Selection (Activity 5).

𝑸𝟔

Determine the finite element meshes files (output of Preprocessing -Activity 2) associated to the tasks that are finishing with error status.

𝑸𝟕

List the 5 computing nodes with the greatest number of Preprocessing activity tasks that are consuming tuples that contain wind speed values greater than 70 Km/h.

Monitoring

Another important form to help gaining insights from the data is by monitoring in a passive way. It means that users can set up some monitoring analyses and wait for the monitoring results to be generated. Results might be delivered to end-users as graphical dashboards or three-dimensional in-situ scientific data visualizations. As users gain insights from monitoring results, if the SWMS has dynamic analytical support, they can adapt previously set up monitoring configurations or add new monitoring analysis [START_REF] Mattoso | Dynamic steering of HPC scientific workflows: A survey[END_REF]. Also, from these new insights, new data exploration through interactive analysis can be done.

If the SWMS allows for provenance data analysis during workflow execution, monitoring becomes more effective, since the SWMS can exploit the continuously populated wf-Database. By doing this, all the aforementioned data provenance analysis and queries executed by users may be used by a monitoring engine.

Human adaptation

After users have analyzed partial data and gained insight from them, they may decide to adapt the workflow execution. Adaptation brings powerful abilities to end-users, putting the human in full control of scientific workflow executions. Many aspects can be adapted by humans, but very few systems support human-in-the-loop actions [START_REF] Mattoso | Dynamic steering of HPC scientific workflows: A survey[END_REF]. The human-adaptable aspects range from computing resources involved in the execution (e.g., adding or removing nodes), to checking-point and rolling-back (debugging), loop break conditions, reducing datasets, modification of filter conditions, and very specific parameter finetuning.

Populating the wf-Database during workflow execution can help all these aspects. In the Chiron SWMS, it has been shown that it particularly facilitates steering. For example, in [START_REF] Dias | Supporting dynamic parameter sweep in adaptive and user-steered workflow[END_REF], it was shown that it is possible to change filter conditions during execution. Also, in [START_REF] Dias | Data-centric iteration in dynamic workflows[END_REF], it is proposed an algebraic approach to enable steering and dynamic changes of loop conditions during execution of iterative workflows (e.g., modify number of iterations or loop stop conditions), and such approach was evaluated in Chiron. These works show that these adaptations can significantly reduce overall execution time, since domain expert users are able to identify a satisfactory result before the programmed number of iterations. Prior to this work, although [START_REF] Dias | Data-centric iteration in dynamic workflows[END_REF] has highlighted its advantages, no work has been developed in Chiron to tackle usersteered data reduction online.

Since provenance data is so beneficial, we consider that when a user interacts with the workflow execution, new data (user interaction data) are generated, and thus their provenance must be registered. In a long-running execution, many interactions can occur and many adaptations may be made. If the SWMS does not adequately register the provenance of interaction data, the users can easily lose track of what they have changed in the past. This is critical if the entire computational experiment takes days to finish and many specific adaptations had to occur, since it may be impossible for the users to remember in the last day of execution what they have steered in the first days. Furthermore, adding interaction data to the wf-Database enriches its content and enables future user interaction analysis. One example of how such data can be exploited is that the registered interaction data could be used by artificial intelligence algorithms for understanding interaction patterns and recommend future adaptations. For these reasons, the SWMS that enables computational steering should collect provenance of user interaction data. To the best of our knowledge, this has not been done before.

ONLINE DATA REDUCTION

In this section, we show our main contribution. Suppose that after analyzing the monitoring results, a user identifies, within the entire dataset, the subset that contains those values can be removed, hence reducing the dataset.

However, reducing a dataset to be processed has specific constraints that need to be addressed so the execution remains in a consistent state, i.e., with valid data and with guarantee that the execution will not crash. As previously described, Chiron implements a relational data model in a tuple-oriented algebraic approach for scientific workflows [START_REF] Ogasawara | An algebraic approach for data-centric scientific workflows[END_REF].

We propose to represent the input dataset as a database relation, which is a set of tuples. In the tuple-oriented approach [START_REF] Ogasawara | An algebraic approach for data-centric scientific workflows[END_REF], tuples represent a domain-specific dataset to be consumed or produced by a workflow activity. As examples, tuples may be composed of parameter values of a computational model, file paths to a large raw data file (e.g., genomic sequences, finite element meshes, textual data, binary files), or calculated values. In the tupleoriented approach, removing a subset of the entire dataset to be processed means removing a set of tuples to be consumed by a workflow activity. As a consequence of this removal, the tasks that would process the tuples within the removed set of tuples will not be executed, hence, reducing both workflow execution time and data processing. Data processing reduction becomes more evident if the removed tuples contain paths to large raw data files that would be consumed by tasks if the tuples were not removed. Not only this prevents execution of tasks that would consume uninteresting data, but also the non-executed tasks will not produce more data, reducing overall generated data amount in a workflow execution. Furthermore, if a tuple of a given activity is removed, the following tuples forming the tuple-flow of the next linked activities will not be processed too, reducing data and, more importantly, execution time in cascade.

Addressing which specific subset will be removed is quite important. So, we first formalize the subset to be removed (Section 4.1). In Section 4.2, we describe how we implemented this in d-Chiron, which is a modified version of Chiron that manages the wf-Database in an in-memory distributed database system and is significantly more scalable than the original Chiron [START_REF] Souza | Parallel execution of workflows driven by a distributed database management system[END_REF]. We also highlight that even though we implemented our solutions in d-Chiron, other SWMS could be used. The only requirement is that the SWMS engine needs to manage workflow data as datasets in a tuple-oriented approach and manage domain, provenance, and workflow execution data online in the same database.

Removing slices of input datasets

In the tuple-oriented approach, to address a subset of the dataset to be removed, we first define a slice, which is a subset of tuples to be removed according to a criteria defined by the user. Let 𝑅 with data schema ₰ = {𝐶} be the relation that represents a workflow activity input dataset to be reduced. {𝐶} is the set of attributes 𝑐 ! , 1 ≤ 𝑗 ≤ | 𝐶 |, from 𝑅 and each 𝑐 ! assumes a data type (integer, string, boolean, etc). Moreover, we split 𝑅 into two subsets 𝑃 and 𝑆, where 𝑃 is the subset of 𝑅 containing tuples that have already been processed and 𝑆 is the subset of 𝑅 containing tuples that will be processed. Thus, 𝑅 ← 𝑃 ∪ 𝑆 | 𝑃 ∩ 𝑆 = ∅. 𝑃 and 𝑆 have the same schema ₰.

Then, we define a slice § as a subset of 𝑆, which is represented as a primary horizontal fragment of 𝑆, defined by the selection relational algebraic operation 𝜎 [START_REF] Özsu | Principles of distributed database systems[END_REF]. Thus, § ← 𝜎 ! 𝑆 , where 𝐹 is the selection formula to obtain the primary horizontal fragment. The formula 𝐹 may either be a simple predicate (e.g., 𝑐 !" = ′𝐹𝐴𝑇𝐼𝐺𝑈𝐸′) or a minterm predicate (e.g., 𝑐 ! > 38 ∧ 𝑐 ! > 0.1 ∧ 𝑐 ! < 1.0) [START_REF] Özsu | Principles of distributed database systems[END_REF]. Figure 2 shows a workflow example on the left:

Act. 1 consumes input relation 𝑅 and produces an output relation that also works as an input relation to be consumed by Act. 2, which produces the final output relation. Although in this illustration we show a data reduction in the first activity, we highlight that input data of any workflow activity can be reduced, including intermediary ones, as shown in Section 6, where input data from the second activity is reduced. The input relation 𝑅 is magnified on the right of Figure 2, where we show the subsets 𝑃 and 𝑆, and the slice § defined by the formula 𝐹.

Once the user has selected the slice to be removed (based on userdefined criteria), the slice can be cut off. For this, we define the operation 𝐶𝑢𝑡 using the difference relational algebraic operator, so that 𝐶𝑢𝑡 𝑅, § ← 𝑅 - §.

By doing so, we ensure that only tuples from 𝑆 will be removed, since § only contains tuples from 𝑆. This is necessary because only tuples that have not been processed yet (i.e., they are "ready" to be processed) can be removed. Otherwise, either the data or the workflow execution may become inconsistent. We note that our solution is applicable to reduce input data of any workflow activity that needs a dataset, as long as the SWMS is aware of the data elements composing the dataset.

Implementation

In this section, we describe how we implement slice removal and cut in d-Chiron. In the SWMS that implements the tuple-oriented approach and manages execution data in the wf-Database, each input tuple (or set of tuples, depending on the dataflow operator) to be consumed is related to a task. For this reason, removing tuples means removing the tasks that would consume the referenced tuples. In the PROV-Wf data model [START_REF] Costa | Capturing and querying workflow runtime provenance with PROV: a practical approach[END_REF], which d-Chiron supports, tasks' data and input tuples are stored in different relations, with a relationship in between. Thus, to implement the set 𝑆, as defined previously, we need to semi-join [START_REF] Özsu | Principles of distributed database systems[END_REF] input tuples from the input relation 𝑅 with tasks in READY state in order to only select the tuples that will be processed. Then, to get the identifiers of the ready tasks 𝜋 !"#$% to be removed, we project over the task identifiers. Using relational algebra:

𝜋 !"#$% ← Π !"#$_!" 𝜎 ! 𝑅 ⋉ 𝜎 !"#"$!!"#$% 𝑇𝑎𝑠𝑘 ,
where the formula 𝐹 is analogous to that in Section 4.1, which is the criteria to select the slice § to be removed. We emphasize that such verifications are necessary to guarantee consistency and are the SWMS's responsibility only, not the users'. The users would only need to specify the formula 𝐹 to select the slice.

To ease slice removal in d-Chiron, we developed a Steer module. With the Steer module, users can issue command lines to inform the name of the input relation 𝑅 and the formula 𝐹 to select the slice to be removed. Then, the module is responsible for retrieving the identifiers 𝜋 !"#$% of the ready tasks to be removed (analogous to the set 𝑆 needed for the slice definition). Instead of physically removing the tasks from the wf-Database, we choose to mark them with the state REMOVED_BY_USER. By doing so, we enable these tasks to be later analyzed with provenance queries and to be consistently related within the table Modified_Task.

To guarantee consistency, we take advantage of d-Chiron's database system [START_REF] Souza | Parallel execution of workflows driven by a distributed database management system[END_REF]. d-Chiron uses a transaction-optimized inmemory distributed database system that provides atomicity, consistency, isolation, and durability (ACID). In a data reduction, both d-Chiron's engine and the Steer module need to concurrently update a shared resource, the Task table in the wf-Database. While d-Chiron's engine updates the Task table to get tasks for execution and to mark them later as executed, the Steer component needs to update the Task table to mark the tasks with the identifiers in the 𝜋 !"#$% slice as removed by user, so that the engine will not get them for execution.

The wf-Database tables are distributed, thus making concurrency control of the Task table partitions even more complex. In d-Chiron, distributed concurrency control in the Task table is outsourced to the distributed database system that guarantees the ACID properties [START_REF] Özsu | Principles of distributed database systems[END_REF]. Thus, concurrency caused by the aforementioned updates is controlled by the database system, guarantying that both execution and data remain consistent.

To store provenance of removed tuples, we extend the wf-Database schema with the table User_Query to store the queries that select the slice of the dataset to be removed. The description for each User_Query column is described in Table 3. We also keep track of the removed tasks in table Modified_Task, which is a table that represents a many-to-many relationship between User_Query and Task tables. In Section 5.2, we will give the necessary extensions to the PROV-Wf data model implemented in d-Chiron to accommodate User_Query and modified tasks.

ADAPTIVE MONITORING

In this section, we combine monitoring and computational steering into an adaptive monitoring approach. Our workflow monitoring approach relies on online queries to the continuously populated wf-Database. Users can set up monitoring queries (as in Table 1 and Table 2) and analyze monitoring results.

In Section 5.1, we present a formal description and describe the implementation in Section 5.2 with the extensions to PROV-Wf to accommodate adaptive monitoring and online data reduction.

Formal description of adaptive monitoring

Monitoring works as follows. There is a set {𝑄} composed of monitoring queries 𝑚𝑞 ! , 0 ≤ 𝑖 ≤ | 𝑄 |, each one to be executed at each 𝑑 ! > 0. Users do not need to specify queries at the beginning of execution, since they do not know everything they want to monitor. This is why {𝑄} starts empty. After users gain insights from the data, after some interactive provenance data analyses, they can add monitoring queries to {𝑄} in an ad-hoc manner. Each 𝑑 ! can be adapted by users, meaning that users have control of the time frame of each 𝑚𝑞 ! during execution.

Each 𝑚𝑞 ! execution generates a monitoring query result set 𝑚𝑞𝑟 !" , 𝑡 = 𝑘𝑑 ! | 𝑘 ∈ ℕ !! , at each time interval 𝑑 ! . We constrain that each 𝑚𝑞𝑟 !" must deliver one column only. If users want more columns, they can write different monitoring queries for each new column. However, the number of rows in the result set is not limited. This means that each monitoring result set 𝑚𝑞𝑟 !" should be either a scalar value or an array.

To improve human-in-the-loop, the end-users have the flexibility to adapt monitoring during workflow execution. To do so, at each instant 𝑡 after each monitoring query result 𝑚𝑞𝑟 !" has been generated, the values for 𝑑 ! and 𝑚𝑞 ! are reloaded from the wf-Database. If any change has happened, it will be considered in the next iteration 𝑡 + 𝑑 ! . Moreover, at each certain amount of time during execution (also configured by the user), the system checks if the user has added new monitoring queries in {𝑄}. Our adaptive monitoring approach takes full advantage of the data stored online in the wf-Database. More importantly, it enables users to dynamically steer monitoring settings (including which data will be monitored and how), highly benefiting them in finding uninteresting subsets to be removed.

Implementation

To implement our approach, we first need to extend the wf-Database schema. To store {𝑄}, we add the table Monitoring_Query, shown in Table 4.

Table 4. Monitoring_Query table description.

Column name Description monitoring_id

Auto increment identifier interval Interval time (in seconds) between each monitoring query (𝑑 ! ) monitoring_query Raw SQL query to be queried wkfid Relationship between the monitoring queries and the current execution of the workflow. In d-Chiron's wf-Database, there may be data from past executions for a same workflow.

The main advantage of storing monitoring results in the wf-Database (and adequately linking the results with the remainder of the data already stored in this database) whenever a monitoring query result is executed is that users are able to query the results immediately after their generation. The wf-Database can also serve as data source for data visualization applications. To store monitoring results in the wf-Database, we add another table: Monitoring_Query_Result, shown in Table 5. Figure 3 shows the steps executed at each time interval. To enable all these monitoring capabilities and human-adaptation, three of these steps represent queries to the wf-Database, including reads and writes. The stored results can be further analyzed a posteriori or, more interestingly, used as input for runtime data visualization tools, since results are immediately made available after they are generated.

Another contribution of this paper is that we add three concepts to PROV-Wf [START_REF] Costa | Capturing and querying workflow runtime provenance with PROV: a practical approach[END_REF], which is W3C PROV-compliant [START_REF] Moreau | the PROV data model[END_REF]. Our main motivations to adhere to the W3C PROV recommendations are to help on query specification, to maintain compatibility between different SWMS and facilitate interoperability between different databases.

These concepts are: UserQuery, MonitoringQuery, and MonitoringResult, as in Figure 4. Using PROV nomenclature, UserQuery is a PROV Activity that stores the user queries that remove sets of tuples and thus influence the state of the associated tasks (i.e., remove them). MonitoringQuery is a PROV Activity that contains the monitoring queries submitted by the user in specific time intervals. The monitoring queries generate PROV Entity MonitoringResult that stores the query results.

EXPERIMENTAL VALIDATION

In this section, we validate our solution (for online data reduction and adaptive monitoring) based on a real data. In Section 6.1, we show the experimental setup, Section 6.2 shows a test case where an expert monitors the execution and removes slices of the dataset. In Section 6.3, we analyze the added overhead.

Experimental setup

Scientific workflow. As a proof of concept for this work, we use a synthetic parameter sweep workflow of the Riser Fatigue Analysis example (see Figure 1), which is based on a real case study. The workflow manipulates approximately 300 GB of raw data. In all executions, we use the same dataset, which spans over approximately 12,000 data elements to be processed in parallel.

Depending on the workflow activity, tasks may take few seconds (e.g., Activity 7) or up to one minute on average (e.g., Activity 3).

Software. In all executions, we use d-Chiron [START_REF] Souza | Parallel execution of workflows driven by a distributed database management system[END_REF], which uses MySQL Cluster 7.4.9 as its in-memory distributed database system to manage the wf-Database. The code to run d-Chiron and setup files are in github.com/hpcdb/d-chiron.

Hardware. The experiments were conducted in Grid5000 using a cluster with 39 nodes, containing 24 cores each (936 cores). Every node has two AMD Opteron 1.7 GHz 12-core processors, 48GB RAM, and 250GB of local disk. All nodes are connected via Gigabit Ethernet and access a shared storage of 10TB.

Test case

Let us consider the following scenario. Peter is an offshore engineer, expert in riser analysis and learned how to set up monitoring, analyze d-Chiron's wf-Database, and use the Steer module developed in this work. In Peter's project, the Design Fatigue Factor is set to 3 and service life is set to 20 years, meaning that fatigue life must be at least 60 years (see from Section 2). Peter is only interested in analyzing risers with low fatigue life values, because they are critical and might need repair or replacement. During workflow execution, it would be interesting if Peter could inform the SWMS, which input values would lead to low risk of fatigue, so they could be removed. However, this is not simple because it is hard to determine the specific range of values (i.e., the slice to be removed). For this, Peter first needs to understand the pattern of input values associated to low risk of fatigue life values. In the workflow (Figure 1), the final value of fatigue life is calculated in Activity 6, but input values are obtained as output of Activity 1, gathered from raw input files. Keeping provenance is essential to associate data from Activity 1 with data from Activity 6.

To understand which input values are leading to high fatigue life values, Peter monitors the generated data online. For simplicity, we consider wind speed, which is only one out of the many environmental condition parameter values captured by Activity 1 to serve as input for Activity 2. Peter knows that wind speed has a strong correlation with fatigue life in risers. He expects that with low speed winds, there is a lower risk of accident.

When workflow execution starts, the Monitor module is initialized. Then, Peter adds two monitoring queries: 𝑚𝑞 ! shows the average of the 10 greatest values of fatigue life calculated in the last 30s of workflow execution, setting 𝑑 ! = 30s; and 𝑚𝑞 ! shows the average wind speed associated to the 10 greatest values of fatigue life calculated in the last 30s, also setting the query interval 𝑑 ! = 30s. We recall from Table 1 that 𝑚𝑞 ! is similar to 𝑄1, but only considering data processed in the last 30s. 𝑚𝑞 ! and 𝑚𝑞 ! queries are added to the Monitoring_Query table.

Peter monitors the results using the Monitoring_Result table.

These results can be a data source for a visualization that plots dashboards dynamically, refreshed according to the query intervals. After gaining insights from the results and understanding patterns, he can start removing the undesired values for wind speed. The monitoring query results 𝑚𝑞𝑟 !! and 𝑚𝑞𝑟 !! for the two previously listed queries, as well as when the user reduced the data, are plotted along the workflow elapsed time, as shown in Figure 5. It presents 𝑚𝑞𝑟 !! in full black line with square markers and 𝑚𝑞𝑟 !! in full gray line with triangle markers. These markers determine when the monitoring occurred.

The workflow execution starts at 𝑡 = 0, but only after approximately 150s, the first output results from Activity 6 starts to be generated. From the first results, at 𝑡 = 150 and 𝑡 = 180, Peter checks that when wind speed is less than 16 Km/h (see horizontal dashed line in 𝑤𝑖𝑛𝑑 𝑠𝑝𝑒𝑒𝑑 = 16 in Figure 5), the results lead to the largest fatigue life values. Since risers with large fatigue life values are not interesting in this analysis, he decides, at 𝑡 = 190, to remove all input data elements that Similarly, Peter continues to monitor and steer the execution. He issues 𝑞 ! at 𝑡 = 370 to remove input data with wind speed < 30.5 Km/h, making a decrease of 4.9% in large fatigue life (comparing fatigue life in 𝑡 = 360 and 𝑡 = 390). Then, he issues 𝑞 ! at 𝑡 = 430 to remove input data with wind speed < 34.5, attaining a decrease of 1.7% in large fatigue life (comparing fatigue life in 𝑡 = 420 and 𝑡 = 450). Despite this small decrease, he decides at t = 520 to further remove data, but with wind speed < 35.5 Km/h. However, no decrease greater than 1%

in the large fatigue life values was registered after this last Peter's steering. Thus, he keeps analyzing the monitoring results, but does not remove input data anymore until the end of execution.

We store each interaction in the User_Query table and map (in table Modified_Task) its rows with rows in the Task table, to consistently keep provenance of which tasks were modified (in this case, removed) by each specific user steering. Thus, keeping provenance of user steering helps analyzing how specific interactions impacted the results. Figure 5 shows that some specific interactions imply significant changes in lines' slopes.

Queries on the wf-Database can show finer details about how many tuples each user interaction made the SWMS not process, as shown in Table 6. Each issued time follows Figure 5 and is registered with the timestamp of when the first activity started. Finally, we run the exact same workflow and input datasets, but with no monitoring or interactions to compare how such slice removals help decrease overall execution time. The workflow with no interaction processes all input data, including those containing wind speed values that lead to risers with low risk of fatigue, which are not considered in Peter's analyses. In total, Peter's steering yields the removal of 1469 input data elements (out of approximately 12,000). This reduces the execution time for this test case by 37% compared with no steering. Furthermore, these removed input data would make the workflow process and generate more raw data files if the input data elements were not removed. By querying the wf-Database in the end of execution, we found that the execution with no user steering processed approximately 300GB of raw data files, whereas with steering the total was 258GB, representing 14% of data reduction.

Analyzing monitoring overhead

A monitoring query 𝑚𝑞 ! in {𝑄} is run by a thread at each 𝑑 ! seconds. Depending on the number of threads (|{𝑄}|) and on the interval 𝑑 ! there may be too many concurrent accesses to the wfq 1 q 2 q 3 q 4 q 5 60.0 Database, which may add overhead. The goal of this experiment is to analyze such overhead.

We set up the Monitor module to run queries, which are variations of the queries 𝑄1-𝑄7 presented in Table 1 and Table 2. For example, in 𝑄2, we vary the curvature value. We also modify them to calculate only the results over the last 𝑑 seconds, at each 𝑑 seconds. To evaluate the overheads, we measure execution time without monitoring and then with monitoring, but varying the number of queries |{𝑄}| and the interval 𝑑, which is considered the same for all queries in {𝑄} in this experiment. The experiments were repeated until the standard deviation of workflow elapsed times was less than 1%. The results are the average of these times within the 1% margin. Figure 6 shows the results, where the gray portion represents the workflow execution time when no monitoring is used; and the black portion represents the difference the workflow execution time with and without monitoring (i.e., the monitoring overhead).

From these results, we observe that when the interval 𝑑 is equal to 30s, the overhead is negligible. When the interval is 1s, the overhead is higher when the number of monitoring threads is greater. This happens because three queries are executed in each time interval (see Figure 3), for each thread. In the scenarios with 30 threads, there will be 120 queries in a single time interval 𝑑. In that case, if 𝑑 is small (e.g., 𝑑 = 1), there are 120 queries being executed per second, just for the monitoring. The database that is queried by the monitors is also frequently queried by the SWMS engine, thus adding higher overhead. However, even in this specific scenario that shows higher overhead (|{𝑄}| = 30 and 𝑑 = 1), it is only 33s or 3.19% higher than when no monitoring is used. Most of the real monitoring cases do not need such frequent (every second) updates. If 30s is frequent enough for the user, there might be no added overhead, like in this test case.

We also evaluated the same scenarios without storing monitoring results in the wf-Database, but rather appending in CSV files, which is simpler. The results are nearly the same as in Figure 6. This suggests storing all monitoring results in the wf-Database at runtime, which enables users to submit powerful queries as they are generated, with all other provenance data. This would not be possible with a solution that appends data to CSV.

RELATED WORK

Considering our contributions, we discuss the SWMS with parallel capabilities with respect to human adaptation (especially data reduction), online provenance support, and monitoring features.

Although online human adaptation is the core of computational steering, there are few parallel SWMS [11][12][19] that support it. These solutions have monitoring services and are highly scalable, but do not allow for online data reduction as a means to reduce overall execution time. WorkWays [START_REF] Nguyen | WorkWays: interacting with scientific workflows[END_REF] is a powerful science gateway that enables users to steer and dynamically reduce data being processed online by dimension reduction or by reducing the range of some parameters, sharing similar motivations to our work. It uses Nimrod/K as its underlying parallel workflow engine, which is an extension of the Kepler workflow system [START_REF] Abramson | Towards massively parallel dynamic grid workflows[END_REF]. WorkWays presents several tools for user interaction in human-inthe-loop workflows, such as graphic user interfaces, data visualization, and interoperability among others. However, WorkWays does not provide for provenance representation and users do not have query access to simulation data, execution data, metadata, and provenance, all related in a database, which limits the power of online computational steering. For example, it prevents ad-hoc data analysis using both domain and workflow execution data, such as those presented in Table 1 and Table 2, which support the user in defining which slice of the dataset should be removed. In contrast, our work uses a robust in-memory distributed database system to manage and relate analytical data involved in the workflow execution. Moreover, the lack of provenance data support in WorkWays, either online or postmortem, does not support reproducibility and prevents from registering user adaptations, missing opportunities to determine in detail how specific user interactions influenced workflow results. Another notable SWMS example is WINGS/Pegasus [START_REF] Gil | Wings: Intelligent workflow-based design of computational experiments[END_REF], which especially focus on assisting users in automatic data discovery. It helps generating and executing multiple combinations of workflows based on user contraints, selecting appropriate input data, and eliminating workflows that are not viable. However, it differs from our solution in the sense that it tries to explore multiple workflows until finding the most suitable one, whereas we often model our experiments as one single scientific workflow to be processed. Also, it does not aim at putting users in the loop to actively eliminate subsets of an input dataset, especially based on extensive ad-hoc intermediary data analysis online. Additionally, as WorkWays, provenance data is not collected online, nor is it integrated with domain-specific and execution data for enhanced analysis.

While human adaptation is less explored in parallel SWMS, monitoring is widely supported in several existing SWMS [13][14]. For example, Pegasus [START_REF] Deelman | Pegasus, a workflow management system for science automation[END_REF] and Triana may be integrated to analytical tools such as Stampede [10][22], which provides a framework to monitor workflow executions and has rich capabilities for online performance monitoring, troubleshooting, and debugging. However, in these solutions, it is not possible to monitor workflow execution data associating them to provenance and domain data, as we do using queries to the wf-Database. To the best of our knowledge, there is no related work that allows for online data reduction based on a rich analytical support with adaptive monitoring and provenance registration of human adaptations in scientific workflows. These features allow for performance improvements of scientific workflows, while keeping data reduction consistency and provenance queries that can show the history of human-in-the-loop actions and results.

CONCLUSION

This work contributes to putting the human in the loop of scientific workflow systems, especially when users can actively steer and reduce data online to improve performance. As a solution to the input data reduction problem, we made use of a tuple-oriented algebraic approach that organizes workflow data to be processed as sets of tuples stored in a wf-Database, managed by an in-memory distributed database system at runtime. We developed a mechanism coupled to d-Chiron, a distributed version of Chiron SWMS, which allows for reducing data, while maintaining both data integrity and execution consistency. A major challenge to the problem of data reduction is to address which subset of the data should be removed. As a solution to this, we proposed an adaptive monitoring approach that aids users in analyzing partial result data at runtime. Based on the evaluation of input data elements and its corresponding results, the user may find which subset of the input data is not interesting for a particular execution, hence can be removed. The adaptive monitoring allows users not only to follow the evolution of the workflow, but also to dynamically adjust monitoring aspects during execution. We extended our previous workflow provenance data model to be able to represent provenance of the online data reduction actions by users and the monitoring results. Although we implemented our solution in d-Chiron, other SWMS could be used if provenance, execution, and domain dataflow data are managed in a database at runtime.

To validate our solution, we executed a data-intensive parameter sweep workflow based on a real case study from the oil and gas industry, running on a 936-cores cluster. A test case demonstrated how the user can monitor the execution, dynamically adapt monitoring settings, and especially remove uninteresting data to be processed, all during execution. Results for this test case show that the user interactions reduced the execution time by 37% comparing with the execution that processed the whole dataset. Although the test case was from the oil and gas domain, any other workflow application could have been used, as long as a domain expert can tell which slice is not interesting, removed with no harm to the final results.

To the best of our knowledge, this is the first work that explores user-steered online data reduction in scientific workflows steered by ad-hoc queries and adaptive monitoring, while maintaining provenance of user interactions. The results motivate us to extend our solution and explore different aspects that can be adapted by humans based on sophisticated workflow data analysis support. Our solution is currently dependent on the domain expert's knowledge to identify correlations between input and output data to determine which subsets are uninteresting. We plan to address in-situ data visualization based on the adaptive monitoring and interactive queries results and develop recommendation models to suggest correlations based on history stored in the wf-Database. Other future work include: enabling users to set priorities to different slices of the data in a way that the SWMS system will process critic slices before; improving usability of the system by developing intuitive user interfaces to decrease the learning curve, especially related to the query interface, to take full advantage of the wf-Database. We also plan to expand our experiments and analyze how reducing each specific type of data (relation tuples, raw data files not processed and not generated) impact final results.

Figure 2 .

 2 Figure 2. Relation 𝑹 with subsets 𝑷 and 𝑺, and a slice

1 . 4 .Figure 3 .

 143 Figure 3. Steps executed by each thread within a time interval.

Figure 4 .

 4 Figure 4. Extended PROV-Wf data module to accommodate modified tasks and monitoring

Figure 5 .

 5 Figure 5. Use case plot to analyze impact of user steering comparing Wind Speed (input) with Fatigue life (output).

Figure 6 .

 6 Results of adaptive monitoring overhead.

Table 1 . Domain dataflow provenance interactive queries.

 1 𝑸𝟑What are the top 5 raw data files that contain original data that are leading to lowest fatigue life value?𝑸𝟒What are the histograms and finite element meshes files related when computed fatigue life based on stress analysis is lower than 60?

𝑸𝟏

What is the average of the 10 environmental conditions that are leading to the largest fatigue life value?

𝑸𝟐 What are the water craft's hull conditions that are leading to risers' curvature lower than 800?

Table 3 .

 3 User_Query table description.

	Column name	Description
	query_id	Auto increment identifier
	slice_query	Query that selects the slice of the dataset to be removed.
	tasks_query	Query generated by the SWMS to retrieve the ready tasks associated.
	issued_time	Timestamp of the user interaction
		Field that determines how the user interacted.
		It could be "Removal", "Addition", and
	query_type	others. We currently only implemented
		"Removal" of tuples, but it can be extended in
		future work.
	user_id	Relationship with the user who issued the interaction query
	wkfid	To maintain relationship with the rest of workflow execution data.

Table 5 . Monitoring_Result table description.

 5 Similar to what we did for the Steer module, we also developed the module Monitor to facilitate utilization. The Monitor should start at any cluster node that is able to access the distributed database system and should start after the workflow execution has begun, whenever users want to monitor the workflow execution.Similar to what we did for the Steer module, we also developed the module Monitor to facilitate utilization. The Monitor should start at any cluster node that is able to access the distributed database system and should start after the workflow execution has begun, whenever users want to monitor the workflow execution.A command line starts the Monitor module that runs in background. It establishes a connection with the distributed database system (connection settings are provided in the XML configuration file). Chiron (and d-Chiron) makes use of this XML file to define the workflow design, workflow general settings, and other user-defined variables. Then, the Monitor program keeps querying the Monitoring_Query table at each 𝑠 to check if a new monitoring query was added. The default value for 𝑠 is 30s, as the time interval to check if monitoring queries were added or removed. However, users can customize this. After the Monitor has started, users can add (or remove) monitoring queries to (or from) the Monitoring_Query table. Currently, users can add monitoring queries using a command line to inform the SQL query to be executed at each time interval and the time interval. Whenever the Monitor module identifies that the user added a new monitoring query, it launches a new thread. Each thread is responsible for executing each monitoring query in

	Column name	Description
	monitoring_result_id Auto increment identifier
	monitoring_id	Relationship with the monitoring query that generated this result
	monitoring_values	Results of the monitoring_query

result_type Data type of the result values of both queries. Currently, "Integer", "Double", "Array[Integer]", and "Array[Double]" Monitoring_Query at each defined time interval. A thread is finished when a monitoring query is removed or when the workflow stops executing (in that case, all threads are finished).

  contain wind speed less than 16 Km/h. For this, the first user query 𝑞 ! is issued with a command line to the Steer module. User queries are represented with circles in the horizontal axis (Elapsed time) in Figure5. The exact time a user issued an interaction query is stored in User_Query table.The next markers after 𝑞 ! happens at 𝑡 = 210. Comparing with the previous monitoring mark, at 𝑡 = 180, we can observe that this Peter's steering (𝑞 ! ) increases the minimum wind speed values to be considered from 14.2 Km/h to 24.1 Km/h. Also, we observe a significant decrease in the slope of the largest values for fatigue life (10.6% lower). This means that the removal of these input data containing wind speed less than 16 Km/h made the SWMS not process data containing low wind speed which would lead to larger fatigue life results.Then, monitoring continues, but that slope decrease calls Peter's attention. To obtain a finer detail of what is happening, he decides to adjust the monitoring interval time (𝑑 ! and 𝑑 ! ) at runtime, by reducing to 10s to get monitoring feedbacks more frequently. We can observe that for both lines 𝑚𝑞𝑟 !! and 𝑚𝑞𝑟 !! , the markers become more frequent during 𝑡 = [220, 270]. This is because a monitoring is registered at every 10s. We highlight that, although in test use case we are only showing monitoring correlating wind speed and fatigue life, other monitoring correlations could also be analyzed and users can add, remove or adjust monitoring queries at any time during execution.After verifying that the results are reasonable, Peter decides to increase back the monitoring query intervals for both queries to 30s after 𝑡 = 270. He then observes that since 𝑞 ! , wind speed less than 25 Km/h are leading to large fatigue life values. Then, at 𝑡 = 310, he calls Steer again to issue 𝑞 ! that removes input data for wind speed < 25 Km/h. The next markers after 𝑞 ! shows that this steering made the wind speed value associated to large fatigue life be at least 30.5 Km/h and a decrease of 6.5% in large fatigue life values between 𝑡 = 300 𝑎𝑛𝑑 𝑡 = 330.

Table 6 . Provenance of slices removed by the user

 6

	Inter act.	Issued time (s)	Slice query	Number of removed data elements
	𝑞 !	190	wind_speed < 16	623
	𝑞 !	310	wind_speed < 25	373
	𝑞 !	370	wind_speed < 30	355
	𝑞 !	430	wind_speed < 34.5	115
	𝑞 !	520	wind_speed < 35.5	3

ACKNOWLEDGMENTS

This work was partially funded by CNPq, FAPERJ and Inria (MUSIC project), EU H2020 Programme and MCTI/RNP-Brazil (HPC4E grant no. 689772), and performed (for P. Valduriez) in the context of the Computational Biology Institute (www.ibcmontpellier.fr). The experiments were carried out using the Grid'5000 testbed (https://www.grid5000.fr).


Farnaz Sedighin
email: fsedighin@ee.sharif.edu

Massoud Babaie-Zadeh

Bertrand Rivet
email: bertrand.rivet@gipsa-lab.grenoble-inp.fr

Christian Jutten
email: christian.jutten@gipsa-lab.grenoble-inp.fr

Two Multimodal Approaches for Single Microphone Source Separation

Keywords: Single microphone source separation, Nonnegative matrix factorization, Multimodal source separation

In this paper, the problem of single microphone source separation via Nonnegative Matrix Factorization (NMF) by exploiting video information is addressed. Respective audio and video modalities coming from a single human speech usually have similar time changes. It means that changes in one of them usually corresponds to changes in the other one. So it is expected that activation coefficient matrices of their NMF decomposition are similar. Based on this similarity, in this paper the activation coefficient matrix of the video modality is used as an initialization for audio source separation via NMF. In addition, the mentioned similarity is used for post-processing and for clustering the rows of the activation coefficient matrix which were resulted from randomly initialized NMF. Simulation results confirm the effectiveness of the proposed multimodal approaches in single microphone source separation.

I. INTRODUCTION

Single microphone speech separation is a challenging task in source separation. In this problem, the observed signal is a superposition of a set of original speech signals, that is

x(t) = L i=1 s i (t), (1)
where x(t) is the observed signal, s i (t)'s are original sources and L is the number of sources. The goal is to separate the original sources from x(t). Different approaches have already been proposed for single microphone source separation. In [START_REF] Jang | Single-channel signal separation using time-domain basis functions[END_REF] single microphone source separation is achieved by learning a set of time domain basis functions. Single channel Independent Component Analysis (ICA), wavelet ICA, Ensemble Empirical Mode Decomposition (EEMD) ICA and using deep learning are some of other methods [START_REF] Mijović | Source separation from single-channel recordings by combining empirical-mode decomposition and independent component analysis[END_REF], [START_REF] Huang | Deep learning for monaural speech separation[END_REF].

One of the approaches used for single microphone source separation is Nonnegative Matrix Factorization (NMF) [START_REF] Wang | Investigating single-channel audio source separation methods based on non-negative matrix factorization[END_REF], [START_REF] Duong | Temporal annotation-based audio source separation using weighted nonnegative matrix factorization[END_REF]. This work has been partly supported by the European project ERC-2012-AdG-320684-CHESS.

NMF decomposes a nonnegative data matrix as the product of two matrices with nonnegative components 1 as [START_REF] Wang | Investigating single-channel audio source separation methods based on non-negative matrix factorization[END_REF], [START_REF] Duong | Temporal annotation-based audio source separation using weighted nonnegative matrix factorization[END_REF] V WH,

where V ∈ F ×N , W ∈ F ×K and H ∈ K×N . V is called observation matrix, W is called basis dictionary matrix and H is called activation coefficient matrix [START_REF] Wang | Investigating single-channel audio source separation methods based on non-negative matrix factorization[END_REF], [START_REF] Duong | Temporal annotation-based audio source separation using weighted nonnegative matrix factorization[END_REF].

For achieving good results in source separation via NMF, W should be clustered so that basis dictionary vectors corresponding to the same source are clustered together. So some algorithms for single microphone source separation via NMF such as [START_REF] Wang | Investigating single-channel audio source separation methods based on non-negative matrix factorization[END_REF] use a prior training for estimating W. In these algorithms, W is learned from training data and kept fixed during the updating process, so during the estimation process only H is updated. Manual clustering of basis vectors after NMF decomposition is also proposed in [START_REF] Wang | Investigating single-channel audio source separation methods based on non-negative matrix factorization[END_REF]. Initialization of W and H highly affects the quality of NMF decomposition. Good initialization results in a good decomposition. Many NMF based algorithms are initialized randomly, so they have to be run several times and the best final result is picked up [START_REF] Boutsidis | Svd based initialization: A head start for nonnegative matrix factorization[END_REF]. However, such a procedure can be expensive especially when the algorithm has a high computational load. So different initialization algorithms are proposed. In [START_REF] Boutsidis | Svd based initialization: A head start for nonnegative matrix factorization[END_REF] an initialization algorithm based on Singular Value Decomposition (SVD) named as NonNegative Double SVD (NNDSVD) is proposed for initializing W and H. In [START_REF] Kim | A method of initialization for nonnegative matrix factorization[END_REF] another initialization approach based on hierarchical clustering of attributes is presented. Other initialization algorithms can be found in [START_REF] Langville | Initializations for the nonnegative matrix factorization[END_REF]. For single microphone source separation, user guided audio source separation is proposed in [START_REF] Duong | Temporal annotation-based audio source separation using weighted nonnegative matrix factorization[END_REF], [START_REF] Ozerov | Multichannel nonnegative tensor factorization with structured constraints for user-guided audio source separation[END_REF]. In these algorithms, an end user is asked to manually annotate the activity of each source. This information is then used for initializing the activation coefficients matrix.

Recently, separating audio signals using multimodal nature of speech is studied in different source separation algorithms such as [START_REF] Rivet | Mixing audiovisual speech processing and blind source separation for the extraction of speech signals from convolutive mixtures[END_REF], [START_REF]Visual voice activity detection as a help for speech source separation from convolutive mixtures[END_REF]. Different aspects of a multimodal phenomenon are measured by different instruments [START_REF] Lahat | Challenges in multimodal data fusion[END_REF]. Each of these measurements is called a modality. A human speech is an example of a multimodal (bimodal) phenomenon which consists of audio and video modalities of the speaker. Bimodal nature of speech is also exploited in NMF for various applications such as speaker diarization [START_REF] Seichepine | Soft nonnegative matrix co-factorizationwith application to multimodal speaker diarization[END_REF].

In this paper, we use the bimodal nature of speech along with NMF to enhance the quality of single microphone source separation. Basic source separation approach of this paper is similar to [START_REF] Wang | Investigating single-channel audio source separation methods based on non-negative matrix factorization[END_REF] but no prior training or manual clustering of basis vectors is needed. Special initialization is used in this paper to force the algorithm to converge to a proper minimum. Moreover, instead of manually annotating the activity of each source, which is used in [START_REF] Duong | Temporal annotation-based audio source separation using weighted nonnegative matrix factorization[END_REF], the similarity of the activation coefficient matrices of the respective modalities is used. In addition, this similarity is used for post-processing of the NMF based single microphone source separation which is initialized randomly. This post-processing is done by clustering the rows of the activation coefficient matrix after NMF decomposition.

The paper is organized as follows. In Section II, NMF model and its usage for single microphone source separation is reviewed. The main ideas are presented in Section III and finally experimental results are presented in Section IV.

II. PRELIMINARIES

A. NMF Model

The goal of NMF is to factorize a matrix with nonnegative entries as the product of the two matrices with nonnegative entries, as in [START_REF] Mijović | Source separation from single-channel recordings by combining empirical-mode decomposition and independent component analysis[END_REF]. It can be achieved by solving the optimization problem [START_REF] Duong | Temporal annotation-based audio source separation using weighted nonnegative matrix factorization[END_REF] min

H≥0,W ≥0 D(V WH), (3)
where D is a measure of the difference between V and WH. Different functions have already been used as the above measure. One of the popular functions is Itakura-Saito (IS) divergence, defined as [START_REF] Duong | Temporal annotation-based audio source separation using weighted nonnegative matrix factorization[END_REF], [START_REF] Ozerov | Multichannel nonnegative tensor factorization with structured constraints for user-guided audio source separation[END_REF]

D IS (V Ṽ) = ij v(i, j) ṽ(i, j) -log v(i, j) ṽ(i, j) -1,
where Ṽ WH and v(i, j) and ṽ(i, j) are the (i, j)-th entries of V and Ṽ respectively. H and W are updated during an optimization process. For example, in [START_REF] Duong | Temporal annotation-based audio source separation using weighted nonnegative matrix factorization[END_REF], [START_REF] Ozerov | Multichannel nonnegative tensor factorization with structured constraints for user-guided audio source separation[END_REF] a multiplicative update rule for optimization of the above cost function is presented as follows,

h(a, µ) ← h(a, µ) i w(i, a)v(i, µ)/ṽ 2 (i, µ) k w(k, a)/ṽ(k, µ) , (4)
w(i, a) ← w(i, a) µ h(a, µ)v(i, µ)/ṽ 2 (i, µ)

ν h(a, ν)/ṽ(i, ν) , (5)
where w(i, a) and h(a, µ) are the components of W and H respectively.

B. Single Microphone Source Separation Based on NMF

The use of NMF for single microphone source separation is studied in different papers such as [START_REF] Wang | Investigating single-channel audio source separation methods based on non-negative matrix factorization[END_REF], [START_REF] Duong | Temporal annotation-based audio source separation using weighted nonnegative matrix factorization[END_REF], [START_REF] Badawy | On-the-fly audio source separation[END_REF]. Short Time Fourier Transform (STFT) matrix of a signal is a matrix whose n-th column is the Fourier transform of the n-th frame of that signal. The n-th frame of a time domain signal such as y(k) is calculated as

y n (k) = y(k + nM )W(k) k = 0, 1, ..., M -1, (6)
where y n (k) is the n-th frame of y and W is a finite-length window with length M , and M is the amount of the shift of the window. Since STFT is a linear transform, for a mixture of L sources

x(f, n) = L i=1 s i (f, n), (7)
where x(f, n) is the (f, n)-th component of the STFT matrix of the mixture and s i (f, n) is the (f, n)-th component of STFT matrix of the i-th source signal. The power spectrum matrix V is a matrix whose components are equal to

v(f, n) |x(f, n)| 2 . ( 8
)
V is a F × N matrix where F is the number of frequency bins and N is the number of time frames. The use of NMF for single microphone source separation is based on the assumption that the power spectrum matrix of the i-th source is factorized to W i H i [START_REF] Wang | Investigating single-channel audio source separation methods based on non-negative matrix factorization[END_REF]. So separation of sources is achieved by the following NMF decomposition [4]

V WH = L i=1 W i H i , (9)
where W = [W 1 , ..., W L ] and H = [H T 1 , ..., H T L ] T . Finally, by using Wiener filtering, the STFT matrix of the i-th separated signal is estimated as the matrix whose (f, n)-th component is given by

ŝi (f, n) = p i (f, n) ( L i=1 p i (f, n)) x(f, n), (10)
where

p i (f, n) is the (f, n)-th component of P i = W i H i .
In the above single microphone source separation approach, the main challenging task is grouping W to W i 's, so that each W i corresponds to the i-th source.

For achieving good results in single microphone source separation, in [START_REF] Wang | Investigating single-channel audio source separation methods based on non-negative matrix factorization[END_REF] W i is learned from training data for each individual source and then only H i is updated during the update procedure. However, training data is not always available so there are other algorithms like [START_REF] Duong | Temporal annotation-based audio source separation using weighted nonnegative matrix factorization[END_REF], [START_REF] Ozerov | Multichannel nonnegative tensor factorization with structured constraints for user-guided audio source separation[END_REF] that use manual annotation of activity of sources for initializing H. But, when the parameters of the NMF based single microphone source separation approach (W, H) are initialized randomly and no training data is available, clustering of basis vector after decomposition is needed. So in [START_REF] Wang | Investigating single-channel audio source separation methods based on non-negative matrix factorization[END_REF], this clustering of basis vectors has been done manually.

III. MAIN IDEAS

Bimodal nature of speech is exploited in different speech source separation algorithms [START_REF] Rivet | Mixing audiovisual speech processing and blind source separation for the extraction of speech signals from convolutive mixtures[END_REF], [START_REF]Visual voice activity detection as a help for speech source separation from convolutive mixtures[END_REF]. In this paper, we use the lip surface of the speaker as the video modality. Since respective audio and video modalities have a similar physical origin (human speech), changes in audio modality usually corresponds to changes in the lip surface of the speaker.

Due to this similarity, it is expected that the activation coefficient matrices of the NMF factorization of the power spectra of respective audio (H a ) and video (H v ) modalities have similar shapes. More specifically, it is expected that respective modalities have zeros in nearly the same components of their activation coefficients matrices. It means that their inactive periods are expected to be nearly the same. Based on this similarity two different ideas are proposed in the rest of the paper: one for initialization of the NMF based separation approach of Section II-B and one for clustering of basis vectors of NMF approach of Section II-B when the parameters (W, H) are initialized randomly and no training data is available. The ideas are compared with each other.

A. First Idea: Multimodal Initialization Approach

Since the update rules (4), ( 5) are multiplicative, zero components do not change during the update process. So the zero components of the finalized and the initialization matrices are mostly the same. Having in mind the similarity of activation coefficient matrices of respective modalities, in this paper we propose that the activation coefficient matrix of the video modality (H v ) is used as an initialization for the single microphone audio source separation. (Note that in this paper the video modality is a one dimensional signal consisting of the lip surface of the speaker extracting from its three dimensional video. So computing its STFT matrix, NMF decomposition of its power spectrum matrix and consequently computing its activation coefficient matrix is similar to what is done in the previous section. In addition, video modalities (lip surface signals) can have constant nonzero values during their inactive periods. In this situation, derivative of the lip surface signals is used. So, the lip surface signals are first differentiated and then their activation coefficient matrices are extracted.) So H v can be used instead of manual annotation of the activity of audio signals which is used in [START_REF] Duong | Temporal annotation-based audio source separation using weighted nonnegative matrix factorization[END_REF], [START_REF] Ozerov | Multichannel nonnegative tensor factorization with structured constraints for user-guided audio source separation[END_REF]. Therefore we initialize the single microphone audio source separation algorithm of [START_REF] Wang | Investigating single-channel audio source separation methods based on non-negative matrix factorization[END_REF], with the following initialization matrix

H a init = [H v 1 T , ..., H v i T , ..., H v L T ] T ,
where H v i is the activation coefficients matrix of the i-th video modality. Each row of H v i is normalized to have unity summation. It should be noted that rows of H v i 's have similar zero patterns but they are not exactly the same. The number of the rows of H v i (the model order of NMF decomposition of the video modalities) is set to a predetermined positive integer κ. H a init is then used as an initialization for the activation coefficient matrix in NMF decomposition of power spectrum matrix of the audio mixture. It is clear that for the NMF decomposition of the mixture of L signals, K is set to L × κ.

Zero components do not change during the update procedure, so the finalized H after the update procedure has mostly the same zero components as its initialization matrix (H a init ). Since the (κ(i -1) + 1 : κi)-th rows of the activation coefficient matrix have been initialized by H v i , it is expected that after the update procedure, the (κ(i -1) + 1 : κi)-th rows of the finalized H and the (κ(i -1) + 1 : κi)-th columns of the finalized W will correspond to s i . The original signals are then reconstructed using [START_REF] Rivet | Mixing audiovisual speech processing and blind source separation for the extraction of speech signals from convolutive mixtures[END_REF].

B. Second Idea: Multimodal Clustering Approach

The second idea is clustering of the basis vectors resulted from NMF based single microphone source separation when the parameters (W, H) are initialized randomly. As mentioned before, in [START_REF] Wang | Investigating single-channel audio source separation methods based on non-negative matrix factorization[END_REF] manual clustering of basis vectors is done after NMF decomposition. In this paper, it is proposed to use the similarity between activation coefficient matrices of modalities for clustering the rows of activation coefficient matrix and consequently the basis vectors. This should be noted again that this clustering is used after a randomly initialized NMF algorithm. The number of clusters is equal to the number of original sources (L). Center of the i-th cluster is the average of the rows of H v i , i.e.:

C i = κ j=1 H v i (j, :) κ , (11)
where C i is a row vector of size 1 × N (N is the number of the columns of H v i ) and is the center of the i-th cluster. H v i (j, :) is the j-th row of H v i . Since activation coefficient matrices of respective audio and video modalities have nearly the same form, it is expected that the matrix [C T i , H(j, :) T ] T

2×N

be nearly a rank one matrix. Matrix rank can be measured using the following criterion [START_REF] Bhattacharya | Analysis and characterization of photo-plethysmographic signal[END_REF]

ρ ij ([C T i , H(j, :) T ] T ) = σ 2 1 p m=2 σ 2 m , (12)
where H(j, :) is the j-th row of the activation coefficient matrix of the mixture, σ i is the i-th singular value of the SVD of [C T i , H(j, :) T ] T and p is the number of singular values. Then H(j, :) is clustered to the m-th cluster if

ρ mj ([C T m , H(j, :) T ] T ) > ρ ij ([C T i , H(j, :) T ] T ) ∀i = m.
The rows of H belong to the m-th cluster are clustered into H m c = [H(m 1 , :) T , ..., H(m κ , :) T ] T , where m 1 , ..., m κ are the rows of H belong to the m-th cluster. So the clustered activation coefficient matrix is calculated as

H c = [H 1 c T , H 2 c T , ..., H L c T ] T . ( 13
)
then W is also clustered to W c such that it corresponds to the above clustering in H. Separated sources then reconstructed using H c and W c .

IV. NUMERICAL EXPERIMENTS

In this Section, the effect of the proposed multimodal initialization and multimodal clustering approaches for source separation is investigated. Signals used for simulation are pairs of audio and video modalities of different speeches [START_REF] Sodoyer | A study of lip movements during spontaneous dialog and its application to voice activity detection[END_REF]. It should be noted that in this paper, the one dimensional signal of the lip surface of the speaker is used as the video modality. The audio signals are sampled at 16 KHz and the lip surface signals are sampled at 50 Hz. So the lip surface signals are upsampled by factor 320 using "interp.m" function of MATLAB. Mixtures are produced by mixing two real audio signals (L = 2). There is a video modality (lip surface signal) correspond to each of the audio signals. Each of the respective pairs of audio and lip surface signals are modalities of a human speech. It is worth noting that there is no mixture of video modalities, i.e., each video modality corresponds to a single speech. The duration of the original audio signals is 32 sec. NMF decomposition of lip surface modalities is initialized randomly. STFT is computed using 0.0625 sec length window (1024 samples). Quality of the separated signals is measured using the following SNR SNR = 10 log 10 i,j S(i, j)

2 i,j (S(i, j) -Ŝ(i, j)) 2 , ( 14
)
where S and Ŝ are the magnitude spectra of original and estimated signals respectively. In the first simulation, the effect of the proposed initialization approach (first idea) comparing to NNDSVD and random initialization approaches is investigated. 10 mixtures are separated using NMF with the proposed, NNDSVD and random initialization approaches. κ is set to 10. Source separation results are presented in Table I. The first two columns of the table ( Ŝ1MM and Ŝ2MM ) correspond to the SNRs of the two separated signals using the proposed initialization. Ŝ1SVD and Ŝ2SVD correspond to NNDSVD initialized separation algorithm and the two last columns correspond to the randomly initialized separation algorithm. It is inferred from the results that NMF separation algorithm with the proposed multimodal initialization has a better quality compared to NNDSVD and randomly initialized NMF algorithms. As mentioned before, this better quality is due to the proposed initialization which prevents disordering of the rows of the matrix H.

In the second simulation, the proposed clustering algorithm (second idea) is investigated. In Table II, the quality of the classical NMF (randomly initialized NMF) source separation algorithm with the proposed clustering algorithm is compared to the classical NMF without clustering of basis vectors. In addition, the proposed initialization approach and the proposed clustering algorithms are also compared with each other in Table II. In this simulation κ is set to 5.

In Table II, Ŝ1c and Ŝ2c are the separated signals estimated after the proposed clustering algorithm. Ŝ1MM , Ŝ2MM , Ŝ1 and Ŝ2 are defined before. It is clear from the results that the proposed clustering algorithm increases the quality of source separation comparing to the randomly initialized NMF source separation algorithm without clustering. But the quality of source separation algorithm with the proposed initialization is greater than the randomly initialized algorithms with or without clustering.

In the third simulation, the situation when the video information (lip surface signal) is only available for one of the sources, say s 1 , is investigated. This occurs when we are only interested to extract one of the sources from the mixture. So only first κ rows of H are initialized by using the video information and the rest of the rows are initialized randomly. Separation results for the mentioned, NNDSVD and randomly initialized NMF based single microphone source separation approach are presented in Table III It can be inferred from the results that using the video information of only one of the sources increases the quality of separation comparing to NNDSVD and random initializations. However, separation quality using the video information of only one of the sources is less than the separation quality when the video information of all of the sources is used. So separation quality for the mentioned initializations can be sorted as: multimodal initialization for all of the sources > multimodal initialization for one of the sources> NNDSVD initialization ≥ random initialization.

V. CONCLUSION

In this paper, single microphone source separation via NMF was addressed. We proposed to use the similarity of activation coefficient matrices of audio and video modalities coming from a single speech to initialize an already known approach for single microphone speech separation based on NMF (Section II-B). In addition, this similarity was used for clustering of basis vectors when the parameters of the mentioned approach are initialized randomly without using any training data. Then the effectiveness of the proposed algorithms was verified by some simulations. The experiments show the crucial aspect of initialization for NMF. Moreover, the proposed multimodal approach is a good choice for initializing the NMF based single microphone source separation approach even if only one video modality is known. So based on the proposed algorithms, single microphone source separation can be achieved without using any training data, manual clustering of basis vectors or manual annotation the activity of sources.

  . The first two columns of the Table III ( S1MM and S2MM ) correspond to the mentioned initialization. SiMM is used instead of ŜiMM to indicate that only first κ rows of H are initialized by using video (lip surface signal) information.

TABLE III SNR

 III (IN dB) FOR THE SEPARATED SIGNALS WHEN THE VIDEO INFORMATION IS AVAILABLE ONLY FOR THE FIRST SOURCE COMPARING TO NNDSVD AND RANDOM INITIALIZATIONS FOR 10 TRIALS.

	#	S1MM	S2MM	Ŝ1SVD	Ŝ2SVD	Ŝ1	Ŝ2
	1	5.80	8.69	-0.72	1.97	-1.94	1.21
	2	3.50	6.14	-1.29	0.56	1.91	4.25
	3	1.99	3.91	0.02	1.93	0.30	2.75
	4	4.03	3.10	1.85	-0.38	1.51	-0.54
	5	1.87	2.29	2.06	0.91	0.88	1.58
	6	2.15	1.51	1.85	-0.31	-0.01 -1.83
	7	3.26	-0.99	4.11	-1.42	2.22	-2.40
	8	3.05	-0.89	3.50	-0.72	1.80	-1.62
	9	3.26	1.18	5.70	0.31	1.82	-1.91
	10	0.89	3.51	0.29	0.46	-0.47	0.05
	avg	2.98	2.84	1.73	0.33	0.8	0.15

We will explain in Section II how the model (1) leads to (2) in the timefrequency domain, using Short Time Fourier Transform.