Suraj Kumar
email: suraj.kumar@pnnl.gov

Lionel Eyraud-Dubois
email: lionel.eyraud-dubois@inria.fr

Sriram Krishnamoorthy
email: sriram@pnnl.gov

Performance Models for Data Transfers: A Case Study with Molecular Chemistry Kernels

Keywords: • Computer systems organization → High Performance Computing, • Computing Methodologies → Modeling and Simulation, • General → Performance Communication Scheduling, Memory Nodes, Runtime Systems, Communication-Computation Overlap, Molecular Chemistry

In distributed memory systems, it is paramount to develop strategies to overlap the data transfers between memory nodes with the computations in order to exploit their full potential. In this paper, we consider the problem of determining the order of data transfers between two memory nodes for a set of independent tasks with the objective of minimizing the makespan. We prove that, with limited memory capacity, the problem of obtaining the optimal data transfer order is NP-complete. We propose several heuristics to determine this order and discuss the conditions that might be favorable to different heuristics. We analyze our heuristics on traces obtained by running two molecular chemistry kernels, namely, Hartree-Fock (HF) and Coupled Cluster Singles Doubles (CCSD), on 10 nodes of an HPC system. Our results show that some of our heuristics achieve significant overlap for moderate memory capacities and resulting in makespans that are very close to the lower bound.

CCS CONCEPTS

INTRODUCTION

With the advent of multicore, and the use of accelerators, it is notoriously cumbersome to exploit the full capability of a machine. Indeed, there are several challenges that come into picture. First, every architecture provides its own efficacy and interface. Therefore, a steep learning curve is required for programmers to effectively utilize all resources. Second, scheduling is a well known NP-complete optimization problem, and hybrid and distributed resources make this problem harder (we refer Brucker and Knust [9] for a survey on the complexity of scheduling problems and Bleuse et al. [7] for a recent survey in the case of hybrid nodes). Third, due to shared buses and parallel resources, it is challenging to obtain a precise model based on prediction of computation and communication times. Fourth, the number of architectures has increased drastically in recent years, making it almost impossible to develop hand-tuned optimized code for all these architectures. All these observations led to the development of different task-based runtime systems. Among several runtimes, we may cite QUARK [26] and PaRSEC [8] from ICL, Univ. of Tennessee Knoxville (USA), StarPU [4] from Inria Bordeaux (France), Legion [5] from Stanford Univ. (USA), StarSs [10] from Barcelona Supercomputing Center (Spain), KAAPI [14] from Inria Grenoble (France). All these runtime systems allow programmers to express their algorithms at the abstract level in the form of direct acyclic graphs (DAG), where vertices represent computations and edges represent dependencies among them. Sometimes some static information is also provided along with the DAG, such as distance to exit (last) node as a priority or affinity of computation toward resources. The runtime is then responsible for managing the schedule of computations and communications, data transfer among different memories, computation-communication overlap, and load balance.

In the last few decades, we have witnessed a drastic improvement in the hardware to provide higher rate of computation, but comparatively smaller improvement has been achieved for the rate of data movement. With extreme-scale computing, supercomputers face bottlenecks due to the need for large amount of data [3,27]. Therefore, the HPC community is now focused on avoiding, hiding, and minimizing communication costs.

Certain applications such as dense linear algebra kernels have regular structure. Therefore, it is possible to associate priorities to computations, based on the task graph structure, and to use them at runtime to make the execution efficient. In irregular applications, programmers do not know the precise structure of the task graphs in advance: tasks are added recursively based on certain sentinel constraints. For such applications, the runtime system sees a set of tasks and schedules them on processing units of different memory spaces. It is extremely important for runtimes to decide the order of data transfers for these scheduled computations so as to maximize the overlap between computation and communication. This is the main topic of this paper. We prove that determining the order of communications on two memory nodes with the objective of minimizing the makespan is a NP-complete problem if the memory of the target node is limited. Our proof is inspired by [19], which applies a similar technique for 2-machine flowshop problem with bounded capacity. The main difference between both approaches is that they consider all tasks have the same occupation on the second machine and the memory occupation starts when the processing finishes on the first machine. On the contrary, our approach is designed for tasks appearing in scientific workloads whose memory requirements are highly irregular and we consider that memory is acquired before starting the data transfer on the communication resource. We propose different runtime strategies in order to maximize the computation-communication overlap. We evaluate our strategies on the context of a cluster of homogeneous nodes. However, our approach is generic and easily adaptable to any system that operates on different memory spaces. Here are the important contributions of this paper:

• NP-completeness proof for the general data-transfer problem • Different scheduling strategies with the objective to minimize the makespan • Linear programming formulation of the problem • Numerous experiments to assess the effectiveness of our strategies on molecular chemistry kernels

The rest of the paper is organized as follows. Section 2 describes past work on the computations with limited memory and similar problems in the literature. In Section 3, we present an algorithm to obtain the order of data transfers when there is not any memory capacity restriction. Also, we prove that, in general, the data transfer problem is NP-complete. In Section 4, we propose several heuristics and discuss the conditions that might be favorable to them. We mainly consider three categories of heuristics: static order based heuristics, dynamic selection based heuristics, and static order with dynamic correction based heuristics. Sections 5 describes our experimental setup and we evaluate our proposed heuristics on two molecular chemistry kernels in Section 6. Our results show that static order with dynamic correction based heuristics achieve good performance in most cases. We finally present conclusions and perspectives in Section 7.

RELATED WORK

Historically there has been a great emphasis on the development of parallel algorithms and minimizing the complexity of computations. As the number of computation cores has increased in recent years, supercomputers face bottlenecks due to the communication required by an application. Hence, the focus has shifted toward developing communication avoiding algorithms, strategies to hiding communications, and minimizing the data accessed by applications [27].

The problem of scheduling tasks has been highly studied in the literature and many formulations are known to be NP-complete [11]. Many static and dynamic strategies have been proposed and analyzed for scheduling tasks on heterogeneous resources [1,6,24]. There is also a number of studies in the direction of task scheduling with the emphasis on improving locality and minimizing the communication cost [4,24]. Stanisic et al. [23] proposed a heuristic to schedule tasks on a computational resource where most of its data is available. A similar approach has been adopted by Agullo et al. for the scheduling of sparse linear algebra kernels [2]. Predari et al. proposed heuristics to partition the task graph across a number of processors such that inter-processor communication can be minimized [20].

The problem we consider also can be viewed as a flowshop problem: the communication link can be seen as a processing resource, and each task needs to first be handled by the communication link and then by the computational resource. Communication and computation times of a task can thus be considered as processing times on different machines. Johnson presented scheduling strategies for 2 and 3-machine flowshop problems with infinite memory capacity [15]. Two-machine flowshop problem with finite buffer was proven NP-complete by Papadimitriou and Kanellakis [19], in which a constraint is imposed on the number of tasks that can await execution on the second machine.

A

PROBLEM FORMULATION

To exploit the full potential of a system it may be necessary to execute tasks on processing units where all of their data does not reside. A task may require all of its input data in local memory before starting the computation. There may be multiple tasks scheduled on a processing unit, which require to transfer data from the same memory node. Ordering data transfers for such tasks is very crucial for the communication-computation overlap, thus for the overall performance. In general, order of task execution with input and output data transfers can be viewed as a 3-machine flowshop problem, where processing time on the first machine is input data transfer time, processing time on the second machine is task computation time, and processing time on the third machine is output data transfer time; and the objective is to minimize the total makespan. This is a well-known NP-complete problem [12].

In many cases, output data to be retrieved after task execution is much smaller than the input data. It is often the case that future tasks running on the same memory node require output data of the past tasks. Therefore, most runtime systems transfer data to other memory nodes based on the demand -not immediately after they were produced. It is also possible that all output data can be stored in a preallocated separate buffer on a memory node. We do not consider output data explicitly in our analysis and assume that output data is negligible or stored in a separate buffer for each task. Thus the problem we consider is more similar to a 2-machine flowshop problem. We prove that ordering the execution of such tasks with finite memory capacity is a NP-complete problem:

Problem DT : A set of tasks ST = {T 1 , • • • ,T n }
is scheduled on a processing unit P with memory unit M of capacity C. Input data for tasks of ST reside on another memory unit M ′ . COMM i is the communication time to transfer input data from M ′ to M for task i and COMP i is the computation time of task i on P. We assume that these tasks do not produce any output data. There can be only one communication at a time, and P can only process one task at a time. A task uses an amount of memory in M from the start of its communication to the end of its computation. Given L, is there a feasible schedule S for ST such that makespan of S, µ(S) ≤ L?

Given a schedule, S COMM (i) and S COMP (i) represent the start times of task i on communication and computation resources. A schedule is feasible if for every time t, the amount of memory required by all tasks such that S COMM (i) ≤ t ≤ S COMP (i) + COMP i is not more than the memory capacity C. For simplicity, we assume throughout the paper that tasks require memory only to store their input data, and thus that the amount of memory required by a task is proportional to its communication time. Without loss of generality, we consider in all examples of Sections 3 and 4 that the memory requirement of a task is equal to its communication time.

We call a task i compute intensive if COMP i ≥ COMM i , and communication intensive otherwise.

Special Case: Infinite Memory

When the computational resource has a very large memory, our problem becomes a classic 2-machine flowshop problem: communication time is the processing time on the first machine and computation time is the processing time on the second machine. Johnson's algorithm [15] is known to provide an ordering for the tasks which results in an optimal makespan. This algorithm is rewritten in Algorithm 1.

Algorithm 1: Johnson's [15] algorithm (infinite memory case).

1: Divide ready tasks in two sets S 1 and S 2 . If computation time of a task T is not less than its communication time, then T is in S 1 otherwise in S 2 . 2: Sort S 1 in queue Q by non-decreasing communication times 3: Sort S 2 in queue Q ′ by non-increasing computation times

4: Append Q ′ to Q 5: τ COMM ← 0 {Available time of communication resource} 6: τ COMP ← 0 {Available time of computation resource} 7: while Q ∅ do 8:
Remove a task T from beginning of Q for processing 9:

S COMM (T ) ← τ COMM 10: S COMP (T ) ← max(S COMM (T ) + COMM T , τ COMP ) 11: τ COMM ← S COMM (T ) + COMM T 12: τ COMP ← S COMP (T ) + COMP T 13: end while
We provide an alternative optimality proof of Algorithm 1 in the extended version [16].

Finite Memory

We now consider the general case, in which the memory limit is a constraint for the schedule. This is related to previous work by Papadimitriou and Kanellakis [19], in which the second machine can only handle a bounded number of tasks. Our problem generalizes this work to heterogeneous memory consumption among tasks, with an additional difference: memory usage starts at the beginning of the first part of a task (instead of at the end of the first part). This requires us to provide a slightly different NP-completeness proof, as given below.

T 3.1. Problem DT is NP-complete.

P

. It is easy to see that the DT belongs in NP: given a schedule, one can check in linear time that at each start of a communication, the memory constraint is satisfied, and that task starts computation only after its input data is transferred to M.

To prove NP-hardness, we use a reduction from the well-known 3-Partition NP-complete problem [11]:

Three Partition Problem (3P ): Given a set of 3m integers A = {a 1 , • • • , a 3m }, is there a partition of A into m tripletsT R i = {a i 1 , a i 2 , a i 3 }, such that ∀i, a i 1 + a i 2 + a i 3 = b, where b = (1/m) a i ?
Let us first show that 3P problem reduces in polynomial time to problem DT . Suppose that we are given an instance

A = {a 1 , • • • , a 3m } of 3P .
It is immediately obvious that a i > 1, since we can always add sufficiently large integers to the a i values and scale the problem accordingly. This scaling will not affect in any way the existence of a solution for the instance of 3P problem.

From such an instance, we define x = max {a i : 1 ≤ i ≤ 3m}, and we construct an instance I of the problem DT with 4m + 1 tasks, whose characteristics are given in Table 1.

Task

Communication time Computation time

K 0 0 3 K 1 , • • • , K m-1 b ′ = b + 6x 3 K m b ′ = b + 6x 0 1 ≤ i ≤ 3m, A i 1 a ′ i = a i + 2x Memory capacity: C = b ′ + 3 Target makespan: L = m(b ′ + 3)
Table 1: Definition of tasks in the reduction from 3P .

We show that I has a schedule S with makespan at most L if and only if the original 3P instance has a solution. Notice that the sum of communication times and the sum of computation times are both equal to L, therefore a valid schedule of makespan at most L has makespan exactly L, with no idle time on both resources. It indicates that the first task is K 0 and the last task is

K m . t Comp. Comm. A 1,1 A 1,2 A 1,3 K 0 K 1 A 1,1 A 1,2 A 1,3 A 2,1 A 2,2 A 2,3 K 1 3 3 b ′ Figure 1: Pattern of feasible schedule S.
If the 3P instance has a solution, A can be partitioned into

m triplets T R i = {a i 1 , a i 2 , a i 3 } such that ∀i, a i 1 + a i 2 + a i 3 = b,
then we can construct a feasible schedule S without idle times by the pattern depicted in Figure 1. The communications of tasks in T R i take place during the computation of task K i-1 , and the computations of tasks in T R i take place during the communication of task K i . Since the memory capacity is C = b ′ + 3, all tasks from a triplet can fit in memory with a task K i , and their computation times are exactly equal to the communication time of K i . This schedule is thus feasible, and has length exactly L.

We now prove that any feasible schedule of I corresponds to a valid decomposition of A for 3P . Indeed, we argue that every feasible schedule has to consist of m segments like the one shown in Figure 1. Each segment provides a triplet {a i 1 , a i 2 , a i 3 } such that

a i 1 + a i 2 + a i 3 = b.
Any schedule S of I having no idle time must start with K 0 . We first show that no other K i task can be active with K 0 , otherwise we would get idle time on the computation resource. Indeed, the communication of such a task K i would end at time at least b ′ > 3 + 6x, but at most two A i tasks can be computed, and they end at time at most 3 + 2max {a ′ i : 1 ≤ i ≤ 3m} = 3 + 6x. Hence three A i tasks must follow K 0 . The memory requirement of other K i tasks is b ′ and 2b ′ > C, therefore at any point in the schedule at most one K i task can be active. Since the total duration of all

K i tasks is 3 + (m -1)(b ′ + 3) + b ′ = m(b ′ + 3) = L, at each point in S exactly one K i task is active.
With these K i tasks in place, the schedule on the computation resource contains m slots of length exactly b ′ , in which all 3m A i tasks must fit without preemption. We thus define T R i as the set of tasks which execute during the communication phase of task K i . At each point in S, exactly one K i task is active and the total memory capacity is b ′ + 3, hence T R i contains exactly 3 A i tasks. Since S has no idle time on the computation resource, the total computation time of tasks in T R i is exactly b ′ , and thus a i 1 + a i 2 + a i 3 = b. This partition is thus a valid solution for the 3P instance A.

This theorem shows that adding a memory constraint to our problem makes it more difficult, just like it does for limited capacity 2-machine flowshop [19]. One additional difficulty for our problem is that it may not be optimal to consider the same order on both machines.

P 3.2.

There exists an instance of DT for which in all optimal schedules, the communication order of tasks is different from their computation order.

Task Memory Req Comm Time Comp Time

A 0 0 5 B 4 4 3 C 1 1 6 D 3 3 7 E 6 6 0.5 F 7 7 0.5
Table 2: Example instance where ordering on both resources has to be different.

P

. Consider the instance described on Table 2, in which memory capacity is C = 10. Figure 2a shows the best possible schedule when tasks are scheduled in the same order on both resources (obtained by exhaustive search). On the other hand, Figure 2b shows another schedule with lower makespan, in which the order is different.

In the infinite memory case, the standard proof that an optimal schedule exists with the same order on both resources claims that it is possible to swap two tasks which do not satisfy this property. On Figure 2

DATA TRANSFER ORDER HEURISTICS

Algorithm 1 presented in Section 3 achieves an optimal makespan when there is no memory constraint. This optimal value indicates a lower bound on the makespan of the constrained case. We denote this value with optimal makespan infinite memory (OMI M). In this section, we propose different heuristics for the limited memory case, and we assess their efficiency with respect to OMI M in Section 6. We classify our heuristics into mainly three categories. In the first category, the order of all computations and communications is computed in advance and the same order is followed on both resources. In the second category, the next task to schedule is dynamically chosen based on different criteria. The final category is based on combining strategies from the first two categories: a static order is precomputed and corrected dynamically to avoid idle time caused by memory limitations. In all of our strategies (except linear programming based strategy), communication and computation orders of tasks are same.

Static Ordering

In this class of strategies, we compute the order of processing in advance based on criteria such as communication time and computation time. After computing the order, we follow the same sequence on computation and communication resources and make sure that the memory constraint is respected at each point in the schedule.

In Algorithm 1, compute intensive tasks are sorted in increasing order of communication times. It allows tasks to utilize the computation resource maximally and make enough margin on the communication resource to accommodate more communication intensive tasks with maximum overlap. Communication intensive tasks are sorted in decreasing order of computation time, which allows tasks to utilize the margin created on communication resource. Hence, in this section, we obtain the orders by sorting tasks based on different combinations of communication and computation times. 3.

Task Memory Req Comm Time Comp Time A 3 3 2 B 1 1 3 C 4 4 4 D 2 2 1
Table 3: A task set for static order schedules.

i) order of optimal strategy infinite memory (OOSI M): This heuristic uses the order given by Algorithm 1, but respects the memory constraint at each point in the schedule. Hence the makespan of this heuristic may be completely different from OMI M. ii) increasing order of communication strategy (IOCMS): Tasks are ordered in non-decreasing order of communication time. iii) decreasing order of computation strategy (DOCPS): Tasks are ordered in non-increasing order of computation time. iv) increasing order of communication plus computation strategy (IOCCS): Tasks are ordered in non-decreasing order of the sum of their communication and computation times. v) decreasing order of communication plus computation strategy (DOCCS): Tasks are ordered in non-increasing order of the sum of their communication and computation times.

In order to highlight the different behaviors of these static heuristics, we propose on Table 3 an example instance, and on Figure 3 the corresponding schedules for all these heuristics.

Dynamic Selection

Dynamic strategies are based on the following principle: when the communication resource is idle, a task is chosen based on a selection criterion which differs depending on the heuristic, among those which fit in memory and induce minimum idle time on the computation resource. For example, if the selection criterion is to choose a highly compute intensive task, then we compute the ratio of computation time and communication time for all tasks, and we select a task with the maximum ratio among those which induce minimum idle time on the computation resource and fit in the currently available memory. If no task fits in memory then we leave the resource idle at that point and proceed to the next event point.

i) largest communication task respects memory restriction (LCMR):

A task with the largest communication time is chosen. ii) smallest communication task respects memory restriction (SCMR):

A task with the smallest communication time is chosen. iii) maximum accelerated task respects memory restriction (MAMR):

A task with the maximum ratio of computation time to communication time is chosen.

Similar to the schedules of the previous class, Figure 4 highlights the schedules of this class of heuristics for the example instance of Table 4.

Static Order with Dynamic Correction

In this class of strategies, we precompute the order of tasks based on some criterion and then follow this ordering as much as possible. But when the communication resource is idle because the memory requirement of the next task is too high, then we select a task with a dynamic strategy. After a task is selected, we update the remaining order without this task. This class of strategies takes advantage of static information in the form of precomputed order and dynamic correction to minimize the idle time due to memory constraint.

In all strategies of this class, the initial order is OMI M order obtained by Algorithm 1. We define the following heuristics based on how we select a task from the set of tasks which fit in memory and induce minimum idle time on the computation resource. If no task fits in memory then we leave the resource idle and forward to the next event point.

i) optimal order infinite memory largest communication task respects memory restriction (OOLCMR): A task with the largest communication time is chosen from the set. ii) optimal order infinite memory smallest communication task respects memory restriction (OOSCMR): A task with the smallest communication time is chosen from the set. iii) optimal order infinite memory maximum accelerated task respects memory restriction (OOMAMR): A task with the maximum ratio of computation time to communication time is chosen from the set. As previously, we propose on Table 5 an example instance for this class of heuristics, and the corresponding schedules in Figure 5.

Additional Heuristics from Previous Work

We also consider two other static heuristics for evaluation. The first heuristic is based on an algorithm, proposed by Gilmore and Gomory, to obtain the minimal cost sequence for a set of jobs [13]. This is a classical algorithm for 2-machine no-wait flowshop problem. In this algorithm, each job has a start and end state and a cost is associated to change the state. In our context, this cost can be seen as non-overlap time of computation for two adjacent tasks. Here is the main idea of this algorithm. Initially, a partial sequence of jobs is represented by a graph such that their overlap is maximum. Subsequently edges are greedily added to this graph to connect two components while minimizing the total non-overlap cost. When the graph is connected, an edge interchange mechanism is used to determine the sequence of jobs, which ensures that the sequence has minimal cost. We refer the original paper [13] for more details. This algorithm does not take memory constraints into account and only provides the sequence of processing. We use this sequence with a memory capacity restriction just like for other static heuristics, and we call this heuristic Gilmore-Gomory (GG).

The second heuristic is based on the First-Fit algorithm for the bin packing problem. The idea of this heuristic is to identify groups of tasks which can fit in memory together, called bins. In First-Fit, tasks are considered in an arbitrary order and added to the first bin in which they can fit. If no suitable bin is found then a new bin is created and this task is added to it. When all tasks have been assigned to bins, we consider the sequence made of all tasks from the first bin, then tasks for the second bin, and so on. We call this heuristic Bin Packing (BP).

Solving Linear Program Iteratively

We use a mixed integer linear program to obtain the order of communications and computations. Recall that COMP i and COMM i represent computation and communication times of task i, and the memory capacity of the target system is C. In the linear program formulation, s i and e i (resp. s ′ i and e ′ i ) represent the start and end times of communication (resp. computation) for task i, and MC(i) is the memory capacity requirement of task The formulation also contains for each pair of tasks i and j i) a boolean variable a i j to denote the order of i and j on the communication resource ii) a boolean variable b i j to denote the order of i and j on the computation resource, and iii) a boolean variable c i j to denote the order of s i and e ′ j . Let L = i (COMP i + COMM i ). It is evident that e i = s i + COMM i and e ′ i = s ′ i + COMP i . The linear program is given below. Minimize l subject to:

∀i, e ′ i ≤ l (task i completes)

∀i, e i ≤ s ′ i (task i valid ordering)

∀i, ∀j i, e j ≤ s i + (1a i j )L

e i ≤ s j + a i j L (exclusive use of communication link) ∀i, ∀j i, e ′ j ≤ s ′ i + (1 -b i j )L e ′ i ≤ s ′ j + b i j L (exclusive use of computation resource) ∀i, ∀j i, e ′ j ≤ s i + (1 -c i j )L s i < e ′ j + c i j L (respect ordering of variables c i j ) ∀i, r i (a ir -c ir )MC(r ) + MC(i) ≤ C (memory constraint)
The value 0 for a i j (resp. b i j ) indicates that task i ends before task j starts on the communication (resp. computation) resource, while 1 indicates task j ends before task i starts. If task j ends on the computation resource after task i starts on the communication resource then c i j attains the value 0 else 1. We use GLPK solver v4.65 to solve the above formulation. We also add the following constraints to help the solver: ∀i, ∀j i, a i j + a ji = 1, b i j + b ji = 1, c i j ≤ a i j , c i j ≤ b i j , and c i j + c ji ≤ 1. The solver was unable to solve this MILP at the scale of our interest in limited time. Hence, we solve the linear program iteratively for a small subset of size k = 3, 4, 5, 6: at the boundary of successive iterations we fix the event (communication or computation) of an unfinished task started before the boundary point and consider other events flexible. The subsets are formed in the order in which tasks are submitted, which is arbitrary. We compute the makespan of the complete set by applying above linear program iteratively on a subset of size k and represent it by lp.k. We assess various lp.k values for different memory capacities and observe that most of the other heuristics perform better than this heuristic. Hence, we do not include this heuristic for the comparison in Section 6. Performance comparison of different heuristics with MILP based heuristics is available in the extended version [16].

Favorable Situations for Heuristics

Based on the definition of proposed heuristics and the optimality of Algorithm 1, we discuss the scenarios which might be more favorable for each heuristic in Table 6. This allows programmers to use appropriate strategies to maximize communication-computation overlap for their applications. In this table, "moderate memory capacity" refers to the case where memory is constrained, but close to the maximal memory requirement of the OMI M schedule.

Heuristic

Favorable Some of these favorable scenarios can be clearly observed in our experimental results, on Figures 7 and9. For example, HF compute intensive tasks have small communication times, which explains why the SCMR heuristic exhibits very good performance in limited memory cases. CCSD has significant percentage of large as well as small types of slightly communication intensive tasks, and indeed the performance of OOLCMR and OOSCMR is very close to optimal in moderate memory cases.

EXPERIMENTAL SETTINGS

We consider a machine called Cascade1 , available at PNNL, for our experiments. We obtain traces by running two molecular chemistry applications, double precision version of HF and CCSD of NWChem [25] package on 10 nodes of this machine. Each node is composed of 16 Intel Xeon E5-2670 cores. NWChem takes advantages of a Partitioned Global Address Space Programming Model called Global Arrays (GA) [18] to use shared-memory programming APIs on distributed memory computers. GA dedicates one core of each node to handle communication, hence we can view a node as being composed of 15 computational cores. We use 150 processes for each application and obtain 150 trace files. We run CCSD with Uracil molecules input and HF with SiOSi molecules (for Uracil molecules, HF has a much smaller workload, each processor executes only around 20 tasks, that is why we chose SiOSi input for HF execution). Each process executes around 300-800 tasks. Our data transfer model is quite simple and we consider that all data transfers between the local memory of each process and the GA memory take the same route. Modeling of different routes of data transfers for the same source-destination pair, bandwidth sharing for different source-destination pairs and network congestion is more challenging and part of our future work. This simple model is enough to provide insight to the application developers (or runtime system) about the ordering of data transfers for the same sourcedestination pair so as to maximize communication-computation overlap. Our model is easily adaptable to any source-destination pair when there is one fixed route between source and destination (such as between CPU and GPU, one copy engine to transfer data from CPU (resp. GPU) to GPU (resp. CPU) ).

Both applications mainly perform two types of computations, tensor transpose and tensor contraction. HF expects to specify a tile size and we set it to 100, so that each core can be used efficiently. CCSD automatically determines tile sizes at different program points based on the input molecules. Hence, HF operates on almost homogeneous tiles while CCSD uses more heterogeneous tiles. To get more insights into the considered workloads, we provide information about the instances we consider in Figure 6. For each

Workload Characteristics

Figure 2 :

 2 Figure 2: Schedules for the instance of Table2with a memory capacity of 10.

Figure 3 :

 3 Figure 3: Static order heuristic schedules for Table3.

Figure 4 :

 4 Figure 4: Different dynamic heuristic schedules for a task set of Table4with a memory capacity of 6.

Figure 5 :

 5 Figure 5: Different static order dynamic correction heuristic schedules for a task set of Table5with a memory capacity of 9. The OMI M order is BCDEA.

Figure 6 :

 6 Figure 6: Characteristics of HF and CCSD tasks workloads. Outliers are shown with black dots.

  Table 2 with a memory capacity of 10. communication of task E can not start earlier because it would not fit in memory with tasks B and C, and delaying the computation of task E after task D would delay task F because E and F do not fit in memory together. We can see that this claim does not hold in the constrained memory case.

Table 4 :

 4 Table 4 with a memory capacity of 6. A task set for dynamic schedules.

	Task Memory Req Comm Time Comp Time
	A	3	3	2
	B	1	1	6
	C	4	4	6
	D	5	5	1

  Table 5 with a memory capacity of 9. The OMI M order is BCDEA.

	Task Memory Req Comm Time Comp Time
	A	4	4	1
	B	2	2	6
	C	8	8	8
	D	5	5	4
	E	3	3	2
	Table 5: A task set for static order dynamic correction sched-
	ules.

Table 6 :

 6 Heuristics and their favorable scenarios.

https://www.emsl.pnl.gov/emslweb/10.25582/inst.34218


Ramon Sanabria
email: ramons@cs.cmu.edu

Ozan Caglayan

Shruti Palaskar

Desmond Elliott

Loïc Barrault

Lucia Specia

Florian Metze

How2: A Large-scale Dataset for Multimodal Language Understanding

Human information processing is inherently multimodal, and language is best understood in a situated context. In order to achieve human-like language processing capabilities, machines should be able to jointly process multimodal data, and not just text, images, or speech in isolation. Nevertheless, there are very few multimodal datasets to support such research, resulting in a limited interaction among different research communities. In this paper, we introduce How2, a large-scale dataset of instructional videos covering a wide variety of topics across 80,000 clips (about 2,000 hours), with word-level time alignments to the ground-truth English subtitles. In addition to being multimodal, How2 is multilingual: we crowdsourced Portuguese translations of the subtitles. We present results for monomodal and multimodal baselines on several language processing tasks with interesting insights on the utility of different modalities. We hope that by making the How2 dataset and baselines available we will encourage collaboration across language, speech and vision communities.

Introduction

Multimodal sensory integration is an important aspect of human concept representation, language processing and reasoning [START_REF] Lawrence W Barsalou | Grounding conceptual knowledge in modality-specific systems[END_REF]. From a computational perspective, major breakthroughs in natural language processing (NLP), computer vision (CV), and automatic speech recognition (ASR) have resulted in improvements in a wide range of multimodal tasks, including visual question-answering [START_REF] Antol | VQA: Visual Question Answering[END_REF], multimodal machine translation [START_REF] Specia | A shared task on multimodal machine translation and crosslingual image description (WMT)[END_REF], visual dialogue [START_REF] Das | Visual Dialog[END_REF], and grounded ASR [START_REF] Palaskar | End-to-end multimodal speech recognition[END_REF]. Despite these advances, state-of-the-art computational models are nowhere near integrating multiple modalities as effectively as humans. This can be partially attributed to a lack of resources that are pervasively multimodal: existing datasets are typically focused on a single task, e.g. images and text for image captioning [START_REF] Chen | Microsoft COCO captions: Data collection and evaluation server[END_REF], images and text for visual-question answering [START_REF] Antol | VQA: Visual Question Answering[END_REF], or speech and text for ASR [START_REF] Godfrey | Switchboard: Telephone speech corpus for research and development[END_REF]. These datasets play a crucial role in the development of their fields, but their single-task nature limits the collective ability to develop general purpose artificial intelligence.

We introduce How2, a dataset of instructional videos paired with spoken utterances, English subtitles and their crowdsourced Portuguese translations, as well as English video summaries. The pervasive multimodality of How2 makes it an ideal resource for developing new models for multimodal understanding. In comparison to other multimodal resources, How2 is a naturally occurring dataset: I'm very close to the green but I didn't get it on the green so now I'm in this grass bunker. Eu estou muito perto do green, mas eu não pus a bola no green, então agora estou neste bunker de grama.

In golf, get the body low in order to get underneath the golf ball when chipping out of thick grass from a side hill lie. neither the subtitles, nor the summaries have been crowdsourced. Furthermore, the visual content is inherently related to the spoken utterances. Figure 1 shows an example in which the presenter is explaining how to play a golf shot. If one only has access to the text, it is unclear whether the "green" in the subtitles refers to the colour green ("verde" in Portuguese), or the surface type ("green" in Portuguese). The textual context alone is not enough to disambiguate the meaning of the subtitles, and at the time of writing, both Google Translate and Microsoft Translator incorrectly translate "green" as "verde". However, given additional visual context (green grass with a flag pole), or the audio context (outside with the sound of chipping a golf ball), our multimodal models can correctly interpret this utterance. See Appendix A.1 for more examples.

The value of additional modalities can also be demonstrated for ASR. Object and motion level visual cues can filter out systematic noise that co-occurs with activities. Scene information from an image can be used to learn a common auditory representational space for different environmental characteristics such as indoor vs. outdoor settings [START_REF] Miao | Open-domain audio-visual speech recognition: A deep learning approach[END_REF]. Entities in an image can also be used to adapt a language model towards a domain [START_REF] Gupta | Visual features for contextaware speech recognition[END_REF].

Together with the dataset, we also release code to perform baseline experiments on several tasks covering different subsets of How2. We find that action-level visual features improve automatic speech recognition, video summarization and speech-to-text translation. These results demonstrate the potential of the How2 dataset for future multimodal research.

HowDataset

The How2 dataset consists of 79,114 instructional videos (2,000 hours in total, with an average length of 90 seconds) with English subtitles. The corpus can be (re-)created using the scripts and metadata we made available at https://github.com/srvk/how2-dataset. The website also contains information on how to obtain pre-computed features for validation or saving computation, and how to reproduce the experimental results we present using nmtpy [START_REF] Caglayan | NMTPY: A flexible toolkit for advanced neural machine translation systems[END_REF].

Collection We downloaded the videos from YouTube, along with various types of metadata, including ground-truth subtitles and summaries (referred to as "descriptions") in English, written by the video creators. Videos were scraped from the YouTube platform using a keyword based spider as described in [START_REF] Yu | Instructional videos for unsupervised harvesting and learning of action examples[END_REF]. In order to produce a multilingual and multimodal dataset, the English subtitles were first re-segmented into full sentences, which were then aligned to the speech at the word level. The visual features were extracted from the video clips that correspond to these sentence-level alignments. The distribution of the duration of segments can be seen in Figure 2b. See Appendix A.2 for more details on the alignment process.

To generate translations, we used the Figure Eight crowdsourcing platform. After conducting a pilot experiment with a small set of languages, we chose Portuguese as target language because of the availability of workers and the quality and reliability of the annotations performed by them. In order to reduce the amount of time it would take to annotate the dataset, we posed translation as a post-editing task: in another pilot experiment, we instructed crowd workers to "choose the best translation" from English to Portuguese among candidate translations provided by three state-of-the-art online neural machine translation systems. We then selected the system that was preferred most often, and had crowdworkers post-edit the candidate translations. This process is still ongoing.

During annotation on Figure Eight, the worker population was restricted to those living in Portugal or Brazil. Workers were paid US$ 0.05 to watch a short video and post-edit the automatically translated Portuguese segment into correct Portuguese. Workers thus performed the annotation (and the pilots) in a multimodal setting. To ascertain worker reliability, a content word of each 5 sentences of the candidate translations was replaced by another random content word that was not part of the translation. If the word inserted was still present in the final translation, the annotations from that worker were discarded and the worker was banned from further contributing to the project.

At the time of writing, we had completed the collection of Portuguese translations for a 300h subset of the entire dataset from 200 workers (each was limited to 5,000 segments to post-edit but none of them reached this limit). We discarded and re-annotated 18% of the 300h. The total cost for data collection thus far was US$ 8,771.

In a verification experiment, we determined found that training an English-Portuguese neural MT system on 300h of machine generated training data degrades performance by about 1 BLEU point, when compared to a system that has been trained on the post-edited translations, when evaluated against expert-validated post-edited translations, showing that the approach is justified.

Topic distribution

We clustered the English subtitles using Latent Dirichlet Allocation (LDA) [START_REF] David M Blei | Latent dirichlet allocation[END_REF]. Upon analyzing the clusters with top words in each topic, inter-topic and intra-topic distances, we found that a good representation for the 300h subset consists of 22 topics. We hand-labeled these topics based on top words in each cluster, as shown in Figure 2a.

Splits

Experiments

To demonstrate and explore the potential of the How2 dataset, we propose several tasks and developed systems for them using a sequence-to-sequence (S2S) approach. Table 2 summarizes the baseline results on the 300h training set for all tasks; only the summarization task uses the entire 2000h set. More details can be found in the Appendix A.4.

1. Automatic speech recognition. We use an S2S model with a deep bi-directional LSTM encoder [START_REF] Hochreiter | Long short-term memory[END_REF]. For multimodal ASR, we apply visual adaptive training [START_REF] Palaskar | End-to-end multimodal speech recognition[END_REF][START_REF] Gupta | Visual features for contextaware speech recognition[END_REF] where we re-train an ASR model by adding a linear adaptation layer which learns a video-specific bias to additively shift the speech features. All parameters of the network all jointly trained in this re-training step. The adaptation layer increases the model size by less than 1%.

2. Machine translation. We train an S2S MT model for English→Portuguese using a bidirectional GRU [START_REF] Cho | Learning phrase representations using rnn encoderdecoder for statistical machine translation[END_REF]. For multimodal MT (MMT), we apply the same adaptive approach as we did for ASR but the inputs to be shifted are now word embeddings instead of speech features.

The adaptation layer increases the model size by 8%.

3. Speech-to-text translation. We directly translate from English speech to Portuguese using the same ASR architecture but with a different target vocabulary, which is similar to previous approaches [START_REF] Weiss | Sequence-tosequence models can directly translate foreign speech[END_REF][START_REF] Bérard | End-to-end automatic speech translation of audiobooks[END_REF]. For multimodal STT, we apply the same adaptive scheme as in ASR.

4.

Summarization. The baseline is again the same S2S MT model. For multimodal summarization, we follow the hierarchical attention approach [START_REF] Libovický | Attention strategies for multi-source sequence-tosequence learning[END_REF][START_REF] Libovický | Multimodal abstractive summarization of open-domain videos[END_REF] to combine textual and visual modalities by using a sequence of action-level features instead of an average-pooled one as in the other experiments. This latter increases the model size by 14%.

Related work

Lying at the intersection of NLP and CV [START_REF] Bernardi | Automatic description generation from images: A survey of models, datasets, and evaluation measures[END_REF], image captioning (IC) is the multimodal task with the largest number of datasets available. The most widely used datasets in this field are the ones with human crowdsourced descriptions, such as Flickr8K [START_REF] Hodosh | Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics[END_REF], Flickr30K [START_REF] Young | From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions[END_REF], MSCOCO [START_REF] Chen | Microsoft COCO captions: Data collection and evaluation server[END_REF] and their extensions to other languages. A closely related task to IC is multimodal machine translation. So far, MMT has been addressed using captioning datasets extended with translations in different languages such as IAPR-TC12 [START_REF] Grubinger | The IAPR TC-12 benchmark: A new evaluation resource for visual information systems[END_REF] and Multi30K which is an extension of Flickr30K into German [START_REF] Elliott | Multi30k: Multilingual english-german image descriptions[END_REF], French [START_REF] Elliott | Findings of the second shared task on multimodal machine translation and multilingual image description[END_REF], and Czech [START_REF] Barrault | Findings of the shared task on multimodal machine translation (WMT)[END_REF]. One major pitfall of these datasets is that they lack syntactic and semantic diversity.

A similar task to IC is that of automatically describing videos (VD). The most popular datasets for VD are MSR-VTT [START_REF] Xu | MSR-VTT: a large video description dataset for bridging video and language[END_REF], LSMDC [START_REF] Rohrbach | Movie description[END_REF], and Microsoft Research Video Description (MSVD) corpus [START_REF] Chen | Building a persistent workforce on Mechanical Turk for multilingual data collection[END_REF] which is the only multilingual resource of this type providing 122K crowdsourced descriptions. However, two-thirds of the descriptions are in English and the ones in other languages are not parallel.

How2 offers a larger amount of data, all of which is in two languages.

Lipreading can be seen as a form of multimodal ASR, albeit not fusing information at the semantic level. Popular and large-scale datasets include Grid [START_REF] Cooke | An audio-visual corpus for speech perception and automatic speech recognition[END_REF] and Lip Reading in The Wild [START_REF] Son | Lip reading in the wild[END_REF]. How2 is the first dataset that allows to perform multimodal ASR, using images as acoustic and linguistic context to improve accuracy. How2 is also a valuable resource for speech-to-text translation, which EN CZ (V) 492,402 pairs is otherwise often performed using Fisher-Callhome [START_REF] Post | Improved Speech-to-Text translation with the Fisher and Callhome Spanish-English speech translation corpus[END_REF] and Audiobooks [START_REF] Ali Can Kocabiyikoglu | Augmenting Librispeech with French Translations: A Multimodal Corpus for Direct Speech Translation Evaluation[END_REF]. How2 is the only corpus for multimodal STT currently available.

Multimodal neural abstractive summarization is an emerging field for which there are no well established benchmarking datasets yet. Li et al. [START_REF] Li | Multi-modal summarization for asynchronous collection of text, image, audio and video[END_REF] collected a multimodal corpus of news articles containing 500 videos of English news articles paired with human annotated summaries. UzZaman et al. [START_REF] Uzzaman | Multimodal summarization of complex sentences[END_REF] collected a corpus of images, structured text and simplified compressed text for summarization of complex sentences. More traditional text-based summarization is commonly based on CNN/Daily Mail [START_REF] Moritz Hermann | Teaching machines to read and comprehend[END_REF][START_REF] Nallapati | Abstractive text summarization using sequence-to-sequence rnns and beyond[END_REF], Gigaword [START_REF] Napoles | Annotated gigaword[END_REF] and the Document Understanding Conference challenge data [START_REF] Over | Duc in context[END_REF]. An older version and non-released version of How2 was used for experiments on learning action examples in videos in [START_REF] Yu | Instructional videos for unsupervised harvesting and learning of action examples[END_REF].

Conclusions

We have introduced How2, a multimodal collection of instructional videos with English subtitles and crowdsourced Portuguese translations. We have also presented sequence-to-sequence baselines for machine translation, automatic speech recognition, spoken language translation, and multimodal summarization. By making available data and code for several multimodal natural language tasks, we hope to stimulate more research on these and similar challenges to obtain a deeper understanding of multimodality in language processing.

Actually I use this beautiful moroccan oil and it's really wonderful on the hair. Na verdade eu uso este belo óleo marroquino e é realmente maravilhoso no cabelo.

Relaxed African-American hair should be moisturized and washed to maintain good and healthy hair. Care for African-American, biracial or ethnic hair with tips from a professional hairstylist in this free video on hair care.

Like said you can cook this pretty quickly with your family.

Como disse você pode cozinhar isso muito rapidamente com sua família.

Learn how to cook and serve picadillo con arroz with expert cooking tips in this free classic Cuban recipe video clip.

When your wide receivers get into their stance they want to have one foot forward, they want to have good position and be ready to fire off the line.

Quando seus receptores largos entram em sua posição eles devem ter um pé para a frent,e eles devem ter uma boa posição e estar pronto para disparar a linha.

Learn some great tips on how to line up as a receiver in this free video clip on how to play football. The image is a randomly selected frame from a concrete segment of the video, the text in bold is the utterance pronounced during this segment, the Portuguese text in italic are the translations corresponding to the utterance and finally, the summary of the whole video is placed inside the rectangle.

A Appendix

A.1 How2 Examples

In Figure 3, we list three typical instances from the How2 dataset. In these examples, we can see the correspondence between the content of the video frame, the summary, and the utterance. This multimodal correspondence is what systems can exploit by using the How2 dataset. In the first example, a hairdresser is explaining how to use a specific hair product. In that case, the visual elements (i.e., hair product, hairdresser) and the scene, a hairdressing salon, are a rich source of context. In the second example, a woman is cooking in a kitchen with many cooking devices. In the third example, we can infer the acoustic environment (i.e., outdoors) by the scene.

A.2 Modality Alignment and Data Checks

To combine all modalities (i.e., speech, video, transcriptions, and translations) successfully, we need to establish and validate their correspondence in time. While the audio transcription is generated from subtitles, these do not always correspond to the actual audio. We thus decided to generate token-level (e.g. word-level) time stamps that link text, audio, and video modality. From these, utterance-level start and end times were also calculated.

features described in Section 2 are now used as-is, i.e. as a sequence of 2048-dimensional vectors rather than being average pooled into a single vector.

A.5 Training Details

Unless otherwise specified, we use ADAM [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] as the optimizer with an initial learning rate of 0.0004. The gradients are clipped to have unit norm [START_REF] Pascanu | On the difficulty of training recurrent neural networks[END_REF]. The training is early stopped if the task performance on validation set does not improve for 10 consecutive epochs. Task performance is assessed using Word Error Rate (WER) for speech recognition, BLEU [START_REF] Papineni | BLEU: A method for automatic evaluation of machine translation[END_REF] for translation tasks and ROUGE-L [START_REF] Lin | Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics[END_REF] for the summarization task. The learning rate is halved whenever the task performance does not improve for two consecutive epochs. All systems are trained three times with different random initializations. The hypotheses are decoded using beam search with a beam size of 10. We report the average results of the three runs. We use nmtpytorch [START_REF] Caglayan | NMTPY: A flexible toolkit for advanced neural machine translation systems[END_REF] to train the models.

Figure 1 :

 1 Figure 1: How2 contains a large variety of instructional videos with utterance-level English subtitles (in bold), aligned Portuguese translations (in italics), and video-level English summaries (in the box). Multimodality helps resolve ambiguities and improves understanding.

  (a) Topic distribution. (b) Segment duration.

Figure 2 :

 2 Figure 2: LDA topic distributions and segment duration for the 300h subset. The 2000h overall corpus exhibits very similar characteristics.

Figure 3 :

 3 Figure 3: Examples of the How2 dataset that reflect its sample variability and modality correlation.The image is a randomly selected frame from a concrete segment of the video, the text in bold is the utterance pronounced during this segment, the Portuguese text in italic are the translations corresponding to the utterance and finally, the summary of the whole video is placed inside the rectangle.

Table 1 :

 1 Statistics of How2 dataset.

			Videos	Hours Clips/Sentences	Per Clip Statistics
	300h	train 13,168	298.2	184,949 5.8 seconds & 20 words
		val	150	3.2	2,022 5.8 seconds & 20 words
		test	175	3.7	2,305 5.8 seconds & 20 words
		held	169	3.0	2,021 5.4 seconds & 20 words
	2000h train 73,993 1,766.6	-
		val	2,965	71.3	-
		test	2,156	51.7	-

Table 1

 1

presents summary statistics of the 2000h set and 300h subset: the val and test sets can be used for early-stopping, model selection and evaluation; the held set is reserved for future evaluations or challenges. The total set (i.e. 2000h) contains around 22.5M words. The tokenized

Table 2 :

 2 Results of the automatic speech recognition, machine translation, speech-to-text translation, and summarization experiments on test set. The arrows indicate direction of improvement. ASR (% WER ⇓) MT (BLEU ⇑) STT (BLEU ⇑) SUM (ROUGE-L ⇑)

	Baseline	19.4	54.4	36.0	53.9
	Multimodal	18.0	54.4	37.2	54.9
	training set of 300h subset contains around 3.8M (43K unique) and 3.6M (60K unique) words for
	English and Portuguese respectively.

Table 3 :

 3 Comparison with previous datasets: (IC) and (VD) stand for image and video captioning. Language names are encoded in ISO-639-1.

	Task	Dataset	Languages	Audio Visual Size
	IC	Flickr8K [19]	EN, TR [20], ZH [21]	(I)	8K
	IC	Flickr30K [22]	150K EN, DE [23]	(I)	30K
	IC	MSCOCO [6]	414K EN	(I)	82K
	IC		820K JA [24]	(I)	164K
	MMT	Multi30K [23]	EN, DE, FR [25], CZ [26]	(I)	30K
	VD	MSVD [27]	122K total in many	(V) 5.3 hours
	VD	LSMDC [28]	EN	(V) 150 hours
	VD	MSRVTT [29]	EN	(V) 41 hours
	AV-ASR	Grid [30]	EN	(V) 50 hours
	AV-ASR	ViaVoice [31]	EN	(V) 34.9 hours
	AV-ASR	LRW [32]	EN	(V) 800 hours
	STT	Fisher [33]	EN, ES		150 hours
	STT	Audiobooks [34]	EN, FR		236 hours
	SUM	CNN/DMC [35, 36] EN		286,817 pairs
	SUM	DUC [37]	EN		500 pairs
	SUM	[38]

https://www.clsp.jhu.edu/workshops/18-workshop/

Acknowledgements

This work was mostly conducted at the 2018 Frederick Jelinek Memorial Summer Workshop on Speech and Language Technologies, 1 hosted and sponsored by Johns Hopkins University. Lucia Specia received funding from the MultiMT (H2020 ERC Starting Grant No. 678017) and MMVC (Newton Fund Institutional Links Grant, ID 352343575) projects. Loïc Barrault and Ozan Caglayan received funding from the CHISTERA M2CR (No. ANR-15-CHR2-0006-01).

To align text and audio, we perform a Viterbi alignment between the transcriptions, which were provided by the users who uploaded the videos, and the audio track of How2, using Kaldi's [START_REF] Povey | The Kaldi speech recognition toolkit[END_REF] Wall Street Journal (WSJ) GMM/ HMM acoustic model. This alignment process estimates the start and end times of each sentence in the audio track. Finally, by using the estimated alignments, we can segment the audio and video track according to the utterances.

To make sure the data will be suitable for the proposed use, we validated two properties: First, we verified that the word alignment is indeed accurate by manually inspecting randomly chosen utterances. The 300 h sub-set was selected to give good alignment scores, and the WSJ model seemed to perform best in that respect: "good" (according to the score) utterances were indeed accurately aligned, when compared to alignments generated with other models, including those developed for speech synthesis. Second, the "transcription" data has been generated from video subtitles, which were not meant to be verbatim and highly accurate "transliterations" of the spoken content. Rather, the "transcription" text is a somewhat canonical form of the spoken word, which is fine for our proposed uses, although it may lead to slightly higher overall word error rates for the speech-to-text tasks.

A.3 Feature Extraction and Processing

Speech Features We used Kaldi [START_REF] Povey | The Kaldi speech recognition toolkit[END_REF] to extract 40-dimensional filter bank features from 16kHz raw speech using a time window of 25ms with 10ms frame shift and concatenated 3-dimensional pitch features to obtain the final 43-dimensional speech features. A per-video Cepstral Mean and Variance Normalization (CMVN) is further applied to account for speaker variability.

Visual Features A 2048-dimensional feature vector per 16 frames is extracted from the videos using a CNN trained to recognize 400 different actions [START_REF] Hara | Can spatiotemporal 3d CNNs retrace the history of 2d cnns and imagenet[END_REF]. It should be noted that this results in a sequence of feature vectors per video rather than a single/global one. In order to obtain the latter, we average pooled the extracted features into a single 2048-dimensional feature vector which will represent all sentences segmented out of a single video.

Text Features All texts are normalized, lowercased and filtered from punctuation. A SentencePiece model [START_REF] Kudo | SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing[END_REF] is learned separately for English and Portuguese resulting into vocabularies of 5K each except for summarization which uses word-level tokens.

A.4 Architecture Details

Automatic Speech Recognition We use a 6-layer pyramidal encoder [START_REF] Chan | Listen, attend and spell: A neural network for large vocabulary conversational speech recognition[END_REF] (with interleaved tanh projection layers) where the middle two layers skip every other input resulting into a subsampling rate of 4. The decoder is a 2-layer conditional GRU (CGRU) decoder [START_REF] Sennrich | Nematus: a toolkit for neural machine translation[END_REF], a transitive decoder where the hidden state of the second GRU is fed back to the first GRU. The first GRU is initialized with the mean encoder hidden state transformed using a tanh layer. A feed-forward attention mechanism [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF] is used inside the decoder and the input and output embeddings are tied [START_REF] Press | Using the output embedding to improve language models[END_REF].

Multimodal Automatic Speech Recognition For multimodal ASR, we apply video adaptive training [START_REF] Palaskar | End-to-end multimodal speech recognition[END_REF][START_REF] Gupta | Visual features for contextaware speech recognition[END_REF] with a learned linear transformation of the visual feature vector v that will act as a visual bias over a given speech feature x t at time t. The shifted speech feature xt is thus computed as follows:

To train this model, we first initialize the model parameters using a previously trained ASR and jointly optimize all parameters including W and b above.

Machine Translation

We train a sequence-to-sequence neural MT model with a 2-layer bidirectional GRU [START_REF] Cho | Learning phrase representations using rnn encoderdecoder for statistical machine translation[END_REF] encoder and a 2-layer conditional GRU decoder [START_REF] Sennrich | Nematus: a toolkit for neural machine translation[END_REF]. A dropout [START_REF] Srivastava | Dropout: A simple way to prevent neural networks from overfitting[END_REF] with 0.3 drop probability is used after source embeddings, source encodings and before the final sof tmax operation.

Multimodal Summarization We follow the hierarchical attention approach [START_REF] Libovický | Attention strategies for multi-source sequence-tosequence learning[END_REF] to combine textual and visual modalities. Unlike previous multimodal ASR, MT and STT architectures, the visual