Frédéric Boniol
email: frederic.boniol@onera.fr

Julien Forget
email: julien.forget@univ-lille1.fr

Claire Pagetti
email: claire.pagetti@onera.fr

WATERS Industrial Challenge 2017 with Prelude

The software model 1 is composed of: 1250 runnables. A runnable is a piece of code that reads a set of labels and writes another set of labels (these two sets are not necessarily disjoint). As an example, Runnable 100ms 246 reads the labels {Label 4530, Label 3583, Label 4338, Label 316, Label 2354, Label 4455, Label 5707, Label 7334, Label 7864} and writes the labels {Label 65, Label 2560, Label 9604, Label 9660}.

A best-case execution time (BCET) and a worst-case execution time (WCET) are provided for each runnable. For instance, the execution time of Runnable 100ms 246 belongs to [8951,32363] (in micro-seconds).

The activation of a runnable can be of two types:

• Periodic: for instance Runnable 100ms 246 has a period of 100 ms; • Sporadic: for instance the inter-arrival time of runnable Runnable sporadic 900us 1000us 4 is 900µs at minimum and 1000µs at maximum. 10000 labels. These serve as the communication variables between runnables. A label can be read by several runnables but can only be written by at most one, which we will call the producer of that label. We assume that a label that is not consumed by any runnable is an external output of the system (i.e. data sent to an actuator) and that a label that is not produced by any runnable is an external input (i.e. data originating from a sensor). Since variables shared between runnables are all explicitly declared as labels (no hidden sideeffects), we can analyze data-dependencies, latencies, etc. 21 tasks. A task is a group of runnables. There are two types of tasks:

• Periodic task: it regroups runnables of the same period.

For instance Task 100ms contains a set of runnables Runnable 100ms X with X ∈ [0, 246]. Inside a task, runnables are called in sequence, as described in the task call graph (in this example, the call graph follows the order defined by the X index mentioned previously).

There is a total of 10 periodic tasks as described in table I. There are two types of periodic tasks:

1 ChallengeModel w Label-Stats fixedLabelMapping App4mc v072.amxmi.

-Preemptive: when such a task is released, it can preempt at any time any task with a lower priority. All periodic tasks with period lower than 10 ms are preemptive; -Cooperative: when such a task is released, it can preempt tasks with lower priority, but only between two runnable executions.

• Sporadic task: it regroups runnables that have the same minimal and maximal inter arrival times. These tasks are named as ISRX with X ∈ [START_REF] Amalthea | An open platform project for embedded multicore systems[END_REF][START_REF] Puffitsch | Mapping a multi-rate synchronous language to a many-core processor[END_REF]. These tasks are all of type Preemptive (as defined above).

B. Platform model

The architecture considered for the challenge [START_REF] Hamann | Demo abstract: Demonstration of the fmtv 2016 timing verification challenge[END_REF], [START_REF] Hamann | Waters industrial challenge 2017[END_REF]

is shown below. core 0 LRAM0 core 1 LRAM1 core 2 LRAM2 core 3 LRAM3 Crossbar GRAM
Each core X has a frequency of 200MHz and is associated with a private local memory (scratchpad memory) denoted as LRAM X. Memory access times, assuming no contention, are the following:

• Access to the local LRAM: 1 cycle;

• Access to a distant LRAM: 9 cycles; • Access to the GRAM: 9 cycles. The LRAMs are single-ported, meaning that if two cores access the same local memory concurrently there is conflict that will be solved with a FIFO policy.

C. Execution model

The execution model of AMALTHEA assumes the following rules:

1) Rule 1 -fixed-priority partitioned scheduling: tasks are statically allocated to cores, this a priori mapping is given in table I. On each core, a RM (rate monotonic) scheduling policy is applied. Task priorities are thus assigned as follows: the highest priority tasks are the ISR tasks, they are ordered by priority in Table I (decreasing from left to right), then come the periodic tasks (ordered according to the same rule). Preemptions depend on the task type (preemptive or cooperative). task executes, along with all the labels produced by the runnables of this task. Note that a runnable may need to access to a distant LRAM to read some labels produced by runnables allocated to a different core; 3) Rule 3 -LET communication (between tasks): The LET model originates from GIOTTO [START_REF] Henzinger | Giotto: a timetriggered language for embedded programming[END_REF], which follows a time-triggered approach to specify and implement embedded systems. With LET communication tasks read data at their activation and write data (to local LRAM) at their deadline; 4) Rule 4 -Implicit communication (between runnables of the same task): The implicit communication semantics, proposed by AUTOSAR, is quite similar to LET except that data is updated at the start and at the end of the runnable execution (instead of at activation/deadline). The top part of figure 1, strongly inspired from [START_REF] Farcas | Real-time component integration based on transparent distribution[END_REF], illustrates the LET semantics; whereas the bottom part illustrates the implicit communication semantics. Dashed rectangles depict at what time labels are updated. According to Rule 3, Task 10ms has a LET of [0,10]. Thus at every time t = k × 10 + (with k ∈ N), the executive layer is in charge of copying all labels read by the task runnables into local variables. At every time t = k × 10 -(with k ∈ N\{0}), the executive layer is in charge of copying all local variables produced by the task runnables into labels. Note that since all periodic tasks are synchronous, at time 0 (which is the critical instant) all the labels are copied first before starting the execution of any task. Now, considering the runnables inside Task 10ms, according to Rule 4, when Runnable 10ms X starts executing, it first copies all the read labels produced by Runnable 10ms Y to local variables. When it completes execution, it updates the its produced labels for the Runnable 10ms Y.

II. PRELUDE IMPLEMENTATION

PRELUDE [START_REF] Forget | A multi-periodic synchronous data-flow language[END_REF], [START_REF] Pagetti | Multitask implementation of multi-periodic synchronous programs[END_REF] is a Synchronous Language for programming real-time embedded control systems, which provides real-time primitives to support multi-periodic systems. The preludec compiler generates multi-task C code, that is independent of the target scheduling policy and of the number of cores. We propose to specify the AMALTHEA system with PRELUDE and to address part of the challenge thanks to PRELUDE features and tools. In this section, we explain how to translate the system into a PRELUDE program.

A. Restrictions compared to the original specification

Restriction 1: Each sporadic task is translated into a periodic task with period equal to the minimal inter arrival time.

PRELUDE can currently only deal with periodic tasks and extending the language to sporadic tasks would require an important effort.

Restriction 2: Labels types are not considered.

The AMALTHEA model only provides the size of each label (in bits), while PRELUDE requires their data-type. This has very limited impact on the challenge results.

Restriction 3: AMALTHEA tasks do not appear explicitly in the resulting PRELUDE program.

Lifting this restriction would require to represent a task as a hierarchical node. We did not do this part purely due to lack of time. This has no impact on the results detailed in remainder of this paper.

B. Structure of the PRELUDE challenge implementation

First all runnables are declared as imported node which is the terminology to express that a program element is implemented by external user-provided code.

Then, we assemble the imported nodes in the main node. The main node also specifies the system inputs, or sensors, and the system outputs, or actuators. The code 1 gives an extract of this assembly node. For instance, Label 5707 is a sensor and Label 9944 is an actuator. We chose to assign the rate of a sensor based on the rate of its fastest consumer, so for instance Label 5707 has period of 100000µs. The rate of an actuator is the rate of its producer, so for instance, Label 9944 has period 5000µs.

Code 1 (Part of the assembly description in PRELUDE):

node main ( L a b e l 5 7 0 7 : i n t r a t e ( 1 0 0 0 0 0 , 0 ) ; . The main node instantiates the imported nodes and describes their data-dependencies using a set of equations (the let..tel block). In the remainder of this section we detail our implementation of label communications.

a) Multi-periodic task communications: In data-flow languages, task execution rates are driven by data rates. Furthermore, the synchronous semantics of PRELUDE requires that the inputs and outputs of an imported node all have the same rate. For instance, since input Label 5707 has a period of 100000µs, so does node Runnable 100ms 246 and so does label Label 65.

The clock calculus raises an error if an imported node tries to combine flows of different rates. So, when a runnable consumes a label produced by a runnable operating at a different rate, the programmer must specify how to perform the rate transition. In PRELUDE, this is done via the operators /ˆand * ˆ: /ˆq produces a flow q times slower, while * ˆq produces a flow q times faster. The equation of Runnable 5ms 0 illustrates rate transitions. Let us for the moment ignore the fby operator, which will be explained later. Label Label 744 is accelerated by a factor of 2, because it was produced by the task Task 10ms. The label Label 1779 is decelerated by 10 and accelerated by 3, because it was produced by ISR4 which has a period of 5000/10 × 3 = 1500µs.

b) Intra-task communications: In the AMALTHEA model, inside a task runnables are called in sequence and consume labels of runnables of the same task according to the implicit communication semantics (Rule 4). In PRE-LUDE, the execution order depends on data-dependencies: data-consumer executes after data-producer. For instance, Runnable 100ms 246 consumes the label Label 4530 which is produced by Runnable 100ms 239. The communication is direct (no operator applied), so Runnable 100ms 239 must execute before Runnable 100ms 246. The labels produced by Runnable 100ms 246 and consumed by other runnables of the same task period cannot be consumed with a direct communication (in the AMALTHEA model, Runnable 100ms 246 is the last runnable of the call graph of Task 100ms). Therefore, the communication is indirect and requires a delay of one period. This is achieved in PRELUDE by using operator fby (for instance Label 65 for Runnable 100ms 36). c) LET semantics: According to Rule 3 any communication between runnables with different periods must follow the LET model. This is achieved in PRELUDE by adding a fby before the rate transition. For instance, for Runnable 5ms 0, Label 744 is first delayed and then accelerated.

C. Automatic translation of Amalthea specification in PRE-LUDE program

We have implemented an automatic translator from AMALTHEA to PRELUDE, which is not restricted to the program provided for the challenge but should also be applicable to other AMALTHEA programs. It was written in Ocaml and consists of about 2000 lines of code, half of which is code reuse from the Prelude compiler. We pruned some data unused for our analysis: the hardware model, the OS model and the event model (which only states that all runnable have deadlines equal to their periods), are ignored. The mapping model is used for analysis but is not part of the resulting PRELUDE program. The PRELUDE program file resulting from the translation is approximately four times smaller than the AMALTHEA program (mainly due to the verbosity of the xml format used by AMALTHEA).

III. LATENCY COMPUTATION

We answer in this section to the challenge issue 4, and we show that we can compute end-to-end latencies directly at the specification level (i.e. directly on the PRELUDE program) without considering the actual implementation and its execution platform.

A. Effect chains and end-to-end latencies

An effect chain is a functional path C = (r

1 x1 → r 2 x1 → . . . xn-1 → r n ) composed of n communicating runnables. Each
x i is a Label (i.e., a data-flow) produced by the runnable r i and consumed by the runnable r i+1 . An end-to-end latency is the time required for performing this chain of computation, from the start date of r 1 to the end date of r n .

In the case of a multi-periodic chain (as EffectChain_2 and EffectChain_3), analyzing the end-to-end latency requires to carefully consider how rate transitions are implemented. For instance, let us consider an effect chain composed of two runnables

C = (r 1 x1 → r 2 )
such that the period of r 1 is 100ms and the period of r 2 is 200ms. r 1 and r 2 belong to two different tasks. First let us consider that the tasks follow the LET semantics. The dependencies scheme between r 1 and r 2 is depicted figure 2. The PRELUDE model of this chain is

C LET = (r 1 x1 → fby /ˆ2 → r 2 )
As shown figure 2, the first instance of the chain begins at t=0 and goes through r 1 1 (the first occurrence of r 1 ), x 1 1 (which is delayed by one period compared to r 1 1 ) and terminates at t = 400ms at the end of r 2 2 (the second occurrence of r 2 ). Its latency is 400ms. Similarly, the second instance of the chain begins at t = 100ms, goes through r 2 1 , x 2 1 and terminates also at t = 400ms at the end of r 2 2 . Its latency is 300ms. Thus, the worst case latency of the chain is 400ms, and it is achieved every 200ms. Fig. 2. Latency example in a biperiodic effect chain with the LET semantics Fig. 3. Latency example in a biperiodic effect chain with the direct semantics Let us consider now consider the case of a directcommunication semantics: outputs are produced as soon as the runnables terminate. In that case, r 2 must wait for x 1 before starting its execution. The behavior of the chain following the direct semantics is depicted Figure 3. The PRELUDE model of this behaviour is:

C direct = (r 1 x1 → /ˆ2 → r 2 )
In that case, the first instance of the chain goes through r 1 1 , x 1 1 , r 1 2 . The maximal end date of r 1 2 being 200ms, then the maximal latency of the first instance is 200ms. The second instance of the chain begins with r 2 1 at t = 100ms and terminates at the end of r 2 2 at most at t = 400ms. Its maximal latency is 300ms. The worst case latency of the chain is then 300ms.

As shown in these examples, PRELUDE allows to automatically compute maximal end-to-end latencies of functional chains directly at the specification level [START_REF] Wyss | End-to-end latency computation in a multi-periodic design[END_REF]. We compute latencies based solely on the semantics of the language, thus the computed latency bounds are valid for any correct implementation of the PRELUDE program, independently from the scheduling policies (PRELUDE supports several of them) and from the hardware platform (mono/multi-core, etc). That way, latency requirements can be checked early on in the development cycle, even without the target platform.

B. Effect Chain 1

Let us first consider

EffectChain_1 = (r10ms149 L3423 → r10ms243 L3968 → r10ms272 L2276 → r10ms107) (
where r10msX is a shorthand for Runnable 10ms X and LY for Label Y). This chain only involves runnables of period 10ms, it is a mono-periodic effect chain. The AMALTHEA model specifies that the runnable r10ms107 is executed before r10ms272. Therefore in PRELUDE, L2276 has to be delayed by a fby operator before consumption by r10ms107. All the other communications in the chain are direct. The PRELUDE model of this chain is:

EC 1,prelude = (r10ms149 L3423 → r10ms243 L3968 → r10ms272 L2276 → fby → r10ms107).
The maximal latency of the chain is simply one period, plus one period for each fby in the chain, that is 20ms in total.

C. Effect Chain 2

EffectChain_2 = (r100ms7

L4258 → r10ms19 L2157 → r2ms8).
This chain involves nodes executing at different rates, it is a multi-periodic chain, which contains a slow-to-fast rate transition. We consider that the slow-to-fast rate transitions follow the LET semantics. At each rate transition, we insert a delay (for the LET semantics) and rate transition operators:

• From r100ms7 to r10ms19 : L4258 is delayed by one period of 100ms, and over-sampled by a factor of 10; the period of the resulting data-flow is 10ms. • From r10ms19 to r2ms8: L2157 is delayed by one period of 10ms, and over-sampled by a factor of 5. The PRELUDE implementation of EffectChain_2 is:

r100ms7 L4258 → fby * ˆ10 → r10ms19 L2157 → fby * ˆ5 → r2ms8.
The PRELUDE end-to-end analysis method [START_REF] Wyss | End-to-end latency computation in a multi-periodic design[END_REF] begins by computing which occurrences of the first and the last runnable of the chain are related by data-dependencies. Here, because of the LET semantics, the dependencies are the following:

• The earliest occurrence of r2ms8 which is impacted by r100ms7 1 is r2ms8 56 ; • More generally, the earliest occurrence of r2ms8 which is impacted by r100ms7 i is r2ms8 56+50 * (i-1) The latency of the chain is a constant value 112ms, i.e., the latest possible end date of r2ms8 56 minus the earliest possible start date of r100ms7 1 . Notice how this latency is computed without taking into account the actual scheduling. Only the semantics of the program is considered.

D. Effect Chain 3

EffectChain_3 = (r700us800us3

L4576 → r2ms3 L646 → r50ms36
). The first runnable of EffectChain_3 is sporadic. PRELUDE has no dedicated support for sporadic nodes so we implement the sporadic runnables as periodic nodes. To ensure deterministic execution, the translator chooses the minimal inter arrival time. For the worst case latency however, we need to consider the maximal inter arrival time. So for instance, in the case of EffectChain_3, r700us800us3 is implemented as a periodic node of period 800us.

Rate transitions are handled as follows:

• From r700us800us3 to r2ms3 : L4576 is under-sampled by a factor of 5 and afterwards over-sampled by 2; the period of the resulting data-flow is 2ms; • From r2ms3 to r50ms36: L646 is under-sampled by a factor of 25; the resulting period is 50ms. The PRELUDE implementation of EffectChain_3 is:

r700us800us3 L4576 → fby / ˆ5 * ˆ2 → r2ms3 L646 → fby /ˆ25 → r50ms36.
The dependencies along the chain are the following:

• r700us800us3 i with i = 1 . . . 60 impact r50ms36 2 ;

• r700us800us3 i with i = 61 . . . 120 impact r50ms36 3 ;

• r700us800us3 i with i = 121 . . . 185 impact r50ms36 4 .

The maximal latency is 103.3, it is achieved for r700us800us3 122 , which starts at 96.8ms and which impacts r50ms36 4 , which itself ends at the latest at 200ms. We did an experiment to measure the impact of the LET semantics on the latency. For fast-to-slow rate transitions, we considered an IMPLICIT communication (instead of the LET semantics). The PRELUDE implementation of EffectChain_3 is then (r700us800us3 [START_REF] Sofronis | A memory-optimal buffering protocol for preservation of synchronous semantics under preemptive scheduling[END_REF]. The size of the buffers, and the read and write positions inside the buffers for each node execution are determined by the compiler in a way that ensures data consistency. These communication mechanisms also prevent race conditions and priority inversions like the Logical Execution Time model.

L4576 → / ˆ5 * ˆ2 → r2ms3 L646 → / ˆ25 → r50ms36).
The language allows some flexibility in the implementation of the buffer associated to each label and of the read and write operations. Such an implementation is considered correct provided that: 1) it respects causality, meaning that the writer completes before the reader begins (except for fby communications); 2) tasks respect their deadlines. The implementation choice has an impact on WCET and performances, since it impacts the memory access contentions, especially for multicore targets. In the past, we proposed a shared variable based approach [START_REF] Forget | A real-time architecture design language for multi-rate embedded control systems[END_REF], and a distributed approach on the many-core Intel Single-chip Cloud Computer (SCC) [START_REF] Puffitsch | Mapping a multi-rate synchronous language to a many-core processor[END_REF], which has been introduced directly in preludec 1.6.0.

More recently, we have specialized this generic distributed approach for the AER model [START_REF] Durrieu | Predictable flight management system implementation on a multicore processor[END_REF] on an ARM-based multicore very similar to the one of the challenge. This compilation scheme based on AER for synchronous programs has been integrated in the WCC compiler [START_REF] Pagetti | Space and time aware compilation of synchronous programs[END_REF] for WCET purpose. The code associated to any runnable is split into three C functions:

• phase A: data consumed by an imported node is first read during an acquisition phase A, i.e. copied from the memory where it is stored to a local variable; • phase E: the imported node executes in isolation from other cores and without accessing any shared resources; • phase R: data consumed by other imported nodes is written in the LRAM during this phase.

The phases are then scheduled according to a non-preemptive schedule computed off-line. The model assumes that when a flow is produced by an imported node and consumed with a direct communication, phase R of the producer must execute before phase A of the consumer. b) (Manual) Adaptation for the challenge: The AER implementation together with the PRELUDE communication protocol fits the challenge semantics. However, it is not optimal in terms of memory size and memory contentions. We propose an AER adaptation that reduces these overheads. First, we split the phase A into two sub-phases:

Fig. 1 .

 1 Fig. 1. LET communication and implicit communication

  . . ) r e t u r n s ( L a b e l 9 9 4 4 ; . . . ) var L a b e l 6 5 , . . . l e t ( L a b e l 6 5 , L a b e l 2 5 6 0 , L a b e l 9 6 0 4 , L a b e l 9 6 6 0 ) = Runnable 100ms 246 ( L a b e l 4 5 3 0 , L a b e l 3 5 8 3 , L a b e l 4 3 3 8 , L a b e l 3 1 6 , L a b e l 2 3 5 4 , L a b e l 4 4 5 5 , L a b e l 5 7 0 7 , L a b e l 7 3 3 4 , L a b e l 7 8 6 4 ) ; ( L a b e l 1 4 1 8 , L a b e l 4 5 3 0 , L a b e l 2 5 5 6 , L a b e l 7 4 2 ) = Runnable 100ms 239 ( . . . ) ; . . . = Runnable 100ms 36 ( . . . , 0 f b y L a b e l 6 5 , . . . ) ; ( L a b e l 2 4 9 2 , L a b e l 3 5 4 3 , L a b e l 4 7 0 4 , L a b e l 9 9 4 4 ) = Runnable 5ms 0 ( ( 0 f b y L a b e l 7 4 4 ) * ˆ2 , ( 0 f b y L a b e l 5 8 6 ) * ˆ2 , 0 f b y L a b e l 1 3 9 4 , L a b e l 1 7 7 9 / ˆ1 0 * ˆ3 ) ; . . . t e l

  The maximal latency is achieved for r700us800us3 62 , which starts at 48.8ms and which impacts r50ms36 3 ending at most at 150ms. The resulting latency is 150 -48.8 = 101.2ms. The LET implementation is only 2ms greater than the IMPLICIT implantation. Current implementation in PRELUDE: Communications in the C code generated by the Prelude compiler rely on buffering mechanisms, similar to the protocol by Sofronis et al

	Effect chains EC 1	EC 2	EC 3
	Max latency 20ms 112ms 103.2ms
	IV. IMPLEMENTATION OF THE COMMUNICATIONS
	We answer in this section to the challenge issues 1 and
	2, that is how to implement efficient LET communication and
	provide an assessment of the overhead in terms of extra cycles
	and memory.
	A. Buffers (or Labels) allocation
	a)

• A let: labels from other tasks are copied at task activation; • A implicit: labels from other runnables of the same task are copied at the beginning of a runnable execution.

Let us illustrate the wrapping of Runnable 5ms 0 shown in figure 2. The runnable comes with 4 functions, namely r5ms0 X with X ∈ {a let, a implicit, e, r}. In the A let phase, each consumed label Label l is copied in a local variable named lr Label l r5ms0. In the R phase, each produced label Label l is copied from a local variable named lw Label l. In the A implicit phase, each consumed label Label l is obtained simply by copying the variable lw Label l, allocated as a static variable on the local core, in a local variable named as lr Label l r5ms0. During the E phase, only local variables are used (lr, lw). We define the following implementation rules:

• spatial rule: for each core c, for each Label l, -For each consumer runnable rX executing on c, there is a local copy lr Label l rX; -If Label l is mapped on c and produced by runnable r, there is local copy lw Label l managed by r;

• temporal rule:

-At the activation of a task t, all rX a let are called in sequence for all runnables rX belonging to t; -At the start of a runnable r, r a implicit is first called; -At the termination of a task, all rX r are called in sequence.

B. Cost and optimization

In this section, we discuss the communication overheads. First, in terms of memory, the total allocated memory is:

where nb consumers(Label l) is the number of runnables consuming Label l and producer(Label l) is equal to 0 if Label l is an external input and 1 otherwise.

We can optimize the memory allocation according to the following rules:

• on each core c, for each label Label l, for each task t consuming Label l and not producing it, there is a single local copy of lr Label l t; • For each task t, for each label Label l produced in t, all consumer runnable will directly use lw Label l. Then the required memory will only be: l∈ [START_REF] Amalthea | An open platform project for embedded multicore systems[END_REF]10000] (nb task consumers(Label l) + 1) × size(Label l)

where nb task consumers(Label l) is the number of tasks consuming Label l. Now, we consider the memory contentions. Let n÷k denote that the rest of the division of n by k is 0. We propose the following scheduling strategy to reduce contentions:

• Pre-compute off-line the set of activation dates p over the hyper-period H = lcm(period(t)). More precisely, {p ≤ H|∃t, p ÷ period(t)}

• Create for each core c the new tasks c let p and c r p with the highest priority; -For all runnable r executing on c such that p ÷ period(r), c let p calls in sequence all its a let functions; -At the termination p -, c r p calls in sequence the r phases of terminating runnables.

• At each date p, we first execute the phases c0 let p on core 0, then to distant memory is done in isolation, we can compute the latency of a read. Thus the code for core i would be: wait(d 0,p + . . . + d i-1,p ); c0 let p; wait(d i+1,p + . . . + d 3,p ).

Thanks to this mechanism, the phases E and R will be fully in isolation. Another approach would be to compute the maximal conflicts encountered by each piece of code.

V. CONCLUSION

In this paper, we presented the main principles of a translation from AMALTHEA to PRELUDE and discussed alternative communication implementations for the challenge. We also presented the C code generation from PRELUDE. Using the translator in combination with the preludec compiler actually provides an AMALTHEA to C compilation chain. It took us 2 days to understand the challenge, 4 days to implement the translator from AMALTHEA to PRELUDE, .5 days to analyze latencies and 1 day to design the adaptations and optimizations presented at the end of the paper. An URL for downloading the translator will be posted on the WATERS forum by the end of the submission week.