Avinash Maurya

Bogdan Nicolae
email: bnicolae@anl.gov

Ishan Guliani

M Mustafa Rafique
email: mrafique@cs.rit.edu

CoSim: A Simulator for Co-Scheduling of Batch and On-Demand Jobs in HPC Datacenters

Keywords: High-performance computing, batch job preemption, job checkpointing

The increasing scale and complexity of scientific applications are rapidly transforming the ecosystem of tools, methods, and workflows adopted by the high-performance computing (HPC) community. Big data analytics and deep learning are gaining traction as essential components in this ecosystem in a variety of scenarios, such as, steering of experimental instruments, acceleration of high-fidelity simulations through surrogate computations, and guided ensemble searches. In this context, the batch job model traditionally adopted by the supercomputing infrastructures needs to be complemented with support to schedule opportunistic on-demand analytics jobs, leading to the problem of efficient preemption of batch jobs with minimum loss of progress. In this paper, we design and implement a simulator, CoSim, that enables on-the-fly analysis of the trade-offs arising between delaying the start of opportunistic on-demand jobs, which leads to longer analytics latency, and loss of progress due to preemption of batch jobs, which is necessary to make room for on-demand jobs. To this end, we propose an algorithm based on dynamic programming with predictable performance and scalability that enables supercomputing infrastructure schedulers to analyze the aforementioned trade-off and take decisions in near real-time. Compared with other state-of-art approaches using traces of the Theta pre-Exascale machine, our approach is capable of finding the optimal solution, while achieving high performance and scalability.

I. INTRODUCTION

Big data analytics and deep learning are rapidly gaining traction both in the industry and scientific computing. A key driver for this trend has been the unprecedented accumulation of big data, which exposes plentiful learning opportunities thanks to its massive size and variety. Unsurprisingly, there has been a significant interest to adopt deep learning at a very large scale on supercomputing infrastructures in a wide range of scientific areas, e.g., fusion energy science [START_REF] Tang | Performance portability of hpc discovery science software: Fusion energy turbulence simulations at extreme scale[END_REF], computational fluid dynamics [START_REF] Kozelkov | Investigation of supercomputer capabilities for the scalable numerical simulation of computational fluid dynamics problems in industrial applications[END_REF], lattice quantum chromodynamics [START_REF] Vranas | The bluegene/l supercomputer and quantum chromodynamics[END_REF], virtual drug response prediction [START_REF] Ellingson | Polypharmacology and supercomputer-based docking: opportunities and challenges[END_REF], and cancer research [START_REF] Aocnp | Watson will see you now: a supercomputer to help clinicians make informed treatment decisions[END_REF].

One of the main use cases of big data analytics and deep learning in scientific computing is to use them as a tool to complement high-performance computing (HPC) simulations running on supercomputing infrastructures in a variety of scenarios: steering of experimental instruments (e.g., calibrate scientific instruments in real-time to correct anomalies in experimental data and/or refocus dynamically on areas of interest), acceleration of high-fidelity simulations through surrogate computations (e.g., under the right circumstances, expensive steps of an HPC simulation can be replaced with faster deep learning predictions), guided ensemble searches (e.g., when running a set of simulations to find a molecule that docks to a protein, deep learning can be used to predict the next most promising simulations to try next).

These scenarios require running opportunistic on-demand jobs when certain conditions are triggered, e.g., an analytics job that looks for anomalies in the experimental data collected by the instrument, a deep learning training and/or inference. These jobs need to start within a given deadline, often in the order of minutes. Failure to start them by the given deadline leads to a missed opportunity (e.g., it's too late to calibrate the instrument) and/or incur a performance penalty (e.g., idle simulations that wait for the next deep learning prediction or otherwise take alternative suboptimal decisions). On the other hand, HPC datacenters traditionally adopt a batch job scheduling model where users request compute and accelerator (e.g., GPU) resources of the datacenters for the required amount of time (wall time), while the scheduler decides when to run each job based on various trade-offs such as the need to maximize the utilization of machines, and job priority. Popular HPC datacenter schedulers, e.g., SLURM (Simple Linux Utility Resource Manager) [START_REF] Yoo | Slurm: Simple linux utility for resource management[END_REF], COBALT [START_REF] Desai | Cobalt: an open source platform for hpc system software research[END_REF], and TORQUE (Terascale Open-source Resource and QUEue manager) [START_REF] Staples | Torque resource manager[END_REF], cannot co-schedule batch jobs with opportunistic on-demand jobs.

A naive solution to address this problem could simply reserve a set of nodes for on-demand jobs and use the rest of the nodes for batch jobs. Although applied in practice, such a solution is not desired as it is hard to predict how many nodes are needed by the on-demand jobs. Using too few nodes for on-demand jobs leads to missed opportunities, whereas, using too many nodes leads to idle nodes and slow progress of the batch jobs. Furthermore, even if such predictions were perfect, there may be significant fluctuations in datacenter utilization patterns that make it hard to dynamically move the nodes back and forth between on-demand and batch queues. At the other extreme, an alternative naive solution could simply use all nodes for batch jobs and start killing batch jobs to make room for on-demand jobs when needed. This solution does not lead to missed opportunities but may incur significant overhead on the batch jobs due to loss of progress.

In this paper, we propose an alternative solution to address these challenges that relies on checkpointing for suspending and resuming batch jobs, if required, to make room for time-978-1-7281-7343-6/20/$31.00 ©2020 IEEE sensitive on-demand jobs, thereby minimizing the amount of lost progress by the batch jobs. To this end, we introduce CoSim, a simulation framework that aims to identify the optimal combination of jobs that should be either checkpointed or killed to free a fixed number of nodes that are required to run the on-demand job. Unlike other approaches, CoSim simulates all outcomes resulting from a variable deadline up to the given maximum in a single pass, thereby eliminating the need to run separate simulations for each fixed deadline. Using this approach, the scheduler can make more informed decisions by considering the various trade-offs arising from delaying the start of the on-demand jobs and losing progress on the batch jobs. Specifically, we make the following contributions in this paper:

• We formulate the problem statement, introducing a series of assumptions and general considerations for simulating the outcomes of checkpointing batch jobs to vacate nodes for running on-demand jobs (Section II). • We introduce a series of design principles and an algorithm based on the dynamic programming to find the optimal combination of batch jobs that incurs the minimum loss of progress while satisfying the deadline of the ondemand jobs. Our algorithm produces an optimal solution for every possible deadline up to a given maximum in a single pass (Section III). • We evaluate our approach in a series of experiments using three scenarios extracted from the batch job traces of Argonne's Theta pre-Exascale machine. We compare our approach with an exhaustive search based on backtracking and a greedy approach. The results show significant performance and scalability improvement as compared to backtracking, as well as a significant improvement in the quality of the solution compared as compared to the greedy approach (Section IV).

II. PROBLEM FORMULATION

The problem of co-scheduling batch jobs with opportunistic on-demand jobs in an HPC datacenter can be formulated as follows. Let's assume that N batch jobs are running at time t 0 , and each of these jobs is characterized by the tuple id, jn, loss, tckpt , where id is a unique identifier of the job, jn is the number of compute nodes the batch job is running on, loss quantifies the amount of lost progress if the job is killed (e.g., node-hours since the last checkpoint or since the beginning if no checkpoint was taken), tckpt is the time required to checkpoint the job to successfully suspend its execution without loss of progress.

Given K nodes that need to be released not later than t 0 +T (for the purpose of starting opportunistic on-demand jobs), the goal is to find all optimal subsets of batch jobs S i ⊂ N and corresponding killing or checkpointing strategy for all t 0 < i < t 0 + T . A subset S i is optimal if it satisfies the following properties simultaneously: (1) at least K nodes are released by the deadline t 0 +i; (2) the accumulated loss of work due to killing jobs is minimized; (3) if there are multiple subsets S i for which the accumulated loss due to job kills is minimized, then prefer the subset for which the checkpointing overhead is minimized. We refer to this as the eviction problem.

Using the subsets S i and corresponding strategy, an HPC datacenter scheduler can simulate the outcome of multiple hypothetical scenarios with a variable deadline i corresponding to the trade-off between maximizing the value of the ondemand jobs and minimizing the loss of the batch jobs.

We note that while we formulate the problem of HPC datacenters, a similar formulation can be done for opportunistic jobs in cloud computing architectures where there is an upper bound on elasticity, e.g., the user cannot afford to run on more than a fixed amount of virtual machines (VMs) at a time and must evict existing jobs if necessary. Without loss of generality, CoSim can be applied in such scenarios as well.

III. DESIGN PRINCIPLES AND APPROACH

This section introduces the high-level design principles of our proposed approach and explains aspects related to the checkpointing model and exploration algorithm that implement these design principles.

A. Design principles

CoSim is based on the following design principles: 1) Mix of system-level and application-level checkpointing: We differentiate between system-level and application-level checkpointing, because they present an interesting trade-off: system-level checkpointing techniques, such as DMTCP [START_REF] Ansel | DMTCP: Transparent checkpointing for cluster computations and the desktop[END_REF], are application-agnostic and can be performed at any moment t 0 . However, they involve large checkpoint sizes because the entire memory space of all application processes needs to be persisted to a stable storage, e.g., a parallel file system (PFS). Therefore, system-level checkpointing may take a long time to complete. On the other hand, application-level checkpointing is typically performed by HPC applications regularly using either a custom solution or a checkpointing library, such as, VELOC [START_REF] Nicolae | Veloc: Towards high performance adaptive asynchronous checkpointing at large scale[END_REF]. In this case, the checkpoint size is smaller as each application process needs to save only the critical data structures needed for a restart and therefore faster to write to the stable storage. However, it is necessary to wait for the application to reach a moment t 1 > t 0 when it is safe to checkpoint. Depending on how far away t 1 is from t 0 and how much larger a system-level checkpoint is compared with an application-level checkpoint, one or the other may be faster. Furthermore, it is important to note that even if the system and application-level checkpointing overheads are equal, it is still important to choose the application-level checkpoint over the system-level checkpoint, because using application-level checkpoint enables the batch job to make additional progress during the interval (t 0 , t 1 ). We incorporate such considerations in our simulator.

2) Simultaneous exploration of the full on-demand deadline range: As discussed in Section II, our goal is to solve the eviction problem for all deadlines in the range (t 0 , t 0 + T ), because the scheduler needs to consider the trade-off between delaying the on-demand jobs, which may lead to lower quality of the results due to slow reaction time, and losing progress of the batch jobs. Thus, a naive strategy would be to iterate over all deadlines i in range (t 0 , t 0 + T ) and solve the problem independently for each i. However, such a strategy is sub-optimal, because the problems resulting from fixing all deadlines i in range (t 0 , t 0 + T ) have identical inputs except for the deadline i, therefore they may be decomposed into sub-problems that are shared across several instances and thus need to be solved only once. Our approach leverages this observation to construct an algorithm based on dynamic programming that is capable of taking advantage of such decompositions to solve all deadlines in a single pass. This algorithm is explained in Section III-C.

3) Polynomial response time: A key requirement for exploring the full on-demand deadline range is to ensure fast response time so that the scheduler can decide quickly, preferably at moment t 0 , about the jobs that must be checkpointed to the PFS to run the incoming on-demand jobs. Thus, an algorithm that is not polynomial in any variable, such as, the number of jobs N , maximum deadline T , or the number of nodes to be released K, will lead to unacceptable response time, considering that modern HPC datacenters routinely run several batch jobs simultaneously and may need to release a large number of nodes for on-demand jobs to accommodate bursts of opportunistic events. Therefore, our proposed solution is designed to satisfy such constraints, and delivers response times in the order of milliseconds or less.

B. Loss and checkpointing model

We estimate the loss incurred by killing a batch job as the number of node-hours that have elapsed since its last application-level checkpoint until t 0 , the moment when the nodes need to be evicted to make room for the on-demand jobs. This is based on a configurable interval that can be independently adjusted for each job in our simulator. In practice, the interval is fixed based on empirical observations, e.g., every hour, because the checkpoints are used both to survive failures and to record intermediate results. However, if checkpoints are only used for fault tolerance, then an optimal checkpointing interval can be computed [START_REF] Daly | A higher order estimate of the optimum checkpoint interval for restart dumps[END_REF].

In order to simulate alternatives that mix application-level checkpointing with system-level checkpointing, we consider the time to checkpoint each batch job:

tckpt = max( jn i=1 sckpt(i)/B a , max jn i=1 (sckpt(i)/B c )) (1)
where jn is the number of nodes occupied by the batch job, B a is the aggregated I/O bandwidth of the PFS, B c is the maximum I/O bandwidth of a compute node, and sckpt(i) is the size of the checkpoint on node i ∈ [1 . . . jn]. The intuition behind this is that the checkpointing time is bounded either by the maximum aggregated bandwidth of the PFS or the slowest node (if the nodes do not consume the maximum aggregated bandwidth).

In the case of application-level checkpointing, we must wait for the next checkpoint to happen, which introduces a delay in addition to tckpt. Therefore, the application-level checkpointing duration is ta = tckpt + delay, where delay is the difference between the next scheduled checkpoint and t 0 . Since system-level checkpointing can be performed instantly at t 0 , its duration, ts, will be equal to tckpt, i.e., ts = tckpt. However, the two approaches will have a different checkpoint size on each node, resulting in the trade-off that is explained in Section III-A.

In a typical HPC datacenter, the size of each batch job is usually large enough to saturate the aggregated I/O bandwidth of the PFS. Therefore, we consider a simple checkpointing model where the jobs are checkpointed serially. In this case, the total time required for checkpointing a set of batch jobs is the sum of their corresponding ta or ts. In fact, under such circumstances, checkpointing multiple batch jobs in parallel would perform worse than checkpointing the batch jobs serially, because of over-subscribing the aggregated I/O bandwidth of the PFS. Nevertheless, we note that our model can be further refined to simulate concurrent checkpointing of the batch jobs in the case of small jobs that do not saturate the aggregated I/O bandwidth of the PFS.

C. Exploration algorithm

In this section we propose a dynamic programming algorithm based on the aforementioned design principles.

The key observation that inspires our algorithm is the fact that the eviction problem is related to the discrete backpack problem: given N items, where W i and V i represent the weight and value of the i th item, fill a backpack that can carry a maximum weight K such that the combined value of all items is maximized without overflowing K. By analogy, we can consider the batch jobs as items and the backpack as the set of nodes where the jobs are running. This problem has a simple dynamic programming decomposition: maximum value for N items is the greater of: (1) the maximum value for N -1 items and capacity K (excludes item N ); (2) V N plus the maximum value obtained for N -1 items and capacity K-W N (includes item N ). By solving this decomposition recursively and applying memoization techniques, a runtime complexity of O(N • K) can be achieved. Note that this decomposition solves the problem not only for a backpack of capacity K, but at the same time for all backpacks of capacity i such that 0 < i ≤ K.

Starting from this observation, we adopt a similar strategy but with two important differences. First, we need to free at least K nodes, which means that the optimal solution may involve more than K nodes. Therefore, we need to consider up to M nodes, where M is the number of nodes occupied by all batch jobs at the moment t 0 . Second, the eviction problem introduces a new dimension in the decomposition, i.e., the deadline T to start the on-demand jobs. Specifically, it is not enough to release at least K nodes within a deadline T when considering N -1 batch jobs and then try for the N th batch job all four alternatives, i.e., ignore, kill, application-level checkpoint, and system-level checkpoint, because the optimal solution for N -1 batch jobs may get close to the deadline Algorithm 1: Dynamic programming algorithm to free K nodes within a range of deadlines [0 . . . T ] with minimal loss of compute progress.

Input: List J of N batch jobs running at t0, K, T Output: List of job eviction strategies Si,

0 < i < T 1 a[0, 0] ← 0 2 u[0, 0] ← ∅ 3 M ← 0 4 for (id, jn, loss, ts, ta) ∈ J do 5 M ← M + jn 6 for (id, jn, loss, ts, ta) ∈ J do 7 b ← a 8 v ← u 9 for (n, t) ∈ a do 10 if b[n + jn, t] > a[n, t] + loss then 11 b[n + jn, t] ← a[n, t] + loss 12 v[n + jn, t] ← u[n, t] ∪ {(id, "kill")} 13 if t + ta <= T ∧ b[n + jn, t + ta] > a[n, t] then 14 b[n + jn, t + ta] ← a[n, t] 15 v[n + jn, t + ta] ← u[n, t] ∪ {(id, "app")} 16 if t + ts <= T ∧ b[n + jn, t + ts] > a[n, t] then 17 b[n + jn, t + ts] ← a[n, t] 18 v[n + jn, t + ts] ← u[n, t] ∪ {(id, "sys")} 19 a ← b 20 u ← v 21 for i ∈ [0 . . . T ] do 22 (x, y) ← argmin(a[x = K . . . M, y = 0 . . . i]) 23 Result[i] ← (a[x, y], u[x, y])
24 return Result T , thereby limiting the set of valid choices for job N (e.g., no further checkpointing is possible within T ).

As a consequence, we propose a two-dimensional decomposition based on both the number of nodes and the deadline. We denote with the tuple jn N , loss N , ts N , ta N the number of nodes, loss of progress due to job killing, system-level checkpointing duration, and application-level checkpointing duration for job N . Then, the minimum loss for N jobs, M nodes, and deadline T denoted as a[N, M, T ] is the lesser of: (1) ignore job N , i.e., a[N -1, M, T ]; (2) kill job N , i.e., loss N + a[N -1, M -jn N , T ]; (3) take an application-level checkpoint of job N , i.e., a[N -1, M -jn N , T -ta N ]; and (4) take a system-level checkpoint of job N , i.e., a[N -1, Mjn N , T -ts N ]. Algorithm 1 presents our approach to solve this decomposition with a runtime of O(N •M •T ). The output of this algorithm is a list S i for 0 < i < T , where S i is the set of jobs to be evicted using an optimal strategy such that the compute loss is minimized, and, in case of multiple solutions with minimal compute loss, the checkpointing overhead is minimized too.

We note that Algorithm 1 uses a temporary minimum loss matrix b and a corresponding solution matrix v to hold the updates resulting from considering all alternatives for job id. This is needed in order to avoid repeatedly selecting the same id in subsequent decompositions. Furthermore, the application-level checkpointing strategy takes precedence over the system-level checkpointing during the updates, and thus becomes a preferred choice in the case of equal loss and checkpointing time.

Another important observation is that Algorithm 1 solves the eviction problem not only for at least K nodes, but the entire node spectrum [0 . . . M ]. This enables the scheduler to consider more advanced trade-offs for on-demand jobs, such as running the on-demand jobs with more or less than K requested nodes, which can be used to dynamically adjust the latency and/or the quality of the on-demand results. Such trade-offs can be incorporated at no additional simulation cost using our proposed approach.

IV. PERFORMANCE EVALUATION

To evaluate our proposal, we study the traces of Argonne's Theta pre-Exascale machine and extract three representative scenarios that create a challenging situation with respect to the eviction problem: most of the nodes are occupied by a relatively large number of batch jobs, leading to many possible combinations that need to be explored. For each scenario, we augment the traces with additional data that enables us to apply our model in order to extract the parameters of each batch job: compute loss and application-level/systemlevel checkpointing duration. We then compare our dynamic programming algorithm with two other approaches: a greedy algorithm (linear complexity) and a backtracking algorithm that performs an exhaustive search (exponential complexity). For the rest of this section, we introduce the methodology of our proposal and discuss the results of the comparison.

A. Batch job traces

In this paper, we consider the case of Argonne's Theta supercomputer, a 11.69 petaflops pre-Exascale Cray XC40 system based on the second-generation KNL Intel Xeon Phi 7230 SKU. The system has 4392 nodes, each equipped with 64 core processors (256 hardware threads), 16 GB of highbandwidth MCDRAM (300-450 GB/s), 192 GB of main memory (DDR4 RAM, 20 GB/s), and a 128 GB SSD (700 MB/s). The interconnect topology is based on Dragonfly with a total bisection bandwidth of 7.2 TB/sec. Durable storage is provided by a Lustre parallel file system that is accessible to the compute nodes through a POSIX mount point. The total aggregated bandwidth is 250 GB/s.

First, we study the DIM_JOB_COMPOSITE trace1 of batch jobs executed on Theta between 2017 and 2019. Specifically, we extract for each job the required fields pertaining to the runtime (start time, execution time) and the number of nodes. Then, we aggregate this information to obtain the number of batch jobs and the number of nodes utilized by the batch jobs per time unit. We focus in particular on the year 2019, which reflects the most recent utilization pattern: a total of 91,217 batch jobs were executed during the entire year.

We zoom on the node utilization (Figure 1a) and the number of jobs (Figure 1b pattern can be observed for the rest of the year. These figures reveal several interesting observations:

• During the entire year, all 4392 nodes were occupied for only around 1.5 days. However, the most frequent number of occupied nodes is 4352, which is close to the maximum capacity and appears for a total of 34 days. Therefore, the likelihood of having to run on-demand jobs when the machine runs batch jobs close to full capacity is very high. • When the machine is operating close to capacity (4352 occupied nodes), the number of jobs is relatively high, peaking at around 25 jobs. • About 61% of the batch jobs reported an execution time of less than 30 minutes. We consider these batch jobs expendable, such that killing them incurs negligible loss. Based on these observations, we construct three representative scenarios, each of which occupies 4352 nodes at moment t 0 and a variable number of jobs: 12, 16, 24. We deliberately avoid expendable jobs in these scenarios (i.e., no expendable job is running at moment t 0 ) in order to create a challenging situation where all jobs may incur a significant loss of nodehours. The scenarios are illustrated in Figure 2, Figure 3 and Figure 4. The regions of interest during which all jobs are running are marked with between two vertical timestamps relative to the beginning of the earliest job. For example, Figure 2 captures a scenario of 12 jobs running for a total of 10 hours and 15 minutes, where, all batch jobs overlap for about 2 hours, i.e., from 02:41 to 04:52. The moment t 0 is chosen within these regions of interest.

B. Augmentation of the traces with checkpointing parameters

The DIM_JOB_COMPOSITE trace does not capture any information about the checkpointing behavior of the batch jobs. Lacking such information, we augment the scenarios with a series of synthetically generated checkpointing information based on empirical observations. Specifically, we assume that all batch jobs conduct application-level checkpoints at an hourly interval. Since the batch jobs have different start times, the likelihood that their application-level checkpoints are written concurrently to the PFS is very small. Furthermore, we assume that each batch job allocates between 40%-90% of the memory available on each node. In this case, the size of the system-level checkpoint on each node coincides with the allocated memory. Out of this memory, we assume 20%-60% holds critical data structures that are written by application-level checkpointing approaches. This is the size of the application-level checkpoints. We use a random threshold for each batch job, both for the application-level and systemlevel checkpoints, which is then used in Equation 1 to calculate the application-level and system-level checkpointing duration.

C. Compared approaches

Throughout our evaluations, we compare three approaches that can be used to solve the eviction problem: 1) Greedy: This algorithm implements a greedy strategy that tries to minimize the loss by checkpointing the most expensive jobs (high loss), while killing the least expensive jobs (low loss). To this end, it sorts the batch jobs in descending order of loss and tries to checkpoint them using the fastest available checkpointing method (application or system level). When the total checkpoint duration becomes larger than the deadline T , it iterates over the sorted jobs in reverse order starting from the end, killing them one by one until at least K nodes have been released. While it does not produce an optimal solution, this algorithm has linear complexity and therefore has a very fast response time.

2) Backtracking: This algorithm implements an exhaustive search of all possible choices for each batch job: keep running (exclude), kill, checkpoint at application-level, checkpoint at system-level. It optimizes the search by early abandoning of all combinations that cannot achieve a lower loss than the best combination found so far. Unlike Greedy, this approach always produces an optimal solution, however it has an exponential complexity and therefore may become untractable for large problem sizes.

3) CoSim: This is our proposal that implements Algorithm 1. It guarantees an optimal solution just like Backtracking, but at the same time it has a fast response time thanks to its polynomial complexity.

D. On-demand job configurations

For each of the three scenarios mentioned in Section IV-A, we consider the maximum deadline T = 15 minutes. We are interested in the optimal eviction strategy for all deadlines in the range [0 . . . 15] with a granularity of one minute. Furthermore, for each of the three scenarios, we consider three different values for K, the minimum number of nodes that need to be evicted in order to make room for the on-demand jobs, i.e., 512, 1024, and 2048 for Scenario-1, Scenario-2, and Scenario-3, respectively. This roughly corresponds to 12.5%, 25% and 50% of the total capacity of Theta.

E. Results

First, we focus on the performance and scalability of the three approaches. To this end, we measure the response time taken by each approach in order to produce the optimal eviction strategy for all deadlines in the range [0 . . . T ]. As a consequence, in the case of Greedy and Backtracking, a separate run is executed for each i ∈ [0 . . . T ]. Therefore, for an increasing i, the response time measures the accumulated runtime of all i runs. In the case of CoSim, a single run is sufficient to obtain the full solution thanks to the memoization of overlapping sub-problems. This metric is important because it determines how soon the scheduler can take decisions, which in turn impacts the value that can be extracted from the ondemand jobs (i.e., faster response time leads to better ondemand job results).

The results for each of the three scenarios are depicted in Figure 5, Figure 6 and Figure 7 respectively. Note that due to the large differences in algorithmic complexity between the three approaches, the y-axis is represented in a log scale. As expected, CoSim keeps a constant response time regardless of the deadline T . Despite the accumulation of response time from an increasing number of runs, the Greedy approach is still at least 30x faster than the other two approaches thanks to its linear complexity. It is interesting to observe that for a small K and a small number of batch jobs (as illustrated by Scenario-1), Backtracking is faster than our approach. However, with increasing K and number of batch jobs (as illustrated by Scenario-2 and Scenario-3), the limitation of the exponential search becomes clearly visible, despite the aggressive early pruning optimization. In this case, our approach is up to five orders of magnitude faster. As a general conclusion, we observe that our approach has the advantage of providing the optimal solution within a predictable constant time, which is well suited for real-time scheduling decisions.

Next, we focus on the quality of the results of the Greedy approach. Since both our approach and the Backtracking approach produce the optimal solution, we use it as a baseline that we substract from the minimum loss found by the Greedy approach. We call this the relative compute loss. This metric is important, because it indicates what result quality degradation can be expected in order to benefit from faster response time.

As can be observed in Figure 8, the relative compute loss is very high, indicating that degradation in the quality of the result found by Greedy is unacceptable. In fact, with the exception of T > 13 for Scenario-3, the relative compute loss is increasing for an increasing T , which means Greedy suffers from an increasing degradation in the quality of the result. Also, it is important to note that in absolute terms, the minimum compute loss is decreasing with an increasing T for all three approaches, because more checkpointing opportunities become available. In fact, the optimal minimum loss is found by CoSim and Backtracking is often 0 (meaning no job needs to be killed), especially for larger T . Therefore, even when the relative compute loss seems to decrease for an increasing T , it is still missing the optimal compute loss by a large margin.

Based on this observation, we conclude that sacrificing the result quality for faster response time is not beneficial, especially when considering that our approach runs in the order of milliseconds in the worst case.

V. RELATED WORK

Scheduling of batch and on-demand jobs for concurrent execution where resources sharing is limited to each type of job has been widely studied [START_REF] Tyagi | A survey on scheduling algorithms for parallel and distributed systems[END_REF]- [START_REF] Ahmad | Data-intensive workflow optimization based on application task graph partitioning in heterogeneous computing systems[END_REF] in the past. However, not much work has been done for collocating both batch and on-demand jobs on the same set of resources [START_REF] Wang | Supporting Real-Time Jobs on the IBM Blue Gene/Q: Simulation-Based Study[END_REF]. SPRUCE (Special Priority and Urgent Computing Environment) [START_REF] Trebon | Enabling urgent computing within the existing distributed computing infrastructure[END_REF] supports on-demand jobs by considering a basic preemptive scheduling scheme with no checkpointing. However, this leads to a significant loss of progress for the batch jobs.

Checkpointing based preemptive scheduling has been traditionally used at the operating system level for multi-tasking. However, recent checkpointing-based preemptive scheduling schemes [START_REF] Leung | Parallel job scheduling policies to improve fairness: A case study[END_REF], [START_REF] Snell | Preemption based backfill[END_REF] focus on reducing their overheads and improving their effectiveness in reducing the average job turnaround time. Nevertheless, these techniques do not directly address the challenges of co-scheduling batch and on-demand jobs in HPC settings.

Large-scale datacenters operated by industry (e.g., Facebook [START_REF] Meza | A large scale study of data center network reliability[END_REF] and Google [START_REF] Jain | B4: Experience with a Globally Deployed Software Defined WAN[END_REF]), leverage centralized job execution environments where the centralized system accumulates jobs from multiple datacenters, and then runs the computation [START_REF] Corbett | Spanner: Google's globally distributed database[END_REF]. However, it leads to increased network traffic and job completion time when the data volume grows exponentially [START_REF] Vulimiri | Global analytics in the face of bandwidth and regulatory constraints[END_REF], [START_REF] Muralidhar | f4: Facebook's Warm BLOB Storage System[END_REF]. Furthermore, regulations may restrict moving data across continents due to security and privacy constraints, thus making such approaches impractical to adopt in production environments at large.

VI. CONCLUSIONS

In this paper, we present CoSim, a simulator that enables onthe-fly analysis of the trade-offs arising between delaying the start of opportunistic on-demand jobs, which leads to longer analytics latency, and loss of progress due to preemption of batch jobs, which is necessary to make room for such ondemand jobs. The key idea of our proposal is to implement preemption through a combination of either killing or checkpointing (at application-level or system-level) a subset of batch jobs running on the compute nodes to free enough nodes by a given deadline. To this end, we introduce a checkpointing and loss model to develop a dynamic programming algorithm to minimize the loss for a variable deadline up to a given threshold, which gives the scheduler high flexibility in exploring a wide range of alternatives. CoSim finds the optimal solution up to 5 orders of magnitude faster than backtracking approaches and offers a predictable response time in the order of milliseconds, thereby eliminating the need for greedy approaches that are fast but find only approximate solutions.

In the future, we plan to investigate several avenues: (1) applicability of our proposal to cloud computing; (2) refinement of checkpointing model (interval, interactions with PFS); (3) integration with the workload schedulers at Argonne National Laboratory's supercomputers to validate CoSim for real-life on-demand workloads.

Fig. 1 :

 1 Fig. 1: Trace analysis for January 2019.

Fig. 2 :

 2 Fig. 2: Scenario-1: 12 batch jobs running on 4352 nodes.

Fig. 3 :

 3 Fig. 3: Scenario-2: 16 batch jobs running on 4352 nodes.

Fig. 4 :

 4 Fig. 4: Scenario-3: 24 batch jobs running on 4352 nodes.

Fig. 5 :Fig. 6 :

 56 Fig. 5: Response time for Scenario-1 consisting of 12 batch jobs. Note the log scale on the Y axis. Lower is better.

Fig. 7 :Fig. 8 :

 78 Fig. 7: Response time for Scenario-3 consisting of 24 batch jobs. Note the log scale on the Y axis. Lower is better.

https://reports.alcf.anl.gov/data/theta.html

ACKNOWLEDGMENTS

This material is based upon work supported by the U.S. Department of Energy (DOE), Office of Science, Office of Advanced Scientific Computing Research and Argonne National Laboratory. Results presented in this paper are obtained using the Chameleon and CloudLab testbeds supported by the National Science Foundation.


Amélie Penot
email: amelie.penot@chu-limoges.fr

Pierre-Marie Preux

Sandra Le Guyader

Albert Collignon

Aurélie Herry

Vinciane Dufour

Alain Monnereau

Anne-Sophie Woronoff

Xavier Troussard

Elisabeth Pons

Am É Lie Penot

Aur É Lie Herry

Dominique Bordessoule

Marc Maynadi

Incidence of chronic myeloid leukemia and patient survival: Results of 5 french population-based cancer registries 1980-2009

Keywords: Chronic myeloid leukemia, incidence, relative survival, population-based registries

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Successive advances in treatment and molecular monitoring of the therapeutic response have revolutionized the management and prognosis of chronic myeloid leukemia (CML). Th e fi rst therapeutic target was identifi ed 40 years after discovery of the Philadelphia (Ph) chromosome (reciprocal translocation 9;22) [START_REF] Brenner | [END_REF], 25 years after the development of the fi rst methods to quantify Ph-positive metaphases in bone marrow [2], and 15 years after the introduction of techniques for identifying and measuring BCR -ABL transcripts with deregulated tyrosine kinase activity [3]. Th e fi rst tyrosine kinase inhibitor (TKI) was introduced in the early 2000s, along with therapeutic monitoring by means of real-time quantitative polymerase chain reaction (RQ-PCR). Th e prognosis of CML has thus been transformed, using oral medication.

In the 1990s, treatment options for newly diagnosed patients with CML consisted of bone marrow transplant (BMT), interferon alpha (IFN α ) and chemotherapy (busulfan, cytarabine) or hydroxyurea. Five-year survival rates were disappointing, at around 57% with IFN α and 42% with chemotherapy [4]; the probability of long-term survival was 50 -80% among the 20 -30% of patients eligible for allogenic BMT [5,6]. In 2001, the fi rst phase I clinical trial [7] of imatimib mesylate (IM) showed cytogenetic responses in patients with unsatisfactory responses to standard treatments. IM became the fi rst-line treatment for CML following the IRIS study (International Randomized Study of Interferon vs. STI571) [8], and was shown in 2004 to markedly enhance patient survival [9]. Th e second generation of TKIs (dastinib, nilotinib) seems to be even more eff ective than IM [10 -12]. However, few data are available on the long-term survival of the general population of patients with CML treated with TKIs.

Th e aim of this study was to examine trends in the incidence of CML and patient survival in France over a 30-year period. We conducted a retrospective cohort study of consecutive, unselected patients with CML registered in fi ve French population-based cancer registries between 1980 and 2009. cases of cancer within the relevant administrative area. Th e C ô te-d ' Or registry specializing in hematologic malignancies and the Doubs general cancer registry each represents a population of about 500 000 inhabitants; the Limousin general cancer registry covers about 720 000 inhabitants; and the Basse-Normandie and Gironde registries specializing in hematologic malignancies cover about 1 440 000 inhabitants each. Exhaustive data were available for 1980 -2009 in the oldest registries (C ô te-d ' Or and Doubs), 1993-2009in the Limousin registry, 1997[START_REF] Brenner | [END_REF]

Patient selection

All cases of CML registered between 1980 and 2009 were included. Before the publication of the third edition of the International Classifi cation of Diseases for Oncology (ICD-O-3), a unique code (9863/3) was used for CML [13]. After publication of in 2000, two supplementary codes were created: 9875/3 (Ph ϩ CML with positive genetic [reciprocal translocation 9;22] or molecular biology [Bcr -Abl chimeric protein] documentation) and 9876/3 (myeloproliferative syndrome described in ICD-O-3 as CML with negative genetic and molecular biology studies: Ph Ϫ CML or atypical CML). Th e former code 9863/3 was conserved from the previous edition of ICD-O. Th e ICD-O-3 classifi cation has often been applied retrospectively to code cases before 2000. We chose to include patients coded 9863/3 in order to avoid underestimating the incidence of CML, but this code included both patients with Ph ϩ CML and patients with Ph Ϫ CML. It was therefore necessary to study patients with Ph Ϫ CML to understand the results for patients coded 9863/3.

Statistical analysis

Crude and age-standardized incidence rates and their confi dence intervals were calculated according to Bouyer et al . [15] with data of french population [START_REF]d é partement[END_REF]. All results are reported as the number of cases per 100 000 inhabitants per year with 95% confi dence intervals (CIs). We estimated incidence rates according to age at diagnosis (0 -14, 15 -44, 45 -54, 55 -64, 65 -74 and 75 -99 years). Age-standardized incidence rates were estimated using the direct method and calculated for reference populations: the Europe standard population (ESP) defi ned according to Waterhouse [17] and the World standard population (WSP) corresponding to Segi ' s world standards [18]. Th e cut-off date for determining vital status was 31 December 2013. Relative survival (RS) was defi ned as the ratio between observed survival and expected survival in the corresponding general population, thereby expressing the probability of CML survival after adjustment for competing causes of death. We used French mortality data published by the Human Mortality Database [START_REF] Usa | Max Planck Institute for Demographic Research (Germany) . Available from: www.mortality.org[END_REF]. RS was calculated using the Maya Pohar-Perme method implemented with the relsurv package for R [20]. All analyses were performed with R 3.0.2 open access software. 16.7). Th e male/ female sex ratio was 1.36. Patients classifi ed as 9875/3 represented 82.2% of the cohort. Only 13.6% of patients overall were classifi ed as 9863/3 (CML-not otherwise specifi ed [NOS]), and the proportion of these patients fell with time, from 22% in 1980 -1986 to 17% in 1987 -1999 and 11% in 2000 -2009. Th e proportion of patients classifi ed as 9876/3 was 4.2%, and this population had diff erent sociodemographic characteristics from the rest of the cohort, particularly a higher mean age (71.8 vs. 56.7 years, p Ͻ 0.0001). Th e estimated mean incidence of patients with CML in metropolitan France was 751 cases per year ( Ϯ 92).

Results

Study population

Incidence

Th e crude incidence rate was 1.14 [1.14 -1.14] per 100 000 person-years overall, 1.31 [1.31 -1.13] in men and 0.98 [0.98 -0.98] in women. 3) 20 ( 4) 33 ( 4)

Ph ϩ _CML, Philadelphia chromosome-positive chronic myeloid leukemia; CML_NOS: chronic myeloid leukemia not otherwise specifi ed; uMPS/SPMD: unclassifi able myelodysplastic syndrome/myeloproliferatif syndrome; SD: standard deviation

Crude incidence rates ranged from 0.07 per 100 000 person-years in the age group 0 -14 years to 0.60 at 15 -44 years, 1.49 at 45 -54 years, 2.29 at 55 -64 years, 2.79 at 65 -74 years and 2.34 at 75 -99 years.

Th e crude incidence rates among patients with CML 9875/3 were 0.05 at 0 -14 years, 0.55 at 15 -44 years, 1.36 at 45 -54 years, 1.97 at 55 -64 years, 2.28 at 65 -74 years and 1.58 at 75 -99 years.

Th e age-standardized incidence rate of cohort and Ph ϩ CML was respectively 1.02 [0.93 -1.11] and 0.97 [0.88 -1.05] relative to the European standardized rate (ESR), and 0.81 [0.72 -0.90] and 0.78 [0.71 -0.85] relative to the world standardized rate (WSR). Age-standardized incidence rates are reported in Table II.

Treatments

Th e treatments received were available for 679 patients. Table III describes the treatments in three calendar periods of diagnosis.

As shown in Table III, most patients diagnosed as 9875/3 in 1980 -1986 received chemotherapy (hydroxyurea or other). New treatments such as IFN α and autologous stem cell transplant were used in 1987 -1999. Only a few patients (mainly young patients with matched donors) received BMT. Among the patients diagnosed in 2000 -2009, 96.3% received a TKI. Some long-term survivors diagnosed earlier also received a TKI: 4% of patients diagnosed in 1980 -1986 and 29% of patients diagnosed in 1987 -1999. Patients with CML-NOS diagnosed in 1980 -1986 received chemotherapy (33.3%), BMT (33.3%) and/or hydroxyurea (66.7%), alone, in combination or sequentially. In 1987 -1999, hydroxyurea and BMT were the most frequently prescribed treatments, but more than 29.4% of these patients subsequently received a TKI. Among patients diagnosed in 2000 -2009, 93.3% were prescribed a TKI.

Patients with Ph Ϫ CML diagnosed in 1980 -1986 received chemotherapy (50.0%) or hydroxyurea (50.0%). In 2000 -2009, TKIs were prescribed to 29.4% of these patients.

Relative survival

Median follow-up was 51 months. Th e 5-year RS rate among patients diagnosed in 1980 -1986, before the use of IFN α , was only 40. 2% [28.7 -56.4]. Th is rate improved to 59.4% [53.0 -66.5] for patients diagnosed in 1987 -1999 and to 82.6% [78.3 -87.1] in the TKI era. Th e 5-year RS rate was 77. 4% [73.6 -81.3] among patients diagnosed with Ph ϩ CML in 1980 -2009, 52.8% [42.2 -66.0] in patients with CML-NOS and 12.1% [4.6 -32.1] in patients with Ph Ϫ CML. Figure 1 shows RS rates according to ICD-O-3 code and calendar period of diagnosis.

Among patients with Ph ϩ CML ( n ϭ 642), the median RS time was 48 months for those diagnosed in 1980 -1986, 90 months in 1987 -1999 and not reached in 2000 -2009. Th e 5-and 8-year RS rates were respectively 43.7% [30.9 -61.9]

Acute transformation

Data on transformation to acute leukemia were available for 424 patients. Th e rate of acute transformation fell from 51.8% among patients diagnosed in 1980 -1986 to 33.5% in 1987 -1999 and only 7.4% in 2000 -2009 ( p Ͻ 0.0001). Acute transformation was less frequent in women than in men (19.1% vs. 28.4%, p ϭ 0.028), and more frequent in patients with Ph Ϫ CML than in the rest of the cohort (53.3% vs. 23.2%, p ϭ 0.0076). Five-and 8-year RS rates among patients with Ph ϩ CML aged 45 -64 years were respectively 48.6% [35.0 -67.6] and 38.4% [25.5 -57.6] for those diagnosed in 1980 -1986, 62.4% [54.7 -71.1] and 50. 4% [42.6 -59.7] in 1987 -1999 and 92.0% [88.2 -95.9] and 89.7% [84.7 -94.9] in 2000 -2009. Figure 2 shows RS curves for patients with Ph ϩ CML.

Likewise, the 5-and 8-year RS rates among patients with Ph ϩ CML aged 65 years or more improved from respectively 26.7% [8.3 -86.0] and 1.4% [0.1 -17.6] for patients diagnosed

Discussion

With 781 cases, this is one of largest French multicenter cohorts of patients with CML. Our analysis of data from various geographic and socioeconomic areas provides original information on patients with CML in France.

Median age at diagnosis of Ph ϩ CML was 55.8 years, and 33.6% of patients were aged 65 years or more. Th e ESR was 0.97 [0.88 -1.05] and the WSR was 0.78 [0.71 -0.85]. Th ese results are similar to those reported in northern France (Nord-Pas de Calais), where median age was 56 years, 32% of patients were 65 years or older and the ESR was 1.3 for men and 0.9 for women [21]. Likewise, a population-based study in south-east England conducted in 1999 -2000 showed an ESR of 1.09 and a WSR of 0.83 [22].

Osca-Gelis et al . published similar incidence rates, with an ESR of 0.96 in the province of Girona, Spain, between 1994 and[START_REF] Brenner | [END_REF], and a median age of 64 years for men and 60 years for women [23]. Th e Surveillance of Rare Cancers in Europe (RARECARE) project showed a standardized incidence rate of 1.2 cases (standard error 0.01) per 100 000 person-years in an analysis of 64 European population-based cancer registries of patients diagnosed from 1995 to 2002 [24]. Th e C ô te-d ' Or specialized registry of hematologic malignancies, which was included in our study, has published a CML WSR of 0.9 (1.1 for men and 0.7 for women) [START_REF] Maynadi É | Twenty-fi ve [years of epidemiological recording on myeloid malignancies: data from the specialized registry of hematologic malignancies of C ô te d ' Or (Burgundy, France)[END_REF]. Crude and standardized incidence rates based on published registry data are shown in Table IV. Recently, the HAEMACARE project, designed to improve the standardization and availability of population-based data on hematologic malignancies archived by EUROCARE cancer registries, published incidence rates of hematologic malignancies in Europe for the period 2000 -2002 [26], using data from 44 cancer registries operating in 20 countries. Th e C ô te-d ' Or registry was the only French registry specializing in hematologic malignancies to participate in this study. Cases of CML with ICD-O-3 codes 9863/3 and 9875/3 were included. With 2468 cases, the crude incidence rate was 1.10 [1.06 -1.15] per 100 000 person-years (1.23 [1.17 -1.30] for men and 0.98 [0.92 -1.04] for women), and the incidence of CML increased with age: the incidence was higher at age 75 years or more than between 65 and 74 years. Th ese rates are higher than in our cohort, in which the incidence among patients aged 75 years or more tended to decrease (2.58 vs. 2.24 per 100 000 person-years), except in the Doubs registry in which it increased. Th is result is difficult to explain, but might be due to under-diagnosis among older patients, particularly in rural France.

Th e RS rate improved between 1980 -1986 and 1987 -1999 when IFN α was the main therapy, and more signifi cantly after 2000, when TKIs showed effi cacy in the general population of patients with Ph ϩ CML. Estimates of net survival among patients with CML in France between 1989 and 2004, including patients included in the FRANCIM network of cancer registries, showed that the net 5-year age-standardized survival rate increased from 47% to 69% for patients diagnosed between 2001 and 2004, corresponding to the advent of TKIs [27]. Effi cacy was higher in patients younger than 75 years. In the IRIS study, which compared fi rst-line IM versus IFN α and cytarabine, the 5-year overall survival rate estimated for selected patients aged 18 -70 years in the chronic phase was 89% [86 -92] [28]. In this trial, a large proportion of patients in the IFN α and cytarabine arm crossed over to the IM arm. Th e M. D. Anderson Cancer Center CML database shows an 8-year survival rate of 87% since 2001 [29]. In our populationbased study, we found that RS improved in each age category, including Ն 65 years. Osca-Gelis et al . published a 5-year RS of 61 months for patients diagnosed in 1994 -2008 in Girona, Spain among a heterogeneous population: 43 patients of 99 received TKIs [30]. Th ey performed a multivariate analysis of relative excess risk (RER) of death adjusted for TKI treatment, gender and age. Th e RER was statistically signifi cantly inferior for patients treated with TKIs (RER ϭ 0.19 [0.06 -0.60], p Ͻ 0.001). In our cohort, long-term survivors (4% of patients diagnosed in 1980 -1986, 29% of patients diagnosed in 1987 -1999) received TKIs after 2000. We chose not to compare patients according to TKI treatment, because of a survival population-based study conducted before the TKI era showed signifi cantly higher standardized incidence ratios (SIRs) for stomach cancer (SIR ϭ 2.76, 95% CI: 1.33 -5.08), urogenital cancers (SIR ϭ 1.61, 95% CI: 1.15 -2.21) and skin cancer (SIR ϭ 5.36, in patients with CML compared to the general population [39].

Our survival results are coherent with trends in CML mortality reported after IM introduction in Japan and the USA [40]. In our cohort, patients diagnosed as 9876/3 (Ph Ϫ CML) had a higher mean age than the overall cohort (71.8 vs. 56.7 years, p Ͻ 0.0001), and also experienced more acute transformations and a lower survival rate. Th ese patients, formerly classifi ed as having atypical CML (aCML), were reclassifi ed as having unclassifi able myelodysplastic/myeloproliferative syndromes (MDS/MPS) in the 2008 World Health Organization (WHO) " classifi cation of tumors of hematopoietic and lymphoid tissues " [START_REF]World Health Organization (WHO) classifi cation of myeloid neoplasms and acute leukemia: rationale and important changes[END_REF]. aCML is characterized by splenomegaly and neutrophilic leukocytosis associated with dysgranulopoiesis and circulating immature granulocytes. In a retrospective study from the M. D. Anderson Cancer Center [42], the median overall survival time of these patients was 24 months. In our cohort it was less 20 months, and did not diff er across the three study periods. Except for BMT, there is no eff ective therapeutic option for these patients. Among patients diagnosed as 9876/3 in 2000 -2009, 29.4% received a TKI with no improvement in survival, confi rming the heterogeneity of the pathological mechanism. Code 9876/3 should be removed from future versions of the ICD-O.

Patients classifi ed as CML-NOS have no available cytogenetic or molecular analyses and probably form a heterogeneous group. Th is lack of information on BCR -ABL status may be related to the presence of adverse clinical prognostic factors, in elderly patients for example. Nevertheless, some of these patients benefi ted from TKI therapy, as the 8-year survival rate was higher among patients diagnosed in 2000 -2009 than in those diagnosed in 1987 -1999, in whom TKIs were used in respectively 93.3% and 29.4% of cases ( p ϭ 0.0003). Currently, the proportion of CML-NOS is rather low, probably due to better access to appropriate genetic and molecular analyses for diagnosis.

As cytogenetic aberrations at diagnosis [trisomy 8, second Ph chromosome and isochromosome (17)(q10)] may have an important impact on the prognosis of CML (the 5-year survival rate is only 53% despite treatment with IM [START_REF] Fabarius | Impact of additional [] cytogenetic aberrations at diagnosis on prognosis of CML: long-term observation of 1151 patients from the randomized CML study IV[END_REF]), exhaustive recording of cytogenetic aberrations in registries might prove useful. Th e complexity of classifi cation and recording of pertinent information in hematologic malignancies underlines the need for specialized registries. RS rates for patients diagnosed during the last decade suggests a margin of possible progression for observance improvement and for new drugs based on molecular targeting.

Conclusion

During the last 30 years, major therapeutic advances have led to a signifi cant improvement in the RS of patients with CML. For example, the 5-year RS rates among patients with Ph ϩ CML were 43.7% [30.9 -61.9] when diagnosed in 1980 -1986, 63.8% [56.9 -71.5] in 1987 -1999 and 88.7% bias for patients for patients diagnosed before 2000: indeed only long-term survivors could be treated with TKIs during follow-up.

Th e diff erence in survival between patients diagnosed in 1987 -1999 and 2000 -2009 was signifi cant for all age classes, contrary to the results of the Surveillance, Epidemiology and End Results Program of the United States National Cancer Institute [31], which showed a signifi cant improvement in the 5-and 10-year RS rates in all age categories except for 65 -74 and Ն 75 years. Five-year RS rates among patients with Ph ϩ CML are shown in Table V for diff erent populationbased studies and calendar periods.

In an Italian study, cumulative complete cytogenetic and molecular remission rates were the same in patients aged 65 years or more as in younger patients [32]. Most of the patients aged 65 years or more who died during follow-up were in complete hematologic remission, suggesting more severe treatment-related toxicity in this age range. A large Swedish population-based study [33] of RS among patients with CML according to the calendar period of diagnosis showed a signifi cant improvement in the estimated 5-year RS rate for all age categories from 1973 -1979 to 2001 -2008, but the increase between 1994 -2000 and 2001 -2008 among patients aged 79 or more was smaller than in other age categories, despite a large increase among patients aged 70 -79 years. As in an American study [START_REF] Wiggins | Age disparity in the [] dissemination of imatinib for treating chronic myeloid leukemia[END_REF], we found disparities in access to TKIs between patients aged 75 years or more and younger patients ( p ϭ 0.0008). It is possible that prescribing physicians believe that older patients are more likely to experience adverse eff ects of TKIs (fatigue, musculoskeletal disorders), but this is not supported by published data. On the contrary, younger patients and women may be more susceptible to these adverse eff ects [35].

CML patient survival in the TKI era remains poorer than that of the general population of the same age and gender. Th is could be explained by the occurrence of blast crisis before an optimal therapeutic response is achieved, and also by the onset of mutations conferring resistance to IM [START_REF] Yeung | Th erapeutic targeting of BCR-ABL: [prognostic markers of response and resistance mechanism in chronic myeloid leukaemia[END_REF]. Second-generation TKIs may reduce mortality related to these factors. Epidemiological studies have shown that some second malignancies occur during TKI therapy, but no causative relationship has been found [37,38]. A

  in the Basse-Normandie registry and 2002 -2008 in the Gironde registry.

  in1980 -1986 to 67.5% [53.6 -84.9] and 40.1% [26.9 -59.9] in1987 -1999 and 82.9% [74.5 -92.3] and 73.7% [62.6 -86.8] in 2000 -2009. Among patients diagnosed in 2000 -2009, the 5-year RS rate of patients with CML-NOS ( n ϭ 106) was 63.6 [46.6 -86.8] and the 3-year RS rate of patients with Ph Ϫ CML ( n ϭ 33) was 12.0 [3.9 -36.8]. RS curves for the overall cohort are shown in Figure 2 according to the ICD-O-3 code and period of diagnosis.

  and 30.6%[19.2 -48.8] for patients diagnosed in1980 -1986, 63.8% [56.9 -71.5] and 47.0% [39.9 -55.5] in1987 -1999 and 88.7% [84.5 -93.0] and 73.3% [77.5 -89.4] in2000 -2009. Th e 8-year RS rate among patients with Ph ϩ CML aged less than 65 years increased across the three periods, from 38.4%[25.5 -57.6] in 1980 -1986 to 50.4% [42.6 -59.7] in 1987 -1999 and 89.7% [84.7 -94.9] in 2000 -2009. Th e 8-year RS rate among patients diagnosed with Ph ϩ CML at age 65 years or more also increased, from 0% in1980 -1986 to 40.1% [26.9 -59.9] in 1987 -1999 and 73.7% [62.6 -86.8] in 2000 -2009.

Figure 1 .

 1 Figure 1. (A) Relative survival of patients diagnosed in 1986 -2009 according to ICD-O-3 code. (B) Relative survival of patients diagnosed in 1980 -2009 according to ICD-O-3 code.

Figure 2 .

 2 Figure 2. (A) Relative survival of patients with Ph ϩ CML according to period of diagnosis. (B) Relative survival of patients with Ph ϩ CML aged 0 -64 years according to period of diagnosis.

Table I .

 I Characteristics of study cohort of 781 chronic myeloid leukemia patients.

			Period of diagnosis
		1980 -1986	1987 -1999	2000 -2009	Total
		( n ϭ 69)	( n ϭ 250)	( n ϭ 462)	( n ϭ 781)
	Age class at diagnosis, years, (n, %)
	0 -14	1 (1)	5 (2)	2 (0)	8 (1)
	15 -44	23 (33)	59 (24)	90 (19)	172 (22)
	45 -54	7 (10)	44 (18)	85 (18)	136 (18)
	55 -64	14 (20)	62 (25)	89 (19)	165 (21)
	65 -74	15 (22)	47 (19)	111 (24)	173 (22)
	75 -99	9 (13)	33 (13)	85 (18)	127 (16)
	Mean (SD)	54.3 (18.5)	55.8 (16.4)	58.7 (16.5)	57.4 (16.7)
	Sex (n, %)
	Men	43 (62)	139 (56)	258 (56)	440 (56)
	Women	26 (38)	111 (44)	204 (44)	341 (44)
	ICD-O-3 code (n, %)
	9875/3: Ph ϩ _CML	48 (70)	201 (80)	393 (85)	642(82)
	9863/3: CML_NOS	15 (22)	42 (17)	49 (11)	106 (14)
	9876/3: uMDS/MPS	6 (9)	7 (

Table II .

 II Age-standardized incidence for 100 000 person-years during1990 -2009.

		ESR [95% CI]	WSR [95% CI]
	Ph ϩ _CML
	Men	1.12 [0.99 -1.24]	1.09 [0.96 -1.22]
	Women	0.85 [0.74 -0.95]	0.74 [0.63 -0.84]
	All	0.97 [0.88 -1.05]	0.78 [0.71 -0.85]
	Ph ϩ _CML & CML_NOS
	Men	1.27 [1.14 -1.41]	1.25 [1.11 -1.39]
	Women	0.96 [0.86 -1.08]	0.83 [0.72 -0.93]
	All	1.10 [1.01 -1.19]	0.89 [0.81 -0.96]

ESR, European standardized ratio, WSR: World standardized ratio.

Table III .

 III Treatments received during follow-up according to period of diagnosis and ICD-O-3 codes ( n ϭ 679).

				Period of diagnosis
	Treatments received	1980 -1986	1987 -1999	2000 -2009
	Morphologic type	during follow-up	( n ϭ 57)	( n ϭ 199)	( n ϭ 423)	Total
	9875/3 Ph ϩ _CML, n (%)
		BMT	3 (6)	13 (7)	8 (2)	24 (4)
		Chemotherapy	25 (53)	13 (7)	4 (11)	42 (7)
		Hydroxyurea	38 (81)	140 (79)	62 (17)	240 (40)
		IFN α	3 (6)	113 (64)	33 (9)	149 (25)
		ASCT	2 (4)	31 (18)	3 (1)	36 (6)
		TKI	2 (4)	52 (29)	362 (96)	416 (69)
		Total	47	177	376	600
	9863/3 CML-NOS, n (%)
		BMT	2 (33)	5 (29)	2 (10)	9 (17)
		Chemotherapy	2 (33)	0	0	2 (4)
		Hydroxyurea	4 (66)	8 (47)	3 (7)	15 (28)
		IFN α	0	3 (18)	0	3 (6)
		ASCT	0	0	0	0
		TKI	0	7 (41)	28 (93)	35 (66)
		Total	6	17	30	53
	9876/3 atypical CML, n (%)
		BMT	0	0	0	0
		Chemotherapy	2 (50)	0	0	2 (8)
		Hydroxyurea	2 (50)	5 (100)	11 (65)	18 (70)
		IFN α	0	0	0	0
		ASCT	0	0	0	0
		TKI	0	0	5 (29)	5 (19)
		Total	4	5	17	26

Ph ϩ _CML: Philadelphia chromosome-positive chronic myeloid leukemia; CML_NOS: chronic myeloid leukemia not otherwise specifi ed; atypical CML: atypical chronic myeloid leukemia; BMT: bone marrow transplantation; IFN α : interferon alpha; ASCT: autologous stem-cell transplant; TKI: tyrosine kinase inhibitor.

Table V .

 V Relative 5-year suvival of Ph ϩ _CML patients according population-based studies and period. 84.5 -93.0] in2000 -2009. Th e 8-year RS rate of patients with Ph ϩ CML diagnosed in2000 -2009 was 83.3% [77.5 -89.4]. TKIs have thus dramatically improved the prognosis of all patients with Ph ϩ CML, not only those included in clinical trials. Disclosure forms provided by the authors are available with the full text of this article at www.informahealthcare.com/lal.

	Potential confl ict of interest:
	Article	n	Period

[

Ph ϩ _CML, Philadelphia chromosome-positive chronic myeloid leukemia; CML_NOS: chronic myeloid leukemia not otherwise specifi ed; CI, confi dence interval.