Andrew Yale

Saloni Dash

Ritik Dutta

Isabelle Guyon

Adrien Pavao

Kristin P Bennett

Generation and Evaluation of Privacy Preserving Synthetic Health Data

We develop metrics for measuring the quality of synthetic health data for both education and research. We use novel and existing metrics to capture a synthetic dataset's resemblance, privacy, utility and footprint. Using these metrics, we develop an end-to-end workflow based on our generative adversarial network (GAN) method, HealthGAN, that creates privacy preserving synthetic health data. Our workflow meets privacy specifications of our data partner:

(1) the HealthGAN is trained inside a secure environment; (2) the HealthGAN model is used outside of the secure environment by external users to generate synthetic data. This second step facilitates data handling for external users by avoiding de-identification, which may require special user training, be costly, or cause loss of data fidelity. This workflow is compared against five other baseline methods. While maintaining resemblance and utility comparable to other methods, HealthGAN provides the best privacy and footprint. We present two case studies in which our methodology was put to work in the classroom and research settings. We evaluate utility in the classroom through a data analysis challenge given to students and in research by replicating three different medical papers with synthetic data. Data, code, and the challenge that we organized for educational purposes are available.

Protection Regulation (GDPR)[3] in the European Union. While beneficial, these laws severely limit access to patient level medical data thus stagnating innovation and limiting educational and research opportunities. The process of

Introduction

Teaching data analysis and doing research with actual patient level medical data such as electronic healthcare records (EHR) are greatly restrained by laws protecting patients' privacy, such as the Health Insurance Portability and Accountability Act (HIPAA) [START_REF] States | The health insurance portability and accountability act[END_REF][START_REF] Annas | Hipaa regulations-a new era of medical-record privacy?[END_REF] in the United States and the General Data obfuscation of medical data is costly and time consuming with high penalties for accidental release. Research and education using EHR are highly skewed to a few shareable datasets such as MIMIC-III (Medical Information Mart for Intensive Care) [START_REF] Johnson | Mimic-III, a freely accessible critical care database[END_REF], which consists of de-identified ICU (intensive care unit) longitudinal data from 2001 to 2012 that adheres to the HIPAA restrictions and therefore can be shared. The only requirement is that the user completes a "Data or Specimens Only Research" certification. Datasets like MIMIC protect patients' privacy with classical anonymization techniques consisting of removing or regrouping quasi-identifiers in higher level categories (such as broad geographical areas) and removing or obfuscating sensitive information. Hence data utility can be severely altered. While the MIMIC data is extremely useful and has generated many research papers, it is limited to ICU data. It does not give access to the entire medical history of patients, hence limiting the type of analyses that can be carried out. This paper addresses this problem by proposing to use generators of synthetic data. The balance that needs to be hit in this project is to create synthetic data with enough quality to be useful for teaching purposes and ideally even for research, while preserving the privacy of the real data. In order to be useful in an education setting, synthetic data must preserve the relationships that exist in real patient-level data, so that assignments and projects using or discovering these relationships can be taught to students with the privacy preserving synthetic data. Other synthetic data generators like Synthea [START_REF] Walonoski | Synthea: An approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care record[END_REF] pursue a similar goal but are based on publicly available summary statistic data, and therefore do not provide the flexibility of creating generative models faithfully resembling real data.

Our proposed workflow (Figure 1) consists of training a generative model of synthetic data, using real data in a secure sand-boxed environment, exporting the model to the outside, and then synthesizing data. This procedure complies with our healthcare partners' regulatory requirements. We use novel and existing metrics to capture [START_REF] States | The health insurance portability and accountability act[END_REF] resemblance: data generated are sufficiently close to the real data and (2) privacy: data generated are significantly different from training samples. We also assess [START_REF] Union | General data protection regulation[END_REF] utility: data generated preserves some utility (for research and education purposes) and ( 4) footprint: trained model may not contain or require real data to generate synthetic data and should not be on the order of the real data in size. We develop a novel Wasserstein GAN-based method called HealthGAN and conduct a benchmark study on MIMIC data comparing it to five other approaches using a battery of metrics of utility, resemblance, and privacy.

We emphasize that privacy and resemblance are conflicting goals. By overfitting the data, the generative models can memorize the data thus potentially generating the actual real data points [START_REF] Yeom | Privacy risk in machine learning: Analyzing the connection to overfitting[END_REF][START_REF] Yeom | The unintended consequences of overfitting: Training data inference attacks[END_REF][START_REF] Khatri | Preventing overfitting in deep learning using differential privacy[END_REF][START_REF] Hardt | The reusable holdout: Preserving validity in adaptive data analysis[END_REF]. Models such as Parzen Windows can potentially accurately capture the data, but may reveal actual data points in the generated data or the modeling code making their footprint unacceptable.

In this paper, we extend the European Symposium on Artificial Neural Networks (ESANN) 2019 conference paper "Privacy Preserving Synthetic Health Data" [START_REF] Yale | Privacy preserving synthetic health data[END_REF] to include a more complete description and evaluation of HealthGAN and our proposed approaches for evaluating the quality of synthetic data. Section The data is gathered, processed, and used to train the generator model inside the secure environment. Then the model, which does not contain any real data, or require real data to generate synthetic data, is exported outside the secure environment. Finally, data is generated using the model and used for multiple types of applications.

2 discusses classical privacy preservation methods that are traditionally used to protect datasets such as MIMIC. These methods work to maintain as much of the real data as possible while still being private. In Section 3, we present new metrics for measuring the resemblance and privacy of synthetic data using nearest neighbors methods. Section 4 explores a previous GAN method for generating synthetic medical data called medGAN [START_REF] Choi | Generating multi-label discrete patient records using generative adversarial networks[END_REF] and discusses how the method fares on other datasets. Section 5 contains descriptions of the six different methods that will be evaluated including our proposed solution, HealthGAN. Evaluation of the methods using our new metrics is presented in Section 6. Further results in the form of case studies using the HealthGAN for both education and research applications are in section 7. Finally, in Section 8, we provide conclusions and discuss future possibilities for this work.

Related Work

Classical privacy preservation techniques are focused on creating a new version of the real data set that ensures that no record in the data can exclusively identify an individual ("Unique Identity Disclosure"). These methods must also ensure that an attacker having prior knowledge about an individual is not able to obtain sensitive information from the disclosed attributes ("Sensitive Attribute Disclosure"). This can be done using processes such as generalization, anatomization, and perturbation [START_REF] Mendes | Privacy-preserving data mining: methods, metrics, and applications[END_REF]. The effectiveness of these methods is proved theoretically and ensures a specific level of privacy preservation.

In general, privacy models can be categorized as follows [START_REF] Mendes | Privacy-preserving data mining: methods, metrics, and applications[END_REF]:

• Generalization : Replacement of a value for a more general one (parent).

For instance, a zip code of 12345 can be replaced by 123** or a Profession of Actor can be replaced by Artist.

• Suppression : Removal of some attribute values to prevent information disclosure (identifiers).

• Anatomization : De-associates quasi-identifiers (QIDs) and sensitive attributes in two separate tables.

• Perturbation : Adding noise to real data to create new data that preserves summary statistical information.

Our approach deviates from these classical methods by focusing on methods that create new data points that approximately mimic the real data rather than altering real data points. We also quantify the loss in privacy rather than relying on theoretical guarantees. Since our generative methods attempt to create data that look exactly like the real data, they minimize the utility loss, as compared to techniques such as generalization which would remove details from the data. Utility loss measures how well the synthetic data performs the task for which the real data was used. For example if a classification task is being performed the loss measures how close performance metrics for the classification task is between the synthetic data and the real data. Empirical evaluation is important to our research because privacy and utility loss can be concretely measured on the synthetic data with respect to organizations' sensitive data before releasing it to the public. As a baseline, the ubiquitous differential privacy data obfuscation method [START_REF] Dwork | Differential privacy, Automata, Languages and Program-ming 4052[END_REF][START_REF] Dwork | Differential privacy: A survey of results[END_REF], an instance of the perturbation model of privacy, will be used to compare how well these data altering methods perform against generative methods.

In addition to classical techniques, there are newer methods that use GANs to create synthetic medical data. MedGAN, developed a GAN on MIMIC data as discussed in "Generating multi-label discrete patient records using generative adversarial networks" [START_REF] Choi | Generating multi-label discrete patient records using generative adversarial networks[END_REF]. The medGAN implementation was tested with diagnosis data derived from MIMIC discrete patient records using generative adversarial networks. Each row in the data represents a patient and contains binary columns or count columns for each diagnosis, although the binary data was the focus of most of the work. In the binary data, for every instance where a patient had ever been diagnosed with a condition in the MIMIC data, the diagnosis was set to one, and the rest are zero. The many real-valued features in MIMIC were not synthesized using medGAN.

The medGAN version of the GAN architecture was also slightly different from the original GAN implementation in several ways, in an attempt to fix problems and make the GAN work for discrete data. In their version, they added an autoencoder to the middle of the network to help the generator create more realistic samples. This autoencoder was trained from the real data so that the decoder part could be used on the data created from the generator. They also used minibatch averaging which helped enforce the column mean of the real data on the synthetic data. Finally, in the initial GAN structure discrete values could not be generated. This implementation took the continuous values from zero to one created by the GAN and rounded them based on the cutoff of 0.5 to create binary data.

To measure the privacy preserved by the data generated by medGAN, they sampled 1% of the training set R and a subset s of known attributes from each record. Then using the synthetic data, they inferred the missing attributes using k-nearest neighbors. They tried this for multiple values of k, s, and the total number of synthetic samples. Throughout all of their experiments they consistently showed that the attacker would have poor sensitivity and precision in this method when inferring the missing attributes. In this case the sensitivity is the percentage of positive diagnoses inferred out of the total number of positive diagnoses in the compromised record. Precision is measure the percentage of inferred diagnoses that were in fact true in the compromised record. This notion of nearest neighbors being used in privacy metrics led us to the metrics we have created for testing privacy and resemblance in our synthetic data.

Metrics of Resemblance and Privacy

We introduce metrics of resemblance and privacy. Consider two data distributions P T and P S , where T and S designate a Target and a Source domain respectively, for instance True (real) and Synthetic data. We draw empirical samples

S T = {(x 1 T , y 1 T ), • • • (x n T , y n T )} from P T and S S = {(x 1 S , y 1 S ), • • • (x n S , y n S )} from P S .
We assume that in all cases x variables belong to a common metric space e.g., IR d and y is a categorical or continuous variable (i.e., defining classification or regression tasks). We also assume that all variables have been normalized, e.g. by subtracting the minimum and dividing by the range of the data.

The proposed metrics are based on nearest neighbors. We call d T S (i) = min j x i T -x j S the distance (Euclidean or otherwise) between x i T ∈ S T , a point in the sample from the Target distribution, and its nearest neighbor in S S , the sample from the Source distribution. Therefore d ST (i) is the opposite measuring the distance between x i S ∈ S S , a point in the sample from the Source distribution, and its nearest neighbor in S T , the sample from the Target distribution. Therefore d ST (i) is the opposite me measuring We call d T T (i) = min j,j =i ||x i T -x j T || the "leave-one-out" distance to the nearest neighbor in a sample of size (n -1) drawn from the same distribution. We define AA T S , the nearest neighbor Adversarial Accuracy between T and S as:

AA T S = 1 2 1 n n i=1 1 (d T S (i) > d T T (i)) + 1 n n i=1 1 (d ST (i) > d SS (i)) (1)
where the indicator function 1(.) takes value one if its argument is true and zero otherwise. If we think of T as the true data and S as the synthetic data, by this definition, a real point i in T , which is sufficiently far away from any point in S, is a "true positive" point with respect to privacy. Similarly, a simulated point j in S must be sufficiently far from any point in T in order to be a "true negative" point. We can think of AA T S as the performance of an adversarial classifier that distinguishes between real versus synthetic data. The AA definition is a "balanced accuracy", which averages the true positive rate and the true negative rate. If datasets T and S are indistinguishable, then AA T S should be 0.5. We use various datasets, all of size n, to define resemblance and privacy: R tr is the real data training set used to train the generator; R te is the real data test set, drawn independently from the same distribution as R tr ; A 1 and A 2 are any two artificial datasets generated by the generator network, G. We denote by E(.) the mean value over all A i and define three kinds of losses:

TrResemblLoss (T rain Adversarial Acc.) = E[AA RtrA1 ] TeResemblLoss (T est Adversarial Acc.) = E[AA RteA2 ] (2) PrivacyLoss (T est AA -T rain AA) = E[AA RteA2 -AA RtrA1 ]
Intuitively, if the generator G does a good job, then the adversarial classifier cannot distinguish between generated data and real data, train and test adversarial accuracy should both be 0.5, and the privacy loss will be zero. If G does a poor job and underfits, it will serve generated data that does not resemble real data. Thus the adversarial classifier will have no problem classifying real vs. artificial so the train and test adversarial accuracy will both be high (>0.5) and similar, and the privacy loss will also be near zero. In this last case, privacy is preserved but the utility of the data may be low. If the generator overfits the training data, the Train AA will be near zero (good training resemblance), but the Test AA will be around 0.5 (poor test resemblance). Thus the privacy loss will be high (near 0.5). Figure 2 provides a two-dimensional synthetic example of these three cases in which blue is the real data and red is the artificial data. We generate train and test data (n = 50 from a semicircle plus Gaussian noise then standardized). We generate two artificial datasets A 1 and A 2 of the same size with the Parzen Windows density estimator (this method approximates a density by a mixture of local continuous density functions centered at data points and having a certain bandwidth size), using a Gaussian kernel of varying bandwidth to create three models, from left to right: (1) underfitted, (2) properly fitted, and (3) overfitted. The Train and Test adversarial accuracy (AA) is shown for each case. For the same example, Figure 3 provides curves representing Train AA, Test AA, and Privacy Loss for decreasing Parzen Windows kernel widths.

HealthGAN Formulation

Based on the improvements the medGAN makes on the initial GAN, and the fact that it actually uses medical data, we used this as a first attempt at a GAN method. Through evaluating the data produced from medGAN, we found that there are some major issues with the implementation. First is the fact that it is only built for binary data. Second, our assessment reveals it has some flaws in the resemblance of the generated data to the real data.

The problem with the resemblance of the data from medGAN is that the architecture has been optimized to preserve the probability of each diagnosis occurring. Stated another way, it is optimizing for matching each column's mean to the real data. This is shown in Figure 4 where the real data column mean is on the X-axis and the synthetic data column mean is on the Y-axis. Even though most of the diagnoses are very rare the synthetic data matches the univariate probabilities closely. While this metric seems to show that the datasets have a very close resemblance, looking at the other dimension shows the true story. Another way that the synthetic data should resemble the real data is in the row sum. The row sum can be interpreted as the total number of unique diagnoses that a patient has. Patients typically have a small number of diagnoses. Comparing the overall distribution of row sums reveals that medGAN overestimates the number of diagnoses for some patients. In Figure 5, the row sums of medGAN and the real data are compared. The X-axis shows different values of row sums, or total unique diagnoses. The Y-axis shows the counts of the occurrences of these bins on a log scale in order to show the tail of the synthetic data. This plot can be interpreted to say that the generator is creating synthetic patients with four times as many diagnoses as the patient with the highest number of diagnoses in the real data. While the number of patients generated in that tail might not be high, this exposes another issue which is that these synthetic patients with over 100 diagnoses become a catch-all for the rare diagnoses. They become patients that are only placed there to make the column means balance. This issue was indirectly referenced in the paper when they showed their synthetic patients and real patients to a medical doctor and ask the medical doctor to distinguish between the two. In some of the instances where the doctor was able to distinguish between the two, the reason was the patient was marked down for a female specific and male specific diagnosis at the same time. MedGAN introduces spurious comorbidities that are not in the real dataset, thus limiting the resemblance and utility of the synthetic data. Even though medGAN was created to solve a similar problem to ours and included some good results, it was ultimately ruled out as a basis for the generative method because of problems with resemblance. First, the fact that it could only generate binary data was not flexible enough. Second, the issues with resemblance in the row sums indicated that unrealistic data was being generated. Finally, the model had been so specifically crafted for the binary diagnosis dataset, that it was not robust enough to work with many different datasets of different sizes and columns, which is the final goal. This led us to creating a new method we are calling HealthGAN.

The HealthGAN architecture and design took ideas from medGAN and combined them with the Wasserstein GAN gradient penalty (WGAN-GP) [START_REF] Arjovsky | [END_REF][START_REF] Gulrajani | Improved training of wasserstein gans[END_REF] to accommodate multiple data types. The WGAN-GP was used as a basis for the HealthGAN because it uses the earth mover's distance or Wasserstein distance versus the Kullback-Leibler (KL) divergence used in original GAN [START_REF] Goodfellow | Generative adversarial nets[END_REF] or the medGAN [START_REF] Choi | Generating multi-label discrete patient records using generative adversarial networks[END_REF]. That paired with a data transformation to accommodate mixed continuous and categorical data, such as MIMIC-III. EHR are always mix of continuous features (e.g., age, bmi, lab results) and categorical features (e.g., diagnosis codes and providers) and therefore this new GAN was built to accommodate multiple data types from the start. Furthermore, a patient has many EHR records through time. While this version of the HealthGAN does not accommodate time series data it is an improvement being attempted on other datasets [START_REF] Dash | Synthetic event time series health data generation[END_REF].

The architecture is similar to the original GAN with a generator network with three layers, and a discriminator network with four layers. Due to the large variance present in medical datasets a large batch size is used to ensure that outliers and rare values are captured in each batch and therefore learned by the generator. The batch size is determined based on the size of the input data to maximize the size. The dimensions of the generator network is also determined by the number of features in the dataset to scale in complexity according to the data it is modeling. HealthGAN represents an attractive black box method with a very compact footprint (parameters of the model) since the bottleneck in HealthGAN is constructed to prevent memorization.

Data Generation Methods

We performed a comparison of 6 data generative methods1 on the MIMIC-III mortality problem: (1) Gaussian Multivariate [START_REF] Duda | Pattern classification[END_REF], (2) HealthGAN, (3) Parzen Windows [START_REF] Parzen | On estimation of a probability density function and mode[END_REF], (4) Additive Noise Model (ANM) [START_REF] Hoyer | Nonlinear causal discovery with additive noise models[END_REF], (5) Differential Privacy preserving data obfuscation (DP) [START_REF] Dwork | Differential privacy, Automata, Languages and Program-ming 4052[END_REF], and [START_REF] Yeom | Privacy risk in machine learning: Analyzing the connection to overfitting[END_REF] Copy the real data (CP)2 . Our usage of HealthGAN and ANM in this context are novel, to the best of our knowledge.

Data Generation Methods

We describe the 6 data generation methods in more detail.

• Gaussian Multivariate: This method simply consists of modeling data by a multivariate Gaussian distribution whose parameters are then found using Maximum Likelihood Estimation (MLE), i.e., using the mean and covariance matrix of the training data. This method fulfills our footprint specifications because the model is much smaller in size than the real data and does not directly represent any sample (provided that the means are not actual data points).

• HealthGAN: Described in the preceding section, a GAN based method for generating mixed continuous and categorical data.

• Additive Noise Model: Inspired by methods used for imputation of missing data, a suitable predictor (here we use Random Forests) is trained to predict one feature of a given sample, given all the other features.

Predicting each feature for each sample in this way gives a dataset A 0 consisting entirely of predicted values, which can then be sampled from to generate synthetic datasets. Noise is drawn from a Gaussian distribution with zero mean and variance equal to the mean-square-error of the fit and is added to each predicted value to increase the diversity of the data produced. The model itself has a small footprint, but data generation requires storing A 0 and therefore exporting data, which rules out this model for our application purposes. We keep it as a baseline method.

• Parzen Windows: Parzen Windows density estimation approximates a density by a mixture of local continuous density functions K, called kernels, centered at data points and with bandwidth equal to h:

fh (x) = 1 Z n i=1 K( x-xi h ) with x 1 , .
.., x i the data points and Z a proper scaling factor. Generating data boils down to picking a data sample at random, then drawing a sample at random around the sample by applying the kernel density function. This method has an unacceptable footprint since each data point is represented in the Parzen Windows function.

• Copy Real Data: We exactly duplicate the data; more precisely we use the train set instead of synthetic data. Resemblance is high but the model maximally overfits, thus privacy is at a minimum. The footprint duplicates the data and thus is of course unacceptable.

• Privacy-preserving Data Obfuscation: Differential Privacy is a widely accepted privacy requirement for data publishing [START_REF] Dwork | Differential privacy, Automata, Languages and Program-ming 4052[END_REF]. We generated a , δ Differentially Private version of the MIMIC-III dataset by creating generalization hierarchies for the seven quasi-identifier attributes3 using ARX, an open source anonymization tool for medical data [START_REF] Prasser | A tool for optimizing de-identified health data for use in statistical classification[END_REF] based on the SafePub Algorithm [START_REF] Raffael | Safepub: A truthful data anonymization algorithm with strong privacy guarantees[END_REF]. The footprint of this method is unacceptable because it requires export of most of the real data and privacy is limited to quasi-identifiable fields.

Data Transformation

Data transformation was essential for the success of many of the methods including HealthGAN. Recall MIMIC-III contains a mix of categorical and discrete variables. We adapted data transformation strategies used in the Synthetic Data Vault (SDV) [START_REF] Patki | The synthetic data vault[END_REF]. We map all features to a range between zero and one, synthesize the data, and finally transform the synthetic data back to its original form, using the mapping from the real data. Numeric variables are scaled by subtracting the min and dividing by (max-min). For each categorical variable, we first sort from most frequent to least frequent. Then we split the interval from zero to one into sections based on the cumulative probability for each category. Finally, lining up each category with its section on the interval from zero to one, we take a sample from that section using a truncated Gaussian distribution. The reverse transformation maps the synthetic data to the original categories. This transformation is part of all of the methods discussed in the previous section that cannot accommodate categories by default.

Experimental Results

We evaluated the synthetic generation data on the MIMIC-III dataset which contains records for about 40,000 intensive care unit (ICU) patients and indicates whether they died in the ICU. It includes demographics, vital signs, diagnoses, and procedures performed. The dataset we used is tabular data related to 48-hour mortality in the ICU. The data has 342 features, the output column of mortality and approximately 27,000 observations. We generated synthetic data and then evaluated the different approaches using visualization techniques and the proposed metrics.

Principal Component Analysis Plots

We found principal component analysis (PCA) plots created using projection of the real train data to be very useful for getting a quick understanding of resemblance of the real test data (black dots) to the generated synthetic data (red dots). Here we can see that data generated by the Gaussian Multivariate and Parzen Windows methods span a larger space than the real data, which aligns with the fact that those methods create differences in the data in both directions uniformly. The PCA of the Differential Privacy data obfuscation method data spans a smaller space, which represents fact that the quasi-identifiers are changed enough to not reveal outlier data.

Both the real and synthetic data distributions of HealthGAN and the ANM have high resemblance, which aligns with their greater ability to define relationships that exist in the real data and apply that to their generated synthetic data.

Other dimensionality reduction methods were also explored, but the best tool for visual assessing the synthetic data was determined to be the PCA. For example T-distributed Stochastic Neighbor Embedding (t-SNE) produced plots that were more uniform looking across the methods and therefore didn't show obvious differences between the methods.

Adversarial Accuracy Results

We compared the adversarial accuracy (Equation ( 1)) in terms of TrainRe-semblanceLoss, TestResemblanceLoss, and PrivacyLoss = TestResemblanceLoss -TrainResemblanceLoss (Equation ( 2)). As shown in Table 1, Gaussian Multivariate preserves privacy, but suffers from high testing adversarial accuracy (0.55). A well fitted Parzen Window (optimized kernel width) and HealthGAN both perform well with respect to resemblance and privacy. But the footprint of Parzen Windows rules it out for this purpose since the real data . While the DP method obscures the quasi-identifiers, it leaves open the rest of the data and therefore scores very poorly on the training data. For the ANM, we used few and deep trees to illustrate a case of overfitting: indeed the ANM overfits the data badly, completely exposing the real data. It is possible to tune the ANM hyperparameters to prevent overfitting, however, its footprint would still make it unacceptable for our applications. In both methods, the privacy of the data is at its worst. In Table 1 the colors for adversarial accuracy indicate how far the value is from optimal. In the case of adversarial accuracy the optimal value is 0.50 and 0.50±.01 is in the blue or excellent range. Yellow indicates a good value and specifically is a value of 0.50±.03. Anything outside that range is orange or poor. For privacy loss the optimal value is zero and anything less than or equal to .01 is excellent, less than or equal to 0.03 is good, and above that is poor.

We also assessed utility of the data generated by the methods by using the synthetic data to train a classifier to predict patient mortality, then testing the classifier on the real test dataset. A logistic regression classifier was selected and by comparing the area under the curve on the test data, we can see that the DP and CP methods have the best performance, but also have unacceptable privacy scores. The next best methods are Parzen Windows, ANM, and HealthGAN. The Additive Noise Model predictably performs poorly on privacy because it is overfitting, but cannot be used as a final method in any form due to the model footprint requiring real data. The Parzen Windows and HealthGAN perform well for both privacy and area under the curve, but the Parzen Windows method does not fulfill the model privacy requirements. Finally, Gaussian Multivariate, As discussed throughout the results the footprint is a major factor in selecting the final model as well. The footprint measures whether the information needed to generate synthetic data, specifically the model and any inputs, contains real data or is on the order of the size of the real data. This is critical to being able to export the model from the secure environment and keep the real data secure. The Gaussian Multivariate and HealthGAN are the only methods that satisfy this condition.

Education and Research Case Studies

After ensuring that the synthetic data generated by HealthGAN is satisfying the privacy and resemblance metrics in the previous section, the next goal was to ensure the synthetic data was of high enough utility to be used in the education and research settings. To do this we conducted both education and research case studies.

The education studies involve setting up a challenge for students with synthetic data in which they must create a classifier and are scored based on the performance of the classifier. This synthetic data challenge was designed for use in undergraduate health informatics curricula at Rensselaer Polytechinic Institute developed with support from the United Health Foundation. This curricula is designed to rapidly recruit and prepare undergraduate students to be data scientists in healthcare using early data analytics courses and experiential research projects centered on real-world health challenges.

In the research case study, we attempt to replicate three different published medical papers that use MIMIC data with synthetic versions of the datasets generated using HealthGAN. The data replicated from the research papers can also be used for educational purposes as students can either replicate the papers themselves with the synthetic data or be given the synthetic data and attempt to find their own method of accomplishing the goals set forth in the papers. All of these results show how well the synthetic data performs at modeling specific relationships in datasets.

Education Case Study

To assess educational utility, we used synthetic data in classroom challenges. "To be, or not to be?" is a mortality prediction challenge hosted on CodaLab 4 . used in two courses The mortality prediction challenge has been used in two undergraduate courses at Rensselaer: "Introduction to Data Mathematics" and Health Analytics Challenge Lab" at Rensselaer Polytechnic Institute. This challenge and other subsequent ones support the curriculum goal of exposing students in early data science courses to compelling healthcare problems in order to attract them to careers in health informatics.

We give students 80,000 synthetic records generated from the MIMIC mortality dataset used for testing in the previous section. This data includes demographic data, vital signs, diagnoses, and mortality. Using this dataset the students create models in R [START_REF] Ihaka | R: a language for data analysis and graphics[END_REF] on the training data and then predict mortality for 20,000 synthetic test records. Their predictions are uploaded to the server and evaluated. They never get to see the mortality values for the test dataset. The models will then be evaluated with the balanced accuracy metric and ranked. A major advantage of using the synthetic data challenge was that the students did not have to undergo the training required to use the real MIMIC data.

The best students achieved values of 0.77 and 0.76 balanced accuracy. When the model is trained on real data and evaluated on the same test set the result is 0.80, which shows how high the level of utility is on the generated data. This case study demonstrates the utility of synthetic data for health informatics education. Students get the experience of working with patient level data while preserving privacy and avoiding additional precautions needed to work with actual patient data. Potentially, synthetic generation could make data from many published papers and electronic health care records available to enhance many types of educational programs including medical and public health programs.

Research Case Study

To assess utility of synthetic data in a research setting, we examined the effectiveness of synthetic data on three of the numerous medical studies published using the MIMIC data. We attempted to replicate the analysis in the three papers using synthetic versions of the MIMIC data. The results were also replicated with real MIMIC data to enable a comparison of the results using real and synthetic data. The papers were selected to utilize a wide variety of analysis methods.

Impact of Race on ICU Mortality Study

The first paper used MIMIC data to try to evaluate the impact of race on 30-day mortality [START_REF] Mundkur | Use of electronic health record data to evaluate the impact of race on 30-day mortality in patients admitted to the intensive care unit[END_REF]. The authors took different demographic variables and comorbidities and found the odds ratios for each of them to find out whether race specifically impacted the 30-day mortality of the patient.

The original paper utilized the MIMIC-II dataset, [START_REF] Saeed | Multiparameter intelligent monitoring in intensive care ii (mimic-ii): a public-access intensive care unit database[END_REF] while we used MIMIC-III. The main difference between these datasets is that MIMIC-III includes years 2001 to 2012 in the data whereas MIMIC-II is only 2001 to 2007. In addition to those extra years of data, MIMIC-III also enriches the dataset with more variables to be looked at and fixing up some of the older values. Therefore the MIMIC-III dataset is just a better version of the MIMIC-II dataset and a better basis for doing this analysis and creating synthetic data. We also did cleaning with the race (called ethnicity in MIMIC), insurance, and age variables to give them better categories. For race there is a long tail of races which are put into "Other" and another category "Unknown" for missing values. Insurance had several categories that are separated into "Private", "Medicaid", and "Medicare". Age is put into buckets to more easily determine the effect of an age range. Finally, 30-day mortality isn't tracked that accurately in the MIMIC data, so we will use mortality in the ICU, which is another example in the paper.

The authors of the paper used Kruskal-Wallis, Wilcoxin two-sample, Pearson's chi-squared, and Fisher's exact tests [START_REF] Saeed | Multiparameter intelligent monitoring in intensive care ii (mimic-ii): a public-access intensive care unit database[END_REF] to account for collinearity between race and covariates. Two multivariate logistic regression models are trained in order to account for confounders. The first regression does not include the "Unknown" race group and the second does. The second model is the focus of the results below.

The model creates odds ratios for each of the variables. In Table 2 we can see a comparison of the odds ratios computed from the real data versus the synthetic data. In this case we can see that 15 out of the 17 values have overlapping confidence intervals, thus indicating the results for the synthetic and real data were not significantly different from each other on the 15 variables. These results are promising because they seem to replicate almost all of the relationships that we see in the real data. This satisfies another targeted utility example and shows that students can work with and learn the relationships between different features specifically found in health data. For research purposes it is harder to determine if quality of the synthetic data was high enough. This is because the results from the paper are not able to be exactly replicated in the real data and therefore also do not appear in the synthetic data. In the paper they found that the "Black" and "Asian" categories for race had odds ratios significantly less than one, meaning they did not overlap with one. Using the newer MIMIC-III dataset, this result does not appear in the data. This could be caused by many things including the specific definition of the features, model, or the fact that there are more records in MIMIC-III. The result is that our model shows inconclusive results about the effects of race on mortality in the real and synthetic data.

Mortality of Elderly Patients in the ICU study

The next paper attempted to analyze the characteristics and mortality of elderly patients in the ICU [START_REF] Fuchs | Icu admission characteristics and mortality rates among elderly and very elderly patients[END_REF]. This study seeks to evaluate the association between the demographic and clinical characteristics of patients over the age of 65 and their 28-day and one-year mortality. Logistic regression is used to analyze 28-day mortality and a Cox regression model is used to analyze one-year mortality.

The data used in this paper are age, gender, sequential organ failure assessment (SOFA) [START_REF] Singer | The third international consensus definitions for sepsis and septic shock (sepsis-3)[END_REF], do not resuscitate (DNR), and the Elixhauser Comorbidity Index (ELIX) [START_REF] Elixhauser | Comorbidity measures for use with administrative data[END_REF]. They only include patients over 65 and categorize the patients into age 65-74, 75-84, and 84+. In the MIMIC-III data we found 15,771 patients in the first category, 5,664 in the second, and 3,517 in the third.

Using the same significant variables as the paper we created a logistic regression model with both the real and synthetic data to see how they compared.

The odds ratios confidence intervals for those models are in Table 3. The results between the synthetic and real data are similar but in some cases don't overlap. In the case of SOFA and DNR, the odds ratios don't overlap but have similar effects in that both show an odds ratio greater than one. The other variable that doesn't overlap is the 75-84 age range which in the synthetic data doesn't seem to have an effect as the confidence interval straddles one, but in the real data the value is decidedly greater than one. The discrepancies between the real and synthetic data highlight the challenges of synthesizing imbalanced data. The results suggests that oversampling rare classes during training of HealthGAN may lead to improved results.

Variable

Real These results mostly match the results in the paper. In the paper all of the selected factors besides being male have a greater than one odds ratio and are significant as we found for the real data.

For the one-year mortality prediction a Cox regression model was used. The same variables were used as in the logistic regression model. In Table 4 Using the real data we were able to replicate most of the results of the original paper; the synthetic data was not able to at all. This again falls into an imbalanced class issue. The chance of a 28-day mortality was about 14.7% in the real data and was replicated in the synthetic data at 12.4%. On the other hand, one-year mortality occurred 1.1% of the time and therefore was much harder to replicate in the synthetic data. The fact that the one-year percentage is so much lower than the 28-day value doesn't make sense, and stems from the fact that the ability for the ICU to track patients for a year after they leave the ICU is poor.

Mortality in Acute Kidney Injury

In this final paper, the authors predict mortality of patients with Acute Kidney Injury (AKI) [START_REF] Celi | A clinical database-driven approach to decision support: Predicting mortality among patients with acute kidney injury[END_REF]. To ensure the patients have an AKI diagnosis we look at patients with a longer than 72 hours stay in the ICU. To predict mortality we use the Simplified Acute Physiology Score (SAPS) [START_REF] Gall | A new simplified acute physiology score (saps ii) based on a european/north american multicenter study[END_REF]. The SAPS score looks at systolic blood pressure, heart rate, temperature, urine output, blood urea nitrogen, white blood count, serum potassium, serum sodium, serum bicarbonate, serum bilirubin, and age. Just like SOFA or comorbidity indexes each one of these variables results in points based on its value. By selecting patients who are in the ICU for at least three days we can get a score for each one of these categories for each day. Therefore we have 33 variables for eleven categories across three days.

With the variables selected we then ran a logistic regression to predict mortality for these patients using the SAPS variables. In Table 5 we can see the resulting coefficient and significance level of each variable using the real data. Based on these results we can specifically select the significant variables and look at those. These variables are urine, age, and heart rate from day one, ACI and sodium from day two, and WBC, ACI, BUN, bilirubin, and sodium from day three.

Using these variables, we then construct five different logistic regression models. The first three use just day one, two, and three variables respectively. With the real data, as the days go on we have a better chance of predicting mortality, based on the area under the curve (AUC) measure. The fourth model uses all of the variables, and achieves a slightly better value AUC than just the day three variables. Finally, the fifth model just uses the significant variables from the previous figure to predict mortality. This model performs roughly equivalently to the all-variable model. These results are seen in Table 6. These same five models were constructed for the synthetic data. The results were almost identical except for the day one and day three AUC values being slightly worse on the synthetic data than the real data. Given such a close result across five different models, we can say that the synthetic data achieves the desired level of targeted utility for this task.

Real Data Synthetic

In all three papers, the synthetic data sets exhibited high utility for education usage. In many cases, the conclusions found were not qualitatively different from those obtained from the real data. HealthGAN synthetic data can be published in lieu of the real data for cases where privacy does not permit publication of the real data. New algorithms and methods could be developed and compared on the public synthetic data. Ideally, the results of these methods should be verified on the real data in a secure environment that maintains privacy.

Conclusions and Future Work

Although GANs have increased in popularity, their effectiveness in the health domain was not clear. Through replicating the results from with the medGAN [START_REF] Choi | Generating multi-label discrete patient records using generative adversarial networks[END_REF] architecture, the potentials and limitations of these methods became apparent. HealthGAN and the proposed evaluation metrics provide both an improved algorithm and better metrics for evaluating the quality of synthetic health data in the future. The workflow that we presented for generating synthetic data from real data and exporting a model only outside a data-secure environment has become operational with the introduction of HealthGAN. Generated data is competitive in resemblance with other methods, while meeting the requirements of privacy preservation and small model footprint. Our methodology includes novel metrics, based on nearest neighbor adversarial accuracy, for defining the resemblance and privacy of synthetic data generated from real data. We evaluated these metrics as well as utility and footprint on six methods using the MIMIC-III 20 mortality data. HealthGAN was the only effective method that maintained privacy and that allowed model export. HealthGAN was then tested further in case studies in education and research. In an education case study, the synthetic data showed high levels of utility as students attempted a mortality prediction challenge. In research, three different papers using MIMIC data were replicated with synthetic data generated from HealthGAN. This workflow can be used to address the vital need to create datasets for health education and research without undergoing obfuscation, which can be both costly and risky and lose information. In addition, our proposed metrics will continue to be useful to monitor progress in synthetic data generation. All data, code, and the mortality prediction challenge that we organized are publicly available 5 .

Future work on this research is to use the HealthGAN method on medical datasets that extend beyond the ICU setting of MIMIC-III. This comparison will be done in the same style as the MIMIC-III paper replications, but with much more varied datasets. These synthetic datasets can then be used in curricula to teach students including creating challenges for them to solve health care problems on more diverse synthetic datasets. Beyond replicating papers, new datasets can be used to generate synthetic data and help create workflows for solving real research problems.

Figure 1 :

 1 Figure 1: Workflow used to generate synthetic data securely. The data is gathered, processed, and used to train the generator model inside the secure environment. Then the model, which does not contain any real data, or require real data to generate synthetic data, is exported outside the secure environment. Finally, data is generated using the model and used for multiple types of applications.

Figure 2 :

 2 Figure 2: Parzen Windows, toy example. Blue markers represent real 2-d data samples (Rtr and Rte) and red markers represent synthetic data generated with Parzen Windows (A 1 and A 2 ). Top row: Rtr and A 1 . Bottom row: Rte and A 2 . Form left to right: Large kernel ⇒ underfitting; optimized kernel ⇒ fitting right; and small kernel ⇒ overfitting.

Figure 3 :

 3 Figure 3: Parzen Windows, learning curves. The Train AA keeps decreasing, but not the test AA. The privacy is good when the difference Test AA -Train AA is small. The best compromise is attained around the point where the black curve crosses the dashed blue line at 0.5. The pink dashed line shows the difference (Test AA -Train AA).

Figure 4 :

 4 Figure 4: medGAN Dimension-wise Probability (Column Mean) Comparison. Each point represents a column mean in the real and synthetic data. A perfect match would be indicated by all the points lying on the line y = x.

Figure 5 :

 5 Figure 5: medGAN Row Sum Distribution Comparison. A perfect match would be indicated by matching histograms.

Figure 6 :

 6 Figure 6: Comparison of generative methods using PCA projection created using the real data. Blue is the real data and red is the synthetic data.

Table 1 :

 1 Comparison of models with respect to various metrics. Blue: Excellent; Yellow: Good; Orange: Poor. Our advocated method marked with (*) performs best. Train AA and Test AA measure resemblance loss. P rivacyLoss = T estAA -T rainAA. Utility measures test accuracy of predicting mortality. Footprint indicates whether we can export a small footprint model out of the secure area.performs the worst on utility, but still has good data and model privacy. The utility metric is important to consider, because it roughly captures usefulness of the synthetic data in the classroom setting. The colors in the table correspond to excellent being a value from 0.80 to 1.00, good from 0.65 to 0.80, and poor for any value below that.

					Methods
			HealthGAN	Gaussian	Fitted	Overfitted	Differential	Copy
			(*)	Multivariate	Parzen Win.	ANM	Privacy (DP)	(CP)
	Adversarial Accuracy	Train AA Test AA Privacy Loss 0.00 0.50 0.51	0.53 0.55 0.02	0.50 0.50 0.00	0.00 0.50 0.50	0.05 0.52 0.47	0.00 0.50 0.50
	Utility	Area Under the Curve	0.66	0.62	0.77	0.74	0.87	0.88
	Footprint	Up-to-specs	yes	yes	no	no	no	no

Table 2 :

 2 Comparing Odds Ratios for Each Variable in Real Data and Synthetic Data. Blue indicates overlapping intervals, red indicates non-overlapping intervals

Table 3 :

 3 Logistic Regression Results. Blue indicates overlapping intervals, red indicates non-overlapping intervals

Table 4 :

 4 Cox Regression Results. Blue indicates overlapping intervals, red indicates nonoverlapping intervals. INF indicates there wasn't enough synthetic data generated in that range and therefore the interval could not be computed.

Table 5 :

 5 Coefficient values and significance per variable per day

Table 6 :

 6 Area Under the Curve results

	Data

https://github.com/yknot/ESANN2019

medGAN was not included as a compared method as it can only generate discrete data

'Insurance','Language','Religion','Marital-Status','Ethnicity','Gender' and 'Age'.

https://competitions.codalab.org/competitions/19365

https://github.com/yknot/ESANN2019


Saloni Dash

Andrew Yale

Isabelle Guyon

Kristin P Bennett

Medical Time-Series Data Generation using Generative Adversarial Networks

Keywords: Synthetic Data, Generative Adversarial Networks, Time-Series

Medical data is rarely made publicly available due to high deidentification costs and risks. Access to such data is highly regulated due to it's sensitive nature. These factors impede the development of data-driven advancements in the healthcare domain. Synthetic medical data which can maintain the utility of the real data while simultaneously preserving privacy can be an ideal substitute for advancing research. Medical data is longitudinal in nature, with a single patient having multiple temporal events, influenced by static covariates like age, gender, comorbidities, etc. Extending existing time-series generative models to generate medical data can be challenging due to this influence of patient covariates. We propose a workflow wherein we leverage existing generative models to generate such data. We demonstrate this approach by generating synthetic versions of several time-series datasets where static covariates influence the temporal values. We use a state-of-the-art benchmark as a comparative baseline. Our methodology for empirically evaluating synthetic timeseries data shows that the synthetic data generated with our workflow has higher resemblance and utility. We also demonstrate how stratification by covariates is required to gain a deeper understanding of synthetic data quality and underscore the importance of including this analysis in evaluation of synthetic medical data quality.

Introduction

Medical data in the form of Electronic Medical Records (EMR) has been widely used by hospitals in the United States for aiding hospital processes like quality improvement, monitoring patient safety etc. [START_REF] Parasrampuria | Hospitals' use of electronic health records data[END_REF]. Furthermore, EMR data has also been used for advancing healthcare research for decades [START_REF] Harris Nordo | Use of ehrs data for clinical research: Historical progress and current applications[END_REF]. However, high de-identification costs and risks severely limit public access to such data. This imposes huge restrictions on data-driven clinical research and makes studies that use this data difficult to reproduce.

A generative model that samples from the distribution of the health data, while simultaneously preserving it's privacy is an ideal solution to the problem.

Generative models like Generative Adversarial Networks (GANs) [START_REF] Yale | Privacy preserving synthetic health data[END_REF][START_REF] Choi | Generating multi-label discrete electronic health records using generative adversarial networks[END_REF] (Health-GAN, medGAN) explicitly generate snapshots of EMR type data. However, real EMR data is longitudinal in nature and falls in the domain of time-series generative modelling. A key aspect of medical data is static covariates that heavily influence temporal variables. For instance, a patient record not only contains details of hospital visits over a period of time but also static demographic details like gender, ethnicity, comorbidities etc. We characterize a good medical timeseries generative model as one that jointly models the distribution of the static as well as the temporal variables.

We address this problem in the paper and provide a simple baseline that can be used for comparison against future medical time-series generative models. The primary contributions of this paper † are:

1. Illustration of an efficient, flexible workflow to facilitate joint modelling and synthesis of static and temporal variables. 2. Explicit qualitative evaluation of influence of static covariates on time-series variables. 3. Reproducing clinical time-series benchmarks on synthetic versions of a publicly available and widely used medical dataset.

Related Work

An open source synthetic patient generator called Synthea [START_REF]Synthetic patient generation[END_REF] uses hand-crafted modules aided by health care practitioners and statistics derived from real data to generate patients from their birth day to the present day. It does not violate any privacy restrictions because it does not use real patients to generate the data. It also claims to maintain utility as the generator uses underlying rules manually derived from the real data. However, the time-series generated for a record are not necessarily representative of real patient trajectories. Additionally, the custom designed rules severely limit the type of data that can be generated and are not easily extendable to other distributions of data. Recurrent (Conditional) GAN (RCGAN) [START_REF] Esteban | Real-valued (medical) time series generation with recurrent conditional gans[END_REF] uses recurrent neural networks (RNNs) in the GAN framework, to generate real valued time-series medical data like respiratory rate, heart rate etc. In the conditional setting, both the generator and discriminator are conditioned on labels sampled from the real data during training, and generated from independent distributions during the generation process. The labels guide the generative process but are not modelled jointly with the time-series variables.

Time-Series GAN [START_REF] Yoon | Time-series generative adversarial networks[END_REF] explicitly models time-series distributions as a joint distribution of static and temporal variables. It produces realistic time-series by jointly optimizing adversarial and supervised losses. We found TimeGAN to be the only time-series generative model that addresses the problem of jointly modelling static and temporal variables, and use it as a comparative baseline for our methodology.

Method

We illustrate our approach by generating time-series datasets for three time-series datasets where the covariates have a strong influence on the time-series variables. The datasets are (1) PJM Hourly Energy Consumption Dataset (Kaggle) [12],(2) Sleeping Patterns from American Time Use Survey (ATUS) [START_REF]American time use survey[END_REF], (3) Medical Information Mart for Intensive Care (MIMIC-III) [START_REF] Alistair | Mimic-iii, a freely accessible critical care database[END_REF].

The workflow is as follows:

1. Identify appropriate summary statistic(s) for time-series variables (e.g. mean, median, skew, count etc.) 2. Compute summary statistic(s) for fixed time-intervals over the whole time period. 3. Append summary statistic(s) to static variables. This maps the time-series data frame to a cross-sectional data frame. 4. Use a generative model of your choice to generate this transformed data.

For (1), we choose summary statistics inspired by downstream applications of the synthetic data. For (4), we use HealthGAN, a Wasserstein GAN [START_REF] Yale | Privacy preserving synthetic health data[END_REF], to generate the transformed data. The HealthGAN includes encoding mappings for categorical, numerical and ordinal variables of which the ordinal mappings particularly boosted our results for the ATUS dataset. We evaluate resemblance of the synthetic data to real data by assessing summary statistics conditioned on covariates and the utility by reproducing published research results.

This workflow is best suited for data where the time-series can be dissected into meaningful intervals to compute summary statistics relevant to the downstream application of synthetic data. An appropriate transformation of computing mean, variance, skew, kurtosis etc. for the whole time-series is always feasible.

We use TimeGAN † as a comparison for this workflow for two of the above datasets. It should be noted that we use the default parameters for TimeGAN and do not fine-tune the model while generating the data.

Results

PJM Hourly Energy Consumption

Our first dataset is not a medical dataset, but it provide an illustration of the challenge of synthesizing time series datasets wiith covariates. PJM Interconnection LLC (PJM) is a transmission organization (RTO) which is part of the Eastern Interconnection grid operating an electric transmission system serving specific regions of the United States. The dataset † primarily comprises of a datetime stamp and the average energy consumed in Mega Watts (MW) in that hour.

A natural summary statistic for the dataset is the average energy consumed per hour. We set the time period to be one day. We hence get twenty-four time-series statistics, one for each hour of the day which we append to the static variables of day of week and month derived from the datetime stamp. The transformed data now has twenty-six variables which are generated by HealthGAN. We also separately generate the original data using TimeGAN.

We then qualitatively evaluate the synthetic datasets by comparing trends in the real data. Figure 1 shows close resemblance of the summary statistic of average hourly energy consumption across twenty-four hours for the real and synthetic datasets (derived from HealthGAN and TimeGAN). A Welsch t-test of the samples binned by hour shows that the hourly means from HealthGAN (p-value = 0.51) as well as the hourly means from TimeGAN (p-value = 0.18) do not significantly differ from the hourly means of the real data. Both generations methods are seemingly performing well.

Figure 2 analyses the influence of the static covariates of day of week and month on the average energy consumption in the real data, which reports highest energy consumption during the weekdays in the evening hours and the summer months. These trends are mimicked in the synthetic data generated by HealthGAN. In the synthetic data generated by TimeGAN, the hourly and weekly trends are captured reasonably well but when examining by the covariate months, the peak at months 7 and 8 is missed.

American Time Use Survey (ATUS)

We generate sleeping patterns from American Time Use Survey (ATUS), a federally administered, annual survey on time use in the United States [START_REF]American time use survey[END_REF]. The survey records how Americans divide their time among life's activities in a nationally representative sample. There are many different types of events per person. However, we choose to restrict our data to only the sleep activities of the people over a period of thirty hours. (Please refer to [START_REF] Dash | Synthetic event time series health data generation[END_REF] for more details). We divide the thirty hours into thirty events of one hour each, and compute average sleep in that hour. We then append them to the static variables of age, sex, day of the week and month of the year. The data is now in matrix form, consisting of 34 (including covariates) features per patient, ready for the HealthGAN to generate this data. Sleep data is synthesized from TimeGAN as well to use as a comparative baseline. To analyse the relationship between the static covariates and time-series variables, we reproduce a sleep study [START_REF] Basner | American time use survey: sleep time and its relationship to waking activities[END_REF] which analyses the average sleeping time stratified by age and day of week. Figure 4 shows the variations in sleep depending on age group and day of the week. In the real data, the average sleeping time on weekends is significantly different from that on weekdays. Overall, teenagers and young adults sleep (age group 15-24) significantly more than other age groups, whereas adults between 35-54 years in general require less sleep than other age groups. These trends are captured well in the synthetic data from HealthGAN. The variations in average sleep times binned by group and day of the week are so high in the real data that although the synthetic plot appears to be shifted by an hour, it still falls within the 95% confidence intervals of the means in the real data. For the synthetic data from TimeGAN, the distributions for days Tuesday through Saturday are missing completely, suggesting mode collapse, and the trends captured are significantly different from those in the real data. The MIMIC -III dataset [START_REF] Alistair | Mimic-iii, a freely accessible critical care database[END_REF] is a publicly available critical care database which is widely used in research studies [START_REF] Bose | Impact of intensive care unit discharge delays on patient outcomes: A retrospective cohort study[END_REF][START_REF] Lokhandwala | One-year mortality after recovery from critical illness: A retrospective cohort study[END_REF][START_REF] Octávio Deliberato | Severity of illness scores may misclassify critically ill obese patients[END_REF]. Specifically we use three clinical prediction time-series benchmarks derived from MIMIC -III [START_REF] Harutyunyan | Multitask learning and benchmarking with clinical time series data[END_REF]. These tasks consist of:

1. In-Hospital Mortality Prediction -Predicting In-Hospital mortality based on 48 hours of ICU data. 2. Decompensation Prediction -Predicting whether a patient's health will worsen over the next 24 hours. 3. Phenotype Classification -Predicting which of the 25 acute care conditions are present in a patient record For each of the above tasks, the logistic regression baseline specifies which summary statistics to extract from the time-series. We attempt to reproduce these baseline results in the generated data as well. The paper identifies 17 clinical variables as primary temporal variables. For each variable, six different sample statistic features (mean, std dev, skew etc.) are computed on seven different subsequences of a given time series (full, first 10%, first 25% etc.). Please refer to [START_REF] Harutyunyan | Multitask learning and benchmarking with clinical time series data[END_REF] for more details. In total there are 714 temporal variables. In (1) the static covariates are age, gender and the mortality label. This results in a total of 717 variables for each patient. The task is a binary classification task with the primary metric being AUC-ROC. In (2) the static variables are age, gender and decompensation label, resulting in 717 variables. This is also a binary classification task with the primary metric being AUC-ROC. The results for (1) and ( 2) are summarized in Figure 5. In (3) the static variables are age, gender and the 25 acute care conditions, resulting in 741 variables. This is a multi-label classification problem to predict phenotype with the primary metric being AUC -ROC for each variable treated independently. The results for the 25 prediction prediction tasks are illustrated in Figure 6.

In figures 5 and 6, RR refers to train on real test on real, RS to train on real test on synthetic, etc. RS scores indicate whether the synthetic data can be substituted for the real data for a downstream application, while SR score indicates whether the synthetic data has realistic features. Overall, across all 27 MIMIC-III tasks, we report RS and SR scores to be reasonably close to RR scores. Note, however, that the SS scores tend to usually overshoot the RR scores indicating that the generated distribution is more regular than the real data.

Conclusion and Future Work

Medical time-series data sets are characterized by both static as well as temporal variables which must be modelled jointly to generate realistic medical time-series data. We provide a simple, flexible and effective workflow to generate this kind of data. We test our methodology by synthesizing three different time-series datasets, two of which are health datasets. We empirically show that the data generated by HealthGan not only shows close univariate resemblance with the real data but also captures trends influenced by static covariates. We use a state-of-the-art benchmark † as a comparative baseline. We highlight the importance of evaluating synthetic medical data with respect to critical covariates and the importance of including such analysis in time-series generative models for medical data. We plan to use super-resolution multivariate GANs trained on varying length interval summaries to capture more complex EMR data in the future. † We would like to thank the authors of TimeGAN for their help with implementation and evaluation details.

Fig. 1 :

 1 Fig. 1: Hourly Energy Consumption -YTrue is the hourly energy consumption in the real data and YPred is the hourly energy consumption in the generated data. The values in both the real and synthetic datasets match closely.

Fig. 2 :

 2 Fig. 2: Average Energy Consumed vs Hour, Day of Week and Month for real and synthetic data sets shows HealthGAN synthetic data highly resembles real data even when covariates are considered.

Figure 3

 3 shows the average sleep trends in the real data and synthetic datasets. The average sleep per hour in the HealthGAN synthetic data closely resembles the real data. Most people are awake by 10:00 am and asleep by 12:00 am. The synthetic data from TimeGAN does not follow this distribution. A Welsch t-test of the sleep times binned by hour shows that the mean sleep time per hour in the data from HealthGAN (p-value = 0.58) is not statistically different from that in the real data. However, for the data from TimeGAN (p-value = 0.012), the means appear to be significantly different.

Fig. 3 :Fig. 4 :

 34 Fig. 3: Average Hourly Sleep Trends of real (blue) and synthetic data (yellow) generated by HealthGann and TimeGAN

Fig. 5 :

 5 Fig. 5: AUC ROC for MIMIC-III Mortality and Decompensation tasks. First letter indicates training set (Real or Synthetic). Second letter indicates testing set

Fig. 6 :

 6 Fig. 6: Phenotype Classification Heatmap -In general, RR, RS, SR scores fall in the same range. SS scores tend to overshoot RR scores significantly, suggesting increased regularity in synthetic data as compared to the real data.

† We use the source code available at https://bitbucket.org/mvdschaar/ mlforhealthlabpub/src/master/alg/timegan/ † https://www.kaggle.com/robikscube/hourly-energy-consumption