Corentin Hardy
email: corentin.hardy@technicolor.com

Erwan Le Merrer
email: erwan.le-merrer@inria.fr

Bruno Sericola
email: bruno.sericola@inria.fr

MD-GAN: Multi-Discriminator Generative Adversarial Networks for Distributed Datasets

A recent technical breakthrough in the domain of machine learning is the discovery and the multiple applications of Generative Adversarial Networks (GANs). Those generative models are computationally demanding, as a GAN is composed of two deep neural networks, and because it trains on large datasets. A GAN is generally trained on a single server.

In this paper, we address the problem of distributing GANs so that they are able to train over datasets that are spread on multiple workers. MD-GAN is exposed as the first solution for this problem: we propose a novel learning procedure for GANs so that they fit this distributed setup. We then compare the performance of MD-GAN to an adapted version of federated learning to GANs, using the MNIST, CIFAR10 and CelebA datasets. MD-GAN exhibits a reduction by a factor of two of the learning complexity on each worker node, while providing better or identical performances with the adaptation of federated learning. We finally discuss the practical implications of distributing GANs.

I. INTRODUCTION

Generative Adversarial Networks (GANs for short) are generative models, meaning that they are used to generate new realistic data from the probability distribution of the data in a given dataset. Those have been introduced by Goodfellow et al in seminal work [START_REF] Goodfellow | Generative Adversarial Networks[END_REF]. Applications are for instance to generate pictures from text descriptions [START_REF] Reed | Generative Adversarial Text to Image Synthesis[END_REF], to generate video from still images [START_REF] Vondrick | Generating Videos with Scene Dynamics[END_REF], to increase resolution of images [START_REF] Ledig | Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network[END_REF], or to edit them [START_REF] Perarnau | Invertible Conditional GANs for image editing[END_REF]. Application to the chess game [START_REF] Chidambaram | Style transfer generative adversarial networks: Learning to play chess differently[END_REF] or to anomaly detection [START_REF] Hyunsun | Generative ensembles for robust anomaly detection[END_REF] were also proposed, which highlights the growing and cross-domain interest from the machine learning research community towards GANs.

A GAN is a machine learning model, and more specifically a certain type of deep neural networks. As for all other deep neural networks, GANs require a large training dataset in order to fit the target application. Nowadays, the norm is then for service providers to collect large amounts of data (user data, application-specific data) into a central location such as their datacenter; the learning phase is taking place in those premises. The image super-resolution application [START_REF] Ledig | Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network[END_REF] for instance leverages 350, 000 images from the ImageNet dataset; this application is representative of new advances: it provides state of the art results in its domain (measured in terms of quality of image reconstruction in that example); yet the question of computational efficiency or parallelism is left aside to future works.

The case was made recently for geo-distributed machine learning methods, where the data acquired at several datacenters stay in place [START_REF] Hsieh | Gaia: Geo-distributed machine learning approaching LAN speeds[END_REF], [START_REF] Cano | Towards geo-distributed machine learning[END_REF], as the considered data volumes would make it impossible to meet timing requirements in case of data centralization. Machine learning algorithms are thus to be adapted to that setup. Some recent works consider multiple generators and discriminators with the goal to improve GAN convergence [START_REF] Durugkar | Generative Multi-Adversarial Networks[END_REF], [START_REF] Hoang | Multi-Generator Generative Adversarial Nets[END_REF]; yet they do not aim at operating over spread datasets. The Parameter Server paradigm [START_REF] Li | Scaling distributed machine learning with the parameter server[END_REF] is the prominent way of distributing the computation of classic (i.e., non-GAN) neural networks: workers compute the neural network operations on their data share, and communicate the updates (gradients) to a central server named the parameter server. This framework is also the one leveraged for geodistributed machine learning [START_REF] Hsieh | Gaia: Geo-distributed machine learning approaching LAN speeds[END_REF].

In this paper we propose MD-GAN, a novel method to train a GAN in a distributed fashion, that is to say over the data of a set of participating workers (e.g., datacenters connected through WAN [START_REF] Hsieh | Gaia: Geo-distributed machine learning approaching LAN speeds[END_REF], or devices at the edge of the Internet [START_REF] Hardy | Distributed deep learning on edge-devices: Feasibility via adaptive compression[END_REF]). GANs are specific in the sense that they are constituted of two different components: a generator and a discriminator. Both are tightly coupled, as they compete to reach the learning target. The challenges for an efficient distribution are numerous; first, that coupling requires fine grained distribution strategies between workers, so that the bandwidth implied by the learning process remains acceptable. Second, the computational load on the workers has to be reasonable, as the purpose of distribution is also to gain efficiency regarding the training on a single GPU setup for instance. Lastly, as deep learning computation has shown not to be a deterministic process when considering the accuracy of the learned models facing various distribution scales [START_REF] Zhang | Staleness-aware async-sgd for distributed deep learning[END_REF], the accuracy of the model computed in parallel has to remain competitive.

a) Contributions: The contributions of this paper are: (i) to propose the first approach (MD-GAN) to distribute GANs over a set of worker machines. In order to provide an answer to the computational load challenge on workers, we remove half of their burden by having a single generator in the system, hosted by the parameter server. This is made possible by a peer-to-peer like communication pattern between the discriminators spread on the workers. (ii) to compare the learning performance of MD-GAN with regards to both the baseline learning method (i.e., on a standalone server) and an adaption of federated learning to GANs [START_REF] Mcmahan | Federated learning of deep networks using model averaging[END_REF]. This permits head to head comparisons regarding the accuracy challenge. (iii) to experiment MD-GAN and the two other competitors on the MNIST, CIFAR10 and CelebA datasets, using GPUs. In addition to analytic expectations of communication and computing complexities, this sheds light on the advantages of MD-GAN, but also on the salient properties of the MD-GAN and federated learning approaches for the distribution of GANs.

b) Paper organization: In Section II, we give general background on GANs. Section III presents the computation setup we consider, and presents an adaptation of federated learning to GANs. Section IV details the MD-GAN algorithm. We experiment MD-GAN and its competitors in Section V. In Section VI, we review the related work. We finally discuss future works and conclude in Section VII.

II. BACKGROUND ON GENERATIVE ADVERSARIAL NETWORKS

The particularity of GANs as initially presented in [START_REF] Goodfellow | Generative Adversarial Networks[END_REF] is that their training phase is unsupervised, i.e., no description labels are required to learn from the data. A classic GAN is composed of two elements: a generator G and a discriminator D. Both are deep neural networks (DNN). The generator takes as input a noise signal (e.g., random vectors of size k where each entry follows a normal distribution N (0, 1)) and generates data with the same format as training dataset data (e.g., a picture of 128x128 pixels and 3 color channels). The discriminator receives as input either some data from two sources: from the generator or from the training dataset. The goal of the discriminator is to guess from which source the data is coming from. At the beginning of the learning phase, the generator generates data from a probability distribution and the discriminator quickly learns how to differentiate that generated data from the training data. After some iterations, the generator learns to generate data which are closer to the dataset distribution. If it eventually turns out that the discriminator is not able to differentiate both, this means that the generator has learned the distribution of the data in the training dataset (and thus has learned an unlabeled dataset in an unsupervised way).

Formally, let a given training dataset be included in the data space X, where x in that dataset follows a distribution probability P data . A GAN, composed of generator G and discriminator D, tries to learn this distribution. As proposed in the original GAN paper [START_REF] Goodfellow | Generative Adversarial Networks[END_REF], we model the generator by the function G w : R -→ X, where w contains the parameters of its DNN G w and is fixed. Similarly, we model the discriminator by the function D θ : X -→ [0, 1] where D θ (x) is the probability that x is a data from the training dataset, and θ contains the parameters of the discriminator D θ . Writing log for the logarithm to the base 2, the learning consists in finding the parameters w * for the generator:

w * = arg min w max θ (A θ + B θ,w ), with A θ = E x∼Pdata [log D θ (x)] and B θ,w = E z∼N [log (1 -D θ (G w (z))] ,
where z ∼ N means that each entry of the -dimensional random vector z follows a normal distribution with fixed parameters. In this equation, D adjusts its parameters θ to maximize A θ , i.e., the expected good classification on real data and B θ,w , the expected good classification on generated data. G adjusts its parameters w to minimize B θ,w (w does not have impact on A), which means that it tries to minimize the expected good classification of D on generated data. The learning is performed by iterating two steps, named the discriminator learning step and the generator learning step, as described in the following.

1) Discriminator learning: The first step consists in learning θ given a fixed G w . The goal is to approximate the parameters θ which maximize A θ + B θ,w with the actual w. This step is performed by a gradient descent (generally using the Adam optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF]) of the following discriminator error function J disc on parameters θ:

J disc (X r , X g ) = A(X r ) + B(X g ), with A(X r ) = 1 b x∈Xr log(D θ (x)); B(X g ) = 1 b x∈Xg log(1-D θ (x)),
where X r is a batch of b real data drawn randomly from the training dataset and X g a batch of b generated data from G. In the original paper [START_REF] Goodfellow | Generative Adversarial Networks[END_REF], the authors propose to perform few gradient descent iterations to find a good θ against the fixed G w .

2) Generator learning: The second step consists in adapting w to the new parameters θ. As done for step 1), it is performed by a gradient descent of the following error function J gen on generator parameters w:

J gen (Z g ) = B ({G w (z)|z ∈ Z g }) = 1 b x∈{Gw(z)|z∈Zg} log(1 -D θ (x)) = 1 b z∈Zg log(1 -D θ (G w (z)))
where Z g is a sample of b -dimensional random vectors generated from N . Contrary to discriminator learning step, this step is performed only once per iteration. By iterating those two steps a significant amount of times with different batches (see e.g., [START_REF] Goodfellow | Generative Adversarial Networks[END_REF] for convergence relaessentiallyted questions), the GAN ends up with a w which approximates w * well. Such as for standard deep learning, guarantees of convergence are weak [START_REF] Arjovsky | Towards Principled Methods for Training Generative Adversarial Networks[END_REF]. Despite this very recent breakthrough, there are lots of alternative proposals to learn a GAN (e.g., more details can be found in [START_REF] Arjovsky | Wasserstein GAN[END_REF], [START_REF] Odena | Conditional image synthesis with auxiliary classifier GANs[END_REF], and [START_REF] Salimans | Improved Techniques for Training GANs[END_REF]).

III. DISTRIBUTED COMPUTATION SETUP FOR GANS

Before we present MD-GAN in the next Section, we introduce the distributed computation setup considered in this paper, and an adaptation of federated learning to GANs. a) Learning over a spread dataset: We consider the following setup. N workers (possibly from several datacenters [START_REF] Hsieh | Gaia: Geo-distributed machine learning approaching LAN speeds[END_REF]) are each equipped with a local dataset composed of m samples (each of size d) from the same probability distribution P data (e.g., requests to a voice assistant, holiday pictures). Those local datasets will remain in place (i.e., will not be sent over the network). We denote by B = N n=1 B n the entire dataset, with B n the dataset local to worker n. We assume in the remaining of the paper that the local datasets are i.i.d. on workers, that is to say that there are no bias in the distribution of the data on one particular worker node.

The assumption on the fix location of data shares is complemented by the use of the parameter server framework we are now presenting.

b) The parameter server framework: Despite the general progress of distributed computing towards serverless operation even in datacenters (e.g., use of the gossip paradigm as in Dynamo [START_REF] Decandia | Dynamo: Amazon's highly available key-value store[END_REF] back in 2007), the case of deep learning systems is specific. Indeed, the amounts of data required to train a deep learning model, and the very iterative nature of the learning tasks (learning on batches of data, followed by operations of back-propagations) makes it necessary to operate in a parallel setup, with the use of a central server. Introduced by Google in 2012 [START_REF] Dean | Large scale distributed deep networks[END_REF], the parameter server framework uses workers for parallel processing, while one or a few central servers are managing shared states modified by those workers (for simplicity, in the remaining of the paper, we will assume the presence of a single central server). The method aims at training the same model on all workers using their given data share, and to synchronize their learning results with the server at each iteration, so that this server can update the model parameters.

Note that more distributed approaches for deep learning, such as gossip-based computation [START_REF] Blot | Gossip training for deep learning[END_REF], [START_REF] Hardy | Gossiping GANs[END_REF], have not yet proven to work efficiently on the data scale required for modern applications; we thus leverage a variant of parameter server framework as our computation setup. c) FL-GAN: adaptation of federated learning to GANs: By the design of GANs, a generator and a discriminator are two separate elements that are yet tightly coupled; this fact makes it nevertheless possible to consider adapting a known computation method, that is generally used for training a single deep neural network. 1 Federated learning [START_REF] Konečný | Federated Learning: Strategies for Improving Communication Efficiency[END_REF] proposes to train a machine learning model, and in particular a deep neural network, on a set of workers. It follows the parameter server framework, with the particularity that workers perform numerous local iterations between each communication to the server (i.e., a round), instead of sending small updates. All workers are not necessarily active at each round; to reduce conflicting updates, all active workers synchronize their model with the server at the beginning of each round.

In order to compare MD-GAN to a federated learning type of setup, we propose an adapted version of federated learning to GANs. This adaptation considers the discriminator D and generator G on each worker as one computational object to be treated atomically. Workers perform iterations locally on their data and every E epochs (i.e., each worker passes E times the data in their GAN) they send the resulting parameters to the server. The server in turn averages the G and D parameters of all workers, in order to send updates to those workers at the next iteration. We name this adapted version FL-GAN; it is depicted by Figure 1 b).

We now detail MD-GAN, our proposal for the learning of GANs over workers and their local datasets.

IV. THE MD-GAN ALGORITHM

A. Design rationale

To diminish computation on the workers, we propose to operate with a single G, hosted on the server2 . That server holds parameters w for G; data shares are split over workers. To remove part of the burden from the server, discriminators are solely hosted by workers, and move in a peer-to-peer fashion between them. Each worker n starts with its own discriminator D n with parameters θ n . Note that the architecture and initial parameters of D n could be different on every worker n; for simplicity, we assume that they are the same. This architecture is presented on Figure 1 a).

The goal for GANs is to train generator G using B. In MD-GAN, the G on the server is trained using the workers and their local shares. It is a 1-versus-N game where G faces all D n , i.e., G tries to generate data considered as real by all workers. Workers use their local datasets B n to differentiate generated data from real data. Training a generator is an iterative process; in MD-GAN, a global learning iteration is composed of four steps:

• The server generates a set K of κ batches K = {X (1) , . . . , X (κ) }, with κ ≤ N . Each X (i) is composed of b data generated by G. The server then selects, for each worker n, two distinct batches, say X (i) and X (j) , which are sent to worker n and locally renamed as

X (g) n and X (d)
n . The way in which the two distinct batches are selected is discussed in Section IV-B1.

• Each worker n performs L learning iterations on its discriminator D n (see Section II-1) using

X (d) n and X (r)
n , where X (r) n is a batch of real data extracted locally from B n .

• Each worker n computes an error feedback F n on X (g) n by using D n and sends this error to the server. We detail in Section IV-B2 the computation of F n .

• The server computes the gradient of J gen for its parameters w using all the F n feedbacks. It then updates Table I: Table of notations its parameters with the chosen optimizer algorithm (e.g., Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF]). Moreover, every E epochs, workers start a peer-to-peer swapping process for their discriminators, using function SWAP(). The pseudo-code of MD-GAN, including those steps, is presented in Algorithm 1.

Note that extra workers can enter the learning task if they enter with a pre-trained discriminator (e.g., a copy of another worker discriminator); we discuss worker failures in Section V.

B. The generator learning procedure (server-side)

The server hosts generator G with its associated parameters w. Without loss of generality, this paper exposes the training of GANs for image generation; the server generates new images to train all discriminators and updates w using error feedbacks. n . This two-batch generation design is required, for the computation of the gradients for both D and G on separate data (such as for the original GAN design [START_REF] Goodfellow | Generative Adversarial Networks[END_REF]). A possible way to distribute the X (i) among the N workers could be to set

X (g) n = X ((n mod k)+1) and X (d) n = X (((n+1) mod κ)+1) for n = 1, . . . , N .
2) Update of generator parameters: Every global iteration, the server receives the error feedback F n from every worker n, corresponding to the error made by G on X

e ni = ∂ B(X (g) n ) ∂x i , with x i the i-th data of batch X (g) n . The gradient ∆w = ∂ B ∪ N n=1 X (g) n /∂w is deduced from all F n as ∆w k = 1 N b N n=1 xi∈X (g) n e ni ∂x i ∂w k , with ∆w k the k-th element of ∆w. The term ∂x i /∂w k is com- puted on the server. Note that ∪ N n=1 X (g) n = {G w (z)|z ∈ Z g }. Minimizing B ∪ N n=1 X (g) n
is thus equivalent to minimize J gen (Z g ). Once the gradients are computed, the server is able to update its parameters w. We thus choose to merge the feedback updates through an averaging operation, as it is the most common way to aggregate updates processed in parallel [START_REF] Recht | Hogwild: A lock-free approach to parallelizing stochastic gradient descent[END_REF], [START_REF] Dean | Large scale distributed deep networks[END_REF], [START_REF] Lian | Asynchronous parallel stochastic gradient for nonconvex optimization[END_REF], [START_REF] Chen | Revisiting Distributed Synchronous SGD[END_REF]. Using the Adam optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF], parameter w k ∈ w at iteration t, denoted by w k (t) here, is computed as follows:

w k (t) = w k (t -1) + Adam(∆w k ),
for i ← 1 to I do 4: X (r) n ← SAMPLES(B n , b) 5: X (g) n , X (d) n ← RECEIVEBATCHES(C) 6:
for l ← 0 to L do 7:

D n ←DISCLEARNINGSTEP(J disc ,D n ) 8:
end for 9:

F n ← { ∂ B(X (g) n ) ∂xi |x i ∈ X (g) n } 10: SEND(C, F n ) Send F n to server 11: if i mod ( mE b ) = 0 then 12: D n ←SWAP(D n ) 13:
end if Initialize w for G 26:

for i ← 1 to I do 27:

for j ← 0 to κ do 28:

Z j ←GAUSSIANNOISE(b)
29:

X (j) ← {G w (z)|z ∈ Z j } 30:
end for 31:

X (d) 1 , . . . , X (d)
n ← SPLIT(X (1) , . . . , X (κ) )

32:

X (g) 1 , . . . , X (g)
n ← SPLIT(X (1) , . . . , X (κ) )

33:

for n ← 1 to N do

34: SEND(W n , (X (d) n , X (g) n )) 35:
end for 36:

F 1 , . . . , F N ← GETFEEDBACKFROMWORKERS() 37:
Compute ∆w according to F 1 , . . . , F N

38:

for w k ∈ w do 39:

w k ← w k +ADAM(∆w k ) 40:
end for 41:

end for 42: end procedure where the Adam optimizer is the function which computes the update given the gradient ∆w k .

3) Workload at the server: Placing the generator on the server increases its workload. It generates κ batches of b data using G during the first step of a global iteration, and then receives N error feedbacks of size bd in the third step. The batch generation requires κbG op floating point operations (where G op is the number of floating operations to generate one data object with G) and a memory of κbG a (with G a the number of neurons in G). For simplicity, we assume that G op = O(|w|) and that G a = O(|w|). Consequently the batch generation complexity is O(κb|w|). The merge operation of all feedbacks F n and the gradient computations imply a memory and computational complexity of O(b(dN + κ|w|)).

4) The complexity vs. data diversity trade-off: At each global iteration, the server generates κ batches, with κ < N . If κ = 1, all workers receive and compute their feedback on the same training batch. This reduces the diversity of feedbacks received by the generator but also reduces the server workload. If κ = N , each worker receives a different batch, thus no feedback has conflict on some concurrently processsed data. In consequence, there is a trade-off regarding the generator workload: because κ = N seems cumbersome, we choose κ = 1 or κ = log(N ) for the experiments, and assess the impact of those values on final model performances. n of generated images is used to compute the error term F n of generator G. Once computed, F n is sent to the server for the computation of gradients ∆w.

1) The swapping of discriminators: Each discriminator n solely uses B n to train its parameters θ n . If too many iterations are performed on the same local dataset, the discriminator tends to over specialize (which decreases its capacity of generalization). This effect, called overfitting, is avoided in MD-GAN by swapping the parameters of discriminators θ n between workers after E epochs. The swap is implemented in a gossip fashion, by choosing randomly for every worker another worker to send its parameters to.

2) Workload at workers: The goal of MD-GAN is to reduce the workload of workers without moving data shares out of their initial location. Compared to our proposed adapted federated learning method FL-GAN, the generator task is deported on the server. Workers only have to handle their discriminator parameters θ n and to compute error feedbacks after L local iterations. Every global iteration, a worker performs 2bD op floating point operations (where D op is the number of floating point operations for a feed-forward step of D for one data object). The memory used at a worker is O(|θ|). III, for both MD-GAN and FL-GAN. Table IV instantiates those complexities with the actual quantities of data measured for the experiment on the CIFAR10 dataset. The first observation is that MD-GAN requires server to workers communication at every iteration, while FL-GAN performs mE/b iterations in between two communications. Note that the size of workersserver communications depends on the GAN parameters (θ and w) for FL-GAN, whereas it depends on the size of data objects and on the batch size in MD-GAN. It is particularly interesting to choose a small batch sizes, especially since it is shown by Gupta et al. [START_REF] Gupta | Model accuracy and runtime tradeoff in distributed deep learning: A systematic study[END_REF] that in order to hope for good performances in the parallel learning of a model (as discriminators in MD-GAN), the batch size should be inversely proportional to the number of workers N . When the size of data is around the number of parameters of the GAN (such as in image applications), the MD-GAN communications may be expensive. For example, GoogLeNet [START_REF] Szegedy | Going deeper with convolutions[END_REF] analyzes images of 224 × 224 pixels in RGB (150, 528 values per data) with less than 6.8 millions of parameters.

We plotted on Figure 2 an analysis of the maximum ingress traffic (x-axis) of the FL-GAN and MD-GAN schemes, for a single iteration, and depending on chosen batch size (y-axis). This corresponds for FL-GAN to a worker-server communication, and for MD-GAN for both worker-server and workerworker communications during an iteration. Plain lines depict the ingress traffic at workers, while dotted lines the traffic at the server; these quantities can help to dimension the network capabilities required for the learning process to take place. Note the log-scale on both axis. As expected the FL-GAN traffic is constant, because the communications depends only on the model sizes that constitute the GAN; it indicates a target upper bound for the efficiency of MD-GAN. MD-GAN lines crossing FL-GAN is indicating more incurring traffic with increasing batch sizes. A global observation is that MD-GAN is competitive for smaller batch sizes, yet in the order of hundreds of images (here of less than around b = 550 for MNIST and b = 400 for CIFAR10).

2) Computation complexity: The goal of MD-GAN is to remove the generator tasks from workers by having a single one at the server. During the traning of MD-GAN, the traffic between workers and the server is reasonable (Table III). The complexity gain on workers in term of memory and computation depends on the architecture of D; it is generally half of the total complexity because G and D are often similar. The consequence of this single generator-based algorithm is more frequent interactions between workers and the server, and the creation of a worker-to-worker traffic. The overall operation complexities are summarized and compared in Table II, for both MD-GAN and FL-GAN; the Table indicates a workload of half the one of FL-GAN on workers.

V. EXPERIMENTAL EVALUATION

We now analyze empirically the convergence of MD-GAN and of competing approaches.

A. Experimental setup

Our experiments are using the Keras framework with the Tensorflow backend. We emulated workers and the server on GPU-based servers equipped of two Intel Xeon Gold 6132 processor, 260 GB of RAM and four NVIDIA Tesla M60 GPUs or four NVIDIA Tesla P100 GPUs. This setup allows for a training of GANs that is identical to a real distributed deployement, as computation order of interactions for Algorithm IV are preserved. This choice for emulation is thus oriented towards a tighter control for the environment of competing approaches, to report more precise head to head result comparisons; raw timing performances of learning tasks are in this context inaccessible and are left to future work. a) Datasets: We experiment competing approaches on two classic datasets for deep learning: MNIST [START_REF] Lecun | The mnist database of handwritten digits[END_REF] and CIFAR10 [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF]. MNIST is composed of a training dataset of 60, 000 grayscale images of 28 × 28 pixels representing handwritten digits and another test dataset of 10, 000 images. Theses two datasets are composed respectively of 6, 000 and 1, 000 images for each digits. CIFAR10 is composed of a training set 50, 000 RGB images of 32×32 pixels representing the followings 10 classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck. CIFAR10 has a test dataset of 10, 000 images. We later validate our conclusions on the CelebA dataset [START_REF] Liu | Deep learning face attributes in the wild[END_REF], which is composed by 200K images of celebrities (128 × 128 pixels).

b) GAN architectures: In the experiments, we train a classical type of GAN named ACGAN [START_REF] Odena | Conditional image synthesis with auxiliary classifier GANs[END_REF]. We experiment with three different architectures for G and D: a multi-layer based architecture (MLP), a convolutional neural network based architecture (CNN) for MNIST and a CNN-based architecture for CIFAR10. Their characteristics are: c) Metrics: Evaluating generative models such as GANs is a difficult task. Ideally, it requires human judgment to assess the quality of the generated data. Fortunately, in the domain of GANs, interesting methods are proposed to simulate this human judgment. The main one is named the Inception Score (we denote it by IS), and has been proposed by Salimans et al. [START_REF] Salimans | Improved Techniques for Training GANs[END_REF], and shown to be correlated to human judgment. The IS consists to apply a pre-trained Inception classifier over the generated data. The Inception Score evaluates the confidence on the generated data classification (i.e., generated data are well recognized by the Inception network), and on the diversity of the output (i.e., generated data are not all the same). To evaluate the competitors on MNIST, we use the MNIST score (we name it MS), similar to the Inception score, but using a classifier adapted to the MNIST data instead of the Inception network. Heusel et al. propose a second metric named the Fréchet Inception Distance (FID) in [START_REF] Heusel | GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium[END_REF]. The FID measures a distance between the distribution of generated data P G and real data P data . It applies the Inception network on a sample of generated data and another sample of real data and supposes that their outputs are Gaussian distributions. The FID computes the Fréchet Distance between the Gaussian distribution obtained using generated data and the Gaussian distribution obtained using real data. As for the Inception distance, we use a classifier more adapted to compute the FID on the MNIST dataset. We use the implementation of the MS and FID available in Tensorflow 3 .

d) Configurations of MD-GAN and competing approaches: To compare MD-GAN to classical GANs, we train the same GAN architecture on a standalone server (it thus has access to the whole dataset B). We name this baseline standalone-GAN and parametrize it with two batch sizes b = 10 and b = 100.

We run FL-GAN with parameters E = 1 and b = 10 or b = 100; this parameter setting comes from the fact that E = 1 and b = 10 is one of the best configuration regarding computation complexity on MNIST, and because b = 50 is the best one for performance per iteration [START_REF] Mcmahan | Federated learning of deep networks using model averaging[END_REF] (having b = 100 thus allows for a fair comparison for both FL-GAN and MD-GAN). MD-GAN is run with also E = 1; i.e., for FL-GAN and MD-GAN, respective actions are taken after the whole dataset has been processed once.

Figure 3: MNIST score / Inception score (higher is better) and Fréchet Inception Distance (lower is better) for the three competing approaches, with regards to the number of iterations (x-axis).

For MD-GAN and FL-GAN, the training dataset is split equally over workers (images are sampled i.i.d). We run two configurations of MD-GAN, one with κ = 1 and another with κ = log(N ) , in order to evaluate the impact of the data diversify sent to workers. Finally, in FL-GAN, GANs over workers perform learning iterations (such as in the standalone case) during 1 epoch, i.e., until D n processes all local data B n .

We experimented with a number of workers N ∈ [1, 10, 25, 50]; geo-distributed approaches such as Gaia [START_REF] Hsieh | Gaia: Geo-distributed machine learning approaching LAN speeds[END_REF] or [START_REF] Cano | Towards geo-distributed machine learning[END_REF] also operate at this scale (where 8 nodes [9] and 22 nodes [START_REF] Hsieh | Gaia: Geo-distributed machine learning approaching LAN speeds[END_REF] at maximum are leveraged). All experiments are performed with I = 50, 000, i.e., the generator (or the N generators in FL-GAN) are updated 50, 000 times during a generator learning step. We compute the FID, MS and IS scores every 1, 000 iterations using a sample of 500 generated data. The FID is computed using a batch of the same size from the test dataset. In FL-GAN, the scores are computed using the generator on the central server.

B. Experiment results

We report the scores of all competitors, with regards to the iterations, on the Figure 3. The resulting curves are smoothed for readability. . This explains why MD-GAN obtains very similar scores than standalone GAN with b = 100 (except with the CNN on MNIST). We note that, as highlighted in discussion in Section IV-B4, the hyper-parameter κ has a significant impact on the learning process. The more the data diversity sent by the server to workers, the higher the generator scores.

For the experiments on MLP, FL-GAN does not converge, whereas MD-GAN has better scores (FID and MS) than the standalone competitor. We propose a multi-discriminator vs one generator game; some recent works [START_REF] Durugkar | Generative Multi-Adversarial Networks[END_REF] have shown that some central strategies based on one generator and multiple discriminators, or a mixture of generators and one discriminator [START_REF] Hoang | Multi-Generator Generative Adversarial Nets[END_REF], can as well exceed the performances of a standalone GAN. In the CNN experiments on MNIST, the FID and MS scores obtained by MD-GAN and FL-GAN are close to equivalent. In the CNN experiments on CIFAR10, MD-GAN obtains better IS and MS than FL-GAN over this more complex learning task.

These three experiments show that MD-GAN exploits the advantage of having a single generator to train, that faces multiple discriminators.

2) Scalability and the impact of worker to worker communications: We present on Figure 4 the evolution of the final accuracy score for MD-GAN (after 20, 000 iterations), as a function of the number of workers using the MLP model. Because the dataset is split over workers, increasing the number of participants reduces the size of local datasets

(|B n | = |B|/N ).
Two variants of MD-GAN are executed. The first one is the discussed MD-GAN algorithm, and the second one depicts on the dotted curves MD-GAN where no swapping between workers occurs (i.e., with respectively E = 1, and E = ∞). The blue curves present the MD-GAN scores when the workload on workers (i.e., the number of images to process) remains constant, while the orange curves present a constant workload of the central node. We note that Figure 4 also illustrates a varying size of mini-batches b used by workers on the curve with a constant workload on server: the larger N is, the lower b is in consequence, to maintain the same workload on the server.

We note that interesting phenomenons appear at scale after N = 10; for lower values of N the workers appear to have enough data locally to reach satisfying scores.

The first observation is that considering a constant workload on workers leads to better results. This yet comes at the price of a higher cost on the server (cf Table II andIII).

The swapping process between workers leads to better results. We yet observe that, despite the better result in MS, the FID score improvement using swapping is marginal in the case of the constant workload on server setting. This indicates that data available locally to workers is enough, and that their is a marginal gain to await from the diversity brought by swapping discriminators.

3) Fault tolerance of MD-GAN facing worker crashes: In order to assess the tolerance of a MD-GAN learning task facing worker fail-stop crashes (workers' data also disappear from the system when the crash occurs), we conduct the following experiment, presented in Figure 5. We operate in the same scenario than for experiments in Figure 3, and for the best performing MD-GAN setup (with κ = log(N ) ), but this time we trigger a worker to crash every I/N iterations (appearing as the curve in green). Consequence is that at I = 50, 000, all workers have crashed. For comparison with a baseline, standalone GAN (i.e., single server GAN learning) are reploted for two batch sizes (b ∈ [START_REF] Durugkar | Generative Multi-Adversarial Networks[END_REF]100]), and so is the non crashing run on the blue curve with same parametrization.

First observation is that this crash pattern has a no significant impact on the result performance for the MNIST dataset, for both MS and FD metrics. The MLP architecture even exhibits the smallest FD at the end of the experiment. This highlights that for this dataset, the MD-GAN architecture manages to learn fast enough so that crashes, and then the removal of dataset shares, are not a problem performancewise.

Both metrics are affected in the case of the CIFAR10 dataset: we observe a divergence due to crashes, and it happens early in the learning phase (around I = 5, 000, corresponding to the first crashed worker). This experiment shows the sensitivity of the learning to early failures, because GANs did not have enough time to accurately approximate the distribution of the data, and then misses the lost data shares for reaching a competitive score. Scores are yet comparable to the standalone baseline up to 8 crashed workers.

We nevertheless note that in the geo-distributed learning frameworks [START_REF] Hsieh | Gaia: Geo-distributed machine learning approaching LAN speeds[END_REF], [START_REF] Cano | Towards geo-distributed machine learning[END_REF] that our work is aiming to support, the crashes of several workers will undoubtedly trigger repair mechanisms, in order to cope with diverging learning tendencies.

4) Validation on a larger dataset: In this experiment, we validate the convergence of MD-GAN, and its interest with regards to the standalone and FL-GAN approaches. The goal is to train a GAN over the CelebA dataset [START_REF] Liu | Deep learning face attributes in the wild[END_REF]. We use 10K of the 200K images in CelebA as the test dataset, while the remaining images are distributed equally (i.i.d.) over the N = {1, 5} workers. The GAN architecture is a variant of the one used for the CIFAR10 dataset: G is composed of one fully-connected layer of 16, 384 neurons and two transposed convolutional layers of respectively 128 and 3 kernels of size 5×5; D is composed of six convolutional layers of respectively [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF][START_REF] Szegedy | Going deeper with convolutions[END_REF]64,128,256 cessed to compute one generator update). In this experiment, we use two different settings for the Adam optimizer, leading to better results for each competitor. The standalone GAN and FL-GAN use a learning rate of α = 0.003 for G (resp. α = 0.002 for D), and hyperparameters β 1 = 0.5, β 2 = 0.999 for both, whereas MD-GAN uses a learning rate of α = 0.001 for G (resp. α = 0.004 for D), and hyperparameters β 1 = 0.0, β 2 = 0.9 for both. The resulting FID and Inception scores during the 30, 000 iterations we considered are reported in Figure 6. We observe that all IS scores are comparable (MD-GAN is slightly above); yet regarding the FID, MD-GAN (as well as FL-GAN) is distanced by the standalone approach (as this is the case for the CNN experiment on MNIST).

VI. RELATED WORK: DISTRIBUTING DEEP LEARNING

Distributing the learning of deep neural networks over multiple machines is generally performed with the Parameter Server model proposed by J. Dean et al in [START_REF] Dean | Large scale distributed deep networks[END_REF]. This model was adapted in different works [START_REF] Gupta | Model accuracy and runtime tradeoff in distributed deep learning: A systematic study[END_REF], [START_REF] Zhang | Staleness-aware async-sgd for distributed deep learning[END_REF], [START_REF] Zhang | Deep learning with elastic averaging SGD[END_REF]. The first interest is to speed up the learning in large data-centers [START_REF] Li | Scaling distributed machine learning with the parameter server[END_REF], [START_REF] Chilimbi | Project adam: Building an efficient and scalable deep learning training system[END_REF], [START_REF] Chen | Revisiting distributed synchronous SGD[END_REF]. This parameter server model was used for privacy reasons in [START_REF] Shokri | Privacy-preserving deep learning[END_REF]. The federated learning is a most accomplished method using the parameter server model with auxiliary workers to reduce communications [START_REF] Konečný | Federated Learning: Strategies for Improving Communication Efficiency[END_REF] or increase the privacy [START_REF] Bonawitz | Practical secure aggregation for privacy preserving machine learning[END_REF].

We experimented in a position paper [START_REF] Hardy | Gossiping GANs[END_REF] the distribution of the generator function. In this fully decentralized setup where compute nodes exchange their generators and discriminators in a gossip fashion (there are n couples of generator and discriminators, one per worker), the experiment results are favorable to federated learning. We then propose MD-GAN as a solution for a performance gain over federated learning.

Finally, a recent work [START_REF] Im | Generative Adversarial Parallelization[END_REF] proposes to multiply the number of discriminators and generators in a datacenter location: the authors propose to train several couples of GAN in parallel and to swap generators and discriminators every fix amount of iterations. Durugkar et al. [START_REF] Durugkar | Generative Multi-Adversarial Networks[END_REF] propose a centralized multidiscriminators architecture to improve the discriminator judgment on generated data. In the same way, Hoang et al. [START_REF] Hoang | Multi-Generator Generative Adversarial Nets[END_REF] study a centralized multi-generator architecture is proposed to improve the generator capacities and to reduce the so called mode collapse problem [START_REF] Arjovsky | Towards Principled Methods for Training Generative Adversarial Networks[END_REF]. The works [START_REF] Wang | Ensembles of Generative Adversarial Networks[END_REF] and [START_REF] Tolstikhin | AdaGAN: Boosting Generative Models[END_REF] improve the mixture of generative adversarial models. Wang et al. [START_REF] Wang | Ensembles of Generative Adversarial Networks[END_REF] use ensemble of GANs trained separately organized as a cascade to build an aggregated model. In the work of Tolstikhin et al. [START_REF] Tolstikhin | AdaGAN: Boosting Generative Models[END_REF], GANs are trained sequentially using boosting strategies to incrementally improve the performance of the final model. Note that all these works are proposed to improve GAN convergence, but not to distribute the learning (discriminators have access to the whole dataset). Our contribution is a method leveraving multiple adversaries in a distributed setup, and taking the network constraints into account.

VII. PERSPECTIVES AND CONCLUSION

Before we conclude, we highlight the salient questions on the way to a widespread distribution of GANs.

1) Asynchronous setting: Instead waiting all F every global iteration, the server may compute a gradient ∆w and apply it each time it receives a single F n . Fresh batches of data can be generated frequently, so that they can be sent to idle workers. All workers can operate without global synchronization, contrarily to federated learning methods as FL-GAN. In this setting, the waiting times of both workers and the server are reduced drastically. However, because of asynchronous updates, there is no guarantee that the parameters w of a worker n at time t (used to generate X (g) n ) are the same at time t + ∆t when it sends its F n to the server.

In the parameter server model, asynchrony implies inconsistent updates by workers. In practice, the training task nevertheless works well if the learning rate is adapted in consequence [START_REF] Zhang | Staleness-aware async-sgd for distributed deep learning[END_REF], [START_REF] Gupta | Model accuracy and runtime tradeoff in distributed deep learning: A systematic study[END_REF], [START_REF] Hardy | Distributed deep learning on edge-devices: Feasibility via adaptive compression[END_REF].

2) The central server communication bottleneck: The parameter server framework, despite its simplicity, has the obvious drawback of creating a communication bottleneck towards the central server. This has been quantified by several works [START_REF] Li | Scaling distributed machine learning with the parameter server[END_REF], [START_REF] Hardy | Distributed deep learning on edge-devices: Feasibility via adaptive compression[END_REF], and solutions for traffic reduction between workers and the server have been proposed. Methods such as Adacomp [START_REF] Hardy | Distributed deep learning on edge-devices: Feasibility via adaptive compression[END_REF] propose to communicate updates based on gradient staleness, which constitutes a form of data compression.

In the context of GANs, those methods may be applied on generated data before they are sent to workers, and to the error feedback messages sent by workers to the server. In particular, concerning images, there are many techniques from their compression (with or without loss of information, see e.g., [START_REF] Weinberger | The loco-i lossless image compression algorithm: principles and standardization into jpegls[END_REF]).

A fine grained combination of techniques for gradient and data object compression would make the parameter server framework more sustainable for GANs and increasingly larger datasets to learn on. A second direction might be to mix the federated learning approach with ensemble of GANs training independently in cascades (as presented by Wang et al. [START_REF] Wang | Ensembles of Generative Adversarial Networks[END_REF]). Federated learning would act as the scheduling mechanism for the parallel ensembles; this would restrict the burden on the server to critical only communications (up-to-date model hosting and dispatching), while most of the training occurs on edge workers, hierarchically.

3) Adversaries in generative adversarial networks: The current deployment setup of GANs in the literature is assuming an adversary-free environment. In fact, the question of the capacity of basic deep learning mechanisms to embed byzantine fault tolerance has just been recently proposed for distributed gradient descent [START_REF] Blanchard | Machine learning with adversaries: Byzantine tolerant gradient descent[END_REF]. In addition to the gradient updates in GANs, and more specifically, the learning process is most likely prone to workers having their discriminator lie to the server's generator (by sending erroneous or manipulated feedback). The global convergence, and then the final performance of the learning task will be affected in an unknown proportion. This adversarial setup, and more generally better fault tolerance, are a crucial aspect for future applications in the domain.

4) Scaling the number of workers: We experimented MD-GAN over up to 50 parallel workers. The current scale at which parallel deep learning is operating is in the order to tens (e.g., in Gaia [START_REF] Hsieh | Gaia: Geo-distributed machine learning approaching LAN speeds[END_REF] or in [START_REF] Cano | Towards geo-distributed machine learning[END_REF]) to few hundreds of workers (experiments in TensorFlow [START_REF] Abadi | Tensorflow: A system for large-scale machine learning[END_REF] for instance reach 256 workers maximum). It is still not well understood what is the bottleneck for reaching larger scales: is the dataset size imposing the scale? Or is this the conflicting asynchronous updates [START_REF] Zhang | Staleness-aware async-sgd for distributed deep learning[END_REF] from workers to the server limiting the benefit of scale after a certain threshold? We note that federated learning can be used on a large number of workers (e.g., 2, 000 in some works [START_REF] Konečný | Federated Learning: Strategies for Improving Communication Efficiency[END_REF]) by using only a random subset of the available devices at every round. MD-GAN can be adapted in a similar way, with fewer discriminators than workers: because discriminator models are swapped during the learning process, the whole distributed dataset could be leveraged.

Those general questions for deep learning are also applying to the learning of GANs, as they are themselves constituted by couples of deep neural networks. The unknown spot comes from the specificity of GANs, because of the coupling of generators and discriminators; that coupling will most likely play an additional major role in the future algorithms that will be dedicated to push the scalability of GANs to a new standard.

This paper has presented generative adversarial networks in the novel context of parallel computation and of learning over distributed datasets; MD-GAN aims at being leveraged by geo-distributed or edge-device deep learning setups. We have presented an adaptation of federated learning to the problem of distributing GANs, and shown that it is possible to propose an algorithm (MD-GAN) that removes half the computation complexity from workers by using a discriminator swapping technique, while still achieving better results on the two reviewed datasets. GANs are computationaly and communication intensive, specially in the considered datadistributed setup; we believe this work brought a first viable solution to that domain. We hope that raised perspectives will trigger interesting future works for the system and algorithmic support of the nascent field of generative adversarial networks.

Figure 1 :

 1 Figure 1: The two proposed competitors for the distribution of GANs: a) The MD-GAN communication pattern, compared to b) FL-GAN (federated learning adapted to GANs). MD-GAN leverages a single generator, placed on the server; FL-GAN uses generators on the server and on each worker. MD-GAN swaps discriminators between workers in a peer-to-peer fashion, while in FL-GAN they remain fix and are averaged by the server upon reception from the workers.

1 )

 1 Distribution of generated batches: Every global iteration, G generates a set of k batches K = {X (1) , . . . , X (κ) } (with κ ≤ N ) of size b. Each participating worker n is sent two batches among K, X (g) n and X (d)

  (g) n . More formally, F n is composed of b vectors {e n1 , . . . , e n b }, where e ni is given by

Algorithm 1

 1 The MD-GAN algorithm 1: procedure WORKER(C, B n , I, L, b) 2: Initialize θ n for D n 3:

C

  . The learning procedure of discriminators (worker-side) Each worker n hosts a discriminator D n and a training dataset B n . It receives batches of generated images split in two parts: X (d) n and X (g) n . The generated images X (d) n are used for training D n to discriminate those generated images from real images. The learning is performed as a classical deep learning operation on a standlone server [1]. A worker n computes the gradient ∆θ n of the error function J disc applied to the batch of generated images X (d) n , and a batch or real image X (r) n taken from B n . As indicated in Section II-1, this operation is iterated L times. The second batch X (g)

D

  . The characteristic complexities of MD-GAN 1) Communication complexity: In the MD-GAN algorithm there are three types of communications: • Server to worker communication: the server sends its κ batches of generated images to workers at the beginning of global iterations. The number of generated images is FL-GAN MD-GAN Computation C O(IbN (|w| + |θ|)/(mE)) O(Ib(dN + κ|w|)) Memory C O(N (|w| + |θ|)) O(b(dN + κ|w|) Computation W O(Ib(|w| + |θ|)) O(Ib|θ|) Memory W O(|w| + |θ|) O(|θ|)TableII: Computation and memory complexities for MD-GAN and FL-GAN. The rows in grey highlight the reduction by a factor of two for MD-GAN on workers.

n

  , all workers send their error term F n to the server. The size of error term is bd per worker, because solely one float is required for each feature of the data.• Worker to worker communications: after E local epochs, each discriminator parameters are swapped. Each worker sends a message of size |θ n |, and receive a message of the same size (as we assume for simplicity that discriminator models on workers have the same architecture). Communication complexities are summarized in Table

Figure 2 :

 2 Figure 2: Maximal ingress traffic, per communication, for two types of GANs (for both MD-GAN and FL-GAN).

Figure 4 :

 4 Figure 4: MNIST score and Fréchet Inception Distance with regards to the varying number of workers (x-axis) for MD-GAN using the MLP model. Experiments include the disabling of the swapping processing for comparison purposes.

Figure 5 :

 5 Figure 5: MNIST score or Inception score and Fréchet Inception Distance over the number of iterations for MD-GAN with crash faults, compared to MD-GAN without any crash and to a standalone GAN.

  and 512 kernels of size 3 × 3, and one fully-connected layer of one neuron. The batch size for the standalone GAN and FL-GAN is b = 200 whereas the batch size of MD-GAN is b = 40 (corresponding to 200 images pro-

Figure 6 :

 6 Figure 6: Inception scores and Fréchet Inception Distance of the three competitors, on the CelebA dataset.

Table III :

 III Communication complexities for both MD-GAN and FL-GAN. C and W stand for the central server and the workers, respectively.

κb (with κ ≤ N ), but only two batches are sent per worker. The total communication from the server is thus 2bdN (i.e., 2bd per worker).

• Worker to server communications: after computing the generator errors on X

Table IV :

 IV Illustration of communication costs for both MD-GAN and FL-GAN, in the CIFAR10 experiment with 10 workers.

•

  In the MLP-based architecture for MNIST, G and D are composed of three fully-connected layers each. G layers contain respectively 512, 512 and 784 neurons, and D layers contain 512, 512 and 11 neurons. The total number of parameters is 716, 560 for G and 670, 219 for D. The total number of parameters is 628, 058 for G and 286, 048 for D. • In the CNN-based architecture for CIFAR10, G is composed of one full-connected layer of 6, 144 neurons and three transposed convolutional layers of respectively 192, 96, and 3 kernels of size 5 × 5. D is composed of six convolutional layers of respectively 16, 32, 64, 128, 256 and 512 kernels of size 3 × 3, a mini-batch discriminator layer and one full-connected layer of 11 neurons. The total number of parameters is 628, 110 for G and 100, 203 for D.

• In the CNN-based architecture for MNIST, G is composed of one full-connected layer of 6, 272 neurons and two transposed convolutional layers of respectively 32 and 1 kernels of size 5 × 5. D is composed of six convolutional layers of respectively 16, 32, 64, 128, 256 and 512 kernels of size 3 × 3, a mini-batch discriminator layer

[START_REF] Salimans | Improved Techniques for Training GANs[END_REF]

and one full-connected layer of 11 neurons.

We note that more advanced GAN techniques such as those by Wang et al.[START_REF] Wang | Ensembles of generative adversarial networks[END_REF] or by Tolstikhin et al.[START_REF] Tolstikhin | Adagan: Boosting generative models[END_REF] might also be distributed and serve as baselines; yet this distribution requires a full redesign of the proposed protocols, and is thus out of the scope of this paper.

In that regard, MD-GAN do not fully comply with the parameter server model, as the workers do not compute and synchronize to the same model architecture hosted at the server. Yet, it leverages the parallel computation and the iterative nature of the learning task proposed by the the parameter server framework.

Code available at https://github.com/tensorflow/models/blob/master/research/ gan/mnist/util.py.


Blazo Nastov

Vincent Chapurlat

Christophe Dony

François Pfister

A verification approach from MDE applied to Model Based Systems Engineering: xeFFBD dynamic semantics

. Among other principles, it promotes creating and analyzing models all along systems engineering. These models are used to discuss, to argue and finally to make decisions that impact the achieved system (in terms of functioning, costs, safety, etc.). One of the main expectations of MBSE is to permit engineers to dispose of models with a high level of confidence. For this purpose, several model Verification and Validation (V&V) approaches exist, aiming to ensure models' quality in terms of construction (models are correctly built) and in terms of relevance for reaching design objectives and stakeholders' requirements. This paper aims at discussing and evaluating an approach originally developed in the field of Model Driven Engineering by proposing some adaptations. The approach is illustrated on a well-known functional modeling language dedicated to MBSE field.

Introduction

Systems Engineering (SE) [START_REF]Systems and software engineering -System life cycle processes[END_REF] is a process and a system thinking oriented approach for designing complex systems. In this field, Model Based System Engineering (MBSE) is considered as "the formalized application of [system] modeling" [START_REF] Estefan | Survey of Model-Based Systems Engineering ( MBSE ) Methodologies 2 . Differentiating Methodologies from Processes , Methods , and Lifecycle Models[END_REF] all along engineering activities requested in SE processes [START_REF] Incose | Systems Engineering Vision 2020[END_REF]. A model represents a system under study so called System of Interest (SOI), considering it under one of the various possible views e.g. functional, behavioral, physical, requirements, contextual, etc. Once created, models allow system engineers to argue and make architectural decisions. These decisions have impact on the realized system, its functioning, its safety, induced costs and so on [START_REF]System Engineering Book of Knowledge[END_REF]. Therefore, engineers should have a high level of confidence in models they are handling, before making any decision based on them. So first, models have to be "well-constructed" (correctly built, respecting (meta) modeling requirements and building rules) and second, they have to be the "right models" in terms of 1) relevance for reaching design objectives and 2) respect to a part of, or all, stakeholder's requirements. Moreover, the level of confidence increases if models verify their "well-constructiveness" and "model-rightness", considered separately and in interaction with other models of the SOI. [START_REF] Incose | Systems Engineering Vision 2020[END_REF] LGI2P, Ecole des Mines d'Alès, Parc Scientifique G. Besse, 30000 Nîmes, France {Blazo.Nastov, Vincent.Chapurlat, Francois.Pfister}@mines-ales.fr 2 LIRMM, 161 rue Ada, 34392 Montpellier, France -dony@lirmm.fr Classically, models are the subject of study of Model Driven Engineering (MDE) [START_REF] Kent | Model Driven Engineering[END_REF] and nowadays they are built using Domain Specific Modeling Languages (DSMLs). So first and foremost is the creation of DSMLs, a process, defining the languages' abstract and concrete syntaxes. An abstract syntax is provided by a metamodel representing, through a graph of classes and associations, the concepts of a domain and their relations. A concrete syntax defines how instances of abstract syntax concepts (forming a model) are concretely represented in a textual or graphical form. In MBSE context, we focus on graphical representation of models. So proposing the abstract syntax and a graphical concrete syntax of a DSML makes it operational to create models, each of them seen as a graphical representation of a particular point of view of a SOI. The literature highlights several frameworks for creating DSMLs and graphical editors [START_REF] Pfister | A light-weight annotation-based solution to design Domain Specific Graphical Modeling Languages[END_REF], [START_REF] Steinberg | EMF: Eclipse Modeling Framework[END_REF], [START_REF] Pontisso | TOPCASED Combining Formal Methods with Model-Driven Engineering[END_REF]. Unfortunately, such editors are of "two dimensional drawing-board" nature because they do not deal with model wellconstructiveness or rightness. As a consequence, created models turn-out to be "simple two dimensional drawings" of modeled concepts and relationships.

The main idea of this work is to add a third dimension, allowing designers to create models that can be simulated and questioned, achieving some of the Verification and Validation (V&V) objectives and to ensure the coherence between all models of the same SOI. Some solutions are proposed in the field of MDE [START_REF] Jézéquel | Model driven language engineering with Kermeta[END_REF], [START_REF] Mellor | Executable UML: A Foundation for Model-Driven Architecture[END_REF], [START_REF] Combemale | A Design Pattern for Executable DSML[END_REF], but unfortunately they remain, not or hardly applied in the field of SE due to their insufficiency for reaching SE objectives. In this paper, we discuss and evaluate an existing approach coming from MDE by applying it on a DSML, often considered by systems engineers and architects as a relevant functional description language named eFFBD [START_REF] Dod | Systems Engineering Fundamentals[END_REF] in order to emerge an extension that reaches SE objectives.

The paper is structured as follows. Section 2 discusses the importance of DSML semantics and introduces approaches and concepts that allow defining such semantics. Some limitations of studied approaches are discussed and afterwards an approach exceeding discussed limitations is proposed. Section 3 evaluates that approach by applying it on the eFFBD language in order to create an executable extension. Section 4 proposes contributions that allow the adaptation of the approach in the field of SE, before concluding about research perspectives in Section 5.

Towards execution and validation of DSMLs

Abstract syntaxes of DSMLs partially define language semantics through their underlying structure and the vocabulary naming concepts and relationships. Unfortunately, such semantics may sometimes be ambiguous, since different engineers may have different understanding of a single model. Therefore, in order to have equal and non-ambiguous understanding, it is essential to define in a precise and non-ambiguous manner DSMLs semantics.

Semantics are either static, independent of any behavior, or dynamic, describing the dynamic comportment of models' elements (can be advisedly called "dynamic model" or "dynamic comportment" or "DSML behavior"). There are three ways to formalize dynamic semantics description. First, operational semantics describes model comportment as a sequence of states, transitions between states and a machine that executes such a state model. Second, denotational (translational) semantics transforms DSML concepts into other DSML concepts with predefined dynamic comportment. Last, axiomatic semantics describes in a declarative way the evolution of model properties [START_REF] Combemale | Approche de métamodélisation pour la simulation et la vérification de modèle --Application à l'ingénierie des procédés[END_REF]. In this paper we focus on defining DSMLs behavior using dynamic semantics.

Literature highlights several approaches and tools for defining dynamic semantics for a given DSML. For instance, Kermeta [START_REF] Jézéquel | Model driven language engineering with Kermeta[END_REF] is an executable metamodeling language that defines operational semantics for a given DSML (in imperative way). Another example is the Atlas Transformation Language (ATL) [START_REF] Jouault | ATL: a QVT-like transformation language[END_REF] that (in declarative way) defines operational semantics through endogenous transformations and denotational semantics through exogenous transformations. Additionally, metamodeling languages together with constraints definition languages can be used to define axiomatic semantics. Meta Object facilities (MOF) [START_REF] Omg | MOF Core specification[END_REF] is usually used to define metamodels and OCL (Object Constraint Language) [START_REF] Omg | OCL : Object Constraint Language[END_REF] to add constraints to metamodel e.g. pre and post conditions, invariants and so on. However, these tools and approaches are related to software engineering and programming languages which somehow make them difficult to use for SE experts. Indeed, dynamic semantics of dedicated DSML is to be described and formalized with minimal efforts from experts by assisting them and automating the process as much as possible. [START_REF] Combemale | A Design Pattern for Executable DSML[END_REF] Another approach, supporting state-based execution of DSMLs is proposed in [START_REF] Combemale | A Design Pattern for Executable DSML[END_REF]. The approach is schematized in Figure 1 as a pattern composed of four structural parts related to each other and of a fifth part providing the dynamic semantics relying on the previous four. Modeling concepts and relationships between them are defined in the Domain Definition MetaModel (DDMM) package. The DDMM does not usually contain execution-related information. Such kind of information is defined in the State Definition MetaModel (SDMM) package through several sets of states assigned to DDMM concepts that can evolve during execution. Model execution is described as successive state changes of DDMM concepts provoked by stimuli. The Event Definition MetaModel (EDMM) package defines different types of stimuli (events), together with their relationship to DDMM concepts and SDMM states. Applied stimuli are either injected by the environment (exogenous kind) or produced internally by the system in response to other stimuli (endogenous kind). The Trace Management MetaModel (TM3) provides monitoring mechanism of model execution by capturing stimuli. The last and key part is the package Semantics, composed of evolution rules, describing how the running model SDMM evolves over DDMM concepts (changing their state) according to the stimuli defined in the EDMM. Evolution rules can be either defined as operational semantics or as denotational semantics.

MetaModel (M2)

MetaMetaModel (M3)

Application in the field of SE

This section proposes to extend the eFFBD functional modeling language by building and tooling its dynamics in order to become able to interpret and animate eFFBD models, and to prove their evolution rules. Similar solution has been proposed by [START_REF] Seidner | EFFBDs Verification: Model checking in Systems Engineering[END_REF] using translational semantics, transforming eFFBD models into Petri Nets models. [START_REF] Chapurlat | Verification, validation, qualification and certification of enterprise models: Statements and opportunities[END_REF] argues that translational semantics solutions limit V&V objectives. Fortunately, such limitations are exceeded by operational semantics that allow achieving V&V objectives on a source model rather than a third party model. So the expected result is xeFFBD i.e. an executable eFFBD that we consider as "selfsufficient" in achieving these V&V objectives. In [START_REF] Haskins | Systems Engineering Handbook: A Guide for System Life Cycle Processes and Activities[END_REF] a short history and various evolutions of a particular DSML named FFBD (Functional Flow Block Diagram) and its main evolution named eFFBD (enhanced FFBD) is presented. It is considered as a functional-modeling language preferred in the SE community [START_REF] Chesnut | Systems Engineering Methods[END_REF]. This DSML provides system designers with a framework to describe the behavior of complex, distributed, hierarchical, concurrent and communicating systems [START_REF] Aizier | xFFBD: towards a formal yet functional modeling language for system designers[END_REF]. For instance, Figure 2 shows the functional architecture of an interface highlighting parallelism, selection, loop and other complex constructs allowed in eFFBD. Creating an executable DSML is divided into two major phases. First, a language executable metamodel is defined containing domain (in DDMM), execution-related (in SDMM and EDMM) and monitoring (in TM3) information. Second, semantics are defined describing how execution-related information evolves over domain concepts.

Phase 1: executable metamodel definition

Executable metamodel definition phase includes four construction stages: DDMM definition, SDMM definition, EDMM definition and TM3 definition. We apply such process to eFFBD as shown in Figure 3. First, language or domain concepts and relationships are defined. This construction stage is identical to a (non-executable) DSML metamodel definition. The created metamodel is called DDMM since it contains domain concepts and relationships. In order to reduce language complexity and to improve readability and understandability, we split the DDMM into three packages: xeFFBD Diagram, xeFFBD Construct and xeFFBD Flow, each one representing a different aspect of the eFFBD language. xeFFBD DDMM is then created by merging all three packages, using the "merge" predefined package operator of MOF.

The xeFFBD language defines three kinds of core elements: Function, Resource and Item. Function describes what a system must do. They transforms one or more input Items in one or more output Items respecting transformation rules, possibly under control of triggers. Resource is something (data, material or energy e.g. human operator, consumable, plans, etc.) that is requested and utilized or consumed during an inputs/outputs transformation. Requested resources are considered as independent from transformation goal and they are requested for function execution that modifies them. Item is something (data, material or energy) that is requested and transformed by function in order to provide another(s) distinct Item(s). Taking into account its type, an Item can be consumed or can remain available during certain time duration after which its value becomes obsolete and unusable. These core elements are characterized by temporal attributes e.g. minimal and maximal time of execution, life time, etc.

xeFFBD Diagram is the core package describing a xeFFBD diagram as a quadruplet of begin and end operators, main branch and set of input/output objects carried by flows. Begin and end describe starting and finishing points in a diagram. The branch is composed of several control constructions named exFFBD Constructs, described hereafter. Two sorts of input/output objects are then available: items and resources respectively carried out by item flows and resource flows as detailed below. Last, a diagram is temporized element, having started and finished execution time.

xeFFBD Construct package represents different constructions recurring into a xeFFBD Diagram. These constructions allow engineer to describe how functions are chained and the different manners of their execution, introducing the possibility to describe function parallelism, sequence, exclusion, and selection. A construct can either be 1) a function control construct composed of a set of functions (eventually one unique function) put in a sequence, or 2) an operator control construction containing minimum one branch beginning on a begin operator and ending on an end operator. Four types of operator control construction are introduced: AND, OR, Iteration and Loop. A fifth one, named replication construction, is not considered at this moment. AND and OR constructions contain minimum two branches and they represent respectively parallel and exclusive execution of branches. Iteration and Loop constructions represent two possibilities of repetitive execution of one branch differing in the stop condition. Iteration fixes a number of iterations, while loop stops on a Boolean condition. Constructions are temporized elements having started and finished execution time.

xeFFBD Flow package describes three types of flows that can be represented in a xeFFBD model: functional flow, item flow and resource flow. A functional flow describes the order in which functions are executed (related to the primitive relation successor/predecessor between two functions). It is represented by the functional flow class connecting functional flow connectable elements which are either operators or functions. A Resource Flow describes requested Resources of a function that consumes them and restores them after execution, modifying eventually some of resource characteristics such as its quality and quantity levels. For this a Resource Flow is characterized by two attributes: quantity and quality. Quantity attribute indicates the requested amount of resource, consumed as an input by a function in order to execute it (requested quantity), and provided as an output after execution of related functions (provided quantity). Quality attribute indicates the level of resource quality, requested as an input in order to execute related functions (requested quality), and restituted after function execution as an output altering then eventually the level of quality of the resource (provided quality) i.e. mixing for instance its availability and its efficiency. Item flow relates Item with function by input or output relationships. These relationships describe items that are needed and consumed as inputs for function execution and items that are provided as output after execution. Provided items are a result from transformation of inputs flows and eventually under the help or the control of resource flows. Note that there is a special kind of triggering items and resources that can trigger function execution, controlling then function start and/or stop conditions. Functional and resource flow have attributes (comment, condition and quantity, etc.), so they are represented in the metamodel using the classassociation pattern, while item flow is represented using associations. Once a DDMM is defined, the second construction stage consists in defining SDMM. In this stage, we define in the package xeFFBD SDMM, the possible states of some of previously defined domain concepts. First, domain concepts that may evolve have to be chosen. For instance, we chose the following: Construct, Function, Item and Resource. The third construction stage consists in defining the events requested for the evolution of evolving concepts together with their relationship with corresponding concepts. Such information is defined in the EDMM. For instance, we defined three types of events: construct event, function event and item event.

Additionally, Figure 4 shows finite states automate associated to the concept Function. Here, on the one hand, function states are represented by automates' states and on the other hand, different types of function event are represented by automates' transitions. The evolution of concepts is represented as transition firing. In this sense, we consider here the input/output transformation described by a Function, is first possible (Authorized) i.e. the function can start but wait for Items (and eventually Resources) before being able to make the real transformation of energy, material and / or data (Execution) providing then the outputs items and resources (Finished). Due to external events, a function can be suspended and even aborted (Suspended, Aborted) e.g. in case of dysfunction of the component on which the function has been allocated. In the case of Item and Resource, the state model is replaced by defining state variables named quantity and quality allowing us to reduce the number of possible states. For example, a resource of oil can be in state 50 liters, but also in state 100 liters if a function provides as output another 50 liters of that same resource.

Sleep Authorised

The executable metamodel definition process ends by defining a monitoring mechanism considered as fourth stage of this process. For this, we propose the generic trace mechanism described in the approach, the TM3 package, shown in Figure 3.

Phase 2: semantics definition

As previously defined, xeFFBD metamodel contains execution-related (dynamic) information (e.g. packages SDMM and EDMM). Yet, xeFFBD metamodel is static, until the package Semantics (Figure 1) is defined. This package defines how and when dynamic information is executed on domain concepts, allowing state and property changes. For this, we adopt a proposed property-driven approach detailed in [START_REF] Combemale | A Property-Driven Approach to Formal Verification of Process Models[END_REF]. This approach describes how to formally define execution rules under the form of properties (described below), and how to become able to check some of those properties. Three types of properties can be expressed: structural properties, temporal properties and quantitative properties. The approach distinguishes properties checked on each model execution called universal properties from those checked once called existential properties.

Model evolution is first, defined through universal and existential properties by preconditioning events, second, through transitions that are defined between domain concepts states and finally, through event-based transition firing. When fired, transitions invoke state changing of domain concepts. Figure 4 illustrates different states of Function, event-based transitions between states and corresponding events. For instance, if event StartFunction is applied on an instance of Function that is in state Sleep, a transition is fired changing its state into Authorized. We consider that execution of lower level embedded constructions is controlled (i.e. started and finished) by higher level embedding constructions respecting an ordering given by a functional flow. Figure 5 illustrates an example of such execution control describing applied events and invoked state changes of components represented by different type of arrows. A simple xeFFBD diagram is represented by a starting point (entering arrow), an ending point (exiting arrow) and a main branch. A sequence is placed inside that branch, containing three functions: F1, F2 and F3. In this example, input and output object flows are not represented in order to ease the readability of the figure. The execution occurs as detailed hereafter. Each Construct controls the execution of Branches and Constructs it contains. So, the diagram starts the main branch which starts the sequence. Since a sequence contains functions, it also controls their execution. First, it starts the beginning function (F1), and afterwards it waits for finished functions, to end their execution and to start the execution of following functions (end F1 and start F2). This process is repeated until the ending function (F3) ends and then the sequence itself finishes execution. The main branch ends the sequence, before finishing its proper execution. The diagram waits for the main branch to finish execution, in order to end it. The end of the main branch execution means that the diagram can first, finish and then end its execution.

Functions are contained in a sequence, so the formal definition of their execution starting and ending is defined by the dynamic behavior of the Sequence construct as described in previous example. Next, due to lack of space we formally define only a dynamic behavior of functions using the previously described property-driven approach. An input/output transformation described by the Function is first possible i.e. the function can start but has to wait for Items and eventually Resources (Figure 6, Eq.1) before being able to make the real transformation of energy, material and / or data (Figure 6, Eq.2) providing then the outputs items and resources and finishing its execution respecting minimal and maximal execution time (Figure 6, Eq.3).

For f  Function (Eq. 1) { (f.state==authorised) AND (  i  f.itemInputs,(i.state==present)) AND (  j  f. resourceFlowInputs, ( (j.requestedQuantity >= j.sourceResource.availableQuantity) AND (j.requestedQuality == j.sourceResource.quality)))) implies executeFunction(f) } (Eq. 2) { (f.state==execution) implies ( ( i  f. itemInputs, (consumeItem(i))) AND ( j  f. resourceFlowInputs, (j.sourceResource.availableQuantity -= j.requestedQuantity)) } (Eq. 3)

{ ((f.state==execution) AND ((internalTime -f.startedTime) >= minimalTime) AND ((internalTime -f.startedTime) <= maximalTime)) implies ( finishFunction(f)) } (Eq. 4)

{ (f.state==finished) implies ( (  i  f. itemOutputs, (provideItem(i))) AND ( j  f. resourceFlowOutputs, (j.targetResource.availableQuantity += j.providedQuantity)))} Let us note that due to external events, a function can be suspended temporarily, can resume its execution or can abort (Suspended, Aborted). These external events can be then shared with other constructs from other modeling languages. For instance, the function behavior can depend on the component behavior that performs this function. So, the event Suspended can be a common event shared between xeFFBD and a future DSML named xPBD (executable Physical Block Diagram currently under study).

Application discussion and expected contributions

State notion and formalization. Considered approach describes concepts' execution as successive state change. This induces to define a set of states considered by the user as sufficient for his V&V objectives. Unfortunately, some concepts, such as Resource from xeFFBD, may be characterized by a continuum of states. For this, we propose three solutions. The first solution consists in defining a finite number of states. For instance, resource states model can be reduced to a two state model, containing: sufficient or insufficient states. However, this solution is too limitative for V&V objectives. The second solution consists in introducing a set of state variables describing possibly infinite number of states that can evolve continuously. This solution allows describing high level of detail. For instance resource state model can be represented by quality and quantity variables. The third one consists in mixing the previous two solutions, linking discrete states defined in the state model and state variables. For instance, Item state model is composed of a state quality variable and two states: present and absent. These three solutions are applied in SDMM package and are now under development.

Towards condition and event based transition approach. In order to understand concepts' evolution, one has to simultaneously visit three packages: DDMM, SDMM and EDMM, and the evolution rules defined in the semantics package. We consider that this makes created languages difficult to read and understand. In order to ease readability and improve understandability, we propose a representation of previously stated packages, using finite state automata. Conditions and events are responsible for a transition firing. First, a Condition (True by default) is a Boolean function computed on variables, attributes of any concept from the local DDMM and external variables corresponding to other concepts from another DDMM. So inter and intra conditions are distinguished. Intra conditions have to be satisfied by a currently manipulated model, while inter conditions correspond to conditions that have to be satisfied by one or several other models from the same SOI whose behavior interacts with the behavior of studied model. Second, it is always possible to distinguish two events and there exist a default event e always occurring. A Transition can be then fired when associated event is received, and if and only if, associated condition is verified. The work now consists of formalizing these notions and linking the transition behavior with discrete event system theory.

Towards model transient states detection and management. Temporal evolution rules (named properties by [START_REF] Combemale | Approche de métamodélisation pour la simulation et la vérification de modèle --Application à l'ingénierie des procédés[END_REF]) are currently defined using Temporal OCL (TOCL). This induces the examination of defined properties taking into account a unique scale of time. However, the notion of "model stability" is by hypothesis essential for models representing critical, parallel or distributed systems. A "transient state" of a concept is a state such that it is possible to change that state without modifying the inputs, as defined for instance in the case of Sequential Function Chart in [START_REF]Specification language GRAFCET for sequential function charts[END_REF]. A model is stable if and only if each instance of a modeling concept used in this model is itself in stable state. We propose here to extend considered approach by a double scale of time named external and internal time modeled by two independent logical clocks. Values of each variable vi appearing in conditions and occurring events associated to transitions of a state model M are read and then frozen in external time. M evolves taking into account vi by using then an internal scale when performing execution rules allowing then to detect transient states and to reach the next stable state of M. The external time depends on environment evolution and is a logical modeling of physical scale time. It is defined as a set of moments ordered by taking into account events apparition. It is initialized when a simulation starts. The internal time is initialized at each moment defined in external time and there are no common temporal dimensions between internal and external scales. The evolution algorithm allowing transient states detection is schematized in Figure 7.

Towards properties modeling languages and checking techniques. Literature highlights several property-driven approaches with associated V&V techniques [START_REF] Dasgupta | A roadmap for formal property verification[END_REF], [START_REF] Chapurlat | UPSL-SE: A model verification framework for Systems Engineering[END_REF] that will be explored and applied in the prosed frame of work.

Towards modeling languages and models interoperability. As illustrated in Section 3, we propose to become able to link formally the resulting interpretation and execution of several DMSLs each dedicated to the description and the analysis of a view of a given SOI (behavioral, physical, functional, etc.). This will allow contributing to become able to check the coherence of SOI models even considering different points of view and different modeling objectives. It is a question of linking the dynamic semantics i.e. SDMM and EDMM have to be extended introducing requested and shared concepts and evolution rules.

Tooling. Unfortunately, tools supporting the considered approach do not exist at this moment. An extension of Diagraph [START_REF] Pfister | A light-weight annotation-based solution to design Domain Specific Graphical Modeling Languages[END_REF] is now under development taking into account proposed improvements.

Conclusion and Perspectives

This paper presents an approach from the field of MDE for defining semantic of a DSML. The approach is considered here as a formal and relevant way for achieving models V&V objectives. It is applied to a functional modeling language largely used in MBSE domain. This application however, makes appear some questions that seem crucial and remain partially or completely uncovered. Conceptual as technical improvements are then proposed in order to complement this approach. The research and development work is now on going intending to fully support executable DSMLs creation process and to deploy it on MBSE domain.

Figure 1 :

 1 Figure1: The executable DSML Pattern[START_REF] Combemale | A Design Pattern for Executable DSML[END_REF]

Figure 2 :

 2 Figure 2: eFFBD example Our proposition to achieve executable eFFBD along the above discussed methodology is presented in the following section.Creating an executable DSML is divided into two major phases. First, a language executable metamodel is defined containing domain (in DDMM), execution-related

Figure 3 :

 3 Figure 3: Creating executable metamodel for eFFBD construction stages

Figure 4 :

 4 Figure 4: A finite states automate for the concept Function representing state evolution as transition firing

F1Figure 5 :

 5 Figure 5: execution of a simple exFFBD model

Figure 6 :

 6 Figure 6: Semantics mapping by defining evolution properties of Function concept

Figure 7 :

 7 Figure 7: proposed evolution algorithm including stability reaching objectives