Babajide O Ayinde
email: babajide.ayinde@louisville.edu

Keishin Nishihama

Jacek M Zurada
email: jacek.zurada@louisville.edu

Diversity Regularized Adversarial Deep Learning

Keywords: Deep learning, feature correlation, generative model, adversarial learning, feature redundancy, generative adversarial networks, regularization

The two key players in Generative Adversarial Networks (GANs), the discriminator and generator, are usually parameterized as deep neural networks (DNNs). On many generative tasks, GANs achieve state-of-the-art performance but are often unstable to train and sometimes miss modes. A typical failure mode is the collapse of the generator to a single parameter configuration where its outputs are identical. When this collapse occurs, the gradient of the discriminator may point in similar directions for many similar points. We hypothesize that some of these shortcomings are in part due to primitive and redundant features extracted by discriminator and this can easily make the training stuck. We present a novel approach for regularizing adversarial models by enforcing diverse feature learning. In order to do this, both generator and discriminator are regularized by penalizing both negatively and positively correlated features according to their differentiation and based on their relative cosine distances. In addition to the gradient information from the adversarial loss made available by the discriminator, diversity regularization also ensures that a more stable gradient is provided to update both the generator and discriminator. Results indicate our regularizer enforces diverse features, stabilizes training, and improves image synthesis.

Introduction

Convolutional neural networks (CNNs) have become the powerhouse for tackling many image processing and computer vision tasks. By design, CNNs learn to automatically optimize a well-defined objective function that quantifies the quality of results and their performance on the task at hand. As shown in previous studies [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF], designing effective loss functions for many image prediction problems is daunting and often requires manual effort and in-depth experts' knowledge and insights. For instance, naively minimizing the Euclidean distance between predicted and ground truth pixels have shown to result in blurry outputs since the Euclidean distance is minimized by averaging all conceivable outputs [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF][START_REF] Pathak | Context encoders: Feature learning by inpainting[END_REF][START_REF] Zhang | Colorful image colorization[END_REF][START_REF] Ayinde | Deep learning of constrained autoencoders for enhanced understanding of data[END_REF]. One plausible way of training models with high-level objective specifications is by allowing CNNs to automatically learn the appropriate loss functions that satisfy these desired objectives. One of such objectives could be as simple as asking the model to make the output not distinguishable from the groundtruth.

As established in [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF][START_REF] Goodfellow | Generative adversarial nets[END_REF][START_REF] Radford | Unsupervised representation learning with deep convolutional generative adversarial networks[END_REF][START_REF] Salimans | Improved techniques for training gans[END_REF], GANs are trained to automatically learn an objective function using a discriminator network to classify if its input is real or synthesized while simultaneously training a generative model to minimize the loss. In GAN framework, both the discriminator and generator aim to minimize their own loss and the solution to the game is the Nash equilibrium where neither player can independently improve their individual loss [START_REF] Goodfellow | Generative adversarial nets[END_REF][START_REF] Kurach | The gan landscape: Losses, architectures, regularization, and normalization[END_REF]. This framework can also be interpreted from the viewpoint of a statistical divergence minimization between the learned model distribution and the true data distribution [START_REF] Nowozin | f-gan: Training generative neural samplers using variational divergence minimization[END_REF][START_REF] Arjovsky | Wasserstein generative adversarial networks[END_REF][START_REF] Mao | Least squares generative adversarial networks[END_REF].

Even though GANs have resulted in new and interesting applications and achieved promising performance, they are still hard to train and very sensitive to hyperparameter tuning. A peculiar and common training challenge is the performance control of the discriminator. The discriminator is usually inaccurate and unstable in estimating density ratio in high dimensional spaces, thus leading to situations where the generator finds it difficult to model the multi-modal landscape in true data distribution. In the event of total disjoint between the supports of model and true distributions, a discriminator can trivially distinguish between model distribution and that of true data [START_REF] Arjovsky | Towards principled methods for training generative adversarial networks[END_REF], thus leading to situations where generator stops training because the derivative of the resulting discriminator with respect to the input has vanished. This problem has seen many recent works to come up with workable heuristics to address many training problems such as mode collapse and missing modes.

We argue in line with the hypothesis that some of the problems associated with the training of GANs are in part due to lack of control of the discriminator. In light of this, we propose a simple yet powerful diversity regularizer for training GANs that encourages the discriminator to extract near-orthogonal filters. The problem abstraction is that in addition to the gradient information from the adversarial loss made available by the discriminator, we also want the GAN system to benefit from extracting diverse features in the discriminator. Experimental results consistently show that, when correctly applied, the proposed regularization enforces diverse features in the discriminator and better stabilize the GAN training with mostly positive effects on the generated samples.

The contribution of this work is two-fold: (i) we propose a new method to regularize adversarial learning by inhibiting the learning of redundant features and availing a stable gradient for weights updates during training and (ii) we show that the proposed method stabilizes the adversarial training and enhances the performance of many state-of-the-art methods across many benchmark datasets. The rest of the paper is structured as follows: Section II highlights the state-of-the-art and Section III discusses in detail the formulation of diversity-regularized adversarial learning. Section IV discusses the detailed experimental designs and presents the results. Finally, conclusions are drawn in Section V.

Related Work

As originally introduced in [START_REF] Goodfellow | Generative adversarial nets[END_REF], GANs consist of generator and the discriminator that are parameterized by deep neural networks and are capable of synthesizing interesting local structure on select datasets. The representation capacity of original GAN was extended in conditional GANs [START_REF] Mirza | Conditional generative adversarial nets[END_REF] by incorporating an additional vector that enables the generator to synthesize samples conditioned on some useful information. This extension has motivated several conditional variants of GAN in diverse applications such as edge map [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF][START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF], image synthesis from text [START_REF] Reed | Generative adversarial text to image synthesis[END_REF], super-resolution [START_REF] Ledig | Photo-realistic single image super-resolution using a generative adversarial network[END_REF], style transfer [START_REF] Azadi | Multi-content gan for few-shot font style transfer[END_REF], just to mention a few. Learning useful representation with GANs has shown to heavily rely on hyperparameter-tuning due to various instability issues during training [START_REF] Kurach | The gan landscape: Losses, architectures, regularization, and normalization[END_REF][START_REF] Arjovsky | Towards principled methods for training generative adversarial networks[END_REF][START_REF] Grewal | Variance regularizing adversarial learning[END_REF]. GANs are remarkably hard to train in spite of their success on variety of task. Robustly and systematically stabilizing the training of GANs has come in many forms such as selective architectural design [START_REF] Radford | Unsupervised representation learning with deep convolutional generative adversarial networks[END_REF], matching of intermediate features [START_REF] Salimans | Improved techniques for training gans[END_REF], and unrolling the optimization of discriminator [START_REF] Metz | Unrolled generative adversarial networks[END_REF]. Many recent advances inspired by either theoretical insights or practical considerations have been attempted in form of regularization and normalization to address some of the issues associated with training of GANs. Imposing Lipschitz constraint on the discriminator has shown to stabilize the adversarial training and avoid an overoptimization scenario where the discriminator still distinguishes and allots different scores to nearly indistinguishable samples [START_REF] Arjovsky | Towards principled methods for training generative adversarial networks[END_REF]. By satisfying the Lipschitz constraint, the discriminator's joint/compressed representation of the true and synthesized data distributions is guaranteed to be smooth; thus ensuring a non-zero learning signal for the generator [START_REF] Arjovsky | Towards principled methods for training generative adversarial networks[END_REF][START_REF] Grewal | Variance regularizing adversarial learning[END_REF]. Enforcing the discriminator to satisfy the Lipschitz constraints has been approximated and implemented via ancillary means such as gradient penalties [START_REF] Gulrajani | Improved training of wasserstein gans[END_REF] and weight clipping [START_REF] Arjovsky | Towards principled methods for training generative adversarial networks[END_REF]. Using a Gaussian classifier over the real/fake indicator variables has also been shown to have a smoothing effect on the discriminator function [START_REF] Grewal | Variance regularizing adversarial learning[END_REF]. Injecting label noise [START_REF] Salimans | Improved techniques for training gans[END_REF] and gradient penalty have equally been shown to have a tremendous regularizing effect on GANs. Schemes such as weighted gradient [START_REF] Roth | Stabilizing training of generative adversarial networks through regularization[END_REF] and missing modes penalty [START_REF] Che | Mode regularized generative adversarial networks[END_REF] have been utilized to alleviate some training and missing modes issues in GAN learning.

Weight vectors of discriminator have been l 2 -normalized with Frobenius norm, which constraints the sum of the squared singular values of the weight matrix to be 1 [START_REF] Salimans | Improved techniques for training gans[END_REF]. However, normalizing using Frobenius norm translates to utilizing a single feature to discriminate the model probability distribution from the target thus, reducing the rank and hence the number of discriminator features [START_REF] Miyato | Spectral normalization for generative adversarial networks[END_REF]. In addition to weight clipping [START_REF] Arjovsky | Wasserstein generative adversarial networks[END_REF][START_REF] Arjovsky | Towards principled methods for training generative adversarial networks[END_REF], weight normalization approaches yield primitive discriminator model that maps the target distribution only with select few features. The most closely related work to ours is orthonormal regularization of weights [START_REF] Brock | Neural photo editing with introspective adversarial networks[END_REF] that sets all the singular values of weight matrix in the discriminator to one, which translates to using as many features as possible to distinguish the generator distribution from the target distribution. Our approach, however, imposes much softer orthogonality constraint on the weight vectors by allowing a degree of feature sharing in upper layers of the discriminators. Other related work is spectral normalization of weights that guarantees 1-Lipschitzness for linear layers and ReLu activation units resulting in discriminators of higher rank [START_REF] Miyato | Spectral normalization for generative adversarial networks[END_REF]. The advantage of spectral normalization is that weight matrices are constrained and Lipschitz. However, bounding the spectral norm of the convolutional kernel to 1 does not bound the spectral norm of the convolutional mapping to unity.

Method

The training of GAN can be abstracted as a non-cooperative game between two players, namely the generator G and the discriminator D. The discriminator tries to distinguish if the generated sample is from the real (p data ) or fake data distribution (p z ), while G tries to trick D into believing that generated sample is from p data by moving the generation manifold towards the data manifold. The discriminator aims to maximize E x∼p data (x) [logD(x)] when the input is sampled from real distribution and given a fake image sample G(z), z ∼ p z (z), it is trained to output probability, D(G(z)), close to zero by maximizing

E z∼pz(z) [log(1 -D(G(z)))].
The generator network, however, is trained to maximize the chances of D producing a high probability for a fake image sample G(z) thus by minimizing

E z∼pz [log(1 -D(G(z)))].
The adversarial cost is obtained by combining the objectives of both D and G in a min-max game as given in 1 below:

J adv = min G max D E x∼p data (x) [logD(x)] + E z∼pz(z) [log(1 -D(G(z)))] (1)
Training D can be conceived as training an evaluation metric on sample space [START_REF] Che | Mode regularized generative adversarial networks[END_REF] that enables G to use the local gradient ∇ log D(G(z)) information made available by D to improve itself and move closer to the data manifold.

Feature diversification in GAN

Both D and G are commonly parameterized as DNNs and over the past few years, the general trend has been that DNNs have grown deeper, amounting to huge increase in number of parameters. The number of parameters in DNNs is usually very large offering possibility to learn very flexible high-performing models [START_REF] Liu | Pruning deep neural networks by optimal brain damage[END_REF]. Observations from many previous studies [START_REF] Xie | On the generalization error bounds of neural networks under diversity-inducing mutual angular regularization[END_REF][START_REF] Rodríguez | Regularizing cnns with locally constrained decorrelations[END_REF][START_REF] Dundar | Convolutional clustering for unsupervised learning[END_REF][START_REF] Ayinde | Building efficient convnets using redundant feature pruning[END_REF] suggest that layers of DNNs typically rely on many redundant filters that can be either shifted version of each other or be very similar with little or no variations. For instance, this redundancy is evidently pronounced in filters of AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] as emphasized in [START_REF] Rodríguez | Regularizing cnns with locally constrained decorrelations[END_REF][START_REF] Zeiler | Visualizing and understanding convolutional networks[END_REF][START_REF] Ayinde | Clustering of receptive fields in autoencoders[END_REF]. To address this redundancy problem, we train layers of the discriminator under specific and well-defined diversity constraints.

Since G and D rely on many redundant filters, we regularize them during training to provide more stable gradient to update both G and D. Our regularizer enforces constraints on the learning process by simply encouraging diverse filtering and discourages D from extracting redundant filters. We remark that convolutional filtering has found to greatly benefit from diversity or orthogonality of filters because it can alleviate problems of gradient vanishing or exploding [START_REF] Brock | Neural photo editing with introspective adversarial networks[END_REF][START_REF] Ayinde | Regularizing deep neural networks by enhancing diversity in feature extraction[END_REF][START_REF] Saxe | Exact solutions to the nonlinear dynamics of learning in deep linear neural networks[END_REF][START_REF] Ayinde | Nonredundant sparse feature extraction using autoencoders with receptive fields clustering[END_REF].

Typically, both D and G consist of input, output, and many intermediate processing layers. By letting the number of channels, height, and width of input feature map for l th layer be denoted as n l , h l , and w l , respectively. A convolutional layer in both D transforms input x l ∈ R p into output x l+1 ∈ R q , where x l+1 is the input to layer l + 1; p and q are given as n l × h l × w l and n l+1 × h l+1 × w l+1 , respectively. x l is convolved with n l+1 3D filters χ ∈ R n l ×k×k , resulting in n l+1 output feature maps. Unrolling and combining all layer l th filters into a single matrix results in kernel matrix

θ D i ∈ R m corresponds to the i-th column of the kernel matrix

(l) Θ D = [ (l) θ D 1 , ... (l) θ D n l ] ∈ R m×n l+1
; the bias term of each layer is omitted for simplicity.

Given that

(l)
Θ D ∈ R m×n l contain n l normalized filter vectors as columns, each with m elements corresponding to connections from layer l -1 to i th neuron of layer l, then, the diversity loss J D for all layers of D is given as:

J D (θ D ) = L l=1   1 2 n l i=1 n l j=1 (l) Ω D ij 2 (l) M D ij   (2)
where

(l) Ω D ∈ R n l ×n l denotes ( (l) Θ D ) T (l)
Θ D which contains the inner products of each pair of columns i and j of (l)

Θ D in each position i,j of (l)
Ω D in layer l;

(l)

M D ∈ R n l ×n l is a binary mask for layer l defined in [START_REF] Goodfellow | Generative adversarial nets[END_REF]; L is the number of layers to be regularized.

(l)

M D ij = 1 τ ≤ | (l) Ω D ij | ≤ 1 0 i = j 0 otherwise (3)
Similarly, the diversity loss J G for generator G is given as:

J G (θ G ) = L l=1   1 2 n l i=1 n l j=1 (l) Ω G ij 2 (l) M G ij   (4) and (l)
M G ij = 1 τ ≤ | (l) Ω G ij | ≤ 1 0 i = j 0 otherwise (5)
In order to enforce feature diversity in both G and D while training GANs, the diversity regularization terms in (4) is added to the conventional adversarial cost J adv in (1) as given in [START_REF] Radford | Unsupervised representation learning with deep convolutional generative adversarial networks[END_REF].

J net = J adv + J div (6)
where

J div = λ G J G (θ G ) -λ D J D (θ D
), λ G and λ D is the diversity penalty factors for generator and discriminator, respectively. The derivative of diversity loss J D with respect to weights of D is given as

∇ Θ (l) i,j J D (θ D ) = n k=1 (l) Θ D i,k (l)
Ω D k,j (l)
M D k,j (7)
and the derivative of diversity loss J G with respect to weights of G is

∇ Θ (l) i,j J G (θ G ) = n k=1 (l) Θ G i,k (l) Ω G k,j (l)
M G k,j (8)
The idea behind diversifying features is that in addition to adversarial gradient information provided by D, we provide additional diversity loss with more stable gradient to refine both G and D. The diversity loss encourages weights of both generator and discriminator to be diverse by pushing them towards the nearest orthogonal manifold. Our proposed regularization provides more efficient gradient flow, a more stable optimization, richness of layer-wise features of resulting model, and improved sample quality compared to benchmarks and baseline. The diversity regularization ensures the column space of (l)

Θ D and (l)
Θ G for l th layer does not concentrate in few direction during training thus preventing them to be sensitive in few and limited directions. The proposed diversity regularized adversarial learning alleviates some of the main failure mode of GAN by ensuring features are diverse.

Experiments

All experiments were performed on Intel(r) Core(TM) i7-6700 CPU @ 3.40Ghz and a 64GB of RAM running a 64-bit Ubuntu 16.04 edition. The software implementation has been in PyTorch library1 on two Titan X 12GB GPUs. Implementation of DiReAL is available at https://github.com/keishinkickback/DiReAL. Diversity regularized adversarial learning (DiReAL) was evaluated on MNIST dataset of handwritten digits [START_REF] Lecun | The mnist database of handwritten digits[END_REF], CIFAR-10 [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF], STL-10 [START_REF] Coates | An analysis of single-layer networks in unsupervised feature learning[END_REF], and Celeb-A [START_REF] Liu | Deep learning face attributes in the wild[END_REF] databases. In the first set of experiments, an ubiquitous deep convolutional GAN (DCGAN) in [START_REF] Radford | Unsupervised representation learning with deep convolutional generative adversarial networks[END_REF] was trained using MNIST digits. The standard MNIST dataset has 60000 training and 10000 testing examples. Each example is a grayscale image of an handwritten digit scaled and centered in a 28 × 28 pixel box. Both the discriminator and generator networks contain 5 layers of convolutional block. Adam optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] with batch size of 64 was used to train the model for 100 epochs and τ and learning rate in DiReAL were set to 0.5 and 0.0001, respectively. In similar vein, λ D and λ G were to 1.0 and 0.01, respectively. Adam optimizer (β 1 = 0.0, β 2 = 0.9) [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] with batch size of 64 was used to train the model for 100 epochs Fig. 2 shows the diversity loss of both generator and discriminator for DiReAL and unregularized counterpart. It can be observed that DiReAL was able to minimize the pairwise feature correlations compared to the highly correlated features extracted by the unregularized counterpart. Specifically, DiReAL was able to steadily minimize the diversity loss as training progresses compared to the unregularized DCGAN, where extraction of similar features grows with epoch of training, thus increasing the diversity loss. The divergence between discriminator output for real handwritten digits and generated samples over 30 batches for regularized and the unregularized networks is shown in Fig. 3a. The divergence was measured using the Wasserstein distance measure [START_REF] Vallender | Calculation of the wasserstein distance between probability distributions on the line[END_REF] and it can be observed that the regularizing effect of DiReAL stabilizes the adversarial training and prevents mode collapse. For unregularized network, however, the mode started to collapse around 45th epoch. Closer look into the diversity of the generator in Fig. 2a, it is evident that just around the epoch of collapse the generator starts extracting more and more redundant filters. We suspect that DiReAL was able to stabilize the training by pushing features to lie close to the orthogonal manifold, thus preventing learned features from collapsing to an undesirable manifold. Fig. 3b shows the handwritten digit samples synthesized with and without DiReAL and it can be observed that diversification of features is beneficial for stabilizing adversarial learning and ultimately improving the samples' quality. Another observation is that DiReAL also prevents learned weights from collapsing to an undesirable manifold thus highlighting some of the benefits of pushing weights near the orthogonal manifold.

In the second large-scale experiments, CIFAR-10 dataset was used to train GAN using DiReAL and the results compared to the unregularized training. The dataset is split into 50000 and 10000 training and testing sets, respectively. Similar to experiments with MNIST, Fig. 4b shows the diversity loss of the discriminator with and without DiReAL trained on CIFAR-10 database. It can be observed that DiReAL was able to minimize the diversity loss and encourages diverse features that benefit the adversarial training. On the other hand, Fig. 4b shows that the diversity loss of the unregularized is higher and unconstrained compared to that of DiReAL. The images Method Inception Score Real data 9.04 -Standard CNN-Unregularized [START_REF] Radford | Unsupervised representation learning with deep convolutional generative adversarial networks[END_REF] 4.00 ± 0.15 DiReAL (ours) 4.17 ± 0.03 Batch Normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] 5.48 ± 0.19 Layer Normalization [START_REF] Ba | Layer normalization[END_REF] 5.05 ± 0.12 Weight Normalization [START_REF] Salimans | Weight normalization: A simple reparameterization to accelerate training of deep neural networks[END_REF] 4.66 ± 0.14 Spectral Normalization [START_REF] Miyato | Spectral normalization for generative adversarial networks[END_REF] 6.50 ± 0. synthesized with DiReAL was compared and contrasted with state-of-the-art methods such as batch normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF], layer normalization [START_REF] Ba | Layer normalization[END_REF], weight normalization [START_REF] Salimans | Weight normalization: A simple reparameterization to accelerate training of deep neural networks[END_REF], and spectral normalization [START_REF] Miyato | Spectral normalization for generative adversarial networks[END_REF]. It is remarked that DiReAL can be used in tandem with the other regularization techniques and could also be deployed as stand-alone regularization tool for stabilizing adversarial learning. In this light, DiReAL was also combined with these techniques. It must be noted that spectral normalization uses a variant of DCGAN architecture with an eight-layer discriminator network. See [START_REF] Miyato | Spectral normalization for generative adversarial networks[END_REF] for more implementation details.

It can be observed in Fig. 5 that diversity regularization was able to synthesize more diverse and complex images compared to unregularized counterpart. Other benchmark regularizers were able to generate better image samples compared to using only Di-ReAL. However, when DiReAL was combined with other regularizers the quality of the generated samples was significantly improved. For quantitative evaluation of generated examples, inception score metric [START_REF] Salimans | Weight normalization: A simple reparameterization to accelerate training of deep neural networks[END_REF] was used. Inception score has been found to highly correlate with with subjective human judgment of image quality [START_REF] Miyato | Spectral normalization for generative adversarial networks[END_REF][START_REF] Salimans | Weight normalization: A simple reparameterization to accelerate training of deep neural networks[END_REF]. Similar to [START_REF] Miyato | Spectral normalization for generative adversarial networks[END_REF][START_REF] Salimans | Weight normalization: A simple reparameterization to accelerate training of deep neural networks[END_REF], inception score was computed for 5000 synthesized images using generators trained with each regularization technique. Every run of the experiment is repeated five times and averaged to combat the effect of random initialization. The average and the standard deviation of the inception scores are reported.

The proposed regularization is also compared and contrasted in terms inception score with many benchmark methods as summarized in Table 1. It can be again observed that DiReAL was able to improve the image generation quality compared to unregularized counterpart and when combined with spectral normalization, we observed a 6% improvement in the inception score. By combining DiReAL with layer normalization, an improvement of 11.68% on inception was observed. However, no significant improvement was observed when DiReAL was combined with batch normalization and weight normalization. It must be remarked that the calculation of Inception Scores is library dependent and that is why the scores reported in Table 1 is different for those reported by Miyato et al. [START_REF] Miyato | Spectral normalization for generative adversarial networks[END_REF]. While our implementation was in PyTorch, [START_REF] Miyato | Spectral normalization for generative adversarial networks[END_REF] was in Chainer2 .

In the next set of large-scale experiments, STL-10 dataset was used to train generator under diversity regularization and compared with other state-of-the-art regularization techniques. As can be observed in Fig. 6, images synthesized by generator trained with DiReAL was able to generate images with competitive quality in comparison with other regularization methods considered. Performance of DiReAL was also observed to be competitive to regularization methods such as WGAN-GP and spectral normalization. In Fig. 7 we show the images produced by the generators trained with DiReAL using Celeb-A dataset. It can be again be observed that DiReAL was able to stabilize the training and avoid mode collapse in comparison to the unregularized counterpart.

Conclusion

This paper proposes an interesting and effective method of stabilizing the training of GANs using diversity regularization to penalize both negatively and positively correlated features according to features differentiation and based on features relative cosine distances. It has been shown that diversity regularization can help alleviate a common failure mode where the generator collapses to a single parameter configuration and outputs identical points. This has been achieved by providing additional stable diversity gradient information in addition to adversarial gradient information to update both the generator and discriminator's features. The performance of the proposed regularization in terms of extracting diverse features and improving adversarial learning was compared on the basis of image synthesis with recent regularization techniques namely batch normalization, layer normalization, weight normalization, weight clipping, WGAN-GP, and spectral normalization. It has also been shown on select examples that extraction of diverse features improves the quality of image generation, especially when used in combination with spectral normalization. This concept is illustrated using MNIST handwritten digits, CIFAR-10, STL-10, and Celeb-A Dataset.

Fig. 1 .

 1 Fig. 1. Schema of Diversity Regularized Adversarial Learning (DiReAL)

Θ

  D ∈ R m×n l+1 where m = k 2 n l . Then, (l) θ D i , i=1,...n l , denotes filters in layer l, each

Fig. 2 .

 2 Fig. 2. Diversity loss of (a) generator JG with no regularization (b) generator JG with diReAL (c) discriminator JD with no regularization, and (d) discriminator JD with DiReAL trained on MNIST dataset.

Fig. 3 .Fig. 4 .

 34 Fig. 3. (a) Divergence, as measured by Wasserstein distance, between the discriminator output for synthesized and real MNIST samples (b) Synthesized hand-written digits with and without diversity regularization.

Fig. 5 .

 5 Fig. 5. Generated images with and without DiReAL trained on CIFAR-10 dataset.

Fig. 6 .

 6 Fig. 6. Qualitative comparison of generated images with four regularization techniques for models trained on STL-10 dataset.

Fig. 7 .

 7 Fig. 7. Generated images with and without diversity Regularization trained on CELEB-A dataset.

Table 1 .

 1 Inception Scores with unsupervised image generation on CIFAR-10

	30

https://pytorch.org/

https://chainer.org/

Acknowledgement

This work was supported partially by the NSF under grant 1641042.


Alexander Wendt
email: alexander.wendt@tuwien.ac.at

Maximilian Götzinger
email: maximilian.goetzinger@tuwien.ac.at

Thilo Sauter
email: thilo.sauter@tuwien.ac.at

An Agent-Based Framework for Complex Networks

Keywords: Multi-Agent System, ACONA, Middleware, Communication, Industry 4.0, Interoperability, Distributed Application Platform, Internet of Things, Self-adaption, MQTT. 1

This paper presents the ACONA Framework (Agent-based COmplex Network Architecture). It is an agent-based middleware for component-based software development. Its purpose, which particularly applies to research groups with an interdisciplinary focus, is to provide a general middleware for several domains instead of several specialized middleware for each domain. Therefore, the design emphasizes a lightweight and flexible infrastructure. Its performance is demonstrated in three applications: A cognitive architecture with around 40 interconnected modules; a stock market simulator with elements of evolutionary programming; and an industry 4.0 application of a conveyor belt.

Introduction

In a heterogeneous system, each system module might be developed in a particular language. The modules might be running in a distributed fashion. Modules have individual tasks like being device drivers or controllers for applications. Many research and industrial projects show a demand for module-based development.

In many cases, it is enough to use some message bus, which provides essential means of communication. The developer then implements the custom communication methods tailored for each module. In the case of a typical Smart Grid [START_REF] Kollmann | Local intelligence for a customer energy management system equipped with smart breakers[END_REF] and IoT [START_REF] Weyrich | Reference architectures for the internet of things[END_REF] applications, a message bus for basic communication between sensors and actuators is often enough. Optionally, a higher instance provides the application with directives on how to control the system.

However, in projects that implement multi-agent systems, simulations [START_REF] Luke | Mason: A multiagent simulation environment[END_REF] or cognitive architectures [START_REF] Wendt | Usage of Cognitive Architectures in the Development of Industrial Applications[END_REF], a bare message bus as a middleware requires a very high effort of implementing the infrastructure, as they are too simple for the target application. A cognitive architecture consists of up to 30 modules, which run in cycles and interact on-demand. In this type of systems, common programming patterns are repeatedly used. It demands a flexible agent system that offers basic functionality for designing agent functions. It would allow a fast setup of an industrial network.

This paper proposes a solution to these demands, namely, the Agent-based COmplex Network Architecture (ACONA) Framework. It will be shown how an agent-based middleware is created as a layer in the communication protocol MQTT. Typical for a middleware, the developer shall not take care of communication between agents. Instead, the framework provides the necessary functionality to help to implement a selfadapting complex system. Three applications will be discussed to demonstrate the flexibility and performance of the framework.

Related Work

In a smart grid setup, smart meters and other sensors measure the grid's state. Based on these measurements smart breakers connect or disconnect devices if overloads are detected. A SCADA system assesses and displays an overload condition, e.g., through a traffic light system. In these applications, the middleware OpenMuc [START_REF] Kollmann | Local intelligence for a customer energy management system equipped with smart breakers[END_REF] can be used. It relies on a component-based programming framework OSGi1 in Java. Channels available to all components handle the data access, and the core automatically synchronizes data between modules. However, OSGi comes with a crucial drawback for agent systems as it only allows one instance of a message bus to run on a Java VM. It makes it hard to implement parallel working agents. Other middleware solutions in this area are Siemens Gridlink [START_REF] Cejka | A framework for communication and provisioning in an intelligent secondary substation[END_REF] and TU Wien Demo Facility [START_REF] Wendt | Software architecture for a smart grids test facility[END_REF], which both provide a basic low-level infrastructure based on either VertX2 and Hazelcast3 or Google Protobuf 4 . These frameworks offer enough functionality to manage basic communication between modules. However, they are not suited for creating a hierarchical agent system, i.e., agents with sub-functions.

A framework like the LIDA framework [START_REF] Snaider | The LIDA framework as a general tool for AGI[END_REF] could be used to create a complex agent system like a cognitive architecture. Unfortunately, it uses the observer pattern between modules, which is fast but also implies a strong coupling between modules. MASON [START_REF] Luke | Mason: A multiagent simulation environment[END_REF] is a Java-based simulation tool that offers an extensive user interface and a world simulator, where actors can interact. Agents are stepped through serially in each simulation cycle. Compared to MASON, ACONA shall provide agents, whose functionality is loosely coupled, i.e., the agents shall not be seen as a single entity, but as a composition of several individual functions.

In the first version, ACONA extended the widely spread multi-agent system Java Jade [START_REF] Wendt | A Multi-Agent-Based Middleware for the Development of Complex Architectures[END_REF], [START_REF] Bellifemine | JADE-a java agent development framework[END_REF]. Jade provides containers equivalent to a message bus, where agents run. Jade agents are threads, which get their functionality from scheduled behaviors. It has the advantage that it is very customizable. It enables negotiation between agents by using the FIPA protocol. On the other side, the developer has to implement all communication from scratch. It turned out that communication methods were hard to separate from behavior classes. The purpose was to hide the communication layer. Finally, Jade is designed to work as a state machine in a single thread per agent, which does not allow blocking calls, making a request-response pattern hard to implement with a method call. In the end, this was not an efficient solution. While Jade proposes a system that is supposed to have many agents with simple functionality, ACONA adds much functionality within a few agents.

ZeroMQ 5 would be an attractive alternative as it offers a broker-less architecture, which fits well into an agent concept. However, each client needs an own port that has to be addressed. The system does not scale well for replicating agents.

ACONA Framework Model

The basic idea is to define an agent, which modifies its internal state or the environment through functions. An agent encapsulates its functions from the environment. Internally, functions share data through a data storage. The communication between functions and other agents applies an onion model, which separates the communication infrastructure from the function logic. To achieve a general agent, which can replicate, all functions can be generated entirely from a textual configuration. Fig. 1 left part shows the components of an agent. Each component will be described in the following chapter.

Functions

Each agent consists of a set of functions; see Fig. 1, right part. Anytime, a function can be registered in an agent instance to add functionality. Each function offers the following features, which implements some common development patterns:

 Services, which are accessible through other functions (request-response pattern)  A thread, which runs code on demand or scheduled (parallel execution)  Listeners, which are triggered by subscribed data points (publish-subscribe pattern)

Services are accessed through a subscribed data point on a message bus, e.g., <agent1>/function1/service1. A request with parameters in JSON format that is sent to that data point. The service returns a response to a predefined callback data point. Functions offer customized services as well as some default services. Such a service is the "command" service that receives commands to start, stop or pause the function thread. The caller function blocks until it receives a response. The core functionality of a function is the running thread. It consists of three segments: pre-processing for reading data, main execution for processing and post-processing for writing data to destination addresses. A typical case is to combine services with running threads, e.g., in a controller. There, the service blocks the caller by providing a delayed response, while the thread runs its functions independently.

Communication and Data Structures

For the communication between functions, an encapsulated communicator is used. In contrast to the previous implementation of the framework in Java Jade, each function gets its communicator to prevent unintended locks. To make the ACONA framework natively compatible with IoT devices, we used the lightweight MQTT6 protocol. It serializes messages bytes, uses low bandwidth and can cope with large messages.

Because each function needs a communication client and the addressing of clients shall be scalable, we decided to use the Mosquitto7 broker. Thus, it is single threaded and performance could be an issue.

MQTT only implements the publish-subscribe pattern. A communicator client publishes data to an address in the broker. Other clients subscribe to them and are notified by a callback function. However, many use cases need a request-response pattern in their communication functionalities. It is solved by using a callback address for the response. It turns out that the only needed communicator functionality to develop complex systems is to implement a method that executes a service in another function. MQTT topics define the addresses of the functions and their services. As each function belongs to an agent, the first part of the address is the agent name, followed by the function and service name. Messages use the JSON syntax to unify and simplify overhead and allow serialized objects to be passed between functions.

The desired functionality was to have a persistent database to share data between functions. MQTT does not support polling topics without subscribing them. To solve this problem, a key-value database was added to each agent instance. As all functions contain MQTT clients, the agent is defined by a common database. It uses the same addressing syntax as used in MQTT. The agent instance implements an access function for reading, writing, and subscribing values. The message broker handles subscriptions by mirroring those values. Fig. 2 presents an example of how a function <function1> sends a request to its write service. The writing service writes the value <Hello> and returns a response with an acknowledgment to the caller.

Configuration

To provide usage in self-adapting industrial networks or self-healing, the system is completely configurable. A significant feature of ACONA is that all functions that define a particular agent can be composed like Lego blocks in a text-based configuration. This is possible as each function is connected through MQTT and completely decoupled from all other agent functions. Java functions are instantiated per reflections from the class and the function name. Other properties are also passed here.

The composition of functions makes it easy to perform tests by replacing real functions with mockups. The message broker links functions as they share data, which makes the setup of agent networks for prototyping easy.

To enable self-adaptation of an agent system, they are equipped with a replication function. We note that an agent does not make a binary copy of itself, but it instantiates a modifiable configuration of itself or another agent. In the latter case, that would be the functionality of a virus, as it replicates a foreign agent.

Complex Structures: Controller

It is possible to develop complex functionalities like a controller. In this context, a controller is a scheduler, called a codelet handler. Codelets are independent functions that provide only an execution service and register themselves in the codelet handler. Fig. 3 shows such a codelet handler with registered codelets. It provides services to register/unregister and to execute the codelets. Codelets can be executed in parallel or sequence by setting an execution order. All codelets with the same execution order are pooled and run in parallel, and each of these pools is executed sequentially. To instantiate codelets, they are assigned to a codelet handler function in the configuration, where also the execution order is set.

Application and Results

To demonstrate the flexibility and potential of the ACONA framework, it was implemented in three applications in different domains. In each application, the system complexity is presented together with a performance metric.

Cognitive Architecture in Building Automation

Enabling the ACONA framework for implementing applications of cognitive architectures in industrial applications was one primary goal of our work. A system based on cognitive architecture reads sensor inputs and provides options for various actions. These options are based on the inferred beliefs of the environment and current goals. After the system has evaluated by considering cost and gain of all possible options, it executes one of them. Instead of hard-coded functions, most reasoning is done through learning and knowledge retrieval, which makes cognitive architectures general-purpose software [START_REF] Wendt | Usage of Cognitive Architectures in the Development of Industrial Applications[END_REF]. ACONA was used as infrastructure and middleware for the modules in the cognitive architecture of the project KORE [START_REF] Wendt | Usage of Cognitive Architectures in the Development of Industrial Applications[END_REF]. The cognitive architecture, inspired by the cognitive model SiMA [START_REF] Schaat | Interdisciplinary Development and Evaluation of Cognitive Architectures Exemplified with the SiMA Approach[END_REF], was designed as a general-purpose software in the area of building automation. In a simulation of a building, each room was equipped with temperature, CO2, and occupancy detectors as well as ventilation and heating controllers. The setup was stored in an ontology. The research problem was to use the ontology to generate a set of control rules for the building automatically. The objective was to keep the comfort level while reducing energy consumption. The cognitive architecture was implemented as a controller for the generation of rules. It had the task to get optimization requests from a REST interface, select which algorithms to run for the generation and parameterization of the control strategies as well as to evaluate, how well a specific control strategy performed in the simulator. The results showed that a Fig. 4. The cognitive architecture of the project KORE based on the ACONA framework cognitive architecture could be applied, but it lacks efficiency as the functionality could be programmed in a straight forward manner.

Fig. 4 demonstrates the complexity of such a system as it depicts the design of the cognitive architecture. A single ACONA agent implemented a top-codelet handler to run the cognitive process, starting from process step B to I. For each process step, a sub-codelet handler activates several parallel codelets that execute tasks such as generating an option for actions based on beliefs or making evaluating an option based on external user requests. In total, eight codelet handlers were used hierarchically, 37 codelets were running. Additionally, 20 other agent support functions were implemented. The native data storage of the agent was sufficient to use as different short-term memories. The implementation demonstrates how the ACONA framework manages to be used as infrastructure in a comprehensive cognitive architecture.

Self-Evolving Agents in a Multi-Agent System

To explore the potential of the framework regarding self-replication, evolution, and simulation capabilities, a stock market trading game was implemented. In this game, every agent is a trader, which earns money by buying and selling a stock. Real market values represent the stock. For the test, ~5000 closing prices of the OMXS30 index were used. For this simulation, the price patterns are relevant and therefore, data is looped to provide more extended periods than 5000 samples. Each trader uses two exponential moving averages (EMA) as buy and sell signals, i.e., two parameters control its trading behavior. If the shorter EMA > longer EMA, the signal is to buy, otherwise to sell. The configuration defined that if a trader increases his deposit value by 30%, the agent replicates by splitting its depot into a new agent. If the deposit value decreases with 90%, the agent dies. Each of the two trading parameters "mutates" with 30% probability, 51% mutation probability for any change. Agents with equal trading parameters define a species.

An example is a species <L200S50>, where "L" is the long EMA 200 periods and "S" is the short EMA of 50. To measure the success of a species, the number of individuals per species is counted. Species that frequently replicate produce the newest species and those parameters with modifications are inherited. By producing more agents for "fitting" parameters, the evolutionary system explores different local maxima.

Fig. 5 shows the setup of the stock market simulator with the ACONA framework.

Step 1 to 10 describes the simulation process. First, a codelet handler executes the price generator. Then, it triggers the traders on execution order 2. They access the broker on demand for trading. Finally, the evaluation function in the broker agent is executed to update the statistics. The simulation test was executed on a Dell Precision 3510 laptop with a Mosquitto MQTT server. As the system was running, agents replicated as their deposits grew. To test the upper limit of the agent system, the replication process continued until there were so many agents, that the system stopped working correctly. Fig. 6 left shows the results. The duration depends on how many trader agents were instantiated and run in parallel. The duration increases more than linear with the number of agents, probably because more agents need to get data from a single point, the broker. At the same time, the single threaded Mosquitto server has to handle more agents in long lists. The limit was reached after about 9000 steps at around 1670 agents, which manifests in long cycle durations. At that time, ~50000 threads and ~8000 agent functions were running from the ACONA system.

To increase the number of agents, another MQTT broker that runs in more threads and therefore manages more connections that Mosquitto could be chosen. Decoupled functions through MQTT guarantee the modularity, which makes mocking easy. The drawback is that it demands many resources as every function is an own thread and maintains its individual MQTT client. It sets an upper limit to the number of agents that can be used within a multi-agent system. An improvement to increase performance would be to make two types of functions with the same interface. The first function type of function access the MQTT message bus and can be used for external communication. The other function type only accesses other agent internal functions. It would require that each agent maintain its message bus. Another method to increase capacity is to apply the simulations distributed on multiple computers for parallel execution. The simulation also shows interesting results. In Fig. 6 right, the number of agents per species is shown for each simulation step. The legend shows the most successful species, namely <L146S50> and <L95S91> with 21 respectively 17 individuals. These trading strategies would generate the highest profit. On the other hand, most species with only a few agents. With this setup, 61% of the agents belong to species with less than four agents, i.e., they hardly make any profit but do not lose either.

The stock market game shows that the ACONA framework can be used directly as a multi-agent simulator, where each simulator component is a function or an agent. The potential for evolutionary programming is provided through self-modification during replication. In this application, only parameters were modified. In upcoming applications, thanks to the replaceable functions, also the functions themselves will be the topic of modifications.

4.3

Infrastructure in an Industry 4.0 Application

In the project SAVE (Self-monitoring based process Adaptation for quality assurance in heterogeneous Versatile manufacturing). The simulator part represents a conveyor, which translocate conrods for combustion engines between seven machines. The raw part that has 27 features, such as diameters of the holes and the thickness of a conrod. It passes through seven machines, and each machine modifies one or more of these features. The simulator is a closed loop and implements a model of a machine. The model receives commands that influence how features are generated, e.g., the replacement of a drill results in a smaller hole. The goal of this project/system is to reduce the loss of conrods, which is around 3% at project start. The approach is to add a monitoring and control system called an Autonomous Cooperative Object (ACO) to each machine, which analyzes the features. Together with the ACOs of other machines, it then decides about mitigations as anomalies occur. The ACONA framework serves as a fast prototyping simulator for the conveyor and as middleware for the modules used within every ACO. Fig. 7 shows the setup of the system with one machine and one ACO. To the left, ACONA is used to model the simulator and to the right the ACOs. Through the connectivity of the framework, the real ACO system connects to the simulator without any non-ACONA interfaces.

The simulator is started by a controller that triggers several (1..n) machines in a sequence. Each machine gets the conrod from the data storage of the simulator, modify features through a model and puts the conrod back on another address in the memory. The memory, i.e., the data storage, serves as the conveyor belt. Each machine triggers its associated ACO to run and process measured values. For this application, the whole system runs as a state machine, meaning that the ACO is blocking the simulator while running. This is implemented by using a service call to the codelet handler of the connected ACO. As a result, the ACO sends actions that are applied to the machine model in the next step.

To demonstrate the performance of ACONA, similar to the stock market example, a load test was performed on a Dell Precision 3510. In each cycle, a machine with associated ACO was added. A codelet for statistics collection recorded the number of agents and the cycle duration. As Fig. 8 shows, it was possible to create and add 896 machines with ACOs before the cycle duration started scattering. From about 700 machines and ACOs, the duration for completing a cycle increases much more than linearly. Finally, the system consisted of 8000 running ACONA functions, similar to the stock market example.

This implementation of the first mock system in the project SAVE shows that it is possible to generate a very complex architecture of as well the ACO agent and the simulator for the machines. As the project SAVE is still in the design phase, all software modules were programmed mocks in Java. Eventually, the modules of an ACO will be written in Java, C++, and Python.

Conclusion and Future Work

In this paper, we present the agent-based Complex Network Architecture (ACONA) framework. It was initially developed to meet the demands of cognitive architectures in an industrial system, which many other middleware systems can hardly meet it. Agent functions natively offer several common development patterns, reducing the time of having to implement infrastructure. The edge of ACONA in comparison to common middleware is that it provides high flexibility in designing controller systems, simulations, and cognitive architectures. It allows building many types of modular systems and network topologies. Therefore, it qualifies as a middleware in many domains. Through a commonly used communicator, IoT-projects can apply the framework, in many cases without having to write adapters. Another feature that is usually not found in middleware is the ability to apply evolutionary programming in the replication of agents.

The next steps will be to implement clients to allow other programming languages to be used within a single agent. Because functions can be added at runtime, the code is written in other languages than Java will be addable after an agent has started. The potential for further applications of the system is high. In upcoming projects in the Smart Grids domain, the framework will be directly compared to existing solutions. For public use, the ACONA framework is maintained as an open source software at Github8 .

Fig. 1 .

 1 Fig. 1. Left: Agent with its components; right: function features with assigned data points in a data storage

Fig. 2 .

 2 Fig. 2. A function sends a write request to the data storage

Fig. 3 .

 3 Fig. 3. Codelet handler and codelets

Fig. 5 .

 5 Fig. 5. Design of the stock market simulator

Fig. 6 .

 6 Fig. 6. Left: Cycle duration dependency of the number of trader agents; right: development of species per simulation step

Fig. 7 .

 7 Fig. 7. Design of the conveyor belt simulator and controllers

Fig. 8 .

 8 Fig. 8. Cycle duration as a function of the number of machines

https://www.osgi.org

https://vertx.io

https://hazelcast.com

https://developers.google.com/protocol-buffers

http://zeromq.org

http://mqtt.org

https://mosquitto.org

https://github.com/aconaframework/aconaStep Duration[s]

Acknowledgment

We gratefully acknowledge the financial support provided to us by the BMVIT and FFG (Austrian Research Promotion Agency) program Production of the Future in the SAVE project (864883).


Flora Sakketou

Nicholas Ampazis
email: n.ampazis@fme.aegean.gr

On the Invariance of the SELU Activation Function on Algorithm and Hyperparameter Selection in Neural Network Recommenders

Keywords: Recommender Systems, Neural Networks, Activation Functions

In a number of recent studies the Scaled Exponential Linear Unit (SELU) activation function has been shown to automatically regularize network parameters and to make learning robust due to its selfnormalizing properties. In this paper we explore the utilization of SELU in training different neural network architectures for recommender systems and validate that it indeed outperforms other activation functions for these types of problems. More interestingly however, we show that SELU also exhibits performance invariance with regards to the selection of the optimization algorithm and its corresponding hyperparameters. This is clearly demonstrated by a number of experiments which involve a number of activation functions and optimization algorithms for training different neural network architectures on standard recommender systems benchmark datasets.

Introduction

The literature review of studies on recommender systems [START_REF] Koren | The BellKor Solution to the Netflix Grand Prize[END_REF]24,[START_REF] Piotte | The Pragmatic Theory Solution to the Netflix Grand Prize[END_REF], quickly makes apparent that latent factor models are among the ones most suited to the problem. The most successful realizations of latent factor models are based on matrix factorization. In its basic form, matrix factorization for recommender systems describes both users and items by vectors of (latent) factors which are inferred from the user-by-item rating matrix. High correlation between user and item factors leads to a recommendation of an item to a particular user [START_REF] Koren | Matrix factorization techniques for recommender systems[END_REF].

The matrix factorization model can be considered as learning within a multilayer feedforward network framework where all neurons have identity activation functions, and which consists of U inputs, K hidden nodes and I outputs, corresponding to the U users and I items of the recommendation problem [START_REF] Takács | On the Gravity recommendation system[END_REF]. User features U uk can be considered as the weights between the uth input and the kth hidden neuron, and item features I ki as the weights between the kth hidden neuron and the ith output neuron. For the (u, j)th rating, in order to obtain output activations we can set the input x u to 1 and all other x v =u = 0. In this case the network can be trained with Stochastic Gradient Descent (SGD) with regularization. In the testing phase, we set the input as in the training phase, and the network's outputs y i (i = 1 • • • I) predict the active user's rating on the ith item.

In recent years other neural network architectures have been proposed for recommender systems, e.g. Restricted Boltzmann Machines (RBMs) [START_REF] Salakhutdinov | Restricted Boltzmann machines for collaborative filtering[END_REF], Deep AutoEncoders [START_REF] Kuchaiev | Training Deep AutoEncoders for Collaborative Filtering[END_REF], and Variational AutoEncoders (VAE) [START_REF] Liang | Variational autoencoders for collaborative filtering[END_REF]. In this work we explore the utilization of Feedforward Neural Network architectures (FNNs), and in particular architectures which take as input trainable embeddings of users and items, and propagate their contribution to subsequent layers. As is typical in FNN training, non-linear activation functions such as sigmoid or tanh are utilized for calculating the activations of the nodes. However recently a new activation function has been proposed, namely the Scaled Exponential Linear Unit (SELU) [START_REF] Klambauer | Self-normalizing neural networks[END_REF]. The use of the SELU activation function helps to regularize network parameters and makes learning robust due to its self-normalizing properties. SELU has been demonstrated to outperform other activation functions when utilized in deep FNN architectures in a number of standard machine learning benchmark problems such as MNIST [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF] and CIFAR [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF].

SELU is a variant of the Rectified Linear Unit (ReLU) activation function [START_REF] Nair | Rectified linear units improve restricted boltzmann machines[END_REF] whose derivative is constant for all positive input values. Due to the nature of the recommendation problem where the predicted values should lie within a positive range (typically in a scale between 1 and 5 "stars"), this means that positive values will remain positive during the learning process, thus reinforcing high correlations between user and item factors in suitable FNN architectures. Adding this observation to the beneficial self-normalizing properties that SELU exhibits, we expect its utilization in training FNN recommender systems to exhibit interesting characteristics. To the best of our knowledge, in the context of recommender systems, SELU has only been utilized in deep autoencoder networks [START_REF] Kuchaiev | Training Deep AutoEncoders for Collaborative Filtering[END_REF].

In this paper we explore the utilization of SELU in training various FNN architectures for training recommender systems and demonstrate that it not only outperforms other activation functions as is already known from the literature, but that it also exhibits interesting invariance with regards to the selection of the optimization algorithm and its corresponding hyperparameters. We have performed a number of experiments with various optimization algorithms for training different FNN architectures on standard recommender systems benchmark datasets, which clearly demonstrate this invariance property.

Activation Functions

ReLU

FNN training with non-linear activation functions such as sigmoid or tanh suffer by the vanishing gradient problem towards the lower (input) layers [START_REF] Bengio | Learning long-term dependencies with gradient descent is difficult[END_REF]. Therefore the FNNs that perform well are usually shallow. A common way to tackle the vanishing gradient problem is the utilization of the ReLU activation function [START_REF] Nair | Rectified linear units improve restricted boltzmann machines[END_REF] which prevents gradients from saturating since its gradient has a constant value. The ReLU activation function is given by:

f (x) = max(0, x) (1)
Thus ReLU is linear (identity) for all positive values and returns zero if it receives any negative input hence the mean activation of the nodes is greater than zero. In essence these units act as bias for units in the next layer causing a bias shift which hinders learning.

ELU

In contrast to ReLU, the Exponential Linear Unit (ELU) activation function has negative values which bring mean unit activations centered around zero. Zero means can accelerate learning because they bring the gradient closer to the unit natural gradient [START_REF] Clevert | Fast and accurate deep network learning by exponential linear units (elus)[END_REF]. The ELU activation function is given by the following formula:

f (x) = x, if x > 0 αe x -α, if x ≤ 0 (2)
ELU is comprised of two parts: the positive part of the function is the identity and the negative part exponentially skews the negative values. The hyperparameter α controls the convergence rate when the network inputs tend to be negative infinite.

SELU

During training, the distribution of the network activations changes at every training step which may slow down training. This is known as the internal covariate shift problem [START_REF] Shimodaira | Improving predictive inference under covariate shift by weighting the log-likelihood function[END_REF] which renders training neural networks hard since it demands smaller learning rates and careful parameter initialization. One way to address this problem is by using Batch Normalization (BN) [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] which normalizes the layer inputs in order to follow the standard normal distribution with zero mean and unit variance. SELU [START_REF] Klambauer | Self-normalizing neural networks[END_REF] deals with the covariant shift problem and has additional advantages over BN since it comes with lower computational complexity and has self-normalizing properties, in the sense that node activations remain centered around zero and have unitary variance. Due to its properties, SELU makes learning highly robust and allows to train networks that have many layers. SELU is defined as:

selu(x) = λ x, if x > 0 αe x -α, if x ≤ 0 (3)
For standard scaled inputs (zero mean and standard deviation one) the values of the parameters α and λ are α ≈ 1.6732 and λ ≈ 1.0507 [START_REF] Klambauer | Self-normalizing neural networks[END_REF]. The initial weights of nodes with the SELU activation function should be drawn from a normal distribution with zero mean and variance equal to 1/n where n is the size of the input. In addition, in [START_REF] Klambauer | Self-normalizing neural networks[END_REF] a new dropout technique [START_REF] Srivastava | Dropout: A simple way to prevent neural networks from overfitting[END_REF] is proposed, called Alpha Dropout. The proposed dropout technique randomly sets inputs to -λα which is the point that the network inputs tend to be negative infinite. Then in order to ensure the self-normalizing property, an affine transformation is applied to the inputs using parameters that are relative to the dropout rate, targeted mean and variance. The proposed Alpha Dropout rates are 0.05 or 0.1. Neural Network embeddings have been proven to be very powerful both for modeling language and for representing categorical variables. For example, the Word2Vec word embeddings [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] map a word to a vector based on training a neural network on large corpora. These embeddings can be used in any supervised model because they are just numerical representations of categorical variables. In the context of recommender systems, we can utilize the concatenation of two embedding layers [START_REF] Gal | A theoretically grounded application of dropout in recurrent neural networks[END_REF] as input to an FNN architecture. The first embedding layer corresponds to the encoding of all users whereas the second embedding layer encodes all items. In all our experiments presented in section 4 we have utilized embeddings layers of dimensionality 100. In addition the dot product of the embeddings of each user/item pair in the training set can be added as an extra input node to the network in order to reinforce positive correlations between user and item latent factors (embeddings) as shown in Figure 1(a). The network has only a single output node with a bias weight. Network regularization is implemented by the utilization of a dropout layer. The output of the dropout layer is then added to (trainable) user and item biases as it is customary practice in recommendendation models to account for the variations between the ratings that each user provides and conversely for the variations between the ratings that each item receives [START_REF] Koren | The BellKor Solution to the Netflix Grand Prize[END_REF].

Neural Network Architectures

Single Layer Feedforward Neural Network with Global Mean

This network architecture is shown in Figure 1(b) and is a variation of the FNN shown in Figure 1(a). The variation consists of the addition of the global mean of the ratings in the training set to the dot product of the user and item embeddings. The inclusion of the global mean has been proved to have a beneficial effect to the training of many recommendation models as it maintains the output of the prediction centered around this value. This results in smaller gradients of the loss function which helps to prevent oscillations and accelerates learning [24].

Double Layer Feedforward Neural Network

A double layer FNN with a single hidden layer can be utilized for the recommendation problem in order to encapture higher order dependencies. This network architecture is shown in Figure 2. The input consists of the concatenation of the user and item embeddings which are fed to a hidden layer. In all our experiments we set the hidden layer size to 128 nodes. Each node also has a bias weight. A dropout layer is then added for regularization and the output of that layer is then added to the trainable user and item biases as in the case of the single layer FNN architecture.

Experimental Results

In this section, we evaluate the performance of ReLU, ELU and SELU for the three different neural network architectures mentioned in section 3. For the evaluation we utilized the Movielens-100K and Movielens-1M benchmark datasets [START_REF] Harper | The movielens datasets: History and context[END_REF]. For each dataset we performed k-fold cross validation (with k = 5) and split the data using stratified sampling per user, so that 80% of the ratings of each user were used for training and the remaining 20% of each user's ratings were used for validation. We report the average Mean Absolute Error (MAE) on the validation sets over the k folds, as given by the following formula:

M AE = 1 k k j=1 1 |(R tr ) j | Rui∈(Rtr)j (R ui -Rui ) ( 4
)
where k is the nuber of folds, |(R tr ) j | is the size of each validation set (R tr ) j , and R ui , Rui are the actual and predicted ratings of user u on item i respectively. Three optimization algorithms were employed on the experiments, namely Stochastic Gradient Descent (SGD) [START_REF] Kiefer | Stochastic estimation of the maximum of a regression function[END_REF], Adam [START_REF] Kingma | A method for stochastic optimization[END_REF] and RMSprop [START_REF] Tieleman | RMSprop Gradient Optimization[END_REF]. For each algorithm a grid search was performed, the parameters of which are shown in Table 1. The grid search was complemented by employing each activation function with various dropout rates as shown in Table 2. For each trial, the initial weights of all network architectures were drawn from the normal distribution with zero mean and standard deviation of 0.1, starting from the same random seed in order to obtain fair comparisons. For the SELU trials the weights of the network were initialized from the normal distribution with zero mean and variance equal to 1/n where n is the size of input according to its specifications guidelines [START_REF] Klambauer | Self-normalizing neural networks[END_REF].

All the experiments were implemented within the Tensorflow framework [START_REF] Ramsundar | TensorFlow for Deep Learning: From Linear Regression to Reinforcement Learning[END_REF] and we have made all the codes publicly available on a Github repository Figure 3b shows the experimental results for the Single Layer Feedforward Neural Network with global mean architecture described in section 3.2. Again, for each activation function we report the average validation MAE over the 5-folds versus various combinations of hyperparameter choices, grouped by optimization algorithm. Similarly to the previous architecture, SELU retains consistently low MAE results contrary to both ReLU and ELU which oscillate regardless of the choice of optimization algorithm and its hyperparameters. The second row of Table 3 shows that for this FNN architecture ReLU's, ELU's and SELU's minimum results are practically the same, with ELU slightly outperforming the other two. However, SELU's average MAE (0.7568) is significantly lower than the average MAE of both ReLU (0.8299) and ELU (0.8291). The invariance of SELU is reflected in its MAE variance (6.44e-04), which, in this case too, is practically 10 times lower than the MAE variances of both ReLU (6.92e-03) and ELU (6.92e-03). The p-values of the Wilcoxon test indicate that the differences of SELU's average MAE result compared to the other two activation functions is statistically significant.

Finally for this dataset, Figure 3c shows the experimental results for the Double Layer Feedforward Neural Network architecture described in section 3.3. Again, for each activation function we report the average validation MAE over the 5-folds versus various combinations of hyperparameter choices, grouped by optimization algorithm. Compared to the two previous architectures, SELU exhibits some small oscillations, especially when the RMSprop algorithm is used, but continues to obtain lower and more consistent MAE results than ReLU and ELU, which both oscillate practically in the same way as in the two previous cases. This behavior of SELU is depicted in the third row of Table 3, which shows an increase in the MAE variance (1.10e-03) in comparison to the previous two architectures but still remains 5 times lower than the MAE variances of both ReLU (5.30e-03) and ELU (5.13e-03). In addition SELU's average MAE (0.7807) is lower than the average MAE of ReLU (0.8283) and ELU (0.8290). We also note that SELU (0.7425) achieves better MAE than ReLU and ELU (0.7476 and 0.7444 respectively). The p-values of the Wilcoxon test indicate that the differences of ReLU's and ELU's results compared to the best perfoming method is statistically significant. Experimental results on the Movielens-1M dataset: Figure 4a shows the experimental results for the Single Layer Neural Network architecture described in section 3.1. As in the Movielens-100K dataset, for each activation function we report the average validation MAE over the 5-folds versus various combinations of hyperparameter choices, grouped by optimization algorithm. From this Figure we can see that ReLU's and ELU's MAE results are practically the same Table 3: Experimental results for MovieLens-100K. The Table shows the minimum and maximum value of MAE as well as the mean and variance of the MAE results over the grid search results shown in Figure 3. The last column shows the p-value of the Wilcoxon test. Figure 4b shows the experimental results for the Single Layer Neural Network with global mean architecture described in section 3.2. For each activation function we report the average validation MAE over the 5-folds versus various combinations of hyperparameter choices, grouped by optimization algorithm. SELU retains consistently low MAE results (with a slight increase when Adam is used), contrary to ELU and ReLU which oscillate practically in the same way as in the previous FNN architecture. The second row of Table 4 shows that, similarly to the previous FNN architecture, SELU (0.9058) outperforms ReLU (0.9238) and ELU (0.9237) in terms of minimum MAE and SELU's average MAE (1.0069) is again significantly lower than the average MAE of ReLU (1.2721) and ELU (1.2727). Furthermore, the invariance property is noted in SELU's MAE variance (9.35e-03), which, in this case, is 11 times lower than ReLU's (8.10e-02) and ELU's (8.07e-02) MAE variance. Again, the p-values of the Wilcoxon test indicate that the differences of SELU's average MAE result to the other two functions is statistically significant.

Finally, Figure 4 shows the experimental results for the Double Layer Feedforward Neural Network architecture described in section 3.3. For each activation function we report the average validation MAE over the 5-folds versus various combinations of hyperparameter choices, grouped by optimization algorithm. With this FNN architecture, ELU exhibits stronger oscillatory behavior. Due to these oscillations and only when Adam is used with certain hyperparameters ELU manages to achieve slightly better results than SELU. However, except for those few occasions, SELU obtains lower and and more consistent MAE results than ReLU and ELU across the grid search, thus preserving its invariance. This is clearly depiscted in the third row of Table 4, which shows SELU's MAE variance (1.10e-03) which is almost 5 times lower than the MAE variances of ReLU (5.30e-03) and ELU (5.13e-03). As with the previous architecture cases, SELU's average MAE (1.0156) is notably lower than ReLU's (1.287) and ELU's (1.2718) average MAE. Moreover, SELU achieves noticably better MAE (0.8838) than ReLU (0.9265) and ELU (0.9267). In this case also, the p-values of the Wilcoxon test show that the benefits of the utilization of the SELU as activation function are statistically significant.

Conclusion

In this paper we examined the utilization of the SELU activation function in FNN architectures for recommender systems. Experimental results on standard recommender systems benchmark datasets (Movielens-100K and Movielens-1M) demonstrated that SELU is invariant on the choice of optimization algorithm and its corresponding hyperparameters. SELU performed consistently across all the combination of datasets, algorithms and hyperparameters reinforcing the belief that its self-normalizing properties for the network parameters is especially beneficial for neural network recommender systems due to the nature of the problem. Our future plans include the utilization of deeper neural network architectures which will also address recommendation as a ranking problem.

3. 1

 1 Fig. 1

Fig. 2 :

 2 Fig. 2: Double Layer Feedforward Neural Network

Fig. 4 :

 4 Fig.4: Experimental results on the Movielens-1M dataset for the three different neural network architectures: each subfigure shows the average validation MAE over the 5-folds versus various combinations of hyperparameters grouped by optimization algorithm (SELU is depicted with a green solid line, ELU with a blue dashed line, and ReLU with a red dotted line).

Table 1 :

 1 Algorithm parameters

		Learning Rates	Momentum
	SGD	0.01, 0.02	0, 0.2, 0.5
	Adam 0.0005, 0.0007, 0.001, 0.002, 0.003	-
	RMSprop 0.0005, 0.0007, 0.001, 0.002, 0.003	-

1

.

Table 2 :

 2 Dropout rates In order to evaluate the statistical significance of the results, we ranked the activation functions by the average MAE result and performed a paired Wilcoxon test[START_REF] Wilcoxon | Individual Comparisons by Ranking Methods[END_REF]. The last column of that Table shows the p-values of the Wilcoxon test and portray that the differences of SELU's results to the other two functions is statistically significant. Furthermore, in terms of optimum results, SELU obtains better minimum MAE (0.7382) than that of both ReLU and ELU (0.7428 in both cases).

		Dropout Alpha Dropout
	ReLU 0, 0.2, 0.5	-
	ELU 0, 0.2, 0.5	-
	SELU	-	0, 0.05, 0.1

  and oscillate strongly regardless of the choice of optimization algorithm and its hyperparameters. We note that the oscillations are stronger when RMSprop is used. Compared to the Movielens-100K dataset SELU's MAE results are slightly more sensitive to the choice of algorithm but still remain within a small range of the algorithm's hyperparameter grid search, and are consistently better than those of ReLU and ELU. The first row of Table4shows that in terms of optimum results, SELU obtains a considerably better MAE (0.8) than both ReLU and ELU (≈ 0.917 in both cases). Interestingly from that Table we can see that there is also a significant difference between the average MAE of SELU

		Activation Function	Minimum value	Maximum value	Mean	Variance	p-value
	Single Layer FNN	ReLU ELU SELU	0.7428 0.7428 0.7382	1.0348 1.0349 0.7982	0.8299 0.8309 0.7627	4.81e-03 4.69e-03 4.81e-04	6.83e-09 8.21e-09
	Single Layer	ReLU	0.7128	1.0477	0.8299	6.96e-03	6.47e-08
	FNN with	ELU	0.7123	1.0471	0.8291	6.92e-03	9.64e-08
	global mean	SELU	0.7127	0.7959	0.7568	6.44e-04
	Double Layer FNN	ReLU ELU SELU	0.7476 0.7444 0.7425	1.0693 1.0588 0.8609	0.8283 0.8290 0.7807	5.30e-03 5.13e-03 1.10e-03	1.95e-06 1.36e-06
	(1.0107) and the average MAE of ReLU (1.2820) and ELU (1.2834). In addition
	SELU's MAE variance (1.27e-02) is almost 7 times lower than ReLU's (8.37e-02)
	and ELU's (8.41e-02) MAE variances. The p-values of the Wilcoxon test show
	that the differences of SELU's results to the other two functions is statistically
	significant.

Table 4 :

 4 Experimental results for MovieLens-1M. The Table shows the minimum and maximum value of MAE as well as the mean and variance of the MAE results over the grid search results shown in Figure4. The last column shows the p-value of the Wilcoxon test.

		Activation Function	Minimum value	Maximum value	Mean	Variance	p-value
	Single Layer FNN	ReLU ELU SELU	0.9172 0.9170 0.8000	1.9015 1.8995 1.2695	1.2820 1.2834 1.0107	8.37e-02 8.41e-02 1.27e-02	2.62e-07 2.22e-07
	Single Layer	ReLU	0.9238	1.9051	1.2721	8.10e-02	1.60e-07
	FNN with	ELU	0.9237	1.9038	1.2727	8.07e-02	1.51e-07
	global mean	SELU	0.9058	1.2445	1.0069	9.35e-03
	Double Layer FNN	ReLU ELU SELU	0.9265 0.9267 0.8838	1.9059 1.9016 1.2975	1.2870 1.2718 1.0156	7.89e-02 7.70e-02 1.56e-02	5.14e-08 3.22e-06

https://github.com/flo3003/recsys_fnn.git