Mayalen Chris Reinke
email: chris.reinke@inria.fr

Pierre-Yves Oudeyer
email: pierre-yves.oudeyer@inria.fr

INTRINSICALLY MOTIVATED DISCOVERY OF DIVERSE PATTERNS IN SELF-ORGANIZING SYSTEMS

In many complex dynamical systems, artificial or natural, one can observe selforganization of patterns emerging from local rules. Cellular automata, like the Game of Life (GOL), have been widely used as abstract models enabling the study of various aspects of self-organization and morphogenesis, such as the emergence of spatially localized patterns. However, findings of self-organized patterns in such models have so far relied on manual tuning of parameters and initial states, and on the human eye to identify "interesting" patterns. In this paper, we formulate the problem of automated discovery of diverse self-organized patterns in such high-dimensional complex dynamical systems, as well as a framework for experimentation and evaluation. Using a continuous GOL as a testbed, we show that recent intrinsically-motivated machine learning algorithms (POP-IMGEPs), initially developed for learning of inverse models in robotics, can be transposed and used in this novel application area. These algorithms combine intrinsicallymotivated goal exploration and unsupervised learning of goal space representations. Goal space representations describe the "interesting" features of patterns for which diverse variations should be discovered. In particular, we compare various approaches to define and learn goal space representations from the perspective of discovering diverse spatially localized patterns. Moreover, we introduce an extension of a state-of-the-art POP-IMGEP algorithm which incrementally learns a goal representation using a deep auto-encoder, and the use of CPPN primitives for generating initialization parameters. We show that it is more efficient than several baselines and equally efficient as a system pre-trained on a hand-made database of patterns identified by human experts.

INTRODUCTION

Self-organization of patterns that emerge from local rules is a pervasive phenomena in natural and artificial dynamical systems [START_REF] Ball | The self-made tapestry: pattern formation in nature[END_REF]. It ranges from the formation of snow flakes, spots and rays on animal's skin, to spiral galaxies. Understanding these processes has boosted progress in many fields, ranging from physics to biology [START_REF] Camazine | Self-organization in biological systems[END_REF]. This progress relied importantly on the use of powerful and rich abstract computational models of self-organization [START_REF] Stuart A Kauffman | The origins of order: Self-organization and selection in evolution[END_REF]. For example, cellular automata like Conway's Game of Life (GOL) have been used to study the emergence of spatially localized patterns (SLPs) [START_REF] Gardener | Mathematical games: The fantastic combinations of john conway's new solitaire game" life[END_REF], informing theories of the origins of life [START_REF] Gardener | Mathematical games: The fantastic combinations of john conway's new solitaire game" life[END_REF][START_REF] Randall | Autopoiesis and cognition in the game of life[END_REF]. SLPs, such as the famous glider in GOL [START_REF] Gardner | Wheels, life, and other mathematical amusements[END_REF], are self-organizing patterns that have a local extension and can exist independently of other patterns. However, finding such novel self-organized patterns, and mapping the space of possible emergent patterns, has so far relied heavily on manual tuning of parameters and initial states. Moreover, the dependence of this exploration process on the human eye to identify "interesting" patterns is strongly limiting further advances.

We formulate here the problem of automated discovery of a diverse set of self-organized patterns in such high-dimensional, complex dynamical systems. This involves several challenges. A first challenge consists in determining a representation of patterns, possibly through learning, enabling to Source code and videos at https://automated-discovery.github.io/ 1 arXiv:1908.06663v2 [cs.LG] 30 Sep 2019 Preprint incentivize the discovery of diverse "interesting" patterns. Such a representation guides exploration by providing a measure of (di-)similarity between patterns. This problem is particularly challenging in domains where patterns are high-dimensional as in GOL. In such cases, scientists have a limited intuition about what useful features are and how to represent them. Moreover, low-dimensional representations of patterns are needed for human browsing and the visualization of the discoveries. Representation learning shall both guide exploration, and be fed by self-collected data.

A second challenge consists in how to automate exploration of high-dimensional, continuous initialization parameters to discover efficiently "interesting" patterns, such as SLPs, with a limited budget of experiments. Sample efficiency is important to enable the later use of such discovery algorithms for physical systems [START_REF] Grizou | Exploration of selfpropelling droplets using a curiosity driven robotic assistant[END_REF], where experimental time and costs are strongly bounded. For example, in the continuous GOL used in this paper as a testbed, initialization consists in determining the values of a real-valued, high-dimensional 256×256 matrix besides 7 additional dynamics parameters. The possible variations of this matrix are too large for a simple random sampling to be efficient. More structured methods are needed.

To address these challenges, we propose to leverage and transpose recent intrinsically motivated learning algorithms, within the family of population-based Intrinsically Motivated Goal Exploration Processes (POP-IMGEPs -denoted simply as IMGEPs below, [START_REF] Baranes | Active learning of inverse models with intrinsically motivated goal exploration in robots[END_REF]; [START_REF] Péré | Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration[END_REF]). They were initially designed to enable autonomous robots to explore and learn what effects can be produced by their actions, and how to control these effects. IMGEPs self-define goals in a goal space that represents important features of the outcomes of actions, such as the position reached by an arm. This allows the discovery of diverse novel effects within their goal representations. It was recently shown how deep neuronal auto-encoders enabled unsupervised learning of goal representations in IMGEPs from raw pixel perception of a robot's visual scene [START_REF] Laversanne-Finot | Curiosity driven exploration of learned disentangled goal spaces[END_REF]. We propose to use a similar mechanism for automated discovery of patterns by unsupervised learning of a low-dimensional representation of features of self-organized patterns. This removes the need for human expert knowledge to define such representations.

Moreover, a key ingredient for sample efficient exploration of IMGEPs for robotics has been the use of structured motion primitives to encode the space of body motions [START_REF] Pastor | From dynamic movement primitives to associative skill memories[END_REF]. We propose to use a similar mechanism to handle the generation of structured initial states in GOL-like complex systems, based on specialized recurrent neural networks (CPPNs) [START_REF] Kenneth | Exploiting regularity without development[END_REF].

In summary, we provide in this paper the following contributions:

• We formulate the problem of automated discovery of diverse self-organized patterns in high-dimensional and complex game-of-life types of dynamical systems.

• We show how to transpose POP-IMGEPs algorithms to address the associated joint challenge of (learning to) represent interesting patterns and discovering them in a sample efficient manner.

• We compare various approaches to define or learn goal space representations for the sample efficient discovery of diverse SLPs in a continuous GOL testbed.

• We show that an extension of a state-of-the-art POP-IMGEP algorithm, with incremental learning of a goal space using a deep auto-encoder, is equally efficient than a system pretrained on a hand-made database of patterns.

RELATED WORK

Automated Discovery in Complex Systems Automated processes have been widely used to explore complex dynamical systems. For example, evolutionary algorithms have been applied to search specific patterns or rules of cellular automata [START_REF] Mitchell | Evolving cellular automata with genetic algorithms: A review of recent work[END_REF][START_REF] Sapin | Research of a cellular automaton simulating logic gates by evolutionary algorithms[END_REF]. However, their objective is to optimize a specific goal instead of discovering a diversity of patterns.

Another line of experiments represent active inquiry-based learning strategies which query which set of experiments to perform to improve a system model, i.e. a mapping from parameters to the system outcome. Such strategies have been used in biology [START_REF] Ross D King | Functional genomic hypothesis generation and experimentation by a robot scientist[END_REF][START_REF] Ross D King | The automation of science[END_REF], chemistry [START_REF] Raccuglia | Machine-learningassisted materials discovery using failed experiments[END_REF][START_REF] Brandon J Reizman | Suzuki-miyaura cross-coupling optimization enabled by automated feedback[END_REF][START_REF] Duros | Human versus robots in the discovery and crystallization of gigantic polyoxometalates[END_REF] and astrophysics [START_REF] Joseph W Richards | Active learning to overcome sample selection bias: application to photometric variable star classification[END_REF]. However, these approaches have relied on expert knowledge, and focused on automated optimization of a pre-defined target property. Here, we are interested to automatically discover and map a diversity of unseen patterns without prior knowledge of the system. An exception is the concurrent work of [START_REF] Grizou | Exploration of selfpropelling droplets using a curiosity driven robotic assistant[END_REF], which showed how a simple POP-IMGEP algorithm could be used to automate discovery of diverse patterns in oil-droplet systems. However, it used a low-dimensional input space, and a hand-defined low-dimensional representation of goal spaces, identified as a major limit of the system.

Intrinsically motivated learning Intrinsically-motivated learning algorithms [START_REF] Baldassarre | Intrinsically motivated learning in natural and artificial systems[END_REF][START_REF] Baranes | Active learning of inverse models with intrinsically motivated goal exploration in robots[END_REF] autonomously organize an agent's exploration curriculum in order to discover efficiently a maximally diverse set of outcomes the agent can produce in an unknown environment. They are inspired from the way children self-develop open repertoires of skills and learn world models. Intrinsically Motivated Goal Exploration Processes (IMGEPs) [START_REF] Baranes | Active learning of inverse models with intrinsically motivated goal exploration in robots[END_REF][START_REF] Forestier | Intrinsically motivated goal exploration processes with automatic curriculum learning[END_REF] are a family of curiosity-driven algorithms developed to allow efficient exploration of high-dimensional complex real world systems. Population-based versions of these algorithms, which leverage episodic memory, hindsight learning, and structured dynamic motion primitives to parameterize policies, enable sample efficient acquisition of high-dimensional skills in real world robots [START_REF] Forestier | Intrinsically motivated goal exploration processes with automatic curriculum learning[END_REF][START_REF] Rolf | Goal babbling permits direct learning of inverse kinematics[END_REF]. Recent work [START_REF] Laversanne-Finot | Curiosity driven exploration of learned disentangled goal spaces[END_REF][START_REF] Péré | Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration[END_REF] studied how to automatically learn the goal representations with the use of deep variational autoencoders. However, training was done passively and in an early stage on a precollected set of available observations. Recent approaches [START_REF] Ashvin V Nair | Visual reinforcement learning with imagined goals[END_REF][START_REF] Vitchyr H Pong | Skewfit: State-covering self-supervised reinforcement learning[END_REF] introduced the use of an online training of VAEs to learn the important features of a goal space similar to the methods in this paper. However, these approaches focused on the problem of sequential decisions in MDPs, incurring a cost on sample efficiency. This problem is observed in various intrinsically motivated RL approaches [START_REF] Bellemare | Unifying count-based exploration and intrinsic motivation[END_REF][START_REF] Burda | Exploration by random network distillation[END_REF]. The approaches are orthogonal to the automated discovery framework considered here with independent experiments allowing the use of memory-based sample efficient methods. A related family of algorithms in evolutionary computation is novelty search [START_REF] Lehman | Exploiting open-endedness to solve problems through the search for novelty[END_REF] and quality-diversity algorithms [START_REF] Pugh | Quality diversity: A new frontier for evolutionary computation[END_REF], which can be formalized as special kinds of population-based IMGEPs.

Representation learning We are using representation learning methods to learn autonomously goal spaces for IMGEPs. Representation learning aims at finding low-dimensional explanatory factors representing high-dimensional input data [START_REF] Bengio | Representation learning: A review and new perspectives[END_REF]. It is a key problem in many areas in order to understand the underlying structure of complex observations. Many state-of-theart methods [START_REF] Michael Tschannen | Recent advances in autoencoder-based representation learning[END_REF] have built on top of Deep variational autoencoders (VAE) (Kingma & Welling, 2013), using varying objectives and network architectures. However, studies of the interplay between representation learning and autonomous data collection through exploration of an environment have been limited so far.

ALGORITHMIC METHODS FOR AUTOMATED DISCOVERY

POPULATION-BASED INTRINSICALLY MOTIVATED GOAL EXPLORATION PROCESSES

An IMGEP is an algorithmic process generating a sequence of experiments to explore the parameters of a system by targeting self-generated goals (Fig. 1). It aims to maximize the diversity of observations from that system within a budget of N experiments. In population-based IMGEPs, an explicit memory of the history H of experiments and observations is used to guide the process.

The systems are defined by three components. A parameter space Θ corresponding to the controllable system parameters θ. An observation space O where an observation o is a vector representing all the signals captured from the system. For this paper, the observations are a time series of images which depict the morphogenesis of activity patterns. Finally, an unknown environment dynamic D: Θ → O which maps parameters to observations. To explore a system, an IMGEP uses a goal space T that represents relevant features of its observations, computed using an encoding function ĝ = R(o). For the exploration of patterns, such features may describe their form or extension. The exploration process iterates N times through: 1) sample a goal from a goal sampling distribution g ∼ G(H); 2) infer corresponding parameter θ using a parameter sampling policy Π = Pr(θ; g, H)

; rameters depend on a history of explored parameters, an initial set of N init parameters are randomly sampled and explored before the intrinsically motivated goal exploration process starts.

Different goal and parameter sampling mechanisms can be used within this architecture [START_REF] Baranes | Active learning of inverse models with intrinsically motivated goal exploration in robots[END_REF][START_REF] Forestier | Modular active curiosity-driven discovery of tool use[END_REF]. In the experiments below, parameters are sampled by 1) given a goal, selecting the parameter from the history whose corresponding outcome is most similar in the goal space; 2) then mutating it by a random process. The goal sampling policy is a uniform distribution over a hypercube in T chosen to be large enough to bias exploration towards the frontiers of known goals to incentivize diversity.

ONLINE LEARNING OF GOAL SPACES WITH DEEP AUTO-ENCODERS

For IMGEPs the definition of the goal space T and its corresponding encoder R are a critical part, because it biases exploration of the target system. One approach is to define a goal space by selecting features manually, for example by using computer vision algorithms to detect the positions of a pattern and its form. The diversity found by the IMGEPs will then be biased along these pre-defined features. A limit of this approach is its requirement of expert knowledge to select helpful features, particularly problematic in environments where experts do not know in advance what features are important, or how to formulate them.

Another approach is to learn goal space features by unsupervised representation learning. The aim is to learn a mapping R(o) from the raw sensor observations o to a compact latent vector z ∈ R d . This latent mapping can be used as a goal space where a latent vector z = g is interpreted as a goal.

Previous IMGEP approaches already learned successfully their goal spaces with variational autoencoders (VAE) [START_REF] Laversanne-Finot | Curiosity driven exploration of learned disentangled goal spaces[END_REF][START_REF] Péré | Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration[END_REF]. However, the goal spaces were learned before the start of the exploration from a prerecorded dataset of observations from the target environment. During the exploration the learned representations were kept fixed. A problem with this pretraining approach is that it limits the possibilities to discover novel patterns beyond the distribution of pretraining examples, and requires expert knowledge.

In this paper we attempt to address this problem by continuously adapting the learned representation to the novel observations encountered during the exploration process. For this purpose, we propose

H[ĝ] ← R(o)
an online goal space learning IMGEP (IMGEP-OGL), which learns the goal space incrementally during the exploration process (Algorithm 1). The training procedure of the VAE is integrated in the goal sampling exploration process by first initializing the VAE with random weights (Appendix E). The VAE network is then trained every K explorations for E epochs on the observation collected in the history H. Importance sampling is used to give more weight to recently discovered patterns.

STRUCTURING THE PARAMETER SPACE IN IMGEPS: FROM DMPS TO CPPNS

A key role in the generation of patterns in dynamical systems is their initial state A t=1 . IMGEPs sample these initial states and apply random perturbations to them during the exploration. For the experiments in this paper this state is a two-dimensional grid with 256 × 256 cells. Performing directly a random sampling of the 256 × 256 grid cells results in initial patterns that resemble white noise. Such random states result mainly in the emergence of global patterns that spread over the whole state space, complicating the search for spatially localized patterns. This effect is analogous to a similar problem in the exploration of robot controllers. Direct sampling of actions for individual actuators at a microscopic time scale is usually inefficient. A key ingredient for sample efficient exploration has been the use of structured primitives (dynamic motion primitives -DMPs) to encode the space of possible body motions [START_REF] Pastor | From dynamic movement primitives to associative skill memories[END_REF].

We solved the sampling problem for the initial states by transposing the idea of structured primitives. Indeed, "actions" consist here in deciding the parameters of an experiment, including the initial state.

We propose to use compositional pattern producing networks (CPPNs) [START_REF] Kenneth | Exploiting regularity without development[END_REF] to produce structured initial patterns similar do DMPs. CPPNs are recurrent neural networks that allow the generation of structured initial states (Appendix B, Fig. 9) . The CPPNs are used as part of the parameters θ. They are defined by their network structure (number of neurons, connections between neurons) and their connection weights. They include a mechanism for random mutation of the weights and structure. The number of parameters in θ is therefore not fixed (yet starts small) and open-ended.

EXPERIMENTAL METHODS

We describe here the continuous Game of Life (Lenia) we use as a testbed representing a large class of high-dimensional dynamical systems, as well as the experimental procedures, the evaluation methods used to measure diversity and detect SLPs, and the used algorithmic baselines and ablations.

CONTINOUS GAME OF LIFE AS A TESTBED

Lenia (Chan, 2018) is a continuous cellular automaton [START_REF] Wolfram | Statistical mechanics of cellular automata[END_REF] similar to Conway's Game of Life [START_REF] Gardener | Mathematical games: The fantastic combinations of john conway's new solitaire game" life[END_REF]. Lenia, in particular, represents a high-dimensional complex dynamical system where diverse visual structures can self-organize and yet are hard to find by manual exploration. It features the richness of Turing-complete game-of-life models. It is therefore well suited to test the performance of pattern exploration algorithms for unknown and complex systems. The fact that GOL models have been used widely to study self-organization in various disciplines, ranging from physics to biology and economics [START_REF] Bak | Self-organized criticality in the'game of life[END_REF], also supports their generality and potential of reuse of our approach for discovery in other computational or wet high-dimensional systems.

Lenia consists of a two-dimensional grid of cells A ∈ [0, 1] 256×256 where the state of each cell is a real-valued scalar activity A t (x) ∈ [0, 1]. The state of cells evolves over discrete time steps t (Fig. 2,a). The activity change is computed by integrating the activity of neighbouring cells. Lenia's behavior is controlled by its initial pattern A t=1 and several settings that control the dynamics of the activity change (R, T, µ, σ, β 1 , β 2 , β 3 ). Appendix A describes Lenia and its parameters in detail.

Lenia can be understood as a self-organizing morphogenetic system. Its parameters for the initial pattern A t=1 and dynamics control determine the development of morphological patterns. Lenia can produce diverse patterns with different dynamics (stable, non-stable or chaotic). Most interesting, spatially localized coherent patterns that resemble in their shapes microscopic animals can emerge (Fig. 2,b,c). These pattern types, which we will denote "animals" as a short name, are a key reason scientists have used GOL models to study theories of the origins of life [START_REF] Gardener | Mathematical games: The fantastic combinations of john conway's new solitaire game" life[END_REF][START_REF] Randall | Autopoiesis and cognition in the game of life[END_REF]. Therefore, in our evaluation method based on measures of diversity (see below), we will in particular study the performance of IMGEPs, and the impact of using various approaches for goal space representation, on finding a diversity of animal patterns. We implemented for this purpose different pattern classifiers to analyze the exploration results (Appendix A.2). Initially we differentiate between dead and alive patterns. A pattern is dead if the activity of all cells are either 0 or 1. Alive patterns are separated into animals and non-animals. Animals are a connected areas of positive activity which are finite, i.e. which do not infinitely cross several borders. All other patterns are non-animals whose activity usually spreads over the whole state space.

EVALUATION BASED ON THE DIVERSITY OF PATTERNS

The algorithms are evaluated based on their discovered diversity of patterns. Diversity is measured by the spread of the exploration in an analytic behavior space. This space is externally defined by the experimenter as in previous evaluation approaches in the IMGEP literature. For example, in [START_REF] Péré | Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration[END_REF] is the diversity of discovered effects of a robot that manipulates objects measured by binning the space of object positions and counting the number of bins discovered. A difference here is that the experimenter does not have access to an easily interpretable hand-defined low-dimensional representation of possible patterns, equivalent to the cartesian coordinate of rigid objects. The space of raw observations O, i.e. the final Lenia patterns A t=200 , is also too high-dimensional for a meaningful measure of spread in it. We constructed therefore an external evaluation space. First, a latent representation space was build through the training of a VAE to learn the important features over a very large dataset of Lenia patterns identified during the many experiments over all evaluated algorithms. This large dataset enabled to cover a diversity of patterns orders of magnitude larger than what could be found in any single algorithm experiment, which experimental budget was order of magnitude smaller. We then augmented that space by concatenating hand-defined features (the same as for the HGS algorithm). See Appendix C for more information.

For each experiment all explored patterns were projected into the analytic behavior space. The diversity of the patterns is then measured by discretizing the space into bins of equal size by splitting each dimension into 7 sections (results were found to be robust to the number of bins per dimension, see C). This results in 7 13 bins. The number of bins in which at least one explored entity falls is used as a measure for diversity.

We also measured the diversity in the space of parameters Θ by constructing an analytic parameter space. The 15 features of this space consisted of Lenia's parameters (R, T , µ, σ, β 1 , β 2 , β 3 ) and the latent representation of a VAE. The VAE was trained on a large dataset of initial Lenia states (A t=1 ) used over the experimental campaign. This diversity measures also used 7 bins per dimension.

ALGORITHMS

The exploration behaviors of different IMGEP algorithms were evaluated and compared to a random exploration. The IMGEP variants differ in their way how the goal space is defined or learned. Appendices D and E provide details and hyperparameters.

Random exploration:

The IMGEP variants were compared to a random exploration that sampled randomly for each of the N exploration iterations the parameters θ including the initial state A t=1 .

IMGEP-HGS -Goal exploration with a hand-defined goal space: The first IMGEP uses a handdefined goal space that is composed of 5 features used in Chan (2018). Each feature measures a certain property of the final pattern A t=200 that emerged in Lenia: 1) the sum over the activity of all cells, 2) the number of activated cells, 3) the density of the activity center, 4) an asymmetry measure of the pattern and 5) a distribution measure of the pattern.

IMGEP-PGL -Goal exploration with a pretrained goal space: For this IMGEP variant the goal space was learned with a VAE approach on training data before the exploration process started. The training set consisted of 558 Lenia patterns: half were animals that have been manually identified by Chan (2018); the other half randomly generated with CPPNs, see Section 4.4.

IMGEP-OGL -Goal exploration with online learning of the goal space: Algorithm 1.

IMGEP-RGS -Goal exploration with a random goal space: An ablated IMGEP using a goal space based on the encoder of a VAE with random weights.

EXPERIMENTAL PROCEDURE AND HYPERPARAMETERS

For each algorithm 10 repetitions of the exploration experiment were conducted. Each experiment consisted of N = 5000 exploration iterations. This number was chosen to be compatible with the application of the algorithms in physical experimental setups similar to [START_REF] Grizou | Exploration of selfpropelling droplets using a curiosity driven robotic assistant[END_REF], planned in future work. For IMGEP variants the first N init = 1000 iterations used random parameter sampling to initialize their histories H. For the following 4000 iterations each IMGEP approach sampled a goal g via an uniform distribution over its goal space. The ranges for sampling in the hand-defined goal space (HGS) are defined in Table 5 (Appendix D). The ranges for the VAE based goal spaces (PGL, OGL) were set to [-3, 3] for each of their latent variables. Then, the parameter θ k from a previous exploration in H was selected whose reached goal ĝk had the minimum euclidean distance to the current goal g within the goal space. This parameter was then mutated to generate the parameter θ that was explored.

Preprint

The parameters θ consisted of a CPPN (Section 3.3) that generates the initial state A t=1 for Lenia and the settings defining Lenia's dynamics:

θ = [CPPN → A t=1 , R, T, µ, σ, β 1 , β 2 , β 3 ].
The CPPNs were initialized and mutated by a random process that defines their structure and connection weights as done by [START_REF] Kenneth | Exploiting regularity without development[END_REF]. The random initialization of the other Lenia settings was done by an uniform distribution and their mutation by a Gaussian distribution around the original values. The meta parameters to initialize and mutate the parameters were the same for all algorithms (Appendix B). They were manually chosen without optimizing them for a specific algorithm.

RESULTS

We address several questions evaluating the ability of IMGEP algorithms to identify a diverse set of patterns, and in particular diverse "animal" patterns (i.e. spatially localized patterns).

Does goal exploration outperform random parameter exploration? In robotics/agents contexts where scenes are populated with rigid objects, various forms of goal exploration algorithms outperform random parameter exploration [START_REF] Laversanne-Finot | Curiosity driven exploration of learned disentangled goal spaces[END_REF]. We checked whether this still holds in continuous GOL which have very different properties. Measures of the diversity in the analytic behavior space confirmed the advantage of IMGEPs with hand-designed (HGS) or learned goal spaces (PGL/OGL) over random explorations (Fig. 3,b). The organization resulting from goal exploration is also visible through the diversity in the space of parameters. IMGEPs focus their exploration on subspaces that are useful for targeting new goals. In contrast, random parameter exploration is unguided, resulting in a higher diversity in the parameter space (Fig. 3,b).

What is the impact of learning a goal space vs. using random or hand-defined features?

We compared also the performance of random VAE goal spaces (RGS) to learned goal spaces (PGL/OGL). For reinforcement learning problems, using intrinsic reward functions based on ran- The different number and size of clusters of animals or non-animals can explain the differences in their resulting diversity between the algorithms (Fig. 3).

dom features of the observations can result in a high or boosted performance [START_REF] Burda | Large-scale study of curiosity-driven learning[END_REF]2019). In our context however, using random features (RGS) collapsed the performance of goal exploration, and did not even outperform random parameter exploration for all kinds of behavioural diversity (Fig. 3). Results also show that hand-defined features (HGS) produced significantly less global diversity and less "animal" diversity than using learned features (PGL/OGL). However, HGS found an equal diversity of "non-animals". These results show that in this domain, the goal-space has a critical influence on the type and diversity of patterns discovered. Furthermore, unsupervised learning is an efficient approach to discover a diversity of diverse patterns, i.e. both efficient at finding diverse animals and diverse non-animals.

Is pretraining on a database of expert patterns necessary for efficient discovery of diverse animals? A possibility to bias exploration towards patterns of interest, such as "animals", is to pretrain a goal space with a pattern dataset hand-made by an expert. Here PGL is pretrained with a dataset containing 50% animals. This leads PGL to discover a high diversity of animals. However, the new online approach (IMGEP-OGL) is as efficient as PGL to discover diverse patterns (Fig. 3, b,c,d). Taken together, these results uncover an interesting bias of using learned features with a VAE architecture, which strongly incentivizes efficient discovery of diverse spatially localized patterns.

How do goal space representations differ? We analyzed the goal spaces of the different IMGEP variants to understand their behavior by visualizing their reached goals in a two-dimensional space. T-SNE [START_REF] Van Der Maaten | Visualizing data using t-sne[END_REF] was used to reduce the high-dimensional goal spaces. It puts points that were nearby in the high-dimensional space also close to each other in the two-dimensional visualization.

The hand-defined (HGS) and learned (OGL) goal spaces show strong differences between each other (Fig. 4). We believe this explains their different abilities to find either a high diversity of non-animals or animals (Fig. 3,c,d). The goal space of the IMGEP-HGS shows large areas and several clusters for non-animal patterns (Fig. 4,a). Animals form only few and nearby clusters. Thus, the hand-defined features seem poor to discriminate and describe animal patterns in Lenia. As a consequence, when goals are uniformly sampled within this goal space during the exploration process, then more goals are generated in regions that describe non-animals. This can explain why IMGEP-HGS explored a higher diversity of non-animal patterns but only a low diversity of animal patterns. In contrast, features learned by IMGEP-OGL capture better factors that differentiate animal patterns. This is indicated by the several clusters of animals that span a wide area in its goal space (Fig. 4,b).

We attribute this effect to the difficulty of VAEs to capture sharp details (Zhao et al., 2017b). They therefore represent mainly the general form of the patterns but not their fine-grained structures. Animals differ often in their form whereas non-animals occupy often the whole cell grid and differ in their fine-grained details. The goal spaces learned by VAEs seem therefore better suited for exploring diverse sets of animal patterns.

CONCLUSION

We formulated a novel application area for machine learning: the problem of automatically discovering self-organized patterns in complex dynamical systems with high-dimensions both in the action space and in the observation space. We showed that this problem calls for advanced methods requiring the dynamic interaction between sample efficient autonomous exploration and unsupervised representation learning. We demonstrated that population-based IMGEPs are a promising algorithmic framework to address this challenge, by showing how it can discover diverse self-organized patterns in a continuous GOL. In particular, we introduced a new approach of learning a goal space representation online via data collected during the exploration process. It enables sample efficient discovery of diverse sets of animal-like patterns, similar to those identified by human experts and yet without relying on such prior expert knowledge (Fig. 2). We also analyzed the impact of goal space representations on the diversity and types of discovered patterns.

The continuous game of life shares many properties with other artificial or natural complex systems, explaining why GOL models have been used in many disciplines to study self-organization, see [START_REF] Bak | Self-organized criticality in the'game of life[END_REF]. We therefore believe this study shows the potential of IMGEPs to automated discovery in other systems encountered in physics, chemistry or even computer animation. In further work, we aim to apply this approach in roboticized wet experiments such as the one presented in [START_REF] Grizou | Exploration of selfpropelling droplets using a curiosity driven robotic assistant[END_REF] and addressing fundamental understanding of how proto-cells can self-organize.

A TARGET SYSTEM: CONTINUOUS GAME OF LIFE (LENIA)

The Lenia model is a particular implementation of continuous Game of Life models [START_REF] Wang | Lenia-biology of artificial life[END_REF]. It was used as the target system for all exploration experiments. The following section describes Lenia and the parameters to control its behavior in detail. It is followed by a description of the classifiers used to categorize dead, animal and non-animal Lenia patterns. Finally, statistical measures about the patterns are introduced which were used to define goal and analytic spaces.

A.1 IMPLEMENTATION DETAILS AND PARAMETERS Lenia (Chan, 2018) is a cellular automaton [START_REF] Wolfram | Statistical mechanics of cellular automata[END_REF]

A t+1 (x) = A t (x) + ∆t G K * A t (x) 1 0 ,
where G is the growth mapping, K is the kernel, ∆t = 1 T with T ∈ N is the time step and [n] a b = min(max(n, a), b) is the clip function. For all experiments an exponential growth mapping was used:

G(u; µ, σ) = 2 exp - (u -µ) 2 2σ 2 -1 ,
with µ ∈ R and σ ∈ R being perimeters that control its shape.

The kernel integrates the activity of the current cell x and its neighbours by a convolution with a kernel function K(n):

K * A t (x) = n∈N (x) K(n)A t (x + n)∆x 2 , ( 1
)
where N is the neighborhood around the cell x and ∆x = 1 R with R ∈ N is the site distance. The neighborhood is defined by a circle around x with radius R:

N (x) = {y : ||x -y|| 2 ≤ R}. The kernel is constructed by a kernel core function K C : [0, 1] → [0, 1] and a kernel shell function K S : [0, 1] → [0, 1].
The kernel core creates a ring around the center coordinate and is defined by an exponential:

K C (r) = exp α - α 4r(1 -r)
with α = 4 .

The kernel shell K S takes a vector parameter β = (β 1 , β 2 , β 3 ) ∈ [0, 1] 3 and copies the kernel core into concentric rings. The rings are of equal thickness with peak heights β i :

K S (r; β) = β Br K C (Br mod 1) .
Finally, the kernel is normalized:

K(n) = K S (||n|| 2 ) |K S | .
In total 8 parameters θ = {A t=1 , R, T, µ, σ, β 1 , β 2 , β 3 } controlled the behavior of Lenia for all experiments. A t=1 is the starting pattern of the system. R is the radius of the circle around a cell x whose enclosed cells influence the activity of x. T controls the growth strength update per time step. The growth mapping is controlled by µ and σ. The form of the kernel function is controlled by

β 1 , β 2 , β 3 .
We based our Python implementation of Lenia on the code provided by https://github.com/ Chakazul/Lenia.

A.2 CLASSIFIER

We categorized 3 types of patterns that are observed in Lenia. The categories were used to analyze if the exploration algorithms showed differences in their exploration behaviors by identifying different types of patterns. The 3 categories are dead, animals and non-animals. For each class is a classifier defined. The classifiers only classify the final pattern in which the Lenia system morphs after M = 200 time steps.

Dead Classifier: For dead patterns is the activity of all cells either 0 or 1 in the last time step.

Animal Classifier: The final Lenia pattern is classified as an animal if it is a finite and connected pattern of activity. Cells x, y are connected as a pattern if both are active (A(x) ≥ 0.1 and A(y) ≥ 0.1) and if they influence each other. Cells influence each other when they are within their radius of the kernel K as defined by the parameter R (Eq. 1).

Furthermore, the connected pattern must be finite. In Lenia finite and infinite patterns can be differentiated because the opposite borders of Lenia's cell grid are connected, so that the space is similar to a ball surface. Thus, a pattern can loop around this surface making it infinite. We identify infinite patterns by the following approach. First, all connected patterns are identified for the case of assuming an infinite grid cell, i.e. opposite grid cell borders are connected. Second, all connected patterns for the case of a finite grid cell, i.e. opposite grid cell borders are not connected, are identified. Third, for each border pair (north-south and east-west) it is tested if cells within a distance of R from both borders exists, that are part of a connected pattern for the infinite and finite grid cell case. If such a pattern exists than it is assumed to be infinite, because it loops around the grid cell surface of Lenia (Fig. 6,a). All other patterns are considered to be finite (Fig. 6,b). Please note that this method has a drawback. It can not identify certain infinite patterns that loop over several borders, for example, if a pattern exists that connects the north to east and then the west to south border (Fig. 7).

Moreover, there are two additional constraints that an animal pattern must fulfill. First, the cells of the connected pattern P = {x 1 , . . . , x n } must have at least 80% of all activation, i.e. x∈P A(x) ≥ 0.8 ∀y A(y). Second, a pattern must exists for the last two time steps (t = M and t = M -1). Non-Animal Classifier: We also classified non-animal patterns which are all entities that were not dead and not an animal. These patterns spread usually over the whole state space and are connected via borders. See Fig. 27, 29, 30 and 31 for examples of non-animal patterns.

A.3 STATISTICAL MEASURES FOR LENIA PATTERNS

We defined five statistical measurements for the final patterns A t=M that emerge in Lenia. The measures were used as features for hand-defined goal spaces of IMGEPs and to define partly the analytic behavior space in which the results of the exploration experiments were compared.

Activation mass M A : Measures the sum over the total activation of the final pattern and normalizes it according to the size of the Lenia grid:

M A = 1 L 2 x A t=M (x) ,
where L 2 = 256 • 256 is the number of cells of the Lenia system.

Activation volume V A : Measures the number of active cells and normalizes it according to the size of the Lenia grid:

V A = 1 L 2 ∀x : A t=M (x) > with = 10 -4 .
Activation density D A : Measures how dense the activation is distributed on average over all active cells:

D A = M A V A .
Activation asymmetry A A : Measures how symmetrical the activation is distributed according to an axis that starts in the center of the patterns activation mass and goes along the last movement direction of this center. This measure was introduced to especially characterize animal patterns such as shown in Fig. 5. The center of the activity mass is usually also the center of the animals and analyzing the activity along their movement axis measures how symmetrical they are.

As a first step, the center of the activation mass is computed for every time step of the Lenia simulation and the Lenia pattern recentered to this location. This ensures that the center is all the time correctly computed in the case the animal moves and reaches one border to appear on the opposite border in the uncentered pattern. The center (x, ȳ) t for time step t is calculated by:

(x, ȳ) t = M 10 M 00 , M 01 M 00 with M pq = x y x p y q A t (x, y) ,
where M pq measures the image moment (or raw moment) of order (p + q) for p, q ∈ N.

Based on the center (x, ȳ) t the pattern A t is recentered to A t C by shifting the x and y indexes according to the center:

A t C (x, y) = A t ((x -x) mod L, (y -ȳ) mod L) , (2)
where L is width and length of the Lenia grid and the indexing is x, y = 0, . . . , L -1. After each time step the center is recomputed and the pattern recentered:

A t=1 -→ recenter A t=1 C -→ Lenia step A t=2 -→ recenter A t=2 C -→ Lenia step . . . .
Please note, the simulations and all figures of patterns in the paper are done with the uncentered pattern. The centered version is only computed for the purpose of statistical measurements.

The recenter step by (x, ȳ) t defines also the movement direction of the activity center:

(m x , m y ) t = (x, ȳ) t -x mid , y mid = x -x mid , ȳ -y mid ,
where x mid , y mid = L-1 2 are the coordinates for the middle point of the grid. A line can be defined that starts in the midpoint x mid , y mid of the final centered pattern A t=M C and goes in and opposite to the final movement direction of the activity mass center (m x , m y ) t=M . This line separates the grid in two equal areas. The asymmetry is computed by comparing the amount of activity in the grid right M right A and left M lef t A of the line. The normalized difference between both sides is the final asymmetry measure:

A A = 1 M A (M right A -M lef t A ).
Activation centeredness C A : Measures how strong the activation is distributed around the activity mass center:

C A = 1 M A x y w xy • A t=M C (x, y) with w xy = 1 - d(x, y) max y,x d(x, y) 2 ,
where d(x, y) = (x -x mid ) 2 + (y -y mid ) 2 is the distance from the point (x, y) to the center point x mid , y mid . A t=M C (x, y) is the centered activation that is updated every time step as for the asymmetry measure (Eq. 2). The weights w xy decrease the farer a point is from the center. Thus, patterns that are concentrated around the center have a high value for C A close to 1. Whereas, patterns whos activity is distributed throughout the whole grid have a smaller value. For patterns that are equally distributed (∀ x,x : A(x) = A(x )) is C A = 0 defined as centeredness measure.

B SAMPLING OF PARAMETERS FOR LENIA

All exploration algorithms explore Lenia patterns by sampling the parameters θ that control Lenia. The parameters are comprised of the initial pattern A t=1 and the parameters which control the dynamic behavior (R, T, µ, σ, β 1 , β 2 , β 3 ). There are two operations to sample parameters: 1) random initialization and 2) mutating an existing parameter θ. CPPNs are used for the random initialization and mutation of the initial pattern A t=1 . The details of this process are described in the next section. Afterwards, the initialization and mutation of Lenia's parameter that control its dynamics are described.

B.1 SAMPLING OF START PATTERNS FOR LENIA VIA CPPNS

Compositional Pattern Producing Networks (CPPNs) are recurrent neural networks that were developed for the generation and evolution of gray-scale 2D images [START_REF] Kenneth | Exploiting regularity without development[END_REF]. We used CPPNs to generate and mutate the initial state of Lenia A t=1 which resembles an image. CPPNs generate images pixel by pixel by taking as input a bias value, the x and y coordinate of the pixel in the image and its distance d to the image center (Fig. 8). Their output is the pixel value as a gray scale between 0 and 1 for the given (x, y) coordinate. For the generation of initial Lenia patterns is as input the x and y coordinate of the grid cells used. They were mapped to x = [-2, 2] and y = [-2, 2]. The distance to the grid center is given by d(x, y) = x2 + y 2 . The final activity of a cell is the remapped output p of the CPPN via A(x, y) = 1 -|p|.

CPPNs consist of several hidden neurons (typically between 4 to 6 in our experiments) that can have recurrent connections and self connections. Each CPPN has one output neuron. Two activation functions were used for the hidden neurons and the output neuron. The first is Gaussian and the second is sigmoidal: gauss

(x) = 2 exp -(2.5x) 2 -1 , (3) sigm
(x) = 2 1 1 + exp(-5x) -1 . (4)
To randomly initialize a Lenia initial pattern A t=1 a CPPN is randomly sampled by sampling the number of hidden neurons, the connections between inputs and neurons and neurons to neurons, their connection weights and the activation functions for neurons. Afterwards the initial pattern is generated by it. In the history H of the IMGEPs is then the CPPN as part of the parameter θ added. If the parameter is mutated, then the weights, connections and activation functions of the CPPN are mutated and the new initial pattern A t=1 generated by it. A CPPN is defined over its network structure (number of nodes, connections of nodes) and its connection weights. The number of parameters in θ is therefore variable and not fixed.

We used the neat-python 2 package for the random generation and mutation of CPPNs. It is based on the NeuroEvolution of Augmenting Topologies (NEAT) algorithm for the evolution of neural networks [START_REF] Kenneth | Efficient evolution of neural network topologies[END_REF]. The meta-parameters for the initialization and mutation of CPPNs are listed in Table 1. The random sampling and mutation of CPPNs allows to generate complex patterns as illustrated in Fig. 9.

The random sampling of a new CPPN is done by the following steps. All CPPNs are initialized with 4 hidden neurons and 1 output neuron. Their activation functions are randomly assigned. Each input-hidden, hidden-hidden and hidden-output neuron pair is connected with a probability of 0.6. The weights of each connection are sampled via a Gaussian distribution: w ij ∼ N (µ = 0, σ = 0.4).

The maximum and minimum weights for a connection are -3 and 3.

An existing CPPN is mutated by the following procedure. At first, structural mutations are performed. With probability 0.02 a new neuron z with a random activation function is added. The neuron is connected to the network by choosing randomly an existing connection. This connection is deleted. A connection from the source i of the deleted connection to the new neuron is added with weight w iz = 1.0. Additionally, a new connection from the new neuron to the target j of the deleted connection is added with the old connection weight w zj = w ij , finishing the addition of a new neuron. With probability 0.02 one of the hidden neurons is deleted. With probability 0.05 a new connection is added between a random input-hidden, hidden-hidden or hidden-output neuron pair. The connection weight is sampled by the same method as for the sampling of new CPPNs. With probability 0.01 one random existing connection is removed. After the structural mutations the activation functions and weights are mutated. For each neuron the activation function is changed with probability 0.1 by randomly assigning a new activation function (either gauss or sigm). For each connection the weight is mutated by the following steps. With probability 0.05 the weight of the connection is changed according to: Please note, the neat-python package allows also the setting and mutation of response and bias weights for each neuron. Those settings were not used for the experiments. Moreover, we adjusted the sigmoid and Gaussian function in the neat-python package to the ones defined in Eq. 3 and Eq. 4 to be able to replicate similar images as in [START_REF] Kenneth | Exploiting regularity without development[END_REF].

w ij ← [w ij + N (0, σ M )]

B.2 SAMPLING OF LENIA'S DYNAMIC PARAMETERS

The parameters that control the dynamics of Lenia (R, T, µ, σ, β 1 , β 2 , β 3 ) are initialized and mutated via uniform and Gaussian distributions. Table 2 lists for each parameter the meta-parameters for their initialization and mutation. Each parameter is initialized by an uniform sampling θ i ∼ U(a, b) with a and b as upper and lower border. An existing parameter θ i is mutated by the following equation:

θ i ← [θ i + N (0, σ M )]
M R N [2, 20] 0.5 T N [1, 20] 0.5 µ R [0, 1] 0.05 σ R [0.001, 0.3] 0.01 β[1] R [0, 1] 0.05 β[2] R [0, 1] 0.05 β[3] R [0, 1] 0.05
Table 2: Settings for the initialization and mutation of Lenia system parameters θ.

C MEASUREMENT OF DIVERSITY IN THE ANALYTIC PARAMETER AND BEHAVIOR SPACE

The algorithms are compared on their ability to explore a diverse set of patterns. The next section introduces the diversity measure, followed by sections that introduce the spaces in which the algorithms are compared.

C.1 DIVERSITY MEASURE

Diversity is measured by the area that explored parameters cover in the parameter space of Lenia or that the identified patterns cover in the observation space. For the experiments the parameter space consisted of the initial start state of Lenia (A t=1 ∈ [0, 1] 256×256 ) and the settings for Lenia's dynamics (R, T, µ, σ, β 1 , β 2 , β 3 ). The space consist therefore of 256 2 dimensions, each for a single grid cell of the initial pattern, plus 7 dimensions for the dynamic settings. The observation space consists of the final patterns A t=200 ∈ [0, 1] 256×256 resulting in 256 2 dimensions for the space. Each single exploration results in a new point in those spaces.

The diversity measures how much area the algorithms explored in those spaces (Fig. 10). The measurement is done by discretizing the space with a spatial grid and counting the number of discretized areas in which at least one point falls. For the discretization each dimension of the space is given a range, i.e. a minimum and maximum border. Each dimension is then split in a certain number of equally sized bins between those borders. The areas with values falling below the minimum or above the maximum border are counted as two additional bins.

The number of dimensions of the original parameter and observation space are too large to measure diversity in a meaningful manner. The initial pattern and the final pattern have 256 2 dimensions. We constructed therefore an analytic parameter and behavioral space where the latent representations of a β-VAE were used to reduce the high-dimensional patterns to 8 dimensions. The diversity in those spaces was compared between the algorithms. 5 bins (7 with the out of range values) per dimension were used for the discretization of those spaces for all experiments in the paper.

C.2 ANALYTIC PARAMETER SPACE

The analytic parameter space was constructed by the 7 Lenia parameters that control its dynamics and 8 latent representation dimensions of a β-VAE (Table 3). The β-VAE was trained on initial patterns A t=1 used during the experiments. The dataset was constructed by randomly selecting 42500 patterns (37500 as training set, 5000 as validation set) from the experiments of all algorithms and each of their 10 repetitions. The β-VAE uses the same structure, hyper-parameters, loss function and learning algorithm as described in Section E. It was trained for more than 1400 epochs with β = 5 (Fig. 11). The encoder which resulted in the minimal validation set error during the training was used. According to its reconstructed patterns it can represent the general form of patterns but often not individual details such as their texture (Fig. 12).

Preprint

Analytic Parameter Space Definition Parameter min max Parameter min max

R 1 20 β-VAE latent 1 -5 5 T 2 10 β-VAE latent 2 -5 5 µ 0 1 β-VAE latent 3 -5 5 σ 0 0.3 β-VAE latent 4 -5 5 β 1 0 1 β-VAE latent 5 -5 5 β 2 0 1 β-VAE latent 6 -5 5 β 3 0 1 β-VAE latent 7 -5 5 β-VAE latent 8 -5 5
Table 3: Features of the analytic parameter space and their min and max values

Learning curve of the β-VAE for the Analytic Parameter Space

M A 0 1 β-VAE latent 1 -5 5 volume V A 0 1 β-VAE latent 2 -5 5 density D A 0 1 β-VAE latent 3 -5 5 asymmetry A A -1 1 β-VAE latent 4 -5 5 centeredness C A 0 1 β-VAE latent 5 -5 5 β-VAE latent 6 -5 5 β-VAE latent 7 -5 5 β-VAE latent 8 -5 5
Table 4: Features of the analytic behavior space and their min and max values

Learning curve of the β-VAE for the Analytic Behavior Space The analytic behavior space was constructed by combining the 5 statistical measures for final Lenia patterns (Section A.3) and 8 latent representation dimensions of a β-VAE (Table 4). The β-VAE was trained on final patterns A t=200 observed during experiments. The dataset was constructed by randomly selecting 42500 patterns (37500 as training set, 5000 as validation set) from the experiments of all algorithms and each of their 10 repetitions. The dataset consists of 50% animal and 50% non-animal patterns. The β-VAE uses the same structure, hyper-parameters, loss function and learning algorithm as described in Section E. It was trained for more than 1400 epochs with β = 5 (Fig. 13). The encoder which resulted in the minimal validation set error during the training was used. Its reconstructed patterns show that it is able to represent the general form of patterns but often not individual details such as their texture (Fig. 14).

D RANDOM EXPLORATION AND IMGEPS WITH HAND-DEFINED GOAL SPACES

Two random explorations and several IMGEPs with different hand-defined goal spaces were evaluated and compared. The main paper and the additional results in Section F only report the results for the best random exploration and one IMGEP variant with a hand-defined goal space. This section introduces the implementation details and diversity results of all evaluated random explorations and IMGEPs with hand-defined goal spaces.

D.1 RANDOM EXPLORATIONS

We evaluated two random exploration strategies: Random Initialization and Random Mutation. The main paper and the additional results in Section F only discuss the Random Initialization approach.

Random Initialization: This approach sampled for each of the 5000 explorations a random parameter θ including a random CPPN to generate the initial state A t=1 . The approach can be replicated by using Algorithm 2 with N init = 5000.

Random Mutation: This approach is closer to the principle of IMGEPs. It first performs N init = 1000 random explorations and adds each explored parameter θ to a history H. Afterwards, it randomly samples a parameter from the history and mutates it. The new parameter is also added to history H. The approach can be replicated by using Algorithm 2 where line 6 is skipped and the parameter sampling distribution Π(g, H) is selecting a random parameter from the history and mutating it.

D.2 IMGEPS WITH HAND-DEFINED GOAL SPACES

We evaluated several IMGEP variants with goal spaces that were hand-defined (IMGEP-HGS). Each space was constructed by a different combination of statistical measures of the final Lenia patterns (Tables 5 and6) which are described in Section A.3. The main paper and the additional results in Section F only discuss the IMGEP-HGS 9 approach. Algorithm 2 lists the steps of the IMGEP-HGS variants. They begin with N init = 1000 random explorations, followed by 4000 explorations based on randomly generated goals. Each goal was sampled from a uniform distribution within the ranges defined in Table 5. Then the parameter from a previous exploration that resulted in the closest outcome to the current goal was mutated and explored. The random explorations and IMGEP-HGS variants are compared by their resulting diversity in the analytic parameter and behavior space (Fig. 15). The diversity is measured by the number of reached bins in each space using a binning of 7 bins per dimension.

The Random Initialization approach reached for all diversity measures a higher diversity than the Random Mutation approach. Therefore, the Random Initialization approach is used for the comparison to IMGEP approaches in the main paper and the additional results in Section F.

Preprint

Most IMGEP-HGS variants had a higher diversity in the analytic behavior space compared to random explorations, although their diversity in the analytic parameter space is lower. This shows the advantage of IMGEPs over random searches in discovering a wider range of patterns in the target system. The best overall diversity had IMGEP-HGS 3, 4 and 9. We chose IMGEP-HGS 9 to compare it with learned goal spaces in the main paper and for the additional results in Section F. It identified the highest diversity of non-animals of the three variants (3, 4, 9) reaching a higher diversity for non-animals than any IMGEP with a learned goal space. It was therefore selected to show that the choice of the goal space has an influence on the patterns that IMGEPs identify.

Depending on the statistical measures used to define the goal space the diversity between the IMGEP-HGS variants varied. IMGEPs that use the volume measure (HGS 1 -4) reach in general a higher overall diversity which can be attributed to their higher diversity of animal patterns than goal spaces with the density measure (HGS 5 -8) (Fig. 15,b,c). In terms of diversity of identified animals showed the inclusion of several measures the best performance (HGS 4 and HGS 8 in Fig. 15,c). In terms of diversity of identified non-animals showed the inclusion of several measures besides the centeredness C A measure the best performance (HGS 3 and HGS 7 in Fig. 15,d). The results show that the choice of the goal space has an important influence on the diversity of identified patterns and their type (animal or non-animal).

Feature min max mass M

A 0 1 volume V A 0 1 density D A 0 1 asymmetry A A -1 1 centeredness C A 0 1
Table 5: HGS Goal Space Ranges

HGS-Variants

Feature 1 2 3 4 5 6 7 8 9 15: Although all IMGEP-HGS variants have lower diversity in the analytic parameter space compared to the Random Initialization approach, most of them have a higher diversity in the analytic behavior space. Each dot besides the boxplot shows the diversity of found patterns for each repetition (n = 10). The box ranges from the upper to the lower quartile. The whiskers represent the upper and lower fence. The mean is indicated by the dashed line and the median by the solid line.

mass M A × × × × × × × × × volume V A × × × × × density D A × × × × × centeredness C A × × × × × asymmetry A A × × × × ×

E IMGEPS WITH RANDOM AND LEARNED GOAL SPACES USING DEEP VARIATIONAL AUTOENCODERS

We considered three random initializations for the VAE representation used in the IMGEP-RGS as well as three different training objectives for learning the VAE goal space used in IMGEP-PGL and IMGEP-OGL. Variational Autoencoders (VAEs) [START_REF] Diederik | Auto-encoding variational bayes[END_REF][START_REF] Preprint | Stochastic backpropagation and approximate inference in deep generative models[END_REF] are commonly used deep generative models that can unsupervisedly learn a latent representation of the data. The latent representation has a reduced number of dimensions and should capture the important features of the input data. We use VAEs to learn the important features that describe Lenia patterns. The features are then used to define goal spaces for IMGEPs. This section details the different variants that were implemented for IMGEP with random and learned VAE goal spaces, the implementation details and compares the diversity results as well as the VAEs reconstruction accuracy.

E.1 IMGEP WITH RANDOM VAE GOAL SPACES

To study the impact of learning representations we implemented IMGEP-RGS as an ablated version of IMGEPs where the goal space is based on the encoder of a VAE with random weights. We evaluated three variants to randomly set the weights of the VAE encoder: Pytorch [START_REF] Paszke | Automatic differentiation in pytorch[END_REF], Xavier [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF] and Kaiming [START_REF] He | Delving deep into rectifiers: Surpassing human-level performance on imagenet classification[END_REF]. The VAE is composed of four 2D convolutional layers (with ReLU activations) followed by three fully-connected layers.

Table 7 shows the different sampling distributions from which the encoder parameters are initialized. We used uniform distributions for both Xavier and Kaiming variants and set all the layers bias parameters to zero. Table 7: IMGEP-RGS variants initialization schemes. fan in is the number of input units in the weight tensor and fan out is the number of output units in the weight tensor.

Convolutional Layers Linear Layers

RGS

E.2 IMGEP WITH LEARNED VAE GOAL SPACES E.2.1 VAE FRAMEWORK

VAEs have two components: a neural encoder and decoder. The encoder q(z|x, χ) represents a given data point x in a latent representation z. In variational approaches the encoder describes a data point by a representative distribution in the latent space of reduced dimension d. A standard Gaussian prior p(z) = N (0, I) and a diagonal Gaussian posterior q(z|x, χ) = N (µ, σ) are used for this purpose. Given a data point x, the encoder outputs the mean µ and variance σ of the representative distribution in the latent space. The decoder p(x|z, ψ) tries to reconstruct the original data x from a sampled latent representation z for the distribution given by the encoder.

Under these assumptions, training is done by maximizing the computationally tractable evidence lower bound (with β = 1):

L(χ, ψ) = E z∼qχ(z|x) [log p ψ (x|z)] a -β × D KL [q χ (z|x) p(z)] b . (5)

Preprint

The first term (a) represents the expected reconstruction accuracy while the second (b) is the KL divergence of the approximate posterior from the prior.

b = D KL [N (µ(x), Σ(x)) N (0, I)] = d i=1 D KL [N (µ(x) i , σ(x) i ) N (0, 1)] bi . (6)

E.2.2 VAE VARIANTS

The recent growing interest in unsupervised representation learning, and therefore in VAEs, resulted in a plethora of proposed losses, network designs and choices of family for the encoder, decoder and prior distributions [START_REF] Michael Tschannen | Recent advances in autoencoder-based representation learning[END_REF]. In order to enhance desired properties such as interpretability and disentanglement of the latent variables, many current state-of-the-art approaches build on the VAE framework and augment the VAE objective [START_REF] Higgins | beta-vae: Learning basic visual concepts with a constrained variational framework[END_REF][START_REF] Christopher P Burgess | Understanding disentangling in beta-vae[END_REF][START_REF] Kim | Disentangling by factorising[END_REF][START_REF] Qi | Isolating sources of disentanglement in variational autoencoders[END_REF][START_REF] Kumar | Variational inference of disentangled latent concepts from unlabeled observations[END_REF].

In this paper, we couple the VAE architecture with three different objectives: the classical VAE objective (Kingma & Welling, 2013) (equation 5 with β = 1), the β-VAE objective [START_REF] Higgins | beta-vae: Learning basic visual concepts with a constrained variational framework[END_REF] equation 5 with β > 1) and an augmented β-VAE objective (equation 7).

The β-VAE objective re-weights the b term by a factor β > 1, aiming to enhance the disentangling properties of the learned latent factors. We are interested in such properties as it has been shown that it can benefit exploration [START_REF] Laversanne-Finot | Curiosity driven exploration of learned disentangled goal spaces[END_REF]. However, heavily penalizing b can result in the network learning to "sacrifice" one or more of the learned latent variables in order to nullify their contribution b i (equation 6). Those dimensions become completely uninformative and useless for further exploration in the learned latent space. This phenomenon is known as posterior collapse and is a common problem when training VAEs [START_REF] Samuel R Bowman | Generating sentences from a continuous space[END_REF][START_REF] Chen | Variational lossy autoencoder[END_REF][START_REF] He | Lagging inference networks and posterior collapse in variational autoencoders[END_REF]Kingma et al., 2016).

To prevent this phenomenon to happen, we then considered an augmented β-VAE objective with a new term that encourages the network to decrease together the individual contributions b i of the different latent variables. This augmented loss term not only minimizes the averaged contribution (sum) but also the variance of the individual contributions:

b aug = d i=1 b i + Var ([b 1 , . . . , b d ]) . (7)
Similarly other modifications of the training objective can be found in the literature to avoid posterior collapse [START_REF] Tolstikhin | Wasserstein autoencoders[END_REF]Zhao et al., 2017a). All VAEs use the same architecture (Table 8). The encoder network has as input the Lenia pattern and as outputs for each latent variable z i the mean µ i and log-variance log(σ 2 i ). The decoder takes as input during the training for each latent variable a sampled value z i ∼ N (µ i , σ 2 i ). For validation runs and the generation of all reconstructed patterns shown in figures the decoder takes the mean z i = µ i as input. Its output is the reconstructed pattern.

The training objectives of all three variants are given in section E.2.1. The resulting loss function (Eq. 8) of all VAE variants for a batch is:

Loss(x, x, µ, σ) = -a + β d i=1 b i + γVar([b 1 , . . . , b d ] ,
Preprint patterns. The best VAE model obtained during the training phase, i.e. the one with with the highest accuracy on the validation data, is used for the exploration.

The dataset used to train the VAE has 558 patterns which are distributed into a training (75%), validation (10%) and testing (15%) datasets. Half of the patterns (279) were manually identified animal patterns by Chan (2018) (Fig. 5). The other half ( 279) are randomly initialized CPPN patterns as described in Section B.1 (Fig. 9).

During the intrinsically motivated iterations, goals are uniformly sampled in the hypercube [-3, 3] 8 . This values were chosen because the encoder of the VAE is trained to match a prior standard normal distribution (through the KL divergence term), therefore we can assume that most area of the covered goal space will fall into that hypercube. The datasets were constructed incrementally during the exploration by gathering non-dead patterns. One pattern every ten is added to the validation set (10%) and the rest is used in the training set. At the initial period of training, the training dataset amounts approximately 50 patterns and at the last period of training the dataset amounts approximately 3425 patterns (Fig. 16). The validation dataset only serves for checking purposes and has no influence on the learned goal space.

During the intrinsically motivated iterations, goals are uniformly sampled in the hypercube [-3, 3] 8 .

E.4 RESULTS

We compared the different IMGEP-RGS variants as well as the different objective variants for IMGEP with learned goal spaces (PGL and OGL) with each other on the basis of the diversity of their identified patterns. Furthermore, the pattern reconstruction ability of the VAEs is analyzed.

E.4.1 DIVERSITY

The algorithms are compared by their diversity in the analytic parameter and behavior space (Section C). Diversity is measured by the number of discretized bins that were explored by the algorithms in each space if each dimension of the space is seperated in 7 bins.

Preprint

All the IMGEPs with learned goal spaces reached a higher diversity in the analytic behavior space compared to random explorations (Fig. 17,b), although random explorations have a higher diversity in the analytic parameter space (Fig. 17,a). This result confirms further the advantage of IMGEPs over random explorations in terms of identifying diverse patterns.

Furthermore, all the IMGEPs with learned goal spaces outperformed the IMGEP with random goal space. This result shows the importance of learning relevant pattern features that, combined with an effective exploration process, is key to discover a high diversity of patterns.

There is no significant differences between Xavier and Kaiming IMGEP-RGS variants. They both seem to present a higher variance than the Pytorch variant and reach therefore a higher average performance, but it is unclear why. Because the Xavier initialization performed slightly the best for IMGEP-RGS, it was used for the results in the main paper and in Section F.

The difference between the PGL and OGL variants were small for all diversity measures. The OGL showed a slight advantage over the PGL versions in all diversity measures. Thus, an online version of the IMGEP can learn an appropriate goal space during the exploration. A precollected dataset as for the PGL is not necessary to successfully use IMGEPs.

The difference between the VAE objective variants (VAE, β-VAE and augmented β-VAE) was very small. The β-VAE was slightly better than the other two variants for the diversity in the analytic parameter space and for both IMGEP variants. All VAEs seemed to learn similar features for our datasets. It might be possible that the different VAE variants show different behaviors if their parameters are fine-tuned, such as the β parameter, but this was out of the scope of this paper. Because the β-VAE objective performed slighly the best for IMGEP-PGL and IMGEP-OGL, it was used for the results in the main paper and in Section F.

E.4.2 VAE PATTERN RECONSTRUCTION

All learned VAE variants showed similar learning curves on the precollected dataset and the online collected dataset (Fig. 18 and20). Their ability to reconstruct patterns based on the encoded latent representation is also qualitatively similar. For both datasets the VAEs are able to learn the general form of the activity pattern (Fig. 19 and21). Nonetheless, the compression of the images to a 8dimensional vector results in a general blurriness in the reconstructed patterns. As a result, the VAEs are not able to encode finer details and textures of patterns (Fig. 22). We believe this is the reason for their ability to identify more animals compared to the random exploration or the IMGEP-RGS and IMGEP-HGS. Different animals have often a different form, whereas non-animals span often over the whole area of Lenia's grid and differentiate mainly in their textures and small details. Because the VAE seem to encode more the general form a goal space based on them is more appropriate to find patterns with different forms such as the animals and not different textures which are important for non-animals.

F ADDITIONAL RESULTS

This section lists additional results. The results are only for a subset of all algorithm variants that have been evaluated. The results correspond to the following algorithms: Random to Random Initialization (Section D), IMGEP-RGS to IMGEP-RGS Xavier (Section E), IMGEP-HGS to IMGEP-HGS 9 (Section D), IMGEP-PGL to IMGEP-PGL with a β-VAE (Section E) and IMGEP-OGL to IMGEP-OGL with a β-VAE (Section E).

F.1 NUMBER OF IDENTIFIED PATTERNS

The main paper used the measure of diversity of the found patterns per algorithm to compare their performance. Another measure to compare the algorithms is the number of the patterns they identified for each of the three pattern classes: dead, animals, non-animals (Fig. 23).

The results deviate slightly from the diversity measures. In terms of identified non-dead patterns, all IMGEP approaches outperform a random exploration by finding between 10 to 20% more patterns.

Although the IMGEP-RGS and IMGEP-HGS find more non-dead patterns than the IMGEPs with learned goal spaces (OGL, PGL) its overall diversity in the analytic behavior space is smaller (Fig. 3, b of the main paper).

In the case of animal patterns, all IMGEP approaches outperform the random exploration (8%). Within the IMGEP approaches the online learned goal space approach (IMGEP-OGL, 34%) and the pretrained goal space approach (IMGEP-PGL: 35%) find a similar amount. The hand-defined goal space approach identified less animal patterns (IMGEP-HGS: 19%). The random goal space approach is the one that finds the least (IMGEP-RGS: 10%). For non-animal patterns, the random goal space approach identifies most patterns (IMGEP-RGS: 79%) , followed by the hand-defined approach (IMGEP-HGS: 67%), the random exploration (56%) and both learned goal space approaches (IMGEP-OGL: 45%, IMGEP-PGL: 43%). Although the number of identified non-animal patterns for the learned goal space approaches is low, their diversity is higher than for a random exploration and the random goal space approach, and only slightly lower than for the hand-defined goal space approach (Fig. 3, d of the main paper).

F.2 DEPENDENCE OF THE DIVERSITY MEASURE ON THE NUMBER OF BINS PER DIMENSION

The diversity of identified patterns measures the spread of the area which the identified patterns cover in the analytic behavior space (Fig. 3 in the main paper). The measure is defined by dividing the space in a number of discrete areas or bins (Section C.1). The diversity is then measured by how many bins are covered during an exploration. The bins are created by dividing each dimension of the space into a number of equally-sized bins. We analyzed how the number of bins per dimension influences the diversity measure (Fig. 24).

Although the diversity difference between the algorithms depends on the number of bins per dimension for each space, the order of the algorithms, i.e. which algorithm has a higher diversity, is generally constant. Only if the number of bins per dimension grows large (>10) the order of the algorithms changes for some spaces and subpatterns. The order starts to follow the order seen for the number of identified patterns (compare the diversity with 25 bins per dimension in Fig. 24 with the number of identified patterns in Fig. 23). In this case the discretization of the space becomes too fine and each pattern falls into its own discretized area. We chose therefore a smaller number of bins per dimension of 7 (including the out of border bins) for all other diversity plots in the main paper and the Supplementary Material to compare the algorithms in a meaningful way.

F.3 DIMENSION REDUCTION OF THE ANALYTIC PARAMETER AND BEHAVIOR SPACE

A two-dimensional reduction of the identified patterns in the analytic parameter and behavior space (Section C) visualizes the diversity of the parameters and identified patterns. The dimension reduction of the parameter space is based on all explored parameters encoded in the analytic parameter space from the first repetition experiment of all 4 algorithms. All encoded points were normalized so that the overall minimum value became 0 and the maximum value 1 for each dimension. Afterwards a principle component analysis (PCA) was performed to detect the 2 principle components [START_REF] Jolliffe | Principal Component Analysis and Factor Analysis[END_REF]. The found patterns for each algorithm are plotted according to those components. The results show that the random exploration has a stronger uniform distribution than any of the IMGEP algorithms in the analytic parameter space (Fig. 25). The IMGEP algorithms show concentrations of explorations in specific regions of the parameter space. The visualization shows also that In the analytic behavior space IMGEPs reach a higher diversity compared to a random exploration, except IMGEP-RGS which covers similar areas. The HGS approach explores more non-animal areas and the PGL and OGL more animal areas. PCA dimension reduction of the analytic behavior space which illustrates all identified patterns by the first repetition experiment per algorithm.

Preprint it is not possible to define distinct regions in the parameter space that allow to differentiate between dead, animal and non-animal patterns.

The same analysis was performed for the identified patterns of each algorithm encoded in the analytic behavior space (Fig. 26). It is visible that the random exploration and the random goal space (RGS) approaches are more concentrated compared to the hand-defined (HGS) or learned goal spaces approaches (PGL and OGL), especially in a region with many non-animal patterns (north-west). The IMGEP-RGS has a similar spread than random exploration, with an even higher concentration of non-animals (north-west). The IMGEP-HGS has a wider spread in the non-animal area. The IMGEPs with a learned goal space (PGL and OGL) show a stronger distribution in an area that encodes mostly animals (south).

F.5 VISUALIZATION OF GOAL SPACES

The goal space of IMGEPs is their most important element because it defines which type of patterns are set as goals for the exploration. This section provides complementary material for the analysis made in Section 5.2 of the main paper and shows further visualizations of goal spaces. The goal spaces of all IMGEP algorithms are visualized via a two-dimensional reduction of each goal space. Two techniques for dimensionality reduction were applied: PCA [START_REF] Jolliffe | Principal Component Analysis and Factor Analysis[END_REF]) and t-Distributed Stochastic Neighbor Embedding (t-SNE) [START_REF] Van Der Maaten | Visualizing data using t-sne[END_REF].

The visualization was constructed by using for each exploration algorithm its goal space representations of all patterns it explored from a single repetition experiment. All goal representations were normalized so that the overall minimum value became 0 and the maximum value 1 for each goal space dimension. Afterwards the PCA was performed to detect the 2 principle components. T-SNE was executed by using the default standard Euclidean distance metric and default hyper-parameters (perplexity set to 50).

The resulting two-dimensional visualizations of the goal spaces make the differences between the algorithms visible (Fig. 32). For both aproaches (PCA, t-SNE), the random goal space (RGS) and hand-defined goal space (HGS) have only a small area and a few clusters for animal patterns. In contrast, the learned goal spaces based on β-VAEs (PGL and OGL) have larger areas and more clusters for animal patterns. As a result, the learned goal spaces explore more animal patterns and find a higher diversity of them (Fig. 3,c) compared to the hand-defined goal space and the random goal space. The reason for this effect seems to be that the β-VAE which defines the goal space for the PGL and OGL is learning to represent the shape of patterns. The shape is an important feature of animals. Whereas, non-animals often cover the whole Lenia grid and differ mainly in their textures which the β-VAE does not represent well (Section E).

The visualization serves as a support in qualitatively evaluating and comparing the efficiency of each algorithm in extracting a diversity of patterns from the data. Integrated into an interactive interface, these graphs are also useful for a potential human end-user to easily explore and visualize the different type of found patterns during the exploration phase. Videos and demonstrations of the interface can be found on the website https://automated-discovery.github.io/.

Figure 1 :

 1 Figure 1: Population-based intrinsically motivated goal exploration process with incremental learning of a goal space (IMGEP-OGL algorithm) used to explore a continuous GOL model.

Figure 2 :

 2 Figure 2: Example patterns produced in the continuous GOL system (Lenia). Illustration of the dynamical morphing from an initial CPPN image to an animal (a). The automated discovery (b) is able to find similar complex animals as a human-expert manual search (c) by Chan (2018).

Figure 3 :Figure 4 :

 34 Figure3: (a) Although random explorations reach the highest diversity in the analytic parameter space, (b) IMGEPs reach a higher diversity in the analytic behavior space (except when using random representations). (c) IMGEPs with a learned goal space discovered a larger diversity of animals compared to a hand-defined goal space. (d) Learned goal spaces are as efficient as a hand-defined space for finding diverse non-animals patterns. Overall, IMGEPs with unsupervised learning of goal features are efficient to discover a diversity of diverse patterns. Depicted is the average diversity (n = 10) with the standard deviation as shaded area (for some not visible because it is too small). SeeFig. 27, 28, 29, 30 and 31 in Annex for a qualitative visual illustration of these results.

Figure 5 :Figure 6 :

 56 Figure 5: Patterns identified by a human-expert[START_REF] Wang | Lenia-biology of artificial life[END_REF]. The activity of each cell is mapped from [0, 1] to the color scheme visualized at the bottom. An activity of 0 is white. This color scheme is used for all illustrated patters throughout the paper.

Figure 7 :

 7 Figure 7: Examples of infinite patterns that are misclassified as a finite patterns.

Figure 8 :

 8 Figure8: CPPNs are recurrent neural networks which take as input a bias of 1, the x and y coordinate of a point in the generated pattern and its distance r to the center of the pattern. Their output is the activity value of a grid cell.

3 - 3

 3 , where σ M = 1 is the mutation power and [n] a b = min(max(n, a), b) is the clip function. With probability 0.06 the connection weight is completely replaced by sampling a new one as done for the sampling of weights of new CPPNs.

  Figure 9: CPPNs can generate complex patterns via their random initialization and successive mutations. Each row shows generated patterns by one CPPN and its mutations.

  where σ M is the mutation power and [n] a b = min(max(n, a), b) is the clip function with a and b as upper and lower border. For natural numbers θ i ∈ N the resulting value is rounded towards the nearest natural number. Parameter Type Value Range Mutation σ

Figure 10 :

 10 Figure10: Illustration of the diversity measure in a two-dimensional space. The ranges for the dimensions were set to[-5, 5] and [0, 0.3]. The number bins per dimension is 5. Including the outlier areas the number of discretized bins is 7 2 = 49. The diversity is the number of bins in which points exist (grey areas) which are 12 in this example.

Figure 11 :

 11 Figure 11: Learning curve of the β-VAE whose latent encoding was used for the analytic parameter space. Reconstruction Examples of the Analytic Parameter Space β-VAE

Figure 13 :

 13 Figure 13: Learning curve of the β-VAE whose latent encoding was used for the analytic behavior space. Reconstruction Examples of the Analytic Behavior Space β-VAE

Algorithm 2 :

 2 IMGEP-HGS Initialize goal space representation R by hand-defined features for i ← 1 to N do if i < N init then // Initial random iterations to populate H Sample θ ∼ U(Θ) else // Intrinsically motivated iterations Sample a goal g ∼ G(H) based on the space represented by R Choose θ ∼ Π(g, H) Perform an experiment with θ and observe o Append (o, θ, R(o)) to the history H D.3 RESULTS

  b, b) U(-b, b) U(-b, b) U(-b, b)

  By writing the VAE training objective as stated in equation 8, the three different variants outlined above correspond to the following set of hyper-parameters {β = 1, γ = 0}, {β = 5, γ = 0} and {β = 5, γ = 1}. L = -a + β (b + γ × c) the IMGEP approaches (RGS, PGL and OGL) and the network architecture, training procedure, hyper-parameters and datasets for the training of their VAEs.

Algorithm 4 :

 4 IMGEP-PGL Initialize goal space encoder R with the pretrained VAE for i ← 1 to N do if i < N init then // Initial random iterations to populate H Sample θ ∼ U(Θ) else // Intrinsically motivated iterations Sample a goal g ∼ G(H) based on space represented by R Choose θ ∼ Π(g, H) Perform an experiment with θ and observe o Append (o, θ, R(o)) to the history H IMGEP-OGL (online learned goal space): IMGEP (Algorihm 1 in the main paper) that trains the VAE which defines the goal space during the exploration. The VAE is trained on Lenia patterns discovered by the algorithm. Every K = 100 explorations the VAE model is trained for 40 epochs resulting in 2000 epochs in total (less if there is not enough data after the first T runs to start the training). Importance sampling is used to give the patterns in the training dataset a different weight during the training. A weighted random sampler is used that samples newly discovered patterns from the training dataset half of the time. Each pattern that has been added to the training dataset during the last period of 100 explorations has a probability of 0.5 N to be sampled (N is the total number of new patterns in the dataset). Older patterns are also sampled half of the time each one with probability 0.5 |D T |-N . As a result, newer discovered patterns have a higher weight and a stronger influence on the training of the VAE model.

Figure 16 :Figure 18 :Figure 20 :

 161820 Figure16: The IMGEP-OGL collects during the exploration animal and non-animal patterns to add them to its dataset for the training of the VAE. The figure shows the development of the averaged dataset size over all repetitions (n = 10) of the IMGEP-OGL algorithm with a β-VAE. Standard deviation is depicted as a shaded area but for some not visible because it is too small.

Figure 23 :Figure 24 :

 2324 Figure23: IMGEPs found less dead patterns compared to the random exploration. In terms of animals, the learned goal space approaches (IMGEP-PGL and OGL) found most animal patterns. For non-animals, the random goal space (IMGEP-RGS) found most patterns, followed by the handdefined goal space (IMGEP-HGS). The plot illustrates the percentage of found patterns for each class. Each dot besides the boxplot shows the percentage of found patterns for each repetition (n = 10). The box ranges from the upper to the lower quartile. The whiskers represent the upper and lower fence. The mean is indicated by the dashed line and the median by the solid line.

Figure 25 :

 25 Figure25: A random exploration covers the analytic parameter space more uniformly than IMGEP algorithms which form clusters at certain areas. PCA dimension reduction of the analytic parameter space which illustrates all explored parameters by the first repetition experiment per algorithm.

F. 4

 4 Fig. 27, 28, 29, 30 and 31 illustrate examples of identified pattern per class (animal, non-animal, dead) and their ratio for the random exploration, IMGEP-RGS, IMGEP-HGS, IMGEP-PGL and IMGEP-OGL. The patterns have been randomly sampled from the results of the first exploration repetition experiment of each algorithm.

Figure 32 :

 32 Figure32: PCA and t-SNE visualization of the goal spaces for the IMGEP variants show that HGS has more area (PCA) and clusters (t-SNE) for non-animals compared to learned goal spaces (PGL and OGL) and vice versa for animals. t-SNE shows that the hand-defined goal space (HGS) and learned goal spaces (PGL and OGL) structure and cluster more the discovered patterns compared to random goal space (RGS).

  . It consists of a two-dimensional grid of cells A ∈ [0, 1] L×L with L = 256 for all experiments. The cell grid is similar to the surface of a ball. Cells on the north border are neighbors to the south border cells. The east and west border are also connected. The state of each cell is a real-valued scalar activity A(x) ∈ [0, 1]. The states of cells A t evolve over discrete time steps t = [1, . . . , M ] with M = 200 for all experiments. The activity change of a cell is computed by integrating the previous activity of its neighbouring cells:

Table 1 :

 1 Configuration for the initialization and mutation of CPPN networks that generate the initial state for the Lenia system.

	Parameter

Table 6

 6

	: IMGEP-HGS Variants

https://github.com/CodeReclaimers/neat-python

ACKNOWLEDGMENTS

We thank Bert Wang-Chak Chan for his helpful discussions about the Lenia system and Jonathan Grizou for his comments on the visualization of our results. Furthermore, we thank Cédric Colas for his useful comments on the script.

given by a binary cross entropy with logits:

where the index j is for the single cells (pixel) of the pattern and n for the datapoint in the current batch, N is the batch size and σ(x) = 1 1+e -x . The KL divergence terms b i are given by:

All VAEs were trained for 2000 epochs and initialized with pytorch default initialization. We used the Adam optimizer (Kingma & Ba, 2014) (lr = 1e-3, β 1 = 0.9, β 2 = 0.999, = 1e-8, weight decay=1e-5) with a batch size of 64.

The patterns from the datasets were augmented by random x and y translations (up to half the pattern size and with probability 0.3), rotation (up to 40 degrees and with probability 0.3), horizontal and vertical flipping (with probability 0.2). The translations and rotations were preceded by spherical padding to preserve Lenia spherical continuity.

Three types of IMGEPs were evaluted:

IMGEP-RGS (random goal space): IMGEP with a goal space defined by an encoder network with random weights (Algorihm 3). The network architecture of the encoder is the same that the one of the VAEs used for IMGEP with learned goal spaces. In the other IMGEP algorithms (HGS/PGL/OGL), the goals are sampled uniformly within fixed-range boundaries that are chosen in advance. However, in the case of random goal spaces, we do not know in advance in which region of the space goals will be encoded. Therefore, we set the range to [-min, max] for each of the latent variables, to also bias exploration towards the boundaries of the discovered goal space.

IMGEP-PGL (prelearned goal space): IMGEP (Algorihm 4) with a goal space defined by a VAE that was trained before the exploration starts. The VAE is trained on a dataset with precollected Lenia