Anil Yaman
email: anilyaman@kaist.ac.kr

Nicolas Bredeche

Onur Ç Aylak

Joel Z Leibo

Sang Wan Lee

Onur C ¸aylak

Meta-control of social learning strategies

Introduction

Learning is one of the basic requirements for animal survival. The ultimate goal of learning is to acquire reliable knowledge from a limited amount of interactions with the environment. However, the environment is often uncertain and volatile, making it difficult to learn.

Decades of studies found that animals have multiple learning strategies. For example, an animal can learn associations between environmental cues and outcomes (Pavlovian learning) or learn actionoutcome associations (model-free reinforcement learning). It is a simple strategy but less adaptive to environmental changes because the action-outcome associations are gradually updated based on experience. A more sophisticated strategy is to learn the internal model of the environmental structure and to use this information to quickly perform actions in a more predictive manner (modelbased reinforcement learning). While this strategy can accelerate the adaptation, encoding the internal model of the environment requires additional memory and computations [START_REF] Lee | Neural computations underlying arbitration between model-based and model-free learning[END_REF][START_REF] John | Dissociable roles of ventral and dorsal striatum in instrumental conditioning[END_REF].

Recent studies in neuroscience suggest that the brain can find a compromise between these learning strategies via a process called meta-control [START_REF] Nathaniel D Daw | Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control[END_REF][START_REF] Wang | Prefrontal cortex as a meta-reinforcement learning system[END_REF][START_REF] Olsson | The neural and computational systems of social learning[END_REF][START_REF] Eppinger | Meta-control: From psychology to computational neuroscience[END_REF]. Meta-control is based on the premise that learning strategies have different levels of sensitivity to environment variability and this can be measured by perceptual uncertainty concerning the association of actions and rewards [START_REF] Nathaniel D Daw | Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control[END_REF]. Thus, perceptual uncertainty can be used to arbitrate between learning strategies. For example, in a stable environment, the brain prefers to use a sample-efficient model-based strategy, followed by a gradual transition to a computationally-efficient model-free learning strategy [START_REF] Gläscher | States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning[END_REF]. On the other hand, in environments with high perceived uncertainty, model-free learning are preferred over model-based learning because they are less susceptible to environmental uncertainty [START_REF] Lengyel | Hippocampal contributions to control: The third way[END_REF]. Accumulating evidence suggests that a part of the prefrontal cortex implements meta-control of various learning strategies, which provides a cost-effective solution to environmental uncertainty [START_REF] Lee | Neural computations underlying arbitration between model-based and model-free learning[END_REF][START_REF] Kim | Task complexity interacts with state-space uncertainty in the arbitration between model-based and model-free learning[END_REF][START_REF] John | Why and how the brain weights contributions from a mixture of experts[END_REF]. Ultimately, computational models of the brain's meta-control principle should find a way to efficiently avoid complications arising from environmental variability [START_REF] Jee | Toward highperformance, memory-efficient, and fast reinforcement learning-lessons from decision neuroscience[END_REF].

Taking full advantage of the brain's meta-control capability of learning strategies, this paper proposes a meta-control approach to social learning, which we term meta-social learning. The proposed method aims to resolve environmental uncertainty by arbitrating between individual learning and two different social learning strategies, each of which exhibits a different uncertainty-sensitive performance-cost profile.

Both the individual and social learning strategies play a crucial role in learning as a population.

Innovations are usually made by individual learning (IL) and spread throughout the population via social learning (SL) [START_REF] Henrich | The secret of our success: how culture is driving human evolution, domesticating our species, and making us smarter[END_REF][START_REF] Lewis G Dean | Human cumulative culture: a comparative perspective[END_REF][START_REF] Gabora | An evolutionary framework for cultural change: Selectionism versus communal exchange[END_REF]. However, these strategies involve advantages and drawbacks, suggesting the need to trade social learning strategies off with individual learning [START_REF] Charpentier | A neuro-computational account of arbitration between choice imitation and goal emulation during human observational learning[END_REF][START_REF] Heyes | Who knows? metacognitive social learning strategies[END_REF].

Individual learning can explore and discover useful innovations, however, it can be costly due to exploration, risk of injury, mortality, etc. [START_REF] Kendal | The evolution of social learning rules: payoff-biased and frequency-dependent biased transmission[END_REF]. It is only worthwhile to bear these costs if learning is required to adapt to the environmental changes. In static environments on the other hand, it would be unnecessary to pay these costs. To resolve this dilemma, one can suggest adapting the exploration rate so that it depends on the environment's variability (e.g. [START_REF] Tokic | Adaptive ε-greedy exploration in reinforcement learning based on value differences[END_REF]). However, in this case, individual learners would still need to explore the action space by themselves to find the optimum behavior. This is inefficient if the optimum behavior was already discovered by other individuals in the population, and it can be readily copied. In that case it would be beneficial to make use of the knowledge explored by others in order to avoid paying the cost of exploring oneself.

In group-living animals in nature, social learning has evolved to take advantage of the exploration performed by others via copying their behavior, thereby reducing the cost of learning. Therefore, it does not involve these costs related to individual learning [START_REF] Heyes | When does social learning become cultural learning[END_REF][START_REF] Rachel L Kendal | Social learning strategies: Bridge-building between fields[END_REF][START_REF] Toyokawa | Human collective intelligence under dual exploration-exploitation dilemmas[END_REF][START_REF] Boyd | Culture and the evolutionary process[END_REF]. On the other hand, social information can be less accurate since it depends on the observation of previously performed behaviors (i.e. may be outdated in case of environment change). Moreover, it requires identifying the individuals with reliable knowledge to copy. The term social learning strategy (SLS) refers to any of a variety of methods by which individuals can choose others to copy [START_REF] Laland | Social learning strategies[END_REF][START_REF] Rachel L Kendal | Social learning strategies: Bridge-building between fields[END_REF][START_REF] Whiten | The burgeoning reach of animal culture[END_REF][START_REF] Heyes | When does social learning become cultural learning[END_REF][START_REF] Tjh Morgan | The evolutionary basis of human social learning[END_REF].

The efficiency of individual and social learning strategies in stable and dynamic environments has been demonstrated through theoretical and empirical studies [START_REF] Henrich | The evolution of conformist transmission and the emergence of between-group differences[END_REF][START_REF] Kameda | Cost-benefit analysis of social/cultural learning in a nonstationary uncertain environment: An evolutionary simulation and an experiment with human subjects[END_REF][START_REF] Aoki | The emergence of social learning in a temporally changing environment: a theoretical model[END_REF][START_REF] Kendal | The evolution of social learning rules: payoff-biased and frequency-dependent biased transmission[END_REF][START_REF] Kandler | Tradeoffs between the strength of conformity and number of conformists in variable environments[END_REF]. For instance in a computer tournament, Rendell et al. [START_REF] Rendell | Why copy others? insights from the social learning strategies tournament[END_REF] noted the success of strategies that rely heavily on social learning over individual learning. They tested competing strategies on a fundamental decisionmaking problem known as the multi-armed bandit problem [START_REF] Richard | Reinforcement learning: An introduction[END_REF] (or k-armed bandit). Furthermore, they modeled a changing adaptive environment by adjusting the reward association of the arms during the task (it was a "restless" or "non-stationary" multi-armed bandit problem) [START_REF] Rendell | Why copy others? insights from the social learning strategies tournament[END_REF][START_REF] Schlag | Why imitate, and if so, how?: A boundedly rational approach to multi-armed bandits[END_REF][START_REF] Koulouriotis | Reinforcement learning and evolutionary algorithms for non-stationary multi-armed bandit problems[END_REF][START_REF] Groß | Simple learning rules to cope with changing environments[END_REF]. Despite these efforts, a fundamental issue in social learning still remains unaddressed: whether to use success-based social learning and copy the behavior of the successful individuals or simply follow the conformist strategy and copy the behavior that is the majority in the population. This poses a fundamental challenge for the individual, forcing them to confront a trade-off between performance and exploration cost [START_REF] Jee | Toward highperformance, memory-efficient, and fast reinforcement learning-lessons from decision neuroscience[END_REF].

To fully examine this issue, we performed an evolutionary analysis on individual learning and two social learning strategies, success-based and conformist [START_REF] Krebs Denton | Cultural evolution of conformity and anticonformity[END_REF][START_REF] Rachel L Kendal | Social learning strategies: Bridge-building between fields[END_REF][START_REF] Henrich | The evolution of conformist transmission and the emergence of between-group differences[END_REF][START_REF] Kendal | The evolution of social learning rules: payoff-biased and frequency-dependent biased transmission[END_REF][START_REF] Nakahashi | The evolution of conformist transmission in social learning when the environment changes periodically[END_REF]. Despite several investigations into these strategies, the effect of environment uncertainty on their performance has remained largely unaddressed, making it hard to connect to the meta-control idea in neuroscience.

To fill this gap, we designed non-stationary multi-armed bandit tasks with variable amounts of uncertainty, for which we systematically manipulated the reward distributions. This setup has been used as an abstracted model of fundamental decision making processes in nature, such as foraging, predator avoidance, symbiosis, and mutualism [START_REF] Tjh Morgan | The evolutionary basis of human social learning[END_REF][START_REF] Coolen | Species difference in adaptive use of public information in sticklebacks[END_REF][START_REF] Mike | Social learning strategies and predation risk: minnows copy only when using private information would be costly[END_REF], as well as human decision making processes, such as human-robot interactions, investment decisions in stock markets, consumer decision making, dynamics of social networks, etc.

Our simulations confirmed the view that while the success-based strategy is vulnerable to environmental uncertainty, the conformist strategy serves as an alternative that can effectively resolve the adverse effect of uncertainty. These results show that neither individual learning, success-based, nor conformist social learning strategies are on their own sufficient to achieve a optimal policy for lifetime learning. This view motivates us to hypothesize that there exists an ideal combination of these strategies to cope with the environment volatility and uncertainty. In doing so, we propose a meta-social learning strategy that uses estimated uncertainty to arbitrate between individual and social learning strategies.

To test our hypothesis, this model was pitted against a large set of algorithms implementing various strategies based on reinforcement learning [START_REF] Richard | Reinforcement learning: An introduction[END_REF], genetic algorithms [START_REF] Agoston E Eiben | Introduction to evolutionary computing[END_REF] and neuroevolution [START_REF] Yaman | Limited evaluation cooperative co-evolutionary differential evolution large-scale neuroevolution[END_REF][START_REF] Kenneth O Stanley | Designing neural networks through neuroevolution[END_REF] in environments with various levels of volatility and uncertainty. The results demonstrate that the proposed model serves to achieve near-optimal lifetime learning in the sense that it resolves the performance-exploration cost dilemma. We also show that the version of the models In earlier stages of the process (initial and after environment change, shown in green dash lines), populations with success-based social learning strategy achieves higher average population reward faster relative to the conformist strategy. In later stages of the process (shown in purple dashed lines), populations with success-based social learning strategy achieves higher average population reward in environments with low uncertainty, whereas, populations with conformist social learners achieves higher average population reward in environments with high uncertainty.

Popula�on

implementing our hypothesis tends to have a higher ratios in the populations and survive longer relative to the others throughout our evolutionary analysis.

Results

Uncertainty-invariance in social learning

To examine whether and to what extent environmental uncertainty influences performance of individual and social learning strategies, we compared adaptation performance of various types of learning strategies in different levels of environmental uncertainty. We considered independent populations consisting of individual learners using the success-based social learning strategy, and individual learners with the conformist strategy. To model the evolutionary dynamics of these populations, we used two distinct approaches: (1) a mathematical model based on the replicator-mutator equa-tion [START_REF] Komarova | Replicator-mutator equation, universality property and population dynamics of learning[END_REF][START_REF] Nowak | Evolutionary dynamics: exploring the equations of life[END_REF], and (2) an agent-based evolutionary algorithm [START_REF] Agoston E Eiben | Introduction to evolutionary computing[END_REF]. These approaches allow us to tract the change in the ratios of the individual and social learners within a certain environment.

In the former case, the change in the frequencies of individual/social learners selecting a given arm are modeled by a system of coupled first order differential equations. This approach has largely been used in evolutionary game theory [START_REF] Maynard | Evolution and the Theory of Games[END_REF][START_REF] Hofbauer | Evolutionary game dynamics[END_REF]. The fitness of the types of individuals was defined based on the rewards received. Note that the individual learners carry a constant computational cost of learning. To the contrary, the social learners avoid this issue by simply copying the others' choice, although they are deemed to endure a latency issue arising from the necessity of observing the past choices of the others.

In the latter case, we simulate the evolutionary process involving a population of individual and social learners, each of which were modelled separately. The individual learners have the capacity to improve their behavioral policy over time based on their experience. They were implemented using the -greedy algorithm [START_REF] Richard | Reinforcement learning: An introduction[END_REF] in which an exploration parameter is the probability per timestep of taking a random (uniformly distributed) action instead of taking the current greedy action with the highest average reward. Note that this exploration carries an extra cost when the individual is already making an optimal choice (see Section 4). The social learners, on the other hand, perform their actions by copying others with a certain level of latency. This process is repeated in each generation where all the individuals made their choices and receive their rewards. At the end of each generation cycle, individuals were sampled with replacement for the next generation proportional to their fitness.

An example illustration of a binary decision-making task given in Figure 1 (a) and (b). The reward distributions are parameterized by Gaussian distributions with a mean (µ) and standard deviation (σ). Since the performance of success-based social learning is contingent on correctly identifying agents making optimal choices, we hypothesized that the high uncertainty in the environment would make it hard to identify successful individuals, leading to the degradation of its learning outcomes. To test this, we designed novel tasks with different levels of uncertainty, controlled by the degree of overlap between reward distributions of different arms. The fitness values of the individuals are defined as the amount of rewards received following their choice. To assess the individuals' adaptation ability to environmental changes, at the midpoint of each simulation we reversed the association between arms and their reward distributions.

Conformist

Figure 2: The success-based strategy shows the best performance in environments with low uncertainty, however, it suffers when there is uncertainty in the environment. On the other hand, the conformist strategy achieves similar performance independent of the environment uncertainty.

the uncertainty in the environment is low, the success-based social learning strategy shows the best performance in terms of adaptation after an environment change (p < 0.01; Wilcoxon rank-sum test [START_REF] Wilcoxon | Individual comparisons by ranking methods[END_REF]), and at the end of the process, the success-based and conformist strategies show similar performance, and they are superior to individual learning only. However, when the uncertainty in the environment is increased, populations with conformist social learners achieve higher average population reward (p < 0.01).

Uncertainty as a predictor of social learning performance

To further investigate the relationship between environmental uncertainty and social learning strategies, we measured the performance difference between the success-based and conformist strategy as a function of the amount of uncertainty in reward distributions. We refer to this uncertainty as the optimum distribution prediction uncertainty (ODPU) because it undermines the ability of the success-based strategy to correctly identify individuals making optimal choices. We computed the ODPU directly by the probability of receiving better rewards from the sub-optimal reward distributions. For example, if the ODPU is high, it is more likely to mistake an individual making sub-optimal choice as a successful individual and copy its action.

Considering a population consists of M and N individuals making choices to collect rewards from the environment associated with certain reward distributions. In this case, the ODPU depends not only on the sufficient statistics of the reward distributions but also on the size of the subgroup of individuals making optimal and sub-optimal choices, denoted by M and N , respectively. In Figure 3, we illustrate the performance difference between the success-based and conformist strategies as a function of the ODPU on two Gaussian reward distributions. The generalized version of the ODPU that can be applied to more than two reward distributions is provided in Section 4.4.

In Figures 3c and3d, we show the ODPU when M = 5, N = 95 and M = 50, N = 50 depending on σ 1 , σ 2 respectively. Overall, the smaller the number of individuals making optimal choices, the larger the ODPU. In addition, an increase in σ 1 and a decrease in σ 2 causes an increase in the ODPU. Figure 3e shows the strong correlation between the ODPU and the difference between performance of the conformist and of the success-based strategy (with Pearson's correlation coefficient r = 0.9791).

Note that after a certain ODPU value (i.e. approximately around 0.1 -0.2 in Figure 3e), their performance difference becomes highly significant. The maximum possible performance difference depends on the difference in their µ.

Meta-social learning hypothesis

In this section, we propose meta-social learning as a way for agents to arbitrate between individual and social learning strategies during their lifetime. We explored this hypothesis using several approaches. First, we used context encoding approach to determine the "context" of the environment by estimating three environmental variables, namely, environment change (EC(t)), conformity (C(t)) and uncertainty (U (t)) that play crucial role in the performance of the individual and social learning strategies. Then, based on our analysis, we defined meta-social learners that can arbitrate between these strategies depending on the context of the environment. Furthermore, as alternative approaches, we used evolutionary algorithms and reinforcement learning to optimize the meta-social learners. Finally, we defined several baseline strategies that perform individual and social learning strategies randomly with a predefined probabilities.

Context encoding

We utilize the "social information" in estimating environment change, conformity and uncertainty.

the social information is assumed be available for all individuals in the population and it consists of the action distribution of the population h(a j , t) ∈ H and the rewards received by the individuals r i (a j , t) ∈ R where h(a j , t) denotes the frequency of action a j in the population and r i (a j , t) denotes the reward of individual i by performing action a j at time t. From the reward distribution, it is possible to estimate average rewards µ j (t) and standard deviations σ j (t) of actions a j . Note that the social information is the same as the information2 required to perform the success-based and conformist strategies. Environment change. It is defined as the difference between the current and previous values of average rewards of the estimated optimum action:

EC(t) =    1, if |µ * (t) -µ * (t -δ)| > th ec , 0, otherwise.
where subscript * denotes the estimated optimum action (that is the action with the highest average reward), δ is a parameter for comparing previous values of the average rewards, and th EC is threshold for triggering the environment change detection. Threshold th EC can depend on the task. In our experiments, we assign 0.15 for this threshold.

Conformity. It is based on the estimation whether the majority of the individuals are performing the behavior with the highest average reward. Thus, it is defined as:

C(t) =          1 (conformity), if arg max j µ j (t) = arg max j h(a j , t), 0 (non-conformity), if EC(t) = 1, 0 (non-conformity), otherwise.
Furthermore, if an environment change is detected, conformity is reset to 0. Uncertainty. Estimated based on the ODPU (the probability of sampling higher reward values from sub-optimum distributions, see Sections 2.2 and 4.4). We use average rewards µ j (t) and standard deviations σ j (t) to compute the ODPU. Then, the uncertainty is detected as:

U (t) =    1 (high uncertainty), if ODP U > th u , 0 (low uncertainty), otherwise.
where uncertainty threshold th u is set to 0.1 in our experiments based on our uncertainty analysis in Section 2.2.

Meta-control of social learning strategies

A generic representation of the strategy selection process of a meta-social learner is shown in Equation 1. A strategy S ∈ {individual learning, success-based and conformist} is selected by meta-social learner M SL() based on the context of the environment.

S := M SL(EC(t), C(t), U (t)) (1)
In addition, we implement several other meta-social learning control mechanisms using various approaches and discuss under four types as follows (for implementation details of these algorithms, see Section 4. Evolutionary control. The control policies for arbitrating between individual and social learning strategies were achieved by evolutionary algorithms. We used different environments (provided in Supplementary Material) for training and testing. We performed evolutionary-based training for 10 independent runs and selected the best performing strategy. Then, we tested this strategy on the test environments that were not encountered during the training processes and reported the results.

The goal of this separation was to demonstrate the generalization capability of the train model. We implemented two versions.

• SL-GA (trained by the genetic algorithms) perform the task based on the rules optimized with the genetic algorithms [START_REF] Agoston E Eiben | Introduction to evolutionary computing[END_REF]. These rules are optimized to select individual and social learning strategies depending on the binary states of EC(t), C(t) and U (t) (see Section 4.6.2). It is possible to explore all possible rules (based on all possible states of EC(t), C(t) and U (t))

in this space to identify the "optimal" rule. We note that the best performing evolutionary control mechanism3 found by SL-GA converged to SL-EC-Conf-Unc.

• SL-NE (artificial neural network trained by neuroevolution) utilizes an artificial neural network to control meta-social learning, whose parameters were optimized by an evolutionary algorithm (known as neuroevolution approach [START_REF] Kenneth O Stanley | Designing neural networks through neuroevolution[END_REF]). The fully connected feedforward network (FCN) with one hidden layer takes the average rewards of arms (µ 1 (t), . . . , µ k (t)), standard deviations (σ 1 (t), . . . , σ k (t)) and the frequencies of the individuals that select k arms (h(a 1 , t), . . . , h(a k , t)), and chooses a strategy as follows:

S := F CN (µ 1 (t), . . . , µ k (t), σ 1 (t), . . . , σ k (t), h(a 1 , t), . . . , h(a k , t)) (2)
Note that unlike other versions of meta-social learning, this one does not require identifying the context of the environment such as environment change, uncertainty or conformity.

Multi-armed bandit control. We used SL-RL ( -greedy algorithm), SL-UCB (upper confidence bound algorithm) and SL-QL (Q-learning) algorithms to learn to choose between individual, success-based and conformist social learning strategies. SL-RL and SL-UCB does not use environment context, rather, they perform a strategy selection based on the estimated rewards of selecting these strategies. The estimated rewards of these strategies are updated based on the rewards received after their selection. The SL-QL uses a value learning, intended to maximize the expected amount of future rewards. We used the context of the environment (environment change, uncertainty and conformity) as the states.

Other baseline strategies. We further implemented a set of baseline strategies: SL-Rand, SL-Prop, SL-Conf, SL-Succ, and IL-Only that perform a random strategy with a predefined fixed probability (see Section 4.6 for details). These meta-social learning strategies do not make use if the context of the environment. On the right of each figure, the labels of the meta-social learning strategies (A through M) were ranked from best to worst based on their performance values. Decimal numbers on the left indicate their average ranks (the lower is the better), and the differences in their ranks that are not statistically significant at α = 0.05 are linked by vertical black lines.

SL-GA (I) SL-NE (J) SL-EC-Conf-Unc (K) SL-EC-Conf (L) SL-EC-Succ (M) IL-Only (A) SL-Rand (B) SL-Prop (C) SL-Conf (D) SL-Succ (E) SL-RL (F) SL-UCB (G) SL-QL (H) Name (ID)
I SL-GA K SL-EC-Conf-Unc M SL-EC-Succ J SL-NE L SL-EC-Conf E SL-Succ C SL-Prop F SL-RL G SL-UCB H SL-QL B SL-Rand A IL-Only D SL-Conf Rank ID (c)
I SL-GA K SL-EC-Conf-Unc M SL-EC-Succ J SL-NE L SL-EC-Conf E SL-Succ C SL-Prop F SL-RL G SL-UCB H SL-QL B SL-Rand A IL-Only D SL-Conf ID (d) Volatile

Uncertainty based meta-control resolves the trade-off between performance and exploration cost

To compare the performance of the meta-social learning algorithms on a task with uncertainty changes, first we defined stable and volatile environments with low and high uncertainty (See Supplementary Material, Section A.5). To construct these environments, we arbitrarily generated a set of six reward distributions (Figures 13a through 13f Overall, both the SL-GA and SL-EC-Conf-Unc achieved the highest performance with lowest exploration cost (the performance of the two models are not significantly different; Figure 4). This is due to the fact that the evolved controller in SL-GA is converged to the same controller used in the SL-EC-Conf-Unc. We note that the SL-NE provides one of the top five ranking results even though it uses low level population based features with artificial neural networks.

In general, the models utilizing the conformist strategy showed better performance in uncertain environments, compared to the ones with the success-based strategy. From the exploration cost point of view, we note that the IL-Only suffers from the highest cost with about three times more costly than the second most costly meta-social learner. In the case of the uncertain environment with gradual environment change (Experiment3; see Figure 4f), it is not surprising that the algorithms relying on the threshold-based environment change detection (SL-EC-Conf, SL-EC-Succ, SL-EC-Conf-Unc) did not perform well, which is ascribed to the failure in detecting environment change. To the contrary, SL-NE showed reliable performance robust against environment changes even though it was trained on the environments with different conditions. Note that this remarkable adaptation ability does not require an explicit environment change detection mechanism.

Uncertainty based meta-control dominates the evolution in volatile environments

What if there is a competition between different meta-social learning strategies in environments with various levels of volatility and uncertainty? Which ones would persist in the populations and become dominant relative to others? To assess that, we conducted an evolutionary analysis on meta-social learning. We further recorded how long they stay in the population (age) to assess their resilience.

This analysis shows us successful strategies that are not being invaded by other strategies even in changing environmental conditions.

In the beginning of the evolutionary processes, we assigned each individual a specific type of metasocial learning strategy, randomly sampled from the complete set of the social learning strategies (i.e.

IL-Only, SL-Rand, SL-Prop, ..., SL-EC-Conf-Unc). The individuals then used their own meta-social learning strategy to perform the tasks. After each generation, meta-social learners are selected based on the probability proportional to their fitness values (rewards received in the previous generation).

Furthermore, we introduce a mutation operator that re-samples the type of meta-social learning strategy of each individual at each generation based on a small probability controlled by mutation rate mr. At every generation, the age of all strategies is increased by one. It is possible to pass multiple copies of a strategy to the next generation during the selection process. In this case, multiple copies are treated as offspring where only the age of the first copy is preserved while the age of the others is set to zero. Similarly, after mutation the age of the strategy is set to zero.

Figure 5 shows the population dynamics of meta-social learning algorithms during the evolutionary processes 5 . The meta-social learning strategies that perform well relative to the others show increase in their ratios in the population, whereas, the ones that cannot perform well show decrease in their ratios, and eventually die out 6 .

Overall, in the environments with low uncertainty, seven meta-social learning strategies show an increase in their ratios at the end of the processes relative to their starting ratios, whereas, in the environments with high uncertainty, only four of them show increase in their ratios. The most dominating four meta-social learners are: SL-EC-Conf-Unc, SL-GA, SL-EC-Conf and SL-QL.

These meta-strategies, with the exception of SL-EC-Conf, make use of the environment uncertainty information. Remarkably, SL-EC-Conf is able to compete with the others using conformity bias without the need of using environment uncertainty.

We note that even though meta-social strategies such as SL-GA, SL-EC-Conf-Unc that failed to perform well in gradual environment (see Figure 4f) by themselves, they can show domination over other strategies in this experiment (see Figure 5f). This is due to the fact that the populations that consist of only these strategies cannot detect the environment change leading to the failure of exploring the other arm after an environment change. However, when they are used in combination with the other meta-social strategies (e.g. SL-Rand, SL-IL, SL-NE, ...), the exploration performed by other meta-social strategies helps overcome this effect. Consequently, these strategies (that fail due to the environment detection mechanisms) can perform well and dominate the populations.

We performed an analysis on the age distributions of the meta-social learning strategies during the evolutionary processes (provided in Figure 16 in Supplementary Material A.6). Sudden changes in the age distribution due to the environment change can be observed. The dominant meta-social learners show higher life expectancy throughout the processes especially in the environments with high uncertainty.

IL
We further hypothesize that the life expectancy of the dominant meta-social learning strategies should be consistent with key variables of the evolutionary process, such as the selection strength and mutation rate. To test this we performed further experiments, in which we ran the same simulations while varying the level of selection strengths and mutation rates (as low, moderate and high). The selection strength is given by p i = f s i / m j=1 f s j , where p i is the selection probability of a meta-social learning strategy i, to pass to the next generation, f i is its fitness value, and m is the total number of meta-social learners.

While the selection strength increases, the life expectancy of the dominant strategies increases, whereas, the life expectancy of the others decreases (provided in Supplementary Material A.7, see Figure 17). Furthermore, it can be observed that, while the mutation rate increases, the life expectancy of the dominant strategies decreases. This is due to the fact that, when the mutation rate is high, the probability of randomly mutating a dominant strategy increases, causing their life expectancy to decrease.

Discussion

While previous research examined individual and social learning strategies in the context of a changing environment [START_REF] Rendell | Why copy others? insights from the social learning strategies tournament[END_REF][START_REF] Henrich | The evolution of conformist transmission and the emergence of between-group differences[END_REF][START_REF] Kameda | Cost-benefit analysis of social/cultural learning in a nonstationary uncertain environment: An evolutionary simulation and an experiment with human subjects[END_REF][START_REF] Aoki | The emergence of social learning in a temporally changing environment: a theoretical model[END_REF][START_REF] Kendal | The evolution of social learning rules: payoff-biased and frequency-dependent biased transmission[END_REF][START_REF] Kandler | Tradeoffs between the strength of conformity and number of conformists in variable environments[END_REF], this study tested a new hypothesis that measurements of environmental uncertainty can be used as a means to implement a reliable and cost-efficient learning strategy, regardless of environmental changes. To test this hypothesis, we performed an analysis on individual learning and two social learning strategies, namely success-based and conformist, on volatile and uncertain environments. Our analysis showed that the performance of the success-based strategy, the most direct way to explore to find the optimal policy, is susceptible to uncertainty in the environment, whereas that of the conformist strategy, though it does not guarantee the optimal performance, is highly reliable. Motivated by these results, we proposed several meta-social learning algorithms. Overall, the proposed meta-social learning strategies showed significantly better performance, and in the evolutionary analysis, they dominated other meta-social learning approaches in terms of survival rate.

The proposed meta-social learning scheme is motivated by the recent theoretical idea in neuroscience, suggesting that the brain uses meta-learning strategies to find a compromise between different types of learning, such as Pavlovian, model-free, and model-based learning [START_REF] Nathaniel D Daw | Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control[END_REF][START_REF] Wang | Prefrontal cortex as a meta-reinforcement learning system[END_REF][START_REF] John | Why and how the brain weights contributions from a mixture of experts[END_REF][START_REF] John | The structure of reinforcementlearning mechanisms in the human brain[END_REF].

Accumulating evidence suggests that meta-learning allows individuals to resolve environmental uncertainty efficiently [START_REF] Lee | Neural computations underlying arbitration between model-based and model-free learning[END_REF][START_REF] Kim | Task complexity interacts with state-space uncertainty in the arbitration between model-based and model-free learning[END_REF][START_REF] Jee | Toward highperformance, memory-efficient, and fast reinforcement learning-lessons from decision neuroscience[END_REF][START_REF] Charpentier | A neuro-computational account of arbitration between choice imitation and goal emulation during human observational learning[END_REF][START_REF] Collette | Neural computations underlying inverse reinforcement learning in the human brain[END_REF]. Critically, this view is fully consistent with our finding that meta-social learning strategy mitigates the adverse effect of environmental uncertainty on performance. Our computational framework can thus be used to examine neural mechanisms of meta-social learning [START_REF] Olsson | The neural and computational systems of social learning[END_REF].

Our meta-social learning framework provides a means to examine complex population dynamics, thereby helping us better understand the fundamental nature of biological learning and decision making [START_REF] Tjh Morgan | The evolutionary basis of human social learning[END_REF][START_REF] Charpentier | A neuro-computational account of arbitration between choice imitation and goal emulation during human observational learning[END_REF][START_REF] Coolen | Species difference in adaptive use of public information in sticklebacks[END_REF]. For example, our meta-social learning principle would provide theoretical insight into why animals or humans often copy others' behavior and why society needs to achieve conformity, especially in highly volatile situations. It would also be possible to examine how animal societies cope with environmental uncertainty and volatility. Moreover, the meta-social learning scheme can be extended to explain various types of ecological interactions, such as symbiosis, mimicry, and mutualism.

In artificial intelligence and robotics applications, nature inspired approaches have proven to be successful in modeling intelligent behavior [START_REF] Bredeche | Embodied evolution in collective robotics: A review[END_REF][START_REF] Agoston E Eiben | Embodied, on-line, on-board evolution for autonomous robotics[END_REF][START_REF] Bredeche | Environment-driven distributed evolutionary adaptation in a population of autonomous robotic agents[END_REF][START_REF] Haasdijk | Combining environment-driven adaptation and task-driven optimisation in evolutionary robotics[END_REF].

Accordingly, social learning aims to benefit from the collective property of multi-agent systems to provide efficient learning and adaptation as a population. As illustrated in this work, exploiting the behaviors of other individuals can reduce exploration cost significantly. It can also improve learning efficiency in uncertain and volatile environments. This may prove to be important in real-world applications such as the internet of things and swarm robotics [START_REF] Yaman | Distributed embodied evolution over networks[END_REF][START_REF] Atzori | The internet of things: A survey[END_REF][START_REF] Lin | A survey on internet of things: Architecture, enabling technologies, security and privacy, and applications[END_REF][START_REF] Michael Rubenstein | Programmable self-assembly in a thousand-robot swarm[END_REF][START_REF] Ebert | Multi-feature collective decision making in robot swarms[END_REF]. For instance, one recent example of distributed learning approach has been used in healthcare to detect illnesses [START_REF] Warnat-Herresthal | Swarm learning for decentralized and confidential clinical machine learning[END_REF].

Meta-social learning strategies can play a key role in these kinds of distributed learning applications to improve the efficiency of learning.

Rational choice theory in economics and game theory suggests that individuals choose their best action through a cost-benefit analysis which we usually conceptualize as involving explicit deduction (thinking through pros and cons) [START_REF] Satz | Rational choice and social theory[END_REF][START_REF] Scott | Understanding contemporary society: Theories of the present[END_REF]. Since our results suggest that an individual can make cost-effective decisions instead via social information, i.e., the decisions of others and their outcomes, it may be useful to consider models based on such foundations as well. For instance, it would be possible to estimate the environment uncertainty and volatility simply by measuring individual choice variability. This inference based on social information improves sample efficiency significantly compared to individual learning.

In addition to the computational and theoretical implications of meta-social learning, another exciting research direction is to use meta-social learning to examine the fundamental nature of social networks [START_REF] Degenne | Introducing social networks[END_REF]. For instance, complex dynamics of individual interactions can lead to the emergence of various "social learning networks". Investigation of fundamental computations underlying the emergence and evolution of social networks would allow us to understand and predict the future of animal societies. Furthermore, the computational framework of meta-social learning be used to test new hypotheses about multi-agent social learning [START_REF] Joel Z Leibo | Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research[END_REF][START_REF] Baker | Emergent tool use from multi-agent autocurricula[END_REF]. For instance, it is possible to test whether complex internal dynamics arising from meta-social learning promote the natural emergence of curriculum.

Methods

Multi-armed bandit problem

The multi-armed bandit problem is a classic problem in reinforcement learning that [START_REF] Rendell | Why copy others? insights from the social learning strategies tournament[END_REF][START_REF] Koulouriotis | Reinforcement learning and evolutionary algorithms for non-stationary multi-armed bandit problems[END_REF][START_REF] Rendell | Why copy others? insights from the social learning strategies tournament[END_REF][START_REF] Richard | Reinforcement learning: An introduction[END_REF] where individuals are required to perform actions to choose one of k alternative choices (also known as k-arms). Performed actions provide rewards based on their underlying distributions that is unknown to the individual. In our work, We model the reward distribution of each action as a Gaussian distribution {N (µ 1 , σ 1 ), . . . , N (µ k , σ k )}. The goal of an individual is to perform actions to choose repetitively one of the choices and collect the rewards for a certain period of time such a way that can maximize the cumulative sum of the rewards received during the process.

Individual learning model

Individual learning is modeled as a reinforcement learning agent using -greedy algorithm [START_REF] Richard | Reinforcement learning: An introduction[END_REF]. We denote estimated reward of an action a at time t as Q(a, t). The reward estimation is updated based on the rewards r(a, t) when a is chosen using Equation [START_REF] Baker | Emergent tool use from multi-agent autocurricula[END_REF].

Q(a, t + 1) = Q(a, t) + β [r(a, t) -Q(a, t)] (3)
where 0 < β ≤ 1 is the step size which was set to 0.2 in our experiments. Equation ( 4) shows the -greedy algorithm where the behavior with the highest estimated reward or a random behavior is chosen with the probabilities of 1 -and respectively. We set = 0.1 in our experiments. (4)

Social learning models

Social learners copy the behavior of other individuals in the population based on a certain strategy [START_REF] Rachel L Kendal | Social learning strategies: Bridge-building between fields[END_REF]. We implement two social learning strategies as: success-based and conformist given in below:

a(t) =    arg max a r(a, t -τ ), if success-based strategy, arg max a h(a, t -τ ), if conformist strategy.
where r(a, t -τ ) and h(a, t -τ ) denote the reward and frequency of an action a at time t -τ with some latency τ . Social learning is performed only when t -τ > 0.

In success-based strategy, social learners copy the behavior of the individual with the best reward at t-τ , and in conformist strategy, social learners copy the most frequent behavior in the population at time t -τ .

Optimum distribution prediction uncertainty

Uncertainty of the environment (U (t)) is estimated based on the probability of sampling higher reward values from sub-optimum distributions. We refer to this probability as the optimum distribution prediction uncertainty (ODPU) and define as:

Let {X 1 j1 } j1 , . . . , {X k j k } j k be k sets of normally distributed random variables, where the random variables of set i are independently drawn from N (µ i , σ i ). All k sets are finite, where j i represents an integer value such that 0 < j i ≤ N i < ∞ for all i = 1, . . . , k. The notation X i (Ni) is used to indicate the N i -th order statistic. That is, the maximum of all random variables of a given set {X i ji } ji . Now, using the fact that the random variables are independently drawn from normal distributions, one can write the probability density function, f , and the cumulative distribution function, F , of the N (i) -th order statistic as

f X i (N i ) (x) = N i σ i φ x -µ i σ i Φ x -µ i σ i Ni-1 , F X i (N i ) (x) = Φ x -µ i σ i Ni .
The optimum distribution prediction uncertainty is then be formulated by

ODPU = 1 -P(X 1 (N1) ≥ X i (Ni) ; for all i ≥ 2), = 1 - ∞ -∞ f X 1 (N 1 ) (y) • F X 2 (N 2 ) (y) • • • • • F X k (N k ) (y) dy,
which describes the probability of sampling higher values from the distributions with lower means relative to µ 1 .

Evolution of social learning

Mathematical model

This section provides our mathematical model for analysing the evolution of social learning strategies on the multi-armed bandit problem for number of arms k = 2. Let a i , i ∈ {1, 2} denote actions with corresponding payoffs r(a i , t) at time t.

The frequency of the individual learners in the population at time t is denoted as IL(t). We further distinguish two types of individual learners A 1 (t) and A 2 (t) to indicate the frequencies of the individuals that perform actions a 1 and a 2 . The sum of the frequencies of the behaviors satisfy the following condition:

A 1 (t) + A 2 (t) = IL(t) for all t ≥ 0.
We denote the frequencies of the social learners in the population as SL(t) at time t. Overall, the sum of the individual and social learners in the population satisfy the following condition:

IL(t) + SL(t) = 1 for all t ≥ 0.
The change in the frequencies of A 1 , A 2 and SL are modeled using the replicator-mutator equation [START_REF] Komarova | Replicator-mutator equation, universality property and population dynamics of learning[END_REF][START_REF] Nowak | Evolutionary dynamics: exploring the equations of life[END_REF] given as a system of coupled first order ordinary differential equations below7 :

               Ȧ1 (t) = F(t)[I T (t) • col 1 (M)] -A 1 (t)ψ(t), t > 0, Ȧ2 (t) = F(t)[I T (t) • col 2 (M)] -A 2 (t)ψ(t), t > 0, ṠL(t) = F(t)[I T (t) • col 3 (M)] -SL(t)ψ(t), t > 0, A 1 (t) = A 1,0 , A 2 (t) = A 2,0 , SL(t) = SL 0 , t = 0. ( 5
)
where

F(t) := [f A1 (t), f A2 (t), f SL (t)] is a row vector of fitness values, I(t) := [A 1 (t), A 2 (t), SL(t)] is
a row vector of individual frequencies, • denotes the element-wise multiplication operator, col k () is a function that returns the k-th column of a matrix, and ψ(t) is the average fitness of the population found as:

ψ(t) := F(t)I T (t) (6)
The replication may not be perfect. The mutation probabilities are provided by M where M ij indicates the probability that type j is produced by type i, and M i: indicates row vector with an index of i. M is a row-stochastic matrix thus satisfies the following condition:

M ∈ {A ∈ R 3×3 ≥0 : 3 j=1 A ij = 1, 1 ≤ i ≤ 3}.
In our experiments, we set mutation matrix M as follows: Individual learners perform actions a 1 and a 2 . However, they try the other action with a small frequency for exploration (e.g. analogous to -greedy algorithm in reinforcement learning [START_REF] Richard | Reinforcement learning: An introduction[END_REF]).

M =    0
Thus they suffer from an exploratory cost. On the other hand, this may become useful for learning new action in case if the environment changes (i.e. change in the payoffs of the actions). Consequently, the fitness f Ai of type A i is found by the weighted average of payoffs obtained from performing different actions as shown in Equation 7.

f Ai (t) = (1 -)r(A i , t) + r(A j , t) (7)
for all i, j = {1, 2} where i = j.

Fitness of the the social learners f SL (t) updated based on a specific social learning strategy. We define four SLSs in following sections.

Success-based. the social learners copy the behavior of successful individual from a previous time (t-τ ). Thus, the fitness values of the social learners equal to the reward received by performing the optimum action.

f SL (t) = r(a * , t) (8)
where a * denotes the optimum action r(a * , t) is its reward at time t.

Conformist. the social learners copy the behavior with the highest frequency in the population.

We keep track of the behavior frequencies in the population and introduce some latency represented as τ . The frequencies of the behaviors performed by the social learners are also included into the model. We first show how the frequencies of the behaviors are computed and then define the fitness of conformist strategy.

Let h(a i , t) denote the frequencies of actions a i at time t. Furthermore, we define H SL (a i , t) to denote the frequencies of the actions performed by the social learners. When t = 0, some portion (H SL (0)) of the population consists of social learners, however they do not perform any action because the information of the frequencies of the actions in previous times (t -τ ) is not available.

Therefore, we set the initial values when (t -τ <= 0) as H SL (a i , t) = 0, h(a n , 0) = A i (0), and

f SL (0) = 0.
When t > τ , we update the frequencies of the actions as follows:

H SL (a i , t) =    1, if i = arg max i h(a i , t -τ ), 0, otherwise. (9)
h(a i , t) = A i (t) + SL(t)h SL (a i , t) (10)
Finally, the fitness of social learners, given in Equation [START_REF] Degenne | Introducing social networks[END_REF], is found by the average payoffs of the social learners that perform each behavior type.

f SL (t) = n i=1 h SL (a i , t)r(a i , t). (11)

Evolutionary algorithm

Algorithm 1 provides the pseudocode for the evolutionary algorithm [START_REF] Agoston E Eiben | Introduction to evolutionary computing[END_REF] we use to analyse the evolution of a population of individual and social learners [START_REF] Kameda | Cost-benefit analysis of social/cultural learning in a nonstationary uncertain environment: An evolutionary simulation and an experiment with human subjects[END_REF][START_REF] Kameda | Does social/cultural learning increase human adaptability?: Rogers's question revisited[END_REF]. Individuals are assigned one of these types randomly during the initialization process. In each generation, the individuals can perform their actions based on their type. Their fitness is computed based on their actions and used for the selection process for the next generation. We use only a mutation operator which alters the type of a selected individual with a small probability.

The individual learners are modeled as reinforcement learning agents where they perform their behavior based on their model. Moreover, their models can be updated based on the rewards received as response to their behaviors. We use -greedy algorithm, discussed in detail in Section 4.2 [START_REF] Richard | Reinforcement learning: An introduction[END_REF], to Algorithm 1 the Evolutionary Algorithm for the evolution of social learning strategies. while t ≤ T do 8:

for each i ∈ I t do 9:

if (isIndividualLearner(i) or t = 1) then Individual Learning 10:

r i (a j , t) = individualLearning( )

11:
updateDecisionModel(r i (a j , t)) See Equation ( 3)

12:
else Social Learning

13:
r i (a j , t) = socialLearning(sls)

14:
updateDecisionModel(r i (a j , t))

f i = updateFitness(r i (a j , t)) 17:
end for 18:

I := select(I t , F ) 19:
I t+1 := mutate(I , mr) 20:

t = t + 1 21:
end while 22: end procedure model the learning process of the individual learners.

The social learners on the other hand, perform their behaviors based on the behaviors of others.

We model the same strategies, namely, conformist and success-based discussed in Section 4.5.1. In case of conformist strategy, the social learners select the behavior with maximum frequency in the population. For the success-based strategy, social learners copy the behavior of the individual with the best fitness value in the previous generation. Depending on the outcome r i (a j , t), that is the reward received by the i-th individual performing action a j , their fitness values f i ∈ F are updated. We simply use the reward as the fitness of an individual i at generation t as: f i (t) = r i (a j , t). The average population reward ψ(t) is the average of the fitness values of the individuals in the population:

ψ(t) = 1 m m i f i (t)
where m is the number of individuals in the population.

Selection of the individuals for the next generation t + 1 is performed based on their fitness values at t. We use the roulette wheel selection (also known as fitness proportionate selection [START_REF] Agoston E Eiben | Introduction to evolutionary computing[END_REF]) to simulate natural selection process. According to this scheme, m number of individuals are selected based on the probability that is proportional to their fitness values as given below:

p i (t) = f i (t) m j=1 f j (t) , (12)
where p i (t) is the probability of selecting individual i from the population, and f i (t) is the fitness of the individual. The same individuals can be selected multiple times to construct the population for the next generation. There is no mating process involved. However, individuals are mutated by changing their type with a small probability controlled by the mutation rate (mr).

Meta-social learning

The meta-social learners can switch between individual and social learning during their lifetime. A generic algorithm for meta-social learning is provided in Algorithm 2. In this section, we provide the implementation details of all the algorithm variants used to control the meta-social learning strategies.

Algorithm 2 Meta-social learning algorithm based on the environmental variables: environment change, uncertainty and conformity. end while 18: end procedure

Observation-based control

The SL-EC-Conf-Unc algorithm uses three environmental variables, environment change (EC(t)), conformity (C(t)) and uncertainty (U (t)), to decide the type of learning strategy. Initially, and after an environment change, the algorithm uses individual learning strategy for exploration.

During individual learning, the conformity of the population and uncertainty of the environment are estimated and used for switching between conformity and success-based strategies as shown in Table 1. We use a standard GA with population size of 50 individuals, roulette wheel selection with four elites, 1-point crossover operator with 0.8 probability and a mutation operator which selects discrete genes with the probability of 1/(L -2) where L -2 is the length of the genotype excluding the genes that encode continuous variables, replaces one of three possibilities for the discrete genes, and performs Gaussian perturbation with zero mean and 0.1 standard deviation on the continuous genes. 2. Thresholds of the evolved rule for uncertainty and environment change is th u = 0.05 and th ec = 0.15.

Yes

We use a separate training environment for optimizing the meta-social learning policies using the GA (provided in Supplementary Material). The GA aims to maximize the fitness values of the policies which is computed by the median of the cumulative sum of the average population reward of 112 runs as follows:

f = median T t=1 ψ i (t) , ∀i = {1, . . . , 112} (13)
The GA process on the training environment is executed for 10 independent runs. We stop the evolutionary process if the algorithm fails to find a better fitness value for 20 subsequent generations.

At the end of the runs, we select the best policy to be tested on the test environment reported in Results section.

Figure 6 shows the best evolved policy over 10 independent GA runs. The details of the optimization process provided in Supplementary Material. The evolutionary process is performed on a separate environment different than the environments we used for test in Results section. Note that the best evolved policy converged to the policy suggested by our analysis, and implemented by the SL-EC-Conf-Unc.

SL-NE (ANN based trained by neuroevolution) uses neuroevolution (NE) [START_REF] Kenneth O Stanley | Designing neural networks through neuroevolution[END_REF] approach to optimize artificial neural network (ANN) based policies. In NE, evolutionary algorithms are used for the optimization processes of the topologies and/or weights of the networks.

Illustrated in Figure 7, we use feed-forward ANNs with one hidden layer to perform individual learning, success-based and conformist social learning strategies. The input to the networks are the average and standard deviations of the estimated rewards of the actions, and frequencies of the individuals that perform each action (see Equation ( 2)). For two actions, we used 6, 12 and 3 neurons in the input, hidden and output layers respectively. We include an additional bias neuron (constant +1) in input and hidden layers. Therefore, the total number of network parameters is we use roulette wheel selection with 5 elites, 1-point crossover operator with the probability of 0.8 and Gaussian mutation operator as: N (0, 0.1), that performs independent perturbation for each dimensions in the genotype.

In the case of the DE, we use "rand/1" mutation strategy and uniform crossover with parameters of F = 0.5 and CR = 0.1 respectively [START_REF] Yaman | Limited evaluation cooperative co-evolutionary differential evolution large-scale neuroevolution[END_REF][START_REF] Storn | Differential evolution-a simple and efficient heuristic for global optimization over continuous spaces[END_REF].

Both algorithms aim to maximize the median of the total rewards, given in Equation ( 13), on the training environment provided in Supplementary Material. We run the GA and DE for 10 independent runs each, and use the ANN that achieved the best fitness value for the comparison in Results section. 2. SL-QL (Q-learning): we represent the estimate rewards of a certain action a in a certain state of the environment s as Q(s t , a t ) at time t. An action is selected based on the following: where QL is exploration parameter for the Q-learning. Then, estimate rewards are updated based on the Bellman equation as follows:

a(t) =    arg max a Q(s t ,
Q(s t , a t ) = Q(s t , a t ) + α r(a, t) + γ max a Q(s t+1 , a) -Q(s t , a t )
where α and γ are the learning rate and the discount factor.

Here, we use there environmental variables as states as: environment change EC(t), conformity C(t) and uncertainty U (t). All of these variables are binary, thus, there are a total of eight states. There are three possible actions as: individual learning, success-based or conformist social learning strategies. We trained the SL-QL on the training environment provided in Supplementary Material. We performed experiments with different parameter settings of the algorithm and found that QL = 0.2, α = 0.01 and γ = 0 value assignments provided the best results.

3. SL-UCB (upper confidence bound): to control the degree of the exploration, the equation for selecting actions is modified as follows:

a(t) = arg max a Q(a, t) + c ln t N (a, t)
where c is exploration parameter and N (a, t) is the number of time a is selected until time t.

We use the same update rule for Q(a, t) given in Equation (3).

In UCB selection, the square root term is the uncertainty in the estimate of a. While N (a, t) increases, uncertainty terms decreases, whereas, while N (a, t) keeps the same, uncertainty increases (since t increases) [START_REF] Richard | Reinforcement learning: An introduction[END_REF].

A.2 Training environment

A.3 Evolutionary optimization of the SL-GA

In this section, we provide the results of the evolutionary processes of the GA used to optimize the decision policies in the SL-GA. We performed 10 independent GA runs on the environment shown in Figure 10. Fitness values are the median of the cumulative rewards of 112 processes.

The fitness trends during the GA processes, and the fitness versus standard deviations of the best solution (SL-GA policies) are shown in Figure 11.

A.4 Evolutionary optimization of the SL-NE

A.5 Experiment design and results

We designed three sets of experiments with various volatility and uncertainty in terms of environment change and the overlap between the reward distributions. In Experiment1, we designed four environments as: stable low uncertainty, stable high uncertainty, volatile low uncertainty and volatile high uncertainty. In stable environments, the reward distributions were changed two, and in volatile environments five times. We defined six reward distributions (shown in Figure 13) and assigned to a time period in the processes as illustrated in Figures 15a,15b, 15c and 15g. Low and high uncertainty environments have low and high ODPUs respectively.

In Experiment2 (random volatile environment), we defined random environments by selecting the number of environment change between [START_REF] Lewis G Dean | Human cumulative culture: a comparative perspective[END_REF][START_REF] Kendal | The evolution of social learning rules: payoff-biased and frequency-dependent biased transmission[END_REF] from uniform distribution, and assigned a reward distribution (from Figure 13) to each period between environment change points randomly.

In Experiment3 (gradual environment), we defined gradual environment change by defining the reward distributions based on sinusoidal functions as shown in Figure 14.

Figure 1 :

 1 Figure1: (a) A population of individuals perform a binary decision-making task based on individual and social learning strategies and collect their rewards (r(a j , t)) based on their actions. The individual learners can perform their actions based on their decision models that can be improved by experience. The social learners use success-based or conformist strategies to copy the actions of successful individuals or the majority respectively. (b) Binary decision-making task (2-armed bandit) is iteratively performed for a certain period of time with specified reward distributions that are unknown to the individuals. At some point, an environment change occurs by changing the reward distributions (a.k.a reward reversal). In earlier stages of the process (initial and after environment change, shown in green dash lines), populations with success-based social learning strategy achieves higher average population reward faster relative to the conformist strategy. In later stages of the process (shown in purple dashed lines), populations with success-based social learning strategy achieves higher average population reward in environments with low uncertainty, whereas, populations with conformist social learners achieves higher average population reward in environments with high uncertainty.

Figure 2

 2 Figure 2 shows the performance comparison of the populations consisting of only individual learners, success-based and individual learners, and conformist and individual learners after the environment change (the average of the period between t = [200, 250]) and at the end of the process (average of the period between t = [350, 400]). To formally quantify the effect of uncertainty on performance, we also used critical difference (CD) diagrams 1 . The population dynamics throughout the evolutionary processes are shown in Figures 8 and 9 in Supplementary Material A.1. When

Figure 3 :

 3 Figure 3: The higher the uncertainty between two distributions (measured by the ODPU) the higher the performance difference between conformist and success-based strategies in terms of average population reward at the end of simulation processes. Figures (a) and (b) illustrate two cases where σ 1 = 0.4, σ 2 = 0.2 and σ 1 = 0.2, σ 2 = 0.4 respectively. (c) and (d) show the ODPU, formalized as the probability of sampling the highest reward value from the sub-optimum distribution, depending on σ 1 and σ 2 and the ratios of samples drawn independently from the optimum and sub-optimum reward distributions. In (c) the ratios of samples drawn from the optimum and sub-optimum distributions are 0.05 and 0.95, and in (d) the ratios are 0.5 and 0.5 respectively. (e) shows the relation between the ODPU and the difference in average reward at the end of the process between the populations with conformist and success-based strategies.

  6): Observation-based control. Here, we implement three versions. All of these versions start the process by using individual learning strategy. Similarly, they switch back to individual learning after an environment change. Otherwise, they use success-based or conformist social learning depending on the conformity and uncertainty. • SL-EC-Conf (uses environment change and conformity) switches to the conformist social learning strategy if conformity (C(t)) is satisfied. • SL-EC-Succ (uses environment change and uncertainty) switches to the success-based strategy in low uncertainty environment. Otherwise, they use individual learning. • SL-EC-Conf-Unc (uses environment change, conformity and uncertainty) arbitrates between social learning strategies depending on conformity and uncertainty. If the conformity is observed in the population, then the individuals switch to the conformist strategy. Otherwise, if the environment is with low uncertainty, then they use the success-based strategy. If none of the above conditions is met, they perform individual learning.

Figure 4 :

 4 Figure4: SL-GA and SL-EC-Conf-Unc show the best performance vs. exploration cost on diverse set of environments. Overall, the meta-social learning strategies that utilize conformist strategy show better performance environments with high uncertainty. On the right of each figure, the labels of the meta-social learning strategies (A through M) were ranked from best to worst based on their performance values. Decimal numbers on the left indicate their average ranks (the lower is the better), and the differences in their ranks that are not statistically significant at α = 0.05 are linked by vertical black lines.

  ) from the highest to the lowest levels of uncertainty. Then, we created a task consisting of multiple periods, each of which is associated with a reward distribution selected from this set (Experiment1; Figures15a, 15b, 15c and 15g). Moreover, we ran additional tests with two challenging tasks: random volatile environment where the number of environment changes and distributions were chosen randomly (Experiment2), and uncertain environment with a gradual environment change (Experiment3). The performance-exploration cost4 trade-off and their ranks (based on CD diagrams) are shown in Figure4. The change of the average reward and cumulative average reward during the learning processes on these environments are shown in Figure15in Supplementary Material A.5.

Figure 5 :

 5 Figure5: Based on the ratios in the populations, SL-GA, SL-EC-Conf-Unc, SL-QL and SL-EC-Conf are the most dominating meta-social learning strategies in wide range of environments. On the right of each figure, the ranks of the algorithms (higher to lower in terms of ratios) at the end of the processes are shown. The differences that are not statistically significant (at p > 0.05, Wilcoxon rank-sum test) are linked using black vertical lines. The points of environment change are indicated with orange arrows. The highlighted areas show the standard deviation 112 runs.

  a Q(a, t), based on the probability 1 -, random(a), based on the probability .

  rate of 0.005 from the individual learners to social learner and vice versa.

1 :

 1 procedure EA( , sls) Evolution of individual and sls type social learners 2: // : exploration parameter of individual learning 3:// sls: the type of social learning strategy (i.e. success-based or conformist)

Table 1 :

 1 The strategies implemented by the SL-EC-Conf-Unc based on the conformity and uncertainty. Conformity (C(t) = 1) Non-conformity (C(t) = 0) Low uncertainty (U (t) = 0) Conformist Success-based High uncertainty (U (t) = 1) Conformist Individual learning Other variants in this class include SL-EC-Conf and SL-EC-Unc. These two variants use environment change but does not make use off full functionality of SL-EC-Conf-Unc. In case of SL-EC-Conf, individuals can use only individual learning and conformist strategies depending on the environment change and conformity, whereas, in case of SL-EC-Unc, they can use only individual and success-base strategies based on the environment change and uncertainty.4.6.2 Evolutionary controlSL-GA (trained by the genetic algorithms) uses genetic algorithms (GAs[START_REF] Agoston E Eiben | Introduction to evolutionary computing[END_REF]) to optimize the metasocial learning policies of the individuals for switching between the individual and social learning strategies. Shown in Table2, we encode the type of strategy s j (i.e. individual learning, successbased or conformist) for a given state of the environmental variables, namely environment change, conformity and uncertainty. Since these variables can take binary values (see Section 4.6), there are 8 possible states (discrete variable) which can take three strategies. Thus, there are total of 3 8 = 6561 possible distinct policies.In addition to these discrete variables, we include two continuous variables into the genotype of individuals for determining the thresholds used in environment change (th ec ) and uncertainty (th u ). Consequently, the genotype of the individuals consist of 10 genes, 8 discrete and 2 continuous variables.

Figure 6 :

 6 Figure 6: Discrete part of the SL-GA policy that achieved the highest cumulative reward. "Yes" and "No" indicate 1 and 0 states of the environmental variables shown in Table2. Thresholds of the evolved rule for uncertainty and environment change is th u = 0.05 and th ec = 0.15.

12( 6 + 1 )Figure 7 :

 617 Figure 7: Neuroevolution scheme used to optimize the social learning policies. (a) Feed-forward artificial neural network topology with one hidden layer can take the average, standard deviations and frequencies of two actions a 1 and a 2 and decides to perform individual learning, success-based or conformist social learning strategies. (b) The weights of the networks between input and hidden layers (W hi ), and hidden and output layers (W oi ) are directly mapped to the genotype of the individuals and represented as real valued vectors. (c) Evolutionary algorithms are used to optimize the genotype of the individuals.

4. 6 . 3

 63 Multi-armed bandit control 1. SL-RL ( -greedy algorithm): the action-value based approach used in individual learning model (discussed in Section 4.2) is used for selecting the social learning strategy. In this case, the action space consists of performing one of the followings: individual learning, success-based and conformist strategies at time t. Q(a, t) is the estimate reward of these learning approaches and updated based on the rewards received as shown in Equation (3). One of the actions are selected based on Equation (4).

  a), based on the probability 1 -QL , random(a), based on the probability QL .

4. 6 . 4

 64 Other baseline strategies 1. IL-Only (individual learning): individuals perform only individual learning throughout the processes. 2. SL-Rand (random strategies): individuals perform randomly one of individual learning, successbased and conformist social learning strategies with equal probability. 3. SL-Prop (proportional strategy selection): individuals perform success-based, conformist, individual learning strategies with probabilities of 0.45, 0.45 and 0.1 respectively. 4. SL-Conf (conformist with individual learning): individuals perform conformist and individual learning strategies with probabilities of 0.95 and 0.05 respectively. 5. SL-Succ (success-based with individual learning): individuals perform success-based and individual learning strategies with probabilities of 0.95 and 0.05 respectively.

Figure 10

 10 Figure 10 illustrates the environment used for training phase of the algorithms. The trained algorithms then tested on separate environments and reported in the main text of the paper.

Figure 10 :

 10 Figure 10: The environment used for training processes for algorithms: SL-GA and SL-NE. (a) through (f) show arbitrarily defined reward distributions and their ODPUs for 2 arms. (g) shows the training environment generated by using the specified reward distributions for specified lengths of periods. The complete period consists of 220 time steps. Dashed vertical lines indicate the change of the reward distribution points. The periods with uncertainty (ODPU≥ 0.1) are highlighted in gray.

Figure

  Figure reffig:runNE shows the evolutionary optimization processes of the ANN based policies using genetic algorithms and differential evolution. The evolutionary processes are terminated if there is no fitness improvement for 50 consecutive generations.

Figure 11 :Figure 12 :

 1112 Figure 11: (a) Fitness trend of 10 independent GA runs, and (b) fitness versus standard deviations of the best solution for each independent GA runs.

0.5 1 (Figure 13 :

 113 Figure 13: The reward distributions of the arms defined in each period of the environments shown in Figures (a), (b), (c) and (g).

Figure 14 :

 14 Figure 14: Reward distributions and their standard deviations (highlighted) in gradual environment. We used two sinusoidal functions to model environment change. The standard deviation of the first arm kept constant while an additional sinusoidal function is used to model the change in the standard deviations of the second arm depending on time.

Figure 17 :

 17 Figure 17: The effect of selection strength and mutation rate to the domination of the successful strategies. While the selection strength increases, domination of successful strategies increases; however, while the mutation rate increases, domination of successful strategies decreases.

  Volatile low uncertainty

	115					Rank
	110					1.56 1.56
	105					2.93 4.41
						4.59
	100					6.21
						7.44
	95					8.27 8.44
	90					10.1 11.1
						11.4
	85	1	Exploration cost (log) 4	8	12	12.7

Table 2 :

 2 Possible states of the environment and their strategy assignments that can take one of the strategies as: individual learning, success-based or conformist.

	Environment Change Conformity Uncertainty Strategy
	0	0	0	s 1
	0	0	1	s 2
	. . .	. . .	. . .	. . .
	1	1	1	s 8

The critical difference (CD) diagrams allow comparison of results from multiple strategies. They show the average ranks of the algorithms from the best to the worst. The algorithms that do not have significant rank difference are linked (post-hoc Nemenyi test[START_REF] Nemenyi | Distribution-free multiple comparisons[END_REF] at α = 0.05[START_REF] Demšar | Statistical comparisons of classifiers over multiple data sets[END_REF]).

Success-based social learning requires finding the action with the highest reward and conformist strategy requires finding the action with the highest frequency.

We perform 10 independent evolutionary runs and select the best control mechanism based on their cumulative reward. The results of the training processes are provided in Supplementary Material.

Exploration is provided only through individual learners and controlled by , therefore, we measure the exploration cost by the number of individual learners used throughout a process multiplied by their .

For statistical significance, we used a population consisting of 5000 individuals and perform the evolutionary processes for 112 independent runs. The statistical significance of the results measured by pairwise p-values and differences that are not statistically significant shown on the right side of each figure.

Due to the use of mutation (with rate of 0.005), strategies are not completely eliminated from the evolutionary processes.

Dot notation represents time derivative (i.e. ẋ = dx/dt).

Acknowledgement. This work was supported by Institute of Information & Communications Technology Planning & Evaluation(IITP) grant funded by the Korea government (MSIT) (No.2019-0-01371, Development of brain-inspired AI with human-like intelligence) and Samsung Research Funding Center of Samsung Electronics under Project Number SRFC-TC1603-52.

A Supplementary Material A.1 Population dynamics

We present the results of the mathematical model at the top row of Figure 8 and the evolutionary algorithm at the bottom row of Figure 8. Notably, these approaches produce very similar results. The social learners using the success-based strategy show dominance over individual learners throughout the learning process as well as rapid adaptation after the reward reversal. On the other hand, the social learners using the conformist strategy show dominance only after the majority of the individuals learn to make optimum choices. When the latency is increased, the success-based strategy shows similarity to the results of the conformist strategy.

In highly uncertain environment, the success-based social learners lose their dominance in the population throughout the evolutionary processes (see Figure 9). This indicates that the fitness of individual learners is higher than that of the success-based social learners. On the other hand, the behavior of conformist learners in uncertain environments is similar to that in static environments. The age distributions of the dominant strategies are higher relative to others.