Nobuyuki Hanaki

Alan Kirman
email: alan.kirman@univ-amu.fr]

Paul Pezanis-Christou

Ido Erev

Mamoru Kaneko

Rajiv Sethi

Counter intuitive learning: An exploratory study *

Keywords: multi-armed bandit, reinforcement learning, eureka moment, pay-off patterns, observational learning JEL Code: D81, D83

The literature on learning in unknown environments emphasises reinforcing on actions which produce positive results. But, in some cases, success requires shifting from a currently successful actions to others. We examine, experimentally and theoretically in a very simple framework, how individuals initially learn by exploiting information from the pay-offs of actions taken but also from exploring new actions. We analyse if and how they learn that pay-offs are inter-temporally dependent. We then ran the same experiments but where individuals could observe the actions taken or the pay-offs obtained by others or both. Such observations improved pay-offs if one of the pair had learned to obtain the maximum pay-off.

Introduction

When individuals have to learn which of the many courses of action open to them yield the best results, they typically start exploring the various possibilities that they perceive. They start with a set of possible actions and try to choose amongst them but may learn that there are other options available to them as they proceed.

Most of the emphasis in the learning literature is on reinforcing actions which have produced positive results [START_REF] Bush | A mathematical model for simple learning[END_REF] whether the reinforcement is based on the realized payoffs [START_REF] Erev | Predicting how people play games: reinforcement learning in experimental games with unique, mixed strategy equilibria[END_REF], the foregone payoffs [START_REF] Camerer | Experience-weighted attraction learning in normal form games[END_REF], or their regrets [START_REF] Marchiori | Predicting human interactive learning by regretdriven neural networks[END_REF]. However, they should also explore alternatives other than those which they have tried in the past even if some of the latter have given positive results. This is what is referred to in the literature as the "exploration, exploitation" trade-off [START_REF] March | Exploration and Exploitation in Organizational Learning[END_REF] and we will use this as our basic framework. We will assume that agents start with a model of their environment and try to learn within it.

But, what if this model is an inadequate representation? Can agents learn not only which actions to take in their currently perceived model but also to change the latter? In our theoretical model, we will model the process involved in doing this.

To focus the discussion, consider the case in which the success of an endeavor depends on shifting from a currently successful action to another. The simplest case is that of fisheries. Having fished at one period in a certain area, fishermen move on to other areas knowing that otherwise their catch will diminish and that the remaining fish will not be sufficient to replace the current population. A more subtle problem but still a classic example is crop rotation where sustainable success depends on changing the crop cultivated each year. The yield from a crop grown on a given plot of land this year depends on what was grown there in the past. The use of such inter-temporal strategies depends on the farmer becoming aware of the existence of inter-temporal dependencies in the payoffs across various options. Once this happens he will then have to explore among the many possible inter-temporal choice strategies to discover the ones that result in higher payoffs. This problem is particularly difficult because farmers may well learn quite quickly that leaving the land fallow will improve the yield of the crop which was planted previously, but it is more complicated to understand that growing other crops on the same land may actually improve the yield compared to what it would have been if it had been left fallow. The process by which people become aware of such features of the environment that they have not known before, and learn to adopt potentially more complex, but better, behavioral strategies is not yet very well understood. How individuals learn which actions to take within their perceived model has been thoroughly investigated but how they learn to change their perception of their model is much less clear. This raises an important question which has generated a large literature. What are the mental processes at work that lead individuals to learn how to understand and operate in their environment? This has been the subject of considerable interest in the neurosciences where attention has been focused on the mechanisms which lead a person to start exploring new alternatives and thus to enlarge his vision of the world he lives in.

One approach has been to identify the neural processes at work when an individual suddenly perceives the answer to a problem he or she has been wrestling with. This is often referred to as the "aha" or, referring to Archimedes, the "eureka" moment. There is now a substantial literature trying to identify the changes in neural activity that take place when such a realisation occurs (see [START_REF] Auble | Effort toward comprehension: Elaboration or "aha![END_REF][START_REF] Kounios | The Origins of Insight in Resting-State Brain Activity[END_REF][START_REF] Topolinski | Gaining Insight Into the "Aha" Experience[END_REF]. However, this literature has focused on solving a well defined problem to which it is known that there is a "solution" such as the Rubik's cube or a mathematical puzzle.

We are concerned with a more general situation in which individuals take actions and try to improve their performance but do not know whether there is necessarily a "solution." What interests us, in particular, is how agents come to decide to try alternative actions to those which they have already used. It is now argued, in the neuroscience literature, that different mechanisms operate in the brain when an individual switches from exploiting the information about the various options he has tried to exploring new ones (see [START_REF] Laureiro-Martínez | Frontopolar cortex and decision-making efficiency: comparing brain activity of experts with different professional background during an exploration-exploitation task[END_REF][START_REF] Greiner | An Online Recruitment System for Economic Experiments[END_REF]. 1

1 Not only are different networks of neurons activated in the two cases but the process of switching is different depending on the activity that an individual normally pursues. The authors compared two groups of subjects, managers and entrepreneurs, and found that the threshold at which a manager passed to exploration is significantly higher than that of an entrepreneur. In other words, entrepreneurs tend to function in the exploratory mode more often than managers. Indeed, the distinction that emerges between people with different professions suggests that

In this paper, we focus on experiments to investigate how individuals gain a better understanding of their environment and, as a result, obtain better results. We study a case in which the payoffs from actions are correlated and have a temporal structure. More specifically, we study a case in which subjects are faced with a multi-armed bandit and try to obtain the highest pay-off they can get from choosing the different arms in succession. Our experiment is very different from other experiments on "standard" stationary multi-armed bandit problems in which all the arms generate stochastic payoffs from predetermined distributions, and the task for a subject is to find which one to choose (see, for example, [START_REF] Banks | An experimental analysis of the bandit problem[END_REF][START_REF] Brown | Observing evidence accumulation during multi-alternative decisions[END_REF][START_REF] Efferson | Learning, productivity, and noise: an experimental study of cultural transmission on the Bolivian Altiplano[END_REF][START_REF] Hu | Nonparametric learning rules from bandit experiments: The eyes have it![END_REF][START_REF] Mcelreath | Applying evolutionary models to the laboratory study of social learning[END_REF][START_REF] Steyvers | A Bayesian Analysis of Human Decision-Making on Bandit Problem[END_REF]. In that framework, agents will optimally try arms to form an idea as to their expected pay-offs and will have a stopping rule which tells them when to stop experimenting and stick to that arm which has proved most successful up to that point. Such a strategy is not well adapted to a framework like ours in which payoffs have temporal structure and are not independent of each other.

How individuals learn which action to take, in the sort of context that corresponds to our experimental framework, and how to condition that choice on previous experience has been a subject of considerable interest in other fields such as machine learning. There, a problem which corresponds to our simplest case, is what is referred to as the "contextual bandit problem." In this, as in our experiments, an agent collects rewards for actions taken over a sequence of rounds. At each point in time, the agent chooses an action to take on the basis of two things: the context for the current round, and the feedback, in the form of rewards, obtained in previous rounds. That literature has focused on the optimal choice of the probabilities with which to explore different actions. The strongest known results [START_REF] Auer | Using Confidence Bounds for Exploitation-Exploration Trade-offs[END_REF][START_REF] Mcmahan | Tighter Bounds for Multi-Armed Bandits with Expert Advice[END_REF][START_REF] Beygelzimer | Contextual bandit algorithms with supervised learning guarantees[END_REF] provide algorithms that carefully control the distribution of the probabilities with which actions are explored, to achieve an optimal regret after T rounds. However, in this paper, we will use a rather simple exploratory rule, "entropic" exploration. The logistic rule which we employ can be derived as an optimal trade-off between exploitation and exploration [START_REF] Nadal | A Formal Approach to Market Organization: Choice Functions, Mean Field Approximation and Maximum Entropy Principle[END_REF].

this process is also present in longer term situations.

An important feature of our problem, as mentioned, is the sequential nature of the pay-offs and, until the subject recognises this structure, he may continue to try to reinforce on the experience from choosing single arms. The important change that is necessary for him to learn how to improve his gain, is to recognise that there is a sequential pattern in the pay-offs and then reinforce on those. Thus he has to change the space of possible choices to one of choices conditioned on what has happened in the past, and this, in our framework, is the "context" which is referred to in the machine learning literature to which we have just alluded. Indeed, the literature on machine learning has dealt with similar issues when analyzing problems in "sequence extrapolation" [START_REF] Laird | Discrete Sequence Prediction and Its Applications[END_REF]. That literature typically tries to develop deterministic algorithms that may be used for successful sequence predictions in general. And where there is an underlying deterministic process generating pay-offs as in our case, such an approach is appropriate.

Our paper provides a link between two approaches, that of machine learning in which the recognition of patterns in the data being observed is the primary objective, and that developed in game theory in which the objective is to recognise patterns in opponents' behaviour (see, [START_REF] Sonsino | Learning to learng, pattern recognition, and Nash equilibrium[END_REF][START_REF] Spiliopoulos | Pattern recognition and subjective belief learning in a repeated constant-sum game[END_REF]Spiliopoulos, , 2013)). Although the mechanics of the processes have many aspects in common, the goal is very different. In machine learning, the aim is to recognise and identify patterns in data. In the game theory literature, the objective is to react to the patterns in the best way knowing that the reaction will provoke a change in the patterns. The ultimate aim is to find an equilibrium in which the patterns no longer change.

We will proceed by first presenting our experimental framework in which the subjects have to understand that the order in which they choose actions is important if they are to achieve good results. Until subjects realise that there is such a sequential structure, they cannot obtain a high or, although they do not know it, the optimal pay-off. Their problem is precisely that they have no a priori information about how actions and pay-offs are linked. Our question is not whether they can find the solution to a well defined problem, such as solving the Rubik's cube, but rather, whether, by exploring the various possibilities and using the information they gain in doing so, they improve their payoffs and may even arrive, without realising it, at the maximum attainable. We present some examples of how subjects performed in the experiments and then develop a model to try to capture how subjects learn about the interdependence in the pay-offs from their actions and how they learn to try new strategies. We then compare the behaviours the model generates with those observed in the experiments.

In the second part of the paper, we consider another important aspect of the learning problem.

In many cases, individuals do not learn in isolation but observe what others do or what they gain from their actions. In a second series of experiments, we took this into account and allowed subjects to observe another individual's choices, pay-offs or both. There is a substantial literature on what is referred to as "observational learning" [START_REF] Bandura | Influence of social reinforcement and the behavior of models in shaping children's judgment[END_REF]Bandura et al., 1963;[START_REF] Fryling | Understanding Observational Learning: An Interbehavioral Approach[END_REF] and we investigate to what extent individuals benefit from such information by accelerating their learning process.

The baseline experiment

In our baseline experiment, 30 subjects participated in an experiment that involve 200 rounds of play. In each of these round, participants were asked to choose one of four options without being given any information about the possible payoffs that each option could generate. The underlying payoff generating process was such that the first three options generate a payoff of either 0 or 1 following a deterministic cycle whereas the fourth generates a constant payoff of 0.3. The payoff cycle of the first three options was such that in round t, option a ∈ {1, 2, 3} generates a payoff of 1 if M od(t -1, 3) = 3 -a and 0 otherwise. That is, a payoff of 1 can be achieved in every round if the participant selects the right option a at the right time t, i.e., the optimal option choice cycles in the order of 3, 2, 1, 3, 2, 1, 3,... from round 1, ..., 200.

At the end of each round, participants received no other information than the payoff outcome of their own choice. Further, they were not allowed to record any information on paper or else either, so that they had to rely on their memory of past play and outcomes to make their decisions. To incentivize their decisions, their payoffs (expressed in Experimental Currency Units, ECUs) were cumulated over the 200 rounds and converted into cash at the end of the experiment at the rate of 20 Australian cents per ECU.

Once the 200 rounds of play were over, participants were given a new set of instructions that outlined the lottery choice questionnaire to measure their risk preferences proposed by [START_REF] Holt | Risk Aversion and Incentive Effects[END_REF]. Again, to incentivize decisions, it was explained to them that, once the questionnaire was completed, one of the lottery questions would be randomly chosen to determine their individual payoffs for this second part of the experiment. Subjects received the outcome of the lottery if they accepted the chosen lottery and nothing if they did not. The lottery payoffs, which were expressed in Australian Dollars, are provided in the Appendix along with the full set instructions.

The average individual reward from participating in this experiment, including the lottery choice questionnaire, was A$30 (which includes a A$5 show-up fee) for a maximum of one hour and a half spent in the laboratory, including the time needed to read the instructions.

We are interested in understanding how quickly subjects discover, if ever, that there was a sequential structure to the pay-offs, and how many actually made the best choices given the true hidden payoff generating pattern even though they were not aware of the existence of such a pattern.

In other words, how many subjects "learned to optimise."

Descriptive results

We start with providing a brief model-free overview of the most salient types of behaviour. Out of the 30 subjects, exactly half of them discovered the hidden cycle that generates a payoff of 1 in each round but they did so at very different times: five of them found it within 50 rounds, 13 within 100 rounds and 15 within 200 rounds. The other half of subjects either never found it or settled for the safe option (option 4) which generates a constant payoff of 0.3.2

Figure 1 shows the time-series of choices and payoff outcomes for six participants who displayed different types of behaviour. While Subject 19 discovered the hidden cycle within 50 rounds, other subjects explored for a few rounds and settled for the safe option. Among the latter, some never got a payoff of 1 when choosing other options (e.g., Subject 10) while others decided to exploit the safe choice even though they sometimes got a payoff of 1 when choosing other options (e.g., Subject 1). Yet, other participants settled into more complex and less profitable cyclical choice patterns: Subject 20 settled into a 9-round cycle repeating the choice sequence (3,4,4,4,4,1,4,4,4), Subject 25 settled into the 6-round cycle (1, 3, 2, 4, 4, 4) while Subject 30 settled into another, almost optimal, 6-round cycle (3, 2, 1, 3, 2, 4).

One might reasonably conjecture that more risk averse individuals are more likely to exploit the safe option than to explore risky ones, so we checked whether the individuals' frequencies of choosing the safe bandit is positively correlated to their aversion toward risk. Recall that the Holt and Laury ( 2002) questionnaire consists of a series of lottery choice tasks in which participants are asked to choose between two lotteries, one of which is riskier than the other in the early tasks and safer in the later ones. A risk averse subject should therefore choose the safer lottery in the early tasks and switch to the other lottery in the later ones; the later this switch occurs and the more risk averse is the subject.3

Figure 2 shows a scatter plot of the degree of risk aversion (the subjects' switching points) versus the frequency of choosing the safe option (freq4) along with a Spearman correlation coefficient. As the reported coefficient is not statistically significant at α = 0.05, we discard risk aversion as a possible explanation for the observed behaviour and develop a model that attempts to capture the observed behaviour.

The baseline model

We assume that agents, who are initially not aware of the possible payoffs that each option generates, remember the payoffs that result from choosing each option if it is observed once. Thus, for our simple problem, after enough trials, agents will learn about all the possible outcomes that each option generates. Let Π i t (a) represent the set of payoffs agent i has observed each time he chose option a until period t. Thus, if agent i has experienced all the possible outcomes from choosing all four options at least once, before period t, we have Π i t (a) = {0, 1} for a ∈ {1, 2, 3} and Π i t (4) = {0.3}.

Of course, agents cannot be certain that what they have observed are indeed all the possible outcomes. In addition, we need to decide how the agents in our model make their choices amongst these different options. To what extent should they rely on their experience in trying the different options and to what extent should they switch to trying on other options. This is the well known "Explore v.s. Exploit" trade-off which has given rise to an extensive literature (see, for example, [START_REF] Hills | Exploration versus exploitation in space, mind, and society[END_REF], for a review). The idea is that the individual weighs up the gain that he might get from trying different options against the value of pursuing previously successful actions. In this literature, it is generally assumed that the success obtained from taking an action is independent of the choices of other actions in the past. Or put alternatively, the agents are assumed to make that assumption. This, we assume, is their initial model of the environment. Now the question is, how do individuals learn come to understand that their model is an inadequate representation of reality and to change their model?4

It may well be the case that individuals do not learn the "correct model." Indeed, there have been a number of accounts showing that individuals can have a wrong model in mind, but, as they learn what to do within that framework, their actions lead them to believe that the model is correct. Furthermore, their actions lead to the results that they expect (see, e.g., [START_REF] Arrow | Notes on Expectations Equilibria in Bayesian Settings[END_REF][START_REF] Kirman | Learning by firms about demand conditions[END_REF]Kirman, , 1983;;[START_REF] Bray | Learning, estimation and stability of rational expectations[END_REF][START_REF] Woodford | Learning to believe in sunspots[END_REF]. In those cases the pay-offs are, themselves, modified by the actions taken by the economic agents. Here, this is not the case. But the agents may be led, depending on the pay-offs that they receive in the early rounds, to conclude that they should use the safe arm, for example. But we are faced with a much more difficult question as to what it is that individuals might observe that might lead them to change their model of the world.

This would lead them to be faced with a different set of possible actions and possibly arrive at results quite different from those obtained in the past. One reason for making such a change would be that the results obtained using the original perception were consistently different from those anticipated. In many cases, such as those discussed in the literature just mentioned, this would not be evident. An individual may not understand that there were alternative courses of action which would be superior to the best of those that he has already tried. He "does not know what he does not know." However, our experimental evidence shows that many subjects do become aware that there is an inter-temporal structure in the way pay-offs are generated.

The first step will be for individuals to recognise that their gain does not simply depend on the particular action taken but also on when that action is taken as opposed to some alternative. Thus, the subjects in our experiments should not just condition their choices on past experience with those choices but also the "context." In our case, the context is the order in which actions are taken. A substantial literature on learning about "contextual bandits" has developed in machine learning.

In the simplest case, corresponding to our baseline treatment, the feedback is incomplete: in any given round, the agent observes the reward only for the chosen action but is not informed as to what he would have obtained had he chosen another action. The use of features to encode context is inherited from supervised machine learning and in our case, when we come to consider other experimental treatments, this will allow us to introduce the history or sequence of previous choices unlikely that this is the procedure that ordinary subjects would adopt.

and not just the pay-offs from each. On the other hand, that literature emphasises that exploration is necessary for good performance as it is in reinforcement learning. 5 In our particular case, the recognition that sequences matter amounts to detecting patterns in the payoff sequence and again in the machine learning literature considerable attention has been paid to pattern detection.6

For our model then, to take these considerations into account, we need to make an assumption as to how agents become aware of the existence of a pattern in the payoff generating process. We assume that agents become aware of the possibility of the existence of such a pattern in the payoffs from options a ∈ {1, 2, 3} if they observe the sequence (choice, payoff) of length l i ≥ 2, starting with a payoff of 1, r i ≥ 1 times. For example, if l i = 2 and r i = 2, after observing the sequence of choice payoff pair (1, 1), (1, 0) over two consecutive periods twice, agent i will start considering the existence of a dependency (namely, choosing the option 1 after choosing it and obtaining payoff of 1 will result in payoff of 0).7

Once i starts considering such dependencies, he will start choosing the option conditional on the outcome in the previous period. In particular, i will start remembering the possible payoffs each conditional choice generates. Let Π i t (a|h) represent the set of payoffs agent i has observed by choosing option a conditional on history h until period t. For example, Π i t (1|(2, 1)) will be either empty or 1. While it is possible that an agent may condition his choice of options on the outcomes of two or more previous periods, we restrict our attention to those choices which are conditioned only on the outcome of the most recent period (i.e., the choice of period t, a t , depends on the outcome of period t -1 (h i t-1 = (a i t-1 , π i t-1 ) )). Because of this assumption, we also assume l i = 2 for all i. Note that for the problem we consider in this paper, the set of all the possible outcomes in the previous period, h i t-1 , is {(1, 0), (1, 1), (2, 0), (2, 1), (3, 0), (3, 1), (4, 0.3)}.

How will these outcomes contribute to determining choices? We consider two sets of strategies:

unconditional and conditional. Unconditional strategies, those used before a subject becomes aware of sequences, are simply choices of options a ∈ {1, 2, 3, 4}. Conditional strategies are period t choices of options conditional on the outcomes in period t -1, s = a|h. Agents start using conditional strategies only after they have become aware of existence of a sequential pattern as we have described above.

Let us first describe an unconditional strategy. A i t (a) summarizes, at the beginning of period t, the past experience for agent i from choosing option a. Let A i 0 (a) = 0.5 for all i and a. We assume that A i t (a) evolves as follow:

A i t+1 (a) =        α i A i t (a) + (1 -α i )π i t if a = a i t A i t (a) otherwise
where a i t and π i t denote the option chosen by agent i in period t and the resultant payoff, and α ∈ (0, 1) captures the weight put on past experience.

Given A i t (a), we assume that the probability of agent i choosing option a in period t is

P r(a i t = a) = e λ i A i t (a) k e λ i A i t (k)
Let B i t (a|h) summarize i's experience from choosing a conditional strategy a|h at the beginning of period t. Let B i t (a|h) = A i t (a) for all h after i became aware of the inter-temporal dependencies.

And B i t (a|h) evolves as follows

B i t+1 (a|h) =        β i B i t (a|h) + (1 -β i )π i t if a|h = a i t |h i t-1 B i t (a|h) otherwise
Based on B i t (a|h), the agent chooses option a in period t according to P r(a i t = a|h i t-1 ) =

e µ i B i t (a|h i t-1 ) k e µ i B i t (k|h i t-1 )
For simplicity, we assume α i = β i and λ i = µ i for all i.

We fit our model to the experimental data for each subject by searching for a set of (α i , λ i , r i ) from the predefined parameter space that maximizes

200 t=1 ln(P i t (a i t ))
where P i t (a i t ) is the probability of observing choice a i t according to the model described above.

The parameter space we consider is α i ∈ [0.01, 0.99] with a step of 0.01, λ i ∈ [0.1, 20.0] with a step of 0.1, and r i ∈ [1, 10] with a step of 1. When multiple sets of parameter values generate the same sum of log likelihood, we selected the one with the smallest r i as this parameter was the unique source of multiplicity.

Figure 3 shows the time series of observed and simulated choices (left panel) and payoffs (right panel) for three of the six participants displayed in Figure 1. Our model replicates quite well the behaviour of Subject 19 who discovered the hidden cycle within 50 rounds, and of Subject 10 who settled for the safe choice from early on. This, however, is less so for Subject 30 who ended-up using an almost optimal cyclical pattern of length 6 (i.e., 3, 2, 1, 3, 2, 4).

Once we fitted and determined the set of best parameter values for each individual (α i , λi , ri ) , we considered a population of 1000 artificial agents with parameters randomly drawn (with replacement) from this set and we compare the simulated outcomes to the observed ones in terms of individuals' discovery times and total payoffs. Clearly, as participants did not know the exact structure underlying the bandits' payoffs, their objective function is ill-defined and many choice patterns could be believed to be 'optimal'. Here we focus on the discovery of the hidden cycle generating a payoff of 1 in each round. More precisely, we diagnose a discovery if a participant earns a payoff of 1 for twelve consecutive rounds for the first time 8 and we define participant i's 8 As already observed, some participants occasionally 'deviated' from the hidden pattern after having exploited it discovery time, ti by the median round number of this twelve-round window. 9

The observed and simulated cumulative distributions of discovery times and total payoffs and their respective averages (standard deviations) reported in Figure 4 indicate that the model fits these data remarkably well. This is confirmed by the Permutation tests (for independent samples) which do not reject the null of equivalent data samples (p = 0.814 for discovery times and p = 0.597 for total payoffs, two-tailed tests).

Additional experiments: Learning from others

We now extend our baseline setting, henceforth named the NI (No Information) treatment, to allow participants to observe the choices made or the payoffs obtained by another participant.

This "observational learning," that is learning from others, came to the forefront with the work of Bandura and his colleagues [START_REF] Bandura | Influence of social reinforcement and the behavior of models in shaping children's judgment[END_REF]Bandura et al., 1963). They argued that the process of learning is greatly influenced by observing the behaviour of other individuals and [START_REF] Fryling | Understanding Observational Learning: An Interbehavioral Approach[END_REF] argue that "The important conclusion of these studies is that behavior for long time, but most of them quickly returned to it. 9 For example, if a subject got a payoff of 1 from round 40 until 200, then his her discovery time is round 45. Note that we get qualitatively the same results if we change the criterion to six consecutive rounds instead of twelve.

change can and does occur through observation, even when such observation is incidental, occurring in the context of other activities." Indeed, when individuals are looking at the other's behaviour or rewards what may cause them to modify their own behaviour is the observation of systematic patterns in the behaviour of the person they are observing. In our framework, the observation of consistently high pay-offs or of regular patterns in the choices may lead a subject to change his behaviour. 10 As [START_REF] Bossan | The evolution of social learning and its economic consequences[END_REF] show, and which is intuitive, imitation of good performance may do better than individual learning. This is simply because imitating successful individuals accelerates the learning process for the less successful. They add, however, an important caveat: in a non-stationary environment, imitation may produce inertia in behavior as imitation induces herd behaviour and discourages individual exploration. In the experiments that we will discuss in this paper, the environment is, in fact, stationary, but the subjects were not aware that this was the case.

Imitation or social learning has also been taken into account in the neuroscientific field by conducting experiments in which subjects, in addition to their own experience also profit from feedback from other participants. It is also probably appropriate when considering phenomena such as the emergence of agricultural practices which involve substantial periods of time before the results of certain actions are known, and where actions are taken in a noisy environment, where weather and other stochastic natural variations can make the interpretation of the consequences of an action difficult. In such environments one might expect the learning process to be a collective one and still the adoption of innovative actions to be very slow. Indeed, in the optimal foraging literature in which individuals "learn" to behave optimally (see, [START_REF] Stephens | Foraging theory[END_REF], no indication is given as to the time necessary for such convergence to optimal behaviour to be accomplished. Furthermore, one might argue that the process never converges since the environment may change more quickly than either individual or collective learning. In fact, one can either think of the problem of individuals learning how to solve a problem which is well defined even if they may not 10 Similar work has been done by [START_REF] Nedic | A Decision Task in a Social Context: Human Experiments, Models, and Analyses of Behavioral Data[END_REF] who faced subjects with a two armed bandit whose arms, unknown to the subjects paid off depending on the number of times that the arms had been chosen. Many subjects managed to make the pay-offs from the two arms equal but did not find the optimal behaviour. In their case observing the behaviour of others actually had negative effects in some cases. There is a considerable literature on observational learning that is relevant here (e.g. [START_REF] Smith | Observational learning[END_REF]. However, most (if not all) of it deals with interactive situations where pay-offs to individuals depend on those to others, which is different from what we do. What we investigate here is the relative impact on subjects' behaviour of three different sorts of information: observing another player's choice and pay-off, observing another's choice only, and observing only the other's pay-off. To be more precise each of the following three experiments (treatments) involve 32 different subjects who were randomly matched in pairs for the 200 rounds of play. In one treatment, they got to know both the other subject's last choice and payoff outcome. In another treatment, they got to know the other subject's last payoff outcome but not the corresponding choice whereas in the last one, they got to know the other's last choice but not the payoff outcome of that choice. We will refer to these treatments as the CPI (Choice and Payoff Information), PI (Payoff Information) and CI (Choice Information) treatments, respectively. All other aspects of these treatments were the same as those of the NI treatment -Table 1 summarizes the four treatments' main features.

11 It is worth noting that some work has been done in neuroscience to try to detect what mechanisms are at work when individuals shift to another way of looking at their environment, (see, [START_REF] Cohen | Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration[END_REF] in which individuals were faced with a changing environment. Thus they were not faced with a problem to be solved but rather with a problem of adapting their choices to this evolving environment. in CI). In contrast to the NI treatment, however, we find no evidence of participants settling into more complex and less profitable choice patterns.

Descriptive Results

Before proceeding with the modeling of behaviour, we check again whether the observed frequencies of safe choices were driven by the participants' risk aversion. The scatter plots in Figure 8 suggest no clear pattern in this regard, and since the reported correlation coefficients are not significant (p > 0.06) we reject the conjecture that behaviour in these information treatments is driven by risk aversion.

Extending the baseline model

In what follows we extend our baseline model to account for the additional information provided in each of the above treatments. We note, however, that subjects were not informed that if they chose the same arm as the other subjects with whom they are matched, they would receive the same pay-off. This reduces the inferences they could make from their observations.

The CPI case

We assume that agents keep track not only of their own choice and payoff, but also of the other's, and thus learn on the basis of these two information sets. Let A i t (a) summarize, at the beginning of period t, the past experience for agent i from choosing option a. Let A i 0 (a) = 0.5 for all i and a. We assume that A i t (a) evolves as follows:

A i t+1 (a) =                α i A i t (a) + (1 -α i )π i t if a = a i t α i A i t (a) + (1 -α i )π j t if a = a j t A i t (a) otherwise
As for the NI treatment, we assume that agent i becomes aware of inter-temporal dependencies when i observes the same sequence of (choice, payoff) pairs. However, this sequence of (choice, payoff) pairs can now be based on the sequence of his own (choice, payoff) pairs as well as on the sequence of j's (choice, payoff) pairs. Let B i t (a|h) = A i t (a) for all h from the moment when i becomes aware of the inter-temporal dependencies. We assume B i t (a|h) to evolve as follows:

B i t+1 (a|h) =                β i B i t (a|h) + (1 -β i )π i t if a|h = a i t |h i t-1 β i B i t (a|h) + (1 -β i )Γ i t π j t if a|h = a j t |h j t-1 B i t (a|h) otherwise
Based on B i t+1 (a|h), the choice of option a for agent i in period t is thus be defined as:

P r(a i t = a) = e λ i B i t (a|h i t-1 ) + e λ i B i t (a|h j t-1 ) k e λ i B i t (k|h i t-1 ) + k e λ i B i t (k|h j t-1 )

The PI case

The basic idea here is that when i observes j obtaining a sequence of payoff 1s, i tries to figure out how j is obtaining such a stream of high payoffs based on (i) what i knows about the set of possible (choice, payoff) pairs and (ii) their inter-temporal dependencies. On the one hand, if i is not yet aware of conditional strategies, then we assume that by observing j's sequence of high payoffs, i becomes aware of the existence of such strategies. Adopting the same reasoning, i would need to observe j obtaining a sequence of consecutive π j t = 1 of length l i = 2, at least r i times.

If i is already aware of conditional strategies and has just experienced the (choice, payoff)

sequence (1, 1), (1, 0), then i will infer (assuming he already knows that Π(a) i t = {0, 1} for a ∈ {1, 2, 3} and Π(4) i t = {0.3}) that choosing either 2 or 3 instead of 1 after (1, 1) could have generated a payoff of 1. In fact, this type of inference is possible in our set-up only after i obtains a payoff of 1 in the previous period and 0 in the current one. 13 Notice also that this means that i will not use the information about j's payoff when s/he is not using a conditional strategy. We therefore assume that A i t (a) evolves as follows:

A i t+1 (a) =        αA i t (a) + (1 -α)π i t if a = a i t A i t (a) otherwise
On the other hand, if i is already using a conditional strategy (or has become aware of such strategies due to the observation of j's payoff) then,

B i t+1 (a|h) =                βB i t (a|h) + (1 -β)π i t if a|h = a i t |h i t-1 βB i t (a|h) + (1 -β)1 if h = h i t-1 , a = a i t , π i t = 0, π i t-1 = 1, 1 ∈ Π i t (a) B i t (a|h) otherwise
Again, we are basically assuming that j's payoff information is used only if j obtains a payoff of 1 in two consecutive periods. The choice of a conditional strategy in period t + 1 will be based on B i t+1 (a|h) just as in the baseline case.

The CI case

Here, intuitively, the decision will involve mimicking. We will, however, implement it within our modeling framework through the evolution of attractions for consistency with the analysis of the NI treatment (to keep the modeling framework consistent across our four conditions instead of modeling it directly as agents copying the observed choice patterns). The basic idea will be that when agent i detects a pattern in the choices made by j, i will assume that j is doing it because it generates high payoff. So the attraction for conditional strategies will be updated with "presumed" high payoffs based on i's observation about j's choices (of course, given i's knowledge about the set of possible payoffs for each options and conditional choices).

First i has to recognize patterns in j's choices. He does so when he observes that j has been making the same sequence of l i consecutive choices r i times in the row. Say i has observed j making the sequence of choices 3, 2 twice in row. If i has already learned about the performance of conditional strategies, this will translate into i assuming that strategy 2|(3, 1) will result in payoff of 1.

In the case where, i has not yet become aware of conditional strategies then the first time i notices the pattern in j's choice sequences, we assume that i then becomes aware of the existence of such strategies and starts learning about their performance.

CPI Discovery Time

Total Payoff outcomes of Kruskal-Wallis (KW) and Permutation tests that assess their stochastic equivalence.

Both the simulated and the observed distributions (and the test statistics) clearly support the conjecture that the more complete the information one has about the other subject, the faster the hidden pattern will be discovered. However, partial information of the type provided in our experiments participants receive (i.e, choices or payoffs) does not significantly affect the variables of interest; as both the simulated and the observed distributions indicate. The data actually suggest no significant difference between NI and CI or PI (p > 0.05) and significantly smaller (higher) discovery times (total payoffs) in CPI than in the other treatments.

Does "observation" facilitate discovery?

Although it is quite intuitive that observing the choices and/or payoffs of another person who has discovered the hidden pattern will help the observer to discover the hidden pattern, it is less clear whether observing the choices and/or payoffs of another participant who has not found any pattern is equally helpful. We investigate this by first identifying in each pair of subjects, the one who discovered the hidden pattern first, henceforth the Early participant, or second, henceforth the Late participant. In case the hidden pattern was not discovered, the labels are randomly assigned.

We also randomly match participants in the NI treatment in pairs and assign labels by comparing their respective discovery times to get some benchmark for Early and Late participants who do not observe any other participant.

Figure 11 shows the distributions of discovery times of each type of participant in each treatments along with the outcomes of nonparametric tests to assess the observed differences. The plots and test statistics for Late participants (right panel) indicate that they greatly benefited from this additional information. It appears that those in CPI benefited most and significantly more than those in CI or PI whose distributions are not significantly different. The plots also indicate that when compared to the NI treatment, the partial information feedback of these treatments significantly improved their time of discovery. However, for the Early participants (left panel) we find no significant difference across treatments so that the additional information provided in the information treatments had no significant effect on the performance of Early participants.

Concluding remarks

This paper examines the problem of how people learn to improve the results that they obtain from taking different actions. In particular we focus on situations in which individuals have no knowledge of the structure of the environment within which they operate. In many problems of this sort, it is assumed that there is a simple mapping from actions to pay-offs. We looked at a situation in which pay-offs depend on the sequence of actions taken. To make improvements in such a case, subjects, who may start by reinforcing on the pay-offs from single actions, have to realise that there is an inter-temporal dependence.

Our experiments involved multi-armed bandits with a hidden deterministic payoff structure and we wished to see whether subjects could discover this structure or to behave as if they had done so. In a baseline treatment (NI), subjects were only informed of the payoff they themselves have obtained after each choice. While some of our subjects were surprisingly fast in "discovering" the hidden mechanism in this treatment, half of them failed to do so within 200 rounds. Most of our subjects initially explored among four different options. At some point, however, those who were successful seem to had, what has been referred to in the psychology literature as an "aha" moment, and started to reinforce on sequences of actions taken in the past. We developed a simple model that tries to incorporate this type of "counter intuitive learning" and found that it generates (simulated) distributions of discovery times and of total payoffs that match remarkably well those observed.

We then considered a further series of experiments through which we examined how the possibility of observing what others do or obtain may help subjects to improve their gains. This is the problem that is raised in the literature on "observational learning." In these additional paired treatments, a subject could, in addition to the own payoff, observe the last choice made (CI), the last payoff obtained (PI), or both (CPI) by another subject. In these treatments, however, the additional observation did not seem to significantly facilitate the task for subjects who were the first in their pair to discover the pattern to do so compared to those in the NI treatment who discovered the pattern earlier in randomly created pairs. However, compared to those who were slower to discover the pattern in NI treatment, those subjects in paired treatments who discovered the pattern later than the other discovered it significantly faster. This was true even if subjects were only observing the payoffs, and not the choices, obtained by the other and thus had to discover the exact choice sequence leading to consecutive payoffs of one, the maximum attainable in our framework. As might be expected, the CPI case led to the most improvement for these slower subjects.

We extended our basic model to capture the observed behaviour in these additional experiments. We found that our model gave discovery times which corresponded well to the experimental evidence in the CI and the PI treatments. In the CPI treatment, however, our model resulted in later discovery times than was observed in our experiment. This gap between the model and the experimental observations may reflect some compounding effect of information that is not captured well in our model.

We believe our modelling approach is the first to study this type of learning with(out) observational learning and hope that it will induce further research that will investigate whether our results are specific to the particular deterministic payoff generating mechanism we employed in our experiment. For example, one can investigate how the existence of a safe option that returned a low but sure payoff would affect the results by eliminating such an option or changing it to a risky option with the same expected payoff. One could also extend the memory of the subjects, by displaying the past outcomes, to see if this facilitates their learning.

Another fruitful avenue for future research would be to consider several models of learning that allows an agent to become aware of features of the environment and start expanding the set of possible choice strategies, and compare which model does a better job in replicating the observed experimental outcomes by running a competition à la [START_REF] Erev | From anomalies to forecasts: Toward a descriptive model of decision under risk, under ambiguity, and from experience[END_REF]. Among many possibilities, building upon the models proposed by [START_REF] Donoso | Foundations of human reasoning in the prefrontal cortex[END_REF] or [START_REF] Plonsky | Reliance on Small Samples, the Wavy Recency Effect, and Similarity-Based Learning[END_REF] seem to be fruitful ways forward.

Finally, as we have mentioned, a number of the questions that are raised in this paper are related to issues raised in the psychology and in the neuroscience literature. For example, to better understand the mechanisms involved in producing an "aha" moment, a moment in which the subjects change their perspective on the environment about which they are learning, would be of considerable interest. To understand the neural processes at work when such an insight occurs, possibly by using fMRI or other instruments, is a promising avenue for future research. one of the ten questions that you have answered and you will be rewarded according to your decision (ie., Option A or Option B) for that question.

Even though you will make ten decisions, only one of these will end up affecting your earnings, but you will not know in advance which decision will be used. Obviously, each decision has an equal chance of being used in the end.

Figure 1 :

 1 Figure 1: Exemples of choice and payoff patterns in the baseline treatment.

Figure 3 :Figure 4 :

 34 Figure 3: Comparison of the experimental data and simulated model for three subjects.

Figures 5 ,

 5 Figures 5, 6 and 7 display the time-series of choices and payoffs for four pairs of subjects in each of these three additional treatments, CPI, PI, and CI, respectively. The plots indicate that while some pairs never discovered the hidden pattern (Pair 7 in CPI and in CI and Pair 12 in PI), in a few others, one participant discovered it whilst the other did not (Pair 1 in CPI, Pair 3 and 14 in PI and Pair 5 in CI). Yet, in other pairs, both participants eventually discovered patterns (Pairs 5 and 16 in CPI, Pair 1 in PI and Pairs 1 and 4 in CI). As in the NI treatment, a few participants who discovered the pattern occasionally deviated from it in subsequent rounds (Subjects 5 and 6 of Pair 5 in CPI, Subject 27 of Pair 14 in PI and Subject 19 of Pair 4 and Subject 23 of Pair 5

  12

Figure 5 :Figure 6 :Figure 7 :

 567 Figure 5: Examples of choice and payoff patterns in the CPI treatment.

Figure 9 :

 9 Figure 9: CDFs of discovery times and total payoffs. Legend: Observed (solid), Simulations (dashed). Mean (standard deviation) are also shown. Pvalues are based on Permutation tests (for independent samples), two-tailed.

Figure 10 :

 10 Figure 10: CDFs of discovery times and total payoffs (with p-values of Kruskal-Wallis tests for multiple sample comparisons and permutation tests for pair-wise comparisons (one-tailed)). Legend: NI (blue), CPI (red), PI (dashed orange), CI (light blue).

Table 1 :

 1 Summary of experimental treatments

	Treatment No. of subjects Description
	NI	30	Each subject independently tries our bandit problem
	CPI	32 (16 pairs)	Subject can observe both the choice made by one another
			subject and the payoff s/he obtained (the pair is fixed
			throughout the experiment).
	PI	32 (16 pairs)	Subject can observe the payoff obtained by one another
			subject (the pair is fixed throughout the experiment).
	CI	32 (16 pairs)	Subject can observe the choice made by one another sub-
			ject (the pair is fixed throughout the experiment).

be aware of the full structure of the problem, or think of individuals trying constantly to adapt to an environment which is non-stationary. 11

  Figure 11: CDFs of discovery times of Early and Late participants (with p-values of Kruskal-Wallis tests for multiple sample comparisons and permutation tests for pair-wise comparisons (one-tailed)). Legend: NI (blue), CPI (red), PI (dashed orange), CI (light blue).Next, to determine who benefited most from the information provided, we tracked the evolution of average payoffs of each type in the four treatments. Figure12reports the median of the average pay-offs from each round for Early and Late participants for each block of 10 rounds in each treatment. The plots and the reported KW test statistics (c.f. lower panel of Figure12) suggest that round average payoffs are not significantly different across treatments for the first five blocks and are actually very similar across types. Differences emerge from the sixth block onwards as the round average payoffs of Late participants vary widely across treatments whereas those of Early participants become identical. This confirms that the information provided did not affect the discovery time of Early participants whereas it significantly helped Late participants in the three information treatments considered. To repeat, observing a successful subject is very helpful but observing an unsuccessful subject does not speed up learning. Figure 12: Dynamics of median round average payoff (over block of 10 rounds) of Early and Late participants (with p-values of Kruskal-Wallis tests).

			Discovery Time of Early participants							Discovery Time of Late participants
	CDF 1.0 Median Π	the Early participants					CDF 1.0 Median Π	the Late participants
	1.0 0.8									1.0 0.8
										NI					NI
	0.8 0.6								NI CI	0.8 0.6			NI CI
	0.6 0.4								CI PI	0.6 0.4			CI PI
										PI CPI					PI CPI
	0.4 0.2								CPI	0.4 0.2			CPI
	0.2	0	5	10		15	20	Τ		0.2	0	5		10	15	20	Τ
	0 0.0		50	100	150	200 t		0 0.0	50		100	150	200 t
		p = 0.527 (KW) p-values CI PI NI 0.575 0.647 0.221 CPI p-values from Kruskal-Wallis test p < 0.001 (KW) p-values CI PI NI τ 1 2 3 4 5 6 7 8 9 0.001 0.005 < 0.001 CPI 10 Early participants 0.249 0.709 0.814 0.416 0.801 0.288 0.430 0.261 0.852 0.300
		CI Late participants	0.575 0.172 0.098 0.204	0.592	0.067	0.981	CI 0.008	0.044	0.551 0.039 0.004	0.027 0.018
		PI Early participants τ	11 0.126	0.147 12 0.317	13 0.638	14 0.852	15 0.693	PI 0.759 16	17 0.690	18 0.511	19 0.885	20 0.035 0.676
		Late participants	0.003	0.019		0.014	0.005	0.014	0.012		0.008	0.032	0.010	0.054

Legend: NI (blue), CI (solid light blue), PI (dashed light blue), CPI (red).

Table 2 :

 2 List of questions fro Holt and Loury task.

	Option A	Option B
	1. A$2.0 with 10% chance or A$1.60 with	A$3.85 with 10% chance or A$0.1 with
	90% chance	90% chance
	2. A$2.0 with 20% chance or A$1.60 with	A$3.85 with 20% chance or A$0.1 with
	80% chance	80% chance
	3. A$2.0 with 30% chance or A$1.60 with	A$3.85 with 30% chance or A$0.1 with
	70% chance	70% chance
	4. A$2.0 with 40% chance or A$1.60 with	A$3.85 with 40% chance or A$0.1 with
	60% chance	60% chance
	5. A$2.0 with 50% chance or A$1.60 with	A$3.85 with 50% chance or A$0.1 with
	50% chance	50% chance
	6. A$2.0 with 60% chance or A$1.60 with	A$3.85 with 60% chance or A$0.1 with
	40% chance	40% chance
	7. A$2.0 with 70% chance or A$1.60 with	A$3.85 with 70% chance or A$0.1 with
	30% chance	30% chance
	8. A$2.0 with 80% chance or A$1.60 with	A$3.85 with 80% chance or A$0.1 with
	20% chance	20% chance
	9. A$2.0 with 90% chance or A$1.60 with	A$3.85 with 90% chance or A$0.1 with
	10% chance	10% chance
	10. A$2.0 with 100% chance or A$1.60 with	A$3.85 with 100% chance or A$0.1 with
	0% chance	0% chance

By "discover", we mean that a subject uses the "best" pattern and earn a payoff of 1 for successive rounds. We define a participant as having discovered the best pattern if he or she earned a payoff of 1 for twelve consecutive rounds for the first time and used the middle round of this sequence as the discovery time.

Six out of thirty participants reverted to the "safe lottery" after having stopped choosing it. As such behaviour might be considered aberrant since it seems to be inconsistent with rational choice theory under uncertainty, we conducted our analysis with and without these participants but found no significant change in our conclusions. For the analyses including these subjects, we have used their first switching point to measure their degree of risk aversion. It is also worth noting that in our context, if an individual who has chosen the safe arm, switches unsuccessfully to another choice it would not be irrational to come back to the safe arm.

One approach is that of econometricians who argue that what is being learned are the characteristics of the data generating process with which agents are faced (see,[START_REF] Dufour | Model selection[END_REF], for a survey). However, whilst various forms of Bayesian learning have been explored, the problem of how to act when the actual data generating process does not fall into the class which the agent has considered, remains a difficult one. One approach is to start with a very general specification in which all possible variables, functional forms, lags etc are included in the set of candidate models. However, it has been argued that to start with such a general specification and then to narrow it down to a specific model is simply too costly. But[START_REF] Castle | Evaluating Automatic Model Selection[END_REF] give a full discussion of this problem and argue that it is now technically possible and not necessarily very costly if one uses automatic selection procedures. Nevertheless it seems

[START_REF] Agarwal | Taming the monster: A fast and simple algorithm for contextual bandits[END_REF] point out, contextual bandit problems are found in many important applications such as online recommendation and clinical trials. The use of the term contextual is important because it suggests that other features of the choices can be taken into consideration, thus one can argue that this sort of model represents an intermediate point between "supervised learning" and pure "reinforcement learning."

An even more difficult problem that has received some attention when dealing with games is that of recognising patterns in an opponent's behaviour. Here the difficulty is that the pay-off to each player depends on both players' choices. One attempt to do that was made by[START_REF] Sonsino | Learning to learng, pattern recognition, and Nash equilibrium[END_REF] and his idea of "learning to learn" in this case, is simply illustrated when he says, "As a simple example, consider the case where three strategy profiles, say C, A, B, have been played repeatedly, for a long time, in that specific order, so that the history of play at some stage is . . .C, A, B, C, A, B, C, A, B, C, A, B, C, A. We claim that the players must " recognize" the repeated pattern if it has been repeated successively with no interruptions a large enough number of times." Recently,[START_REF] Spiliopoulos | Pattern recognition and subjective belief learning in a repeated constant-sum game[END_REF]Spiliopoulos ( , 2013) ) showed that introducing ability to recognize pattern in the opponent's choices in belief-based learning model greatly improves the fit of the model to the experimental data. In our case the subject can be thought of as playing "a game against nature" where nature is playing a deterministic sequence. We do not need to give such an interpretation however, and the problem boils down to one of identifying a structure in the pay-offs without considering that they are generated by an "opponent."

We can easily allow an agent to start considering dependencies based on (choice, payoff) sequences that do not start with payoff of 1. This will in fact speed up the learning of the inter-temporal dependencies. We did not do so here because (choice, payoff) sequence starting with payoff of zero, such as (1, 0) can result in payoff of both 0 and 1 for all the possible choices among three options.

[START_REF] Bayer | Do we learn from our own experience or from observing others?[END_REF] report evidence that suggests that subjects actually learn mostly from their own experience rather than from other's. However, their finding pertains to Bertrand duopolies which involve strategic interactions whereas our setting involves no such interactions.

After receiving a 0 payoff in the previous period, all choices (but 4) can result in both payoff of 0 or 1.

, and the Universities of Nice (GREDEG) and Toulouse for useful comments and suggestions. Support from the School of Economics of the University of New South Wales where the experiments were conducted, LABEX OT-MED (ANR-11-LABX-0061), a JSPS-ANR bilateral research grant "BECOA" (ANR-11-FRJA-0002) and the Australian Research Council through a Discovery Project grant (DP140102949) are gratefully acknowledged. The usual disclaimer applies.

Recall, however, that i does not observe j's payoff directly. Thus, we assume that when i is making unconditional choices, the observation of j's choice will not be used in the evolution of A i t (a); and that this will happen only when i's choices are conditional.

if a|h = (a j t |(a j t-1 , 1)), a j t ∈ {1, 2, 3}

The choice of a conditional strategy in period t + 1 will be based on B i t+1 (a|h) just as in the baseline case.

Simulation results

We simulate the above models using the parameter values best that best fit the baseline model described in Section 3. So for each individual in each pair of 500 simulated pairs of agents, we randomly draw (with replacement) each of the parameter values from the set {(α i , λi , ri )} obtained for the NI treatment. The resulting distributions of individual discovery times and total payoffs are displayed in Figure 9 along with the observed ones.

The plots suggest that, overall, subjects discovered the hidden cycle somewhat faster than our simulated agents (c.f. left panels) which consequently led them to earn higher payoffs (c.f., right panels). However, this only partially holds for the CPI treatment since the outcomes of Permutation tests indicate a significant difference at α = 0.05 between simulated and observed only for total payoffs (p = 0.058 for discovery times and p = 0.024 for total payoffs). Otherwise, we find no significant difference so that, as the reported average statistics also indicate, our modeling approach provides a remarkably good overall out-of-sample fit to the data ( p = 0.500 in PI and p = 0.189 in CI for discovery times, and p = 0.482 in PI and p = 0.277 in CI for total payoffs).

We proceed with cross-treatment comparisons of the models' simulations and the observed data.

The upper panel of Figure 10 displays the simulated distributions of discovery times (left panel) and total payoffs (right panel) whereas the lower panel displays the observed distributions along with the

Appendix

The experiments were conducted in the laboratory for experimental economics of the Australian Business School (ASBLab) at the University of New South Wales. Participants were students in Economics, Commerce, Marketing, and Engineering who were recruited on campus using the ORSEE software [START_REF] Greiner | An Online Recruitment System for Economic Experiments[END_REF]. Upon arrival in the laboratory, they were randomly assigned to individual cubicles and given the following sets of instructions that were read aloud. Once the experiments finished, they were individually called to collect their reward in cash. The experiments were programmed using z-tree [START_REF] Fischbacher | z-Tree: Zurich toolbox for ready-made economic experiments[END_REF].

Instructions for treatments CPI, PI and CI

Welcome to the ASBLab.

If you read the following instructions carefully, you can, depending on your decisions, earn a considerable amount of money. It is therefore very important that you read these instructions carefully. The instructions are the same for all participants.

It is prohibited to communicate with the other participants during the experiment. If you have a question at any time raise your hand and the experimenter will come to your desk to answer it. Please switch off your mobile phone or any other devices which may disturb the experiment. Please use the computer only for entering your decisions. Please do not start or end any programs, and do not change any settings.

This Experiment

You are about to participate in an experiment which consists of two parts.

The first part consists of 200 rounds of play. The second part will be explained to you when you finished the first part.

First Part Task

In each of the 200 rounds of play, you are asked to choose one of 4 one-armed bandits. Once you made your choice, you will be informed and will receive the bandits' payoff (in Experimental Currency Units, ECUs). The experiment then proceeds to the next round.

Information

You are not aware of the payoffs that you may receive from each of these bandits but you are told that a bandits' payoff outcome in one round does not depend on which bandit you chose in the previous round.

You are not allowed to collect any written information on the bandits' payoff outcomes.

== [CPI treatment only] ==

Throughout the experiment, you will be matched with one other participant. At the end of each round, you will get to know this participant's bandit choice and the payoff outcome of that choice, and s/he will get to know your bandit choice and the associated payoff. Note that the information displayed is about bandit choices and the payoff outcomes associated to these choices.

== [CI treatment only] ==

Throughout the experiment, you will be matched with one other participant. At the end of each round, you will get to know this participant's bandit choice, and s/he will get to know your bandit choice.

== [PI treatment only] ==

Throughout the experiment, you will be matched with one other participant. At the end of each round, you will get to know the payoff outcome for this participant's, and s/he will get to know your payoff outcome.

Payment

Your reward from participating to this first part will be equal the sum of the payoffs that you realised, and this sum will be converted to the rate of $0.2 per ECU and individually paid to you.

Second Part Task

In the second part of the experiment, you are asked 10 times to choose between "Option A" and "Option B." Please read carefully the questions asked. (The questions were displayed in sequence and were phrased as in Table 2 below)

Payment

Once you have answered all questions, your reward from participating to this second part will be determined by 1) your answer to one of these ten questions and 2) by chance. The computer will randomly select


Yann Quinsat
email: yann.quinsat@lurpa.ens-cachan.fr

Claire Lartigue

Christopher A Brown

Lamine Hattali

Multi-scale surface characterization in additive manufacturing using CT

Keywords: CT measurement, additive manufacturing, multi-scale analysis

In additive manufacturing, the part geometry, including its internal structure, can be optimized to answer functional requirements by optimizing process parameters. This can be performed by linking process parameters to the resulting manufactured geometry. This paper deals with an original method for surface geometry characterization of printed parts (using Fused Filament Fabrication FFF) based on 3D Computer Tomography (CT) measurements. From 3D measured data, surface extraction is performed, giving a set of skin voxels corresponding to the internal and external part surface. A multi-scale analysis method is proposed to analyse the relative internal area of the total surface obtained at different scales (from sub-voxel to super-voxel scales) with different process parameters. This analysis turns out to be relevant for filling strategy discrimination.

Introduction

AM product examples • Porous structures

In additive manufacturing, the weight of parts can be optimized to support a given load by designing an internal structure, including porosity, using a given filling strategy (figure 1). Relations between strength and filling strategies are required for product and process design optimization. Our preliminary study has shown the significant influence of FFF (Fused Filament Fabrication) process parameters on both the mechanical and fracture properties of the manufactured parts.

Within the context of fracture study, a series of bending samples has been carried out by considering variable filling parameters such as layer thickness, filling mode, and density. All the specimens presented a small cracking directly issued from manufacturing (figure 2(a)). Following cracking initiation, a 3-point bending test was performed with the view to determining K C , critical stress intensity factor corresponding to fracture initiation. Results displayed in table 1 clearly highlight the great influence of the process parameters on the value of Kc. Therefore, the relations between strength and filling strategies are necessary for product and process design optimization. In the paper, we propose a first approach to determine the relation between the filling strategy and the resulting surface topography, both internal and external, which can be linked to strength. Surface topography is thus considered the key element between the manufacturing process and the part function. The study of 3D surface topography obtained by FFF is a problem welladdressed in the literature. The most classical approximation consists in modeling the external surface as a juxtaposition of elliptical pipes (or tubes) [START_REF] Ahn | Representation of surface roughness in fused deposition modeling[END_REF]. The influence of scanning and deposition directions on external topography during filling has been studied in [START_REF] Galantucci | Experimental study aiming to enhance the surface finish of fused deposition modeled parts[END_REF][START_REF] Pandey | Improvement of surface finish by staircase machining in fused deposition modeling[END_REF]. Zeng et al. show the influence of the process on local curvatures in a multi-scale analysis [START_REF] Zeng | Multi-scale evaluations of the roughness of surfaces made by additive manufacturing[END_REF]. An approach based on a finite element analysis is proposed in [START_REF] Jamiolahmadi | Surface topography of additive manufacturing parts using a finite difference approach[END_REF] in order to predict 3D surface topography issued from FFF based on a few measured points. Meanwhile, Pupo et al. [START_REF] Yurivania | Influence of process parameters on surface quality of cocrmo produced by selective laser melting[END_REF] establish the link between process parameters and the obtained topography in the case of sintering of metallic powders. Hence, most studies address the influence of process parameters on external surface topography, but few works have studied the effect of process parameters on porosity. One difficulty lies in measuring the internal part surface. Computer Tomography (CT) now allows non-destructive 3D measurements of internal and external part geometry with uncertainties down to micrometers [START_REF] Wang | Review of the mathematical foundations of data fusion techniques in surface metrology[END_REF][START_REF] Chiffre | Industrial applications of computed tomography[END_REF][START_REF] Bartscher | Enhancement and proof of accuracy of industrial computed tomography (ct) measurements[END_REF]. Data issued from tomography represent the different levels of X-ray attenuation of the part for each point of the measured volume. The information is stored as a numerical value (gray level) in a volume element called voxel [START_REF] Yage-Fabra | A 3d edge detection technique for surface extraction in computed tomography for dimensional metrology applications[END_REF]. Surface extraction from these 3D data (also referred to as 3D edge detection) is the first necessary stage for metrology applications [START_REF] Dewulf | Uncertainty determination and quantification for dimensional measurements with industrial computed tomography[END_REF][START_REF] Lifton | On the uncertainty of surface determination in xray computed tomography for dimensional metrology[END_REF], and is strongly linked to the voxel size which represents the tomography resolution [START_REF] Kruth | Computed tomography for dimensional metrology[END_REF]. Some studies show that uncertainty in surface extraction can be reduced by making use of a sub-voxel resolution method which numerically improves CT resolution [START_REF] Lifton | On the uncertainty of surface determination in xray computed tomography for dimensional metrology[END_REF][START_REF] Yage-Fabra | A 3d edge detection technique for surface extraction in computed tomography for dimensional metrology applications[END_REF]. Uncertainty can achieve 1/10 of the voxel size when sub-voxel resolution is used [START_REF] Carmignato | Accuracy of industrial computed tomography measurements: Experimental results from an international comparison[END_REF], which makes CT applied for dimensional metrology applications.

Within this context, this paper deals with a method for surface characterization in additive manufacturing to discriminate the filling strategy based on CT measurements. The objective is to propose a tool to discriminate the internal geometry based on the relative internal area evaluation. The originality of this approach lies in the use of a multi-scale discrimination method, which analyses the relative internal area of the total surface, including internal surfaces of pores from CT measurements. For the experimentation, parts were made using 3-D printer (Ultimaker 2) with three different filling strategies as displayed in figure 3. For each specimen, the same layer thickness ep = 0.25mm was used. This paper is organized as follows. Section 2, is dedicated to the presentation of our method for CT data treatment. In section 3, the surface area, including that of the pores, is characterized as a function of scale. The method application is proposed in section 4.

CT data treatment

Data obtained by CT measurements correspond to a set of images defined in 3 perpendicular directions (see figure 4(b)) which constitute a set of voxels. For me-trology applications, the part geometry is defined from the contours of its surface (both external and internal). Ontiveros et al [START_REF] Ontiveros | Computer tomography 3d edge detection comparative for metrology applications[END_REF] propose a review of surface extraction methods in relation with metrology applications. Among all the methods, threshold methods are commonly used for industrial applications: a gray level value is defined as a limit to differentiate the air and the material [START_REF] Ontiveros | Computer tomography 3d edge detection comparative for metrology applications[END_REF].

To conduct a surface characterization by multi-scale analysis, an automatic CT data treatment method is required. Hence, a threshold method, close to the algorithm proposed in [START_REF] Shahabi | Simulation and measurement of surface roughness via grey scale image of tool in finish turning[END_REF], and using Matlab© Image Processing Toolbox, has been developed. The first step is then image binarizing according to a threshold value determined with the well-known Otsu's method [START_REF] Otsu | A threshold selection method from gray-level histograms[END_REF]. Otsu's thresholding chooses the threshold to minimize the intraclass variance of the thresholded black and white pixels. This step is followed by a morphological operation on the binary image which removes interior pixels and keeps the outline of the shapes. As surface extraction is the key point of surface characterization, and the basis for the evaluation of metrological quantities, we propose to assess our method.

For this purpose, a gauge block made in ZrO2 is measured. This gauge ensures a flatness defect of 0.10 μm and a parallelism defect between its two faces of 0.10 μm. The calibrated distance between the 2 faces is 1.3 mm with an uncertainty of ± 0.12 μm at 20°C. CT measurements are performed with a Zeiss Metrotom for which the voxel size is 5.9μm×5.9μm ×5.9μm. Finally, the measured volume consists of 200×300×1132 voxels. Our extraction method is applied leading to 2 sets of points, each one corresponding to one of the 2 gauge faces (figure 5). To test the interest of the sub-voxel technique, the extraction method is applied considering a smaller voxel-size, 30% of the original voxel-size. This is simply obtained by a sub-voxel linear interpolation.

Assessment is thus performed considering two quality indicators, as proposed in [START_REF] Mehdi-Souzani | A knowledge database of qualified digitizing systems for the selection of the best system according to the application[END_REF]: noise and trueness. Noise accounts for the measurement dispersion around each extracted face (a plane in practice), and trueness measures the difference between the calibrated distance and the measured distance obtained after face extraction (plane-plane distance). Results, displayed in table 2, are consistent with the gauge quality, which illustrates the efficiency of our surface extraction method. It also can be observed that the noise along with the trueness is decreased when using a smaller voxel-size. This interesting result shows that sub-pixel refinement, as suggested by numerous studies [START_REF] Jiménez | An optimized segmentation algorithm for the surface extraction in ncomputed tomography for metrology applications[END_REF][START_REF] Ontiveros | Computer tomography 3d edge detection comparative for metrology applications[END_REF][START_REF] Yage-Fabra | A 3d edge detection technique for surface extraction in computed tomography for dimensional metrology applications[END_REF] actually improves surface extraction.

Surface characterization by multi-scale analysis

To link process parameters with the resulting surface, a multi-scale discrimination method is proposed to analyze the relative internal area of the total surface. The surface area, including that of the pores, is characterized as a function of scale. Prior to the description of our multi-scale analysis method, the calculation of the surface area based on the identification of the skin voxels is first detailed.

Relative area calculation

The area of the part surface can be defined from the 3D extracted contours, these contours defining a kind of skeleton structure. Hence, the skeleton is a 3D element. To simplify the representation, an example of the skeleton's construction in 2D is reported in figure 6. Indeed, the operation of surface extraction leads to the identification of all the voxels containing a portion of surface (internal or external) referred to as skin voxels, V skin (figure 7(b)). The relative internal area is calculated as the sum of all the areas of the skin voxel faces:

      skin V i 6 1 j j , i j , i A A  Where       

 

The area so calculated depends on the dimensions of the studied zone. To avoid this problem, the relative internal area A r is introduced, defined by A r =A/A b , where A b is the area of the big voxel bounding the studied zone (figure 7(a)). Considering a 2D representation of a specimen measurement, the set of skin voxels corresponds to the white voxels in figure 7(c).

Multi-scale analysis

In order to conduct a multi-scale analysis, the surface area is characterized as a function of scale [START_REF] Brown | Scale-sensitivity, fractal analysis and simulations[END_REF]. The voxel size defines the scale, and we propose to vary this value to have scales smaller and larger than the original dimension related to the measurement scale. The value of the material density belonging to the considered voxel is obtained by linear interpolation. The influence of such a variation on the surface extraction is illustrated in figure 9. The parameter A r can thus be calculated as a function of scale. The scale is chosen as the area of the voxel surface for a considered voxel size. The presented method proposes to vary this voxel size (i.e. the scale) and to evaluate the corresponding internal surface area, the scale varying from sub-voxel to meta-voxel. The relative internal area evolves, increasing linearly on a decreasing logarithmic scale to a plateau (see figure 10). The plateau begins at the scale of the original measured voxel (identified by a vertical blue line in the figure). Results highlight that, in this study, the numerical sub-voxel treatment (scales < 10 -2 in the present case) does not provide additional information. In the same way, meta-voxels are too far from reality. When the size of the voxel is close to the size of the test part, the relative internal area becomes lesser than 1, which is not realistic. The study can thus be restricted to the zone in which the relative internal area is above 1.

Application and discussion

The interest of our surface characterization method is illustrated through 2 applications. The first one concerns the use of such a method to discriminate filling modes, and the second one more particularly addresses the influence of process parameters on resulting printed surfaces.

Filling mode discrimination

For this application, 3 different filling modes are used as described in figure 3 with the same process parameters: thickness is set to 0.25 mm, and the filling is imposed to 60 %. The original voxel sizes are 37.8μm × 37.8μm × 37.8μm, and the bounding voxel size is set to 5mm × 5mm × 5mm. The multi-scale analysis is applied to the CT measurement obtained for the 3 specimens. Results, displayed in figure 11, show that the evolution of the relative internal area is different depending on the filling mode. This is particularly distinct at low scales.

The linear decrease with the increase of the scale is in turn greater when the original relative internal area is the largest, corresponding to the zig-zag strategy. This analysis turns out to be relevant for filling strategy discrimination: multiscale analysis of the relative internal area enables printed surface characterization.

Study of process parameters

The second illustration aims at studying the influence of process parameters on the manufactured surface. The zig-zag mode is considered for this study. The process parameters retained are level thickness, direction of filling, and filling rate. A design of experiment is proposed according to a table L9 with 3 factors for 3 levels (table 3). The 9 manufactured specimens are measured using CT and a multi-scale analysis of the relative internal area is afterwards performed, leading to the results reported in figure 12. As displayed in the figure, the value of the relative internal area at the low scales is different in function of the considered specimen. Furthermore, in the linear zone, the slope also varies for each specimen. This study confirms the relevance of using the relative internal area to discriminate surface topographies obtained using different process parameters. It is interesting to notice that due to process uncertainties, specimen for which the filling rate is 100% present some pores, which explains that the relative internal area is not constant for these specimens. This is likely due to process uncertainty.

Conclusion

The set of skin voxels defined the 3D surface topography, both internal and external. To link process parameters to the manufactured surface, a multi-scale discrimination method is proposed which analyses the relative internal area of the total surface. The surface area (area of all the skin voxels), including that of the pores, is characterized as a function of scale. To show the relevance of multi-scale relative internal area analysis, 2 different applications are proposed. The first one clearly highlights the interest of our approach to discriminate filling modes when using FFF. For the second application, the filling mode is the same (zig-zag mode), and the study focuses on the influence of some process parameters (level thickness, filling direction, and filling rate) on the printed surface. This study confirms the relevance of using the relative internal area to discriminate surface topographies obtained with different process parameters. Hence, the geometry and the structure of the part can be analyzed according to process parameters to answer a given part function. This work is a first approach to determine the relationship between mechanical properties and surface topography. We propose to use the relative internal surface area to discriminate the internal geometry. According to multiscale analysis, further work will focus on linking strength and internal relative area. A functional correlation could be found by regressing the relative internal areas at each scale versus the mechanical properties.

Fig. 1 .

 1 Fig. 1. Example of internal structure

(a) 3 - 2 .

 32 point bending setup (b) Typical load evolution vs displacement Fig. Description of the cracking tests

Fig. 3 .

 3 Fig. 3. The 3 different filling strategies: (a) Hilbert curve (b) Honey curve(c) ZigZag curve

Fig. 4 .

 4 Fig. 4. Measurement from CT

Fig. 5 .

 5 Fig. 5. Contour extraction for the gauge block: (left) Gauge Measurement, (middle) Contour extraction (original voxel-size) (right) Contour extraction (30% of the original voxel-size)

Fig. 6 .

 6 Surface extraction in the XY-plane

Fig. 7 .

 7 Fig. 7. Voxel representation of surface extraction

Fig. 9 .

 9 Fig. 9. Surface extraction as a function of scale (defining from initial voxel size) -from left to right: Measurement scale 0.3, scale 1, and scale 3, Surface extraction scale 0.3, scale 1 and scale 3

Fig. 10 .

 10 Fig. 10. Evolution of the relative internal area as a function of the scale

Fig. 11 .

 11 Fig. 11. Filling mode comparison using mutli-scale analysis

FactorFig. 12 .

 12 Fig. 12. Multi-scale analysis for different process parameters

Table 1 .

 1 Values of Kc (from DOE define in table 3)

	Tests	1	2	3	4	5	6	7	8	9

Kc (MPa.sqrt(m)) 0.54 1.27 1.68 0.99 2.03 0.78 0.55 0.68 0.61

Table 1 .

 1 Assessment of the surface extraction method

	Voxel-size	Noise (Face 1) (μm)	Noise (Face 2) (μm)	Trueness (μm)
	original size	3	3.3	25.5
	30% of original size	2.4	2.7	21.2

Table 3 .

 3 Design of experiment.

Acknowledgments Authors would thank Zeiss Industrial Metrology, LLC which provides all the CT measurements.