Djallel Bouneffouf
email: djallel.bouneffouf@orange.com

Romain Laroche
email: romain.laroche@orange.com

Tanguy Urvoy
email: tanguy.urvoy@orange.com

Raphael Feraud
email: raphael.feraud@orange.com

Robin Allesiardo
email: robin.allesiardo@orange.com

Contextual Bandit for Active Learning: Active Thompson Sampling

Keywords: Contextual Bandits, Active learning, Thompson sampling

The labelling of training examples is a costly task in a supervised classification. Active learning strategies answer this problem by selecting the most useful unlabelled examples to train a predictive model. The choice of examples to label can be seen as a dilemma between the exploration and the exploitation over the data space representation. In this paper, a novel active learning strategy manages this compromise by modelling the active learning problem as a contextual bandit problem. We propose a sequential algorithm named Active Thompson Sampling (ATS), which, in each round, assigns a sampling distribution on the pool, samples one point from this distribution, and queries the oracle for this sample point label. Experimental comparison to previously proposed active learning algorithms show superior performance on a real application dataset.

Introduction

Active Learning (AL) has emerged as a popular approach for solving machine learning problems with limited labelled data [START_REF] Settles | Active Learning Literature Survey[END_REF]. In this approach the learning algorithm is "active", and is allowed to query, an oracle O, for the label of points that are maximally informative for the learning process. The result is that, by using few but well chosen labels, the active learning algorithm is able to learn as well as a passive learning algorithm that has access to more labelled data.

In selective sampling, the choice of examples to be labelled can be seen as the dilemma between exploration and exploitation (exr/exp) on the training data. On one hand, an active learning strategy that just exploits the data will be specialized in certain areas of the input space X but will be very poor in generalization. On the other hand, a strategy which uses that exploring data does not focus on regions where X is known to improve the predictive model. These two situations illustrate the need for an active learning to find a compromise between exr/exp of labelling data strategy.

In [START_REF] Ganti | Building bridges: Viewing active learning from the multi-armed bandit lens[END_REF], a similar analysis of the problem led the authors to model the active learning problem as a multi-armed bandit problem. They suppose that the different hypotheses of distribution h ∈ H of the data are the arms and use an adapted UCB (upper confident bound) to select the most promising hypothesis of distribution to select the points to be labelled. The drawback of this approach is in the non consideration of the context or the different features that characterize the points. For example the number of points in a area space, the proportion of labelled points, the ratio of the classes in the area or the density of points in the area can be useful to determine the most interesting to label.

To tackle this problem, we propose to model the active learning as a contextual bandit problem, where we have at first clustered the input space: each cluster is considered as an arm and the different features of the cluster are the context of the arm. Then, we implement a novel algorithm named Active Thompson Sampling (ATS) adapting the Thompson Sampling to the active learning problem. Finally, we evaluate ATS on actual data and find out that ATS outperforms all other algorithms in our panel.

The remaining of the paper is organized as follows. Section 2 reviews related works. Section 3 describes our multi armed contextual bandit model and the ATS algorithm. The experimental evaluation is illustrated in Section 4. The last section concludes the paper and points out possible directions for future works.

Related Work

We refer, in the following, recent works that address Active Learning problem and the exr/exp trade-off (bandit algorithm).

Active Learning. A variety of AL algorithms have been proposed in the literature employing various query strategies. One of the most popular strategy is called uncertainty sampling (US), where the active learner queries the point whose label is the most uncertain [START_REF] Lewis | A sequential algorithm for training text classifiers[END_REF]. Usually the uncertainty in the label is calculated using variance of the label distribution [START_REF] Settles | Active Learning Literature Survey[END_REF]. The authors in [START_REF] Seung | Query by committee[END_REF] introduced the query-by-committee (QBC) strategy where a committee of potential heterogeneous models, is learnt from the labelled data, and used to select for querying, the point where most committee members disagree. Other strategies include the maximum expected reduction in error [START_REF] Zhu | Combining active learning and semisupervised learning using gaussian fields and harmonic functions[END_REF] or variance reducing query strategies [START_REF] Zhang | A probability analysis on the value of unlabeled data for classification problems[END_REF] to querying the optimal point. All above proposed approaches only exploit the data.

Contextual Multi-armed Bandit. Multi-armed bandit (MAB) problems model the exr/exp trade-off inherent in many sequential decision problems. A particularly useful version is the contextual multi-armed bandit problem. In this problem, in each iteration, an agent has to choose between arms. Before making the choice, the agent sees a d-dimensional feature vector (context vector), associated with each arm. The learner uses these context vectors along with the rewards of the arms played in the past to make the choice of the arm to play in the current iteration. Overtime, the learner's aim is to collect enough information about how the context vectors and rewards relate to each other, so that it can predict the next best arm to play by looking at the feature vectors.

Recently, the contextual bandit has been used in different domains such as recommender system (RS) and information retrieval. For example, in [START_REF] Bouneffouf | A contextual-bandit algorithm for mobile context-aware recommender system[END_REF][START_REF] Bouneffouf | A Dynamic Risk-Aware Recommender System[END_REF], authors model RS as a contextual bandit problem. The authors propose an algorithm called Contextual--greedy which sequentially recommends documents based on contextual information about the users. In [START_REF] Chapelle | An empirical evaluation of thompson sampling[END_REF], authors analyse the Thompson Sampling (TS) in contextual bandit problem. The study demonstrates that it has a better empirical performance compared to the state-of-art methods. The TS is one of the oldest heuristics for multi-armed bandit problems and it is a randomized algorithm based on Bayesian ideas. Authors in [START_REF] Chapelle | An empirical evaluation of thompson sampling[END_REF][START_REF] Bouneffouf | A contextual-bandit algorithm for mobile context-aware recommender system[END_REF] describe a smart way to balance exr/exp, but do not study the contextual bandit in the active learning problem.

Multi-armed Bandit for Active Learning. To our knowledge there has been only two papers bridging the world of active learning and MAB. [START_REF] Osugi | Balancing exploration and exploitation: a new algorithm for active machine learning[END_REF] adapted the EXP4 algorithm which is a MAB algorithm with expert advice, where the different active learning algorithms are the various experts and the different points in the pool are the arms of the MAB. An each iteration, every expert provides a sampling distribution on the pool. EXP4 maintains an estimation of the error rate for each expert, and uses exponential weight to select the optimal sampling distribution on the pool. Authors in [START_REF] Ganti | Building bridges: Viewing active learning from the multi-armed bandit lens[END_REF] propose an adaptation of UCB called LCB algorithm, the authors suggested minimizing an unbiased estimator of risk of h, and a sampling distribution that was in proportion to the entropy of the prediction on the pool. The authors consider the arms of the bandit as the different hypothesis, and querying a data point, as the process of improving their estimation of the risk of the different hypothesis.

Our Contributions. As it is observed above, none of the described works has dressed the active learning problem from a contextual bandits view, although the consideration of the pool context might be a very informative feature for an active learning algorithm. This is precisely what we intend to do by exploiting the following new features: [START_REF] Agrawal | Thompson sampling for contextual bandits with linear payoffs[END_REF] We model the active learning as a contextual bandit problem, where each cluster of points in the space is an arm and the different features of the cluster are the context of the arms. [START_REF] Bouneffouf | A Dynamic Risk-Aware Recommender System[END_REF] We propose a new algorithm named Active Thompson Sampling (ATS), that adapts the TS to the active learning problem. [START_REF] Bouneffouf | A contextual-bandit algorithm for mobile context-aware recommender system[END_REF] We evaluate it against other methods form the state-of-the-art.

Key Notions and Proposed Model

This section focuses on the proposed model, beginning by introducing the key notions used in this paper.

In pool based AL we are provided with a pool U 0 = {x 1 , ...x n } of unlabelled points, an empty set of labelled points L 0 = {} and a labelling oracle O, which when queried for the label of x, returns y. Algorithms in the pool based setting have to decide which points to query by looking at the entire pool.

Definition (Contextual Bandit Problem with Linear Payoffs). In a contextual bandits problem with linear payoffs, there are N arms. At time t = 1, 2, ..., a context vector b i (t) ∈ R d , is revealed for every arm i. History Q t-1 = {a τ , r τ , b i (τ ), i = 1, ..., N, τ = 1, ..., t -1} where a τ denotes the arm played at time τ and that triggered reward r τ . Given b i (t), the reward for arm i at time t is generated from an (unknown) distribution with mean b i (t) µ, where µ ∈ R d is a fixed but unknown parameter. An algorithm for the contextual bandit problem needs to choose, at every time step t, an arm a t to play, using history Q t-1 and current contexts b i (t), i = 1, ..., N.

Let a * t denote the optimal arm at time t, i.e. a * t = argmax i b i (t) µ. And let ∆ i (t) be the difference between the mean rewards of the optimal arm and the arm i played at time t, i.e., ∆ i (t) = b a * t (t) µ -b i (t) µ. Then, the regret at time t is defined as regret(t) = ∆ at (t). The objective is to minimize the total regret R(T ) = T t=1 regret(t). The time horizon T is finite and known in our case. To model the active learning problem as a contextual bandit with linear payoffs we need to define the arms of the bandit, the rewards of the environment and the context of each arms.

Construction of the Arms. We cluster corpus points {x 1 , x 2 , ..., x n }. The resulted clusters c ⊂ U are considered as the arms of the bandit.

Context of the arms. We consider a context vector b i (t) that describes the the arms (the clusters), and contains the features that characterise the clusters.

Reward. A metric is used to measure the variation of the hypothesis learned by the model between two iterations. More the hypothesis learned by the model varies more is the received reward. We now define the function d(h t-1 , h t ) that we use to get the variation of the model. Let U 0 = {x 1 , ..., x n } = L t ∪ U t be the set of labelled and unlabelled training examples that we have. Then for each of the two real-valued hypotheses h t-1 (.), h t (.), we define the vectors H t-1 = (h t-1 (x 1 ), h t-1 (x 2 ), ..., h t-1 (x n )) and H t = (h t (x 1 ), h t (x 2 ), ..., h t (x n )), i.e. vectors of the real-valued predictions of h t-1 and h t .

d(h t-1 , h t ) = H t-1 .H t ||H t-1 ||.||H t || (1)
In Eq. 1, we compute the cosine similarity between the two vectors H t-1 and

H t . Thus d(h t-1 , h t ) ∈ [-1, +1
] is the cosine of the angle between H t-1 and H t , and we normalise the result in the interval [0, 1] using y(t) = 2.cos -1 (d(ht-1|ht))

π . Stationarity of the reward. We have observed that, more an area is sampled by the model less is the received reward (nonstationarity of the rewards). We have confirmed this common sense idea from an off-line evaluation (see Fig. 2). In order to circumvent this nonstationarity we assume that the reward function y(t) = r t • D(t), where D(t) is a decreasing function that follows the decreasing reward given by the environment and r t is the stationary reward. The process for obtaining the function D(t) is described in Section 4.

Contextual Bandit Algorithm. A Contextual bandits algorithm determines a cluster c ⊂ U t to be sampled at each time step t, based on the previous observation sequence Q t-1 = {c τ , r τ , b c (τ ), c = 1, ..., N, τ = 1, ..., t -1}, and its current context b c (t).

Active Thompson Sampling

Thompson sampling is understood in a Bayesian setting as follows. The set of past observations Q is made of triplets (c t , r t , b c (t)) and are modelled using a parametric likelihood function P r(r t |μ) depending on some parameters μ. Given some prior distribution P r(ũ) on these parameters, the posterior distribution of these parameters is given by the Bayes rule, P r(μ|r t ) ∝ P r(r t |μ)P r(μ).

From [START_REF] Agrawal | Thompson sampling for contextual bandits with linear payoffs[END_REF], we can say that the posterior distribution at time t + 1, P r(μ|r t ) ∝ P r(r t |μ)P r(μ) were given by a multivariate Gaussian distribution N (μ(t + 1), v 2 B(t + 1) -1 ), where B(t

) = I d + t-1 τ =1 b cτ (τ )b cτ (τ ) with d the size of the context vectors, v 2 ∈]0, 1]
is a constant fixed to 0.25 according to [START_REF] Chapelle | An empirical evaluation of thompson sampling[END_REF] andμ

= B(t) -1 ( t-1 τ =1 b cτ (τ )b cτ (τ )). Every step t consists of generating a d-dimensional sample μ from N (μ(t), v 2 B(t) -1 )

Experimental Evaluation

To conduct our evaluation, we have got from our company a corpus containing French utterances collected from a commercial spoken dialogue system. There are 7 765 utterances annotated by human experts. The unannotated part consists of 3 911 695 utterances. We use a corporate supervised algorithm (rule-based algorithm), being a part of a spoken language dialogue system. We simulate in the experiments an expert (oracle) on the unannotated corpus by using the rule-based algorithm which was designed using the 7 765 annotated utterances. In our experiments, the clustering algorithm (k-means in our case) uses the cosine similarity as a similarity metric between utterances. We have considered different features in the context vector of the clusters b c (t) = (M dis c , V dis c , |c|, plb c,t , M ixRate c,t ), where M dis c and V dis c are respectively the average distance between individuals in the cluster, and its variance. |c| gives the number of points in the cluster. plb c,t gives the proportion of labelled individuals in the cluster at time t and M ixRate c,t gives the ratio of the classes in the cluster at time t (the proportion of examples labelled in each class in the cluster). To obtain the decreasing function D(t), we assume D(t) = αe -βt , and we compute the parameters α and β of D(t) by sampling uniformly the different clusters and drawing the rewards in Fig. 1, then we lit D(t) on the reward curve. We obtain α = 0.61 and β = 0.12 We average the regret over 1000 times with a time horizon of 2000 sentences to label which correspond to our budget in term of labelling. To compute the regret, we have supposed that the optimal policy is given by the oracle. In addition to the random (baseline), we have compared our algorithm to the ones described in the related work (Sec. 2). QBC, US, and the different approaches that consider the bandit algorithms in the active learning as EXP4 used in [START_REF] Osugi | Balancing exploration and exploitation: a new algorithm for active machine learning[END_REF] and LCB used in [START_REF] Ganti | Building bridges: Viewing active learning from the multi-armed bandit lens[END_REF]. In Fig. 2, the horizontal axis represents the number of iterations and the vertical axis gives the cumulative regret (performance metric) which is the sum of the regrets from the first iteration to the current iteration. From the Fig. 2 we observe that over all strategies gives better result than a random selection. Neither QBC nor US gives a good results. This confirms that a pure exploitation is not efficient, and it confirms the need of the exr/exp tradeoff. While EXP4 algorithm gets a low cumulative regret, its overall performance is not as good as LCB and ATS. ATS and LCB indeed have the best cumulative regrets, ATS decreases the cumulative regret with 29% over the baseline and LCB, with 23%. The improvement comes from a dynamic exr/exp. These algorithms take full advantage of exploration from the beginning of the exploration rather than other strategies like uncertainty sampling or request by committee that need enough iteration to construct their models. Finally, as expected, ATS outperforms LCB, which is explained by the consideration of the context, and also that TS performs better exr/exp trade-off than UCB.

Conclusion

In this paper, we study the active learning problem from the side of contextual bandit and propose a new approach that adaptively balances exr/exp regarding the context of the cluster (arms). We have validated our work with data from real-world application and shown that the proposed algorithm offered promising results. This study yields to the conclusion that considering the contextual bandit model for the active learning significantly increases the results. Considering these results, we plan to study the theoretical regret of the proposed algorithm.

Algorithm 1 1 : 2 : 5 :

 1125 , and solving the problem argmax c⊂Ut∧|c|>0 b c (t) μ (select the cluster c that maximizes b c (t) μ. After that the algorithm selects randomly an individual x ∈ c, requests a labelling from the oracle O and observes reward y(t). The Active Thompson Sampling algorithm Require. B = I d set μ = 0 d , f = 0 d . Foreach t = 1, 2, ..., T do 3: Sample μ from the N (μ, v 2 B -1 ) distribution. 4: Select cluster ct = argmax c⊂U t ∧|c|>0 bc(t) μ xt = Random(ct). 6: Query O for label yt of xt 7: Observe y(t) and compute rt 8: B = B + bc t (t)bc t (t) T , f = f + bc t (t).rt else μ = B -1 f 9: End

Fig. 1 .

 1 Fig. 1. Reward function

Fig. 2 .

 2 Fig. 2. Cumulative Regret for Active learning algorithms


Agnès Festré
email: festre@gredeg.cnrs.fr

Criisea

GREDEG Pierre Garrouste
email: pierre.garrouste@gredeg.cnrs.fr

Somebody may scold you! A dictator experiment

Keywords: JEL Classification codes: C72, C91, D03 PsycINFO Classification codes: 3020, 3040 social psychology, game theory, communication, beliefs, observation, altruistic behavior 1 JEL Classification codes: C72, C91, D03 PsycINFO Classification codes: 3020, 3040 social psychology, game theory, communication, beliefs, observation, altruistic behavior 1. Introduction

In this contribution, we investigate the effects of observation-only and observation with feedback from a third-party in a one-shot dictator game (DG). In addition to a baseline condition (DG), a third-party anonymous subject was introduced who either silently observed or observed and got to give feedback by choosing one of seven messages consisting of varying degree of (dis)satisfaction. We found that observation coupled with feedback increased significantly dictators' propositions, while no significant effect is found for observation-only. We conclude that regard by others matters only if it linked to social factors such as communication. This complements the literature arguing that altruistic behavior is instrumental in serving other selfish (or non-purely altruistic) ends such as self-reputation or social approval. This experiment also contributes to the growing literature that aims at decreasing the artificiality of dictator game designs by increasing their practicability and external validity.

Experimental design and participants

Reporting of results

The 66 observations of the post-play intentions do not correspond to specific treatments but are linked with the observation with feedback ones. Accordingly we have 66 x 2 = 132 subjects for the control treatments, 66 x 3 x 2 = 198 for the observation only treatments and 66 x 3 x 2 = 198 for the observation with feedback treatments. That means that 528 subjects have been concerned by our experiments. F' corresponds to what the subjects would have been given if they would to play again. Accordingly the design is a between one but for those sessions.

We add t-tests in appendix C. The differences between treatments O and F and C and F are statistically significant. As for table 1, the percentages are adjusted (1 subject corresponds to an exact percentage of 1.51515151515152). 15 subjects gave 3 in the observation only treatment. The exact percentage is then 22.7272727272727. We have adjusted it to 22.73%.

As for the treatment C, thank you, you are right; the values were not correctly reported. Accordingly table 1 has been modified. Our results however do not changed because they are calculated with the right values. The fact that 33.33% of the subjects given half or more of their endowment is in line with [START_REF] Andreoni | Social image and the 50-50 norm: A theoretical and experimental analysis of audience effect[END_REF].

We agree with the fact that the averages of the gifts in the control and observation only treatments are equal up to the 6 th position after the decimal point has an almost nil probability to appear. It is however the case.

We changed the regression according to your advice.

Minor comments

P. 3/4: The reference to Mohlin and Johannesson (208) has been dropped since it is an example for the reduction of social distance as communication is introduced ex ante.

Footnote 7 (p. 10) now footnote 9 (p. 10) has been reworded for clarity. By 'very fast' formation, we mean that subjects are passive (they read on the computer that they are going to be matched with two others) and have not the time to realize (due to the speed of the flow information they are subject to) that have been grouped.

Figure 2/figure 3 has been standardized p. 12. The cross-reference has been revised. It refers to footnote 6 instead of footnote 1.

We removed a figure ( the histograms of the results) and an appendix Figure 5 that becomes figure 4 has been changed according to your advice.

We have changed the regression (we ran a tobit one) and put the SD in parentheses

The typo error (Johanensson instead of Johannesson) has been corrected

Highlights

we study the role of observation and sanction on individual behavior observation is provided by introducing a third-party in a dictator game sanction is provided by means of a written feedback from the receiver in a dictator game we show that observation alone has no effect on behavior observation with feedback increases the generosity of dictators *Highlights (for review)

As early as 1759 Adam Smith, in the Theory of Moral Sentiments, emphasized the social approbation and disapprobation dimensions of individual behavior: "we are pleased to think that we have rendered ourselves the natural objects of approbation (…) and we are mortified to reflect that we have just merited the blame of those we live with." [START_REF] Smith | Theory of moral sentiments[END_REF][START_REF] Smith | Theory of moral sentiments[END_REF] a key role in the process of approbation or disapprobation of the conduct of others, and in turn, our self-approbation or self-disapprobation1 .

A related idea emerges in social psychology in the work by [START_REF] Cooley | Human nature and the social order[END_REF] who coined the phrase 'looking-glass self', meaning that a person's self grows out of society's interpersonal interactions and the perceptions of others. The term refers to people shaping themselves based on other people's perceptions, which leads to a reinforcement of other people's perspectives on themselves. Recently, this notion was formalized by [START_REF] Bénabou | Intrinsic and extrinsic motivation[END_REF] among others, using a principal-agent framework, and analyzing the consequences of such a 'looking-glass self' effect in terms of costs2 . The motive of social approbation has also been proposed by economists who do not consider that emotions such as shame or guilt which play a highly complementary role in maintaining one's commitments, can be reduced to costs to be avoided. For Elster (1999, 281-3), shame is defined in terms of the 'action tendencies' it is like to produce, and involves a commitment to certain principles, standards, or norms. This echoes references in the psychological literature to anticipatory shame (see Tracy et al. 2007).

In a nutshell, anticipatory shame (in contrast to exposed shamed) is felt by the actor, in the absence of exposure, by imagining how he (or she) would feel were his (or her) normative transgression exposed. Our results are consistent with the idea that anticipatory shame plays a crucial role in shaping behaviour in a dictator game (DG) experiment.

In order to test for the role of social approval on altruistic behavior, we try to disentangle the intertwining influences of observation-only and observation coupled with ex post communication. We use a (one-shot) DG framework where the influence of social factors is manipulated via the introduction of a neutral third-party subject who 1) is incentivized independently of any of the randomly matched partners of the DG, and 2) either silently observes or observes and is required to give ex post written feedback to the giver.

The observation-only treatment can be related to several contributions on the effects of observation in experiments in economics and cognitive psychology. The seminal work of [START_REF] Hoffman | Preferences, property rights, and anonymity in bargaining games[END_REF] on dictator and ultimatum, testing the anonymity hypothesis by using a double-blind design, suggests that "other-regarding preferences may have an overwhelming social, what-do-other-know component" so that fairness may not be one's own preference but a derivative of the judgment of others (Hoffman et al. 1994, p. 371). Some recent work argues that selfish concerns such as guilt aversion (Charness andDufwenberg 2006, Battigalli and[START_REF] Charness | Promises & partnership[END_REF]) and self-image maintenance through self-signaling [START_REF] Bodner | The diagnostic value of actions in a self-signaling model[END_REF] or self-deception [START_REF] Dana | What you don't know won't hurt me: Costly (but quiet) exit in dictator games[END_REF][START_REF] Fischbacher | z-Tree: Zurich toolbox for ready-made economic experiments[END_REF], can help explain seemingly altruistic behavior.

Other theoretical contributions on social or interdependent preferences explore the suggestion that altruistic behavior can be promoted by reducing social distance3 [START_REF] Hoffman | Social distance and other-regarding behavior in dictator games[END_REF].

This literature offers several means to manipulate social distance: by making potential victims of selfishness more identifiable (cf. Schelling 1968 on the 'identifiable victim effect') through the use of photographs [START_REF] Burnham | Engineering altruism: A theoretical and experimental investigation of anonymity and gift giving[END_REF], by contrasting varying kinds (one-way/both ways, silent/non silent) of identification (Bohnet andFrey 2001a et 2001b), by using tacit visual or auditing clues (Haley and Fessler 2005[START_REF] Bateson | Cues of being watched enhance cooperation in a real-world setting[END_REF][START_REF] Nettle | The watching eyes effect in the Dictator Game: it's not how much you give, it's being seen to give something[END_REF] or simulating an audience effect by varying the probability of nature to play [START_REF] Andreoni | Social image and the 50-50 norm: A theoretical and experimental analysis of audience effect[END_REF].

Another strand in the related literature addresses the role of third-party informal punishment, via social approval or disapproval, for guiding behavior. Although indirectly linked to the literature on the disciplinary effect of third-party punishment in games [START_REF] Fehr | Third-party punishment and social norms[END_REF]), our experimental setting departs company from it since it involves 1) informal and non pecuniary punishment, and 2) communication (at least anticipated). From this standpoint, it is more in line with the economic and psychology literature which has established that informal sanctions such as communication, permitting the expression of disapproval, can favor prosocial behaviors. For instance, [START_REF] Ellingsen | Anticipated verbal feedbacks induce altruistic behavior[END_REF] provide a one-shot dictator experiment in which they compare a feedback treatment -i.e. a treatment where an anonymous verbal written message is sent by the recipient to the proposer after the pass, with a no feedback treatment, and show that anticipated rewards induce altruistic behavior. [START_REF] Xiao | Avoiding the sharp tongue: Anticipated written messages promote fair economic exchange[END_REF] find similar results although there are more restrictions in their experimental design on the amount to pass (e.g. the divider cannot take more than 90%) and on the content of the feedback written message. In a DG, [START_REF] Andreoni | The power of asking: How communication affects selfishness, empathy, and altruism[END_REF] show that when the receiver can speak the dictator is more generous compared to the case when only the dictator speaks. [START_REF] Xiao | Emotion expression in human punishment behavior[END_REF] find complementary results in ultimatum games.

They show that ex-post verbal written feedback messages by receivers significantly decrease the likelihood of rejecting unfair results. This suggests that verbal written communication is an expression of disapproval, which can be a non-costly substitute for monetary punishment.

In the same vein, [START_REF] Xiao | Justification and Cooperation, Working Paper[END_REF] shows that in a one-shot anonymous interaction, compared with the case when the behavior is simply observed by the audience, individuals are more likely to act on what they believe the other thinks they should do (a proxy for social approval), and therefore are more reluctant to violate social norms when they have to provide the audience with justification for their decisions. The explanation provided by the author is that justification makes social norms more salient to the subjects than observation by the audience only. Finally, [START_REF] Charness | Bare promises: An experiment[END_REF] find limited support for lying aversion and guilt aversion4 .

In order to test the respective role of observation and social disapproval, we provide a framework that permits us to disentangle those two effects and to clarify their underlying motivations. To our knowledge, this is the first study to isolate the impact of observation with ex post feedback from the impact of observation-only. Moreover, it is the only contribution to deal with feedback communication from a third-party5 . The paper is organized as follows. In section 2 we present the experimental design and procedures. Section 3 provides the results, which are discussed in section 4. The paper concludes in section 5.

Experimental design and procedures

In order to avoid both reputational effects due to repeated games and conditional reciprocity (i.e. reciprocal motivations based on future material payoffs), we conduct a one-shot anonymous DG experiment with and without ex post communication from a third-party which we call the observer. Regarding the role of observation-only, the motivation for our design is to test whether the introduction of a neutral observer randomly chosen among the subjects, has any effect in addition to the hypothetical effect of observation by the experimenter (see [START_REF] Levitt | What do laboratory experiments tell us about the real world[END_REF] on the scrutiny effect). A DG was chosen in preference to the more popular ultimatum game in order to avoid confounding altruism with risk aversion or false beliefs. Feedback from a non incentivized third-party is chosen over feedback by the recipient in order to avoid outcome-oriented punishment.

Ex-post communication takes the form of an anonymous, closed-form6 written message consisting of a list of 7 possible messages to be sent to the dictator by the third-party (the observer), ranked from 1 to 7 according to the degree of (dis)approval of the dictator's donation expressed by the observer. 'bastardo' (1) (bastard in Italian) is very harsh while 'bravissimo' (7), meaning 'this is very nice of you', is the kindest message. The item 'nessuno commento', or no comment, has the value 0 and corresponds to cases where the third-party chooses not to send any feedback. We experienced only 8 cases of no feedback. The screen seen by subjects appears as follows:

[Screen 1around here]

We ran three 'between' treatments in order to distinguish between the observation effect and the social disapproval effect. The first treatment (C) is a standard DG (control treatment). The second treatment (O) is a DG with an observer informed about the allocator's donation (observation-only treatment). The third treatment (F) allows the observer to send written feedback to the dictator (observation with feedback treatment). In the F treatment, the dictator is asked also how much he/she would keep were he/she to have the opportunity to play again.

That is, the allocators are informed about the content of the feedback message from the observer attached to their group (e.g., in the screen below 'taccagno' means 'stingy') and then decide how much of their total endowment they would expend if they had the opportunity to replay. The response to this question provides some information that allows us to evaluate what we call 'ex post feedback effects' (see below).

[screen 2around here]

In order to test for the existence of a socialization effect 7 , we ran two variants of each treatment with different subjects. We hypothesized that the order in which subjects are randomly matched in pairs (for the control treatment) or groups of three (for treatments O and F) and attributed a role, might matter. The intuition is that grouping subjects first might decrease the social distance, and therefore affect the behavior of the allocators. We ran variants of each treatments in which subjects first were randomly matched in pairs or groups of three, and then attributed a role (proposer, recipient or possibly observer); and vice versa, that is subjects were first attributed a role and then matched into pairs or groups of three. We labeled these experiments C1, O1, and F1 respectively for the first variant, and C2, O2, and F2 for the second variant. (See instructions in appendix A.)

The experiment was conducted in October 2011 at the "Centro Sperimentale A Roma

Est" (CESARE) on one of the campus locations of the LUISS Guido Carli University in 7 The idea that grouping subjects in a DG might influence donations by dictators was introduced by [START_REF] Cason | A laboratory study of group polarisation in the team dictator game[END_REF]. They compared an individual DG where the dictator decides to transfer an amount of y to the recipient, with a team DG where two subjects dictate a donation of 2y to be transferred. The results showed that there was group polarization in the latter context and the data indicated that team choices tended to be dominated by the more other-regarding member. However, other studies of DGs show contrasting results, i.e., an egoistic shift [START_REF] Luhan | Group Polarization in the Team Dictator Game Reconsidered[END_REF], or no significant difference between single and group decisions [START_REF] Franzen | Giving according to preferences: Decision-making in the group dictator game[END_REF].

Rome. The subjects were recruited by e-mail using ORSEE (Online Recruitment System for Economic Experiments, devised by Ben Greiner at the University of New South Wales in 1994). They were randomly allocated among the six treatments and we ran 18 sessions (6 each for the control, observation-only, and observation with feedback treatments). A total of 528 subjects participated in the experiment yielding 198 pairs or trios of observations (66 pairs of observations for the control treatments, 66 trios of observations for the observationonly treatments, and 66 trios of observations for the observation with feedback treatments).

We also obtained 66 observations8 linked to the observation with feedback treatments that correspond to the post-play intentions of donations of allocators after receiving feedback from the observer.

Subjects in pairs or groups of three were anonymous to one another, and the decision of one subject was observable only by the receiver and the observer of his or her specific group of three but not by the other subjects. The instructions (translated from Italian) of the treatments C1, O1 and F2 are given in appendix A (remember that treatments indexed by 1 differ from those indexed by 2 only in relation to the order in which subjects were matched into pairs or groups of three and attributed the roles of proposer, recipient, or observer). At the end of the experiment, participants were asked to complete an anonymous questionnaire (professional status, discipline, age, gender). Finally, subjects were called one by one by the experimenter in order to be paid privately before leaving the lab.

The amount of the show-up was 3 ECU (Experimental Currency Unit), and the endowment of allocators 6 ECU. The observer was given a fixed amount of 6 ECU. The final gains were calculated in euros at a conversion rate of 1 ECU equals 1 euro. The minimum and maximum gains were respectively 3 euros and 9 euros. The average time of the experiments was 15 minutes.

Results

Dictators pass an average 1.98 for all treatments taken together (33% of the dictators' endowment). In treatment C, the mean is 1.78 with a standard deviation (SD) equal to 1.38, in treatment O it is 1.78 (SD = 1.45) and in treatment F it is 2.37 (SD = 1.65). Intentions of allocations by dictator after receiving the feedback message from the observer (F') amounts to 2.22 (SD=1.89) (see figure 1). The donations go from 29.7% of the endowment in control and silent observation, to 39.5% in the treatment with feedback, i.e., that is, donations increase by 33.1% from the two first treatments to the feedback treatment.

[Figure 1 around here]

Table 1 presents the gifts as percentages for the different treatments. Descriptive statistics are provided in appendix B.

[Table 1 around here]

A Mann-Whitney test shows that there is no significant socialization effect. Recall that we defined this effect in relation to the order in which subjects were matched into groups of two or three and attributed roles. The absence of a significant socialization effect therefore, means that the order of the sequence of grouping and attributing roles does not modify the behavior of allocators substantially 9 . However, the Mann-Whitney test reveals that, if there is 9 One possible explanation for the absence of a socialization effect might be that the random procedure we use to form groups in the experiment is 'very fast', not self-involving and does not seem to catch the subject's attention. More precisely, subjects are passive (they hear and read on the computer that they are going to be a significant difference between the amounts given by the proposers in treatments F and O, this difference is not significant when we compare treatments C and O (see appendix C).

Those results are corroborated by the t-tests (see appendix C).

Since there is no significant socialization effect, data are aggregated over treatments O1 and O2, over C1 and C2, as well as over F1 and F2, to give treatments C, O and F respectively.

All the remaining statistics are derived from these aggregated treatments.

Interestingly, as Figure 2 illustrates, the percentages of donations in treatment F show a different distribution from those in treatments O and C. That is, the percentages of donations under the 50:50 norm (i.e., when donations are strictly less than 3 or up to 2) are systematically lower in the F treatment than in the C and O treatments, while the percentages of donations over the 50:50 norm (i.e., when donations are strictly more than 3 or 4 and over) are systematically higher in the F treatment than in the C and O treatments10 . An unusual result is that the percentage of subjects giving more than half of their endowment is high (9% for treatment C and 18% for treatment F). This result is difficult to explain. As for treatment F, concerning those choices the messages received have an average value of 4.6. Moreover the correlation between the value of the messages received and the amount of money the proposers would have sent if they would have to play again equals 0.52, although the correlation between the value of the messages and the amount sent during the game is -0.39 and the correlation between the amount sent during the game and the amount of money the proposers would have sent if they would have to play again is -0.28. Accordingly a possible explanation is that people giving more than half of their endowment want, ex ante to test the fairness of the received message and because it is not the case they consequently reduce the matched with one or two other subjects) and have not the time to realize (due to the speed of the flow information they are subjected to) that they have been grouped.

amount they would have given if they would have played another time. This explanation is however only a rough one and we have to admit that facts are sometimes rebel."

[Figure 2 around here]

For the observation with feedback treatment (F), the mean of the amount that the proposer would have retained had he/she the opportunity to replay, is significantly higher (3.77) than the amount he/she effectively retained during the game (3.22). As for the message, only 36.36 percent of the messages sent were 'soft' (i.e., ranked from 4 up to 7).

As hinted in footnote 6, we constructed a proxy variable in order to capture the intensity of evaluation of observers, defined as the difference between the degree of (dis)approval of the message sent by the observer (scaled from 1 to 7) and the effective amount of the donation (scaled from 0 to 6). Therefore, the intensity of evaluation of observers thus defined can take values from -5 (a seemingly excessive level of disapproval) to 7 (a seemingly excessive level of approval) (see appendix D). Following this logic, we consider that the value of a 'balanced' or 'objective' intensity of evaluation corresponds to1.

The percentages of the different values taken by the intensity of evaluation are shown in figure 3.

[Figure 3 around here]

The student t-test 11 shows that the mean of the distribution of the evaluation intensities (EI) is significant and equals 1 12 . This means that the evaluations by observers are balanced, i.e., the content of the feedback messages is close to the 'objective' one.

Te 11 We used the Skewness-Kurtosis test to control for whether the distribution of intensities of evaluation is normal (Prob> chi2 = .81).

12 The student t-test gives: Pr(|T|>|t|) = 0.019 confidence interval: [-.3587919, .8739434].

The amount the proposer would have given, had he/she had the opportunity to play again (moneygiven2) and the amount that he/she effectively gave during the game (moneygiven1) are significantly different (Pr(|T > |t|) = 0.0230)

We run a Tobit regression (to account for left-censored dependent variable) of the amount the proposer would have given had he/she had the opportunity to play again (moneygiven2) on an interaction variable (the amount effectively given (moneygiven1) times the message), and we obtain a significant coefficient (see appendix E) 13 .

This means that the interaction between the feedbacks combined with the amounts the proposer gave during the game, impact on his (or her) virtual choices. Figure 4 illustrates the proposers' percentages as a function of the amount they would have given if they had the opportunity to play again, and the percentages of the various degrees of disapproval of the messages the proposers received. It shows that if the messages received are soft (from 4 to 6), the amount of money the proposers would have given had they had the opportunity to play again is low. If the messages are hard (from 1 to 3) the opposite holds: the amount of money the proposers would have given if they had had the opportunity to play again is high. Thus there is a discontinuity between messages 3 and 4.

[Figure 4 around here]

This result is interesting and rather intuitive although it should be taken with caution since it relates to cost-free intentions which could be suspected of hypocrisy. However, this result suggests that the feedback has an ex post effect since proposers intend to give more if they receive a harsh message. This result needs to be mitigated. Figure 4 shows that dictators give more if the message is strong and give less if the message is soft.

Discussion

13 This idea was suggested by one of the journal reviewers.

Our experiments allow discussions of five theoretical elements: the first concerns the influence of social distance on individual behavior; the second is linked to the impact of observation on behaviors; the third is due to the possible effect of anticipated sanctions on individual decisions; the fourth concerns the evaluation intensity of the message; and the fifth refers to the possible ex post effect of a sanction.

In relation to the first issue, we argued in the introduction that altruistic behavior may be promoted by decreasing social distance, as suggested by [START_REF] Hoffman | Social distance and other-regarding behavior in dictator games[END_REF]. Our results

do not seem to corroborate this hypothesis since we found no differences in terms of dictators' decisions of gift if they are first matched with others and then ascribed the role of dictator or vice versa. The apparent absence of any socialization effect may be explained by a limited attention effect. If we assume that people pay attention to information conditional on its salience, our design could be criticized for giving insufficient salience to the difference in the order of the sequence presented above. This explanation could be complemented by another one. The degree of social distance is not a 0-1 variable. In our experiment, the difference in social distance between the two versions of a given treatment appears very small. Thus, it is not surprising that such a small perturbation of social distance has no discernible effect on the amount retained.

In relation to the second issue, we also argue in the introduction that social factors such as observation, underpin social interactions so that would-be altruistic behavior may be the efficient response to the perceived situation. The results of our experiments do not support this claim since we found no significant effect of observation-only. But the absence of significance in the differences between the means of donations in treatments C and O may be due to the fact that we ran one-shot experiments. However, it corroborates the idea developed by [START_REF] Levitt | What do laboratory experiments tell us about the real world[END_REF] that the presence of an experimenter impacts on the subjects'

behavior. The presence of a silent observer replicates the presence of an experimenter, and therefore we should not expect any additional effect of observation by a third-party. Another interpretation might be that at least some of the allocators have some internalized social norms which induce them to transfer an amount significantly different from zero. This may explain why the presence of an observer does not have a significant impact on donations.

Moreover, since subjects have no information concerning the status of the observer, it is impossible for them to infer his/her opinion or his/her preferences in terms of fairness for instance. Taking account of this uncertainty might explain why observation-only is not sufficient to have an impact on behavior.

From this standpoint, and with reference to the third theoretical element listed above, introducing the possibilitywhich is common knowledgeof a feedback removes the uncertainty and makes the role of the observer explicit or more salient, and therefore effective. This suggests that in our experiment, observation by others matters only if it is complemented by the likelihood of disapproval. This is in line with findings in cognitive psychology that perception can be even more powerful than 'reality' insofar as people use cues or heuristics to evaluate the likelihood that their actions are observable, and possibly disapproved. The existence of feedback therefore is a deciding factor in order for the proposers to know particulars about the third party (or to infer how this third-party thinks they are or should be), not directly concerned by the proposer's decision of donation as is the case in [START_REF] Ellingsen | Anticipated verbal feedbacks induce altruistic behavior[END_REF]. In other words, the anticipated feedback is a means of engineering altruism (see [START_REF] Burnham | Engineering altruism: A theoretical and experimental investigation of anonymity and gift giving[END_REF]. The fact that proposers consequently adjust their behavior should come as no surprise. The feedback effect is economically relevant since it indicates that individuals are sensitive to anticipated possible sanctions. This suggests that feedbacks are non-costly substitutes for effective pecuniary sanctions.

Fourth, the results obtained with the proxy variable for the intensity of evaluation of observers plead for an interpretation related to reciprocity. Feedbacks are essentially 'balanced', i.e., they are consistent with the assumption that the degree of approval is proportional to the generosity of the proposer, so that observers reciprocate proposers' donations 'fairly'.

Fifth, the fact that the degree of disapproval of messages has an impact on the amount the dictator would have given had he/she had the opportunity to replay, is interesting due to the positive relationships between the two14 . In other words, the softer the message (from level 4 to level 7), the less the amount the dictator would have given if he had played one more time. Conversely, the harsher the message (from leve1 1 to level 3), the higher would be this amount. This result supports an interpretation of the feedback acting as a slightly complex disciplinary device.

Conclusion

This contribution which complements [START_REF] Ellingsen | Anticipated verbal feedbacks induce altruistic behavior[END_REF] by introducing a third-party feedback, could be extended by new experiments: where the proposer plays a real (as compared to a virtual) second game after receiving feedback, which would corroborate our results concerning the amount retained by the proposer after the feedback has been received;

or where a treatment is compared with the feedback from the recipient (as in [START_REF] Ellingsen | Anticipated verbal feedbacks induce altruistic behavior[END_REF]) and a treatment with feedback from a third party (our own contribution).

Proxy of the evaluation intensity

FigureFigure 1 :

 1 Figure

Figure 2 :

 2 Figure 2: Percentages of donations in the control treatment (gift C), in the observation-

Figure 3 :

 3 Figure 3: percentages of the evaluation intensity of observers

  , p. 116). Moreover, he pointed out that this dimension is intrinsically intertwined with a related self-image notion, i.e., self-approbation or self-esteem: "[w]e endeavour to examine our own conduct as we imagine any other fair and impartial spectator would examine it. If, upon placing ourselves into his [the impartial spectator's] situation, we thoroughly enter into all the

passions and motives which influenced it, we approve of it, by sympathy with the approbation of this supposed equitable judge. If otherwise, we enter into his disapprobation, and condemn it." (Ibid, p. 110). Smith also emphasized that conscious feelings such as praise or blame, play

Table 1 :

 1 Gifts in percentages in the treatments (with the Mann-Whitney tests of significance of the differences between the gifts)

		2.5
		2
		1.5
		1
		.5
		0
			mean of gift C	mean of gift O
			mean of gift F	mean of gift F'
	Gifts	C1	C2	O1	O2	F1	F2
	0	21.21	21.21	24.24	18.18	9.09	18.18
	1	30.30	24.24	24.24	30.30	15.15	9.09
	2	18.18	18.18	21.21	18.18	30.30	33.33
	3	24.24	24.24	24.24	21.21	21.21	27.27
	4	6.06	3.03	3.03	3.03	9.09	3.03
	5	0	3.03	0	9.09	6.06	3.03
	6	0	6.06	3.03	0	9.09	6.06
	Mann-Whitney test
	Prob>|z|	0.49		0.66		0.78
	Gifts	C		O		F
	0	21.21		21.21		15.15
	1	27.27		27.27		12.12
	2	18.18		19.70		30.30
	3	24.24		22.73		24.24
	4	4.55		3.03		6.06
	5	1.52		4.55		4.55
	6	3.03		1.52		7.58
	Mann-Whitney test
	Prob>|z|		0.888		0.037

We agree with one reviewer that the impartial spectator is a concept that is related more to moral norms of the kind discussed by Immanuel Kant (moral imperative) than to social norms such as fairness, which are considered in our experiment.

This approach is in line with[START_REF] Becker | Spouses and Beggars: Love and Sympathy[END_REF].

Social distance need to be distinguished from anonymity.Hoffman et al. (1996, p. 654) propose that social distance means a "sense of coupling between dictator and his or her counterpart." In contrast, anonymity of subjects circumvents that experimenters and/other participants find something out about the subject's decision.[START_REF] Koch | Giving in Dictator Games: Regard for Others or Regard by Others[END_REF] try to disentangle the effect of regard for others from the effect of regard by others.

They add to[START_REF] Charness | Promises & partnership[END_REF] a 'bare promises' treatment where the second player, before the game, can send a message to the first, either 'I promise to choose Roll' (playing the die) or a blank message.

In contrast to[START_REF] Ellingsen | Anticipated verbal feedbacks induce altruistic behavior[END_REF], we avoid a possible reciprocal effect on the receiver's feedback just because the third party is not directly concerned by the dictator's donation.

We chose closed-form messages in particular because we wanted to create a proxy for the evaluation intensity of observers (see below). In order to reduce the dimensional problem, we selected messages whose intervals were dimensionally similar to 1 ECU. The value of 'bravissimo' is comparable to

ECU.

These observations do not correspond to specific treatments but are linked to the observation with feedback treatment.

42% of the offers are equal at least to 3 in the observation with feedback treatment (F) while they respectively reach 35% and 32% in the control (C) and observation-only (O) treatments.

We stress again that this interpretation should be taken with care because of the lack of monetary incentives linked to this part of the game.

Acknowledgments

We offer warm thanks to Jean Beuve, Anne Corcos, Werner Güth, Nicolas Jacquemet, Agnès Labrousse, Marie-Claire Villeval, and Erte Xiao and participants to the PPE seminar at George Mason University, the PPP seminar at University Paris 1, and the ASFEE meeting in La Martinique for helpful suggestions and comments. We thank the University LUISS Guido Carli for allowing us to run the experiments at CESARE (Centro di Economia Sperimentale A Roma Est). We also thank Daniela di Cagno and Francesco Farina for helping us to find volunteers from LUISS students, to run the experiments. Finally, we thank Andrea Lombardo who wrote the program for the experiments using Z-tree software (Fischbacher, 2007). This research was funded by the ANR (Agence Nationale de la Recherche).

Appendix A

Instructions are translated from Italian, the language in which they were delivered.

They were posted on the subjects' screens and read out aloud by a randomly selected subject.

Instructions of treatment C1 [i.e., the control treatment where groups are formed first and then roles are attributed using random procedures]

Welcome. You have agreed to participate to an experiment at the end of which you will receive at least a show-up fee of 3 euro. You may get more than this depending on your performance during the experiment. The total amount gained will be given to you at the end of the experiment. At the end of the experiment, before being paid you will be asked to complete an anonymous questionnaire (level of studies, gender, etc.).

Determining the pairs

During the experiment you will be matched randomly in groups of two persons. You will not know the identity of the other member of your pair, either during or after the experiment.

Determining roles.

You will be randomly assigned to the role of proposer (A) or receiver (B). Each proposer has an amount of 6 euro and has the possibility to transfer part of this amount to the receiver with whom he/she is matched. (For example, if A decides to transfer 2 euro to B, he or she types the amount in euro that he or she has decided to keep, namely, 4 euro).

Instructions for treatment O1 [i.e., the treatment with observation alone where first groups are formed and then roles are attributed using random procedures] Welcome. You have agreed to participate in an experiment at the end of which you will receive a show-up fee of 3 euro. You may earn more depending on your performance during the experiment. The total amount of your gain will be given to you at the end of the experiment. At the end of the experiment, before being paid, you will be asked to complete an anonymous questionnaire (level of studies, gender, etc.).

Determining the groups of three

During the experiment you will be matched randomly in groups of three persons. You will not know the identity of the members of your group either during or after the experiment.

Determining roles.

You will be randomly assigned the role of proposer (A), receiver (B), or observer (C). Each proposer has an amount of 6 euro and the possibility to transfer part of this amount to the receiver with whom he/she is matched. C receives a fixed amount of 3 euro in addition to the show-up fee. (For example, if A decides to transfer 2 euro to B, he/she types in the amount in euro he or she has decided to keep, namely, 4 euro). The observer of his or her group is immediately notified of the amount.

Instructions for treatment F2 [i.e., the treatment with observation and feedback where roles are attributed and then the groups formed using random procedures] Welcome. You have agreed to participate in an experiment at the end of which you will receive a show-up fee of 3 euro. You may earn more depending on your performance during the experiment. The total amount of your gains will be given to you at the end of the experiment. At the end of the experiment, before being paid you will be asked to complete an anonymous questionnaire (level of studies, gender, etc.).

Determining roles

During this experiment, you will be randomly assigned the role of proposer (A), receiver (B), or observer (C). You will not know the roles of the members of group to which you belong either during or after the experiment. Each proposer has an amount of 6 euro and has the possibility to transfer part of this amount to the receiver to whom he/she is matched. C knows what amount is transferred by A to B, and has the possibility to send a written message to A to offer his/her opinion concerning the amount transferred. C receives a fixed amount of 3 euro in addition to the show-up fee.

A is then asked to decide the amount he/she would like to keep were he/she given the opportunity to replay.

Determining the groups of three

You will now be matched randomly in a group of three persons consisting of one proposer (A), one receiver (B), and one observer (C).

Each proposer (A) decides how much he/she is willing to transfer to the receiver (B) in his or her group. (For example, if A decides to transfer 2 euro to B, he/she types the amount in euro he/she has decided to keep, namely, 4 euro). The observer (C) in his or her group is immediately notified of the amount. C has the possibility to send a written message to A to offer his/her opinion concerning the amount transferred. C receives a fixed amount of 3 euro in addition to the show-up fee.

A is then asked to decide the amount he/she would like to keep were he/she given the opportunity to replay.

Appendix B -