Vande Corentin

Samuel Kerckove

Pascal Martin

Jason Gend

Julien M Rentfrow

Vincent Hendrickx

Blondel

Corentin Vande Kerckhove

Samuel Martin
email: samuel.martin@univ-lorraine.fr

Pascal Gend

Peter J Rentfrow

Julien M Hendrickx

Vincent D Blondel

Modelling influence and opinion evolution in online collective behaviour

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Many individual judgments are mediated by observing others' judgments. This is true for buying products, voting for a political party or choosing to donate blood. This is particularly noticeable on the online world. The availability of online data has lead to a recent surge in trying to understand how online social influence impact human behaviour. Some in vivo large scale online experiments were devoted to understand how information and behaviours spread in online social networks [START_REF] Centola | The spread of behavior in an online social network experiment[END_REF], others focused on determining which sociological attributes such as gender or age were involved in social influence processes [START_REF] Aral | Identifying influential and susceptible members of social networks[END_REF].

Although decision outcomes are often tied to an objective best choice, outcomes can hardly be fully inferred from this supposedly best choice. For instance, predicting the popularity of songs in a cultural market requires more than just knowing the actual song quality [START_REF] Salganik | Experimental study of inequality and unpredictability in an artificial cultural market[END_REF]. The decision outcome is rather determined by the social influence process at work [START_REF] Lorenz | How social influence can undermine the wisdom of crowd effect[END_REF]. Hence, there is a need for opinion dynamics models with a predictive power.

Along these lines and complementarily to the in vivo experiments, other recent studies used online in vitro experiment to identify the micro-level mechanisms susceptible to explain the way social influence impacts human decision making [START_REF] Moussaïd | Social influence and the collective dynamics of opinion formation[END_REF][START_REF] Mavrodiev | Quantifying the effects of social influence[END_REF]. These recent 1 arXiv:1511.02647v2 [cs.SI] 14 Dec 2015 online in vitro studies have lead to posit that the so-called linear consensus model may be appropriate to describe the way individuals revise their judgment when exposed to judgments of others. To date, the predictive power of such a mechanism remains to be assessed.

Trying to describe how individuals revise their judgment when subject to social influence has a long history in the psychological and social sciences. The consensus model used in this article draws from this line of work. These works were originially developped to better understand small group decision making. This occurs for instance when a jury in civil trials has to decide the amount of compensation awarded to plaintiffs [START_REF] Hastie | Inside the jury[END_REF][START_REF] Horowitz | Effects of trial complexity on decision making[END_REF][START_REF] Hinsz | Assimilation to Anchors for Damage Awards in a Mock Civil Trial1[END_REF]. Various types of tasks have been explored by researchers. These includes the forecasts of future events e.g., predicting market sales based on previous prices and other cues [START_REF] Fischer | Combining forecasts: What information do judges need to outperform the simple average?[END_REF][START_REF] Harvey | Using advice and assessing its quality. Organizational behavior and human decision processes[END_REF], the price of products [START_REF] Schrah | No decision-maker is an Island: integrating expert advice with information acquisition[END_REF][START_REF] Sniezek | Improving judgement with prepaid expert advice[END_REF], the probability of event occurence [START_REF] Budescu | Confidence in aggregation of expert opinions[END_REF][START_REF] Budescu | The effects of asymmetry among advisors on the aggregation of their opinions[END_REF], such as the number of future cattle deaths [START_REF] Harvey | Taking advice: Accepting help, improving judgment, and sharing responsibility[END_REF], or regional temperatures [START_REF] Harries | Combining advice: The weight of a dissenting opinion in the consensus[END_REF]. The central ingredient entering in models of judgment revision is the weight which individuals put on the judgments of others, termed influenceability in the present article. This quantity is also known as the advice taking weight [START_REF] Sniezek | Improving judgement with prepaid expert advice[END_REF][START_REF] Harvey | Taking advice: Accepting help, improving judgment, and sharing responsibility[END_REF] or the weight of advice [START_REF] Yaniv | The benefit of additional opinions[END_REF][START_REF] Yaniv | Receiving other people's advice: Influence and benefit[END_REF][START_REF]Do we listen to advice just because we paid for it? The impact of advice cost on its use[END_REF]. It is represented by a number taking 0 value when the individual is not influenced and 1 when they entirely forget their own opinion to adopt the one from other individuals in the group. It has been observed that in a vaste majority of the cases, the final judgment falls between the initial one and the ones from the rest of the group. Said otherwise, the influenceability lies between 0 and 1. This has been shown to sensibly improve the accuracy of decisions [START_REF] Yaniv | Using advice from multiple sources to revise and improve judgments[END_REF]. A 20% improvment has been found in an experiment when individuals considered the opinion of another person only [START_REF] Yaniv | The benefit of additional opinions[END_REF]. However, individuals do not weight themselves and others equally. They rather overweight their own opinions [START_REF] Bonaccio | Advice taking and decision-making: An integrative literature review, and implications for the organizational sciences[END_REF]. This has been coined egocentric discounting [START_REF] Yaniv | Advice taking in decision making: Egocentric discounting and reputation formation. Organizational behavior and human decision processes[END_REF]. Many factors affect influenceability. These include the perceived expertize of the advisor [START_REF] Harvey | Taking advice: Accepting help, improving judgment, and sharing responsibility[END_REF][START_REF] Soll | Strategies for revising judgment: How (and how well) people use others' opinions[END_REF] which may result from age, education, life experience [START_REF] Feng | Predicting receptiveness to advice: Characteristics of the problem, the advice-giver, and the recipient[END_REF], the difficulty of the task [START_REF]Do we listen to advice just because we paid for it? The impact of advice cost on its use[END_REF], whether the individual feels powerful [START_REF] See | The detrimental effects of power on confidence, advice taking, and accuracy[END_REF] or angry [START_REF] Gino | Blinded by anger or feeling the love: how emotions influence advice taking[END_REF], the size of the group [START_REF] Mannes | Are we wise about the wisdom of crowds? The use of group judgments in belief revision[END_REF], among others. A sensitivity analysis has been carried out to determine which factors most affect advice taking [START_REF] Azen | The dominance analysis approach for comparing predictors in multiple regression[END_REF].

This line of work has focused on determining the factors impacting influenceability. None has yet answered whether judgment revision models could be used to predict future decisions. Instead, the models were validated on the data which served to calibrate the models themselves. This pitfall tends to favor more complex models over more simple ones and may result in overfitting. The model would then be unable to predict judgment revision from a new dataset. One reason for this literature gap could be the lack of access to large judgment revision database at the time, now made more readily available via online in vitro experiments. The predictability assessment is a necessary step to grow confidence in our understanding and in turn use this mechanism as a building block to design efficient online social systems. Revising judgments after being exposed to others' judgments takes an important role in many online social systems such as recommendation system [START_REF] Pope | Reacting to rankings: evidence from "America's Best Hospitals[END_REF][START_REF] Dellarocas | The digitization of word of mouth: Promise and challenges of online feedback mechanisms[END_REF] or viral marketing campaign [START_REF] Bessi | Science vs Conspiracy: collective narratives in the age of misinformation[END_REF] among others. Unlike previous research, the present work provides an assessment of the model predictive power through crossvalidation of the proposed judgment revision model.

The prediction accuracy of a model is limited to the extend the judgment revision process is a deterministic process. However, there are theoretical [START_REF] Steyvers | Probabilistic inference in human semantic memory[END_REF][START_REF] Kersten | Bayesian models of object perception[END_REF][START_REF] Ma | Bayesian inference with probabilistic population codes[END_REF] and empirical [START_REF] Vul | Measuring the crowd within probabilistic representations within individuals[END_REF] evidence showing that the opinion individuals display is a sample of an internal probabilistic distribution. For instance, Vul and Pashler [START_REF] Vul | Measuring the crowd within probabilistic representations within individuals[END_REF] showed that when participants were asked to provide their opinion twice with some delay in between, participants provided two different answers. Following these results, the present article details a new methodology to estimate the unpredictability level of the judgment revision mechanism. This quantifies the highest prediction accuracy one can expect.

The results presented in this article were derived using in vitro online experiments, where each participant repeated several times estimation tasks in very similar conditions. These repeated experiments yielded to two complementary sets of results. First, it is shown that, in presence of social influence, the way individuals revise their judgment can be modeled using a quantitative model. Unlike the previously discussed studies, the gathered data allow assessing the predictive power of the model. The model casts individuals' behaviours according to their influenceability, the factor quantifying to what extent one takes external opinions into account. Secondly, a measure of intrinsic unpredictability in judgment revision is provided. Estimating the intrinsic unpredictability provides a limit beyond which no one can expect to improve predictions. This last result was made possible through a specific in vitro control experiment. Although models of opinion dynamics have been widely studied for decades by sociologist [START_REF] French | A formal theory of social power[END_REF] from a theoretical standpoint, to the best of our knowledge, it is the first time the predictive power of a quantitative model of opinion dynamics is tested against a real dataset.

Results

To quantify opinion dynamics subject to social influence, we carried out online experiments in which participants had to estimate some quantities while receiving information regarding opinions from other participants. In a first round, a participant expresses their opinion x i corresponding to their estimation related to the task. In the two subsequent rounds, the participant is exposed to a set of opinions x j of other participants who performed the same task independently, and gets to update their own opinion. The objective of the study is to model and predict how an individual revises their judgment when exposed to other opinions. Two types of games were designed : the gauging game, in which the participants evaluated color proportions and the counting game, where the task required to guess amounts of items displayed in a picture. Participants to this online crowdsourced study were involved in 3-round judgment revision games. Judgment revision is modeled using a time-varying influenceability consensus model. In mathematical terms, x i (r) denotes the opinion of individual i at round r and its evolution is described as

x i (r + 1) = x i (r) + α i (r) • (x(r) -x i (r)) (1)
where r = 1, 2 and where x(r) is the mean opinion of the group at round r (see Opinion revision model section in Material and Methods for details). This model is based on the influenceability α i (r) of participants, a factor representing to what extent a participant incorporates external judgments.

Influenceability of participants

The influenceability is described for each participant by two parameters : α i (1), the influenceability after first social influence and α i (2), the influenceability after second social influence. The distribution of couples (α i (1), α i (2)) were obtained by fitting model (1) to the whole dataset via mean square minimization for each type of games independently. The marginal distributions of (α i (1), α i (2)) are shown in Fig. 1. Most values fall within interval [0, 1], meaning that the next judgment falls between one's initial judgment and the group judgment mean. Such a positive influenceability has been shown to improve judgment accuracy [START_REF] Yaniv | Using advice from multiple sources to revise and improve judgments[END_REF] (see also the Practical Implications of the Model section). Most individuals overweight their own opinion compared to the mean opinion to revise their judgment with α i (r) < 0.5. This fact is in accordance with the related literature on the subject [START_REF] Bonaccio | Advice taking and decision-making: An integrative literature review, and implications for the organizational sciences[END_REF]. The average prediction error given in terms of RMSE is provided for each bin of the distributions in Fig. 1 through a color scale. This information shows that participants with a small but non-negative influenceability are the ones leading to the best predictions. The distribution of influenceabilities of the population is subject to evolution over time. A two-sample Kolmogorov-Smirnov test rejects equality of distributions between α i (1) and α i (2) for both types of games (p-val < 10 -6 ). A contraction toward 0 occurs with a median going from 0.34 for α i (1) to 0.18 for α i (2) in the counting game and 0.32 to 0.20 in the gauging game. In other words, the participants continue to be influenced after round 2 but this influence is lightened. Fig. 1-C shows the discrepancy between the cumulated distribution functions over rounds.

(A) Gauging

Model Performance

Prediction scenarios

When one wishes to predict how a participant revises their opinion in a decision making process, the level of prediction accuracy will highly depend on data availability. More prior knowledge on the participant should improve the predictions. When little prior information is known about the participant, the influenceability derived from it will be unreliable and may lead to poor predictions. In this case, it may be more efficient to proceed to a classification procedure provided that data from other participants are available. These hypotheses are tested by computing the prediction accuracy in several situations reflecting data availability scenarios.

In the worst case scenario, no data is available on the participant and the judgment revision mechanism is assumed to be unknown. In this case, predicting constant opinions over time is the only option. This corresponds to the null model against which the consensus model ( 1) is compared.

In a second scenario, prior data from the same participant is available. The consensus model can then be fitted to the data (individual influenceability method). Data ranging from 1 to 15 prior instances of the judgment process are respectively used to learn how the participant revises their opinion. Predictions are assessed in each of these cases to test how the predictions are impacted by the amount of prior data available. In a final scenario, besides having access to prior data from the participant, it is assumed that a large body of participants took part in a comparable judgment making process. These additional data are expected to reveal the most common behaviours in the population and enable to derive typical influenceabilities (population influenceability methods).

If available, the prior information on the participant can then serve to determine the best suited influenceability among the typical ones computed in the previous step. If no prior data is available on the targeted participant, only one typical influenceability value should be computed from the population and used as predictor.

Prediction accuracy

We assess the predictions when the number of training data is reduced, accounting for realistic settings where prior knowledge on individuals is scarce. Individual parameter estimation via the individual influenceability method is compared to population influenceability method. This last method uses one or two (α(1), α(2)) couples of values derived on independent experiments. Fig. 2 presents the RM SE (normalized to the range 0-100) obtained on validation sets for final round predictions using the model [START_REF] Centola | The spread of behavior in an online social network experiment[END_REF] with parameters fitted on training data of varying size. The methodology was also assessed for second round predictions instead of final round predictions. The results aslo hold in this alternative case, as described in Second round predictions section in Material and Methods.

The individual influenceability and population influenceability methods are compared to a null model assuming constant opinion with no influence (i.e., α(r) = 0). As expected, the null model does not depend on training set size (small variations are artefacts and would vanish if taking infinite number of randomly selected partitions in the crossvalidation). By contrast, the individual influenceability method which, for each individual, fits parameters α i [START_REF] Centola | The spread of behavior in an online social network experiment[END_REF] used for training is higher than 5 in both types of games but its predictions become poorer otherwise, due to overfitting. Overfitting is alleviated using the population influenceability methods which restrict the choice of α(1) and α [START_REF] Aral | Identifying influential and susceptible members of social networks[END_REF], making it robust to training size variations. The population method which uses only one typical couple of influenceability as predictor presents one important advantage. It provides a method which does not require any prior knowledge about the participant targeted for prediction. This method improves by 31% and 18% the prediction error for the two types of games compared to the null model of constant opinion.

The population methods based on two or more typical couples of influenceability require to possess at least one previous game by the participant to calibrate the model. These methods are more powerful than the former if enough data is available regarding the participant's past behaviour (2 or 3 previous games depending on the type of games). The number of typical couples of influenceabilities to use depends on the data availability regarding the targeted participant. This is illustrated in Fig. 3. The modification obtained using more typical influenceabilities for calibration is mild. Moreover, too many typical influenceabilities may lead to poorer predictions due to overfitting. This threshold is reached for 4 couples of influenceabilities in the gauging game data. As a consequence, it is advisable to restrict the choice to 2 or 3 couples of influenceabilities. This analysis shows that possessing data from previous participants in a similar task is often critical to obtain robust predictions on judgment revision of a new participant. The results of the control experiments are displayed as shaded areas in Fig. 2-A,B. This shaded areas correspond to the amount of prediction error which is due to the intrinsic unpredictability of judgment revision. No model can make better predictions than this threshold (see Control experiment section in Material and Methods). The gauging game obtains an unpredictable RMSE of 5.35 while the counting game obtains 8.23. By contrast, the average square variation of the judgments between first and final rounds are respectively 11.76 and 11.84 for both types of games (corresponding to the RMSE of the null model). Taking the intrinsic unpredictable variation thresholds as a reference, the relative prediction RMSE is more than halved when using the time varying influenceability model [START_REF] Centola | The spread of behavior in an online social network experiment[END_REF] with one couple of typical influenceabilities instead of the null model with constant opinion. In other words, more than two thirds of the prediction error made by the consensus model is due to the intrinsic unpredictability of the decision revision process.

Practical Implications of the Model

Do groups reach consensus ?

Because of social influence, groups tend to reduce their disagreement. However, this does not necessarily implies that groups reach consensus. To test how much disagreement remains after the social process, the distance between individual judgments and mean judgments in corresponding groups is computed at each rounds. The results are presented for the gauging game. The same conclusions also hold for the counting game. Fig. 4 presents the statistics summary of these distances. The median distance are respectively 5.5, 4.2 and 3.5 for the three successive rounds, leading to a median distance reduction of 24% from round 1 to 2 and 16% from round 2 to 3. Assuming homogeneous parameters α i (r) among participants and constant mean judgments in model ( 1), the median distance evolution from round 1 to round 2 and round 3 provides estimates of α(1) = 0.24 and α(2) = 0.16 which is consistent to the values found in the influenceability distributions and with a decrease of influenceability over rounds. If we assume a constant ratio α(r + 1)/α(r) = 0.66 and a constant judgment mean, the model (1) would extrapolate, yielding that the distance between judgment and mean will settle away from consensus, at 5.5 [START_REF] Mavrodiev | Quantifying the effects of social influence[END_REF] which is close to half of the initial median distance. Opinions are settling while no consensus has been reached. Moreover, 90% of this movement is reached by round 3 : opinions have already almost settled after three rounds. This suggests that the will to find the true value may not be a sufficient incentive for the group to reach a consensus, opinions settle and disagreement remains. However, the assumption that the ratio α(r + 1)/α(r) is constant over rounds remains to be tested. Is displayed only the data coming from games where for all 3 rounds at least 5 out of 6 participants provided a judgment. This ensures that the judgment mean and standard deviation can be compared over rounds.

∞ t=0 (1-α i •0.66 t ) = 2.

Influenceability and individual performance

Each game is characterized by a true value, corresponding to an exact proportion to guess (for gauging games) or an exact amount of items displayed to the participants (for counting games). Whether social influence promotes or undermines individual performance can be measured for the two tasks. At each round r, a participant's success is characterized by the root mean square distance to truth, denoted E i (r). The error E i (r) depicts how far a participant is to truth. Errors are normalized to fit in the range 0 -100 in both types of tasks so as to be comparable. A global error is defined as the median over participants of E i (r), and the success variation between two rounds is given by the median value of the differences E i (r) -E i (r + 1). A positive or negative success variation corresponds respectively to a success improvement or decline of the participants after social interaction. The results are reported for the gauging game. The same conclusions replicate for the counting game. The global error for rounds 1, 2 and 3 are respectively 11.8, 10.0 and 9.8 (with std of 6.9, 5.6 and 5.5). It reveals an improvement with a success variation of 1.3 and 0.6 for E i (1)-E i (2) and E i (2)-E i (3) respectively, showing that most of the improvement is made between first and second round. We examine relationships between success variation and the model parameters α i by computing partial Pearson correlations ρ controlling for the effect of the rest of the variables. Only significant Pearson correlations are mentioned (p-val < 0.05). All corresponding p-values happen to be smaller than 0.001. Influenceability α i (1) between round 1 and 2 and improvement are positively related with ρ(α i (1), E i (1) -E i (2)) = 0.41, showing that participants who improve more from round 1 to 2 are those who give more weight to the average judgment. This is in accordance to the wisdom of the crowd effect which is a statistical effect stating that averaging over several judgments yields a more accurate evaluation than most of the individual judgments would (see the early ox experiment by Galton in 1907 [START_REF] Galton | Vox populi (the wisdom of crowds)[END_REF] or more recent work [START_REF] Ariely | The effects of averaging subjective probability estimates between and within judges[END_REF]). A similar effect relates success improvement and the influenceability α i (2) between round 2 and 3 with ρ(α i (2) , E i (2) -E i (3)) = 0.36. As may be expected, higher initial success leaves less room for improvement in subsequent rounds, which explains that ρ(E i (1),

E i (1) -E i (2) = 0.68 and ρ(E i (1), E i (2) -E i (3)) = 0.25.
This also means that initially better participants are not better than average at using external judgments.

Modelling influenceability across different types of games

The assessment of the predictive power of model ( 1) on both types of games provides a generalisability test of the prediction method. The two types of games vary in difficulty. The root mean square relative distance E i (1) between a participant's first round judgment and truth is taken as the measure of inaccuracy for each participant. The median inaccuracy for the counting game is 23.1 while it is 11.8 for the gauging game (Mood's median test supports the rejection of equal median, p = 0). Moreover, a Q-Q plot shows that inaccuracy is more dispersed for the counting game, suggesting that estimating quantities is more difficult than gauging proportion of colors. The accuracy of model ( 1) is compared for the two datasets in Fig. 2. Interestingly, the model prediction ranks remains largely unchanged for the two types of games. As depicted in Fig. 1-C, influenceability distributions do not vary significantly between the two games. A two-sample Kolmogorov-Smirnov test fails to reject the equality of distribution null hypothesis of equal median with p > 0.65 for both α i (1) and α i (2). This means that although the participants have an increased difficulty when facing the counting game, they do not significantly modify how much they take judgments from others into account. Additionally, the relationships between participants' success and influenceability are preserved for both types of games. The preserved tendencies corroborate the overall resemblance of behaviours across the two types of games. These similarities indicate that the model can be applied to various types of games with different level of difficulty.

Discussion

The way online social systems are designed has an important effect on judgment outcome [START_REF] Muchnik | Social influence bias: A randomized experiment[END_REF]. Operating or acting on these online social systems provides a way to significantly impact our markets, politics [START_REF] Bond | A 61-million-person experiment in social influence and political mobilization[END_REF] and health. Understanding the social mechanisms underlying opinion revision is critical to plan successful interventions in social networks. It will help to promote the adoption of innovative behaviours (e.g., quit smocking [START_REF] Christakis | The Collective Dynamics of Smoking in a Large Social Network[END_REF], eat healthy) [START_REF] Valente | Network interventions[END_REF]. The design and validation of models of opinion revision will enable to create a bridge between system engineering and network science [START_REF] Liu | Controllability of complex networks[END_REF].

The present work shows that it is possible to model opinion evolution in the context of social influence in a predictive way. When the data regarding a new participant is available, parameters best representing their influenceability are derived using meansquare minimization. When the data is scarce, the data from previous participants is used to predict how the new participant will revise their judgments. To validate our method, results were compared for two types of games varying in difficulty. The model performs similarly in the two experiments, indicating that our influenceability model can be applied to other situations.

The decaying influenceability model after being fit to the data suggests that despite opinion settlement, consensus will not be reached within groups and disagreement will remain. This indicates that there needs to be incentives for a group to reach a consensus. The analysis also reveals that participants who improve more are those with highest influenceability, this independently of their initial success.

The degree to which one may successfully intervene on a social system is directly linked to the degree of predictability of opinion revision. Because there must always be factors which fall out of the researcher's reach (changing mood or motivations of participants), part of the process cannot be predicted. The present study provides way to assess the level of unpredictability of an opinion revision mechanism. This assessment is based on a control experiment with hidden replicated tasks.

The proposed experiment type and validation method can in principle be generalized to any sort of continuous judgment revision. The consensus model can also serve as a building block to more complex models when collective judgments rely on additional information exchange.

Material and Methods

Experiment

Our research is based on an experimental website that we built, which received participants from a crowdsourcing platform. When a participant took part in an experiment, they joined a group of 6 participants. Their task was to successively play 30 games of the same sort related to 30 distinct pictures.

Criteria for online judgment revision game

The games were designed to reveal how opinions evolve as a result of online social influence. Suitable games have to satisfy several constraints. First, to finely quantify influence, the games ought to allow the evolution of opinion to be gradual. Numbers were chosen as the way for participant to communicate their opinion. Multiple choice questions with a list of unordered items (e.g., choosing among a list of holiday locations) were discarded. Along the same lines, the evolution of opinion requires uncertainty and diversity of a sufficient magnitude in the initial judgments. The games were chosen to be sufficiently difficult to obtain this diversity. Thirdly, to encourage serious behaviours, the participants were rewarded based on their success in the games. This required the accuracy of a participant to be computable. Games were selected to have an ideal opinion or truth which served as a reference. Subjective questions involving for instance political or religious opinions were discarded.

Additionally, the game had to satisfy two other constraints related to the online context where, unlike face-to-face experiments, the researcher cannot control behavioural trustworthiness. Since the educational and cultural background of participants is a priori unknown, the game had to be accessible, i.e., any person which could read English had to be able to understand and complete the game. As a result, the games had to be as simple as possible. For instance, games could not involve high-level mathematical computations. Despites being simple to understand our games were still quite difficult to solve, in accordance with the first constraint. Lastly, to anticipate the temptation to cheat, the solution to the games had to be absent from the Internet. Therefore, questions such as estimating the population of a country were discarded.

Gauging and counting games

Each game was associated with a picture. In the gauging game, the pictures were composed of 3 colors and participants estimated the percentage as a number between 0 and 100 of the same given color in the picture. In the counting game, the picture was composed of between 200 and 500 many small items, so that the participant could not count the items one by one. The participants had then to evaluate the total number of these items as a number between 0 and 500. A game was composed of 3 rounds. The picture was kept the same for all 3 rounds. In each round, the participant had to make a judgment. During the first round, each of the 6 participants provided their judgment, independently of the other participants. During the second round, each participant ananymously received all other judgments from the first round and provided their judgment again. The third round was a repetition of the second one. Accuracy of all judgments were converted to a monetary bonus to encourage participants to improve their judgment at each round. Screenshots of the games' interface are provided in the Design of the Experiment section.

Design of the experiment

The present section describes the experiment interface. A freely accessible single player version of the games was also developped to provide a fird hand experience of the games. In the single player version, participants are exposed to judgments stored on our database obtained from real participant in previous games. The single player version is freely accessible at http://collective-intelligence.cran.univ-lorraine.fr/eg/login. The interface and the timing of the single player version is the same as the version used in the control experiment. The only difference is that the freely accessible version does not involve redundent games and provides accuracy feedback to the participants.

In the multi-player version which was used for the uncontrolled experiment, the participants came from the CrowdFlower ® external crowdsourcing platform where they received the URL of the experiment login page along with a keycode to be able to login. The add we posted on CrowdFlower was as follows :

Estimation game regarding color features in images You will be making estimations about features in images. Beware that this game is a 6-player game. If not enough people access the game, you will not be able to start and get rewarded. To start the game : click on <estimation-game> and login using the following information : login : XXXXXXXX password: XXXXXXXX You will receive detailled instruction there. At the end of the game you will receive a reward code which you must enter below in order to get rewarded : < >

The particpants were told they will be given another keycode at the end of the experiment which they had to use to get rewarded on the crowdsourcing platform, this forced the participants to finish the experiment if they wanted to obtain a payment. Secondly, the participants arrived on the experiment login page, chose a login name and password so they could come back using the same login name if they wanted to for another experiment (see supplementary Fig. S1). Once they had logged in, they were requested to agree on a consent form mentioning the preservation of the anonymity of the data (see the Consent and privacy section below for details). Thirdly, the participants were taken to a questionnaire regarding personality, sex, highest level of education, and whether they were native English speaker or not (all the experiment was written in English). The questions regarding personality come from a piece of work by Gosling and Rentfrow [START_REF] Gosling | A very brief measure of the Big-Five personality domains[END_REF] and were used to estimate the five general personality traits (no result are reported on this questionnaire in the present article because no relationship was found between participants' social behaviours and the questionnaires). The questionnaire page is reported for completeness in supplementary Fig. S2. Once the questionnaire submitted, the participants have access to the detailed instructions on the judgment process (supplementary supplementary Fig. S3). After this step, they were taken to a waiting room until 6 participants had arrived at this step. At this point, they started the series of 30 games which appeared 3 at a time, with one lone round where they had to make judgments alone and two social rounds where the provided judgment being aware of judgments from others. An instance of the lone round is given in supplementary Fig. S4-(A) for the counting game while a social round is shown in supplementary Fig. S5. Instances of pictures for the gauging game are provided in supplementary Fig. S4-(B). In the gauging game, the question was replaced by "What percentage of the following color do you see in the image ?". For this type of games, a sample of the color to be gauged for each pictures was displayed between the question and the 3 pictures. At the end of the 30 games, the participants had access to a debrief page where was given the final score and the corresponding bonus. They could also provide a feedback in a text box. They had to provide their email address if they wanted to obtain the bonus (see supplementary Fig. S6).

Consent and privacy

Before starting the experiment, participants had to agree electronically on a consent form mentioning the preservation of the anonymity of the data : Hello! Thank you for participating in this experiment. You will be making estimations about features in images. The closer your answers are to the correct answer, the higher reward you will receive. Your answers will be used for research on personality and behaviour in groups. We will keep complete anonymity of participants at all time. If you consent you will first be taken to a questionnaire. Then, you will get to a detailed instruction page you should read over before starting the game. Do you understand and consent to the terms of the experiment explained above ? If so click on I agree below.

In this way, participants were aware that the data collected from their participation were to be be used for research on personality and behaviour in groups. IP addresses were collected. Email addresses were asked. Email addresses were only used to send participants bonuses via Paypal ® according to their score in the experiments. IP addresses were used solely to obtained the country of origin of the participants. Behaviours were analyzed anonymously. Information collected on the particpants were not used in any other way than the one presented in the manuscript and were not distributed to any third party. Personality, sex and country of origin presented no correlation with influenceability or any other quantity reported in the manuscript. The age of participants was not collected. Only adults are allowed to carry out microtasks on the CrowdFlower platform : CrowdFlower terms and conditions include : "you are at least 18 years of age". The experiment was declared to the Belgian Privacy Comission (https://www.privacycommission.be/) as requested by law. We did not ask the ethics committees/IRBs to approve this consent procedure because the data were analyzed anonymously.

Control experiment

Human judgment is such a complex process that no model can take all its influencing factors into account. The precision of the predictions is limited by the intrinsic variation in the human judgment process. To represent this degree of unpredictability, we consider the variation in the judgment revision process that would occur if a participant were exposed to two replicated games in which the set of initial judgments happened to be identical. A control experiment served to measure this degree of unpredictability.

To create replicated experimental conditions, the judgments of five out of the six participants were synthetically designed. The only human participant in the group was not made aware of this, so they would act as in the uncontrolled experiments. Practically, participants took part in 30 games, among these, 20 games had been designed to form 10 pairs of replicated games with an identical picture used in both games of a pair. To make sure the participants did not notice the presence of replicates, the remaining 10 games were distributed between the replicates. The order of appearance of the games with replicates is as follows : 1, 2, 11|3, 4, 12|6, 13, 5|14, 7, 8|9, 1, 15|4, 10, 16|2, 6, 17|7, 18, 3|8, 5, 19|10, 9, 20, where games 1 to 10 are the replicated games. The games successively appeared three at a time from left to right. The 15 synthetic judgments (5 participants over 3 rounds) which appeared in the first instance of a pair of replicates were copies of past judgment made by real participants in past uncontrolled experiments. The copied games collected in uncontrolled experiments were randomly selected among the games in which more than 5 participants had provided judgments. Since the initial judgment of the real participant could not be controlled, the 15 synthetic judgments in the second replicate had to be shifted in order to maintain constant the initial judgment distances in each replicate. The shift was computed in real time to match the variation of the real participant initial judgments between the two replicates. The same shift was applied to all rounds to keep the synthetic judgments consistent over rounds (see Fig. 5 for the illustration of the shifting process). This provided exactly the same set of initial judgments up to a constant shift in each pair of replicated games. Such an experimental setting allowed assessing the degree of unpredictability in judgment revision (see the Prediction accuracy section in Results for details).

Participants

Uncontrolled experiment

The data were collected during July, September and October 2014. Overall, 654 distinct participants took part in the study (310 in the gauging game only, 308 in the counting game only and 36 in both). In total, 64 groups of 6 participants completed a gauging game, while 71 groups of 6 participants completed a counting game. According to their IP addresses, participants came from 70 distinct countries. Participants mostly originated from 3 continents : 1/3 from Asia, 1/2 from Europe and 1/6 from South America. As detailed at the end of the paragraph, most participants completed most of the 30 games and played trustworthfully. The others were ignored from the study via two systematic filters. First, since the prediction method was tested using up to 15 games in the model parameter estimation process, the predictions reported in present study concern only the participants who completed more than 15 out of the 30 games. This ensures that the number of games used for parameter estimations is homogeneous over all participants. The prediction performance can then be compared The intrinsic variation in judgment revision is the distance between second round judgments, reduced by the shift in initial judgments. The dotted cross is not a judgment and is only displayed to show how the second round intrinsic variation is computed using the initial shift. All shifts are equal. The same shift is also applied to all other second round judgments, and used to compute final round intrinsic variations.

among participants. The median number of fully completed games per participants was 24 with 10.5 std for the gauging game and 27 with 10.5 std for the counting game.

Lower numbers are possibly due to loss of interest in the task or connexion issues. The first filter lead to keep 68% of the participants for the gauging game and 71% for the counting game (see Fig. 6-A,C for details). Secondly, the prediction were only made on judgments of trustworthy participants. Trustworthiness was computed via correlation between participant's judgments and true answers. Most participants carried out the task truthworthfully with a median correlation of 0.85 and median absolute deviation (MAD) of 0.09 for the gauging game and 0.70 median and 0.09 MAD for the counting game. A few participants either played randomly or systematically entered the same aberrant judgment. A minimum Pearson correlation thresholds of 0.61 for the gauging game and 0.24 for the counting game were determined using Iglewicz and Hoaglin method based on median absolute deviation [START_REF] Iglewicz | How to detect and handle outliers[END_REF]. The difference between the two thresholds is due to the higher difficulty of the counting game as expressed by the difference between median correlations. This lead to keep 91% and 96% of the participants which had passed the first filter (see Fig. 6-B,D for details).

Control experiment

The data were collected during May and June 2015. Overall, 207 distinct participants took part in the study (113 in the gauging game only, 87 in the counting game only and 7 in both). The 120 gauging game participants took part in 139 independent games while the 94 counting game participants were involved in 99 independent games. Each independent game was completed by 5 synthetic participants to form groups of 6. The same filters as those used in the uncontrolled experiment were applied to the participants in the control experiment. This lead to keep 88% of the counting games and 80% of the gauging games.

Opinion revision model

To capture the way individual opinions evolve during a collective judgment process, a consensus model is used. These models have a long history in social science [START_REF] French | A formal theory of social power[END_REF] and their behaviour has been thoroughly analyzed in a theoretical way [START_REF] Olfati-Saber | Consensus and cooperation in networked multi-agent systems[END_REF][START_REF] Martin | Continuous-time consensus under persistent connectivity and slow divergence of reciprocal interaction weights[END_REF]. Our model [START_REF] Centola | The spread of behavior in an online social network experiment[END_REF] assumes that when an individual sees a set of opinions, their opinion changes linearly in the distance between their opinion and the mean of the group opinions. There is recent evidence supporting this assumption [START_REF] Mavrodiev | Quantifying the effects of social influence[END_REF]. The rate of opinion change as a result of social influence is termed the influenceability of a participant. The model also assumes that this influenceability may vary over time. The decrease of influenceability represents opinion settling. The model is described in mathematical terms in equation ( 1), with α i (r) being the influenceability of participant i after round r. The ratio α i (2)/α i (1) represents the decaying rate of influenceability. Parameters α i (1) and α i (2) are to be estimated to fit the model to the data. It is expected that α i (1), α i (2) ∈ [0, 1]. If α i (r) = 0, the participant does not take into account the others and their opinion remains constant. If α i (r + 1) = α i (r), the influenceability does not change in time and the opinion x i (t) eventually converges to the mean opinion x. Instead, if α i (r + 1)/α i (r) < 1, influenceability starts positive but decays over time, which represents opinion settling.

Prediction procedure

Regression procedure

The goal of the procedure is to predict the future judgment of a given participant in a given game. The set of first round judgments of this game are supposed to be known to initialize the model [START_REF] Centola | The spread of behavior in an online social network experiment[END_REF], this includes the initial judgment from the participant targeted for prediction and the initial judgments from the five other participants who possibly influenced the former.

To tune the model, prior games from the same participant are assumed to be available. These data serve to estimate the influenceability parameters α i (1) and α i [START_REF] Aral | Identifying influential and susceptible members of social networks[END_REF]. In one scenario, the influenceability parameters of the participant are estimated independently of the data from other participants (individual influenceability method). This is the only feasible method when no prior data on other participants is available (see the Prediction scenarios section in Results for details on the data availability scenarios). In this first case, parameters α i (1) and α i (2) are determined using a mean square error minimization procedure (this procedure amounts to likelihood maximization when the errors between model predictions and actual judgments are normally distributed and have zero mean, see for instance [49, p27]).

In situations where the number of prior games available to estimate the influenceability parameters is small, little information is available on the participant's past behaviour. This may result in unreliable parameter estimates. To cope with such situations, another scenario is considered : besides having access to prior data from the targeted participant, part of the remaining participants (half of them, in our study) are used to derive the typical influenceabilities in the population. The expectation-maximization (EM) algorithm is used to classify the population into groups of similar influenceabilities [START_REF] Bishop | Pattern recognition and machine learning[END_REF]. These typical influenceabilities serve as a pool of candidates. The prior games of the targeted participant are used to determine which candidate yields the smallest mean square error (population influenceability method). The procedure to determine which typical candidate best suits a participant requires less knowledge than the one to accurately estimate their influenceability without a prior knowledge on its value (see the results in the Prediction accuracy section in Results).

Validation procedure

The three scenarios presented in the Prediction scenarios section in Results are validated via crossvalidation. The validation procedure of the last scenario (i.e., access to prior data from participants, existence and access to typical influenceabilities) starts by randomly splitting the set of participants into two equal parts. The prediction focuses on one of the two halves while the remaining half is used to derive the typical influenceabilities in the population influenceability method. In the half serving for prediction, our model is assessed via repeated random sub-sampling crossvalidation : for each participant, a training subset of the games is used to assign the appropriate typical influenceabilities to each participant. The rest of the games serves as the validation set to compute the root mean square error (RMSE) between the observed data and predictions. The error specific to participant i is denoted as RM SE i . The results are compared for various training set sizes. The RMSE is obtained from averaging errors over 300 iterations using a different randomly selected training set each time. To compare scenarios 1 and 2 with scenario 3, we only consider the half serving for prediction. Scenario 1 (no data available) does not require any training step. For scenario 2 (access to prior data from participants), instead of learning affectation of typical influencabilites to each participant, the learning process directly estimates the influenceabilities α i (1) and α i (2) for each participant without prior assumption on their values. The whole validation process is also carried out reversing the role of the two halves of the population and a global RMSE is computed out of the entire process.

Intrinsic unpredictability estimation

Unpredictability of the second round judgment

Even though the study mainly focuses on the predictions of third round judgments, we first focus in the present section on the two first rounds of the games, for the sake of clarity. The prediction procedure can easily be adapted to predict second round rather than third round judgments. Results for second round predictions based on the consensus model ( 1) are presented in Second round predictions section below.

The control experiment described in Control experiment section provides a way of estimating the intrinsic variations in the human judgment process. When a participant takes part in a game, their actual second judgment depends on several factors : their own initial judgment x i (1), the vector of initial judgments from other participants denoted as x others [START_REF] Centola | The spread of behavior in an online social network experiment[END_REF] and the displayed picture. As a consequence, the second round judgment of a participant can always be written as

x i (2) = f 1 i (x i (1), x others (1), picture) + η, (2)
where f 1 i describes how a participant revises their judgment in average depending on their initial judgment and external factors. The term picture is the influence of the picture on the final judgment. The quantity η captures the intrinsic variation made by a participant when making their second round judgment despite having made the same initial judgment and being exposed to the same set of judgments and identical picture. Formally η is a random variable with zero mean. The standard deviation of η is assumed to be the same for all participants to the same game, denoted as std(η). The standard deviation std(η) measures the root mean square error between f 1 i and the actual judgment x i (2) ; this error measures by definition the intrinsic unpredictability of the judgment revision process. If it was known, the function f 1 i would provide the best prediction regarding judgment revision. By definition, no other model can be more precise. The function f 1 i is unknown, but it is reasonable to make the following assumptions. First, the function f 1 i is assumed to be a sum of (i) the influence of the initial judgments and and (ii) the influence of the picture. Thus f 1 i splits into two components :

f 1 i (x i (1), x others (1), picture) = λg 1 i (x i (1), x others (1)) + (1 -λ)h 1 i (picture)
, where g 1 i represents the dependence of the second round judgment on past judgments while h 1 i contains the dependency regarding the picture. The parameter λ ∈ [0, 1] weights the relative importance of the first term compared to the second and is considered unique for each particular type of game. It is further assumed that if the initial judgment x i (1) and the others judgment at round 1 are shifted by a constant shift, the g 1 i component in the second round judgment will on average be shifted in the same way, in other words, it is possible to write

g 1 i (x i (1) + s, x others (1) + s) = g 1 i (x i (1), x others (1)) + s,
where s is a constant shift applied to the judgments. Under this assumption, the control experiment provides a way of measuring the intrinsic variation. The intrinsic variation estimation std(η) can be empirically measured as the root mean of

1 2 (x i (2) -x i (2) -λ(x i (1) -x i (1))) 2 (3)
over all repeated games and all participants, where the prime notation is taken for judgments from the second replicated game in the control experiment (see Participants section in Material and Methods. The derivation of equation ( 3) is provided in supplementary section S1. Since η is assumed to have zero mean, the function f i properly describing the actual judgment revision process is the one minimizing std(η). Correspondingly, the constant λ ∈ [0, 1] is set so as to satisfy this minimization. The intrinsic variation estimation std(η) is displayed in Fig. 7. The optimal λ values are found to be 0.67 and 0.74 and correspond to intrinsic unpredictability estimations of 5.38 and 7.36 for the gauging game and the counting game, respectively. These thresholds can be used to assess the quality of the predictions for the second round (see Second round predictions section).

Unpredictability of the final round judgment

The procedure to estimate third round intrinsic unpredictability, whose results are shown in Fig. 2, varies slightly from second round estimation procedure. For third round judgments, a function with the same input as f 1 i , depending only on the initial judgments and the picture, cannot properly describe the judgment revision of one participant independently of the other participant's behaviour. In fact, the third round judgments depend on the second round judgments of others, which results from the revision process of other players. In other words, in a situation where a participant were to be faced successively to two groups of other participants, who by chance had given an identical set of initial judgments, the second round judgments of the two groups could vary due to distinct ways of revising their judgments.

Since the initial judgments does not suffice in describing third round judgments, function f 1 i is modified to take the second round judgments of others x others (2) as an additional input. Then, the control experiment provides a way to estimate the intrinsic variations occuring in judgment revision up to the third round. This is described formally in the rest of the section. However, it should be noted that this description of judgment revision does not strictly speaking provide the exact degree of intrinsic variation included in the final round prediction error made in the uncontrolled experiment, it is rather an under-estimation of it : The predictions via the consensus model presented in the Prediction performance section in Results, are based solely on the initial judgments, whereas, the second round judgment from other participants are also provided in the present description. This additional piece of information necessarily makes the description of third round judgment more precise. As a consequence, the intrisic variation estimated here (see details below) is an under-estimation of the actual intrinsic variation included in the prediction error of the consensus model. From a practical point of view, this means that the actual intrinsic unpredictability threshold is actually even closer to the predictions made by the consensus model than displayed in Fig. 2. In other words, there is even less space to improve the predictions provided by the consensus model, since more than two thirds of the error comes from the intrinsic variation rather than from the model imperfections.

Formally, the deterministic part of the third round judgment of a participant x i (3) is fully determined as a function of their initial judgment x i (1), the initial judgment of others x others (1), the second round judgment of others x others (2) and the picture. So, the third round judgment can be written as x i (3) = f 2 i (x i (1), x others (1), x others (2), picture) + η,

where η is the estimation of the intrisic variation occuring after three rounds under the same initial judgments, same picture and same second round judgments from others. Under the same assumptions on function f 2 i as those made on f 1 i , and analogously to equation (3), std(η) is measured by the root mean of 1 2 (x i (3) -x i (3) -λ(x i (1) -x i (1))) 2 (5) over all repeated games and all participants. This intrinsic variation is provided as a function of parameter λ in Fig. 7, (C)-(D).

Second round predictions

The prediction procedure based on the consensus model ( 1) is applied to predict the second round judgments. Crossvalidation allows to assess the accuracy of the model. Results are presented in Fig. 8. These results are qualitatively equivalent to the prediction errors for the third round as shown in Fig. 2, although the second round predictions lead to a lower RMSEs, as expected since they correspond to shorter term predictions.

(A) Gauging

Modelling influence and opinion evolution in online collective behaviour

Corentin Vande Kerckhove, Samuel Martin, Pascal Gend, Peter J. Rentfrow, Julien M. Hendrickx, Vincent D. Blondel

S1 Derivation of the measure of unpredictability

Equation ( 3) is derived using the following reasoning. The judgments made in two replicated games of a control experiment by a same participants are described as

x i (2) = f 1 i (x i (1), x others (1), picture) + η, x i (2) = f 1 i (x i (1), x others (1), picture) + η ,

where the prime notation is taken for judgments from the second replicated game and η and η are two independent draws of the random intrinsic variation. By design, the set of judgments are all shifted by the same constant :

x i (1) = x i (1) + s,

x others (1) = x others (1) + s, where s = x i (1) -x i (1) is known. According the assumption made on function f 1 i ,

x i (2) = λg 1 i (x i (1), x others (1)) + (1 -λ)h 1 i (picture) + η, x i (2) = λg 1 i (x i (1), x others (1)) + (1 -λ)h 1 i (picture) + η , the second round judgment made in the second replicate is then

x i (2) = λ g 1 i (x i (1), x others (1)) + s + (1 -λ)h 1 i (picture) + η , where the invariance by translation of g 1 i was used. Taking the difference makes the unknown terms g 1 i (x i (1), x others (1)) and h 1 i (picture) vanish to obtain

x i (2) -x i (2) = λs + η -η.

Since η and η are independent with zero mean, i.e., , E(η) = E(η ) = 0, the theoretical variance of η is

E(η 2 ) = 1 2 E(η 2 ) + E(η 2 ) = 1 2 (E(η 2 ) -2E(η )E(η) + E(η 2 )) = 1 2 (E(η 2 -2η η + η 2 )) = 1 2 E((η -η) 2 ).

Figure 1 .

 1 Figure 1. Influenceability over rounds and games. (A) Gauging game, (B)Counting game : distributions of the α i (1) influenceability after the first round and α i (2) influenceability after the second round for the time-varying influenceability model[START_REF] Centola | The spread of behavior in an online social network experiment[END_REF]. The colormap corresponds to the average prediction RM SE of participants in each bin. For visualization purposes, one value at α i (1) = -1.5 has been removed from the histogram in the gauging game. (C) Cumulated distributions of α i (1) and α i (2) for each type of games.

Figure 3 .

 3 Figure 3. Predictive power of the consensus model when the number of typical couples of influenceability used for model calibration varies. Root mean square error (RMSE) plotted for training set size from 1 to 15 games. (A) gauging game, (B) counting game. In (B), RMSE has been scaled by a factor of 5 to be comparable to the (A) plot.

Figure 4 .

 4 Figure 4. Distance of participants' judgment to mean judgment for rounds 1, 2 and 3 in the gauging game.Is displayed only the data coming from games where for all 3 rounds at least 5 out of 6 participants provided a judgment. This ensures that the judgment mean and standard deviation can be compared over rounds.

Figure 5 .

 5 Figure 5. Illustration of the shift applied to the synthetic judgment in order to preserve the distances between initial judgments. The shift made by the participant in the first round was synthetically applied to all other initial judgments. The intrinsic variation in judgment revision is the distance between second round judgments, reduced by the shift in initial judgments. The dotted cross is not a judgment and is only displayed to show how the second round intrinsic variation is computed using the initial shift. All shifts are equal. The same shift is also applied to all other second round judgments, and used to compute final round intrinsic variations.

Figure 6 .

 6 Figure 6. Filters for participants' trustworthiness. (A),(C) gauging game, (B),(D) counting game. (A,B) : Histograms of the number of full games played by participants. (C,D) : Histograms of the correlations between judgments and true values for each participant. Rejected participants are displayed in red while kept participants are in blue, to the right of the black arrow.

Figure 7 .

 7 Figure 7. Root mean square intrinsic variation as a function of the λ parameters. (A), (B) : the second round judgments, as given in equation (3). (C), (D) : the third round judgments, as given in equation (5). (A), (C) gauging game, (B), (D) counting game.

Figure 8 .

 8 Figure 8. Root mean square error (RMSE) of the predictions for the second round. The RMSEs are obtained from crossvalidation for training set size from 1 to 15 games. (A) gauging game, (B) counting game.In (B), the RM SE has been scaled by a factor of 5 to be comparable to the (A) plot. Top black horizontal line (round dots) corresponds to the null model of constant opinion. The decreasing blue curves (triangle dots) correspond to fitting with the individual influenceability method. Points lying above the null model error are not displayed. The bottom horizontal red lines (crosses) correspond to fitting using the same typical couple of influenceability for the whole population. The slightly decreasing green curves (dots) correspond to fitting choosing among 2 typical couples of influenceability. The shaded gray areas correspond to the prediction error due to intrinsic variation in judgment revision. All RM SE were obtained on validation games.

(A) Gauging

  and α i (2) based on training data, is sensitive to training set size : it performs better than the null model when the number of games

	RMSE
	12						Null Indiviualdα
							1dtypicaldα
	11						2dtypicaldα
	10
	9
	8
	5.35 5.35
	2	4	6	8	10	12	14
			Numberdofdgamesdindtraining
			(B) Counting
	RMSE
	12						Null Indiviualvα
							1vtypicalvα
							2vtypicalvα
	11
	10
	9
	8.23 8.23
	2	4	6	8	10	12	14
			Numbervofvgamesvinvtraining

Figure 2. Root mean square error (RMSE) of the predictions for the final round. The RMSEs are obtained from crossvalidation for training set size from

  null model error are not displayed. The bottom horizontal red lines (crosses) correspond to fitting using the same typical couple of influenceability for the whole population. The slightly decreasing green curves (dots) correspond to fitting choosing among 2 typical couples of influenceability. The shaded gray areas correspond to the prediction error due to intrinsic variations in judgment revision. All RM SE were obtained on validation games.

	1 to
	15 games. (A) gauging game, (B) counting game. In (B), the RM SE has been scaled
	by a factor of 5 to be comparable to the (A) plot. Top black horizontal line (round
	dots) corresponds to the null model of constant opinion. The decreasing blue curves
	(triangle dots) correspond to fitting with the individual influenceability method. Points
	lying above the

Acknowledgments

C. V. K. is a F.N.R.S./FRIA research fellow. Research partly supported by the Belgian Interuniversity Attraction Poles, by an Actions incitatives 2014 of the Centre de

Author contributions statement

C. V. K., S. M., J. M. H., P. J. R. and V. D. B. conceived the experiment(s). C. V. K., S. M. and P. G. conducted the experiments. C. V. K. and S. M. analysed the results. All authors reviewed the manuscript.

Additional information

Competing financial interests: The authors declare no competing financial interests.

Supplementary Information

Consequently, the variance of η is empirically measured as the average of

over all repeated games and all participants. This corresponds to equation (3).