N Le Floc'h

C Knudsen

T Gidenne

L Montagne

E Merlot

O Zemb

Keywords: feed restriction, growth, hygiene, microbiota, pigs Implications

Feed restriction could be a relevant strategy to preserve gut health, reduce systemic inflammatory response and finally limit antibiotic use. This study assessed the effect of feed restriction on growing pigs submitted to a moderate inflammatory challenge induced by the degradation of the environmental hygiene that is known to alter growth rate. The experiment was run on 80 pigs selected at 7 weeks of age according to a 2 × 2 factorial design: two feeding levels, ad libitum (AL) and feed restricted (FR) at 60% of AL, and two conditions of environmental hygiene, clean and dirty. Pigs were housed individually throughout the experiment. From 61 to 68 days of age (day 0 to 7), pigs were housed in a post weaning unit and feed restriction was applied to half of the pigs from day 0 to day 29. At 68 days of age (day 7 of the experiment), pigs were transferred in a growing unit where half of FR and half of AL pigs were housed in a dirty environment (poor hygiene) and the other half in a clean environment (good hygiene) until day 42. Growth performance was recorded weekly. Blood and faeces samples were collected to measure indicators of inflammation, nutrient digestibility and microbiota composition. Faecal consistency was monitored daily to detect diarrhoeas. Feed restriction decreased daily weight gain (-35% to -50%, P < 0.001), increased the feed conversion ratio (+15%, P < 0.001) and CP digestibility (+3%, P < 0.05) and reduced the occurrence of diarrhoeas irrespective of hygiene conditions. Poor hygiene conditions decreased growth performance (-20%, P < 0.05) and total tract digestibility of all nutrients ( P < 0.001). Haptoglobin (+50%) concentrations and lymphocyte (+10%) and granulocyte (+40%) numbers were higher in poor hygiene conditions ( P < 0.05), confirming that the model was effective to induce a systemic inflammatory response. Both feed restriction and hygiene modified the profile of the faecal microbiota. In this study, feed restriction did not reduce the systemic inflammatory response caused by poor hygiene conditions despite the limitation of the occurrence of digestive disorders. However, our study opens discussions regarding the impact of hygiene and feed restriction on gut microbial communities and digestive health.

Impact of feed restriction on health, digestion and faecal microbiota of growing pigs housed in good or poor hygiene conditions

Introduction

To avoid the dissemination of antibiotic resistances and the reduced efficiency of antibiotics for human and animal † E-mail: nathalie.lefloch@rennes.inra.fr Animal (2014), 8:10, pp 1632-1642 © The Animal Consortium 2014 doi:10.1017/S1751731114001608 animal medicine, there is now a general consensus regarding the necessity to reduce the use of antibiotics in farm animals. Alternatives to antibiotics, such as feed additives and feeding practices share the common goal of preventing health problems and maintaining the productivity of farm animals [START_REF] Kil | Management and feeding strategies to ameliorate the impact of removing antibiotic growth promoters from diets fed to weanling pigs[END_REF]. Strategies based on a moderate feed restriction could be used without increasing feed costs.

For healthy young growing pigs, the major objective is to optimize the feed intake to maximize the growth. During the critical phases, ad libitum (AL) feeding could be suboptimal since pigs may consume more than they can digest. If a large amount of undigested nutrients is fermented in the large intestine, it could unbalance the microbial ecosystem which in turn could result in diarrhoeas as it was shown in newly weaned piglets [START_REF] Kil | Management and feeding strategies to ameliorate the impact of removing antibiotic growth promoters from diets fed to weanling pigs[END_REF]. Thus, feed restriction applied during the first days after weaning reduces diarrhoeas as well as the proportion of hemolytic Escherichia coli in the faeces [START_REF] Rantzer | Effects of a strategic feed restriction on pig performance and health during the post-weaning period[END_REF]. Feed restriction strategies are now currently used from weaning and throughout almost all the growing period in rabbit farming and allow reducing mortality and morbidity [START_REF] Gidenne | Feed intake limitation strategies for the growing rabbit: effect on feeding behaviour, welfare, performance, digestive physiology and health: a review[END_REF]. Data from other species also suggest that feed restriction is not only beneficial for the digestive tract, but may also efficiently alleviate the consequences of a systemic inflammatory response in growing and adult animals. In rodents, feed restriction attenuated the inflammatory response caused by a lipopolysaccharide administration [START_REF] Matsuzaki | Inflammatory responses to lipopolysaccharide are suppressed in 40% energy-restricted mice[END_REF]. These results reinforce the necessity to better understand the consequences of a transient feed restriction on the health and physiology of growing pigs, for which health disorders are not limited to digestive disturbances, and growth rate may be strongly impacted by subclinical and multifactorial diseases.

The objective of this study was to determine the effect of a substantial but transient feed restriction on the ability of growing pigs to face a moderate inflammatory challenge induced by the degradation of environmental hygiene conditions (Le [START_REF] Floc | Importance of sanitary environment for growth performance and plasma nutrient homeostasis during the post-weaning period in piglets[END_REF][START_REF] Pastorelli | Sanitary housing conditions modify the performance and behavioural response of weaned pigs to feed-and housing-related stressors[END_REF]. To do so, we measured growth performance, nutrient digestibility as well as blood indicators of inflammation and diarrhoea in pigs submitted to a feed restriction prior and during the hygiene challenge. Additionally, the impact of hygiene and feed restriction on the gut microbiota was evaluated by sequencing the 16S rRNA of the ribosomal small-subunit from faecal samples.

Material and methods

Animals

The experiment was performed at the INRA experimental facilities in Saint-Gilles (France) in compliance with the guidelines of the French Ministry of Agriculture for animal experimentation and care. The protocol was approved by the regional ethical committee (C2EA-07). The experiment was conducted with 80 castrated male and female Piétrain × (Large White × Landrace) pigs from INRA (UMR1348 PEGASE, Saint-Gilles), in two replicates of 40 pigs each. Pigs were weaned at 4 weeks of age. Twenty blocks of four half-sibling pigs (piglets from the same boar) with a similar BW were constituted at 8 weeks of age (average weight of 20.6 ± 2.44 kg). Within a block, each pig was affected to one of the four experimental treatments described below.

Experimental design

The experiment consisted in a complete 2 × 2 factorial design comparing two environmental hygiene conditions (Good (G) and Poor (P)) and two feeding levels (AL and feed restricted (FR) at 60% of AL). At 8 weeks of age, pigs were adapted to individual housing. Two weeks later (day 0), two pigs per block were FR (Period 1, Figure 1). At 68 days of age (day 7 of the Feed restriction in pigs experiment), all pigs were transferred in a growing unit. Within each block, one FR and one AL pig were transferred in good (G) hygiene conditions (FR-G and AL-G pigs), and one FR and one AL pig were housed in poor (P) hygiene conditions (FR-P and AL-P pigs). Feed restriction was maintained for 3 additional weeks in FR pigs (Period 2). From day 30 to day 42 of the trial (Period 3), pigs were housed in the same hygiene conditions as during the Period 2, but all were fed AL. From day 43 of the trial (Period 4), the hygiene challenge was stopped and all pigs were housed in conventional hygiene conditions. Pigs were vaccinated against Haemophilus parasuis at 7 and 10 weeks of age and no other medication was applied during the experimental period.

Housing

Pigs were housed individually throughout the experiment. From day 0 to day 7, they were housed in individual 80 × 60 cm cages in a post weaning unit where the room temperature was of 25°C. Then, until day 42, they were housed in 85 × 265 cm pens in a growing unit where the room temperature was of 22°C. Different hygiene conditions were obtained through a modification of the procedure described by [START_REF] Floc | Importance of sanitary environment for growth performance and plasma nutrient homeostasis during the post-weaning period in piglets[END_REF]. Poor hygiene conditions were created by housing pigs in a room previously occupied by non-experimental pigs. The room was neither cleaned nor disinfected before and during the experimental period. Additionally, non-experimental pigs were also housed in the same room to increase the microbial pressure. In good hygiene conditions, pigs were housed in a room that was cleaned and disinfected before and daily throughout the experimental period. There was no mixing with nonexperimental pigs and the staff put clean boots and clothes before entering the room.

Diets and feeding

Pigs were fed a standard weaner diet during the period of adaptation and Period 1, then a standard growing diet containing 0.4% of titanium dioxide as an indigestible marker. The weaner and growing diets (Table 1) provided 9643 and 9634 kJ net energy/kg and 186 and 179 g/kg of CP, respectively. During the period of adaptation to the individual housing as well as during Periods 3 and 4, all pigs were fed AL and feed refusals were recorded every morning to calculate feed intake. During Periods 1 and 2, FR pigs were fed 60% of their feed consumption/kg BW measured during the 2 weeks of adaptation. The daily feed allowance was recalculated every day to take into account the increase in BW. Feed refusals, if any, were daily recorded. Water was provided AL throughout the experiment.

Measurements and biological samplings

The mornings before the allocation to each experimental group, at the beginning of the feed restriction and the transfer to the growing unit, the pigs were weighed after being fasted overnight. Then, the pigs were weighed every week without being fasted. Growth performance (average daily gain (ADG), average daily feed intake (ADFI) and feed to gain ratio (F/G)) were calculated for Periods 1, 2 and 3 whereas only ADG was calculated for Period 4. Backfat thickness was measured during Period 3 at day 30 (end of the feed restriction period) and day 42 (end of the hygiene challenge) by ultrasonic scanning (Vetko plus echograph; Noveko Inc., Boucherville, Canada).

Three blood samples were taken between 0800 and 0900 h on all the pigs the day before starting the feed restriction then on day 14 and day 28 (middle and end of Period 2) by jugular puncture. The blood sampling procedure was timed and limited to 2 min in order to limit the effect of stress on blood parameters. Blood (2 × 5 ml) was collected in K2-EDTA and Heparin Vacutainer tubes. The possible negative effects of feed restriction on the well-being of the animals was evaluated by measuring cortisol in the salivary samples collected on days 0, 2, 6 and 8 between 1100 and 1145 h, by allowing piglets to chew on cotton buds voluntarily until they were moistened. Blood and salivary samples were kept on ice until their transfer to the laboratory. Faeces samples were collected twice on 2 consecutive days (days 27 to 28 and days 37 to 38) for nutrient digestibility measurements. Faeces were immediately chilled on ice after collection and then freeze-dried. Samples of 2 consecutive days The weaning diet was distributed during the Period 1 and the growing diet during the Periods 2 to 4.

were pooled, ground and kept at 4°C until analysis. For microbiological analyses, faeces were collected on day 1 and day 28 and stored at -80°C. The consistency of the faeces was monitored daily during Periods 1 and 2 and scored using a 3-levels scale (0 = solid or normal, 1 = soft or moist and 2 = diarrhoeic faeces).

Blood analyses

The total number of leukocytes and the differential count of lymphocytes and polymorphonuclear leukocytes (PMN) were measured on whole blood collected in EDTA tubes with a hematology automated cell counter calibrated for pigs (MS--9R, Melet Schloesing laboratories, Osny, France).

The blood was then centrifuged (2500 × g, 20 min, 4°C) and the plasma was aliquoted and kept at -20°C until analysis. Plasma concentration of haptoglobin, a major acute-phase protein in pigs, was measured by a colorimetric method (Phase Haptoglobin assay T801, Tridelta Development Limited, Maynooth, Ireland). Plasma concentration of cortisol was assessed by a I125 radio-immunoassay (Immunotech, Prague, Czech Republic). Saliva was collected by centrifugation of the cotton buds at 3000 × g for 15 min at 4°C, and stored at -20°C until cortisol measurement with a luminescence immunoassay kit (LIA, IBL, Hamburg, Germany).

Faecal scores, faecal analyses and digestibility calculations

The sum of faecal scores was calculated for both Periods 1 and 2. The number of pigs with solid, soft or diarrhoeic faeces was calculated per day during Periods 1 and 2. Then, for each period, three categories of pigs were identified as followed: pigs with solid faeces throughout the period, for at least half the duration of the period and for less than half the duration of the period. The percentage of pigs within each class was then calculated for each experimental group. Diets and faeces were analysed for dry matter, organic and mineral matters, CP (N × 6.25; Dumas Method) using the AOAC procedures. Gross energy was measured in diets and faeces using an adiabatic bomb calorimeter (IKA, Staufen, Germany). The concentration of titanium dioxide in the growing diet and in the faeces was determined photometrically (Cobas Mira, Horiba ABX, Montpellier, France). The total tract apparent digestibility of nutrients and energy was calculated using the nutrients to marker (titanium dioxide) ratio in the diet and faeces as previously described [START_REF] Wilfart | Sites of nutrient digestion in growing pigs: effect of dietary fiber[END_REF].

Microbial analyses of faecal samples

The microbial DNA was extracted from 200 mg of frozen faeces by beadbeating according to a previously published protocol [START_REF] Combes | Postnatal development of the rabbit caecal microbiota composition and activity[END_REF] and sent to a molecular research laboratory (Lubbock, TX, USA) for FLX 454 pyrosequencing using the 27F and 530R primers targeting the 16S rRNA gene [START_REF] Dowd | Evaluation of the bacterial diversity in the feces of cattle using 16S rDNA bacterial tag-encoded FLX amplicon pyrosequencing (bTEFAP)[END_REF]. Sequences with ambiguous base calls, homopolymer exceeding 6 bp or insufficient length (200 bp) were removed. The resulting sequences were preclustered using the naïve Bayesian classifier embedded in Mothur 1.30.1 with a cutoff of 50% [START_REF] Schloss | Introducing mothur: open-source, platform-independent, community-supported software for describing and comparing microbial communities[END_REF] trained on the data set 9 of the ribosomal database project (Cole et al., 2007). Each precluster was then independently divided into operational taxonomic units (OTUs) using Esprittree [START_REF] Sun | A large-scale benchmark study of existing algorithms for taxonomy-independent microbial community analysis[END_REF], singletons were eliminated and non-redundancy of the OTUs was verified. Finally the phylogenetic affiliation of one representative sequence per OTU was performed using the LTP database (www.arb-silva.de/ projects/living-tree).

Statistical analyses

The MIXED procedure of SAS (version 8.1, 2000; SAS Institute Inc., Cary, NC, USA) was used for all data other than faecal scores and microbial OTUs. The experimental unit was the pig. The effects of hygiene conditions (H), feeding levels (F), replicate and their interactions were first tested per period for growth performance and per date for blood, saliva and faecal analyses. The interactions of hygiene conditions and feeding levels with replicate were non-significant and were removed from the model. The block was used as a random effect. The initial BW and the weight recorded on the day of measurement were used as covariates for the analyses of growth performance and backfat thickness, respectively. Blood leukocyte numbers, haptoglobin and cortisol concentrations were submitted to a log transformation and salivary cortisol concentrations to a square root transformation to fit normal distribution. The effects were considered as significant if P < 0.05 and adjusted means (LS means) were compared using the Bonferroni test.

The sum of faecal scores were analysed by Kruskal-Wallis test (Anastat software) and the frequency of pigs within each class was analysed by a χ 2 test (FREQ procedure of SAS). The statistical significance of feed restriction and hygiene on gut microbiota was determined by the discriminant analysis of the principal components or DAPC procedure with 500 group randomizations [START_REF] Jombart | Discriminant analysis of principal components: a new method for the analysis of genetically structured populations[END_REF].

Results

Three pigs (one AL-G, one AL-P and one FR-P) were removed from the experimental design because of health disorders (diarrhoea requiring antibiotic administration, rectal prolapses and lameness).

Feed intake and growth performance Observed average feed intake differed slightly from what was expected (Table 2). The difference between AL and FR was of 30% on average during Period 1 and, irrespective of the hygiene conditions, of 45% during Period 2, instead of the expected 40%. Both hygiene and feeding levels reflected in the performance but the interaction between hygiene and feeding level was not significant. Initial average BW was equivalent between treatments. FR pigs had lower growth rate and greater F : G than AL pigs during Period 1 (Table 2). At the beginning of the hygiene challenge, FR pigs were 2.3 kg lighter than pigs fed AL (P F < 0.001). Both feed restriction and hygiene challenge reduced the growth performance and increased F : G measured during Period 2. During Period 3, growth rate did not differ between pigs Feed restriction in pigs previously FR and pigs previously fed AL, whereas ADFI of pigs previously FR remained lower than that of pigs previously fed AL. Consequently, F : G was lower in pigs previously FR than in pigs previously fed AL. The effect of the hygiene challenge was significant on ADG, ADFI and F : G leading to lower growth performance in pigs housed in poor compared with good hygiene conditions. At the end of both Periods 2 and 3, backfat thickness was lower in FR pigs than in AL pigs and was not affected by the hygiene challenge. During Period 4, pigs previously FR and pigs previously housed in good hygiene conditions grew faster than pigs previously fed AL and pigs previously housed in poor hygiene conditions. However, pigs previously FR were still more than 8 kg-lighter than pigs fed AL, whereas the effect of hygiene challenge on BW was no longer significant.

Total tract digestibility coefficients

The total tract digestibility coefficients of the main dietary components were affected by hygiene conditions in both periods of measurement, that is end of Period 2 and 3 (Table 3). Pigs housed in good hygiene conditions had greater coefficients compared with pigs submitted to the hygiene challenge (P H < 0.001). Feed restriction did not impact on the digestibility coefficients of dry and organic matter, and coherently of energy but negatively affected that of minerals (P F < 0.05) especially in pigs submitted to the hygiene challenge (-7.3% between AL and FR in poor hygiene conditions; P H × F < 0.05). Irrespectively of the hygiene conditions, FR pigs had greater digestibility of CP than AL pigs (75.9% v. 73.6%, P < 0.01) at the end of Period 2. Inversely, at the end of Period 3, 1 week after the end of the feed restriction, digestibility of CP was greater in AL pigs than in FR pigs (74.3% v. 72.8%, respectively, regardless the hygiene conditions).

Faecal scores

The sum of faecal scores was greater in AL pigs than in FR pigs during Period 2 but not during Period 1. The effect of hygiene was not significant. During Period 1, feed restriction (Table 4) did not affect the proportion of pigs with normal faeces within the three categories defined in the Material and methods section (P F = 0.38). During Period 2 (hygiene challenge and feed restriction), feed restriction decreased the frequency of diarrhoeas and moist faeces as 41% of the FR pigs had normal faeces all the days of Period 2 compared with 10.5% of the AL pigs (P F = 0.003). Hygiene challenge (data not shown) did not modify the proportion of pigs within each category (P H = 0.74) with on average, 26% of the pigs with normal faeces during all the days of Period 2, 70% during more than half of the period and 4% during less than half of the period. Probability values for the effect of hygiene conditions (H), feeding level (F) and the interaction (H × F); *P < 0.05, **P < 0.01, ***P < 0.001 and ns = non-significant.

3

Pigs were housed in one of the two hygiene conditions during periods 2 and 3. Poor hygiene conditions correspond to a room that was neither cleaned after being occupied by pigs of a previous band, nor during the experimental period. Good hygiene conditions correspond to a room that was cleaned and disinfected before and during the experimental period.

Plasma haptoglobin concentrations and blood formula Concentrations of plasma haptoglobin (Figure 2a), blood numbers of lymphocytes and PMN (Figure 2b) were measured before the start of the experiment (day 1) and during Period 2 (days 14 and 28). The blood variables did not differ between the four experimental groups on day 1 and were affected neither by the feeding level, nor by the interaction between the feeding level and hygiene conditions. One week after the transfer into the growing unit (day 14), pigs housed in poor hygiene conditions had greater PMN numbers (P H < 0.001), than those housed in good hygiene conditions, whereas the number of lymphocytes did not differ. Plasma haptoglobin tended to be higher in pigs housed in poor hygiene conditions (0.05 < P H < 0.1). At the end of Period 2 (day 28), pigs housed in poor hygiene conditions had greater plasma haptoglobin concentration (+105%, P H < 0.001), blood PMN number (+69%, P H < 0.001) and lymphocyte number (+10%, data not shown, P H < 0.05) than pigs housed in good conditions.

Salivary and plasma cortisol Plasma cortisol concentrations were not affected by feeding level and hygiene conditions at day 1 (42 ± 2 ng/ml) and day 14 (34 ± 2 ng/ml). On day 28, FR pigs tended to have lower cortisol concentrations than AL pigs (23 v. 27 ± 3 ng/ml, 0.05 < P F < 0.1). Salivary cortisol was slightly increased in FR pigs in comparison with the AL pigs 3 h after the first meal on day 0 (P F < 0.01), but no more on day 2 and day 6, and after the transfer to the growing unit on day 8 (Figure 3).

Gut microbiota

The analysis of the microbial DNA in the faeces collected at the end of Period 2 yielded 362 871 sequences that were clustered in 4106 OTUs. DAPC on the relative abundances of the 4106 OTUs separated all four biological treatments (i.e. FR-G, FR-P, AL-G and AL-P) better than any other 500 randomized groups (P = 0.0075, Figure 4). In contrast randomized groups offered a discriminative power similar to the real groups at the beginning of the experiment (day 1), Probability values for the effect of hygiene challenge (H), feeding level (F) and the interaction (H × P); *P < 0.05, **P < 0.01, ***P < 0.001 and ns = non-significant.

3

Pigs were housed in one of the two hygiene conditions during Periods 2 and 3. Poor hygiene conditions correspond to a room that was neither cleaned after being occupied by pigs of a previous band, nor during the experimental period. Good hygiene conditions correspond to a room that was cleaned and disinfected before and during the experimental period.

Table 4 Consequences of feed restriction on the proportion (%) of pigs with normal faeces throughout the period (Category 1), more than half of the period (Category 2) or less than half of the period (Category 3)

Ad libitum Restricted

Feeding Probability values (χ 2 test) for the effect of feeding level; *P < 0.05 and ns = non-significant.

Feed restriction in pigs

showing that the pig microbiota of the 8 weeks old animals are initially undistinguishable but then evolve differently depending on the hygiene and feed restriction. Furthermore it shows that both feed restriction and hygiene conditions can modulate the gut microbiota. Interestingly the 10 species with the highest DAPC loadings correspond to 18 non-redundant microbial species which are sufficient to distinguish the four groups by DAPC (P = 0.032). These 18 species represent between 3% and 27% of the total number of sequences of the individual samples. Out of the 18 separating phylotypes, the phylotypes related to Clostridium butyricum, Clostridium bartletti, Lactobacillus animalis and Bifidobacterium choerinum are significantly favoured by feed restriction (Figure 5).

Discussion

Consequences of poor environmental hygiene on pig performance and health

The present results are in accordance with previous data reporting a major effect of poor hygiene on growth performance of pigs from weaning to the first weeks of the growing period (Le [START_REF] Floc | Importance of sanitary environment for growth performance and plasma nutrient homeostasis during the post-weaning period in piglets[END_REF]. Reduced feed intake and digestibility as well as metabolic and behavioural changes caused by poor hygiene probably act together to impair growth rate [START_REF] Pastorelli | Sanitary housing conditions modify the performance and behavioural response of weaned pigs to feed-and housing-related stressors[END_REF].

Poor hygiene conditions induced a stimulation of the immune system as indicated by greater haptoglobin concentrations, a major acute phase protein in pigs [START_REF] Eckersall | The acute phase response of acid soluble glycoprotein, α1-acid glycoprotein, ceruloplasmin, haptoglobin and C-reactive protein, in the pig[END_REF], and changes in blood cell counts. The proportion of white blood cells, and mainly that of granulocytes, increased while that of lymphocytes remained almost constant. Those changes in blood formula have been described in pigs suffering from inflammation in response to a bacterial infection [START_REF] Odink | Hematological and clinicochemical profiles of healthy swine and swine with inflammatory processes[END_REF]. Additionally, the synthesis of haptoglobin by the liver is also enhanced during the inflammatory response [START_REF] Moshage | Cytokines and the hepatic acute phase response[END_REF] and greater plasma haptoglobin concentrations are correlated with poor health and sanitary status of the pig farm [START_REF] Harding | Association of tumour necrosis factor and acute phase reactant changes with post arrival disease in swine[END_REF][START_REF] Lipperheide | Effects of farm hygiene on blood chemical variables in fattening pigs[END_REF]. Accordingly, our results indicated that the poor hygiene conditions led to faecal ecosystem modifications. Pigs housed in good and poor hygiene conditions exhibit different microbial communities. Among the 18 species that suffice to distinguish the four groups by DAPC, poor hygiene significantly favours a phylotype related to B. choerinum (0.17% v. 0.05%), which is known to correlates with cytokine expression and profile in blood and intestine [START_REF] Splichalova | Interference of Bifidobacterium choerinum or Escherichia coli Nissle 1917 with Salmonella Typhimurium in gnotobiotic piglets correlates with cytokine patterns in blood and intestine[END_REF]. Interestingly, good hygiene also significantly favours a Clostridium-related phylotype (0.8% v. 0.4%) but additional data would be required to determine if this phylotype is more closely related to Clostridium difficile strain 630 harbouring pathogenic pathways (Scaria et al., 2011) or to Clostridium bartlettii, which might be beneficial by limiting the adhesion of enterotoxigenic E. coli [START_REF] Messori | Effect of susceptibility to enterotoxigenic Escherichia coli F4 and of dietary tryptophan on gut microbiota diversity observed in healthy young pigs[END_REF].

Pigs housed in poor hygiene conditions ate 10% less than pigs housed in good environmental hygiene. This is similar to what was reported for growing pigs after 2 weeks of hygiene challenge [START_REF] Pastorelli | Sanitary housing conditions modify the performance and behavioural response of weaned pigs to feed-and housing-related stressors[END_REF]. The lower feed intake probably results from the moderate systemic inflammation caused by the poor environmental hygiene since the release of inflammatory mediators, like cytokines, are known to be responsible for appetite reduction [START_REF] Plata-Salaman | Cytokines and feeding suppression: an integrative view from neurologic to molecular levels[END_REF]. In a previous experiment, the impact of poor hygiene conditions on the growth rate was observed on pair fed pigs confirming that the depressed growth rate was not caused only by the lower feed intake (Le [START_REF] Floc | Importance of sanitary environment for growth performance and plasma nutrient homeostasis during the post-weaning period in piglets[END_REF]2009). Indeed, inflammation is known to be associated with dramatic changes in metabolism leading to a repartitioning of nutrients away from growth towards functions associated with body defences [START_REF] Klasing | Monokines in growth and development[END_REF].

The depressed of the apparent total tract digestibility measured in pigs reared in poor hygiene could result in a real impairment of the digestion (including hydrolysis and absorption) and/or an increase in endogenous losses. This may be involved in the depressed growth rate through a reduction of the amount of nutrients delivered to the body by digestion. Faecal scores indicated that poor environmental hygiene did not increase diarrhoeas. Thus, malabsorption caused by digestive disorders may not explain the impact of poor hygiene on digestibility. For example, the fact that the digestibility of minerals was impaired in pigs housed in poor hygiene conditions compared with pigs in good hygiene conditions was quite unexpected since diarrhoeas limit mineral absorption [START_REF] Scrimshaw | Synergism of nutrition, infection, and immunity: an overview[END_REF]. This result contrasts with a previous experiment conducted with the same hygiene model applied after weaning and that clearly established that the proportion of diarrhoeas was greater in poor hygiene conditions [START_REF] Montagne | Comparative effects of level of dietary fiber and sanitary conditions on the growth and health of weanling pigs[END_REF].

Consequences of feed restriction on pig performance and stress response FR pigs grew less than pigs fed AL during the period where feed restriction was applied. Feed restriction also depressed feed efficiency and reduced backfat thickness, possibly because restricted pigs support their maintenance metabolism rather than growth. Moreover, the difference in backfat thickness indicates that FR pigs had a higher protein and less fat deposition than pigs fed AL. The fact that protein deposition is less energetically efficient than fat deposition probably contributed to reduce feed efficiency [START_REF] Lovatto | Effects of feed restriction and subsequent refeeding on energy utilization in growing pigs[END_REF]. On the other hand, it is unlikely that variations in nutrient digestibility were involved. Indeed, the digestibility of nutrients was unaltered except for minerals, which digestibility slightly decreased, but only in poor hygiene conditions. In other species like the rabbit, total tract nitrogen digestibility improvement caused by the feed restriction was associated to prolonged transit time [START_REF] Gidenne | Feed intake limitation strategies for the growing rabbit: effect on feeding behaviour, welfare, performance, digestive physiology and health: a review[END_REF].

When all pigs returned to an AL feeding level (Period 3), feed efficiency was greater in the pigs previously FR than in the pigs fed AL as reported previously [START_REF] Lovatto | Effects of feed restriction and subsequent refeeding on energy utilization in growing pigs[END_REF]. However, this was not sufficient to compensate entirely for their reduced growth by the end of Period 4. Our results suggest that the improvement of feed efficiency was not solely caused by an increase in nitrogen digestibility, but also by metabolic changes [START_REF] Whang | Effects of protein deprivation on subsequent growth performance, gain of body components, and protein requirements in growing pigs[END_REF]. For example, at the end of the feed restriction period, blood cortisol tended to be lower in FR pigs compared with AL pigs. Likewise, Booth et al. (1994) reported lower cortisol levels in restricted growing pigs in the post-prandial state, which might reflect a better anabolic state.

Our study was designed to determine whether a substantial feed restriction could help the growing pigs to cope with moderate health deterioration. However, such a strategy would only be acceptable provided that it does not generate welfare problems. An elevation in salivary cortisol can be considered as a good stress marker in pigs [START_REF] Merlot | Endocrine response of gilts to various common stressors: a comparison of indicators and methods of analysis[END_REF]. The slight increase in salivary cortisol in the FR pigs compared with the pigs fed AL 3 h after the first application of restriction indicated that this restriction might generate a discomfort, probably due to food frustration. However, this increase was of small amplitude in comparison to the response obtained after another stressor such as the transfer to a new building on day 8. Furthermore, no increase was observed in FR pigs in comparison to AL pigs at the later time points (on days 2, 6 and 8). This might suggest that piglets easily adapted to their new feeding level. However, De [START_REF] Leeuw | Effects of feeding level and the presence of a foraging substrate on the behaviour and stress physiological response of individually housed gilts[END_REF] showed that food frustration in pigs could lead to the expression of behaviours revealing a situation of stress, even when cortisol levels are unaffected.

Consequences of feed restriction on health parameters and microbiota Limiting energy supply in pigs does not reduce the haptoglobin or leukocyte response (data not shown). Therefore feed restriction seems to be inefficient to reduce the inflammation generated by poor hygiene and the consequences of poor hygiene on growth performance. Contrary to what was reported in rodents injected with a bacterial Figure 5 Phylogenetic tree of the 18 most discriminating phylotypes and P-values per phylotype for the effects of the feed restriction (F), the hygiene (H) and their interaction; the names of the strains already tested as probiotics regulating inflammation by inducing cytokines are printed in grey; the confidence of the phylogenetic affiliation is reported in brackets; the strains without an acceptable phylogenetic affiliation are compared with the closest cultivable relatives.

endotoxin [START_REF] Matsuzaki | Inflammatory responses to lipopolysaccharide are suppressed in 40% energy-restricted mice[END_REF][START_REF] Macdonald | Calorie restriction attenuates LPS-induced sickness behavior and shifts hypothalamic signaling pathways to an anti-inflammatory bias[END_REF], feed restriction had no effect on the inflammatory response caused by poor hygiene conditions. The intensity of feed restriction, 25% and 50% of the AL intake for [START_REF] Macdonald | Calorie restriction attenuates LPS-induced sickness behavior and shifts hypothalamic signaling pathways to an anti-inflammatory bias[END_REF], as well as the intensity of the inflammatory response and the species may explain this discrepancy. Yet, in growing and unchallenged pigs, caloric restriction applied for a few days modified the expression of genes involved in the immune response [START_REF] Lkhagvadorj | Gene expression profiling of the short-term adaptive response to acute caloric restriction in liver and adipose tissues of pigs differing in feed efficiency[END_REF] suggesting that short-term feed restriction may potentially modify the immune response. Moreover, feed restriction did not exert any positive effect on growth performance of pigs housed in poor hygiene conditions. This suggests that metabolic changes induced by feed restriction were not effective to overcome the metabolic consequences of poor hygiene conditions. In this experiment, pigs were fed standard diets formulated to cover their nutritional requirements. However, feed restriction may also induce deficiencies in some nutrients that are specifically involved in the regulation of the inflammatory response, thus preventing an efficient positive effect of feed restriction.

In accordance with data obtained in weaning pigs [START_REF] Rantzer | Effects of a strategic feed restriction on pig performance and health during the post-weaning period[END_REF], feed restriction was effective to reduce the proportion of pigs with diarrhoeas and soft faeces, but this was observed irrespectively of hygiene conditions. Using feed restriction to favour bacteria that may reduce the local and systemic inflammation might contribute to the lower diarrhoea occurrence observed in the FR pigs. Interestingly most of the phylotypes distinguishing the four groups have been tested as probiotics to reduce inflammation: attempts using B. choerinum in pigs were unsuccessful [START_REF] Splichalova | Interference of Bifidobacterium choerinum or Escherichia coli Nissle 1917 with Salmonella Typhimurium in gnotobiotic piglets correlates with cytokine patterns in blood and intestine[END_REF] but secretion of cytokines in mice is modified by the ingestion of C. butyricum [START_REF] Hayashi | A single strain of Clostridium butyricum induces intestinal IL-10-producing macrophages to suppress acute experimental colitis in mice[END_REF] or L. animalis [START_REF] Karunasena | Effects of the probiotic Lactobacillus animalis in murine Mycobacterium avium subspecies paratuberculosis infection[END_REF]. It should be noted that lactobacilli are naturally abundant in the pig gut [START_REF] Leser | Culture-independent analysis of gut bacteria: the pig gastrointestinal tract microbiota revisited[END_REF] and that they correlate to an increased immune response when piglets are raised outdoors rather than indoors [START_REF] Mulder | Environmentally-acquired bacteria influence microbial diversity and natural innate immune responses at gut surfaces[END_REF]. Thus feed restriction simultaneously favours several phylotypes that may reduce the inflammation with the exception of Megasphaera elsdenii, which is hindered in restricted pigs. Surprisingly hygiene and feeding levels interplay to yield the abundance of several of the 18 discriminating phylotypes. For example, Butyricicoccus is only favoured by good hygiene in pigs fed AL, which might strengthens the epithelial barrier function [START_REF] Eeckhaut | Butyricicoccus pullicaecorum in inflammatory bowel disease[END_REF]. Similarly Blautia wexlerae and Turicibacter sanguinis are favoured in the FR pigs, but only in good hygiene.

Conclusion

Our study showed no beneficial effect of a substantial feed restriction on the inflammatory response caused by poor hygiene conditions. However, feed restriction reduced the occurrence of diarrhoeas irrespective of hygiene conditions. Together these results suggest that microbiota communities are more sensitive to the interplay between hygiene and feed restriction than the general indicators of inflammation such as haptoglobin or leukocytes. The fact that the four groups can be distinguished by their faecal microbial communities challenges the relative stability observed in the pig microbial communities from 4 weeks onwards [START_REF] Thompson | A window of environmental dependence is evident in multiple phylogenetically distinct subgroups in the faecal community of piglets[END_REF], possibly because the window of dependence to environmental changes is larger than previously thought. Moderate feed restriction strategies should be explored to limit the performance deterioration while preserving a digestive health benefit.

Figure 1

 1 Figure1Schematic representation of the experimental design. The experimental groups are AL-G, pigs housed in good environmental hygiene conditions and fed ad libitum, FR-G, pigs housed in good environmental hygiene conditions and feed restricted, AL-P, pigs housed in poor environmental hygiene conditions and fed ad libitum, FR-P, pigs housed in poor environmental hygiene conditions and feed restricted. AL = pigs fed ad libitum; FR = feed-restricted pigs.

Figure 2 Figure 3

 23 Figure2Consequences of the deterioration of environmental hygiene conditions on (a) plasma haptoglobin concentrations and (b) blood polymorphonuclear (PMN) leukocyte numbers; # and *** for 0.05 < P < 0.10 and P < 0.001, respectively. G, pigs housed in good environmental hygiene conditions; P, pigs housed in poor environmental hygiene conditions.

Figure 4

 4 Figure 4 Separation of the 4 groups by DAPC based on the total 16S microbial communities of the feed restricted (FR) or ad libitum (AL) pigs housed in poor (P) or good (G) environmental hygiene housings (Left) and the 18 most discriminating phylotypes (Right). The 3D-ellipsoids are centered on the barycentre of each group.

Table 1

 1 Ingredients and chemical composition of the diets

	Diet 1
	Weaning	Growing

Table 2

 2 Consequences of feed restriction on the performance of pigs housed in good or poor environmental hygiene conditions

	Feeding level 1	Ad libitum	Restricted			P-values 2
	Hygiene conditions 3	Good	Poor	Good	Poor	s.e.m.	H	F	H × F
	Period 1 (days 0 to 7): feed restriction alone
	Initial BW (kg)	20.7		20.4		0.5		ns
	ADG (g/day)	711		457		19		***
	ADFI (g)	1180		839		33		***
	F : G	1.66		1.92		0.05		***
	Final BW (kg)	26.4		24.1		0.6		***
	Period 2 (days 8 to 29): feed restriction and hygiene challenge
	ADG (g/day)	779	616	384	303	28	***	***	ns
	ADFI (g/day)	1710	1556	935	866	57	*	***	ns
	F : G	2.19	2.60	2.74	2.90	0.12	*	***	ns
	Final BW (kg)	43.7	40.2	32.7	30.5	1.2	***	***	ns
	Backfat thickness (mm)	7.15	6.93	6.49	6.40	0.18	ns	**	ns
	Period 3 (days 30 to 42): hygiene challenge alone
	ADG (g/day)	1293	1124	1273	1114	43	***	ns	ns
	ADFI (g/day)	2545	2267	2165	2034	82	**	***	ns
	F : G	1.96	2.03	1.71	1.85	0.06	**	***	ns
	Final BW (kg)	60.6	54.8	49.3	45.0	1.6	***	***	ns
	Backfat thickness (mm)	8.63	8.11	7.81	7.91	0.23	ns	*	ns
	Period 4 (days 43 to 78): recovery
	ADG (g/day)	996	1110	1090	1122	25	**	*	ns
	Final BW (kg)	95.4	93.9	87.4	84.2	1.9	ns	***	ns

ADG = average daily gain; ADFI = average daily feed intake; F : G = feed to gain ratio. 1 Feed restriction (60% of ad libitum) was applied during Periods 1 and 2. 2

Table 3

 3 Consequences of feed restriction on apparent total tract digestibility (%) of nutrients and energy of pigs housed in good or poor environmental hygiene conditions

	Feeding level 1	Ad libitum	Restricted			P-values 2
	Hygiene conditions 3	Good	Poor	Good	Poor	s.e.m.	H	F	H × F
	Period 2 (days 27 to 28): feed restriction and hygiene challenge
	Dry matter	81.7	78.7	82.8	77.8	0.6	***	ns	ns
	Minerals	51.8	41.8	51.0	34.5	1.6	***	*	*
	Organic matter	83.6	81.1	84.8	80.5	0.6	***	ns	ns
	CP	77.3	70.0	80.4	71.5	0.8	***	*	ns
	Energy	81.3	77.8	82.6	77.2	0.7	***	ns	ns
	Period 3 (days 37 to 38): hygiene challenge alone
	Dry matter	82.0	79.8	81.6	79.1	0.5	***	ns	ns
	Minerals	49.9	44.4	51.4	44.1	0.9	***	ns	ns
	Organic matter	84.1	82.0	83.5	81.3	0.5	***	ns	ns
	CP	77.4	71.2	76.1	69.4	0.7	***	*	ns
	Energy	81.8	78.9	81.0	78.1	0.5	***	ns	ns
	1 Feed restriction (60% of ad libitum) was applied during Periods 1 and 2.

2

Acknowledgements

This study received the financial support from the Animal Physiology and Livestock System division of INRA. The authors thank B. Carissant, R. Comte, S. Daré, J. Delamarre, H. Demay, F. Guérin, Y. Jaguelin, P. Roger, F. Thomas and P. Touanel from INRA UMR PEGASE for animal care and laboratory expertise and B. Gabinaud from INRA UMR TANDEM for the laboratory expertise in microbiology.


Stéphane Airiau
email: stephane.airiau@dauphine.fr

Umberto Grandi
email: umberto.grandi@irit.fr

Filipo Studzinski Perotto
email: filipo.perotto@gmail.com

Toulouse Archive Ouverte Learning Agents for Iterative Voting

Keywords: Computational social choice, Iterative voting, Bandit algorithms

This paper assesses the learning capabilities of agents in a situation of collective choice. Each agent is endowed with a private preference concerning a number of alternative candidates, and participates in an iterated plurality election. Agents get rewards depending on the winner of each election, and adjust their voting strategy using reinforcement learning. By conducting extensive simulations, we show that our agents are capable of learning how to take decisions at the level of well-known voting procedures, and that these decisions maintain good choice-theoretic properties when increasing the number of agents or candidates.

Introduction

In a situation of collective choice, we say that an agent is voting strategically, or that she is manipulating, when the agent does not submit her sincere view to the voting system in order to obtain a collective result that she prefers to the one that would be obtained had she voted sincerely. A classical result in social choice theory showed that all sensible voting rules are susceptible to strategic voting [START_REF] Gibbard | Manipulation of voting schemes: A general result[END_REF][START_REF] Allen | Strategy-proofness and Arrow's conditions: Existence and correspondence theorems for voting procedures and social welfare functions[END_REF]. In fact, strategic voting may be exploited to make better decisions in several situations where, for instance, agents are confronted with a sequence of repeated elections, from where an interesting compromising candidate can be elected.

The plurality rule, aka. first-past-the-post, selects the alternatives that have been voted for by the highest number of agents. Its computation is quick and its communication costs very low, but in view of its simplicity it suffers from numerous problems. For example, it is possible that the plurality winner would lose in pairwise comparison against all other alternatives3 . If however the plurality rule is used in an iterative fashion, staging sequential elections in which at each point in time one of the voters is allowed to manipulate, then it constitutes an effective tool for selecting an outcome at equilibrium with good properties [START_REF] Meir | Convergence to equilibria in plurality voting[END_REF]. This setting is known as iterative voting, and several recent papers explored its convergence using different voting rules, and assessed the quality of the winner [START_REF] Lev | Convergence of iterative voting[END_REF][START_REF] Reijngoud | Voter response to iterated poll information[END_REF][START_REF] Grandi | Restricted manipulation in iterative voting: Condorcet efficiency and Borda score[END_REF][START_REF] Meir | A local-dominance theory of voting equilibria[END_REF][START_REF] Obraztsova | On the convergence of iterative voting: How restrictive should restricted dynamics be?[END_REF][START_REF] Rabinovich | Analysis of equilibria in iterative voting schemes[END_REF].

Most works in this field suffer from two main drawbacks. First, agents are highly myopic in not taking into account the history of their interactions, and in having an horizon for strategic thinking of one single iteration. It creates an artificial asymmetry between the available knowledge and the strategic behavior. Second, to ensure convergence it is required that agents manipulate one at a time, a property that is difficult to enforce.

In this paper, we tackle both aspects by studying a concurrent manipulation process in which agents have the capability of learning from their past interaction. In our setting, iterative voting is seen as a repeated game in which voters use reinforcement learning to cast their ballots. We limit the information available to the learning agents to only the winner of each iteration step (when classic iterative voting methods require more information). Our goal is to show that multiagent learning can be a solution in the context of iterative voting, even when the information available to agents is severly limited: a learning agent bases her decisions on the history of past interactions, and because of the learning rate, the choice of the vote is not purely myopic. In addition, in our model all the learning agents are allowed to change their ballot at the same time. Such possibility cannot be given to classic iterative voting methods because there may not be convergence to a single winner [START_REF] Meir | Convergence to equilibria in plurality voting[END_REF]. The question we ask in this paper is whether learning can help making a good collective decision [START_REF] Shoham | If multi-agent learning is the answer, what is the question[END_REF]: do we observe convergence, and is the winner good according to choice-theoretic criteria?

We show experimentally that our learning agents are able to learn how to make collective decisions under standard measures of decision quality, such as the Condorcet efficiency and the Borda score. The contribution of this paper is twofold: we show that iterative learning 1) outdo all iterative voting methods using less information, and 2) is comparable to a well-known procedure called single transferable vote.

The paper is organized as follows. Section 2 provides the basic definitions and reviews the literature on iterative voting and multiagent reinforcement learning. Section 3 presents the specifics of our simulation setting, and Section 4 discusses the obtained results. Section 5 concludes the paper.

Basic Definitions and Related Works

We now provide all definitions that are needed for the construction of our setting. We introduce the basics of iterative voting and of multiagent reinforcement learning.

Voting Rules

Let C be a finite set of m candidates or alternatives and N be a finite set of n agents. Based on their preferences, individuals in N need to make a decision on which alternative in C to choose. Agents are typically assumed to have preferences over candidates in C in the form of a linear order, i.e., a transitive, anti-symmetric and complete binary relation over C. We denote with > i the preference of agent i and with P = (> 1 , . . . , > n ) the profile listing all individual preferences. Hence, we write b > i a to denote that agent i prefers candidate b to candidate a. A (non-resolute) voting rule is a function w that associates with every profile P a non-empty subset of winning candidates w(P ) ∈ 2 C \∅. The simplest voting rule, and the one involving as little communication as possible among the agents, is the plurality rule: each agent votes for a single candidate, and the candidates with the highest number of votes win. The Borda rule is a scoring rule in which a candidate c is given m-j points for each voter that is ranking c in j-th position. The score of a candidate is the sum of her points over all voters. For the Copeland rule, the score of a candidate c is the number of pairwise comparisons she wins (i.e., contests between c and another candidate a such that there is a majority of voters preferring c to a) minus the number of pairwise comparisons she loses. For Borda and Copeland, the candidates with the highest score win. Finally, Single Transferable Vote (STV) can be viewed as an iterative process: at the first round the candidate that is ranked first by the fewest number of voters gets eliminated (ties are broken following a predetermined order). Votes initially given to the eliminated candidate are then transferred to the candidate that comes immediately after in the individual preferences. This process is iterated until one alternative is ranked first by a majority of voters.

Iterative Voting

Agents face the choice of submitting their truthful ballot, i.e., a ballot corresponding to their individual preferences, or to vote strategically. In iterative voting, agents start from a voting situation: they fill their ballots and a winner is announced. Then, one at a time, an agent changes her ballot, and a new winner is announced, creating a sequence of ballot profiles and consequent winners.

Iterative voting is guaranteed to converge for the plurality rule with linear tiebreaking [START_REF] Meir | Convergence to equilibria in plurality voting[END_REF] when agents know the score of each candidate at each round, though for most other voting rules convergence cannot be guaranteed [START_REF] Lev | Convergence of iterative voting[END_REF]. Restricted dynamics, defined by limiting the possible actions available to players, have therefore been studied to guarantee convergence [START_REF] Reijngoud | Voter response to iterated poll information[END_REF][START_REF] Grandi | Restricted manipulation in iterative voting: Condorcet efficiency and Borda score[END_REF][START_REF] Obraztsova | On the convergence of iterative voting: How restrictive should restricted dynamics be?[END_REF]. In this paper we focus on iterative voting with the plurality rule and on the following two strategies for individual manipulation:

Best response: at time t, given the plurality score for each candidate c at time t -1, the voter computes her best response(s) and votes for one; 3-Pragmatists [START_REF] Reijngoud | Voter response to iterated poll information[END_REF]: at time t, given the top three plurality candidates at time t -1, the voter manipulates in favour of her preferred candidate amongst them.

Convergence with 3-pragmatists manipulation is guaranteed by the fact that the set of 3 most-voted candidates is not changed by every manipulation step, hence each agent will manipulate the election only once.

Two main critiques have been raised to the setting defined above. First, one agent at a time can change her ballot and the new winner is announced before another agent can change her ballot: this sequential aspect is unrealistic but is key to guarantee the convergence of iterative voting. As was already observed by [START_REF] Meir | Convergence to equilibria in plurality voting[END_REF], there is no convergence if individuals are allowed to move at the same time. Second, individuals are highly myopic, since their strategic horizon only considers one-step forward in the iterative process and they do not make use of the history of previous manipulations by other agents when making their next choice.

Multiagent Learning

Multiagent reinforcement learning has been used both in cooperative domains (where the set of agents share the same goal) and in non-cooperative ones (where each agent is trying to optimize its own personal utility). For cooperative domains, the key issue is that learners obtain a local/personal reward but need to learn a decision that is good for the society of agents. For example, agents that try to optimise air-traffic [START_REF] Agogino | A multiagent approach to managing air traffic flow[END_REF] care about individual preferences as well as the global traffic. In this paper, agents are not concerned about the quality of the outcome for the entire population: each voter would like one of her favourite candidates to win. We are in a non-cooperative setting similar to the one of learning in games: the actions are the different ballots and agents have preferences over the joint actions (i.e. voters have preferences over the candidates).

One key difference is that preferences are typically ordinal in voting whereas they are cardinal for games (see Section 3.1 describing how we generate cardinal utilities from ordinal ones).

In this paper, we use a basic multiarmed bandit style reinforcement learning algorithm [START_REF] Sutton | Introduction to Reinforcement Learning[END_REF] for testing whether agents can learn to make a collective decision, experimenting with different exploration strategies (see Section 3 for a detailed description). Many reinforcement learning algorithms have been used for playing normal form games, e.g. joint-action learning [START_REF] Claus | The dynamics of reinforcement learning in cooperative multiagent systems[END_REF], gradient-based algorithms such as IGA-WoLF or WoLF-PHC [START_REF] Bowling | Multiagent learning using a variable learning rate[END_REF], to name a few. Since no algorithm can be claimed to be best, we focus on showing that the most basic learning algorithm is able to perform well. For a similar reason we also choose that agents will only get to observe the current winner, and no other information is available to them, such as the score of all candidates (as done in standard iterative voting).

Evaluation Criteria

Because there is no consensus on the quality of a collective outcome, we will study the results on multiple criteria. Given a profile of preferences (> 1 , . . . , > n ), a Condorcet winner (CW) is a candidate that beats every other candidate in pairwise comparisons. A CW is not guaranteed to exists, but a first parameter in assessing a voting rule is the percentage of profiles in which it elects a CW when there exists one:

Condorcet efficiency: the ratio of profiles where a CW is elected out of all profiles where a CW exists.

Many voting rules are designed to elect a CW whenever it exists, such as the Copeland rule, which hence have a Condorcet efficiency of 1. Other voting rules, such as plurality, Borda and STV may elect a candidate that is not a Condorcet winner. Related work estimates the Condorcet efficiency of plurality and Borda for large electorates using Monte Carlo simulations [START_REF] Lepelley | Computer simulations of voting systems[END_REF].

A second parameter that can be used to measure the quality of the winner is the Borda score itself: Borda Score: a candidate c is given m -j points for each agent ranking c in j-th position in her truthful preference

The Borda score provides a good measure of how the rule compromises between topranked candidates and candidates ranked lower in the individual preferences. One interpretation of the Borda score is that it estimates the average rank of candidates, and the Borda winner is the candidate with the highest average rank over all candidates. Obviously, the Borda rule is the best rule according to this criterion.When varying the number of voters or candidates, we measure the ratio between the Borda score of the elected winner and the maximal Borda score that can be obtained, i.e., if B(c) is the Borda score of a candidate c then BR(c) = B(c)/ max a∈C (B(a)).

Learning and Simulation Setting

We now describe the settings of our simulations. Each simulation is defined by the parameters m = |C| (the number of candidates), n = |N | (the number of voters), T as the number of iterations, or repeated elections, the agents dispose to learn. We use iterative voting with plurality rule and lexicographic tie-breaking. Note that the choice of the tie-breaking method has been shown to be an important factor in guaranteeing the convergence of iterative voting rules [START_REF] Lev | Convergence of iterative voting[END_REF]. We also performed experiments with a randomised tie-breaking rule, obtaining comparable results.

Preferences and Utilities

While voting is based on ordinal information, reinforcement learning needs cardinal utility. Hence the need to translate a preference order > i of agent i into a utility function u i : C → R. Given an ordering >, let pos(c) be candidate c's position, where position 0 is taken by the most preferred candidate, and |C| -1 by the least preferred. We considered three possibilities:

Linear utilities: u lin i (c) = 1 -pos(c) |C|-1 ; Exponential utilities: u exp i (c) = 1 2 pos(c) ; Logistic utilities: u sig i (c) = 1 - 1 1+e -k(pos(c)-|C|-1 2
) ,

The parameter k controls the steepness of the curve in the last definition. These three different methods represent distinct satisfaction contexts. Linear utilities correspond to the Borda values, meaning that the satisfaction with a given candidate decreases linearly following the preference order. Exponential utilities are a more realistic representation, especially in large domains where alternatives at the top bear more importance than those at the bottom. They can also be used to simulate partial orders, since the alternatives below a certain threshold of utility count as non-ranked. In this case, the voters have precise choices, and the satisfaction decreases quickly as soon as the winner is not the preferred candidate. In contrast, logistic utilities decrease slowly in a neighbourhood of the top preferred candidates.

Profiles Generation

Our experiments are averaged over 10.000 preference profiles generated using the following two distributions:

Impartial culture assumption (IC): linear orders are drawn uniformly at random.

Urn model with correlation α ∈ [0, 1): The preference order of the first voter is drawn with uniform probability among all possible linear orders that are present in an urn. A number of copies of the first drawn preference is then put into the urn depending on the parameter α (more precisely, m!

( 1 α -1)
), and the preference of the second voter is then drawn. The process is repeated until all n preference orders have been selected.

The urn model is also known as the Polya-Eggenberger model [START_REF] Berg | Paradox of voting under an urn model: The effect of homogeneity[END_REF]. The interest of such scheme is to have some correlation between voters, where some observed preference is more likely to be observed again. The higher the correlation parameter α the more likely it is that a Condorcet winner exists, and the less likely it is that a single voter can change the winner of the plurality rule with a single manipulation. Note that IC is equivalent to the Urn model with α = 0.

We developed a generator for the urn model that avoids the manipulation of all permutations over the set of candidates, based on the following intuition. At first the probability of creating a random preference order is equal to 1, and the list of defined voters is empty. At each iteration a new preference is generated, either as a copy of a previous preference, or by generating a random preference. The probability of using a random preference decreases with the number of preference orders generated. The method is described in Algorithm 1, the function CreateRandomPreference() returns a sample from the uniform distribution over all possible linear orders, and CopyPref-erenceFrom(N) picks a voter uniformly at random from N , returning a copy of her preferences.

Algorithm 1 Efficient Urn Model Generator (α, C, n)

/* α is correlation, C is the set of candidates, n is the size of the profile */ N ← ∅ // preference profile

β ← 1 α -1 for i from 0 to n -1 do v ← CreateRandomPreference(), with prob. β β+|N |
CopyPreferenceFrom(N), otherwise N ← N ∪ {v} end for

Learning Algorithm

Since the voting rule is plurality and only the winner is announced at each iteration, the learner has to decide, at each iteration, the candidate she will vote for. From the point of view of each voter, the proposed setting correspond to the well-known multiarmed bandit problem (MAB), a wide studied case of computational reinforcement learning (RL). A MAB is equivalent to a Markovian Decision Process (MDP) with a single state [START_REF] Sutton | Introduction to Reinforcement Learning[END_REF][START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF].

The learning mechanism must evaluate the utility of each possible action during the sequence of interactions, only based on the feedback suggested by the reward. In our case, the reward is the preference value of the elected candidate (the winner) for the agent. The agent then learns a function Q : C → R + that estimates the expected utility of voting for each candidate. Once a voter i has voted for candidate c and knows the winner w, it can compute its reward r (the elected winner is w and we have r = u i (w)), and from there, i updates its Q value using the following update rule:

Q(c) ← βr + (1 -β) • Q(c).
where α is the learning rate used to control the impact of new information: when β = 0, the new information is not used, when β = 1, only the new information matters. Initially, we fix β = 0.1.

For controlling the exploration, we have several choices (e.g. using -greedy, softmax exploration schemes or UCB [START_REF] Auer | Finite-time analysis of the multiarmed bandit problem[END_REF]). We experimented several mechanisms, and the one described below obtained the best results. We use a simple implementation of the "optimism in face of uncertainty" principle for exploration: the voter always picks the candidate with the highest Q-value. To ensure exploration, the Q-values are initialized with a copy of the preference values of the agent, and in case of ties the agent prefers to chose the action that has been tested the least. Thus, a voter is most likely to vote for candidates she prefers at the beginning. If a different candidate wins the iteration, the bad reward decreases the Q-value, and the agent has incentives to try other candidates in the coming iterations.

Both -greedy and softmax algorithms make the agents very explorative at the beginning, and when an agent explores, sub-optimal actions are chosen (even the least preferred ones), disturbing the learning progress of all the agents. The same happens with classic version of the optimistic-greedy method, where all the utilities are equally initialized to the maximal reward possible. The UCB method is also very conservative, the exploration is made sufficient to guarantee near optimal performance in stationary MAB problems. As the environment is not stationary (agents are concurrently learning), exploration is no longer adequate and UCB performs poorly.

Simulation Results

We now present the main results, showing that a society of agents provided with very simple learning capabilities can make a "good" collective decision, comparable to that taken by well-known voting rules and often better than what standard iterative voting would recommend. In all results we present next, we use the urn model with α = 0.1 for generating the preferences. With the exception of Section 4.3, all experiments in this section use exponential utilities, but the results for linear and logistic utilities are similar and not presented here.

Learning Dynamics

By considering the result of iterated plurality with learning agents as a voting rule per se (recall that in our setting iterated plurality is guaranteed to converge), we are able to evaluate its performance in social-choice-theoretic terms, measuring both its Condorcet efficiency and the Borda score of the winner at the end of the iteration. In Figures 1 and2 we plot the progress of the first parameter depending on the number of learning iterations that is allowed (a similar figure can be obtained for the second parameter). In order to interpret our findings, we also plot the Condorcet efficiency (CE) and Borda score of one-round plurality, best-response iterative voting, 3-pragmatists iterative voting, STV, Copeland and Borda. We present the results in one scenario with a population of 9 voters and 7 candidates. Utilities of voters are generated using exponential utilities and the results averaged over 10,000 elections.

First, let us consider the Borda score as evaluation criteria in Figure 1. By design, the Borda voting rule is best. The averaged rank of the winner over all the elections is about 1.7. The next best mechanisms are Copeland and STV. All these voting rules require the knowledge of the complete ordinal preference of the agents. We observe that iterative voting comes next, either in the standard mechanism or our new mechanism with learning agent. Note that our learning agents are using less information as they only know the current winner whereas standard iterative voting uses the plurality score of all candidates. The level of performance is still quite acceptable (the winners' averaged rank is about 1.94). Finally, we observe that our mechanism outperforms one round plurality and 3-pragmatists. Now, let us turn to Figure 2 in which we evaluate the performances using Condorcet efficiency. Among the voting rules we consider, only Copeland always elects a Condorcet winner when it exists. For the other rules, STV performs well, followed by iterative voting with learning agents. The best results are obtained for a high number of candidates and low number of voters. Previous work showed that a 10% increase could be obtained with restricted iterative voting in a similar setting (25 candidates and 10 vot- ers, with preferences generated using the urn model) [START_REF] Grandi | Restricted manipulation in iterative voting: Condorcet efficiency and Borda score[END_REF], and comparable figures with 50 voters and 5 alternatives using the impartial culture generation of preferences [START_REF] Reijngoud | Voter response to iterated poll information[END_REF]. With standard iterative voting, only one voter at a time can manipulate, and it will do so only if it changes the outcome. If the winner c is winning by two votes or more, standard iterative voting will not be able to elect the Condorcet winner. In our framework, all voters can manipulate. However, some learners may not notice they have a chance to improve their utilities (because they voted for cw in the past but cw never won, so the Q-value for cw is low), others may decide to explore. In addition, voters do not know whether a Condorcet winner exists. But the improvement we observe shows that learners manage to coordinate their vote which results in electing a Condorcet winner.

Observe that, while Borda and Copeland are obviously scoring the maximum, respectively, in Borda score and CE, the Borda rule can score worse than iterative learning in terms of Condorcet efficiency, and a complex voting rule such as STV can score worse under both parameters (this occurs in simulations performed with 3 voters and 15 candidates). Note that we also conducted the same experiments under the impartial culture assumption, obtaining similar results. As a last remark, observe that in view of our initialization the first election is always truthful, i.e. its result coincide with oneround plurality. This is not the case anymore when using other exploration strategies, for which we obtained slower but similar learning dynamics.

Scalability

One drawback of using learning agents is the number of iterations for convergence. Obviously, it is not reasonable for a human agent to participate in such an iterated process. In the results presented in this section simulations are run with 500 iterations, and we show that learners can still perform well. In addition, the number of voters and the number of candidates are two parameters that, when increased, could significantly deteriorate the performance of learning agents in iterative voting. Figures 3 and4 shows instead that the deterioration is comparable to those voting rules we considered. When we keep the number of voters fixed and we add more candidates, the Condorcet efficiency decreases at a similar rate as the other voting rules (results are equivalent to the ones of STV, and learning agents perform better than iterative best response and 3-pragmatist with a similar margin). Typically, in learning in games adding additional actions requires more iterations for learning well (and we usually observe a slight drop in performance). When the number of candidates is high, candidates low in the ranking will have utilities that are negligible compared to the top candidates. Therefore, under those circumstances, the loss in Condorcet efficiency is acceptable.

What is perhaps the most interesting result is that the number of voters does not affect significantly the performance of the learning agents. This is surprising since the environment is less stationary and the noise level is higher with more agents trying to learn concurrently. Typically, it is much more difficult to reach convergence with a high number of voters. On the other hand, with the increasing number of agents, the likelihood of being pivotal decreases, which may help convergence.

Social Welfare

The last criteria we want to consider are the measures of the social welfare that aggregate the individual utility. We considered the following two definitions: Observe that if individual utilities are defined as the Borda score, i.e., giving m -j points to the individual in j-th position, then the USW of a candidate corresponds to its Borda score. The Borda score, Borda ratio and Condorcet Efficiency measure the performance of the voting rules. On the other hand, USW is a measure of efficiency, often used to study the performance of a (cooperative) multiagent system. Each learning agent is trying to maximise its private utility function. Using plurality at each round, a majority of agents will be satisfied, so we do not necessarily expect to maximise USW, but we should observe that a majority of agents improves their utilities. Indeed, this is what we observed in our simulations, in which initially the performance of the learning agents is comparable with those of other voting rules but the USW quickly deteriorates. Remember that the utilities of each candidate (from the most preferred to the least preferred) are 1, 1 2 , 1 4 , etc. If we look at the distribution of utilities, we have more very high values initially (but less than a majority) and at convergence, we have a majority of middle or high utility values. The overall sum decreases over time, but more agents are happier. As a proof of this observation we measured the egalitarian social welfare -the utility of the poorest voter -observing that initially ESW starts pretty low and raises with the number of iterations. Using that criterion, iterative voting with learning agents scores third behind the Borda rule and Copeland.

Our method seems to perform poorly in terms of USW, although this should not be interpreted as a negative result. Note that the second worse voting rule is the Borda rule for the same reason. We also ran experiments with linear and logistic utilities, observing an increasing social welfare as measured by the Borda score (recall that USW with linear utilities is the same as the Borda score).

Conclusions and Future Work

Motivated by the emergent works on iterative voting, this paper assesses the learning capabilities of autonomous agents in making good collective decisions. Voting theory tells us that agents always have incentives to manipulate. The idea of iterative voting is then to use a simple voting rule such as plurality, and ask each voter one at a time whether she wants to change her ballot to obtain a preferred winner in the next round.

In this paper we address two weaknesses of existing models of iterative voting: it is not realistic that voters change their ballot one at a time, nor is the myopic-agent assumption that does not allow voters to profit from past interactions. In our model we allow all agents to change their ballots at the same time if they wish to do so. In order to avoid a completely chaotic process, we use learning agents as a mean to learn a good compromise. We show that by using a simple learning algorithm with scarce information (only the winner of the current election is shown to learning agents), the performance of winning candidates are quite good. We evaluate the winner of the iterative process using extensive simulations both in terms of Borda score and of Condorcet efficiency against various voting rules, and we show that we obtain reasonable performances from around 300 iterations. While this number is of course too large for any human to use this method, it is manageable for artificial agents.

We leave it for future work to use more sophisticated learning mechanisms for decreasing the number of iterations to obtain a reasonable performance, and to explore settings in which more information is available to the agents. For instance, using the score of each candidate appears quite reasonable, but has consequences on the learning space.

Fig. 4 .

 4 Fig. 4. Scalability of the performance of iterative voting with learning agents at 500 iterations, increasing voters (Condorcet efficiency, 7 candidates)

  Performance of learning agents in terms of Borda score of the winner (9 voters, 7 candidates). The number of profiles which actually have a Condorcet winner is 9359 profiles.

		39
		38.5
	Borda Score of Winner	36 36.5 37 37.5 38
		35.5
		35
		0	50	100	150	200	250	300
					Number of Iterations
	One Round Plurality Copeland Borda STV	terative 3 Pragmatist
	Iterative Best Response Iterative Learning Optimistic Greedy
	Fig. 1.

  Performance of learning agents in terms of Condorcet efficiency (9 voters, 7 candidates). The number of profiles which actually have a Condorcet winner is 7331 profiles.

		1
		0 95
	Condorcet Efficiency	0.8 0 85 0.9
		0.75
		0.7
		0	50	100	150	200	250	300
					Number of Iterations
	One Round Plurality Copeland Borda STV	terative 3 Pragmatist
	Iterative Best Response Iterative Learning Optimistic Greedy
	Fig. 2.

  Scalability of the performance of iterative voting with learning agents at 500 iterations, increasing candidates (Condorcet efficiency, 9 voters).

		1
		0.98
		0.96
		0.94
		0.92
		0.9
		0.88
	Condorcet Efficiency	0.74 0.76 0.78 0.8 0.82 0.84 0.86
		0.72
		0.7
		0.68
		0.66
		0.64
		0.62
		0.6
		2	4	6	8	10	12	14	16	18	20	22	24	26	28
							Number of Candidates
			One Round Plurality	Copeland	Borda	STV Iterative 3 Pragmatist	Iterative Best Response	Iterative Learning
	Fig. 3.

See, e.g.,[START_REF] Pacuit | Voting methods[END_REF] for a proof.