Héloïse Théro
email: heloise.thero@sciencespo.fr

Emmanuel Vincent
email: emmanuel.vincent@sciencespo.fr

Investigating Facebook's policies to tackle misinformation during the COVID-19 pandemic

Keywords: Disinformation, Fact-checking, Social media analysis, Algorithmic bias and transparency

Using a dataset of URLs marked as 'False' by Science Feedback, an organization verifying the credibility of science-related viral information, we investigated the reach of groups and pages sharing misinformation on Facebook during 2019 and 2020, and Facebook's actions to curb its spread. We found that, consistently with what the company publicly announced, Facebook removed large QAnon conspiracy accounts in August 2020, which were instrumental in spreading science misinformation. We also investigated Facebook's 'repeat offenders' policy and found that only some accounts that repeatedly share misinformation display periods in which the popularity of their posts was temporarily reduced. Despite these measures, we have witnessed that most Facebook accounts spreading scientific misinformation have increased their reach at the beginning of the COVID-19 pandemic (March -June 2020), and that a drastic drop in their reach suddenly occurred around the 9th of June, 2020. Meanwhile the reach of a set of mainstream news accounts remained stable across the entire period. No public information was given by Facebook about this sudden decrease.

INTRODUCTION

While the new coronavirus keeps propagating, so are misleading or false information regarding the pandemic. It has led the director of the World Health Organization to declare: "we're not just fighting an epidemic; we're fighting an infodemic. " [START_REF] Adhanom | Munich Security Conference[END_REF] The term 'infodemic' has been used to express the threat related to the massive spread of misinformation, as it can undermine public health measures to contain the spread of the virus [START_REF] Zarocostas | How to fight an infodemic[END_REF].

Today, an increasing proportion of the public get their information online [START_REF] Mitchell | Pathways to news[END_REF], mainly through search engines, social media and video platforms. In April 2020, a questionnaire from the Reuters Institute in the UK found that people use more online than offline sources when looking for information about the coronavirus. Among social media platforms, Facebook was the most widely used with 24% of the respondents saying they used Facebook to access COVID-19 information in the last 7 days [START_REF] Fletcher | Information inequality in the UK coronavirus communications crisis[END_REF]. The predominance of Facebook in the media landscape is confirmed by Parse.ly's dashboard, showing that the visitors to their 2500+ online media sites are referred by Facebook in 26% of the cases (only Google had a higher referral volume with 52% of the traffic) [START_REF]Parse.ly's Network Referrer Dashboard[END_REF].

These platforms have come under heavy criticism over the past few years for enabling the circulation of misinformation on massive scales [START_REF] David Mj Lazer | The science of fake news[END_REF][START_REF] Shao | The spread of low-credibility content by social bots[END_REF]. Facebook has publicly announced a three-part policy to fight against 'misleading or harmful content' : they remove harmful information, reduce the spread of misinformation and inform people with additional context [START_REF] Lyons | The Three-Part Recipe for Cleaning up Your News Feed[END_REF]. Facebook is indeed communicating regularly about deleting accounts spreading hate speech or incitation to violence, and about promoting official sources of information about elections, health or climate change [START_REF]Additional Steps to Protect Myanmar's 2020 Election[END_REF][START_REF]Stepping Up the Fight Against Climate Change[END_REF][START_REF] Gleicher | Removing Coordinated Inauthentic Behavior[END_REF][START_REF] Jin | Keeping People Safe and Informed About the Coronavirus[END_REF].

Regarding how they handle misinformation, Facebook has announced more specific measures:

-first to identify potential false news and seek the expertise of fact-checkers to review content and clearly label misinformation, and -second to ensure that fewer people see misinformation, and to take action against repeat offenders by reducing their distribution [START_REF]Fact-Checking on Facebook[END_REF].

The application of the first measure can be transparently verified on Facebook, with some posts being publicly labeled as misinformation. However, there is little data available to measure the efficiency and track the impact of the latter measure.

A study analysed the reach of a set of websites identified as sources of false stories on Facebook and Twitter from January 2015 to July 2018. They found that during the 2016 American elections, total engagement on Facebook and Twitter for these sites had more than doubled compared to pre-election levels. Following the election, however, Facebook engagements fell sharply, while Twitter shares continued to increase for these sites, suggesting that Facebook might have taken measures to contain misinformation [START_REF] Allcott | Trends in the diffusion of misinformation on social media[END_REF].

A more recent article measured the level of interactions on Facebook with articles from outlets that repeatedly publish false content from 2016 to 2020, and found results contrasting with Allcott and colleagues'. Although they did observe a decrease in the first and second quarter of 2017, interactions with 'deceptive' outlets have increased since, and are now 242 percent higher on Facebook than during the run-up to the 2016 election [START_REF] Kornbluh | New Study by Digital New Deal Finds Engagement with Deceptive Outlets Higher on Facebook Today Than Run-up to 2016 Election[END_REF].

Our analysis is based on Science Feedback's fact-checking dataset. Science Feedback is a scientific organization verifying the credibility of science-related claims and articles for a general audience. The organization tracks the most influential press articles or social media posts, invites scientists with domain expertise to evaluate their credibility and is one of Facebook's third-party fact-checkers [START_REF] Vincent | Science Feedback partnering with Facebook in fight against misinformation[END_REF]. We used the URLs marked as 'False' by Science Feedback (misinformation links) to investigate their reach on Facebook and Facebook's efforts to reduce the distribution of 'repeat offenders'. [START_REF]An Update to How We Address Movements and Organizations Tied to Violence[END_REF], consistently with Facebook's Dangerous Individuals and Organizations policy. Twitter and Youtube also announced to take action on activity associated with QAnon [START_REF]Managing harmful conspiracy theories on YouTube[END_REF][START_REF] Twittersafety | [END_REF].

In this section, we verify the impact of Facebook's policy in our dataset.

To that end, we rely on the 1,290 URLs labeled as 'False' by Science Feedback (i.e., we excluded the URLs labeled as 'Partly False', 'Missing Context', 'False headlines' or 'True'). The list was obtained on June 2, 2020. From the CrowdTangle API (using the /links endpoint) and using the minet library [START_REF]Minet: a webmining library and command line tool written in python[END_REF], we gathered the Facebook pages and groups that publicly shared at least one 'False' URL over a 12 months period (between June 2, 2019 and June 1, 2020). Due to the API limitations at the time, if a URL was shared in more than 100 posts, we collected only the 100 posts that received the highest number of interactions.

On August 27, 2020, we repeated the above operation with the then 1,691 URLs labeled 'False' collecting all the Facebook accounts that shared at least one of the links over a 12 months period (between September 1, 2019 and August 31, 2020).

We first filtered the Facebook groups and pages to keep only the ones that shared at least 10 different 'False' URLs (misinformation links), and manually identified the groups and pages linked to the QAnon theory. In the dataset collected in June, we found 12 accounts whose name was associated with Q or QAnon among the total of 204 accounts having shared 10 misinformation links or more. In the dataset collected in August, we found only 5 accounts linked to 1 for the full lists of names). The total number of followers of these accounts went from almost 400,000 in June to only 60,000 in August.

From the lists shown in Table 1, we can see that all the QAnon related groups can be automatically identified by searching groups containing the letter 'Q' in their name. To capture a broader set of accounts, we ran a 'Q' search on all the accounts having shared at least 3 misinformation links on June (1,384 accounts) and August (1,670 accounts). The list was then manually sorted to remove accounts with a 'Q' in their name that were not related to the QAnon theory (such as 'AMERICANS Against Excessive Quarantine!' or 'Info Pro-Trump Québec').

For this broader set of Facebook accounts, the number of QAnon related accounts has actually increased from 35 to 46 between June and August 2020, but their total follower count has decreased by half (Table 2). This suggests that Facebook has mainly removed popular QAnon accounts, while many new accounts have been created over the three months period.

Our results confirmed that Facebook suppressed many popular accounts linked to QAnon, some of which have been instrumental in spreading misinformation. Despite this policy, we still found 46 QAnon groups sharing 3 misinformation links or more after August 19, 2020. In a recent update to their official communication [START_REF]An Update to How We Address Movements and Organizations Tied to Violence[END_REF], Facebook announced they would act more firmly: "Starting today [October 6, 2020], we will remove any Facebook Pages, Groups and Instagram accounts representing QAnon, even if they contain no violent content. " Further data collection will be needed to track the impact of this stated policy.

A SUDDEN DROP IN THE REACH OF MOST MISINFORMATION ACCOUNTS

In this section, we turn to the overall evolution in the interactions generated by posts in groups repeatedly sharing misinformation during the COVID-19 pandemic.

To that end, we focused on the 90 Facebook accounts (75 groups and 15 pages) that have shared 15 or more different misinformation links in the data collected in August. From the CrowdTangle API (using the /posts endpoint), we collected all the posts they published between September 1, 2019 and August 31, 2020. The collection was performed between the September 1 and 7, 2020. We computed the daily number of posts, as well as the mean daily number of comments, shares and reactions (aggregating the 'like', 'love', 'favorite', 'ahah', 'wow', 'sad' and 'angry' reactions) per post, averaged over the 90 accounts (Figure 1).

During the period from March to June 2020, we observe a doubling in the number of daily posts, from 60 to a peak at 120 posts per day (Figure 1 middle panel), while the number of reactions, shares and comments per post vary both upwards and downwards alternatively (Figure 1 top panel). As a result, the number of reactions, comments and shares per day has risen from March to June 2020 (Figure 1 bottom panel), leading to an increase in total engagement for these accounts.

Surprisingly, we observe a sudden and large decrease in all three engagement metrics around June 9, 2020, with a decrease of 42% in the number of reactions per post, of 50% in the number of shares and of 46% for comments between June 8 and 10, 2020.

To verify whether this drop could be driven by the sudden disappearance of one (or many) popular group, we computed the same metrics aggregated over the 64 Facebook accounts that did not either appear or disappear between September 1, 2019 and August 31, 2020. We still observed a drop in engagement at the same date with the number of reactions per post declining by 42%, by 46% for the shares and 46% for the comments. We can thus reject this hypothesis.

To compare the Facebook pages and groups sharing misinformation repeatedly with a control set of accounts, we gathered Facebook pages and groups associated with 'mainstream news' outlets. A report from NewsWhip [START_REF]Coverage of the Coronavirus on Web and Social[END_REF] was used to identify the 10 media that communicated the most about COVID-19 online, i.e. NBC, The Daily Mail, CNN, Fox News, The Independent, BBC, The New York Times, The Washington Post, Yahoo and The New York Post. We searched on Facebook for the name of the outlets, and listed 10 pages and 6 groups with the verified 'blue check'. Using Crowd-Tangle, we collected all the posts published by these 16 accounts between September 1, 2019 and August 31, 2020. The collection was performed on the September 7, 2020.

Figure 2 displays the same statistics as for the 'misinformation accounts'. Contrary to what we observe for the 'misinformation accounts', there is no drop in reach around June 9, 2020 (dotted line on Figure 2) for 'mainstream news accounts'. As for trends, their daily number of posts remained stable throughout the period, while the number of engagements per post increased by roughly 100%. As a result, the total engagement for these accounts increased with the same proportion. We note that a post on a mainstream news account is generating about two orders of magnitude more reactions, shares and comments per day than a post on a misinformation account.

To investigate whether the 'June drop' could be explained by Facebook taking action against accounts that repeatedly shared misinformation, we correlated the magnitude of the drop with the number of misinformation links shared by a page and contrast this with other account characteristics such as the number of followers and the mean engagement (reactions + comments + shares) per post. The drop was measured for each account by the negative growth rate in the sum of reactions, comments and shares between June 8 and 10, 2020. We found no correlation between the drop and the number of followers (Pearson correlation coefficient: r = 0.11), the mean engagement per post (r = 0.09) or the number of shared misinformation links (r = 0.16). We thus cannot provide an explanation for how Facebook selected the reduced reach groups.

FACEBOOK MOMENTARILY REDUCING THE REACH OF SOME MISINFORMATION ACCOUNTS

According to Facebook, "Pages and websites that repeatedly share misinformation rated False or Altered will have some restrictions, including having their distribution reduced. " [START_REF]Fact-Checking on Facebook[END_REF]. Accounts repeatedly sharing misinformation are labeled 'repeat offenders', but in the official communication it is not clear whether a group is classified as such after sharing 2 misinformation links or more. A non-official source precised that "The company operates on a 'strike' basis, meaning a page can post inaccurate information and receive a onestrike warning before the platform takes action. Two strikes in 90 days places an account into 'repeat offender' status" [START_REF] Solon | Sensitive to claims of bias, Facebook relaxed misinformation rules for conservative pages[END_REF].

To verify this policy, we computed the daily number of reactions and comments per post for each of the 90 accounts spreading the most misinformation (Figure 3). The number of shares per post was excluded as it was more erratic, and thus less reliable to spot reduced reach periods.

For some accounts such as 'Chemtrails Global Skywatch', 'Women SCOUTS for TRUMP (c)', 'THRIVE MOVEMENT' or 'Drain The Swamp', we observe periods during which the numbers of reactions and of comments per post were unusually low for several weeks. These periods of reduced reach may correspond to a time when Facebook labeled the groups as 'repeat offenders'. Other accounts such as 'Conspiracy Theory & Alternative News' or 'The Shift -Being The Change' appear to have a more stable engagement per post throughout the year (Figure 3).

If the post sharing a 'False' URL got published after the fact-check for the corresponding link, we used the date of the post as the date of the strike. In the case when the account first shared a link, which was then fact-checked as 'False', the fact-check publication date was used as the strike date. The strike dates are shown as red lines on Figure 3.

From the list of strike dates, we computed the 'repeat offender' periods for each account using the following rule: at any given time, if an account has shared two or more misinformation links over the past 90 days, it was defined as a 'repeat offender' (periods shaded in red on Figure 3). One can see that the '2 strikes in less than 90 days = reduced reach' rule does not seem to apply, as most accounts display stable or increasing engagement levels during the periods shaded in red.

To further analyse whether the number of shared misinformation links could be correlated with observed periods of reduced reach, we devised a simple algorithm that detects the reduced reach periods objectively. The numbers of reactions and of comments per post were summed in a global engagement metric. When a period of 14 days had a mean engagement metric below the average engagement over a reference period divided by 1.5, and an average z-score inferior to -1 (i.e., the engagement values were on average lower than a standard deviation below the mean), this period was labeled as 'reduced reach', and shaded in green on Figure 3. We computed the reference average and standard-deviation over the period from September 1, 2019 to June 8, 2020, before the 'June drop' happened.

One can see that our rule is able to detect some identifiable periods of reduced interactions, although its specificity can surely be improved (Figure 3). We then computed the percentage of time spent in the reduced reach periods, as measured by our rule, between September 1, 2019 and June 8, 2020, and investigated whether it correlates with the number of strikes or other account characteristics. We found no correlation between the percentage of reduced reach periods and the number of misinformation links shared (r = 0.30), the number of followers (r = 0.07), or the mean engagement per post (r = 0.16).

CONCLUSION

Misinformation on social media is a potential threat to the fight against the COVID-19 pandemic. Using a fact-checking organization dataset, we investigated the reach of a set of accounts that repeatedly shared misinformation on Facebook over the past year, and tested whether Facebook's stated policies to tackle misinformation were applied to these accounts.

We compared the temporal evolution of engagement metrics of Facebook accounts spreading misinformation to accounts of 'mainstream news' outlets. While the daily engagement metrics of 'misinformation' accounts doubled during the first half of 2020, mainly because of an increase in posting, the 'mainstream news' accounts' activity remained stable and the total engagement with their posts increased. This observation draws parallels with another study finding that between 2015 and 2017, Facebook interactions for major and small news sites have remained relatively stable with a modest upward trend, while interactions for websites known to publish false information peaked during the 2016 election [START_REF] Allcott | Trends in the diffusion of misinformation on social media[END_REF]. A more recent study showed that during the second quarter of 2020, Facebook interaction levels with articles from all news sites (deceptive and legitimate) have overall increased. However, interactions with articles from 'deceptive sites' remained high during the third quarter of 2020, while interactions with articles from legitimate journalistic outlets fell close to their pre-pandemic levels [START_REF] Kornbluh | New Study by Digital New Deal Finds Engagement with Deceptive Outlets Higher on Facebook Today Than Run-up to 2016 Election[END_REF].

Regarding Facebook's actions against misinformation, some of our findings are consistent with Facebook's public announcements: we did witness that QAnon groups with a large number of followers have disappeared after August 19, 2020 [START_REF]An Update to How We Address Movements and Organizations Tied to Violence[END_REF], and that certain Facebook accounts instrumental in spreading misinformation had their reach momentarily reduced for several weeks. However, while Facebook publicly announced to apply reduced reach to 'repeat offenders' [START_REF]Fact-Checking on Facebook[END_REF], we did not observe a reduction in the number of reactions or comments per day for accounts that repeatedly share misinformation links, nor did we observe a correlation between the number of 'False' flag received by a page and the proportion of time spent with reduced reach.

We also observed on June 9, 2020 a large drop in engagement per post metrics for most misinformation accounts, but this 'June drop' does not correspond to any official communication on that matter. Further investigations will be needed to understand the nature and origin of this change.

One limitation of this study is that we only took into account the URLs labelled as 'False' by one fact-checking organization (Science Feedback), while Facebook partners with over 60 fact-checking organizations [START_REF] Rosen | An Update on Our Work to Keep People Informed and Limit Misinformation About COVID-19[END_REF]. Data from other fact-checkers would help to further investigate the relationships between 'False' flags and consequences for a Facebook account.

We hope these results can be useful to those working on addressing online misinformation. The code used to collect and clean the data, and to plot the figures is available on GitHub (the URL will be given later because the name of the GitHub account and of the contributors would impair the double-blind procedure). We have listed the URLs of the Facebook groups and pages collected in the GitHub repository so that the figures could be as easy to reproduce as possible.

Figure 1 :

 1 Figure 1: Temporal evolution of Facebook interaction metrics averaged for 75 Facebook groups and 15 pages repeatedly sharing 'False' URLs. The dotted line marks the date of June 9, 2020, when a sudden drop in reactions, shares and comments can be observed.

Figure 2 :

 2 Figure 2: Temporal evolution of Facebook interaction metrics averaged for 6 Facebook groups and 10 pages associated with 'mainstream news' outlets. The dotted line marks the date of June 9, 2020.

Figure 3 :

 3 Figure 3: Mean number of reactions and comments per post over the past year for 10 Facebook accounts among the 90 spreading misinformation repeatedly. Each red line at the bottom of a subplot represents the date of a known strike (a shared link labelled 'False' by Science Feedback), and the areas shaded in red represent the 'repeat offender' periods as defined by the '2 strikes in less than 90 days' rule. The areas shaded in green represent the 'reduced reach' periods automatically defined by a combination of a 'one standard deviation below the average' rule and a period average below a reference average.

Table 1 :

 1 List of QAnon related groups or pages that have shared 10 misinformation links or more found in the data collected on June 2 and August 31, 2020, with their number of followers in parentheses. The number of followers was retrieved at the time of the CrowdTangle request, which explains the differences between the June and August columns.

	QAnon related accounts	QAnon related accounts
	10+ misinformation links	10+ misinformation links
	June 2, 2020	August 31, 2020
	OFFICIAL Q / QANON (167,223)	Q The Greatest Story Ever Told (27,384)
	We Are Q (44,838)	Real Qanon Follow The White Rabbit (16,495)
	QAnon -Posts by Q (35,821)	Q ANON QUÉBEC (12,008)
	WWG1 WGA...Q (29,977)	QAnon Arts & Info Pub	WWG1WGA (2,162)
	QANON PIZZAGATE FULL DISCLOSURE GROUP (28,102)	Qanon Suomi (2,144)
	Q The Greatest Story Ever Told (23,539)
	Q The Great Awakening (19,462)
	Real Qanon Follow The White Rabbit (13,417)
	Q ANON QUÉBEC (10,958)
	Truth Bomb News! #Qanon #The Great Awakening (9,278)
	MemeLab / QAnon / Q-Map (9,168)
	QAnon 8ch Uncensored Research (7,727)
	12 accounts	5 accounts
	399,510 total followers	60,193 total followers
	2 FACEBOOK REMOVING QANON
	ACCOUNTS, POTENT MISINFORMATION
	SPREADERS
	On August 19, 2020, Facebook publicly announced having removed
	over 790 groups and 100 pages tied to QAnon, a growing right-
	wing conspiracy theory, for "hav[ing] demonstrated significant
	risks to public safety"

Table 2 :

 2 Number of QAnon related accounts that have shared 3 misinformation links or more and their total follower number, found in the data collected on June 2 and August 31, 2020.

	QAnon related accounts	QAnon related accounts
	3+ misinformation links	3+ misinformation links
	June 2, 2020	August 31, 2020
	35 accounts	46 accounts
	723,742 total followers 371,288 total followers
	the conspiracy theory among the total of 185 accounts (see Table

ACKNOWLEDGMENTS

To Guillaume Plique, Benjamin Ooghe-Tabanou and all the médialab technical team for their help. This research was supported by the Programme d'Investissements d'Avenir (ANR-19-MPGA-0005).


Andrei Mitriakov

Panagiotis Papadakis

Sao Mai Nguyen

Serge Garlatti

Staircase Negotiation Learning for Articulated Tracked Robots with Varying Degrees of Freedom

Tracked robots capable of negotiating 3D terrains require delicate control, most often tailored to a specific platform or setting. For staircase traversal in particular, autonomous robot behaviours are difficult to obtain due to the increased risk of accident and stochasticity. Based on a previously developed reinforcement learning based framework that allows learning staircase ascent for an articulated tracked robot, in this work we extend our work to allow also staircase descent and further investigate the role of a manipulating arm in the stability and smoothness of the traversal. By relying on a precise simulation of geometry and kinematics of a real robot, we demonstrate prototype policies for staircase ascent and descent, optionally under the influence of an integrated active arm and different penalty criteria. The obtained results are qualitatively and quantitatively compared and show that the robot can learn plausible behaviors effectively, when guided by appropriate reward and penalty criteria.

I. INTRODUCTION

Robots capable of 3D navigation bear a strong potential for applications pertaining to hazardous or extreme environments. In such cases, a resilient robot construction may compensate collisions with the environment as the robot passively negotiates the traversed terrain. On the other hand, applications where robot as well as environment safety is crucial require active control of the articulated robot parts.

Most previous works on the development of active 3D obstacle negotiation by tracked robotic vehicles are customized to specific platforms [START_REF] Otsu | Fast approximate clearance evaluation for rovers with articulated suspension systems[END_REF], [START_REF] Ilhan | Autonomous stairwell ascent[END_REF], inevitably relying on complex inverse kinematics and accurate estimates of environment geometry. Such solutions are not easily transferable to different platform/environment setups and developing alternative negotiation strategies is not straightforward. Additionally, the development of a robot controller necessitates domain expertise or experience which may not be easy or even feasible to acquire, e.g. for search and rescue environments [START_REF] Kruijff | Rescue robots at earthquake-hit mirandola, italy: A field report[END_REF]. Learning from demonstration [START_REF] Billard | Handbook of Robotics[END_REF] can alleviate some of these challenges yet providing good demonstrations is costly, partly because controlling multiple degrees of freedom (DOF) in parallel is not intuitive and hence cumbersome. To increase autonomy in the process of learning robot mobility, we perform the acquisition of such skills via reinforcement learning (RL) where we can favor the emergence of the desired control properties through the appropriate design of reward/penalization functions and state/action domains. Since learning in reality is risky and tedious due to the need to perform manual environment reset, we opt for learning such skills through simulation.

Lab-STICC, UMR 6285, F-29238, team RAMBO, IMT Atlantique Bretagne/Pays de la Loire, Brest, France A recent approach for simulating robot tracks that provides more realistic robot track simulation and has shown highly plausible results and faster simulation is the Contact Surface Motion (CSM) [START_REF] Pecka | Fast simulation of vehicles with non-deformable tracks[END_REF], which we employ in the present work. Relatedly, we recently developed a RL-based framework for tracked robots equipped with flippers, for learning ascent control policies of staircases of varying size [START_REF] Mitriakov | Staircase traversal via reinforcement learning for active reconfiguration of assistive robots[END_REF]. In continuation of that work, we present here our latest developments for learning more elaborate mobility skills that are altogether critical in enabling tracked robots to undertake the commonly required task of object search and transport. The main contributions of this work can be summarized as follows:

• We upgrade our previous RL-problem formalization to allow learning of staircase ascent and descent and further investigate the influence of an active arm during staircase negotiation. • We update our simulation model using the geometrical and physical properties of the robot Jaguar V4 with Arm (jaguar.drrobot.com/specification_ V4Arm.asp) and adopt the CSM approach for the simulation of robot tracks (see Fig. 1). The remainder of the paper is organized as follows. Section II presents most relevant works that deal with the problem of staircase negotiation for articulated tracked robots. In section III we augment our previous formalization [START_REF] Mitriakov | Staircase traversal via reinforcement learning for active reconfiguration of assistive robots[END_REF] to further allow learning of staircase descent and consider the influence of an articulated arm. Finally, in section IV the experimental setup is presented together with the obtained qualitative and quantitative results.

II. RELATED WORKS

The domain of Urban Search and Rescue (USAR) has drawn considerable attention to the problem of kinematic modelling in order to allow 3D terrain negotiation [START_REF] Martínez | Approximating kinematics for tracked mobile robots[END_REF]- [START_REF] Zhang | System centroid position based tipover stability enhancement method for a tracked search and rescue robot[END_REF]. Our particular focus here regards staircase negotiation, generally considered as an important milestone in enabling robots to explore 3D indoor environments.

1) Learning-free approaches: Relatedly, Mourikis et al. [START_REF] Mouriakis | Autonomous stair climbing for tracked vehicles[END_REF] presented an algorithm for autonomous stair ascent through the use of gyroscopic data and estimation of stair step edges. Stair descent was studied in [START_REF] Ben-Tzvi | A mobile robot with autonomous climbing and descending of stairs[END_REF] together with stair traversal with the aid of an actively rotating pendulum in a complex track system, yet the obtained behavior is specific to the particular kinematics and hence not generalizable. An approach using dense laser sensory data and passive flippers was proposed in [START_REF] Nagatani | Semi-autonomous traversal on uneven terrain for a tracked vehicle using autonomous control of active flippers[END_REF] where the robot orientates flippers tangentially to the traversing obstacle while being tele-operated. A planning and control method for 4-DOF tracked robots in autonomous ascent and descent of known staircase was proposed in [START_REF] Endo | Stair climbing control for 4-dof tracked vehicle based on internal sensors[END_REF] where the robot estimated its state through internal sensors. The main limitation of that work is that the robot can only negotiate staircases known beforehand. Finally, an elaborate analysis of robot stability and control was presented in [START_REF] Zhang | System centroid position based tipover stability enhancement method for a tracked search and rescue robot[END_REF], taking into account active arm adaptation during stair ascent and descent. The derived analytical solution is characteristic of the high complexity of direct modelling of the interaction between a multi-DOF robot and a negotiated obstacle. Still, it requires full knowledge of the platform kinematic, geometric and physical characteristics, it does not employ congruent control of flippers, main tracks and the arm, nor does it account for the influence of a transported object.

2) Learning-based approaches: Learning-based control approaches on the other hand, tend to be more straightforward when the dynamics of physical interaction are hard to be estimated. Authors of [START_REF] Paolo | Towards continuous control of flippers for a multi-terrain robot using deep reinforcement learning[END_REF] were among the first who treated the problem of staircase traversal via deep reinforcement learning in the end-to-end fashion. Using front and rear cameras they showed that the robot can learn to ascend a previously known staircase, yet at a high computational cost. Reinforcement learning was also applied to the problem of flipper configuration and compliance autonomous control by [START_REF] Zimmermann | Adaptive traversability of unknown complex terrain with obstacles for mobile robots[END_REF] in order to reduce the operator's cognitive load. That approach enabled negotiation of single palettes and uneven terrain obstacles, nonetheless, assuming that main tracks are independently controlled. Another more sophisticated approach was demonstrated in [START_REF] Pecka | Autonomous flipper control with safety constraints[END_REF] where authors incorporated constraints into the optimization problem of relative entropy policy search using a RL algorithm [START_REF] Deisenroth | A survey on policy search for robotics[END_REF]. Still, results were limited to the traversal of a palette.

Endowing a system with constraint respect such as stability can be performed in different manners. It is a common practice to introduce safety constraints into the RL algorithm [START_REF] Pecka | Autonomous flipper control with safety constraints[END_REF], [START_REF] Ha | Learning to walk in the real world with minimal human effort[END_REF], often giving promising results at the cost of exhaustive exploration effort. Alternatively, one can associate safety and reward inside a single function [START_REF] Zimmermann | Adaptive traversability of unknown complex terrain with obstacles for mobile robots[END_REF], [START_REF] Bagnell | Learning decision: Robustness, uncertainty, and approximation[END_REF], which raises the question of determining their relative influence. In the following, we describe our approach for RL-based staircase ascent and descent under the influence of an integrated arm transporting an object. The diversification of the task incurred by the inclusion of new DOF leads to a harder problem and higher risk of accident, requiring a more comprehensive treatment. In comparison with related works, our framework is easily applicable to different platforms, allows to generalize over a variety of obstacles, accounts for arm control and considers ascent and descent traversals. III. METHODOLOGY At time step t, the system being in the state s t ∈ S executes the action a t ∈ A with respect to its policy π, obtains a scalar reward r t ∈ R and transits into a new state s t+1 ∈ S. The total set of transitions from the episode start to the end is called a roll-out τ = {s 0 , a 0 , .., s T -1 , a T -1 } where T is its number of time steps. To solve this RL problem, we will use policy gradient algorithms where the policy is directly optimized. The main idea consists in performing policy ascent over policy parameters θ to maximise the expected gradient ∇ θ J(π θ ) return. In the sequel, we present the variables used to describe the problem of ascent and descent staircase negotiation learning. Then, we propose a reward function to entice stair traversal while taking into account the stability constraints.

A. Problem description

In our previous work [START_REF] Mitriakov | Staircase traversal via reinforcement learning for active reconfiguration of assistive robots[END_REF] where we opted for independent tracks and flipper control, the robot was able to learn a behavior required for accomplishing staircase ascent while respecting safety constraints. However, with the inclusion of DOF of an articulated arm, separate control of the main tracks with the arms actuators can be sub-optimal.

This motivated us to explore two types of action spaces; (i) when the robot controls 3 DOF consisting of the robot base linear velocity, front flipper and rear flipper angles and (ii) 3 + 2 = 5 DOF where the 2 additional DOF correspond to the joints of an arm. With reference to (i), the robot selects 3 action parameters, namely, front and rear flippers rotation angles and velocity of the base forming an action vector:

a = (ψ f ront a , ψ rear a , v a ) ∈ [ψ min a , ψ max a ] 2 × [v min , v max ] (1)
where ψ f ront a and ψ rear a are front and rear flipper angles, v a is the applied velocity. For case (ii), the previous action vector is extended with the joint angles of the arm:

(φ 1 a , φ 2 a ) ∈ [φ min 1 , φ max 1 ] × [φ min 2 , φ max 2 ] (2)
Fig. 2: Front and rear obstacle (step edge) coordinate systems that correspond respectively to the angles of the first and second joints (see Fig. 2).

The sensory space structure is set as follows. For case (i), the observation vector is represented as:

B. Reward function design

In this section we present reward functions used for learning staircase ascent and descent. In detail, we employ the same positive reward as in [START_REF] Mitriakov | Staircase traversal via reinforcement learning for active reconfiguration of assistive robots[END_REF] representing the total travelled distance on the stairs which drives learning, namely:

R tr (τ ) = Nτ t=1 x t D max ( 4
)
where R tr (τ ) is a cumulative return along a rollout τ whose maximal value can attain 1, N τ is the number of time steps in the rollout, x t is a travelled distance during one time step, D max is the maximal possible travelled distance. As stated earlier, learning with constraints can be performed through accounting for a negative reward (penalty) within an episode. We first propose a reward function that addresses the problem of robot stability for ascent and descent negotiation. Then, a reward function meant to reduce the pitch angular velocity experienced by the platform during descent is proposed. 1) Center of Gravity stabilization: The center of gravity of the robotic platform is known to be instrumental in preventing tip-over accidents and improving stability [START_REF] Zhang | System centroid position based tipover stability enhancement method for a tracked search and rescue robot[END_REF], [START_REF] Sasaki | Stair-climbing wheelchair with lever propulsion control of rotary legs[END_REF]. In [START_REF] Mitriakov | Staircase traversal via reinforcement learning for active reconfiguration of assistive robots[END_REF] we enabled a robot to learn a stable behavior by favoring robot poses with low center of gravity (COG) with respect to the underlying surface A that represents the Normalized Energy Stability Margin (NESM) [START_REF] Hirose | Normalized energy stability margin and its contour of walking vehicles on rough terrain[END_REF]. The presence of an arm nonetheless may place the COG low but also close to the "safety zone" border [START_REF] Zhang | System centroid position based tipover stability enhancement method for a tracked search and rescue robot[END_REF], without violating the NESM. To overcome this problem, we use the Stability Margin (SM) [START_REF] Mcghee | On the stability properties of quadruped creeping gaits[END_REF] that estimates the distance between the lower footprint and the COG projection on the ground. Since it is generally difficult to estimate where exactly the lower footprint touches the ground, we instead minimize the deviation of the COG projection on the stair surface C x from the projection point O of the centroid of the robot base. In this manner, we can guarantee that the robot respects the SM criterion. We consider that the arm has to place the COG as close as possible to the point O -the most stable point (see Fig. 3) along both the X and Y axes, optimizing both for the SM and the NESM. Thus, we minimize the deviation D = √ d 2 + h 2 where d and h are the distances between O and the projections of the COG on X and Y axes. This minimization serves the purpose of improving stability through the SM and NESM criteria but also redistribute symmetrically the weight forces exerted by the moving robot tracks to the staircase.

We further wish to penalize the robot when it loses stability and tips over that happens when the pitch angle of the robot reaches π/2. Thus, letting the COG deviation at every time step be D t , the penalty term is defined as:

r D t = -1, if tip over -K D * D t , otherwise (5)
where the scaling coefficient K D is used for normalisation as will be explained later in III-B.3.

Fig. 3: Side schematic view of the robot on a staircase

2) Drop impact reduction: Robot dynamics are different between ascending and descending a staircase, as gravity hinders the accomplishment of the former and it may conduct the robot to fall in the latter. Even if no accident happens, repetitive collisions caused by every step negotiation would have impact on the robot. Such events appear when the COG crosses step edges, at this moment, the base starts rotating downwards which yields an increase of pitch angular velocity that we refer as drop impact (bumpiness). As a way to prevent such behavior, the robot can learn to influence these dynamics through a pitch angular velocity based penalty. The robot may experience a front tip-over when the pitch angle of the robot reaches -π/2, that we wish to penalize as well. Letting W t denote the average pitch angular velocity, then the corresponding negative reward is:

r W t = -1, if tip over -K W * W t , otherwise (6)
3) Adjustment of scaling coefficient: The learned policy is largely determined by the reward function that guides learning of a specific task. At the same time, it can endow the policy with specific characteristics such as improvement of traction and respect of safety. While there is no ambiguity if we learn a specific task with only positive rewards, direct summing up the latter with another reward, for example the negative one, may mislead -bias the robot to concentrate on respecting auxiliary constraints rather than the main task.

To control the arbitrary bias of the penalty term, we normalize it by a scaling coefficient whose approximation can balance the learning of the main task along with the desired property. Towards this goal, we initiate a small set of trial experiments for the first n steps where we do not assign negative rewards, in order to acquire sufficient statistics related to the target value V , for example, COG distance D or angular velocity W in our case. This results in making the robot optimize its behavior only with respect to the main task as specified by the positive reward, setting aside the consideration of the penalty term. Upon the completion of the n th step, we calculate the "sub-optimal" mean target value V = n t=1 V t /n , then the scaling coefficient as follows:

K V = ( V • N episode ) -1 (7)
where N episode is the maximum episode length. Thus, the cumulative episode return could be expressed as follows:

R V (τ ) = R tr (τ ) + Nτ t=1 r V t ( 8
)
Normalization guarantees that the absolute episode return R tr (τ ) varies from 0 to 1 as well as the absolute episode penalty without tipping over penalization Rpenalty V (τ ). This can be easily seen by defining:

Rpenalty V (τ ) = Nτ t=1 V t V • N episode • N τ N τ = Vτ N τ V • N episode ( 9
)
where Vτ is the mean target value during the episode after trial experiments. We consider that in the beginning of learning the policy is sub-optimal, thus the "sub-optimal" mean target value V calculated in trial experiments relates to further target value mean observations as Vτ ≤ V . Since the rollout length does not exceed its maximal length N τ ≤ N episode , we can conclude:

Vτ N τ ≤ V • N episode (10)
It was finally observed that the tip-over penalty prevents learning of sub-optimal policies, therefore we added auxiliary negative reward -1 and end the episode in such cases, thus, R penalty V (τ ) ∈ [-2, 0]. Finally, we obtain the limits for the episode cumulative return as R V (τ ) ∈ [-2, 1].

IV. EXPERIMENTS

In this section we present the experiments that we conducted, allowing us to obtain staircase ascent and descent policies with desired properties. We also provide a qualitative analysis of learned behaviours in the supplementary video (also available at partage.imt.fr/index.php/ s/JBdmEXaWcjLmgmB/download).

1) Environment set-up: We developed a simulation environment as illustrated in Fig. 1, using the physics-based Gazebo simulator (gazebosim.org). Staircase size was varied in ranges corresponding to real-world staircases (cf. [START_REF] Mitriakov | Staircase traversal via reinforcement learning for active reconfiguration of assistive robots[END_REF] for details), allowing to learn behaviours in randomly generated staircases further taking into account the influence of noise in the state estimation process.

Furthermore, we increased the maximum total number of steps from 5 to 10. More importantly, we have shifted from a wheeled-based simulation of robot tracks to the more realistic CSM model [START_REF] Pecka | Fast simulation of vehicles with non-deformable tracks[END_REF] that is more lightweight in terms Fig. 4: Illustration of stair traversal task set-up of surface collision calculations and increases the level of accuracy. We instantiate a model of the robot Jaguar V4 and add a manipulator arm platform (see Fig. 1) incorporating its geometrical and weight parameters. To study the influence of transporting objects we perform learning with a cargo added to the robot end-effector that constitutes 10% of total robot mass and the heaviest permissible load, then we show resilience to its variation via test roll-outs with cargo masses that constitutes 5% and 15% of total robot mass.

A two-layer perceptron is used for policy representation. Its parameters are optimised using Proximal Policy Optimization [START_REF] Schulman | Proximal policy optimization algorithms[END_REF]. In the beginning of the ascent tasks, the robot starts at the start point (see Fig. 4) being orientated towards the stairs. It has to traverse the distance from the start to the finish line. For the descent tasks, start and goal positions are swapped. We assign both positive and negative rewards from the beginning of the episode to its end. The episode ends when either the robot reaches the finish line, goes out of the training zone (rectangle defined by red, blue and green lines), tips over or number of time steps exceeds the maximal episode length. We studied learning a total of 5 variants of the staircase traversal task, distinguished by the staircase negotiation direction, DOF involvement and penalty optimization criterion as summarized in Table I.

For all experiments the robot arm starts with the same, vertically stretched, initial configuration perpendicular to the robot base (see Fig. 1 middle). We select this initial arm pose as it does not alter the COG projection of the robot when it moves on flat terrain but it may severely hinder staircase traversal and should therefore incite the robot to learn to move the arm at a better pose. We recall that in the case of a 3DOF action space, the arm remains fixed in this pose.

2) Performance for staircase ascent: Three independent learning trials were performed for each task (i) and (ii), after which we average the obtained performances. Fig. 5 (a) presents the average over all trials of the exponentially smoothed cumulative reward with min-max bands. The smoothing factor is set to 0.95, considering the trade-off between curve smoothing while highlighting local changes. At the end of learning, a total relative reward gain of 0.21 is observed between the tasks, clearly suggesting that the robot learns to control the arm in a way that optimizes the respective criterion. Cumulative reward curves do not reach their maximum positive cumulative return value of 1 because of the minimal COG bias that results from its inability to put the COG closer to the base centroid than a certain distance.

It is worth noticing of the considerable distance between the two curves by the end of learning, explained by the fact that the robot with static raised arm and only using its flippers, is unable to attain the same level of positive reward because of the higher COG deviation. Fig. 5 (b) shows how the COG deviation evolves during the learning process. We can see that the robot learns its dynamics in both tasks. It shows that in both cases the robot converges to a better behaviour in a similar pace. Eventually, the arm control allows to decrease the COG deviation by 0.06m compared to the beginning of learning. This figure contains the COG evaluation curve obtained for the task (ii) without reward shaping. This illustrates how absence of reward scaling worsens final optimal control policy. We can see that the sub-optimal policy increases the COG deviation and converges to the mean value which exceeds the corresponding optimal one by 0.15m.

3) Performance for staircase descent: Cumulative reward curves are presented in Fig. 6a. We can see that the reward for all tasks reaches approximately the value 0.6 and does not attain the maximal reward 1 because of constant presence of the minimal negative reward. Fig. 6b shows the evaluation of the COG deviation during learning. As expected, we observe that the most unstable robot behavior is obtained for task (iii), where the reduction of COG distance due to the flipper control from the centroid projection point is low. Results of pitch angular velocity optimization with moving arm (iv) improves stability and the robot has further achieved to reduce the COG deviation by a total of 0.02m. Nevertheless, direct minimization of the COG distance (v) provides the best overall results, wherein the COG decreases by more than 0.03m and attains the lowest absolute value among tasks.

The pitch angular velocity evaluation is presented in Fig. 6c. We can see that COG and pitch angular velocity optimizations (iv) and (v) behave quite similarly, with the former yielding a slightly lower angular velocity. Comparing it with the performance of task (iii), we could claim that the mean pitch angular velocity is smaller if the arm does not move. Also, it may seem that the robot control policy fails to converge to an optimal behaviour but this could be partially explained. We have seen that the initial, vertically stretched arm performs worst of all in terms of stability easily leading to front tip-over, end of episode and penalization by -1. Thus, starting in the vertical arm position, the robot would experience more tip-overs that drives the arm control to more stable configuration even if it increases pitch angular velocity during movements. This seems a plausible behavior in reality, as we would prefer to undergo small bump impacts due to increased pitch angular velocity instead of a drastic accident.

4) Overall performance: We refer the interested reader to the qualitative results provided in the accompanying video, which presents the policies learned for the ascent and descent tasks in simulation. Policy learning time is constrained by action execution duration in the simulation. Using a contemporary machine, a 10000 time step simulation required approximately 20 minutes. As an empirical check that our simulation environment is realistic, we also transferred a learned policy from simulation to a real robot in an example staircase for the task (iii) policy.

Fig. 8a presents mean values with min-max of the COG distance during a test episode for ascent tasks (i) and (ii). We can see that after the initial phase of the task progress, i.e. from 30% of the ascent of the staircase, the COG deviation for the policy (ii) is lower than for the policy (i), further illustrated by Fig. 9 which presents mean values of COG deviations and angular velocities with their standard deviations during simulation test deployment for corresponding experiments of Table I. The mean COG deviation of the policy (i) is higher of the policy (ii) by 0.03m. Fig. 7 provides indicative snapshots of robot configurations during ascent (a) and descent (b) transitions while respecting the COG criterion. We can see that the robot pushes its arm in front during ascent owing to which tip-over of the platform is largely avoided. The front flippers are raised during ascent while rear flippers are pushed down. Such configuration improves robot traction with front and rear step edges.

Mean transition curves given in Fig. 8b show the evolution of the angular velocity during descent negotiation. To evaluate policy effectiveness in the descent tasks we compare quantitatively corresponding policies with the help of Fig. 9. Angular velocity optimization shows better results for the policies (iii) and (iv) in contrast to the policy (v) which is correspondingly higher by around 0.04rad/s and 0.05rad/s. Flipper rotation angles are reversed in the manner that front and rear flippers touch correspondingly lower and upper obstacles making transitions smoother. Thus, front flippers prevent the robot from dropping forward while rear flippers allow the robot to descend smoother from the rear step edge. We favor pitch angular velocity reduction in the descent tasks (iii) and (iv), but the stability (v) remains important.

In reality, we would prefer to expose the robot to some drop impacts rather to tip-over that could damage the robot.

We have performed additional tests in the ascent and descent tasks for the same policies when the end effector holds loads constituting 5 to 15% of the robot mass. We can observe from Fig. 9 that the robot is capable of controlling the COG deviation if load mass changes in the ascent tasks. For example, the robot controls the arm in a manner that it improves the target value for tests with loads of 5, 10 and 15% of the robot mass by 0.046, 0.044 and 0.083m in comparison with the stretched arm tests. In the descent tasks, the COG deviation is the highest in the task (iii) for all load types. Then, it is decreased if the robot controls its arm to minimize the angular velocity, but the minimal target value corresponds to the policy of the task (v). The mean angular velocity decreases along with load mass augmentation for all tasks. We can conclude that the angular velocity is smaller for tasks (iii) and (iv) than for task (v).

V. CONCLUSION

We have presented an improved RL-based framework for the problem of control policy learning during staircase descent and ascent, further investigating the influence of an integrated arm. Within 150 episodes, the robot is able to We are currently working on transferring the learned behavior to the real robot, integrating auxiliary components such as robot localization while we further consider comparing against conventional control or other learning-based techniques to different platform configurations.

Fig. 1 :

 1 Fig. 1: Jaguar V4 Manipulator with raised arm (a), the corresponding CSM simulated model (b) and simulated ascent staircase negotiation using a learnt policy (c)

  of the robot centroid relative to the next and previous nearest step edge along the X-axis of the robot as shown in Fig. 2, with p f ront y , p rear y corresponding to the relative distances along the Y -axis and v s being the current linear velocity. The state further includes ψ f ront s and ψ rear s as the current flipper angles. For case (ii), the state vector is extended with the arm joint angles φ 1 s , φ 2 s .

Fig. 5 :

 5 Fig. 5: Ascent task learning analysis

Fig. 6 :

 6 Fig. 6: Descent task learning analysis

Fig. 7 :Fig. 8 :

 78 Fig. 7: Robot during transitions on the staircase; front arm configuration (a), rear arm configuration (b)

Fig. 9 :

 9 Fig. 9: Performance of learned policies with varying weight of transported object

TABLE I :

 I Description of tasks

	Task id Direction DOF Criterion
	i	Ascent	3	COG
	ii	Ascent	5	COG
	iii	Descent	3	Ang. vel.
	iv	Descent	5	Ang. vel.
	v	Descent	5	COG

VI. ACKNOWLEDGEMENTS

The work is performed in the context of the project RE-ACT, project VITAAL and is financed by Brest Metropole, the region of Brittany and the European Regional Development Fund. Sao Mai Nguyen acknowledges the support of U2IS, ENSTA, IP Paris and of Inria FLOWERS, France.