As I write this contribution to this ALH forum in the spring of 2020, the world is facing the pandemic of coronavirus Covid-19, one of the most challenging global health crisis in recent history. For a few months, while countries have gone into lockdown one after the other, before beginning to ease restrictions in an equally uncoordinated fashion amidst fears of a second wave of infections, life has come to a relative standstill, initially bringing with it, at least for me, an But is to "suspend the Breath" indeed "the most we can," as literary scholars, in a time of crisis?

Is literature merely the record of our helplessness in the face of the unfolding of history and the unravelling of the world ?

Although initially resonant in this respect, while time itself seemed to suspend during the first weeks of the pandemic, Dickinson's antiphrastically " [i]gnorant" claim also speaks to the poem's knowledge of a provisional and precarious "balancing" between past and future, brought about by the confrontation between "Life" and "Death" which "Crisis" consists in-a "balancing" which threatens to turn into a toppling at any moment and which literature registers and criticism arguably seeks to account for. In this view, the function of literary criticism is none other than to read the "great variety of morbid symptoms" which, as Gramsci famously observed, appear during the crisis' "interregnum," while "the old is dying and the new cannot be born" (Selections 276). Despite recent calls to move beyond symptomatic readings in the field of literary studies, literary history reminds us forcefully at this particular historical juncture, as pleas for a "new normal" vie with pressures to go back to "business as usual," that it does offer a rich archive of such moments of crisis, where signs and figures are seen "Wandering between two worlds, one dead, / The other powerless to be born," as Matthew Arnold put it in 1855 in "Stanzas from the Grande Chartreuse" (301), a decade before he published his landmark meditation on "The Function of Criticism at the Present Time."

Whereas some texts surely are successful at grasping "the auspicious signs of the coming days[] as they glimmer already" in the present (Emerson,"The American Scholar" 68) or the fateful tokens of a world on the brink of collapse, literary historians however flesh out after the fact the diagnosis that these texts make, and almost inevitably so, once history has indeed unfolded, 3 in a belated recognition of the potential of literature to illuminate its own time (e.g., Pétillon).

The gap, in this account, between literature's timeliness and the historical hindsight that underwrites criticism foregrounds more generally the fraught question of the relation between literature, criticism, and the times, between the literary past and the critical present, which the pandemic compels me, as a nineteenth-century Americanist, to reconsider with renewed urgency. By doing so, the crisis may be seen to fashion its own critical model, as one in which it is the function of criticism to respond to the exigencies of the times and to meet the fast pace of our constantly changing circumstances-even when those seem for a while to elongate and suspend time-in order to reflect on, and emphasize, the resonance and the relevance of literary history in the present. This model of literature's and criticism's timely relevance intervenes in two conversations animating literary studies for a long time. One pits presentist against historicist conceptions of literature and literary criticism; the other wonders whether and how literature can be made to matter, both within and outside the academy. The terms of the first debate are by now familiar. Annales historian Marc Bloch offered a standard version of the presentist side of the argument in the opening pages of Strange Defeat (1946), his account of the French army's debacle during the Battle of France in 1940 and a text which has enjoyed renewed currency in France during the pandemic for its diagnosis of the state's failure to prepare adequately for a shattering crisis. "[W]e can truly understand the past," Bloch wrote, "only if we read it by the light of the present" (2). Historicist-minded critics on the other hand usually discard such presentist approaches as, in fact, unhistorical and reductionist, taking exception to their "naïve tendency . . . to read their own period's assumptions and values into the texts and events of the past" (Insko 2019, 14). By contrast, historicism customarily deploys a discourse of otherness between past and present, out of a proclaimed respect to the past's utter difference from the present. As L. P. Hartley memorably remarked in the opening to The Go-Between (1953): "The 4 past is a foreign country; they do things differently there" ( 5). Yet this ethics of difference remains subtended by a position of intellectual superiority, whereby present readers consider themselves better equipped than their historical counterparts to uncover the uniqueness of the past and pierce its unspeakable secrets. Interestingly, however, this superiority complex is something that historicists share with presentists who, following "a Whiggish sense" of history, also posit that "the anxieties of the past" have largely remained obscure to those who experienced them, but can be helpfully illuminated by the concerns of the present (Hutner, "The Pleasure of the Past in the Present" 10). Historicism and presentism thus come across as the two sides of the same critical coin: both claim, albeit for different purposes, that we read the past from the present, whether the reading produced ultimately heightens or elides the difference between past and present. One of the functions of literary criticism in this regard may well be to imagine ways of "[n]icely balancing," per Dickinson, these two extremes, between the presumption of the past's distinctiveness and its indistinguishable collapse with the present, and to recognize the dialectics of distance and proximity that inescapably underscores our readerly relation to literary history.

From this perspective, both the somewhat naïve presentism that criticism ought to eschew and the more refined presentism that haunts historicism are also symptomatic of a deeper desire: that literary criticism and literature in general continue to matter in our time. The complex of critical superiority vis-à-vis past texts therefore appears to be underwritten by its contrary: a complex of disciplinary inferiority, a dogging anxiety that literature and literary criticism be condemned as irrelevant. From Rita Felski's Uses of Literature (2008), to Italo Calvino's 1986 book by the same title, back to Emerson asking about "the right use" of books in his 1837 address "The American Scholar" (57), the inextricable issues of the uses of literature and the functions of criticism have been central to the field of literary studies, as it "had to grapple with questions of legitimation" (Fluck,"Shadow Aesthetics" 11). In "Experience" 5 (1844), Emerson himself ventriloquizes the classic charge that criticism is of no practical use, which the outbreak of the Covid-19 pandemic confronts me with in turn: "But what help from these fineries or pedantries?" he asks. "What help from thought? Life is not dialectics. We, I think, in these times, have had lessons enough of the futility of criticism" (478). In the wake of the New Historicist turn, discussions of the relevance of literature and literary criticism have often wagered on the politics of literary texts as providing a critique of a world increasingly driven by the forces of capitalist exploitation while modeling alternative forms of commonality, in an effort to claim thinking and writing as a ways of acting, pace Emerson, then, for whom "[o]ur young people have thought and written much on labor and reform, and for all that they have written, neither the world nor themselves have got on a step" ("Experience" 478). It is the case, though, that Emerson's own disavowal of thinking as ineffective is complicated, if not altogether challenged, by the essay's assertion of its own thinking procedures and its foregrounding of the community that this thinking produces ("We, I think"). Yet the kind of accusation that "Experience" voices in the first place has continued to find its way into academic discourse to this day. In one of the latest rejoinders to date, Victorian scholars Elaine Freedgood and Michael Sanders remark sardonically that: "We will not defeat neoliberalism with literary theory" ("Response" 118)-nor will we, I may add, defeat a pandemic, but it is surely not the issue and literary criticism might nonetheless provide useful interpretive procedures to understand, and maybe conceptual tools to resist, some of the more dire social and political consequences of the economic, environmental, and health crises we live through.

Challenging however the relevance of relevance as the index of scholarly endeavor, Freedgood and Sanders also note, more pointedly perhaps, that it is "never value-free" ("Response" 118), and thus never entirely divorced from the logic of capitalism that it seeks to expose. In a similar vein, Elisa Tamarkin helpfully problematizes the notion of relevance as, on the one hand, implicitly tied to notions of utility and marketability and, on the other hand, as being a function 6 of timeliness and immediate interest ("Reading for Relevance"). The dual meaning of currency, as indexing the course of time and the circulation of money, contemporaneity and profitability, speaks powerfully to the contradictory measure of literature's relevance, which aims to defy, but simultaneously reinscribes, the laws of the marketplace.

In a related attempt to reclaim "critical theory in dark times" from the accusation of irrelevance, Wendy Brown gives the idea of relevance a further turn of the screw by theorizing it provocatively not as "punctuality" but as "untimeliness." Noting that from Arendt to Brecht and back to ancient Greece, the notion that we live in "dark times" has been a recurrent one, she observes that the phrase names "an experience of darkness . . . rife with diabolical forces that can neither be mastered or comprehended, forces that frighten as they spook and heighten a felt impotence," leaving us "disoriented, frightened, and stumbling in the dark" ("Untimeliness and Punctuality" 9-10). Writing in 2005, Brown identifies "terrorism" and "empire" as the main causes of the specific darkness of our contemporary times, yet her description is equally fitting for the complex sense of panic, confusion, numbness and anxiety The idea that we ought to "grasp the times by thinking against the times," which Brown formulates through Nietzsche and Benjamin, resonates with Giorgio Agamben's own theorizing of contemporaneity as untimeliness, which he draws from Roland Barthes's reading of Nietzsche. "Contemporariness," Agamben writes, "is that relationship with time that adheres to it through a disjunction and an anachronism." And he concludes with the paradox that: "Those who are truly contemporary . . . are thus in this sense irrelevant" ("What Is the Contemporary?" 11).

The dialectics which consists in conceiving of criticism-literary and otherwise-as relevant because it is irrelevant, and therefore as constituting a form not of inaction but of paramount action, is one that, as I argue elsewhere ("The Dial"), Emerson and his fellow Transcendentalists repeatedly mobilized. In a journal entry of 1847, for instance, Emerson dismissed the imperative of "centrality," or relevance, and claimed "irrelevant" and "impertinent" reading as offering fruitful vistas on "the hour that passes" (JMN 10: 34-5). In "Self-Reliance," he further equated living "in the present" to living "above time" (270). Rather than a flight from history, as has often been argued, the insistence on living "above time" actually registers-or so the critical consensus now goes-"an ability to live . . . fully in the present tense" (Guthrie 93; see also Allen 186-216;Insko 2019, 93-126). Yet this ability remains predicated on a paradox, as well as on an important tropological shift, whereby the relation of the individual to the present is conceived spatially rather than temporally, and specifically in terms of spatial distance. To live fully in the present thus means to be at a remove from it and only the true contemporary is able to discern and inhabit this gap within time itself.

For Emerson-as for Agamben for that matter-this singular ability is the preserve of the poets.

8

They are the true contemporaries, for their powers of vision and expression enable them to see and say what others cannot. They deserve therefore a better mode of criticism than the one they are routinely subjected to:

Criticism is infested with a cant of materialism, which assumes that manual skill and activity is the first merit of all men, and disparages such as say and do not, overlooking the fact that some men, namely poets, are natural sayers, sent into the world to the end of expression, and confounds them with those whose province is action but who quit it to imitate the sayers. ("The Poet" 449) Sublating his Transcendentalism as in fact a superior form of materialism, Emerson denounces the blind spot of traditional criticism, which mischaracterizes both sayers and doers and "confounds" the ones with the others, all the while offering a reinvigorated "legitimation" of both poetry and criticism as more authentic forms of social and political work ("The Poet" 459).

Readings of this sort have certainly helped to recover the political force that drives Emerson's philosophy and to redeem him from the charge of inaction. They have also helped, by the same token, to warrant the relevance of many of our politicized readings of his works, and of nineteenth-century American literature more generally (e.g., Castronovo and Nelson). This kind of critical apparatus, where the function of criticism is to extract from literary texts a critical politics that eventually speaks both to these texts' historical context and to their readers' present circumstances, has been put to productive use from the very beginning of the pandemic of Covid-19, as had been the case following the high point of the AIDS epidemic in the 1980s and 1990s (e.g., [START_REF] Wald | Contagion and Culture[END_REF]. Building on advances in the field of medical humanities, Sari Altschuler and Elizabeth Maddock Dillon for instance have put together from the start of the pandemic a collaborative "Humanities Coronavirus Syllabus" with AOV 16 June 2020 9 a view to "help us think and teach about contagion, global health, and community in a time of social distancing and fear" (1). The reading list includes, for the US in the long nineteenth century, novels and tales by Charles Brockden Brown, Edgar Allan Poe, Nathaniel Hawthorne, Elizabeth Stuart Phelps, or Jack London-about smallpox, yellow fever, cholera, and plague outbreaks or the wearing of masks in the public space-along with critical studies demonstrating the longstanding entanglement of health and politics, as well as race, gender, and class, and exploring the narratives, characters, and tropes that have emerged and circulated in relation to the spread of infectious diseases (e.g., Otis 2000;[START_REF] Wald | Contagious: Cultures, Carriers, and the Outbreak Narrative[END_REF]).

Other resonant possibilities from the nineteenth-century literary archive would include Louisa May Alcott's Hospital Sketches (1863) and its grim satire of "Hurly-burly" hospital during the Civil War, where "disorder" (72), contagion, patient priorization, and death on the ward are the sad norm amidst acts of medical and nursing heroism, or Harriet Jacobs's Incidents in the Life of a Slave Girl (1861), in which the trope of slavery as a "poisonous" disease ( 53) and the space and the figure of the "loophole of retreat" (91-94) help to put both the racial politics of health and life under lockdown into perspective. Another possibility still, though perhaps more oblique-or irrelevantly relevant-would be Emerson's 1851 "'Address to the Citizens of Concord' on the Fugitive Slave Law," at the start of which he declared:

We do not breathe well. There is infamy in the air. I have a new experience. I wake in the morning with a painful sensation, which I carry about all day, and which, when traced home, is the odious remembrance of that ignominy which has fallen on Massachusetts, which robs the landscape of beauty, and takes the sunshine out of every hour. I have lived all my life in this State, and never had any experience of personal inconvenience from the laws, until now. They never came near me to my discomfort before. I find the like sensibility in my neighbors. And in that class who take no interest 10 in the ordinary questions of party politics. There are men who are as sure indexes of the equity of legislation and of the sane state of public feeling, as the barometer is of the weight of the air; and it is a bad sign when these are discontented. For, though they snuff oppression and dishonor at a distance, it is because they are more impressionable: the whole population will in a short time be as painfully affected. (54-5) What remains for Emerson a metaphor, however embodied-slavery troped as contamination-may read, for those who have experienced the pandemic of Covid-19 as well as other epidemics, as a literal description. Figured as malaria, the bad air of slavery is no longer circumscribed to the swamps of the Southern US plantation world: it has reached the heart of Massachusetts and now threatens to affect or infect the nation's whole population" with its "painful sensation." Yet that model of contamination actually reads both ways for Emerson: as a looming threat but also as a promise of empowerment. Describing the horrors and the pain of slavery in 1844, on the occasion of the anniversary of the emancipation of slaves in the British West Indies, he concluded: "Well, so it happened; a good man or woman, a country-boy or girl, it would so fall out, once in a while saw these injuries, and had the indiscretion to tell on them.

The horrid story ran and flew; the winds blew it all over the world." ("An Address" 10) In this instance, airborne contamination by word of mouth turns into virtuous and global contagion rather than deadly infection, as the "horrid story" touches those who hear it and transforms them, through the operations of sympathetic exchange, into full-blown abolitionists and activists. The germs of slavery can thus potentially become their own cure and their circulation "all over the world," though an "infamy" and an "ignominy," is also a boon, insofar as it fuels the cause of abolition.

Rather than to "suspend the Breath" in a time of crisis, as Dickinson seems to recommend (F1067), Emerson emphasizes what Cynthia Davis has called "the utopian AOV 16 June 2020 11 possibilities of contagion" ("Contagion as Metaphor" 831) and invites us to breathe in the blowing "winds" of history and change, all the while warning us of the dangers of the bad air of slavery. Emerson's entanglement of contagion and race, as well as his insistence on the viral power of words and stories, sound an unexpectedly compelling echo as the unfolding of the pandemic of Covid-19 intersects with the wave of demonstrations across the Western world in the wake of the death of African American George Floyd during a police arrest in Minneapolis in May 2020. In the space of a few weeks, the world has moved from panic following the outbreak of the epidemic to paralysis under lockdown life, to civil unrest in support of the Black Lives Matter movement. Tragically and soberingly fast-changing circumstances indeed, yet ones with which American literature continues to resonate for us, as it captures the "air" or the atmosphere of history and equips us with models to take it in, while "balancing," as Dickinson would say, the risks but also the potentialities of our exposure. Such is, I believe, the (un)timely relevance of American literature and American literary criticism at the present time.

Works

  been revised and published as "American Literary Criticism in a Time of Pandemic" in American Literary Scholarship, Volume 34, Issue 1, Spring 2022, Pages 67-76, https://doi.org/10.1093/alh/ajab099

  uneasy sense of both numbness and urgency about what role literature and literary criticism might have to play in such a time of crisis. "Crisis is a Hair," Emily Dickinson writes, Toward which the forces creep Past which forces retrograde If it come in sleep To suspend the Breath Is the most we can 2 Ignorant is it Life or Death Nicely balancing. (F1067)

  that the pandemic of Covid-19 has elicited. She further argues that what makes critical theory necessary and necessarily untimely is precisely the imperative to resist the dominant conception of time-as-fear that the times and their frightful specters would impose upon us: To insist on the value of untimely political critique is not, then, to refuse the problem of time or timing in politics but rather to contest settled accounts of what time it is, what the times are, and what political tempo and temporality we should hew to in political life. Untimeliness deployed as an effective intellectual and political strategy, far from being a gesture of indifference to time, is a bid to reset time. Intellectual and political strategies of successful untimeliness therefore depend on a close engagement with time 7 in every sense of the word. They are concerned with timing and tempo. They involve efforts to grasp the times by thinking against the times. (4)

  Cited Agamben, Giorgio. "What Is the Contemporary?" Nudities. Trans. David Kishik and Stefan Pedatella. Stanford: Stanford UP, 2011. 10-19. Alcott, Louisa May. Hospital Sketches. Boston: James Redpath, 1863. Allen, Thomas M. A Republic in Time: Temporality and Social Imagination in Nineteenth-Century America. Chapel Hill: U of North Carolina P, 2008.


Aurélien Morel

Yakumo Kunimoto
email: yakunimoto@yahoo.fr

Alex Coninx
email: coninx@isir.upmc.fr

Stéphane Doncieux
email: doncieux@isir.upmc.fr

Aurelien Morel
email: aurelien.morel.arthur@gmail.com

Automatic Acquisition of a Repertoire of Diverse Grasping Trajectories through Behavior Shaping and Novelty Search

Grasping a particular object may require a dedicated grasping movement that may also be specific to the robot end-effector. No generic and autonomous method does exist to generate these movements without making hypotheses on the robot or on the object. Learning methods could help to autonomously discover relevant grasping movements, but they face an important issue: grasping movements are so rare that a learning method based on exploration has little chance to ever observe an interesting movement, thus creating a bootstrap issue. We introduce an approach to generate diverse grasping movements in order to solve this problem. The movements are generated in simulation, for particular object positions. We test it on several simulated robots: Baxter, Pepper and a Kuka Iiwa arm. Although we show that generated movements actually work on a real Baxter robot, the aim is to use this method to create a large dataset to bootstrap deep learning methods.

I. INTRODUCTION

Grasping is a mandatory step for many daily object manipulation tasks. We do it without even thinking about it, but despite this apparent simplicity, it is a challenging movement that has been studied for decades and is still not completely solved for robotic agents, in particular when the 3D model of the target objects are not known [START_REF] Billard | Trends and challenges in robot manipulation[END_REF]. If robots are to be deployed in our every-day environment, for instance as home assistants, they will have to deal with very diverse situations and will thus need a strong adaptivity. In this context, grasping synthesis algorithms that are robust and efficient on a large set of objects having diverse shape and weight will be required. Learning methods are expected to help improve the generalization ability of grasping controllers [START_REF] Sahbani | An overview of 3d object grasp synthesis algorithms[END_REF], [START_REF] Bohg | Data-driven grasp synthesis-a survey[END_REF], [START_REF] Kleeberger | A survey on learning-based robotic grasping[END_REF], but learning to grasp requires to face a major issue: grasping is a hard exploration problem, where only a small subset of a large policy space is relevant. It creates a challenge for exploration-based learning algorithms, as without strong constraints or prior knowledge on the task or on the policy space to explore, it is unlikely that a purely random exploration could discover relevant grasping motions. A learning algorithm may thus fail to bootstrap and find solutions to improve on, may it be through gradient descent or through a gradient-free trial and error process. A possible solution is to decompose the task and start by estimating object pose before determining where to grasp it [START_REF] Kleeberger | A survey on learning-based robotic grasping[END_REF], but these approaches require to know object models. Another approach consists in exploiting available knowledge about the end-effector. The features of a parallel-jaw gripper, for instance, can be used to limit the number of grasping candidates [START_REF] Kraft | Development of object and grasping knowledge by robot exploration[END_REF], [START_REF] Mahler | Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics[END_REF], [START_REF] Depierre | Jacquard: A large scale dataset for robotic grasp detection[END_REF]. Unfortunately, this strategy does not easily transfer to other kinds of end-effectors. Another strategy consists in constraining the grasping movements to an easier top-down grasping motion [START_REF] Levine | Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection[END_REF], but, by construction, it limits the range of possibilities. Human demonstrations can also be used to guide the learning process [START_REF] Lenz | Deep learning for detecting robotic grasps[END_REF], [START_REF] Sharma | Multiple interactions made easy (mime): Large scale demonstrations data for imitation[END_REF], [START_REF] Zhang | Deep imitation learning for complex manipulation tasks from virtual reality teleoperation[END_REF], [START_REF] Song | Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations[END_REF], but generating large enough datasets is costly and would need to be repeated for every new robot.

In this work, we introduce a method to autonomously build grasping datasets. The method can be applied to any kind of robot arm and end-effectors with negligible adaptation. It has been tested in simulation on grippers (Kuka Iiwa and Baxter robots) as well as on a multi-fingered hand (Pepper robot). The transfer to a real robot has been studied on the Baxter. The proposed approach relies on a Quality Diversity algorithm [START_REF] Pugh | Quality diversity: A new frontier for evolutionary computation[END_REF], [START_REF] Cully | Quality and diversity optimization: A unifying modular framework[END_REF] built on top of Novelty Search [START_REF] Lehman | Abandoning objectives: Evolution through the search for novelty alone[END_REF]. Previous works with these algorithms have generated repertoires of legged-robots movements [START_REF] Cully | Robots that can adapt like animals[END_REF] or ball throwing and joystick manipulation movements on a Baxter [START_REF] Kim | From exploration to control: learning object manipulation skills through novelty search and local adaptation[END_REF]. Novelty Search has been shown to have an efficient exploration ability as it samples uniformly in a given behavior space [START_REF] Doncieux | Novelty search: a theoretical perspective[END_REF], [START_REF] Doncieux | Novelty search makes evolvability inevitable[END_REF]. To deal with the sparsity of successful grasping movements, we introduce an improvement of its exploration skill through the simultaneous management of multiple behavior spaces. It drastically improves the number of grasping movements discovered. The approach generates sets of open-loop grasping movements for a given object at a given location (Fig. 1). It cannot be used to directly grasp unknown objects at any location, but can generate large-scale grasping datasets for deep learning approaches. Although the grasping movements are generated and tested in simulation, we show that the size and diversity of the dataset result in a large number of policies that transfer to the real world.

II. RELATED WORK

Large scale data collection can rely on human demonstrators observation [START_REF] Saudabayev | Human grasping database for activities of daily living with depth, color and kinematic data streams[END_REF], [START_REF] Ekvall | Learning and evaluation of the approach vector for automatic grasp generation and planning[END_REF], but these methods are hard to scale since they require large human resources, are biased by human semantic priors and are specific to one type of robotic arm. Automating the data collection is challenging given the sparsity of grasping movements, but it has been achieved in restricted planar grasp conditions [START_REF] Pinto | Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours[END_REF], [START_REF] Levine | Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection[END_REF]. These restrictions are sufficient to get a successful grasp rate of 10% to 30%, which allows to collect a sufficient dataset to train a neural network achieving grasp control from real video images [START_REF] Levine | Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection[END_REF]. Other approaches focus on parallel-plate grippers to define an autonomous grasping movement generator [START_REF] Depierre | Jacquard: A large scale dataset for robotic grasp detection[END_REF]. The approach introduced here does not constrain grasping movements and can be used directly on any kind of robot.

Discovering diverse solutions is an important challenge in a variety of Machine Learning fields. Novelty-based evolutionary methods [START_REF] Lehman | Abandoning objectives: Evolution through the search for novelty alone[END_REF], [START_REF] Pugh | Quality diversity: A new frontier for evolutionary computation[END_REF], [START_REF] Mouret | Illuminating search spaces by mapping elites[END_REF], as well as related goal exploration process methods [START_REF] Forestier | Intrinsically motivated goal exploration processes with automatic curriculum learning[END_REF], were designed to tackle this problem and illuminate search spaces. Novelty search, one of these approaches [START_REF] Lehman | Abandoning objectives: Evolution through the search for novelty alone[END_REF], was actually shown to tend towards a uniform sampling in a user-defined behavior space [START_REF] Doncieux | Novelty search: a theoretical perspective[END_REF], [START_REF] Doncieux | Novelty search makes evolvability inevitable[END_REF], a property that is hard to get given the complexity of the mapping the policy parameter space and the behavior space. These methods have been used to generate repertoires of primitive actions for robot locomotion [START_REF] Mouret | Illuminating search spaces by mapping elites[END_REF], [START_REF] Cully | Robots that can adapt like animals[END_REF] or simple manipulations with a robot arm [START_REF] Kim | From exploration to control: learning object manipulation skills through novelty search and local adaptation[END_REF]. The generated repertoires have already been used to bootstrap deep learning approaches [START_REF] Jegorova | Behavioral repertoire via generative adversarial policy networks[END_REF], [START_REF] Keller | Model-Based Quality-Diversity Search for Efficient Robot Learning[END_REF]. We propose to extend novelty-based repertoire generation methods with exploration of multiple behavior spaces such as in [START_REF] Doncieux | Open-ended evolution with multi-containers qd[END_REF], [START_REF] Cazenille | Ensemble Feature Extraction for Multi-Container Quality-Diversity Algorithms[END_REF] but with an additional focus on setups with very sparse interactions. We then apply it to grasping movements generations.

III. METHOD

Fitness shaping is a convenient mean to turn a sparse reward into a dense reward in order to facilitate exploration [START_REF] Ng | Policy invariance under reward transformations: Theory and application to reward shaping[END_REF]. Novelty-based algorithms algorithms are not driven by reward and therefore do not suffer from the sparse reward problem, but if the behaviors they look for are very sparse in the space of all possible behaviors, as is the case for grasping, a similar issue arises as the search will have trouble bootstrapping and discovering relevant solutions. In the previously studied robotics tasks such as locomotion, navigation and ball throwing, all policies (even randomly generated ones) yield a valid behavior: the robot reaches some final position, or the ball reaches some location. The first discovered behaviors are usually very simple (such as remaining very close to the start position or just dropping the ball), but they allow the algorithm to bootstrap and then iteratively discover more novel and higher quality policies.

By contrast, trying to bootstrap grasping policies with random generation in a large policy space with little or no prior knowledge will mostly produce behaviors that do not engage the target object at all.

To mitigate this issue, we introduce an approach for behavior shaping that has a goal similar to fitness shaping, but in behavior spaces. The idea is to introduce new behavior spaces whose exploration aims at facilitating the generation of "target" behaviors, in a curriculum-like way. Building upon the ability of Novelty Search to continuously generate novel solutions [START_REF] Doncieux | Novelty search makes evolvability inevitable[END_REF], Novelty Search with Multiple Behavior Spaces (NSMBS) extends it to take into account multiple behavior spaces and thus propose a framework in which behavior shaping can be implemented.

A. Definitions and notations

In this paper, we consider policies π θ parameterized by a vector θ ∈ Θ ⊂ R ng , with n g the policy dimension. The policies, as well as the transition model of the robot and environment, are assumed deterministic. Novelty-based methods rely on the policies' behavior characterization, which results from mapping the trajectory of the robot when applying the policy from an initial state s 0 into a smaller dimension vector in a carefully selected behavior space [START_REF] Doncieux | Novelty search: a theoretical perspective[END_REF]. In the following and as usual in novelty-based algorithms, we assume that the initial state is fixed, and the behavior therefore only depends on the policy parameter θ.

NSMBS proposes a framework to explore in multiple behavior spaces, here called behavior components. A policy is thus be described by n b behavior components. While in classic novelty-based algorithms, each policy is always associated to a behavior characterization, NSMBS relaxes this assumption and some policies may have undefined behavior components. It allows us to characterize grasping behaviors, for instance, and give an undefined grasping behavior component to policies that do not succeed in grasping the object. To this end, an eligibility criterion is associated to each behavior component. This gives rise to the following notations:

• B i ⊂ R n B i is the i-th behavior component space; • ξ i (θ) is the eligibility criterion associated to B i . If ξ i (θ) = True, the corresponding behavior is defined. • b i (θ) ∈ B i ∪ {∅} is the i-th component of the robot behavior when it follows policy π θ . b i (θ) = ∅ i.f.f. ξ i (θ) = False; • n b is the number of behavior components.
Although some recent work try to automatically learn the behavior spaces [START_REF] Paolo | Unsupervised learning and exploration of reachable outcome space[END_REF], [START_REF] Cully | Autonomous skill discovery with quality-diversity and unsupervised descriptors[END_REF], here the behavior components are supposed to be known and given by the experimenter.

B. NSMBS

NSMBS (Algorithm 1) derives from Novelty Search [START_REF] Lehman | Abandoning objectives: Evolution through the search for novelty alone[END_REF], [START_REF] Doncieux | Novelty search: a theoretical perspective[END_REF]. As for other evolutionary algorithms, an initial population of solutions is randomly generated, evaluated, and a parent set is selected. The parents are then copied and modified by mutation and crossover operators to get new solutions, that are then evaluated and selected to constitute the next generation of parents. In Novelty Search, an archive of explored behaviors is also maintained, and the selection process relies on maximizing a novelty objective, which is the average distance to the K-nearest neighbors in the behavior space among the archive and current population.

In NSMBS, the outcome of a policy π θ is described by a list of behavior components: (b 1 (θ), b 2 (θ), . . . , b n b (θ)). The selection process in charge of finding the individuals of the next generation relies on these elements (Algorithm 2). Policies are selected one by one from a set of individuals S initialized with the current population and offspring. The individual selection starts by randomly choosing a behavior component among the ones that are defined at least for one policy in S. The most novel individual in the selected component is then selected; it is removed from S and the process starts again until the new population is filled. 2)

with b i (θ) = ∅ i.f.f. ξ i (θ) = False • selectParents is a random selection process of λ indi-
3)

Fig. 2: Description of the multiBCSel() method for 4 individuals and 3 behavior components. The third (red) component cannot be selected because there is no eligible individual. Among the two remaining components, the second (blue) one is randomly selected, and the eligible individuals (inds 1, 2 and 4) are sorted according to nov2. The most novel individual, ind 1, is selected.

IV. EXPERIMENTAL SETUP

A. Simulated environments description

In order to demonstrate the algorithm's ability to generate rich, diverse grasping movements for multiple robots, we rely on simulated environments using the Gym framework [START_REF] Brockman | Openai gym[END_REF]. A robot is placed in front of a table, where an object is spawned at a fixed position. Three robots are used: A Rethink Robotics Baxter using only the left arm, with 7 degrees of freedom and a parallel gripper; A Kuka Iiwa arm with 7 degrees of freedom and a two-fingers clamp gripper, both simulated using pyBullet [START_REF] Coumans | Pybullet, a python module for physics simulation for games[END_REF]; and a SoftBank Robotics Pepper, using only the left arm, with 5 degrees of freedom and a multi-fingered hand, simulated using Qibullet [START_REF] Busy | qibullet, a bullet-based simulator for the pepper and nao robots[END_REF]. The episode's length T is adapted to each robot and simulator's dynamics, all other experimental hyperparameters being the same between the three robots. Five target objects are used: a simple 5 cm cube, a simple 5 cm ball, a miniature plastic bowling pin, a mug with a handle, and a gamepad.

The policies are defined by three waypoints in the joint space, giving the robot pose at T /3, 2T /3 and T . Smooth motions are interpolated using third order polynomials. An extra parameter t grasp defines the time when the gripper closes. The policy dimension n g is therefore 3 × n dof + 1.

Besides the length of the episode T , we define t touch the time when the gripper first touches the object (undefined if the object is not touched), obj pos t and grip pos t the position of resp. the object and the gripper at time t in the Cartesian space (R 3 ), and grip or t ∈ R 4 the orientation of the gripper at t, defined as a quaternion. b 1 is always eligible. b 2 is only eligible if the gripper touched the object (we only search diverse trajectories that engage the object). b 3 and b 4 are only eligible if the object was successfully grasped. A successful grasp is defined as a trajectory where the gripper touches the object shortly after it is closed while the object is on the table, and that ends with the object in a stable position in the air, with minor conditions on penetrations to avoid unrealistic grasps.

B. Behavior Descriptors

C. Measures

To monitor the progress of the exploration and the resulting diversity, we use the following metrics:

• The number of successful grasping solutions found, and their coverage of the behavior space B 4 [START_REF] Doncieux | Novelty search makes evolvability inevitable[END_REF]. This gives an estimation of the raw success of the repertoire generation process. • The sample efficiency, defined as the proportion of all the evaluations that resulted in a successful grasp. This measures the ability to generate a rich repertoire while minimizing the computational cost. In order to compare our method to the state of the art, we introduce two other measurements:

• The first success generation, which is the number of iterations after which the algorithm first discovers a successful grasp. This evaluates the ability of an algorithm to quickly bootstrap in this sparse, difficult task. • The successful run rate, which measures the proportion of runs that generated at least a single grasping movement. This measures the reliability of an algorithm.

V. RESULTS

A. Grasping synthesis on multiple robots

The NSMBS algorithm is first run on all the three robots, with the mug object, for 1000 generations, a population size µ = 100 individuals, an offspring size λ = 50 individuals, and a value of K = 15 for the KNN algorithm. The search is repeated 10 times for each condition.

The generated repertoires are very different between robots, with sizes of 13148 ± 4574 successful grasps for the Baxter, 3490 ± 4706 for the Pepper, and 3929 ± 2262 for the Kuka. This reflects the variable difficulty of the grasping task with different robots, the high variance for the Pepper and Kuka robots being due to the presence of failed runs where no grasping was discovered. Despite this, the search finds a variety of grasping behaviors for all robots, some of which are shown in Fig. 1.

Fig. 3 shows the final diversity coverage and the evolution of the sample efficiency for the three robots. We can see that the algorithm quickly manages to bootstrap and discover grasping motions, and then leverages those existing motions to improve diversity, as shown by the constantly growing sample efficiency during the process. Furthermore, we can sort the ways the mug object can be grasped in four different categories, which allows us to define four grasping styles, not all of them available to all robots due to their design:

• Grasping the handle (handle style): this grasping style is possible on all three robots; • Grasping the mug from the top, with part of the gripper inside the mug and the other outside (in-out style): only the Kuka and Baxter robots are capable of this grasping style, the Pepper robot's weaker hand being unable to hold the mug this way; • Grasping the mug from the sides, with the gripper grasping the outside of the mug (out style): only the Kuka robot is capable of this grasping style, thanks to its much larger gripper; • Grasping the mug from the inside, by inserting the entire gripper (in style): only the Baxter robot with its thin parallel gripper is capable of this grasping style. We categorize the discovered grasping motions on all three robots. The results (Fig. 4) show that NSMBS is general enough to fully make use of the capabilities of each robot, discovering all the possible styles for all robots.

B. Comparative analysis

In order to highlight the benefits of NSMBS, we compare the performance with the following variants and baselines: Red cross means grasp style not generated for this robot.

• A classic Novelty Search (NS) algorithm [START_REF] Lehman | Abandoning objectives: Evolution through the search for novelty alone[END_REF] using a single behavior descriptor, which is built by concatenating the four b i from NSMBS. If one of the components is not eligible, its values are set to 0. • A Map-Elites algorithm [START_REF] Mouret | Illuminating search spaces by mapping elites[END_REF], using the same concatenated behavior descriptor as NS with 1000 CVT cells [START_REF] Vassiliades | Using centroidal voronoi tessellations to scale up the multi-dimensional archive of phenotypic elites algorithm[END_REF]. Map-Elites uses a quality measure; it is not important here and set to a simple energetic criterion; • A random search baseline where a similar number of individuals are generated by random uniform sampling in policy parameter space. The algorithms are used on the Baxter environment with the mug object, with the same hyperparameters as above except the process is run for 2000 generations. Each condition is repeated 20 times. The main results are shown in Fig. 5. The NSMBS variants overperforms the baselines, yielding higher sample efficiency and diversity coverage. NS discovers some grasping motions, but is generally much less sample-efficient, resulting in a lower coverage. Map-elites is generally unsuccessful ; this may be due to the absence of a population combined with a random selection and a very large composite behavior space, which fails to focus the evolutionary budget on promising individuals [START_REF] Coninx | Younger is better: A simple and efficient selection strategy for map-elites[END_REF].

Fig. 6 gives further insights into the benefits of our behavior shaping technique. Although most NS and NSMBS runs are eventually successful at generating at least some grasping behaviors (Fig. 6a), the full NSMBS algorithm with all the behavior components does so earlier in the evolutionary process (Fig. 6b). It also quickly achieves and maintains higher sample efficiency (Fig. 6c), with 25% of evaluated individuals (in average) being successful grasps after only 400 generations.

C. Transfer on real-world robot

The grasping motion repertoires are only relevant if at least some of the generated policies transfer to the real world. We therefore generate repertoires for five objects with the Baxter robot and evaluate them in real world experiments.

NSMBS is first run 10 times for 1000 generations in the simulated Baxter environment for each of the five objects. Repertoires are successfully generated for all objects (Fig. 8). Results for the mug were previously discussed in section V-B; sample efficiency is lower for other objects, but NSMBS still reliably generates large repertoires of successful grasps. All 10 runs succeed in discovering grasping motions for the pin, 9 succeed for the mug, the sphere and the gamepad, and 7 succeed for the ball. For each successful run, 10 individuals are randomly sampled from the final repertoire, and evaluated on the robot, for a total of 70 to 100 real world evaluations depending on the object. Some successful grasps for each object are shown in Fig. 7 and in the video in annex.

The transfer success rates are reported in Fig. 9a. The highest transfer rate is achieved with the mug, where 30 % of policies succeeded, followed by the cube, the sphere, and finally the gamepad and the pin at 8 %. It is noteworthy that this success rate is directly correlated with the sample efficiency of the process in simulation (Fig. 8). The lower sample efficiency for the gamepad and the pin can be explained by their characteristics : the gamepad is quite heavy (0.21 kg) compared to other objects, and it has a complex shape and moving parts that are not fully represented in the simulated environment. The pin is standing, thus it can easily fall and the dynamics then differ from the simulation. We can use these results to estimate the sample efficiency of the full repertoire generation process, i.e. the expected proportion of all the evaluations in simulation that result in a grasp that transfers to reality, by multiplying the sample efficiency in simulation (as defined in section IV-C) by the transfer success rate. Results are shown in Fig. 9b. Values range from about 1.6 % for the pin to 7.1 % for the mug.

VI. CONCLUSION

The simulation results show that our NSMBS method is able to efficiently generate grasping behaviors for various robot arms, and that it outperforms the state of the art in both diversity coverage and sample efficiency. The comparison to ablated versions of the method with only some of the behavior components highlights the importance of choosing adequate behavior characterizations for the method to be able to bootstrap and learn, and the comparison to classic diversity methods showcases the ability of NSMBS to take advantage of those multiple components for behavior shaping.

Real world transfer rates range from 8 % to 30 % depending on the object. This is comparable to the technique used by Levine et al. [START_REF] Levine | Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection[END_REF], which makes strong hypotheses about the experimental setup and uses motor primitives that constrains the diversity of the possible grasps, whereas our method is applicable to a large variety of robots without any modification and promotes grasping diversity.

The real world sample efficiency ranges from 1.6 % to 7.1 %, which corresponds to one out of 14 to one out of 62 simulated policies being a valid grasping policy in the real world. It may seem low, but it does not prevent the generation of a large, diverse repertoire, as simulated experiments are cheap enough that more than 100 000 evaluations can be performed in a few hours on a modern workstation, resulting in thousands of transferable grasps. An open question is how to filter the generated repertoire for individuals that transfer to the real world, as the transfer rate makes random sampling wasteful. Further work could address this by training a model to predict transferability from a limited real world dataset.

Our method allows for the generation of large-scale, diverse datasets for robotics, including for challenging tasks like grasping, without limitations about the applicable robots or policies. Instead, it requires defining relevant behavior spaces and the corresponding eligibility criteria. This is usually not a major issue, as it is often quite straightforward to define behavior components to promote diversity in a way that helps discover and explore the task solutions space.

The final test of NSMBS shall be the use of the generated repertoires as datasets to train closed loop policies [START_REF] Levine | Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection[END_REF], [START_REF] Jegorova | Behavioral repertoire via generative adversarial policy networks[END_REF], and the evaluation of those policies on real robots. The large size and high diversity of the generated datasets, the good sample efficiency of the method and its applicability to various problems and robots without adaptation or parameter tweaking, make it promising in this regard.

Fig. 1 :

 1 Fig. 1: Examples of grasping behaviors found for each robot.

  Input: set of individuals S, number of individuals to select µ Result: set of individuals P P ← ∅ ; while size(P) < µ do i ← randomSelComponent(S) ; T ← {θ ∈ S | ξi(θ) = True} ; chosen ← arg max θ∈T novi(θ) ; P ← P ∪ {chosen} ; S ← S -{chosen} ; end • generateRandomPopulation generates a population of size µ: {θ 0 , ...θ µ-1 } by uniform sampling; • eval evaluates an individual's policy θ and outputs its behavior components bd = (b 1 (θ), b 2 (θ), . . . , b n b (θ)),

b 3

 3 (θ) = grip pos t touch and b 4 (θ) = grip or t touch . Each b i pushes the exploration in one direction and generates a useful incentive: b 1 pushes for trajectories that move the object around, b 2 promotes diverse movements during exploration, b 3 and b 4 explicitly push towards diverse grasping poses.

  Evolution of sample efficiency during the search.

Fig. 3 :

 3 Fig. 3: Coverage and sample efficiency of the search for grasping behavior with NSMBS for the three robots and the mug object.

Fig. 4 :

 4 Fig. 4: Mug grasping styles proportions (10 runs, 1000 generations). Red cross means grasp style not generated for this robot.

  Overall sample efficiency of each algorithm.

  Final diversity coverage of each algorithm.

Fig. 5 : 3 N

 53 Fig. 5: Sample efficiency and diversity coverage of the search for grasping behavior on the Baxter robot after 2000 generations.

3 N

 3 r a n d o m M A P -E li te s N S N S M B S n o B D 2 N S M B S n o B D Generation at which the first grasp is found, for each algorithm. Unsuccessful runs are ignored. Evolution of sample efficiency during the runs.

Fig. 6 :

 6 Fig.6: Proportion of successful runs, bootstrap time for successful runs, and evolution of the sample efficiency for each algorithm (map-elites hidden behind random).

Fig. 7 :Fig. 8 :

 78 Fig. 7: Grasping example on real Baxter robot for each object. The end effector trajectory, the pose at which the robot grasps the object and the final robot and object pose are shown.

  Estimated sample efficiency for the full repertoire generation process in reality.

Fig. 9 :

 9 Fig.9: Success rate and sample efficiency in reality on the robot Baxter. Repertoire generation is run 10 times per object, and 10 individuals are randomly selected from each successfully generated repertoire.

  viduals in pop; • operate copies the parents and applies gaussian bounded mutation and crossover operators to generate offsprings; • getNov computes a list of per component novelties nov = (nov 1 , nov 2 , . . . , nov n b (θ)) for an individual. (θ), b i (θ k ))) where b i (θ k ) are the behavior components of the K nearest neighbors of b i in refSet (∅ values are ignored). If b i (θ) = ∅, nov i is undefined. • multiBCSel is the main selection process, described in Fig. 2 and Algorithm 2. • randomSelComponent selects a behavior component by uniform selection among components for which at least one individual is eligible.

	K-1
	nov i = (dist (b i ind 1 k=0 ind 2	ind 3	ind 4
	1)
			random selection
	novelty	ind 4 ind 1 ind 2	ind 1

• Our full NSMBS algorithm (NSMBS); • The same algorithm without the object touching BD (NSMBS no BD2) and without the grasping position BD ((NSMBS no BD3);

ACKNOWLEDGMENT

This work was supported by ANR projects InDex and Learn2Grasp.