Geoffrey Mead

Bourdieu and conscious deliberation

Social theorists have in recent years concerned themselves with the question of the kind and intensity of people's everyday reflective capacities. In this respect, Bourdieu's theory has mostly been found wanting. In an effort to counter this sentiment, I intend to find in Bourdieu's theory of practice an adequate response to this question. This is accomplished first by examining the dominant 'mechanistic' interpretation of Bourdieu's theory, within which practice is reduced to programmatic action detached from conscious thought. While recognizing that mechanistic traces persist within Bourdieu's theory, I advocate an alternate reading that accentuates his manifestly 'anti-mechanistic' intentions. I argue that by reifying 'consciousness', opposing it to a mechanistic habitus, and then positing a triangular relation to a reified social world, commentators have manufactured theoretical problems that a different way of reading Bourdieu dissolves. In this alternate reading, the sociologist makes no wager on the causal efficacy of consciousness or habit, and allows for a conception of the relation between actor and world that locates 'agency" in improvised struggles undertaken over time.

Perhaps the greatest testament to the success of Pierre Bourdieu's sociological project is the fact that, in significant instances, even his critics take their leave from premises his project has established. Notwithstanding his repeated claims to have been 'misinterpreted' on account of a number of factors (Bourdieu, 2005a: 33), it remains that these purported misunderstandings can be accurately described as 'well-founded' misunderstandings. That is, they are not merely trading in alien impositions on Bourdieu's work but are touching on some of the profound unresolved and, perhaps, irresolvable tensions that Bourdieu himself strove to overcome. Or so I argue. I propose that certain recent discussions, on the whole rather critical of Bourdieu's positions, nevertheless mobilize particular conceptions of sociological phenomena that originate in Bourdieu's own work-conceptions that, upon reflection, reveal themselves to be confused and hastily expressed: I intend here the relation between the social agent and her environment and the place of consciousness in this relation.

By returning to Bourdieu's own discussion of these concepts, and their development in his oeuvre, we are afforded other means of construing the important phenomena that these ideas seek to capture.

The tensions to which I refer are exemplified in the concept of habitus, whose nameas Bourdieu (1977: 218) was well aware -both calls to mind a mechanistic philosophy 1 and, at the same time, seeks to combat it 'from within', by defining itself in terms of the dimensions of 'tact, dexterity, [and] savoir-faire ' (1977: 10). As I will show, it is not simply a misinterpretation to suppose that the habitus is mechanistically deployed, for it does appear in this mode in Bourdieu's work. Yet its appearance under this guise should not be taken as paradigmatic or as exhausting the possibilities inherent in the concept-especially since its underlying premises militate against this reading. Nevertheless, among critics and those deploying the concept, habitus is often expressed in mechanistic terms, as if it were simply 'habit', rendering practice a mere execution of an 'internalized program' (as it is put by Boltanski, 2012: 339). From such a schema it follows that the 'environment' serves only as a constant background, a support for habitual practice, rendering the social agent a 'fish in

water'-at least until the environment no longer serves as a support and, according to this mechanistic narrative, conscious direction takes over.

Since commentators have clustered their arguments around the place of consciousness in Bourdieu's work -either to decry its ostensible denigration or to see in it an opportunity for political action -it is important to examine its shifting role and successive manifestations in his work. I will suggest that rather than denoting a kind of necessary predicate of practice or, worse, an anthropological supposition, the purported 'unconscious' quality of practice simply represents a means of pointing out the non-subjective nature of meaning characterizing any social practice, a means of stating that such significance is irreducible to one's conscious intentions. This has the consequence of supposing that one can indeed be lucid in one's actions and at the same time be 'unconscious', for the former is an anthropological phenomenon and the latter is an epistemological statement. Still, while this should result in Bourdieu remaining at cross-purposes from recent debates about the place of deliberation in 'practice theory', it is complicated by the fact that Bourdieu made efforts to enter this debate, under the guise of 'position [ing] habitus against voluntaristic philosophies of action' (Elder-Vass, 2007: 328)-to the detriment, I believe, of his initial point. This article takes a common triad of concerns -agent, environment, consciousnessand aims to extirpate from the relations among them the mechanistic traces that remain in Bourdieu's discussions and that repeatedly surface in commentary around them. I discuss, in turn, the ambivalent role of 'consciousness', which oscillates between methodological postulate and anthropological datum; the confused depiction the social agent's relation to the world; and, more positively, the manner by which social agents forge this world in the struggle over its stakes. I ultimately come to argue that a mechanistic approach reifies terms -such as 'consciousness' and 'world' -that ought to remain relational and subject to definition in their practical interrelation-an interrelation, I must add, that necessarily occurs in time. To his own disservice, with respect to this latter point, Bourdieu's phenomenological penchant for concrete examples like moves on a football field, encourages the kind of hypostatizing of the social space that neglects that it is in fact comprised of a structure of probabilities and is only ever produced in time, through the practices and struggle that take place within it.

Consciousness as a functional principle

In the vexed domain of 'consciousness' -the frequency of its presence, its explanatory weight vis-à-vis 'unconscious' determination, its practical potency, and its political potential -the situation I referred to at the outset, where certain formulations of Bourdieu have come to be taken as unquestioned points of departure for his commentators and critics, is most palpable: sketched broadly, in this picture Bourdieu proposes first that in comparison to unconscious determination, we operate with some kind of 'conscious' lucidity but a small part of the time (Jenkins, 1992: 46;Bourdieu, 1984: 474); second, that unconscious determination is analogically aligned with the body (Noble and Watkins, 2003: 529;Bourdieu, 1977: 94); third, that lucidity emerges in moments of major or minor 'crisis' (Crossley, 2013: 151;Ermakoff, 2010: 541;Bourdieu and Wacquant, 1992: 131); and fourth, that this resultant lucidity offers some means of counteracting symbolic violence, necessarily so inasmuch as the latter operates unconsciously and in the folds of the body (McNay, 1999: 35;[START_REF] Bourdieu | Masculine domination[END_REF]. This article challenges the picture comprised by these elements, and suggests that Bourdieu offers an alternate one that better expresses his overall project, accounting for phenomena in a more rigorous fashion. In particular, the above picture rests on the idea of consciousness as some kind of anthropological presupposition, whereas this alternate picture remains mostly indifferent to this issue. So, despite all his evocative sketches of unconscious practice, the notion of 'unconscious' is first deployed as an epistemological presupposition -as 'non-conscious' practice -rather than as any kind of anthropological or psychological characteristic:

What might be called the principle of non-consciousness, conceived as the sine qua non for the constitution of sociological science, is nothing other than the reformulation in the logic of that science of the principle of methodological determinism which no science can reject without disowning itself as a science. This is what is obscured when the principle of nonconsciousness is expressed in the vocabulary of the unconscious. Those who do so thereby transform a methodological postulate into an anthropological thesis (Bourdieu et al., 1991: 16) At least at this stage and in this domain 'non-consciousness' or, if the word be used, the 'unconscious', operates as a functional rather than substantial term, denoting a presupposition that sociology must make if it is to be considered 'scientific'. It refers not to a kind of hidden determinant of practice, but to the objectivity of meaning, to the fact that the one who, willy-nilly, produces meaning does not remain its 'master and possessor ' (1991: 17).

Bourdieu's position is aphoristically captured in the following: 'it is because subjects do not, strictly speaking, know [savent] what they are doing that what they do has more meaning than they know' (Bourdieu, 1977: 79). This is to state neither that actors are in some sense robotically -and so 'unconsciously' -following a kind of 'program' deposited deep within, nor that, as Jenkins puts it, 'although actors may understand their behavior as the pursuit of known goals and objectives, the sociologist (Bourdieu) knows better' (Jenkins, 1992: 45;Sayer, 2005: 29). Conscious goals and awareness of one's actions, not to mention the existence of a constant 'stream of consciousness', are in no sense inconsistent with this epistemological principle of non-consciousness. To give an example of Bourdieu's own as a means of drawing the distinction, take a literary author, somebody about whom it would be difficult to argue that he is simply following a program or is acting with mere automaticity:

If you ask a canonical writer: "What is it that you are doing?", and he responds: "I'm writing a novel", he knows what he is doing. They are intentions. But he does not know that he enacts a whole theory-cum-practice of the novel that orients all his choices from its first moment (Bourdieu, 2013b: 89) If it be protested that this is giving short shrift to the canonical author, who perhaps 'knows' in some explicit and thematized sense what he is doing, we can remove all doubt by taking an avant-garde, 'ironic' author, who -if nothing else -knows what she is doing. Yet the argument still stands: while she knows what the naïve canonical author ignored, and perhaps even parodies it, she does not therefore know the sociologically pertinent information, the objective conditions that impel her toward her ironic pursuit. Strictly speaking, then, lucidity and deliberative capacity do not, especially in the moment of deliberation, grant one escape from 'non-conscious' determination, which again is not some deposited program leaving us more or less 'at the mercy of our habitus' (Elder-Vass, 2007: 344), but is a gesture towards the endlessly 'thrown' nature of social action: its meaning, in the absence of a great deal of social-scientific labor, extends beyond our ability to appropriate it.

To reduce the distinction to one of automatism, routine, or habit rather than consciousness is to miss Bourdieu's point. There is implied in such a reading an erroneous conflation of automatism and habitus, as if together these vie with conscious deliberation for causal supremacy. Such is demonstrably erroneous insofar as one can be 'to some degree aware of [one's] habitual responses as they occur' (Throop and Murphy, 2002: 199), and for all that still orient oneself according to a set of socially structured criteria, 'internalized' in the form of habitus. Social determination need not be experienced as some kind of alien imposition. Instead, at least according to Bourdieu, it is experienced in terms of the "socially innocent language of likes and dislikes" (Bourdieu, 1984: 243). Taking up the phenomenological strain of Bourdieu's work, we can express these possibilities of like and dislike, "attraction" and "repulsion", in a number of ways: in Merleau-Ponty's (2012: 52) objects which call out to us and which we grasp "in [their] signification for us", and in Gibson's (1979: 127) "affordances", the aspects of the environment that show up and beckon or threaten an organism so disposed to recognize them as such. It must be noted that interactions with the surrounding world do not represent arbitrary designations assigned to otherwise indifferent configurations of stimuli, but a profound attunement to a reality with which one has had prolonged experience. Thus, a cliff that looks dangerous is dangerous (Martin, 2011: 184) and a social position that looks like it cannot be occupied will have a hard time being occupied by the one who so beholds it-regardless of whether one's action in relation to it is "automatic" or not (pace Leschziner and Green, 2013: 123).

Ultimately, for Bourdieu, what is crucial is that it appear reasonable to adopt a particular posture before the world, it being given that one has been prepared by a personal and -to an extent -collective history for a certain future.

The diversity of relations to objective possibilities can be imagined on a scale: at one pole, where "habitual" and automatic reactions predominate, certain objectively existent opportunities are not even perceived (e.g., the elite education that barely even impinges on the conscious awareness of the one who has "no chance" of appropriating it). As we progress away from this pole, such "opportunities" can be perceived as existing, but are just not "for me" or "for us", whether for reasons of explicit distaste or for reasons of inability of access. That is, they experientially lack a "vital" or "affordance" character. While perceived, they do not "call out" to those who, in effect, perceive them as inaccessible or as undesirable. The next point on the scale posits that certain opportunities might appear as attractive and even be chosen despite one's disposition. This is the ideal-typical extreme preferred by those who exalt conscious deliberation, and for whom one's conscious resolution is determinant.

In such a case -which Bourdieu's theory in no way "prohibits", but would certainly presume improbable -it would be expected that attempted occupation of such positions result in significantly higher levels of "attrition" amongst those not otherwise disposed to occupy them. Bourdieu's probabilistic logic assumes not that these seeming miracles do not happen, but that they are anomalous and that the "tension" experienced by the "pretender" would bring about either his eventual relinquishing of his hopes or a different mode of occupying the position (so that the pretender who occupies the throne represents a vastly different phenomenon than its occupation by the one "destined" to do so) (Bourdieu, 1984: 337).

Here the danger of conflating habitus with routine and automaticity is exposed.

Habitus amounts to more than a corollary of the simple unconscious "complicity" of conditions of production and conditions of existence. It is the evolving product of an embodied history that prepares one for a particular future which makes other futures improbable-if not, strictly speaking, "unthinkable" and certainly not logically "impossible". So as we verge toward the other end of the scale, we reach the point at which there dawns a sense that something is "for me/us", even if this is not necessarily the case. (It is in these middle areas that "allodoxia" -the belief that something is "for me", when it is more or less unlikely to be so -is most likely to occur.) Next, it becomes rather obvious that something is "for me", and is perhaps even undoubtedly so. Finally we must mention "amor fati": in this case not only is such an objective possibility obviously a destination, but it is desirably so. Yet even at this end of the scale there is still much effort necessary to realize one's position. This presence of effort ought to signal the firm incompatibility between practice through the lens of habitus and simple automatisms and routine behaviors. (It even, perhaps, allows for the possibility of a peculiarly Bourdieusian notion of "will" [cf. Throop 2010: 29].)

While the picture of consciousness that I described at the outset can indeed be found in Bourdieu, and has certainly been productive in terms of contributing a position to current debates around the role of "cognition" in sociology, I argue that the alternate one has a sounder basis in Bourdieu's work, much of which it undergirds, and better reflects the reality of social life, where it is difficult to deny that "a partial, lacunary, discontinuous form of consciousness always accompanies practices" (Bourdieu, 1972: 200). I implied that the most pernicious effect of this construal was the conflation of habitus and a kind of thoughtless routine and "habit". Such a conflation, I now assert, has important implications for how we conceive of practice in relation to a "world" (or, among its many guises, a "space", "field", or "environment"). Specifically, a hasty mobilization of some of Bourdieu's own comments around the relation between agent and environment results in a reification of both terms of this relation and a failure to appreciate the crucial roles of time and of practice as processes.

Routine, crisis, and reflection

Having argued in the first section that by conflating habitus and automatic practice we reify a functional notion, I wish to argue here that a hypostatized conception of the social world results from holding tight to such a picture (of practice as the automatic and unconscious execution of a program). In my enumeration of the various aspects of the standard Bourdieu "picture" of practice, I listed the proposition that conscious lucidity emerges in times of crisis-in times where there is a breakdown in the "fit" between an agent's dispositions and the objective "conditions" in which they ordinarily flourish. Here I wish to focus on this aspect and suggest that the construction underlying it is riddled with troublesome issues deriving from certain presumptions about the character of these objective "conditions", about what we think the "world" is. Conventionally, it suggests a static world, typified by the experience of the individual amidst her concrete surrounds, within which everything is always already laid out, and the social agent either matches these surrounds or does not. It thereby neglects time, and with it the labors of anticipatory adjustment, deferral, and attempted subversion through struggle, each of which result in this state of the world being brought about. This view of a hypostatized world is common to several sociologists who mobilize a form of the idea that conscious lucidity emerges in the breakdown of the relationship between habit and world. Certain among them make a historical argument: the increasing rapidity of change in various domains, concomitant with the proliferation of such domains, has brought about an intensification in the number or severity of dislocations between objective structures and their internalized counterparts.2 A corollary to this argument is that Bourdieu's theory becomes obsolete. This is especially so, Archer argues, since such a concept as habitus was formed to account for the static reproduction of an undifferentiated society, far from the one that we encounter daily (see also Haber, 2004: 197). Indeed, this latter, contemporary society is far even from 1960s France, where the concept was most frequently deployed: "the young of the new millennium…no longer live in Bourdieu's world" (Archer, 2012: 68). In this changed world, the "habitual, routinized, or customary action" that served past generations can only founder (Archer: 47): "Swift change renders habitual guidelines to action of decreasing relevance or positively misleading". In their place come to stand conscious efforts and resolute struggles on the part of the acting agent, who "internally converse[s] about the person they would like to become and the job which will best express this" (Archer, 2000: 290-291). Lahire mounts a similar critique to Archer but turns his focus away from ultimate life decisions to relatively minor, "polymorphous crises that beset actors in their everyday lives" (Lahire, 2011: 45). These minor crises in their turn impose the need for a kind of reflection on practice in the moment, as well as the need for recourse to auxiliary tools to compensate for what Lahire sees as the failures of a habitual, practical sense. Like Archer, Lahire (2011: 21) sees a concept forged for "traditional Algerian peasant society" as unsuitable for analysis of what he calls our present-day "ultradifferentiated societies" (Lahire: 28). In these latter formations, the clash of multiple determinations, each characteristic of one of many persistent logics, demands from the actor a conscious computation and weighing up of options on offer.

Other commentators make a similar argument but instead of suggesting a kind of historical obsolescence of the concept, make reference to its unsuitability for capturing generic "mismatches", necessarily occasioned in everyday life insofar as unforeseen and "chance" occurrences are present. In this vein, Crossley compares Bourdieu unfavorably to Dewey regarding their respective levels of sensitivity to the minor "disruptions" of "the taken-for-granted world that our habits are attuned to" (2013: 151; see also Elder-Vass: 331). Smith (2004: 110), similarly, finds in the novelist Marcel Proust a "more balanced picture of life" than that provided by Bourdieu, for whom "the self [is] imprisoned by habit and engaging in repetitive tasks and mental routines" (109). Proust's depiction, by contrast, is ostensibly superior because of its recognition of the "oscillations between habituated action and reflexive self-awareness", amidst "those chance collisions of life that provide a shock and with them some distance from the routinized and habituated" (110). Each such attempt, whether making a historical or generic argument about the relative frequency and the effect of these "shocks", takes for granted a "routine/reflexivity dialectic" (Lahire, 2013: 147) and the mechanistic assumptions bound up with it.

Each of these engagements with Bourdieu's thought clearly expresses itself in the terms that I rehearsed above in my discussion of consciousness. This pertains especially to the reduction of practice associated with habitus to some kind of "routine". With conscious deliberation opposed so firmly to routine, and with the latter defined in such close association to habitus, it can only follow that a break in routine introduces conscious deliberation-and so habitus, conflated with "routine", is defined out of relevance. I propose in the remainder of this section that this usage ignores the other potentials the concept offers, for it either restricts its level of application to some kind of immediate, concrete, and tangible "situation", or it takes this situation as exemplary and abstracts from it accordingly. Nevertheless, as my initial statements express, I maintain that Bourdieu's own work -or, more specifically, Bourdieu's occasional statements about his work -give a source for this peculiar mobilization of the concept, above all through his rhetoric and means of exemplification.

The troublesome component in many of Bourdieu's statements, inherited by those deploying the critical "routine/reflection" usage, centers on the character of the second half of the "agent and world" dyad (Bourdieu, 1990a: 10). Here, the world functions as a concrete stage upon which practice unfolds, and is indifferent to it, so that the world is, paradoxically, only analytically pertinent when it is no longer present, when there is a "breakdown" in the concordance between objective and internalized structures. This derives from a hasty abstraction from certain of Bourdieu's seductive examples of practical sense "in action". Take for instance a sample of Bourdieu's illustrations of practice-the football "move" (Bourdieu, 1990b: 82), the missed shot of the tennis player, the elevator stopping prematurely (2000: 211), the co-ordination of traffic around the Place de la Bastille (2001: 1), all of which are intended to convey the idea contained in Bourdieu's notion of the "forthcoming". This phenomenological manner of expression encourages a form of thinking about sociological phenomena that unwittingly transposes them into the realm of the concrete, hypostatizing a space of distributions of capital into an alwaysexisting "setting".

The result is a hypostatizing of the social world, encouraging one to forget that the "field" is not just that upon which players make their moves but is the product of those moves. A hypostatized conception interferes with an adequate understanding of Bourdieu's attempts to elucidate the agent's relation to the environment and the breakdowns encountered between them. I refer here to the notion of a "coincidence" of, or "concordance" between, internalized and objective structures. Bourdieu is never clear on precisely what this entails. For instance, within the space of a few lines of one article (Bourdieu, 1974: 4), Bourdieu oscillates between significantly different formulations of the relation between the conditions of acquisition and implementation of habitus: while the two conditions must "coincide perfectly" and be "identical", at the same time Bourdieu suggests they merely be "similar", "homothetic", and "objectively compatible" (1974: 5). Later Bourdieu employs the even more benign suggestion that dispositions be "adjusted" to the conditions of their implementation.

This obscurity is repeated in several other places throughout his oeuvre, leaving the reader with no means of determining precisely how proximate dispositions must be to conditions if practice is to remain "ontologically complicit". Yet I suggest that this obscurity is in some sense necessary, and all these adjectives -however incompatible -are accurate, for the relation between these two terms cannot be precisely delineated by theoretical "dictate", since this assumes the relation to be characterized by two constants, when in fact the two variables remain for the agent to resolve in practice.

From his earliest writings Bourdieu warned of succumbing to the hypostatizing tendency, which he then labeled the "realism of the structure" [START_REF] Bourdieu | Structuralism and theory of sociological knowledge[END_REF]. This consists in "hypostatiz[ing] the systems of objective relations in already constructed totalities, outside the history of the individual or the group" (1968: 705). It is avoided by recalling that …ultimately, objective relations do not exist and do not really realize themselves except in and through the system of dispositions of the agents, produced by the internalization of objective conditions. Between the system of objective regularities and the system of directly observable conducts a mediation always intervenes which is nothing else but the habitus, geometrical locus of determinisms and of individual determination, of calculable probabilities and of lived-through hopes, of objective future and subjective plans (705) Bourdieu refers to this "hypostasis" structuralist model as a "mechanistic" one, wherein agents contribute nothing in particular to the execution of the model forged by the sociologist, but remain epiphenomenal "bearers" of its tendencies. I suggest that Bourdieu's own conception, by contradistinction, is closer to Canguilhem's (1991: 197-198), who wrote of the domain of the living: Certainly this environment, which science defines, is made of laws but these laws are theoretical abstractions. The living creature does not live among laws but among creatures and events which vary these laws…. For the living being life is not a monotonous deduction, a rectilinear movement, it ignores geometrical rigidity, it is discussion [débat] or explanation (what Goldstein calls Auseinandersetzung) with an environment where there are leaks, holes, escapes and unexpected resistances This notion of a "debate", or Auseinandersetzung, in which each term in the relation is constituted by its place within the relation appears to capture quite well the "vision" (Schumpeter, 1946: 501) driving Bourdieu's work. On the one hand, the "subject" term, the habitus, is formed out of the agent's "dialectical confrontation" (Bourdieu, 2005a: 31) with a given situation or social space. On the other hand, the "object" term, the situation or social space, functions insofar as the habitus activates the "possibilities inscribed" therein (Maître and Bourdieu, 1994: xix). Bourdieu writes not only that habitus is necessary to "keep [institutions] in activity, continuously pulling them from the state of dead letters" but also that by reviving them in this way, the habitus "impos[es] the revisions and transformations that reactivation entails" (Bourdieu, 1990b: 57). It is in these revisions and transformations that much of my argument is located: it makes little to think of the relation between agent and world as characterized by some simple correspondence between pre-existing dispositions and positions. Rather, a constant process of adjustment, an interested striving to "meet" the environment, prevails-with the environment, or elements thereof, in turn re-constructed in a manner befitting the striver. The world is made in this twin process of adjustment and construction, for the agent is always "looking forward" to something, and is not simply facing a series of instantaneous presents. So the social "world" -as a "structure of probabilities" -whether in the form of a social space, or in its derivation as field, should not be conceived as analogous to a mere concrete setting or situation. Such a reification entails a decidedly atemporal conception of practice, where one relates to a world that is always already there. If we take as paradigmatic the kind of practice that occurs in a concrete, immediate "situation", as Lahire does in his investigations of quotidian lapses in practical sense, it is understandable that we would arrive at his conclusions. After all, we are surrounded by concrete things that patently persist beyond our immediate practices.

The keyboard hardly requires my typing to pull it from the state of a "dead letter", and the action of my fingers imposes no revisions on its being. Yet such thinking, when applied to sociological phenomena, falls victim to what Bachelard (1984: 40) termed a "thingist" approach, in which an everyday experience of reality -here an experience of concrete matter, with all its inert and massy connotations and evocations -is substituted for a scientific one. It remains nevertheless that Bourdieu himself is at ease in abstracting from concrete practices, taking such phenomena as representative of a generic practical sense (Bourdieu, 2000: 162). In one such example, he captures the breakdown in practical sense by reference to "the experience, familiar to all of us, of the unexpected feeling that occurs when a lift, instead of going straight down to the ground floor, stops at the first floor, where someone has called it" (2000: 211). While such exemplifications have the obvious advantage of evoking for the reader a "concrete intuition" (2005a: 28), they bear a specific difficulty when attempting to articulate the kinds of sociological phenomena that such concepts as habitus and field are invoked to explain. These latter kinds of phenomena -like diploma value, class future, and the emergence of occupational categories -bear little resemblance to such an experience as travelling on an elevator, for the significant reason that in the latter, time is unproblematic: I experience a sudden and instantaneous shock when the elevator stops prematurely, but the temporal gap between the "objective" devaluation (or the initial trigger for the devaluation) of my university degree and my "subjective" realization of this fact offers innumerable opportunities for, and even successive experiences of, denial, deferral, compensation, or subversion-all of whose outcome may be a kind of belated "shock" experience. We are here a long way from the quotidian conception, from Smith's (2004: 110) "chance collisions of life".

The struggle over the production of time

Despite at one time insisting that the "forthcoming" [à venir] should not be mistaken for an imminent future, so that even distant moments, "provided they are tied to [the present] by the unity of signification", can "offer opportunities [potentialités] perceived as co-present" (Bourdieu, 2013a: 65), Bourdieu tends to express himself in terms that presuppose the imminence of the forthcoming, with almost dramatic effect. This is clear in the analogies listed above, all predicated on instantaneous "shock" experiences, but it is also present in the "instantaneist" way of expressing certain sociological phenomena. For example, Bourdieu refers to the disentanglement of objective chances and subjective hopes as occurring by an "abrupt slump" of the former in relation to the latter (Bourdieu, 1984: 168). Elsewhere he and Wacquant (1992: 131) write of this relation being "brutally disrupted". Finally, Bourdieu makes reference to the "sudden, brutal devaluation" of those whose hopes were bound up with a previous state of the matrimonial market (Bourdieu, 2007: 64). 3The point is not simply that what are taken as sudden shifts, akin to the shock of a premature halt in an elevator, are actually processes, but that they are transformations actively brought about or realized in duration. In this latter conception, equally locatable in Bourdieu's work, time is not the mechanical unfolding of events impervious to one's actions (Bourdieu, 2000: 206). Rather, it is the product of practice: "practice is not in time, but makes time". The ascent and eclipse of aesthetic movements provide examples of this manner of "making" time: in the event of their decline, dominants are brought down by an interested effort on the part of upstarts to displace and out-mode them, rather than by any kind of foregone "mechanical sliding into the past", as is the case with natural ageing (Bourdieu, 1996a: 157). This similarly describes another example of Bourdieu's: the "devaluation" of university degrees occasioned by the expansion of the student population. Therein we observe the processes bringing about the production of time. I refer here to the "temporal gap", imbued with significance, between the instant in which there appear within the university field the first intimations of change and the moment of realization, the "outbreak of open crises" (Bourdieu, 1988: 166). This realization is hardly reducible to a "shock" experience -that is, the mechanistic failure of an expectation to be fulfilled -especially because the temporal gap affords numerous potentialities and alternate historical courses. Rather, the misalignment of "slump" is actively brought about. Recall that the objective environment here, notwithstanding its immense "weight" and objectivity, consists in a structure of probabilities, that is, in chances regarding the future (Bourdieu, 2000: 211). We are not dealing with a world in which everything has already happened, as in the concrete exemplification of the "situation", in which the world is simply laid out before the actor. It is, rather, a practical accomplishment, subject to constant re-production.

At its most elaborate, drawing on that strain in Bourdieu's thought that concerns the potential symbolic efficacy of political struggle, my argument implies that the realized world is affected by the manner of its realization-one's "approach" or "attitude" toward it. Since the world, produced through practice of a temporal nature, always remains to be made or re-made, and since institutions and possibilities always await their reanimation and concordant deformation, we must recognize the importance of what is owed to the relation between the agent's approach and the world that is produced or reproduced in the encounter. From this principle derives Bourdieu's interest, beginning in the 1970s, in performative statements -"naming" and "social magic" -and in such phenomena as the "self-fulfilling prophecy", in each of which is posed the political and symbolic question of the definition of reality.

These are urgently present in a series of writings in the 1970s, when the fate of the French peasantry was at stake and appeared to pivot on whether the peasants themselves would accept as realistic a particularly pessimistic representation of their future (Bourdieu and Boltanski, 1978: 215).

At stake in such a struggle over a representation of reality is the character of reality itself (Bourdieu refers to it as a struggle over "the principles of di-vision"). The vision of the world really impacts upon its contours and divisions. To put this in terms consistent with the example, it can be said that the peasants' manner of approaching reality will -if only to an extent, in conjunction with other factors (Bourdieu, 1990b: 141) -come to determine this reality, by determining the "space of possibles" open to them. This suggestion, that the manner of approaching reality, of "realizing" it, impacts on the constitution of that reality itself, remains nonsensical so long as we conceive of the relation to the social world on the paradigm of the relation of the individual to a concrete situation. Heidegger's hammer does not yield to the dimensions of the fingers, but the social world, at least in certain respects, yields to the expectations of those who face it.

While this performative dimension obviously entails the possibility of "change", perhaps even political change for the better, Bourdieu certainly has, for various reasons, a predilection for identifying the continuities and repetitions, the "reproduction" of existing relations. Yet we must recall that such reproduction is by no means a fait accompli, but is the product of a process brought about through practice oriented by a particular view of the future, leading to what Bourdieu calls the "causality of the probable": the causality of the probable is the result of this kind of dialectic between the habitus, whose practical anticipations rest on all previous experience, and the probable significations, that is, the given that it gives itself by a selective apperception and a biased appreciation of the indices of the future that it must contribute to bringing about [faire advenir] (things "do to", "to say", etc.): practices are the result of this encounter between a predisposed and forewarned agent, and a presumed world, that is, a foreseen and prejudged world, the only one that he can ever know (Bourdieu, 1974: 28) Of course, if it need be noted again, the causality or "fatalism" of the probable is but one course of action, and sociological inquiry -producing "knowledge of the probable" -offers the possibility for "a rejection of the probable based on the scientific mastery of the laws of production governing the eventuality rejected" (Bourdieu, 1991: 136).

The struggle over the course of the world Since Bourdieu's world, an eminently historical one, is temporally structured, the gap between the first movement of a transformation and its plain manifestation should be seen less as an intellectual "realization" than as a realization in the sense of a practical accomplishment. The "environment" is only ever revealed as it is realized through agents' practices. This is to take seriously what Bourdieu means by the social world being the object of struggle: it is never "settled" and to register it in statistical analysis is but to take a snapshot of "a given moment" of the struggle, to "freeze" it in its "endless fluctuations" (Bourdieu, 1984: 245-246;Bourdieu, 1990b: 141). I shall conclude this paper with a brief survey of the auxiliary components of Bourdieu's model, as a means of further clarifying what I posit as a "flexible" relation between the social agent and the social world. The components in question are "transposition" and "reconversion". In different ways they subtend several of the processes identified by such concepts as habitus and field.

Transposition is a primordial mode by which the agent relates to the world. It is inherent in the notion of habitus, as demonstrated by Bourdieu's recollection of the empirical origins of the latter concept: "I was struck to see, for example, in the case of peasants, how their attitude to the development of their farming is at one with their attitude toward the education of their children. Here it is a single attitude toward the future" (Bourdieu, 1966a: 7). The transposition of dispositions entails a decidedly non-mechanistic approach and as such cannot be reduced to a "transfer" of practices across different domains, as Lahire attempts to do in suggesting that the motor skills bound up with skiing and rock-climbing meet their limit of transferability in moving, respectively, "from mountain skiing to water-skiing" (Lahire, 2010: 85) and from indoor to outdoor rock climbing. But this is firstly to fall into tautology, defining a motor skill as "skiing" and then claiming that it can only be deployed in "ski-like" situations, and it is secondly to negate Bourdieu's relationalist, sociological approach from the outset: such a habitus as the one underlying rock climbing -the "most ascetic form of the aesthetic disposition" (Bourdieu, 1984: 267) -is transposable to a practice like museum-going (and others of "the culturally most legitimate and economically cheapest practices"), all the while not supposing that the "same motor skills" -strictly defined -are being deployed. We are here dealing with something more profound than simple psychological schemes (pace [START_REF] Lizardo | The cognitive origins of Bourdieu's habitus[END_REF].

The idea of a transfer of habits, furthermore, would take us back to the idea of an ahistorical "context" that serves merely as a backdrop to the activation of inclinations and skills. From here derives the superficial "shock" model, of breakdown incurred by travel between fields. The latter are, for those authors who make use of this notion, conceived analogous to physical spaces. In place of such a statically defined habitus specific to each field -in which habitus would be ineffectual outside of that domain in which it was acquired -we ought to think of an underlying principle "of practices performed in fields governed by different logics and therefore inducing different forms of realization" (Bourdieu, 1984: 101). The habitus is accessed through "the typically Leibnizian method of possible worlds, [uncovering] several realizations of the same habitus" (Bourdieu, 1996a: 379).

Transposition implies flexibility to respond in stylistically homologous manners to the diverse realities or faces of reality presented to the actor. Here, a program -which operates on a ceteris paribus assumption (Dreyfus, 1992: 57) -fails. The capacity of a generative principle to have recourse to manifest practices different than those undertaken as a matter of automatic routine confirms the central advantage of a nonmechanistic conception of habitus. Transposition entails on the one hand similarity enough between the initial set of conditions and the conditions of "deployment", and on the other enough dissimilarity to render some "bridging" action necessary: "two 'realities' are never entirely alike in all respects but are always alike in some respect, at least indirectly" (Bourdieu, 1990b: 88;Hage, 2013: 87). Far from being reducible to a mechanical "scheme transfer", then, transposition figures as another indicator of the specificity of habitus vis-à-vis 'habit': given a practical imperative for the 'economical use of polysemy, fuzzy logic, vagueness, approximation' (Bourdieu, 2000: 57), habitual routine and strictly defined specific skills cannot be relied upon, as they would result in 'breakdown' each time an unanticipated situation appears.

Thinking of habitus, rather, as a generative principle offers a way of going beyond this 'given'. This is habitus as 'dexterous' and inventive, emerging from a confrontation between a set of practical principles and the world to which it relates.

As seen through the prism of habitus, practice is far from a repetition of the same practice in the same conditions. Rather it involves a series of more or less creative acts undertaken over time, implying 'a margin of for the inconstancies of the environment' (Canguilhem, 1991: 197). In the face of unexpected circumstances, one is hardly fated to catastrophe, but instead strives to apply one's dispositions in other ways; that is, one is encouraged to transpose them into another 'key'. In this way the habitus 'provides a practical mastery of situations of uncertainty' (Bourdieu, 2005b: 214), and enables 'agents to cope with unforeseen and constantly changing situations' (1990b: 61). 4 (The question becomes an empirical one of determining precisely how tolerant a habitus can be of variably volatile circumstances for practice to remain in the state of 'going without saying'.) Transposition can thus be seen as a form of 'dexterity', an option whose full significance is realized when we consider 'reconversion'. Reconversion refers to the strategy undertaken to maintain or augment one's position at the cost of relinquishing a declining form of capital in the hopes (by no means assured) of exchanging it for an ascendant or stable form. Here one intuits the inception of a crisis or decline and seeks to forestall it. Reconversion also offers yet another reason why habitus is irreducible to anything like routines or 'skills' and why the sense of orientation it implies cannot be exhausted by the subjective experience or reporting of the latter: as Bourdieu notes, the practices grouped together under the term reconversion can be experienced as a life-changing personal 'conversion', perhaps implying a shift in one's calling, yet with respect to the one's position in social space it involves a maintenance of the same (1996b: 291). That is, the same internalized sense of social space, or habitus, prevails amidst a 4 An unexplored avenue remains regarding this question: rather than supposing that increasing specialization and the proliferation of minor crises automatically bring about recurrent breakdown necessitating conscious deliberation, it would be as fair, from Bourdieu's perspective, to suppose that people undertake strategies to 'make lives in fragmented and volatile worlds rather than waiting for normalisation and reconfiguration' (Vigh 2008: 8;see Berlant 2011: 4). transformation of condition and all the habits and routines bound up with it (1984: 156).

Social agents undertake these contingency strategies, shifting habits and routines, as a means of retaining their social position, accomplishing this by playing on the pliability of the social world. In this light we should take seriously Bourdieu's statement that 'one makes for oneself an environment in which one feels "at home" and in which one can achieve that fulfilment of one's desire to be which one identifies with happiness' (2000: 150-emphasis added). This 'environment', I have sought to assert throughout, is most strictly conceived of as a space of social positions, rather than any kind of phenomenal environment, so that a shift in observable customs and clothing, for example, would not in itself constitute the kind of shift in the environment of interest to Bourdieu. Bourdieu's sociology defines environment by relative positions rather than by absolute conditions. Consequently, the habitus is primarily the 'sense' of social space (the structure of positions) that the agent possesses. The conditioned skills and habits bound up with their positions remain secondary, as suggested by agents willing, through reconversion, to dispense with them in order to retain position (1984: 455).

Conclusion

The foregoing primes us to explore some difficult issues in Bourdieu's work. First, a strong tension remains between condition and position (note, e.g., the language used in 1984: 172). Bourdieu remarks that class condition, a substantialist hold-over, was rightly disposed of as he came to a relational conception of social space, as defined by position (in Sapiro, 2004: 87;Bourdieu, 1966b: 204). While this might be so, it remains that the concept of habitus has, so to speak, one foot in each camp: primarily and most profoundly, it implies a deep attunement to a relationally-defined social position; yet it also suggests, with the language of 'conditioning', an unyielding place for class condition. Most of the commentary has heretofore favored this latter dimension at the expense of attention to relational position, a fact that concords with the intense focus on habit. By retaining a confused sense of the two facets -or by ignoring their difference completely -we have not yet begun to examine the interrelations between them.

Still, if the space of positions is granted primacy, then we must ask what place there is for 'condition' and all that is bound up with it. I refer here in particular to the body. If, against Lahire, I argue that transposition involves something more 'profound' than the transfer of motor skills from one activity to another, but rather a social sense of orientation allowing for one principle to conceivably result in apparently vastly different behaviors, then what do we gain from imputing this principle to the body, or to the body in any recognizable form? Formulated otherwise, if it be granted that the same disposition underlies rock-climbing and museum visitation -an eminently defensible proposition from a relational perspective -then why is it necessary to refer to the 'body' as the pivot and unifying point of practice in the world? What specific properties does the body possess that would grant it such a role? Put most bluntly, what is the status of the body in Bourdieu? While his innovation certainly makes the important step of overcoming a strictly 'semiotic' conception of the body and emphasizes the material basis of human finitude, we ought to ask whether the body plays any greater role than as a functional term: that into which is projected, and through which is extolled, all non-cognitive properties hitherto derided by the dominant philosophical tradition (Martin and George, 2006: 126). 5In any case, I have offered a reading of Bourdieu that attempts to bring the 'relationalist' elements of his work into the finest focus, and in doing so clarifies much of the commentary, particularly regarding the topic of consciousness, the character of the social world, and the relations between them. Most importantly, I have suggested that manners of 'constructing' sociological phenomena -whether on the part of social agents themselves or those seeking to understand them -have real effects on the 'world' that is encountered.

In this article I understand by mechanism, in Bourdieu's terms, a way of construing events (such as actions and responses) in the form of an interlinked chain of 'moments in a sequence of programmed actions produced by a mechanical apparatus'(Bourdieu 1977: 73).

This can be seen as a Bourdieusian version of the Beck "reflexive modernization" thesis: "the more societies are modernized, the more agents (subjects) acquire the ability to reflect on the social conditions of their existence and to change them accordingly"(Beck, 1994:174). See[START_REF] Adkins | Reflexivity: freedom or habit of gender? Theory[END_REF],[START_REF] Crossley | The phenomenological habitus and its construction[END_REF]: 114), Sweetman (2003: 541), and McNay (1999) with direct respect to Bourdieu.

This is obviously not to deny the empirical existence of situations of "crisis or sudden change" (2000: 161), but it is to deny that these ought to be taken as paradigmatic instances of "change" or transformation.

The Bourdieusian thinker could respond that it is an error to conflate the "body" with the motor cortex, as if all practices have to be reduced to their specifically motor component (which would amount to reducing habitus to bodily habit). Yet this does nothing to rid us of the question of what the body specifically offers.


Matthieu Zimmer
email: matthieu.zimmer@lip6.fr

Paolo Viappiani
email: paolo.viappiani@lip6.fr

Paul Weng
email: paul.weng@lip6.fr

Teacher-Student Framework: A Reinforcement Learning Approach

Keywords: reinforcement learning, reinforcement learning with rich feedback, teaching on a budget

. Moreover, we propose a technique for a student to take into account advice more efficiently and we experimentally show that performances are improved in Torrey and Taylor's setting.

Introduction

Reinforcement learning (RL) [START_REF] Kober | Reinforcement learning in robotics: A survey[END_REF] is a framework for solving sequential decision problems where an agent interacts with the environment and adapt her policy taking into account a numerical reward signal. RL agents can autonomously learn somewhat difficult tasks, like navigating a maze or playing a video game. While the basic setting of RL is now well established, recently a number of researchers have been studying variants where agents have access to demonstrations of successful task completion (learning by demonstration or by imitation [START_REF] Argall | A survey of robot learning from demonstration[END_REF][START_REF] Billard | Learning human arm movements by imitation: : Evaluation of a biologically inspired connectionist architecture[END_REF]), or where the knowledge acquired for a particular task needs to be used to solve a similar related task (transfer learning [START_REF] Taylor | Transfer learning for reinforcement learning domains: A survey[END_REF]) or other settings where agents interacts with each other. The common ground of these works is that learning can be faster by exploiting additional information for the task at hand; this intuition goes back to the idea behind reward shaping [START_REF] Ng | Policy in variance under reward transformations: Theory and application to reward shaping[END_REF][START_REF] Laud | The influence of reward on the speed of reinforcement learning: An analysis of shaping[END_REF].

In this paper, we consider an agent charged with the task of teaching another agent how to perform a particular task. The idea of integrating teaching in RL is not new (e.g., [START_REF] Lin | Programming Robots Using Reinforcement Learning and Teaching[END_REF][START_REF] Thomaz | Reinforcement Learning with Human Teachers: Evidence of Feedback and Guidance with Implications for Learning Performance[END_REF][START_REF] Knox | Understanding Human Teaching Modalities in Reinforcement Learning Environments: A Preliminary Report[END_REF]). Recently, a number of researchers have proposed similar settings but with different assumptions and communication protocols [START_REF] Cohn | Comparing action-query strategies in semi-autonomous agents[END_REF][START_REF] Akrour | April: Active preference learning-based reinforcement learning[END_REF][START_REF] Fürnkranz | Preference-based reinforcement learning: a formal framework and a policy iteration algorithm[END_REF][START_REF] Akrour | Interactive robot education[END_REF]. However, in this literature, the teacher is always assumed to be a human. We assume here that the teacher is an artificial agent. This teaching problem was studied by Torrey and Taylor [START_REF] Torrey | Teaching on a budget: Agents advising agents in reinforcement learning[END_REF] who introduced a teacher-student framework for reinforcement learning and proposed a number of simple heuristic methods for choosing in which situations to provide advice (the best action) to the student. An important assumption in this framework is that a "budget" (maximum number of suggestions that can be given) is fixed, and therefore it is really important to carefully decide when (and which) advice to give. The constraint of a limited budget for teaching is justified in two situations [START_REF] Torrey | Teaching on a budget: Agents advising agents in reinforcement learning[END_REF]. In some problems, communication is constrained and limited (e.g., cost of communication), even if all agents are artificial. This framework could also be extended to human students and in this case, obviously the number of advice has to be small.

The setting in this paper assumes that the teacher (who knows the optimal policy) observes the student (her state) but cannot monitor nor change anything internal to the student. Communications rely on a very simple protocol: at each step of an episode, the student announces her intended action and the teacher can decide whether to provide some advice. A piece of advice consists in suggesting the action that the student should do (according to the optimal policy, known to the teacher but not to the student). In this paper, we try to answer two questions. From the teacher's point of view, how advice can be given efficiently? From the students' point of view, how can a piece of advice be efficiently exploited (in the RL setting)?

In this paper we assume a principled decision-theoretic approach where teaching is itself formulated as a reinforcement learning problem. Our intuition is that in this way we can provide efficient techniques for choosing the right moment at which advice should be given. For instance, the teacher should be able to autonomously learn that a student who chooses a good enough action does not need advice, or that most of the advice (but not all) will be needed in the first episodes of training, and that in some states it is more critical to perform a good action than in others. Moreover, with our approach, the teacher could adapt to particular types of learners and overcome differences in state representation.

Reinforcement Learning

Reinforcement learning (RL) [START_REF] Sutton | Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning)[END_REF] is a framework that models sequential decision problems where an agent learns to make better decisions while interacting with the environment. After an agent performs an action, the state changes and the agent receives a numerical value, called reward, that encodes "local" information about the quality of the action just taken. The goal of the agent is to maximize her long-term expected total reward. Because it is possible that actions associated with low reward will allow the agent to reach high-reward states, the agent needs to estimate somehow (by trial-and-error) the value of making an action in a given state.

The underlying formalism of RL is that of Markov Decision Processes (MDPs). An MDP is formally defined as a tuple S, A, T, R where S is a set of states, A a set of actions, T : S × A × S → [0, 1] are transition probabilities between states (T (s, a, s ) = p(s |a, s) is the probability of reaching state s from state s after executing action a) and R : S × A → R is a reward signal. A (deterministic) policy π : S → A is a mapping from states to action (the action to be taken in each state) and encodes how the agent will behave. The optimal policy π * is the policy maximizing the expected total reward, that is (for episodic reinforcement learning with a horizon T max ):

π * = arg max π E[ Tmax t=0 R(s t , π t (s t ))] (1)
where t denotes a time step and T max the maximum number of steps. In MDPs, since T and R are given, an optimal policy π * can be computed offline using dynamic programming (e.g., value iteration, policy iteration) [START_REF] Howard | Dynamic Programming and Markov Processes[END_REF] for instance.

In the RL setting, the agent however does not know the transition probabilities T and the reward function R. There are two main approaches for learning an optimal policy in RL: model-based approach and model-free approach. While model-based methods aim at learning T and R, model-free methods are computationally more viable techniques that perform online learning storing an estimation of Q : S × A → R, the (sequential) values of actions in each state. Whenever an action is taken and a reward is observed, the agent updates the value of Q(s, a) using the following formula:

Q(s, a) = Q(s, a)+α[ R(s, a) -Q(s, a)
adjustment given observed reward + γQ(s t+1 , a t+1 )

estimation of future rewards ] (2)
where α ∈]0, 1[ is the learning rate (empirically, if the task is highly stochastic α needs to be low, alternatively it can be set to a high value and then the agent might be able to learn faster) and γ the discount rate.

In RL the agent faces the exploitation versus exploration dilemma, that means choosing whether to focus on high reward actions (but at the cost of ignoring potentially more valuable regions of the world) or performing actions with the intent of exploring other regions (but at the cost of giving up reward in the short term). One simple strategy to solve this tension is -greedy, that exploits most of the time, but with a small probability performs a (random) exploration move, i.e., π(s) = arg max a Q(s, a) with probability 1-and π(s) is a random uniformly drawn action in A with probability . To determine a good value for one often relies on experimental observations. Among the most common algorithms for reinforcement learning are Sarsa and Q-learning [START_REF] Sutton | Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning)[END_REF].

Notice however that in many tasks, the number of states and actions are too large so that the agent cannot store Q-values in a table. The problem of continuous states can be tackled with discretization; however the number of states will often explode. Even with the memory and computational power to handle very large tables, reinforcement learning with Q values represented in the tabular form suffers from the lack of generalization, meaning that what is learned in a state does not impact our knowledge about the value of similar states. A solution to these problems is to use an approximation function, that provide a means for both efficient computation and support for generalization. The Q values is approximated by a function ψ θ : S × A → R, i.e., Q(s, a) ≈ ψ θ (s, a) parametrized by a vector θ whose values are updated whenever receiving reward. In this way, we can represent Q-values for continuous domains. Moreover this allows generalizing what it is learned in a state to a different, but similar, state (notice that ψ θ can take many forms of supervised learning methods: neural networks, linear regression, mixture of Gaussians, ...). In this paper, we assume that the Q-values are expressed as a linear approximation function

Q(s, a) ≈ ψ θ (s, a) = i θ i × f i (s, a) (3)
where the functions f i : S × A → R are given basis functions.

3 Teaching on a Budget Torrey and Taylor [START_REF] Torrey | Teaching on a budget: Agents advising agents in reinforcement learning[END_REF] introduced a teacher-student framework for reinforcement learning. In their framework, a student agent learns to perform a task through reinforcement learning, while a teacher agent may suggest an action in order to help the student learn faster. A few assumptions are made in this framework. First of all, the teacher is also a RL learner and has already learned an optimal policy. Moreover, the teacher and the student share a common action set. Besides, the teacher can only give a limited number of advice, called budget. Finally, the only means of communication between the two agents consists in the possibility for one agent to communicate an action to the other. More specifically, the student announces the action that it is about to perform (before actually performing it) and then the teacher may decide to give advice, in the form of an action that the student should perform instead (an action better than the announced one). Formally, the framework can be described as follows. Given a task to be learned by the student agent, let M task = S task , A task , T task , R task be an MDP representing it. The state representation of the student is denoted by R student . The state representation of the teacher, which is not necessarily identical to R student , is denoted by R teacher . Typically, R student and R teacher are defined by the shape of the tiles and their numbers; they can be viewed as "how the agents perceive the environment".

The student starts in an initial state and learns through trial and error (i.e., with any RL algorithm). We call episode a sequence of time steps from an initial state to a goal state if the student manages to reach it or when a fixed maximum number of steps is reached. When an episode ends, the student starts over in a new initial state. We call session the sequence of episodes before the student learns a good enough policy.

During a student learning session, the teacher may choose to advise the student (i.e., communicate the best action according to the teacher's optimal policy to take in the current state of the student). The teacher can decide to spend her budget (i.e., the limited number of advice) through the whole learning session, not only on one episode. Torrey and Taylor propose the following heuristic methods (where advice is given if a condition is satisfied) for the teacher to decide to give advice or not. We review them below:

Early advising This very simple strategy consists in the teacher spending all her budget at the very beginning of the session, providing advice at each step until the budget is over. In this strategy, the student agent does not need to announce the action that it is going to perform. Importance advising This strategy consists in giving advice only in states that are deemed "important". For evaluating the importance I(s) of a state s, Torrey and Taylor [START_REF] Torrey | Teaching on a budget: Agents advising agents in reinforcement learning[END_REF] adopt the following measure (initially proposed by Clouse [START_REF] Clouse | On integrating apprentice learning and reinforcement learning[END_REF]):

I(s) = max a Q(s, a) -min a Q(s, a)
. Then, at any timestep of a session, until the whole budget is spent, advice is given only if the importance I(s), of the state s the student is currently in, is greater than a given (empirically optimized) threshold. In this strategy, the student does not announce her action. Mistake Correcting In the previous heuristics, the teacher wastes her budget on situations where the student would have chosen on her own the recommended action. Denote π * the optimal policy learned by the teacher. Mistake correcting consists in giving advice only if π * (s) = a student and the state importance I(s) is greater than a threshold.

Actually, Torrey and Taylor proposed another heuristic, Predictive Advising, extending Mistake Correcting, where the student does not need to announce her action. Using a classifier (in their implementation, a support vector machine) the teacher learns to predict what action the student is likely to do next. While these heuristics have proved successful in experiments, in the following we show that by modeling the decision of when to give advice in a principled decision-theoretic way (as a sequential decision problem), we can provide more effective teaching agents. In the next section, we present how we model the teaching task as a RL problem.

Learning to Teach

There are two main components in our approach. First, we model the teaching task as a reinforcement learning problem (Section 4.1), i.e., the teacher agent needs to learn a policy for deciding when to give advice. Doing so, we expect that the teacher can learn a better policy for giving advice than those based on the previous heuristics. Second, the student can also exploit more efficiently the pieces of advice given by the teacher. We present a technique for improving the estimation of the learner's Q-values in order to be consistent with the information provided by the teacher (Section 4.2). Note however that such a technique can be used independently from the policy used by the teacher.

The Teacher Model

In our approach, the teacher learns how to teach RL students efficiently, without resorting to the previous heuristics, using reinforcement learning as well. As in the teacher-student framework proposed by Torrey and Taylor [START_REF] Torrey | Teaching on a budget: Agents advising agents in reinforcement learning[END_REF], the teacher first learns an optimal policy π * for M task in her state representation R teacher . Then, the teacher tries to learn a teaching policy, which would solve the same problem as the heuristics methods recalled in Section 3.

Let M teacher = S teacher , A teacher , T teacher , R teacher be a MDP describing the teaching task. The set of states S teacher is defined by S task × A task × {0, 1, . . . , b max } × N where b max is the initial budget: the current state in S task of the student, the action in A task announced by the student, a remaining budget and a learning episode number of the student. Note that the state of the student is observed in the teacher's state representation R teacher . Recall that it may differ from that of the student R student . An action is defined as an element of A teacher = {true, f alse}, telling whether an advice is given or not. If the teacher decides to give advice for a state s, he sends π * (s) to the student. The reward function defined for M teacher is

R teacher (s, a) = r max - nstep r d if student reached goal -1 otherwise ( 4
)
where n step is the number of time steps the student needed to reach a goal state, positive constant r max is the greatest reward obtainable in M task and r d is a positive constant such that the maximum number of time steps of an episode divided by r d is greater than r max . The definition of this reward function is motivated as follows. The longer it takes for the student to reach her goal, the less the teacher is rewarded. Thus, this implies that the teacher will try to minimize the number of learning time steps for the student. Note that, due to r d , the teacher is always better off when the student reaches a goal state, however long it takes. We call session one episode of this MDP, i.e., a full learning session of one student over M task . Through several sessions, the teacher experiments on multiple learners.

Effect of Advice

In their paper, Torrey and Taylor [START_REF] Torrey | Teaching on a budget: Agents advising agents in reinforcement learning[END_REF] considers a piece of advice as an informed exploration : the recommended action is performed (instead of the announced action) as if it was chosen by the -greedy strategy, and the Q-value is updated using the usual formula. This simple strategy has the advantage that the modification of the student's RL algorithm is minimal. However, to make it effective, all Q(s, a)'s need to be initialized in a pessimistic way, i.e. at a very low negative value. Doing so guarantees that a recommended action performed once has a greater chance of being chosen again as the best action when visiting the same state in the future.

In this section, we propose a more sophisticated strategy for the student to take into account a recommended action. This new approach will change more the student RL algorithm, but at the benefit of better estimations of Q-values. In this approach, called max update, we want the student to exploit a piece of advice not only once, but several times if needed. To that aim, we endow the student with a memory of the previous recommended actions and the corresponding states (of course, in the student state representation). We present the approach in the tabular version first to make it easier to understand, then we extend it to the case where approximation functions are used.

The first time an action a r is recommended in a state s r , the student performs it like in the informed exploration approach. Although the student does not know yet the correct value for Q(s r , a r ), it nonetheless knows that the following property shall be kept true:

∀a ∈ A, Q(s r , a r ) ≥ Q(s r , a) (5)
If this property is not satisfied for s r , then there exists an action a max , different from a r , such that a max = arg max a Q(s r , a). The naïve update where Q(s r , a r ) is set to the value of Q(s r , a max ) may not work as there is no guarantee that Q(s r , a max ) has already converged and that it may not be a good approximation of the true Q-value. Instead, we propose to make the following update:

Q(s r , a r ) = Q(s r , a r ) + α × (Q(s r , a max ) -Q(s r , a r )) (6)
at every time steps. These updates help Q(s r , a r ) converge faster towards the target value Q(s r , a max ) as their aim is to minimize the quadratic difference of the two Q-values. Besides, as after each time step in the student RL algorithm, the estimation of Q(s r , a max ) may improve, the updates of Equation 6 may use better and better approximations. If at any time, the property described by Equation 5 is satisfied, these updates are not needed anymore and they are stopped.

For the version using a function approximation (Equation 3), the update needs to be performed on parameter θ. A component i of θ can be updated with the following formula

θ i = θ i + α j θ j (f j (s r , a max ) -f j (s r , a r ))(f i (s r , a max ) -f i (s r , a r )). (7)
This is a standard gradient descent for minimizing the quadratic difference of Q(s r , a r ) and Q(s r , a max ). We believe that this strategy is efficient in both settings (tabular RL and function approximations) and in Section 5.4 we indeed show that can boost the performance of a teacher. Another advantage of this new method is that we do not need to initialize the Q(s, a) to a very low value (pessimistic initialization), since the property of Equation 5 is ensured by a series of specific updates (Equation 6 and Equation 13); however this comes at the expenses of additional computation at each step of the execution.

Experiments

We evaluate our two strategies comparing it to the heuristic methods proposed by Torrey and Taylor [START_REF] Torrey | Teaching on a budget: Agents advising agents in reinforcement learning[END_REF]. In the next three subsections, we recall the mountain car domain [START_REF] Satinder | Reinforcement learning with replacing eligibility traces[END_REF], a popular benchmark for reinforcement learning and give some details about the implementations of our two approaches. Then, we present our experimental results in the last subsection. A standard testing domain in reinforcement learning is the mountain car problem [START_REF] Satinder | Reinforcement learning with replacing eligibility traces[END_REF]: an under-powered car must drive up a steep hill (Figure 1). The gravity is stronger than the car's engine so, even at full throttle, the car cannot simply accelerate up the steep slope; the agent must learn to gain momentum as on a swing. The state is an element of a 2 dimensional continuous space: (x, y) ∈ P osition × V elocity with

Mountain Car

P osition = [-1.2, 0.6] V elocity = [-0.07, 0.07] (8)
The actions are valued in a one dimensional discrete space: a ∈ {-1, 0, +1} corresponding to the directions of the acceleration, i.e., respectively backward, no acceleration and forward. The reward for every step is -1 except when the goal is reached (reward is 0). The transitions are defined as follows:

y t+1 = y t + a × 0.001 + cos(3 × x t ) × -0.0025 x t+1 = x t + y t (9)
If the position or the velocity go out of the bounds, their values are set at the extreme bound of the respective domains. For the position, this corresponds to an inelastic wall on the left hand side and the goal on the right hand side. The car begins each episode with zero velocity at the bottom of the mountain (initial position (x 0 , y 0 ) = (-0.5, 0)), and the episode ends when it reaches the goal at the top (whenever x ≥ 0.6), or after 500 steps.

Since the state space is continuous, we rely on a function approximation of Q, with the common approach of using a linear approximation function with a binary tile coding:

Q(s, a) ≈ ψ θ (s, a) = i θ i × f i (s, a) (10)
where f i : S × A → {0, 1} with the semantics that a feature f i is "active" only in some states (and therefore parameter θ i only impacts the computation of Q for some regions of the world). We denote by F the set of features that are active for a state-action pair, i.e., F (s, a) = {i | f i (s, a) = 1}. In our implementation, we assume a common tiling structure (of rectangular shape) for all actions and, moreover, a feature can only depend on one action at a time (i.e., for a given state s and two different actions a and a , the number of active features is the same, while their intersection is empty). More formally, the two following properties are satisfied:

(i) ∀s ∈ S, ∀a, a ∈ A, a = a ⇒ (F (s, a) ∩ F (s, a ) = ∅) independence ∧(|F (s, a)| = |F (s, a )|) (ii) ∀a, a ∈ A, ∀f i , ∃f j , {s ∈ S | f i (s, a) = 1} = {s ∈ S f j (s, a ) = 1}.
A common way of increasing the learning speed is to keep a trace of the last updated pairs (s, a). In our implementation we use an accumulative trace:

e t (s, a) = γ × λ × e t-1 (s, a) if s = s t or a = a t 1 + e t-1 (s, a) otherwise ( 11
)
where λ ∈ [0, 1] is the size of the trace, γ is the discount factor. The update step will adjust the Q-value of all pairs (s, a) for which e t (s, a) = 0, using the formula:

Q(s, a) = Q(s, a) + α × [R(s, a) -Q(s, a) + γ × Q(s t+1 , a t+1 )] × e t (s, a) (12)
The algorithm used for students and teachers is Sarsa(λ). In our simulations, the different reinforcement learning parameters for the student are = 0.05, α = 0.08, λ = 0.9, γ = 1 and those for the teacher are = 0.001, α = 0.02, λ = 0.99, γ = 0.99. For R student , the number of tilings is 8 and the number of segments in one tiling is 16 × 16.

Implementation of the RL teacher

The state of the teacher is represented by families of independent features. Recall that the teacher state s teacher = (s student , a student , b, n e ) is a tuple containing the student's current state s student in S task , the student's declared action a student in A task , the remaining budget b and the learning episode number n e . The number of tilings is set to 8 for each component of s teacher . The families of features are:

-A set of features g i (s student , a teacher ) jointly representing the student's position and her velocity and a teacher's action a teacher corresponding to the decision of advising or not. The tiling structure corresponding to the student's state is qualitatively similar to the student's state representation (but the actual positions of the tilings differ from those of the student's own representation). The number of g i 's is therefore n g = 16 × 16 × 8 × 2 = 4096. -A set of features h i (a student , a teacher ) representing the action a student announced by the student (backward, no acceleration,forward). There are therefore n h = 3 × 8 × 2 = 48 of features of this type. -A set of features m i (b, a teacher ) representing the available budget (the maximum allowed budget b max is discretized into b max /3 slots). For the m i 's, the number of features is

n m = b max /3 × 8 × 2.
-A set of features l i (n e , a teacher ) that discretize the number of episodes into n max /3 subintervals (to have a fine-grained representation of the temporal position of the current episode in the training session) where n max is the maximum allowed number of episodes of a session. There are n l = n max /3 × 8 × 2 features of type l i . Thus, the functional approximation of Q(s, a) for the teacher is computed as

Q(s teacher , a teacher ) ≈ψ θ (s teacher , a teacher )
where

ψ θ (s teacher , a teacher ) = ng i=1 g i (s student , a teacher )θ g i + n h i=1 h i (a student , a teacher )θ h i + nm i=1 m i (b, a teacher )θ m i + n l i=1
l i (n e , a teacher )θ l i with the parameter vector θ decomposed as (θ g , θ h , θ m , θ l ).

Implementation of max-update

Recall the set of indices of active features in state-action pair (s, a) is denoted F (s, a) = {i | f i (s, a) = 1}. We denote with θ(F (s, a)) the tuple of the corresponding components of parameter θ. Thanks to the two assumptions (i) and (ii) of the tiling structure, the update (Equation 7) for our max-update approach can be simplified and written as follows:

θ(F (s r , a r )) = θ(F (s r , a r )) + α × [θ(F (s r , a max )) -θ(F (s r , a r ))] (13)
Vector θ(F (s r , a r )) can be seen as the output vector of a neural networks, and θ(F (s r , a max )) as the desired vector. This update is a well-known gradient descent and helps make θ(F (s r , a r )) converge to θ(F (s r , a max )).

Experimental Results

In the experiments, we compare the heuristics proposed by Torrey and Taylor [START_REF] Torrey | Teaching on a budget: Agents advising agents in reinforcement learning[END_REF] and our approach based on representing the teacher as an RL agent. All plots are averaged over several runs, and the threshold of mistake correcting has been optimized to obtain the best performance (indeed if the threshold is too high, the heuristic will perform suboptimally as it will not spend all the available budget; conversely, if it is too low it will perform poorly as advice will be given also when not particularly relevant). We do not display the importance advising strategy because it is always outperformed by mistake correcting, since we assume that we are in a setting in which the learner agent announces her intended action before the teacher has the possibility of providing advice (importance sampling ignores this piece of information).

The RL teacher needs to interact with several learners before it learns a nearoptimal teaching policy. In Figure 2 we plot the student's reward in function of the number of episodes of training for a teacher at different levels of training (different number of sessions with students). As expected, the student's performance improves when the teacher is more experienced. The teacher finishes to learn after approximately 8000 sessions (learning ends when Q-values converge).

In the following experiments we consider the performance of a fully trained RL teacher (meaning that the agent has already completed a number of sessions with different students, each composed of several episodes, until the teaching policy has converged). In Figures 3456we compare our RL teacher to the heuristics presented in Section 3 for a variety of values of the total available budget. We Fig. 3. Average reward of students with the fully trained RL teacher, compared to early advising and mistake correcting (budget=25; threshold for mistake correcting=76) Fig. 4. Average reward of students with the fully trained RL teacher, compared to early advising and mistake correcting (budget=50; threshold for mistake correcting=148) Fig. 5. Average reward of students with the fully trained RL teacher, compared to early advising and mistake correcting (budget=75; threshold for mistake correcting=128) Fig. 6. Average reward of student with the fully trained RL teacher, compared to early advising and mistake correcting (budget=100; threshold for mistake correcting=4) can notice that our RL teacher performs very well, outperforming the other strategies in all but one scenario (the high budget setting where all strategies perform roughly the same). Indeed, the bigger is the budget, the better are the performances of all teachers. The simplest strategy, early advising, performs poorly in most cases, but when the budget is high it becomes successful: the student will complete the task following the teacher's advice in each step of the first episode; given the pessimistic initialization of Q-values, the student will subsequently follow the same trajectory in successive episodes (except in some exploratory moves occurring with low probability).

As mentioned above, the threshold for mistake correcting has been tuned for best performance. It is interesting to notice that there is not a simple correlation between this threshold and the available advice budget. Indeed, the best threshold is somewhat low with a budget of 25 or 100 (Figures 3 and6), and high for a budget of 50 or 75 (Figures 4 and5). An explanation of this may be that if there is not enough budget, mistake correcting cannot be of too much help, on the other end, if one can give advice often, the problem becomes trivial because the student can reach the goal in the first episode just following the teacher's advice all the time.

These experimental results show that our RL teacher (when fully trained) is effective in advising students; moreover we do not have to rely on manual parameter optimization as mistake correcting does. The superior performance of our RL teacher can be explained in many ways. Intuitively, we postulate that the RL teacher is capable of autonomously discovering which states are more important, also in relation to the point of the session. Since actions are represented as features, the teacher can also learn that putting the motor in neutral position is a less severe error than moving forward when instead one should move backwards (so in the first case giving advice might not be necessary, while in the second case it will be). The availability of information about the episode in the teacher's state is crucial for learning an effective teaching policy. Indeed, for a given pair (state, action) in two different episodes of the session, we do not necessarily give advice in both.

In a final experiment, we compare the two alternative ways of using a piece of advice given by the teacher: informed exploration and max-update. In Figures 78we show (for different values of the budget) how our proposed method (maxupdate) consistently enhances the heuristic strategies presented in Section 3. We are currently working on integrating this technique into our RL teaching agent.

Conclusions

In this paper we introduced a reinforcement learning approach to learning how to give advice to another RL agent (the student) in order to optimize the use of advice since only a limited number of suggestions can be given. The agent acting as a teacher observes the student's actions but cannot monitor nor change anything internal to the student. This "teaching on a budget" setting was introduced by Torrey and Taylor [START_REF] Torrey | Teaching on a budget: Agents advising agents in reinforcement learning[END_REF], with the specific goal of studying "the idea of how an RL agent can best teach another RL agent" but the authors did not pursue this vision all the way through, as in fact their teacher agent follows very simple if-then-else heuristic rules to decide whether to give advice or not.

In our work the teaching task is modeled as an RL problem whose state, in addition to the state of the student, includes information about the current teaching session (the action that the student is intended to perform, the available advice budget, the number of the episode); her possible actions represent the decision of whether giving advice or not in the current state of the student. With simulations, we showed that our techniques are competitive with respect to previously proposed heuristic techniques. Our RL teacher is effective because it can adaptively learn in which states and in which moment of the training session advice is more needed, it can learn how to spend well her budget when the budget is critically low and it is able to learn that a student agent who chooses a good enough action does not need advice.

We also considered an improved way on how the student can exploit the advice it receives from the teacher. Our max-update strategy manipulates the estimated Q-values in order to be consistent with the information that the suggested action is the best in that particular state. This come at the expense of a considerable modification of the student's architecture and of additional computation. We experimentally proved this idea to be successful in improving the mistake-correcting heuristics, but we did not yet implement it in our RL teacher.

We can extend our approach empowering our RL teacher with a classifier able to predict what the agent will do next (in the same way as the predictive advising strategy of Torrey and Taylor [START_REF] Torrey | Teaching on a budget: Agents advising agents in reinforcement learning[END_REF]) so that the communication between the two agents can be simplified, removing the assumption that the learner agent pre-announces her intended action to the teacher.

Other possible directions for future works include experimentation in other domains, evaluating how effective advising is when the teacher's policy is nearoptimal, a setting with multiple teachers and/or multiple learners, and different types of feedbacks (including pairwise comparisons of actions as in the work of Cohn et al. [START_REF] Cohn | Comparing action-query strategies in semi-autonomous agents[END_REF], advice requested by the student, noisy advice).

Fig. 1 .

 1 Fig. 1. The Mountain Car Problem

Fig. 2 .

 2 Fig. 2. Average reward of students at different levels of training of a RL teacher.

Fig. 7 .Fig. 8 .

 78 Fig. 7. Student's reward considering different heuristics with either informedexploration or max-update (budget = 50; threshold for mistake correcting with informed exploration: 148; threshold for mistake correcting with max update: 160)

Acknowledgments

Funded by the French National Research Agency under grant ANR-10-BLAN-0215.