Marc Calvini-Lefebvre

"Women! Your Country Needs You!": Fleeing Feminism or Gendering Citizenship in Great War Britain? *

Keywords: National Union of Women's Suffrage Societies, feminism, citizenship, gender and war, First World War, Millicent Fawcett, gendering, ideology

" Women andWar" Debate in British Feminism, 1905-1918. He is a founding member of the History of Feminism Network (http://historyfeminism.wordpress.com).

Introduction: War in, feminism out?

When the Great War broke out in August 1914, women across the United Kingdom turned, like Vera Brittain, to "the only work it seems possible as yet for women to dothe making of garments for the soldiers" (Quoted in Kent 1993, 14). In those first few months of war, Ray Strachey recalled, "[it] seemed almost as if the old anti-feminist argument was true, and that in a time of national crisis women were superfluous and irrelevant, passive creatures to be fought for, whose only personal function was to sit at home and weep" [START_REF] Strachey | The Cause. A Short History of the Women's Movement in Great Britain[END_REF](Strachey [ ] 1978, 338), 338). Perhaps S. Bulan captured the spirit animating the country best in declaring that "[a]t the call of war, the first thought of every man is to fight, of every woman to nurse" (Bulan September 1914, 267). In other words, the eruption of total war had "acted as a clarifying moment" that "revealed the [British system] of gender in flux and thus highlighted [its] workings" (Higonnet et al. 1987, 5).

And those workings were clearly along the most traditional of "separate spheres" lines, with men expected to fight and protect the literal and metaphoric home, and women expected to support their protectors unflinchingly and mind the home dutifully in their absence. In short, the outbreak of the Great War gave renewed currency and plausibility to ideas and practices that were strongly identified with the Anti-Suffrage movement.

It was against this background that the constitutionalist National Union of Women's Suffrage Societies (NUWSS), the largest organization in the British women's movement, decided to temporarily suspend its suffrage agitation and turn its considerable organizational abilities instead to relief work targeted specifically at women and children.

As it appeared to accept and even sanction the renewed currency that separate spheres thinking was enjoying, this abrupt turn to work so clearly defined along traditional gender lines has been presented as conclusive evidence of a broader trend: British feminism's ideological collapse in the face of war. This case has been put most forcefully by Susan Kent. Describing the NUWSS's reaction as "unreflective, almost knee-jerk" (Kent 1993, 16), she draws on the work of Denise Riley to argue that its flight from feminism was twofold:

The NUWSS failed to challenge [the dominant] images and representations of women; its valorizing of women's fundamental identity as mothers and homemakers constituted an embracing of what Riley calls "the social". In contrast to its prewar stance, however, feminists in the National Union failed to take the next step, to utilize women's association with "the social" in order to make claims for participation in "the political" [Kent 1993, 22]. 1 Yet, if the activities of the National Union certainly drew on the association of women with "the home," to claim that it did not attempt to use these actions as a stepladder into "the political" is to ignore the countless claims by both its leaders and members that in engaging in relief work they were, in the words of their president Millicent Fawcett, showing themselves "worthy of citizenship, whether our claim to it be recognized or not" (Fawcett August 7 1914, 376, emphasis added). Entry into "the political," however tentatively formulated, was certainly part of the agenda. This may not yet make the agenda feminist however. Jo Vellacott has indeed suggested that Fawcett's instinctive adoption of "a traditional view of the appropriate role of women in wartime" was "reinforced by the realization that women's 'good behavior' during the war might further their admittance to the political system -on men's terms" (Vellacott 1987, 121). So admittance to "the political" may well have been sought, but on principles which owed little to pre-war feminist ideas. A more subtle form of ideological capitulation perhaps, but an ideological capitulation nonetheless.

Yet behaving in ways that (anti-suffragist) men find palatable is not the same thing as adopting their vision of the world. This has been a central contention of scholars such as Jacqueline de Vries and Nicoletta Gullace, who have sought to recover the distinctively feminist dimension of another alleged symptom of feminism's war-time ideological collapse: the xenophobic jingoism of Christabel and Emmeline Pankhurst, the charismatic leaders of the militant Women's Social and Political Union (de Vries 1994, [START_REF] Gullace | The Blood of Our Sons : Men, Women, and the Renegotiation of British Citizenship During the Great War[END_REF], see also [START_REF] Purvis | Emmeline Pankhurst: A Biography[END_REF]Purvis 2007). By paying close attention to the context, their analyses show that the Pankhursts' patriotic discourse was not identical to that of the conservative press in either its specific content or intended effects. What they were doing, they reveal, was not capitulating to patriotism but gendering it with a specific goal of obtaining votes for women. 2 Far from demonstrating the ideological collapse of British feminism in the face of war, their pro-war stance is suggestive instead of the resilience of feminist thought. As Gullace puts it: "the Great War gave patriotic suffragists (…) a more auspicious environment in which to justify claims that had a long and important history within the feminist movement" (Gullace 2002, 6). 3 Such ideological resilience also marked, I will argue in a similarly contextually sensitive reading in this article, the response of Fawcett and her followers in the NUWSS to the outbreak of war. 4 Drawing on Common Cause and the highbrow monthly that was closely linked to it, The Englishwoman, this article shows that the main object of that response was to appropriate the concept and language of citizenship for women. 5 This was done firstly by arguing that the gendered nature of citizenship had been confirmed by the war itself. This allowed for a reframing of the language of separate spheres as a non-hierarchical, complementary discourse, whilst radically redrawing the boundaries between men and women's respective spheres, to the benefit of the latter. It was done secondly by developing a rhetorical strategy of inclusion that sought to appropriate the language of citizenship for women whilst simultaneously rejecting all attempts at excluding them from the national community. 6 The result was an insistence on women's identity as "citizens," an identity which in turn had important consequences for the kind of feminism that could be articulated in its name.

Gendering citizenship and redefining the "woman's sphere" A month into the war, the front page of Common Cause strongly encouraged readers to turn to its correspondence section, where "A Member of the Newcastle Society" had written in to reject the "war argument," that is the claim that women could not be citizens because they could not fight ("Opportunity" 28 August 1914, 405). 7 Suffragists had heard this argument before. It had a long ancestry, plunging deep into classical theorizing on citizenship as a status of freedom that one must be both able and prepared to die for to defend. 8 Yet, although the outbreak of war seemed to vindicate this classical conception of citizenship, Common Cause's correspondent felt it could be easily rebuffed. One needed simply to repeat in war-time the standard suffragist response given to it in peace time: "the work of women in the nurture and care of humanity [is] at least as valuable as that of men in armed defence." This war thus appeared to her as an opportunity to demonstrate the value of women's contribution to the nation: "[n]o one would suggest that women share in the warfare" but the "lion's half of the work that is to be done at home" must fall on their shoulders. For suffragists not to engage in this work would be to "shirk the citizenship we have so long asked to be allowed to share." By taking part in this work at home, the NUWSS demonstrated to British men that "those who claim the privileges of the State are also the first to offer to do its work."

The sex-specific roles that war imposed on men and women were not, then, a demonstration of the impossibility of women's citizenship but rather a vindication of feminists' long-standing claim that there were two faces to citizenship: one male, one female (for a similar point see Allen 2005, 111). It followed that to accomplish those roles was not so much to take part in the reproduction of one's ancestral calling as a member of one's sex, but rather to respond to one's duty as a citizen who happened to be of a particular sex. The sexed citizens were, moreover, strictly equal in their contribution to national survival and complemented one another. Where men must go and fight, women must concentrate on their sphere of expertise and devise solutions for guaranteeing that the impact the war had on it was not detrimental to the nation's struggle (See Fawcett August 7 1914, 376). As we will now see, in the process of discussing what this entailed, promoters of a gendered citizenship, having already reframed separate spheres as a mutually supportive and non-hierarchical discourse, pushed the boundaries of the "woman's sphere" beyond the well-defined limits it was usually associated with.

Ultimately but importantly, the only boundary they accepted was the role of the soldier.

In its first war-issue, The Englishwoman reacted strongly to the "perfect orgy of needlework" and the "sewing-mania" that it felt had gripped the nation's women ("Echoes of War" September 1914, 303). It reproached the authorities for thinking and saying that women should sew in this crisis, and women for listening to that message:

[a]re we still in the mid-Victorian age struggling with our Butterick patterns and our needles and threads, to supplement the stupid omissions of a blundering commissariat? Or are we really women of the new era, the era of social science and electric sewing-machines?

Similarly, Common Cause welcomed the coverage by newspapers such as The Evening Standard of the numerous activities women were involving themselves in: "[i]t is distinctly refreshing to find that some newspapers recognise that women's work in time of war embraces something wider than sewing parties and knitting teas" ("Press Department" 2 October 1914, 465). This is not to say that sewing parties and knitting teas were not also on the agenda. Indeed, focusing on the "woman's sphere" of expertise meant multiplying articles on how to limit the impact of the war on issues ranging from food supplies to caring for infants, from soldier's clothing to children's entertainments. Entire columns were given over to detailing ways in which to conserve food in order to prepare for the long haul and to announcing the creation of maternities, as well as toy and shirt-making shops. 9 It must be noted however that the latter were opened to help women thrown out of work by the war. Thus, focusing on the woman's sphere also meant devising solutions to tackle women's unemployment caused by war, fighting the corner of soldiers' dependents against illtreatment by the authorities, finding homes for stranded Belgian allies, accompanying "alien enemies" back to their homeland, or creating and funding all-female hospital units for the front. 10 In fact, the boundaries of the "woman's sphere" seemed to be almost endlessly expandable in war. Witness this amused reaction from Common Cause to the surprise raised in some quarters at the National Union's enthusiastic turn to relief work: "our members are drawn from that class of women (to be found in all classes) who have held that the world is their home and their charge is to make it home-like" ("The World Our Home" 21 August 1914, 393, emphasis added). It is tempting to dismiss this as merely rhetorical flourish. And yet, when Common Cause offered an inventory of the activities women should be engaging in, it did not content itself with quoting the four core areas outlined by Fawcett in a letter to the Manchester Guardian, i.e. "(1) Doctoring and nursing (2) Care of the young womanhood of the nation (3) Care of children (4) Care of child-bearing women," but also suggested that women should "work as far as we are trained and able, in shop and factory and office and school" ("Our Active Service League" 11 September 1914, 431).

Nor did the argument stop at the expansion of the "woman's sphere" into nontraditional sectors of activity at home. The Englishwoman, for instance, called for some women to be allowed onto the front lines to pick up the wounded even during combat.

Drawing on two reported cases of women nursing soldiers in the line of fire, it argued that: "we should perhaps reconsider the customary refusal to let women, either nurses or doctors, work at the front. This, perhaps, is another practice which has been put out of date by the military methods of the Germans" ("The Utilisation of Voluntary Aid Detachments" November 1914, 107). The authorities, it concluded, should allow women a "share in the work [and] in the risks" ("The Utilisation of Voluntary Aid Detachments" November 1914, 108). Thus, far from confining women physically and occupationally to "the home," the notion that war imposes sex-specific duties only really meant, from the perspective of this discourse, that women could not fight. For in lieu of two rigidly separate spheres, this discourse suggests the existence of three broad spheres of activity in war: one specific to men (fighting); one specific to women (minding the home, caring for children, young and pregnant women); and one that consists of activities that men alone do, or overwhelmingly do, but that women could and should do as well (working in shops, factories, offices, schools and so on). Nor are women confined to working on the home front: those who have the ability and desire should be allowed to risk their lives on the war front. Men, by contrast, do not find their duties expanding into woman's sphere of responsibilities: it is not suggested that they should care for babies, the elderly, or pregnant women and even less that they should take care of the home. However, they do retain one monopoly: the role of the soldier. Indeed, even advocates of allowing women into the line of fire limit the possible roles women would take on the front line to "nurses and doctors." Even here, women remain carers.

The role of the soldier thus appears as the ultimate outer limit of the expansion of the woman's sphere in war. One could argue that this is the result of feminists' careful treading. After all, The Englishwoman's suggestion is framed largely as a necessary response in the face of Germany's "military methods," implying that such a change in "custom" is required as a result of changed conditions and of the nature of the evil faced, not as a demand of irresponsible feminists hell-bent on equality with men. But far from being a case of uncharacteristic shyness, this non-advocacy of the recruitment of women into the armed forces is best understood as self-imposed. Indeed, for this vision of gendered citizenship to hold, women simply cannot be soldiers. As Common Cause put it, "[t]he part of women, even in time of war, is still not to destroy, but to heal; not to strike down, but to raise up and support. About our immediate duty there can be no doubt for any of us; it is to give, to work, to share" ("The Wine Poured Fourth " 9 October 1914, 474-5). If women could be soldiers, they could be killers and would thus not be essentially different from men. 11 Yet if the association of women with "the home" means anything, it means that where men excel in the physical domination of the world, women excel in its "moral" or, in the language of the time, "spiritual" domination. And this in turn justifies both their claim to expertise in certain spheres of life and their claim to citizenship: as they are spiritually or morally better equipped than men, they will bring an important moral dimension to public decision making.

The outbreak of war certainly gave renewed currency to the language and practice of separate spheres. But by the same token, it presented these feminists with a unique opportunity to promote their gendered understanding of the concept of citizenship. This in turn allowed them to frame separate spheres as a non-hierarchical, equalitarian discourse and to radically redraw the boundaries between the two gendered spheres without discarding them altogether, thus harnessing the power of the language of separate spheres for their cause.

However, to present bottling fruit, opening a toyshop and even nursing soldiers on the front line as acts of citizenship is one thing. It is quite another to convince a wider audience that these are just as much acts of citizenship as men's taking up of arms. To this end, these feminists consistently pursued a rhetorical strategy of inclusion which involved appropriating the discourse of citizenship for women.

Women as citizens

Using the language of citizenship to describe actions to which it was not customarily applied required the use of two discursive techniques: one which might be called "mirroring," and the more familiar operation of gendering.

Mirroring involved the straightforward application of keywords of the discourse of citizenship to women's actions. Women's caring roles, for instance, were regularly framed in language more usually associated with military service. Common Cause explained its members' enthusiastic plunge into relief work as a response to a "call for service," and further described them as "[standing] to their posts" and "[working] for the common weal" ("The World Our Home" 21 August 1914, 393). Similarly, one of the paper's correspondents proposed that women too should "enlist" (Milner 11 September 1914, 429). But she did not mean by this that they should join the ranks. She meant that women who have "no men of [their] own to encourage and send" should put their names, as men do, on a national list of volunteers prepared to replace a man who enlists in supporting his dependents. "Just as the country calls on the men to fight," she concluded, "she calls on the women to bring up and care for those whom they leave behind." More illuminating still are the profound parallels between the manner in which these feminists attempted to mobilize women and the manner in which the authorities mobilized men.

Witness Millicent Fawcett's call to women, emblazoned in bold letters on Common Cause -"Women! Your Country Needs You!," which exactly paralleled Kitchener's famous recruitment poster, the absence of Fawcett's picture (and pointing index finger) not withstanding ("'Women! Your Country Needs You!'" 14 August 1914, 385).

Similarly, when the NUWSS, following attacks in the press on young women who were behaving frivolously with soldiers, decided to create an Active Service Girls' Cadet Corps to put their energy and enthusiasm to better use, it entrusted its leadership to Katherine M. Harley, sister of Field Marshal Sir John French. 12 Just as her brother was leading the young men of the nation in their service on the war front, so would she lead the young women of the nation in their service on the home front: "we have mobilised the soldier for the front in France and Belgium: we must mobilise the girl for the front in Great Britain" (A.M.R. December 4 1914, 575). The parallel with the army did not stop at this slogan. Like their male counterparts, these young women were to wear uniforms, be given medals, have a hierarchy of superiors, and follow rigorous training. Only the latter set them apart from men however for they were to be trained not in the art of war, but rather in the arts of homemaking and relief of suffering:

[d]rilling, marching, first aid to the injured, and musical practice, both in band instruments and in singing, will form a large part of the training of cadets. (…)

The other activities will include: -Camp Management, Cooking, Knitting, Sewing, Dancing, Acting, Swimming, Organised Games, &c. 13 This mirroring strategy could not always be used, however, because some of the most important features of the discourse of citizenship were so bound up with masculinity that their straightforward application to women's actions was highly problematic. Applying them to women would thus require their gendering. The concepts of force and honor are two cases in point.

It is by force that the nation is defended, by force that it protects both itself and its allies. Good citizens, therefore, use force in defense of the nation. And women, as Common Cause was keen to demonstrate, were good citizens: "[i]t is the vocation (…) of all women, to be foremost in defence of their country -not by arms but by moral force" ("Defenders of the Country" 7 August 1914, 371). The "force" that characterized citizenship in war thus found itself gendered. Men's "armed force," used on the war front, was complemented and completed by women's "moral force," used on the home front. Where the former sought to defend allies and destroy enemies, the latter sought to maintain morale at home. In using force, soldiers were said to be defending the nation's honor, by which was meant its pledged word to defend Belgium in case of invasion. Shortly before the creation of Katherine Harley's Active Service Girls' Cadet Corps, another initiative to tackle the issue of young girls' misbehavior had been launched by suffragists: the League of Honour. Its aim was to bring girls from a privileged background together with their less fortunate sisters who, it was argued, were engaging in morally suspect behavior with recruits as a result of a lack of character and education. The motivation for this initiative, however, went well beyond reforming individuals. Indeed, the third of the League's four objects was to [d]eepen, among women and girls of all classes, the sense of their responsibility for the honour of the nation, and to make clear the fact that the manhood of our country is either raised or lowered by the influence of its womanhood ["The League of Honour" 30 October 1914, 506].

Girls who chose to participate in the League took the following pledge: "I promise, by the help of God, to do all that is in my power to uphold the honour of the nation and its defenders in this time of war, by prayer, purity, and temperance" ("The League of Honour" 30 October 1914, 507). We thus find the concept of the nation's honor, habitually described as solely falling within the male remit via the honoring of the pledge to Belgium to defend her in case of attack, extended to include women. This is done by claiming that the home front is, above all, a moral front. The threat to the nation's honor that Germany poses in the realm of international relations is paralleled by the threat that the war poses to the nation's moral standards at home. Whereas men were the only ones who could defend the nation's honor abroad, women were the only ones who could defend it at home, because the "morality" of their sexual conduct determined the morality of men's and as a result determined the moral tone of the nation as a whole. The implication, furthermore, was that an "immoral" nation at home would be more likely to behave "dishonorably" abroad, an implication which gave women's role even greater importance. Through both of these cases we find reiterated the notion that men and women have separate spheres of specialty which map on perfectly to the two "spheres" of the war: the battle front is ideally suited for men's sex-specific abilities, the home front for women's. In both also we find a central feature of the separate spheres discourse, women's moral superiority to men, claimed and asserted as characteristic of their different but equal face of citizenship. 14 In sum, by appropriating the language and ceremony of citizenship via these two discursive techniques, feminists attempted to redefine women's sex-specific actions in war as duties of citizenship. Doing one's duty, however, was only one half of citizenship.

The other was proving that one stood united with the rest of the nation. And the suspicion that women were not doing so or simply could not do so had thus to be continually fought.

Combating the exclusion of women from the national community Two months into the war, the Anti-Suffrage Review accused the National Union of not respecting the "political truce" that everyone otherwise seemed to agree was called for as Nor were its opponents alone concerned. One Secretary of a society also felt the National Union was not behaving sufficiently like its male counterparts:

[h]ave not men forgotten that they are Liberals, Unionists, Ulstermen, Nationalists? What we want to prove now is that we are, above all, citizens.

(…) We ask for votes in order that we may the better serve the nation. This is the true basis of any extension of the franchise. If it can be said with any suspicion of truth that we have served the nation so that we might afterwards obtain a vote, then our work will, spiritually, if not materially, have lost its value ["Keeping the Union Alive" 4 September 1914, 417].

At issue in both of these attacks was the central concept of unity and its relationship to citizenship: women must be united with men if they are to lay claim to being citizens; they cannot be seen to be divisive. And to claim that patriotic work was suffrage work was to demonstrate one's inability to transcend one's own selfish cause for the greater good. Common Cause's response was to repeat these feminists' central claim: that the war had demonstrated the truth of their position.

We believe in Women's Suffrage because we believe the expression of woman's point of view is necessary for the health of the nation. We believe this even more in time of war than in time of peace, because the tendency of war is to trample on all the half of life which more especially belongs to women ["Keeping the Union Alive" 4 September 1914, 413].

Thus, the war, by treading on women's "half of life," had allowed the common agenda (both sexes working together for the common good) to finally catch up with their own. In addition to rejecting claims that their own position was divisive, it was necessary to reject positions articulated by other women that would lead to women as a group being perceived as divisive or separate from men. Two typical instances were the White Feather Movement and feminist pacifism.

Following the birth of the White Feather Movement, in which women would hand a white feather to men not in uniform as a symbolic accusation of cowardice, Fawcett wrote to the Manchester Guardian to condemn it: "I do not think it is the function of men or women to lecture each other on the special duties of the opposite sex. These duties are sufficiently obvious" (quoted in "Our Active Service League" 11 September 1914, 430). 15 Common Cause concurred: Emmeline Pankhurst believed, by contrast, that women had a right to tell men to fight by virtue of men's promise to women to defend them (see for instance Pankhurst 23 April 1915, 25-6). But for those pursuing a strategy of inclusion, a group that accuses another of not doing its share of the common load is acting divisively and thus failing to live up to the standards of good citizenship.

Pacifism posed a different kind of problem. Feminist pacifists were often constitutionalists who published in the same organs under review here, and worked closely with proponents of this "women as citizens" discourse. They too argued that women belonged to a different sphere than men. But for them this forced the conclusion that women must, of necessity, act immediately for peace. Because such an argument would set women apart from the united nation and separate them from men who were fighting, it needed to be resisted. But such resistance was made more difficult by the fact that it drew on an identical understanding of women's special qualities. The response of these feminists to their pacifist colleagues, therefore, was mostly an attempt at containment ("A Sacred Trust" 20 November 1914, 550-1). On the one hand, pacifists were invited, in the name of consistency, to take part in relief work:

[t]he most pacific of us should (…) prove her love of peace to be not a "selfregarding pacifism," but a self-regardless pity. Those who hate war the mostand what woman is there who does not hate war? -must be the readiest to give their lives to the sacred task of alleviating the horrors of war.

On the other hand, it was conceded that it was women's "sacred trust" to build up public opinion in such wise that if and when our rulers are in a position to consider terms of peace, they will find behind them a great and mighty force making for justice, for self-control, for wisdom.

To remedy this ignorance women must study war: "We are bound (…) to consider what is the cause of war. We cannot be satisfied only to heal its wounds so far as we may, though this healing is a duty from which no woman will dream of holding back."

Following the publication of this view, a reader asked Common Cause to launch an "educational campaign" in its pages to facilitate this greater understanding (Lyon 27 November 1914, 503). There followed a new series of articles entitled "Problems of War and Peace" which attempted to enlighten Common Cause readers on issues ranging from Britain's treaty obligations to disarmament, from conscientious objection to the Swiss militia system. 16 In both cases, the aim was clearly to disarm the threat that feminist pacifism represented to women's inclusion in the national community by either co-opting pacifist women into relief work -the work for which women were being praised -or taking the internationalist and activist sting out of thinking about peace by framing it as yet another duty for women to engage in "as citizens." When containment failed, however, and pacifists decided to take part, with other women from around the world, in the Women's Peace Congress at The Hague, their attitudes were flatly denounced. 17 Fawcett put it most strongly. For her, as long as German troops had not been repelled, it was "akin to treason to talk of peace" (Quoted in Liddington 1989, 96, emphasis original).

The entire women-as-citizens discourse rested on the claim that women were one half of a united nation: they could neither be seen as undermining that unity by pointing the finger at men for not holding up their end of the bargain, nor could they declare themselves to be somehow outside of this national community, as having different priorities from those of the nation. However, the greatest challenge to this discourse did not come from white feather women or feminist pacifists. Indeed, a testimony to the uphill battle these feminists faced is the fact that women's discursive exclusion from the language of citizenship was more often than not to be found in banal forms of speech, such as jokes, sayings, and turns of phrase.

Journalists joked that the following advert had appeared in The Times: "Wantedpetticoats for all able-bodied youths in this country who have not yet joined the Navy or Army" ("A Bad Joke" 11 September 1914, 426). To which Common Cause angrily responded:

[w]e express no opinion whatever about the young men who do not volunteer, doubtless they have their reasons -but the innuendo clearly is that they are unmanly -therefore they are to be dressed in women's clothes. No, thank you! An unmanly man is not fit to wear a woman's clothes. It is time men learnt that a strong, capable woman is not necessarily "virile," nor a feeble man woman-like.

In contesting this limitation of courage to the male gender, Common Cause was performing two interrelated acts. On the one hand, it was challenging the superimposition of positive/negative dichotomies on the male/female dichotomy that was central to the separate spheres discourse. On the other, it was resisting women's exclusion from citizenship. Indeed, if women could be successfully associated with non-courageous men, they could ipso facto be excluded, like those men, from "true" citizenship. 18 Not only jokes, but sayings too could be potent vehicles for excluding women from the community of citizens. Witness the ubiquitous: "For men should work and women should weep." In its second issue after the war was declared, Common Cause rejected widespread talk of "weeping women" as offensive. Women, it argued, were "keeping the race while those we love are slaughtering each other" ("Woman, the Adaptable. The Queen's Appeal to Women" 14 August 1914, 388-9). Again, the aim was to resist the portrayal of women as feeble and thus incapable of rising up to the challenge, as men were doing, of war, i.e. of citizenship.

Turns of phrase, finally, also attracted the ire of Common Cause, even when they emanated from such respected an authority as Lord Kitchener himself. In his first speech to the troops, the national hero warned them against the temptations of "women and wine." The suffragist paper pounced:

Now, when all women are plunged in grief and dread; when so many women have only one thought -how best they can make good the infinite disaster; now, in modern England, we find him warning the men with the foul old tag "women and wine." It seems to us that the necessary warning might have been conveyed in manlier phrase; that the men might have been appealed to, as they left sorrowing wives, and mothers, and sisters to keep the home together, that they should respect womanhood, and not merely that they should safeguard their own health ["An Unhappy Phrase" 21 August 1914, 395].

To portray women as a danger to the troops was, once again, to present them as outside of the national community. By insisting on the fact that these soldiers had female relatives who were playing a vital role, Common Cause was inserting them back in.

That feminists should have spent precious column inches vigorously countering these apparently unimportant utterances is only understandable in light of their strategy of defining women "as citizens." To let such jokes, sayings and turns of phrase stand, would have been to let stand the idea that women were not qualified to be citizens. Such exclusion of women from citizenship, however, did not only take place in the discursive arena. Indeed, a number of decisions taken by the authorities either implicitly or explicitly framed women as outside the national community or, worse, a problem for it to solve.

The British Nationality and Status of Aliens Bill was debated throughout 1914 and opposed by women's suffrage campaigners because it would strip British women who had married foreigners of their citizenship, whilst British men kept their citizenship regardless of their wives' nationalities. As a result of the outbreak of war these women now fell under the category of "alien enemies." The Englishwoman was pained: hundreds of English women must now register themselves as alien enemies, and become liable to removal from their homes if these are in a forbidden area. (…) This is not a small matter to us, we are accustomed to be denied citizenship; but to be branded as alien enemies is a deeper humiliation (…) it with is illustrated by a leader published in Common Cause which, four months into the war, reflected on the mixed reactions that were greeting women's actions. On the one hand, they were being eulogized for their relief work, on the other there were "bitter and repeated complaints of the women who are hindering" (""Woman's Part in War Time"" 11 December 1914, 596). Of these, there were three types: women who were wasting the money they had received on account of being "dependents" of soldiers or sailors; young girls who were accused of pestering soldiers and engaging in immoral conduct; and professional prostitutes.

The first two cases showed, for Common Cause, not that women and girls were somehow individually to blame but rather that their social circumstances were the source of the ill. The dependents of fighting men were often of a poor background and not used to having such amounts of money. To address their poverty was the solution. As for the young girls, they were behaving in this way because, as suffragists had been pointing out for a long time, they had been socialized into thinking that convincing a man to marry them was their sole function in life. The solution was to change their education.

The prostitute was a different case altogether:

[s]he represents, in an extreme and final form, the woman exploited by society. (…) It is useless (…) to appeal to their patriotism. They have none. A prostitute is without nationality. Her hand is against every man, and every man's hand against her. Society has exploited her, and she exploits society.

What has her country done for her, in heaven's name, that she should know "patriotism"? Nevertheless, she is human, and every proposal that assumes that she is not is bound to fail ["" Woman's Part in War Time"" 11 December 1914, 596].

It concludes that suffragists must help prostitutes as far as possible because their battle from the beginning has also been to obtain recognition of their status as human beings. Yet, it is significant that the setting of reference for dealing with prostitutes is not the nation, as it is for soldiers' wives and for young girls, but humanity. By saying that they have no country, the suffragists are making a clear demarcation between the nation as a whole and prostitutes. Certainly, the fault lies not with prostitutes, but with the nation and yet this conveniently distances "women," who are contributing to the nation's struggle, from "prostitutes" who, although not purposefully, can only undermine it. By saying that prostitutes must be treated as human beings, these feminists are not saying anything they had not said about German women stranded in London at the outbreak of war. 19 This exclusion of prostitutes from the nation alerts us to the fact that gendering citizenship in order to define women's response to war as that of "citizens" had important consequences not only for the concept and language of citizenship, but also for the kind of feminism that one could articulate.

Defining women "as citizens" in the context of total war: consequences for feminism To define women "as citizens" in the context of total war led to the articulation of a feminism marked by five principal features.

Firstly, an unconditional acceptance of "the nation" as the only possible frame of reference for a feminist politics. Witness Common Cause's justification for its decision not to turn itself into a propaganda machine either for or against the involvement of Britain in the war: "Great Britain has gone to war. This is a British paper. We accept the war as our condition for the time being, and our immediate concern is to bear ourselves as good citizens under these conditions" ("Accepting Facts" 14 August 1914, 386). War is enough to command one's loyalty to the nation. Thus, whereas Common Cause had carried numerous articles throughout the month of July on the International Suffrage Alliance's meeting in London, in which it celebrated the leaders of sister suffrage societies around the globe who were in attendance (see issues of 10 July, 17 July, 24 July), its first issue discussing the war (31 July) made no mention whatsoever of their continued presence in London and even less of the possibility of international (as distinct from imperial) sisterly action.

The second feature of this feminism is its rejection of xenophobic nationalism. Indeed, this discourse's ideal citizen is fair-minded, heroic and honorable, someone who loves their country, defends it because they are bound to but without hatred for nonnationals. When "German atrocity reports" began to spread in early September 1914, Common Cause issued a characteristic call for Britain not to respond in kind but to continue to behave according to the laws of "civilised warfare" ("The Crime of Vengeance" 18 September 1914, 440). We also saw above that the NUWSS took part in schemes that gave relief to "alien enemies." These examples, as well as the insistence that one of women's duties is to prepare public opinion for peace by thinking through the causes of war and how to avoid them, all point to this discourse's solid rooting in the liberal ideological tradition of reason over passion and right over might. To construct women "as citizens" in war was thus to weld this feminism's fortunes firmly to those of British liberal nationalism.

A third striking feature of this feminism is the narrowness with which it defines its constituency, that is those whom it recognizes as "women." Because non-nationals could not, by definition, be good citizens, they fell out of the picture altogether, as we have just seen. This is not to say, however, that all British females qualified as "women." The prostitute, for instance, did not come under the category "woman" because she failed both tests of "good citizenship." Indeed, she did not accomplish any act that could be described as fulfilling a duty to the nation, nor could she be said to stand united with the national community in the common struggle because of the threat she represented to soldiers' health. The pacifist, similarly, fell outside of feminism's constituency for failing to stand united in the common cause of national struggle. That constituency, then, was made up of British women who could be described as acting in support of their country, as "good citizens."

The fourth key feature of this feminism is its rhetorical strategy of inclusion. This provided suffragists with an opportunity to challenge accepted limitations on the "woman's sphere" by theorizing opportunities that the war opened up for women as proof of the validity of a gendered understanding of citizenship. The "woman's sphere" thus found itself expanding out of the home into the public domain of men and out of the country, into the front line. The success of this theorizing, however, was also its limitation, as indeed is the limitation of all strategies of inclusion: even as they attempt to transform from within and to their advantage the heavily gendered discourses within which they operate, they tend to highlight and reinscribe their male bias. Thus, the campaign to mobilize women could not escape the male-centeredness of the discourse of citizenship, the uneasy, "unnatural" relationship between "women" and "citizenship." 20 When Kitchener pointed his finger at passers by, the poster did not read "Men! Your Country Needs You!." By contrast, Fawcett must preface her call to service with "Women!," thus involuntarily highlighting the fact that it was by no means obvious that women were actually needed. Harley does not compare the "girl" that she is mobilizing with the "boy" that the government mobilized but with the "soldier."

The final distinctive feature of this feminism is its vision of the male-female relationship as an essentially harmonious one. Because it locked women into a gendered national community that was meant to be harmonious, it could not read men's symbolic, legal and literal attacks on women as anything other than irrational bigotry, a lack of understanding borne of prejudice which could and would be overcome by reasoning and argument. To understand it as "sex-war," as Emmeline and Christabel Pankhurst did, would have undermined the basic notion of a united nation. To understand it as a symptom of militarism, as pacifists did, would have undermined the premise of the righteousness of the nation's cause in this war.

Conclusion: Feminism in war

The aim of this article has been to understand the response of Fawcett and her followers to the outbreak of war by placing it in its linguistic context. This has allowed me to draw two main conclusions.

First, far from representing a knee-jerk flight from feminism, their response was a self-conscious continuation of a long standing feminist argument about the gendered nature of citizenship. By plunging into relief work they were not capitulating to the notion of separate spheres or seeking entry into the political system on men's terms but gendering the concept and language of citizenship in order to obtain votes for women on resolutely suffragist terms. 21 In a context where men and women's essential difference was heavily insisted upon, and where citizenship was welded to the separate spheres discourse, it was a response that made considerable sense. Indeed, it gave women's daily relief work a political significance, vindicated the legitimacy of their claim to citizenship, and framed the ubiquitous language of "separate spheres" as an acceptable, efficient and empowering political division of labor between women and men. Rather than an exemplar of British feminism's ideological collapse in the face of war, the case of Fawcett and her followers provides us rather with further evidence of the resilience of feminist thinking in this period. 22 In addition to strengthening the "resilience thesis," I have sought in this article to make a first step in moving the debate beyond the collapse/resilience dichotomy by asking not only "did feminism survive the outbreak of war?" but also "what form of feminism emerged as a result of the interaction between a long-standing feminist argument and the context of total war?." Thus, my second conclusion is that to use the long-standing gendered concept of citizenship in a context of total war had a significant impact on the shape of the feminism that Fawcett and her followers promoted. Theirs was a feminism that anchored British women firmly into the British political community, reframed separate spheres discourse to women's advantage, and tied feminism firmly to liberal nationalism. This made it a powerful language with which to theorize the war-time expansion of the "woman's sphere" as evidence of women's rightful claim to enfranchisement. However, by excluding women both at home and abroad from its constituency and by positing a harmonious national community, this feminist language left its proponents with few conceptual resources to deal with British men's reluctance to treat them as equals. Indeed, they could only fall back on an appeal to reason, having no possibility of returning to a critique of male domination as either sex-interest or as the symptom of militarism.

Notes * This article has benefited greatly from the comments and criticisms of the two anonymous reviewers of this journal. For their insightful remarks on previous versions, I would also like to thank Richard Grayson, Rodney Barker, Michael Levin, Simon In seeking to challenge one such interpretation, this article cannot but continue that suffrage-centric trend. Yet this should not be taken to mean that I disagree with those historians who have insisted that the organisations, individuals and ideas that constituted the movement for women's suffrage do not encapsulate the entirety of "British feminism" in this period. For a recent brilliant example, see Delap 2007. 4 The same can be said of course of the well documented position of Fawcett's opponents in the NUWSS, the pacifists led by such impressive figures as Catherine Marshall, Helena Swanwick or Maude Royden. See for instance [START_REF] Wiltsher | Woman's Part in War Time[END_REF][START_REF] Liddington | The Long Road to Greenham: Feminism and Anti-Militarism in Britain since 1820[END_REF], Vellacott 1993, Vellacott 2007. Indeed, the rich diversity of feminist thought in Britain before the war allowed each of these competing factions to plausibly claim the label "feminist" for themselves, despite the radically different positions they expounded. The "ideological collapse" thesis tends to conceal, by denying the label to this or that group, precisely this diversity in pre-war feminist thought. 5 These were not the only forums in which this particular discourse was articulated (nor was it the only discourse presented in those forums), but they were certainly amongst those in which it was presented the most clearly. The value of focusing on periodicals for historians of feminism has recently been highlighted by a marvellous collection edited by Lucy Delap, Maria DiCenzo and Louise Ryan (Delap, et al. 2006).

12 For a perceptive discussion of the moral panic over young women's alleged collective bout of "khaki fever" in the first months of war, which some argued resulted from their non-mobilisation, see Woollacott April 1994. For subsequent moral panics involving women's sexuality, see Grayzel 1999, chapter 4. 13 My concern in this article is with the articulation, not the reception, of these feminists' attempts to appropriate the language of citizenship for women. It is worth noting in passing, however, that such female organisations styled on the military were largely frowned upon, precisely for their perceived transgression of proper gender roles. Women could claim for themselves the virtues of citizenship, but not don its costume. See for instance Watson 2004, especially chapter 2. 14 The notion that the war had opened a "moral front" at home was not an invention of these feminist commentators. Indeed, their response has much in common with that of the most conservative commentators, who called on women to behave "properly". See for instance Grayzel 1999, especially chapter 4. Where they part ways, however, is in the feminists' insistence that to do so was an act of citizenship in addition to being an act of proper womanhood. 15 For an influential study of the White Feather Movement, see Gullace 2002, chapter 4. 16 See respectively Blease 4 December 1914, 581, Hobson 24 December 1914, 621, Brockway 29 January 1915, 681-2, and Gunter 22 January 1915, 667. 17 18 Conscientious objectors were disenfranchised for five years by The Reform Bill of 1918. See Gullace 2002, chapter 8, especially 178-84.

a

  result of war: The first indication that the Suffragist moral code differed from the one accepted by the rest of the community appeared in The Common Cause of August 28 th . (…) It was pointed out that the Union had abandoned nothing, and the notice continued: "It has announced that it has temporarily 'suspended ordinary political work,' but it is doing a good deal of extraordinary political work, and with excellent result." The subtle distinction is worthy of the German Press Bureau ["The Broken Truce" October 1914, 162].

  [w]e are sick to death, we Suffragists, of being told by men what we may doought to do -what is "womanly!" In the name of commonsense let us not now copy the folly and set out to tell men what they may do and ought to do -what is "manly!" They know in their hearts what is manly, just as we know what is womanly, if they see us going about our business we may trust them to go about theirs ("Our Active Service League" 11 September 1914, 431).

  is an unforgettable blow to what we hold dearest ["Echoes of War" September 1914, 308]. Nor did the outbreak of war result in women being treated any more equitably by the authorities. Two cartoons published in Common Cause in consecutive weeks in December 1914 highlight this amply. Although the subjects were different -one addresses the War Office's decision to put soldiers' and sailors' wives under police surveillance, the other the decision to bar women from pubs before 11:30 a.m. -but the message was the same. INSERT IMAGE 1 HERE. DELETE CAPTION BELOW IF READABLE BUT KEEP REFERENCE 'THE CRIME OF BEING A SOLDIER'S WIFE. Magistrate: Do you confess that your husband is serving his country at the front?

  Griffiths, Saul Newman, Leda Glyptis, David Ferguson, Sonja Grussendorf, Thierry and Mary Lefebvre, my colleagues in the History of Feminism Network -Daniel Grey, Esme Cleal, Angela Granger, and Laura Schwartz -and the participants in the "Violence, Bodies Selves (IV): Women and War" panel at the Annual Conference of the British International Studies Association held in Cambridge in December 2007, as well as the participants in the "History Lab" seminar held in London at the Institute of Historical Research in January 2008. Finally, I doubt that I can ever thank Marion Calvini-Lefebvre enough for her patience and support. Errors and shortcomings remain my own. 1 See Riley 1988, particularly chapters 1 and 3. 2 That attention to the linguistic context of political thought and action allows us to understand more clearly what actors were doing by performing those verbal and nonverbal actions is a central insight of "Cambridge school" historians, particularly Quentin Skinner. His main theoretical articles on this matter and his critics' responses have been usefully brought together by James Tully in Tully and Skinner 1988. 3 On the resilience of suffragism see also Smith 2005. It will be clear by now that the debate over the capitulation/resilience of feminist thought in the face of total war has disproportionately focused on the responses to the war of the main suffrage organisations.

  For a detailed account of the Women's Peace Congress in The Hague see Wiltsher 1985.

Soldier's wife: Yes, sir.

Magistrate: Then you must be put under police surveillance at once' ("The Crime of Being a Soldier's Wife" 11 December 1914, 589). In all three cases -marriage law, surveillance of soldier's wives, and drinking lawsthe same argument was articulated: women were making considerable but unrecognized sacrifices in this war. Worse still, they were either humiliated by the way the law treated them, or were being punished for the sacrifices they had made. In every case, men failed to value their contributions to the nation and, as suffragists had long been arguing, their unequal treatment resulted from their absence from the bodies that had taken those decisions.

But of much greater concern to the authorities than misbehaving soldiers' wives or drunken women, were prostitutes who were blamed for spreading disease amongst recruits. Defending these women, however, was more difficult: for what contribution to the nation could they be said to be making? The way in which this problem was dealt 19 See for instance "International Women's Relief Committee" 28 August 1914, 403. According to this article, the NUWSS's participation in a relief committee's work to help "alien enemy" women stranded in London stems from "a deeply human sympathy with anyone in trouble." 20 This uneasy relationship was acutely felt by contemporaries. In the words of Janet Watson: "Women could only be equal-but-different, and their efforts were always perceived as those of women in particular, not just citizens" (Watson 2004, 7). 21 For a similar, if brief, assessment of the NUWSS's relief work as "feminist" but from a perspective that is anchored in social and political rather than intellectual history, see Holton 1986, 132. For a longer defense on similar lines see Vellacott 2007, 15-32. 22 In fairness to Jo Vellacott it must be noted that her most recent work happily accepts the pro-war wing of the NUWSS under the label "feminism," although it remains reluctant to do the same for the jingoism of Emmeline and Christabel Pankhurst. See " 11 September 1914. The Common Cause, 426. "Accepting Facts." 14 August 1914. The Common Cause, 386.


Emmanuel Hadoux

Aurélie Beynier

Paul Weng

Solving Hidden-Semi-Markov-Mode Markov Decision Problems

Hidden-Mode Markov Decision Processes (HM-MDPs) were proposed to represent sequential decision-making problems in non-stationary environments that evolve according to a Markov chain. We introduce in this paper Hidden-Semi-Markov-Mode Markov Decision Processes (HS3MDPs), a generalization of HM-MDPs to the more realistic case of non-stationary environments evolving according to a semi-Markov chain. Like HM-MDPs, HS3MDPs form a subclass of Partially Observable Markov Decision Processes. Therefore, large instances of HS3MDPs (and HM-MDPs) can be solved using an online algorithm, the Partially Observable Monte Carlo Planning (POMCP) algorithm, based on Monte Carlo Tree Search exploiting particle filters for belief state approximation. We propose a first adaptation of POMCP to solve HS3MDPs more efficiently by exploiting their structure. Our empirical results show that the first adapted POMCP reaches higher cumulative rewards than the original algorithm. However, in larger instances, POMCP may run out of particles. To solve this issue, we propose a second adaptation of POMCP, replacing particle filters by exact representations of beliefs. Our empirical results indicate that this new version reaches high cumulative rewards faster than the former adapted POMCP and still remains efficient even for large problems.

. In all those works, non-stationary environments are represented with multiple modes. The model of Hidden-Mode Markov Decision Processes (HM-MDPs) proposed by Choi et al.[8] formalizes this idea. HM-MDPs constitute a subclass of Partially Observable MDPs. In HM-MDPs, the environmental changes are described by a Markov chain and thus occur at each decision step. However, we argue that this assumption is not always realistic. Indeed, in the elevator problem for instance, allowing, even with a small probability, the environment to be able to change between different rush modes at every move of the elevator is debatable.

 what HM-MDPs are to hidden Makov models. In HS3MDPs, when the environment stochastically changes to a new mode, it stays in that mode during a stochastically drawn duration. While HM-MDPs assume that environmental changes follow a geometric law, this assumption is relaxed in HS3MDPs.

, an online algorithm proposed for approximately solving POMDPs, based on Monte Carlo Tree Search and particle filters for belief state approximation. We present two improvements of POMCP for solving HS3MDPs more efficiently. The first adaptation exploits the special structure of HS3MDPs and the second furthermore represents belief states exactly instead of using particle filters. Finally, we experimentally validate those algorithms showing their effectiveness on a diverse range of domains.

In Sect. 2, we recall the necessary notations and definitions. Then, in Sect. 3, we introduce our new model. In Sect. 4, we present two adapted algorithms for solving HS3MDPs. Experimental results are presented in Sect. 5. Finally, we conclude in Sect. 6.

Background

Markov Decision Process. A Markov Decision Process (MDP) [14] is defined by S, A, T, R where S is a finite set of states, A is a finite set of actions, T (s, a, s ) is the probability of reaching state s from s after executing action a and R(s, a) ∈ IR is the immediate reward obtained after performing action a in s. A policy π is a sequence (δ 0 , δ 1 , . . . , δ t , . . .) of decision rules such as each

Introduction

Markov Decision Processes (MDPs) provide a general formal framework for sequential decision-making under uncertainty. They have proved to be powerful for solving many planning problems [START_REF] Puterman | Markov Decision Processes: Discrete dynamic stochastic programming[END_REF]. However, MDPs run under the assumption that the environment is stationary, i.e., the transition function and/or the reward function do not evolve through time. In many real-world applications, this assumption does not hold and the sources of non-stationarity are diverse. For instance, the environment may change due to external events. In finance, when investing on the stock market, a financial crisis changes the dynamics of stock prices. Another example of non-stationary environment concerns multiagent systems. Indeed, from the viewpoint of one agent, a change of behavior (e.g., due to learning) of another agent may affect the environment of the first agent.

Fig. 1: HM-MDP representation with 3 modes and 4 states decision rule δ t : S → A dictates which action to take for each state at timestep t. In a state s, a policy π can be valued by the expected discounted total reward it yields:

V π (s) = E π ( t γ t R(S t , A t ) | S 0 = s) .
(

) 1
where γ ∈ [0, 1[ is a discount factor. Function V π is called the value function of π. Solving an MDP consists in finding an optimal policy, i.e., a policy that maximizes the expected discounted sum of rewards. One of the main limitations of the standard MDP framework is that it requires the transition and reward functions to be stationary.

Hidden-Mode MDP. Hidden-Mode MDPs (HM-MDPs) formalize a subclass of non-stationary problems where environmental changes are limited to a fixed and known number n of modes. Each mode represents a possible stationary environment, formalized as an MDP. Transitions between modes represent environmental changes. Formally, an HM-MDP is defined as follows [START_REF] Choi | Solving Hidden-Mode Markov Decision Problems[END_REF]. For i ∈ {1, . . . , n}, let m i = S, A, T i , R i be a mode, i.e., an MDP. An HM-MDP is characterized by M, C where M = {m 1 , . . . , m n } and C : M × M → [0, 1] is a transition function over modes. Notice that S and A are shared by all m i 's and that an HM-MDP with n = 1 is a standard MDP. In HM-MDPs, the only observable information is the current state s ∈ S, the current mode m ∈ M is not observable. Figure 1, showing a 3-mode, 4-state HM-MDP, depicts how HM-MDPs can be visualized. To illustrate further the definition of an HM-MDP, we present a simple example:

Example 1. The elevator problem consists in controlling e elevators in a f -floor building. At each decision step, a user may call an elevator at any floor and, once inside, select any desired floor to go. The number of states is then 2 f (e+1) × f e . The modes are the different types of rush-hour and therefore have an influence on which buttons are pressed, which is described by different transition functions over states. Three actions can be applied to each elevator: go up/down by one floor and open the doors, leading to an action set of size 3 e . Finally, in this problem, the reward function is identical for all modes, the controller receives a penalty for each unsatisfied user.

Considering an office building of 2 floors with 1 elevator: M = {morning rush-hour, late-afternoon rush-hour, non-rush-hour}, S = {1 st floor call button states} × {2 nd floor call button states} × {1 st floor drop-off button states} × {2 nd floor drop-off button states} × {elevator positions}, A = {open, up, down}. In this small example, there are 32 states, 3 actions and 3 modes. The transition function in the morning rush-hour mode describes the situation where it is more probable for the elevator to be called at the first floor. In the lateafternoon rush-hour mode, it describes the opposite situation where users tend to leave the office. For the non-rush-hour mode, the transition function models the normal operating situation.

Partially Observable MDP. Partially Observable MDPs (POMDPs) extends MDPs to partially observable settings [START_REF] Puterman | Markov Decision Processes: Discrete dynamic stochastic programming[END_REF] and are defined by S, A, O, T , Q, R where S is a set of POMDP states, A a set of actions, O a set of observations, T : S × A × S → IR is a transition function over POMDP states, Q : S ×O → IR is a probability distribution over observations and R : S ×A → IR is a reward function. Since the agent does not observe the POMDP state, she has to act based on her only available information (i.e., at step t, the probability distribution over the initial states and her history of observations and actions up to the current step t) which can be represented as a probability distribution over states, called belief state [START_REF] Aström | Optimal control of markov decision processes with incomplete state estimation[END_REF]. Optimal algorithms have been proposed to solve POMDPs [START_REF] Kaelbling | Planning and acting in partially observable stochastic domains[END_REF][START_REF] Cassandra | Incremental Pruning: A simple, fast, exact method for Partially Observable Markov Decision Processes[END_REF], but they do not scale to large-sized problems. Indeed, finding an optimal policy for infinite-horizon POMDPs is PSPACE-Complete [START_REF] Papadimitriou | The complexity of Markov Decision Processes[END_REF].

Choi They have also proposed algorithms to optimally solve HM-MDPs [START_REF] Choi | Reinforcement learning in nonstationary environments[END_REF][START_REF] Choi | Solving Hidden-Mode Markov Decision Problems[END_REF]. They have adapted exact POMDP solving methods in order to exploit the structure of HM-MDPs. Those adapted methods can solve larger instances of HM-MDPs than the original ones, but they also suffer from the curse of dimensionality. Indeed, solving an HM-MDP is still PSPACE-Complete [START_REF] Chadès | MOMDPs: A solution for modelling adaptive management problems[END_REF]. Like exact POMDP solving algorithms, exact HM-MDP solving algorithms does not scale. In that case, one has to resort to approximate algorithms like POMCP.

POMCP. The Partially Observable Monte-Carlo Planning (POMCP) algorithm [START_REF] Silver | Monte-Carlo planning in large POMDPs[END_REF] is one of the most efficient online algorithms to approximately solve large-sized POMDPs. To choose an action at a given timestep, POMCP (Alg. 1) runs an effective version of Monte-Carlo Tree Search (MCTS), called UCT (Upper Confidence Bounds (UCB) applied to Trees) [START_REF] Kocsis | Bandit based Monte-Carlo planning[END_REF], using a black-box simulator of the environment and a particle filter to approximate a belief state. This search tree is built iteratively. POMCP uses the simulator to run a fixed number of simulations in order to evaluate the actions before performing in the real environment the best action found in the search tree. At one decision step, to choose which action to perform, search(τ ) is invoked with τ the current history, i.e., the sequence of past observations and actions. This history can be expanded with an action a giving τ a and an observation o giving τ ao. The root of the search tree is a node matching the last seen observation. Its children are all possible actions, whose own children are the respective possible observations given an action. A node of the tree is a triplet N (τ ), V (τ ), B(τ ) associated to τ where the components are respectively the number of times τ has been visited, its mean value and the set of particles (i.e., POMDP states) for this history. During a simulation, the algorithm randomly draws a particle p from the particle set B(τ ) and uses the simulator G(p, a) to get the new particle p , the observation o and the reward r. Actions are selected (Line 19 of Alg. 1) following the UCB1 procedure guaranteeing a good exploration-exploitation compromise. Once all simulations have been done, a step is performed in the real environment with the action returned by search, i.e., the best action found in the search tree. The algorithm sets the new root to the node matching this observation and prunes the tree.

At the beginning, POMCP is initialized with an empty history and an initial (e.g., uniform) distribution I over states. Two important parameters have to be set to guarantee that a good action is selected: the tree depth and the number of simulations. The tree depth d can be deduced from the discount factor γ for a given precision > 0 as follows: d = log( )/ log(γ) . The higher the number of simulations, the better the estimation of the values of the actions but the longer it takes to run. This parameter is generally determined by time constraints. However, as the number of simulations tends to infinity, this algorithm is theoretically guaranteed to choose the optimal action at each step. Finally, notice that the size of the initial particle filter is generally set in function of the number of simulations.

HS3MDP

The HM-MDP framework is not always the most suitable model for representing sequential decision-making in non-stationary environments as it assumes that the environment may change at every timestep. For instance, modeling the elevator problem with an HM-MDP is problematic as decisions have to be made every (say) second, while a mode (rush hour or not) can last several hours. In a problem where this assumption does not hold, the usual modeling trick is to set a low probability of transition between modes. However, from a theoretical viewpoint, this is more than questionable when mode transitions are not geometrically distributed. One of the main contributions of this paper is to propose a more natural model for such cases where the environment dynamics evolve according to a semi-Markov chain. More precisely, the new model we propose, called Hidden-Semi-Markov-Mode MDP, represents environmental changes with hidden semi-Markov models [START_REF] Yu | Hidden Semi-Markov Models[END_REF] while in HM-MDPs, they were represented with hidden Makov models. At each timestep, after a state transition in current mode m, the next mode m and its duration h are determined as follows:

   if h > 0 m = m and h = h -1 if h = 0 m ∼ C(m, •) h = k -1 where k ∼ H(m, m , •) (2)
where h is the duration of current mode m. If h is positive, the environment does not change. But, if h is null, the environment evolves according to transition function C and the number of steps to stay in the new mode is drawn following conditional probability H. Like HM-MDPs, HS3MDPs form a subclass of POMDPs. An HS3MDP can be reformulated as a POMDP S, A, O, T , Q, R whose components are defined by:

S = M × S × IN, A = A, O = S, T ( m, s, h , a, m , s , h ) = αT m (s, a, s ) with α =    C(m, m ) × H(m, m , h ) if h = 0, 1 if h = h -1 and m = m, 0 otherwise (3) Q( m, s, h , a, o) = 1 if s = o and 0 otherwise, R( m, s, h , a) = R m (s, a).
Discussions. When considering non-stationary environments in MDPs, each component of the quadruplet S, A, T, R may be impacted by an environmental change. Indeed, some states may become impossible or new states may become reachable, some actions may become infeasible or new actions may appear, the transition function and the reward function can of course also change after the environment evolves. Interestingly, a change in the set of states and/or the set actions may always be modeled by a change in the transition and reward functions by considering the set of all possible states for S and the set of all possible actions for A at the beginning.

It is easy to notice that HM-MDPs form a subclass of HS3MDPs. In fact, a problem represented as an HS3MDP can also be exactly represented as an HM-MDP by augmenting the modes. The two models are equivalent in the following sense. A model M is equivalent to a model M if and only if a problem that can be represented in model M can also be exactly represented in model M and vice-versa.

Proposition 1. HM-MDPs are equivalent to HS3MDPs.

Proof. ⇒ Given an HM-MDP, we can define an equivalent HS3MDP by setting a mode duration function H such that ∀m, m , H(m, m , 1) = 1 and H(m, m , h) = 0, ∀h = 1. At each timestep, h = 0, thus leading only to the first alternative of 3. This turns out to be the exact formulation of an HM-MDP.

⇐ Given an HS3MDP, we show how to build an equivalent HM-MDP. To that aim, we build a sequence of equivalent HS3MDPs. Denote M 1 , C 1 , H 1 the initial HS3MDP. We repeat the following operation to build the sequence: If, for M i , C i , H i , there exist m, m ∈ M i and h = 1 such that H i (m, m , h) > 0, we define the next HS3MDP M i+1 , C i+1 , H i+1 as follows:

M i+1 = M i ∪ h =1 {m 0 , . . . , m h -1 |H i (m, m , h ) > 0} C i+1 (m, m h -1 ) = C i (m, m ) × H(m, m , h ) C i+1 (m j , m j-1 ) = 1, ∀j > 0 C i+1 (m 1 , m 2 ) = C i (m 1 , m 2 ), ∀(m 1 , m 2 ) = (m, m ) H i+1 (m, m h -1 , 1) = H i+1 (m j , m j-1 , 1) = 1, ∀h > 0, j > 0 H i+1 (m 1 , m 2 , h ) = H i (m 1 , m 2 , h ), ∀(m 1 , m 2 ) = (m, m ), ∀h (4)
where for all j, m j is a duplicate of m and C i+1 and H i+1 are null for the unspecified cases. When this operation cannot be iterated, in the last HS3MDP, unreachable modes can be removed. Finally, the resulting HS3MDP corresponds to an equivalent HM-MDP.

However, representing HS3MDPs in such a way feels unnatural and leads to a higher number of modes, which moreover, would have a negative impact on the solving time. It is also obvious that, if the maximum duration is unbounded, the equivalent HM-MDP would have an infinite number of modes, making it difficult to solve.

As a final note, the models of HM-MDPs and HS3MDPs are particular instances of Mixed-Observable MDPs (MOMDPs) [START_REF] Ong | POMDPs for robotic tasks with mixed observability[END_REF][START_REF] Araya-López | A closer look at MOMDPs[END_REF], a subclass of POMDPs. Therefore, MOMDPs algorithms could be used for solving HS3MDPs. We chose to base our solving method on POMCP, because it tends to be more efficient than specialized algorithms on MOMDPs and more generally on factored POMDPs, even when POMCP is run on the non-factored representations [START_REF] Silver | Monte-Carlo planning in large POMDPs[END_REF].

Solving an HS3MDP

As HM-MDPs form a subclass of HS3MDPs, solving exactly an HS3MDPs is a PSPACE-complete problem [START_REF] Chadès | MOMDPs: A solution for modelling adaptive management problems[END_REF]. In order to be able to tackle large instances of problems, we therefore focus on an approximate solving algorithm. A first naive approach is to apply POMCP (see Sect. 2) to directly solve the POMDP derived from an HS3MDP. In that case, a particle in POMCP represents a mode m, a state s and a duration h of the HS3MDP. We propose in this section two possible improvements to this naive approach. Notice that, as a subclass of HS3MDPs, these solving methods can also be applied to HM-MDPs. In the remaining of the paper, we will therefore focus only on HS3MDPs.

Adaptation to the structure. In large instances, POMCP can suffer from a lack of particles to approximate the belief state especially if the number of states in the POMDP and/or the horizon are large. To tackle this issue, a particle reinvigoration technique can be used in the original algorithm. However, it is often insufficient. When POMCP runs out of particles, it samples the action set according to a uniform distribution, which obviously leads to suboptimal decisions.

We propose a first adaptation of POMCP that exploits the structure of HS3MDPs to postpone the lack of particles. In fact, in the derived POMDP, as the agent observes a part of the state of the POMDP, a particle needs only to represent non-observable information, that is, the mode m and the number of steps to stay h. This adaptation allows us to initially distribute particles over a set whose cardinality is much smaller. However, the size of the particle set |B(τ )| still depends on the number of simulations. This modification of POMCP is introduced at line 3 of Alg. 1.

Exact representation of the belief state. When solving large-sized problems, the above adaptation of POMCP still suffers from lack of particles. We thus propose a second adaptation where we replace the particle set B by an exact representation of the belief state. This representation consists of a probability distribution µ over M × IN (modes and duration in the current mode).

Lines 3 and 5 of Alg. 1 are modified as particles are now drawn according to a probability distribution. Line 22 is not needed anymore. This probability distribution is updated after a new observation using the following equation:

µ (m , h ) = 1 K T m (s, a, s ) × µ(m , h + 1) + (5) m∈M C(m, m ) × T m (s, a, s ) × µ(m, 0) × H(m, m , h + 1) .
where K is the normalization term and elements s, s , a are respectively the previous observation, the new observation given by the real environment and the action performed and given by the procedure search. This update is performed after every action executed in the real environment.

In HM-MDPs we can rewrite the above equation knowing µ(m , h + 1) = 0, ∀m , h and H(m, m , 1) = 1. We then obtain:

µ (m ) = 1 K m∈M C(m, m ) × T m (s, a, s ) × µ(m) . (6)
We fall back to the HM-MDP update equation described by [START_REF] Choi | Solving Hidden-Mode Markov Decision Problems[END_REF].

Unlike the previous adaptation, the spatial complexity of this one does not depend on the number of simulations. Indeed, µ is a probability distribution over M × IN. Assuming a finite maximum number h max of timesteps to stay in a mode, which is often the case in practice, there always exists a number of simulations N for which the size of the particle set is greater than the length of this distribution. In such a case, this second adaptation will be more interesting to consider. The time complexity of the update of the exact representation is O(|M| × h max ). It is to be compared to the particle invigoration of the original POMCP and the first adaption which is O(N ) with N being the number of simulations.

Experimental Results

We tested POMCP and our two adapted versions on four non-stationary problems. The first three environments are problems of the literature [START_REF] Choi | Solving Hidden-Mode Markov Decision Problems[END_REF]. We solved an extended version of each problem modeled as an HS3MDP. Recall that those adapted versions of the problems cannot be represented as efficiently with HM-MDPs (see Prop. 1). Results for this model are thus not reported.

H(m, m , •) is defined as a truncated Gaussian probability distribution on duration h of the mode m after a transition from m. The mean of the Gaussian is uniform randomly drawn between 1 and 5 when creating the environment.

We present the results for the original POMCP and for our adaptations of POMCP: the Structure Adapted (SA) and Structure Adapted combined with the Exact Representation (SAER) of belief states. We also show the results of the optimal policy when it could be computed, using Finite Grid [START_REF] Cassandra | pomdp-solve[END_REF] and MO-IP [START_REF] Araya-López | A closer look at MOMDPs[END_REF]. We also used MO-SARSOP [START_REF] Ong | POMDPs for robotic tasks with mixed observability[END_REF] with one hour of policy computation time when the model could be generated for offline computing. We present the performances of the algorithms for several numbers of simulations to study how the quality of the solutions evolves. For each number of simulations we averaged the cumulative discounted rewards over 1000 runs. We reported results that could be obtained within one hour on a computer equipped with an Intel XeonX5690 4.47 Ghz core. We chose to present the raw results for the original POMCP and percentages for the others. Reported percentages correspond to the improvement in the average cumulated discounted rewards between our modified versions and the original POMCP.

Traffic light. In the traffic light problem, the environment is a two-way road where the system has to choose which side to let pass. It has to decide which traffic light to switch on, knowing only the current state of the lights and the presence or not of cars on each side of the road. In this problem, the HS3MDP has two modes: rush on the left or on the right and two actions to choose which light to switch on. The model contains eight states depending on the presence or not of cars on the left, on the right and on the light state. The reward function gives a negative reward when a car waits on a side of the road whose light is shut off. At each timestep, the environment has a probability of 0.9 to stay in the same mode and 0.1 to change. Finally, the transition function over the state depends on the probability of cars arriving on each side, according to the current mode. Exact probabilities for the original problem can be found in [START_REF] Choi | Reinforcement learning in nonstationary environments[END_REF].

Table 1 describes results for the traffic light problem, using different algorithms: original POMCP (orig.), Structure Adapted (SA), Structure Adapted combined with Exact Representation of belief states (SAER) and Finite Grid, MO-IP and MO-SARSOP. The last three algorithms yield the same results, which are presented in column "Opt." to give an idea of the optimal value. The performances of the original POMCP almost strictly increase with the number of simulations. They therefore get closer to the optimal value, which translates into decreasing percentages in Column "Opt." of Table 1. Since our modified versions of POMCP performs better than the original one (positive percentages for columns "SA" and "SAER"), they also get closer to the optimal. For instance, with 512 simulations, 4.7% of improvement for SAER compared to 9.3% for Column "Opt." means that the performances of SAER are half-way between those of the original POMCP and the optimal value. Note that a decreasing percentage does not mean a raw decrease in the performances. It means that the increase of the performances of the original POMCP is higher than those of the other algorithms. Nonetheless, the percentages being positive, the later still perform better.

Theoretically, POMCP converges towards the optimal solution while the number of simulations increases. Experimental results (Table 1) show that it is also the case for our adapted versions whose performances are always at least as good as the original POMCP.

In the traffic light problem, both adaptations of POMCP are roughly even. In fact, the size of the problem is quite small so the original POMCP and the structured adapted POMCP do not lack particles. Moreover, there are enough particles to draw a high quality estimation of the belief state. That is why, the exact representation of belief states does not significantly outperform other POMCP versions. Nonetheless, our adaptations of POMCP both outperform the original version since exploiting the structure of the HS3MDP leads to more accurate belief states.

Sailboat. The sailboat problem is about controlling a boat from a corner of a finite grid to the opposite corner. The states are possible positions in the grid and the modes are the different wind directions, limited to North, South, West and East. Two possible actions manage the sail orientation between North-South and East-West. The transition function over states depends on the sail orientation given the wind direction. The environment has a probability of 0.5 to stay in the same mode, 0.2 to go to an adjacent one and 0.1 to go to the opposite one. The reward function gives a reward of 1 when the goal is reached. This problem can be enlarged as needed by increasing the size of the grid. Results for a 7 × 7 grid are reported in Table 2.

Due to probabilities of transition between modes, the environment can stay several steps in the same mode thus giving the same wind direction. When the boat is on an edge of the grid, it cannot move until the wind changes to a more favorable configuration. This particularity of the environment leads to a big set of runs where the boat cannot reach the goal and gets stuck on an edge until the end of the run. Moreover, the small drops in the original POMCP performances can be explained with the low number of simulations. If this number is not high enough to explore efficiently, the impact of the random can lead to a high variance. Results show that our adaptations always perform better than the original method and that SAER performs almost always better than SA. Column "MO" stands for the results of MO-SARSOP. We can see that SAER converges toward those results as the number of simulations increases.

Elevators. In the elevator problem, the environment can stay in the current mode (see Example 1) with a probability of 0.1 and has a probability 0.45 to change to the other two when the duration is null. Table 3 contains results for an instance with 7 floors and 1 elevator whereas Table 4 shows results for a 4 floors and 2 elevators instance. We were not able to compute the optimal policy for these instances because of their high dimensionality. The results of our adaptations are roughly even since the size of the problem remains quite limited and does not lead to lack particles. However, it is important to note that our methods always outperform the original POMCP whose performances increase with the number of simulations and converge to the optimal solution.

The low number of simulations reached during the computation time is explained by the representation of the transition function. In this problem, transitions are not represented by a matrix of probabilities because of the high number of state components. The transitions are based on a set of rules, leading to a longer computation time.

Randomly generated environments. These environments allow us to study in a controlled setting the scalability of our algorithms. To create an instance, a number of states n s , actions n a and modes n m have to be defined. Random MDPs are then automatically generated such that, in each state, each action can lead to |S| 10 states and |S|

5

states can yield a positive reward. To enlarge those environments, we varied the size of the sets of states, actions and modes. We averaged results from 10 different instances with different state/mode transition and reward functions for each parameter set.

Tables 5, 6 and 7 describe results for randomly generated environments with respectively 5, 10 and 20 modes. We were not able to compute the optimal policy for these instances because of their high dimensionality. We can see that our methods significantly outperform the original POMCP method. In fact, the exact representation of belief states always outperforms POMCP versions based on particles filter on sufficiently large environments. Indeed, these methods quickly lack particles to accurately represent the belief state.

Moreover, the computation time of our adaptations are promising for application to large-sized real life problems. For instance, in the random environment with 20 modes (Table 7), one run of 1024 simulations took 1.15 seconds for solving the HS3MDP with structured adapted POMCP and 1.48 seconds for solving the HS3MDP with POMCP and exact representation of the belief state.

Conclusion and discussions

In this paper, we introduced Hidden-Semi-Markov-Mode Markov Decision Processes (HS3MDPs), a new generalization of Hidden-Mode Markov Decision Processes (HM-MDPs) to handle in a more natural and efficient way non-stationary environments. We proposed to use the Partially Observable Monte-Carlo Planning algorithm as a solving method for HS3MDPs. As a subclass of our model, HM-MDPs can be solved efficiently using the same methods. However, this algorithm does not solve large-sized problems modeled with HS3MDPs in the most efficient way. We developed two adaptations of POMCP to improve its performances. The first adaptation exploits the structure of HS3MDPs to alleviate particle deprivation. The second adaptation uses an exact representation of the belief state to reach better results with less simulations than the other two methods. Experimental results on various domains of the literature show that those adaptation significantly improve the performance.

As future work, different research directions could be explored. In HM-MDPs and HS3MDPs, transition functions over modes do not depend on the performed action. This assumption does not hold in environments like stock markets where buying big volumes may influence the market. An extension of HS3MDPs to handle such situations would be interesting. Another research direction is to relax the assumption that the transition function between modes is known and to learn it in a multi-armed bandit or reinforcement learning setting.

  et al. have shown that an HM-MDP can be seen as a POMDP S, A, O, T , Q, R where S = M×S, A = A, O = S, T ( m, s , a, m , s ) = T m (s, a, s ) × C(m, m ), Q( m, s , a, o) = 1 if s = o and 0 otherwise, R( m, s , a) = R m (s, a).

V 8 if γ depth < then 9 return 0 10 athen 16 forall the a ∈ A do 17 T

 89101617 Definition of HS3MDP. Formally, Hidden-Semi-Markov-Mode Markov Decision Process (HS3MDP) is defined by M, C, H where M = {m 1 , . . . , m n } is a set of modes, C : M × M → [0, 1] is a transition function over modes and H : M × M × IN → [0, 1] is a mode duration function. Transition C(m, m ) repre-(τ b) procedure rollout(p, τ, depth) ∼ π rollout (τ, •) 11 (p , o, r) ∼ G(p, a) 12 return r + γ.rollout(p , τ ao, depth + 1) procedure simulate(p, τ, depth) 13 if γ depth < then ree(τ a) ← (Ninit(τ a), Vinit(τ a), ∅) 18 return rollout(p, τ, depth) 19 a ← arg max b V (τ b) + c log N (τ ) N (τ b) 20 (p , o, r) ∼ G(p, a) 21 R ← r + γ.simulate(p , τ ao, depth + 1) 22 B(τ) ← B(τ ) ∪ {p} 23 N (τ ) ← N (τ ) + 1 24 N (τ a) ← N (τ a) + 1 25 V (τ a) ← V (τ a) + R-V (τ a) N (τ a) 26 return Rsents the probability of moving to new mode m from current mode m knowing that the duration in m (i.e., the number of remaining timesteps to stay in m) is null. Value H(m, m , h) represents the probability of staying h timesteps in new mode m when the current mode is m. Both the mode and the duration are not observable.

Table 7 :

 7 Results for random environments with ns = 50, na = 5 and nm = 20

	Sim. Orig.	SA SAER	Sim. Orig.	SA SAER	Sim. Orig.	SA SAER
	1	0.41 0.0% 5.6%	1	0.39 0.1% 8.9%	1	0.39 0.8% 11.9%
	2	0.41 4.9% 51.4%	2	0.39 21.0% 57.5%	2	0.40 2.6% 51.1%
	4	0.42 11.5% 140.9%	4	0.40 9.9% 149.0%	4	0.40 2.7% 138.9%
	8	0.44 30.9% 209.6%	8	0.41 24.0% 224.6%	8	0.41 11.8% 225.2%
	16	0.48 34.6% 234.7%	16	0.43 33.0% 261.3%	16	0.41 22.3% 270.8%
	32	0.58 46.0% 223.0%	32	0.48 58.2% 275.8%	32	0.45 42.9% 290.3%
	64	0.77 53.1% 187.2%	64	0.60 76.2% 248.7%	64	0.51 77.5% 305.5%
	128 1.08 45.7% 123.4%	128 0.83 75.4% 184.5%	128 0.63 102.2% 261.1%
	256 1.52 33.5% 70.0%	256 1.16 64.1% 115.9%	256 0.85 102.7% 186.8%
	512 1.98 19.6% 34.5%	512 1.61 41.5% 61.5%	512 1.23 73.3% 107.7%
	1024 2.30 12.5% 17.3%	1024 2.05 2.2% 28.8%	1024 1.66 43.6% 55.3%
	Table 5: Results for	Table 6: Results for
	random environments	random environments
	with ns = 50, na = 5	with ns = 50, na = 5
	and nm = 5		and nm = 10

Acknowledgments

Funded by the French National Research Agency under grant ANR-10-BLAN-0215.