Léo Magnin

The hedgerow: industrial farming's "useful idiot"? The contributions and limitations of radical criticism in sociological analysis

Keywords: Common Agricultural Policy, greening, ecologisation

Introduction

What can we, as sociologists, do with radical political criticism? The publication of the study Reprendre la terre aux machines1 (Reclaiming the land from the machines) by the cooperative L'Atelier [START_REF] Paysan | Reprendre la terre aux machines. Manifeste pour une autonomie paysanne et alimentaire[END_REF] offers a particular answer to this age-old question. Sociologists accustomed to the complexity of case studies are often uncomfortable with overarching explanations that leave little room for nuance. This discomfort generally gives way to polite ignorance, distant sympathy or a sociological study of the critique [START_REF] Boltanski | De la critique : précis de sociologie de l'émancipation[END_REF], but rarely to a robust discussion of the critique's arguments supported by field surveys. However, just as the social sciences sometimes fuel political reflection, the positions formulated by politically and socially committed stakeholders can offer fruitful avenues of analysis for research that would be regrettable to ignore [START_REF] Magnin | Savoirs d'éleveur. Un traité d'expérience[END_REF].

The Atelier Paysan is a "cooperative of peasant technologies" (L'Atelier Paysan, 2021, p. 255) whose main activity is self-building agricultural equipment. By training groups of farmers in wood and metalwork, the cooperative examines uses and needs to create tools that can be reproduced, repaired and adapted to each farm. The cooperative members' approach is opposed to industrial logic: the objective is to promote farmers' collective autonomy by enabling them to make tools that require less financial investment. This liberating aim combined with defending agroecology manifests itself, however, in a "quantitatively derisory result" (p. 154) in the words of the Atelier Paysan itself with its 22 permanent paid staff, and 700 farmers trained each year. The starting point of this "manifesto for peasant and food autonomy" announced by the book's subtitle is, then, the authors' dissatisfaction with the results of their own efforts. The originality of their writing, somewhere between essay and pamphlet, is that their critique is not reduced to identifying opposing forces but is also always self-critical. This reflexivity is not only aimed at the cooperative but also at all the components of the agricultural left-wing that advocate for agroecological farming. Effectively, although the authors' identities are not mentioned in the book (probably to avoid personalising the spokespersons), the actors who present it in the media were involved before or in parallel with the Atelier Paysan in the left-wing, associative peasant trade union and popular education.

The self-criticism of alternative agriculture consists in denouncing its integration into "the agro-industrial complex" (p. 83), which it nevertheless wishes to depose. The latter includes industries both upstream (fertilisers, pesticides, seeds, machinery) and downstream (agrifood, mass distribution), banks and leading executives of the agricultural world (majority trade unionists, elected representatives and senior civil servants). Despite not using the same concepts, this analysis is similar to the one developed in a political economy book published around the same time [START_REF] Ansaloni | L'expropriation de l'agriculture française : pouvoirs et politiques dans le capitalisme contemporain[END_REF] 2 . In both cases, the enlargement of farms and the decrease in their number are analysed as fundamental movements that increase inequalities in the agricultural world and, more generally, social inequalities with regard to food. According to the authors, the industrialisation of farming is fundamentally incompatible with the deployment of agroecology, which requires the despecialisation of farms dependent on large quantities of inputs. This state of affairs is locked in by an economic policy based on international competition that establishes a framework within which agricultural policies have little room for manoeuvre. In these agricultural policies, ecologisation (i.e. the integration of environmental criteria) is therefore structurally restricted to marginal measures.

Reading studies such as these can raise questions for researchers working on comparable initiatives. In my thesis, I studied one of the measures of this ecologisation: the protection of hedgerows in force in the Common Agricultural Policy since 2015 [START_REF] Magnin | La haie requalifiée. Enquête sur un dispositif d'écologisation de la Politique agricole commune (2014-2019[END_REF]. By analysing the genesis and implementation of the measure, I showed that the rule was negotiated as a matter of urgency, that for this reason it includes many potential circumventions and that it is only rarely applied by weakened administrations. The findings of my monograph identify certain mechanisms denounced by the Atelier Paysan (the weight of the history of agricultural practices, the power of the majority agricultural unionism, and the low impact of agri-environmental measures) from a particular angle. Nonetheless, the manifesto led me to address a perspective previously overlooked in my approach, probably owing to its dimension of research conducted with associative actors. With Afac-Agroforesteries, the main national association that brings together hedgerow professionals working with farmers, I have co-written reports for the Ministry of Agriculture that identify levers for better protection of hedgerows in the CAP (Afac-Agroforesteries and Magnin 2020; 2021). Although critical of the planned implementation of this ecologisation, I did not question the implicit idea that increasing hedgerow protection in the CAP, and more generally taking better account of the environment in agricultural policies, were politically desirable objectives. However, one of the striking theories put forward by the Atelier Paysan is precisely that the institutional advances of agroecology are not small steps that make it possible to progressively transform farming but instead residual environmental guarantees that only reinforce the dominant agricultural model. This process of compartmentalising ecologisation through marginal measures may be intentional [START_REF] Fouilleux | Agriculture durable. Comment la nocivité de l'agriculture industrielle est-elle dissimulée ?[END_REF] or unintentional or even completely contrary to the actors' initial intentions. The mechanism is not theorised as such in Reprendre la terre aux machines, even though it is, in my opinion, its most powerful political charge. To avoid euphemistic periphrases, I suggest naming this perverse effect "the useful idiot", after the wellknown political expression initially used to designate left-wing intellectuals favourable to the East despite Soviet totalitarianism.

By way of example, organic farming is also included in the critique formulated by the Atelier Paysan. According to the authors, the strategy of moving upmarket is, of course, not environmentally harmful in itself. But it is "a complement to industrial agriculture" (p. 100), a complement that cannot count on becoming mainstream because it exists only by virtue of its comparative advantages and the social hierarchy of ways of feeding oneself. The label "useful idiot" expresses this contradiction between what is aimed for and what is obtained. It is an example of an old criticism of reformism that, while it relies on the gradual transformation of a system, ultimately only actively perpetuates the very system it seeks to transform. In addition to this structural definition, there is a moral dimension: according to the authors, proponents of organic farming, and even peasant agriculture as a whole (consumers, activists, farmers and researchers), have in common a tendency towards selfsatisfaction that reinforces the relegation of the political balance of power by basing social transformation on individual choices.

The authors do not apply this radical criticism to the protection of hedgerows in agricultural policies. However, there is an interrelated theme in the "zones de non traitement" or nontreated areas near homes and residential areas where the use of phytosanitary products is prohibited. For the authors, this measure, championed and obtained in 2018 by environmental organisations, serves only to allow farmers to continue to use toxic products without fundamentally challenging the production and marketing methods that make them indispensable, thus "further entrenching pesticides in agricultural practices" (L'Atelier Paysan, 2021, p 142). Indeed, after the size of these zones was reduced by the State, aid worth 30 million euros was granted to "co-finance equipping farmers with state-of-the-art high-tech spraying equipment in order to 'reduce' the quantities of pesticide spread and to 'limit' their drifting towards neighbouring homes" (L'Atelier Paysan, 2021, p. 142). In this example, nontreated areas and their environmentalist advocates are part of a technological 'solutionism' fostered by the latter's 'depoliticisation'. The authors treat this attitude with derision: "To those calling on public authorities to monitor compliance with non-treated areas near homes, we're tempted to reply: let's also set up fridge and cupboard inspection brigades on market days to check that everything that goes into them is organic! Joking aside, let's stop making these dramatic problems simply a matter of conscience, morality and 'best practice'".

(L'Atelier Paysan, 2021, p. 164) Consequently, the aim of this short paper is to put hedgerows to a similar test: are hedgerows, and with them all those who defend their greater consideration in agricultural policies, the useful idiots of the dominant agricultural model? This reflexivity is not introspection but rather the re-investing of a research topic based on dialogue with a critical discourse. The discussion is therefore organised in two stages. Firstly, it presents the arguments showing that hedgerows can support consensual ecologisation that marginalises a more profound transformation of the agricultural economy. Secondly, however, I then explore the limitations of this position by arguing that if greening via hedgerows is indeed marginal, it is not reduced to being a useful idiot but participates in ecologisation from the margins.

I. The consensual hedgerow: a driver for the marginalisation of structural ecologisation

The hedgerow is a cultural object that now has positive connotations. For several years, there has been a craze for trees [START_REF] Dupré | Faire lutte de tout arbre[END_REF] as seen in numerous publications aimed at the general public. The phenomenon is spreading to the academic world, which some people regret, seeing in this, sometimes rightly, an arbitrary poetisation of the human sciences [START_REF] Poupeau | Ce qu'un arbre peut véritablement cacher[END_REF]. As ordinary bushes and trees, hedgerows are enjoying this renewed public curiosity, and this can been seen in the frequent articles in the local press reporting on hedges planted by schoolchildren. The act of planting is not politically neutral, and conveys meanings that are used more or less openly by funders, whether public or private. Planting a hedge is a way of acting for the environment that is positive (it does not involve reducing consumption), local (plantations cannot be delocalised) and visible (the action becomes a long-term part of the landscape). Furthermore, the hedgerow is presented as an environmental "Swiss penknife" that may sometimes be confused with a panacea: water filtration, restricting air erosion, shade for livestock, carbon sequestration, local wood production, biodiversity corridor, etc. Landscape ecology [START_REF] Baudry | De la haie aux bocages. Organisation, dynamique et gestion[END_REF] provides scientific justifications for the peasant practice of bocage, which is part of an environmental history that also has an aesthetic value.

These physical and social characteristics give rise to the consensual politicisation of hedgerows: they appear as a single solution to multiple problems. In other words, they are both a material response to environmental problems (without the need for advanced technology) and an effective tool for political communication. This politicising of hedgerows is not only a matter for local spatial planning policies, but has become a matter of general policy. Recently, President Emmanuel Macron showcased the plantation of 7,000km of hedgerows thanks to the France Relance Programme launched in response to the Covid-19 pandemic3 . Julien Denormandie, his Minister for Agriculture from 2020 to 2022, pursued and strengthened this gesture by repeatedly announcing that the next CAP will be the "Hedgerow CAP" [START_REF] Sanson | Cette PAC sera celle de la haie ![END_REF] 4 .

It could be hypothesised that this hedgerow consensus stems from the fact that it is marginal both literally, in that it surrounds fields, and figuratively, as it does not directly affect agricultural production -hedgerows cannot be eaten5 . However, conventional farming, defined by Benoît Daviron as "chemical farming", is precisely a type of farming that has historically focused on the production of food biomass, thanks to the development of fossil fuels and synthetic chemistry [START_REF] Daviron | Biomasse. Une histoire de richesse et de puissance[END_REF]. From this angle, it can be argued that the political focus on this marginal element can lead to the marginalisation of a more systemic ecologisation. From this perspective, the hedgerow is no longer a framework to spread greening but rather a fence that prevents it from reaching the heart of the production system.

A "Hedgerow CAP" would indeed illustrate a consensual environmental willingness that could, however, be a way of circumventing deeper disagreements. The measures to plant and protect hedgerows, as legitimate as they are, may give the impression that agriculture's greening is now only a matter of time even though environmental issues such as reducing the sector's greenhouse gas emissions, restricting the use of phytosanitary products, limiting water consumption and reducing dependence on synthetic mineral fertilisers essentially remain in the trees' shadow. Effectively, the question of productive and food democracy is not posed through the partial prism of hedges: what should we eat and produce in order to limit environmental shocks and adapt to them, come what may?

The media and political success of hedgerows also masks the fundamental question of who, tomorrow, will look after the hedges planted today. Available surveys unequivocally show that hedgerows are disappearing because the size of plots of land is increasing along with the size of farms per farmer, and therefore farmers have less and less time to maintain their remaining trees [START_REF] Jouin | Les techniques et le temps de travail pour l'entretien des haies[END_REF][START_REF] Perichon | L'impossible reconstruction des bocages détruits[END_REF][START_REF] Perichon | Les représentations sociales associées au bocage. Analyse des relations agriculteurs/territoire en Haute-Bretagne[END_REF]. The social issue of taking over farms is crucial here because it reveals a conflict of timeframes: hedges are planted, admittedly, but the lifespan of trees is decades while 43% of farmers are aged 55 or over [START_REF] Agreste | Âge des exploitants et devenir des exploitations[END_REF]. Similarly, the coexistence of a "Hedgerow CAP" and fiscal policies that encourage over-equipping with machinery -a phenomenon that concerns not only the Atelier Paysan (p. 133) but also the Ministry of Agriculture itself (CGAAER 2021) -is paradoxical given that the size of tractors leads to the destruction of old paths lined with hedges, which are now too narrow.

For all these reasons, it is no exaggeration to claim that the hedgerow's success may, effectively, play the role of the useful idiot for an agricultural model that has remained unchanged in its technical, economic and social foundations. In this regard, the Atelier Paysan's critique is instructive -decreeing that hedges must be planted and protected without, however, addressing the socio-economic mechanisms that have determined their destruction for decades is an inherently illusory 'solution'. This is reinforced by the fact that the announcement of a 'Hedgerow CAP' not only coexists with the status quo of a political economy focused on the accumulation of capital within an ever smaller number of farms (-20% in ten years), but accompanies a policy of prolonging and intensifying this trend. In any case, this is one interpretation that could be made of the "third agricultural revolution" announced by President Macron6 , which, focused on progress in "genetics, robotics and digital technology", is likely to prolong an economic model of selecting farms according to their investment capacity rather than their agroecological commitment.

II.

Reclassifying the hedgerow: ecologisation from the margins?

If the reference to hedgerows sometimes plays the role of useful idiot in today's agricultural policies, does this mean that any action carried out to promote hedges can be reduced to this?

In order to answer this question, we need to move away from the focus of political communication and consider how agricultural policies are negotiated and implemented.

Hedges are not only landscape images but also physical elements that are not as popular as the 'bocage' aesthetics. In the 1950s, French law described hedges as "obstacles to rational land-use" because they prevented expanding plots of land and thus mechanising them.

Because of the shade they cast, they are also accused of lowering crop yields. And lastly, they require time for their maintenance even though they no longer provide the services expected by the farmers of the first half of the 20th century: barbed and electric wire have replaced hedges as fences, and electricity and fossil fuels have made the bundles from their pruning superfluous. They have therefore gradually become physical 'constraints' for specialised agriculture based on standardising spaces.

However, from the 1980s onwards, planting policies were implemented in the departments of western France. For decades, these replanting policies coexisted with direct or indirect support for the destruction of hedges. Since 2015, CAP cross-compliance has made the maintenance of hedgerows mandatory. Although the measure is only rarely applied, it remains a regulatory ban that applies to all farms. The parallel with organic farming is therefore no longer relevant here since it is not a question of a voluntary commitment based on a niche market. It also differs from the "decentralised" and "fragmented negotiation" (p. 139) of non-treated areas on a local level that the authors deplore.

This measure reflects the hedgerow's progression in agricultural policies, but also puts its consensual success into perspective -it remains a contentious political object. To dwell only on the hedge's success in the State summit's political communication would ultimately mean only taking into account the surface of the political facts. In fact, the protection of hedgerows in 2015 was only obtained by the defenders of bocage thanks to European budgetary pressure, against the wishes of the majority agricultural unions [START_REF] Magnin | La haie requalifiée. Enquête sur un dispositif d'écologisation de la Politique agricole commune (2014-2019[END_REF]. These power struggles have influenced the definition of hedgerows protected by the CAP in France: the definition adopted in 2015 is restrictive, does not cover the diversity of French hedgerows, and has not been modified in the CAP 2023 despite the appeals of tree professionals.

The same tensions are at play in the new CAP that takes hedgerows into account to an unprecedented degree: in addition to being protected by cross-compliance and supported by agri-environmental measures under the second pillar, they play a greater role in the first pillar with environmental aids known as "eco-schemes" [START_REF] Sanson | Cette PAC sera celle de la haie ![END_REF]. Nevertheless, this state of affairs must be seen in the context of CAP negotiations, which are beset from start to finish by power struggles with the majority agricultural trade unions. The FNSEA (France's national union of farmers' syndicates) opposed measures affecting hedgerows, in particular by trying to influence hedges' equivalence coefficient so that one linear metre would be worth 100m² instead of 10m² of ecological surface area. These objections show that hedgerows are not just the useful idiot of chemical agriculture, but that they are a nuisance to its representatives. The denunciation of the "systematic avoidance of conflict" (p. 176) that the authors of the Atelier [START_REF] Paysan | Reprendre la terre aux machines. Manifeste pour une autonomie paysanne et alimentaire[END_REF] diagnose in agroecological organisations does not apply to the professional world of hedgerows.

For agroforestry advisors, integrating hedgerows into the regulations is as much, if not more, a cause for concern as it is for satisfaction. In their everyday work, rural tree professionals see the discrepancy between the law and the facts: ripping up hedges continues, and ageing hedgerows are not maintained because of lack of time and skills lost over three generations of farm specialisation. If the State, local authorities and companies wish to finance plantations, these funds arrive as a matter of urgency in structures that lack resources, because the financiers generally wish to finance one-off spectacular actions (so many kilometres planted) without providing long-term support to professionals, for example by funding jobs. The superficial political success of hedgerows exposes them to political recuperation by private and public actors whom they distrust, as well as to competition with newcomers who are potentially opportunistic and who do not have the technical knowledge or the relationships of trust they have built up in their areas, sometimes since the 1980s for the oldest structures. This is how agroforestry advisors perceive the political reappropriation of their work, far from the "triumphalist declarations" (L'Atelier Paysan, 2021, p. 158) of certain organic farming representatives whom the authors of Reprendre la terre aux machines lambast for their optimism.

This is not to say that the social logics of opposition are absent from local level, but real political negotiation can take place there, unlike in national CAP negotiations where the proand anti-hedgerow camps are more rigid. By providing a technical discourse on hedgerows, agroforestry advisors participate in 'agroecological diplomacy', in that they make sense of hedgerow protection regulations from the point of view of the farmers they work with (Magnin 2020). Although they sometimes have to deal with tense situations such as the CAP suspension of land consolidation, they do not adopt an overbearing position by imposing environmental requirements on farmers' practices. Instead, they provide an opportunity for consensus with regard to hedgerows. The social polysemy of the hedgerow allows for these shifts in meaning: it is an agroecological infrastructure but also a marker of landscape identity, a source of wood, an object with long-lost technicity, not to mention that its destruction is costly. These interactions are opportunities for socialisation to agroecology, all the more effective as it is not imposed as such7 . The task of these advisors is, precisely, to reach out to 'historical' farmers (L'Atelier Paysan, 2021, p. 253), in other words, conventional farmers who inherited their farms, with whom the authors encourage establishing connections.

If agroforestry advisors act as diplomats between the administration and farmers, this is not by nature but because they are already obliged to negotiate and compromise within their own organisations. The Afac-Agroforesteries brings together organisations that, on paper, are varied or even antagonistic: the Chambers of Agriculture, under the aegis of the majority agricultural trade unions; hunters' federations, which maintain a population of wild fauna that may damage crops; environmental associations, not known for their cordial relations with mainstream conventional farmers or hunters; local authorities, subject to the vagaries of local votes; companies, whose economic model is radically different from the structures mentioned here, etc. At national and local levels, the Afac's work is an example of an attempt -sometimes successful, sometimes not -to overcome the "sociological confinement" (L'Atelier Paysan, 2021, p. 166) into which, according to the authors, the left-wing peasant and ecologist movements risk sinking.

We should also take note of the fact that this constant compromise does not lead, as one might expect, to a purely technical political position or to the weakening of the ambitions to transform agriculture, i.e. to the "depoliticisation" (p. 173) denounced by the Atelier Paysan (2021). On the contrary, even though Afac-Agroforesteries positions itself as an expert body on a specific subject, in 2019 it joined the 'Pour une Autre PAC' platform, which became the 'Nourrir' collective in 2022, bringing together environmental associations and the peasant left. Taking this stand, after twelve years of existence, is relatively risky for an association because it places it in a defined camp in the agricultural arena. This positioning stems from a desire to overcome the obstacles that Afac members have encountered in their work for hedgerows, locally in technical advising, and nationally by participating in CAP negotiations.

Conclusion

The main lesson of this paper is to highlight the benefits for sociology to take seriously the political analyses of stakeholders, not only as objects of study, but also as sparks to inspire the sociological imagination. The method developed here is not a theoretical assessment of the authors' positions, but the testing of some of their arguments in a case study related to their discourse. This challenge allowed me to take a fresh look at my object of study, hedgerows, from an original perspective while identifying its strengths and limitations, in the hope that researchers and field workers may find useful analyses.

For a review, see(Meyer

2022).

For a review, see(Magnin 2021a).

See his speech in the Ministry of Agriculture's video "Plantons des haies", 8 April 2021.

"Les chantiers que le ministre doit boucler avant l'élection présidentielle", Terre-Net, 30 September 2021.

At least since the massive disappearance of subsistence practices that took from hedges blackberries for jam or hazelnuts for oil[START_REF] Lizet | Brider le buisson" : une logique de la haie paysanne[END_REF].

https://www.elysee.fr/emmanuel-macron/2021/10/12/presentation-du-plan-france-2030, consulted 25 October 2022.

However, it cannot be said either that hedgerows are a gateway that, once pushed open, naturally leads to farms' total transformation. The consequences of reclassifying hedgerows as an integral part of the farm on the rest of its components is an avenue of research that future empirical studies could address.


V Varagapriya
email: varagapriyav@gmail.com

Vikram Vikas

Abdel Singh

Lisser

Vikas Vikram Singh
email: vikassingh@iitd.ac.in

Abdel Lisser
email: abdel.lisser@l2s.centralesupelec.fr

Joint Chance-Constrained Markov Decision Processes

Keywords: Constrained Markov decision process, Joint chance constraint, Second order cone programming problem, Copula, Elliptical distribution, Queueing problem. 1 Joint Chance-Constrained Markov Decision Processes

published or not. The documents may come   L'archive ouverte pluridisciplinaire

Introduction

Markov decision process (MDP) is a decision making framework for a stochastic system which evolves over finite/infinite horizon. The system can have a finite or infinite number of states, and at each state, a controller chooses an action from a set of available actions at that state. The system moves from one state to another state according to a controlled Markov chain. At every stage, the system incurs cost whenever a state is visited and an action is taken. The objective is to find a policy which minimizes the expected cost incurred over a period of time. The discounted and average cost criteria are most commonly used in the literature. For a finite state-action MDP with known transition probabilities and running costs, there exists a stationary deterministic optimal policy, and it can be obtained by using dynamic programming and Linear programming (LP) based methods [START_REF] Altman | Constrained Markov Decision Processes[END_REF][START_REF] Bertsekas | Dynamic Programming and Optimal Control, 2nd edn[END_REF][START_REF] Puterman | Markov Decision Process[END_REF]. MDPs can also be solved via policy iteration [START_REF] Ye | The simplex and policy-iteration methods are strongly polynomial[END_REF]. In many real-life examples, there are more than one running costs incurred at every stage whenever the Markov chain visits a state and an action is taken. For instance, in [START_REF] Altman | Constrained Markov Decision Processes[END_REF], the author studies the example of service and admission control in a queueing system in which the controller incurs three different costs at every stage. In [START_REF] Varagapriya | Constrained Markov decision processes with uncertain costs[END_REF], the authors study a variant of a machine replacement problem by considering the case when the controller incurs opportunity cost together with maintenance cost at every stage. In such cases, one type of running cost defines a long-run expected cost function which needs to be minimized and other types of running costs define the long-run expected cost functions on which constraints are imposed. This class of MDP is called a constrained Markov decision process (CMDP). For a finite state-action CMDP problem, when running costs and transition probabilities are stationary and exactly known, there exists a stationary randomized optimal policy which can be obtained by solving an equivalent LP problem [START_REF] Altman | Constrained Markov Decision Processes[END_REF].

In practice, the MDP model parameters such as running costs and transition probabilities are not exactly known. They are either learnt with experience or from historical data, thereby leading to errors in the optimal policies [START_REF] Mannor | Bias and variance approximation in value function estimates[END_REF]. The early literature on MDPs with uncertain parameters dates back to the 1970s, where MDPs under uncertain transition probabilities were studied [START_REF] Satia | Markovian decision processes with uncertain transition probabilities[END_REF][START_REF] White | Markov decision processes with imprecise transition probabilities[END_REF]. Previous studies have accounted for the uncertainties in MDPs under a robust optimization framework by considering the worst case realization of the uncertain parameters [START_REF] Iyengar | Robust dynamic programming[END_REF][START_REF] Nilim | Robust control of Markov decision processes with uncertain transition matrices[END_REF][START_REF] Wiesemann | Robust Markov decision processes[END_REF]. Recently, the equivalent reformulations of robust CMDP problem, with uncertain costs and known transition probabilities, for various uncertainty sets are proposed by Varagapriya et al. [START_REF] Varagapriya | Constrained Markov decision processes with uncertain costs[END_REF]. Delage and Mannor [START_REF] Delage | Percentile optimization for Markov decision processes with parameter uncertainty[END_REF] observed that robust optimization problems can give highly conservative optimal solutions. As an alternative approach, they considered an uncertain MDP problem under discounted cost criterion where either running costs or transition probabilities are defined as random variables and proposed a chance constraint programming based formulation. The chance constraint based MDP is equivalent to a second-order cone programming (SOCP) problem when the running cost vector follows a normal distribution and transition probabilities are known [START_REF] Delage | Percentile optimization for Markov decision processes with parameter uncertainty[END_REF]. When the prior transition probabilities follow Dirichlet distribution and running costs are known, the uncertain MDP problem becomes intractable and optimal policies can be computed using approximation methods [START_REF] Delage | Percentile optimization for Markov decision processes with parameter uncertainty[END_REF].

To the best of our knowledge, CMDPs where either running costs or transition probabilities are defined using random variables are not considered in the literature. In this paper, we consider a finite state-action CMDP problem under both discounted and average cost criteria where running cost vectors are random vectors and transition probabilities are known. We use chance constraint programming [START_REF] Charnes | Chance-constrained programming[END_REF][START_REF] Prékopa | Stochastic Programming[END_REF] to model the uncertainties present in the objective function and the constraints of the CMDP problem. For a given policy, the chance constraint based objective function of the CMDP problem gives a minimum long-run expected cost which is incurred with at least a given probability level. The random constraints of the CMDP problem are defined as a joint chance constraint which guarantees that the random constraints are jointly satisfied with at least a given probability level. We call this uncertain CMDP as a joint chance-constrained Markov decision process (JCCMDP). In general, the random constraint vectors present in the joint chance constraint can be dependent random vectors. In order to study the dependence structure of random variables, the concept of copula was introduced by Abe Sklar in 1959 [START_REF] Sklar | Fonctions de répartition à n dimensions et leurs marges[END_REF]. We consider the case when running cost vectors follow multivariate elliptically symmetric distribution and dependence among random constraint vectors is driven by a Gumbel-Hougaard copula. The choice of elliptical distributions is due to the fact that it is a generalization of normal distributions and contains a list of important symmetric distributions [START_REF] Fang | Symmetric Multivariate and Related Distributions[END_REF] and they are closed under linear transformations. The Gumbel-Hougaard copula is considered because it is separable in nature. Using the fact that the elliptical distributions are closed under a linear transformation and Gumbel-Hougaard copula is separable, we propose a deterministic non-convex optimization problem which is equivalent to the JCCMDP problem. We approximate the equivalent non-convex problem by two SOCP problems whose optimal values give lower and upper bounds of the optimal value of the JCCMDP problem. As an application, we study the problem of admission and service control in a queueing system and perform numerical experiments by considering various instances of different sizes.

The rest of the paper is organized as follows. Section 2 introduces the definition of a CMDP. Section 3 presents a JCCMDP problem and its SOCP based approximations. Section 4 gives the results of numerical experiments carried out on the queueing control problem and on random CMDPs. We conclude the paper in Section 5.

Constrained Markov decision processes

We define an infinite horizon CMDP by the following objects [START_REF] Altman | Constrained Markov Decision Processes[END_REF]:

(i) S is a finite set of states whose generic element is denoted by s.

(ii) A is a finite set of actions and A(s) denotes the set of actions available at state s.

(iii) K = {(s, a) | s ∈ S, a ∈ A(s)} is the set of all state-action pairs; |K| denotes the cardinality of K. (iv) γ = (γ(s)) s∈S is a probability distribution according to which an initial state is chosen. (v) p ∶ K → ℘(S) is a transition probability function where ℘(S) is the set of probability distributions on S; p(s ′ |s, a) is the probability of moving from state s to s ′ when the controller chooses an action a ∈ A(s).

∈ A(s) is chosen. (viii) (ξ k ) K
k=1 are the bounds for the constraints. We consider a discrete-time vector stochastic process (X t , A t ) ∞ t=0 , where (X t , A t ) denotes the state-action pair at time t. At time t = 0, the state of the Markov chain is s 0 ∈ S according to an initial distribution γ and the controller chooses an action a 0 ∈ A(s 0 ). As a consequence, running costs c(s 0 , a 0 ), d k (s 0 , a 0 ), k = 1, 2, . . . , K, are incurred. The Markov chain moves to state s 1 at time t = 1 with probability p(s 1 |s 0 , a 0 ) and the same thing repeats at s 1 and it continues for the infinite horizon. In general, the decision at time t may depend on the history h t = (s 0 , a 0 , s 1 , a 1 , . . . , s t-1 , a t-1 , s t ) at time t. Let H t be the set of all possible histories at time t. A history dependent decision rule f t at time t is a function f t ∶ H t → ℘(A), where for every h t ∈ H t with final state s t , f t (h t ) ∈ ℘(A(s t )). A sequence of history dependent decision rules

f h = (f t ) ∞
t=0 is called a history dependent policy. When each f t in the sequence (f t ) ∞ t=0 depends only on the state at time t, it is called a Markovian policy. A Markovian policy (f t ) ∞ t=0 is called a stationary policy if there exists a decision rule f such that f t = f for all t. We write a stationary policy (f, f, . . . ) by f with abuse of notation, and define f = (f (s)) s∈S such that f (s) ∈ ℘(A(s)) for every s ∈ S. According to a stationary policy f , whenever the Markov chain visits state s, the controller chooses an action from A(s) according to the decision rule f (s). We denote the set of all history dependent and stationary policies by F HD and F S , respectively.

The policy f h and initial distribution γ define a probability measure P f h γ over the state and action trajectories (for details see Section 2.1.6 of [START_REF] Puterman | Markov Decision Process[END_REF]), and E f h γ denotes the expectation operator corresponding to probability measure P f h γ . For a given history dependent policy f h ∈ F HD and an initial distribution γ, the expected discounted costs at a discount factor α ∈ (0, 1) are defined as

[1] C α (γ, f h ) = (1 -α) ∞ ∑ t=0 α t E f h γ c(X t , A t ) = ∑ s∈S ∑ a∈A(s) g α (γ, f h ; s, a)c(s, a), D k α (γ, f h ) = (1 -α) ∞ ∑ t=0 α t E f h γ d k (X t , A t ) = ∑ s∈S ∑ a∈A(s) g α (γ, f h ; s, a)d k (s, a),
for all k = 1, 2, . . . , K, where (1α) is a normalization constant. The set {g α (γ, f h ; s, a)} (s,a) is the occupation measure defined by

g α (γ, f h ; s, a) = (1 -α) ∞ ∑ t=0 α t P f h γ (X t = s, A t = a), ∀ (s, a) ∈ K.
For a given history dependent policy f h and an initial distribution γ, the expected average costs are defined as [1]

C ea (γ, f h ) = lim sup n→∞ 1 n n-1 ∑ t=0 E f h γ c(X t , A t ) = ∑ s∈S ∑ a∈A(s)
g ea (γ, f h ; s, a)c(s, a),

D k ea (γ, f h ) = lim sup n→∞ 1 n n-1 ∑ t=0 E f h γ d k (X t , A t ) = ∑ s∈S ∑ a∈A(s) g ea (γ, f h ; s, a)d k (s, a),
for all k = 1, 2, . . . , K. The set {g ea (γ, f h ; s, a)} (s,a) is an occupation measure obtained from the accumulation points of {{g n ea (γ, f h ; s, a)} (s,a) } n , where

g n ea (γ, f h ; s, a) = 1 n n-1 ∑ t=0 P f h γ (X t = s, A t = a), ∀ (s, a) ∈ K.
The optimal policy of a CMDP problem can be obtained by solving the following optimization problem min

f h ∈F HD C v (γ, f h ) s.t. D k v (γ, f h ) ≤ ξ k , ∀ k = 1, 2, . . . , K, (1)
where v = α and v = ea represent a CMDP problem with discounted cost criterion and average cost criterion, respectively.

We make the following assumption on the average cost criterion throughout the paper.

Assumption 1. For the CMDP problem with average cost criterion, we assume that the CMDP is unichain, i.e., under any stationary deterministic policy, the Markov chain is aperiodic and irreducible (with possibly some transient states) [START_REF] Altman | Constrained Markov Decision Processes[END_REF][START_REF] Puterman | Markov Decision Process[END_REF].

Assumption 1 ensures that the accumulation points of {{g n ea (γ, f h ; s, a)} (s,a) } n is unique and independent of the initial distribution γ. Therefore, the expected average costs are well defined. We do not need Assumption 1 in the case of discounted cost criterion. It is well known that the CMDP problem (1) for v ∈ {α, ea} can be restricted to the class of stationary policies without loss of optimality when the running costs and transition probabilities are stationary, i.e., they depend only on states and actions but time. This follows from the fact that the set of occupation measures defined with respect to history dependent policies and the set of occupation measures defined with respect to stationary policies are equal (see Theorems 3.1 and 4.1 of [START_REF] Altman | Constrained Markov Decision Processes[END_REF]). Moreover, it follows from Theorem 3.2 (resp. Theorem 4.2) of [START_REF] Altman | Constrained Markov Decision Processes[END_REF] that the set of occupation measures defined with respect to stationary policies for the discounted (resp. average) cost criterion and the set Q

α (γ) (resp. Q ea (γ)) are equal. For v ∈ {α, ea}, the set Q v (γ) is defined as Q v (γ) ={ρ ∈ R |K| | ∑ (s,a)∈K ρ(s, a)(δ(s, s ′ ) -ηp(s ′ |s, a)) = (1 -η)γ(s ′ ), ∀ s ′ ∈ S, ∑ (s,a)∈K ρ(s, a) = 1, ρ(s, a) ≥ 0, ∀ (s, a) ∈ K},
where η ∈ (0, 1] and δ(s, s ′ ) is the Kronecker delta which takes the value 1 when s = s ′ , and 0 otherwise. If η < 1, we take it as the discount factor α and Q

v (γ) = Q α (γ), while if η = 1, Q v (γ) = Q ea (γ). Therefore, the CMDP problem (1) is equivalent to the following LP problem min ρ ∑ s∈S ∑ a∈A(s) ρ(s, a)c(s, a) s.t. ∑ s∈S ∑ a∈A(s) ρ(s, a)d k (s, a) ≤ ξ k , ∀ k = 1, 2, . . . , K, ρ ∈ Q v (γ). (2)
If ρ * is an optimal solution of (2), the optimal stationary policy f * of the CMDP problem is defined as f * (s, a) = ρ * (s,a) ∑ a∈A(s) ρ * (s,a) for all s ∈ S, a ∈ A(s), provided the denominator is non-zero (if it is zero, we choose f * (s) arbitrarily from ℘(A(s))).

CMDP under chance constraints

In real-life scenario, the model parameters such as running costs and transition probabilities are subject to errors and are not exactly known. In this paper, we assume that the transition probabilities are stationary and exactly known while the running cost vectors are random vectors defined on a probability space (Ω, F, P). We denote the running cost vectors corresponding to objective function and constraints by ĉ and dk , k = 1, 2, . . . , K, respectively. Therefore, for each realization ω ∈ Ω, ĉ(s, a, ω), dk (s, a, ω) represents the real valued costs which are incurred at state s when an action a is chosen. We assume that the uncertainty in running costs is stationary in nature, i.e., the random cost vectors ĉ and dk , k = 1, 2, . . . , K, do not vary with time. For a given f h ∈ F HD , the costs corresponding to the objective function and the constraints of (1), denoted by, Ĉv (γ,

f h ), Dk v (γ, f h ), k = 1, 2, .
. . , K, are random variables and it makes (1) a stochastic optimization problem. We consider the case where the controller is interested in the minimum long-run expected cost which is incurred with at least p 0 probability, and the random constraints are jointly satisfied with at least p 1 probability. This leads to the following joint chance constraint programming based formulation of the CMDP problem

min t,f h ∈F HD t s.t. P( Ĉv (γ, f h ) ≤ t) ≥ p 0 , P( Dk v (γ, f h ) ≤ ξ k , k = 1, 2, . . . , K) ≥ p 1 . (3)
We call (3) a joint chance-constrained Markov decision process (JCCMDP) problem.

Proposition 1. Let Assumption 1 holds for average cost criterion. Then, the JCCMDP problem (3) can be restricted to the class of stationary policies without loss of optimality. Moreover, it can be equivalently written using the decision vector (t, ρ) as

min t,ρ t s.t. (i) P( ∑ (s,a)∈K ρ(s, a)ĉ(s, a) ≤ t) ≥ p 0 , (ii) P( ∑ (s,a)∈K ρ(s, a) dk (s, a) ≤ ξ k , k = 1, 2, . . . , K) ≥ p 1 , (iii) ρ ∈ Q v (γ). (4)
If ρ * is a part of an optimal solution vector of (4), the optimal stationary policy f * of the JCCMDP problem is given by f * (s, a) = ρ * (s,a) ∑ a∈A(s) ρ * (s,a) for all s ∈ S, a ∈ A(s), provided the denominator is non-zero (if it is zero, we choose f * (s) arbitrarily from ℘(A(s))).

Proof Fix a realization ω ∈ Ω. Let (t, f h ) be a feasible solution of (3). It follows from Theorems 3.1 and 4.1 of [START_REF] Altman | Constrained Markov Decision Processes[END_REF] that the set of occupation measures corresponding to history dependent strategies and stationary strategies are the same, i.e., there exists f ∈ F S such that gv(γ, f h ; s, a) = gv(γ, f ; s, a) for all (s, a) ∈ K. Hence, Ĉv(γ,

f h , ω) = Ĉv(γ, f, ω) and Dk v (γ, f h , ω) = Dk v (γ, f, ω), k = 1, 2, .
. . , K. Thus, we can restrict the JCCMDP problem (3) to the class of stationary policies and obtain the following equivalent problem min

t,f ∈F S t s.t. P( Ĉv(γ, f ) ≤ t) ≥ p 0 , P( Dk v (γ, f ) ≤ ξ k , k = 1, 2, . . . , K) ≥ p 1 . (5)
Now, we show that ( 4) and ( 5) are equivalent. Let (t, f ) be a feasible point of (5). This implies that (t, f ) is a feasible solution of (5). Thus, ( 5) is equivalent to [START_REF] Charnes | Chance-constrained programming[END_REF]. □

The problem ( 4) is an LP problem with a joint chance constraint [START_REF] Luedtke | An integer programming approach for linear programs with probabilistic constraints[END_REF]. In general, such problems are hard to solve. This is because, for any given ρ, finding the feasibility of the joint chance constraint requires multi-dimensional integration, which is NP-hard [START_REF] Geng | Data-driven decision making in power systems with probabilistic guarantees: Theory and applications of chance-constrained optimization[END_REF]. Moreover, the feasible set of this constraint is not convex. In order to obtain an equivalent deterministic optimization problem, we need the information of the distributions of random variables ρ T ĉ and ρ T dk . These random variables are linear combinations of random running cost vectors ĉ and dk . Therefore, we assume that the running cost vectors ĉ, dk follow multivariate elliptically symmetric distributions. The class of elliptically symmetric distributions is a generalization of the normal distribution [START_REF] Fang | Symmetric Multivariate and Related Distributions[END_REF] and it is closed under linear transformation (Theorem 2.16 of [START_REF] Fang | Symmetric Multivariate and Related Distributions[END_REF]). Thus, the distribution of running cost vectors extends to the associated long-run expected costs. Further, we assume the dependence among the random constraints present under the joint chance constraint of ( 4) is driven by Gumbel-Hougaard copula. The separable nature of this copula enables us to express the joint chance constraint as a set of individual chance constraints. We first review a few definitions and results on copulas and elliptical distributions, which will be used in subsequent analysis.

Preliminaries

Definition 1 (Copula [6, 13, 16]) A K-dimensional copula, where K ≥ 2, is a distribution function on [0, 1] K such that all its one dimensional marginals follow a uniform distribution on [0, 1].

The relation between the joint distribution of a random vector X = (X 1 , X 2 , . . . , X K ) and a copula is given by the Sklar's theorem [START_REF] Nelsen | An Introduction to Copulas[END_REF].

Theorem 1 (Sklar's theorem [START_REF] Nelsen | An Introduction to Copulas[END_REF]). For a given K-dimensional distribution function Φ and all its one dimensional marginals Φ1 , Φ2 , . . . , ΦK , there exists a copula C that satisfies

Φ(v 1 , v 2 , . . . , v K ) = C( Φ1 (v 1 ), Φ2 (v 2 ), . . . , ΦK (v K )), ∀ v 1 , v 2 , . . . , v K ∈ R.
When all the marginals are continuous, C is unique throughout its domain; else it is unique on ⨉ K i=1 Range Φi ,

where Range Φi denotes the set of all possible values Φi can take for i = 1, 2, . . . , K. In this paper, we consider a particular class of copulas, namely, the Gumbel-Hougaard family of copulas, denoted by C θ , defined as in [6]

C θ (u) = exp ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ -[ K ∑ k=1 ( -ln(u k )) θ ] 1 θ ⎫ ⎪ ⎪ ⎬ ⎪ ⎪ ⎭ , (6)
where u = (u 1 , u 2 , . . . , u K ) T ∈ [0, 1] K , θ ≥ 1 and exp is the exponential function. When θ = 1, C 1 (u) = ∏ K k=1 u k is the product copula. The joint distribution function of a random vector X defined by a product copula implies that the random variables X 1 , X 2 , . . . , X K are independent.

Definition 2 (Elliptical distribution [START_REF] Fang | Symmetric Multivariate and Related Distributions[END_REF]) A K-dimensional random vector X is said to follow a multivariate elliptically symmetric distribution if its characteristic function has the form E(e ib T X

) = e ib T µ φ(b T Σb), where µ ∈ R K is the location parameter, the matrix Σ ∈ R K×K is the scale parameter and φ is the characteristic generator. We denote it as X ∼ E K (µ, Σ, φ).

If X ∼ E K (µ, Σ, φ) and the density function of X, denoted by ϕ exists, then it has the form

ϕ(x) = 1 √ det(Σ) g((x -µ) T Σ -1 (x -µ)),
where the scale matrix Σ is positive definite, det(Σ) is the determinant of Σ, and the function g is the density generator. In such a case, we also write X ∼ E K (µ, Σ, g). From Theorem 2.16 of [START_REF] Fang | Symmetric Multivariate and Related Distributions[END_REF], a linear combination of the components of an elliptically distributed random vector follows a univariate elliptical distribution. Thus, if

X ∼ E K (µ, Σ, φ), for a b ∈ R K , b T X ∼ E 1 (b T µ, b T Σb, φ). Moreover, if Σ is a positive definite matrix, then b T X-b T µ ∥Σ 1 2 b∥2
is a spherically distributed random variable with characteristic generator φ.

Second order cone approximations

By using Sklar's theorem given in Section 3.1, the joint chance constraint (ii) of ( 4) can be equivalently written as

P( ∑ (s,a)∈K ρ(s, a) dk (s, a) ≤ ξ k , k = 1, 2, . . . , K) = C θ,ρ (P(ρ T d1 ≤ ξ 1 ), P(ρ T d2 ≤ ξ 2 ), . . . , P(ρ T dK ≤ ξ K )) ≥ p 1 ,
where C θ,ρ denotes the copula that defines the joint distribution of the random vector (ρ T d1 , ρ T d2 , . . . , ρ T dK ) T . We make the following assumption on this copula.

Assumption 2. There exists a K-dimensional Gumbel-Hougaard copula, C θ given by (6), such that C θ,ρ = C θ for all ρ ∈ Q v (γ).

Proposition 2. Let Assumption 2 holds, and ĉ ∼ E |K| (µ ĉ, Σ ĉ, φ ĉ), dk ∼ E |K| (µ dk , Σ dk , φ dk ), k = 1, 2, . . . , K, such that Σ ĉ, Σ dk , k = 1, 2, . . . , K, are positive definite matrices. Then, the JCCMDP problem (4) is equivalent to the following non-linear programming problem,

min t,ρ,(y k ) K k=1 t s.t. (i) ρ T µ ĉ + Φ -1 ĉ (p 0 )∥Σ 1 2 ĉ ρ∥ 2 ≤ t, (ii) ρ T µ dk + Φ -1 dk (p y 1 θ k 1 )∥Σ 1 2 dk ρ∥ 2 ≤ ξ k , ∀ k = 1, 2, . . . , K, (iii)
K ∑ k=1 y k = 1, y k ≥ 0, ∀ k = 1, 2, . . . , K, (iv) ρ ∈ Q v (γ), (7)
where p 0 , p 1 ∈ (0.5, 1) and Φ -1 ĉ (⋅), Φ -1 dk (⋅) are the quantile functions of spherical distributions corresponding to ĉ, dk , respectively.

Proof Since, ĉ ∼ E |K| (µ ĉ, Σ ĉ, φ ĉ), the constraint (i) of ( 4) is equivalent to constraint (i) of (7) (see Lemma 2.2 of [START_REF] Henrion | Structural properties of linear probabilistic constraints[END_REF]). It follows from Lemma 1 of [6] that the joint chance constraint (ii) of ( 4) can be equivalently written as

P(ρ T dk ≤ ξ k ) ≥ p y 1 θ k 1 , ∀ k = 1, 2, . . . , K, K ∑ k=1 y k = 1,
y k ≥ 0, ∀ k = 1, 2, . . . , K. (8)
We provide the proof to make our paper self-contained. Let ρ be a feasible point of the joint chance constraint (ii) of ( 4). Define ȳk , y k , k = 1, 2, . . . , K, such that From the joint chance constraint (ii) of ( 4), we have

P(ρ T dk ≤ ξ k ) = p
p 1 ≤ P( ∑ (s,a)∈K ρ(s, a) dk (s, a) ≤ ξ k , k = 1, 2, . . . , K) = C θ (P(ρ T d1 ≤ ξ 1 ), P(ρ T d2 ≤ ξ 2 ), . . . , P(ρ T dK ≤ ξ K )) = C θ (p ȳ 1 θ 1 1 , p ȳ 1 θ 2 1 , . . . , p ȳ 1 θ K 1 ) = exp ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ -[ K ∑ k=1 ( -ln(p ȳ 1 θ k 1 )) θ ] 1 θ ⎫ ⎪ ⎪ ⎬ ⎪ ⎪ ⎭ = exp ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ -[ K ∑ k=1 ȳk ( ln 1 p 1 ) θ ] 1 θ ⎫ ⎪ ⎪ ⎬ ⎪ ⎪ ⎭ = exp ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ ( K ∑ k=1 ȳk ) 1 θ ln(p 1 ) ⎫ ⎪ ⎪ ⎬ ⎪ ⎪ ⎭ = p ( ∑ K k=1 ȳk ) 1 θ 1 ,
where the first equality follows by Sklar's theorem. The condition p 1 ≤ p

( ∑ K k=1 ȳk ) 1 θ 1 implies that ∑ K k=1 ȳk ≤ 1.
Then, from the definition of y k , we have ȳk ≤ y k for all k, which in turn implies that

P(ρ T dk ≤ ξ k ) ≥ p y 1 θ k 1 , ∀ k = 1, 2, . . . , K.
The other two constraints of (8) follow from the definition of y k . Conversely, suppose (ρ, (y k ) K k=1 ) is feasible for [START_REF] El Asri | Score-based inverse reinforcement learning[END_REF]. By Sklar's theorem and the first constraint of (8),

P( ∑ (s,a)∈K ρ(s, a) dk (s, a) ≤ ξ k , k = 1, 2, . . . , K) = C θ (P(ρ T d1 ≤ ξ 1 ), P(ρ T d2 ≤ ξ 2 ), . . . , P(ρ T dK ≤ ξ K )) ≥ C θ (p y 1 θ 1 1 , p y 1 θ 2
1 , . . . , p

y 1 θ K 1 ) = exp ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ -[ K ∑ k=1 ( -ln(p y 1 θ k 1 )) θ ] 1 θ ⎫ ⎪ ⎪ ⎬ ⎪ ⎪ ⎭ = p 1 .
Hence, ρ is a feasible solution of (ii) of (4). Since, dk ∼ E |K| (µ dk , Σ dk , φ dk ), k = 1, 2, . . . , K, it follows from Lemma 2.2 of [START_REF] Henrion | Structural properties of linear probabilistic constraints[END_REF] that the set of constraints in (8) can be equivalently written as

ρ T µ dk + Φ -1 dk (p y 1 θ k 1 )∥Σ 1 2 dk ρ∥ 2 ≤ ξ k , ∀ k = 1, 2, . . . , K, K ∑ k=1 y k = 1, y k ≥ 0, ∀ k = 1, 2, . . . , K.
Thus, ( 4) is equivalent to [START_REF] Delage | Percentile optimization for Markov decision processes with parameter uncertainty[END_REF]. □ Remark 1. When K = 1, the constraint (iii) of (7) gives y 1 = 1 and (7) reduces to the following SOCP problem

min t,ρ t s.t. ρ T µ ĉ + Φ -1 ĉ (p 0 )∥Σ 1 2 ĉ ρ∥ 2 ≤ t, ρ T µ d1 + Φ -1 d1 (p 1 )∥Σ 1 2 d1 ρ∥ 2 ≤ ξ 1 , ρ ∈ Q v (γ).
For K ≥ 2, the optimization problem ( 7) is a non-convex optimization problem due to the product terms Φ -1 dk (p

y 1 θ k 1 )∥Σ 1 2
dk ρ∥ 2 present in its constraints. We approximate [START_REF] Delage | Percentile optimization for Markov decision processes with parameter uncertainty[END_REF] by two SOCP problems whose optimal values give lower and upper bounds on the optimal value of the JCCMDP problem. Our approximations hold under the following assumption. Assumption 3. For each k = 1, 2, . . . , K, the quantile function Φ -1 dk (p

y 1 θ k 1 ) is a convex function of y k .
For the spherical distributions listed in Table 1, Assumption 3 holds true. This follows from the fact that Φ -1 dk (p

y 1 θ k 1 )
(Φ -1 dk ) ′′ (y) = -ϕ ′ dk (Φ -1 dk (y)) (ϕ dk (Φ -1 dk (y))) 3 ≥ 0, ∀ y ∈ [p 1 , 1),
where ϕ dk (⋅) is the probability density function of the spherical distribution whose cumulative distribution function is Φ dk (⋅). The convexity of Φ -1 dk (y k ) for all k, on [p 1 , 1] holds because it is a continuous function at y k = 1. We Table 1 List of some spherical distributions.

Probability Distribution

Density Function (ϕ dk (u))

Normal ce -u 2 2
Pearson Type VII c(1

+ u 2 s ) -N ; N > 1 2 , s > 0 t c(1 + u 2 m ) -(1+m) 2 ; m > 0 an integer Cauchy c(1 + u 2 s ) -1 ; s > 0 Logistic ce -u 2 (1+e -u 2 ) 2 Laplace ce -|u|
The constant c > 0 is the normalization factor that ensures the density function gives 1 on integration.

approximate the functions Φ -1 dk (p

y 1 θ k
1 ), k = 1, 2, . . . , K, using piecewise tangent and linear approximations to obtain two SOCP problems whose optimal values give the lower and upper bounds for the optimal value of (7).

Lower bound approximation

For every k, we approximate the term Φ -1 dk (p

y 1 θ k
1 ) in ( 7) by choosing N tangent points y 1 k < y 2 k < . . . < y N k from the interval (0, 1]. We denote the Taylor series expansion around the point y i k by Φ -1 dk (p

y 1 θ k
1 ) i and define it as

Φ -1 dk (p y 1 θ k 1 ) i = Φ -1 dk (p y i k 1 θ 1 ) + (y k -y i k ) d dy k (Φ -1 dk (p y i k 1 θ 1 )) = Φ -1 dk (p y i k 1 θ 1 ) - (y i k ) 1 θ p y i k 1 θ 1 ln(p 1 )
θϕ dk (Φ -1 dk (p

y i k 1 θ 1
))

+ p y i k 1 θ 1 ln(p 1 )(y i k ) 1 θ -1 y k θϕ dk (Φ -1 dk (p y i k 1 θ 1
))

= āi k + bi k y k , ∀ i = 1, 2, . . . , N,
where āi k = Φ -1 dk (p

y i k 1 θ 1 ) - (y i k ) 1 θ p y i k 1 θ 1 ln(p 1 )
θϕ dk (Φ -1 dk (p

y i k 1 θ 1
))

, bi k = p y i k 1 θ 1 ln(p 1 )(y i k ) 1 θ -1 θϕ dk (Φ -1 dk (p y i k 1 θ 1
))

.

Theorem 2. Let Assumption 3 holds. A lower bound for the optimal value of the equivalent JCCMDP problem (7) is given by the optimal value of the following SOCP problem

min t,ρ,(x kj ) K k=1, |K| j=1 ,(z kj ) K k=1, |K| j=1 t s.t. ρ T µ ĉ + Φ -1 ĉ (p 0 )∥Σ 1 2 ĉ ρ∥ 2 ≤ t, ρ T µ dk + ∥Σ 1 2 dk z k ∥ 2 ≤ ξ k , ∀ k = 1, 2, . . . , K, z kj ≥ (ā i k ρ j + bi k x kj ), ∀ k = 1, 2, . . . , K, j = 1, 2, . . . , |K|, i = 1, 2, . . . , N, K ∑ k=1
x kj = ρ j , ∀ j = 1, 2, . . . , |K|,

x kj ≥ 0, ∀ k = 1, 2, . . . , K, j = 1, 2, . . . , |K|, ρ ∈ Q v (γ), ( 9
)
where ρ j is the j-th component of ρ ∈ R |K| and z k = (z kj )

|K| j=1 for every k = 1, 2, . . . , K.
Proof It is sufficient to show that for every feasible solution of ( 7) there exists a feasible solution of ( 9). The proof is similar to Theorem 3.1.1 of [START_REF] Cheng | A second-order cone programming approach for linear programs with joint probabilistic constraints[END_REF]. Suppose the vector (t, ρ, (y k ) K k=1 ) is a feasible solution of [START_REF] Delage | Percentile optimization for Markov decision processes with parameter uncertainty[END_REF]. Under Assumption 3, we obtain Φ -1 dk (p

y 1 θ k 1 ) ≥ max 1≤i≤N (ā i k + bi k y k ) for all k. Therefore, ρ T µ dk + max 1≤i≤N (ā i k + bi k y k )∥Σ 1 2 dk ρ∥ 2 ≤ ξ k , ∀ k = 1, 2, . . . , K.
The vectors z k , k = 1, 2, . . . K, where z kj = max 1≤i≤N (ā i k ρ j + bi k y k ρ j ) for all k = 1, 2, . . . , K, j = 1, 2, . . . , |K|, satisfy

ρ T µ dk + ∥Σ 1 2 dk z k ∥ 2 ≤ ξ k , ∀ k = 1, 2, . . . , K, z kj ≥ āi k ρ j + bi k y k ρ j , ∀ k = 1, 2, . . . , K, j = 1, 2, . . . , |K|, i = 1, 2, . . . , N.
Then, the vector (t, ρ,

(x kj ) K k=1, |K| j=1 , (z kj ) K k=1, |K| j=1
) where

x kj = y k ρ j , ∀ k = 1, 2, . . . , K, j = 1, 2, . . . , |K|,
is a feasible solution for the problem [START_REF] Fang | Symmetric Multivariate and Related Distributions[END_REF]. □

Upper bound approximation

For every k, we approximate the term Φ -1 dk (p

y 1 θ k
1 ) in ( 7) by choosing N interpolation points y 1 k < y 2 k < . . . < y N k from the interval (0, 1], such that the point y 1 k is sufficiently close to 0. We denote the linear interpolating polynomial passing through the points (y i k , Φ -1 dk (p

1 θ 1
)

y i+1 k -y i k .
Theorem 3. Let Assumption 3 holds. An upper bound for the optimal value of the equivalent JCCMDP problem (7) is given by the optimal value of the following SOCP problem

min t,ρ,(y k ) K k=1 ,(z k ) K k=1 t s.t. ρ T µ ĉ + Φ -1 ĉ (p 0 )∥Σ 1 2 ĉ ρ∥ 2 ≤ t, ρ T µ dk + z k V * k ≤ ξ k , ∀ k = 1, 2, . . . , K, z k ≥ âi k + bi k y k , ∀ k = 1, 2, . . . , K, i = 1, 2, . . . , N -1, K ∑ k=1 y k = 1, y k ≥ 0, ∀ k = 1, 2, . . . , K, ρ ∈ Q v (γ), (10)
where

ρ j is the j-th component of ρ ∈ R |K| , (Σ 1 2 dk ) j is the j-th column of the matrix Σ 1 2
dk and,

V * k = max ρ∈Qv(γ) |K| ∑ j=1 ρj ∥(Σ 1 2
dk ) j ∥ 2 for every k = 1, 2, . . . , K.

Proof We show that for every feasible solution of (10) there exists a feasible solution of [START_REF] Delage | Percentile optimization for Markov decision processes with parameter uncertainty[END_REF]. Suppose the vector (t, ρ, (y k ) K k=1 , (z k ) K k=1 ) is a feasible solution of [START_REF] Geng | Data-driven decision making in power systems with probabilistic guarantees: Theory and applications of chance-constrained optimization[END_REF]. It is enough to show that ρ is a feasible solution of constraint (ii) of [START_REF] Delage | Percentile optimization for Markov decision processes with parameter uncertainty[END_REF]. Under Assumption 3, we obtain Φ -1 dk (p

y 1 θ k 1 ) ≤ max 1≤i≤N -1 (â i k + bi k y k ) for all k. For an arbitrarily fixed k ∈ {1, 2, . . . , K}, consider ∥Σ 1 2 dk ρ∥ 2 = ∥ρ 1 (Σ 1 2 dk ) 1 + ρ 2 (Σ 1 2 dk ) 2 + . . . + ρ |K| (Σ 1 2 dk ) |K| ∥ 2 ≤ ρ 1 ∥(Σ 1 2 dk ) 1 ∥ 2 + ρ 2 ∥(Σ 1 2 dk ) 2 ∥ 2 + . . . + ρ |K| ∥(Σ 1 2 dk ) |K| ∥ 2 ≤ max ρ∈Qv(γ) |K| ∑ j=1 ρj ∥(Σ 1 2 dk ) j ∥ 2 = V * k .
The first inequality holds by triangular inequality of ∥⋅∥ 2 , while the second inequality holds because ρ ∈ Qv(γ). Thus, V * k is an upper bound of ∥Σ

1 2
dk ρ∥ 2 . Consequently, the following inequality holds true

ρ T µ dk + Φ -1 dk (p y 1 θ k 1 )∥Σ 1 2 dk ρ∥ 2 ≤ ρ T µ dk + max 1≤i≤N -1 (â i k + bi k y k )V * k ≤ ρ T µ dk + z k V * k . Therefore, ρ T µ dk + Φ -1 dk (p y 1 θ k 1 )∥Σ 1 2 dk ρ∥ 2 ≤ ξ k , ∀ k = 1, 2, . . . , K. Hence, the vector (t, ρ, (y k ) K k=1
) is a feasible solution for the problem [START_REF] Delage | Percentile optimization for Markov decision processes with parameter uncertainty[END_REF]. □

Numerical experiments

We perform numerical experiments using MOSEK solver of CVX package in MATLAB optimization toolbox, on an Intel(R) 64-bit Core(TM) i5-8250U CPU @ 1.60GHz with 8.0 GB RAM machine. We study our approximations on a queueing control problem [START_REF] Altman | Constrained Markov Decision Processes[END_REF] in Section 4.1, and on randomly generated CMDPs in Section 4.2. In all our numerical experiments, we fix p 0 , p 1 = 0.95, α = 0.99, and assume γ to be a uniformly distributed vector. For our approximations, we vary N , pick uniformly spaced points between 10 -5 and 1 and keep them as both tangent and interpolation points for all the constraints. We note that the interpolation point y 1 k is sufficiently close to 0.

Queueing control problem

We consider a stochastic version of a discrete-time service and admission control problem in a single queue introduced by Altman (see Chapter 5 of [START_REF] Altman | Constrained Markov Decision Processes[END_REF]).

The system has two controllers, namely, service and admission controllers. We consider them as a single controller because their joint objective is to minimize the expected cost incurred in the long-run. The states represent the queue length with buffer size L. Thus, S = {0, 1, . . . , L}, where s = 0 indicates that the queue is empty, while s = L indicates that the queue is full. The action set of the controller is A 1 × A 2 , where A 1 and A 2 are finite sets of probability values. If (a 1 , a 2 ) ∈ A 1 × A 2 , then a 1 and a 2 denote the probabilities with which the controller finishes the service and admits a customer into the system, respectively. We assume that

A 1 ⊆ [a 1 min , a 1 max ] and A 2 ⊆ [a 2 min , a 2 max ], where 0 < a 1 min ≤ a 1 max < 1 and 0 ≤ a 2 min ≤ a 2 max < 1. Moreover, when s = L,
we assume no admission takes place. The transition probabilities from [START_REF] Altman | Constrained Markov Decision Processes[END_REF] are given by

p(s ′ |s, (a 1 , a 2 )) = ⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ a 1 (1 -a 2 ) 1 ≤ s ≤ L -1, s ′ = s -1, a 1 a 2 + (1 -a 1 )(1 -a 2 ) 1 ≤ s ≤ L -1, s ′ = s, (1 -a 1 )a 2 0 ≤ s ≤ L -1, s ′ = s + 1, 1 -(1 -a 1 )a 2 s ′ = s = 0, 1 -a 1 s = L, s ′ = L, a 1 s = L, s ′ = L -1.
If at state s ∈ S, the controller chooses an action (a 1 , a 2 ), it incurs running costs c(s), d 1 (a 1 ) and d 2 (a 2 ). The running cost c(s) is the holding cost incurred due to customers waiting in the queue. Therefore, it only depends on the size of the queue. The holding cost is zero for an empty queue, i.e., c(0) = 0. The running cost d 1 (a 1 ) is a service cost incurred when a customer is served with probability a 1 and d 2 (a 2 ) denotes the cost when a customer is not admitted into the system with (1a 2 ) probability. The service cost d 1 (a 1 ) increases with a 1 and no admission cost d 2 (a 2 ) decreases with a 2 . These costs do not depend on the size of the queue. The controller aims to minimize the expected holding cost in the long-run by keeping the long-run expected service and admission costs below certain threshold values. Usually the costs in the queueing system are not realized before the decisions are taken. Therefore, the running cost vectors c, d 1 and d 2 are better modelled using random vectors. We assume that they follow normal distribution and denote them as ĉ ∼ N (µ ĉ, Σ ĉ), dk ∼ N (µ dk , Σ dk ), k = 1, 2, where ĉ = (ĉ(s)) s∈S , d1 = ( d1 (a 1 )) a 1 ∈A 1 and d2 = ( d2 (a 2 )) a 2 ∈A 2 . We consider the case where the number of states is 10, i.e., L = 9, and the controller finishes the service with probability a 1 = 0.75. The admission controller has two possible actions: 'No Admission (a 2 = 0)' or 'Admission with probability 0.8 (a 2 = 0.8)'. We take ĉ, d2 to be normally distributed random vectors and d1 to be fixed, thus K = 1. Hence, from Remark 1, it follows that the JCCMDP problem (4) reduces to an SOCP problem. We take ξ 2 = 9, µ ĉ = (0, 1, 2, . . . , 9) T , the matrix Σ ĉ ∈ R 10×10 with all diagonal values 0.9 and off-diagonal values 0.35, and

µ d2 = ( 10.0 7.60
) , Σ d2 = ( 0.80 0.24 0.24 0.61 ) .

The optimal values of the JCCMDP problem with the above data for discounted and average cost cases are 5.7963 and 6.2296, respectively. Figure 1 shows the optimal probability of not admitting a customer into the queue at various states. For both the cases, we conclude that the optimal probability of 'No Admission' is 1 at the last state. We consider a system where the action set of the controller is given by A 1 × A 2 = {(0.2, 0.75, 0.9), (0, 0.5, 0.8)}. Thus, K = 2, and we compute the Fig. 1 The solid lines marked with ′ * ′ and ′ ○ ′ denote the optimal probability of not admitting a customer into the queue for discounted and average cost problems, respectively.

lower bound and upper bound for the optimal value of the JCCMDP problem [START_REF] Delage | Percentile optimization for Markov decision processes with parameter uncertainty[END_REF] by solving the SOCP problems ( 9) and [START_REF] Geng | Data-driven decision making in power systems with probabilistic guarantees: Theory and applications of chance-constrained optimization[END_REF], respectively. For L = n -1 (|S| = n), we take µ ĉ = (0,

⎞ ⎟ ⎠

.

The mean vector µ ĉ, where µ ĉ(s) = s, captures the fact that the holding cost increases with s. We take the mean vectors µ d1 and µ d2 such that µ d1 (a 1 ) = 3(1 + a 1 ) 2 and µ d2 (a 2 ) = 10 -3a 2 . It is motivated from the fact that the service cost increases with a 1 and no admission cost decreases with a 2 . We take ξ 1 = 11.30, ξ 2 = 11.35. We solve the SOCP problems ( 9) and [START_REF] Geng | Data-driven decision making in power systems with probabilistic guarantees: Theory and applications of chance-constrained optimization[END_REF] for different values of θ and |S|. The higher values of θ indicate stronger dependence between the random constraints driven by Gumbel-Hougaard copula. The numerical results are summarized in Tables 2 and3 for the case of discounted and average cost criteria, respectively. In the average cost criterion, for all states except s = 0, 1, the corresponding ρ becomes significantly low, such that it has a negligible effect on the optimal value. Thus, the optimal value remains almost the same with the number of states. We define the gap between the lower and upper bounds for the optimal value of the JCCMDP problem (4) by From Tables 2 and3, we observe that the gap ranges from 0 to 1.3565. The CPU time analysis shows that the SOCP problems ( 9) and ( 10) can be solved efficiently for our example. and denote the corresponding states as (s i )

|B f | i=1 .
To fix the non-zero probability values for these states, we randomly generate |B f | -1 values in the interval (0, 1) using the function sort(rand(|B f |-1, 1)) and denote them as (q i )

|B f |-1 i=1
. We take q 0 = 0 and q |B f | = 1. Following the rule given in Section 4.1 of [START_REF] El Asri | Score-based inverse reinforcement learning[END_REF], the transition probabilities are given by p(s ′ |s, a) = ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ q iq i-1 if s ′ = s i , i = 1, 2, . . . , |B f |, 0 otherwise.

For simplicity, we generate only irreducible transition probability matrices. This ensures that Assumption 1 is trivially satisfied. We fix θ = 6, |S| = 500, and |A|, K = 10. Thus, ĉ ∼ N (µ ĉ, Σ ĉ), dk ∼ N (µ dk , Σ dk ), k = 1, 2, . . . , 10 are random vectors of dimension 5000 × 1. We randomly generate µ ĉ in (30, 200), µ dk in (30, 150), and ξ k in (75, 120) for k = 1, 2, . . . , 10 using the function rand.

We generate the square roots of Σ ĉ, Σ dk , k = 1, 2, . . . , 10, each of dimension 5000×5000 using the function sprandsym(5000, 0.0005, 3rand(5000, 1)). The generated matrices are symmetric with density 0.0005 and randomly generated eigenvalues in the interval (0, 3). We consider two cases: (i) |B f | = 100 (20% of the states are reachable from each state-action pair), (ii) |B f | = 400 (80% of the states are reachable from each state-action pair). We solve the SOCP problems ( 9) and [START_REF] Geng | Data-driven decision making in power systems with probabilistic guarantees: Theory and applications of chance-constrained optimization[END_REF], and summarize the results in Tables 4 and5 for the case of discounted and average cost criteria, respectively. In both the tables, the average gap between the lower and upper bounds reduces as N increases, irrespective of |B f |. Furthermore, the average gaps for |B f | = 100 are larger than the corresponding gaps for |B f | = 400, while the gaps obtained for the discounted cost criterion are smaller than the corresponding gaps for the average cost criterion. For the data we generate, the gap ranges from 1.5155 to 12.9035. This indicates that gaps obtained are data-dependent. The average CPU time to solve the problem [START_REF] Geng | Data-driven decision making in power systems with probabilistic guarantees: Theory and applications of chance-constrained optimization[END_REF] is less than [START_REF] Fang | Symmetric Multivariate and Related Distributions[END_REF], possibly since the latter problem has K additional second order constraints. Furthermore, on average it takes more time to solve the problems with |B f | = 400 than with |B f | = 100 and more time to solve the discounted cost criterion than its average cost criterion counterpart. However, we conclude that both our approximations can be solved efficiently for both the cost criteria.

Conclusion

We consider a JCCMDP problem under discounted and average cost criteria. The dependence among the random cost constraint vectors is driven by a Gumbel-Hougaard copula. When the random cost vectors follow multivariate elliptically symmetric distributions, we propose SOCP based approximations which give lower and upper bounds on the optimal value of the JCCMDP problem. We illustrate our results on queueing control problem and on a class of randomly generated CMDP problems by considering various small and large instances. The gap between the lower and upper bounds shows the effectiveness of the SOCP approximations. The CPU time analysis shows that both the approximations can be solved efficiently for large instances.

  (vi) c = (c(s, a)) (s,a)∈K denotes the running cost vector associated with the objective function, where c(s, a) is the cost incurred at state s when an action a ∈ A(s) is chosen. (vii) d k = (d k (s, a)) (s,a)∈K , k = 1, 2, . . . , K, denote the running cost vectors associated with the constraints, where d k (s, a) is the cost incurred at state s when an action a

  Define a vector ρ such that ρ(s, a) = gv(γ, f ; s, a) for all (s, a) ∈ K. It follows from Theorems 3.2 and 4.2 of [1] that ρ ∈ Qv(γ) and Ĉv(γ, f, ω) = ∑ (s,a)∈K ρ(s, a)ĉ(s, a, ω) and Dk v (γ, f, ω) = ∑ (s,a)∈K ρ(s, a) dk (s, a, ω), k = 1, 2, . . . , K. Hence, (t, ρ) is a feasible point of (4). Conversely, if (t, ρ) is a feasible solution of (4), then there exists f ∈ F S defined as f (s, a) = ρ(s,a) ∑ a∈A(s) ρ(s,a) , whenever the denominator is nonzero (if it is zero, we choose f (s) arbitrarily from ℘(A(s))). For such an f , we have Ĉv(γ, f, ω) = ∑ (s,a)∈K ρ(s, a)ĉ(s, a, ω) and Dk v (γ, f, ω) = ∑ (s,a)∈K ρ(s, a) dk (s, a, ω), k = 1, 2, . . . , K.

ȳ 1 θ k 1

 1 , i.e., ȳk = (ln P(ρ T dk ≤ξ k ) ln(p1)

Gap

  is a composition of functions p

	1
	θ k y
	1
	and Φ -1 dk (y k ), where former is a convex function of y k ∈ [0, 1] and latter is non-
	decreasing and convex function of y k ∈ [p 1 , 1]. The convexity of Φ -1 dk (y k ) for
	the probability distributions listed in Table 1 can be verified by checking the
	following condition for all k,

  1, 2, . . . , n -1) T , the matrix Σ ĉ ∈ R n×n with all diagonal values 0.9 and off-diagonal values 0.35, µ d1 = (4.32, 9.1875, 10.83) T , µ d2 = (10.00, 8.50, 7.60) T , and

	Σ d1 =	⎛ ⎜ ⎝ 0.15 0.05 0.10 0.05 0.10 0.15 0.10 0.15 0.40	⎞ ⎟ ⎠	, Σ d2 =	0.80 0.35 0.24 ⎛ 0.35 0.70 0.20 ⎜ ⎝ 0.24 0.20 0.61

Table 2

 2 Queueing control problem: Discounted cost criterion.

					N = 5					N = 10			N = 20
	θ States	Lower Bound (LB) Upper Bound (UB)	Gap(%)	Lower Bound (LB) Upper Bound (UB)	Gap(%)	Lower Bound (LB) Upper Bound (UB)	Gap(%)
			Opt.	CPU(s)	Opt.	CPU(s)		Opt.	CPU(s)	Opt.	CPU(s)		Opt.	CPU(s)	Opt.	CPU(s)
		500	177.4043	68.25	178.639 11.8281	0.696	177.4071 75.7813 178.5013 12.9375	0.6168	177.408 97.7344 178.3905 11.9219	0.5538
	1	1000 422.0911 342.1875 427.8169 31.6875	1.3565 422.2185 353.9531 427.266 27.5938	1.1955	422.226 412.1875 427.2463 39.5156	1.189
		1500 671.2592 943.6094 677.8156 56.7656	0.9767 671.4757 877.5938 677.2248 62.8438	0.8562 671.5145 980.75	677.2431 65.8281	0.8531
		2000 920.8955	1908.6 927.8149 85.8281	0.7514 921.2717	2004.6	927.2242 87.7813	0.6461 921.3652	2318.6 927.2425 111.9219 0.6379
		500	177.3957 63.2188 177.3996 9.9063	0.0022 177.3955 69.7656 177.3988 11.125	0.0019 177.3959 83.4531 177.3984 11.4688	0.0014
	3	1000 420.2786 299.2813 423.1495 33.875	0.6831 420.2828 310.9688 423.1099 28.7031	0.6727 420.2915 345.9844 423.0939 32.1719	0.6668
		1500 669.3847 829.7344 673.0587	58.5	0.5489 669.4349 892.6406 672.9414 62.3594	0.5238 669.4366 951.6875 672.9065 72.1094	0.5183
		2000 919.0816	1809.2 923.0578 96.6875	0.4326 919.1713	1930.1	922.9407 110.0625 0.4101 919.1847 2105.8	922.9059 104.6875 0.4048

Table 3

 3 Queueing control problem: Average cost criterion.We generate random finite CMDPs with normally distributed cost vectors to study the efficiency of our proposed methods. Previous literature has conducted experiments on randomly generated finite MDPs known as Garnets[START_REF] Archibald | On the generation of Markov decision processes[END_REF][START_REF] El Asri | Score-based inverse reinforcement learning[END_REF]. We construct our system similar to the description in[START_REF] El Asri | Score-based inverse reinforcement learning[END_REF], using the parameter tuple (|S|, |A|, |B f |), where |S| and |A| are the number of states and actions, respectively, while |B f | is the branching factor that indicates the number of states reachable from a given state-action pair. For each state-action pair (s, a), we draw |B f | states using the function sort(randsample(|S|, |B f |))

					N = 5					N = 10					N = 20
	θ States	Lower Bound (LB) Upper Bound (UB)	Gap(%)	Lower Bound (LB) Upper Bound (UB)	Gap(%)	Lower Bound (LB) Upper Bound (UB)	Gap(%)
			Opt.	CPU(s)	Opt.	CPU(s)		Opt.	CPU(s)	Opt.	CPU(s)		Opt.	CPU(s)	Opt.	CPU(s)
		500	1.5606	73.75	1.5607	16.6094	0.0064 1.5606	77.2188	1.5607	14.7656	0.0064 1.5606	92.7031	1.5607	14.2188	0.0064
	1	1000 1.5606 331.9063 1.5607	33.8594	0.0064 1.5606 353.1406 1.5607	32.7656	0.0064 1.5606 412.4063 1.5607	31.875	0.0064
		1500 1.5606 980.5313 1.5607	71.875	0.0064 1.5606 927.5313 1.5606 103.5938	0	1.5606 975.5781 1.5607	72.125	0.0064
		2000 1.5606	1776.8	1.5606 272.3438	0	1.5605	2195.8	1.5607 119.5313	0.0128 1.5606	2231.2	1.5606	136	0
		500	1.5606	65.2188	1.5607	12.75	0.0064 1.5606	68.7969	1.5607	12.6563	0.0064 1.5606	84.3906	1.5607	11.3125	0.0064
	3	1000 1.5606	294	1.5607	30.5938	0.0064 1.5606 320.2656 1.5607	33.0625	0.0064 1.5606 346.2656 1.5607	31.6406	0.0064
		1500 1.5606 882.6875 1.5607 104.1563	0.0064 1.5606 933.2031 1.5607	88.8125	0.0064 1.5606	941	1.5607	68.625	0.0064
		2000 1.5606	1973.1	1.5606 176.5625	0	1.5605	1948.7	1.5606 124.0781	0.0064 1.5605	2099	1.5606 136.4219	0.0064
	4.2 Random CMDPs

Table 4

 4 Random CMDPs: Discounted cost criterion.

						|Bf | = 100									|Bf | = 400
	Instance		N = 5			N = 10			N = 20			N = 5			N = 10			N = 20
		LB	UB	Gap(%)	LB	UB	Gap(%)	LB	UB	Gap(%)	LB	UB	Gap(%)	LB	UB	Gap(%)	LB	UB	Gap(%)
	1	45.6958 46.5055 1.7719 45.6958 46.4991 1.7579	45.6958	46.498	1.7555	45.6793 46.5135 1.8262	45.6793 46.5073 1.8126	45.6793 46.5063 1.8104
	2	46.9667 48.8036 3.9111 46.9667 48.8026 3.9089	46.9668	48.7993 3.9017	47.0762 48.9294 3.9366	47.0762 48.928	3.9336	47.0763 48.9278	3.933

Acknowledgement

The research of first author was supported by CSIR, India. The research of second and third author was supported by DST/CEFIPRA Project No. IFC/4117/DST-CNRS-5th call/2017-18/2 and CNRS Project No. AR/SB:2018-07-440, respectively.

The last two rows of the table give the average CPU time taken to solve the SOCP problems (against the columns with optimal values) and the average gap between the lower and upper bounds (against the columns with Gap(%)).

Conflict of Interest

The authors declare that there is no conflict of interest. The last two rows of the table give the average CPU time taken to solve the SOCP problems (against the columns with optimal values) and the average gap between the lower and upper bounds (against the columns with Gap(%)).

Letters, 325-328 (2012)

[6] Cheng, J., Houda, M., Lisser, A.: Chance constrained 0-1 quadratic programs using copulas. Optimization Letters 9(7), 1283-1295 (2015)

for the Markov decision problem with a fixed discount rate. Mathematics of Operations Research 36(4), 593-603 (2011)


Celia Marin-Collado

Gael Huet

Anir Benihya

Emmanuelle Gastaldi

Florian Monlau

Cecilia Sambuisti

Sandra Domenek

Chemical pre-treatment to improve anaerobic digestion of polylactic acid packaging

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Contact:

Sandra Domenek: sandra.domenek@agroparistech.fr Celia Marín Collado: celia.marincollado@agroparistech.fr

Polylactic acid (PLA) is a promising bio-based alternative to conventional petrol-based plastic packaging. Despite its biodegradable qualities in industrial composting, PLA biodegradability remains low in mesophilic anaerobic digestion (1,2). In anticipation of the selective collection of biowastes, efforts are being made to foster and accelerate PLA food packaging digestibility together with biowaste by applying pretreatments strategies (2). Among the different types of pre-treatments, thermo-alkaline methods seemed to be the most promising (1).

This study aims to better understand the impact of PLA's morphology and geometry during thermo-alkaline hydrolysis, to gain knowledge on PLA kinetic biodegradation and solubilization (2). For this purpose, several PLA available in the market has been selected and are described below.

( Overall, this poster emphasizes the importance of effective end-of-life management of PLA packaging and presents a novel approach to enhancing its biodegradability. The thermo-alkaline hydrolysis pre-treatment offers a promising solution for meeting industrial anaerobic digestion times and fostering PLA degradation for methane production. The findings of this study contribute to the development of sustainable waste management practices and support the transition to a circular economy. A decrease in pH is observed with an increase in pretreatment duration due to the PLA hydrolysis.

The crystallinity degree of all the samples increased with the thermo-alkaline pre-treatment duration. The initial increase of the amorphous samples was faster.

The solubility percentage of all the samples increased with the pre-treatment duration. The different geometry (PLA 1.1 and PLA 1.2) seems to have no impact.

Thermograms show values for melting temperature (T m ) and the enthalpy of crystallization. The enthalpy increased with the pre-treatment duration, the crystallinity degree increases over time and so does the melting temperature (T m ) of the material.

  a , Gaël Huet b , Anir Benihya a , Emmanuelle Gastaldi b , Florian Monlau c , Cecilia Sambusiti d , Sandra Domenek a a UMR SayFood (AgroParisTech/INRAE/Université Paris Saclay), Palaiseau, France. b UMR IATE (Univ Montpellier/INRAE), Montpellier, France. c TotalEnergies, PERL -Pôle DʹEtudes et de Recherche de Lacq, Pôle Economique 2, BP 47 -RD 817, 64170 Lacq, France d TotalEnergies, CSTJF -Avenue Larribau, 64018 Pau, France

  1) Guillaume Cazadehore. Thèse de doctorat. Méthanisation des plastiques biodégradables : performance et diversité microbienne. Université de Pau et Pays de l'Adour, 11/03/2022.(2) Cazaudehore, G., Guyoneaud, R., Vasmara, C., Greuet, P., Gastaldi, E., Marchetti, R., Leonardi, F., Turon, R., & Monlau, F. (2022). Impact of mechanical and thermo-chemical pretreatments to enhance anaerobic digestion of poly(lactic acid). Chemosphere, 297, 133986. https://doi.org/10.1016/j.chemosphere.2022.133986

  Thermo-alkaline pretreatments was performed on the various PLA polymers (amorphous vs semicrystalline, and pellet vs film). KOH was purchased at Sigma Aldrich and used at 20 % w/v (3.57 M equivalents, pH of 13.5). Polymers were hydrolyzed at a solid loading of 50g/L, 20 % w/v KOH, at 80 ºC and 100 rpm stirring in a water bath for 24 h, 48 h and 72 h respectively. The following analysis were performed: Differential San Calorimetry (DSC): program heating-cooling-heating between -20 ºC and 200 ºC, at a rate of 10 ºC/min in a Mettler Toledo DSC (DSC 1, USA) under N₂ 50 ml/min. Results were analyzed using STARe software.