Chenyang Liu

Jian Cao
email: cao-jian@cs.sjtu.edu.cn

Frédéric Le Mouël
email: frederic.le-mouel@insa-lyon.fr

A Low-Latency Service Composition Approach in Mobile Ad Hoc Networks

Keywords: C.2.2 [Computer-Communication Networks]: Network Protocol Algorithms, Performance, Design, Experimentation MANET Service, Service Composition Path, Path Filtering, Path Combination

In order to offer complex services to the users, separate services located at different devices in MANET should be composed in a mobile ad hoc network. A distributed approach to search for the Service Composition Path (SCP) with a low latency is proposed, which is based on two methods, Path Filtering and Path Combination. These two methods avoid unnecessary message transmissions, and greatly improve the searching efficiency. The experiment results show the superiorities of the approach to its counterpart.

INTRODUCTION

In recent years, service composition in pervasive environments [START_REF] Zhou | Context-Aware Pervasive Service Composition[END_REF][START_REF] Ibrahim | A Survey on Service Composition Middleware in Pervasive Environments[END_REF][3][4][5][6], especially in MANETs, has attracted more and more research interests. A MANET consists of multiple stand-alone and self-configurable computing nodes, which use a multi-hop routing mechanism, such as DSR and AODV, to communicate with each other. With the emergence of various intelligent devices, the nodes in a MANET will be equipped with richer hardware and software resources than before. These resources pave the way for constructing micro-service frameworks on MANET nodes to facilitate cross-device service discovery and invocation, which lays the foundation for service composition in the MANET.

Many researchers proposed their frameworks for service composition in the MANET [START_REF] Chakraborty | Service Composition for Mobile Environments[END_REF][START_REF] Choudhury | Mobility Aware Distributed Service Composition Framework in SOA based MANET Application[END_REF][START_REF] Wang | Exploiting Mobility Prediction for Dependable Service Composition in Wireless Mobile Ad Hoc Networks[END_REF][START_REF] Karmouch | Capability Reconciliation for a CSP Approach to Virtual Device Composition[END_REF]. Their approaches can be roughly divided into two categories. In the first category, for each function, a path from the composite service-requesting node to the service-providing node corresponding to this function should be found. Obviously, this approach suffers from low efficiency for service execution, because of the messages transmitted back and forth on the path between the composite service-requesting node and the service-providing node. In another category a composite service is represented as an executable path, in which previous nodes provide services and intermediate results will be transmitted to successive ones through multi-hop transmission. Our approach belongs to this category.

Since there often exist more than one composition solution capable of fulfilling a request, searching for the optimal one in a MANET will lead to packet flooding problem. In order to obtain the optimal solution more efficiently, an approach is proposed in this paper and it has following features:

1. Making use of Path Filtering to discard non-optimal composition solutions to avoid unnecessary message transmissions; 2. Searching for the Forward and Backward partial composition paths at the same time, and then applying Path Combination to obtain the complete solution more efficiently.

The contribution of our approach is improving the quality of the composition solution evidently, and reducing the time and communication cost greatly.

RELATED WORK

In recent years, many researchers proposed their own architectures for service composition in pervasive environment. The approaches proposed in [START_REF] Zhou | Context-Aware Pervasive Service Composition[END_REF][START_REF] Ibrahim | A Survey on Service Composition Middleware in Pervasive Environments[END_REF] are based on service model and user context information, while [3-4] dwell on reliability and availability modeling of service composition through probabilistic analysis on a MANET nodes' history data. In [5-6], parameterbased service matching approaches are proposed in the pervasive service environment, which employs Semantics Networks and Ontology Learning to match two services.

Confronted constantly changing network topology and resource limitations in the MANET environment, some researchers have proposed a mechanism, called Broker Arbitration, which selects the best broker to be responsible for service composition in the MANET [START_REF] Chakraborty | Service Composition for Mobile Environments[END_REF][START_REF] Choudhury | Mobility Aware Distributed Service Composition Framework in SOA based MANET Application[END_REF]. However, Broker-based service composition will lighten heavy burdens on some nodes with poor resources, and may incur message flooding for updating the cached service information. This shortcoming is overcome by the mechanisms suggested in [START_REF] Wang | Exploiting Mobility Prediction for Dependable Service Composition in Wireless Mobile Ad Hoc Networks[END_REF], which apply Group-based Service Discovery (GSD) [START_REF] Karmouch | Capability Reconciliation for a CSP Approach to Virtual Device Composition[END_REF], to search for candidate atomic services.

In fact, a service composition solution can also be described as a sequential workflow. Based on this idea, [START_REF] Sun | Decentralized Execution of Composite Service in MANET[END_REF][START_REF] Chen | A Location-Based Execution Path Selection for Composite Service in MANETs[END_REF][START_REF] Zhou | A Backup Restoration Algorithm of Service Composition in MANETs[END_REF] present algorithms for obtaining the optimal composition workflow with minimum execution time. These algorithms forward the composition request among the nodes, and regard the first complete workflow as the optimal solution. However, due to the lack of limitation on message forwarding, these algorithms will suffer from packet flooding in MANET. Contrast to these methods, our approach introduces Path Filtering, which limits the message forwarding effectively, and Path Combination, which saves an amount of time while obtaining the optimal solution.

PROBLEM FORMATION OF SERVICE COMPOSITION IN THE MANET

In our assumption, every function in the MANET is modeled as a 4-element tuple, <N i , S j , F k , D j >, which represents that service component S j is deployed on MANET node N i , and implements the function F k . D j is the service execution time. While we employ a 3-element tuple, <N i , N j , D ij > represents a bidirectional link between node N i and node N j , whose network communication delay is D ij . Based on these definitions, Service Composition Path (SCP) is described as a sequential loop path, starting and ending at the requesting node, and consisting of all required service nodes and relay nodes and their corresponding communication links. Hence, the total path delay can be computed according to Formula (1):

T i path s N N i j D D   (1)
where

i s D is the D i in <S i , N j , F k , D i >, and i N Nj D is the D ij in <N i , N j , D ij >.

Fig. 1. An example of all SCPs for a 5-function-composition

For every sub-function in a service composition request, we can find several corresponding service components in the MANET so that there exist multiple SCPs satisfying the same composition request. For example in Fig. 1, there is a functional graph that consists of 5 function nodes, and each function can be fulfilled by several services interchangeably. Our problem is to find the latency-optimal SCP in the MANET with few time and network messages as possible. This problem can be abstracted as a Distributed Multi-stage Decision Problem, which means the resolution is based on the collaboration among the MANET nodes.

FORWARD SEARCH AND PATH FILTERING

ForeNoRepSearch algorithm continues forwarding and updating the partial SCP unless it arrives at the requesting node or is discarded. In the searching process, this algorithm searches for a service for each sub-function in sequence. Any node receiving a request message will query the local system for current required service, update the partial SCP, and apply Path Filtering to decide whether to continue forwarding this path or not. When the requesting node receives a SCP who implements all the required sub-functions, it will compare the SCP with current optimal one to obtain a better solution. The algorithm will continue running until there is no request message on any node, which means the search process has converged to an optimal solution. The algorithm of ForeNoRepSearch on one node is shown below. Path Filtering is proposed to judge whether current partial SCP is worth forwarding or not, whose principle is based on the concept of Optimal Substructure in Dynamic Programming. If current partial SCP is included in the optimal SCP, there is no other partial SCP including current node with the same number of services and less path delay. To implement Path Filtering, every node will establish a SCP list to store local delay-optimal partial SCPs. When a node receives a partial SCP, it will decide whether to discard, or forward the partial SCP and delete the worse ones in the list. The pseudo-code of Path Filtering is listed as following:

PATH-FILTER (scCurrPath: SCP) {scCurrPath is the current partial SCP waiting for filtering} begin For each scListPath in SCP list if scListPath has more required services than scCurrPath && its path delay is no more than scCurrPath then discard scCurrPath and return; For each scListPath in SCP list if scListPath has no more required services than scCurrPath && its path delay is more than scCurrPath then delete scListPath from SCP list; Insert scCurrPath into SCP list; end.

FORWARD-BACKWARD SEARCH AND PATH COMBINATION

ForeBackRepSearch Algorithm queries for two required services to update the forward and backward partial SCPs respectively in the request message, and also apply Path Filtering to reduce network messages. At the same time, the current node tries to get a new complete SCP through Path Combination. If the new formed SCP has lower path delay than the current optimal one, it will be sent back to the requesting node. The following codes give out the logic of ForeBackRepSearch algorithm:

Algorithm 2 ForeBackRepSearch

HANDLE-SERV-COMP-REQ (reqMsg: SERV_COMP_REQ) var

currOptSCP: optimal SCP stored on current node; combSCP: current SCP through Path Combination; begin When a node gets a new forward partial SCP after Path Filtering, and there exists a backward partial one in the SCP list on current node, which has the rest required services that the forward one has not discovered, we can combine the two partial SCPs to constitute a complete one. Similarly we can apply these operations for a backward one. The following pseudo-code only describes the method for a forward partial SCP to combine the backward ones in the SCP list:

F 1 F 2 F 3 F 4 F5 S 1 -1 S 1 -2 S 1 -3 S 2 -1 S 2 -2 S 2 -3 S 3 -1 S 3 -2 S 4 1 - S 4 2 - S 4 3 - S5-1 S5-2 S 4
FORE-COMBINE-

EXPERIMENT EVALUATION

We wrote a Java-based platform to simulate MANET service environment. Given the number of nodes, services, and functions, this platform can generate a virtual MANET, simulate all the services that implements different functions, and deploy these services uniformly and randomly on different nodes. As a test-bed, each node was modeled as a Java Thread, and we applied Java Message Queue mechanism to simulate message transmissions among the nodes. In this simulation environment, we implemented the above three service composition algorithms.

We conducted three groups of experiments. In each group, we ran the three algorithms in different environment generated by changing one of the following three configuration parameters: In the first group of experiment, we randomly generated 40 MANET nodes, created 90 services which implemented 20 different functions, and deployed these services uniformly onto the nodes. We generated 10 service composition requests for each number of required services, from 3 to 9. For each request, we executed the above three algorithms, and utilized the mean value to measure the performance of these algorithms, which is shown in Fig. 3 Similarly, in the third group of experiment, we fixed the number of MANET nodes and functions respectively as 30 and 20, made the number of MANET services increase from 55 to 90, and ran three algorithms for different randomly generated 5-functioncomposition requests. The curve graphs of the results are omitted here due to the length limitation of the paper.

CONCLUSIONS

In this paper, we define the MANET service composition as a problem of searching for the optimal Service Composition Path (SCP) in the network, and propose a distributed approach based on two algorithms, ForeNoRepSearch and ForeBackRepSearch. ForeNoRepSearch introduces Path Filtering to impose a constraint on message forwarding, and continue forwarding partial SCPs after finding all the required services, which improves the quality of the result SCP remarkably comparing with ForeRepSearch. On the basis of Path Filtering, ForeBackRepSearch applies Path Combination to obtain complete SCPs from combining forward and backward partial SCPs, which enhances the efficiency of searching process significantly. In our experiments, we made comparative assessment on our two algorithms and ForeRepSearch, and the results prove that our algorithms can obtain better SCPs with evidently fewer time and network messages than ForeRepSearch under different environment configuration.

  i)Number of required services; ii)Number of MANET nodes; iii) Number of MANET services. For each configuration, we generate 10 different requests randomly and evaluate the three algorithms in terms of the following metrics: i)Total time needed for obtaining the solution; ii)Total number of network messages; iii)Number of nodes in the result SCP; ⅳ)Path delay of the result SCP.

  (a)~(d).

Fig. 4 (

 4 Fig. 1(a~b). Time, Messages vs. Required Services

[ 3 ]

 3 Brent Lagesse, Mohan Kumar, Matthew Wright. ReSCo: A Middleware Component for Reliable-Service Composition in Pervasive Systems. Proceedings of 8th IEEE International Conference on Pervasive Computing and Communications Workshops, Pages 486 -491, Mannheim, DE, Mar. 29 -Apr. 2Jianping Wang. Exploiting Mobility Prediction for Dependable Service Composition in Wireless Mobile Ad Hoc Networks. IEEE Transactions for Services Computing, Volume 4, Issue 1, Pages 44-55, January-March 2011. [5] Noha Ibrahim, Stéfane Frénot, Frédéric Le Mouël. User-Excentric Service Composition in Pervasive Environment. Proceedings of the 24th IEEE International Conference on Advanced Information Networking and Applications. Pages 682 -689, Perth, Australia, Apr. 20-23, 2010. Fig. 4(c~d)Number of Nodes, Path Delay vs. MANET Nodes [6] Swaroop Kalasapur, Mohan Kumar, Behrooz A. Shirazi. Dynamic Service Composition in Pervasive Computing. IEEE Transactions on Parallel and Distributed Systems, Volume 18, Issue 7, Pages 907-918, July 2007.


Bernt Øksendal
email: oksendal@math.uio.no

Agnès Sulem
email: agnes.sulem@inria.fr

Tusheng Zhang
email: tusheng.zhang@manchester.ac.uk

Singular control and optimal stopping of SPDEs, and backward SPDEs with reflection

Keywords: Stochastic partial differential equations (SPDEs), singular control of SPDEs, maximum principles, comparison theorem for SPDEs, reflected SPDEs, optimal stopping of SPDEs. MSC2000 subject classification : Primary: 60H15, Secondary: 93E20, 35R60 OR/MS subject classification : Primary: Probability theory and stochastic processes For additional applications:Stochastic partial differential equations, secondary: Systems theory, control For optimal control: Stochastic partial differential equations, Partial differential equations: Partial differential equations with randomness, stochastic partial differential equations

We consider general singular control problems for random fields given by a stochastic partial differential equation (SPDE). We show that under some conditions the optimal singular control can be identified with the solution of a coupled system of SPDE and a reflected backward SPDE (RBSPDE). As an illustration we apply the result to a singular optimal harvesting problem from a population whose density is modeled as a stochastic reaction-diffusion equation. Existence and uniqueness of solutions of RBSPDEs are established, as well as a comparison theorems. We then establish a relation between RBSPDEs and optimal stopping of SPDEs, and apply the result to a risk minimizing stopping problem.

1. Introduction As a motivation for the problem studied here we consider a problem of optimal harvesting from a fish population in a lake D. Suppose the density Y (t, x) of the population at time t ∈ [0, T ] and at the point x is given by a stochastic reaction-diffusion equation of the form dY (t, x) = [∆Y (t, x) + αY (t, x)]dt + Y (t, x)βdB(t)λ 0 ξ(dt, x); (t, x) ∈ (0, T ) × D Y (0 -, x) = y 0 (x) > 0; x ∈ D Y (t, x) = 0; (t, x) ∈ (0, T ) × ∂D, (1.1) where D is a bounded domain in R d and y 0 (x) is a given bounded deterministic function. Here B(t) = B t , t ≥ 0 is an m-dimensional Brownian motion on a filtered probability space (Ω, F, F t , P ),

Mathematics of Operations Research 00(0), pp. 000-000, c 2013 INFORMS α, λ 0 > 0 are given constants, β is a given m-dimensional vector and ∆ :=

d i=1 ∂ 2 ∂x 2 i
is the Laplacian differential operator. We may regard ξ(dt, x) as the harvesting effort rate and λ 0 > 0 as the harvesting efficiency coefficient. The performance coefficient is assumed to be

J (ξ) = E[ D T 0 h 0 (t, x)Y (t, x)ξ(dt, x)dx + D h 0 (T, x)Y (T, x)dx], (1.2)
where h 0 (t, x) > 0 is the net unit price of the fish and T > 0 is a fixed terminal time. Thus J (ξ) represents the expected total net income from the harvesting. The problem is to maximize J (ξ) over all admissible harvesting strategies ξ(t, x). We say that ξ is admissible and write ξ ∈ A if ξ(t, x) is F t -adapted, non-decreasing in t and ξ(0, x) = 0 for each x. In this example we also require that Y (t, x) ≥ 0 for all (t, x) ∈ [0, T ] × D. This optimal harvesting problem is a special case of a general singular control problem of stochastic partial differential equations (SPDE) driven by a multiplicative noise of finite dimension. The aim of this paper is to study these problems. In particular, we want to establish stochastic maximum principles providing optimality conditions, and to study relations with some associated reflected backward SPDEs.

It is well-known that the stochastic maximum principle method for solving a stochastic control problem for SPDEs involves a backward SPDE for the adjoint processes p(t, x), q(t, x) (see [START_REF] Øksendal | Backward stochastic partial differential equations with jumps and application to optimal control of random jump fields[END_REF]). We will show that in the case of singular control problem for SPDE we arrive at a BSPDE with reflection for the adjoint processes.

Several papers are devoted to the study of backward SPDEs (without reflection) and maximum principles of SPDEs, see e.g. [START_REF] Bensoussan | Stochastic maximum principle for distributed parameter systems[END_REF][START_REF] Hu | Adapted solution of a backward semilinear stochastic evolution equation[END_REF][START_REF] Hu | Maximum principle for semilinear stochastic evolution control systems[END_REF][START_REF] Hu | On semi-linear degenerate backward stochastic partial differential equations[END_REF][START_REF] Guatteri | On the existence of optimal controls for spdes with boundary-noise and boundary-control[END_REF]. In a finite dimensional setup, maximum principles for singular stochastic control problems have been studied in [START_REF] Andersson | The relaxed general maximum principle for singular optimal control of diffusions[END_REF][START_REF] Bahlali | A general stochastic maximum principle for singular control problems[END_REF][START_REF] Bahlali | The relaxed stochastic maximum principles in singular optimal control of diffusions[END_REF][START_REF] Bahlali | Optimality necessary conditions in singular stochastic control problems with non-smooth coefficients[END_REF], and in the recent paper [START_REF] Øksendal | Singular stochastic control and optimal stopping with partial information of jump diffusions[END_REF], where connections between singular stochastic control, reflected BSDEs under partial information are also established. For the study of SPDEs with reflection, we refer to [START_REF] Donati-Martin | White noise driven spdes with reflection[END_REF], [START_REF] Haussmann | Stochastic variational inequalities of parabolic type[END_REF], [START_REF] Nualart | White noise driven by quasilinear spdes with reflection[END_REF], [START_REF] Zhang | White noise driven spdes with reflection: strong feller properties and harnack inequalities[END_REF].

The paper is organized as follows: In Section 2, we study a class of singular control problems for SPDEs and prove a maximum principle for the solution of such problems. This maximum principle leads to an adjoint equation which is a reflected backward stochastic partial differential equation. Both the necessary and sufficient properties of the maximum principle are discussed and, similarly to the finite dimensional case, the sufficient condition is established under suitable concavity properties of the coefficients.

As an illustration, at the end of Section 2 we apply the result to the singular optimal harvesting problem above. In Section 3, we study existence and uniqueness of solutions of backward stochastic partial differential equations (BSPDEs) with reflection and in Section 4 we establish comparison theorems for BSPDEs and reflected BSPDEs. In Section 5, we establish connections between reflected BSPDEs and optimal stopping of SPDEs and in Section 6 we consider an application to a risk minimizing stopping problem in a market with mean-field interactions.

Singular control of SPDEs

Let D be a regular domain in R d . Denote by a(x) = (a ij (x)) a matrix-valued function on R d satisfying the uniform ellipticity condition:

1 c I d ≤ a(x) ≤ cI d for some constant c ∈ (0, ∞).
Let b(x) be a vector field on D with b ∈ L p (D) for some p > d and q(x) a measurable real valued function on D such that q ∈ L p 1 (D) for some p 1 > d 2 . Introduce the following second order partial differential operator:

Au(x) = -div(a(x)∇u(x)) + b(x) • ∇u(x) + q(x)u(x).
Suppose the state equation is an SPDE of the form

dY (t, x) = {AY (t, x) + b(t, x, Y (t, x))}dt + σ(t, x, Y (t, x))dB(t) + λ(t, x, Y (t, x))ξ(dt, x) ; (t, x) ∈ [0, T ] × D (2.1) Y (0 -, x) = y 0 (x) ; x ∈ D Y (t, x) = 0 ; (t, x) ∈ (0, T ) × ∂D. (2.2)
Here y 0 ∈ K := L 2 (D) and y 1 ∈ L 2 ([0, T ] × D) are given functions. We assume that b, σ and λ are C 1 with respect to y. Let V = W 1,2 0 (D) be the Sobolev space of order one with zero boundary condition. Then Y is understood as a weak (variational) solution to (2.1), in the sense that

Y ∈ C([0, T ]; K) ∩ L 2 ([0, T ]; V ) and for φ ∈ C ∞ 0 (D), < Y (t, •), φ > K = < y 0 (•), φ > K + t 0 < Y (s, •), A * φ > ds + t 0 < b(s, •, Y (s, •)), φ > K ds + t 0 < σ(s, •, Y (s, •)), φ > K dB(s), (2.3)
where A * is the adjoint operator of A, and <, > denotes the dual pairing between the space V and its dual V * . Under this framework the It formula can be applied to such SPDEs. See [START_REF] Pardoux | Stochastic partial differential equations and filtering of diffusion[END_REF], [START_REF] Prévôt | A concise course on stochastic partial differential equations[END_REF]. The performance functional is given by

J (ξ) = E D T 0 f (t, x, Y (t, x))dtdx + D g(x, Y (T, x))dx + D T 0 h(t, x, Y (t, x))ξ(dt, x)dx , (2.4)
where f (t, x, y), g(x, y) and h(t, x, y) are bounded measurable functions which are differentiable in the argument y and continuous w.r.t. t. We want to maximize J (ξ) over all ξ ∈ A, where A is the set of all adapted processes ξ(t, x), which are non-decreasing and left-continuous w.r.t. t for all x, ξ(0, x) = 0, ξ(T, x) < ∞ and such that the performance functional is finite. We call A the set of admissible singular controls. Thus we want to find ξ * ∈ A (called an optimal control) such that sup ξ∈A J (ξ) = J (ξ * ).

We study this problem by using an extension to SPDEs of the maximum principle in [START_REF] Øksendal | Singular stochastic control and optimal stopping with partial information of jump diffusions[END_REF]: Define the Hamiltonian H by

H(t, x, y, p, q)(dt, ξ(dt, x)) = {f (t, x, y) + b(t, x, y)p + σ(t, x, y)q}dt + {λ(t, x, y)p + h(t, x, y)}ξ(dt, x). (2.5)
To this Hamiltonian we associate the following backward SPDE (BSPDE) in the unknown process (p(t, x), q(t, x)):

dp(t, x) = -A * p(t, x)dt + ∂H ∂y (t, x, Y (t, x), p(t, x), q(t, x))(dt, ξ(dt, x)) + q(t, x)dB(t) ; (t, x) ∈ (0, T ) × D (2.6) p(T, x) = ∂g ∂y (x, Y (T, x)) ; x ∈ D (2.7) p(t, x) = 0 ; (t, x) ∈ (0, T ) × ∂D. (2.8)
Here A * denotes the adjoint of the operator A. We assume that a unique solution p(t, x), q(t, x) of (2.6)-(2.8) exists for each ξ ∈ A.

Mathematics of Operations Research 00(0), pp. 000-000, c 2013 INFORMS Theorem 1 (Sufficient maximum principle for singular control of SPDE). Let ξ ∈ A with corresponding solutions Ŷ (t, x), p(t, x), q(t, x). Assume that y → h(x, y) is concave, (2.9) (y, ξ) → H(t, x, y, p(t, x), q(t, x))(dt, ξ(dt, x)) is concave,

(2.10)

E[ D ( T 0 {(Y ξ (t, x) -Ŷ (t, x)) 2 q2 (t, x) + p2 (t, x)(σ(t, x, Y ξ (t, x)) -σ(t, x, Ŷ (t, x))) 2 }dt)dx] < ∞,
for all ξ ∈ A.

(2.11)

Moreover, assume that the following maximum condition holds:

ξ(dt, x) ∈ argmax ξ∈A H(t, x, Ŷ (t, x), p(t, x), q(t, x))(dt, ξ(dt, x)), (2.12)
i.e.

{λ(t, x, Ŷ (t, x))p(t, x) + h(t, x, Ŷ (t, x))}ξ(dt, x) ≤ {λ(t, x, Ŷ (t, x))p(t, x) + h(t, x, Ŷ (t, x))} ξ(dt, x) for all ξ ∈ A. (2.13)
Then ξ is an optimal singular control.

Remark. By saying that (y, ξ) → H(y, ξ) is concave, we mean that for all a ∈ [0, 1] and all (y, ξ), (ȳ, ξ), we have H(a(y, ξ)

+ (1 -a)(ȳ, ξ)) ≥ aH(y, ξ) + (1 -a)H(ȳ, ξ). Since H is C 1 , this is equivalent to H(y, ξ) -H(ȳ, ξ) ≤ ∂H ∂y (ȳ, ξ)(y -ȳ) + ∇ ξ H(ȳ, ξ)(ξ -ξ)
where ∇ ξ H(ȳ, ξ) is the Fréchet derivative of H with respect to ξ and ∇ ξ H(ȳ, ξ)(ξξ) is the result of applying this linear operator to ξ -ξ.

Proof of Theorem 1 Choose ξ ∈ A and put Y = Y ξ . Then by (2.4) we can write

J (ξ) -J ( ξ) = I 1 + I 2 + I 3 , (2.14)
where

I 1 = E T 0 D f (t, x, Y (t, x)) -f (t, x, Ŷ (t, x)) dxdt (2.15)
I 2 = E D g(x, Y (T, x)) -g(x, Ŷ (T, x)) dx (2.16
)

I 3 = E T 0 D h(t, x, Y (t, x))ξ(dt, x) -h(t, x, Ŷ (t, x)) ξ(dt, x) .
(2.17)

By our definition of H we have

I 1 = E T 0 D {H(t, x, Y (t, x), p(t, x), q(t, x))(dt, ξ(dt, x))
-H(t, x, Ŷ (t, x), p(t, x), q(t, x))(dt, ξ(dt, x))

- T 0 D {b(t, x, Y (t, x)) -b(t,
T 0 Ỹ (t, x)A * p(t, x)dtdx = T 0 < Ỹ (t, •), A * p(t, •) > dt, D T 0 p(t, x)A Ỹ (t, x)dtdx = T 0 < p(t, •), A Ỹ (t, •) > dt,
where <, > stands for the dual pairing between the space V = H 1,2 0 (D) and its dual V * . In view of < Ỹ (t, •), A * p(t, •) >=< p(t, •), A Ỹ (t, •) >, combining (2.14)- (2.19) and concavity of H, we have

J (ξ) -J ( ξ) ≤ E D T 0 {H(t, x, Y (t, x), p(t, x), q(t, x))(dt, ξ(dt, x)) -H(t, x, Ŷ (t, x), p(t, x), q(t, x))(dt, ξ(dt, x)) -Ỹ (t, x) ∂H ∂y (t, x, Ŷ , p, q)(dt, ξ(dt, x)) dx ≤ D T 0 ∇ ξ H(t, x, Ŷ (t, x), p(t, x), q(t, x))(ξ(dt, x) -ξ(dt, x))dx = E D T 0 {λ(t, x, Ŷ (t, x))p(t, x) + h(t, x, Ŷ (t, x))}(ξ(dt, x) -ξ(dt, x))dx ≤ 0 by (2.13).
This proves that ξ is optimal.

For ξ ∈ A we let V(ξ) denote the set of adapted processes ζ(t, x) of finite variation w.r.t. t such that there exists δ = δ(ξ) > 0 (possibly depending on ζ) such that ξ + yζ ∈ A for all y ∈ [0, δ].

Proceeding as in [ØS] we prove the following useful result: We may formulate what we have proved as follows:

Lemma 1.
Theorem 2. (Sufficient maximum principle II) Suppose the conditions of Theorem 1 hold. Suppose ξ ∈ A, and that ξ together with its corresponding processes Y ξ (t, x), p ξ (t, x), q ξ (t, x) solve the coupled system consisting of the SPDE (2.1)-(2.2) together with the reflected backward SPDE (RBSPDE) given by

dp ξ (t, x) = -A * p ξ (t, x) + ∂f ∂y (t, x, Y ξ (t, x)) + ∂b ∂y (t, x, Y ξ (t, x))p ξ (t, x) + ∂σ ∂y (t, x, Y ξ (t, x))q ξ (t, x) dt - ∂λ ∂y (t, x, Y ξ (t, x))p ξ (t, x) + ∂h ∂y (t, x, Y ξ (t, x)) ξ(dt, x) + q(t, x)dB(t); (t, x) ∈ [0, T ] × D λ(t, x, Y ξ (t, x))p ξ (t, x) + h(t, x, Y ξ (t, x)) ≤ 0 ; for all t, x, a.s. {λ(t, x, Y ξ (t, x))p ξ (t, x) + h(t, x, Y ξ (t, x))}ξ(dt, x) = 0 ; for all t, x, a.s. p(T, x) = ∂g ∂y (x, Y ξ (T, x)) ; x ∈ D p(t, x) = 0 ; (t, x) ∈ (0, T ) × ∂D.
Then ξ maximizes the performance functional J (ξ).

It is also of interest to have a maximum principle of "necessary type". To this end, we first prove some auxiliary results. Lemma 2. Let ξ(dt, x) ∈ A and choose ζ(dt, x) ∈ V(ξ). Suppose that the derivative process

Y(t, x) = lim y→0 + 1 y (Y ξ+yζ (t, x) -Y ξ (t, x)) (2.26) exists. Then Y satisfies the SPDE dY(t, x) = AY(t, x)dt + Y(t, x)[ ∂b ∂y (t, x, Y (t, x))dt + ∂σ ∂y (t, x, Y (t, x))dB(t) + ∂λ ∂y (t, x, Y (t, x))ξ(dt, x)] + λ(t, x, Y (t, x))ζ(dt, x) ; (t, x) ∈ [0, T ] × D Y(t, x) = 0; (t, x) ∈ (0, T ) × ∂D Y(0, x) = 0 ; x ∈ D (2.27)
Proof. By (2.1), we have:

lim y→0 + 1 y (Y ξ+yζ (t, x) -Y ξ (t, x)) = lim y→0 + 1 y [ t 0 A(Y ξ+yζ -Y ξ )(s, x)ds + t 0 (b(s, x, Y ξ+yζ (s, x)) -b(s, x, Y ξ (s, x)))ds + t 0 (σ(s, x, Y ξ+yζ (s, x)) -σ(s, x, Y ξ (s, x)))dB(s) + t 0 (λ(s, x, Y ξ+yζ (s, x))(ξ(ds, x) + yζ(ds, x)) -λ(s, x, Y ξ (s, x))ξ(ds, x))] = t 0 AY(s, x)ds + t 0 ∂b ∂y (s, x, Y ξ (s, x))Y(s, x)ds + t 0 ∂σ ∂y (s, x, Y ξ (s, x))Y(s, x)dB(s) + t 0 ∂λ ∂y (s, x, Y ξ (s, x))Y(s, x)ξ(ds, x) + t 0 λ(s, x, Y ξ (s, x))ζ(ds, x).
By (2.2), we have

Y(t, x) = 0; (t, x) ∈ (0, T ) × ∂D Y(0, x) = 0 ; x ∈ D
Remark. The existence of the limit in (2.26) is a nontrivial issue and we do not discuss conditions for this in this paper. Here we simply assume that the limit exists. We refer to [P1] for a study about this issue in a related setting.

Lemma 3. Let ξ(dt, x) ∈ A and ζ(dt, x) ∈ V(ξ). Put η = ξ + yζ; y ∈ [0, δ(ξ)]. Assume that E[ D ( T 0 {(Y η (t, x) -Y ξ (t, x)) 2 q 2 (t, x) + p 2 (t, x)(σ(t, x, Y η (t, x)) -σ(t, x, Y ξ (t, x))) 2 }dt)dx] < ∞ for all y ∈ [0, δ(ξ)], (2.28)
where (p(t, x), q(t, x)) is the solution of (2.5)-(2.7) corresponding to Y ξ (t, x). Then

lim y→0 + 1 y (J (ξ + yζ) -J (ξ)) = E[ D ( T 0 {λ(t, x, Y (t, x))p(t, x) + h(t, x, Y (t, x))}ζ(dt, x))dx]. ( 2
lim y→0 + 1 y (J (ξ + yζ) -J (ξ)) = E[ D { T 0 ∂f ∂y (t, x, Y (t, x))Y(t, x)dt + ∂g ∂y (x, Y (T, x))Y(T, x)}dx + D T 0 ∂h ∂y (t, x, Y (t, x))Y(t, x)ξ(dt, x)dx + D T 0 h(t, x, Y (t, x))ζ(dt, x)dx].
(2.30) By (2.5) we obtain

E[ D T 0 ∂f ∂y (t, x, Y (t, x))Y(t, x)dtdx] = E[ D ( T 0 Y(t, x){ ∂H ∂y (dt, ξ(dt, x)) -p(t, x) ∂b ∂y (t, x)dt -q(t, x) ∂σ ∂y (t, x)dt -(p(t, x) ∂λ ∂y (t, x) + ∂h ∂y (t, x))ξ(dt, x)})dx], (2.31)
where we have used the abbreviated notation

∂H ∂y (dt, ξ(dt, x)) = ∂H ∂y (t, x, Y (t, x), p(t, x), q(t, x))(dt, ξ(dt, x)) etc.
By the Itô formula and (2.27) we see that Then ξ is a directional sub-critical point for J (•), in the sense that as claimed. This proves (i).

E[ D ∂g ∂y (x)Y(T, x)dx] = E[ D p(T, x)Y(T, x)dx] = E[ D ( T 0 {p(t, x)dY(t, x) + Y(t, x)dp(t, x)} + [p(•, x), Y(•, x)](T ))dx] = E[ D ( T 0 [p(t,
lim y→0 + 1 y (J ( ξ + yζ) -J ( ξ)) ≤ 0 for all ζ ∈ V( ξ). ( 2
(ii) Conversely, suppose ξ ∈ A is as in (ii). Then (2.38) follows from Lemma 3.

Application to Optimal Harvesting

We now return to the problem of optimal harvesting from a fish population in a lake D stated in the introduction. Thus we suppose the density Y (t, x) of the population at time t ∈ [0, T ] and at the point x ∈ D is given by the stochastic reactiondiffusion equation (1.1), and the performance criterion is assumed to be as in (1.2). In this case the Hamiltonian in (2.5) is

H(t, x, y, p, q)(dt, ξ(dt, x)) = (αyp + βyq)dt + [-λ 0 p + h 0 (t, x)y]ξ(dt, x) (2.46)
and the adjoint equation

(2.6)-(2.8) is dp(t, x) = -[∆p(t, x) + αp(t, x) + βq(t, x)]dt -h 0 (t, x)ξ(dt, x) + q(t, x)dB(t); (t, x) ∈ (0, T ) × D, p(T, x) = h 0 (T, x); x ∈ D p(t, x) = 0; (t, x) ∈ (0, T ) × ∂D. (2.47)
The variational inequalities (2.34)-(2.35) for an optimal control ξ(dt, x) are:

-λ 0 p(t, x) + h 0 (t, x)Y (t, x) ≤ 0; (t, x) ∈ [0, T ] × D, (2.48) [-λ 0 p(t, x) + h 0 (t, x)Y (t, x)]ξ(dt, x) = 0; (t, x) ∈ [0, T ] × D (2.49)
We can rewrite the variational inequalities above as follows:

p(t, x) ≥ h 0 (t, x)Y (t, x) λ 0 ; (t, x) ∈ [0, T ] × D [p(t, x) - h 0 (t, x)Y (t, x) λ 0 ]ξ(dt, x) = 0; (t, x) ∈ [0, T ] × D.
(2.50)

We summarize the above in the following:

Theorem 4. (a) Suppose ξ(dt, x) ∈ A is an optimal singular control for the harvesting problem

sup ξ∈A E[ D T 0 h 0 (t, x)Y (t, x)ξ(dt, x)dx + D h 0 (T, x)Y (T, x)dx] (2.51)
where Y (t, x) is given by the SPDE (1.1). Then ξ(dt, x) solves the reflected BSPDE (2.47), (2.50).

(b) Conversely, suppose ξ(dt, x) is a solution of the reflected BSPDE (2.47), (2.50). Then ξ(dt, x) is a directional sub-critical point for the performance J (•) given by (1.2).

Heuristically we can interpret the optimal harvesting strategy as follows:

• As long as p(t, x) > h 0 (t,x)Y (t,x) λ 0 , we do nothing

• If p(t, x) = h 0 (t,x)Y (t,x) λ 0
, we harvest immediately from Y (t, x) at a rate ξ(dt, x) which is exactly enough to prevent p(t, x) from dropping below h 0 (t,x)Y (t,x) λ 0 in the next moment.

• If p(0, x) < h 0 (0,x)Y (0 -,x) λ 0 , we harvest immediately what is necessary to bring h 0 (0,x)Y (0,x) λ 0 down to the level of p(0, x).

Existence and uniqueness results of reflected backward SPDEs

In this section we prove existence and uniqueness results for reflected backward stochastic partial differential equations. For notational simplicity, we choose the operator A to be the Laplacian operator ∆. However, our methods work equally well for general second order differential operators like

A = d i,j=1 ∂ ∂x i (a ij (x) ∂ ∂x j ),
where a = (a ij (x)) :

D → R d×d (d ≥ 2
) is a measurable, symmetric matrix-valued function which satisfies the uniform ellipticity condition

λ|z| 2 ≤ d i,j=1 a ij (x)z i z j ≤ Λ|z| 2 , ∀ z ∈ R d and x ∈ D
for some constants λ, Λ > 0. Let V = W 1,2 0 (D) be the Sobolev space of order one with the usual norm || • ||. As before let K = L 2 (D). Consider the reflected backward stochastic partial differential equation:

du(t, x) = -∆u(t, x)dt -b(t, u(t, x), Z(t, x))dt + Z(t, x)dB t -h 0 (t, x)η(dt, x), t ∈ (0, T ), x ∈ D u(t, x) ≥ L(t, x), t ∈ (0, T ), x ∈ D T 0 D (u(t, x) -L(t, x))η(dt, x)dx = 0, u(t, x) = 0; (t, x) ∈ (0, T ) × ∂D u(T, x) = φ(x) a.s. (3.1)
The optimality equations (2.47)-(2.50) for the optimal harvesting problem above are typically of this form.

Theorem 5. Assume that E[|φ| 2 K ] < ∞ and that |b(s, u 1 , z 1 ) -b(s, u 2 , z 2 )| ≤ C(|u 1 -u 2 | + |z 1 -z 2 |).
Let L(t, x) be a measurable function which is differentiable in t and twice differentiable in x such that

T 0 D L ′ (t, x) 2 dxdt < ∞, T 0 D |∆L(t, x)| 2 dxdt < ∞.
Let h 0 (t, x) > 0 be a given bounded predictable process. Then there exists a unique

K × L 2 (D, R m ) × K-valued progressively measurable process (u(t, x), Z(t, x), η(t, x)) such that (i) E[ T 0 ||u(t)|| 2 V dt] < ∞, E[ T 0 |Z(t)| 2 L 2 (D,R m ) dt] < ∞. (ii) η is a K-valued continuous process, non-negative, nondecreasing in t, and η(0, x) = 0. (iii) u(t, x) = φ(x) + T t ∆u(t, x)ds + T t b(s, u(s, x), Z(s, x))ds - T t Z(s, x)dB s + T t h 0 (s, x)η(ds, x); 0 ≤ t ≤ T, x ∈ D, (iv) u(t, x) ≥ L(t, x) a.e. x ∈ D, ∀t ∈ [0, T ]. (v) T 0 D (u(t, x) -L(t, x))η(dt, x)dx = 0 (vi) u(t, x) = 0; (t, x) ∈ (0, T ) × ∂D (3.2)
where u(t) stands for the K-valued continuous process u(t, •) and (iii) is understood as an equation in the dual space V * of V .

Mathematics of Operations Research 00(0), pp. 000-000, c 2013 INFORMS

For the proof of the theorem, without loss of generality we assume h 0 (t, x) ≡ 1 and introduce the penalized BSPDEs:

du n (t, x) = -∆u n (t, x)dt -b(t, u n (t, x), Z n (t, x))dt + Z n (t, x)dB t -n(u n (t, x) -L(t, x)) -dt, t ∈ (0, T ) (3.3) u n (T, x) = φ(x) a.s.
According to [ØPZ], the solution (u n , Z n ) of the above equation exists and is unique. We are going to show that the sequence (u n , Z n ) has a limit, which will be a solution of the equation (3.2). First we need some a priori estimates: Lemma 4. Let (u n , Z n ) be the solution of equation (3.3). We have

sup n E[sup t |u n (t)| 2 K ] < ∞, (3.4)
sup

n E[ T 0 ||u n (t)|| 2 V ] < ∞, (3.5)
sup

n E[ T 0 |Z n (t)| 2 L 2 (D,R m ) ] < ∞. (3.6) Proof. Take a function f (t, x) ∈ C 2,2 0 ([-1, T + 1] × D) satisfying f (t, x) ≥ L(t, x). Applying Itô's formula, it follows that |u n (t) -f (t)| 2 K = |φ -f (T )| 2 K + 2 T t < u n (s) -f (s), ∆u n (s) > ds +2 T t < u n (s) -f (s), b(s, u n (s), Z n (s)) > ds -2 T t < u n (s) -f (s), Z n (s) > dB s +2n T t < u n (s) -f (s), (u n (s) -L(s)) -> ds - T t |Z n (s)| 2 L 2 (D,R m ) ds +2 T t < u n (s) -f (s), f ′ (s) > ds, a.s. (3.7)
where <, > denotes the inner product in K. Now we estimate each of the terms on the right hand side:

2 T t < u n (s) -f (s), ∆u n (s) > ds = -2 T t ||u n (s)|| 2 V ds + 2 T t < ∂f (s) ∂x , ∂u n (s) ∂x > ds ≤ - T t ||u n (s)|| 2 V ds + T t ||f (s)|| 2 V ds (3.8) 2 T t < u n (s) -f (s), b(s, u n (s), Z n (s)) > ds = 2 T t < u n (s) -f (s), b(s, u n (s), Z n (s)) -b(s, f (s), Z n (s)) > ds + 2 T t < u n (s) -f (s), b(s, f (s), Z n (s)) -b(s, f (s), 0) > ds + 2 T t < u n (s) -f (s), b(s, f (s), 0) > ds ≤ C T t |u n (s) -f (s)| 2 H ds + 1 2 T t |Z n (s)| 2 L 2 (D,R m ) ds + C T t |b(s, f (s), 0)| 2 H ds (3.9) 2n T t < u n (s) -f (s), (u n (s) -L(s)) -> ds = 2n T t D (u n (s, x) -f (s, x))χ {u n (s,x)≤L(s,x)} (L(s, x) -u n (s, x))dsdx ≤ 0 (3.10)
Substituting (3.8),(3.9) and (3.10) into (3.7) we obtain

|u n (t) -f (t)| 2 K + T t ||u n (s)|| 2 V ds + 1 2 T t |Z n (s)| 2 L 2 (D,R m ) ds ≤ |φ -f (T )| 2 K + C T t |u n (s) -f (s)| 2 K ds + C T t |b(s, f (s), 0)| 2 K ds + T t ||f (s)|| 2 V ds -2 T t < u n (s) -f (s), Z n (s) > dB s . (3.11)
We take expectation and use the Gronwall inequality to obtain sup

n sup t E[|u n (t)| 2 K ] < ∞ (3.12) sup n E[ T 0 ||u n (t)|| 2 V ] < ∞ (3.13) sup n E[ T 0 |Z n (t)| 2 L 2 (D,R m ) dt] < ∞ (3.14)
By virtue of (3.14), (3.12) can be further strengthend to (3.4). Indeed, by the Burkholder inequality,

E 2 sup v≤t≤T | T t < u n (s) -f (s), Z n (s) > dB s | ≤ CE ( T v |u n (s) -f (s)| 2 K |Z n (s)| 2 L 2 (D,R m ) ds) 1 2 ≤ CE sup v≤s≤T (|u n (s) -f (s)| K )( T v |Z n (s)| 2 L 2 (D,R m ) ds) 1 2 ≤ 1 2 E sup v≤s≤T (|u n (s) -f (s)| 2 K ) + CE T v |Z n (s)| 2 L 2 (D,R m ) ds (3.15)
With (3.15), taking supremum over t ∈ [v, T ] on both sides of (3.7) we obtain (3.4).

We need the following estimates:

Lemma 5. Suppose the conditions in Theorem 5 hold. Then there is a constant C such that

E[ T 0 D ((u n (t, x) -L(t, x)) -) 2 dxdt] ≤ C n 2 . (3.16)
Proof.

For m ≥ 1, define the functions ψ m (z), f m (x) as follows (see [START_REF] Donati-Martin | White noise driven spdes with reflection[END_REF]).

ψ m (z) =    0 if z ≤ 0, 2mz if 0 ≤ z ≤ 1 m , 2 if z > 1 m .
(3.17)

f m (x) = 0 if x ≤ 0, x 0 dy y 0 ψ m (z)dz if x > 0.
(3.18)

Mathematics of Operations Research 00(0), pp. 000-000, c 2013 INFORMS

We have

f ′ m (x) =    0 if x ≤ 0, mx 2 if x ≤ 1 m , 2x -1 n if x > 1 m . (3.19) Then f m (x) ↑ (x + ) 2 and f ′ m (x) ↑ 2x + as m → ∞. For h ∈ K, set G m (h) = D f m (-h(x))dx.
It is easy to see that for

h 1 , h 2 ∈ K, G ′ m (h)(h 1 ) = - D f ′ m (-h(x))h 1 (x)dx, (3.20)
G ′′ n (h)(h 1 , h 2 ) = D f ′′ m (-h(x))h 1 (x)h 2 (x)dx. (3.21)
Applying Itô's formula we get

G m (u n (t) -L(t)) = G m (φ -L(T )) + T t G ′ m (u n (s) -L(s))(∆u n (s)))ds + T t G ′ m (u n (s) -L(s))(b(s, u n (s), Z n (s)))ds + n T t G ′ m (u n (s) -L(s))((u n (s) -L(s)) -)ds + T t G ′ m (u n (s) -L(s))(L ′ (s))ds - T t G ′ m (u n (s) -L(s))(Z n (s))dB s - 1 2 T t G ′′ m (Z n (s), Z n (s))ds =: I 1 m + I 2 m + I 3 m + I 4 m + I 5 m + I 6 m + I 7 m . (3.22)
Now,

I 2 m = T t G ′ m (u n (s) -L(s))(∆u n (s)))ds = - T t D f ′ m (L(s, x) -u n (s, x))(∆(u n (s, x) -L(s, x)))dxds - T t D f ′ m (L(s, x) -u n (s, x))(∆L(s, x))dxds ≤ - T t D f ′′ m (L(s, x) -u n (s, x))|∇(u n (s, x) -L(s, x))| 2 dxds + 1 4 n T t D f ′ m (L(s, x) -u n (s, x)) 2 xds + C n T t D
(∆L(s, x)) 2 dxds, (3.23)

I 3 m = - T t D f ′ m (L(s, x) -u n (s, x))b(s, u n (s, x), Z n (s, x))dxds ≤ 1 4 n T t D f ′ m (L(s, x) -u n (s, x)) 2 ds + C n T t D (b(s, u n (s, x), Z n (s, x))) 2 dxds, (3.24)
I 5 m = - T t D f ′ m (L(s, x) -u n (s, x))(L ′ (s, x))dxds ≤ 1 4 n T t D f ′ m (L(s, x) -u n (s, x)) 2 ds + C n T t D (L ′ (s, x)) 2 dxds. ( 3
E[G m (u n (t) -L(t))] ≤ E[G m (φ -L(T ))] + 3 4 n T t D f ′ m (L(s, x) -u n (s, x)) 2 ds + C n E[ T t D (L ′ (s, x)) 2 dxds] + C n E[ T t D (∆L(s, x)) 2 dxds] + C n E[ T t D (b(s, u n (s, x), Z n (s, x))) 2 dxds] -nE[ T t D f ′ m (L(s, x) -u n (s, x))((u n (s, x) -L(s, x)) -)ds]. (3.26) Letting m → ∞ we conclude that E[ D ((u n (t, x) -L(t, x)) -) 2 dx] ≤ 3 4 nE[ T t D ((u n (s, x) -L(s, x)) -) 2 dxds] -nE[ T t D ((u n (s, x) -L(s, x)) -) 2 dxds] + C ′ n , (3.27)
where the Lipschitz condition of b and Lemma 4 have been used. In particular we have

E[ T t D ((u n (s, x) -L(s, x)) -) 2 dxds] ≤ C ′ n 2 .
(3.28) Lemma 6. Let (u n , Z n ) be the solution of equation (3.3). We have

lim n,m→∞ E[ sup 0≤t≤T |u n (t) -u m (t)| 2 K ] = 0, (3.29) lim n,m→∞ E[ T 0 ||u n (t) -u m (t)|| 2 V dt] = 0. (3.30) lim n,m→∞ E[ T 0 |Z n (t) -Z m (t)| 2 L 2 (D,R m ) dt] = 0. ( 3
|u n (t) -u m (t)| 2 K = 2 T t < u n (s) -u m (s), ∆(u n (s) -u m (s)) > ds +2 T t < u n (s) -u m (s), b(s, u n (s), Z n (s)) -b(s, u m (s), Z m (s)) > ds -2 T t < u n (s) -u m (s), Z n (s) -Z m (s) > dB s +2 T t < u n (s) -u m (s), n(u n (s) -L(s)) --m(u m (s) -L(s)) -> ds - T t |Z n (s) -Z m (s)| 2 L 2 (D,R m ) ds (3.32)
Now we estimate each of the terms on the right side:

2 T t < u n (s) -u m (s), ∆(u n (s) -u m (s)) > ds = -2 T t ||u n (s) -u m (s)|| 2 V ds. (3.33)
By the Lipschitz continuity of b and the inequality ab

≤ εa 2 + C ε b 2 , one has 2 T t < u n (s) -u m (s), b(s, u n (s), Z n (s)) -b(s, u m (s), Z m (s)) > ds ≤ C T t |u n (s) -u m (s)| 2 K ds + 1 2 T t |Z n (s) -Z m (s)| 2 L 2 (D,R m ) ds. (3.34)
In view of (3.28),

2E[ T t < u n (s) -u m (s), n(u n (s) -L(s)) --m(u m (s) -L(s)) -> ds] = 2nE[ T t < u n (s) -L(s), (u n (s) -L(s)) -> ds] +2mE[ T t < L(s) -u n (s), (u m (s) -L(s)) -> ds] +2mE[ T t < u m (s) -L(s), (u m (s) -L(s)) -> ds] +2nE[ T t < L(s) -u m (s), (u n (s) -L(s)) -> ds] ≤ 2mE[ T t < L(s) -u n (s), (u m (s) -L(s)) -> ds] +2nE[ T t < L(s) -u m (s), (u n (s) -L(s)) -> ds] ≤ 2mE[ T t D (u n (s, x) -L(s, x)) -(u m (s, x) -L(s, x)) -dxds] +2nE[ T t D (u m (s, x) -L(s, x)) -(u n (s, x) -L(s, x)) -dxds] ≤ 2m(E[ T t D ((u n (s, x) -L(s, x)) -) 2 dxds]) 1 2 (E[ T t D ((u m (s, x) -L(s, x)) -) 2 dxds]) 1 2 +2n(E[ T t D ((u n (s, x) -L(s, x)) -) 2 dxds]) 1 2 (E[ T t D ((u m (s, x) -L(s, x)) -) 2 dxds]) 1 2 ≤ C ′ ( 1 n + 1 m ). (3.35)
It follows from (3.32) and (3.33) that

E[|u n (t) -u m (t)| 2 K ] + 1 2 E[ T t |Z n (s) -Z m (s)| 2 L 2 (D,R m ) ds] +E[ T t ||u n (s) -u m (s)|| 2 V ds] ≤ C T t E[|u n (s) -u m (s)| 2 K ]ds + C ′ ( 1 n + 1 m ). ( 3
{E[|u n (t) -u m (t)| 2 K ] + 1 2 E[ T t |Z n (s) -Z m (s)| 2 L 2 (D,R m ) ds]} = 0, (3.37) lim n,m→∞ E[ T t ||u n (s) -u m (s)|| 2 V ds] = 0. ( 3
E[ sup 0≤t≤T |u n (t) -u m (t)| 2 K ] = 0. (3.39)
The proof is complete.

Proof of Theorem 5.

From Lemma 6 we know that (u n , Z n ), n ≥ 1, forms a Cauchy sequence. Denote by u(t, x), Z(t, x) the limit of u n and Z n . Put ηn (t, x) = n(u n (t, x) -L(t, x)) -.

Lemma 5 implies that ηn (t, x) admits a non-negative weak limit, denoted by η(t, x), in the following Hilbert space:

K = {h; h is a K-valued adapted process such that E[ T 0 |h(s)| 2 K ds] < ∞} with inner product < h 1 , h 2 > K = E[ T 0 D h 1 (t, x)h 2 (t, x)dtdx]. Set η(t, x) = t 0 η(s, x)ds.
Then η is a continuous K-valued process which is increasing in t. Keeping Lemma 6 in mind and letting n → ∞ in (3.3) we obtain

u(t, x) = φ(x) + T t ∆u(t, x)ds + T t b(s, u(s, x), Z(s, x))ds - T t Z(s, x)dB s +η(T, x) -η(t, x); 0 ≤ t ≤ T. (3.40)
Recall from Lemma 5 that

E[ T t D ((u n (s, x) -L(s, x)) -) 2 dxds] ≤ C ′ 1 n 2
Mathematics of Operations Research 00(0), pp. 000-000, c 2013 INFORMS By the Fatou Lemma, this implies that E[ T t D ((u(s, x) -L(s, x)) -) 2 dxds] = 0. In view of the continuity of u in t, we conclude u(t, x) ≥ L(t, x) a.e. in x, for every t ≥ 0. Combining the strong convergence of u n and the weak convergence of ηn , we also have

E[ T 0 D (u(s, x) -L(s, x))η(dt, x)dx] = E[ T 0 D (u(s, x) -L(s, x))η(t, x)dtdx] ≤ lim n→∞ E[ T 0 D (u n (s, x) -L(s, x))η n (t, x)dtdx] ≤ 0 (3.41) Hence, T 0 D (u(s, x) -L(s, x))η(dt, x)dx = 0, a.s.
We have shown that (u, Z, η) is a solution to the reflected BSPDE (3.1).

It remains to prove that the solution is unique. Let (u 1 , Z 1 , η 1 ), (u 2 , Z 2 , η 2 ) be two solutions of equation (3.2). By Itô's formula, we have

|u 1 (t) -u 2 (t)| 2 K = 2 T t < u 1 (s) -u 2 (s), ∆(u 1 (s) -u 2 (s)) > ds +2 T t < u 1 (s) -u 2 (s), b(s, u 1 (s), Z 1 (s)) -b(s, u 2 (s), Z 2 (s)) > ds -2 T t < u 1 (s) -u 2 (s), Z 1 (s) -Z 2 (s) > dB s +2 T t < u 1 (s) -u 2 (s), η 1 (ds) -η 2 (ds) > - T t |Z 1 (s) -Z 2 (s)| 2 L 2 (D,R m ) ds (3.42)
Similar to the proof of Lemma 6 , we have

2 T t < u 1 (s) -u 2 (s), ∆(u 1 (s) -u 2 (s)) > ds ≤ 0, (3.43) and 2
T t < u 1 (s) -u 2 (s), b(s, u 1 (s), Z 1 (s)) -b(s, u 2 (s), Z 2 (s)) > ds ≤ C T t |u 1 (s) -u 2 (s)| 2 K ds + 1 2 T t |Z 1 (s) -Z 2 (s)| 2 L 2 (D,R m ) ds (3.44) On the other hand, 2E[ T t < u 1 (s) -u 2 (s), η 1 (ds) -η 2 (ds) >] = 2E[ T t D (u 1 (s, x) -L(s, x))η 1 (ds, x)dx] -2E[ T t D (u 1 (s, x) -L(s, x))η 2 (ds, x)dx] +2E[ T t D (u 2 (s, x) -L(s, x))η 2 (ds, x)dx] -2E[ T t D (u 2 (s, x) -L(s, x))η 1 (ds, x)dx] ≤ 0. ( 3
E[|u 1 (t) -u 2 (t)| 2 K ] + 1 2 E[ T t |Z 1 (s) -Z 2 (s)| 2 L 2 (D,R m ) ds] ≤ C T t E[|u 1 (s) -u 2 (s)| 2 K ]ds. (3.46)
Appealing to the Gronwall inequality, this implies

u 1 = u 2 , Z 1 = Z 2
which further gives η 1 = η 2 from the equation they satisfy. This completes the proof of Theorem 5 .

Comparison theorems for BSPDEs and reflected BSPDEs

We establish now some comparison theorems for backward stochastic partial differential equations and BSPDEs without and with reflection, which are useful in the application to risk measures below, and which are also of independent interest. Consider two backward SPDEs: From now on, if u(t, x) is a function of (t, x), we write u(t) for the function u(t, •).

du 1 (t, x) = -∆u 1 (t, x)dt -b 1 (t, u 1 (t, x), Z 1 (t, x))dt + Z 1 (t, x)dB t , t ∈ (0, T ) u 1 (T, x) = φ 1 (x) a.s. ( 4
Theorem 6. (Comparison theorem for BSPDEs) Assume that b i , φ i , i = 1, 2 satisfy the conditions in Theorem 3.1. Suppose φ 1 (x) ≤ φ 2 (x) and b 1 (t, u, z) ≤ b 2 (t, u, z). Then we have u 1 (t, x) ≤ u 2 (t, x), x ∈ D, a.e. for every t ∈ [0, T ].

Proof. For h ∈ K := L 2 (D), set F n (h) = D f n (h(x))dx,
where f n is defined in (3.18). Applying Ito's formula we get

F n (u 1 (t) -u 2 (t)) = F n (φ 1 -φ 2 ) + T t F ′ n (u 1 (s) -u 2 (s))(∆(u 1 (s) -u 2 (s)))ds + T t F ′ n (u 1 (s) -u 2 (s))(b 1 (s, u 1 (s), Z 1 (s)) -b 2 (s, u 2 (s), Z 2 (s)))ds - T t F ′ n (u 1 (s) -u 2 (s))(Z 1 (s) -Z 2 (s))dB s - 1 2 T t F ′′ n (u 1 (s) -u 2 (s))(Z 1 (s) -Z 2 (s), Z 1 (s) -Z 2 (s))ds =: I 1 n + I 2 n + I 3 n + I 4 n + I 5 n , (4.3)
Mathematics of Operations Research 00(0), pp. 000-000, c 2013 INFORMS where,

I 2 n = T t F ′ n (u 1 (s) -u 2 (s))(∆(u 1 (s) -u 2 (s)))ds = T t D f ′ n (u 1 (s, x) -u 2 (s, x))(∆(u 1 (s, x) -u 2 (s, x)))dxds = - T t f ′′ n (u 1 (s, x) -u 2 (s, x))|∇(u 1 (s, x) -u 2 (s, x)| 2 dxds ≤ 0, (4.4)
I 5 n = -n T t D χ {0≤u 1 (s,x)-u 2 (s,x)≤ 1 n } (u 1 (s, x) -u 2 (s, x))|Z 1 (s, x) -Z 2 (s, x)| 2 dxds - T t D χ {u 1 (s,x)-u 2 (s,x)> 1 n } |Z 1 (s, x) -Z 2 (s, x)| 2 dxds. (4.5)
For I 3 n , we have

I 3 n = T t D f ′ n (u 1 (s, x) -u 2 (s, x))(b 1 (s, u 1 (s, x), Z 1 (s, x)) -b 2 (s, u 2 (s, x), Z 2 (s, x)))dxds = T t D f ′ n (u 1 (s, x) -u 2 (s, x))(b 1 (s, u 1 (s, x), Z 1 (s, x)) -b 2 (s, u 1 (s, x), Z 1 (s, x)))dxds + T t D f ′ n (u 1 (s, x) -u 2 (s, x))(b 2 (s, u 1 (s, x), Z 1 (s, x)) -b 2 (s, u 2 (s, x), Z 1 (s, x)))dxds + T t D f ′ n (u 1 (s, x) -u 2 (s, x))(b 2 (s, u 2 (s, x), Z 1 (s, x)) -b 2 (s, u 2 (s, x), Z 2 (s, x)))dxds ≤ T t D f ′ n (u 1 (s, x) -u 2 (s, x))(b 2 (s, u 2 (s, x), Z 1 (s, x)) -b 2 (s, u 2 (s, x), Z 2 (s, x)))dxds +C T t D ((u 1 (s, x) -u 2 (s, x) + ) 2 dxds := I 3 n,1 + I 3 n,2 , (4.6)
where the Lipschitz condition of b and the assumption b 1 ≤ b 2 have been used. I 3 n,1 can be estimated as follows:

I 3 n,1 ≤ C T t D f ′ n (u 1 (s, x) -u 2 (s, x))|Z 1 (s, x) -Z 2 (s, x)|dxds = C T t D χ {0≤u 1 (s,x)-u 2 (s,x)≤ 1 n } n(u 1 (s, x) -u 2 (s, x)) 2 |Z 1 (s, x) -Z 2 (s, x)|dxds +C T t D χ {u 1 (s,x)-u 2 (s,x)> 1 n } [2(u 1 (s, x) -u 2 (s, x)) - 1 n ]|Z 1 (s, x) -Z 2 (s, x)|dxds ≤ C T t D χ {u 1 (s,x)-u 2 (s,x)> 1 n } (2(u 1 (s, x) -u 2 (s, x)) - 1 n ) 2 dxds + T t D χ {u 1 (s,x)-u 2 (s,x)> 1 n } |Z 1 (s, x) -Z 2 (s, x)| 2 dxds + 1 4 C 2 T t D χ {0≤u 1 (s,x)-u 2 (s,x)≤ 1 n } n(u 1 (s, x) -u 2 (s, x)) 3 dxds + T t D χ {0≤u 1 (s,x)-u 2 (s,x)≤ 1 n } n(u 1 (s, x) -u 2 (s, x)) 2 |Z 1 (s, x) -Z 2 (s, x)| 2 dxds ≤ C ′ T t D ((u 1 (s, x) -u 2 (s, x)) + ) 2 dxds
F n (u 1 (t) -u 2 (t)) ≤ F n (φ 1 -φ 2 ) + C T t D ((u 1 (s, x) -u 2 (s, x)) + ) 2 dxds - T t F ′ n (u 1 (s) -u 2 (s))(Z 1 (s) -Z 2 (s))dB s (4.9)
Take expectation and let n → ∞ to get

E[ D ((u 1 (t, x) -u 2 (t, x)) + ) 2 dx] ≤ T t dsE[ D ((u 1 (s, x) -u 2 (s, x)) + ) 2 dx] (4.10)
Gronwall's inequality yields that

E[ D ((u 1 (t, x) -u 2 (t, x)) + ) 2 dx] = 0, (4.11)
which completes the proof of the theorem.

Remark 1. Comparison theorems for BSPDEs were also proved in [MYZ] and [HMY]. However, the results in these articles could not cover our theorem and the proofs are quite different.

We now state the comparison theorem for BSPDEs with reflection For i = 1, 2, consider the reflected backward stochastic partial differential equation:

du i (t, x) = -∆u i (t, x)dt -b i (t, u i (t, x), Z i (t, x))dt +Z i (t, x)dB t -h 0 (t, x)η i (dt, x), t ∈ (0, T ), x ∈ D u i (t, x) ≥ L i (t, x), t ∈ (0, T ), x ∈ D T 0 D (u i (t, x) -L i (t, x))η(dt, x)dx = 0, u i (t, x) = 0; (t, x) ∈ (0, T ) × ∂D u i (T, x) = φ i (x) a.s. (4.12)
Let u i (t, x), i = 1, 2 be solutions of the above equations.

Theorem 7. (Comparison theorem for reflected BSPDEs) Assume that b i , φ i , L i , i = 1, 2 satisfy the conditions in Theorem 5. Suppose φ

1 (x) ≤ φ 2 (x), L 1 (t, x) ≤ L 2 (t, x) and b 1 (t, u, z) ≤ b 2 (t, u, z). Then we have u 1 (t, x) ≤ u 2 (t, x), x ∈ D, a.e. for every t ∈ [0, T ]. Proof. Let u n i (t, x), i = 1, 2
, n ≥ 1 be the solutions of the penalized backward SPDEs: Mathematics of Operations Research 00(0), pp. 000-000, c 2013 INFORMS 5. Link to optimal stopping In this section, we provide a link between the solution of a reflected backward stochastic partial differential equation and an optimal stopping problem. Let S t,T be the set of all stopping times τ satisfying t ≤ τ ≤ T a.s. For τ ∈ S t,T , let (Y, k) = (Y τ , k τ ) be the solution of the BSPDE:

du n i (t, x) = -∆u n i (t, x)dt -b(t, u n i (t, x), Z n i (t, x))dt + Z n i (t, x)dB t -n(u n i (t, x) -L i (t, x)) -h 0 (t, x)dt, t ∈ (0, T ) (4.13) u n i (T, x) = φ i (x)
dY (t, x) = -∆Y (t, x)dt -g(t, x, Y (t, x), k(t, x))dt + k(t, x)dB(t), (t, x) ∈ (0, τ ) × R d Y (τ, x) = L(τ, x)χ τ <T + φ(x)χ τ =T L(τ, x) ; x ∈ R d . (5.1)
which gives in integral form

Y (t, x) = τ t P s-t g(s, x, Y (s, x), k(s, x))ds+P τ -t L(τ, x)χ {τ <T } +P τ -t φ(x)χ {τ =T } - τ t P s-t k(s, x)dB s , (5.
2) where P t denotes the semigroup generated by the Laplacian operator ∆, i.e.

P t f (x) = (4πt) -d/2 R d f (y)exp(- |y -x| 2 4t )dy; f ∈ L 1 (R d ).
Now let u(t, x), Z(t, x), η(t, x) be the solution of the following reflected BSPDE:

u(t, x) = φ(x) + T t ∆u(s, x)ds + T t g(s, x, u(s, x), Z(s, x))ds - T t Z(s, x)dB s +η(T, x) -η(t, x); 0 ≤ t ≤ T, x ∈ R d , u(t, x) ≥ L(t, x), (t, x) ∈ [0, T ] × R d , T 0 D (u(s, x) -L(s, x))η(ds, x)dx = 0 a.s. (5.3)
We have the following result:

Theorem 8. u(t, x) is the value function of the the optimal stopping problem associated with Y τ (t, x), i.e., u(t, x) = ess sup τ ∈S t,T Y τ (t, x).

(5.4)

Moreover, τ := τ (t, x) := inf{s ∈ [t, T )|u(s, x) = L(s, x)} ∧ T (5.5)
is an optimal stopping time.

Proof. Observe that u admits the following mild representation: Combining this with (5.8) we obtain the theorem.

u(t, x) = P T -t φ(x) + T t P s-t (g(s, u(s, x), Z(s, x)))ds - T t P s-t (Z(s, x))dB s + T t P s-t η(ds, x); 0 ≤ t ≤ T. ( 5
6. Application to risk minimizing stopping Let τ ∈ S 0,T , the set of stopping times with values between 0 and T . Suppose that g(t, x, y, k) is concave with respect to (y, k) for all t, x. Let F (t, x) be a given square integrable adapted process. In analogy with the definition of a convex risk measure in finance in terms of (ordinary) backward stochastic differential equations, we may consider F τ (x) = F (τ, x) as the financial standing at (τ, x), and we define the risk ρ(F τ )(t, x) of F τ (x) at time t ≤ τ and at the point x by ρ(F τ )(t, x) = -Y F τ (t, x), (

where Y (t, x) = Y F τ (t, x), k(t, x) is the solution of the BSPDE dY (t, x) = -∆Y (t, x)dtg(t, x, Y (t, x), k(t, x))dt + k(t, x)dB(t), (t, x) ∈ (0, τ ) × R d Y (τ, x) = F τ (x) ; x ∈ R d . (6.2)

Note that the monotonicity and the convexity of such "risk measures" is ensured by the comparison theorem 6. We consider the risk minimizing stopping problem to find τ * ∈ S 0,T and ρ(F τ * )(t, x) such that ρ(F τ * )(t, x) = ess inf τ ∈S t,T ρ(F τ )(t, x) (6.3)

We may consider the space diffusion effect stemming from the Laplacian operator, as a representation of a mean-field effect in a market with many agents with interacting notions of risk.

Conclusion

In this paper we study singular control problems of stochastic partial differential equations (SPDEs). We prove a sufficient and a necessary maximum principle for the solution of such problems and show that the solution is linked to a reflected backward SPDE (RBSPDE). The existence and uniqueness of such equations is established, and we also prove comparison theorems for BSPDEs and reflected BSPDEs, which is of independent interest.

We give two applications of our general results: (i) In the first application we solve a problem of optimal harvesting from a population whose density dynamics is modeled by a stochastic reaction diffusion equation, (ii) In the second we show that the solution of an optimal stopping problem for a BSPDE can be expressed in terms of an associated reflected BSPDE. The result is applied to the problem of risk minimizing stopping in a financial market with mean-field type of interactions between the agents.

D

  .25) Combining (3.22)-(3.25) and taking expectation we obtain

  .45) Combining (3.42)-(3.45) we arrive at

  x)dB(s) + η(T, x)η(t, x); (t, x) ∈ (0, T ) × R d , u(t, x) ≥ F (t, x); (t, x) ∈ (0, T ) × R d , s, x) -F (s, x))η(ds, x)dx = 0 a.s.(6.7)Moreover, the stopping time τ = τ (t, x) defined by τ (t, x) = inf{s ∈ [t, T )|u(s, x) = F (s, x)} ∧ T is optimal.

  ) be the corresponding solution of (2.1)-(2.2) and (2.6)-(2.8) , respectively, and assume that (2.28) holds with ξ = ξ * .

	Then
	λ(t, x, Y (2.34)
	and
	{λ(t, x, Y (2.35)
	(ii) Conversely, suppose that there exists ξ ∈ A such that the corresponding solutions Ŷ (t, x), (p(t, x), q(t, x)) of (2.1)-(2.2) and (2.6)-(2.8), respectively, satisfy
	λ(t, x, Ŷ (t, x))p(t, x) + h(t, x, Ŷ (t, x)) ≤ 0 for all (t, x) ∈ [0, T ] × D, a.s.	(2.36)
				(2.33)
	Therefore, substituting (2.31) and (2.32) into (2.30), we get
	lim y→0 +	1 y	(J (ξ + yζ) -J (ξ))
		= E[

x){AY(t, x)dt + Y(t, x) ∂b ∂y (t, x)dt + Y(t, x) ∂λ ∂y (t, x)ξ(dt, x) + λ(t, x)ζ(dt, x)} + Y(t, x){-A * p(t, x)dt -∂H ∂y (dt, ξ(dt, x))} + Y(t, x) ∂σ ∂y (t, x)q(t, x)]dt)dx], (2.32) where [p(•, x), Y(•, x)](t) denotes the covariation process of p(•, x) and Y(•, x). Since p(t, x) = Y(t, x) = 0 for x ∈ ∂D, we deduce that D p(t, x)AY(t, x)dx = D A * p(t, x)Y(t, x)dx. D ( T 0 {λ(t, x)p(t, x) + h(t, s)}ζ(dt, x))dx]. We can now state our necessary maximum principle: Theorem 3. [Necessary maximum principle] (i) Suppose ξ * ∈ A is optimal, i.e. max ξ∈A J (ξ) = J (ξ * ). Let Y * , (p * , q * * (t, x))p * (t, x) + h(t, x, Y * (t, x)) ≤ 0 for all (t, x) ∈ [0, T ] × D, a.s. * (t, x))p * (t, x) + h(t, x, Y * (t, x))}ξ * (dt, x) = 0 for all (t, x) ∈ [0, T ] × D, a.s. and {λ(t, x, Ŷ (t, x))p(t, x) + h(t, x, Ŷ (t, x))} ξ(dt, x) = 0 for all (t, x) ∈ [0, T ] × D, a.s. (2.37)

  .31)

	Mathematics of Operations Research 00(0), pp. 000-000, c 2013 INFORMS
	Proof. Applying Itô's formula, it follows that

  {u 1 (s,x)-u 2 (s,x)> 1 n } |Z 1 (s, x) -Z 2 (s, x)| 2 dxds (s,x)-u 2 (s,x)≤ 1 n } n(u 1 (s, x)u 2 (s, x)) 2 |Z 1 (s, x) -Z 2 (s, x)| 2 dxds

	Øksendal, Sulem, and Zhang: Article Short Title Mathematics of Operations Research 00(0), pp. 000-000, c 2013 INFORMS	21
			T
	+
		t	T	D
	+	t		D	χ {0≤u 1 (4.7)
	(4.5),(4.6) and (4.7) imply that
					T
					I 3 n + I 5 n ≤ C	((u 1 (s, x) -u 2 (s, x)) + ) 2 dxds	(4.8)
					t	D
	Thus it follows from (4.3), (4.4) and (4.8) that

χ

  a.s. From the proof of Theorem 5 we know that u n

i (t, x) → u i (t, x) as n → ∞. By Theorem 6, we have u n 1 (t, x) ≤ u n 2 (t, x), x ∈ D, a.e. for every t ∈ [0, T ] and n ≥ 1. Hence, u 1 (t, x) ≤ u 2 (t, x), x ∈ D, a.e. for every t ∈ [0, T ].

  Taking conditional expectation with respect to F t on both sides we getu(t, x) ≥ E[Y τ (t, x)|F t ] = Y τ (t, x). = inf{s ∈ [t, T )|u(s, x) = L(s, x)} ∧ T.From the property of η, it is not increasing on the interval [t, τ ]. Thus, τ t P s-t η(ds, x) = 0. So we have from (5.7) thatu(t, x) = P τ -t (u(τ, x))| τ =τ + P s-t (k(s, x))dB s = Y τ (t, x)| τ =τ -

					(5.8)
	As τ is arbitrary, we obtain
				u(t, x) ≥ ess sup	Y τ (t, x)	(5.9)
					τ ∈S t,T
	Now, define
			τ = τ (t, x) τ
					P s-t (g(s, x, u(s, x), Z(s, x)))ds
		τ			τ	t
		-	P s-t (Z(s, x))dB s +
		t		τ	t	τ
				P s-t (Z(s, x))dB s +	P s-t (k(s, x))dB s
				t	t
					.6)
	More generally, for any stopping time τ with t ≤ τ ≤ T , we have
			τ		τ
	u(t, x) = P τ -t (u(τ, x)) +	P s-t (g(s, x, u(s, x), Z(s, x)))ds -	P s-t (Z(s, x))dB(s)
	τ		t		t
	+	P s-t η(ds, x); 0 ≤ t ≤ τ.	(5.7)
	t

Since η(s, x) is increasing in s and u(s, x) ≥ L(s, x) for s ≤ T , it follows that

u(t, x) ≥ Y τ (t, x) -τ t P s-t (Z(s, x))dB(s) + τ t P s-t (k(s, x))dB s = Y τ (t, x) -τ t P s-t (Z(s, x))dB s + τ t P s-t (k(s, x))dB s .

(

5.10)

Taking conditional expectation yields that

u(t, x) = E[Y τ (t, x)|F t ] = Y τ (t, x).

  Mathematics of Operations Research 00(0), pp. 000-000, c 2013 INFORMS Note that the solution of the BSPDE for Y F τ (t, x) isY F τ (t, x) = τ t P s-t g(s, x)ds + P τ -t F (τ, x) -Therefore, comparing with the equation (5.7) above for Y τ (t, x), we see that Y τ F (t, x) coincides with Y τ (t, x) if we choose L(t, x) and φ(x) such thatF (t, x) = L(t, x)χ t<T + φ(x)χ t=T .(6.5)Applying Theorem 4.1 above to this choice of L(t, x) and φ(x) we get the following result, which is a space-time version of a known result (see[START_REF] Quenez | Reflected bsdes and robust optimal stopping for dynamic risk measures with jumps[END_REF]) :

		τ
		P s-t (k(s, x))dB(s).	(6.4)
		t
	Theorem 9. (Risk minimizing stopping theorem)
	ess inf τ ∈S t,T	ρ(F τ )(t, x) = -u(t, x),	(6.6)

where u(t, x), Z(t, x), η(t, x) is the solution of the reflected BSPDE u(t, x) = F (T, x) + T t ∆u(s, x)ds + T t g(s, x, u(s, x), Z(s, x))ds

Acknowledgments.


Bernt Øksendal
email: oksendal@math.uio.

Agnès Sulem
email: agnes.sulem@inria.fr

Tusheng Zhang
email: tusheng.zhang@manchester.ac.uk

A stochastic

come

Introduction {intro}

In classical theory of stochastic control of systems described by a stochastic differential equations (SDEs) there are two important solution methods:

(i) Dynamic programming, which leads to the classical Hamilton-Jacobi-Bellman (HJB) equation. This is a deterministic non-linear partial differential equation (PDE) in the (unknown) value function for the problem.

(ii) The maximum principle, which involves the maximization of the Hamiltonian and an associated backward stochastic differential equation (BSDE) in the (unknown) adjoint processes.

Dynamic programming is a very efficient solution method, but it only works if the system is Markovian. The maximum principle, on the other hand, works also in non-Markovian settings, but the drawback is that it leads to a complicated coupled system of forwardbackward SDEs (FBSDEs) with constraints, and this system is difficult to solve in general.

In view of this it is natural to ask if there is an extension of the HJB approach to non-Markovian systems. The answer has been known to be yes for some time, at least in some cases. See [P], where a stochastic HJB equation is proved for non-Markovian, Brownian motion driven SDEs, in which the diffusion coefficient does not depend on the control. The purpose of this paper is to show that the answer is yes also in a more general context. More precisely, and more generally, we give a method for solving optimal control problems for general non-Markovian systems of forward-backward stochastic differential equations (FBS-DEs) by means of a stochastic HJB equation, which is a backward stochastic partial differential equations (BSPDEs). Our underlying models are Itô-Lévy processes (with jumps). If the system is a Markovian SDE, then our stochastic HJB equation becomes deterministic and coincides with the classical HJB equation. In the last part of the paper we illustrate our theory by studying some applications to finance. In particular we, apply our results to study a problem of risk minimization in a financial market.

In [P] a stochastic version of the classical HJB equation is studied and existence and uniqueness is proved for this type of SPDEs. However, there it is assumed that the control does not enter the diffusion coefficient.

The relation between FBSDEs and B(S)PDEs has been known for several years. See, for example, [MPY] for the Markovian case (which leads to a deterministic backward PDE). For the more general, possibly non-Markovian case, see e.g. the recent paper [MYZ] and the references therein.

For papers on optimal control of general SDEs and associated stochastic HJB equations, see e.g. [P] and [BM] and the references therein. None of the above papers deal with jumps.

The novelty of our paper lies in the application of this connection studied in [MYZ] to optimal control of FBSDEs and in the extension to jump models.

Optimal control of FBSDEs {sec2}

We refer to [ØS1] for information about stochastic calculus and control for jump diffusions.

Consider the following controlled coupled FBSDE: The forward equation in X(t) has the form

             dX(t) = α(t, X(t), Y (t), Z(t), K(t, •), u(t, X(t)))dt +β(t, X(t), Y (t), Z(t), K(t, •), u(t, X(t)))dB(t) + R γ(t, X(t), Y (t), Z(t), K(t, •), u(t, X(t)), ζ) Ñ (dt, dζ) ; t ∈ [0, T ] X(0) = x ∈ R (2.1) {eq1.1}
and the backward equation in Y (t), Z(t), K(t, ζ) has the form

       dY (t) = -g(t, X(t), Y (t), Z(t), K(t, •), u(t, X(t)))dt + Z(t)dB(t) + R K(t, ζ) Ñ (dt, dζ) ; t ∈ [0, T ] Y (T ) = h(X(T )).
(2.2) {eq1.2}

Here

B(t) = B(t, ω) and Ñ (dt, dζ) = N (dt, dζ) -ν(dζ)dt ; t ∈ [0, T ], ω ∈ Ω, ζ ∈ R 0 := R -{0}
is a Brownian motion and an (independent) compensated Poisson random measure, respectively, on a given filtered probability space (Ω, F, F := {F t } t≥0 , P ).

The given functions

α(t, x, y, z, k, u, ω) : [0, T ] × R × R × R × R × V × Ω → R β(t, x, y, z, k, u, ω) : [0, T ] × R × R × R × R × V × Ω → R γ(t, x, y, z, k, u, ζ, ω) : [0, T ] × R × R × R × R × V × R 0 × Ω → R g(t, x, y, z, k, u, ω) : [0, T ] × R × R × R × R × V × Ω → R
are assumed to be F-predictable for each x, y, z, k, u. R denotes the set of functions k(ζ) : R 0 → R and V is a given set of admissible control values u(t, x, ω), where u(t) = u(t, X(t), ω) is our control process. The function h(x, ω) : R × Ω → R is assumed to be F T -measurable for each x.

We let A denote the set of admissible control processes u(t), to be specified further below. We want to find û ∈ A such that

sup u∈A Y u (0) = Y û(0). (2.3) {eq1.3}
To this end, let us first recall the following extension of the Itô-Ventzell formula:

{th5.1}
Theorem 2.1 (The Itô-Ventzell formula with jumps) Suppose y(t, x) solves the SPDE

dy(t, x) = A(y(•), z(•), k(•))(t, x)dt + z(t, x)dB(t) R k(t, x, ζ) Ñ (d, dζ) ; t ≥ 0 (2.4) {eq5.3}
for some partial integro-differential operator A acting on x, and X(t) satisfies an equation of the form

dX(t) = α(t)dt + β(t)dB(t) + R θ(t, ζ) Ñ (dt, dζ) ; t ≥ 0. (2.5) {eq5.4}
for given F -predictable processes α, β, θ. Put

Y (t) = y(t, X(t)) ; t ≥ 0. (2.6) {eq5.5}
Then

dY (t) = A(y(•), z(•), k(•))(t, X(t))dt + z(t, X(t))dB(t) + R k(t, X(t), ζ) Ñ (dt, dζ) + y ′ (t, X(t))[α(t)dt + β(t)dB(t)] + 1 2 y ′′ (t, X(t))β 2 (t)dt + R {y(t, X(t) + θ(t, ζ)) -y(t, X(t)) -y ′ (t, X(t))θ(t, ζ)}ν(dζ)dt + R {y(t, X(t -) + θ(t, ζ)) -y(t, X(t -))} Ñ (dt, dζ) + z ′ (t, X(t))β(t)dt + R {k(t, X(t -) + θ(t, ζ))} Ñ (dt, dζ) + R {k(t, X(t -) + θ(t, ζ)) -k(t, X(t -))}ν(dζ)dt, (2.7)
where y ′ (t, x) = ∂y ∂x (t, x) etc.

Proof. See [ØZ] and the references therein.

We now return to problem (2.3). First we try to write the solution Y (t) of (2.2) on the form

Y (t) = y(t, X(t)) (2.8) {eq1.4}
for some random field y(t, x) = y(t, x, ω) which, together with z(t, x) and k(t, x, ζ), satisfies a BSPDE of the form

       dy(t, x) = A(y(•), z(•), k(•))(t, x)dt + z(t, x)dB(t) + R k(t, x, ζ) Ñ (dt, dζ) ; t ∈ [0, T ] y(T, x) = h(x),
(2.9) {eq1.5}

for some partial integro-differential operator A acting on x. By the Itô-Ventzell formula,

dY (t) = A(y(•), z(•), k(•))(t, X(t))dt + z(t, X(t))dB(t) + R k(t, X(t), ζ) Ñ (dt, dζ) + y ′ (t, X(t))[α(t)dt + β(t)dB(t)] + 1 2 y ′′ (t, X(t))β 2 (t)dt + R {y(t, X(t) + γ(t, ζ)) -y(t, X(t)) -y ′ (t, X(t))γ(t, ζ)}ν(dζ)dt + R {y(t, X(t) + γ(t, ζ)) -y(t, X(t))} Ñ (dt, dζ) + z ′ (t, X(t))β(t)dt + R {k(t, X(t) + γ(t, ζ), ζ) -k(t, X(t), ζ)}ν(dζ)dt + R k(t, X(t -) + γ(t, ζ), ζ) Ñ (dt, dζ), (2.10) {eq1.6}
where we have used the shorthand notation

α(t) = α(t, X(t), Y (t), Z(t), K(t, •), u(t)) etc.
Rearranging the terms we see that

dY (t) = [A(y(•), z(•), k(•))(t, X(t)) + y ′ (t, X(t))α(t) + 1 2 y ′′ (t, X(t))β 2 (t) + R {y(t, X(t) + γ(t, ζ)) -y(t, X(t)) -y ′ (t, X(t))γ(t, ζ)}ν(dζ)dt + z ′ (t, X(t))β(t) + R {k(t, X(t) + γ(t, ζ), ζ) -k(t, X(t), ζ)}ν(dζ)]dt + [z(t, X(t)) + y ′ (t, X(t))β(t)]dB(t) + R {y(t, X(t) + γ(t, ζ)) -y(t, X(t)) + k(t, X(t) + γ(t, ζ), ζ)} Ñ (dt, dζ). (2.11) {eq1.7}
Comparing (2.11) with (2.2) we deduce that

A(y(•), z(•), k(•))(t, X(t)) = -[g(t, X(t), Y (t), Z(t), K(t, •), u(t, X(t))) + y ′ (t, X(t))α(t) + 1 2 y ′′ (t, X(t))β 2 (t) + R {y(t, X(t) + γ(t, ζ)) -y(t, X(t)) -y ′ (t, X(t))γ(t, ζ)}ν(dζ) + z ′ (t, X(t))β(t) + R {k(t, X(t) + γ(t, ζ), ζ) -k(t, X(t), ζ)}ν(dζ)]
(2.12) {eq1.8}

Z(t) = z(t, X(t)) + y ′ (t, X(t))β(t) (2.13) {eq1.9} K(t, ζ) = y(t, X(t) + γ(t, ζ)) -y(t, X(t)) + k(t, X(t) + γ(t, ζ), ζ).
(2.14) {eq1.10}

In the following we will use the shorthand notations

β(t) := β(t, x, y(t, x), z(t, x), k(t, x, •)) etc, (2.15) z(t, x) := z(t, x) + y ′ (t, x)β(t) (2.16) k(t, x, ζ) := y(t, x + γ(t, ζ)) -y(t, x) + k(t, x + γ(t, ζ), ζ).
(2.17)

We summarize what we have proved as follows:

{th2.1}

G u 1 (t, x) ≤ G u 2 (t, x) for all t, x ∈ [0, T ] × R and h 1 (x) ≤ h 2 (x)
for all x ∈ R, we have

y 1 (t, x) ≤ y 2 (t, x) for all t, x ∈ [0, T ] × R.
Sufficient conditions for the validity of comparison principles for BSPDEs can be found in [MYZ] and [START_REF] Øksendal | Comparison theorems for backward SPDEs with jumps[END_REF].

For example, from Theorem 2.13 in [MYZ] we get:

{th2.3}
Theorem 2.4 Assume that the following holds:

• N = K = 0, i.e. there are no jumps

• The coefficients α, β, and g are F -progressively measurable for each fixed (x, y, z) and h is F T -measurable for each fixed x

• α, β, g, h are uniformly Lipschitz -continuous in (x, y, z)

• α and β are bounded and

E[ T 0 g 2 (t, 0, 0, 0)dt + h 2 (0)] < ∞ (2.23)
• α(t, x, y, z, u) does not depend on z

Then G u (t, x) satisfies the comparison principle.

From the above we deduce the following result, which may be regarded as a stochastic HJB equation for optimal control of possibly non-Markovian FBSDEs.

{th1.1}

Theorem 2.5 Suppose that G u (t, x) satisfies the comparison principle. Moreover, suppose that for all t, x, ω there exists a maximizer u = û(t, x) = û(y, y ′ , y ′′ , z, z ′ , k)(t, x, ω) of the function u → G u (t, x).

Suppose the system (2.18) with u = û has a unique solution (ŷ(t, x), ẑ(t, x), k(t, x, •)) and that û(t, X(t)) ∈ A. Then û(t, X(t)) is an optimal control for the problem (2.3), with optimal value sup u∈A Y u (0) = Y û(0) = ŷ(0, x).

(2.24) {eq1.13}

Note that in this general non-Markovian setting the classical value function from the dynamic programming is replaced by the solution ŷ(t, x) of the BSDE (2.18) for u = û. See Example 3.1 below for more details.

Hence the BSPDE for y(t, x) gets the form

   dy(t, x) = (y ′ (t, x)b(t) + z ′ (t, x)σ(t)) 2 2y ′′ (t, x)σ 2 (t) dt + z(t, x)dB(t) ; t ∈ [0, T ] y(T, x) = U (x). (3.8) {eq2.6}
If b, σ and U are deterministic, we can choose z(t, x) = 0 and this leads to the following PDE for y(t, x):

∂y ∂t (t, x) - y ′ (t, x) 2 b 2 (t) 2y ′′ (t, x)σ 2 (t) = 0 ; t ∈ [0, T ]. (3.9) {eq2.7}
This is the classical Merton PDE for the value function, usually obtained by dynamic programming and the HJB equation.

Hence we may regard (3.6)-(3.8) as a generalization of the classical Merton solution (3.9) to the case with stochastic b(t), σ(t) and U (x), where classical dynamic programming cannot be used. Thus we see that the Markovian case corresponds to the special case when ẑ = 0 of the BSDE (3.9). Therefore ŷ(s, x) is a stochastic generalization of the value function

ϕ(s, x) := sup u∈A E[U (X (u) s,x (T ))] (3.10) {eq5.23}
where

dX s,x (t) = u(t)[b 0 (t)dt + σ 0 (t)dB(t)] ; t ≥ s X s,x (s) = x (3.11) {eq5.24}
Compare with the use of the classical HJB:

   ∂ϕ ∂s (s, x) + max v 1 2 v 2 σ 2 0 (s)ϕ ′′ (s, x) + vb 0 (s)ϕ ′ (s, x) = 0 ; s < T ϕ(T, x) = U (x).
(3.12) {eq3.10}

The maximum is attained at

v = û(s, x) = - b 0 (s)ϕ ′ (s, x) ϕ ′′ (s, x)σ 2 0 (s) (3.13) {eq5.26}
Substituted into (3.12) this gives the HJB equation

∂ϕ ∂s (s, x) - ϕ ′ (s, x) 2 b 2 0 (s) ϕ ′′ (s, x)σ 2 0 (s) = 0, (3.14) {eq5.27}
which is identical to (3.9).

{exa5.6}

Example 3.2 (Risk minimizing portfolios) Now suppose X(t) = X (u)

x (t) is as in (3.3), while (Y (t), Z(t)) is given by the BSDE

   dY (t) = -- 1 2 Z 2 (t) dt + Z(t)dB(t) Y (T ) = X(T ).
(3.15) {eq5.28}

Note that g(z) = -1 2 z 2 is concave. We want to minimize -Y (0) = ρ(X(T )), where ρ(X(T )) is the risk of X(T ) with respect to the driver g(z) = 1 2 z 2 . See e.g. [QS], [R] for more information about the representation of risk measures via BSDEs. Here

G u (t, x) = - 1 2 (z(t, x) + y ′ (t, x)uσ(t)) 2 + y ′ (t, x)ub(t) + 1 2 y ′′ (t, x)u 2 σ 2 (t) + z ′ (t, x)uσ(t), (3.16) {eq5.29} which is minimal when u = û(t, x) satisfies û(t, x) = - z(t, x)y ′ (t, x)σ(t) -y ′ (t, x)b(t) -z ′ (t, x)σ(t) ((y ′ (t, x)) 2 -y ′′ (t, x))σ 2 (t) . (3.17) {eq5.30}
This gives

G û(t, x) = - 1 2 ẑ2 (t, x) + (ẑ(t, x)ŷ ′ (t, x)σ(t) -ŷ′ (t, x)b(t) -ẑ′ (t, x)σ(t)) 2 2((ŷ ′ (t, x)) 2 -ŷ′′ (t, x))σ 2 (t) . (3.18) {eq5.31}
and hence (ŷ(t, x), ẑ(t, x)) solves the BSPDE

dŷ(t, x) = -G û(t, x)dt + ẑ(t, x)dB(t) ; 0 ≤ t ≤ T ŷ(T, x) = x. (3.19) {eq5.32}
Let us try to choose ẑ(t, x) = 0. Then (3.19) reduces to the PDE

   ∂ ŷ(t, x) ∂t = - (ŷ ′ (t, x)b(t)) 2 2((ŷ ′ (t, x)) 2 -ŷ′′ (t, x))σ 2 (t) ; 0 ≤ t ≤ T ŷ(T, x) = x. (3.20) {eq5.33}
We try a solution of the form

ŷ(t, x) = x + a(t), (3.21) {eq5.34}
where a(t) is deterministic. Substituted into (3.20) this gives

     a ′ (t) = - 1 2 b(t) σ(t) 2 ; 0 ≤ t ≤ T a(T ) = 0 (3.22) {eq5.35}
which gives

a(t) = T t 1 2 b(s) σ(s) 2 ds ; 0 ≤ t ≤ T.
With this choice of a(t) we see that (3.20) is satisfied and we conclude that the minimal risk is Hence, if we define the measure Q Γ (ω) by Note that Q Γ is the unique equivalent martingale measure for the market (3.1),(3.2).

ρ min (X(T )) = -Y (û) (0) = -ŷ(0, x) = -x - T 0 1 2 b(s) σ(t
Thus we conclude that the negative of the minimal risk is equal to the initial wealth x plus the entropy of the equivalent martingale measure.

It is natural to ask what the corresponding result would be in the incomplete market case.

  (3.23) can be writtenρ min (X(T )) = -x -H(Q Γ | P ) (3.28)

  We say that the driver G u (t, x) given by (2.19) satisfies the comparison principle if the corresponding BSPDE (2.18) satisfies the comparison principle with respect to u. In other words, for all u 1 , u 2 ∈ A and all F T -measurable h 1 , h 2 with corresponding solutions y 1 (t, x), y 2 (t, x), respectively, of (2.18) such that

	Definition 2.3
	+	{k(t, x + γ(t, ζ), ζ) -k(t, x, ζ)}ν(dζ),	(2.19) {eq1.12}
	R
	Then (Y (t), Z(t), K(t, ζ)), given by
		Y (t) := y(t, X(t)),	(2.20)
	Z(t) := z(t, X(t)) + y ′ (t, X(t))β(t),	(2.21)
	K(t, ζ) := y(t, X(t) + γ(t, ζ)) -y(t, X(t)) + k(t, X(t) + γ(t, ζ), ζ),	(2.22)

Theorem 2.2 Suppose that (y(t, x), z(t, x), k(t, x, •)) satisfies the BSPDE

dy(t, x) = -G u (t, x)dt + z(t, x)dB(t) + R k(t, x, ζ) Ñ (dt, dζ) y(T, x) = h(x) (2.18) {eq1.11}

with G u (t, x) := g(t, x, y(t, x), z(t, x), k(t, x, •), u(t, x))

+ y ′ (t, x)α(t) + 1 2 y ′′ (t, x)β 2 (t) + z ′ (t, x)β(t) + R {y(t, x + γ(t, ζ))y(t, x)y ′ (t, x)γ(t, ζ)}ν(dζ)

is a solution of the FBSDE system (2.1)-(2.2).

  ) It is interesting to note that (3.23) can be interpreted by means of entropy as follows:Recall that in general the entropy of a measure Q with respect to the measure P is defined by

														2
														ds	(3.23) {eq5.36}
	Hence by (3.17) the optimal (risk minimizing) portfolio is
			û(t, X(t)) =	b(t) σ 2 (t)	.	(3.24) {eq5.37}
	Remark 3.1 H(Q | P ) := E	dQ dP	ln	dQ dP
	.
	Define	Γ(t) = exp -	0	t	b(s) σ(s)	dB(s) -	1 2	0	t	(	b(s) σ(s)	) 2 ds .	(3.25)
	Then	dΓ(t) = Γ(t) -	b(t) σ(t)	dB(t) ; Γ(0) = 1.	(3.26)
	By the Itô formula we have
	d(Γ(t) ln Γ(t)) = Γ(t) -	b(t) σ(t)	dB(t) -		1 2	b(t) σ(t)	2	dt
		+ (ln Γ(t))Γ(t) -	b(t) σ(t)	dB(t) + Γ(t) -	b(t) σ(t)	-	b(t) σ(t)	dt.

no and Norwegian School The research leading to these results has received funding from the European Research Council under the European Community's Seventh Framework Programme (FP7/2007-2013) / ERC grant agreement no [228087].

Applications {sec4}

We now illustrate Theorem 2.5 by looking at some examples and applications. First we consider the classical Merton problem:

Example 3.1 [Maximizing expected utility from terminal wealth] Consider a financial market consisting of two investment possibilities, as follows: (i) A risk free investment, with unit price

(ii) a risky investment, with unit price

Let u(t, X(t)) be a portfolio, representing the amount invested in the risky asset at time t.

If we assume that is self-financing, then the corresponding wealth X(t) at time t is given by the stochastic differential equation

Let (Y (t), Z(t)) be the solution of the BSDE

where U (x) = U (x, ω) is a utility function, possibly random. Then

In this case we get, from (2.19),

which is maximal when