Nicolò Cesa-Bianchi
email: nicolo.cesa-bianchi@unimi.it

Tommaso Cesari
email: tommaso.cesari@tse-fr.eu

Roberto Colomboni
email: roberto.colomboni@unimi.it

Claudio Gentile
email: cgentile@google.com

Yishay Mansour
email: mansour@tau.ac.il

Nonstochastic Bandits with Composite Anonymous Feedback Nonstochastic Bandits with Composite Anonymous Feedback *

Keywords: Multi-armed bandits, non-stochastic losses, composite losses, delayed feedback, online learning

We investigate a nonstochastic bandit setting in which the loss of an action is not immediately charged to the player, but rather spread over the subsequent rounds in an adversarial way. The instantaneous loss observed by the player at the end of each round is then a sum of many loss components of previously played actions. This setting encompasses as a special case the easier task of bandits with delayed feedback, a well-studied framework where the player observes the delayed losses individually.

Our first contribution is a general reduction transforming a standard bandit algorithm into one that can operate in the harder setting: We bound the regret of the transformed algorithm in terms of the stability and regret of the original algorithm. Then, we show that the transformation of a suitably tuned FTRL with Tsallis entropy has a regret of order (d + 1)KT , where d is the maximum delay, K is the number of arms, and T is the time horizon. Finally, we show that our results cannot be improved in general by exhibiting a matching (up to a log factor) lower bound on the regret of any algorithm operating in this setting.

Introduction

Multiarmed bandits, originally proposed for managing clinical trials, are now routinely applied to a variety of other tasks, including computational advertising, e-commerce, and beyond. Typical examples of e-commerce applications include content recommendation systems, like the recommendation of products to visitors of merchant websites and social media platforms. A common pattern in these applications is that the response elicited in a user by the recommendation system is typically not instantaneous, and might occur some time in the future, well after the recommendation was issued. This delay, which might depend on several unknown factors, implies that the reward obtained by the recommender at time t can actually be seen as the combined effect of many previous recommendations to that user.

The more specific scenario of bandits with delayed rewards has been investigated in the literature under the assumption that the contributions of past recommendations to the combined reward is individually discernible -see, e.g., (Neu et al., 2010;[START_REF] Joulani | Online learning under delayed feedback[END_REF][START_REF] Cesa-Bianchi | Delay and cooperation in nonstochastic bandits[END_REF][START_REF] Vernade | Stochastic bandit models for delayed conversions[END_REF]. [START_REF] Pike-Burke | Bandits with delayed, aggregated anonymous feedback[END_REF] revisited the problem of bandits with delayed feedback under the more realistic assumption that only the combined reward is available to the system, while the individual reward components remain unknown. This model captures a much broader range of practical settings where bandits are successfully deployed. Consider for example an advertising campaign which is spread across several channels simultaneously (e.g., radio, tv, web, social media). A well-known problem faced by the campaign manager is to disentangle the contribution of individual ads deployed in each channel from the overall change in sales. [START_REF] Pike-Burke | Bandits with delayed, aggregated anonymous feedback[END_REF] formalized this harder delayed setting in a bandit framework with stochastic rewards, where they introduced the notion of delayed anonymous feedback to emphasize the fact that the reward received at any point in time is the sum of rewards of an unknown subset of past selected actions. More specifically, choosing action I t ∈ [K] at time t generates a stochastic reward Y t (I t ) ∈ [0, 1] and a stochastic delay τ t ∈ {0, 1, . . . }, where Y t (i), τ t i∈ [K],t∈N is a family of independent random variables such that Y 1 (i), Y 2 (i), . . . have a common distribution ν Y (i) (for all arms i ∈ [K]) and τ 1 , τ 2 , . . . have a common distribution ν τ with expectation µ τ . The delayed anonymous feedback assumption entails that the reward observed at time t by the algorithm is the sum of t components of the form Y s (I s )I{τ s = t-s} for s ∈ {1, . . . , t}. The main result in [START_REF] Pike-Burke | Bandits with delayed, aggregated anonymous feedback[END_REF] is that, when the expected delay µ τ is known, the regret is at most of order of K (ln T )/∆ + µ τ , where ∆ is the suboptimality gap. This bound is of the same order as the corresponding bound for the setting where the feedback is stochastically delayed, but not anonymous [START_REF] Joulani | Online learning under delayed feedback[END_REF], and cannot be improved in general.

In this work, we study a bandit setting similar to delayed anonymous feedback, but with two important differences. First, we work in a nonstochastic bandit setting, where rewards (or losses, in our case) are generated by some unspecified deterministic mechanism. Second, we relax the assumption that the loss of an action is charged to the player at a single instant in the future. More precisely, we assume that the loss for choosing an action at time t is adversarially spread over at most d + 1 consecutive time steps t, t + 1, . . . , t + d. Hence, the loss observed by the player at time t is a composite loss, that is, the sum of (d + 1)-many loss components ℓ (0)

t (I t ), ℓ (1) t-1 (I t-1 ), . . . , ℓ (d) t-d (I t-d ), where ℓ (s)
t-s (I t-s ) defines the s-th loss component from the selection of action I t-s at time t-s. Note that in the special case when ℓ (s) t (i) = 0 for all s = d t , and ℓ (dt) t (i) = ℓ t (i), we recover the model of nonstochastic bandits with delays d 1 , d 2 , • • • ≤ d (which, in particular, reduces to the standard nonstochastic bandits when d = 0). Our setting, which we call composite anonymous feedback, can accomodate scenarios where actions have a lasting effect which combines additively over time. Online businesses provide several use cases for this setting. For instance, an impression that results in an immediate clickthrough, later followed by a conversion, or a user that interacts with a recommended item -such as media content-multiple times over several days, or the free credit assigned to a user of a gambling platform which might not be used all at once.

Our main contribution is a general reduction technique (Composite Loss Wrapper, or CoLoWrapper, Algorithm 1) turning a base nonstochastic bandit algorithm into one operating within the composite anonymous feedback setting. We then show that the regret of CoLoWrapper can be upper bounded in terms of the stability and the regret of the base algorithm (Theorem 4). Choosing as a base algorithm Follow the Regularized Leader (FTRL) with Tsallis entropy, Theorem 4 gives an upper bound of order (d + 1)KT on the regret of nonstochastic bandits with composite anonymous feedback (Corollary 7), where d ≥ 0 is a known upper bound on the delay, K is the number of actions, and T is the time horizon. This result relies on a nontrivial stability analysis of FTRL with Tsallis entropy that could be of independent interest (Theorem 5). Finally, we show the optimality of the (d + 1)KT rate by proving a matching lower bound (up to a logarithmic factor, Theorem 10). In particular, this shows that, in the nonstochastic case with delay d, anonymous feedback is strictly harder than nonanonymous feedback, whose minimax regret was characterized by [START_REF] Cesa-Bianchi | Delay and cooperation in nonstochastic bandits[END_REF] as (d + K)T . See the table below for a summary of results for nonstochastic K-armed bandits (all rates are optimal ignoring logarithmic factors).

no delay delayed feedback anonymous composite feedback √ KT (d + K)T (d + 1)KT [START_REF] Auer | The nonstochastic multiarmed bandit problem[END_REF][START_REF] Cesa-Bianchi | Delay and cooperation in nonstochastic bandits[END_REF] (this paper)

We now give an idea of the proof techniques. Similar to [START_REF] Pike-Burke | Bandits with delayed, aggregated anonymous feedback[END_REF], we play the same action for a block of at least 2d + 1 time steps, hence the feedback we get in the last d + 1 steps contains only loss components pertaining to the same action, so that we can estimate in those steps the "true loss" of that action. Unfortunately, although the original losses are in [0, 1], the composite losses can be as large as d + 1 (a composite loss sums d + 1 loss components, and each component can be as large as 1). This causes a corresponding scaling in the regret, compromising optimality. However, we observe that the total composite loss relative to the same action over any d + 1 consecutive steps can be at most 2d + 1 (Lemma 2). Hence, we can normalize the total composite loss relative to the same action over d + 1 consecutive steps, simply dividing by 2d + 1, obtaining an average loss in the range [0, 1]. This idea leads to the right dependence on d in the regret. The last problem is how to avoid suffering a big regret in the first d steps of each block, where the composite losses mix loss components belonging to more than one action. We solve this issue by borrowing an idea of Dekel et al. (2014b). We build blocks with random endpoints so that their length is (always) at least 2d + 1 and (on average) not much bigger. This random positioning and length of the blocks is the key to prevent the oblivious adversary from causing a large regret in the first half of each block. Moreover, as we prove in Theorem 4, if the distribution over actions maintained by the base algorithm is stable (Definition 3), then the algorithm is not significantly affected by the uncertainty in the positioning of the blocks. Extending our results to the case where d is unknown, [START_REF] Wang | Adaptive algorithms for multi-armed bandit with composite and anonymous feedback[END_REF] show a regret bound of order T 2/3 . When d is known, however, their analysis does not guarantee our faster √ T rate.

Further related work Online learning with delayed feedback was studied in the full information (non-bandit) setting by [START_REF] Weinberger | On delayed prediction of individual sequences[END_REF]; [START_REF] Mesterharm | On-line learning with delayed label feedback[END_REF]; [START_REF] Langford | Slow learners are fast[END_REF]; [START_REF] Joulani | Online learning under delayed feedback[END_REF]; [START_REF] Quanrud | Online learning with adversarial delays[END_REF]; [START_REF] Khashabi | Adversarial delays in online strongly-convex optimization[END_REF]; [START_REF] Joulani | Delay-tolerant online convex optimization: Unified analysis and adaptive-gradient algorithms[END_REF]; [START_REF] Garrabrant | Asymptotic convergence in online learning with unbounded delays[END_REF]; [START_REF] Mann | Learning from delayed outcomes via proxies with applications to recommender systems[END_REF], see also [START_REF] Shamir | Online learning with local permutations and delayed feedback[END_REF] for an interesting variant. The bandit setting with delay was investigated in (Neu et al., 2010;[START_REF] Joulani | Online learning under delayed feedback[END_REF][START_REF] Mandel | The queue method: Handling delay, heuristics, prior data, and evaluation in bandits[END_REF][START_REF] Cesa-Bianchi | Delay and cooperation in nonstochastic bandits[END_REF][START_REF] Vernade | Stochastic bandit models for delayed conversions[END_REF][START_REF] Pike-Burke | Bandits with delayed, aggregated anonymous feedback[END_REF][START_REF] Li | Bandit online learning with unknown delays[END_REF][START_REF] Thune | Nonstochastic multiarmed bandits with unrestricted delays[END_REF][START_REF] Zimmert | An optimal algorithm for adversarial bandits with arbitrary delays[END_REF][START_REF] Vernade | Linear bandits with stochastic delayed feedback[END_REF][START_REF] Ito | Delay and cooperation in nonstochastic linear bandits[END_REF][START_REF] Gael | Stochastic bandits with armdependent delays[END_REF][START_REF] Agrawal | Learning by repetition: Stochastic multi-armed bandits under priming effect[END_REF].

Our delayed composite loss function generalizes the composite loss function setting of Dekel et al. (2014a)-see the discussion at the end of Section 2 for details-and is also related to the notion of loss functions with memory. This latter setting has been investigated, e.g., by [START_REF] Arora | Online bandit learning against an adaptive adversary: from regret to policy regret[END_REF], who showed how to turn an online algorithm with regret guarantee of O(T q ) into one attaining O(T 1/(2-q) )-policy regret, also adopting a blocking scheme. A more recent paper in this direction is [START_REF] Anava | Online learning for adversaries with memory: price of past mistakes[END_REF], where the authors considered a more general loss framework than ours, though with the benefit of counterfactual feedback, in that the algorithm is aware of the loss it would incur had it played any sequence of d decisions in the previous d rounds, thereby making their results incomparable to ours.

Preliminaries

We denote the set of positive integers by N and the set of integers by Z. For all n ∈ N we denote the set {1, . . . , n} of the first n integers by [n]. We will use the handy convention that, if (c t ) t∈Z ⊂ R and m, n ∈ Z are such that m > n, then n t=m c t = 0 and n t=m c t = 1. For any x ∈ R, we denote its positive part max{x, 0} by x + .

We start by considering a nonstochastic multiarmed bandit problem on K actions with oblivious losses in which the loss ℓ t (i) ∈ [0, 1] at time t of an action i ∈ [K] is defined by the sum

ℓ t (i) := d s=0 ℓ (s) t (i) of (d + 1)-many components ℓ (s)
t (i) ≥ 0 for s ∈ {0, . . . , d}. Let I t denote the action chosen by the player at the beginning of round t. If I t = i, then the player incurs loss ℓ (0)

t (i) at time t, loss ℓ (1)
t (i) at time t + 1, and so on until time t + d. Yet, what the player observes at time t is only the combined loss incurred at time t, which is the sum

ℓ (0) t (I t ) + ℓ (1) t-1 (I t-1 ) + • • • + ℓ (d) t-d (I t-d )
of the past d + 1 loss contributions, where ℓ (s) t (i) = 0 for all i and s when t ≤ 0. Then, we define the d-delayed composite loss at time t of a sequence of d + 1 actions i t-d , . . . , i t ∈ [K] as

ℓ • t (i t-d , . . . , i t ) := d s=0 ℓ (s) t-s (i t-s ) . (1)
With this notation, the d-delayed composite anonymous feedback assumption states that what the player observes at the end of each round t is only the composite loss ℓ • t (I t-d , . . . , I t ). The goal of the algorithm is to bound its regret R T against the best fixed action in hindsight,

R T := E T t=1 ℓ • t (I t-d , . . . , I t ) -min i∈[K] T t=1 ℓ • t (i, . . . , i) .
We define the regret in terms of the composite losses ℓ • t rather than the true losses ℓ t because in our model ℓ •

t is what the algorithm pays overall on round t. It is easy to see that a bound on R T implies a bound on the more standard notion of regret

E T t=1 ℓ t (I t ) - min k T t=1 ℓ t (k) up to an additive term of at most O(d).
Our setting generalizes the composite loss function setting of Dekel et al. (2014a). Specifically, the linear composite loss function therein can be seen as a special case of the composite loss (1) once we remove the superscripts s from the loss function components. In fact, in the linear case, the feedback in (Dekel et al., 2014a) allows one to easily reconstruct each individual loss component in a recursive manner. This is clearly impossible in our more involved scenario, where the new loss components that are observed in round t need not have occurred in past rounds.

The CoLoWrapper Algorithm

Our Composite Loss Wrapper algorithm (Algorithm 1) takes as input a standard K-armed bandit algorithm A and a Boolean sequence B. The base algorithm A operates on standard (noncomposite) losses with values in [0, 1], producing probability distributions q 1 , q 2 , . . . over the action set [K]. The wrapper calls the base algorithm A only in a subset of rounds determined by the Boolean sequence B, which we call update rounds.

Definition 1 (Update round) We say that t ∈ N is an update round with respect to a Boolean sequence

B = (b t ) t∈N ⊂ {0, 1} N if t ≥ 2d + 1 and b t 2d s=1 (1 -b t-s ) = 1. Note that if d > 0, the condition is equivalent to b t = 1, and b t-1 = . . . = b t-2d = 0. If d = 0, by our convention, the condition is equivalent to b t = 1.
To help understand our algorithm, we will also define two other types of rounds. We say that t is a draw round if t = 1 or the previous round t -1 was an update round. If t is not a draw round, we say that it is a stay round. Note that, if d = 0, both draw and stay rounds can be update rounds, while if d ≥ 1, only stay rounds can be update rounds.

The CoLoWrapper algorithm proceeds in blocks of (random) length of at least 2d + 1 rounds in which it constantly plays the same action (Figure 1). Blocks in Algorithm 1 are counted by variable n t . Each block n t consists of a draw round followed by (2d or more) stay rounds, with the last round of the block being also an update round. During a draw round t, CoLoWrapper uses its current distribution p t to draw and play an action I t . During stay rounds, it keeps playing the action that was drawn during the latest draw round. After playing the action I t for the current round t, if t is an update round, CoLoWrapper asks the base algorithm A to make an update of its base distribution q nt → q nt+1 as if A played action I t and observed as the loss of I t the quantity The bottom line shows the corresponding types of rounds: each block begins with a (D)raw round, followed by a variable number of (S)tay rounds, the last of which is also an (U )pdate round. Since a round t is an update round only if B t = 1 and B s = 0 for the 2d previous rounds s, the length of each block is at least 2d + 1.

1 2d+1 t s=t-d ℓ • s (I s-d , . . . , I s ). Then, ≥ 2d + 1 ≥ 2d + 1 ≥ 2d + 1 B round 1 D 0 S 1 S 0 S 0 S 0 S 0 S 1 SU 0 D 0 S 1 S 0 S 0 S 0 S 0 S 0 S 0 S 1 SU 0 D 0 S 0 S 0 S 1 SU
Algorithm 1: CoLoWrapper (Composite Loss Wrapper) input: Base K-armed bandit algorithm A and Boolean sequence B initialize: let n 0 = 0 and q 1 be the initial distribution over [K] of A 1 for round t = 1, 2, . . . do 2 if either t = 1 or t -1 was an update round (w.r.t. B) then use the update rule q nt → q nt+1 of A to obtain a new base distribution q nt+1 the block ends and the distribution of CoLoWrapper at the beginning of the next block n t+1 = n t + 1 is p t+1 = q nt+1 . Note that if t is an update round, the quantity

3 let n t = n t-1 + 1, p t = q nt ,
1 2d+1 t s=t-d ℓ • s (I s-d , .
. . , I s ) that is fed back to A relates only to the current action I t , because blocks contain at least 2d + 1 rounds and the same action is played in all of them.

The following lemma shows that this quantity is indeed in [0, 1], so that it is a legitimate feedback to pass to the base algorithm A.

Lemma 2 For all t ≥ 2d + 1 and i ∈

[K], t τ =t-d ℓ • τ (i, . . . , i) ≤ 2d + 1 . Proof For all t ≥ 2d + 1 and i ∈ [K], t τ =t-d ℓ • τ (i, . . . , i) = t τ =t-d d s=0 ℓ (s) τ -s (i) = d s=0 t τ =t-d ℓ (s) τ -s (i) = d s=0 t-s ρ=t-d-s ℓ (s) ρ (i) ≤ d s=0 t ρ=t-2d ℓ (s) ρ (i) = t ρ=t-2d d s=0 ℓ (s) ρ (i) = t ρ=t-2d ℓ ρ (i) ≤ t -(t -2d) + 1 = 2d + 1 .
As a final remark, we point out that, albeit the algorithm is parameterized with an entire sequence B, at each time t, it does not require the knowledge of the sequence at future times t + 1, t + 2, . . . . This implies in particular that these Boolean values could be produced and fed to CoLoWrapper in an on-line fashion.

Upper Bound

We begin by formalizing the notion of stability (of the base algorithm), in terms of which we express the performance of the CoLoWrapper algorithm.

Definition 3 (ξ-stability) Let ξ > 0, A be a K-armed bandit algorithm, and (q n ) n∈N be the (random) sequence of probability distributions over actions [K] produced by A during a run over rounds {1, 2, . . . }. We say that A is ξ-stable if for any round n, we have

E   i∈[K] q n+1 (i) -q n (i) +   ≤ ξ .
We can now state our main result of this section.

Theorem 4 If we run CoLoWrapper with a ξ-stable base K-armed bandit algorithm A and an i.i.d. sequence B = (B t ) t∈N of Bernoulli random variables with bias β ∈ (0, 1) (independent of the randomization of A), then, for any time horizon T ≥ 2d + 1, the regret R T satisfies

R T ≤ 2d + 2d + 1 d + 1 3d + 2dβ(1 -β) 2d ξT + 1 β(1 -β) 2d R ⌊T /(2d+1)⌋
where R ⌊T /(2d+1)⌋ is the worst-case regret after T /(2d+ 1) rounds of A (for an adversarial setting with [0, 1]-valued losses).

Proof Fix an arbitrary horizon T ≥ 2d + 1 and an arm i

⋆ ∈ [K]. Let, for all 1 t ≥ 2d + 1, c t := E ℓ • t (I t-d , . . . , I t ) -ℓ • t (i ⋆ , . . . , i ⋆ ) , a := 2d + 1 , b := T .
Applying the elementary identity in Lemma 11 (Appendix A) in step ( * ) below, we obtain

R T = 2d t=1 c t + T t=2d+1 c t ≤ 2d t=1 E ℓ t (I t ) + T t=2d+1 c t ≤ 2d + T t=2d+1
= 2d + 1 d + 1 a-1 t=a-d (t -a + d + 1) c t + (d + 1) b t=a c t + b+d t=b+1 (b + d + 1 -t) c t - 1 d + 1 a-1 t=a-d (t -a + d + 1) c t - 1 d + 1 b+d t=b+1 (b + d + 1 -t) c t ( * ) = 2d + 1 d + 1 b+d τ =a τ t=τ -d c t - 1 d + 1 a-1 t=a-d (t -a + d + 1)c t - 1 d + 1 b+d t=b+1 (b + d + 1 -t)c t ≤ 2d + 1 d + 1 b+d τ =a τ t=τ -d c t + 1 d + 1 a-1 t=a-d (t -a + d + 1)ℓ • t (i ⋆ , . . . , i ⋆ ) + 1 d + 1 b+d t=b+1 (b + d + 1 -t)ℓ • t (i ⋆ , . . . , i ⋆ ) =: ( )
Now, applying Lemma 2 in steps (•) below, we get ( )

(•) ≤ 2d + 2 2d + 1 d + 1 d + 1 d + 1 b+d τ =a τ t=τ -d c t = 2d + 2 2d + 1 d + 1 d + 1 d + 1 T +d τ =2d+1 τ t=τ -d E ℓ • t (I t-d , . . . , I t ) -ℓ • t (I τ -2d , . . . , I τ -2d ) + 2d + 1 d + 1 E T +d τ =2d+1 1 2d + 1 τ t=τ -d ℓ • t (I τ -2d , . . . , I τ -2d ) -ℓ • t (i ⋆ , . . . , i ⋆ ) (•) ≤ 2d + 3 2d + 1 d + 1 d + 1 d + 1 T +d τ =2d+1 τ t=τ -d E ℓ • t (I t-d , . . . , I t ) -ℓ • t (I τ -2d , . . . , I τ -2d ) + 2d + 1 d + 1 E T τ =2d+1 1 2d + 1 τ t=τ -d ℓ • t (I τ -2d , . . . , I τ -2d ) -ℓ • t (i ⋆ , . . . , i ⋆ ) =: 2d + 3 2d + 1 d + 1 d + (I) + 2d + 1 d + 1 × (II) .
We upper bound the two terms (I) and (II) separately. First, let U be the (random) set of update rounds U := τ ∈ [T ] : τ is an update round (w.r.t. B) .

For the first term (I), we have

(I) = 1 d + 1 T +d τ =2d+1 i∈[K] E τ t=τ -d d s=0 ℓ (s) t-s (i) p t-s (i) -p τ -2d (i) ≤ 1 d + 1 T +d τ =2d+1 i∈[K] E max t∈{τ -d,...,τ },s∈{0,...,d} p t-s (i) -p τ -2d (i) τ t=τ -d ℓ • t (i, . . . , i) (•) ≤ 2d + 1 d + 1 T +d τ =2d+1 i∈[K]
E max t∈{τ -d,...,τ },s∈{0,...,d} p t-s (i)p τ -2d (i)

(♥) = 2d + 1 d + 1 T +d τ =2d+1 i∈[K] E I τ -1 σ=τ -2d {σ ∈ U } p τ (i) -p τ -2d (i) + = 2d + 1 d + 1 T +d τ =2d+1 i∈[K] τ -1 σ=τ -2d E I{σ ∈ U } p τ (i) -p τ -2d (i) + =: (♦)
where (•) follows by Lemma 2 together with the fact that the max in the previous line is always nonnegative (to see this, simply observe that picking t = τ -d and s = d within the max makes p t-s (i) = p τ -2d (i)), and (♥) follows by the facts that the max in the previous line is always greater than or equal to zero, it can be strictly positive only if there is an update in a round σ with τ -2d ≤ σ ≤ τ -1, and there can be at most a single update in 2d + 1 consecutive time steps. Now, using the facts that p τ = q nτ and that n τ = n τ -2d + 1 on the event {σ ∈ U }, we get

(♦) = 2d + 1 d + 1 T +d τ =2d+1 i∈[K] τ -1 σ=τ -2d E I{σ ∈ U } q n τ -2d +1 (i) -q n τ -2d (i) + = 2d + 1 d + 1 T +d τ =2d+1 i∈[K] τ -1 σ=τ -2d n∈N E I{σ ∈ U }I{n τ -2d = n} q n+1 (i) -q n (i) + (♣) = 2d + 1 d + 1 T +d τ =2d+1 τ -1 σ=τ -2d n∈N E I{σ ∈ U }I{n τ -2d = n} E   i∈[K]
q n+1 (i)q n (i)

+   (♠) ≤ 2d + 1 d + 1 ξ T +d τ =2d+1 τ -1 σ=τ -2d n∈N E I{σ ∈ U }I{n τ -2d = n} = 2d + 1 d + 1 ξ T +d τ =2d+1 τ -1 σ=τ -2d P[σ ∈ U ] ≤ 2d + 1 d + 1 2dβ(1 -β) 2d ξT
where (♣) follows by the independence of the Bernoulli sequence B and the randomization of base algorithm A; (♠) by the ξ-stability of A; and the last inequality by the fact that the probability that a time step σ is an update round is 0 if σ ≤ 2d and β(1 -β) 2d otherwise.

For the second term (II), set for brevity

r τ : [K] → R , i → 1 2d + 1 τ t=τ -d ℓ • t (i, . . . , i) -ℓ • t (i ⋆ , . . . , i ⋆ )
for all τ ≥ 2d + 1. We first note that

E τ ∈U r τ (I τ ) = E τ ∈U r τ (I τ -2d ) = E T τ =2d+1 r τ (I τ -2d )B τ 2d s=1 (1 -B τ -s ) = β(1 -β) 2d T τ =2d+1 E r τ (I τ -2d )
where the first identity is a consequence of the fact that if τ ∈ U , then the action I τ played by Algorithm 1 at round τ coincides with the actions played in the previous 2d rounds, i.e., I τ = • • • = I τ -2d , while the last equality follows by the independence of I τ -2d and the vector of Bernoulli random variables (B τ , . . . , B τ -2d ). Thus,

(II) = E T τ =2d+1 r τ (I τ -2d ) = 1 β(1 -β) 2d E τ ∈U r τ (I τ ) = 1 β(1 -β) 2d E τ ∈U 1 2d + 1 τ t=τ -d ℓ • t (I τ , . . . , I τ ) - τ ∈U 1 2d + 1 τ t=τ -d ℓ • t (i ⋆ , . . . , i ⋆ ) (2)
Now define for any τ ≥ 2d + 1,

lτ : [K] → [0, 1] , i → 1 2d + 1 τ t=τ -d ℓ • t (i, . . . , i) . (Note that lτ (i) ∈ [0, 1] for all i ∈ [K] by Lemma 2.) Leveraging again the fact that τ ∈ U implies I τ = • • • = I τ -2d , yields, for each τ ∈ U , 1 2d + 1 τ t=τ -d ℓ • t (I t-d , . . . , I t ) = 1 2d + 1 τ t=τ -d ℓ • t (I τ , . . . , I τ ) = lτ (I τ ) .
This shows that the loss Algorithm 1 feeds the base algorithm A (in line 8) at each update round τ is a bandit feedback (for A) for the arm I τ with respect to the [0, 1]-valued loss lτ . Therefore, by Equation ( 2) and the regret guarantees of the base algorithm, we obtain

(II) = 1 β(1 -β) 2d E τ ∈U lτ (I τ ) - τ ∈U lτ (i ⋆ ) ≤ 1 β(1 -β) 2d R ⌊T /(2d+1)⌋
where in the last inequality we also used the monotonicity of the worst-case regret τ → R τ and the fact that

|U | ≤ T -(2d+1)+1 2d+1 ≤ T 2d+1 .
In conclusion, we have

R T ≤ 2d + 3 2d + 1 d + 1 d + (I) + 2d + 1 d + 1 • (II) ≤ 2d + 2d + 1 d + 1 3d + 2dβ(1 -β) 2d ξT + 1 β(1 -β) 2d R ⌊T /(2d+1)⌋
hence concluding the proof.

We can derive corollaries for various algorithms using Theorem 4. Consider for instance as a base algorithm A the well-known Exp3 algorithm of [START_REF] Auer | The nonstochastic multiarmed bandit problem[END_REF]. It can be easily shown that Exp3 is η-stable (where η is its learning rate, see Lemma 14 in Appendix C). Combining this fact with its worst-case regret bound R N ≤ ln K η + η 2 KN , we obtain that the horizon-T regret of the CoLoWrapper algorithm with A = Exp3(η), η = d ln K KT , and an i.i.d. sequence B of Bernoulli random variables with bias β = 1 2d+1 (independent of the randomization of A), satisfies R T = O (d + 1)KT log K .

Using Follow the Regularized Leader (FTRL) with 1 2 -Tsallis Entropy (Algorithm 2), we can remove the log K term in the above bound. 2

Algorithm 2: Follow The Regularized Leader (FTRL) with ( 1 /2)-Tsallis entropy input: learning rate η ≥ 0 initialize: L 0 = 0 1 for round n = 1, 2, . . . do 2 Play action J n drawn according to

q n ∈ argmin q∈∆ K   i∈[K] L n-1 (i)q(i) -2η i∈[K] q(i)   where ∆ K is the probability simplex in R K 3 Observe loss ℓ n (J n ) and update L n = L n-1 + ℓ n , where ℓ n (i) = ℓ n (i) q n (i) I{J n = i} ∀i ∈ [K]
However, we still need to prove the stability of Algorithm 2. This is established by the following result, whose (non-trivial) proof is given in Appendix B.

Theorem 5 Algorithm 2 run with any learning rate η > 0 is ξ-stable, with

ξ ≤ 2 1 + ln K η
The worst-case regret guarantees of FTRL with 1 2 -Tsallis entropy are as follows. 3

Theorem 6 (Abernethy et al. 2015) For each N ∈ N, the worst-case regret R N after N rounds of Algorithm 2 run with learning rate η = N 2 (in an adversarial setting with

[0, 1]-valued losses) satisfies R N ≤ 2 √ 2KN
Combining Theorems 4 to 6, we obtain the following regret bound for composite losses.

2. The argmin where Algorithm 2 picks each distribution q n is always a singleton if η > 0. This is a consequence of Lemma 12, in Appendix B. For the sake of convenience, we also allow the learning rate η = 0, corresponding to a (non-regularized) Follow The Leader algorithm. In this case, multiple minimizers could exist but, for the sake of our results, ties could be broken in any (measurable) way. 3. The analysis of Abernethy et al. (2015) is presented for a more general class of algorithms. A straightforward application of their Corollary 3.2 shows the validity of Theorem 6 for Algorithm 2.

Corollary 7 For any time horizon T ∈ N, if we run CoLoWrapper using:

• As A, Algorithm 2 with learning rate η = 1 2 ⌊T /(2d + 1)⌋ • As B, an i.i.d. sequence of Bernoulli random variables with bias β = 1 2d+1 (independent of the randomization of A) then, its regret satisfies R T ≤ c (d + 1)KT where c = 2 √ 2 if d = 0, and c = 28 otherwise.

Proof If d = 0, then β = 1 and Algorithm 1 reduces to the base algorithm. The result in this case is therefore implied immediately by Theorem 6.

For the second part, we can assume without loss of generality that T ≥ 2d + 1, so that η > 0. Then, plugging ξ ≤ 2 1+ln K η (Theorem 5) and R ⌊T /(2d+1)⌋ ≤ 2 2K ⌊T /(2d + 1)⌋ (Theorem 6) into Theorem 4 gives the result.

Lower bound

In this section we derive a lower bound for bandits with composite anonymous feedback. We do that through a reduction from the setting of linear bandits (in the probability simplex) to our setting. This reduction allows us to upper bound the regret of a linear bandit algorithm in terms of (a suitably scaled version of) the regret of an algorithm in our setting. Since the reduction applies to any instance of a linear bandit problem, we can use a known lower bound for the linear bandit setting to derive a corresponding lower bound for our composite setting.

Let ∆ K be the probability simplex in R K . At each round t, an algorithm A for linear bandit optimization chooses an action q t ∈ ∆ K and suffers loss ℓ ⊤ t q t , where ℓ t ∈ [0, 1] K is some unknown loss vector. The feedback observed by the algorithm at the end of round t is the scalar ℓ ⊤ t q t . The regret suffered by algorithm A playing actions q 1 , . . . , q T is

R lin T = T t=1 ℓ ⊤ t q t -min q∈∆ K T t=1 ℓ ⊤ t q = T t=1 ℓ ⊤ t q t -min i=1,...,K T t=1 ℓ ⊤ e i (3)
where e 1 , . . . , e K are the elements of the canonical basis of R K and we used the fact that a linear function on the simplex is minimized at one of the corners. Let R lin T (A, ∆ K ) denote the worst case regret (over the oblivious choice of ℓ 1 , . . . , ℓ T ) of algorithm A. Similarly, let R T (A d , K, d) be the worst case regret (over the oblivious choice of loss components ℓ (s) t (i) for all t, s, and i) of algorithm A d for nonstochastic K-armed bandits with d-delayed composite anonymous feedback. For the sake of clarity, we assume below that the time horizon T is a multiple of d + 1. If this is not the case, we can straightforwardly stop at the highest multiple of d + 1 (smaller than T ) up to paying an additive O(d + 1) regret. Our reduction shows the following.

Lemma 8 For any algorithm A d for K-armed bandits with d-delayed composite anonymous feedback, there exists an algorithm A for linear bandits in

∆ K such that R T (A d , K, d) ≥ (d + 1) R lin T /(d+1) (A, ∆ K ).
Our reduction, described in detail in the proof of the above lemma (see Appendix D), essentially builds the probability vectors q t played by A based on the empirical distribution of actions played by A d during blocks of size d + 1. Now, an additional lemma is needed (whose proof is also given in the Appendix D).

Lemma 9 The regret of any algorithm A for linear bandits on

∆ K satisfies R lin T (A, ∆ K ) = Ω √ KT .
In the previous lemma, as well as the following result, the Ω notation is only hiding a √ log T denominator. Using the two lemmas above we can finally prove the lower bound.

Theorem 10 For any algorithm A d for K-armed bandits with d-delayed composite anonymous feedback, R T (A d , K, d) = Ω (d + 1)KT .

Proof Fix an algorithm A d . Using the reduction of Lemma 8 gives an algorithm A such that R T (A d , K, d) ≥ (d + 1) R lin T /(d+1) (A, ∆ K ) = Ω (d + 1)KT , where we used Lemma 9 with horizon T /(d + 1) to prove the Ω-equality.

Although the loss sequence used to prove the lower bound for linear bandits in the simplex is stochastic i.i.d., the loss sequence achieving the lower bound in our delayed setting is not independent due to the deterministic loss transformation in the proof of Lemma 8 (which is defined independently of the algorithm, thus preserving the oblivious nature of the adversary).

Conclusions

We have investigated the setting of d-delayed composite anonymous feedback as applied to nonstochastic bandits. Composite anonymous feedback lends itself to formalize scenarios where the actions performed by the online decision-maker produce long-lasting effects that combine additively over time. A general reduction technique was introduced that enables the conversion of a stable algorithm working in a standard bandit framework into one working in the composite feedback framework. Applying our reduction to the FTRL algorithm with Tsallis entropy, we obtain an upper bounded on the regret of order (d + 1)KT . We then showed the optimality of this rate (up to a logarithmic factor) relying on a lower bound for bandit linear optimization in the probability simplex. Furthermore, defining for all i ∈ [K],

q opt w,η (i) := η 2 w(i) -λ w,η 2 ,
we have that q opt w,η is the unique global minimizer of the function:

∆ K → R , q → i∈[K] w(i)q(i) -2η i∈[K]
q(i) .

Proof Define the two auxiliary functions

f w,η : [0, ∞) K → R , q → i∈[K] w(i)q(i) -2η i∈[K] q(i) ϕ : [0, ∞) K → R , q → i∈[K]
q(i)

Note that f w,η is strictly convex and continuous on [0, ∞) K and differentiable on (0, ∞) K . Thus, by the Lagrange multiplier theorem, a point q ∈ (0, ∞) K is the unique global minimizer of f w,η on the simplex ∆ K if and only if

ϕ(q) = 1 and ∃λ ∈ R, ∇f w,η (q) = λ∇ϕ(q) (4)
A direct verification shows that a pair (q, λ) ∈ (0, ∞) K × R satisfies condition (4) if and only if

λ < min i∈[K] w(i) , i∈[K] η 2 w(i) -λ 2 = 1 , and ∀i ∈ [K] , q(i) = η 2 w(i) -λ 2 (5)
Note that, letting m := min i∈[K] w(i), the function

g : (-∞, m) → R , λ → i∈[K] η 2 w(i) -λ 2
is continuous, strictly increasing, and it satisfies lim λ→-∞ g(λ) = 0 and lim λ→m - g(λ) = +∞ hence, there exists a unique λ w,η ∈ (-∞, m) such that g(λ w,η ) = 1. Therefore, q opt w,η is the unique global minimizer of f w,η on the simplex ∆ K . This lemma controls the variation of the Lagrange multipliers corresponding to two points that vary by quantity δ only in a single coordinate.

Lemma 13 Let w ∈ R K such that w(1) ≤ • • • ≤ w(K) and η > 0. Then, for all δ > 0 and j

∈ [K], λ w+δe j ,η -λ w,η ≤ 1 j δ ,
where λ w+δe j ,η and λ w,η are defined as in Lemma 12 and e 1 , . . . , e K is the canonical basis

of R K . Proof Define X := u(1), . . . , u(K), λ ∈ R K × R | λ < min i∈[K] u(i) and ψ : X → R , u(1), . . . , u(K), λ → i∈[K] η 2 u(i) -λ 2 Λ : R K → R , u → λ u,η
where λ u,η is defined as in Lemma 12. By the implicit function theorem, we have that Λ is infinitely differentiable and for all u ∈ R K ,

D j Λ(u) = - D j ψ u(1), . . . , u(K), λ u,η D K+1 ψ u(1), . . . , u(K), λ u,η = - -2 η 2 (u(j)-λu,η) 3 2 i∈[K] η 2 (u(i)-λu,η) 3 = 1 1 + i∈[K],i =j u(j)-λu,η u(i)-λu,η 3 ,
where we denoted the partial derivative with respect to the j-th coordinate by D j . Now, by the fundamental theorem of calculus,

λ w+δe j ,η -λ w,η = Λ(w + δe j ) -Λ(w) = δ 1 0 D j Λ(w + sδe j ) ds = δ 1 0 1 1 + i∈[K],i≤j-1 w(j)+sδ-λ w+sδe j ,η w(i)-λ w+sδe j ,η 3 + i∈[K],i≥j+1 w(j)+sδ-λ w+sδe j ,η w(i)-λ w+sδe j ,η 3 ds ≤ δ 1 0 1 1 + i∈[K],i≤j-1 w(i)-λ w+sδe j ,η w(i)-λ w+sδe j ,η 3 ds = 1 j δ .
We can finally prove the stability of FTRL with 1 2 -Tsallis entropy. Proof of Theorem 5 Consider an arbitrary sequence of losses (ℓ m ) m∈N ⊂ [0, 1]. Fix any η > 0. For each w ∈ R K , let λ w := λ w,η , where λ w,η is defined as in Lemma 12. Let F 0 be the trivial σ-algebra (containing only the sample space and the empty set) and for all n ∈ N, F n := σ(J 1 , . . . J n ). Note that, for each n ∈ N, we have that L n-1 is F n-1 -measurable and, as a consequence of Lemma 12, for all i ∈ [K],

q n (i) = η 2 L n-1 (i) -λ L n-1 2 .
Let e 1 , . . . , e K be the canonical basis of R K and fix any n ∈ N. Define for each j ∈ [K], ℓ n,j := ℓn(j) q n (j) e j and note that ℓ n = ℓ n,Jn . To make the notation more compact, we also let

E n [•] := E[• | F n-1 ]. Then E n   i∈[K] q n+1 (i) -q n (i) +   = E n   i∈[K]   η 2 L n (i) -λ Ln 2 - η 2 L n-1 (i) -λ L n-1 2   +   = E n   i∈[K]   η 2 L n-1 (i) + ℓ n,Jn (i) -λ L n-1 + ℓ n,Jn 2 - η 2 L n-1 (i) -λ L n-1 2   +   = j∈[K] E n   I{J n = j} i∈[K]   η 2 L n-1 (i) + ℓ n,j (i) -λ L n-1 + ℓ n,j 2 - η 2 L n-1 (i) -λ L n-1 2   +   = j∈[K] E n I{J n = j} i∈[K]   η 2 L n-1 (i) + ℓ n,j (i) -λ L n-1 + ℓ n,j 2 - η 2 L n-1 (i) -λ L n-1 2   + = j∈[K] q n (j) i∈[K]   η 2 L n-1 (i) + ℓ n,j (i) -λ L n-1 + ℓ n,j 2 - η 2 L n-1 (i) -λ L n-1 2   + = (⋆) .
Now, note that, for each j ∈ [K]:

• λ L n-1 + ℓ n,j ≥ λ L n-1 (because, as we show in the proof of Lemma 13, the directional derivatives of u → λ u are positive).

• For each i ∈ [K]\{j},

η 2 L n-1 (i)-λ L n-1 ≤ η 2 L n-1 (i)-λ L n-1 + ℓ n,j = η 2 L n-1 (i)+ ℓ n,j (i)-λ L n-1 + ℓ n,j
(by the previous point).

• η 2 L n-1 (j)-λ L n-1 ≥ η 2
L n-1 (j)+ ℓ n,j (j)-λ L n-1 + ℓ n,j (by the previous point and the fact that

i∈[K] η 2 L n-1 (i)-λ L n-1 = 1 = i∈[K]
η 2 L n-1 (i)+ ℓ n,j (i)-λ L n-1 + ℓ n,j

).

It follows that (⋆) = j∈[K]

q n (j)

i∈[K],i =j   η 2 L n-1 (i) -λ L n-1 + ℓ n,j 2 - η 2 L n-1 (i) -λ L n-1 2   = η 2 j∈[K]
q n (j)

i∈[K],i =j L n-1 (i) -λ L n-1 2 -L n-1 (i) -λ L n-1 + ℓ n,j 2 L n-1 (i) -λ L n-1 + ℓ n,j 2 L n-1 (i) -λ L n-1 2 = η 2 j∈[K]
q n (j)(λ L n-1 + ℓ n,j -λ L n-1 )

i∈[K],i =j L n-1 (i) -λ L n-1 + L n-1 (i) -λ L n-1 + ℓ n,j L n-1 (i) -λ L n-1 + ℓ n,j 2 L n-1 (i) -λ L n-1 2 ≤ 2η 2 j∈[K]
q n (j)(λ L n-1 + ℓ n,j -λ L n-1 )

i∈[K],i =j 1 L n-1 (i) -λ L n-1 + ℓ n,j 3 ≤ 2η 2 j∈[K]
q n (j)(λ L n-1 + ℓ n,j -λ L n-1 )

  i∈[K],i =j 1 L n-1 (i) -λ L n-1 + ℓ n,j 2   3/2 = 2η 2 j∈[K]
q n (j)(λ L n-1 + ℓ n,j -λ L n-1 )

  i∈[K],i =j 1 L n-1 (i) + ℓ n,j (i) -λ L n-1 + ℓ n,j 2   3/2 ≤ 2η 2 j∈[K]
q n (j)(λ L n-1 + ℓ n,j -λ L n-1 )

  i∈[K] 1 L n-1 (i) + ℓ n,j (i) -λ L n-1 + ℓ n,j 2   3/2 = 2 η j∈[K]
q n (j)(λ L n-1 + ℓ n,j -λ L n-1 ) = (⋆⋆) .

Now, let σ be a random permutation of [K] such that L n-1 σ(1) ≤ • • • ≤ L n-1 σ(K) . Then, using Lemma 13, we have

(⋆⋆) = 2 η j∈[K]
q n σ(j) (λ

L n-1 + ℓ n,σ(j) -λ L n-1 ) = 2 η j∈[K]
q n σ(j) λ L n-1 + ℓn(σ(j))

q n (σ(j)) e σ(j)

-λ L n-1 ≤ 2 η j∈[K]
q n σ(j) 1 j ℓ n (σ(j)) q n (σ(j))

≤ 2 η j∈[K] 1 j ≤ 2 1 + ln K η .
In conclusion:

E   i∈[K]
q n+1 (i)q n (i)

+ | F n-1   ≤ 2 1 + ln K η .
It follows that:

E   i∈[K]
q n+1 (i)q n (i)

+   = E   E   i∈[K]
q n+1 (i)q n (i)

+ | F n-1     ≤ 2 1 + ln K η .
Being n and (ℓ m ) m∈N arbitrary, the result follows.

This concludes the proof.

We now prove a lower bound for linear bandits.

Proof of Lemma 9

The statement is essentially proven in (Shamir, 2015, Theorem 5), where the author shows a Ω K/T lower bound on the error of bandit linear optimization in the probability simplex. 4 As explained in [START_REF] Shamir | On the complexity of bandit linear optimization[END_REF], Section 1.1), (cumulative) regret lower bounds for linear bandits can be obtained by multiplying the lower bounds on bandit linear optimization error by T . A possible issue is that the proof in (Shamir, 2015, Theorem 5) uses unbounded Gaussian losses. However, in (Shamir, 2015, Appendix B) it is shown how lower bounds for Gaussian losses can be converted into lower bounds for losses in [-1, 1] at the cost of a 1 √ ln T factor in the regret. Finally, note that our setting requires losses in [0, 1], but this is not an issue either because we are in a linear setting, and thus we can add the (1, . . . , 1) constant vector to all loss vectors without affecting the regret.

Figure 1 :

 1 Figure 1: Sequence of rounds the algorithm is undergoing when d = 2. The top line contains the values of the Boolean sequence B = (B t ) t∈N .The bottom line shows the corresponding types of rounds: each block begins with a (D)raw round, followed by a variable number of (S)tay rounds, the last of which is also an (U )pdate round. Since a round t is an update round only if B t = 1 and B s = 0 for the 2d previous rounds s, the length of each block is at least 2d + 1.

Figure 2 :

 2 Figure 2: On the right-hand side of the equation in Lemma 11, we are summing each row of the squares in the picture. On the left hand side, we are summing the columns. Each column s contains the same constant c s in each component.

It is worth stressing that the lower bound in[START_REF] Shamir | On the complexity of bandit linear optimization[END_REF] is based on stochastic i.i.d. generation of losses, hence it does not violate our assumption about the obliviousness of the adversary.

Acknowledgments

NCB and RC were partially supported by the MIUR PRIN grant Algorithms, Games, and Digital Markets (ALGADIMAR) and the EU Horizon 2020 ICT-48 research and innovation action under grant agreement 951847, project ELISE (European Learning and Intelligent Systems Excellence). The work of TC has benefited from the AI Interdisciplinary Institute ANITI, which is funded by the French "Investing for the Future -PIA3" program under the Grant agreement ANR-19-P3IA-0004. TC also acknowledges the support of the project BOLD from the French national research agency (ANR). YM was supported in part by the European Research Council (ERC) under the European Union's Horizon 2020 research

Appendix A. An Accountants' Lemma

The next elementary lemma can be proved straightforwardly by swapping the order of the sums (see Figure 2).

Appendix B. Stability of FTRL with Tsallis entropy

In this section, we prove a key stability property of FTRL with Tsallis entropy that could be of independent interest. We begin by showing that, when η > 0, there exists a unique solution of the optimization problem that defines q n in Algorithm 2 and provide a formula for it in terms of the corresponding Lagrange multiplier λ. An analogous result was stated in (Zimmert and Seldin, 2021, Section 3.3) for the related algorithm Tsallis-INF.

Appendix C. Stability of Exp3

In this section, we prove the stability of Exp3.

Lemma 14 Exp3 with learning rate η is ξ-stable with ξ = η.

Proof Consider an arbitrary sequence of losses (ℓ n ) n∈N ⊂ [0, 1]. In this case, stability holds pointwise (for all realizations of the actions J 1 , J 2 , . . . played by Exp3 on the sequence of losses (ℓ n ) n∈N ) rather that in expectation. From (Cesa-Bianchi et al., 2016, Lemma 1) we have, for any round n ∈ N and all arms i ∈ [K],

where ℓ n (j) = ℓn(j)I{Jn=j} q n (j)

, q n (j) = w n (j)/ K k=1 w n (k), and if n = 1, w n (k) = 1 while, if n ≥ 2, w n (k) is defined inductively by w n (k) = q n-1 (k)e -η ℓ n-1 (k) . Hence we can write, for any n ∈ N,

Being (ℓ n ) n∈N arbitrary, the result follows.

Appendix D. Lower Bound (missing proofs)

We begin this section by showing a reduction mapping each algorithm for bandits with composite anonymous feedback to one for linear bandits with a better or equal regret.

Proof of Lemma 8 Fix an instance ℓ 1 , . . . , ℓ T /(d+1) of a linear bandit problem and use it to construct an instance of the d-delayed bandit setting with loss components ℓ (s)

where e 1 , . . . , e K are the elements of the canonical basis of R K . These components define the following composite loss incurred by any algorithm A d playing actions I 1 , I 2 , . . .

where q t is defined from I t-d , . . . , I t ∈ [K] as follows

Note that q t (i) is the fraction of times action i was played by A d in the last d + 1 rounds. Given the algorithm A d , we define the algorithm A for playing linear bandits on the loss sequence ℓ 1 , . . . , ℓ T /(d+1) as follows. If t = 0 (mod d + 1), then A skips the round. On the other hand, when t = 0 (mod d + 1), A performs action q t defined in (6), observes the loss ℓ ⊤ ⌈t/(d+1)⌉ q t , and returns to A d the composite loss ℓ • t (I t-d , . . . , I t ). Essentially, A d observes a nonzero composite loss only every d time steps, when t = 0 (mod d + 1). When this happens, the composite loss of A d is d ℓ ⊤ ⌈t/(d+1)⌉ q t , which is d + 1 times the loss of A. Now it is enough to note that, using (3), min


Jeanne Kergomard

Nathalie Barouh

Gilles Paboeuf

Pierre Villeneuve

Olivier Schafer

Timothy Wooster

V Vié

Claire Bourlieu-Lacanal

Stability to oxidation and interfacial behavior at the air-water interface of minimally-processed versus processed walnut oil-bodies

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

*Corresponding author: Jeanne.kergomard@univ-rennes1.fr

Conclusion:

Altogether, this study unveiled the interesting stability of OB and their specific interfacial reactivity, opening the way to the use of these natural lipoproteic assemblies in various food applications. Nevertheless, their physico-chemical stability during the design process should be taken into consideration. Indeed, the alteration of the physical structure of OB during processing may affect their chemical stability during storage and hence the quality of the final product expected by the consumer. This finding underlines the need to rethink food processing to preserve natural assemblies and their beneficial properties. In tree nuts, OB are remarkably stable lipoprotein assemblies that are rich in essential nutrients (polyunsaturated fatty acids, tocopherols, antioxidant phenolic compounds…).

What are OB?

Results -Interfacial behavior How does the structural changes of oxidized OB affect the behaviour at the interface? The consumption of walnuts is associated with many health benefits, attributed to their fatty acid profiles, which is rich in polyunsaturated fatty acids (PUFA). As a result, walnuts are used in several food products, such as walnut-based beverages, where fat is partly dispersed under the form of natural lipoproteic assemblies, which are called oil bodies (OB). Walnuts OB were stable to oxidation on the short term (few days). This phenomenon is related to the "assembly effect" of OB and to their antioxidant content in vitamins E (41.0 ± 20.7 % wt.). The stability of the OB dispersion was higher under the complex matrix form due to the "matrix effect". (Huang et al., 1994) Conclusion: When MP OB adsorb at an air/water interface, the good cohesive ability of their native assembly allows them to keep intact their microstructure. Oxidation phenomenon modified the physical integrity of the OB, decreasing intermolecular forces, which resulted in a different interfacial organization.

Material and methods

TBARS value

MP OB Complex matrix (Kergomard et al., 2021)

  AND INTERFACIAL BEHAVIOR AT AIR-WATER INTERFACE OF MINIMALLY-PROCESSED VERSUS PROCESSED WALNUT OIL-BODIES Two questions remain pressing: what is walnut OB's oxidative behavior and what are the consequences of oxidation on its interfacial reactivity?

  pressure) → molecular interfacial interactions Δ (ellipsometric angle) → amount of matter at the a/

Fresh Isolated MP OB Oxidized Isolated MP OB Jeanne

  Kergomard 1,2* , Nathalie Barouh 3 , Gilles Paboeuf 2 , Pierre Villeneuve 3 , Olivier Schafer 4 , Tim J. Wooster 4 , Véronique Vié 1 and Claire Bourlieu 2 Montpellier, France ; 3 QUALISUD, Univ Montpellier, CIRAD, Institut Agro, IRD, Univ Réunion, Montpellier, France ; 4 Institute of Materials Science, Nestlé Research, Lausanne, Switzerland

1 IPR Institute of Physics, UMR UR1 CNRS 6251, Rennes 1 University, France ; 2 IATE, Univ Montpellier, INRAE, Institut Agro,


Stéphane Gourmelon
email: stephane.gourmelon@ensiacet.fr

Raphaële Hetreux

Pascal Floquet

Philippe Baudet

Olivier Baudouin

Premises for a combined Exergy and Pinch Optimization within ProSimPlus ® simulator

Keywords: Exergy Analysis, Pinch Technology, Process optimization, Process simulation

The recent increase in fossil fuel prices and the more and more stringent environmental regulations have stimulated the search for further improvements. A short term and sustainable solution consists in improving energy efficiency of industrial processes. To tackle this challenge, a combined exergy and Pinch Analysis has been developed for industrial process' optimization. To summarize the presented methodology, the Exergy Analysis firstly aims at making a diagnosis of the existing process by identifying the major inefficiencies; then based upon this diagnosis; the Pinch Analysis enables to give concrete solutions to improve its energy efficiency. Moreover, the strength of this work relies in the implementation of this Exergy Analysis and Pinch Analysis methodologies in the process simulation software ProSimPlus ® which provides an efficient support to the engineer.

Introduction

Industrial sector accounts for one third of global energy consumption. The recent increase in fossil fuel prices and the more and more stringent environmental regulations have stimulated the search for further improvements. A short term and sustainable solution consists in improving energy efficiency of industrial processes. Process Integration may be used to tackle this challenge (Klemeš et al., 2013). Among the several methodologies of process integration, one can find the Pinch Technology and the Exergy Analysis. While the latter [START_REF] Ghannadzadeh | General methodology for exergy balance in ProSimPlus® process simulator[END_REF] can be applied to identify thermodynamic imperfections (exergy losses) of a given process and to propose some hints to reduce such losses, Pinch Technology (Klemeš and Kravanja, 2013) aims at improving heat-exchanger network of a process, solely dealing with heat transfers, excluding chemical or pressure changes. To overcome these limitations, it seems appropriate to implement a methodology combining Exergy Analysis and Pinch Technology as shown in the literature [START_REF] Marmolejo-Correa | A new procedure for the design of LNG processes by combining Exergy and Pinch Analyses[END_REF].

The purpose of this paper is to present a new general approach for optimizing an industrial process from an exergy point of view as well as an energy efficiency view point. More emphasis is placed on the retrofitting and improvement of existing processes operating under steady-state conditions. The concepts of exergy load, and exergy efficiency are combined to the Pinch methodology. The general procedure for process optimization and improvement is described through a very simple cogeneration case study. This paper is part of the COOPERE ANR project involving several partners (Veolia Environnement Recherche et Innovation, Prosim SA, AgroParisTech' and Laboratoire de Génie Chimique).

Methodology

The Computer-aided Exergy and Pinch Analysis (PA) methodology (CEPA) is primarily based on the ProSimplus ® environment's ability to make Pinch and exergy calculations. Therefore, the studied process needs first to be modeled and simulated in ProSimPlus ® .

Exergy Analysis

The Exergy Analysis, which has been proved to be an efficient tool for the energy diagnosis of industrial processes, especially relies on the computation of internal losses and exergy efficiencies. The Exergy Analysis first enables to perform an energy diagnosis of the existing process by evaluating several efficiencies representing the thermodynamic performance from different points of view. By offering the possibility to make automatic calculation of exergy of material and heat streams, ProSimPlus ® simulation software [START_REF] Prosim | Software and services in Process Simulation[END_REF] facilitates Exergy Analysis of industrial processes [START_REF] Ghannadzadeh | General methodology for exergy balance in ProSimPlus® process simulator[END_REF]. As a reminder, the amount of exergy destroyed, I, also known as irreversibilities or exergetic internal losses, of a system is obtained by a simple exergy balance -Eq (1) -where B in and B out respectively represent the whole exergy entering and leaving the system.

out in B B I - = (1)
An in-depth study of exergy based efficiencies has been undertaken. It highlighted that the intrinsic efficiency -Eq (2) - [START_REF] Sorin | Exergy Flows Analysis in Chemical Reactors[END_REF] is the general way to compute a significant exergy efficiency.

consumed Exergy

produced Exergy = η (2)
The intrinsic efficiency must be combined to the exergy load concept in order to compute an exergetic efficiency of a multiple unit operation system -Eq (3).

The computation of the intrinsic efficiency i η , exergy load coefficients p i λ and p i -1 λ which rely on a general methodology is beyond the scope of the study, thus the reader is recommended to consult references [START_REF] Gourmelon | General procedure for the computing of exergy efficiency : a first step of a complete exergy analysis methodology within ProSimPlus process imulation software[END_REF]. For a given unit operation, it is also possible to define the relative exergy loss as the ratio of exergy loss and exergy consumed -Eq (4). [ ]

∑ = - - - = UO N i i p i i p i 1 1 ) 1 .( . η λ η λ η (3)

Complementarity between Pinch and Exergy Analysis

The analysis of these thermodynamic criteria permits to suggest modifications aimed at reducing irreversibilities (exergy internal losses) and to maximize the reuse of external losses (waste streams). The decomposition of exergy into thermal, mechanical and chemical terms enables to get a clear vision of the potential of reuse of waste streams. The way of using such waste streams depends on the contribution of each term (T, P, z) in the exergy balance. As presented in Figure 1, thermal exergy losses should be recovered as hot or cold streams. For chemical exergy losses, some recycling solutions could be considered such as methanisation or combustion. In the case of high mechanical exergy loss, it might be possible to produce power.

Pinch Analysis provides an efficient tool to generate a heat exchanger network in order to maximize the internal heat load reuse. However it is sometimes difficult to decide which heat exchanger, heat pump… represents the best investment for a more efficient energy use in the industrial process. Finally, calculation of the exergetic efficiency and other exergy based coefficients of the different configurations can help the process manager to make a choice between several solutions.

Case study description

Air entering the cogeneration system (Erreur ! Source du renvoi introuvable.Figure 2) is compressed up to 25 atm and is sent to the combustion chamber. Natural gas (NG), supplied by stream S03, is burnt to produce hot gases. The latters are expanded down to 1 atm and recycle to produce steam (E101). The power produced by the turbine T101 is partly used to fulfill the needs of the compressor C101. The rest is sent to another system.

Figure 2: Heat and power cogeneration system

The natural gas and air flowrate are computing according to equation ( 5). The temperature of steams leaving the evaporator E101 is equal to 773.15 K. Stream S06 is assumed to be a waste stream.

Air Gas Natural F F . 047 . 0 =
(5)

Heat and power cogeneration system analysis

Exergy analysis

The first step of the methodology consists of proceeding to an Exergy Analysis of the studied process. Exergy balance is reported in Table 1 and Figure 3 summarizes the results of the exergy analysis by detailing the irreversibility and the exergy efficiency on unit operations. The overall efficiency is 55.22 %. As shown in Figure 3, even though the exergy destroyed represents about 30 % of the exergy consumed by the combustion chamber, chemical reaction is the most significant source of exergy destruction in the cogeneration system. As noticed by Cziesla et al. [START_REF] Cziesla | Avoidable thermodynamic inefficiencies and costs in an externally fired combined cycle power plant[END_REF], the thermodynamic inefficiencies may be reduced by preheating the air or increasing the fuel/air ratio. Consequently, a heat exchanger dedicated to the preheating of air should be introduced in the flowsheet after the compressor. The maximum input temperature is set at 947 K in order to keep the output temperature below 1900 K.

As the stream S06 is a waste stream, and despite the low relative internal losses, the exergy efficiency of the heat exchanger E101 is very low. In fact, the formulation of intrinsic efficiency allows taking into account the difference between waste and utilized streams. When analysing the values of the three components of exergy of the stream S06, it would be interesting to reuse the thermal exergy rather than reject it to the environment.

Pinch analysis

The new process flowsheet with the air preheater, the updated fuel/air ratio and the cooling down of the stream S06 to the ambient temperature is modeled using ProSimPlus ® .Then, applying the automatic Pinch calculation, the process Composite Curve and the Grand Composite Curve are drawn in Figure 4. As illustrated in Figure 4, the S05 hot stream leaving the T101 turbine is sufficient to meet the global hot utility requirement of the cogeneration system. It can be concluded from the design of the new heat exchanger network that a preheater has to be placed before the E101 heat exchanger in the S05 stream flow. . In addition, exergy efficiency is increased as external is reduced by recycling the thermal exergy loss.

Table 2 compares the performances of the retrofitted process with the base case cogeneration system. The increase of generated power and the decrease of irreversibilities are obvious. In addition, exergy efficiency is increased as external is reduced by recycling the thermal exergy loss.

Conclusions

This paper presents a general methodology for computer-aided process improvement tool combining the Pinch Technology and the Exergy Analysis. The Exergy Analysis first enables to make a diagnosis of a process proposing then solutions to recover external losses or to reduce internal losses. Here, with the cogeneration system, we have illustrated the case of thermal losses. In such a case the concerned streams become hot or cold streams for the Pinch Analysis. A more in-depth study could have been done to determine the best input temperature and air/fuel ratio of the combustion chamber. Then starting from cold and hot streams, power sinks and sources deduced from a previous Exergy Analysis, the Pinch methodology proposes different solutions to optimize both process and utility. The next step is to extend the scope of this methodology to industrial scale processes.

  coefficients are part of the exergy based tools package used by engineer to make a process energy diagnosis.

Figure 1 :

 1 Figure 1: CPEA methodology diagram

Figure 4 :

 4 Figure 4: (a) Composite curve and (b) grand composite curve of the cogeneration system4.3. Retrofitted processThe retrofitted cogeneration system, represented in Figure5, has been simulated in ProSimPlus ® modeling and simulation software.

Figure 5 :

 5 Figure 5: Retrofitted cogeneration system

Table 1 :

 1 Exergy balance of the cogeneration system

	Streams	S01	S02	S03	S04	S05	S06	V01	V02
	Temperature (K)	298.15 797.17 298.15 1762 915.03 520.71 298.15 773.15
	Pressure (atm)	1	25	25	24.5	1	1	15	15
	Total exergy (MW)	0.40 74.06 202.86 224.09 53.32 13.53 1.06	26.92
	Physical exergy (MW)	0.00 73.66 1.90 220.99 50.22 10.43 0.03	25.89
	Thermal exergy (MW)	0.00 32.69 0.00 182.13 50.22 10.43 0.00	25.86
	Mechanical exergy (MW)	0.00 40.96 1.90 38.86 0.00	0.00	0.03	0.03
	Chemical exergy (MW)	0.40	0.40 200.96 3.10	3.10	3.10	1.03	1.03
	Figure 3: Exergy efficiencies and internal losses on unit operations

Table 2 :

 2 Base case vs. Retrofitted case

		Base case process	Retrofitted process
	Generated Power (MW)	86.39	98.20
	Overall exergy efficiency (%)	55.22	61.03
	Overall internal losses (MW)	76.42	69.23
	Overall relative internal losses (%)	37.60	34.06
	External losses (MW)	13.53	8.90

Acknowledgement

Financial support of the French National Agency (ANR) is gratefully acknowledged.


Michiko Ahn Furudate

Seungyon Cho

Denis Hagebaum-Reignier

Study on chemical kinetics of HTO + H 2 → H 2 O + HT for design of tritium breeding blanket

Keywords:

Understanding the details of chemical kinetics related to the tritium recovery process is necessary to design highly efficient Tritium Extraction System (TES) in the fusion reactors. In the TES, tritium is expected to be released in the form of HTO from the breeder pebbles, and recovered in the form of HT in the helium purge gas containg small amount of hydrogen. The exchange reaction HTO + H 2 → H 2 O + HT is considered to be one of the important chemical reaction to obtain the tritium in the form of HT in the purge gas flow. In this study, the chemical kinetics of HTO + H 2 → H 2 O + HT is investigated based on quantum chemistry. Firstly, the potential energy surface will be explored to obtain information of reaction paths and its transition states by using a quantum chemistry calculation software. Then, the reaction rate constant of the exchange reaction is estimated from transition state theory.

Introduction

Efficient and reliable tritium recovery process for a breeding blanket is essential for achieving tritium sustainability in a fusion reactor. Tritium Extraction System (TES) is one of main ancillary systems among tritium circuit connected to a solid type breeding blanket which is known as outer fuel cycle contrast to inner fuel cycle having several systems in it such as pumping system, tokamak exhaust processing, isotope separation system, storage and delivery system, fueling system, etc [START_REF] Demange | A new combination of membranes and membrane reactors for improved tritium management in breeder blanket of fusion machines[END_REF].

Main function of TES is to extract bred tritium from the blanket using purge gas that comprises low pressure helium with small amount of hydrogen to facilitate tritium release via isotopic exchange reactions, and to recover tritium before routing to tritium accountancy system then to tokamak exhaust processing in the inner cycle. The bred tritium is known to be released in the form of HTO from breeder to purge gas in a blanket [START_REF] Franza | Tritium transport analysis in HCPB DEMO blanket with the FUS-TPC code[END_REF], however it is expected to be recovered mainly in the form of HT in TES modules due to presence of hydrogen in purge gas. The exchange reaction HTO + H 2 → H 2 O + HT is considered to be one of the important chemical reaction to obtain the tritium in the form of HT in the purge gas flow. In this study, the chemical kinetics of HTO + H 2 → H 2 O + HT is investigated based on quantum chemistry. Firstly, the potential energy surface will be explored to obtain information of reaction paths and its transition states by using a quantum chemistry calculation software. Then, the reaction rate constants of the exchange reaction are estimated from transition state theory.

Calculation methods

ab initio electronic structure calculations

The electronic energies of the chemical species and its complex is calculated by Density Functional Theory (DFT) using Gaussian 09 software [START_REF] Frisch | Gaussian 09, Revision A.02[END_REF]. As the hybrid functional and the basis set, UB3LYP and cc-pVTZ are employed in the present calculations, respectively.

Transition state structure searching by MC-AFIR

Transition states on the reaction paths are searched by Multi-Component Artificial Force Induced Reaction (MC-AFIR) method [4] with LUP using GRRM14 program [5]. The AFIR explores a potential energy surface searching for possible stable structures along the socalled AFIR paths. The AFIR path is a barrier less reaction made by applying artificial attractive force. In the MC-AFIR, these AFIR paths are randomly chosen for various positions of the reactants. The accuracy of the AFIR path are improved by applying LUP. The stable structures found in AFIR searching are the approximate transition states (TSs) and local minimums (LMs). They are re-optimized to obtain true TSs and LMs for the real reaction path. The potential energy calculations and the structure optimization in these procedures are carried out by Gaussian program that mentioned in the previous sub-section.

In the present study, reaction paths of the H 2 O + H 2 → H 2 O + H 2 reaction are searched to determine the transition state structure. Once the transition state structure for the H 2 O + H 2 → H 2 O + H 2 reaction is determined, one appropriate hydrogen atom is replaced by a tritium atom, and then structure optimization and frequency analysis are carried out to determine the properties of the transition states structure for HTO + H 2 → H 2 O + HT reaction. This approach is considered to be valid because isotopes have very similar electronic structures, and mass-difference effects on the equilibrium geometries of the transition state are expected to be negligible as shown in the results for water isotopologues by a non-Born-Oppenheimer approach [START_REF] Császár | On equilibrium structures of the water molecule[END_REF]. The massdifference effects are taken into account in the frequency analysis to determine the kinetic and the thermal properties.

Transition state theory reaction rate constant

The TST reaction constants of HTO + H 2 → H 2 O + HT reaction is deduced by using the KiSThelP [START_REF] Canneaux | KiSThelP: a program to predict thermodynamic properties and rate constants from quantum chemistry results[END_REF] package. The conventional expression of TST reaction constant in the unit of cm 3 /molecule-s is given by

⎜ ⎟ = ⎛ ⎝ - ⎞ ⎠ k σ k T h Q N Q V k T exp TST b TS A R b ‡ ( 1
)
where σ is the reaction path degeneracy, k b the Boltzmann constant, temperature, h the Planck constant, N A the Avogadro's number, Q TS is the partition function of transition state, and Q R is the products of partition function of the products, and V ‡ the activation energy. The KiSThelP package uses the equivalent expression for bimolecular reaction in terms of thermodynamics properties;

⎜ ⎟ ⎜ ⎟ = ⎛ ⎝ ⎞ ⎠ ⎛ ⎝ - ⎞ ⎠ k σ k T h k T p G RT exp ∆ TST b b 0 ‡,0 (2)
where ΔG ‡,0 is the standard Gibbs free energy difference between the reactants and the transitions state in the unit of J/mol. The tunneling effect is considered by multiplying the Wigner's correction factor [START_REF] Wigner | Über das Überschreiten von Potentialschwellen bei chemischen Reaktionen[END_REF], χ, that is,

= k χTk ( ) TST W TST , (3)
where

= + T h v k T χ( ) 1 1 24 Im( ) b ‡ (4)
The reaction constants are also calculated by KiSThelP by using the variational transition state theory (VTST);

= k k Ts min ( , ), VTST s GT ( 5
)
where k GT is the generalized transition state rate constant

⎜ ⎟ = ⎛ ⎝ - ⎞ ⎠ k σ k T h Q T s N Q T s V s k T ( , ) ( , ) exp ( ) , GT b TS A R b ‡ ( 6
)
and s is the distance along the reaction path in inertial coordinate. The tunneling correction factor for the VTST rate is calculated based on an Eckart potential energy barrier [START_REF] Eckart | The penetration of a potential barrier by electrons[END_REF]. The equilibrium constants can be obtained in the form of

= ⎛ ⎝ - ⎞ ⎠ K T G RT ( ) exp ∆ . eq (7)
3. Results

Transition state structure

The AFIR calculations for H 2 O+H 2 → H 2 O + H 2 reaction found four approximate TSs and four approximate LMs. By the optimizations of these structures, one approximate TS converged to a stable structure, and the other approximate TSs and LMs are found to be structures on dissociation channel. The optimized structure and the energies of the TS are summarized in Table 1. The frequency analysis for the TS replaced Atom3 by Tritium is also out, and the results are also shown in Table 1.

The structure has one imaginary frequency, indicating that the structure is in the saddle point of the potential surface.

Properties of the reactants

In order to calculate the reaction rate constants by the transition state theory, we need to have the electronic energies and the vibrational frequencies of the reactants. To be consistent, these properties for H 2 O, HTO, H 2 and HT is calculated by the same ab initio approach as in the calculation for TS. The obtained properties are summarized in Table 2.

IRC

Fig. 1 show the IRC paths calculated for the reactions H 2 O+H 2 → H 2 O+H 2 and HTO+H 2 → H 2 O + HT from the obtained TS. The vertical axis is the energy measure from the sum of the electric energies of H 2 and HTO, that is -203.84 kJ/mol. The activation energy for these path is found to be 279.88 kJ/mol. The path for the reaction HTO +H 2 → H 2 O + HT is slightly broader than that of the H 2 O + H 2 → H 2 O + H 2 , because the horizontal axis is the mass-weighted reaction coordinate.

Reaction rate constants by transition state theory

Assuming the nominal temperature and the pressure of the purge gas in the Korean Helium-Cooled Ceramic Reflector (HCCR) Test Blanket Module (TBM) [START_REF] Ahn | Design change of Korean HCCR TBM to vertical configuration[END_REF], the reaction rate constants for the reaction HTO + H 2 → H 2 O + HT are calculated for the temperature range from 700 K to 900 K at the pressure of 1 atm. Fig. 2 shows the calculated reaction rate constants as a function temperature. As seen in Fig. 2, the reaction rate constants almost linearly increase as the temperature increase. The reaction rate constant calculated by the variational transition state theory (VTST) is generally greater than that of the original transition state theory(TST).

The tunneling effects correction factor evaluated by Wigner's as well as Eckart's methods are shown in Fig. 3. In both methods, the correction factor decreases when the temperature decrease. The Eckart's method gives higher correction factor compared with the Wigner's method, especially at the lower temperatures. With the tunneling effects, the reaction rate constant in the present temperature range approximately doubles the reaction rate constant.

The three parameters for the Arrhenius fitting function are also calculated by the KiSThelP program. The obtained Arrhenius function for the VTST reaction rate constant with the Eckart's correction factor in the temperature range 800 K ± 20% is

= × ⎛ ⎝ - ⎞ ⎠ - k T T RT ( ) 7.
We have taken the same approach for the backward reaction, H 2 O + HT → HTO + H 2 , and obtained the Arrhenius function of

= × ⎛ ⎝ - ⎞ ⎠ - k T T RT ( ) 6.
The equilibrium constant in terms of free energy gap between the reactants and the products for the HTO + H 2 → H 2 O+HT is calculated to be

= ⎛ ⎝ - ⎞ ⎠ K T RT ( ) exp 2950 .
eq [START_REF] Ahn | Design change of Korean HCCR TBM to vertical configuration[END_REF] The equilibrium constant can be also expressed by the ratio of the forward and the backward reaction rate constants. When we take the ratio of Eqs. 8 and 9, we obtain

= = ⎛ ⎝ - ⎞ ⎠ - K T k k T RT ( )
12.429 exp 6930 .

eq f b 0.287 [START_REF] Black | Equilibrium in hydrogen water systems containing tritium[END_REF] In Fig. 4, the reciprocals of the present equilibrium constants are compared with the experimental results obtained by Taylor et al. [START_REF] Black | Equilibrium in hydrogen water systems containing tritium[END_REF] from the partial pressure measurements. Note that the fitting function of Taylor is derived to fit the experimental data in the temperature range from 289.20 K to 576.95 K, while the present fitting function is targeted to represent calculated values for 800 K ± 20%. The present calculations and the fitting functions agree well with the experimental data and its fitting, especially in the temperature range from 400 K to 600 K. It is also shown in Fig. 4 that the fitting functions of Eq. ( 11) well represents the calculated values of equilibrium constants all over the temperature range. The fitting functions of Eq. ( 10), which is deduced from the standard Gibbs free energy difference between the reactants and the transitions state at the temperature of 800 K, lose the accuracy in the lower temperature range.

Discussions

The reaction rate constants for the HTO + H 2 → H 2 O + HT are estimated by the transition state theory, and the fitting expression in the Arrhenius form is given in equation. At 800 K and 1 atm, which is the nominal conditions of the purge gas flow in HCCR TBM, the exchange reaction rate constant is estimated to be 8.5774 × 10 -30 cm 3 /molecules-s. These small rate is due to the relatively high activation energy for the reaction is 232.58 kJ/mol, that is corresponding to 27,973 K in temperature. Although only the direct exchange reaction by from HTO and H 2 to H 2 O and HT are considered in the present study, the possibility of other reaction channels must be studied; i.e., the pre-dissociation HTO or H 2 molecules into atoms, or the reactions with catalysis. Also, kinetics for other chemical reactions expected to occur in the purge gas flow, for example, reactions related to the other isotopes such as deuterium, must be studied in the future.

Conclusions

The reaction paths of HTO + H 2 → H 2 O + HT are explored by AFIR approach by using GRRM program, and one transition state is successfully discovered. From the obtained transitions state, the reaction rate constant is derived by the transition state theory, and fitted in the Arrhenius form as a function of temperature. For the temperature around 800 K, the reaction rate constant is estimated to be the order of 10 -30 -10 -29 cm/molecules-s.

Fig. 1 .

 1 Fig. 1. The calculated IRC path for transition state theory.

Fig. 2 .

 2 Fig. 2. The obtained reaction rate coefficients.

Fig. 3 .

 3 Fig. 3. The tunneling correction factor.

Table 1

 1 Optimized structure and properties of TS.

Table 2

 2 Properties of H2O, HTO, H2 and HT.

Acknowledgments

This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF-2015R1D1A1A01058637).