David L St-Pierre
email: davidls@lri.fr

Olivier Teytaud
email: olivier.teytaud@inria.fr

Nash and the Bandit Approach for Adversarial Portfolios

In this paper we study the use of a portfolio of policies for adversarial problems. We use two different portfolios of policies and apply it to the game of Go. The first portfolio is composed of different versions of the GnuGo agent. The second portfolio is composed of fixed random seeds. First we demonstrate that learning an offline combination of these policies using the notion of Nash Equilibrium generates a stronger opponent. Second, we show that we can learn online such distributions through a bandit approach. The advantages of our approach are (i) diversity (the Nash-Portfolio is more variable than its components) (ii) adaptivity (the Bandit-Portfolio adapts to the opponent) (iii) simplicity (no computational overhead) (iv) increased performance. Due to the importance of games on mobile devices, designing artificial intelligences for small computational power is crucial; our approach is particularly suited for mobile device since it create a stronger opponent simply by biasing the distribution over the policies and moreover it generalizes quite well.

I. INTRODUCTION

The use of portfolio is at the center of a revolution in machine learning [START_REF] Utgoff | Perceptron trees: A case study in hybrid concept representations[END_REF], [START_REF] Aha | Generalizing from case studies: A case study[END_REF], Artificial Intelligence, planning, combinatorial optimization [START_REF] Nudelman | Understanding random sat: beyond the clauses-to-variables ratio[END_REF], [START_REF] Xu | Hydra-mip: automated algorithm configuration and selection for mixed integer programming[END_REF], [START_REF] Kotthoff | Algorithm selection for combinatorial search problems: A survey[END_REF] and games [START_REF] Gaudel | A principled method for exploiting opening books[END_REF], [START_REF] Bouzy | Hedging algorithms and repeated matrix games[END_REF], [START_REF] Swiechowski | Self-adaptation of playing strategies in general game playing[END_REF]. "Portfolio" here refers to the family of algorithms used in the solving, whereas "portfolio combination" or "combination" refers to the combined algorithm. There exist several ways to build such combination. It can be combined either "externally", which refers to a process that do not explicitly enters in each algorithm, through "chaining" [START_REF] Borrett | Towards a formal framework for comparing constraint satisfaction problem formulations[END_REF], which means interrupting one and using its state for another algorithm or even "internally" [START_REF] Vassilevska | Confronting hardness using a hybrid approach[END_REF]. The most famous applications of portfolios are around SAT-solving [START_REF] Xu | Satzilla: Portfoliobased algorithm selection for sat[END_REF].

In this paper, we focus on portfolio of policies. Portfolios of policies have been less widely explored, except for e.g. combinations of local controllers by Fuzzy Systems [12], Voronoi controllers [START_REF] Teraoka | Voronoi coverage control with time-driven communication for mobile sensing networks with obstacles[END_REF] or some case-base reasoning [START_REF] Lejri | Representation in case-based reasoning applied to control reconfiguration[END_REF]. These methods are based on "internal" combinations, using the current state for choosing between several policies. We will here focus on external combinations. Such combinations are sometimes termed "ensemble methods"; however, we simply consider weighted averages of already constructed policies, the simplest case of ensemble methods.

A distribution over a portfolio can be computed either offline [START_REF] Kadioglu | Algorithm selection and scheduling[END_REF] or online [START_REF] Gagliolo | Learning dynamic algorithm portfolios[END_REF], [START_REF] Armstrong | Dynamic algorithm selection using reinforcement learning[END_REF]. In this paper, we use two different methods. First we compute a Nash Equilibrium (NE) over the portfolio of policies in an offline fashion. Second, we compute online a distribution using a bandit approach. The second case is adaptive, the coefficients will depend on the opponent's decisions.

The main contribution of this paper is to propose a methodology that can generically improve the performance of policies without actually changing the policies themselves, except through the policy's options or even the policy's random seed. Incidentally, we establish that the random seed can have a contribution, just because it has an effect on rote learning; while a fixed random seed cannot be strong against an adaptive opponent, our policies are more diversified (for the offline portfolio that we propose, which combines several seeds) or adaptive (for the online portfolio that we propose). It is particularly well suited when the computational power is limited. We study 2 different portfolios of policies:

• The first one is a set of algorithms that can play the game of Go (actually, different options of a same program). The goal here is to find a probability distribution for choosing among these algorithms. • The second portfolio is composed of a set of random seeds for a given algorithm that plays the game of Go.

The difficulty here lies in finding a distribution over the set of random seeds such that it provides a strong strategy either versus a known opponent (an opponent for which we know the set of random seeds) or an unknown one. In all cases we show that there is a way to compute a distribution over the portfolio such that it generates a robust (not exploitable) agent: this is our Nash-portfolio. Moreover we show that we can learn a specialized distribution, adaptively, given a fixed stationary opponent. This is our UCBT-portfolio.

The rest of the paper is divided as follows. Section II formalizes the settings. Section III describes our approach. Section IV presents the results and Section V concludes.

II. PROBLEM STATEMENT

A policy π is defined as follows. Let us consider a state space S, an initial state s 0 ∈ S and a transition function (s, a) → (s , r) where r is the reward when playing action a in state s. A policy π(•) selects an action a ∈ A s given the current state s ∈ S, where A s is the set of legal actions given the state s. The resulting state being s . In several cases, the reward may not be directly observable. By repeating this process until a final state s f ∈ S f ⊆ S is reached, where S f is the set of final states, then the reward can be retrieved.

Moreover, we are interested in adversarial portfolios. This means that more than one player can intervene in the outcome. We limit ourselves to constant-sum 2-player games. In the 2-player case, each state s is equipped with a player index, stating whether, in state s, it is player 1's turn to play or player 2's turn to play. A simulation starts at s 0 , then decisions are made by players, until a final state s f is reached. Thus, to get a reward r, we must know the initial state s 0 , the policy π 1 for player 1 and the policy π 2 for player 2.

We consider here matrix games with sum equal to 1 instead of an arbitrary constant, without loss of generality. Consider a matrix K × K , with values in [0, 1]. This matrix models a game as follows:

• Simultaneously and privately:

-Player 1 chooses i ∈ {1, . . . , K}.

-Player 2 chooses j ∈ {1, . . . , K }.

• Then they receive rewards as follows:

-Player 1 receives reward M i,j .

-Player 2 receives reward 1 -M i,j . A pure strategy (for player 1) consists in playing a given, fixed i ∈ {1, . . . , K}, with probability 1. A mixed strategy, or simply a strategy, consists in playing i with probability p i , where K i=1 p i = 1 and ∀i ∈ {1, . . . , K}, p i ≥ 0. Pure and mixed strategies for player 2 are defined similarly. Pure strategies are a special case of mixed strategies.

It is known since [START_REF] Nash | Some games and machines for playing them[END_REF] that there exist strategies p and q for the first and second player respectively, such that ∀(p , q ), p M q ≤ pM q ≤ pM q .

(1)

p and q are not necessarily unique, but the value v = pM q is unique and it is, by definition, the value of the game. The exploitability of a strategy p for the first player is exploit 1 (p) = v -min q pM q. exploit 1 (p) = 0 is equivalent to the fact that p is the first-player part of a Nash equilibrium (p, q). The exploitability of a strategy q for the second player is exploit 2 (q) = max p pM q-v and it verifies similar properties.

III. APPROACH

Section III-A explains how to combine policies offline, given a set of policies for player 1 and a set of policies for player 2. Section III-B explains how to combine policies online, given a portfolio of policies for player 1 and given an opponent.

A. Nash-portfolios: combining policies Consider two players P1 and P2, playing some game (not necessarily a matrix game). Assume that P1 has a portfolio of K policies. Assume that P2 has a portfolio of K policies. Then, we can construct a static combination of these policies by solving (i.e. finding a Nash equilibrium of) the matrix game associated to the matrix M , with M i,j the winning rate of the i th policy of P1 against the j th policy of P2. Solving this 1-sum matrix game provides p 1 , . . . , p K and q 1 , . . . , q K , probabilities, and the combination consists in playing, for P1, the i th policy with probability p i and, for P2, the j th policy with probability q j . Such a combination will be termed here a Nash-portfolio. By construction,

• the Nash-portfolio can play both as Black and as White;

• the Nash-portfolio does not change over time but is, in the general case, stochastic. Definition: given a set P 1 of K policies for player 1 and a set P 2 of K policies for player 2,

• the strategy (as player 1) playing the i th strategy for player 1 with probability p i ; • and the strategy (as player 2) playing the i th strategy for player 2 with probability q i ; is termed the Nash-portfolio of (P 1 , P 2 ) if the p i and q i are defined as solutions of Eq. 1.

The Nash equilibrium can be found using an exact solving of the matrix M [START_REF] Gale | Linear programming and the theory of games[END_REF]. It can also be found approximately and iteratively, in sublinear time, as shown by [START_REF] Grigoriadis | A sublinear-time randomized approximation algorithm for matrix games[END_REF], [START_REF] Auer | Gambling in a rigged casino: the adversarial multi-armed bandit problem[END_REF]; the EXP3 algorithm is classical for doing so. Coevolution is also a possibility [START_REF] Paredis | Coevolutionary computation[END_REF], [START_REF] Ong | Discovering chinese chess strategies through coevolutionary approaches[END_REF], [START_REF] Drake | Coevolving partial strategies for the game of go[END_REF].

From the properties of Nash equilibria, we deduce that the Nash-portfolio has the following properties:

• It depends on a family of policies for player 1 and on a family of policies for player 2. It is therefore based on a training, as in, e.g., [START_REF] Kadioglu | Algorithm selection and scheduling[END_REF] in the framework of optimization. • It is optimal (for player 1) among all combinations of the pure strategies in the portfolio of player 1, in terms of both worst case among the pure strategies in the portfolio of player 2; worst case among the mixed strategies over the portfolio of player 2.

• It is not necessarily uniquely defined.

In optimization settings, it is well known [START_REF] Samulowitz | Learning to solve qbf[END_REF] that having a somehow "orthogonal" portfolio of algorithms, i.e. algorithms as different from each other as possible, is a good solution for making the combination efficient. It is however difficult, in the context of policies, to know in advance if two algorithms are orthogonal.

B. UCBT-Portfolio

Section III-A assumed that P 1 and P 2 , two sets of strategies, are available, and that we want to define a combination of policies in P 1 (resp. in P 2 ). A different point of view consists in adapting online the probabilities p i and q i , against a fixed opponent. We propose the following algorithm (in the case of player 1 having K policies at hand), directly inspired by the bandit literature [START_REF] Lai | Asymptotically efficient adaptive allocation rules[END_REF], [START_REF] Auer | Finite time analysis of the multiarmed bandit problem[END_REF], and, more precisely, by Upper-Confidence-Bounds-Tuned (UCBT) [START_REF] Audibert | Exploration-exploitation trade-off using variance estimates in multi-armed bandits[END_REF]:

• Define n i = 0, r i = 0, for i ∈ {1, . . . , K}.

• For each iteration t ∈ {1, 2, 3, . . . }.

compute for each i ∈ {1, . . . , K} score(i) = min(1, r i /n i + 1 100 C log(4 pt )/n i + 16 100 log(4 pt )/n i ). using X/0 = +∞ (even for X = 0), p = 2.1 and C = 2 (UCBT, i.e. UCB-Tuned, formula).

choose k maximizing score(k).

play a game using algorithm k in the portfolio.

-if it is a win, r i ← r i + 1. -n i ← n i + 1.
Definition. We refer to this adaptive player as UCBT-Portfolio, or Bandit-Portfolio.

IV. EXPERIMENTS

The game of Go is an ancient oriental game, born in China probably at least 2500 years ago. It is still a challenge for artificial intelligence, as even though Monte-Carlo Tree Search (MCTS [START_REF] Coulom | Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search[END_REF]) revolutionized the domain, the best programs are still far from the professional level. Go is a very deep game [START_REF] Robertie | Backgammon[END_REF], and professional Go players are still able to win games against the best Go programs in 9x9.

Section IV-A focuses on a portfolio of different algorithms (actually, variants of a same program, using different options) and explores the performance of our approach on this setting. Next, Section IV-B studies our approach but this time over a portfolio of deterministic policies, obtained by fixing random seeds in a randomized program. Thanks to very low computational cost, both settings are relevant for the application of our approach on mobile devices.

A. Portfolio of algorithms 1) Test case: variants of a program: We consider the problem of combining several options, leading to several variants (each variant corresponds to a set of options which are enabled), of an artificial intelligence for the game of Go.

Our matrix M is a 32×32 matrix, where M i,j is the winning rate of the i th variant of GnuGo (as black) against the j th variant of GnuGo (as white). We consider all combination of 5 options of GnuGo, hence 32= 2 5 variants. In short, the first option is 'cosmic-gnu', which focuses on playing at the center, thus usually weakening the AI. The second option is the use of fuseki (global opening book). The third option is 'mirror', which consist in mirroring your opponent at the early stage of the game. The fourth option is the large scale attack, which evaluates if a large attack across several groups is possible. The fifth option is the break-in. It consists in breaking the game into territories that require deeper tactical reading and are impossible to detect otherwise. It revises the territory valuations. Further details on the 5 options are listed on our website https://www.lri.fr/ ∼ teytaud/games.html.

In this section, experiments are performed on the convenient 7x7 framework, with MCTS having 300 simulations per move -this setting is consistent with the mobile device setting.

After extracting the matrix M by intensive experiments on a cluster, we solve the matrix game associated to M and get a portfolio for Black and a portfolio for White.

In this section we refer to the i th algorithm for Black as BAIi (Black Artificial Intelligence # i), and WAIj is the j th algorithm for White.

2) Nash-based portfolio: We here check the exploitability of our method, compared to other combinations. The other considered combinations are:

• Uniform combination: each pure strategy is equally likely.

• Maximal pure combination: choose the pure strategy which maximizes the average reward against the uniform combination. By definition, the exploitability of the Nash-portfolio method is zero, because it is a Nash equilibrium. But other methods usually have a positive exploitability. We here quantify this phenomenon. We get the following on the data set described above (variants of a program):

• Value of the 32 × 32 matrix game: 61.18% (i.e. the game is easier for Black).

Conclusion:

We conclude that our Nash-combination is moderately better than each combination of options considered separately. This section is devoted to experiments limited to a learning set of 32 combinations of options for each player, and therefore could be considered as an overfit; we will check the efficiency of our Nash-Portfolio in generalization in Section IV-B4, on the second test case (random seeds), for which results of the Nash portfolio are better.

3) UCBT Portfolio: Here we present the losing rate of our UCBT Portfolio against 2 baselines. The first baseline is the Nash equilibrium (label N ash in Figure 1), which consists in computing the Nash portfolio as in the previous section. The second baseline is the uniform player (label U nif ) which consists in playing each variant with the same probability.

Here, we apply our UCBT Portfolio for learning

• As White against Nash-portfolio.

• As Black against Nash-portfolio.

• As White against each pure Black policy.

• As Black against each pure White policy. Losing rates of the recommended variant are presented in Fig. 1. All results are averaged over 1000 runs. The X-axis shows the number of iterations of UCBT (i.e. number of played games for learning) whereas the Y-axis represents the frequency at which the game is lost.

Conclusion:

We see that (i) UCBT eventually reaches, against Nash-portfolio, approximately the value of the game for each player (ii) the Nash-portfolio is among the most Fig. 1: Losing rate of UCBT-portfolio, versus the online learning time, against (i) Nash-Portfolio (black line) (ii) each variant independently (stars). X-axis: log2(number of iterations of UCBT (i.e. number of played games for learning). Y-axis: frequency at which the game is lost. In this case, the variants under work are GnuGo variants (different options). All experiments are reproduced 1000 times. difficult opponents (the curve decreases slowly only). (iii) each deterministic combination of options is clearly beaten after some runs, which shows the adaptivity of UCBT Portfolio and therefore its efficiency as a eTeacher.

B. Portfolio of Random Seeds

1) Test case: The previous section combines existing algorithms. Now, this section (i) creates variants thanks to different random seeds and (ii) combine them. This means that M i,j is 1 if, with random seed i, Black wins against White with random seed j. Importantly, the number of games to be played for getting the matrix involved in the Nash-portfolio (Eq. 1) is K 2 , with K the number of random seeds under testsno need for playing multiple games as random seeds make it deterministic. For making the random seed principle more convincing, we switch to 9x9, which is harder than 7x7 in terms of rate-learning. [START_REF] Gaudel | A principled method for exploiting opening books[END_REF] proposed a combination of opening book, using tools similar to those we propose in Section III-A for combining policies. We here propose to simply use GnuGo's random seed for having several Gnugo variants (hence, programs playing the opening differently). The random seed of GnuGo makes the program deterministic, by fixing the seed used in all random parts of the algorithm. We define 32 variants, using "GnuGo -level 10 -random-seed V" with V in {1, . . . , 32}.

In other words, we use a MCTS with 80 000 simulations per moves.

2) Nash-based portfolio: We here check the exploitability of our method, compared to other combinations. By definition, the exploitability of the Nash-portfolio method is zero, because it is a Nash equilibrium. But other methods are exploitable. We here quantify this phenomenon. We get the following on the data set described at the beginning of Section IV:

• Value of the game: 54.16% (i.e. the game is easier for Black). • The winning rate at the Nash equilibrium for the Black player is incidentally the value of the game 54.16%. • In the NE, the number of selected options is 11 for Black and 9 for White. This indicates that roughly 1 3 of the random seeds are relevant.

• Against a uniform random combination of the 32 options, the winning rate of the Nash-portfolio as Black (resp. White) is 68.51% (resp. 38.80%), i.e. the exploitability of the uniform random combination is 14.35% (resp. 15.36%).

• Exploitability of Portfolios with one option only:

-Exploitability of the Portfolio restricted to BAIi, with i = arg max K j=1 M i,j : 44.24%. -Exploitability of the Portfolio restricted to WAIj, with j = arg max K i=1 M i,j : 33.38%. Conclusion: Nash-portfolios seems useful even in cases in which an optimal deterministic policy exists, such as fully observable games (Go is such a case). The generalization ability will be tested in Section IV-B4.

3) Learning UCBT: Here we present the losing rate of UCBT against 3 baselines. The first baseline is the Nash equilibrium (label N ash and previously defined in Section III-A. The second baseline is the uniform player (label U nif ) which consists in playing uniformly each option of the bandit. The third baseline consists in playing a single deterministic strategy (only one random seed) regardless of the opponent.

Figure 2 presents the results. The x-axis represents the number of iterations (on logarithmic scale) of UCBT (i.e. number of played games for learning). The y-axis is the frequency at which the game is lost. All results are averaged over 1000 runs.

First and foremost, as the number of iterations grows, there is a clear learning against both N ash and U nif baselines. We see that (i) UCBT eventually reaches, against Nash-portfolio, approximately the value of the game for each player (ii) the Nash-portfolio is among the most difficult opponents (the curve decreases slowly only). We can also observe from Figure 1 that against the U nif baseline UCBT learns a strategy that outperforms this opponent.

When it plays as the Black player, it takes less than 2 7 (128) games to earns the correct strategy and wins with a 100 % ratio against every single deterministic variant. As the White player, it is even faster with only 2 5 games required to always win.

Conclusion: UCBT can learn very efficiently against a fixed deterministic opponent; this confirms its ability as a eTeacher. It performs better than Nash-portfolio against Uniform, showing that even against a stochastic opponent it can perform well, and in particular better than the Nash. This is not a contradiction with the Nash optimality; the Nash portfolio is optimal in an agnostic sense, whereas UCBT tries to overfit its opponent and can therefore exploit it better.

4) Generalization ability of Nash-Portfolio: Here we are interested in the generalization of a Nash-portfolio. In other words, whether it is possible to use a distribution computed over a portfolio of policies (learnt against a given set of opponent policies) against new opponent policies that were not part of the initial matrix. The idea is to select a submatrix of size N , choose a combination in this submatrix (i.e. this is the learning set) and make it play against the remainder of the seeds (i.e. this is the validation set).

Figure 3 presents the results of 3 different approaches. The first approach, labeled N ash, is to use the distribution representing the NE computed on the initial submatrix against the new opponent policies. The second approach, labeled BestArm is to select the single best performing policies. The third, our baseline, is to use the uniform distribution U nif over the initial set of seeds. The x-axis represents the number of policies N considered (hence a matrix M = N 2 ). The y-axis shows the win rate of the different approaches. All experiments are reproduced 1 000 times.

From Figure 3 we can observe that there is a clear advantage to use either the N ash or the BestArm approach when facing a new set of policies. Moreover, as expected, as the size of the initial matrix grows, the winning rates of both N ash and BestArm increase when compared to the baseline.

It does not come as a surprise that the approach BestArm performs slightly better than the N ash against a uniformly random opponent. The BestArm approach is particularly well suited to play against such an opponent. However, the BestArm policy is easily exploitable.

Figure 4 shows the difference between a N ash policy and a BestArm policy in terms of exploitability. The x-axis represents the number of policies considered. The y-axis shows the loss rates. All experiments are reproduced 100 times.

From Figure 4 it clearly appears that BestArm is a strategy very easy to exploit. Thus, even if Figure 3 shows that the use of the BestArm policy outperforms N ash versus the uniform baseline, N ash is a much more resilient strategy.

Conclusion:

The Nash portfolio and the BestArm portfolio provides a stable solution which generalizes against new opponents. The Nash portfolio is by definition optimal as a worst case against a given set of opponents, but the BestArm performs better in generalization against a fixed uniform portfolio.

V. CONCLUSION

We proposed two algorithms for combining policies:

• The Nash-Portfolio, which learns offline, given a family of variants for each player; • The Bandit(UCBT)-Portfolio, which learns online, given an opponent. We have seen that:

• The Nash-Portfolio is more diversified than any of its components; it is harder to learn against than any of its components and harder to learn against than the uniform-Portfolio; • The UCBT-Portfolio can learn a combination until reaching the exploitability of a stationary opponent (this is mathematically guaranteed by properties of UCBT, which is a consistent bandit algorithm in the discrete setting).

In particular, it defeated clearly the deterministic variants in Fig. 2, reaching 100% winning rate. It also performs quite well against the default policy, which is uniformly randomized random seed. Only the Nash-Portfolio resists much better, which shows that we can enhance a randomized algorithm a lot, just by biasing the choice of the random seed. Fig. 2: Losing rate of UCBT-portfolio, versus the online learning time, against (i) Nash-Portfolio (black line) (ii) each option independently (stars). X-axis: log2(number of iterations of UCBT (i.e. number of played games for learning). Y-axis: frequency at which the game is lost. In this case, the variants under work are random seeds. All experiments are reproduced 1000 times. The Nash-portfolio generalizes efficiently to new opponents, as shown by Section IV-B4. Therefore our tools provide an easy improvement, on top of randomized algorithms equipped with a random seed, or on top of variants of an algorithm. The computational cost could potentially become an issue in Section IV-A (as we averaged multiple games for building the matrix). It is not the case in Section IV-B where deterministic games are used. It should also be pointed out that solving Nash was fast, and if needs be, we can further the speed of the computation using algorithms that can -approximate a NE in sublinear time [START_REF] Grigoriadis | A sublinear-time randomized approximation algorithm for matrix games[END_REF], [START_REF] Auer | Gambling in a rigged casino: the adversarial multi-armed bandit problem[END_REF] -so that a number of games linear in max(K, K ) log(KK )/ 2 (less than the number K × K of elements in the matrix) is sufficient.

Results are therefore both

• An improvement in terms of eTeaching; our UCBT-Portfolio algorithm is more difficult to overfit, more diversified, than the uniform random seed, and is adaptive. This does not involve any additional computing power as UCBT is online and has a negligible internal cost. • An improvement in terms of adaptivity; our Nash-Portfolio is harder to overfit. At the end of the offline computation of the Nash equilibrium, it is just a bias in the random seed distribution, so the additional computational cost is negligible. The non-intuitive key point in the "random seed" part of this work is that biasing the random seed has an impact.

• An improvement in terms of playing strength, as our Nash-Portfolio performs better than each of its components.

Fig. 3 :

 3 Fig. 3: Winning rate of 3 baselines against uniform strategy in generalisation. The x-axis represents the number of policies considered. The y-axis shows the win rates. All experiments are reproduced 1000 times.

Fig. 4 :

 4 Fig.4: Loosing rate of the N ash and BestArm policies against an opponent that knows their respective strategy. The x-axis represents the number of policies considered. The yaxis shows the loss rates. All experiments are reproduced 100 times.

  • In the NE, the number of selected options is 5 for Black and also 5 for White, i.e. roughly 15.6% of the variants are in the Nash.• Against a uniform random combination of the 32 options, the winning rate of the Nash-portfolio as Black (resp. White) is 65.52% (resp. 49.71%), i.e. the Nash-portfolio outperforms the uniform random combination by 4.35% as Black (resp. 10.9% as White).

• Exploitability of Portfolios with one option only (selected as the best on average):

-Exploitability of the Portfolio restricted to BAIi, with i = arg max K j=1 M i,j : 3.15%. -Exploitability of the Portfolio restricted to WAIj, with j = arg min K i=1 M i,j : 5.85%.


Emmanuel Hoang

Agathe Dupas

Sami Hlioui

Michel Lécrivain

Mohamed 3d Gabsi

Global

3D Global Windings Hybrid Excitation Switching Flux machine

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

I. INTRODUCTION

Hybrid excitation machines combines advantages of wound excitation machines and permanent-magnet excited machines. Electric excitation offers a control of the magnetic field by controlling the excitation current while permanent magnet benefits from a high-energy density.

Many hybrid excited topologies were proposed in the literature and compared with PM machines. One of these topologies is the hybrid flux-switching machine presented in [1]- [4]. Due to a salient passive rotor, hybrid excited flux-switching permanent magnet machines have high robustness and thanks to permanent magnets localized on the stator the machine have high torque density.

In this study, a new hybrid excitation flux-switching machine is presented. The main characteristic of this machine is a global winding hybrid excitation.

In this paper we will, firstly, present the structure and the operation principles of this new hybrid excitation fluxswitching machine. Secondly, a 3D Finite Element model and the influence of DC Excitation Current on the no-load flux and on the short-circuit current will be presented. In order to validate the principle of operation of this new structure and the 3D FE analyses, a prototype machine was manufactured. The no-load and on-load tests of this machine will be presented.

II. PRESENTATION OF THE STRUCTURE

A. Topology

This machine is composed of a stator that includes armature coils, permanent magnets and a wound inductor (with global windings). The rotor is simply made of salient poles.

In our first version of the hybrid excited flux-switching machine [1]-[2], the magnetic structure has axial symmetry, this is a 2D structure. The excitation windings are composed of 12 windings and are located upon the permanent magnets. The new structure, presented in this paper, is has a global DC excitation windings. Figure 1 shows a 3D view of the proposed 12 slots and 10 poles machine.

The wounded excitation part is composed by the excitation winding and two armatures that are made, in the first prototype, by solid iron materials.

We study the possibility to manufacture them by soft magnetic composites (SMC). The second intrinsic characteristic of the machine is the value of the short-circuit current. In Figure 4, we present the evolution of these short-circuit current depending on the hybrid excitation current. We added the values of the thermal currents. To determine theses values, we applied a direct current as the resistance value increases by 40%.

III. 3D FE ANALYSIS

The magnetic field is computed by using 3D FEA. Noload flux and machine in motor mode are explored.

We can see in Figure 5 and

IV. PROTOTYPE VALIDATION

In order to validate the operating principle of this new structure a prototype is realized. The experimental results for no-load and on-load tests will be presented. In Figure 3, we introduced the no-load flux and the short-circuit current in Figure 4.

In order to test the performance of this new machine, we conducted tests in DC generator mode. The experimental test bench is shown in Figure 7 [4]. The tests were carried out with a value of the DC bus voltage of 300V. We find the characteristic curve of a synchronous machine coupled to a diode rectifier (see figure 8). We show the values of the various losses that we could measure. Copper losses in armature and excitation circuits are determined with the values of resistors and currents. We measure the torque with a balance assembly. We deduce the value of the mechanical power. Measuring the output power is knowing with the value of the continuous current output. The sum of the mechanical losses and iron losses is made by taking the difference between the mechanical power and the copper losses and the DC side power.

We are currently studying the models of iron losses and mechanical losses to distinguish them.

V. COMPARATIVE STUDY OF THE TWO STRUCTURES OF THE EXCITATION CIRCUIT

A. Description of the structure

In our first version of the hybrid excited flux-switching machine, the magnetic structure has axial symmetry. This is a 2D machine as most electrical machines. We can see from the picture shown in the following figure one of the disadvantages of these machines with axial symmetry is what is called the'' heads'' coils, especially if the length of the laminations is low. In this paper we present a new arrangement of the excitation coil, one that is driven by a DC current. In Figure 10, we present the original configuration, where all the conductors are perpendicular to the magnetic circuit, and wherein the windings are cellular, that is to say that they are entirely made in one unit cell. The excitation coil is associated with a magnetic circuit, driven primarily by a continuous flow, as indicated by red arrows in the photo below. The principle is to'' turn'' by 90° the excitation circuit as shown in the following two pictures. As we can see from the previous pictures, the magnetic excitation circuit will need to be modified so that it can achieve a'' bridge'' magnetic while the magnetic circuit of the armature remains unchanged, as shown in next figures.

B. Comparative studies of copper losses

We will compare the copper losses of excitation circuits for the two structures of magnetic excitation circuit.

 Cellular winding (initial solution):

 Global winding (new solution) :

With :

: copper resistivity : excitation current density slot number : winding area : active length : length of winding heads : average radius of the global winding

If one accepts that it is the ampere-turns are necessary to the magnetization of the magnetic circuit, we find that:

If in addition, we compare with the same slot section, the ratio of copper losses is:

(1) Numerical application: = 40 %; = 70 %. N S =12 ; L A =30mm ; L TB = 27mm ; R be_g_moy =52mm. = 3.6

We have verified experimentally, this value, or rather, the magnitude of the value between the copper losses of the excitation circuit for the two types of excitation circuit. For this, we measured, at ambient temperature, the resistance values of the different coils for the two structures: When comparing the two structures of the excitation circuit, for the same value of torque, we find for the annular excitation coil:

I exc = 4.8 A and I ind = 3.9 A P exc = 55 W and P ind = 33 W

In motor mode, to achieve a mean torque of 8 Nm at 600 rpm, we find for:  Cellular winding (initial solution): I exc = 9.8 A and I ind = 10 A P exc = 125 W and P ind = 155 W  Global winding (new solution) : I exc = 4.9 A and I ind = 6.9 A P exc = 57 W and P ind = 100 W

We find the expected trend well by the formula (1).

VI. CONCLUSION

In this paper, we present a new way to achieve the excitation circuit of a synchronous machine flux switching hybrid excitation.

For the study, we performed finite element calculations in 3D. We compared the 3D calculations with experimental tests. Tests in DC generator mode has also been made. This structure of the excitation circuit looks promising in terms of economic copper coil for excitation and in terms of losses. Studies are underway to optimize the shape and constitution of this circuit to minimize the mass of copper used and the copper losses.

VII. REFERENCES

Figure 1 :

 1 Figure 1: 3D view of the machine

Figure 3 :

 3 Figure 3: Maximum no-load flux versus excitation current. Comparison between 3D FE model and experimental results.

Figure 4 :

 4 Figure 4: Short-circuit current versus excitation current. Comparison between 3D FE model and experimental results.

  Figure 6 the values of the flux density and the local saturation. It should be noted that the outer ring was made of steel for the prototype and this is what justifies the above 2 Tesla in the excitation circuit values. We are conducting a study of analytical modeling and optimization of the shape and constitution of this new excitation circuit study.

Figure 5 :

 5 Figure 5: 3D FE analysis. Flux density map.

Figure 6 :

 6 Figure 6: 3D FE analysis. Flux density map.

Figure 7 :

 7 Figure 7: Experimental test bench [4].

Figure 8 :

 8 Figure 8: Experimental results in DC generator mode under 300 VDC.

Figure 9 :

 9 Figure 9: View of the two winding.

Figure 10 :

 10 Figure 10: Presentation of cellular windings.

Figure 11 :

 11 Figure 11: Front view of the excitation circuit.

Figure 12 :

 12 Figure 12: Front view with excitation circuit rotated by 90°.

Figure 12 :

 12 Figure 12: Side view with excitation circuit rotated by 90°.

Figure 13 :

 13 Figure 13: Description of half the excitation circuit.

Figure 14 :

 14 Figure 14: Description of half the excitation circuit with the winding.

Figure 15 :

 15 Figure 15: Picture of half the excitation circuit with the winding.



  Cellular winding : R exc = 1.3  ; R ind = 0.5   Global winding : R exc = 2.4  ; R ind = 0.7  In motor mode, to achieve a mean torque of 5 Nm at 600 rpm, with the initial machine with a winding cell excitation, we found: I exc = 9.8 A and I ind = 4.7 A P exc = 125 W and P ind = 33 WFor a machine with annular excitation coil, there is a relationship between the armature current and the excitation current.

Figure 16 :Figure 17 :

 1617 Figure 16: Excitation and armature currents for a constant torque at 5 Nm, for the machine with global winding.

  Owen and Z. Q. Zhu, "Hybrid-Excited Flux- Switching Permanent-Magnet Machines With Iron Flux Bridges". IEEE Transactions on Magnetics, vol. 46, no. 6, pp. 1726-1729, 2010. [4] B. Gaussens, E. Hoang, P. Manfé, M. Lécrivain, M. Gabsi, ''Improved output power capability of Hybrid Excited Flux-Switching DC-Alternators: Analysis and experiments''. IEEE International Conference on Industrial Technology (ICIT), Athens, Greece, 19-21 March 2012.

	[1] E. Hoang, M. Lécrivain and M. Gabsi, "A new
	structure of a switching flux synchrounous polyphased
	machine with hybrid excitation" in EPE 2007,
	Aalborg, 2007.
	[2] E. Hoang, M. Lécrivain and M. Gabsi, "Flux-switching
	dual excitation electrical	machine".	Patent
	US7868506, 8 march 2007.
	[3] R.

SATIE, ENS Cachan, CNRS, UniverSud, 61, av Président Wilson, F-94230 Cachan, France

SATIE, CNAM, CNRS, ENS Cachan E-mail: hoang@satie.ens-cachan.fr


Dr Young-Hye La

Jacquana Diep

Radwan Al-Rasheed

Majed Nassar

E Idil Mouhoumed

Anthony Szymczyk

Dr Geraud Dubois
email: gdubois@us.ibm.com

The Effect of Cross-contamination in the Sequential Interfacial Polymerization on the RO Performance of Polyamide Bilayer Membranes

Keywords: Reverse Osmosis, Polyamide Bi-layer Membrane, Cross-contamination, Sequential Interfacial Polymerization (SIP), Desalination

In this study, hexafluoroalcohol-containing polyamide layer (HFAPA) was prepared on top of a conventional polyamide under-layer (REFPA) via sequential interfacial polymerization (SIP) to improve RO separation behavior, and the performance of the resulting bilayer membrane was thoroughly optimized by investigating the effect of cross-contamination in the SIP process. When several coupons of the polyamide bilayer membrane were prepared by SIP of MPD(aq), TMC(hx) and hexafluoroalcohol-containing diamine (HFAMDA)(aq) in the manner of subsequent membrane dipping, unreacted MPD monomer (mostly captured in the porous PSF support) carried over from the 1 st interfacial reaction dissolved and accumulated in the 2 nd aqueous solution as verified by UV spectroscopic analysis. The MPD contaminant then participated in the 2 nd interfacial reaction, forming copolyamide with HFAMDA monomer onto the REFPA. Depending on the amount of MPD contaminant accumulated in the 2 nd aqueous solution, the composition of the resulting co-polyamide in the top-layer varied, causing a significant variation of RO performance; the flux was gradually decreased with the increase of MPD contaminants while the salt rejection slightly increased (from 1 st coupon toward 4 th coupon). This result indicated that a trace amount of MPD contaminant may be necessary to maximize RO separation behavior. Through in-depth performance evaluation of polyamide bilayer membranes prepared by adding various known-amount of MPD into 2 nd HFAMDA solution, and also by applying a frame process (2 nd amine solution was applied only top surface of membrane) to eliminate uncontrollable MPD contamination, we have successfully demonstrated consistent RO performance, and identified an optimum material composition to provide superior separation performance. The bilayer membrane prepared by adding 1.2 mol % of MPD to the total amount of HFAMDA in the 2 nd aqueous solution showed 99.8 % NaCl rejection with the water flux of 45 LMH under the cross-flow filtration performed with 2000 ppm NaCl solution at 400 psi, 25 C.

Introduction

Membrane technology, particularly reverse osmosis (RO) is widely used for water desalination and water reuse to overcome global water scarcity. High performance membrane materials and processes for these applications have attracted immense interest from the scientific community [START_REF] Service | Desalination Freshens Up[END_REF][START_REF] Shannon | Science and technology for water purification in the coming decades[END_REF]. The current generation of RO membranes is based on the thin film composite (TFC) structure, where an ultra-thin (50 ~ 200 nm) active layer exists on a thick porous support [START_REF] Cahill | Microscopy and microanalysis of reverseosmosis and nanofiltration membranes[END_REF][START_REF] Matsuura | Progress in membrane science and technology for seawater desalination -a review[END_REF][START_REF] Petersen | Composite reverse osmosis and nanofiltration membranes[END_REF]. The ultra-thin active layer mainly determines the membrane's separation performance while the support (porous polysulfone and a non-woven polyester fabric) provides enough mechanical strength for high operation pressure. To date, various materials such as polyamide [START_REF] Benavente | Determination of some electrical parameters for two novel aliphatic-aromatic polyamide membranes[END_REF][START_REF] Cadotte | Reverse osmosis membrane[END_REF][START_REF] La | Novel thin film composite membrane containing ionizable hydrophobes: pH-dependent reverse osmosis behavior and improved chlorine resistance[END_REF][START_REF] Wang | Nanofiltration membranes prepared by interfacial polymerization on thin-film nanofibrous composite scaffold[END_REF][START_REF] Yung | Fabrication of thin-film nanofibrous composite membranes by interfacial polymerization using ionic liquids as additives[END_REF], poly(ether-amide) [START_REF] Riley | Spiralwound poly(ether/amide) thin-film composite membrane systems[END_REF], polyurea [START_REF] Arthur | Preparation and uses of multilayer reverse osmosis membrane of polyamide-urea[END_REF][START_REF] Liu | Study on a novel polyamide-urea reverse osmosis composite membrane (ICIC-MPD)[END_REF], poly(amide-urethane) [START_REF] Yung | Fabrication of thin-film nanofibrous composite membranes by interfacial polymerization using ionic liquids as additives[END_REF][START_REF] Riley | Spiralwound poly(ether/amide) thin-film composite membrane systems[END_REF], and sulfonated poly(arylene ether sulfone) [START_REF] Kim | Nanocomposite poly(arylene ether sulfone) reverse osmosis membrane containing functional zeolite nanoparticles for seawater desalination[END_REF] have been investigated as potential active layers [START_REF] Lee | A review of reverse osmosis membrane materials for desalination. Development to date and future potential[END_REF]. However, the crosslinked aromatic polyamide invented by Cadotte et al. in the 1980s still appears as the most reliable material showing superior salt rejection and water flux, and remains as a basic platform for manufacturing TFC RO membranes on a commercial scale [START_REF] Cadotte | Reverse osmosis membrane[END_REF].

The TFC membrane comprising aromatic polyamide active layers often suffer from membrane fouling [START_REF] Rana | Surface Modifications for Antifouling Membranes[END_REF] and chemical oxidation by chlorine-based biocidal agents [START_REF] Knoell | Municipal wastewater. Chlorine's impact on the performance and properties of polyamide membranes[END_REF].

Therefore, recent research has focused on surface modification of the crosslinked polyamide TFC membrane to remedy its weakness or to further improve its separation performance. Self-assembly of inorganic nanoparticles (ex. TiO2) or organic functional materials (ex. poly(ethylene amine)) onto commercial polyamide membranes through hydrogen-bond or electrostatic interaction has been introduced as a way to improve a membrane's anti-fouling efficiency and/or chlorine resistance [START_REF] Kim | Design of TiO2 nanoparticle selfassembled aromatic polyamide thin-film-composite (TFC) membrane as an approach to solve biofouling problem[END_REF][START_REF] Choi | Layer-by-Layer Assembly of Graphene Oxide Nanosheets on Polyamide Membranes for Durable Reverse-Osmosis Applications[END_REF][START_REF] Zhou | Surface modification of thin film composite polyamide membranes by electrostatic self deposition of polycations for improved fouling resistance[END_REF]. Surface-initiated grafting of anti-fouling or chlorine resistant materials (ex. polyethyleneglycol methacrylate, imidazolidinyl urea) has been also performed by activating a polyamide membrane surface with an appropriate coupling agent or a redox initiator [START_REF] Mickols | Composite polyamide-polyoxyalkylene graft polymers as reverse osmosis and nanofiltration membranes[END_REF][START_REF] Xu | Reverse osmosis membrane with regenerable anti-biofouling and chlorine resistant properties[END_REF].

Contrary to these strategies, in which a commercial TFC polyamide membrane with residual carboxylate functional groups (COO -) has been used as a base material, direct modification of a freshly prepared polyamide layer by a sequential interfacial reaction scheme has been also proposed [START_REF] La | Enhanced desalination performance of polyamide bi-layer membranes prepared by sequential interfacial polymerization[END_REF][START_REF] Kang | A novel method of surface modification on thin-film composite reverse osmosis membrane by grafting poly(ethylene glycol)[END_REF][START_REF] Zou | Synthesis and characterization of thin film composite reverse osmosis membranes via novel interfacial polymerization approach[END_REF]. In general, a crosslinked polyamide layer is prepared by interfacial polymerization of m-phenylenediamine (MPD) in water and trimesoylchloride (TMC) in an organic solvent. Thus, the membrane surface always contains excessive unreacted acid chlorides (-COCl), which will eventually hydrolyze to carboxylic acids (COOH) as normally observed on the surface of conventional aromatic polyamide RO membranes. The hydrolysis reaction, however, is relatively slow, so the acid chlorides are available for further interfacial reactions with the same or different types of amine-functionalized monomers before hydrolysis occurs [START_REF] Kang | A novel method of surface modification on thin-film composite reverse osmosis membrane by grafting poly(ethylene glycol)[END_REF][START_REF] Zou | Synthesis and characterization of thin film composite reverse osmosis membranes via novel interfacial polymerization approach[END_REF]. This sequential interfacial polymerization is very straightforward and can be done in a short period of time since the reaction between amine and acid chlorides occurs almost instantly if monomers have good mobility [START_REF] La | Enhanced desalination performance of polyamide bi-layer membranes prepared by sequential interfacial polymerization[END_REF][START_REF] Zou | Synthesis and characterization of thin film composite reverse osmosis membranes via novel interfacial polymerization approach[END_REF]. Kang et al. immobilized aminefunctionalized polyethylene glycol (NH2-PEG) onto a conventional aromatic polyamide active layer by using the sequential interfacial reaction scheme and demonstrated improved anti-fouling efficiency [START_REF] Kang | A novel method of surface modification on thin-film composite reverse osmosis membrane by grafting poly(ethylene glycol)[END_REF]. Zou et al. also fabricated an anti-fouling polyamide layer with a large amount of amino groups on the surface by dipping a conventional polyamide made with MPD(aq) and TMC(hx) into MPD aqueous solution again [START_REF] Zou | Synthesis and characterization of thin film composite reverse osmosis membranes via novel interfacial polymerization approach[END_REF]. Similarly, we have developed polyamide bilayer membranes (HFAPA-on-REFPA) composed of an hexafluoroalcohol (HFA)-containing polyamide top-layer (HFAPA) and a conventional polyamide underlayer (REFPA) with enhanced RO separation performance by using the sequential interfacial polymerization process of MPD(aq), TMC(hx), and hexafluoroalcohol-containing diamine (HFAMDA) [START_REF] La | Enhanced desalination performance of polyamide bi-layer membranes prepared by sequential interfacial polymerization[END_REF].

Although the sequential interfacial reaction or polymerization (SIP) enables the formation of a very thin and durable coating (safe from delamination) on a conventional polyamide membrane, each process step must be well controlled to obtain reliable and consistent membrane performance. One key feature of SIP dipping process that we have identified in this study is cross-contamination of water-soluble diamine monomers. As shown in Figure 1, to fabricate a performance enhancing layer on a conventional polyamide (REFPA) via the SIP process, a roll or flat-sheets of REFPA membrane prepared by the interfacial reaction of MPD in the first aqueous solution (I) and TMC in an organic solution will continuously pass through the second aqueous solution (II) containing another amine functional monomers (ex. HFAMDA) in serial order. In this sequential process, the contamination of the second aqueous solution by the water soluble-, unreacted MPD monomers, which are transferred from the previous reaction step, could be a significant issue, causing a huge variation in the material compositions and RO performance of the resulting polyamide membranes; RO performance at the end of the roll (or the last sheet) could be very different from that at the starting roll (or the first sheet) due to the accumulation of MPD contaminant in the 2 nd amine solution. This cross-contamination can be minimized or eliminated by applying reaction solutions onto only top surface of membranes by using a spray or other equivalent methods since most of the MPD monomers are delivered by the thick porous support layer during a membrane dipping process. However, an optimum amount of MPD involvement in the 2 nd interfacial reaction along with a base diamine monomer (HFAMDA) may provide positive influence on the membrane performance, and it is therefore very valuable to investigate the effect of MPD cross-contamination on the material compositions of polyamide membranes and their final RO performance in a systematic way.

Here, we have prepared multiple coupons of polyamide bilayer membranes (HFAPA-on-REFPA) via sequential interfacial polymerization of MPD(aq), TMC(hx) and HFAMDA(aq) in the manners of serial membrane dipping and investigated the effect of MPD contamination on the material composition as well as desalination performance of the resulting polyamide membranes. Quantitative analysis of the MPD leaching and its reactivity in the 2 nd aqueous solution was confirmed by UV spectroscopic study, and the relative material compositions of polyamide bilayer membranes were characterized by Xray photoelectron spectroscopy (XPS). RO desalination performance evaluated by using cross-flow filtration of saline feed water demonstrated that a trace amount of MPD is necessary in the 2 nd diamine solution (HFAMDA(aq)) to optimize overall desalination performance of the polyamide bilayer membranes. The ideal composition between MPD and HFAMDA to form the best performing HFAPA top-layer was also identified through an in-depth study of the performance of copolyamide top-layers, which were prepared by adding various known-amounts of MPD into the 2 nd HFAMDA solution and also by applying a frame process (2 nd amine solution was applied only top surface of membrane) to eliminate uncontrollable MPD contamination.

Experimental

Materials

3,3'-bis(1-hydroxy-1-trifluoromethyl-2,2,2-trifluoroethyl)-4,4'-methylenedianiline (HFA-MDA) was provided by Central Glass, Inc. (Japan) and used as received. m-phenylenediamine (MPD flakes, > 99 %) and trimesoyl chloride (TMC, 98 %) were purchased from Sigma-Aldrich. MPD was purified by sublimation, and TMC was distilled before use. Sodium chloride (NaCl), sodium hydroxide (NaOH, pellets) and sodium carbonate (Na2CO3) were purchased from J. T. Baker. Hexane was purchased from OmniSolv. Deionized (DI) water was used for all experiments. Polysulfone (PSF) ultrafiltration membranes (PS20) were purchased from Sepro Membranes, Inc. and used as supports for RO fabrication.

Fabrication of TFC Polyamide Membranes

Membrane fabrication was performed by either a dipping process or a frame process.

The dipping process required a polysulfone (PSF) ultrafitration membrane to be submerged in solution and held into place by two glass plates at opposite ends. The frame process required the membrane to be clamped between two frames and only the top surface of the membrane was exposed to the reaction solution, allowing for a one-sided interfacial polymerization. Where the frame process was not mentioned, the dipping process was used for the membrane fabrication process.

Reference polyamide TFC membrane (REFPA):

A PSF UF membrane was placed in an aqueous solution of 2% (w/v) MPD for 2 min, and the membrane was rolled with a rubber roller to remove excess solution. The MPD-saturated PSF membrane was then immersed in a solution of 0.1% (w/v) TMC in hexane. After 1 min, the TMC solution was decanted, and the resulting membrane was rinsed with an aqueous solution of 0.2% (w/v) sodium carbonate and stored in DI water until use.

Polyamide Bilayer TFC Membranes (HFAPA-on-REFPA) with same solution bath (for cross-contamination study): A polyamide bilayer membrane, HFAPA-on-REFPA, was synthesized on a pre-formed polysulfone (PSF) ultrafiltration membrane by sequential interfacial polymerization. The PSF membrane was placed in an aqueous solution of 2 % (w/v) MPD for 2 min, and the MPD soaked support membrane was then rolled with a rubber roller to remove excess solution. The MPD saturated membrane was then immersed in a solution of 0.1 % (w/v) TMC in hexane. After 1 min, the resulting membrane was air dried at ambient temperature and placed in an alkaline aqueous solution containing 2 % (w/v) HFA-MDA diamine for 2 min (2 eq. of NaOH per mole of HFA-MDA were added to completely dissolve the HFA-MDA monomer in water). The resulting membrane was subsequently rinsed with an aqueous 0.2 % (w/v) sodium carbonate solution for 5 min, and stored in DI water until use. To investigate the effect of MPD cross-contamination on the final RO performance, 4 coupons (4" x 5") of the polyamide bilayer membrane were prepared by using the same reaction solutions (MPD(aq), TMC(hx), and HFAMDA(aq)).

Polyamide Bilayer TFC Membranes (HFAPA-on-REFPA) by a frame process (Copolymerization of 2 nd layer):

REFPA was first prepared by the dipping process. Then, copolyamide top-layers were synthesized by exposing the top surface of REFPA membranes to the 2 nd aqueous solutions by using a frame process. The 2 nd aqueous solutions to form copolyamide top-layers were formulated by adding various amounts of MPD additives (1 mol % ~ 5 mol % relative to HFAMDA) into 2 wt% HFAMDA base solution.

Characterization of TFC Membranes

X-ray Photoelectron Spectroscopic (XPS) analysis: XPS analysis was performed using a Physical Electronics Quantum 2000 ESCA Microprobe with a monochromatic Al Kα source and charge neutralization. 1100 eV survey spectra were taken at 188 eV pass energy, 200 μm spot size, and 45º take-off angle for each sample to determine the overall elemental composition of the membrane's active layer surface. High resolution spectra (58 eV pass energy, 0.5 eV/step) were taken for carbon (1s), oxygen (1s), nitrogen (1s), and fluorine (1s) to investigate the chemical environment. Binding energies were referenced to C (1s) maximum = 284.8 eV.

Atomic force microscopic (AFM) analysis: Membrane surface topographic images

were acquired by AFM using a Digital Instruments Dimension 3100 atomic force microscope with a Nanoscope IV controller (Woodbury, NY). Samples were dried under a vacuum prior to analysis and the AFM images were acquired under ambient conditions in intermittent contact mode (tapping mode) at a 1 Hz scan rate and 256 × 256 pixel resolution with silicon cantilevers (spring constant: ∼50 N/m). Three different positions were analyzed for each sample, each over a 5 µm × 5 µm area. Surface roughness was calculated using the data analysis software provided by the manufacturer.

Transmission electron microscopic (TEM) analysis: Formation of thin, dense, interfacially polymerized polyamide membranes was confirmed by cross-sectional TEM.

A membrane sample was dried and cut into 5 mm x 5 mm pieces. The non-woven fabric backing layer (of the UF support membrane) was peeled away before embedding the sample in Epotek 301 epoxy and outgassing in a vacuum chamber for 1 hour. The sample pieces were cured in a 60 o C oven for 3 hours. The samples were then prepared for TEM imaging by cutting thin sections using a Leica Ultracut S Ultramicrotome and collecting the pieces on copper grids. All samples were imaged at 200 kV in a Topcon 002B TEM.

Fourier Transform Infrared Sepectroscopic (FTIR) analysis: FTIR spectra of membranes were obtained by using Thermo Nicolet Nexus 670 FT-IR with Smart Omni Sampler designed for attenuated total reflectance infrared spectroscopy (ATR-IR). All membrane coupons were thoroughly dried under vacuum at room temperature for at least 24 hours before recording the measurements. The IR detector (MCT) was cooled with liquid nitrogen for 30 minutes before using and 128 scans for each sample were recorded with the resolution of 4 cm -1 .

Contact angle measurement:

Water contact angles of the TFC membranes were measured by a static sessile drop method using a Dataphysics Contact Angle System OCA 20 (Germany). All samples were dried under vacuum prior to analysis.

Streaming current measurement: Streaming current measurements were performed with a SurPASS electrokinetic analyzer (Anton Paar GmbH). All measurements were conducted with an adjustable-gap cell which makes possible to vary the distance between the two membrane samples with micrometric screws without dismounting the cell. The membrane was cut and adjusted to the dimensions of the sample holders (i.e. L = 2 cm and W = 1 cm) and fixed using double-sided adhesive tape. To prevent any leakage between the membranes and the sample holders, membranes were firmly pressed against sample holders for 30-60 s, paying special attention to the edges exposed to the hydrodynamic flow. The solution flow was created by a pair of syringe pumps and streaming current was measured with a pair of reversible Ag/AgCl electrodes (surface area: 10 cm 2 ).

UV analysis:

The amounts of MPD monomers transferred from previous reaction bath were measured using Agilent 8453 UV-Visible Spectrophotometer. The amounts were estimated using the calibration curve generated from known MPD concentrations. Five molar concentrations, 0.0000462 M, 0.0000925 M, 0.0001849 M, 0.0002774 M, and 0.0004624 M, were used to generate the calibration curve using linear regression to fit the following equation:

A=ε*C*L
Where A is the absorbance, ε is the constant known as extinction coefficient (M -1 cm -1 ), C is the molar concentration (M) of the MPD, and L is the pathlength through the sample.

A quartz cuvette with a pathlength, L, of 1 cm was used for the analysis. An ε value of 2281.3 M -1 cm -1 was obtained from the linear regression with an R 2 of 0.9996.

Evaluation of Separation Performance

NaCl rejection and water permeance measurement: Membrane performance was evaluated using a cross-flow filtration system. Membrane samples were clamped into the cross-flow cells, and 25 o C DI water was circulated throughout the feed loop at 27.6 bar (400 psi) and a cross-flow rate of 3.8 L min -1 for 3 hours to allow the membranes to equilibrate and reach steady state. After reaching steady state, the pure water flux was measured gravimetrically. Subsequently, 2000 ppm NaCl was added to the feed, and after allowing the system to reach steady state, the conductivities of the feed and permeate solutions were measured and converted to salt concentration via a calibration curve. The apparent salt rejection, R, was calculated as follows:

R (%) = 100 x (1-(Cp/Cf)),
where Cp and Cf are the salt concentrations of the permeate and feed solutions, respectively.

Water flux was expressed as permeate volume per membrane area per unit of time (L m -2 h -1 : LMH).

Results and Discussion

3-1. Polyamide Bilayer Membranes: The Effect of Cross-contamination on RO

Performance

As described in the Introduction, cross-contamination during sequential interfacial polymerization (SIP) might be a critical issue causing large variations in membrane performance. In order to investigate the effect of MPD contamination in the 2 nd aqueous solution on the RO performance of the resulting membranes, polyamide bilayer membranes (HFAPA-on-REFPA) composed of an aromatic polyamide under-layer (REFPA) and a HFA-substituted aromatic polyamide top-layer (HFAPA) were fabricated by sequential interfacial polymerization of MPD(aq), TMC(hx), and HFAMDA(aq) in the manner of sequential membrane dipping, and 4 sample coupons were fabricated with the same reaction solutions in serial order as shown in Figure 1. For comparison, TFC membranes with a single layered conventional aromatic polyamide layers (REFPA) were also prepared by one-step interfacial polymerization of MPD and TMC. The formation of thin and dense polyamide active layers was confirmed by cross-sectional TEM images, however it was difficult to identify a difference in the active layer thickness between REFPA and HFAPA-on-REFPA membranes as we demonstrated in our earlier paper (active layer thickness: approximately 80 ~ 85 nm dense film with 250 ~ 300 nm overall thickness including extended branch-like arms, Supporting Information, Figure S1) [START_REF] La | Enhanced desalination performance of polyamide bi-layer membranes prepared by sequential interfacial polymerization[END_REF]. towards the last (4 th ) sample coupon, while salt rejection slightly increased from the 1 st coupon (99.4 %) to the 4 th coupon (99.6 %). When compared with desalination performance of REFPA (water flux: 57  7 LMH, salt rejection: 98.9  0.5), all four samples showed much higher salt rejection with similar or lower water flux (1st sample coupon showed similar water flux to REFPA). However, this trend was not observed when we prepared four coupons of REFPA by dipping into the same MPD(aq) and TMC(hx) solutions in serial order, which indicates that the observed performance trend (flux reduction and rejection increase from 1 st to 4 th coupons) from the polyamide bilayer membranes are solely generated in the 2 nd reaction solution. The most plausible mechanism to explain this phenomenon is that, as we initially expected, the unreacted water-soluble monomers (MPD) carried over from the first reaction step dissolve and accumulate in the second aqueous solution, and ultimately participate in the second interfacial reaction with HFAMDA monomers to form a much denser copolyamide structure as a top-layer. Copolymer-formation on the top-surface was also confirmed by XPS and contact angle analysis. As shown in Table 1, the surface fluorine signal (F(1s)) and an atomic ratio of fluorine to carbon (F/C) decreased from 1 st coupon towards 4 th coupon. This result supports that MPD incorporation to the top HFAPA layer gradually increases as the amount of MPD contaminant in the 2 nd HFAMDA solution increases.

The slight reduction of surface water contact angles depending on the membrane fabrication order (133 o for the 1 st coupon to 120 o for the 4 th ) also indicates that the relative content of hydrophobic HFA groups decreases due to more MPD involvement in the 2 nd interfacial reaction.

3-2. Quantitative Analysis of MPD Leaching

The MPD contamination issue and the performance trend we observed from polyamide bilayer membranes (HFAPA-on-REFPA) brought up the question about actual compositions of the bilayer membranes and even the REFPA membranes we have prepared as our internal control. In order to clarify material compositions, MPD leaching and its involvement in the 2 nd interfacial reaction were quantitatively monitored by using UV-visible spectrophotometer through two case studies as shown in Figure 3. For Case 1 (Figure 3(a)), four coupons of REFPA membrane were prepared by the single interfacial reaction of MPD(aq) and TMC(hx), and the membrane coupons were directly immersed into fresh deionized water (2 min-immersion for each sample coupon) in serial order. Each coupon had a fixed dimension of 4" x 5" for quantitative analysis. In general, a PSF support will capture lots of MPD within its pores (MPD aqueous solution fills up the pores of PSF support) as well as its surface when it was initially soaked to a MPD(aq) solution. By dipping the MPD-saturated PSF support into a TMC(hx) solution, the MPD molecules on the membrane surface react with TMC, but the MPD molecules in the pores still remains without leaching in the TMC-organic solvent. As a result, the REFPA membrane prepared by the 1 st interfacial reaction typically has large amounts of unreacted MPD in the support layer and unreacted TMC deposited on its surface. When this REFPA membrane was immersed into deionized water, water soluble MPD monomers leaches out, and consequently some of them could directly react with the surface-remaining unreacted TMC monomers (2 nd interfacial reaction). If so, the UV signals of the water solution would be originated from only the amount of the leftover-MPDs from the 2 nd interfacial reaction. To confirm this in-situ 2 nd interfacial reaction between the released-MPD and the surface residual TMC molecules, Case Study 2 was also performed as shown in Figure 3(b). In the Case 2, unreacted free TMC monomers deposited on the REFPA was thoroughly rinsed with hexane before dipping into deionized water. Since the REFPA surface also has reactive acid chloride groups, which are chemically bound to the polyamide layer, very small portion of MPD released in the water could still be consumed by some of these surface-anchored acid chlorides. But, MPD amounts detected by the UV intensities of the water solution would be much closer to the actual amount of MPD released from membrane coupons in this case, compared to the Case Study 1.

Figure 4(a) and 4(b) shows the UV intensities of aqueous rinse solutions from the case studies 1 and 2, respectively. In both cases, strong absorption peaks at 290 nm were observed, and the intensity of the peak increased by dipping more membrane coupons in serial order, indicating that unreacted MPD monomers leach out and accumulate in the aqueous rinse solutions. The actual UV absorption values and the concentration of MPD molecules calculated by the UV data were summarized in Table 2. As we initially hypothesized, overall UV intensities observed from the case study 2 were almost 2~ 3 times higher than those observed from the case study 1. This result supports that some portion of the released-MPD have immediately reacted with surface-remaining free TMC molecules for the Case Study 1. Moreover, for the polyamide bilayer membranes (HFAPA-on-REFPA) discussed in the previous section, it could be inferred that almost 1 mol % (for 1 st coupon) to the 5 mol % of MPD contaminants (for 4 th coupon) relative to the total amount of HFAMDA monomers (the concentration of HFAMDA: 0.038 mol/L) were present in the 2 nd aqueous solution although it is still unclear how much of them were actually involved in the 2 nd interfacial reaction.

Materials Compositions and Corresponding RO Performance

The case studies we performed to monitor MPD leaching and its reactivity showed that the material compositions of REFPA could be slightly different depending on its rinsing processes. To understand the structure-property relationship, RO separation performance of the REFPA membranes prepared by Case Study 1 (REFPA rinsed w/ DI water) and Case Study 2 (REFPA rinsed w/Hexane and subsequently w/ DI water) were measured by crossflow filtration of 2000 ppm NaCl solution and compared with that of polyamide bilayer membranes (HFAPA-on-REFPA) discussed in the previous section (section 3.1. Figure 2). Four sample coupons for each membrane type were evaluated and the performance was averaged for comparison. Figure 5 Based on the UV study, approximately 6 x 10 -5 mole of more MPD molecules were consumed to form the additional polyamide bonds on the surface of REFPA_W, compared to REFPA_HW. Since the difference between REFPA_W and REFPA_HW was caused by such a trace amount of MPD molecules, it was very difficult to identify any morphological and physical differences between these two membranes by using surface characterization techniques. Both membranes showed a characteristic "peak-andvalley" topographic structure that is typically observed in commercial polyamide membranes with very similar surface roughness as confirmed by AFM (surface roughness: about 60 nm, Supporting Information, Figure S2). Streaming current measurement also showed similar density of surface charges, which originated from surface carboxyate (COO -) functional groups under neutral pH conditions (about -35 mV at pH = 7, Supporting Information, Figure S3). However, these two membranes showed a clear difference in the desalination performance, and the REFPA_W membrane showed a higher salt rejection value with similar water flux (98.5  0.6 %, 61  6 LMH) compared to REFPA_HW (92.8  2.2 %, 57  4 LMH). It seems that the additional polyamide bonds densify and/or repair the originally-formed polyamide under-layer, resulting in the enhanced salt rejection. The HFAPA-on-REFPA bilayer membrane prepared by a SIP dipping process is expected to be comprised of a copolyamide top-layer (HFA-polyamide and small portion of REFPA) and the original REFPA under-layer (Figure 5(c)). Since a large portion of surface remaining TMC molecules were reacted with HFAMDA monomers as well as a trace amount of MPD contaminants in the 2 nd reaction bath, the HFAPA-on-REFPA membrane showed a relatively lower density of negative charges under neutral conditions (zeta potential: about -20 mV, less COO -groups) compared to two REFPA membranes (Supporting information, Figure S3). The polyamide bilayer membranes showed further improvement in the salt rejection (99.4  0.1 %) compared to REF_W, although water flux was slightly decreased. The enhanced salt rejection must be attributed to the hydrophobic nature of surface HFA functional groups as we've demonstrated previously [START_REF] La | Enhanced desalination performance of polyamide bi-layer membranes prepared by sequential interfacial polymerization[END_REF]. However, to get consistent and reliable desalination performance with the polyamide bilayer membranes, careful control of the SIP process is required, such that the membrane fabrication order doesn't affect the membrane performance. Moreover, it is important to find an optimum composition of the top surface layer to achieve the best combination of salt rejection and water flux.

Polyamide Bilayer Membranes: Optimum Material Composition for High RO

Performance

To find an optimum monomers composition for the best desalination performance, polyamide bilayer membranes were fabricated by adding various amounts of MPD into HFAMDA aqueous solutions (mole concentration of HFAMDA: 0.038 mol/L), and by using a frame process in the 2 nd reaction step (Figure 6). In the frame process, uncontrollable MPD cross-contamination was eliminated since the 2 nd aqueous solution was exposed to only top-surface of a freshly prepared REFPA layer. The molar ratios of MPD additive to the total amount of HFAMDA in the 2 nd aqueous solutions were varied from 0 % (pure HFAMDA), 1.2 %, 2.4 %, 4.8 %, and 100 % (pure MPD). The relative material compositions of the resulting polyamide bilayer membranes were confirmed by FTIR spectroscopic analysis. As shown in Figure 7, all spectra showed characteristic peaks originated from polyamide active layers (amide bonds: 1664 and 1530 cm -1 , aromatic rings: 1585, 1504, 1484, and 1107 cm -1 ) and PSF substrates (1321, 1290, 1172, 1150, and 1245 cm -1 ), and there was no significant difference in these peaks among the samples. However the weak peak at 962 cm -1 , which is generated by HFA functional groups, gradually decreased toward the polyamide bilayer membranes prepared by adding more MPD additives (Green Boxed Region in Figure 7, (a) ~ (d): 0 ~ 4.8 mol % MPD additives), indicating the reduction of HFAMPDA monomer content in the copolyamide top-layers. Finally, the peak at 962 cm -1 disappeared completely when the sample was prepared by using 100 % pure MPD solution (Figure 7(e)).

Figure 8 shows water flux and salt rejection of the resulting polyamide bilayer membranes. It is worthy noting that the membranes for this experiment were prepared by using a new batch of PSF support so that a single-layered REFPA over this new PSF support represented 99.4 % of salt rejection and 44 LMH of water flux. For 0 mol% MPD addition, pure HFAPA top-layer (0 % MPD addition) formed on a REFPA underlayer, and the resulting bilayer membrane showed very good desalination performance (99.6 % salt rejection and 46 LMH), which outperforms REFPA. When 1.2 mol % of MPD monomer was added into the HFAMDA solution to form the copolyamide top-layer, the salt rejection was even further increased and maximized without adding significant resistance to water transport (99.8 % salt rejection and 45 LMH). By adding more than 2.4 mol % of MPD, however, both water flux and salt rejection gradually decreased. The water flux reduction trend is consistent with the result we have obtained during the crosscontamination study in Section 3.1. Finally, when the top surface layer was formed by pure MPD monomers (100 % MPD), the resulting REFPA-on-REFPA showed very low salt rejection as well as low water flux. Significant reduction in the water flux is likely due to the formation of a much tighter polyamide network on the top surface since monophenyl MPD monomer is more reactive and smaller than bi-phenyl HFAMDA monomer.

In general, salt rejection rate is strongly influenced by the surface phenomenon such as an interaction between membrane surface and the salts in the feed. The low salt rejection value obtained in the REFPA-on-REFPA indicates that the top REFPA layer (must be amine-rich surface) is not effective enough to repel salts from the membrane surface, and also verifies that HFA functional groups on the surface play an important role to enhance salt rejection behavior. From this study, it is concluded that the 80:1 molar ratio of HFAMDA and MPD in the 2 nd aqueous solution provides the best-performing copolyamide top-layer. Enhanced desalination performance by the copolyamide top-layer seems to be attributed to the combination of the following two factors: (a) hydrophobic nature of HFA functional groups covering the membrane surface and (2) filling and/or repairing any defect sites by the trace amount of small MPD monomer as confirmed by the case studies in Section 3.3 (REFPA_W vs. REFPA_HW).

Conclusions

The sequential interfacial reaction or polymerization (SIP) is a very straightforward way to modify a conventional polyamide membrane with the aim of improving the membrane's desalination performance, fouling resistance, and/or chlorine resistance. To achieve reliable and consistent performance of the modified membrane, however, basic understanding on the chemistries involved in each reaction process is indeed necessary.

In this study, we have demonstrated the significant impact of cross-contamination in a SIP dipping process while making multiple coupons of polyamide bilayer membranes (HFAPA-on-REFPA) with the same reaction solutions. For the SIP process, unreacted water-soluble MPD monomers carried over from the first interfacial reaction step dissolved and accumulated in the second aqueous solution containing HFAMDA monomers, and ultimately participated in the second interfacial reaction to form copolyamide at the top-layer. Depending on the amount of MPD contaminants accumulated in the second aqueous solution, the composition of the resulting copolyamide in the top-layer was varied, resulting in a significant variation of desalination performance (flux reduction and slight increment of salt rejection from 1 st sample coupon toward 4 th sample coupon).

MPD leaching in the 2 nd aqueous solution and the subsequent reaction of the released MPD with surface-remaining TMC molecules were clearly confirmed by two case studies conduced with UV spectroscopic analysis (Case 1: REFPA  dipping into DI water  UV analysis, and Case 2: REFPA  rinsing with hexane  dipping into DI water  UV analysis). These case studies indicated that the material compositions of REFPA could be slightly different depending on its rinsing processes as well. REFPA_W prepared by Case Study 1 must have additional polyamide bonds when compared to REFPA_HW prepared by Case Study 2 due to the direct reaction of the released-MPD and surface-remaining TMC molecules during the water rinse process. Performance comparison among REFPA_HW, REFPA_W, and HFAPA-on-REFPA (by SIP dipping process) supported that a trace amount of MPD involvement in the 2 nd interfacial reaction might be necessary to improve overall desalination performance (salt rejection: 92.8 % for REFPA_HW  98.5 % for REFPA_W) and also verified that HFA functional groups covering the membrane top surface play an important role in the enhancement of RO separation behavior (salt rejection: 99.4 % for HFAPA-on-REFPA).

Through in-depth evaluation on the performance of top surface layers, which were prepared by adding various known-amount of MPD into 2 nd HFAMDA solution (copolymerization) and also by applying a frame process (no cross-contamination issue), we have successfully identified an ideal composition (molar ratio of 80:1) between HFAMDA and MDA to achieve the best RO desalination performance (99.8 % salt rejection, 45 LMH water flux).

This study clearly addresses an important factor that should be considered during sequential interfacial polymerization to obtain a reliable and high performing coating layer on a conventional polyamide membrane. The 2 nd layer-copolymerization approach (trace amount of MPD addition to the HFAMDA solution), which we have performed to optimize RO separation behaviors of polyamide bilayer membranes, could be also applied to other material systems (ex. anti-fouling materials) to improve overall desalination performance. Tables Table 1. XPS F(1s) concentration , atomic ratio (F/C), and water contact angle of the layered polyamide membranes fabricated by the SIP dipping process in serial order with the same solution baths.

Figures Figure 1.

Figure 2 (

 2 Figure 2(a) and 2(b) shows water flux and salt rejection of four HFAPA-on-REFPA

  represents the structures of two REFPA membranes and HFAPA-on-REFPA membranes analogized from their fabrication processes and UV analysis data. The REFPA membrane treated by hexane and water (REFPA_HW) would have a relatively thin layer of aromatic polyamide (Figure 5(a), by case study 2), while the REFPA membrane rinsed with only water (REFPA_W) would have a slightly thicker and/or denser polyamide layer because some of released-MPD molecules in the water directly reacted with surface-remaining free TMC molecules to form additional polyamide bonds (Figure 5(b), by case study 1).

  Figure 1. A schematic drawing showing a cross-contamination issue in the sequential interfacial polymerization (SIP) process based on the subsequent membrane dipping.

Figure 2 .

 2 Figure 2. Water flux (a) and salt rejection (b) of REFPA and four coupons of polyamide bilayer membranes (1 st ~ 4 th ) prepared by sequential interfacial polymerization in the manner of serial membrane dipping.

Figure 3 .Figure 4 .Figure 5 .

 345 Figure 3. Schemes for the case studies to prove MPD leaching and its reactivity in the 2 nd aqueous solution

Figure 6 .Figure 7 :Figure 8 .

 678 Figure 6. A photograph showing a frame process to form polyamide bilayer membranes featuring HFAPA top-layer (copolymerization of HFAMDA and MPD additive) on a REFPA under-layer without cross-contamination

Table 2 . 4 Figure S3 :

 24S3 Figure S3: pH-dependant zeta-potential curves of REF_W (blue, prepared by Case Study 1), REF_HW (Black, prepared by Case Study 2), and HFAPA-on-REFPA (Red, prepared by the scheme in Figure2). All experiments were performed in a millimolar KCl solution

nd aqueous solution: HFA-MDA + MPD additive REFPA

Acknowledgements

This work was performed under a joint development agreement between IBM Research and the King Abdulaziz City for Science & Technology (KACST).

Supporting Information