Matias Berdakin

Géraldine Féraud

Claude Dedonder-Lardeux

Christophe Jouvet
email: christophe.jouvet@univ-amu.frgpino@fcq.unc.edu.ar

Gustavo A Pino

Excited states of protonated DNA/RNA bases

The very fast relaxation of the excited states to the ground state in DNA/RNA bases is a necessary process to ensure the photostability of DNA and its rate is highly sensitive to the tautomeric form of the bases.

Protonation of the bases plays a crucial role in many biochemical and mutagenic processes and it can result in alternative tautomeric structures, thus making important the knowledge of the properties of protonated DNA/RNA bases. We report here the photofragmentation spectra of the five protonated DNA/RNA bases. In most of the cases, the spectra exhibit well resolved vibrational structures, with broad bands associated with very short excited state lifetimes. The similarity between the electronic properties e.g. excitation energy and very short excited state lifetimes for the canonical tautomers of protonated and neutral DNA bases, suggests that the former could also play an important role in the photostability mechanism of DNA.

Introduction

Understanding the response of the DNA/RNA bases to UV radiation is critical since their photochemistry constitutes a fundamental step in radiation-induced DNA/RNA damage and very likely it had also played an important role in prebiotic chemistry. 1 In the recent years, experimental and theoretical groups have been working extensively and settling benchmark results to disentangle the intrinsic properties of these building blocks of life in the gas phase. These studies have been recently reviewed in three complete papers. 1- 2 3 The neutral DNA bases are structurally rich given that they can exist in many tautomeric forms.

However, nature has chosen those structures (the canonical forms) shown in Fig. 1, that allow for a precise recognition between C (cytosine) and G (guanine) through three H-bonds and between T (thymine) or U (uracil) and A (adenine) through a double H-bond, to guarantee the preservation of the genetic information encoded in the DNA molecule. The excited state lifetimes of DNA bases depend strongly on their tautomeric form, the canonical tautomers being those with the shortest excited state lifetime, which is considered to be the natural photoprotection mechanism of DNA. [1][2][3] DNA bases are susceptible to protonation through proton transfer or other reaction mechanisms. 4,5 The protonation processes are important because alternative tautomeric structures of the DNA bases can be formed. The existence of such non-canonical tautomers could induce longer excited state lifetimes [1][2][3] or development of mutations in the nucleic acid replication because the Hbonding structure changes leading to failures in the recognition process. An excited state proton/hydrogen transfer reaction in the Watson Crick (WC) GC pair, has been identified by Sobolewski and Domcke 11 as one of the most important deactivation pathways of the photoexcited pair in the gas phase. Therefore, in a bottom-up strategy to build up a whole photochemical and photophysical model of DNA, understanding the structure of protonated DNA bases, their tautomeric forms, stability and the effects of protonation on the excited state lifetime and absorption spectra is of imperative relevance to calibrate the calculations with experimental results. However, very few detailed studies have been done on these protonated bases 5-7, 12- 13 14 as compared with the numerous studies performed on the neutral species. [1][2][3] We performed a detailed study of the electronic spectroscopy of the five cold protonated DNA/RNA bases (CH + , TH + , UH + , GH + and AH + ) by means of photofragmentation spectroscopy in a cold ion trap that allows getting information about their excitation energies and excited state lifetimes. Structural assignment is obtained with ab-initio calculations in the ground and excited states.

Methodology

2.a. Experiment

The electronic spectra of the protonated aromatic DNA/RNA bases were obtained via parent ion photo-fragment spectroscopy in a cryogenically-cooled quadrupole ion trap (Paul Trap from Jordan TOF Products, Inc.). (15) The setup is similar to the one developed in several groups based on the original design by Wang and Wang. 16 built at Aarhus University. 19 At the exit of the capillary, ions are trapped in an octopole trap for 90 ms.

They are extracted by applying a negative pulse of c.a. 50 V and are further accelerated to 190 V by a second pulsed voltage just after the exit electrode. This time sequence of pulsed voltages produces ion packets with duration between 500 ns and 1 µs. The ions are driven by a couple of electrostatic lenses toward the Paul trap biased at 190 V so that the ions enter the trap gently avoiding fragmentation induced by collisions. A mass gate placed at the entrance of the trap allows selecting the parent ion. The Paul trap is mounted on the cold head of a cryostat (Coolpak Oerlikon) connected to a water-cooled He compressor. Helium as buffer gas is injected in the trap using a pulsed valve (General Valve) triggered 1 ms before the ions enter the trap as previously reported by Kamrath et al. 18 The ions are trapped and thermalized at a temperature between 20 and 50 K through collisions with the cold buffer gas (see S.I.).

The ions are kept in the trap for several tens of ms before the photodissociation laser is triggered. This delay is necessary to ensure thermalization of ions and efficient pumping of the He buffer gas from the trap to avoid collision induced dissociation of the ions during the extraction towards the 1.5 m long timeof-flight mass spectrometer. After laser excitation, the ions are stored in the trap for a delay that can be varied between 20 and 90 ms before extraction to the TOF mass spectrometer. The complete mass spectrum is recorded on a micro channel plate (MCP) detector with a digitizing storage oscilloscope interfaced to a PC. The photofragment yield spectrum of each detected ion is normalized to the parent ion signal and the laser power.

The photo-dissociation laser is an OPO laser from EKSPLA, which has a 10 Hz repetition rate, 10 ns pulse width, a resolution of 10 cm -1 and a scanning step of 0.02 nm. The laser is shaped to a 1mm 2 spot to fit the entrance hole of the trap and the laser power is around 20 mW in the UV spectral region.

2.b. Calculations

Ab initio calculations have been performed with the TURBOMOLE program package, 20 making use of the resolution-of-the-identity (RI) approximation for the evaluation of the electron-repulsion integrals. 21 The equilibrium geometries of the protonated species in their ground states (S 0 ) have been determined at the MP2 (Møller-Plesset second order perturbation theory) level. Vertical excitation energies of the lowest excited singlet states have been determined at the RI-ADC(2)(second order Algebraic Diagrammatic Construction) level. 22 Due to numerous conical intersections between the n and * states, the excited state optimization is not straightforward and failed in many systems as shown already for protonated adenine. 12,13 Calculations were performed with the correlation-consistent polarized valence double-zeta (cc-pVDZ or aug-cc-pVDZ) basis set. 23 The vibrations in the ground and excited states have been calculated for the enol/1H-2H [START_REF]We use the notation 1H-2H to indicate the hydrogen and proton positions on nitrogen numbered 1 and on the oxygen linked to carbon 2[END_REF] tautomer of protonated cytosine and the electronic spectra simulated using the Pgopher software [START_REF] Western | PGOPHER, a Program for Simulating Rotational Structure[END_REF] for Franck Condon analysis.

Results

The spectra of all the cold protonated DNA/RNA bases are presented in Fig. 2. The laser has been scanned at lower energy up to 410 nm but no signal was detected below the bands shown here.

For all the protonated molecules, the fragments observed in these experiments are the same as those observed in collision induced dissociation (CID) or in infrared multi-photon dissociation (IRMPD):

NH 3 loss, H 2 O loss and HNCO/H 2 NCN loss, which means that very likely these fragments come from dissociation in the ground state following internal conversion (IC). Internal conversion following UV excitation (e.g. 310 nm) leads to the production of ions in the S 0 state with 4 eV (32000 cm -1 ) of vibrational excitation, which is above the dissociation threshold of all the systems studied in this work.

This allows us to neglect the possibility of missing of some transition due to low dissociation yield.

Most of the bands are very broad except those of GH + in the low energy region of the spectrum for which the width of the vibrational bands is limited by the laser bandwidth and the rotational contour (12 cm -1 ) and the low energy band system of UH + , which has only slightly broader bandwidths (19 cm -1 ). In the case of the protonated pyrimidine bases, for CH + and UH + clearly two band systems (hereafter called the blue and red systems according to their absorption region) are observed, which might be due to two electronic states or two isomers. However, only one system is observed for TH + in the (34250 -44250) cm -1 spectral range, while the low resolution spectra in the (33330 -30300) cm -1 spectral region did not show any transition associated to TH + (see S. I.) An enlarged view of the spectra near the band origins for the protonated pyrimidine bases is shown in Fig. 3. The broad character of the vibrational bands is noticeable except for the red system of UH + . Although it has to be taken with caution, the widths of the bands were fitted by a Lorentzian function to extract excited state lifetimes using the uncertainty principle, and the validity of this assumption will be discussed in section 4.a. An enlarged view of the spectra near the band origin for the protonated purine bases is depicted in Fig. 4. Protonated guanine shows a well resolved spectrum with narrow bands (FWHM= 12 ± 2 cm -1 ).

Protonated adenine presents two band systems composed of a very broad absorption band on the red part of the spectrum and a group of broad peaks starting at 1000 cm -1 to the blue of the origin of the red band. It must be noticed that for the very broad part of the spectrum in the low energy region, the intensity of the two fragments (m/z 94 (-H 2 NCN) and 119 (-NH 3 )) is the same whereas for the blue band system the -NH 3 fragment is more intense than the -H 2 NCN fragment, indicating the existence of two electronic states or isomers. The excited state lifetimes for GH + and the blue band system of AH + obtained from the fit of the bandwidth are also reported in Table 1. ) the two fragmentation channels have the same intensity whereas on the bands at 1100 and 1600 cm -1 the m/z 119 fragment (NH 3 loss) is more intense than the m/z 94 fragment (H 2 NCN loss). This most probably reflects the presence of two tautomers.

Analysis and Discussion

4.a. Determination of the excited state Lifetimes

The excited state lifetimes have been estimated from the widths of the bands fitted by

Lorentizan functions as can be seen in Fig. 3, except for GH + and UH + (red) that were fitted to Voigt profiles to account for the rotational contour and the laser Gaussian bandwidth (12 cm -1 ) convolution.

The fitting of the other spectra gives the same Lorentzian bandwidth independently of fitting to Voigt or Lorentzian profiles.

The relationship between the Lorentzian bandwidth and the excited state lifetime is given by the uncertainty principle which can be written as:

12 11 5.3 10 ( ) (2 c ) () x FWHM cm s (Eq. 1)
The Full Width at Half Maximum (FWHM) of the peak is given in cm -1 and the lifetime in seconds. This procedure is valid if it can be assumed that the broadening is only due to the excited state lifetime and not to spectral congestion (e.g. rotational contour or low frequency active modes).

For the red band system of UH + (Fig. 3) as well as for GH + (Fig. 4) that present well resolved narrow vibrational structure, no low frequency progressions (lower than 100 cm -1 ) are observed.

Therefore, there are no reasons to expect low vibrational frequency progressions blended into a spectral background of the other bases for which very broad bands are observed.

The ground state vibrational frequencies were calculated for all the protonated bases and the lowest frequencies corresponding to out-of-plane modes are in the order of 200 cm -1 . At the low temperature of the experiment, the vibrational modes of the electronic ground state are not excited.

Thus, the band broadening is not expected to be due to hot band sequences. Under the assumption that the excited state frequencies are in the same range as the ground state frequencies (200 cm -1 ), any vibrational progression in low frequency modes would have been easily separated in the absence of lifetime broadening, as in the case of the red system of UH + or of GH + . It is thus reasonable to assume that the bandwidths are intrinsic and that lifetimes can be deduced.

The estimated lifetimes are listed in Table 1 and are found between 30 fs and 100 fs (±50 fs) for most of the species except for GH + and the red band system of UH + that shows only a small broadening as compared to GH + , and should have a lifetime longer than a few hundreds of femtoseconds. a The width of the bands in protonated Guanine (< 1 cm -1 ) and in protonated Uracil red system (16 ± 5 cm -1 ) corresponds to the Lorentzian bandwidth of a Voigt profile, given a Gaussian function (FWHM= 12 cm -1 ) that represents the convolution of the rotational contour and the laser bandwidth. For the protonated Uracil red system, the bands are only slightly larger than the guanine bands so that the lifetime may be longer than 330 fs.

We do not observe a large change of the broadening of the bands with the excess energy in contrast with other systems such as neutral adenine, for which a few bands are well resolved at low energy and a continuous absorption is observed at higher energy indicating a shorter lifetime. [START_REF] Kim | [END_REF]27 In protonated bases, the non-radiative processes leading to the fast decays are present on the 0-0 transition and do not change drastically at least for the first 1000 cm -1 above the origin. At higher energy the spectra become structure-less but this is most probably due to spectral congestion.

It is worth noting that the short excited state lifetime determined here corresponds to the * excited state lifetime. These lifetimes do not give direct information on the rate of the dissociation process that leads to the photofragments detected. The photofragmentation takes place most likely in the electronic ground state, following fast IC as in the case of neutral DNA bases, which is expected to be the main deactivation pathway associated with the * excited state lifetime.

In the gas phase isolated system, IC produces highly vibrationally excited ions for which one possibility for energy dissipation is the unimolecular dissociation. However, in bulk systems, the major energy dissipation mechanism is vibrational relaxation through energy transfer to the solvent and the environment or even IVR to the whole DNA molecule, insuring the stability of the molecule. [1][2][3][4]

4.b Tautomers Assignment

Generally for neutral and protonated aromatic molecules, geometry optimization of the first excited state at the ADC(2) level leads to adiabatic transition energies in very good agreement with experimental values, the mean error being around ± 0.2 eV. 15,28 2930 -31 32 In the present study, it was possible to optimize only the excited state of the most stable isomer of CH + (the enol/1H-2H tautomer). The calculated adiabatic transition is 4.62 eV, in very good agreement with the experimental value of the origin of the blue system (4.63 eV), being within the usual 0.2 eV (even much smaller in this case). For this case, the assignment of the spectrum seems quite certain (see below A-Cytosine).

As a general trend, the excited state optimization process for aromatic molecules (neutral or protonated) leads to a stabilization of c.a. 0.5 eV as compared to the vertical excitation energy (E vert ). 15,28- to an adiabatic transition energy E ad = E vert -0.5 eV = 4.65 eV in very good agreement with the experimental value (4.63 eV) and the calculated adiabatic transition energy (4.62 eV). Therefore, the empirical rule:

E exp =E vert -(0.5 ± 0.3 eV) (Eq. 2)
that correlates the experimental excitation energy (E exp ) with the calculated vertical excitation energy (E vert ), was used to assign all the observed electronic spectra, which can all be assigned to the most stable ground state tautomers of the protonated bases.

The calculated vertical excitation energies together with the experimental transition energies and the difference vert -0.5) -E exp for the main tautomers are summarized in Table 2.

Table 2. Assignment of the electronic transitions to different tautomers of the protonated DNA/RNA bases. The crossed enol-keto protonated thymine tautomer might be populated in the ground state due to its low energy, but is not observed. For the ground state (S 0 ) the energies are relative to the energy of the most stable tautomer. All units are in eV.

A-CytosineH +

For CH + we clearly detect the 2 lowest energy isomers. Both have very distinct band origins and the vertical transition energy calculation of higher energy states excludes the assignment of the two band systems to two excited states of the same tautomer. The most stable tautomer has an enol/1H-2H

(see figure 1 for atom numbering) form as already determined by IR-MPD spectroscopy. 5,7 For this tautomer the excited state optimization was possible and the calculated adiabatic transition of 4.62 eV is in very good agreement with the experimental value of the blue band system (4.63 eV). The blue band system (Fig. 1) is thus assigned to enol/1H-2H tautomer. For this tautomer, the ground and excited state frequencies have been calculated and the spectrum can be satisfactorily simulated as shown in Fig. 5. The lowest out-of plane frequency is calculated at 95 cm -1 in the excited state and is only very weakly active with one quantum ( not visible in Fig. 5) and more active with 2 quanta (2 visible in Fig. 5). The agreement between the simulated spectrum and the experimental one is very good, showing active modes which are mainly in-plane modes. Only the mode observed at +950 cm -1 is missing in the simulation.

For the second isomer keto/1H-3H, also characterized by IR-MPD spectroscopy, 5,7 the * optimization leads to a crossing with a n * state through the NH 2 rotation. Thus, the excited state optimization was not successful and the adiabatic energy of the * cannot be obtained. However, taking into account the ground state energy and the calculated vertical energy, the assignment of the electronic spectrum of the red band system to the keto/1H-3H tautomer can be quite ascertained. Both tautomers have short excited state lifetimes (Table 1).

B-ThymineH +

For TH + , the most stable tautomer has an enol-enol/2H-4H structure, and the transition energy calculated with the empirical rule (Eq. 2) agrees with the observed electronic transition energy. For the second isomer, the ground state energy is calculated 0.17 eV higher than the enol-enol/2H-4H form, at the MP2 level of theory, while the energy difference calculated by Salpin et al. 5 at the B3LYP level was 0.1 eV. The electronic transition can be predicted to lie at lower energy (around 4.05-0.5= 3.55 eV/ 350 nm), but this isomer was not detected in our experiment. So we assign the TH + tautomer observed in our experiment to enol-enol/2H-4H tautomer. Similarly to CH + , the excited state lifetime of the observed tautomer is very (Table 1).

system starting at higher energy (+ 0.15 eV) is composed of a broad vibrational progression with the first band at 273 nm (4.54 eV=36630 cm -1 ). In fact, three ground state tautomers have been calculated to lie within less than 0.1 eV and their vertical electronic excitation energies are also within 0.1 eV, 12 thus the empirical rule (Eq. 2) cannot be used unambiguously. The state-of-the-art ab-initio calculations have already been performed by Marian et al., 12 thus we did not perform any calculation for this system but just compared with previous calculations. In the ground state the 1H-9H tautomer is the most stable one, followed by the 3H-7H and the 3H-9H tautomers at 0.02 eV and 0.08 eV above the energy of the 1H-9H tautomer, respectively.

The 3H-9H tautomer can probably be ruled out since its absorption lies in the 320 nm region 13 and, taking into account the ion trap temperature and the ground state energies, the population of this species is expected to be very low.

The other two tautomers (1H-9H and 3H-7H) have very similar vertical electronic transition energies at 4.83 eV and 4.80 eV, respectively. The energy gap between the two first * states of each of these two tautomers are around 0.2 -0.3 eV larger than the observed energy gap between the two experimental electronic transitions. Thus one can probably assign the two systems to different tautomers.

In the low energy system the intensity of the m/z=94 and m/z=119 fragments arising from H 2 NCN and NH 3 loss are the same whereas on the higher energy band system the m/z 119 fragment becomes more intense than m/z 94. Very likely, this difference arises from the different structures of the tautomers responsible for each absorption band and this argument can be used to assign the spectra.

The H 2 NCN fragment can be produced by direct fragmentation of the 3H-7H tautomer while in the case of the 1H-9H tautomer the formation of this fragment would require a subsequent H atom loss or a previous rearrangement of the structure. In any case, the rate of production of the H 2 NCN fragment from 1H-9H tautomer is expected to be lower that from the 3H-7H tautomer and then its branching ratio this paper provides a complete set of data that can be used as experimental benchmark for theoretical excited state dynamics calculations.

Conclusions

We have presented here a complete set of experimental data concerning the excited states of the protonated DNA/RNA bases. When the comparison with neutral homologue can be done the electronic transition energies do not change drastically between neutral and protonated species. In contrast, the excited state lifetime of the protonated species seems to be very short except for guanine and one uracil tautomer. This challenges theoretical calculations and these results are benchmark for testing ab-initio methods, in particular to identify the non-radiative processes. However standard excited state calculations constitute a good tool to allow identification of tautomers. ‡ In this paper, since the excited state lifetimes are extracted from vibrational broadening, the values can neither describe possible non-exponential decays nor variation of these lifetimes with the excess energy. Thus femtosecond pump-probe experiment would be highly desirable.

Figure 1 .

 1 Figure 1. Watson-Crick (WC) structures of the DNA base pairs. The red arrows indicate possible protonation sites of canonical tautomers through a proton transfer reaction within the WC pair.

5

 5

Figure 2 .

 2 Figure 2. Electronic photofragmentation spectra of the cold protonated DNA/RNA bases recorded on different fragmentation channels: HNCO or H 2 NCN loss in black, H 2 O or NCO loss in red and NH 3 loss in blue. Protonated cytosine (upper panel) and protonated uracil (lowest panel) present clearly two band systems, while thymine and guanine present only one. All the electronic transitions can be assigned to the lowest energy ground state tautomers.

Figure 3 .

 3 Figure 3. Low energy part of the spectra of the protonated pyrimidine bases. The vibrational bands (fitted with Lorentzian peaks) are very broad, except for the lowest electronic transition of protonated uracil red band system which was fitted to a Voigt profile with a Gaussian width of 12 cm -1 that corresponds to the laser bandwidth and rotational contour. FWHM is full width at half maximum of the Lorentzian functions used to simulate the bands of each tautomer.

Figure 4 .

 4 Figure 4. Low energy part of the spectra of the protonated purine bases. A ) Protonated guanine in the energy region between -200 and 1000 cm -1 around the 0-0 transition, which is very weak. The width of the first vibrational bands is 12 cm -1 and corresponds to the convolution of a cold rotational contour with the laser bandwidth. B) Protonated adenine in the energy region between -500 and 2000 cm -1 around the photofragmentation onset. In the lower energy part (0 -800 cm -1

b

  Taken from Ref.(12)

Figure 5 .

 5 Figure 5. In black the enol/1H-2H tautomer of CytosineH + experimental spectrum. In blue the calculated spectrum. Above each transition the vibrational mode involved for each calculated band.

Table 1 .

 1 Bandwidth and excited state lifetimes of the different electronic systems of the five protonated bases

	Protonated base	Measured	Estimated lifetimes
		FWHM (cm -1 )	(fs)
	Uracil red system a	16 ± 5	≥
	Uracil blue system	133 ± 15	40 ± 20
	Thymine	91 ± 20	58 ± 20
	Cytosine red system	40 ± 10	133 ± 20
	Cytosine blue system	54 ± 10	98 ± 20
	Adenine blue system	177 ± 20	30 ± 20
	Guanine a	< 1	> 5000

 -18 The protonated ions are produced in an electrospray ionization source

In the case of the enol/1H-2H tautomer of CH + , E vert is 5.15 eV and the stabilization is expected to lead

Acknowledgements

This works was supported by ECOS-MinCyT cooperation program (A11E02) the ANR Research Grant (ANR2010BLANC040501), FONCyT, CONICET and SeCyT-UNC. We acknowledge the use of the computing facility cluster GMPCS of the LUMAT federation (FR LUMAT 2764). ‡ We also confirmed recently the identification of different tautomers of protonated uracil with a new UV-UV hole-burning spectroscopy. 39

C-UracilH +

The spectrum of UH + shows two bands systems, which can be assigned to the two lowest energy ground state isomers. The lowest isomer has again an enol-enol/2H-4H structure as reported previously 5,6 and the first electronic transition is estimated to be at 4.94 eV from the empirical rule (Eq. 2). We can then associate the blue band system starting at 4.83 eV (257 nm) to the first electronic transition of this enol-enol/2H-4H tautomer. The other tautomer (enol-keto/3H-4H) has a ground state energy calculated at 0.08 eV at the MP2 level and 0.05 eV at the B3LYP level 5 and its electronic transition should be observed at lower energy than the enol-enol/2H-4H transition energy, according to the empirical rule that leads to a calculated transition energy around 4.39-0.5= 3.89 eV (318 nm). Indeed, the first band of the red system is observed at 314.8 nm. Therefore, the red band system is assigned to this second tautomer, enol-keto/3H-4H. In this case, as shown in Table 1, both isomers have very different excited state lifetime (Table 1), the enol-keto/3H-4H (red system) being longer lived than the enol-enol/2H-4H (blue system).

D-GuanineH +

In the case of GH + , the 7H-9H tautomer has been shown to be more stable than the other ones by at least 23 kJ/mol (0.24 eV). 33 Once more, the observed spectrum can be assigned to this tautomer using the empirical rule (Eq. 2). The spectrum presents well resolved vibrational bands that can be followed for more than 1000 cm -1 without any sign of spectral broadening, indicating the absence of very fast non-radiative decay.

E-AdenineH +

The experimental spectrum AH + is composed of two systems very close in energy. One extremely broad system, which looks like a continuum, is starting at 282 nm (4.40 eV=35460 cm -1 ) and another should be also lower. Then, we can assume that the higher energy band system is due to the 1H-9H tautomer, while the lower energy system is due to the 3H-7H tautomer. Nevertheless, an unambiguous assignment would require additional work such as IR-UV spectroscopy. Both tautomers have very short excited state lifetime (Table 1).

4.c. Lifetimes of protonated tautomers

The keto/1H-3H and the enol/1H-2H tautomers of CH + and the 1H-9H tautomer of AH + can be obtained by proton transfer (PT) reaction in the canonical WC structures (Fig. 1). We observed short excited state lifetimes for these tautomers. On the other hand, the 7H-9H tautomer of GH + (long excited state lifetime) and the enol-enol/2H-4H tautomer of TH + (short excited state lifetime) cannot be produced by PT reaction within the CG and TA WC base pairs, respectively. Therefore, the electronic properties of the canonical CH + and AH + , which are biological significant, are expected to be relevant for the DNA photostability. In line with the results obtained for neutral isolated DNA/RNA bases, for which the biologically most relevant forms exhibit the shortest excited state lifetimes, 1-3 the same rule seems to hold for the protonated bases. Canonical CH + and AH + have short excited state lifetime, while the excited state lifetime of the non-canonical GH + tautomer is longer by at least three orders of magnitude.

The excited states of the WC bases pairs have also very short lifetime, a property that is related to photostability of DNA. [START_REF] Abo-Riziq | Proc. Natl. Acad. Sci[END_REF][START_REF] Schultz | [END_REF] The processes at play have been the subject of many investigations [1][2][3]11,[START_REF] Abo-Riziq | Proc. Natl. Acad. Sci[END_REF] 36 -37 38 but it is not so easy to get a unified mechanism since there are many possible deactivation pathways, which have not all been explored. The similarity between the electronic properties e.g: excitation energy and very short excited state lifetimes for canonical tautomers of the neutral and protonated DNA bases, reported here, suggests that the latter could also play an important role in the photostability mechanisms of DNA. Thus

TABLE OF CONTENTS

The excited state lifetime of protonated DNA/RNA bases is strongly dependent on the tautomeric form.


Donghang Cheng
email: dcheng@mdanderson.org

#, ¶ Sergio Valente

Sabrina Castellano

Gianluca Sbardella

Roberto Di Santo

Roberta Costi

Mark T Bedford
email: mtbedford@mdanderson.org

Antonello Mai
email: antonello.mai@uniroma1.it.

Novel 3,5-Bis(bromohydroxybenzylidene)piperidin-4-ones as Coactivator-associated Arginine Methyltransferase 1 Inhibitors: Enzyme Selectivity and Cellular Activity

Coactivator-associated arginine methyltransferase 1 (CARM1) represents a valuable target for hormone-dependent tumors such as prostate and breast cancers. Here we report the enzyme and cellular characterization of the 1-benzyl-3,5-bis(3-bromo-4-hydroxybenzylidene) piperidin-4-one (7g) and its analogues 8a-l. Among them, 7g, 8e, and 8l displayed high and selective CARM1 inhibition, with lower or no activity against a panel of different PRMTs or HKMTs. In human LNCaP cells, 7g showed a significant dose-dependent reduction of the PSA promoter activity.

Arginine methylation of mainly nuclear proteins is a reversible post-translational modification process involved in structural remodeling of chromatin. [START_REF] Bedford | Protein arginine methylation in mammals: who, what, and why[END_REF][START_REF] Bedford | Arginine methylation an emerging regulator of protein function[END_REF] Protein arginine methyltransferase (PRMT) enzymes remove the methyl group from the donor molecule Sadenosyl-L-methionine (AdoMet) generating the product S-adenosyl-L-homocystein (AdoHcy), and transfer this methyl residue to the terminal nitrogen atom(s) of the guanidinium side chain of an individual arginine residue in the target protein. [START_REF] Smith | Chemical mechanisms of histone lysine and arginine modifications[END_REF] PRMTs are ubiquitously expressed in most cell types and tissues of the human body with the unique exception of PRMT8, which appears to be restricted to neurons in the brain. [START_REF] Taneda | Specific regional distribution of protein arginine methyltransferase 8 (PRMT8) in the mouse brain[END_REF] Moreover, they differ in their substrate specificities, and are therefore probably involved in different physiological processes. Among PRMTs, PRMT4/CARM1 (coactivator-associated arginine methyltransferase 1) was the first to be identified as a transcriptional regulator. [START_REF] Chen | Regulation of transcription by a protein methyltransferase[END_REF] CARM1 methylates a number of proteins that are involved in transcription and RNA processing, including histone H3 (H3R17 and H3R26), amplified in breast cancer 1 (AIB1), p300/CBP (cAMP-responsive element binding protein [CREB] binding protein), poly(A)-binding protein 1 (PABP1), and co-activator of 150 kDa (CA150).1 CARM1 requires its enzymatic activity for all its in vivo functions. [START_REF] Kim | Enzymatic activity is required for the in vivo functions of CARM1[END_REF] In cancer, CARM1 has been shown to regulate estrogen-stimulated MCF-7 breast cancer cell cycle progression through E2F1 upregulation. [START_REF] Frietze | CARM1 regulates estrogen-stimulated breast cancer growth through up-regulation of E2F1[END_REF] Moreover, CARM1 has been found upregulated in castration-resistant prostate cancer [START_REF] Hong | Aberrant expression of CARM1, a transcriptional coactivator of androgen receptor, in the development of prostate carcinoma and androgen-independent status[END_REF] and in grade-3 breast tumors, [START_REF] El Messaoudi | Coactivator-associated arginine methyltransferase 1 (CARM1) is a positive regulator of the Cyclin E1 gene[END_REF] and CARM1 knockdown by siRNA completely inhibited prostate cancer LNCaP cell proliferation by induction of apoptosis. [START_REF] Majumder | Involvement of arginine methyltransferase CARM1 in androgen receptor function and prostate cancer cell viability[END_REF] All these findings prompted researchers to develop molecules able to inhibit CARM1 activity, as potential anticancer agents. Some pyrazole-containing compounds (1-4) as well as the benzo [d]imidazole (5) have been reported as inhibitors of CARM1, [START_REF] Purandare | Pyrazole inhibitors of coactivator associated arginine methyltransferase 1 (CARM1)[END_REF][START_REF] Huynh | Optimization of pyrazole inhibitors of Coactivator Associated Arginine Methyltransferase 1 (CARM1)[END_REF][START_REF] Allan | N-Benzyl-1-heteroaryl-3-(trifluoromethyl)-1H-pyrazole-5-carboxamides as inhibitors of co-activator associated arginine methyltransferase 1 (CARM1)[END_REF][START_REF] Therrien | 1,2-Diamines as inhibitors of co-activator associated arginine methyltransferase 1 (CARM1)[END_REF][START_REF] Wan | Benzo[d]imidazole inhibitors of Coactivator Associated Arginine Methyltransferase 1 (CARM1)-Hit to Lead studies[END_REF] and the plantderived ellagic acid (6) [START_REF] Selvi | Identification of a novel inhibitor of coactivatorassociated arginine methyltransferase 1 (CARM1)-mediated methylation of histone H3 Arg-17[END_REF] has been recently shown to selectively block methylation at Arg 17 of histone H3 (H3R17), [START_REF] Selvi | Identification of a novel inhibitor of coactivatorassociated arginine methyltransferase 1 (CARM1)-mediated methylation of histone H3 Arg-17[END_REF] the CARM1 histone site for methylation (Chart S1 in Supporting Information).

Despite the fact that all of these compounds showed submicromolar inhibitory activity against CARM1, no inhibitor has been demonstrated to exhibit cellular effects to date.

Pursuing our searches on design, synthesis, and biological validation of small molecule modulators of epigenetic targets, [START_REF] Mai | Synthesis and biological properties of novel, uracil-containing histone deacetylase inhibitors[END_REF] in 2008 we prepared and tested some bis(3-bromo-4hydroxy-and 3,5-dibromo-4-hydroxyphenyl) compounds and their analogues against PRMT1, 18 CARM1, 18 SET7 (an histone lysine methyltransferase, HKMT), 18 p300/CBP (an HAT enzyme), [START_REF] Mai | epigenetic multiple ligands: mixed histone/protein methyltransferase, acetyltransferase, and class III deacetylase (sirtuin) inhibitors[END_REF][START_REF] Costi | Cinnamoyl compounds as simple molecules that inhibit p300 histone acetyltransferase[END_REF] SIRT1, and SIRT2. [START_REF] Mai | epigenetic multiple ligands: mixed histone/protein methyltransferase, acetyltransferase, and class III deacetylase (sirtuin) inhibitors[END_REF] Depending on the extent of bromination of the molecule (presence of four bromine atoms), and on the nature of the linker connecting the two dibromophenol moieties (penta-1,4-dien-3-one, 2,6-dimethylene(hetero)cycloalkanone, 1,1-(1,3-phenylene)diprop-2-en-1-one, and hepta-1,6-diene-3,5-dione), some of such compounds behaved as epigenetic multiple ligands (epi-MLs), they being active against all the tested enzymes. [START_REF] Mai | epigenetic multiple ligands: mixed histone/protein methyltransferase, acetyltransferase, and class III deacetylase (sirtuin) inhibitors[END_REF] Differently, compounds carrying two or three bromine atoms in their structure or featuring a bis(3,5-dibromo-4-hydroxybenzamide) or bis(3,5-dibromo-4hydroxyanilide) scaffold failed to be recognized as epi-MLs, and showed some degree of selectivity against a particular epigenetic target.

Thus, with the aim to identify CARM1-selective inhibitors among them, and taking in account the fluorograph data previously reported, we determined the IC 50 values for selected bis(bromo-and dibromophenol) compounds 7a-m (see Figure S1 and Table S1 in Supporting Information) against PRMT1, CARM1, and the HKMT SET7.

Among the tested compounds, 7b showed high potency and selectivity in inhibiting PRMT1, whereas 7c,d,g,h,l,m preferably inhibited CARM1, 7g being the most potent (IC 50 = 7.1 μM). With the exception of 7a,b, all the tested compounds displayed very low (if any) inhibition against SET7.

Accordingly, we chose 7g as our lead compound for selective CARM1 inhibition, and prepared some 3,5-bis(3-bromo-4-hydroxybenzylidene)-1-benzylpiperidin-4-one analogues 8a-l by insertion of a chlorine atom or a methyl or methoxy group at the ortho, meta, or para position of the N1-benzyl moiety, or by replacing such benzyl group with a 2-phenylethyl, 3-phenylpropyl, or 2-oxo-2-phenylethyl moiety at N1 (Figure 1). These new compounds were tested as CARM1-selective inhibitors, and two of them together with 7g were investigated in more detail in vitro and in vivo. Experimental procedures for compounds 9 and 10, and chemical and physical data (Tables S2-S4) for compounds 8-10 are reported as Supporting Information.

Results and Discussion

The new compounds were tested by fluorograph at 50 and 20 μM against CARM1 using PABP1 as a substrate, [START_REF] Lee | PABP1 identified as an arginine methyltransferase substrate using highdensity protein arrays[END_REF] and at 50 μM against PRMT1 (substrate: the heterogeneous nuclear ribonucleoprotein NPL3) [START_REF] Mcbride | Arginine methylation of yeast mRNA-binding protein Npl3 directly affects its function, nuclear export, and intranuclear protein interactions[END_REF] and SET7 (substrate: histone H3), to assess their potency and selectivity (Figure 2). At 20 μM, the 4-methyl-and the 2-, 3-, and 4-methoxybenzyl analogues of 7g (compounds 8c and 8g-i) as well as the 3-phenylpropyl-piperidone 8k showed no effect against the PABP1 methylation, thus the methoxy-containing compounds were excluded by IC 50 calculation.

IC 50 values for compounds 7g and 8a-f,j-l were determined against CARM1 using PABP1 as a substrate, and against PRMT1 and SET7 using NPL3 and histone H3 as substrates, respectively (Table 1). The corresponding IC 50 curves are reported in Supporting Information. All the tested compounds displayed low micromolar activity against CARM1, the insertion of methyl as well as chloro substituents at the N1-benzyl moiety having only modulator effects on enzyme inhibition. The preferred position to introduce a methyl group at the benzyl portion seems to be the ortho position (compound 8a), while for chlorine insertion the benzyl meta position afforded the highest inhibitory activity (compound 8e), similar to that of the lead compound 7g.

All the tested compounds were selective towards CARM1, they showing very low (if any) activity against PRMT1 and SET7. Among them, we selected 7g, 8e, and 8l for further experiments: 7g and 8e were the most potent inhibitors of CARM1 with PABP1 as a substrate (see Table 1), while 8l was the only analogue carrying a structural diversity, the carbonyl group at the substituent at N1, that could influence someway its binding with the enzyme and its inhibitory behavior.

First, we repeated the CARM1 assay testing 7g, 8e, and 8l at 100 μM by fluorograph and using four different CARM1 substrates: PABP1, CA150, [START_REF] Cheng | The arginine methyltransferase CARM1 regulates the coupling of transcription and mRNA processing[END_REF] the spliceosome protein SmB, [START_REF] Cheng | The arginine methyltransferase CARM1 regulates the coupling of transcription and mRNA processing[END_REF] and histone H3 (Figure 3).

All the three tested compounds strongly inhibited the CARM1 activity on the various substrates; among these, CA150 was the most sensitive whereas the use of histone H3 yielded the lowest CARM1 inhibition. To check the real selectivity of 7g, 8e, and 8l against various PRMTs, we tested them at 100 μM against i) PRMT1 using NPL3 and histone H4 as a nonhistone and histone substrate, respectively, ii) PRMT3 using NPL3 and the ribosome protein rpS2 [START_REF] Swiercz | Ribosomal protein rpS2 is hypomethylated in PRMT3deficient mice[END_REF] as substrates, iii) PRMT5 and histone H4 as a substrate, iv) PRMT6 using NPL3 and histone H3 as substrates (Figure 3).

In addition, 7g, 8e, and 8l were tested at 100 μM against a panel of HKMTs, namely SET7 (substrate: H3), DOTL1 (substrate: nucleosome), Suv39H1 (substrate: H3), and G9a (substrate: H3) (Figure 4). Against PRMTs, 7g and 8e were able to inhibit to some extent PRMT3, and 8e and 8l showed high inhibition of PRMT5 at 100 μM, nevertheless in all cases the observed inhibition values were weaker than those observed with CARM1 when used at the same concentration (see Figure 3). No significant activity at 100 μM was registered for 7g, 8e, and 8l against the tested HKMTs (see Figure 4). Known CARM1 substrates such as PABP1 are hypermethylated in vivo and this methylation is very stable. To test the efficacy of potential PRMT inhibitors in cell may require days of treatment, while waiting for the methylated substrates to turn-over. Under these conditions, compounds with pleiotropic effects would be difficult to investigate in a cell-based assay. To reduce the exposure time of the compound to cells, and bypass this problem, we developed an Flag-tagged PABP1 inducible cell line obtained by engineering a tetracyclinecontrolled transrepressor protein (TetR) in human embryonic kidney HEK293 cells. [START_REF] Deuschle | Tetracycline-reversible silencing of eukaryotic promoters[END_REF] The TetR protein binds to tet operator (tetO) sequences in absence but not in the presence of tetracycline, silencing the transcriptional activities at the promoter.

We can thus easily distinguish between the endogenous PABP1 and the induced Flag-tagged PABP1 due to its slower migration by SDS-PAGE. We tested 7g, 8e, and 8l in this reporter system. Upon addition of tetracycline, Flag-PABP1 is induced in HEK293 cells in the presence of the indicated compound, and its methylation status can be detected by the use of a methyl-specific PABP1 antibody generated in our lab. [START_REF] Deuschle | Tetracycline-reversible silencing of eukaryotic promoters[END_REF] In this reporter system, only 7g was able to inhibit Flag-PABP1 methylation (Figure S4 in Supporting Information).

There is increasing evidence of the involvement of CARM1 in hormone responsive cancers such as prostate cancer. Thus, we determined the effect of 7g, 8e, and 8l on prostate-specific antigen (PSA) promoter in human prostate adenocarcinoma LNCaP cells by using PSA luciferase assay, relative to a CMV-Renilla control (Figure 5, top panel). In particular, we transfected PSA reporter into LNCaP cells, and then we treated the cells with increasing concentration of 7g, 8e, or 8l for two days.

As seen in Figure 5, a dose-dependent decrease of the reporter activity was observed with 7g and 8e up to 8-10 μM, while 8l was effective only at 30 μM. In parallel, we measured the cell viability through Cell Titer-Glo (CTG), based on quantitation of the ATP present (Figure 5, bottom panel). This was done to confirm that the observed PSA effects were the results of CARM1 inhibition, and to rule out involvement of other targets and/or cell death. 7g and 8l displayed no or little effects on cell viability, at concentrations that impacted the luciferase assay.

In conclusion, we have reported on the ability of the 1-substituted-3,5-bis(3-bromo-4hydroxybenzylidene)piperidin-4-ones 7g and 8a-l to selectively inhibit CARM1 activity. Compounds 7g, 8e, and 8l were able to inhibit CARM1-mediated methylation of different substrates (PABP1, CA150, SmB, and H3) up to single-digit micromolar level, displaying low inhibitor activity (if any) against a panel of different PRMTs or HKMTs. In human prostate cancer LNCaP cells, 7g showed a significant dose-dependent reduction of the PSA promoter activity, at concentration that did not affect cell viability.

Experimental Section Chemistry

Melting points were determined on a Buchi 530 melting point apparatus and are uncorrected. 1H NMR and 13C NMR spectra were recorded at 400 MHz on a Bruker AC 400 spectrometer; chemical shifts are reported in δ (ppm) units relative to the internal reference tetramethylsilane (Me4Si).

EIMS spectra were recorded with a Fisons Trio 1000 spectrometer; only molecular ions (M + ) and base peaks are given. All solvents were reagent grade and, when necessary, were purified and dried by standard methods. Organic solutions were dried over anhydrous sodium sulfate. Elemental analysis has been used to determine purity of the described compounds, that is >95%. Analytical results are within ± 0.40% of the theoretical values (See Table S3 in Supporting Information). All chemicals were purchased from Aldrich Chimica, Milan (Italy), or from Alfa Aesar, Milan (Italy), and were of the highest purity.

General Procedure for the Synthesis of N-substituted-3,5-bis(3-bromo-4hydroxybenzylidene)piperidin-4-ones (8a-l). Example: 3,5-Bis(3-bromo-4-(hydroxybenzylidene)-1-(3-chlorobenzyl)piperidin-4-one (8e)

A solution of 10e (0.42 mmol, 0.3 g) in methanol (5 mL) and 3 N hydrochloric acid (5 mL) was stirred at 60 °C for 3 h, then the suspension was neutralized with 1 N sodium hydrogen carbonate, the precipitated solid was filtered and washed with water (3 × 10 mL) and diethyl ether (3 × 10 mL) to give the pure compound 8e as a yellow powder. [START_REF] Bedford | Protein arginine methylation in mammals: who, what, and why[END_REF] H NMR (DMSO-d 6 , 400 MHz, δ ppm) δ 4.43-4.48 (s, 6H, PhCH 2 and CH 2 piperidone), 7.05-7.74 (m, 12H, PhCH and benzene protons), 11.13 (bs, 2H, OH) ppm; [START_REF] Allan | N-Benzyl-1-heteroaryl-3-(trifluoromethyl)-1H-pyrazole-5-carboxamides as inhibitors of co-activator associated arginine methyltransferase 1 (CARM1)[END_REF] C NMR (DMSO-d 6 , 400 MHz, δ; ppm) δ 53.4 (2C), 63.9, 113.7 (2C), 118.0 (2C), 126.0, 126.9, 127.3, 128.7 (2C), 129.6 (2C), 131.3 (2C), 132.2, 134.0, 136.9, 140.6 (2C), 145.9 (2C), 155.8 (2C), 186.0 ppm; MS (EI): m/z: 588.95 (M) + .

Experimental procedures for compounds 9 and 10, and chemical and physical data (Tables S2-S4) for compounds 8-10 are reported as Supporting Information.

Plasmids and antibodies

See Supporting Information.

In Vitro Methylation Assay and IC 50 determination

The assays have been described in detail previously. [START_REF] Mai | Synthesis and biological validation of novel synthetic histone/ protein methyltransferase inhibitors[END_REF] Briefly, all methylation reactions were performed in a final volume of 30-μL of PBS (137 mM NaCl, 2.7 mM KCl, 4.3 mM Na 2 HPO 4 , 1.4 mM KH 2 PO 4 , pH 7.4) and in the presence of S-adenosyl-l-[methyl-3 H]methionine ([ 3 H]AdoMet, 85 Ci/mmol from a 0.5 mCi/mL in dilute HCl/ethanol 9:1, pH 2.0-2.5, PerkinElmer Life Sciences). The reaction contained 0.5-1.5 μM of substrate and 0.1-0.2 μM of recombinant enzyme with 100 μM of each indicated compound for fluorograph (Figures 3 and4) or different doses of each compound for IC 50 determination (Table 1). The reaction was incubated at 30 °C for 90 min and then separated by SDS/PAGE, transferred to a PVDF membrane, sprayed with Enhance (PerkinElmer Life Sciences), and exposed to film overnight for fluorograph. After fluorograph, the same PVDF membrane stained by Ponceau S, and cut the visualized bands of substrate to count dpm by using liquid scintillation analyzer (Tri-Carb; Packard) for graphic depiction or IC 50 value determination.

Cell Lines and Cultures

See Supporting Information.

Luciferase assay

LNCaP cells were cultured in phenol-red-free RPMI1640 supplemented with 10% charcoalstripped fetal calf serum. Approximately 20 h before transfection, cells were seeded into each well of 24-well culture dishes. The cells in each well were transfected with Lipofectamine 2000 transfection reagent (Invitrogen) according to the manufacturer's protocol. For each transfection, 300 ng of PSA(ARE)-LUC and 2 ng humanized CMV-Renilla internal control were used. After 12 h of transfection, cells were treated with 20 nM DHT to induce PSA-firefly and indicated amount of compound. After 42-44 h, the cells were washed twice with PBS and harvested. 5/6 Cells were used to perform luciferase assay using the Dual Luciferase Assay System (Promega) (Figure 5 top panel), and 1/6 cells were used to determine cell viability using CellTiter-Glo luminescent reagent (Promega) according to the manufacturer's protocol (Figure 5 CARM1-selective inhibitors used in this study. Inhibitory activities of compounds 8a-l against CARM1 using PABP1 as a substrate, PRMT1 using NPL3 as a substrate, and SET7 using histone H3 as a substrate. The concentration of the compounds used in each in vitro methylation assay is shown. Inhibitory activity of 7g, 8e, and 8l against CARM1 using PABP1, CA150, SMB, and histone H3 as substrates, and against a panel of PRMTs (PRMT1, PRMT3, PRMT5, and PRMT6) using indicated histone and/or non-histone substrates. The fluorographs are shown in the left panels, and the tritium count for each band is depicted in the right panels. Inhibitory activities of 7g, 8e, and 8l against a panel of HKMTs (SET7, DOTL1, Suv39H1, and G9a) using the indicated histone and/or non-histone substrates. Effects of increasing concentrations of 7g, 8e, and 8l on PSA promoter activity by luciferase assay in LNCaP cells, relative to a CMV-Renilla control (top panel), and on cell viability based on quantitation of the ATP present, which is an indicator of metabolically active cells, and is used to determine the viability of cells in culture (bottom panel). The results are presented as mean ± SD that were calculated from triplicate luciferase assays. a Values were determined from at least two separate experiments. The reaction contained 0.1 μM of GST-CARM1 and 0.5 μM of GST-PABP1, 0.15 μM of GST-PRMT1 and 0.5 μM GST-NPL3, or 0.15 μM of GST-SET7 and 1.1 μM of histone H3 with 0.22 μM [ 3 H] AdoMet and different concentrations of each compound for IC 50 determinations with different concentrations of each compound for IC 50 determinations. The software that we used for fitting curves and determining IC 50 s is SigmPlot. The equation used for fitting is y=y 0 + a/1+(x/x 0 )b.

J Med Chem. Author manuscript; available in PMC 2012 November 02.

Chemistry 3 , 5 -

 35 Bis(3-bromo-4-(methoxymethoxy)benzylidene)piperidin-4-one 9, the key intermediate of the title compounds, was prepared by condensation of 3-bromo-4-(methoxymethoxy)benzaldehyde 18 with 4-piperidone in alkaline medium (barium hydrate). Alkylation reactions of 9, carried out at 60 °C with the opportune alkyl bromide in the presence of dry potassium carbonate in acetonitrile, furnished the N-arylalkyl-3,5-bis(3bromo-4-(methoxymethoxy) benzylidene)piperidin-4-ones 10a-l that were subjected to acidic hydrolysis in methanolic 3 N HCl at 60 °C to afford the desired bis(3-bromo-4hydroxybenzylidene) analogues 8a-l (Scheme S1 in Supporting Information).

  ABBREVIATIONSAdoHcy S-adenosyl-L-homocystein AdoMet S-adenosyl-L-methionine AIB1 amplified in breast cancer 1 CA150 co-activator of 150 kDa CREB cAMP-responsive element binding protein

Figure 1 .

 1 Figure 1.

Figure 2 .

 2 Figure 2.

Figure 3 .

 3 Figure 3.

Figure 4 .

 4 Figure 4.

Figure 5 .

 5 Figure 5.

Table 1

 1 IC 50 values of 7g and 8a-f,j-l against CARM1, PRMT1, and SET7.

	compd		IC 50 (μM)
		CARM1/PABP1 PRMT1/NPL3 SET7/H3
	7g	8.6 ± 0.8	> 667	> 667
	8a	10.3 ± 3.3	> 667	> 667
	8b	15.2 ± 0.9	> 667	> 667
	8c	11.9 ± 2.3	> 667	> 333
	8d	12.5 ± 6.1	> 667	> 667
	8e	8.1 ± 2.2	> 667	174 ± 28
	8f	12.2 ± 3.0	> 667	> 600
	8j	14.8 ± 2.5	> 667	> 667
	8k	16.0 ± 3.8	> 667	> 667
	8l	14.4 ± 1.7	> 667	149 ± 26

J Med Chem. Author manuscript; available in PMC 2012 November 02. $watermark-text $watermark-text $watermark-text

J Med Chem. Author manuscript; available in PMC 2012 November 02.

Acknowledgments

This work was partially supported by grants from Fondazione Roma (AM), COST Action TD09/05 Epigenetics (AM), and by an institutional NIEHS center grant ES007784 and ES015188 (MTB).

Supplementary Material

Refer to Web version on PubMed Central for supplementary material.


Aurélie Beynier

Abdel-Illah Mouaddib

Decentralized Decision Making Process for Document Server Networks

A peer-to-peer server network system consists of a large number of autonomous servers logically connected in a peer-to-peer way where each server maintains a collection of documents. When a query of storing new documents is received by the system, a distributed search process determines the most relevant servers and redirects the documents to them for processing (compressing and storing at the right document base).

In this paper, we model this distributed search process as a distributed sequential decision making problem using a set of interactive Markov Decision Processes (MDP), a specific stochastic game approach, which represent each server's decision making problem. The relevance of a server to a document is regarded as a reward considering the capacity of the storage and the goodness score of a server. We show that using a central MDP to derive an optimal policy of how to distribute documents among servers leads to high complexity and is inappropriate to the distributed nature of the application. We present then interactive MDPs approach transforming this problem into a decentralized decision making process.

I. INTRODUCTION

The Web is real-world application of distributed environment where various entities are massively distributed and loosely connected. Documents become the basic element to manage in large document bases. Consequently their storage and search problems should be carefully considered to maintain a performant system. To alleviate these problems, distributed decision making systems formalize the problem of storing the huge web documents to a large number of autonomous local decision makers (servers) where each of the decision makers (servers) decides about how to store and how to retrieve documents [START_REF] Shen | An mdp-based peer-to-peer search server network[END_REF], [START_REF] Yuwono | Server ranking for distributed text retrieval systems on the internet[END_REF].

In this paper, we adopt a peer-to-peer distributed document storage and search server network, in which each server is autonomous and connected to only a small number of peer servers. The resource consumption for each document storage is probabilistic because of the uncertain output of the compression techniques. However, we assume that each storage technique has a probability distribution on resource consumptions (memory). To optimally store a set of documents, the system should decide how to distribute the documents among the servers. We present, first, how this decision making process can be formalized as an MDP from which a global optimal policy of servers can be derived. It, also, will be shown that using a centralized MDP is computationally high and unsuitable because of the distributed nature of the problem. Consequently, we present another approach based on a specific distributed decision making technique.

The distribution of the document decision making problem is, then, formalized as a decentralized decision making process, a specific stochastic game approach, among servers where each server should make a decision on which documents to store considering its capacity, the relevance of the documents and the ability of the other servers to store it. In order to model this problem, we consider a set of interactive MDPs for a cooperative game theoretic model where each server uses an MDP to compute its local policy which selects documents to store considering the abilities of the other agents.

The system we consider is a set of servers S = {a 1 , . . . a n } and queries q of searching and storing a set of documents {d 1 , . . . d T }. The problem is then how to optimally distribute documents d t among servers a j such that the resource consumption of the system is minimized and the relevance of documents to server is maximized.

To deal with this problem we transform it into a problem of task allocation in a multi-agent system [START_REF] Rothkopf | Computationally manageable combinatorial auctions[END_REF], [START_REF] Sandholm | Emediator: A next generation electronic commerce server[END_REF], [START_REF] Hanna | Task selection problem under uncertainty as decision-making[END_REF], [START_REF] Abdallah | Modeling task allocation using decision theoretic model[END_REF], a specific problem of distributed network resource allocation, where the set of servers becomes a set of agents (or players) A and the set of documents to store become a set of tasks T.

II. FRAMEWORK FOR STORAGE TASKS

The problem of distributing documents to store among servers can be formalized as a decentralized cooperative system which consists of a set of agents A = {a 1 , a 2 , • • • , a n } that have to allocate and to execute a set of tasks

T = {d 1 , d 2 , • • • , d T }.
Each agent a k has a bounded quantity of resources R a k that he uses to execute his allocated tasks. Tasks will be allocated and executed in a commonly known order: without loss of generality, we assume that this ordering is d 1 , d

= l, D k ∩ D l = ∅ and ∪ a k ∈A D k ⊆ T. Definition 1: Each agent a k has a reward function G a k : T → R + such as, G a k (d t ) represents what the system gains if a k stores the document d t .
This function in the context of Peer-to-Peer document storage server architecture considers many criteria such as the memory capacity, the relevance, security and other criteria. In our context, we limit ourselves to memory capacity and the relevance of the server. To do that, we consider the goodness score of a server [START_REF] Baeza-Yates | Modern Information Retrieval[END_REF], [START_REF] Yuwono | Server ranking for distributed text retrieval systems on the internet[END_REF] which assesses the relevance to store a document in comparison with the ones already stored. If the terms of the document to store appear frequently in these documents, then we believe the server is more relevant to the document. In order to avoid considering all the terms of a document, we use a relative term frequency according to the greatest frequency in the document. We call this frequency the local index LI(x, d t ) of a term x in a document d t . More formally:

LI(x, d t ) = |tf (x, d t ) -max y∈dt tf (y, d t )| max y∈dt tf (y, d t )
where tf (x, d t ) is the term frequency of term x in document d t . This index allows us to consider only terms with LI greater than a threshold th (50% for example).

The relevance of a server to a term t is computed as a sum of LI values of all the documents d t in the server a:

Relevance(t, a) = dt∈a LI(t, d t )
The relevance of the server to a document d t is then:

Relevance(d t , a) = t∈dt,LI(t,dt)>th

Relevance(t, a)

This relevance is then used as the reward function gained by an agent (server) a k when it stores a document d t .

G a k (d t ) = Relevance(d t , a k ) = t∈dt,LI(t,dt)>th dt∈a k LI(t, d t )
In this paper, we will be concerned not only with the relevance but also with the utility to store a document d t in a server. This function will consider in addition to the relevance of the server, its capacity of storing new documents which is its available memory resource. The utility function W a k is, then, given by:

W a k (d t ) = G a k (d t ) + Available resource(a k )
the function Available resource gives the available resource of the server which is an observable information.

The uncertainty on task execution (resource consumption) can be expressed by the fact that an agent a k cannot exactly determine the quantity of resources which will be consumed when he executes a task d t . So, he does not know whether he could execute all tasks allocated to him or he will be obligated to ignore ones. To deal with this uncertainty, we use a discrete representation of the resource consumptions for each agent:

Definition 2: The execution of a task d t by an agent a k consumes one of p quantities of resources:

r 1 k , r 2 k , • • • , r p k .
The value of p can vary with the agent and the task.

Definition 3: Each agent a k has a probability distribution P r a k : {r j k : d t ∈ T, j = 1, ..., p} -→]0, 1], where P r a k (d t , r j k ) expresses the probability that the execution of the task d t by a k consumes the quantity of resources r j k . With a higher value of p, the agent handles the uncertainty in a better way. The different quantities r j k , k = 1, ..., n, j = 1, ..., p and the probability distribution P r a k are the knowledge of a k about his environment. The agents' aim is then to allocate tasks (respond to query) in a way which maximizes the system expected reward. We formalize this specific allocation of tasks as a MDP [START_REF] Hanna | Task selection problem under uncertainty as decision-making[END_REF] from which a task allocation policy is derived.

III. DOCUMENT STORAGE TASK AS AN MDP

The problem of distributing documents among servers is formalized as a sequential decision making process where at each step, we decide to which server the document should be allocated. We define the state of the system by the set of documents stored in each server and by the available resources (available memory of servers). In centralized systems, the controller constructs and solves one MDP in order to obtain an optimal distribution of documents among servers. In the following, we describe the MDP1 via: (1) the states, (2) the actions, (3) the transition model and (4) the expected reward.

A. State Representation

A state represents a situation of the task allocation and of the anticipation of resource consumption for all agents. We denote by S t = ((D 1 t , R 1 t ), ..., (D n t , R n t )) the state of the system at step t, where: D k t is the set of tasks allocated to a k up until step t (set of stored documents at a server). The sets D k t satisfy the conditions:

n ∪ k=1 D k t ⊆ {d 1 , ..., d t } and ∀k = l then D k t ∩ D l t = ∅ A task d j≤i belongs to D k
t if there were enough resources to execute it when the decision to allocate it to a k has been made (see next section). The construction of the MDP starts from the initial state: S 0 = ((∅, R 1 0 ), ..., (∅, R n 0 )). The system arrives to a terminal state S T when a decision has been made for all tasks or when agents have no more resources to store new documents. More formally,

S T = ((D 1 T , R 1 T ), ..., (D n T , R n T )) where n ∪ k=1 D k T ⊆ T and R k T ≥ 0, k = 1, • • • , n.

B. Actions and Transition Model

An action E(S t-1 , d t , a k ) consists in allocating, from S t-1 , a task d t to an agent a k (allocating a document to a server) and in anticipating the amount of resources which will be consumed when executing this task. For simplicity, let this action be

E(d t , a k ). When it is applied to S t-1 = ((D 1 t-1 , R 1 t-1 ), ..., (D n t-1 , R n t-1
)), the system moves to one of the p following states:

S j t = ((D 1 t , R 1 t ), ..., (D k t , R k t = R k t-1 -r j k ), ..., (D n t , R n t )) where j = 1, • • • , p, D l t = D l t-1 and R l t = R l t-1 , ∀l = k. • If r j k ≤ R k t-1 then D k t = D k t-1 ∪ {d t } and R k t = R k t-1 -r j k (agent a k has enough resources to store the document). • Otherwise (r j k > R k t-1 ), D k t = D k t-1 and R k t = 0 (agent a k lacks resources).
In fact, there are, at most, p possible states because the execution of d t can consume one of p different quantities of resources (Def. 2). When r j k > R k t-1 , the agent will consume all his resources (R k t-1 ) without achieving d t . So, this task does not belong to D k t . However, arriving to state S j t , where

r j k ≤ R k t-1
, means that a k has executed d t consuming the quantity r j k of resources and the system will gain W a k (d t ). The transition probability is expressed by the distribution P r a k because a k reaches state S j t if the execution of d t consumes the quantity r j k . In the other words, the probability to reach a state S j t is P r a k (d t , r j k ).

C. Expected Reward and Global Optimal Policy

The decision to apply an action depends on the reward that the system expects to obtain by applying this action. An allocation policy π is a mapping from states into actions: for each state S t-1 , π(S t-1 ) = E(d t , a k ) means that being in state S t-1 the action E(d t , a k ) will be applied.

We denote by V (E(d t , a k )) the expected reward associated to the action E(d t , a k ). Being in a state S t-1 , t = 1, • • • , T , the expected reward of a policy π(S t-1 ) = E(d t , a k ) is the expected reward obtained by executing π: V (E(d t , a k )). We define the expected reward EV [S t-1 ] associated with the state S t-1 as an immediate gain α t-1 that the system obtains being in this state (

α t-1 = W a k (d t-1 ), if d t-1 ∈ D k
t-1 , 0 otherwise), accumulated by the expected reward of the policy to come (reward-to-go). An optimal policy π * (S t-1 ) is the one which maximizes the expected reward at each state. This policy is obtained by solving Bellman's equation adapted to our former using value iteration [START_REF] Puterman | Markov Decision Processes[END_REF]:

• for each nonterminal state S t-1 : EV [S t-1 ] = α t-1 + max a k ∈A {V (E(d t , a k ))} (1)
V

(E(d t , a k )) = p j=1 P r a k (d t , r k t ) • EV [S j t ] (2)
• for every terminal state S T :

EV [S T ] = α T (3)
Since the obtained MDP is a finite horizon with no loops2 , the policy guaranteed by Equation ( 1) is then optimal [START_REF] Puterman | Markov Decision Processes[END_REF]. Formally, π * (S t-1 ) = arg(max

a k ∈A {V (E(d t , a k ))}) (4)
The operator arg returns the action whose expected reward is maximal.

An optimal task allocation P * can be obtained from a terminal state such as: S T = ((D 1 T , R 1 T ), ..., (D n T , R n T )) reached by applying the optimal policy at each state starting up from S 0 , thus:

P * = {D 1 T , D 2 T , • • • , D n T }.
Due to the MDP state space size, allocating tasks can only be performed when the number of tasks and agents is "small" (see Section VI). Since a state S t-1 = ((D 1 t-1 , R 1 t-1 ), ..., (D n t-1 , R n t-1 )), in the global MDP, globalizes information (allocated tasks and available resources task) of all agents. Using such a representation while dealing with large sets of servers and documents is not realistic since it quickly limits the size of the problems that can be considered. Furthermore, because of the distributed nature of the problem (peer-to-peer server network), the distribution of the decision making process is more suitable than a centralized process. Indeed, the number of states is exponential in the number of tasks and agents. Besides, the agent who solves the MDP should have precise knowledge about the uncertain behavior of the others. The high complexity of the problem and the distributed nature of the application thus reduce the applicative range of a global MDP for real-life peer-to-peer server architecture. Consequently, we propose to consider a system, more suitable, where each server is an autonomous agent able to make a decision locally about storing or not the document through a local MDP.

In the following, we introduce a new method allowing agents to act in a decentralized way to obtain an optimal task allocation similar to the one obtained by the MDP. We re-formalize the problem of task allocation as decentralized MDPs (decentralized Local-MDP) where each agent solves his local MDP which requires less computation than the global MDP.

IV. DOCUMENT STORAGE TASK AS LOCAL-MDPS

A Local-MDP consists of a set S of states, a set of actions ĀC and a transition model. In a k 's Local-MDP, each state is associated with a reward which represents what the system gains when a k is in this state. We associate each action with an expected reward which represents what the system expects to gain if a k applies this action. In the following, we describe the MDP locally developed by an agent a k .

A. State Representation

A state of S represents a local situation of task allocation and of the anticipation of resource consumption for the agent a k . We denote by: St = ( Dk t , Rk t ) the state of the agent a k at step t, where Dk t is the set of tasks allocated to a k up until step t, and Rk t is a k 's available resources. The development of the Local-MDP starts from the initial state: S0 = (∅, R k 0 ). The agent arrives to a terminal state ST when all tasks have been allocated or when he has no more sufficient resources to execute other tasks. Note that a k 's Local-MDPs state space does not contain any information about the states of the other agents. This loss of information will be covered (see next Section) using a specific action e ∅ which leads to some interactions between agents.

B. Actions and Transition Model

The action e( St-1 , d t , a k ) consists in the allocation of a task d t to a k and in the anticipation of the resource quantity which will be consumed when executing d t . The action e ∅ ( St-

C. Distributed expected reward and joint optimal policy

We define the reward EV [ St-1 ] and the expected rewards associated to each action by a similar way as in the global MDP. We denote by V (e(d t , a k )) and V (e ∅ (d t , a k )) the expected rewards associated to the action e(d t , a k ) and to the action e ∅ (d t , a k ), respectively at a state St-1 . Being in a state St-1 , a policy to follow is an action e(d t , a k ) or e ∅ (d t , a k ) to apply. The expected reward of a policy is the expected reward of the corresponding action. The reward EV [ St-1 ] is defined as an immediate gain ᾱt-1 that the system obtains if a k is in the state St-1 accumulated by the expected reward of the used policy. The immediate gain ᾱt-1 to be in state St-1 is defined as follows:

ᾱt-1 = W a k (d t-1 ) if d t-1 ∈ Dk t-1
, 0 otherwise. In the following, we formulate the reward EV [ St-1 ] and the expected rewards V (e(d t , a k )) and V (e ∅ (d t , a k )) using Bellman's equations and value iteration:

• for each nonterminal state St-1 :

EV [ St-1 ] = ᾱt-1 + max{V (e(d t , a k )), V (e ∅ (d t a k ))} (5)
V (e(d t , a k )) =

p j=1 P r a k (d t , r j k ) • EV [ St ] (6)
V (e ∅ (d t , a k )) = max

a l =a k {V (e(d t , a l ))} (7)
• for every terminal state ST = (D k T , R k T ):

EV [ ST ] = ᾱT (8)
Agent a k uses Equation ( 5) to determine the reward of state St-1 . For that, he locally calculates the immediate gain ᾱt-1 using the definition described above, and the expected value V (e(d t , a k )) given in Equation ( 6). However, expected value V (e ∅ (d t , a k )) given in Equation ( 7) is calculated by receiving V (e(d t , a l )), a l = a k from the other agents via communication. On the other side, each agent a l ∈ A calculates V (e(d t , a l )) from his Local-MDP and sends it to the others. Since the obtained Local-MDP is a finite horizon with no loops, the policy guaranteed by Equation ( 5) is then locally optimal and it leads to a global optimality as it will be shown in Section V. Formally,

π * ( St-1 ) = arg(max{V (e(d t , a k )), V (e ∅ (d t , a k ))}) (9)

D. Coordination of distributed value calculation

As we have shown above, Equation ( 7) generates communication between agents. Indeed, an agent a l calculates V (e(d t , a l )) using Equation ( 6) and then sends it to the other agents. In order to obtain an optimal policy π * ( St-1 ), agents exchange the values V (e(d t , a l )), a l ∈ A that needs an exchange of the values V (e(d t+1 , a l )), a l ∈ A, and so on. At the end, agent a k can directly calculate V (e(d T , a k )) using Equation [START_REF] Yuwono | Server ranking for distributed text retrieval systems on the internet[END_REF]. After exchanging these values, the reward EV [ ST -1 ] can be calculated for any state ST -1 . By backward chaining, EV [ ST -1 ] allows to calculate EV [ ST -2 ], and so on up to state St-1 . Algorithm 1 describes how an agent a k derives his local policy from his local MDP.

COM(V (e(d t+1 , a k )), {V (e ∅ (d t+1 , a k )}) is a communication primitive allowing a k to communicate the value V (e(d t+1 , a k )) to agents a j =k which uses it for computing V (e ∅ (d t+1 , a j )) and to receive from them values V (e(d t+1 , a j )) allowing him to compute V (e ∅ (d t+1 , a k )).The function reachable k (t) returns all states St reached by agent a k using action e(d t , a k ) at step t. reachable k (t) also describes states reached by agent a k at step t.

The optimal policies obtained by the agents do not lead to conflicts between them (see next lemma).

Lemma 1: For any state St-1 , t = 1, ..., T , the optimal policies obtained by the agents according to Equation (9) are consistent. Formally, (eq.( 7))

∀a k ∈ A, t = 1, ..., T, π * k ( St-1 )=e(d t , a k ) =⇒ ∀a l = a k , π * l ( St-1 )=e ∅ (d t ,
=⇒ e(d t , a k ) = arg(max{V (e(d t , a k )), max

a l =k {V (e(d t , a l ))}}) =⇒ V (e(d t , a k )) > V (e(d t , a l )), ∀a l =k ∈ A
As the values V (e(d t , a h )), ∀a h ∈ A are identical in all Local-MDPs (each agent a h calculates the value of V (e(d t , a h )) and sends it to the others), from the precedent inequality we then have: ∀a l =k ∈ A, e(d t , a l ) = arg(max{V (e(d t , a l )), max

a h =l {V (e(d t , a h ))}})
because at least, we have: V (e(d t , a k )) > V (e(d t , a l )). Consequently, ∀a l =k ∈ A, e(d t , a l ) = arg(max{V (e(d t , a l )), V (e ∅ (d t , a l ))})

∀a l =k ∈ A, π * l ( St-1 ) = e ∅ (d t , a l ) ✷ Algorithm 1 Modified value iteration algorithm of M a k : Require: set of tasks (documents to store) Ensure: policy π * k of agent a k 1: for all ST ∈ reachable k (T ) do 2: EV [ ST ] = ᾱT 3: COM(EV [ ST ], {V (e ∅ ( ST -1 , a k ))}) 4: end for 5: for t = T-1 down to 1 do 6:
for all St ∈ reachable k (t) do 7:

EV [ St ] = ᾱ( St ) + max{V (e(d t+1 , a k )), V (e ∅ (d t+1 , a k ))} 8: COM(V (e(d t+1 , a k )), {V (e ∅ (d t+1 , a k )) }) 9:
end for 10: end for 11: for t = 1 to T do 12:

for all St ∈ reachable k (t) do 13: π( St ) = arg(max(ᾱ( St ) + {V (e(d t+1 , a k )), V (e ∅ (d t+1 , a k ))})) 14:
end for 15: end for 16: return π

V. AN OPTIMAL JOINT POLICY

In this section we prove that the optimality obtained in the MDP is saved in the coordinated Local-MDPs. Firstly, we show the relationship between the MDP state space and the Local-MDPs' state spaces. Secondly, we prove the equality between the optimal policy obtained by the MDP and the one obtained by the Local-MDPs.

Lemma 2: For each state S t ((D 1 t , R 1 t ), ..., (D n t , R n t )), t = 0, ..., T in the MDP, it exists a state St in the Local-MDP of each agent a k , where St = (D k t , R k t ), k = 1, ..., n. Proof For t = 0, the agents are in the initial state S 0 ((∅, R 1 0 ), ..., (∅, R n 0 )) in the MDP. The couple (∅, R k 0 ), k = 1, ..., n is the initial state S0 (∅, R k 0 ) in a k 's Local-MDP. Supposing now that the lemma is correct for a value t -1: ∀S t-1 ((D 1 t-1 , R 1 t-1 ), ..., (D n t-1 , R n t-1 )) ∈ MDP k = 1, ..., n, ∃ St-1 ∈ a k 's Local-MDP, where St-1 = (D k t-1 , R k t-1
). We show in the following that the lemma is also correct for the step t. In fact, every state S t , in the MDP, is produced by the application of an action e(d t , a k ), a k ∈ A in a step S t-1 . According to the transition model (see Section III-B), the action e(d t , a k ) has effects only on the set D k t-1 and on the resources R k t-1 . So, for a state S t = ((D 1 t , R

= ((D 1 T -1 , R 1 T -1 ), ..., (D k T -1 , R k T -1 ), ..., (D n T -1 , R n T -1
)) in the MDP, and in the state

ST -1 = (D k T -1 , R k T -1 ) in a k 's Local-MDP.
According to Equation (2), we have:

V (E(d T , a k )) = p j=1 P r a k (d T , r j k ) • EV [S j T ]
(eq.( 3)

) =⇒ V (E(d T , a k )) = p j=1 P r a k (d T , r j k ) • α j T
where α j T is the immediate gain of state S j T . Since, the agent to whom the task d T is allocated by the action E(d T , a k ) is a k , then α j T = ᾱj T , (definitions of α and ᾱ). Moreover,

V (E(d T , a k )) = p j=1 P r a k (d T , r j k ) • ᾱj T (eq.( 8
)) =⇒ V (E(d T , a k )) = p j=1 P r a k (d T , r j k ) • EV [ Sj T ]
(eq.( 6))

=⇒

V (E(d T , a k )) = V (e(d T , a k )).

Supposing now that the lemma is correct for t, formally:

∀S t ((D 1 t , R 1 t ), ..., (D k t , R k t ), ...(D n t , R n t )), V (e(d t+1 , a k )) = V (E(d t+1 , a k )), ∀a k ∈ A
where e is applied in the state St = (D k t , R k t ). We show in the following that the lemma also is correct for t -1.

For every state S t-1 ((

D 1 t-1 , R 1 t-1 ), ..., (D k t-1 , R k t-1 ) , ...(D n t-1 , R n t-1
)), we have:

V (E(d t , a k )) = p j=1 P r a k (d t , r j k ) • EV [S j t ]
(eq.( 1)

) =⇒ V (E(dt, a k )) = p P j=1 P ra k (dt, r j k )•[α j t + max a l ∈A {V (E(d t+1 , a l ))}]
As the lemma is supposed correct for t, then:

V (E(dt, a k )) = p P j=1 P ra k (dt, r j k ) • [ᾱ j t + max a l ∈A
{V (e(d t+1 , a l ))}]

V (E(d t , a k )) = V (e(d t , a k )). ✷ Lemma 4: The optimal policy obtained in the MDP is equivalent to the optimal policies obtained by the coordinated Local-MDPs, formally:

∀a k ∈ A, t = 1, ..., T, π * k ( St-1 ) = e(d t , a k ) ⇐⇒ π * (S t-1 ) = E(d t , a k )
Proof Let a k be the agent verifying that π * k ( St-1 ) = e(d t , a k ) at step t -1. According to Equation (9), we have: e(d t , a k ) = arg(max{V (e(d t , a k )), V (e ∅ (d t , a k ))}) (eq.( 7)) ⇐⇒ e(d t , a k ) = arg(max{V (e(d t , a k )), max

a l =k ∈A {V (e(d t , a l ))}}) ⇐⇒ V (e(d t , a k )) > V (e(d t , a l =k )), ∀a l ∈ A (lem.(3)) ⇐⇒ V (E(d t , a k )) > V (e(d t , a l =k )), ∀a l ∈ A ⇐⇒ E(d t , a k ) = arg(max a l ∈A
{V (e(d t , a l ))}) (eq.( 4)) ⇐⇒ π * (S t-1 ) = E(d t , a k ). ✷

VI. PERFORMANCE AND COMPLEXITY

A. Cost of Computation

We studied the computation time of a problem with different numbers of documents to store (from 10 to 160) and four servers. We recorded the computation time of cases where only one server solves the whole problem (large MDP), two servers (two interactive MDPs), three servers and 4 servers (Figure 1). We observe a gain in computation time when using many servers. The main reason is that the Local-MDP's state space size | S| is defined by: | S| = (p+1) T +1 -1 (p+1)-1 , while the MDP's state space size |S| is defined by |S| =

(n•p) T +1 -1 (n•p)-1 .

C. Reducing communication

Equation 7 uses all information sent by all the other Local-MDPs. This Equation can be solved partially in a distributed way where each Local-MDP, instead of sending all values of reachable states, sends only the max of them. Then, Equation 7 can be solved using only the maximum values coming from all Local-MDPs and then computes their maximum. By this way, we reduce the number of messages and the size of the messages which contain only the maximum value.

There is no effect on the results of Lemma 1-4. Indeed, instead of applying max operator locally on values coming from different Local-MDPs, we apply max operator only on maximum values coming from the other Local-MDPs: V (e ∅ (d t , a k )) = max j =k V max j where V max j = max a∈Aj V (e(d t , a j ) computed in a j 's Local-MDP. This processing does not modify the result of Equation 7. Consequently, results of lemma 1-4 remain valid and the cost of communication is reduced to n.(n -1).Cost(size(message)).

Other techniques based on approximations can be considered such as sending the minimal values (pessimistic approach) or any combination of max and min values but these techniques have an immediate effect on the optimality since the result of Equation 7 should change and the claims could not be valid in these situations. The evaluation of these approximations will be considered in a short time future work.

Figure 2 shows the reduction of the cost of the communication using more local computation. When servers compute the max values, they reduce the communication cost while the quality of the solution remains optimal. In the same way, the computation time is reduced. Figure 1 shows the computation time results with the reduced mechanism of communication as explained in this section. The techniques presented above reduces the number of messages but the complexity is O(n 2 ). This complexity can be reduced by using some specific architectures. A solution is that each server a i communicates only with its neighbor server a i+1 and server a n communicates with server a 1 . This architecture reduces the complexity from polynomial to linear O(n) since the number of messages is 2 • n • T . This architecture presents a drawback because servers waste time waiting for the propagation of the max value. Further work, will be concerned with a grid architecture and the effect on the performance of the approach.

VII. CONCLUSION

In this paper, we presented an approach of applying MDP techniques for the problem of document storage and search using a server network architecture. We discussed the benefit of using a decentralized decision making technique.

This work opens a new direction in using decisiontheoretic techniques to control peer-to-peer server network architectures to improve the performance, the security and the robustness of these systems. Further work will also be concerned with the evaluation of this approach to the security and the fault tolerance. We will deal with problems where some servers are unavailable or become less and less safe and documents should be moved automatically to another server.

We also plan to develop a method allowing to reduce both the number of exchanged messages and the state space size. An agent can develop his Local-MDP considering only a subset of agents (but not all agents). Another one considers a different subset of agents. In this case, the problem of the coordination of the obtained task allocations has to be treated.

Fig. 1 .

 1 Fig. 1. Comparison of computation time

"

  Message exchanged with no local computation" "Messages exchanged with more local computation"

Fig. 2 .

 2 Fig. 2. Reduction of communication cost

  [START_REF] Baeza-Yates | Modern Information Retrieval[END_REF]

  2

  1 , d t , a k ) consists in ignoring the task d t by a k . It represents, in fact, the allocation of d t to another agent a l = a k . We introduce this action in order to represent what happens in the Local-MDPs of the other agents when a k ignores the task d t . For simplicity, we denote e( St-1 , d t , a k ) and e ∅ ( St-1 , d t , a k ) by e(d t , a k ) and e ∅ (d t , a k ). Let a k 's current state be St-1 = ( Dk t-1 , Rk t-1 ), then the application of the action e(d t , a k ) drives the agent a k to one of the p following states: Sj , means that d t has been achieved and the reward W a k (d t ) is obtained. Furthermore, the transition probability of the action e(d t , a k ) is expressed by the distribution P r a k because a k reaches a state Sj ) with probability 1 (no resource consumption). This state summarizes, in a k 's Local-MDP, all states generated, in the MDP, by actions e(d t , a l =k ), a l ∈ A.

	If r j k ≤ Rk t-1 , Dk t = Dk t-1 ∪ {d t } and Rk t = Rk t-1 -r j k . • Otherwise ( r j k > Rk t-1 ), Dk t = Dk t-1 and Rk t = 0.
	When r j k > Rk t-1 , the agent will consume all his resources ( Rk

t = ( Dk t , Rk t ), where j = 1, ..., p. • t-1 ) without achieving d t . So, this task does not belong to Dk t . However, arriving to a state Sj t , where r j k ≤ Rt-1 t if the execution of d t consumes the quantity r j k . On the other hand, the application of the action e ∅ (d t , a k ) drives to state St = ( Dk t = Dk t-1 , Rk t = Rk t-1

  a l ).

	Proof Based on Equations 7 and 9. Let π * k ( St-1 ) be e(d t , a k ): π * k ( St-1 ) = e(d t , a k ) (eq.(9))

=⇒ e(d t , a k ) = arg(max{V (e(d t , a k )), V (e ∅ (d t , a k ))})

  1 t ), ..., (D n t , R n t )) produced by this action, it exists in the a k 's Local-MDP a state St = (D k t , R k t ) produced by the action e(d t , a k ) in the state St-1 = (D k t-1 , R k t-1 ), and it exists in the Local-MDPs of the other agents a l =k ∈ A a state St = (D l t , R l t ) produced by the action e ∅ (d t , a l ) in the state )) in the MDP, the expected reward associated to an action E(d t , a k ), ∀a k ∈ A is equal to the one associated to the action e(d t , a k ) applied in the state St-1 = (D k t-1 , R k t-1 ) in a k 's Local-MDP. Proof We prove this lemma by induction. For the last decision step (t = T -1), we are in a state S T -1

	St-1 = (D l t-1 , R l t-1 ). ✷
	Lemma 3: For every state S t-1 = ((D 1 t-1 , R 1 t-1 ), ..., (D k t-1 , R k t-1 ), ..., (D n t-1 , R n t-1

The MDP described here is based on the MDP presented in[START_REF] Hanna | Task selection problem under uncertainty as decision-making[END_REF].

Being in a state S t-1 , the applicable actions are E(dt, a k ), a k ∈ A and no one can drive to a state S j≤t-1 .