Othman Zennaki
email: othman.zennaki@cea.fr

Laurent Besacier
email: laurent.besacier@imag.fr

Inducing Multilingual Text Analysis Tools Using Bidirectional Recurrent Neural Networks

This work focuses on the rapid development of linguistic annotation tools for resource-poor languages. We experiment several cross-lingual annotation projection methods using Recurrent Neural Networks (RNN) models. The distinctive feature of our approach is that our multilingual word representation requires only a parallel corpus between the source and target language. More precisely, our method has the following characteristics: (a) it does not use word alignment information, (b) it does not assume any knowledge about foreign languages, which makes it applicable to a wide range of resource-poor languages, (c) it provides truly multilingual taggers. We investigate both uni-and bi-directional RNN models and propose a method to include external information (for instance low level information from POS) in the RNN to train higher level taggers (for instance, super sense taggers). We demonstrate the validity and genericity of our model by using parallel corpora (obtained by manual or automatic translation). Our experiments are conducted to induce cross-lingual POS and super sense taggers.

Introduction

In order to minimize the need for annotated resources (produced through manual annotation, or by manual check of automatic annotation), several research works were interested in building Natural Language Processing (NLP) tools based on unsupervised or semi-supervised approaches [START_REF] Collins | Unsupervised models for named entity classification[END_REF][START_REF] Klein | The unsupervised learning of natural language structure[END_REF][START_REF] Andrew | New directions in semi-supervised learning[END_REF]. For example, NLP tools based on cross-language projection of linguistic annotations achieved good performances in the early 2000s [START_REF] Veronis | Arcade ii action de recherche concertée sur l'alignement de documents et son évaluation[END_REF]. The key idea of annotation projection can be summarized as follows: through word alignment in parallel text corpora, the annotations are transferred from the source (resource-rich) language to the target (under-resourced) language, and the resulting annotations are used for supervised training in the target language. However, automatic word alignment errors [START_REF] Fraser | Measuring word alignment quality for statistical machine translation[END_REF] limit the performance of these approaches.

Our work is built upon these previous contributions and observations. We explore the possibility of using Recurrent Neural Networks (RNN) to build multilingual NLP tools for resource-poor languages analysis. The major difference with previous works is that we do not explicitly use word alignment information. Our only assumption is that parallel sentences (source-target) are available and that the source part is annotated. In other words, we try to infer annotations in the target language from sentencebased alignments only. While most NLP researches on RNN have focused on monolingual tasks 1 and sequence labeling [START_REF] Collobert | Natural language processing from scratch[END_REF][START_REF] Graves | Supervised sequence labelling[END_REF], this paper, however, considers the problem of learning multilingual NLP tools using RNN.

Contributions In this paper, we investigate the effectiveness of RNN architectures -Simple RNN (SRNN) and Bidirectional RNN (BRNN) -for multilingual sequence labeling tasks without using any word alignment information. Two NLP tasks are considered: Part Of Speech (POS) tagging and Super Sense (SST) tagging [START_REF] Ciaramita | Broad-coverage sense disambiguation and information extraction with a supersense sequence tagger[END_REF]. Our RNN architectures demonstrate very competitive results on unsupervised training for new target languages. In addition, we show that the integration of POS information in RNN models is useful to build multilingual coarse-grain semantic (Super Senses) taggers. For this, a simple and efficient way to take into account low-level linguistic information for more complex sequence labeling RNN is proposed.

Methodology For training our multilingual RNN models, we just need as input a parallel (or multiparallel) corpus between a resource-rich language and one or many under-resourced languages. Such a parallel corpus can be manually obtained (clean corpus) or automatically obtained (noisy corpus).

To show the potential of our approach, we investigate two sequence labeling tasks: cross-language POS tagging and multilingual Super Sense Tagging (SST). For the SST task, we measure the impact of the parallel corpus quality with manual or automatic translations of the SemCor [START_REF] Miller | A semantic concordance[END_REF] translated from English into Italian (manually and automatically) and French (automatically).

Outline The remainder of the paper is organized as follows. Section 2 reviews related work. Section 3 describes our cross-language annotation projection approaches based on RNN. Section 4 presents the empirical study and associated results. We finally conclude the paper in Section 5.

Related Work

Cross-lingual projection of linguistic annotations was pioneered by [START_REF] Veronis | Arcade ii action de recherche concertée sur l'alignement de documents et son évaluation[END_REF] who created new monolingual resources by transferring annotations from resource-rich languages onto resource-poor languages through the use of word alignments. The resulting (noisy) annotations are used in conjunction with robust learning algorithms to build cheap unsupervised NLP tools [START_REF] Padó | Cross-lingual annotation projection for semantic roles[END_REF]. This approach has been successfully used to transfer several linguistic annotations between languages (efficient learning of POS taggers [START_REF] Das | Unsupervised part-of-speech tagging with bilingual graph-based projections[END_REF][START_REF] Duong | Simpler unsupervised pos tagging with bilingual projections[END_REF] and accurate projection of word senses [START_REF] Bentivogli | Evaluating cross-language annotation transfer in the multisemcor corpus[END_REF]). Cross-lingual projection requires a parallel corpus and word alignment between source and target languages. Many automatic word alignment tools are available, such as GIZA++ which implements IBM models [START_REF] Franz | Improved statistical alignment models[END_REF]. However, the noisy (non perfect) outputs of these methods is a serious limitation for the annotation projection based on word alignments [START_REF] Fraser | Measuring word alignment quality for statistical machine translation[END_REF].

To deal with this limitation, recent studies based on cross-lingual representation learning methods have been proposed to avoid using such pre-processed and noisy alignments for label projection. First, these approaches learn language-independent features, across many different languages [START_REF] Durrett | Syntactic transfer using a bilingual lexicon[END_REF][START_REF] Al-Rfou | Polyglot: Distributed word representations for multilingual nlp[END_REF][START_REF] Täckström | Target language adaptation of discriminative transfer parsers[END_REF][START_REF] Luong | Bilingual word representations with monolingual quality in mind[END_REF][START_REF] Gouws | Simple task-specific bilingual word embeddings[END_REF]Gouws et al., 2015). Then, the induced representation space is used to train NLP tools by exploiting labeled data from the source language and apply them in the target language. Cross-lingual representation learning approaches have achieved good results in different NLP applications such as cross-language SST and POS tagging [START_REF] Gouws | Simple task-specific bilingual word embeddings[END_REF], cross-language named entity recognition [START_REF] Täckström | Cross-lingual word clusters for direct transfer of linguistic structure[END_REF], cross-lingual document classification and lexical translation task (Gouws et al., 2015), cross language dependency parsing [START_REF] Durrett | Syntactic transfer using a bilingual lexicon[END_REF][START_REF] Täckström | Target language adaptation of discriminative transfer parsers[END_REF] and cross-language semantic role labeling [START_REF] Titov | Crosslingual induction of semantic roles[END_REF].

Our approach described in next section, is inspired by these works since we also try to induce a common language-independent feature space (crosslingual words embeddings). Unlike [START_REF] Durrett | Syntactic transfer using a bilingual lexicon[END_REF] and [START_REF] Gouws | Simple task-specific bilingual word embeddings[END_REF], who use bilingual lexicons, and unlike [START_REF] Luong | Bilingual word representations with monolingual quality in mind[END_REF] who use word alignments between the source and target languages2 our common multilingual representation is very agnostic. We use a simple (multilingual) vector representation based on the occurrence of source and target words in a parallel corpus and we let the RNN learn the best internal representations (corresponding to the hidden layers) specific to the task (SST or POS tagging).

In this work, we learn a cross-lingual POS tagger (multilingual POS tagger if a multilingual parallel corpus is used) based on a recurrent neural network (RNN) on the source labeled text and apply it to tag target language text. We explore simple and bidirectional RNN architectures (SRNN and BRNN respectively). Starting from the intuition that low-level linguistic information is useful to learn more complex taggers, we also introduce three new RNN variants to take into account external (POS) information in multilingual SST.

Unsupervised Approach Overview

To avoid projecting label information from deterministic and error-prone word alignments, we propose to represent the word alignment information intrinsically in a recurrent neural network architecture. The idea consists in implementing a recurrent neural network as a multilingual sequence labeling tool (we investigate POS tagging and SST tagging). Before describing our cross-lingual (multilingual if a multiparallel corpus is used) neural network tagger, we present the simple cross-lingual projection method, considered as our baseline in this work.

Baseline Cross-lingual Annotation Projection

We use direct transfer as a baseline system which is similar to the method described in [START_REF] Veronis | Arcade ii action de recherche concertée sur l'alignement de documents et son évaluation[END_REF]. First we tag the source side of the parallel corpus using the available supervised tagger. Next, we align words in the parallel corpus to find out corresponding source and target words. Tags are then projected to the (resource-poor) target language. The target language tagger is trained using any machine learning approach (we use TNT tagger [START_REF] Brants | Tnt: a statistical part-of-speech tagger[END_REF] in our experiments).

Proposed Approach

We propose a method for learning multilingual sequence labeling tools based on RNN, as it can be seen in Figure 1. In our approach, a parallel or multi-parallel corpus between a resource-rich language and one or many under-resourced languages is used to extract common (multilingual) and agnostic words representations. These representations, which rely on sentence level alignment only, are used with the source side of the parallel/multi-parallel corpus to learn a neural network tagger in the source language. Since a common representation of source and target words is chosen, this neural network tagger is truly multilingual and can be also used to tag texts in target language(s).

Common Words Representation

In our agnostic representation, we associate to each word (in source and target vocabularies) a common vector representation, namely V wi , i = 1, ..., N , where N is the number of parallel sentences (bisentences in the parallel corpus). If w appears in i-th bi-sentence of the parallel corpus then V wi = 1.

The idea is that, in general, a source word and its target translation appear together in the same bisentences and their vector representations are close. We can then use the RNN tagger, initially trained on source side, to tag the target side (because of our common vector representation). This simple representation does not require multilingual word alignments and it lets the RNN learns the optimal internal representation needed for the annotation task (for instance, the hidden layers of the RNN can be considered as multi-lingual embeddings of the words).

Recurrent Neural Networks

There are two major architectures of neural networks: Feedforward [START_REF] Bengio | A neural probabilistic language model[END_REF] and Recurrent Neural Networks (RNN) [START_REF] Schmidhuber | A fixed size storage o (n3) time complexity learning algorithm for fully recurrent continually running networks[END_REF][START_REF] Mikolov | Recurrent neural network based language model[END_REF]. [START_REF] Sundermeyer | Comparison of feedforward and recurrent neural network language models[END_REF] showed that language models based on recurrent architecture achieve better performance than language models based on feedforward architecture. This is due to the fact that recurrent neural networks do not use a context of limited size. This property led us to use, in our experiments, the Elman recurrent architecture [START_REF] Jeffrey L Elman | Finding structure in time[END_REF], in which recurrent connections occur at the hidden layer level.

We consider in this work two Elman RNN architectures (see Figure 2): Simple RNN (SRNN) and Bidirectional RNN (BRNN). In addition, to be able to include low-level linguistic information in our architecture designed for more complex sequence labeling tasks, we propose three new RNN variants to take into account external (POS) information for multilingual Super Sense Tagging (SST).

A. Simple RNN

In the simple Elman RNN (SRNN), the recurrent connection is a loop at the hidden layer level. This connection allows SRNN to use at the current time step hidden layer's states of previous time steps. In other words, the hidden layer of SRNN represents all previous history and not just n -1 previous inputs, thus the model can theoretically represent long context.

The architecture of the SRNN considered in this work is shown in Figure 2. In this architecture, we have 4 layers: input layer, forward (also called recurrent or context layer), compression hidden layer and output layer. All neurons of the input layer are connected to every neuron of forward layer by weight matrix I F and R F , the weight matrix H F connects all neurons of the forward layer to every neuron of compression layer and all neurons of the compression layer are connected to every neuron of output layer by weight matrix O.

The input layer consists of a vector w(t) that represents the current word w t in our common words representation (all input neurons corresponding to current word w t are set to 0 except those that correspond to bi-sentences containing w t , which are set to 1), and of vector f (t -1) that represents output values in the forward layer from the previous time step. We name f (t) and c(t) the current time step hidden layers (our preliminary experiments have shown better performance using these two hidden layers instead of one hidden layer), with variable sizes (usually 80-1024 neurons) and sigmoid activation function. These hidden layers represent our common language-independent feature space and inherently capture word alignment information. The output layer y(t), given the input w(t) and f (t -1) is computed with the following steps :

f (t) = Σ(w(t).I F (t) + f (t -1).R F (t)) (1) c(t) = Σ(f (t).H F (t)) (2) y(t) = Γ(c(t).O(t)) (3)
Σ and Γ are the sigmoid and the softmax functions, respectively. The softmax activation function is used to normalize the values of output neurons to sum up to 1. After the network is trained, the output y(t) is a vector representing a probability distribution over the set of tags. The current word w t (in input) is tagged with the most probable output tag. For many sequence labeling tasks, it is beneficial to have access to future in addition to the past context. So, it can be argued that our SRNN is not optimal for sequence labeling, since the network ignores future context and tries to optimize the output prediction given the previous context only. This SRNN is thus penalized compared with our baseline projection based on TNT [START_REF] Brants | Tnt: a statistical part-of-speech tagger[END_REF] which considers both left and right contexts. To overcome the limitations of SRNN, a simple extension of the SRNN architecture -namely Bidirectional recurrent neural network (BRNN) [START_REF] Schuster | Bidirectional recurrent neural networks[END_REF] -is used to ensure that context at previous and future time steps will be considered.

B. Bidirectional RNN

An unfolded BRNN architecture is given in Figure 2. The basic idea of BRNN is to present each training sequence forwards and backwards to two separate recurrent hidden layers (forward and backward hidden layers) and then somehow merge the results. This structure provides the compression and the output layers with complete past and future context for every point in the input sequence. Note that without the backward layer, this structure simplifies to a SRNN.

C. RNN Variants

As mentioned in the introduction, we propose three new RNN variants to take into account low level (POS) information in a higher level (SST) annotation task. The question addressed here is: at which layer of the RNN this low level information should be included to improve SST performance? As specified in Figure 3, the POS information can be introduced either at input layer or at forward layer (forward and backward layers for BRNN) or at compression layer. In all these RNN variants, the POS of the current word is also represented with a vector (P OS(t)). Its dimension corresponds to the number of POS tags in the tagset (universal tagset of [START_REF] Petrov | A universal part-of-speech tagset[END_REF] is used). We propose one hot vector representation where only one value is set to 1 and corresponds to the index of current tag (all other values are 0).

Network Training

The first step in our approach is to train the neural network, given a parallel corpus (training corpus), and a validation corpus (different from train data) in the source language. In typical applications, the source language is a resource-rich language (which already has an efficient tagger or manually tagged resources). Our RNN models are trained by stochastic gradient descent using usual back-propagation and back-propagation through time algorithms [START_REF] David E Rumelhart | Learning internal representations by error propagation[END_REF]. We learn our RNN models with an iterative process on the tagged source side of the parallel corpus. After each epoch (iteration) in training, validation data is used to compute per-token accuracy of the model. After that, if the per-token accuracy increases, training continues in the new epoch. Otherwise, the learning rate is halved at the start of the new epoch. Eventually, if the per-token accuracy does not increase anymore, training is stopped to prevent over-fitting. Generally, convergence takes 5-10 epochs, starting with a learning rate α = 0.1.

The second step consists in using the trained model as a target language tagger (using our common vector representation). It is important to note that if we train on a multilingual parallel corpus with N languages (N > 2), the same trained model will be able to tag all the N languages. Hence, our approach assumes that the word order in both source and target languages are similar. In some languages such as English and French, word order for contexts containing nouns could be reversed most of the time. For example, the European Commission would be translated into la Commission européenne. In order to deal with the word order constraints, we also combine the RNN model with the cross-lingual projection model in our experiments.

Dealing with out-of-vocabulary words

For the words absent from in the initial parallel corpus, their vector representation is a vector of zero values. Consequently, during testing, the RNN model will use only the context information to tag the OOV words found in the test corpus. To deal with these types of OOV words 3 , we use the CBOW model of [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] to replace each OOV word by its closest known word in the current OOV word context. Once the closest word is found, its common vector representation is used (instead of the vector of zero values) at the input of the RNN.

Combining Simple Cross-lingual Projection and RNN Models

Since the simple cross-lingual projection model M1 and RNN model M2 use different strategies for tagging (TNT is based on Markov models while RNN is a neural network), we assume that these two models can be complementary. To keep the benefits of each approach, we explore how to combine them with linear interpolation. Formally, the probability to tag a given word w is computed as

PM12(t|w) = (µPM1(t|w, CM1) + (1 -µ)PM2(t|w, CM2)) (4)
where, C M 1 and C M 2 are the context of w considered by M1 and M2 respectively. The relative importance of each model is adjusted through the interpolation parameter µ. The word w is tagged with the most probable tag, using the function f described as

f (w) = arg max t (P M 12 (t|w)) (5)

Experiments

Our models are evaluated on two labeling tasks: Cross-language Part-Of-speech (POS) tagging and Multilingual Super Sense Tagging (SST).

Multilingual POS Tagging

We applied our method to build RNN POS taggers for four target languages -French, German, Greek and Spanish -with English as the source language.

In order to determine the effectiveness of our common words representation described in section 3.2.1, we also investigated the use of state-of-the-art bilingual word embeddings (using MultiVec Toolkit [START_REF] Bérard | Multivec: a multilingual and multilevel representation learning toolkit for nlp[END_REF]) as input to our RNN.

Dataset

For French as a target language, we used a training set of 10, 000 parallel sentences, a validation set of 1000 English sentences, and a test set of 1000 French sentences, all extracted from the ARCADE II English-French corpus [START_REF] Veronis | Arcade ii action de recherche concertée sur l'alignement de documents et son évaluation[END_REF]. The test set is tagged with the French TreeTagger [START_REF] Schmid | Treetagger| a language independent part-of-speech tagger[END_REF] and then manually checked.

For German, Greek and Spanish as a target language, we used training and validation data extracted from the Europarl corpus [START_REF] Koehn | Europarl: A parallel corpus for statistical machine translation[END_REF] which are a subset of the training data used in [START_REF] Das | Unsupervised part-of-speech tagging with bilingual graph-based projections[END_REF][START_REF] Duong | Simpler unsupervised pos tagging with bilingual projections[END_REF]. This choice allows us to compare our results with those of [START_REF] Das | Unsupervised part-of-speech tagging with bilingual graph-based projections[END_REF][START_REF] Duong | Simpler unsupervised pos tagging with bilingual projections[END_REF][START_REF] Gouws | Simple task-specific bilingual word embeddings[END_REF]. The train data set contains 65, 000 bi-sentences ; a validation set of 10, 000 bi-sentences is also available. For testing, we use the same test corpora as [START_REF] Das | Unsupervised part-of-speech tagging with bilingual graph-based projections[END_REF][START_REF] Duong | Simpler unsupervised pos tagging with bilingual projections[END_REF][START_REF] Gouws | Simple task-specific bilingual word embeddings[END_REF] and [START_REF] Gouws | Simple task-specific bilingual word embeddings[END_REF].

tasks on dependency parsing [START_REF] Buchholz | Conll-x shared task on multilingual dependency parsing[END_REF]). The evaluation metric (per-token accuracy) and the [START_REF] Petrov | A universal part-of-speech tagset[END_REF] universal tagset are used for evaluation.

For training, the English (source) sides of the training corpora (ARCADE II and Europarl) and of the validation corpora are tagged with the English TreeTagger toolkit. Using the matching provided by [START_REF] Petrov | A universal part-of-speech tagset[END_REF], we map the TreeTagger and the CoNLL tagsets to the common Universal Tagset.

In order to build our baseline unsupervised tagger (based on a Simple Cross-lingual Projection -see section 3.1), we also tag the target side of the training corpus, with tags projected from English side through word-alignments established by GIZA++. After tags projection, a target language POS tagger based on TNT approach [START_REF] Brants | Tnt: a statistical part-of-speech tagger[END_REF] is trained.

The combined model is built for each considered language using cross-validation on the test corpus. First, the test corpus is split into 2 equal parts and on each part, we estimate the interpolation parameter µ (Equation 4) which maximizes the per-token accuracy score. Then each part of test corpus is tagged using the combined model tuned on the other part, and vice versa (standard cross-validation procedure).

We trained MultiVec bilingual word embeddings on the parallel Europarl corpus between English and each of the target languages considered.

Results and discussion

Table 1 reports the results obtained for the unsupervised POS tagging. We note that the POS tagger based on bidirectional RNN (BRNN) has better performance than simple RNN (SRNN), which means that both past and future contexts help select the correct tag. Table 1 also shows the performance before and after performing our procedure for handling OOVs in BRNNs. It is shown that after replacing OOVs by the closest words using CBOW, the tagging accuracy significantly increases.

As shown in the same table, our RNN models accuracy is close to that of the simple projection tagger. It achieves comparable results to [START_REF] Das | Unsupervised part-of-speech tagging with bilingual graph-based projections[END_REF], [START_REF] Duong | Simpler unsupervised pos tagging with bilingual projections[END_REF] (who used the full Europarl corpus while we use only a 65, 000 subset of it) and to [START_REF] Gouws | Simple task-specific bilingual word embeddings[END_REF] (who used extra resources such as Wiktionary and Wikipedia). Interestingly, RNN models learned using our common words representation (section 3.2.1) seem to perform significantly better than RNN models using MultiVec bilingual word embeddings.

It is also important to note that only one single SRNN and BRNN tagger applies to German, Greek and Spanish; so this is a truly multilingual POS tagger! Finally, as for several other NLP tasks such as language modelling or machine translation (where standard and NN-based models are generally combined in order to obtain optimal results), the combination of standard and RNN-based approaches (Projec-tion+_) seems necessary to further optimize POS tagging accuracies.

Multilingual SST

In order to measure the impact of the parallel corpus quality on our method, we also learn our SST models using the multilingual parallel corpus MultiSemCor (MSC) which is the result of manual or automatic translation of SemCor from English into Italian and French.

Dataset

SemCor The SemCor [START_REF] Miller | A semantic concordance[END_REF] is a subset of the Brown Corpus [START_REF] Kucera | A standard corpus of present-day edited american english, for use with digital computers[END_REF] labeled with the WordNet [START_REF] Fellbaum | WordNet[END_REF] senses. MultiSemCor The English-Italian MultiSemcor (MSC-IT-1) corpus is a manual translation of the English SemCor to Italian [START_REF] Bentivogli | Evaluating cross-language annotation transfer in the multisemcor corpus[END_REF]. As we already mentioned, we are also interested in measuring the impact of the parallel corpus quality on our method. For this we use two translation systems: (a) Google Translate to translate the English SemCor to Italian (MSC-IT-2) and French (MSC-FR-2). (b) LIG machine translation system [START_REF] Besacier | The lig english to french machine translation system for iwslt 2012[END_REF] to translate the English SemCor to French (MSC-FR-1). Training corpus The SemCor was labeled with the WordNet synsets. However, because we train models for SST, we convert SemCor synsets annotations to super senses. We learn our models using the four different versions of MSC (MSC-IT-1,2 -MSC-FR-1,2), with modified Semcor on source side. Test Corpus To evaluate our models, we used the SemEval 2013 Task 12 (Multilingual Word Sense Disambiguation) [START_REF] Navigli | Semeval-2013 : Multilingual word sense disambiguation[END_REF] test corpora, which are available in 5 languages (English, French, German, Spanish and Italian) and labeled with BabelNet [START_REF] Navigli | Babelnet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network[END_REF] senses. We map BabelNet senses to WordNet synsets, then WordNet synsets are mapped to super senses.

SST Systems Evaluated

The goals of our SST experiments are twofold: first, to investigate the effectiveness of using POS information to build multilingual super sense tagger, secondly to measure the impact of the parallel corpus quality (manual or automatic translation) on our RNN models (SRNN, BRNN and our proposed variants). To summarize, we build four super sense taggers based on baseline cross-lingual projection (see section 3.1) using four versions of MultiSemcor (MSC-IT-1, MSC-IT-2, MSC-FR-1, MSC-FR-2) described above. Then we use the same four versions to train our multilingual SST models based on SRNN and BRNN. For learning our multilingual SST models based on RNN variants proposed in part (C) of section 3.2.2, we also tag SemCor using TreeTagger (POS tagger proposed by [START_REF] Schmid | Treetagger| a language independent part-of-speech tagger[END_REF]).

Results and discussion

Our models are evaluated on SemEval 2013 Task 12 test corpora. Results are directly comparable with those of systems which participated to this evaluation campaign. We report two SemEval 2013 (unsupervised) system results for comparison:

• MFS Semeval 2013 : The most frequent sense is the baseline provided by SemEval 2013 for Task 12, this system is a strong baseline, which is obtained by using an external resource (the WordNet most frequent sense).

• GETALP : a fully unsupervised WSD system proposed by [START_REF] Schwab | Ant colony algorithm for the unsupervised word sense disambiguation of texts: Comparison and evaluation[END_REF] based on Ant-Colony algorithm.

The DAEBAK! [START_REF] Navigli | An experimental study of graph connectivity for unsupervised word sense disambiguation[END_REF] and the UMCC-DLSI systems [START_REF] Yoan | Enriching the integration of semantic resources based on wordnet[END_REF] have also participated to SemEval 2013 Task 12. However, they use a supervised approach6 .

Table 2 shows the results obtained by our RNN models and by two SemEval 2013 WSD systems. SRNN-POS-X and BRNN-POS-X refer to our RNN variants: In means input layer, H1 means first hidden layer and H2 means second hidden layer. We achieve the best performance on Italian using MSC-IT-1 clean corpus while noisy training corpus degrades SST performance. The best results are obtained with combination of simple projection and RNN which confirms (as for POS tagging) that both approaches are complementary. We also observe that the RNN approach seems more robust than simple projection on noisy corpora. This is probably due to the fact that no word alignments are required in our cross language RNN. Finally, BRNN-POS-H2-OOV achieves the best performance, which shows that the integration of POS information in RNN models and dealing with OOV words are useful to build efficient multilingual super senses taggers. Finally, it is worth mentioning that integrating low level (POS) information lately (last hidden layer) seems to be the best option in our case.

Conclusion

In this paper, we have presented an approach based on recurrent neural networks (RNN) to induce multilingual text analysis tools. We have studied Simple and Bidirectional RNN architectures on multilingual POS and SST tagging. We have also proposed new RNN variants in order to take into account low level (POS) information in a super sense tagging task. Our approach has the following advantages: (a) it uses a language-independent word representation (based only on word co-occurrences in a parallel corpus), (b) it provides truly multilingual taggers (1 tagger for N languages) (c) it can be easily adapted to a new target language (when a small amount of supervised data is available, a previous study (Zennaki et al., 2015a;Zennaki et al., 2015b) has shown the effectiveness of our method in a weakly supervised context).

Short term perspectives are to apply multi-task learning to build systems that simultaneously perform syntactic and semantic analysis. Adding out-of-language data to improve our RNN taggers is also possible (and interesting to experiment) with our common (multilingual) vector representation.

Figure 1 :

 1 Figure 1: Overview of the proposed model architecture for inducing multilingual RNN taggers.

Figure 2 :

 2 Figure 2: High level schema of RNN used in our work.

Figure 3 :

 3 Figure 3: SRNN variants with POS information at three levels: (a) input layer, (b) forward layer, (c) compression layer.

Table 1 :

 1 (bi-sentences from CoNLL shared Token-level POS tagging accuracy for Simple Projection, SRNN using MultiVec bilingual word embeddings as input, RNN 5 , Projection+RNN and methods of[START_REF] Das | Unsupervised part-of-speech tagging with bilingual graph-based projections[END_REF],[START_REF] Duong | Simpler unsupervised pos tagging with bilingual projections[END_REF]

	Lang.	French		German	Greek		Spanish
	Model	All words OOV	All words OOV All words OOV All words OOV
	Simple Projection	80.3	77.1	78.9	73.0	77.5	72.8	80.0	79.7
	SRNN MultiVec	75.0	65.4	70.3	68.8	71.1	65.4	73.4	62.4
	SRNN	78.5	70.0	76.1	76.4	75.7	70.7	78.8	72.6
	BRNN	80.6	70.9	77.5	76.6	77.2	71.0	80.5	73.1
	BRNN -OOV	81.4	77.8	77.6	77.8	77.9	75.3	80.6	74.7
	Projection + SRNN	84.5	78.8	81.5	77.0	78.3	74.6	83.6	81.2
	Projection + BRNN	85.2	79.0	81.9	77.1	79.2	75.0	84.4	81.7
	Projection + BRNN -OOV	85.6	80.4	82.1	78.7	79.9	78.5	84.4	81.9
	(Das, 2011)	-	-	82.8	-	82.5	-	84.2	-
	(Duong, 2013)	-	-	85.4	-	80.4	-	83.3	-
	(Gouws, 2015a)	-	-	84.8	-	-	-	82.6	-

Table 2 :

 2 Super Sense Tagging (SST) accuracy for Simple Projection, RNN and their combination.

		Model	Italian		French
	Baseline	Simple Projection	MSC-IT-1 trans man. trans. auto MSC-IT-2 61.3 45.6	MSC-FR-1 MSC-FR-2 trans. auto trans auto. 42.6 44.5
		SRNN	59.4	46.2	46.2	47.0
	SST Based RNN	BRNN SRNN-POS-In SRNN-POS-H1 SRNN-POS-H2 BRNN-POS-In BRNN-POS-H1 BRNN-POS-H2	59.7 61.0 59.8 63.1 61.2 60.1 63.2	46.2 47.0 46.5 48.7 47.0 46.5 48.8	46.0 46.5 46.8 47.7 46.4 46.8 47.7	47.2 47.3 47.4 49.8 47.3 47.5 50
		BRNN-POS-H2 -OOV	64.6	49.5	48.4	50.7
		Projection + SRNN	62.0	46.7	46.5	47.4
		Projection + BRNN	62.2	46.8	46.4	47.5
	Combination	Projection + SRNN-POS-In Projection + SRNN-POS-H1 Projection + SRNN-POS-H2 Projection + BRNN-POS-In Projection + BRNN-POS-H1	62.9 62.5 63.5 62.9 62.7	47.4 47.0 49.2 47.5 47.0	46.9 47.1 48.0 46.9 47.0	47.7 48.0 50.1 47.8 48.0
		Projection + BRNN-POS-H2	63.6	49.3	48.0	50.3
		Projection + BRNN-POS-H2 -OOV	64.7	49.8	48.6	51.0
	S-E	MFS Semeval 2013 GETALP (Schwab et al., 2012)	60.7 40.2		52.4 34.6

to train a bilingual representation regardless of the task

For RNN models, only one (same) system is used to tag German, Greek and Spanish

DAEBAK! and UMCC-DLSI for SST have obtained: 68.1% and

72.5% on Italian; 59.8% and 67.6 % on French


Antoine Carof

Mathieu Salanne

Thibault Charpentier

Benjamin Rotenberg

Collective water dynamics in the first solvation shell drive the NMR relaxation of aqueous quadrupolar cations

Using molecular simulations, we analyze the microscopic processes driving the Nuclear Magnetic Resonance (NMR) relaxation of quadrupolar cations in water. The fluctuations of the Electric Field Gradient (EFG) experienced by alkaline and magnesium cations, which determine the NMR relaxation time, are mainly due to the dynamics of water molecules in their solvation shell. The dynamics of the ion plays a less important role, with the exception of the short-time dynamics in the lighter Li + case, for which rattling in the solvent cage results in oscillations of the EFG autocorrelation function (ACF). Several microscopic mechanisms that may a priori contribute to the decay of the EFG-ACF occur in fact over too long time scales: entrance/exit of individual water molecules into/from the solvation shell, rotation of a molecule around the ion, or reorientation of the molecule. In contrast, the fluctuations of the ion-water distance are clearly correlated to that of the EFG. Nevertheless, it is not sufficient to consider a single molecule due to the cancellations arising from the symmetry of the solvation shell. The decay of the EFG-ACF, hence NMR relaxation, is in fact governed by the collective symmetry-breaking fluctuations of water in the first solvation shell.

I. INTRODUCTION

Among the plethora of experimental techniques to explore the microscopic dynamics in the condensed phase, the measurement of Nuclear Magnetic Resonance (NMR) relaxation times provides access to molecular-scale information over a broad range of time scales. NMR relaxation times characterize the return to equilibrium of an ensemble of previously excited nuclear spins, [START_REF] Abragam | The Principles of Nuclear Magnetism[END_REF][START_REF] Kowalewski | Nuclear Spin Relaxation in Liquids: Theory, Experiments, and Applications[END_REF] as a result of their weak coupling to a fluctuating electrostatic and magnetic environment. Extracting microscopic information from such a macroscopic measurement requires a model for the environment of the nuclei and of its dynamics. Numerical simulation has therefore proved very useful to establish such mechanisms for pure water 3,4 or complex systems such as proteins 5 and porous media. 6 Despite their apparent simplicity and their importance in many contexts, as well as being the focus of several theoretical and simulation studies, the molecular mechanism underlying the NMR relaxation of simple aqueous ions remains elusive. There is therefore a need to address the following points: (i) What can we really learn from their NMR relaxation rates on the dynamics of water around them? (ii) How does the chemical nature of the ions change such dynamics?

In the specific case of quadrupolar mono-atomic ions in water, the coupling between the electrostatic quadrupole of the nuclei and the Electric Field Gradient (EFG) generally controls the relaxation process. While the electrostatic quadrupole moment is a nuclear property, tabulated for the different isotopes, 7 the EFG is a dynamical quantity fluctuating together with the charge distribution around the considered nuclei.

In the present case, the microscopic processes driving the relaxation occur much faster (τ ∼ 10 ps) than the inverse of the Larmor frequency ω 0 = γB 0 , with γ the gyromagnetic ratio and B 0 the static magnetic field (>10 ns). In this so-called extreme narrowing regime, NMR theory relates these microscopic fluctuations to the macroscopic relaxation times,

1 T 1 = 1 T 2 = 3 8 2I + 3 I 2 (2I -1) ( eQ ) 2 J EFG (0), (1)
where I and eQ are, respectively, the spin and the electrostatic quadrupole of the nuclei. J EFG (0) is the spectral density at zero-frequency of the EFG experienced by the nuclei, i.e., Laplace transform (at imaginary frequency -iω) of its time Auto-Correlation Function (ACF),

J EFG (ω) =  ∞ 0 ⟨V z z (t)V z z (0)⟩ e iωt dt. (2)
V z z is the zz Cartesian component of the EFG tensor in the laboratory frame and the brackets stand for an equilibrium ensemble average. Of particular interest for simple aqueous ions is the dependence of the NMR relaxation rate on their chemical nature. One thus needs to model the effect of their mass, radius, and charge on the EFG fluctuations. This not only requires to accurately describe the EFG for each microscopic configuration but also to correctly capture the dynamics of the nucleus environment over the appropriate time scales. While the EFG experienced by the nucleus arises for a large part from the electronic cloud around it, this contribution can be captured as a linear response to the environment within the standard Sternheimer approximation, 8,9 which we recently validated using a combination of advanced quantum mechanical calculations and classical molecular dynamics. 10 Early models of the EFG arising from the surrounding solvent molecules were based on analytical theories. Hertz analyzed the contribution of a single water molecule to the EFG-ACF and assumed that the cross correlations between different molecules are proportional to the latter, 11 via a polarization factor calculated from continuum electrostatics. He further hypothesized that the most important molecular process is the reorientation of water molecules, captured by a Brownian model. Further analytical theories by Hynes and Wolynes 12 and later by Perng and Ladanyi 13 relied instead on a continuous description of the solvent, for which an accurate description of the charge or dipole dynamics is still missing.

The pioneering works of Engström and Jönsson [14][15][16] opened the way to the study of the NMR relaxation of aqueous cations with molecular simulation. 10,[17][18][19][20][21]23 Engström et al. 16 studied cooperative effects between water molecules and indicated that the translation of water may be more important than its reorientation. Roberts and Schnitker 17 investigated cross correlations of the EFG due to different water molecules and casted doubt upon Hertz's polarization factor. Kowalewski and Odelius 18 focused on the quadrupolar and dipolar relaxation mechanisms in the case of the lithium ion, also pointing out that water reorientation may play a minor role compared to translation and offering a new analysis to the self and cross terms. Quantum calculations recently offer new opportunities for the calculation of NMR relaxation rates, 20,21 in order to bypass the need for a classical force field in molecular simulations. However the entailed computational cost currently does not allow to accurately sample the relevant time scales (of the order of ns).

Using molecular simulation of aqueous ions, we recently calculated NMR relaxation rates in good agreement with experimental data. 10,22 This validation of the simulation model opened the way for a detailed analysis of the microscopic properties of the systems of interest. Our simulations highlighted that the fluctuations of the EFG experienced by the quadrupolar nucleus are dominated by the first hydration shell of the ions. 23 We proposed a simple ansatz of the charge distribution, the Radial Dipole Model (RDM), to estimate the EFG-ACF, without further investigating the microscopic dynamics giving rise to the EFG fluctuations. This fundamental question, at the origin of all the above-cited work, remains open and the present work aims at assessing the importance of several microscopic processes.

Here we analyze the effects of the ion and water dynamics on the time-dependence of the EFG, in order to correlate the observed relaxation times with microscopic mechanisms. Specifically, we consider alkaline and magnesium cations and investigate isotopic effects, the entrance/exit from the first solvation shell and several features of water dynamics within this shell: rotation, reorientation, and vibration. This allows us to demonstrate that the decay of the EFG auto-correlation function is governed by the collective, symmetry-breaking fluctuations of water in the first solvation shell.

II. METHODS

A. Theory

The EFG at the nucleus site depends on the local environment of the ion, via the corresponding charge distribution, but also on the deformation of the electronic cloud of the ion itself induced by the latter. This renders the computation of the spectral density at zero frequency J EFG (0) difficult because it requires both the accurate evaluation of the EFG for each microscopic configuration, including in principle core electrons, and the sampling of the relevant time scales. In order to obtained statistically converged J EFG (0), it is necessary to sample the different environments of the ion over several nanoseconds of trajectory, despite the decay of the EFG-ACF over shorter time scales (τ ∼ 10 ps). Such a sampling therefore remains a formidable challenge for ab initio molecular dynamics. 20 Fortunately, the response of the electronic cloud to the external charge distribution can be captured in the linear form of the so-called Sternheimer approximation. 8,9 The NMR relaxation rates then read

1 T 1 = 1 T 2 = 3 8 2I + 3 I 2 (2I -1) ( eQ ) 2 (1 + γ ∞ ) 2 J(0), (3)
where J(0) is the spectral density at zero frequency of the "external" EFG arising from the environment of the ion and γ ∞ is the nucleus-specific Sternheimer factor. 8 We have recently validated this approximation in the present case of aqueous ions and computed the Sternheimer factors corresponding to the microscopic model (see Sec. II B). 10 The dynamics of the EFG is then entirely governed by that of the external EFG via J(0), which we compute using classical molecular dynamics simulations.

B. Simulation details

Alkaline (Li + , Na + , K + , Rb + , and Cs + ) and alkaline earth (Mg 2+ ) cations in water at infinite dilution are modelled by a single ion and 215 water molecules in a cubic box of length 18.65 Å. We use a recently developed force field, based on the Polarizable Ion Model (PIM), 24 in conjunction with the polarizable water model of Dang and Chang. 25 The PIM is parametrized on density functional theory (DFT) calculations in order to reproduce both the ab initio forces and dipoles and was shown to accurately describe the thermodynamic, structural, and dynamical properties of aqueous ions. 24 As explained below, we also consider isotopic effects by considering systems where the mass of the ion is infinite (fixed ion) or where the mass of oxygen (WO model) or hydrogen (WH model) atoms is doubled.

Classical MD trajectories are generated using the CP2K simulation package. 26 For each ion, 5 independent trajectories of 500 ps in the NVE ensemble are generated, obtained after annealing at 1000 K for 150 ps followed by 50 ps equilibration at 298 K, using a time step of 1 fs. The seed configurations for our annealing process are taken from a previous work on similar systems 24 or built by positioning the water molecules and the ion on a regular grid. Periodic boundary conditions in all directions are used. A cutoff of 9.325 Å is used for short-range interactions, while electrostatic interactions and the EFG are computed with Ewald summation. 27 The normalized correlation function is obtained as

C norm EFG (τ) = ⟨V(τ) : V⟩ V 2 = 1 V 2  α, β V α β (τ)V α β , (4)
where α and β indicate the Cartesian components of the EFG tensor. Water molecules are treated as rigid using the SHAKE algorithm. 28,29 During equilibration the temperature is maintained at the target value using a Nosé-Hoover thermostat 30,31 with a time constant of 1 ps. The EFG-ACF is computed from the EFG sampled every 5 fs along the trajectories. Error bars indicate the standard deviation among the 5 independent trajectories.

III. RESULTS AND DISCUSSION

The key microscopic observable for the NMR relaxation of quadrupolar nuclei, the EFG-ACF, depends on the relative positions of the nuclei and its environment and their fluctuations over time. Since the first solvation shell of the ion is the main contribution to the EFG, 23 we examine several microscopic processes associated with their dynamics. We first analyze the effect of isotopic substitution to clarify the role of the ion dynamics, as well as water rotation and translation. We then investigate entrance/exit of water molecules from the first solvation shell. Finally, we analyze the dynamics of water in the solvation shell.

A. Isotopic effects

We first assess the role of the ion dynamics on the EFG-ACF by comparing the results obtained for each ion with that for an otherwise identical ion but with infinite mass (i.e., immobilized). This strategy, already successfully applied to obtain microscopic information on the friction coefficient of ions in water, 32 allows decoupling the ion dynamics from that of the solvent. The normalized EFG-ACF, C norm EFG , is presented in Figure 1 for Li + , K + , Cs + , and Mg 2+ . As noted previously, 23 the EFG-ACF decays in two steps, with a first characteristic time varying from 20 fs for Li + to 90 fs for Cs + and a second one of the order of 1 ps (with a maximum for K + ).

For Li + , which is lighter than water molecules, one observes oscillations with a period of ∼80 fs. These oscillations disappear for the infinitely heavy isotope. They are therefore due to the rattling of the ion in its solvation shell, as suggested by the comparison between the EFG and velocity autocorrelation functions (VACF). 23 The decay of the EFG-ACF for the infinitely heavy Li + proceeds in two steps, as for the other ions, with shallower and slower oscillations. During the first step of the decay, we note that the envelope of oscillations for Li + with a normal mass follows the curve for the infinitely heavy ion. This indicates that two mechanisms overlap for Li + during the first decay of the EFG-ACF: the rattling of this light ion in the solvent cage and the slower evolution of this cage over a few 100 fs.

For larger alkali ions (including Na + and Rb + , not shown), the results obtained with the normal and infinitely heavy ions are very similar. This suggests that most of the EFG fluctuations are due to the dynamics of the solvation shell. Since these ions are heavier than water molecules, their dynamics play a limited role at short times and only in the second step of the decay can we notice an effect of the ion translation. The slightly slower decay in the infinitely heavy case suggests that ion diffusion participates as a secondary mechanism to the decorrelation of the ACF. This long-term effect is best seen in the inset of Figure 1(a), which shows that the integral of C norm EFG converges to the plateau (∝ J(0) in Eq. ( 3)) more slowly and to a larger value than for the ions with normal mass.

We finally note that the EFG-ACF for the heavier K + and Cs + (as well as Na + and Rb + , not shown) is almost identical. This further confirms the decoupling of the ion and water dynamics in this regime. Such a decoupling is also evident in the case of Mg 2+ , for which the mass has very limited influence on the EFG-ACF and its integral, as can be seen in Figure 1(b). The ACF differs markedly from that of other ions and includes a negative minimum after ∼80 fs. Contrary to the Li + case, this cannot be due to ion rattling in the solvent cage, since this would depend on the mass of the ion. Such a feature therefore arises from the solvent dynamics.

Table I reports the spectral densities at zero frequency J(0) and J ∞ (0) for the external EFG (see Eqs. ( 2) and ( 1)) 2) and ( 3)) for aqueous ions with their normal mass, J (0), and for their infinitely heavy isotope, J ∞ (0).

Ions

J (0) (10 27 S.I.) J ∞ (0) (10 obtained with the normal and infinite mass, respectively. Quantitatively, neglecting ion translation overestimates the relaxation rate by 5% for Mg 2+ to 35% for Li + . The larger error in the latter case arises from the failure to capture the oscillations at short times. Summarizing the results of this section, we can conclude that the ion dynamics only plays a limited role in the EFG-ACF, with the exception of Li + , which is lighter than the water molecules. While not unexpected, such a conclusion suggests that one can focus on the water dynamics around the ion, to which we now turn.

As illustrated above, isotopic substitution allows us to assess the influence of molecular motion on the EFG-ACF. We now consider three models to analyze the role of water dynamics. The first two consist in doubling the mass of oxygen (WO model) or hydrogen (WH model) of water molecules. In a last model (WK-WO), the mass of the oxygen is doubled and the ion is fixed. Due to the large difference in mass of O and H, the WO model essentially corresponds to changing the mass of the water molecule while the WH model changes mainly its moment of inertia, thus allowing to separate the contributions of translation and rotation of a single molecule to the EFG fluctuations.

The EFG-ACF for all models is reported in Figure 2. Changing the moment of inertia of the molecule (WH model)

FIG. 2. Normalized correlation function C norm

EFG and their integrals (inset) of the EFG at the nucleus site of an aqueous potassium ion. In addition to the real system, we consider the effect of isotopic substitutions by doubling the mass of the oxygen (WO) or hydrogen (WH) of water molecules. In the last model (WK + ), the mass of the ion is infinite. Error bars indicate standard deviations over five independent trajectories. 1 for an infinitely heavy ion. Such a slowing down is more pronounced when the ion is infinitely heavy and the oxygen is twice heavier (WK-WO). This is consistent with the larger change in the reduced mass for an ion-water pair in this case, as can be seen in Table II. By freezing out translational or rotational water motion in the surrounding of Li + or Na + , Engström et al. 16 presented preliminary evidences that the translation of the water molecules dominates the EFG fluctuations. The present analysis of isotopic substitutions confirms their results and demonstrates that the translation of water molecules is the principal cause of the EFG fluctuations for the alkali and magnesium ions.

The fact that water is lighter than most of the ions considered here explains why the dynamics of the latter plays a secondary role. The present analysis of isotopic effects suggests a simple strategy to model the dynamics of water around the ions to describe the EFG-ACF, in order to assess the effect of its charge and radius. In the first step, one could neglect the motion of the ion and focus on the dynamics of water in the external potential arising from the ion. In the second step, the dynamics of the ions could partially be taken into account by replacing the water mass by the reduced one.

B. Entrance into and exit from the solvation shell

Since the main contribution of the EFG-ACF arises from water molecules in the first solvation shell, 23 one should first examine the entrance into and exit from the latter. The first solvation shell is defined as usual by introducing a cutoff distance at the first minimum of the radial distribution function (rdf). The rdf and corresponding cutoff for the present model of aqueous ions can be found in Ref. 24. Several measures have been proposed to quantify the dynamics of entrance into and exit from the solvation shell. [33][34][35] Here we introduce for each molecule a function S c (τ) equal to 1 if it remains continuously in the first solvation shell between 0 and τ, and to 0 otherwise, the survival probability is given by the canonical ensemble average ⟨S c (τ)⟩. Other prescriptions, such as considering the presence only at times 0 and t and not intermediate ones, result in similar conclusions and will not be discussed here. The survival probability is reported for Li + , K + , Cs + , and Mg 2+ in Figure 3(a). For the alkaline cations, the decay is almost exponential, with characteristic times decreasing along the column of the periodic table (see Table III), reflecting a tighter bound shell with smaller ions due to the stronger electric field. For Mg 2+ no exchange events are observed over tens of ps. The survival probability decays much more slowly than the EFG-ACF, so that we can investigate the role of the dynamics of a given molecule within the first solvation shell (see Section III C).

We now consider events of entrance into and exit from the solvation shell from the collective point of view. To that end, we introduce the survival probability of the solvation shell as whole as S coll c (τ) , where S coll c (τ) is equal to 1 if the identity of the water molecules in the solvation shell remained unchanged between 0 and τ and to 0 otherwise. In contrast to ⟨S c (τ)⟩, which monitors the presence of a given molecule, S coll c (τ) detects the first entrance or exit event leading to a change in the solvation shell. Figure 3(b) shows the results for Li + , K + , Cs + , and Mg 2+ , together with the EFG-ACF. The situation varies substantially between the ions. The survival probability of the solvation shell decays much more slowly than the EFG-ACF for Li + and Mg 2+ , over a similar time scale (in particular for the first decay) for K + , and faster for Cs + . This indicates that the decay of the EFG is not due to such entrance or exit events in the cases of Li + and Mg 2+ , while for Cs + the first entrance or exit occurs faster than the decorrelation of the EFG. In addition, the survival probability decays faster as the ion size increases, i.e., following the trend opposite to the EFG. These observations allow us to conclude that the decorrelation of the EFG is not driven by entrance or exit events (which could, in turn, be associated with the decorrelation of the relative orientations). The similar time scales for K + may therefore be fortuitous.

C. Water dynamics within the first solvation shell

Rotation and reorientation

We finally examine the role of the dynamics of water molecules within the solvation shell. The contribution of an individual water molecule to the EFG experienced by the ion depends on the relative position of the molecule with respect to the ion and on its orientation. Noting r the position of an oxygen of molecule in the frame of the ion, the rotation of the molecule around the ion can be monitored by computing

C rot (τ) = ⟨u(τ) • u(0)S c (τ)⟩ ⟨S c (τ)⟩ , (5)
where u = r ∥r∥ is a unit vector. One can also define a similar correlation function for the reorientation of the molecule by introducing

C reor (τ) = ⟨v(τ) • v(0)S c (τ)⟩ ⟨S c (τ)⟩ , (6)
where v is a unit vector along the symmetry axis of the molecule.

Both correlation functions are given in Figure 4(a) for Li + , K + , Cs + , and Mg 2+ . The rotation of the water molecule around the ion is slower than the decorrelation of its dipole orientation. Nevertheless, the time scales are comparable for each ion and the decay of both correlation functions extends over tens of picoseconds. This decay is approximately exponential in the 500 fs to 5 ps range, allowing to define the characteristic times τ rot and τ reor indicated in Table III. The comparable time scales for both processes (with a factor ∼2 for each ion) suggest that the reorientation of the water dipole is correlated with the rotation of the water molecule around the ion, consistently with the picture of radially oriented dipoles. Both times follow the same trend as for the survival probability τ surv , again reflecting a more rigid solvation shell for the smaller ions due to the stronger electric field. Note however that for K + , Rb + , and Cs + , τ rot and τ reor are longer than τ surv , so that the orientational correlation functions reflect the behaviour only of the molecules that stay in the solvation shell longer than the average.

Overall, the rotation-reorientation around the ion occurs slowly compared to the decorrelation of the EFG-ACF. This conclusion supports similar results reported by Engström et al. in the first molecular simulation study of the NMR relaxation rate of these aqueous ions. 16 It further confirms the weak influence of moment of inertia of the water molecule (see Section III A).

Vibration

The effect of the ion-water distance d = ∥r∥ can be monitored by the correlation function

C dist (τ) = ⟨d(τ)d(0)S c (τ)⟩ ⟨S c (τ)⟩ , (7)
which decays on the picosecond time scale towards a plateau which differs from the initial value d 2 only by a few percents (not shown). In order to obtain more insight into the fast initial decay and to compare the results for the different ions, it is convenient to introduce the normalized function

C norm dist (τ) = C dist (τ) -C dist (∞) C dist (0) -C dist (∞) . (8)
The results for Li + , K + , Cs + , and Mg 2+ are reported in Figure 4. The decay of C norm dist is much faster for the rotation and reorientation and bears strong resemblance to that of the EFG-ACF (Figure 1): The time scales are comparable, the results for K + and Cs + are almost identical, the Li + case displays oscillations with a similar frequency, and a negative part is observed at short times for Mg 2+ .

Together with the above discussion of other processes, these observations suggest that the ion-water vector vibrates in a picosecond time scale around an "average position" that evolves over longer time scales by rotation around the ion and exit from the solvation shell. The features shared by C norm dist and the EFG-ACF indicate that the fluctuations of water molecules around their average position could be the cause of the fluctuations of the EFG. Indeed, as already noted in early molecular simulations studies of the EFG experienced by aqueous cations, the symmetry of the solvation shell may lead to cancellations between various molecules. 17 In this picture, EFG fluctuations emerge from the small displacement of water molecules which break the overall symmetry of the solvation shell. 18

Symmetry-breaking fluctuations

In order to analyze such symmetry-breaking fluctuations, it is necessary to investigate collective effects which are not captured by single-molecule based observables. To address this issue, we introduce here a family of collective variables with suitable symmetry properties and a simple microscopic interpretation,

S (1) p =  i ∈S 1 r p i r i r i , (9)
S (2) p =  i ∈S 1 r p i r 2 i I -3r i ⊗ r i r 2 i , (10)
where as above r i the position of the oxygen of molecule i in the frame of the ion and where the sum runs over molecules in the first solvation shell of the ion, which evolve due to the entrance or exit of molecules. The integer p modifies the weight of the molecules according to their distance from the ion. As an example, S [START_REF] Abragam | The Principles of Nuclear Magnetism[END_REF] -1 corresponds to the position of the center of mass of the first solvation shell, in the ion frame. S [START_REF] Kowalewski | Nuclear Spin Relaxation in Liquids: Theory, Experiments, and Applications[END_REF] -2 is proportional to the inertia tensor of the solvation shell and S [START_REF] Kowalewski | Nuclear Spin Relaxation in Liquids: Theory, Experiments, and Applications[END_REF] 4 is proportional to the EFG created by radially oriented point-dipoles located at the position of the oxygen atoms. For its part, S [START_REF] Kowalewski | Nuclear Spin Relaxation in Liquids: Theory, Experiments, and Applications[END_REF] 0 only depends on the relative orientations of the ion-water vectors, not on their norm. This purely orientational order parameter is closely related to the expansion of the water density around the ion using spherical harmonics. All these functions reflect the deviations of the water molecules from their (slowly varying) average position within the solvation shell and thus the changes in the corresponding symmetry.

The normalized auto-correlation functions of S (1) -1 , S (2) 0 , and S [START_REF] Kowalewski | Nuclear Spin Relaxation in Liquids: Theory, Experiments, and Applications[END_REF] 4 are presented in Figure 5 for Li + , K + , Cs + , and Mg 2+ . Similar results and conclusions are obtained for Na + and Rb + , as well as for other functions of the family such as S [START_REF] Kowalewski | Nuclear Spin Relaxation in Liquids: Theory, Experiments, and Applications[END_REF] -2 . Except for the center of mass S [START_REF] Abragam | The Principles of Nuclear Magnetism[END_REF] -1 in the case of Li + , which displays no oscillation at short times, and for S [START_REF] Abragam | The Principles of Nuclear Magnetism[END_REF] -1 and S [START_REF] Kowalewski | Nuclear Spin Relaxation in Liquids: Theory, Experiments, and Applications[END_REF] 0 in the case of Mg 2+ , which display more oscillations than the EFG-ACF, the decay of all these ACFs is very similar to that of the EFG (see Figure 1). In particular, the decay occurs in two steps, with characteristic times close to that for the EFG, and the trends from one ion to another follow that for the EFG.

-1 (continuous line), S

0 (dotted line), and S The connection between the EFG and S [START_REF] Kowalewski | Nuclear Spin Relaxation in Liquids: Theory, Experiments, and Applications[END_REF] 4 had already been underlined in our previous work introducing the radial dipole model (RDM). 23 The fact that the other collective variables decay with the same dynamics as the EFG is more surprising. The comparison between the EFG and any of these collective variables reveals that there is no clear correlation between them (not shown), with the notable exception of S [START_REF] Kowalewski | Nuclear Spin Relaxation in Liquids: Theory, Experiments, and Applications[END_REF] 4 highlighted in Ref. 23. Nevertheless, the fact that their normalized ACFs look similar reflects that they are all related to the same underlying collective effects-all the more that they all cancel for the same symmetric configurations. In addition, the case of S [START_REF] Kowalewski | Nuclear Spin Relaxation in Liquids: Theory, Experiments, and Applications[END_REF] 0 shows that such symmetry-breaking fluctuations are not only due to the radial motion of water. We finally note that the collective variables introduced here may be useful for the development of analytical theories of the dynamics underlying the decay of the EFG-ACF, since they are simpler than the EFG itself.

The importance of the symmetry of the solvation shell for the EFG had been underlined from the static point of view. The present analysis clearly demonstrates this importance also from the dynamic point of view, and as a result for the decay of the EFG-ACF and the NMR relaxation rate. As a final remark, we note that the collective variables S (n) p are not sufficient to analyze the effect of correlations between water molecules. In the literature, 17,18 such correlations were analyzed by comparing the self-EFG-ACF (for the EFG due to one of the water molecules) to the overall EFG-ACF. However, the self-EFG-ACF suffers from the same drawbacks as C rot (τ) and C dist (τ), with much slower decay than the overall EFG-ACF. 17 The symmetry of the solvation shell implies a correlation between the slowly varying vibrationally averaged position of the molecules, described in Section III C 2. However, the present analysis does not allow to conclude on the possible correlation between the fast vibration of water molecules around these positions, which are responsible for the decorrelation of the EFG. We leave the analysis of such cross correlations between molecules for further study.

IV. CONCLUSION

The EFG fluctuations driving the NMR relaxation of quadrupolar cations are mainly due to the dynamics of water molecules in their solvation shell. The dynamics of the ion plays a less important role, with the exception of the short-time dynamics in the lighter Li + and Mg 2+ case, for which rattling in the solvent cages results in oscillations of the EFG-ACF. Several microscopic mechanisms that may a priori contribute to the decay of the EFG-ACF occur in fact over too long time scales: entrance/exit of individual water molecules into/from the solvation shell, rotation of a molecule around the ion, or reorientation of the molecule. In contrast, the fluctuations of the ion-water distance are clearly correlated to that of the EFG. Nevertheless, as one may have anticipated, it is not sufficient to consider a single molecule due to the cancellations arising from the symmetry of the solvation shell, and one should account for collective effects.

While the origin of the two-step decay of the EFG-ACF remains to be clarified, the present work underlines the importance of collective symmetry-breaking fluctuations. In that respect, future studies would benefit from using collective variables with appropriate symmetry such as the one introduced here or approaches based on effective normal modes. The latter may in addition provide a connection with THz spectroscopies which probe the dynamics over comparable time scales and may thus provide further validation of the force field used for MD simulations. Such a combination (effective normal modes and vibrational spectroscopies) might also shed some light on the peculiar behaviour of the Mg 2+ cation, which could not be explained here.

Even though contribution to the spectral density at zerofrequency arises from the decay of the EFG-ACF over a few ps for in the present case of infinitely dilute solutions, it should be noted that for finite salt concentrations additional slower mechanisms may also play a role in the NMR relaxation. Interpreting the concentration-dependence of the relaxation rate may require to take into account the relative motion of ions and the possible formation of ion pairs. Similarly, interfaces may also contribute to slower fluctuations of the EFG as the nucleus diffuses toward or away from the surface.

FIG. 1 .

 1 FIG. 1. Normalized electric field gradient autocorrelation functions, C norm EFG , and their integrals (inset) for aqueous ions with normal (full lines) and infinite (dashed lines) mass at infinite dilution: (a) Li + , K + , and Cs + and (b) Mg 2+ . Error bars indicate standard deviations over five independent trajectories.

FIG. 3 .

 3 FIG. 3. Survival probability in the first solvation shell: (a) ⟨S c ⟩ monitoring entrance/exit events for a given molecule and (b) S coll cmonitoring the first entrance/exit event (dashed lines). In (b) we also report the normalized EFG-ACF (solid lines).

FIG. 4 .

 4 FIG. 4. Normalized correlation functions for (a) the rotation (C rot , solid lines) and the self-reorientation (C reor , dashed lines) of molecules in the first solvation shell and (b) ion-water distance (C norm dist , solid lines). In (b) we also report the normalized EFG-ACF (dashed lines).

124508- 7 CarofFIG. 5 .

 75 FIG.5. Normalized correlation functions of different functions of the family defined by Eqs. (9) and (10): S

( 2 ) 4 (

 24 dashed line) around different alkali (main figure) and magnesium (inset) cations in water at infinite dilution. Error bars indicate standard deviations over five independent trajectories.

TABLE I .

 I Spectral density at zero frequency of the external electric field gradient (see Eqs. (

  27 S.I.)

	Li +	9.37	12.9
	Na +	3.11	4.26
	K +	2.19	2.62
	Rb +	1.54	1.85
	Cs +	0.90	1.20
	Mg 2+	1.65	1.74

TABLE II .

 II Reduced mass for an ion-water pair for the standard, WK, and WO models (see text).

	Model	m K (g mol -1 )	m O (g mol -1 )	m red (g mol -1 )
	. . .	39.1	16.0	12.3
	WO	39.1	32.0	18.2
	WK	∞	16.0	18.0
	WK, WO	∞	32.0	34.0
	has no effect on the decay of the EFG, showing that the
	reorientation of water molecules plays a limited role on the
	decorrelation of the EFG, hence on the NMR relaxation
	rate. Changing the mass of the molecule (WO model) results
	in a limited slowing down of the decay, comparable to that
	observed in Figure

TABLE III .

 III Characteristic times for the decay of the survival probability ⟨S c ⟩ and the rotation (C rot ) and reorientation (C reor ) of water molecules. For the latter two, τ rot and τ reor are determined by an exponential fit between 500 fs and 5 ps.

	Ions	τ surv (ps)	τ rot (ps)	τ reor (ps)
	Li +	54.0	26.7	13.8
	Na +	10.5	27.3	10.8
	K +	3.2	17.2	9.0
	Rb +	3.1	14.7	7.7
	Cs +	2.5	14.7	7.3
	Mg 2+	. . .	89.5	30.0

Reuse of AIP Publishing content is subject to the terms: https://publishing.aip.org/authors/rights-and-permissions. Downloaded to IP: 132.166.19.195 On: Fri, 30 Sep

ACKNOWLEDGMENTS

A.C. acknowledges financial support from UPMC.