Juan Manuel Coria

Mathilde Veron

Sahar Ghannay

Guillaume Bernard

Hervé Bredin

Olivier Galibert

Sophie Rosset

Analyzing BERT Cross-lingual Transfer Capabilities in Continual Sequence Labeling

whether they are published or not. The documents may come

Introduction

State-of-the-art models for Natural Language Processing (NLP) usually leverage deep neural networks. In particular, pre-trained Transformerbased [START_REF] Vaswani | Attention is All you Need[END_REF] language models like BERT [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF] have proven to perform very well on various NLP tasks, often achieving state-of-the-art results [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF][START_REF] Brown | Language models are few-shot learners[END_REF]. These models are pre-trained in a self-supervised way on large text corpora and rely on knowledge transfer to solve downstream tasks, where the pre-trained model is fine-tuned on the target task. Multilingual versions of these models have also been trained and demonstrate high cross-lingual transfer as well [START_REF] Karthikeyan | Cross-Lingual Ability of Multilingual BERT: An Empirical Study[END_REF][START_REF] Wang | Extending multilingual BERT to lowresource languages[END_REF][START_REF] Conneau | Unsupervised cross-lingual representation learning at scale[END_REF][START_REF] Xue | mt5: A massively multilingual pre-trained text-to-text transformer[END_REF]. Given the interest in these models for cross-lingual transfer, it is of great importance to better understand this phenomenon as well as its limits.

In this work, we analyse the cross-lingual transfer capabilities of multilingual BERT and we work on sequence labeling, where each token of a sentence must be annotated with a specific label. This problem regroups various NLP tasks like Named Entity Recognition (NER), Part-Of-Speech (POS) Tagging, text chunking and slot-filling. We focus our study on two of these tasks using two multilingual corpora1 : MultiATIS++ for slot-filling [START_REF] Xu | End-to-end slot alignment and recognition for crosslingual NLU[END_REF] and MultiCoNER for NER (Malmasi et al., 2022a,b). Experimenting on different corpora allows us to identify which observations may generalize and which ones may be corpus specific.

While most cross-lingual transfer studies about slot-filling or NER focus either on joint training or training on a source and a target language [START_REF] Xu | End-to-end slot alignment and recognition for crosslingual NLU[END_REF][START_REF] Schuster | Cross-lingual transfer learning for multilingual task oriented dialog[END_REF][START_REF] Arkhipov | Tuning multilingual transformers for language-specific named entity recognition[END_REF][START_REF] Mueller | Sources of transfer in multilingual named entity recognition[END_REF][START_REF] Wang | Extending multilingual BERT to lowresource languages[END_REF], our main contribution is a study with special focus on continual cross-lingual transfer, where the model performs one single task but is progressively adapted over a sequence of languages.

We believe this experimental setup to be interesting not only as a novel way of studying crosslingual transfer but also because it is better suited to real case scenarios. Indeed, adaptation to new data over time is a highly desirable feature of most NLP models: oftentimes, collecting data and an- notating them is expensive, which makes training data scarce or incomplete at the beginning of a project. Additionally, model requirements might also evolve with time based on the needs of the users. This means that the model has to adapt sequentially as training data becomes available. An example of this could be a dialogue system that is gradually deployed in different countries. Unfortunately, naive solutions to adapt a previously trained model are costly, as they require either re-training from scratch or maintaining many distinct models. On the other hand, progressively training on multiple datasets that become available one by one is at the heart of continual learning [START_REF] Hadsell | Embracing Change: Continual Learning in Deep Neural Networks[END_REF], where the goal is for a model to improve itself both on past and new data. We refer to these datasets and the order in which they appear as a training sequence (f.i. see Figure 1). Traditional training schemes assume that training examples (in our case annotated sentences) are independent and identically distributed (i.i.d.), which does not usually hold when data becomes available sequentially. Moreover, access to previous data is not allowed2 , as this represents a linear use of resources with respect to the length of the sequence, which can in theory be infinite. In this context, transfer is generally divided in two: forward and backward [START_REF] Hadsell | Embracing Change: Continual Learning in Deep Neural Networks[END_REF][START_REF] Lopez | Gradient Episodic Memory for Continual Learning[END_REF][START_REF] Arora | Does an LSTM forget more than a CNN? an empirical study of catastrophic forgetting in NLP[END_REF], defined in our case as improvement on future and already acquired languages respectively. The biggest challenge of continual learning systems is catastrophic forgetting [START_REF] Hadsell | Embracing Change: Continual Learning in Deep Neural Networks[END_REF][START_REF] French | Catastrophic forgetting in connectionist networks[END_REF], which is defined as a strong performance loss in previously acquired knowledge [START_REF] Xu | End-to-end slot alignment and recognition for crosslingual NLU[END_REF] and Multi-CoNER (Malmasi et al., 2022a).

(i.e. negative backward transfer). While previous studies on continual learning tend to focus on the domain axis for the slot-filling task [START_REF] Lee | Toward continual learning for conversational agents[END_REF][START_REF] Madotto | Continual learning in task-oriented dialogue systems[END_REF], or on the class axis for the NER task [START_REF] Monaikul | Continual learning for named entity recognition[END_REF][START_REF] Xia | Learn and review: Enhancing continual named entity recognition via reviewing synthetic samples[END_REF], we concentrate on the axis of language adaptation. Similar work also investigates cross-lingual transfer of multilingual BERT fine-tuned on sequence labeling tasks, namely NER and POS-Tagging [START_REF] Liu | Preserving Cross-Linguality of Pre-trained Models via Continual Learning[END_REF]. They focus on preserving masked language modeling performance and cross-lingual ability after fine-tuning on one of the two tasks on English only, with a method developed as part of continual learning. Conversely, our work focuses on fine-tuning on a single task over a sequence of many languages.

In this paper, we first describe in Section 2 and 3 the task, the corpora and the model we are working with. Then in Section 4 we define the different continual learning metrics that we use in our experiment. Our study is guided by the following research questions, as presented in Section 5: does cross-lingual transfer exist during continual training or does catastrophic forgetting prevent it? How much transfer can we expect relative to monolingual and multilingual i.i.d. training? In Section 6 we perform an extensive analysis on MultiATIS++ in order to understand how transfer is affected by the training sequence. Finally, in Section 7 we investigate whether lost performance (due to forgetting) can be recovered and at what cost.

2 Task and corpora

Sequence labeling

In sequence labeling, each token of a sentence must be annotated with a specific label. Hence, it is appropriate to identify concepts or entities in sen- [START_REF] Ramshaw | Text Chunking using Transformation-Based Learning[END_REF] labels for an utterance of MultiATIS++ [START_REF] Xu | End-to-end slot alignment and recognition for crosslingual NLU[END_REF] in English. Label "O" (from outside) denotes that no concept is mentioned, "B" (from beginning) denotes the first word of a concept and "I" (from inside) the continuation of a concept. Different slot types are shown in different colors.

tences. In our case, the labels to predict are the same across languages so that the task remains unchanged over the continual learning process.

Sequences are labeled using the IOB format [START_REF] Ramshaw | Text Chunking using Transformation-Based Learning[END_REF], where labels consist of a prefix (B,I or O) and an optional type that categorizes the identified concept. While O indicates that the token is not part of a concept (O for outside), B and I indicate that it is the beginning or continuation of a concept, thus allowing the identification of multi-token concepts. An example of this labeling scheme is shown in Figure 2. This task is usually evaluated using the slot micro F1 score [START_REF] Tjong | Introduction to the CoNLL-2000 Shared Task Chunking[END_REF].

MultiATIS++

The MultiATIS++ multilingual corpus comes from the Air Travel Information System (ATIS) corpus [START_REF] Hemphill | The ATIS spoken language systems pilot corpus[END_REF], consisting in utterances of users asking for flight information. The corpus focuses on the slot-filling task, which is related to task-oriented dialogue systems. It enables the system to identify the important concepts mentioned by the user that are needed to successfully continue the dialogue. These concepts are related to the system's domain and to the tasks that the system should perform. This corpus is the manual translation of the original English (EN) ATIS sentences into 6 different languages: Spanish (ES), Portuguese (PT), German (DE), French (FR), Chinese (ZH) and Japanese (JA). It also includes two additional languages: Hindi (HI) and Turkish (TR), that were added as part of MultiATIS in [START_REF] Upadhyay | almost) zero-shot cross-lingual spoken language understanding[END_REF].

Contrary to the translations added in Multi-ATIS++, the number of utterances of Hindi and Turkish translations are not as many as for the other languages. More details on the composition of Mul-tiATIS++ are shown in Table 1.

MultiCoNER

The MultiCoNER corpus was proposed as part of the SemEval 2022 Task 11 (Malmasi et al., 2022a,b) and focuses on the NER task. While it is usually a generic task consisting in identifying entities like people, organizations, locations or dates in written texts, this corpus focuses on detecting ambiguous and complex entities in short and lowcontext settings. These entities are person, location, group, corporation, product and creative work. MultiCoNER also aims at stimulating the research on multilingual models, as it contains annotations in 11 languages. For a fair comparison with Mul-tiATIS++, we restrict these experiments to also contain 9 languages, namely Bengali (BN), German (DE), English (EN), Spanish (ES), Hindi (HI), Korean (KO), Dutch (NL), Turkish (TR) and Chinese (ZH). More details on the composition of Mul-tiCoNER are shown in Table 1. In the rest of the paper and for both corpora we denote the train, dev and test sets of a given language i with a subscript (e.g. train i ).

Model

We use the multilingual BERT [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF] base model, consisting of 12 multi-head attention layers with 12 heads and hidden size of 768 (177M parameters). This model was trained on large Wikipedia dumps from 104 different languages using masked language modelling and next sentence prediction objectives.

As we use the model for sequence labeling, we append a two-layer feed-forward classifier with hidden size 768 and ReLU (rectified linear unit) activation [START_REF] Nair | Rectified Linear Units Improve Restricted Boltzmann Machines[END_REF]. The input of the classifier are the last layer word hidden states after applying dropout with p = 0.1.

Following [START_REF] Xu | End-to-end slot alignment and recognition for crosslingual NLU[END_REF], we train the model on MultiATIS++ using the Adam optimizer (Kingma and Ba, 2015) with a learning rate of 10 -5 and a batch size of 32 utterances for 50 epochs (unless stated otherwise), selecting the model with the highest slot F1 on the corresponding dev set. We train the model on MultiCoNER the same way, except for the learning rate (optimized on dev and set to 5 × 10 -5 ) and the number of epochs, which is set to 15. We evaluate the model on all test i sets for every language i using the slot F1 calculated with the seqeval library [START_REF] Nakayama | seqeval: A python framework for sequence labeling evaluation[END_REF].

Continual Learning Metrics

Cross-lingual transfer can be defined as the performance improvement of a model on a particular language based on knowledge of other languages. This can take several forms depending on the training structure. In an i.i.d. context, where all data are available from the start, we think of transfer in terms of joint training. If training on language i and j jointly (multilingual) yields better performance on j than training only on j (monolingual), then there is transfer from i to j.

However, continual learning adds a different dimension. Indeed, when training on a language sequence we can identify two types of transfer: forwards and backwards [START_REF] Hadsell | Embracing Change: Continual Learning in Deep Neural Networks[END_REF][START_REF] Lopez | Gradient Episodic Memory for Continual Learning[END_REF]. Forward transfer denotes the performance and learning efficiency improvement on a given language thanks to previously acquired knowledge of other languages. Conversely, backward transfer denotes the performance improvement on a previously acquired language when learning a new one. More formally, and similarly to [START_REF] Lopez | Gradient Episodic Memory for Continual Learning[END_REF], given a sequence of L languages, we define the performance matrix P ∈ R L×L , where P ij is the performance of language i after learning language j. In this context, backward transfer of i is defined as:

BT i = P iL -P ii (1)
Negative backward transfer is also called forgetting, as it denotes performance loss on previous languages. Since P 11 is equivalent to monolingual performance mono 1 , we can define backward transfer of the first language after learning language j:

BT 1j = P 1j -mono 1 (2)
Conversely, we define forward transfer as:

FT mono i = P ii -mono i (3)
where mono i denotes monolingual performance on language i. By comparing performance with a different baseline like multilingual, we can measure how close forward transfer is to joint transfer:

FT multi i = P ii -multi i (4)
where multi i denotes the multilingual performance on language i. These definitions will be useful for the analysis in Section 6.

Cross-lingual Transfer

Does transfer exist during continual training or does catastrophic forgetting prevent it?

Before studying the continual learning scenario, we first measure transfer when training the model on all languages at once (i.e. joint transfer). Then, having this frame of reference, we investigate transfer when training the model on each language sequentially (i.e. continual transfer).

Joint Transfer

In order to measure transfer in unstructured i.i.d. training, we train the model on all languages together (multilingual) and compare the performance we obtain with monolingual training. Note that multilingual training corresponds to concatenating all train i for training and all dev i for validation. We report the mean and standard deviation of test slot F1 per language across 5 runs to reduce the effect of randomness.

Results on MultiATIS++ are reported in Table 2. We observe that multilingual is always stronger than monolingual (except for Chinese and Japanese), which confirms the existence of joint cross-lingual transfer. European languages (German, English, Spanish, French and Portuguese) show modest but visible gains from transfer, whereas Asian languages (Chinese and Japanese) do not seem to benefit from it. However, transfer for the two low resource languages (Hindi and Turkish) is outstanding, with an absolute 4.8% and 13.9% improvement. As noted in [START_REF] Do | To what degree can language borders be blurred in BERT-based multilingual spoken language understanding[END_REF], MultiATIS++ translations keep the same (unrealistic) slot values for particular labels (e.g. American departure city and destination city in Turkish utterances). We suspect this may be the reason why transfer is particularly high in this corpus. The fact that the corpus contains less training data for Hindi and Turkish than for the other languages might also explain why joint transfer is much higher for these two languages.

Table 3 shows results on MultiCoNER. Monolingual results are much lower than in MultiATIS++ even if the number of labels to predict is much lower, suggesting that MultiCoNER is more difficult than MultiATIS++. Although the corpus is not parallel, we observe significant joint cross-lingual transfer (except for Chinese where it is negligible). This is somehow surprising considering that only a maximum of 8% of entity mentions appearing in the test set of a given language are common to those appearing in the train set of other languages.

However, multilingual training assumes that all languages are available at once. As mentioned before, this is not always true in practice, since utterances may be scarce and annotations expensive. Moreover, given N the maximum number of utterances per language and L the number of languages, training on a new language has time cost O(LN ), as the whole model needs to be trained from scratch. A naive solution is to use multiple monolingual models, raising however the space cost to O(LN ). Reducing both costs to O(N ) motivates our decision to structure training as a sequence.

Continual Transfer

Given a training sequence (a list of languages in a given order), continual learning consists in training the model on train i (and validating on dev i ) for each language i in the given order, as depicted in Figure 1. Although having all languages at once is not required and the language addition cost is the lowest, this approach is prone to forgetting previously learned languages.

In the experiments of this section, we report for both forward and backward transfer the average performance per language. The experiments consist of 3 sequences per language and per transfer type repeated 5 times to reduce the effect of randomness, making a total of 54 sequences and 270 experiments. These 3 sequences per language are chosen randomly and maximizing the Kendall rank correlation coefficient [START_REF] Abdi | The Kendall rank correlation coefficient[END_REF] as a distance criterion so that they are as dissimilar as possible.

We first investigate whether forward transfer exists in continual training by looking at the average P LL performance (e.g. model 4 evaluated on English in Figure 1) against monolingual and multilingual. Notice that we look at the performance of the last language, as this allows us to measure whether the model leverages past knowledge to learn a new language. This has the advantage of isolating the effect of forward transfer from that of backward transfer. When generating the sequences we also make sure that each language appears at the end of the sequence the same number of times.

Similarly, we look at backward transfer by comparing the average P 1L performance (e.g. model 4 evaluated on Spanish in Figure 1) against monolingual, making sure that each language appears at the beginning of the sequence the same number of times. This way we can determine whether the initial performance (equal to monolingual) improves with the introduction of new languages to the model. We also look at the performance of the first language, so that the effect of backward transfer is isolated from that of forward transfer.

Notice that whether we focus on the first or the last language, we always look at the performance at the end of the training sequence so that the comparison to multilingual is fair.

Results on MultiATIS++ are reported in Table 2. We observe that continual training benefits from cross-lingual forward transfer. Indeed, P LL is on average closer to multilingual than to monolingual performance. However, although transfer is present for the last language, P 1L suffers from the opposite effect, even falling under monolingual performance. Our results show that contrary to what we expected from the identical slot values of MultiATIS++ (e.g. American departure city and destination city in Turkish utterances), the naturally occurring crosslingual transfer completely vanishes in previous languages.

Similar observations can be made from Multi-CoNER continual experiments from Table 3. Although forward transfer is high in general, it is also lower than the standard deviation for Bengali, Hindi and Korean, and even negative for Chinese. The negative backward transfer values also show that the model forgets a lot about the first language it learnt.

Overall we can see that continual training benefits from forward transfer, although still not performing as well as the multilingual topline, whereas forgetting is clearly present.

Training Sequence

How is transfer affected by the training sequence?

In order to better understand the effect of the training sequence on transfer, we first look at measures of forward transfer at each position relative to monolingual and multilingual. Secondly, we study the impact of the training sequence length on backward transfer measured on the first language. This analysis is conducted only on MultiATIS++ due to time and computational constraints. In the figures of this section, the mean, median and percentiles do take into account eventual outlier languages, while the minimum and maximum do not.

When considering forward transfer, Figure 3a shows that apart from the first position (equal to monolingual), the model consistently benefits from transfer at any point in the sequence, as performance is higher than monolingual. Interestingly, due to some outlier languages (generally Hindi and Turkish), we observe that the means are poor estimates of the distribution when measuring FT mono i . This is an indicator that commonly used continual transfer metrics might over-or underestimate real performance when transfer is not uniformly distributed among languages. Indeed, these metrics usually consist of averages across the adaptation axis [START_REF] Lopez | Gradient Episodic Memory for Continual Learning[END_REF]. In Figure 3b, we also observe that performance gets closer to multilingual as the sequence advances, although it rarely outperforms it.

As per backward transfer, Figure 4 shows that performance of the first language is in general worse than monolingual for any given sequence length. In particular, we observe that performance loss is not strictly monotonic, which means that measuring forgetting between the beginning and the end of the sequence may not be sufficient to explain how the model forgets. Note that a sequence of L = 7 would have shown less forgetting than a sequence of L = 5. Furthermore, as hinted by continual experiments from Table 2, we observe that backward transfer deteriorates as forward transfer improves with the length of the sequence. Since negative backward transfer (i.e. forgetting) tends to be linked to a loss of previously acquired knowledge, it is surprising that new language performance keeps increasing while performance of known languages decreases. Our results indicate that the preserved knowledge that facilitates the acquisition of a new language in multilingual BERT for slot filling is not the same knowledge that preserves previous language performance. This might be explained by a progressive shift of model parameters towards a better multilingual initialization for the ATIS task that might however fail to retain the specificities of previous Figure 4: Distributions of first language backward transfer BT 1j = P 1j -mono 1 (higher is better) on test 1 for different sequence lengths j. We average across 54 sequences and 5 runs. Note that BT 1j = 0 if performance is equal to monolingual. Outliers not shown for readability.

languages. This hypothesis motivates our next research question.

Fast Recovery

Can lost performance due to forgetting be recovered?

Given that forward transfer does not seem to be affected by forgetting, we investigate in this section whether performance lost as a result of forgetting can be recovered quickly after continual training. The ability to recover is especially interesting for MultiCoNER where forgetting is pretty high, but we still conduct experiments on both corpora. To investigate if this is possible, we first set out to discover whether the model shifts towards a better multilingual initialization. Hence we compare the multilingual performance of the initial model 0 (consisting of BERT and a random classifier) against model L , the model at the end of training sequence (e.g. model 4 in Figure 1). In particular, we train both mod-els on all languages jointly for different numbers of epochs and evaluate on each language. Notice that model L comes from our continual P 1L experiments (see Table 2). The results are presented in Tables 4 and5.

The comparison between model 0 multilingual and model L multilingual for both corpora shows two interesting results. On one hand, we observe that even one epoch of multilingual training for model L achieves better performance than the monolingual baseline (model 0 monolingual) and is even close to the multilingual topline (model 0 multilingual)3 , both of which are trained on the maximum number of epochs (50 or 15). This means that model L is capable of achieving good multilingual performance with very little training, hence canceling the effect of forgetting. On the other hand, we see that model L multilingual performance is greatly superior to model 0 multilingual with a single training epoch. This is not surprising given that the classifier is initialized randomly in model 0 , but it shows that the model is capable of retaining knowledge from previous languages, although it is not clear whether that knowledge is preserved in the classifier or in BERT.

We dive deeper into this question by training model L with a random classifier in the same manner (see model L + rnd clf multi. in Table 4). We observe that performance is still greatly superior to model 0 multilingual with a single epoch. However, performance is not as high as model L multilingual (although slightly in MultiCoNER), which keeps its continually trained classifier. This indicates most of the knowledge retained from previous languages is stored in BERT, and that the knowledge stored in the classifier is dependent on the corpus.

Overall, these results lead us to think that for the sequence labeling task, continual training over the language sequence does indeed shift model parameters to a better multilingual initialization. As a result, we explore the possibility to leverage this phenomenon in order to quickly recover lost language specificities due to forgetting for both corpora. To do this, we train model L on the first language of the sequence a second time (i.e. as if it were an (L + 1) th language) and evaluate on the first language only. As shown in Tables 4 and5, when comparing model L monolingual to model 0 monolingual (equal to first language performance P 11 ), we see that the performance of the first language can be recovered and improved upon with as little as a single training epoch 3 . These results are outstanding for MultiCoNER considering the high forgetting that we previously observed. On Mul-tiATIS++, model L monolingual even achieves 50epoch model 0 multilingual performance in most cases after only one epoch, with the remaining languages still showing a big improvement. In particular, Hindi and Turkish improve an absolute 3.9% and 7.8% from model 0 monolingual respectively. Note that for MultiATIS++ increasing the number of recovery epochs for the first language does not bring considerable improvements. The only exception to this observation is Turkish, which might be explained by the small size of its training set. In MultiCoNER however, performance still improves after 5 epochs, getting closer to the multilingual topline. Surprisingly, model L monolingual is even on par with the multilingual topline for Turkish and Chinese. Although the cost of adding a language remains O(N ), the ability to recover all languages raises costs to O(LN ), making it expensive to use in practice. The design of a strategy taking full advantage of these recovery capabilities to limit forgetting with lower cost is left for future work.

Discussion

To summarize, we observe a high level of crosslingual transfer in the i.i.d. setting when learning the sequence labeling task on all languages jointly for both corpora. In a real low resource scenario where data and annotations are scarce, it may be difficult or even impossible to implement either a monolingual or multilingual adaptive approach, as time/space complexity is high and not all languages might be available at once. In a continual learning setting where languages are learned in sequence, these costs are the lowest and cross-lingual transfer is retained in the form of forward transfer. However, forgetting occurs for the first language of the sequence since performance consistently drops below monolingual.

When looking at continual cross-lingual transfer across the entire sequence, we obtain two surprising results. First, commonly used continual transfer metrics may not be a reliable estimate of the performance distribution across languages when transfer is not evenly distributed. Since even in other adaptation axes a considerable variability across datasets is to be expected, we believe a statistic like the median might be a better choice, as we believe it better represents expected performance at any given point. Second, as the sequence progresses, forward transfer improves, while backward transfer diminishes. This might indicate that model parameters remain a good initialization for future languages but that previous language specificities might be lost.

Motivated by this hypothesis, we compare the model at the beginning and at the end of the training sequence. Our results suggest that knowledge from past languages is mostly stored in BERT (as opposed to the task-specific classifier) and that the model may indeed shift towards a better multilingual initialization, making it suitable to quickly recover the performance lost as a result of forgetting. We then measure the recovery capabilities of the model with respect to the first language of the sequence. We empirically show that lost performance can be recovered with as little as a single training epoch even if forgetting is high (like in MultiCoNER). Performance can even greatly improve and approach the i.i.d. multilingual topline after only one training epoch for MultiATIS++ and 5 epochs for MultiCoNER.

In light of the above, we believe that effective continual learning methods for this task would benefit from leveraging recovery capabilities (either for a single language or many languages jointly) to limit the effect of forgetting, while preserving or even boosting forward transfer.

Conclusion

In this paper, we presented an analysis of crosslingual transfer in continual learning for the sequence labeling task using multilingual BERT [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF] as well as the MultiATIS++ [START_REF] Xu | End-to-end slot alignment and recognition for crosslingual NLU[END_REF] and MultiCoNER (Malmasi et al., 2022a) corpora.

Our main finding suggests that although forgetting is present, cross-lingual transfer is retained in the form of forward transfer, which allows the model to have substantial recovery capabilities. Moreover, we empirically show that: 1) high forward transfer is linked to a progressive shift of model parameters towards a better multilingual initialization, and 2) that most knowledge from past languages is stored in the word representation encoder (BERT) and not in the task-specific classifier. Finally, we also find that current continual learning metrics may need to be adapted if we want to better estimate the distribution of transfer across the adaptation axis.

As future work, we would like to reduce training costs by leveraging fast recovery for continual learning across languages. Another interesting research direction would be a study on the continual acquisition of languages not already present in multilingual BERT.

Reproducible Research

In the spirit of reproducible research, we release our code as open source available at github.com/juanmc2005/ContinualNLU.

Figure 1 :

 1 Figure 1: Depiction of a training sequence across 4 languages. For each language in the given order, we train the model on its training set, select the best epoch on the development set and then test on all test sets independently.

Figure 2 :

 2 Figure 2: Example of slot filling IOB[START_REF] Ramshaw | Text Chunking using Transformation-Based Learning[END_REF] labels for an utterance of MultiATIS++[START_REF] Xu | End-to-end slot alignment and recognition for crosslingual NLU[END_REF] in English. Label "O" (from outside) denotes that no concept is mentioned, "B" (from beginning) denotes the first word of a concept and "I" (from inside) the continuation of a concept. Different slot types are shown in different colors.

=Figure 3 :

 3 Figure3: Distributions of forward transfer on test i relative to monolingual and multilingual for different positions i in the sequence. We average over 54 sequences and 5 runs. Note that forward transfer is 0 when performance is equal to (a) monolingual and (b) multilingual. Outliers not shown for readability.

Table 2 :

 2 Slot F1 performance on MultiATIS++ on test i sets for monolingual, multilingual and continual experiments. The latter are calculated as the average of the first (P 1L ) or last (P LL ) language (indicated by the column) at the end of the sequence. See Equations 2 and 3 for the definition of BT 1L and F T mono1L. Reported values are the average of 5 runs with standard deviation shown in parenthesis. Model time cost denotes the cost of adding a new language to the model measured in iterations. Model space cost is the size of the model measured in number of parameters. Data space cost represents the maximum number of training sentences stored in memory at the same time.

	Training	DE	EN	ES	FR	PT	ZH	JA	HI	TR	Model Cost Time Space	Data Cost Space
	Monolingual	94.4 (0.2) 95.6 (0.1) 88.9 (0.4) 93.2 (0.1) 90.3 (0.6) 93.3 (0.4) 93.1 (0.4) 82.4 (0.5) 71.3 (0.9) ≤224K 1.6B	≤4K
	Multilingual	95.0 (0.2) 96.0 (0.2) 90.4 (0.4) 94.0 (0.3) 91.4 (0.2) 93.6 (0.2) 93.0 (0.1) 87.2 (0.3) 85.2 (0.6)	1.7M	178M	33K
	Joint transfer	+0.6	+0.4	+1.5	+0.8	+1.1	+0.3	-0.1	+4.8	+13.9	-	-	-
	Continual (P LL ) 94.9 (0.2) 95.9 (0.1) 89.9 (0.5) 93.9 (0.3) 91.3 (0.3) 93.9 (0.3) 93.1 (0.3) 85.6 (0.7) 84.0 (0.6) ≤224K 178M	≤4K
	F T mono 1L	+0.5	+0.3	+1.0	+0.7	+1.0	+0.6	+0.0	+3.2	+12.7	-	-	-
	Continual (P 1L ) 94.0 (0.7) 95.5 (0.2) 89.2 (0.5) 91.4 (1.7) 88.4 (4.9) 92.0 (1.0) 91.7 (0.7) 80.5 (1.8) 68.1 (3.5) ≤224K 178M	≤4K
	BT 1L	-0.4	-0.1	+0.3	-1.8	-1.9	-1.3	-1.4	-1.9	-3.2	-	-	-
	Training	BN	DE	EN	ES	HI	KO	NL	TR	ZH	Model Cost Time Space	Data Cost Space
	Monolingual	41.6 (3.2) 64.1 (0.8) 61.3 (0.6) 59.0 (0.8) 43.1 (1.2) 56.7 (0.7) 61.4 (0.9) 45.7 (0.7) 57.6 (0.8) 765K 1.6B	15K
	Multilingual	44.9 (1.6) 66.9 (0.4) 64.4 (0.7) 63.8 (0.4) 46.4 (1.2) 59.4 (0.8) 66.5 (0.5) 50.6 (1.0) 58.2 (1.0) 6.9M 178M	138K
	Joint transfer	+3.3	+2.8	+3.1	+4.8	+3.3	+2.7	+5.1	+4.9	+0.6	-	-	-
	Continual (P LL ) 43.4 (1.8) 66.0 (0.6) 63.0 (0.6) 62.1 (0.9) 44.2 (1.0) 57.0 (0.7) 64.6 (0.6) 50.1 (0.8) 56.2 (1.3) 765K 178M	15K
	F T mono 1L	+1.8	+1.9	+1.7	+3.1	+1.1	+0.3	+3.2	+4.4	-1.4	-	-	-
	Continual (P 1L ) 31.7 (4.5) 50.9 (1.5) 52.5 (2.6) 51.1 (2.3) 32.2 (2.4) 43.2 (2.4) 55.4 (3.4) 37.4 (1.9) 40.0 (2.8) 765K 178M	15K
	BT 1L	-9.9	-13.2	-8.8	-7.9	-10.9	-13.6	-6.0	-8.3	-17.6	-	-	-

Table 3 :

 3 Slot F1 performance on MultiCoNER on test i sets for monolingual, multilingual and continual experiments. Same comments from Table 2 apply.

Table 4 :

 4 Slot F1 performance on test i sets for MultiATIS++ fast recovery experiments. model L monolingual performance is averaged over 3 sequences (the P 1L experiment ones starting with the language in question), while model L multilingual is averaged over all 27 sequences from P 1L experiments. Both model 0 and model L experiments are averaged over 5 runs (standard deviation in parenthesis).

	Model	Epochs	BN	DE	EN	ES	HI	KO	NL	TR	ZH
	model 0 multi. (i.i.d.)	1 5 15	36.2 (1.4) 63.1 (0.8) 61.6 (0.6) 60.5 (0.6) 40.5 (1.4) 56.9 (0.4) 63.5 (0.7) 45.5 (0.6) 53.1 (2.4) 43.0 (1.1) 66.6 (1.0) 63.9 (0.2) 63.7 (0.6) 45.4 (1.5) 58.9 (0.7) 66.3 (0.7) 49.7 (1.4) 57.7 (1.5) 44.9 (1.6) 66.9 (0.4) 64.4 (0.7) 63.8 (0.4) 46.4 (1.2) 59.4 (0.8) 66.5 (0.5) 50.6 (1.0) 58.2 (1.0)
	model L	1	42.7 (1.7) 65.8 (0.7) 63.6 (0.7) 63.0 (0.8) 44.8 (1.4) 58.8 (1.0) 65.9 (0.8) 49.8 (1.0) 56.7 (1.3)
	multi. (i.i.d.)	5	43.8 (1.4) 66.4 (0.6) 64.1 (0.5) 63.5 (0.6) 45.4 (1.1) 59.2 (0.8) 66.4 (0.5) 50.6 (0.9) 57.6 (1.2)
	model L	1	42.6 (1.8) 65.5 (0.7) 63.3 (0.6) 62.7 (0.8) 44.7 (1.3) 58.7 (0.8) 65.7 (0.7) 49.6 (1.2) 56.6 (1.4)
	+ rnd clf multi.	5	43.7 (1.4) 66.3 (0.6) 63.9 (0.6) 63.4 (0.7) 45.2 (1.1) 59.1 (0.8) 66.2 (0.6) 50.4 (1.0) 57.6 (1.1)
	model 0 mono. (i.i.d.)	15	41.6 (3.2) 64.1 (0.8) 61.3 (0.6) 59.0 (0.8) 43.1 (1.2) 56.7 (0.7) 61.4 (0.9) 45.7 (0.7) 57.6 (0.8)
	model L	1	41.8 (2.4) 65.5 (0.7) 63.7 (0.8) 61.6 (0.5) 44.2 (1.1) 57.6 (0.4) 64.6 (0.7) 49.5 (1.0) 56.0 (0.9)
	mono.	5	43.6 (1.8) 66.5 (0.5) 64.0 (0.6) 62.4 (0.6) 45.4 (0.7) 57.9 (0.5) 65.0 (0.8) 50.7 (0.7) 58.3 (0.9)

Table 5 :

 5 Slot F1 performance on test i sets for MultiCoNER fast recovery experiments. Same comments from Table4apply.

We do not work on the recent MASSIVE(FitzGerald et al.,

2022) corpus as we consider it too similar to Multi-ATIS++. We also avoid Universal Depedencies[START_REF] Nivre | Universal Dependencies v2: An evergrowing multilingual treebank collection[END_REF] because we consider POS tagging to be too simple for this type of study. Moreover, the amount of per-language data in the latter could bias the transfer we observe.

Access to previous data is sometimes allowed if limited[START_REF] Robins | Catastrophic Forgetting, Rehearsal and Pseudorehearsal[END_REF]

Except for Chinese on MultiCoNER, which is not surprising considering that its joint transfer is negligible.

Acknowledgements

This work has been partially funded by the LIHLITH project (ANR-17-CHR2-0001-03), and supported by ERA-Net CHIST-ERA, and the "Agence Nationale pour la Recherche" (ANR, France). It has also been partially funded by French ANRT under CIFRE PhD contract # 2019/0628. It was also possible thanks to the Saclay-IA computing platform and was performed using HPC resources from GENCI-IDRIS (Grant 2022-AD011012609R1).