Alvaro Santamaria-Gomez

Sari

SARI : interactive & online time series analysis software

de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

• Fit linear, polynomial, sinusoidal, logarithmic, exponential and stepchange models.

• Test the statistical significance of the estimated offsets in the series against colored noise. • Fit time-variable models using a first-order extended Kalman filter (EKF) or an unscented Kalman filter (UKF). See Fig. 1.

• Remove specific periods of data or individual outliers one-by-one or automatically using a threshold. • Band-pass filter the series using the Vondrak smoother. See Fig. 1.

• Average, reduce and regularize the sampling rate. • Estimate the Lomb periodogram. See Fig. 2.

• Estimate the linear trend using the discontinuity-free MIDAS algorithm. • Estimate the wavelet transform. See Fig. 3.

• Automatically detect discontinuities (aka offsets, step-changes). • Estimate the power-law noise content with fixed/free spectral index. • Upload any unevenly sampled time series in different formats.

• Fit a non-parametric periodic waveform not having a sinusoidal shape.

• Estimate the histogram of the residuals and assess their stationarity using two different tests. • Compare and correct the series with a secondary series from a model (loading, post-seismic, etc.) or from a nearby station.

• Save the analysis results in a self-contained file and apply the same model to a different series. • Display equipment changes from a IGS-like station log, a GAMIT-like station.info file or a customized offset file (earthquake dates).

• And many more awesome things … or if they are not implemented you can suggest them to me!

Fig. 1 .ExamplesFig. 2 .Fig. 3 .

 123 Fig.1. Synthetic series with an annual cycle of varying amplitude superimposed on a sinusoid-like varying trend and noise (black dots) fitted with a UKF (red line) and the Vondrák smoother (blue line). Units are whatever you want they to be.

: interactive GNSS position time series analysis software.

  SARI: interactive & online time series analysis softwareAlvaro Santamaría-Gómez GET, Université de Toulouse, CNRS, IRD, UPS, Toulouse, France alvaro.santamaria@get.omp.eu

	What is it?	An online tool to analyze time series interactively
	How much for using it? Where is it? Who can use it? What can I do with it?	Nothing At https://alvarosg.shinyapps.io/sari Anyone having an Internet connection For instance, you can …	More details can be found in Santamaría-Gómez, A. (2019) SARIGPS Solut. 23: 52. DOI: 10.1007/s10291-019-0846-y

4èmes Rencontres Scientifiques et Techniques RESIF, Biarritz, 12-14 novembre 2019


Peter Kop

Fred Janssen

Paul Drijvers

Jan Van Driel

Graphing formulas to give meaning to algebraic formulas

Keywords: Algebra, giving meaning to algebraic formulas, graphing formulas

Many students have difficulties giving meaning to algebra. In this study, we investigated how graphing formulas by hand can help grade 11 students to give meaning to algebraic formulas, defined as the ability to "read through" an algebraic formula: i.e., to recognize its structure and to link it to graphical features. During five 90-minute lessons, 21 students worked on graphing tasks focusing on recognition and heuristic search. To assess the effects of this intervention, a graphing task, and a cardsorting task were administered to the students. In addition, six students were asked to think aloud during the graphing task. The results of the card-sorting task showed that 14 students used categories similar to the ones experts use, but had trouble in consistently categorizing all formulas. The thinkingaloud protocols showed that the students improved their recognition of basic functions and graph features, and their qualitative reasoning, which allowed them to give meaning to algebraic formulas.

Introduction

Many secondary school students have problems with algebraic formulas, which are very abstract for them [START_REF] Arcavi | The Teaching and Learning of Algebra: Ideas, Insights and Activities[END_REF][START_REF] Kieran | Research on the learning and teaching of algebra[END_REF][START_REF] Thompson | In the absence of meaning…[END_REF]. Students, also those in upper secondary school, have difficulties giving meaning to algebraic formulas: they lack symbol sense. Symbol sense has to do with the ability to read through a formula, to recognize its structure and its characteristics [START_REF] Arcavi | Symbol sense: Informal sense-making in formal mathematics[END_REF] and would allow students to give meaning to algebraic formulas. It is insufficiently known how to teach these aspects of symbol sense [START_REF] Arcavi | The Teaching and Learning of Algebra: Ideas, Insights and Activities[END_REF][START_REF] Hoch | Developing Katy's algebraic structure sense[END_REF]. In regular education, teaching for giving meaning often starts with realistic contexts and manipulation of formulas. In the current study, we used graphs to give meaning to formulas and used basic functions as building blocks in reasoning with and about algebraic formulas.

Theory Giving meaning to algebraic formulas

Acquiring meaning is synonymous with assimilating to a cognitive schema, which is a (hierarchal) network of concepts and procedures. Assimilation is the integration of new information into a schema, which results in an adjusted schema [START_REF] Thompson | In the absence of meaning…[END_REF]. To give meaning to algebraic formulas, different sources are suggested in the literature: for instance, via the problem context, via the algebraic structure of the expression, via multiple representations, and via linguistic activity, gestures, body language, metaphors, etc. [START_REF] Kieran | Research on the learning and teaching of algebra[END_REF]. [START_REF] Ernest | The meaning of mathematical expressions: Does philosophy shed any light on psychology?[END_REF] used a syntactical analysis to decompose algebraic expressions into meaningful sub-expressions (building blocks). [START_REF] Hoch | Developing Katy's algebraic structure sense[END_REF] used structure sense to describe the ability to recognize familiar structures by using compound terms as single entities. [START_REF] Janvier | Problems of representation in the teaching and learning mathematics[END_REF] and many others have shown that functions can be investigated by linking different representations. We chose to link algebraic formulas to graphs, which seem to be more accessible for students than formulas [START_REF] Leinhardt | Functions, graphs, and graphing: Tasks, learning, and teaching[END_REF], as they appeal more to the Gestalt aspect of a function and visualize the "story" that an algebraic formula tells. Therefore, graphs can be used to give more meaning to algebraic formulas [START_REF] Eisenberg | On understanding how students learn to visualize function transformations[END_REF][START_REF] Kieran | Research on the learning and teaching of algebra[END_REF][START_REF] Heid | How might computer algebra systems change the role of algebra in the school curriculum?[END_REF]. To link formulas to graphs in order to give meaning to the formulas, graphing tools such as graphic calculators can be used [START_REF] Heid | How might computer algebra systems change the role of algebra in the school curriculum?[END_REF]. Using these tools, it seems easy to make a graph from a formula. However, research has shown that one must know what aspects of graphs to look for [START_REF] Stylianou | The role of visual representations in advanced mathematical problem solving: An examination of expert-novice similarities and differences[END_REF], and that students establish the connection between formula and graph more effectively when they do graphing by hand than when they only perform computer graphing [START_REF] Goldenberg | Mathematics, metaphors, and human factors: Mathematical, technical, and pedagogical challenges in the educational use of graphical representation of functions[END_REF].

The aim of the current study was to improve students' abilities to give meaning to algebraic formulas by teaching them to graph formulas by hand. We defined giving meaning to algebraic formulas as the ability to "read through" an algebraic formula: that is, to recognize its structure and to link it to graphical features.

Teaching graphing formulas for giving meaning

In expertise research, it has been established that for effective and efficient problem solving one needs recognition, and heuristics when recognition falls short [START_REF] Chi | Categorization and representation of physics problems by experts and novices[END_REF]. Based on this idea, a two-dimensional framework to describe strategies for graphing formulas was formulated [START_REF] Kop | Identifying a framework for graphing formulas from expert strategies[END_REF]. In this framework, one dimension is about levels of recognition: from complete or function family recognition, to no recognition of the graph. The other dimension is about heuristic search, from strong heuristics, which provide information about large parts of the graph (e.g., qualitative reasoning about infinity behaviour and/or symmetry), to weak heuristics, which give only local information (e.g., calculating points). See Figure 1. For recognition, a repertoire of basic function families and knowledge of attributes to describe graphs are needed [START_REF] Eisenberg | On understanding how students learn to visualize function transformations[END_REF][START_REF] Slavit | An alternate route to the reification of function[END_REF]. [START_REF] Kop | Graphing formulas: Unraveling experts' recognition processes[END_REF] found, using a card-sorting task, that experts' repertoires of basic function families resembled the basic function families taught in secondary school, like exponential, logarithmic, and polynomial functions. Experts seem to have linked prototypes of these function families to a set of critical graph attributes. For instance, a prototypical logarithmic graph has a vertical asymptote, only positive x-values as a domain, and is concave down. They use their repertoire of basic function families as building blocks in working with formulas, for instance, when decomposing complex functions into simpler basic functions and when reasoning about characteristic graph features from formulas [START_REF] Kop | Identifying a framework for graphing formulas from expert strategies[END_REF][START_REF] Kop | Graphing formulas: Unraveling experts' recognition processes[END_REF].

Experts often start with a global perspective that enables high levels of recognition [START_REF] Even | Factors involved in linking representations of functions[END_REF].

With the global perspective the whole function and/or graph is in view, whereas with a local perspective only a part is in view. The global perspective is more powerful and gives a better understanding of the relation between formulas and graphs, but the pointwise approach is needed to monitor naïve and/or immature interpretations [START_REF] Even | Factors involved in linking representations of functions[END_REF][START_REF] Slavit | An alternate route to the reification of function[END_REF].

The question in the current study was whether we could teach these experts' strategies to students. This led to the following research question: How can graphing formulas based on recognition and heuristic search help pre-university students to develop a knowledge base to give meaning to algebraic formulas?

Method

The teaching experiment was performed in a grade 11 mathematics B class, a regular class of 21 students who were 16-and 17-years old. Mathematics B is a course that prepares students in the Netherlands for university studies in mathematics, science, and engineering. The intervention took place during five lessons of 90 minutes. The students were obliged to work in pairs or groups of three which required them to exchange ideas. Each day, the lesson started with a short plenary discussion (max. 10 minutes) with general feedback on the students' work, reflection on the tasks, and modelling of expert thinking processes aloud. During the rest of the lesson, the students worked on tasks, related to the levels of recognition of the two-dimensional framework (Figure 1). These tasks were formulated as exploratory whole tasks with help, concluded with a reflection question. After the lesson, each pair and group handed in their work for feedback.

To assess the effects of the intervention, we collected students' responses to a written graphing task in a pre-test, post-test, and retention-test (four months after the intervention). Each graph in these graphing tasks was reviewed as correct (score=1) or incorrect (score=0), and a total score was calculated for each student.

Written tests give only limited information about the students' thinking processes. Therefore, six students were asked to think aloud during the graphing task in the pre-test and post-test. These protocols were transcribed and analyzed based on the two-dimensional framework (Figure 1): that is, on the use of recognition (function families, formula decomposed into sub-formulas, and/or graph features), qualitative reasoning, and calculation (points and/or derivatives).

In addition, a card-sorting task was assigned in the post-test to gather information about the students' knowledge base of function families. The students were asked to categorize 40 formulas according to their rough graph, using as many categories as they wished. In addition, the students were asked to give a description and a prototypical formula for each of their categories. Cardsorting tasks are often used in eliciting structured knowledge [START_REF] Chi | Categorization and representation of physics problems by experts and novices[END_REF]. To analyze the cardsorting task, an expert categorization with 12 main categories, similar to the one found by [START_REF] Kop | Graphing formulas: Unraveling experts' recognition processes[END_REF], was constructed and used as the criterion categorization. For each categorization, the student's categories were compared with those of the criterion categorization and it was established which formulas were not consistently categorized. Each corresponding category was graded with a score of two points and an extra point if a distinction between increasing and decreasing had been made. Then the number of correct formulas (consistent with the category) was graded. This method of analysis has been used by others [START_REF] Ruiz-Primo | Problems and issues in the use of concept maps in science assessment[END_REF]; the total score should be an indication of the correctness of a student's categorization.

Results

For the card-sorting task, we found that 14 out of the 21 students used categories almost similar to those in the expert categorization, although different names were used for the categories, like "having a horizontal and vertical asymptote" (linear broken functions), or "only a vertical asymptote" (logarithmic function). Many students made mistakes in categorizing distractors like and often no distinction was made between increasing and decreasing log-functions.

To illustrate the results, we report about two students: K (a high-achieving student) and M (an above-average-achieving student). Student K's categorization is shown in the Appendix. S/he distinguished increasing and decreasing exponential, root, and polynomial functions. S/he categorized only a few formulas inconsistently, but s/he did not make a distinction between logarithmic and root functions, which resulted in a total score of only 45 points. Student M's categories also resembled those of the criterion categorization. When categorizing the formulas, student M used predominantly zeroes and max/min with polynomial functions, like in , asymptotes (horizontal and vertical for broken functions, horizontal for exponential functions and vertical for logarithmic functions), and edge points for root-functions. Although, s/he had more inconsistently categorized formulas, her/his categorization had a total score of 44 points.

The thinking-aloud protocols on the graphing tasks showed that, in the pre-test, the students had trouble recognizing (basic) functions, and only high-achieving students were successful now and then because of their reasoning abilities. In the post-test, these students had improved their recognition of basic function families with their characteristics and their qualitative reasoning.

In the pre-test, K scored 9 correct graphs out of 14. As a high-achieving student, s/he was able to compensate for a lack of recognition through qualitative reasoning. However, s/he often had to calculate many points of the graph. We give two citations to illustrate K's strategies in the pre-test. K did not recognize the global shape of a logarithmic function and used calculations and reasoning about the inverse function to sketch the graph of ln( 3) yx  correctly: "I do not know the ln-graph anymore. When , then …… When , then , so . At x-as the x-axis is intersected. When x is increasing then y increases, so the graph increases. When x is negative … (thinking). Because something to the power of e (e …. ) does not give negative y-values. So, 3 x  cannot be negative; the graph only exists from 3 x  , larger than 3. So, at 3 x  a tangent (asymptote?) and outcomes smaller when x is in the neighbourhood of 3".

At first K showed a correct global shape by gesturing, however her calculating of individual points resulted in an incorrect sketch of :

"…root-functions go like this (makes a correct gesture); the fact that it is 6-2x means that the function only exists for negative values of x, to the point where x is 3; for x larger than 3 the argument is negative; when x=3 than y=0; when x=0, y= 6 , which is about 2.5; when x is more negative, the argument becomes larger; …, 10 3.3  , 12 3.5  , differences become larger (and sketches a graph concave up)"

In the post-test, K's recognition of basic functions had improved and s/he still used her/his abilities of qualitative reasoning, resulting in a score of 13 out of 14. Again, two citations give an illustration.

K sketching correctly: "ln(x) graph goes like this; , so 4 to the right."

K used qualitative reasoning to compose two graphs while sketching correctly:

"2x goes like this; goes like this (sketch); here it is 0; here negative, here 0, and after this it is steeper".

In the pre-test student, M had many problems with graphing formulas. S/he did not know basic function family characteristics, and did not recognize graph features like zeroes and turning points. This resulted in a score of only 2 out of 14. Two citations illustrate her/his strategies.

M recognized the parabola-shape but did not know how to proceed while sketching (S/he stopped talking for a while; after a couple of minutes) I do not know how to proceed."

M recognized a translation but not the shape of the ln(x)-graph; s/he tried to construct the graph of ln( 3) yx  via the inverse function (not finished):

"…graph of ln( )

x that is translated 3 to the right (s/he did not use this but writes

). This is an asymptote; x cannot be 3; ….; when (drew point (4,0) and stopped)".

In the post-test, M's repertoire of basic function families and her/his qualitative reasoning had improved, resulting in a score of 10 out of 14. Two citations illustrate this. In the post-test M recognized more graph features like zeroes and turning points; for instance, while sketching correctly: "…goes downwards; zeroes at 0, 2, 5. At 1 it is negative".

M in the post-test, sketching correctly:

"…asymptotes at 2 and -2; zero at ; no, no zeroes, because 2 x cannot be negative; when x is smaller than 2, then it is positive here, and negative here, so it is negative; when x is a bit larger than 2, positive here, positive here, so positive; the same for -2". Not only had these two students improved their scores in the post-test, but the results of all students showed a significant improvement in scores, with a large effect-size, from a mean score of 3.1 (SD = 2.6) in the pre-test to a mean score of 8.6 (SD = 2.8) in the post-test (t(20) = 10.40, p < .001, d = 2.27). As expected, the scores in the retention-test dropped: a mean score of 6.3 (SD = 3.4). This result is significant when compared with the pre-test scores: t(14) = 3.15, p = .007, d = 0.81.

Conclusion and discussion

The aim of the current study was to enable students to give meaning to algebraic formulas in terms of a graph. A knowledge of basic function families with their characteristics is needed for this. In the intervention, students learned to graph formulas through recognition and qualitative reasoning. The results of the thinking-aloud protocols showed that in the pre-test students were either unable to link formulas to graphs or needed to use a lot of time-consuming reasoning and error-prone calculations. Therefore, without explicit teaching, students might not link formulas to graphs. The post-test results of the graphing task showed that the students had improved their recognition of basic function families and, in general, their reasoning about formulas. In terms of the two-dimensional framework: in the pre-test the students mostly used recognition level E, whereas in the post-test they often used higher levels of recognition (recognition of function families, decomposition of the formula, and recognition of graph features).

The categorizations gave an impression of the knowledge base and thinking processes of the students. The students' categorizations showed that they used more or less the same categories as experts, but still had problems with categorizing less familiar functions, like Student M used zeroes, asymptotes (horizontal and/or vertical), and edge points as main strategies when categorizing. In the post-test graphing task, s/he often used these same strategies. Student K's categorization was very detailed, although s/he combined root-and log-functions. However, in the post-test graphing task s/he showed that s/he knew the differences well (see sketching and in the results section). In the post-test graphing task, K used her/his repertoire of function families often. Both students' categorizations seemed to allow them to recognize function families and use these function families as building blocks in reasoning with and about formulas: these function families are objects that they can use to give meaning to algebraic formulas. These findings suggest that teaching graphing of formulas based on recognition and heuristic search might enable students to develop a repertoire of basic function families, to identify the structure of formulas, to (qualitatively) reason with and about algebraic formulas, and to link formulas to graphical features. This could allow students to "read through" an algebraic formula, that is, to give meaning to algebraic formulas.

  some time) The larger x is, the larger y, so it increases. It is a parabola.

Heuristic search (strong  weak) Levels of Recognition (low  high)

	A A1. Graph is instantly
	recognized as a whole
	B B1. Recognition of family	B2. Search for 'parameters' of	B3. Investigate the family
	(with characteristics)	the graph	characteristics, for
			instance via zeroes
	C C1. Split formula in sub-	C2. Compose the graphs by	C3. Compose the graphs
	formulas	qualitative reasoning	by making a table
	D D1. Graph unknown;
	Characteristic aspect of
	graph is recognized.
	E E1. Graph unknown;	E2. Qualitative reasoning	E3. Algebraic	E4. Strategic	E5. Calculate
	strategic exploration of	about domain, or vertical	manipulation	search, for	strategically
	algebraic formula	asymptote, or symmetry, or		zeroes or	chosen point(s)
		infinity behaviour, …		extreme values
	F F1. No recognition at all	F2. Standard repertoire of	F3. Make random table
		research

Figure 1: Two-dimensional framework for strategies to graph formulas


Marco Dinarelli
email: marco.dinarelli@univ-grenoble-alpes.fr

Loïc Grobol
email: loic.grobol@gmail.com

Hybrid Neural Models For Sequence Modelling: The Best Of Three Worlds

Keywords: Neural Networks, sequence modelling, MEDIA, WSJ, TIGER

We propose a neural architecture with the main characteristics of the most successful neural models of the last years : bidirectional RNNs, encoder-decoder, and the Transformer model. Evaluation on three sequence labelling tasks yields results that are close to the state-of-theart for all tasks and better than it for some of them, showing the pertinence of this hybrid architecture for this kind of tasks.

Introduction

Sequence labelling is an important NLP task, many NLP problems can be modeled indeed as sequence labelling. The most traditional labelling problems are Part-Of-Speech (POS) tagging, syntactic chunking, Named Entity Recognition (NER) [START_REF] Collobert | Natural language processing (almost) from scratch[END_REF], and also Spoken Language Understanding (SLU) in the context of human-machine dialog systems [START_REF] Mori | Spoken language understanding : A survey[END_REF].

Moreover there are other NLP problems that can be decomposed into several processing steps, the first of which can be often modeled as a sequence labelling problem.

We can place among such problems the syntactic constituency parsing, which can be decomposed into POS tagging and constituents analysis [START_REF] Collins | Three generative, lexicalised models for statistical parsing[END_REF] ; the coreference chain resolution [START_REF] Soon | A Machine Learning Approach to Coreference Resolution of Noun Phrases[END_REF][START_REF] Ng | Improving Machine Learning Approcahes to Corefrence Resolution[END_REF], composed of mention recognition from raw text and coreferent mention detection ; and also the structured named entity recognition [START_REF] Grouin | Coreference resolution in clinical reports. the limsi participation in the i2b2/va 2011 challenge[END_REF]Dinarelli & Rosset, 2012a,b).

More widely, machine translation and syntactic constituency parsing can be modeled as endto-end sequence prediction problems [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF][START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF][START_REF] Vaswani | Attention is All you Need[END_REF][START_REF] Curran Associates | Grammar As a Foreign Language[END_REF], as well as a wide class of language understanding tasks [START_REF] Devlin | BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF]. It would be possible thus, at least in principle, to design a model for sequence prediction in a multi-task learning framework, in order to deal with most of the NLP tasks, which shows in turn the interest in the research for sequence modelling problems.

The NLP problems that can be decomposed into several processing steps can be solved also with a single model. For example the syntactic constituency parsing can be performed endto-end with a model performing POS tagging and constituent analysis at the same time, e.g. (Rush et al., 2012). However, even in such cases, pre-training representations with neural networks and fine-tuning them afterwards on simpler tasks, like sequence labelling, can still improve the prediction performance [START_REF] Peters | Deep Contextualized Word Representations[END_REF]. We can also consider this training procedure similar to embedding pre-training, which effectiveness has already beed proved [START_REF] Lample | Neural Architectures for Named Entity Recognition[END_REF][START_REF] Ma | End-to-end sequence labeling via bi-directional lstm-cnns-crf[END_REF].

In this paper we focus on neural architectures for sequence modelling in the sense this is meant in problems such as POS tagging, morpho-syntactic analysis, and semantic chunking and tagging. The latter is performed for SLU tasks in the context of human-machine dialog systems [START_REF] Mori | Spoken language understanding : A survey[END_REF].

We inspired our work from [START_REF] Chen | The Best of Both Worlds : Combining Recent Advances in Neural Machine Translation[END_REF], from which we also inspired the title of our paper, and which proposes neural hybrid models combining the Encoder-Decoder (or Sequence-to-Sequence) and Transformer architectures. We propose an architecture with the main characteristics of the most successful neural models of the last few years : the bidirectional RNNs [START_REF] Lample | Neural Architectures for Named Entity Recognition[END_REF][START_REF] Ma | End-to-end sequence labeling via bi-directional lstm-cnns-crf[END_REF], the Sequence-to-Sequence model [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF][START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF], and the Transformer [START_REF] Vaswani | Attention is All you Need[END_REF].

We evaluate our architecture on three traditional tasks of sequence labelling : Spoken Language Understanding (SLU) in French (MEDIA corpus) [START_REF] Bonneau-Maynard | Results of the french evalda-media evaluation campaign for literal understanding[END_REF], POS tagging in English (WSJ corpus) [START_REF] Marcus | Building a large annotated corpus of english : The penn treebank[END_REF], and morpho-syntactic tagging in German (TIGER corpus) [START_REF] Brants | TIGER : Linguistic interpretation of a german corpus[END_REF]. The results outperform often the state-ofthe-art, and allow in any case to conclude that our architecture finds its place among neural architectures.

Neural Architectures

The proposed neural models are inspired from the LSTM+CRF architectures [START_REF] Lample | Neural Architectures for Named Entity Recognition[END_REF][START_REF] Ma | End-to-end sequence labeling via bi-directional lstm-cnns-crf[END_REF] in the way input encoding (tokens, characters, etc.) is performed. They are inspired from the Sequence-to-Sequence architecture for the overall architecture, and from the models proposed in our previous work (Dinarelli & Tellier, 2016a,b;[START_REF] Dinarelli | Label-dependency coding in Simple Recurrent Networks for Spoken Language Understanding[END_REF][START_REF] Dupont | Label-dependencies aware recurrent neural networks[END_REF][START_REF] Dinarelli | Seq2biseq : Bidirectional output-wise recurrent neural networks for sequence modelling[END_REF] for making predictions based on a bidirectional context on the output side (labels). Afterwards we added to this architecture some of the characteristics of the Transformer model. This work constitutes a step ahead with respect to [START_REF] Dinarelli | Seq2biseq : Bidirectional output-wise recurrent neural networks for sequence modelling[END_REF]). While we have not run full experiments to show the improvements with respect to our previous work, and results are equivalent in some cases, some outcomes suggest that the model proposed here is more effective. In any case the model described here is more general, as the architecture has been slightly modified so that it can be applied also to tasks like machine translation and syntactic parsing, though we do not evaluate on these tasks in this work. Moreover in other cases results are state-of-the-art, though they are obtained on data (TIGER) on which we did not evaluate previously.

Encoder

The encoder of our network is made of a bidirectional GRU [START_REF] Cho | Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation[END_REF], which takes as input a sequence S lex representing input tokens as the concatenation of word embeddings (E w (w i )) and representations computed from their characters h c (w i ). The latter is computed by an ad-hoc recurrent layer, in a fashion similar to those used in the literature [START_REF] Ma | End-to-end sequence labeling via bi-directional lstm-cnns-crf[END_REF], with the difference that we use a GRU recurrent layer instead of a convolution layer.

The character-level representation of a word w is computed like in [START_REF] Dinarelli | Seq2biseq : Bidirectional output-wise recurrent neural networks for sequence modelling[END_REF] :

S c (w) = (E c (c w,1 ), . . . , E c (c w,n )) (h c (c w,1 ), . . . , h c (c w,n )) = GRU c (S c (w), h 0 c ) h c (w) = FFNN(Sum(h c (c w,1 ), . . . , h c (c w,n ))) (1)
where E c is the matrix of character embeddings, c w,i is the i-th character of the word w, S c (w) is the sequence of character embeddings for the word w, GRU c is the GRU layer for characters, h c (c w,i ) is the hidden state associated to the i-th character of the word w. GRU c , like GRU w , is a bidirectional GRU layer.

The full representation of the i-th word w i in a sequence is computed as :

S lex = ([E w (w 1 ), h c (w 1 )], . . . , [E w (w N ), h c (w N )]) (h w1 , . . . , h w N ) = GRU w (S lex , h 0 w ) (2)
Since GRU w scans the sequence forward and backward, h wi depends on the word w i and its context. When additional features are available at token level, they are embedded in the same way as words and concatenated to word embeddings in the sequence S lex . We use a ← ---GRU e backward layer for encoding the right context, and a ---→ GRU e forward layer for the left context with respect to a given labelling position i in the input sequence. These layers take as input the lexical level representation computed by the encoder and the label embeddings E e (e i ), which makes them similar to the decoder used in the original Sequenceto-Sequence architecture for machine translation [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF]; [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF]. An evolution with respect to such architecture is the use of two decoders, one for the left label-level context and one for the right label-level context. This idea of using both left and right context at the output item's level has attracted some interest in the last few years, and has been applied in different variants to machine translation [START_REF] Xia | Deliberation networks : Sequence generation beyond one-pass decoding[END_REF]Zhang et al., 2018a).

Decoders

The right label-level context is computed by the backward ← ---GRU e decoder as follows :

← - h ei = ← --- GRU e ([h wi , E e (e i+1 )], ← --- h ei+1 ) (3)
The left label-level context -→ h ei is computed in a similar way by the forward ---→ GRU e decoder.

Output Layer

In order to allow the model to make global decisions when predicting the current label, we add an output layer on top of the backward decoder, made of an affine transformation followed by a log-softmax activation. This computes log-probabilities of backward predictions :

log-P( ← - e i ) = log-softmax(W bw [h wi , ← - h ei ] + b bw ) ← - e i = argmax(log-P( ← - e i )) (4)
The forward decoder predicts labels using as input the label-level context representations -→ h ei and ←h ei , as well as the lexical level information h wi computed by the encoder :

log-P( - → e i ) = log-softmax(W o [ -→ h ei , h wi , ← - h ei ] + b o ) - → e i = argmax(log-P( - → e i )) (5)
In order to strengthen the global character of the decision, the final output of the model is computed as the arithmetic mean (in the log space, which gives a geometric mean of probabilities) of the forward and backward log-probabilities : 1 2 (log-P( -→ e i ) + log-P( ←e i )). This teaches the model to compute real good predictions also in the backward phase, instead of basing decisions on raw heuristics, or even on the forward (left) context only. Indeed, the backward decoder has only a partial view of the output level information, while the forward decoder uses both left and right context.

Learning

All models described in this paper are learned minimizing the minus log-likelihood LL of the training data. More precisely :

-LL(Θ|D) = - |D| d=1 N d i=1 1 2 (log-P( - → e i ) + log-P( ← - e i )) + λ 2 |Θ| 2 ) (6)
The first sum scans the data D of size |D|, while the second sum is over training examples S d , of size N d .

Since data used in this work for evaluation are relatively small, while our models are relatively complex, we add a Gaussian regularization term L 2 with a coefficient λ.

2.5

The best of three worlds

The neural architecture described so far uses the same ideas introduced in (Dinarelli & Tellier, 2016a;[START_REF] Dinarelli | Label-dependency coding in Simple Recurrent Networks for Spoken Language Understanding[END_REF][START_REF] Dupont | Label-dependencies aware recurrent neural networks[END_REF][START_REF] Dinarelli | Seq2biseq : Bidirectional output-wise recurrent neural networks for sequence modelling[END_REF] for predicting labels using both representations of left (forward) and right (backward) contexts, and for both input-level information (words, characters, etc.) and labels. Our model integrates also the characteristics of bidirectional RNNs and of Encoder-Decoder architectures. Moreover we use two decoders, instead of one like in the original architecture.

Starting from this neural architecture we added also some of the characteristics of the Transformer model.

We note that the paper from which we inspired our work [START_REF] Chen | The Best of Both Worlds : Combining Recent Advances in Neural Machine Translation[END_REF] investigated the combination of two Encoder-Decoder architectures, one based on recurrent layers [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF][START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF], while the other is based on the Transformer model, which is not a recurrent network. This work shows that such combination is possible and effective compared to the two combined architectures.

In this work we combined three different architectures with the goal of putting together their strengths, and trying to overcome some limitations.

First of all, in the tasks of sequence labelling used for evaluation in this work, there is a one-to-one correspondence between input and output items (words and labels, respectively). Our preliminary experiments on mention detection for coreference resolution using a Transformer, as well as some experiments in the literature [START_REF] Guo | Star-transformer[END_REF], suggest that ignoring such correspondence leads to a remarkable performance drop. 1 For this reason the encoder and the decoders used in our architecture exploit the one-to-one correspondence of sequence labelling tasks. We are aware though this information could not be used in real sequence-to-sequence problems such like machine translation or syntactic parsing.

Second, we chose an Encoder-Decoder overall architecture since this proved to be the most effective for sequence-to-sequence problems (including sequence labelling). Moreover, we showed that keeping a label-level context into account, in some sequence labelling problems, using a decoder is more effective than doing the same with a CRF neural layer [START_REF] Dinarelli | Label-dependency coding in Simple Recurrent Networks for Spoken Language Understanding[END_REF], we will show the same also in this work. As we already mentioned, in our architecture we use two decoders. This solution allows for an even more effective modelling of a label-level context. This solution is not possible, at least in principle, with the original Transformer model.

Third, we decided to integrate some of the characteristics of the Transformer in order to overcome limitations of RNNs concerning the learning phase. As discussed in [START_REF] Vaswani | Attention is All you Need[END_REF], RNNs are difficult to train because of the length of paths the learning signal has to back-propagate through. The length of such paths is the consequence of the recurrent nature of the networks.

The Transformer model [START_REF] Vaswani | Attention is All you Need[END_REF] proposed an alternative, non-recurrent Encoder-Decoder architecture based on a multi-head attention mechanism. Recent works [START_REF] Dehghani | Universal transformers[END_REF][START_REF] Dai | Transformer-XL : Attentive Language Models Beyond a Fixed-Length Context[END_REF] suggest however that using some kind of recurrent connections Transformer 's performance can be improved. Another interesting characteristic of the Transformer is the use of residual connections, or skip-connections [START_REF] Bengio | A neural probabilistic language model[END_REF]. This characteristic allows for alleviating the vanishing gradient problem of deep networks. Indeed, the use of gates in recurrent networks reduces the amplitude of this problem, however the literature and our own experiments suggest that for long sequences, gates are not sufficient for an effective learning.

The attention mechanism does not suffer from this problem. Each element of the input sequence is related to a small, fixed number of layers in the attention mechanism. The backpropagation path of the learning signal is thus much shorter than that of RNNs. Moreover the use of skip-connections contributes in keeping a higher learning signal amplitude, allowing to skip a whole layer, over which the learning signal would be weakened (because of gradient multiplications).

We can interpret each block of a Transformer architecture, both encoder and decoder, as the combination of a sub-module which "encodes" contextual features, we call this submodule ctx-encoder, and a feed-forward sub-module which "maps" contextual features into a deeper feature space, we call the second sub-module deep-encoder. The output of the two sub-modules is added to their respective input (skip-connection) and normalized with a layer normalization. This interpretation is shown in the figure 1 in the middle. Skip-connections are highlighted in red.

The Transformer proposed in [START_REF] Vaswani | Attention is All you Need[END_REF] instantiate the generic architecture described above using as ctx-encoder the multi-head attention mechanism, while it uses a two-layer feed-forward network as deep-encoder. A single block of this architecture is shown in the figure 1 on the left.

We modified our neural architecture for instantiating the generic architecture. In our solution we use a bidirectional GRU layer as ctx-encoder, while we use exactly the same sub-module

w 1 w 2 w 3 encoder encoder encoder ← ----- decoder ← ----- decoder ← ----- decoder -----→ decoder -----→ decoder -----→ decoder - → e 1 - → e 2 - → e 3 ← - e 1 ← - e 2 ← - e 3
Figure 2 -Overall architecture of our model, this is the same as the model presented in our previous work [START_REF] Dinarelli | Seq2biseq : Bidirectional output-wise recurrent neural networks for sequence modelling[END_REF] as the Transformer as deep-encoder. A block of our architecture is shown in the figure 1 on the right.

The characteristics of the Transformer model we integrated in our architecture are thus the skip-connections and layer normalization (Add&Norm in the figure 1), as well as the twolayer feed-forward sub-module to re-encode contextual features once the skip-connection has been applied (FFNN in the figure 1). Following the computation chain suggested by [START_REF] Chen | The Best of Both Worlds : Combining Recent Advances in Neural Machine Translation[END_REF], the output of our encoder is computed as follows :

Ŝlex i = Norm(S lex i ) ĥwi = GRU w ( Ŝlex i , h wi-1 ) h wi = FFNN(Norm(Dropout( ĥwi ) + Ŝlex i )) (7)
where we use Norm for the layer normalization, and Dropout for a dropout regularization [START_REF] Srivastava | Dropout : A simple way to prevent neural networks from overfitting[END_REF]. The other GRU layers of our architecture, introduced in previous sections, are modified in a similar way.

In this work we don't use any attention mechanism like in the Transformer, context is granted by the use of GRU layers (cf. section 2.1). Our choice is motivated by the work described in [START_REF] Levy | Long short-term memory as a dynamically computed element-wise weighted sum[END_REF], where it is shown that RNNs, such like GRU, can encode long contexts as a weighted mean of the inputs, in a way that can be considered similar to an attention mechanism. Moreover, using GRU layers avoid the use of positional encoding, the sequential structure of an input sentence is encoded implicitly by the RNNs. 2

The overall architecture of our model is the same as our previous work [START_REF] Dinarelli | Seq2biseq : Bidirectional output-wise recurrent neural networks for sequence modelling[END_REF] (cf figure 2), the difference is in the computation of layer's output described in previous sections.

2. We don't exclude however that using positional encoding would improve our architecture

Evaluation

We evaluate our models on three tasks of sequence labelling :

The French corpus MEDIA [START_REF] Bonneau-Maynard | Results of the french evalda-media evaluation campaign for literal understanding[END_REF] was created for evaluating spoken dialog systems in the context of hotel information and reservation in France. Data have been annotated following a rich concept ontology. Simple semantic components can be combined to create more precise and complex semantic units. 3 Concepts span in general more than one word, the goal is thus to tag words and then reconstruct semantic units from the tags. Statistics of the corpus MEDIA are shown in figure 1.

This task is modelled as sequence labelling splitting semantic units over multiple words using the BIO formalism [START_REF] Ramshaw | Text chunking using transformation-based learning[END_REF]. We already evaluated on this task in the past, using a large variety of probabilistic models (Dinarelli et al., 2009a,b;[START_REF] Quarteroni | What's in an ontology for spoken language understanding[END_REF][START_REF] Hahn | Comparing stochastic approaches to spoken language understanding in multiple languages[END_REF][START_REF] Dinarelli | Spoken Language Understanding : from Spoken Utterances to Semantic Structures[END_REF]Dinarelli & Rosset, 2011;Dinarelli et al., 2011).

In the MEDIA corpus some word classes (clusters) are provided with the annotation. Such classes allow for a better generalization on words belonging to categories for which lists can be easily retrieved, or constructed, in the context of a real human-machine dialog system.

Examples of such classes are the name of cities in France (class CITY ), the hotel brands (class HOTEL), currency amounts (AMOUNTS ), etc. We used these classes in some of our experiments, this is indicated with FEAT in the tables of the following section.

The English Penn TreeBank corpus [START_REF] Marcus | Building a large annotated corpus of english : The penn treebank[END_REF], named WSJ henceforth, is one of the most used corpus for evaluating models for sequence labelling. The task consists in assigning each word its POS tag. For our experiments we use the most common data split : sections 0-18 are used as training data, sections 19-21 as development data, and sections 22-24 are used for testing. Statistics of this corpus are given in the table 2.

The German corpusTIGER [START_REF] Brants | TIGER : Linguistic interpretation of a german corpus[END_REF] is annotated with rich morpho-syntactic information. These include POS tags like in the WSJ corpus, but also gender, number, cases, and other inflection information, as well as conjugation information for verbs. The combination of all these pieces of information constitutes the output labels. This task is similar to the POS tagging from a sequence labelling point of view, however it is made more difficult on one side by the language (German has more complex syntactic structures than English), on the other side by the relatively large number of labels to be predicted (694 in total, while there are 138 labels in MEDIA and 45 in the WSJ corpus). We use the same data split used in [START_REF] Lavergne | Learning the structure of variable-order crfs : a finitestate perspective[END_REF]. Statistics of this corpus are given in the table 3.

As we can see from tables, the size of dictionaries, for both words and labels, is relatively large in the WSJ and TIGER corpora.

We note that the Out-Of-Vocabulary (OOV) rate in the MEDIA corpus is quite small. Indeed there are only two OOV words in both Dev and Test data, moreover they are function words.

There is basically no unknown word in this task. In contrast the OOV rate in both the WSJ and the TIGER corpora is quite high : roughly one word out of five in the first and one out of three in the second.

Settings

We used the corpus MEDIA for fine-tuning the hyper-parameters of our models in [START_REF] Dinarelli | Seq2biseq : Bidirectional output-wise recurrent neural networks for sequence modelling[END_REF]. MEDIA is the smallest corpus among the three used, and allows thus a faster optimization. The settings used in this work are the same as those used in [START_REF] Dinarelli | Label-dependency coding in Simple Recurrent Networks for Spoken Language Understanding[END_REF][START_REF] Dinarelli | Seq2biseq : Bidirectional output-wise recurrent neural networks for sequence modelling[END_REF], where the same data were used for evaluation (except for the TIGER corpus). The settings for the WSJ corpus are the same, except for the word embeddings (300 dimensions) and the learning rate (2,5 × 10 -4 ). We use the same settings for WSJ and TIGER. Since the latter corpus is the largest, we ran few experiments to choose the correct hidden layer size (300).

Like we already explained in [START_REF] Dinarelli | Seq2biseq : Bidirectional output-wise recurrent neural networks for sequence modelling[END_REF], at first our experiments were not fitting in the memory of our GPUs. In order to solve this problem we found two solutions.

In the first we organize the training data as a single stream of items. This is then split into chunks of fixed size, shifting each chunk by one token with respect to the previous (thus consecutive chunks overlap). The second solution is more intuitive and consists in clustering sentences of the same length. This way, very long (rare) sentences are assigned to very small clusters, thus fitting in memory, and small and average clusters form bigger clusters, which still fit in memory. In our experiments we found out that the first solution works far better on the MEDIA corpus. On the WSJ and TIGER corpora the two solutions are equivalent, we thus prefer the second for these two tasks, which is more intuitive and is expected to work better on bigger corpora.

Our neural network system is coded with pytorch 4 , experiments are run under Linux Debian [START_REF] Dinarelli | Label-dependency coding in Simple Recurrent Networks for Spoken Language Understanding[END_REF] -87.36 9.8 Seq2Biseq [START_REF] Dinarelli | Seq2biseq : Bidirectional output-wise recurrent neural networks for sequence modelling[END_REF] 89.57 87.50 10.26 Seq2Biseq 2-opt [START_REF] Dinarelli | Seq2biseq : Bidirectional output-wise recurrent neural networks for sequence modelling[END_REF] 89

Results

Results for the MEDIA task are means over 10 experiments, the parameters are initialized at random 5 for each experiment. On this task we evaluate our models in terms of accuracy and, since the goal is to re-construct semantic units spanning over multiple words, also with F1 measure and Concept Error Rate (CER). CER is computed by aligning the model prediction with the gold annotation with the Levenshtein distance algorithm, then the sum of insertions, deletions and substitutions is divided by the length of the gold annotation. On the other two tasks, since there is no label splitting over multiple words, we evaluate the models only in terms of accuracy.

Since the model presented in this paper is an evolution of the Seq2Biseq model proposed in [START_REF] Dinarelli | Seq2biseq : Bidirectional output-wise recurrent neural networks for sequence modelling[END_REF], the model proposed here is named Seq2Biseq-Transformer, as it adds characteristics of the Transformer on top of the Seq2Biseq model.

Results on Dev and Test data of the MEDIA task are shown in table 4. For comparison we show also results presented in our previous works [START_REF] Dinarelli | Label-dependency coding in Simple Recurrent Networks for Spoken Language Understanding[END_REF][START_REF] Dinarelli | Seq2biseq : Bidirectional output-wise recurrent neural networks for sequence modelling[END_REF], which are state-of-the-art on this task.

As described in [START_REF] Dinarelli | Seq2biseq : Bidirectional output-wise recurrent neural networks for sequence modelling[END_REF], the Seq2Biseq model gives better results when trained with 2 optimizers (see [START_REF] Dinarelli | Seq2biseq : Bidirectional output-wise recurrent neural networks for sequence modelling[END_REF] for details), one for each decoder. The Seq2Biseq-Transformer model is thus always trained with 2 optimizers.

As we can see in [START_REF] Dinarelli | Label-dependency coding in Simple Recurrent Networks for Spoken Language Understanding[END_REF] 96.90 96.91 LSTM+CRF [START_REF] Ma | End-to-end sequence labeling via bi-directional lstm-cnns-crf[END_REF] -97.13 Seq2Biseq [START_REF] Dinarelli | Seq2biseq : Bidirectional output-wise recurrent neural networks for sequence modelling[END_REF] 97.13 97.20 Seq2Biseq 2-opt [START_REF] Dinarelli | Seq2biseq : Bidirectional output-wise recurrent neural networks for sequence modelling[END_REF] 97.33 97.35 Seq2Biseq-Transformer 2-opt 97.33 97.36 Models using pre-trained embeddings 8 Transformer+Char [START_REF] Guo | Star-transformer[END_REF] -97.14 LSTM+CRF + Glove [START_REF] Ma | End-to-end sequence labeling via bi-directional lstm-cnns-crf[END_REF] 97.46 97.55 LSTM+LD-RNN + Glove [START_REF] Zhang | Learning tag dependencies for sequence tagging[END_REF] -97.59 Star-Transformer+Char [START_REF] Guo | Star-transformer[END_REF] -97.64 Keeping both these results and those on the MEDIA task into account, we could conclude that the Seq2Biseq-Transformer model does not really improve our previous model. However we would like to point out that the main goal of this work was to integrate characteristics of the Transformer model, and to come up with a generic architecture, which can be applied not only to traditional sequence labelling tasks, but also to tasks such like machine translation and syntactic parsing. Indeed the Seq2Biseq-Transformer architecture has been modified, with respect to our previous model, in order to perform all these tasks. We consider that obtaining similar results (which are still state-of-the-art) with a more versatile model is still a positive result. Additionally, observing the training procedure of the Seq2Biseq-Transformer model, we found out that while final results are similar, the loss values on both training and Dev data are better than those obtained with the Seq2Biseq model. This could actually mean the model is over-fitting, and we speculate that instead of using the same settings as in [START_REF] Dinarelli | Seq2biseq : Bidirectional output-wise recurrent neural networks for sequence modelling[END_REF], since the Seq2Biseq-Transformer is more complex (at least in terms of parameters, and see below), a fine-tuning of parameters could lead to better performances.

In table 5 we can see that, whether we use one or two decoders for training, our models improve the LSTM+CRF model [START_REF] Ma | End-to-end sequence labeling via bi-directional lstm-cnns-crf[END_REF] without using the embeddings pretrained with GloVe [START_REF] Pennington | Glove : Global vectors for word representation[END_REF]. For a matter of comparison, we show also the best results on this task, obtained with pre-trained embeddings. Though our results are lower than state-of-the-art in terms of accuracy, they are not too far, proving that our hybrid neural architecture is quite effective. In particular, we compare our results with those reported in [START_REF] Guo | Star-transformer[END_REF] obtained with the Transformer and Star-Transformer models. 9 Whether the Transformer model has been trained with or without pre-trained embeddings, our models outperform it by a significant margin. As we mentioned in section 2.5, we have experienced a drop of 3 points in the F1 measure using a Transformer for mention detection in preliminary experiments (not reported in this work). We hypothesize that the one-to-one correspondence between input and output units in sequence labelling tasks is a very useful information, and the lack of performance of the Transformer on sequence labelling tasks can be explained by the fact that this model doesn't use it.

In table 6 we show results obtained on the morpho-syntactic tagging task on the German corpus TIGER. To the best of our knowledge, the best results in the literature for this task are reported in [START_REF] Lavergne | Learning the structure of variable-order crfs : a finitestate perspective[END_REF], obtained using a Variable Order CRF (VO-CRF).

We can see thus that our model improves the state-of-the-art on this task. It is important to note however that the TIGER corpus has been less used for the evaluation of neural models for sequence labelling. Results reported in [START_REF] Lavergne | Learning the structure of variable-order crfs : a finitestate perspective[END_REF] are indeed obtained with a discrete-representation based, though quite sophisticated, CRF model. In table 6 we report also, between parenthesis, POS tagging results. These are obtained from morpho-syntactic tagging results extracting the POS tag from the complete label. That is no model training has been performed for the POS tagging task alone.

Conclusions

In this paper we proposed a neural model for sequence labelling integrating characteristics of the most successful models of the last couple of years : bidirectional RNNs, Sequenceto-Sequence model, and the Transfomer model. An evaluation on three tasks of sequence labelling shows that the proposed model is very effective for this kind of tasks. Indeed it achieves results comparable to the state-of-the-art, but keeping a generic architecture that allows to use the same model on other kind of tasks like machine translation and syntactic parsing. Moreover in some cases our model outperforms state-of-the-art models, proving that our hybrid architecture has its place among current neural architectures.

9. In [START_REF] Guo | Star-transformer[END_REF] it is actually not clear if the Transformer and Star-Transformer models have been trained with or without pre-trained embeddings. Authors report "Advanced Techniques except widely-used pre-trained embeddings" in the caption of table 5. However results are compared with literature models using pre-trained embeddings, and this is not mentioned in the text.

  Our model uses a representation of both left and right context at label level like proposed in Dinarelli & Tellier (2016a); Dinarelli et al. (2017); Dupont et al. (2017); Dinarelli & Grobol (2019).

Figure 1 -

 1 Figure1-High-level architecture, from a conceptual point-of-view, of each block of a Transformer model (in the middle). On the left we show its instantiation in the Transformer, on the right we show its instantiation in our architecture, where we use a bidirectional GRU layer to encode contextual features

Table 1 -

 1 Statistics of the French corpus MEDIA

		Training		Dev		Test
	# sentences	12 908		1 259		3 005
		Words	Concepts	Words	Concepts	Words	Concepts
	# tokens	94 466	43 078	10 849	4 705	25 606	11 383
	dictionary	2 210		99	838	66	1 276	78
	OOV%	-		-	1,33	0,02	1,39	0,04
		Training		Dev	Test
	# sentences	38 219		5 527	5 462
		Words	Labels	Words	Labels	Words	Labels
	# tokens	912 344	-	131 768	-	129 654	-
	dictionary	43 210	45	15 081	45	13 968	45
	OOV%	-	-	22,61	0	20,00	0

Table 2 -

 2 Statistics of the English corpus WSJ

Table 3 -

 3 Statistics of the German corpus TIGER

Table 4 -

 4 Results of different variants of our model for the semantic tagging task of the corpus MEDIA, compared to the state-of-the-art distribution with GeForce GTX 1080 GPUs.

	.79 87.69	9.93

  table 4, results obtained on MEDIA with the Seq2Biseq-Transformer model are slightly worse than Seq2Biseq's results in terms of accuracy, and they are slightly better in terms of F1 and CER. 6 Adding word classes available for the MEDIA task (cf. the begin 5. Using default initialization of the Pytorch library 6. We did not perform significance tests

	Model	Accuracy
		WSJ DEV WSJ TEST
	LD-RNN deep

Table 5 -

 5 Results of different variants of our model for the POS tagging task on the WSJ corpus, compared to the state-of-the-art of section 3, data description) as input to the model, results improve with all evaluation metrics and constitute the new state-of-the-art on this task with manual transcription. 7 .Results of POS tagging on the WSJ corpus are presented in table 5. As we can see in this case, adding functionalities of the Transformer to the Seq2Biseq model give exactly the same result as the Seq2Biseq model on the Dev data, and slightly better on the Test data (on a single experiment).

Table 6 -

 6 Results of different variants of our neural architecture for the morpho-syntactic tagging task on the TIGER corpus, compared with state-of-the-art

	Model	Accuracy
		TIGER DEV TIGER TEST
	Seq2Biseq 2-opt	93.90 (98.30) 91.86 (97.74)
	VO-CRF (Lavergne & Yvon, 2017)	-	88.78

7. SLU is performed on automatic transcription, in this work however we evaluate our models against sequence labelling problems

Acknowledgments

This research is part of the program "Investissements dAvenir" funded by the French National Research Agency (ANR) ANR-10-LABX-0083 (Labex EFL). This work has been partially funded by the French ANR project DEMOCRAT ("Description et modlisation des chanes de rfrence : outils pour l'annotation de corpus et le traitement automatique") project ANR-15-CE38-0008.


Mara Otten
email: m.otten@uu.nl

Marja Van Den Heuvel-Panhuizen
email: m.vandenheuvel-panhuizen@uu.nl

Michiel Veldhuis
email: m.veldhuis@uu.nl

Aiso Heinze
email: heinze@ipn.uni-kiel.de

Fifth-grade students solving linear equations supported by physical experiences

Keywords: Early algebra, linear equation solving, balance model, physical experiences, embodied cognition theory

The aim of this study was to investigate the effect of a six-lesson teaching intervention on fifthgrade students' linear equation solving abilities. A hanging mobile, a balance model consisting of a horizontal beam with on each side a number of bags hanging on a chain, played a central role in this intervention. In total, 213 fifth-graders participated in one of the two intervention conditions or in the control condition. The intervention conditions differed with respect to the type of hanging mobile; either a physical hanging mobile that students could manipulate or a static version on paper. Preliminary analyses of the scores on the pre-and post-test seemed to show an improvement in students' linear equation solving abilities for students in both intervention conditions. Students, who worked in an embodied learning environment with a physical hanging mobile, seemed to show more improvement than students who worked with a paper-based mobile.

Introduction

The importance of laying the foundation for learning algebra at a young age is increasingly being emphasized. (e.g., [START_REF] Kaput | Algebra in the early grades[END_REF]National Council of Teachers of Mathematics [NCTM], 2000). A large number of studies have provided evidence that, by taking students' natural, intuitive ideas and informal reasoning as an entry point, students of primary school age can already engage in algebraic thinking (e.g., [START_REF] Kaput | Algebra in the early grades[END_REF]. Research in this area of early algebra has revealed that several algebraic concepts such as equivalence, expressions, equations, inequalities, generalized arithmetic, functional thinking, variable, and proportional reasoning can be taught to young students [START_REF] Blanton | The development of children's algebraic thinking: The impact of a comprehensive early algebra intervention in third grade[END_REF]. In the current study, we investigated how linear equation solving of fifth-graders can be fostered.

Teaching linear equation solving

Studies on early algebraic thinking often investigated students' linear equation solving ability. These studies generally found positive results. It was for example found that third-grade students can successfully deal with equation-related concepts such as equality and the equal sign, crucial for learning to solve linear equations (e.g., [START_REF] Bush | Prerequisite algebra skills and associated misconceptions of middle grade students: A review[END_REF], can solve simple equations such as 3 × n + 2 = 8, and can use letters to represent unknown quantities [START_REF] Blanton | The development of children's algebraic thinking: The impact of a comprehensive early algebra intervention in third grade[END_REF]. Moreover, when starting from a meaningful context, 10-year old students were found to be able to represent, meaningfully discuss, and solve equations with unknowns on both sides of the equal sign [START_REF] Brizuela | Ten-year-old students solving linear equations[END_REF], and, through informal reasoning, sixth-and seventh-grade students have even shown themselves able to solve systems of linear equations [START_REF] Van Amerom | Focusing on informal strategies when linking arithmetic to early algebra[END_REF][START_REF] Van Reeuwijk | The role of realistic situations in developing tools for solving systems of equations[END_REF]. However, there are only a limited number of studies in which young students had to solve systems of equations and those that were carried out were not set up systematically and contained small samples of students.

A frequently used model to make linear equation solving more accessible to young students is the balance model. This model can support students in different ways when solving linear equations and is particularly deemed suitable for grounding the equality aspect of an equation, as such enhancing the conception of the equal sign as a symbol for representing equality [START_REF] Otten | The balance model for teaching linear equations: A literature review[END_REF]. The balance model also can assist students in providing a language base for solving problems, as was shown by [START_REF] Warren | Young children's ability to use the balance strategy to solve for unknowns[END_REF] in a study involving third-grade students. Furthermore, by means of providing a meaningful context, it can help students to handle problems with unknowns [START_REF] Gavin | A balancing act: Making sense of algebra[END_REF].

Role of physical experiences in learning mathematics

Considering the role of physical experiences for learning has a long tradition. For example, Piaget [START_REF] Piaget | The child's conception of space[END_REF] already assumed that actions form the basis for learning and are important for understanding abstract ideas. A recent study on the efficacy of teaching mathematics through the use of manipulatives, showed benefits for instruction with manipulatives especially for children aged 7-11 and for the mathematical domain of algebra, compared to children that received abstract symbolic instruction [START_REF] Carbonneau | A meta-analysis of the efficacy of teaching mathematics with concrete manipulatives[END_REF]. Using manipulatives has also been found to be beneficial for young students to learn to solve simple symbolically presented equations [START_REF] Sherman | Equivalence in symbolic and nonsymbolic contexts: Benefits of solving problems with manipulatives[END_REF].

Since the emergence of theories of embodied cognition, the attention for physical experiences in learning has been renewed [START_REF] De Koning | Facilitating understanding of movements in dynamic visualizations: An embodied perspective[END_REF]. According to embodied cognition theories, bodily experiences are essential for cognitive learning processes (e.g., [START_REF] Wilson | Six views of embodied cognition[END_REF] and abstract higher-order cognitive processes, such as mathematics, are assumed to be grounded in action and perception [START_REF] Barsalou | Perceptions of perceptual symbols[END_REF]. Hence, embodied learning environments are regarded as important for learning mathematics [START_REF] Abrahamson | Embodiment and embodied design[END_REF].

Current study

In the current study, we investigated the effects of an intervention, consisting of a six-lesson teaching sequence, on fifth-grade students' linear equation solving abilities. The intervention was based on the idea that embodied learning environments can be beneficial for learning mathematics. A hanging mobile, a balance model consisting of a horizontal beam with on each side a number of bags hanging on a chain, played a central role in this intervention. In this study, we formulated the following research questions: (1) What is the effect of an intervention based on a balance model on

students' linear equation solving performance? and (2) What is the difference between the effect of an intervention based on a balance model with a manipulable physical hanging mobile from that with a static hanging mobile on paper on students' linear equation solving performance?

Method

Research design

To investigate these questions, we set up a classroom experiment based on a staged comparison design consisting of two intervention conditions and a control condition. In the first intervention condition, a physical hanging mobile was used that students could manipulate. In the second intervention condition, a paper-based hanging mobile was used. For each condition there were three cohorts, which differed in the timing of the intervention. In each cohort the students' performance on linear equation solving was measured four times (see Table 1). Each measurement contained the same test items. Making use of a staged comparison design made it possible that the same teacher taught all the lessons of the interventions.

M1 Intervention M2 M3 M4 Cohort 2 (n=18) M1 M2 Intervention M3 M4 Cohort 3 (n=25) M1 M2 M3 Intervention M4 Intervention Condition 2 Cohort 1 (n=22) M1 Intervention M2 M3 M4 Cohort 2 (n=21) M1 M2 Intervention M3 M4 Cohort 3 (n=25) M1 M2 M3 Intervention M4 Control Condition Cohort 1 (n=25) M1 Control intervention M2 M3 M4 Cohort 2 (n=30) M1 M2 Control intervention M3 M4 Cohort 3 (n=25) M1 M2 M3 Control intervention M4
Table 1:

Intervention program

In both intervention conditions students were taught linear equation solving by means of the same four-episode (six lessons) teaching sequence (see Figure 1). The lessons were taught by the first author of this paper. The overall aim of this sequence was to elicit algebraic reasoning related to (informal) linear equation solving. In each of the episodes the aim was to provide students with opportunities to develop algebraic strategies related to linear equation solving, such as restructuring strategies (e.g., change the order of terms in an expression, exchange the expressions on both sides of the equation), isolation strategies (i.e., strategies to isolate an unknown, such as taking away similar things on both sides), and substitution strategies (i.e., replacing unknowns by other unknowns or by values). In the first episode, students worked with one hanging mobile to discover relationships between unknowns and were thus reasoning about one equation. Students' main task was to discover all possible ways to maintain the balance of the mobile, for example by exchanging the bags of the left and right side of the mobile, by taking away similar bags from both sides, or by substituting one color of bags by another color. From Episode 2 on, the information from two equations had to be combined to discover relationships between unknowns to solve the problems.

Students were thus reasoning about a system of equations, first still in the context of the hanging mobile (Episode 2) and then in new contexts such as a tug-of-war situation (Episode 3). From Episode 3 on, students were additionally challenged to use more symbolic notations. In the final episode, Episode 4, students had to find the values of unknowns in a system of two symbolically notated linear equations.

The two intervention conditions differed as regards whether students gained physical experiences during the teaching sequence (Figure 1). In the first episode of Intervention Condition 1, an embodied learning environment was created in which students worked in small groups (2-3 students) with a physical hanging mobile. While trying to maintain the balance of the mobile, the tilting beam could be in or out of balance, thus providing students real-time feedback on their actions while manipulating the bags. In Episode 2, students in Intervention Condition 1 worked with paper-based hanging mobiles. The physical hanging mobiles were, however, still present in front of the classroom in this episode, but not all students worked on them. Instead, the physical hanging mobiles were used during classroom discussions by the teacher or by some students to make their reasoning processes explicit. Also, in Episodes 3 and 4 the physical hanging mobile was present in front of the classroom. In Intervention Condition 2, students received the exact same lessons with the exact same assignments; only the physical hanging mobile was replaced by a paper-based hanging mobile so that these students did not gain physical experiences during the lessons. Lastly, students in the Control Condition were not taught any lessons on linear equation solving. Instead they participated in a control intervention consisting of a six-lesson teaching on probabilitya topic which is also not taught at primary schools in the Netherlands. Students were taught the probability lessons as a control group, so that possible differences between the intervention and control conditions could not be attributed to the fact that only in the intervention conditions students received additional lessons on a (to them) new mathematical topic.

Test for linear equation solving

Students' performance on linear equation solving was assessed by a paper-and-pencil test, consisting of four items in which students had to solve (a system of) linear equations (see Figure 2). The same test was administered to the students repeatedly before and after the intervention (see Table 1, Measurement 1-4). The items were formulated in such a way that prior instruction on linear equation solving was not essential to solve the problems. The algebraic strategies that students developed during the lessons (i.e., restructuring, isolation, and substitution strategies) could be used to solve the problems. The open-ended questions explicitly invited students to explain their thinking and to reveal their reasoning.

Data analysis

Each linear equation solving performance item (Figure 2) was scored as incorrect (0) or correct (1), resulting in a total correctness score with scores ranging from 0 to 4. Items 2 and 4 were only scored as correct when both sub questions were answered correctly. Students' explanations were categorized by means of a coding scheme. In each test item, students had to solve the problem by combining the information of two equations. Taking the information from both equations into account and reasoning on the basis of both equations was crucial for solving these problems. In the coding scheme, we therefore coded students' level of reasoning based on their ability to incorporate information from the different equations in their reasoning process. More specifically, we distinguished between students who did not use any of the equations in the description of their reasoning (Level R0), students who reasoned on the basis of only one of the two given equations (Level R1), and students who reasoned on the basis of both given equations by combining the information of both of them (Level R2).

For the current paper, we focus on the data of the tests directly before and after the interventions (see Table 1, measures in bold), consisting of students' correctness and reasoning scores. Mean values and standard deviations were calculated for both scores on the pre-and post-test. Effect sizes [START_REF] Cohen | Statistical power analysis for the behavioral sciences[END_REF] were also calculated for each condition.

First results

For both intervention conditions, students' correctness scores increased. For Intervention Condition 1, the correctness scores increased from M = 2.31 (SD = 1.17) on the pre-test to M = 3.22 (SD = 0.91) on the post-test (d = 0.87) For Intervention Condition 2, the correctness scores increased from M = 2.43 (SD = 1.21) on the pre-test to M = 3.15 (SD = 0.93) on the post-test (d = 0.67). In contrast, students of the Control Condition almost showed no improvement on correctness scores, going from M = 2.65 (SD = 1.26) on the pre-test to M = 2.72 (SD = 1.33) on the post-test (d = 0.05). Students' reasoning also improved for both intervention conditions (see Figure 3). When comparing pre-and post-tests, a decrease in percentage of the lowest level of reasoning (Level R0) was observable, while there was an increase in the highest level of reasoning (Level R2). The percentages of the intermediate level of reasoning (Level R1) remained more or less stable. When comparing both intervention conditions, students in Intervention Condition 1 showed a somewhat larger improvement than students in Intervention Condition 2, both on correctness scores and level of reasoning. For both intervention conditions, there was a decrease in percentage of the lowest level of reasoning (R0) and an increase in the highest level of reasoning (Levels R2), with a somewhat larger decrease of Level R0 and a somewhat larger increase of Level R2 for Intervention Condition 1. The percentages of the intermediate level of reasoning (Levels R1) remained more or less the same for both conditions. For the Control Condition, the percentages of all levels of reasoning (almost) did not change.

To provide an example of an individual learning process of one of the students, we zoom in on the on the pre-and post-test answers to Item 4 of Omar, who participated in Intervention Condition 1. On the pre-test, Omar gave an incorrect answer to this item, with the explanation "I don't know". This response was categorized as Level R0, because he did not use any of the given equations in his explanation. His answer on the post-test is shown in Figure 4. Here, Omar interestingly represented the equations as hanging mobiles. Then he doubled the second equation (S + P = 10) to substitute this in the first equation (S + S + S + P + P = 27), so that unknown S was isolated. In this way he found that S equals 7 and then used the second equation to find the value of P. Omar's solution strategy was categorized as Level R2, because he reasoned on the basis of both given equations by combining the information from both equations.

Discussion

Based on these first results, we can draw the tentative conclusion that our six-lesson intervention in which the balance model in the form of a hanging mobile plays a central role, can improve students' linear equation solving abilities. Moreover, our descriptive data indicate that when this intervention took place in an embodied learning environment in which the students could work with a physical hanging mobile instead of with a version on paper, the performance gain was even larger. This latter finding is in line with the idea that embodied learning environments are beneficial for learning mathematics (e.g., [START_REF] Abrahamson | Embodiment and embodied design[END_REF], and it could add to the use of these environments in whole classroom settings. To be more certain about what we can learn from our study more advanced statistical analyses will be carried out.

  Research design. For the current study we focus on the measures in bold Participants Participants included 213 students (47% boys), with ages ranging from 9 to 11 (M = 10.04, SD = 0.49) from nine fifth-grade classes in seven schools in the Netherlands. The classes were selected by convenience. Three classes participated in Intervention Condition 1 (n = 65), three classes in Intervention Condition 2 (n = 68), and the final three classes formed the Control Condition (n = 80). Students had received no prior instruction on equation solving or other algebra topics.

Figure 1 :

 1 Figure 1: Schematic representation of the intervention (for both intervention conditions)

Figure 2 :

 2 Figure 2: Test items on linear equation solving

Figure 3 :

 3 Figure 3: Percentage of students showing each level of reasoning on the pre-and post-test

Figure 4 :

 4 Figure 4: Solution strategy of Omar (translated from Dutch) on the post-test; categorized as Level R2


Olaf Mercier

Jennifer Arthur Ataam

Nathaniel Langer

Peter Dorfmüller

Lilia Lamrani

Florence Lecerf

Benoit Decante

Philippe Dartevelle

Saadia Eddahibi

Elie Fadel

Abnormal pulmonary endothelial cells may underlie the enigmatic pathogenesis of chronic thromboembolic pulmonary hypertension

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.