Mihai Dascalu
email: mihai.dascalu@cs.pub.ro.

Scott A Crossley

Danielle S Mcnamara

Philippe Dessus

Stefan Trausan-Matu

PLEASE READERBENCH THIS TEXT: A MULTI-DIMENSIONAL TEXTUAL COMPLEXITY ASSESSMENT FRAMEWORK

Keywords: comprehension modeling, learning analytics, automated essay scoring, data analytics, Natural Language Processing

A critical task for tutors is to provide learners with suitable reading materials in terms of difficulty. The challenge of this endeavor is increased by students' individual variability and the multiple levels in which complexity can vary, thus arguing for the necessity of automated systems to support teachers. This chapter describes ReaderBench, an open-source multi-dimensional and multi-lingual system that uses advanced Natural Language Processing techniques to assess textual complexity at multiple levels including surface-based, syntax, semantics and discourse structure. In contrast to other existing approaches, ReaderBench is centered on cohesion and makes extensive usage of two complementary models, i.e., Cohesion Network Analysis and the polyphonic model inspired from dialogism. The first model provides an in-depth view of discourse in terms of cohesive links, whereas the second one highlights interactions between points of view spanning throughout the discourse. In order to argue for its wide applicability and extensibility, two studies are introduced. The first study investigates the degree to which ReaderBench textual complexity indices differentiate between high and low cohesion texts. The ReaderBench indices led to a higher classification accuracy than those included in prior studies using Coh-Metrix and TAACO. In the second study, ReaderBench indices are used to predict the difficulty of a set of various texts. Although the high number of predictive indices (50 plus) accounted for less variance than previous studies, they make valuable contributions to our understanding of text due to their wide coverage.

INTRODUCTION

Intelligent Tutoring Systems have been developed to support learners and human tutors by providing customized instructions and feedback. Measuring text difficulty is of particular interest in learning activities in order to best match resources to reader's comprehension level. However, the automated assessment of text difficulty is a difficult endeavor because multiple text features and learner characteristics (e.g., prior knowledge, language familiarity, reading performance, cognitive capabilities) need to be taken into account in order to provide tailored feedback. Thus, automated systems need to be adaptive to the audience, ensuring that learners are challenged, but are not excessively frustrated or demotivated.

Several automated systems have been developed and adopted in a wide range of educational scenarios. E-Rater [START_REF] Powers | Stumping e-rater®: Challenging the validity of automated essay scoring[END_REF]) is one of the first systems to automatically evaluate essays in which the quality of writing was measured by extracting a set of text indices such as: lexical complexity, spelling mistakes, lexical diction, text organization, as well as basic content analyses based on vocabularies. E-Rater supports a multi-layered textual complexity evaluation and subsequent software releases of the system added new features, including a model for evaluating the complexity of inferences in the discourse [START_REF] Grosz | Centering: a framework for modeling the local coherence of discourse[END_REF]. Newer tools such as TAALES (Kyle, Crossley, & Berger, in press), TAACO [START_REF] Crossley | The tool for the automatic analysis of text cohesion (TAACO): Automatic assessment of local, global, and text cohesion[END_REF], TAASC [START_REF] Kyle | Measuring Syntactic Complexity in L2 Writing Using Fine-Grained Clausal and Phrasal Indices[END_REF], or Coh-Metrix [START_REF] Graesser | Coh-Metrix: Analysis of text on cohesion and language[END_REF] provide comprehensive lists of scores for specific textual complexity indices that can be used for a wide range of text analyses on cohesion (i.e., semantic relations that exist and define a text) and language. Starting from automated essay scoring, the aim has transcended towards building Intelligent Tutoring Systems that provide detailed feedback besides singular evaluation scores. Automated Writing Evaluation systems such as the Writing Pal [START_REF] Roscoe | The Writing Pal intelligent tutoring system: Usability testing and development[END_REF] take the assessment component one step further and provide feedback to learners, thus supporting them to improve their writing skill.

Our implemented framework, ReaderBench [START_REF] Dascalu | Analyzing discourse and text complexity for learning and collaborating[END_REF][START_REF] Dascalu | Mining texts, learner productions and strategies with ReaderBench[END_REF]Dascalu et al., 2017) integrates an extensive list of textual complexity indices centered on cohesion and discourse structure. ReaderBench is built on top of Cohesion Network Analysis (CNA; [START_REF] Dascalu | Cohesion Network Analysis of CSCL Participation[END_REF] which provides an in-depth longitudinal perspective over the cohesive links across the text. Moreover, ReaderBench also integrates a complementary transversal perspective [START_REF] Dascalu | ReaderBench -Automated Evaluation of Collaboration based on Cohesion and Dialogism[END_REF], namely the polyphonic model of discourse [START_REF] Trausan-Matu | Supporting polyphonic collaborative learning[END_REF], which highlights interactions between points of view (i.e., "voices"). Thus, ReaderBench introduces in-depth textual complexity indices that account for certain specificities of the comprehension process, namely: a) automated word learning curves in our Age of Exposure model [START_REF] Dascalu | Age of Exposure: A Model of Word Learning[END_REF]; b) document cohesion flow metrics that model the structure of discourse in terms of global cohesion (Crossley, Dascalu, Trausan-Matu, Allen, & McNamara, 2016); c) metrics inspired from dialogism [START_REF] Dascalu | Modeling Comprehension Processes via Automated Analyses of Dialogism[END_REF]) that consider any text as a dialogue in which multiple points of view or voices interact and interanimate; and d) rhythm features inspired from prosody (Balint, Dascalu, & Trausan-Matu, 2016b).

The multi-layered textual complexity model behind ReaderBench is highly extensible and supports different languages, such as: English (Allen, Dascalu, McNamara, Crossley, & Trausan-Matu, 2016;[START_REF] Crossley | Combining Click-Stream Data with NLP Tools to Better Understand MOOC Completion[END_REF][START_REF] Dascalu | Predicting Academic Performance Based on Students' Blog and Microblog Posts[END_REF], French (Dascalu, Dessus, Bianco, & Trausan-Matu, 2014;[START_REF] Dascalu | Reflecting Comprehension through French Textual Complexity Factors[END_REF], Romanian (Dascalu, Gifu, & Trausan-Matu, 2016;[START_REF] Gifu | Time Evolution of Writing Styles in Romanian Language[END_REF], and Dutch [START_REF] Dascalu | ReaderBench Learns Dutch: Building a Comprehensive Automated Essay Scoring System for Dutch[END_REF]. In addition, several languages including Spanish, Italian, and Latin are partially covered.

At present, ReaderBench has more than a thousand textual indices for English language that cover: classic readability formulas that serve as baseline measures, surface indices (e.g., character, word, sentence and paragraph counts), syntax measures, and, more importantly, semantics and discourse structure indices to highlight in-depth comprehension processes. All the previous categories of indices are included in a multilayered and multi-lingual textual complexity model, and are described in detail in the following section. Thus, the aim of ReaderBench is to ensure a high degree of flexibility to adapt to different learning scenarios, playing the role of an artificial tutor that can, upon request at any moment, gauge multiple features of a text. To this purpose, our framework has been subject to extensive validations and this book chapter is focused on two specific studies, each centered on a different task. The first study considers perceived text difficulty as a function of cohesion in multi-paragraph texts. The second study is centered on predicting text difficulty by replicating the findings for the readability scores from previous studies [START_REF] Crossley | Predicting text comprehension, processing, and familiarity in adult readers: New approaches to readability formulas[END_REF] which used different tools. The vast majority of the aforementioned indices are language independent, and thus only specific semantic models need to be trained once the Natural Language Processing (NLP) pipeline [START_REF] Manning | Foundations of statistical Natural Language Processing[END_REF] is in place, whereas some indices are language-specific and require additional NLP techniques to be set up. ReaderBench includes a comprehensive NLP pipeline [START_REF] Manning | Foundations of statistical Natural Language Processing[END_REF]) that considers: a) stop-word elimination, b) the reduction of inflected forms to their corresponding lemmas, c) named entity recognition, d) the annotation of each word with its corresponding part of speech (POS) tag, e) dependency parsing, and f) co-reference resolution. This chapter is focused on providing an overview of the English indices, which represent the largest subset of available textual complexity indices and are used in the current studies.

READERBENCH -A COMPREHENSIVE MULTI-LAYERED AND MULTI-LINGUAL TEXTUAL COMPLEXITY MODEL

Surface Analysis

Surface analysis considers basic traits of texts derived from the initial studies of [START_REF] Page | The imminence of grading essays by computer[END_REF][START_REF] Page | Analyzing student essays by computer[END_REF] and [START_REF] Wresch | The imminence of grading essays by computer-25 years later[END_REF] that include statistically and easily computable proxes (i.e., computer automated approximations of interest) like: a) paragraph and sentence lengths (average values and standard deviations) in terms of characters, words or unique words; b) commas per sentence or paragraph; c) entropy measures [START_REF] Shannon | A Mathematical Theory of Communication[END_REF][START_REF] Shannon | Prediction and entropy of printed English[END_REF] at character and word levels. Character entropy is in general a language feature, i.e., it has similar values between texts written in the same language based on the distribution of individual letter. In contrast, higher entropy computed on the distribution of word stems that capture the common root of related concepts, reflects a more complex text because more varied concepts are introduced.

Syntax

Word-and sentence-based analyses, including part of speech tagging and dependency parsing, play important roles by providing two different complexity assessment schemes: a) normalized frequencies of each part of speech and b) structural indices derived from the parsing tree. Although nouns and verbs are the most common POSs, our analysis is particularly aimed at prepositions, adjectives, and adverbs that are potentially indicative of a more elaborate and complex text structure. Moreover, pronouns are indicative of a more inter-twined and linked structure of the discourse by revealing potential pronominal co-references. In addition, multiple indices can be derived using the structure of the parsing tree (e.g., an increased number of specific semantic dependencies or a higher maximum depth indicate a more complex discourse structure, yielding increased textual complexity) [START_REF] Gervasi | Quantitative assessment of textual complexity[END_REF].

Entity-density features are included because the number of entities within a text impacts the cognitive resources needed for their understanding, which impacts text readability. In general, named entities introduce conceptual information required for contextualizing the text; thus, in order to quantify the difficulty of a text with regard to newly introduced concepts in a text, we compute counts of entities (unique or not) per paragraph or sentence, as well as the percentages of named entities that are also nouns.

Semantics, Cohesion Network Analysis, and Discourse Structure

Text cohesion relates to explicit lexical, grammatical, or semantic text cues that support readers in making connections among text segments and the underlying ideas. Cohesion characterizes a unified and connected text, with sentences and paragraphs related to one another using explicit cues in the text [START_REF] Mcnamara | Sources of text difficulty: Across the ages and genres[END_REF]. Cohesion relates to humans' perception of that text's overall quality and coherence, and may be present at both local (i.e., sequential relations between neighboring sentences) and global (i.e., relations between paragraphs) levels [START_REF] Crossley | Predicting human scores of essay quality using computational indices of linguistic and textual features[END_REF][START_REF] Mcnamara | The linguistic features of quality writing[END_REF].

ReaderBench makes extensive use of Cohesion Network Analysis [START_REF] Dascalu | Cohesion Network Analysis of CSCL Participation[END_REF], which provides an in-depth view of the cohesive links that connect the discourse. Cohesion is viewed from a computational perspective as a relatedness measure between text chunks computed using multiple semantic models that complement one another [START_REF] Dascalu | Analyzing discourse and text complexity for learning and collaborating[END_REF]. Within ReaderBench, cohesion is the average score between the Wu-Palmer semantic distance in WordNet [START_REF] Wu | Verb semantics and lexical selection[END_REF] and semantic similarities measured using the following models: Latent Semantic Analysis (LSA; [START_REF] Landauer | A solution to Plato's problem: the Latent Semantic Analysis theory of acquisition, induction and representation of knowledge[END_REF], Latent Dirichlet Allocation (LDA; [START_REF] Blei | Latent Dirichlet Allocation[END_REF], and word2vec [START_REF] Mikolov | Efficient Estimation of Word Representation in Vector Space[END_REF]. Therefore, local and global cohesion are reflected within CNA as the strength of intraand inter-paragraph links extracted from the cohesion graph. [START_REF] Crossley | Document Cohesion Flow: Striving towards Coherence[END_REF] developed automated measures of global document cohesion flow based on a CNA that considers the adequacy of paragraph sequences -i.e., the degree to which one paragraph succeeds the previous one in a cohesive manner. Our cohesion flow measures consider the order of different paragraphs and the manner in which they are combined to hold the document together. A text that exhibits a high cohesion flow by linking ideas between adjacent paragraphs tends to be easier to comprehend.

In addition, ReaderBench also implements the polyphonic model [START_REF] Trausan-Matu | Supporting polyphonic collaborative learning[END_REF] inspired from dialogism in which the discourse is perceived as an inter-animation of different points of view (i.e., "voices") that interact with each other. After identifying voices as semantic chains of related words spanning throughout the text [START_REF] Dascalu | Analyzing discourse and text complexity for learning and collaborating[END_REF], several textual complexity indices are computed in order to quantify the impact of each voice and to establish the degree of overlap between voices [START_REF] Dascalu | Modeling Comprehension Processes via Automated Analyses of Dialogism[END_REF]): a) distribution per sentence or paragraph, including span (distance between the last and the first occurrence of words from the same voice), b) recurrence (average and standard deviation of the distance between subsequent words pertaining to the same voice), and c) overlap measures (e.g., co-occurrences or mutual information).

Related to discourse, ReaderBench accounts for three additional dimensions. First, pronominal resolutions are performed [START_REF] Lee | Stanford's Multi-Pass Sieve Coreference Resolution System at the CoNLL-2011 Shared Task[END_REF][START_REF] Manning | The Stanford CoreNLP Natural Language Processing Toolkit[END_REF] and several complexity indices are computed including: a) the average number of co-references per chain, their span and inference distance, b) the average number of active co-reference chains per word (if more words are included in co-reference chains, the text becomes harder to comprehend as it is more inter-twined and more inferences need to be resolved), and c) the number of co-reference chains with a large span (experimentally set at 30% of the document length).

Second, rhythm is an important feature of discourse (Trausan-Matu, [START_REF] Trausan-Matu | PolyCAFe-automatic support for the polyphonic analysis of CSCL chats[END_REF]. Rhythm in a text is related to its communicative purposes and genre (Balint, Dascalu, & Trausan-Matu, 2016a, 2016b). Similar to previous dimensions, multiple indices are integrated into ReaderBench, namely: a) the average number of stressed syllables and of rhythmic units in each sentence, b) the number of deviations from dominant structures divided by total number of syllabic segments, c) the rhythmic index [START_REF] Marcus | Poetica matematică[END_REF], d) the frequency of the maximum rhythmic index, e) the maximum number of consecutive unstressed syllables, and f) the number of alliterations and assonances in a text searching sentence by sentence.

Third, cue phrases are used to quantify the usage of different types of pronouns (e.g., first, second, third, interrogative, or indefinite) and connectives (e.g., conjuctions, contrasts, sentece linking, or conditions), thus providing insights to the structure of each sentence or paragraph, and its corresponding degree of complexity. Finally, specific word lists that capture particular semantic valences are integrated into ReaderBench, namely: General Inquirer (GI, http://www.wjh.harvard.edu/~inquirer/homecat.htm) [START_REF] Stone | The General Inquirer: A Computer Approach to Content Analysis[END_REF], Lasswell [START_REF] Lasswell | The Lasswell Value Dictionary[END_REF] dictionary, SenticNet [START_REF] Cambria | Sentic computing for social media analysis, representation, and retrieval[END_REF], Affective Norms for English Words (ANEW) (M. M. [START_REF] Bradley | Affective norms for English words (ANEW): Stimuli, instruction manual and affective ratings[END_REF], Geneva Affect Label Coder (GALC) [START_REF] Scherer | What are emotions? And how can they be measured?[END_REF], Linguistic Inquiry and Word Count (LIWC) [START_REF] Pennebaker | Linguistic inquiry and word count: LIWC [Computer software[END_REF], and EmoLex or NRC Word-Emotion Association Lexicon [START_REF] Mohammad | Crowdsourcing a word-emotion association lexicon[END_REF]. The cue phrases and taxanomies represent a solid background for covering the most representative linguistic approaches relying on word counts.

Word Complexity and Age of Exposure

Word complexity is computed as a mix of multiple word indices, such as: a) syllable count, b) length of suffices and prefixes expressed in number of characters between a word's inflected form and its lemma or stem, c) specificity computed as the inverse document frequency from the training corpora, d) word polysemy count and average/maximum hypernym tree depth computed using WordNet, and e) our metric of word difficulty that considers contextualization -Age of Exposure (AoE) [START_REF] Dascalu | Age of Exposure: A Model of Word Learning[END_REF]. Similar to [START_REF] Landauer | Word maturity: A new metric for word knowledge[END_REF], our goal was to create a word learning model (AoE) using LDA, an automated alternative for Age of Acquisition (AoA) scores -e.g., [START_REF] Kuperman | Age-ofacquisition ratings for 30,000 English words[END_REF], [START_REF] Bird | Age of acquisition and imageability ratings for a large set of words, including verbs and function words[END_REF] or [START_REF] Schock | Age of acquisition estimates for 3,000 disyllabic words[END_REF] that estimate a learner's age at which a certain word is correctly understood.

In addition, ReaderBench considers the following individual word complexity measures: a) mean syllable count per word: longer, more complex words tend to be perceived as being more difficult; b) mean polysemy count per word: words with multiple senses are more difficult and require more contextual information for disambiguation; c) average and maximum distance within the hypernym tree to the ontology root: more general words are closer to the root, whereas more specific words tend to have a longer path; and d) differences between the inflected form, the lemma and the stem: words with longer prefixes and suffixes tend to be more complex. All word indices are averaged at document, paragraph and sentence levels by considering only the lemmas of content words (i.e., dictionary words, not included in the stop-words lists, and having as part of speech one of the following: noun, verb, adverb or adjective).

PREDICTING COMPREHENSION THROUGH AUTOMATED TEXT ANALYSIS

The following studies represent in-depth comprehension analyses performed using the previously presented ReaderBench textual complexity indices for English language, and are centered on the following research questions:

1) What automated text complexity indices best capture differences in text cohesion? 2) Are linguistic features predictive of judgments of text comprehension, processing, and familiarity?

Study 1 -Text Cohesion

Corpus

We first selected the texts compiled by [START_REF] Mcnamara | Coh-Metrix: Capturing linguistic features of cohesion[END_REF]. These texts were used in experiments that investigated multi-paragraph text cohesion. The dataset contains 19 pairs of texts (low and high cohesion texts) from 12 different studies discourse studies. The texts (n = 38) were selected from textbooks, encyclopedia articles, researcher created texts, or text from books. In each case, the original text was manipulated by discourse processing experts to make the text more cohesive. In this study, we used ReaderBench indices to discriminate between the original texts and those texts modified to make them more cohesive.

Statistical Analyses

A number of pre-selection criteria were required for ReaderBench indices to be included in the final analysis. First, ReaderBench indices that yielded non-normal distributions were removed because they violated statistical assumptions. Of the original 329 indices, this step removed 117 of the indices. A multivariate analysis of variance (MANOVA) was then conducted to examine which ReaderBench variables reported meaningful and significant differences between the original texts and the text modified by experts to make them more cohesive. The MANOVA was followed by a stepwise discriminant function analysis (DFA) using the selected ReaderBench indices that demonstrated significant differences between the original and modified texts. Those indices that demonstrated significant differences between the two text groups and did not exhibit multicollinearity (r > .70) with other indices in the set were used in the DFA. In the case of multicollinearity, the index demonstrating the largest effect size was retained in the analysis. The DFA was used to develop an algorithm to predict group membership through a discriminant function co-efficient. A DFA model was first developed for all of the original and modified texts. This model was then used to predict group membership of the original and modified texts using leave-one-out-cross-validation (LOOCV) in order to ensure that the model was stable across the dataset.

Results

MANOVA

A MANOVA was conducted using ReaderBench indices as the dependent variables and the original and modified texts as the independent variables. The strongest predictors that were not multicollinear with other indices were related to local cohesion indices and syntactic complexity indices. These 12 indices showed medium to strong effect sizes (see Table 1 for the MANOVA results). These indices were used in the subsequent DFA.

Discriminant Function Analysis

A stepwise DFA using the indices selected through the MANOVA retained two variables from the 12 variables selected through the MANOVA as predictors. The remaining ReaderBench variables were removed. The two retained variables were related to local cohesion and syntactic complexity (see Table 2 for indices and unstandardized coefficients).

The results demonstrate that the DFA using these two indices correctly allocated 30 of the 38 texts in the total set, χ2(df = 1) = 12.880, p < .001, for an accuracy of 78.9%. The leave-one-out cross-validation (LOOCV) results reported the same classification accuracy (see the confusion matrix reported in Table 3 for results). The Cohen's Kappa measure of agreement between the predicted and actual completion rate was .79, demonstrating substantial agreement.

Discussion

The results of the DFA indicated that two ReaderBench indices were able to strongly distinguish between original texts and texts modified to increase cohesion. The results of the study demonstrate that the latter texts include more subordinating conjunctions and nominal subject dependencies. The first index is related to local cohesion (i.e., cohesion that connects smaller text segments such as sentences and phrases) and replicates findings in the original study which found that local cohesion in terms of lexical and semantic overlap between sentences was a strong predictor of texts with increased cohesion [START_REF] Mcnamara | Coh-Metrix: Capturing linguistic features of cohesion[END_REF]. The results also replicated a second study that used this dataset [START_REF] Crossley | The tool for the automatic analysis of text cohesion (TAACO): Automatic assessment of local, global, and text cohesion[END_REF] which reported that texts modified to increase cohesion have an increased number of simple subordinators, reason and purpose connectives, and causal connectives. The second index that was predictive in this study (nominal subject dependencies) provides new information about increased text cohesion that was not evaluated in either [START_REF] Mcnamara | Coh-Metrix: Capturing linguistic features of cohesion[END_REF], or [START_REF] Crossley | The tool for the automatic analysis of text cohesion (TAACO): Automatic assessment of local, global, and text cohesion[END_REF]. Similar to McNamara et al., this finding indicates that the process of increasing text cohesion may have unintended effects on syntactic complexity. While relating to the ReaderBench indices, increasing connections between sentences and phrases (and inherently text cohesion) leads to an increase in the number of syntactic dependencies related to the head nouns (i.e., the number of phrases that the head noun controls). As an example, if two sentences such as (1) and ( 2):

1) The woman went to the store. 2) She needed milk. are combined using a simple subordinator such as because, this will result in the sentence "The woman went to the store because she needed milk." While the cohesion in this sentence may increase, it seems to come at the cost of increased syntactic complexity. Nonetheless, added cohesion comes with substantial comprehension benefits for all ages, providing some evidence against theories that focus on processing resources (e.g., working memory), in contrast to theories of discourse comprehension (McNamara et al., 2010[START_REF] Roscoe | The Writing Pal intelligent tutoring system: Usability testing and development[END_REF].

In terms of machine learning accuracy, the indices reported by ReaderBench led to a slightly higher classification accuracy than those reported by Coh-Metrix in the original study [START_REF] Mcnamara | Coh-Metrix: Capturing linguistic features of cohesion[END_REF] and the Tool for the Automatic Analysis of Cohesion (TAACO), which was used in the Crossley, [START_REF] Crossley | The tool for the automatic analysis of text cohesion (TAACO): Automatic assessment of local, global, and text cohesion[END_REF]

Study 2 -Pairwise Text Comprehension Comparisons

Corpus

This study uses 150 news articles from the Guardian Weekly publication which were first reported by [START_REF] Crossley | Predicting text comprehension, processing, and familiarity in adult readers: New approaches to readability formulas[END_REF]. The corpus included 50 original news articles, 50 that were simplified to the beginning level, and 50 simplified to the intermediate level. The articles were truncated using original paragraph breaks in order to ensure an approximately length of 150 words. Afterwards, these texts were ranked in terms of readability using the Mechanical Turk crowdsourcing service available through Amazon.com. Overall, 307 participants were recruited, and each evaluator provided approximately 10 pairwise comparisons of two texts with regards to text comprehension (which text was easier to understand), text processing (which text was quicker to read), and text familiarity (which addressed more familiar or knowledgeable topics). [START_REF] Crossley | Predicting text comprehension, processing, and familiarity in adult readers: New approaches to readability formulas[END_REF] applied a Bradley-Terry model (R. A. [START_REF] Bradley | Rank analysis of incomplete block designs. I. The method of paired comparisons[END_REF] to the 3,011 pairwise comparisons, for each of the previous three ranking criteria, to generate an evaluation of the text's difficulty in terms of its likelihood to be more difficult than all other texts. The Bradley-Terry model provides a global ranking of entries based on multiple individual pairwise comparisons (which in some cases can reflect opposite considerations of the raters); after maximizing the likelihood of the observed data and reaching convergence, texts that take longer to process are scored higher.

Statistical Analysis

Our interest in this ReaderBench analysis is to replicate the findings for the readability scores (i.e., the processing scores) reported by [START_REF] Crossley | Predicting text comprehension, processing, and familiarity in adult readers: New approaches to readability formulas[END_REF]. To this end, we developed a regression model to predict text processing ratings derived from the Bradley-Terry models. Prior to the analysis, we removed any variables that violated a normal distribution to better assure that residuals in the regression model were distributed normally. Pearson correlations were then conducted on the remaining variables to determine whether they were meaningfully correlated with judgments of text processing. Any variables that did not report at least a small effect size (i.e., r ≥ .100) with the text processing scores were removed from the analysis. The remaining variables were checked for multicollinearity to ensure that the final model consisted only of unique indices. For each pair of variables with absolute correlation values of r >= .699, only the variable with the highest correlation with text processing scores was retained. These variables were entered into a stepwise regression (bidirectional) using the stats package in R ( [START_REF] Core | R: A language and environment for statistical computing[END_REF]. Results were checked for significance multicollinearity using variance inflation values (VIF). The final model was checked for normality of residual, homoscedasticity, and constant error variance to ensure assumptions of linearity were met. The final model was then tested using 10-fold cross-validation through the caret package [START_REF] Kuhn | Caret package[END_REF] available in R. The method for inclusion into the model was also stepwise. Values reported for the model were co-efficient strengths and direction, relative importance metrics (predictors explained variance as non-negative contributions) using the relaimpo package [START_REF] Grömping | Relative importance for linear regression in R: the package relaimpo[END_REF], and t and p values. The cross-validated model was also checked for all assumptions of linearity.

Results

We calculated correlations between the selected linguistic indices and the text processing ratings generated by the Bradley-Terry model. After controlling for normal distribution, effect sizes, multicollinearity, and multiple comparisons, the text processing judgment analysis included 51 linguistic indices. Correlations are presented in Table 4.

To analyze which linguistic features predicted the text processing ratings, we conducted a stepwise regression analysis using the selected linguistic indices as the independent variables. This yielded a significant model, F(9, 140) = 10.440, p < .001, R 2 = .402. Nine variables were significant predictors of the text comprehension ratings. These nine variables, when used in a 10-fold cross-validation model yielded a model with an R 2 = .301 and RMSE = .005 (see Table 5 for coefficients, t values, and p values for the model).

DISCUSSION

A unique feature of this analysis and the database is that it focused on text processing and not text comprehension (i.e., how well a text is understood). Most readability studies focus on comprehension (e.g., [START_REF] Kate | Learning to predict readability using diverse linguistic features[END_REF] and not processing even though most theories of text readability include text processing within their definition [START_REF] Chall | Readability revisited: The new Dale-Chall readability formula[END_REF][START_REF] Newbold | The linguistics of readability: the next step for word processing[END_REF]. Thus, this study helps to build on the few studies that focus on text processing as a function of text readability (cf. [START_REF] Crossley | Predicting text comprehension, processing, and familiarity in adult readers: New approaches to readability formulas[END_REF].

The results of the regression model indicate that ReaderBench variables related to sentence length, cohesion, pronoun use, dialogism, and syntactic dependencies were all related to text processing speed. The coefficients suggest that news articles that took longer to read had longer sentences, had a greater number of referents per paragraph, had great paragraph cohesion, and were more syntactically complex. Texts were also more quickly read if they had greater paragraph to text cohesion. These findings support models of text readability which postulate that syntactic and discourse features are important components of readability [START_REF] Just | A theory of reading: From eye fixations to comprehension[END_REF][START_REF] Koda | Insights into second language reading: A cross-linguistic approach[END_REF] and that greater complexity in these features will lead to greater difficulty in text processing. In comparison to [START_REF] Crossley | Predicting text comprehension, processing, and familiarity in adult readers: New approaches to readability formulas[END_REF], ReaderBench indices alone accounted for less variance (i.e., 30% as compared to 47% of the variance) but, again, differences between the two studies can explain these disparities. For instance, Crossley et al. used NLP tools that reported on phrasal indices, one of which (tri-gram incidence) explained 20% of the text processing variance in the original study. Such measures are not currently included in ReaderBench.

The ReaderBench indices in this study also make unique contributions to our understanding of text processing. For instance, to our knowledge, the number of unique pronouns per paragraph, coupled with the number of co-references per chain, has not been a significant predictor of text processing in previous studies. This finding suggests that texts with a greater number of referents are more difficult to process. While unique, the finding does overlap with the model reported in [START_REF] Crossley | Predicting text comprehension, processing, and familiarity in adult readers: New approaches to readability formulas[END_REF] which reported that a greater number of entities per sentence (e.g., people's names, venues, organizations) leads to slower processing. Moreover, indices derived from dialogism sustaining the diversity (i.e., entropy of different points of view) and spread (i.e., span) of ideas were also predictive, in tight connection with the indices related to global cohesion. Lastly, this ReaderBench analysis provides evidence that syntactic complexity (i.e., punctuation and case syntactic dependencies per sentence) in a text can decrease text processing speed. While theorized, little evidence has supported this assertion.

CONCLUSION

To our knowledge, ReaderBench is a unique open-source, multilingual framework, which provides a wide range of textual complexity indices and that can be used to perform various text cohesion and in-depth discourse analyses. It combines several semantic and discourse analysis techniques from Natural Language Processing into CNA and implements the ideas of the polyphonic model together with its associated ideas, such as rhythm and interanimation. The tool and the corresponding web services are available both online (http://readerbench.com) and for download as a desktop client that includes additional functionalities to the ones published online (http://readerbench.com/ deployment).

The results from the first study indicate that the textual complexity indices from ReaderBench can differentiate texts among different levels of cohesion that plays an important role in comprehension. The second study provides evidence for ReaderBench's wide applicability with respect to the high number of significantly correlated textual complexity indices, most of which are centered on semantics and discourse. These findings demonstrate ReaderBench's versatility and generalizability to multiple contexts [START_REF] Botarleanu | ReadME -Generating Personalized Feedback for Essay Writing using the ReaderBench Framework[END_REF]. As such, learners and educators can potentially use the ReaderBench framework across multiple pedagogical scenarios.

Figure 1

 1 Figure 1 depicts an overview of our multi-layered textual complexity model which includes: a) the simplest surface measures that account only for the form of the text;

Figure 1 .

 1 Figure 1. ReaderBench multi-layered textual complexity model.

Table 1 . Study 1 -MANOVA results

 1

	Index

Table 2 . Study 1 -Discriminant Function Coefficients

 2

	Index

Table 3 . Study 1 -Confusion matrix for DFA classifying texts

 3

			Predicted
		Actual	Original	Modified
	Whole set	Original	16	3
		Modified	5	14
	LOOCV	Original	16	3
		Modified	5	14

  study. For instance, the accuracy reported in McNamara at el. was 76% while the ReaderBench analysis was 79%. Lastly, the Coh-Metrix model reported in McNamara et al. reported the results for only the training set, which may have increased the reported accuracy. The accuracy found in Crossley et al. was lower than both the current ReaderBench analysis and Coh-Metrix model at 68%; however, it should be noted that Coh-Metrix relied on cohesion and word frequency, Crossley et al. only used cohesion indices, while the ReaderBench model included lexical, syntactic, semantic, and cohesion-centered indices.

Table 4 . Study 2 -Correlations with reading processing scores

 4

	Index

Table 5 . Study 2 -Summary of multiple regression model for pairwise comparisons (text processing)

 5

	Entry	Relative importance	Coefficient	t	p
	1 Average sentence length expressed in characters	0.2455	0.001	3.110	.002
	2 Avg. unique pronouns per paragraph	0.1594	0.011	4.260 <.001
	3 Avg. case syntactic dependencies per sentence	0.1591	0.001	1.790	.076
	4 Avg. punctuation syntactic dependencies per sentence	0.1014	0.004	2.000	.047
	5 Average paragraph adjacency cohesion using the	0.0746	0.001	3.150	.002
	Jensen-Shannon dissimilarity between LDA
	probability distributions
	6 Avg. paragraph-document cohesion using cosine	0.0706	-0.003	-3.040	.003
	similarity in LSA vector spaces
	7 Avg. co-references per chain	0.0687	0.001	2.640	.009
	8 Avg. voice entropy per paragraph (dialogism)	0.0656	0.001	2.500	.013
	9 Maximum voice span (dialogism)	0.0552	0.001	2.390	.018
	Note: Constant = -0.062.

ACKNOWLEDGMENTS

This research was partially supported by the 644187 EC H2020 Realising an Applied Gaming Eco-system (RAGE) project, the Department of Education, Institute of Education Sciences -Grant R305A130124, as well as by the Department of Defense, Office of Naval Research -Grants N00014140343 and N000141712300.


S Naili
email: naili@univ-paris12.fr

Buckling of a Short Cylindrical Shell Surrounded by an Elastic Medium

The lateral surface of a cylindrical structure, which is composed of a thin tube embedded in a large outer medium, is submitted to a uniform external pressure. The buckling pressure of such a structure, corresponding to a low flexural state of the inner tube wall, is theoretically analyzed on the basis of the asymptotic method. The theoretical results are compared with experimental ones obtained from a compression test realized on an elastic tube inserted in a foam. It is found that the Euler pressure and the associated buckling mode index strongly depend upon the rheological and geometrical parameters of both the tube and the surrounding medium. ͓S0021-8936͑00͒00201-4͔

Formulation of the Problem and Buckling Study

A nonhomogeneous cylindrical structure composed of a thin shell inserted in a surrounding elastic medium was subjected to a state of plane strain by external pressurization and zero axial longitudinal displacement constraint. The onset of the buckling process for such a structure was analyzed. The theoretical results were compared with original experimental ones as derived from a hoop compression test which was conducted with elastic rubber tubes embedded in foamy materials.

Thus, we consider the mechanical behavior of a cylindrical nonhomogeneous structure made of an internal shell confined in a large outer medium, the whole structure being submitted to a uniform pressure p on its external lateral surface. Each solid is elastic, cylindrical-of same axis-with a circular cross section in the reference configuration. In this configuration, the mean radius of the shell is denoted as r 0 . The outer radius r ϱ of the medium is assumed to be very large compared to r 0 . We will denote as e 0 the thickness of the shell. The two solids have the same height which is small in comparison with r ϱ . The outer lateral boundary of the medium is subjected to a uniform pressure p. We assume frictionless contact between the two tubes. Body forces and inertia effects are deemed to be negligibly small.

The work of Forrestal and Herrman ͓1͔ presents solutions for both bonded and smooth conditions at the shell-medium interface; it includes a geometrically nonlinear formulation for both the shell and the medium. Moore and Booker ͓2͔ presented the linear approximation of Forrestal and Herrman's formulation. The physical unknowns which are involved in these theoretical formulations are the buckling pressure of the shell in the first work and the hoop compression stress inside this shell in the second one.

In the present work, we use the dimensionless buckling pressure of the overall structure consisting of the shell and surrounding medium developed by Razakamiadana et al. ͓3͔ and which is given by the relation

p ¯ϭn 2 Ϫ1ϩ ␤ n2 Ϫ1 , ( 1
)
where the dimensionless variables are given as follows:

p ¯ϭ␣p r 0 3 D , ␤ ¯ϭ␤ r 0 4 d ,
and where DϭE 1 e 0 3 /12(1Ϫ 1 2 ) is the flexural rigidity modulus of the shell, E 1 and 1 being, respectively, its Young's modulus and its Poisson's ratio, while the index n characterizes the buckling mode. The parameters ␣ and ␤ are defined by ␣ϭ

ͫ 2 ϩ2 2 2 ϩ 2 ͬͩ 2 e 0 r 0 ͪ 1Ϫ ͩ 1Ϫ2 e 0 r 0 ͪͫ 1Ϫ 2 1 ͬ ϩ 2 1 ϩ 1 , ␤ϭ2 2 r 0 ͑ 2 ϩ 2 ͒ ͫ n 2 Ϫ1 n͑ 2 ϩ2 2 ͒ϩ 2 ͬ ,
in which the Lame's parameters of the shell and of the medium are denoted by 1 , 1 and 2 , 2 , respectively.

In this study, we were interested in the smallest value of the buckling pressure-the Euler pressure-, while varying the index n associated with the buckling mode; such a mode index characterizes the number of axes of symmetry in the actual configuration. Indeed, this minimal pressure is the most frequently observed experimentally while applying incremental loading to the structure. This pressure is expressed as

p e ϭmin nу2 ͫ 1 ␣ ͩ ͑ n 2 Ϫ1 ͒ D r 0 3 ϩ ␤r 0 n 2 Ϫ1 ͪͬ .
(2) It is to be noted here that a classical case corresponds to the particular condition of an external incompressible fluid-i.e., 2 ϭ0 and 2 →ϱ-surrounding the shell, so that ␣ϭ1 and ␤ϭ0 in relations ͑1͒ and ͑2͒.

Experimental Procedure

Hoop compression tests were performed on a cylindrical structure with a circular cross section composed by a thin rubber shell, of external radius r e ϭ13 mm which was inserted in a large foam medium of external radius r ϱ ϭ110 mm. Both tubes had a height Hϭ30 mm. The thin rubber shell was slightly stressed when inserted within the foam medium in order to establish a good contact between the two solids.

A steady loading was applied on the external lateral wall of the surrounding foam medium by means of a tire inner tube connected to a standard pressurization system. The plane strain of the structure was obtained by maintaining it between two circular and polished PMMA transparent plates. In order to avoid significant friction between the foam and the plates, the lower and upper faces of the foam were sprinkled with talc powder.

The applied pressure was measured by using a mercury U-manometer graded every 1 mm in height with a maximum reading error estimated at about 0.5 mm. When the buckling pressure is ''very low,'' the relative accuracy of the pressure measurement was estimated at about seven percent. But, in 90 percent of the cases, the measured pressure was about 50 mm Hg and the relative uncertainty of measurement was estimated, on average, to be one percent.

With this experimental setup, several tests of compression were conducted on the structure with given geometrical and mechanical characteristics. The tested structure was submitted to a gradual and slow loading so that, for each step, the system can be considered in stationary equilibrium state. The shape of the cross section of the shell remains circular before undergoing a change of shape. We monitored the evolution of shape with a CCD camera video placed on the axis of the tubes-see Fig. 1.

The thickness e 0 of the shell, in its reference configuration, was inferred from the mean value of the measurements conducted with a micrometer at various locations on the wall. The variations around the mean value were found to be in the order of two percent. The values of Young's modulus of the shell and of the medium were derived from traction and compression tests, respectively applied on samples of the constitutive materials. In the different experimental setup, four types of latex foam media with different Young's modulus and Poisson's ratio were combined with tubes made of various PCP, PCV, or latex materials having different characteristic mechanical properties as indicated in Table 1. For the deformations up to ten percent each Young's modulus was evaluated with a maximum error of five percent. Besides, these tests have shown that the Poisson's ratio of the shell was about 0.5-, i.e., the material is incompressible-whereas the Poisson's ratio for the foam media were around zero.

Next, the video images were digitized and then automatically processed using a global thresholding method so as to quantify the inner cross section area of the inserted tube and to characterize its shape. In the extreme case corresponding to a significant variation of this shape, the relative uncertainty of area measurement was estimated to be of the order of two percent. Indeed, when the structure is submitted to a gradual and slow loading, we retained as Euler pressure p e , the one which corresponds to a clear change in the inner cross section area, as discussed later on.

For a given structure, the measurement of the buckling pressure was repeated ten times at least and the relative gap compared to the mean value varies between 1 percent and 30 percent.

Analysis and Discussion

We show, in columns 5 and 6 of Table 1, the experimental results obtained on 110 tests implying 12 structures of different geometry and elastic properties. The results are discussed by using the dimensionless geometrical and mechanical parameters e ϭe 0 /r 0 and E ¯ϭE 1 /E 2 in the case of rather thin tubes and exter- It is worth noting that the assessment of the buckling pressure in such an experiment was based on the variation of a global geometrical parameter such as the area of the internal cross section of the inserted shell-see Fig. 2. Other more sensitive parameters, related to the changes in local shape properties could, however, be envisaged but their quantification by an image-processing system would have been more difficult to implement.

In columns 7 and 8 of Table 1, the theoretical results are compared with experimental ones. These results show that the Euler pressures, evaluated theoretically and determined experimentally, agree well accounting for the inherent scatter in experimental measurements. Moreover, the mode index n associated with p ¯e coincides exactly for 50 percent of the cases. Nevertheless, it is worth emphasizing that significant differences arised in the case of very thin tubes for which the mode index are rather high, the large number of lobes being more sensitive to small heterogeneities in material property and geometry. Despite this, the observed experimental results are, in general, well reproduced by our theoretical model.

Fig. 1

 1 Fig. 1 Experimental apparatus for hoop compression tests.The tube inserted in the foam medium is in a buckled state with index of buckling mode equal to two.

Fig. 2

 2 Fig. 2 Top view of the test cell giving an illustrative example of the tube inserted in the foam medium in a buckled state with index of buckling mode equal to four. Circular windows-with a radius in the order 30 mm-were cut on the top and bottom of the PPMA container for a better definition of the image during the recording.

Table 1 Experimental and theoretical results of Euler pressure normalized by E 2 and index of buckling mode n associated for various dimensionless mechanical and geometrical param- eters

 1

			Experiment			Theory
	E ¯E2 ͑kPa͒	e ¯e0 ͑mm͒	p e /E 2 ϫ10 Ϫ2	n	p e /E 2 ϫ10 Ϫ2	n
	14.50	100	0.078	1.00	(8.69Ϯ0.80)	6	8.39	6
	14.50	100	0.189	2.40	(15.34Ϯ2.19)	2	13.00	2
	26.07	79	0.037	0.46	(4.21Ϯ0.40)	4	4.21	6
	28.70	100	0.105	1.30	(11.00Ϯ1.00)	4	9.80	4
	28.70	100	0.136	1.75	(12.89Ϯ0.92)	3	12.05	3
	28.86	79	0.032	0.40	(3.98Ϯ0.33)	3	4.87	6
	28.86	79	0.070	0.90	(6.13Ϯ0.63)	2	7.65	5
	29.85	69	0.037	0.46	(4.66Ϯ0.78)	4	5.24	6
	33.00	69	0.032	0.40	(4.21Ϯ0.21)	3	4.79	6
	33.00	69	0.070	0.90	(6.10Ϯ0.31)	2	7.69	5
	158.46	13	0.037	0.46	(11.76Ϯ3.69)	3	7.60	3
	175.38	13	0.070	0.90	(17.69Ϯ0.10)	2	9.61	2


Guillaume Rosinosky
email: guillaume.rosinosky@loria.fr

Chahrazed Labba
email: chahrazed.labba@loria.fr

Vincenzo Ferme
email: vincenzo.ferme@usi.ch

Samir Youcef
email: samir.youcef@loria.fr

François Charoy
email: francois.charoy@loria.fr

Cesare Pautasso
email: cesare.pautasso@usi.ch

Evaluating Multi-Tenant Live Migrations Effects on Performance

Keywords: Live migration, Multitenancy, BPMS, Performance

Multitenancy is an important feature for all Everything as a Service providers like Business Process Management as a Service. It allows to reduce the cost of the infrastructure since multiple tenants share the same service instances. However, tenants have dynamic workloads. The resource they share may not be sufficient at some point in time. It may require Cloud resource (re-)configurations to ensure a given Quality of Service. Tenants should be migrated without stopping the service from a configuration to another to meet their needs while minimizing operational costs on the provider side. Live migrations reveal many challenges: service interruption must be minimized and the impact on cotenants should be minimal. In this paper, we investigate live tenants migrations duration and its effects on the migrated tenants as well as the co-located ones. To do so, we propose a generic approach to measure these effects for multi-tenant Software as a Service. Further, we propose a testing framework to simulate workloads, and observe the impact of live migrations on Business Process Management Systems. The experimental results highlight the efficiency of our approach and show that migration time depends on the size of data that have to be transferred and that the effects on co-located tenants should not be neglected.

Introduction

A software service provider that wants to provide its service in the Cloud needs to adjust the Cloud resources it consumes to the needs of his customers. A Web based business application requires the deployment of a software stack including application servers and databases. In order to accommodate multiple customers, the provider can rely on a multi-tenant architecture that allows managing several customers on the same server. Thus, to accommodate all customers with the minimal set of resources, it might be necessary to «move» tenants from one installation to another one. During peak hours, many resources might be required, while at night a single resource could respond to all the requests from all customers. Live migrations can be used to ensure such kind of optimization. It can lead to significant gains in the operational costs. Migrating a tenant from a ressource to another has consequences. It takes time during which the service is not available for the customer. It consumes resources as CPU, networking, and disk IO that may affect the Quality of Service (QoS) for other co-located customers. In this work, we investigate the effects of multi-tenant Web applications live migrations. To the best of our knowledge, our work is the first to focus on evaluating the impact of live migrations on both service interruption time and co-located tenants performances for a service including both application and database servers. To investigate the effects of tenants live migration, we propose an approach to evaluate: (i) the duration of service interruption when a tenant is migrated; (ii) the effects of a live migration on the migrated tenant; and (iii) the effects on the performances of other tenants hosted on both origin and destination resources during the migration. The proposed approach is generic and can be applied for all multi-tenant Web applications sharing a set of resources (computing and/or storage). We apply it to a multi-tenant Business Process Management System (BPMS), representing a classical transactional application. To do so, we propose a performance test framework to investigate the efficiency of our approach in determining the effects of migrations for multi-tenant BPMS. The main design goal of the proposed framework is to create a dynamic testing environment that scales in terms of resources and tenants numbers.

To summarize, the main contributions of this paper are as follows:

-A generic approach to measure the impact of the live migrations in terms of service interruption time and performances (execution time) for multi-tenant Web applications. (Section 3) -A performance test framework for multi-tenant BPMS. Indeed a detailed description of the framework architecture is presented. (Section 4) -An example on how the framework can be used for measuring the effects of live migrations on the Performance edition of the BPMS Bonita 7.4.3 4 . (Section 5)

The rest of the paper is organized as follows. Section 2 describes a motivating example that we use throughout this paper to explain the different steps of the proposed method. Our proposed approach for evaluating migrations effects is introduced in section 3. The test framework for multi-tenant BPMS is presented in section 4. Experiments and empirical results are presented in Sections 5 and 6. Related work, Threats to Validity and Conclusion and Future Work are presented respectively in Sections 7, 8, and 9.

Motivating Example

Live migrations of tenants is important for the service provider. Let's consider a Business Process Management as a Service setting consisting of a BPMS deployed on a Cloud provider infrastructure. The Cloud provider offers a set of computing resources. For performance reasons, a BPMS installation is deployed on a resource that consists of at least two compute instances for both a database and an application server. Each resource is characterized by its price per unit of time as well as a minimum and a maximum tasks throughput per second. Each active BPMS instance is used by customers named tenants. Each tenant has a QoS requirement expressed in terms of BPM task throughput. In Figure 1, the tenant T 4 has an initial BPM task throughput of (40 task/second) at time (t=t0). It goes up to (60 tasks/second) at time (t=t1). The widths of the resources and the tenants correspond respectively to the capacity and the requirements in terms of task throughput. At time (t=t0), the tenants are placed on the different resources. For example, in Figure 1, the tenants T 1, T 2 and T 3 share the first installation deployed on the resource R1. At time (t=t1), the tasks throughput of the different tenants (presented in black) changes, which leads to over and under utilization, respectively of R2 and R3. We reorganize the distribution of tenants on the different resources. An appropriate solution consists in migrating T 4 to R3. Migrations may induce a service disruption on the tenant side and is not instantaneous. The interruption time for T 4, includes its deactivation on the origin installation R2, its migration to R3, and its activation on R3. Further, it may have an impact on the performances of the co-located tenants. Thus, we are interested in providing a way to investigate the effects of the migrations on the service interruption time and the performances of the co-located tenants. The next section presents the proposed approach used to solve the aforementioned challenge.

Measuring the Impact of Migration

In this section we present our approach to evaluate the impacts of tenants live migrations on both service time interruption and tenants performances. First, we present the metrics we use to detect these impacts. After this, we describe the assumptions we make related to the effects of live migrations. Then, we explain the generic setup we propose to do the measurements in a way that can be replicated for Web applications including application and database servers.

To the best of our knowledge, there is no zero-downtime technique to perform tenants migrations from an origin to a destination resource. Further, with regular relational databases, there exist no way to perform live migrations from one resource to another, with a zero-downtime for the tenant. During a migration, the tenant may undergo a limited availability of the services provided by the Web application, causing QoS breaks. We characterize the impact of tenants migrations on the overall software operations, using three metrics.

The migration duration or service interruption time is the time that can be measured between the beginning of the migration on origin resource, when it stops accepting requests and the time when it starts accepting requests on the destination installation.

The second and third metric measures the impact of migrations on performances of tenants hosted on origin and destination resources. We measure the processing time for the migrated tenant and for co-located ones on the origin and destination resources, this during, and after the migration. We name these two metrics migration effects on migrated tenant, and migration effects on co-located tenants.

We want to measure the processing time of the executing operations from the customer and the service provider perspectives. Fulfilling a client request does not imply the termination of the associated operations on the service provider side. A BPMS engine may take additional time to terminate the operation after satisfying the client request as there could be long executing processes running after the answer to the client.

We have several assumptions about the negative effects of migrations on the service interruption time and the performances, that we want to verify and measure using our approach :

-Migrations duration is predictable and increases with the size of the data.

Migrations consist of copying Web application data from the origin resource to the destination resource. More data means more time to copy. -Migration effects on migrated tenant exist and are provoked by the creation of a new tenant on the destination installation. This could have various causes such as negative cache hit or asynchronous libraries initialization. -Migration effects on co-located tenants exist and must be taken into account.

The live migration will have effects on both the origin and destination resource of the migration, and their tenants. As tenants are activated, deactivated, and their data is being read, copied across the network and written on the resources, it may induce performance degradation on both resources executing co-located tenants operations.

In order to measure these metrics, we simulate users interactions with the studied Web application. For each measurement, we observe variations such as the nature of the workload and the quantity of operations initiated in the system.

We store every duration of each launched operation on the Web application as well as of each migration.

We propose an experiment where we first execute a defined workload for a tenant, hosted on the origin resource. We then migrate this tenant from the origin resource to the destination resource, and observe the corresponding migration duration. For each experiment, we store each corresponding time stamp, including the relevant internal steps (such as the beginning and end of the movement of data, the eventual phases of deactivation and reactivation of tenants, etc.).

In order to evaluate the Migration effects on migrated tenant, we measure the performances of the tenant before migration on the origin resource, and after migration on the destination resource. We assume we can retrieve Web application response times, if possible. This is usually stored inside the database of the Web application. It should also be possible to retrieve these metrics from the load testing tool point of view. All the duration of the HTTP queries and corresponding processing durations are timestamped and fully identified so we can compare them with the point of view of the application.

We want to compare the effects of a migration on the other tenants currently hosted on the origin and destination resources. In this case, we propose to initiate workloads on tenants on both origin and destination resource. We also initiate a shorter workload for a tenant on the origin resource who will be migrated after to the destination resource, as in previous experiments. A workload is then executed on the migrated tenant, after the migration.

The Migration effects on co-located tenants are retrieved for the co-located tenants hosted on both origin and destination resources. In order to do this one should retrieve response times of the HTTP queries and corresponding processing durations for tenants. We consider as "query concerned by migration" every query running between the beginning of the migration and the end of the migration, i.e., where the timestamp of the beginning of the query is before the end of the migration, and the end timestamp of the query is after the beginning of the migration. We then compare to other processing durations, when there is the same number of tenants, with the same load and on the same resources. The differences in processing durations will show the effects of the migration on the co-located tenants.

Experiment Framework

In order to validate our approach, we developed a testing framework to study the impact of live migrations for multi-tenant BPMS. According to the type of the performance metric to be investigated, the framework takes as input a set of parameters and provides as output a set of measurements. The Figure 2 depicts the architecture of the proposed framework. We distinguish the following elements:

-The migration method used to move the tenants from one installation to another (out of the scope of this paper). -The load testing components used to generate variable workloads to the System Under Test (SUT). -The SUT represents the BPMS to be studied.

In the rest of this section, we present a detailed description of each component as well as the orchestration of the framework to perform the measurements.

The load testing components (Figure 2) emulates the behaviour of users within tenants interacting with the BPMS.

In the case of a BPMS, we used two different categories of load testing components : a load tester for tenant and processes initialization and BPM agents for BPM tasks retrieval and processing.

The goal of the load tester is the following:

-Initialize all the tenants as well as the users within each one.

-Deploy the BPM processes needed for the experiments, and add the authorizations and roles needed for the proper functioning of the BPMS. -Start process instances according to the specific test requirements, by initializing calls to the HTTP API of the BPMS.

The load tester is implemented relying on the Faban framework5 used for the BenchFlow framework [START_REF] Ferme | A framework for benchmarking bpmn 2.0 workflow management systems[END_REF]. It enables the specification of complex workload, as well as the number of simulated users that are interacting with the SUT, and the duration of the interaction. The Faban framework then takes care of executing the specified load test, and ensure a correct and verified execution of the defined load.

A BPMS tenant has an organizational structure that consists of different groups of users. According to their profiles and roles, users carry out different business activities within the organization. We use a multi agent system (MAS) to model and simulate the users involved in the execution of business activities within each tenant. A MAS consists of a number of agents in a common environment (real/virtual) where they can act and cooperate to achieve system objectives [START_REF] Russell | Artificial Intelligence: A Modern Approach[END_REF]. In our work, we focus on the mapping of the organizational settings within tenants to agents. Using agents on behalf of tenants users provides a suitable means of reproducing human behaviour. Due to their distinct capabilities in terms of autonomy, flexibility, and adaptability agents present effective solutions in modeling and simulating behaviours of human resources. We use agents to imitate human resources in order to: (i) work with process instances including various proportions of human tasks, which are similar to those deployed within real organizations; and (ii) maintain a given threshold of active tasks during tenants migrations to investigate its impact on the migration duration as well as on the co-localized tenants on both origin and target installations.

Each agent represents an active entity that performs specific tasks on behalf of a user within a BPMS tenant. The agent user behaviour consists of the following actions: the agent first connects to the BPMS platform. Then, it starts retrieving the available tasks. If the number of ready tasks is under a given threshold, which represents the sought tasks number before tenant migration, the agent waits until more process instances are started. Else, the agent executes the task after assigning it to itself.

In order to fulfill reproducibility, our framework uses Docker containers. Docker permits easy operating-system-level virtualization. We use it to the launch of the SUT, and for the injection and migration scripts and tools. We used Docker Swarm6 for our cluster management, and Docker Compose for the description of the distributed system. Swarm is a functionality of Docker to manage containers on several nodes. Docker Compose allows to describe a complete stack, which can be composed by several containers having dependencies between them, as the use of the database by the Web application.

We also used Faban and BPM agent Docker Compose files in order to launch them easily, with all the required parameters such as the the names of the users, tenants, and processes.

To use the framework, we must prepare a test descriptor containing the various parameters of the experiments. The parameters can be customized in order to evaluate the behaviour depending on the BPM schema, the BPMS, the duration of the BPM processes injection, etc.

Once a test is launched, the following steps are triggered:

1. Test initialization: creation of unique identifier for the test and experiment directory. 2. Deploy initialization: reset and initialize Docker Swarm on the resources and needed files for the experiment. 3. Deploy BPMS: deploy the BPMS and corresponding databases containers on the resources. 4. Launch test: deploy the Faban load testers, the BPM agents on the concerned resources, and execute the tests for the concerned tenants. 5. Copy results: launches queries on BPMS database in order to get processing time, retrieve results and store them.

In step 4 we launch multiple loads on a set of tenants for a defined duration, named "background tenants" while a loaded tenant is migrated. The tests for the background tenants are launched in an asynchronous manner. A launch of the Faban load tester coupled to the BPM agent is triggered on the studied tenant ("migrated tenant") for a short duration. Once this first step is finished, the framework waits for a defined duration to let all processes finish. Then the migration is triggered. After this migration, a second process injection is launched on the destination resource. Results of the tests can be then retrieved from the BPMS, the BPM agents and the Faban agents.

In step 5, we retrieve the measures. The results consist in the duration of the migration, the response time and the duration of processing of each process and tasks launched during the test. These results have a timestamp stored in CSV file with the identifier of the BPM task and its corresponding process, and their durations.

Experimentation

We explain in this section how we have adapted our framework7 for our use case. As shown in the Figure 2, the SUT is composed of five software elements,including the origin and destination stacks consisting of one BPMS engine and its corresponding database, and a reference database whose goal is to host the archive and system data.

For our tests, we use BonitaBPM 7.4.38 in its commercial Performance edition 9 .

We conducted these tests with different parameters for the origin and target resources, and for multiple processes quantities. We needed a method to migrate tenants, a realistic workload, test scenarios, and test infrastructure. All the experiments characteristics are described in the rest of this section. Live migration of tenants requires a series of steps. BonitaBPM allows stopping and restarting the tenant operations. When a tenant or his underlying operations are in a stopped state, new operations cannot be launched, and current operations are put on hold.

We have used specific Docker Compose file referencing two Bonita Docker containers, two Postgresql 10 Docker containers for the application databases, and one for the reference database. Figure 3 shows the corresponding five containers, which will be generated for each experiment.

We use three different business processes models to investigate the impact of the different workflow structures on the migration duration. The processes are modeled using the graphical standard language BPMN10 . The first model consists of a single human task.The other models represent a combination of human and automatic tasks as well as parallel and exclusive gateways. These models were initially defined and used in [START_REF] Ferme | Estimating the cost for executing business processes in the cloud[END_REF]. The second model, in Figure 4, consists of a set of automatic tasks implemented as script tasks and a single human task. The process starts by initializing an integer number (x = 0), which is incremented by the script task situated after the first exclusive gateway. With respect to the value of the variable (x), the upper and lower branches are followed. The model, shown in Figure 5, has a more complex structure compared to the processes models above. It consists of two human tasks executed in parallel. The process starts by initializing an integer number (x = 0), which will be later incremented within the script task situated after the gateway "endAsk". With respect to the value of the variable (x), the upper and lower branches are followed. To have a deterministic behaviour, the upper path, which contains the last exclusive gateway is executed till (x == 4) before the ending of the process.

In order to evaluate the three metrics presented in section 3, we have conducted two sets of experiments:

-The first experiment evaluates the Migration duration. In this case, we have launched the framework for only one tenant. We executed the load on the origin resource for iteratively 0, 5, 10, 30, 60, 120, 300, 600 seconds for the three types of BPM schemas, and then we performed three migrations of the processes from the origin resource to the destination resource, and vice versa. We observed the total duration of the migration, including its different steps: deactivation of tenant, migration of the data and activation of the tenant. -The second experiment observes Migration effects on migrated tenant and Migration effects on co-located tenants tests. In this case, we have three tenants: one background tenant running on the origin resource (tenant1 ), one background tenant running on the destination resource (tenant3 ). Each one has one Faban agent injecting processes for 20 minutes, while 100 BPM agents close their tasks. A third tenant, tenant2, is launched first for 2 minutes on the origin resource. The framework waits for 5 minutes in order to let every process finish, and to have comparison for Test 3. Then a migration from the origin resource to the destination resource is triggered, and a 2 minutes load is launched for this tenant on the destination resource.

We have conducted the tests on Azure Public Cloud. We used the following instances types:

-Databases: Standard E2s v3 (2 vcpus, 16 GB memory) -3 instances -BPMS: Standard F4s (4 vcpus, 8 GB memory) -2 instances -Faban load tester (Harness and Agent): Standard F1s (1 vcpus, 2 GB memory) -1 or 3 instances respectively for the first and second experiment -BPM Agents: Standard F2s (2 vcpus, 4 GB memory) -0 or 3 instances respectively for the first and second experiment -Orchestrator: Standard B2ms (2 vcpus, 8 GB memory) -1 instance E-series are memory-optimized instances 11 . We used it for the database instances. F-series12 are computing-optimized instances. We have chosen it for the BPMS, the BPM agent and the Faban load tester. The instance we have used for the orchestration and data collection is a small size burstable instance.

Empirical Results and Analysis

In this section, we present the we obtained following our experimentation. Due to space limitations, detailed results are shared here13 Fig. 6. Mean duration vs number of processes Figure 6 shows the duration for: the deactivation of the tenant on the origin resource, the migration of the data, the activation on the destination resource, and the total duration of the migration.

The deactivation of tenants is very fast (less than 1 second) and stable regardless the number of active processes or the BPM schema. The activation of tenants is less stable, and last for a few seconds. The copy of the data seems very linear. It is the fastest for the smaller schema, TestHumanTask (about 20 seconds for 10000 processes), then longer for the schema addtionalApproval (more than 30 seconds), and even longer for the M3Process schema. All durations stay the same for 0 processes, and there is some variability, which may be linked to the uncertainties of Cloud behaviour. Figure 7 provides details on the duration of the deactivation and activation of tenants compared to the number of invocation. The duration is stable for the deactivation, between 0.05 and 0.4 seconds for all the experiments. This is not the case for the activation corresponding to the first migration. It always last for more than 3 seconds when most other last around 1.5 seconds. The second migration has a higher duration than the rest. Apart of some longer experiments, which are occasional, the behaviour of both activation and deactivation is similar regardless the BPM schema, and the number of migrated processes.

Duration vary between 5 seconds for no active processes to about 30 seconds for about 10 thousands processes for the TestHumanTask processes. The process comprised of a parallel tasks has a longer duration: 50 seconds for more than 10000 injected processes, this is about 20 seconds more than for sequential processes. This is probably caused by the migration of two active tasks instead of only one for other processes. This shows that in order to evaluate the duration of live migrations, one must consider the BPM schema structure. The duration of the migration of the schema M3process is also a bit longer than the duration of the HumanTask. The presence of numerous automated tasks and gateways could provoke the storage of additional data for this process.

We have seen in the last part that the duration of the activation part of the migrations is longer for the first migrations. The reader must remember that in this experimentation we have launched iteratively 6 migrations, 3 from the origin to the destination and 3 for the destination to the origin, using the same tenant. The probable explanation here is that the first activation of the tenant in a resource needs some initialization in the tenant's metadata (such as library, cache initialization, filesystem based resources, etc.), which do not exist yet in the filesystem or memory of the target resource's installation when it is migrated for the first time . This behaviour happens only one time, and the duration of the activation part stays stable after this first. Now, we present the results for processes durations on the origin resource, and on the destination resource of a migrated tenant. The Table 1 shows the general statistics depending on the observed schema, and when the results have been retrieved in the experiment(before -pre-or after -post-the migration). The mean duration of TestHumanTask process is about 1300 milliseconds shorter after the migration compared to before, but 1500 milliseconds longer for the M3Process schema and 2700 milliseconds for the AdditionalApproval process. The standard deviations are similar except for the TestHumanTask process. We obtain slower mean durations for M3Process and additionalApproval processes after the live migration, and faster mean duration for the TestHumanTask process. At the time of writing, we don't have rational explanation for that.

We compare in figure 8 the duration of processes from the BPM provider perspective (left), and task duration from the BPM agent perspective (right). On both figures, the blue (left) boxplot for each number of active processes is the duration when no migration is executed, and the orange (right) boxplot is the duration during the migration.

There are a lot of variation probably due to the Cloud behaviour, and to some non deterministic effects after the live migrations. Some processes last for more than 200 seconds, up to 1000 seconds. Even with these effects, the duration of both processes and tasks are positively correlated to the number of active processes. Durations are a bit longer and less stable during migrations.

In order to have a more homogeneous comparison, we removed from the data the processes having a duration superior to 200 seconds and tasks superior to 15 seconds. The table 2 shows the corresponding results for the BPM tasks response time, aggregated by tenant. The table shows that the duration is always longer during migrations, both for the tenant on the origin resource (tenant1) and on the destination resource (tenant3). The difference in duration is between 26 and 650 milliseconds. We cannot find correlation with the BPM schema. The performances of tenant1 and tenant3 are similar, sometimes faster, and sometimes slower.

As in the previous experiments, additional process is much longer than M3Process, which is longer than human task. The same behaviour occurs for the composing tasks at a much smaller scale.

The effects of the live migrations on the co-located tenants are not important but they exist. We observe a few hundred more milliseconds for each type of BPM schema. It is higher for the more complex BPM schemas than for the simpler. This is the same when we compare the effects on the tenant of the origin resource to those on the destination resource. Migrations have more effects on the AdditionalApproval process destination resource than on the origin resource. This will require more investigations. The results we obtain are similar for the process duration.

Related Work

Performance is a major concern for multi-tenant service providers. In the literature, performance evaluation for multi-tenant environments is tackled from different perspectives including evaluation of tenant placement [START_REF] Taft | Step: Scalable tenant placement for managing database-as-a-service deployments[END_REF][11], live migrations [START_REF] Elmore | Zephyr: Live migration in shared nothing databases for elastic cloud platforms[END_REF][2] and the suitability of the sharing approach [6][8]. The target applications are mainly multi-tenant databases, whereas in our work, we focus on evaluating the tenants migration effects for a Web application including both application and database servers. Initially, the live migration is used for Virtual Machines [START_REF] Clark | Live migration of virtual machines[END_REF]. The approach consists in moving a virtual machine and its dependencies from one hardware resource to another without being shutdown. The aim is to decommission the hosting of resources, which are not needed anymore, while minimizing the downtime and the negative effects. In [START_REF] Krebs | Multi-tenancy performance benchmark for web application platforms[END_REF], the authors extend the TCP application for benchmark to support multi-tenant platforms. The performance evaluations focus on determining the maximum throughput as well as the tenants number supported by the platform, which can be used later to provide insights about the suitable sharing approach. Although, in our work, the number of the tenants as well as their requirements in terms of throughput are known in advance, our main focus is on investigating tenants live migrations impacts on duration and response time, which is not supported in [START_REF] Krebs | Multi-tenancy performance benchmark for web application platforms[END_REF]. In [START_REF] Taft | Step: Scalable tenant placement for managing database-as-a-service deployments[END_REF], the authors present the STeP framework for scheduling multi-tenant databases on the suitable resources. SteP provides a new set of tenants packing algorithms to optimize both static and dynamic resource allocation. Further, it uses a set of metrics including performance objective violation, the operations cost and the monetary penalty to determine the efficiency of the placement approach. In our work, we focus on the impact of the dynamic placement and its effects on the tenants where the authors consider only scheduling. In [START_REF] Liu | Pmax: tenant placement in multitenant databases for profit maximization[END_REF], the authors present a profit-driven tenant placement strategy for multi-tenant databases. Similar to [START_REF] Taft | Step: Scalable tenant placement for managing database-as-a-service deployments[END_REF], a set of placement algorithms are proposed, which were evaluated based on operations cost and the SLA penalty costs. Although the authors discuss the suitable deployment of multiple tenants, the migration of tenants and its impacts are kept as a future extension of their work. In [START_REF] Lang | Towards multi-tenant performance slos[END_REF], the authors propose a framework that furnishes policies for hardware provisioning and tenants scheduling according to the tenants classes and their performance SLOs. While in our work, we assume we know in advance the required resources to deal with the evolving requirements of tenants and we provide an approach to measure the impact of the live migration in terms of duration and response time.

Threats to Validity

The proposed approach has different limits, that we tried to mitigate when possible:

-We have tested the effects of co-located tenants only for small numbers of active processes (0 to 200). We need to test with more active processes. -Using Public Cloud resources induces variability problems. As we have seen in the results, even when executing multiple tests we still have many outliers, sometimes on a whole experiment. The only solution is to launch even more experiments. We plan to make a more intensive analysis of migration for a next iteration, and the framework will help us in this task. -We have not studied the effects of multitenancy on the performance. In this case we have compared results for a fixed number of tenants. Indeed, there could be additional operations related to the management of tenants, but non related to the workload. Response time could not be totally proportional to the injection if there are more tenants. We plan to study this in a future work. -The results concern only one BPMS and one live migration method. We plan to test with more BPMS from different vendors. -In its current implementation, the framework scales in the number of Faban/BPM agents. However, when we tried to scale up the number of Faban load tester instances, we had issues with the parameters. The SUT should to be tuned in order to take advantage of this (resource-wise, and parameterwise). In this experiment, we have tuned the SUT in the same way for all the experiments. We plan to execute tests with different configurations of the parameters and test different resource configurations in future work.

Conclusion and Future Work

In this paper, we presented two contributions: (i) a generic approach to measure the impact of live migrations in terms of service interruption and performances evaluation for Web applications; and (ii) a performance test framework for multitenant BPMS. We present an example on how the framework can be used for measuring the effects of live migrations using a well known BPMS. The analysis of the experimental results showed that the duration of a migration depends linearly on the size of the active data and that the effects on the co-located tenants cannot be neglected. Indeed, live migrations may last for long and providers should take them into account and study their effects before on their applications if they want to guarantee the same QoS. This fact could included in the migration decision of elasticity algorithms. Further, our framework can be easily used for other Web applications or other live migration methods with the appropriate adaptation mentioned in the paper. As a future work, we plan to perform deeper analysis on the effects of live migrations on co-located tenants, for various number of tenants, live migration methods, and Web applications in order to have better view on their QoS impacts. We also intend to enhance our framework, by improving the BPM agent component through the modeling and simulation of more advanced human behaviours. Our work allows to have a better view on live-migrations effects and pinpoint important criteria Software as a Service (SaaS) providers should not underestimate. It can be used to evaluate deployment decision and to parameterize elasticity algorithms when QoS constraints are very strong.

Fig. 1 .

 1 Fig. 1. Tenants Distribution at Time t = t0 and t = t1

Fig. 2 .

 2 Fig. 2. The framework architecture for studying the impact of live migrations on multitenant BPMS

Fig. 3 .

 3 Fig.3. Proposed database architecture. Software uses a relational database. A "reference" database for system-related and archive data is used. External tables are used for the links between the reference database and the database used by the Web application.

Fig. 4 .Fig. 5 .

 45 Fig. 4. AdditionalApproval BPMN schema

Fig. 7 .

 7 Fig. 7. Duration distribution vs order of launch

  .8 15594.2 466.0 2969.5 12282.0 19553.5 168560.0 AdditionalApproval post 17710.0 113370.0 42819.4 22915.0 78601.0 116355.0 146674.0 199938.0 AdditionalApproval pre 25749.0 110685.8 42710.7 12907.0 83799.0 114749.0 141261.0 199856.0

Fig. 8 .

 8 Fig. 8. Process duration -BPMS point of view-distribution (left) and task duration -BPM agent point of view-distribution (right) vs process schema and number of active processes.

  .6 2359.1 398.0 1313.0 2152.0 3484.00 14949.0 AdditionalApproval tenant3 False 19530.0 3050.3 2109.8 354.0 1458.0 2255.5 4110.00 12673.0 AdditionalApproval tenant3 True 8299.0 3702.0 2349.8 558.0 1751.0 2935.0 5521.00 12882.0

Table 1 .

 1 Duration of tasks grouped by BPM schema and moment compared to the migration.

	BPM schema	When Count Mean Std	Min	25%	50%	75%	Max
	M3Process	post 10292.0 38449.2 16217.

Table 2 .

 2 Duration of tasks during migration compared to before grouped by BPM schema and tenant

	BPM schema	Tenant	Migration running	Count Mean Std Min 25% 50% 75%	Max

http://www.bonitasoft.com

http://faban.org/about.html

https://docs.docker.com/engine/swarm/

https://github.com/guillaumerosinosky/migration_bpms

http://www.bonitasoft.com

Multi-tenancy is only available in this commercial licence.

Business Process Modeling Notation

Standard memory optimized instance: https://docs.microsoft.com/en-us/azure/ virtual-machines/linux/sizes-memory

Standard computing optimized instance: https://docs.microsoft.com/en-us/azure/ virtual-machines/linux/sizes-compute

http://dx.doi.org/10.5281/zenodo.1402632

Acknowledgments

This work has been partly supported by the German Research Foundation (HO 5721/1-1, DECLARE), and by the Swiss National Science Foundation (project no. 178653). This work has been supported by Azure Research Grant. We thank heartfully Bonitasoft without whom this analysis could not have been done.