Oriane Dermy

Armelle Brun

Can we Take Advantage of Time-Interval Pattern Mining to Model Students Activity?

Keywords: Students behavioral patterns, time-interval pattern mining, interval granularities, sequential pattern mining

Analyzing students' activities in their learning process is an issue that has received significant attention in the educational data mining research field. Many approaches have been proposed, including the popular sequential pattern mining. However, the vast majority of the works do not focus on the time of occurrence of the events within the activities. This paper relies on the hypothesis that we can get a better understanding of students' activities, as well as design more accurate models, if time is considered. With this in mind, we propose to study time-interval patterns. To highlight the benefits of managing time, we analyze the data collected about 113 first-year university students interacting with their LMS. Experiments reveal that frequent time-interval patterns are actually identified, which means that some students' activities are regulated not only by the order of learning resources but also by time. In addition, the experiments emphasize that the sets of intervals highly influence the patterns mined and that the set of intervals that represents the human natural time (minute, hour, day, etc.) seems to be the most appropriate one to represent time gap between resources. Finally, we show that time-interval pattern mining brings additional information compared to sequential pattern mining. Indeed, not only the view of students' possible future activities is less uncertain (in terms of learning resources and their temporal gap) but also, as soon as two students differ in their time-intervals, this difference indicates that their following activities are likely to diverge.

INTRODUCTION

The wealth of data that can be collected from a Learning Management System (LMS), mainly the logs of students' interactions with learning resources, provide opportunities to get a more comprehensive understanding of students learning process: point out engaged or at-risk students, identify the most commonly studied or the most difficult resources, highlight recurrent students' activities, etc. In addition to this thorough understanding, inferences or decisions can be drawn: estimate students outcome, predict students future behavior (including dropout), personalize learning by providing students with information or recommendations, etc. To carry out such understanding, inference or decision, data mining methods have been applied. Pattern mining, that discovers frequent patterns of events in data, is one of these methods and is also used in a large number of application fields. Sequential Pattern Mining (SPM) consists of discovering patterns when data is sequential in nature. These patterns, named sequential patterns, are frequent ordered sequences of events. In the educational field, a sequential pattern often represents a recurrent sequence of learning resources, that we call an activity [START_REF] Romero | Educational data mining: A survey from 1995 to 2005[END_REF][START_REF] Cerezo | Different patterns of students interaction with mooodle and their relationship with achievement[END_REF].

The time of occurrence of events is often part of the data to be mined. However, in most of the cases, the patterns mined do not contain temporal information. Nevertheless, the literature has introduced different ways of including such information in patterns. We can, for example, cite temporal patterns, made of events that are associated with their time of occurrence [START_REF] Yoshida | Mining sequential patterns including time intervals[END_REF], their duration [START_REF] Chen | Mining temporal patterns in time interval-based data[END_REF], or the time gap between the events. In [START_REF] Chen | Discovering time-interval sequential patterns in sequence databases[END_REF], gaps between events are grouped into intervals, resulting in time-interval sequential patterns. Since a time-interval pattern conveys more information than its corresponding sequential pattern, they are still the focus of research works [START_REF] Sumalatha | Distributed mining of high utility time interval sequential patterns using mapreduce approach[END_REF]. In the rest of the paper, time-interval patterns will be referred to as ti-patterns and sequential patterns to as s-patterns.

We think that ti-patterns are adequate to represent students' activities. Indeed, it is rare that two students perform exactly the same activities, in both learning resources and time, even though they share underlying sequential activities. To the best of our knowledge, no work in the field of educational data mining has focused on the mining of tipatterns. In this work, we thus rely on the hypothesis that mining ti-patterns will contribute to a better view and understanding of students' learning activities. These patterns do not only indicate in which order students interact with learning resources, but provide also information about the temporal relationship between these resources. For example, let us consider that students tend to interact sequentially with two resources, each of them being lecture slides. The sequence of both resources represents a sequential activity. Suppose that mining ti-patterns highlights that the time gap between both resources tends to be less than 1 minute for some students and between 2 and 4 hours for others. We can thus deduce beyond this sequential activity that there are two typical behaviors. To support our hypothesis, we will conduct a study to evaluate if ti-patterns can be actually identified from students' activity data and evaluate to what extent ti-patterns provide additional information about students' activities.

In the following sections, we will first present an overview of related works on sequential and temporal pattern mining (Section 2). We then present the methodology we adopt to support the hypothesis that we draw (Section 3). Section 4 details the experiments we conduct on a real dataset and presents some ti-patterns. The last sections discuss the results (Section 5), then conclude the work and present our expected future work (Section 6).

LITERATURE REVIEW 2.1 Sequential Pattern Mining (SPM)

Sequential Pattern Mining is a popular task in Data Mining, introduced by Agrawal and Srikant in [START_REF] Agrawal | Mining sequential patterns[END_REF]. SPM aims to discover frequent sequential patterns in sequential databases. A sequential database D is a set of tuples D = {(sidi, di)}, where sidi is the unique identifier of a sequence, and di an input sequence. A sequence is an ordered list of events: s = E1E2 . . . Ex , with Ei ∈ E the set of events. We also say that β is a super-sequence of α, or that it contains α. Let us now define what the support of a sequence is. The support of α, noted supp(α), is the number of sequences in the sequential database D that contain α. Based on both definitions, we can now define that a sub-sequence α is a frequent sequential pattern, if supp(α) δ, for a defined minimum support threshold δ. We define SP as the set of frequent sequential patterns.

Many SPM algorithms have been proposed in the literature. The most commonly cited ones are GSP [START_REF] Srikant | Mining sequential patterns: Generalizations and performance improvements[END_REF], PrefixSpan [START_REF] Pei | Prefixspan: Mining sequential patterns efficiently by prefix-projected pattern growth[END_REF], SPADE [START_REF] Zaki | SPADE: an efficient algorithm for mining frequent sequences[END_REF]. All these algorithms use the "apriori property": "If a sequence s is not frequent, then non of the super-sequences of s is frequent.". Thus, when one pattern is infrequent, it is not extended. Algorithms can be divided into two main approaches. Apriori-like algorithms (also called breadth-first search algorithms), such as Generalized Sequential Pattern Mining (GSP) algorithm [START_REF] Srikant | Mining sequential patterns: Generalizations and performance improvements[END_REF], are the first algorithms that have been proposed. However, these algorithms suffer from scalability problems, mainly due to memory requirements. Depth-first search algorithms, which include pattern-growth algorithms, do not suffer from memory complexity, which explains their popularity. For a couple of years, the most common SPM algorithm is the Prefix-Projected Sequential Pattern Growth (PrefixSpan) algorithm [START_REF] Pei | Prefixspan: Mining sequential patterns efficiently by prefix-projected pattern growth[END_REF], which is a pattern-growth algorithm, that relies on projected databases. Projected databases generally reduce the research space as the size of the projected databases decreases at each iteration. However, the main cost is linked to the generation of these projected databases. The pseudo-code of PrefixSpan is presented in Algorithm 1.

Algorithm 1 PrefixSpan (α,l,D) 1: Inputs: 2: α: a sequential pattern and l its length. call PrefixSpan(α ,l + 1,D| α ) 14: end for In the works mentioned below, both the database and the patterns are sequential. However, in some cases, the database can be temporal, i.e. contain information about the time of occurrence of the events. In these cases a sequence is defined as: s = (t1, E1), (t2, E2), . . . , (tn, En) . where (ti, Ei) represents an event Ei and its time of occurrence ti.

When sequential patterns are mined from these databases, time can be either used as an information or order between events, such as in SPADE [START_REF] Zaki | SPADE: an efficient algorithm for mining frequent sequences[END_REF]. The time of appearance of events can also be used as a constraint. For example, in [START_REF] Hirate | Generalized sequential pattern mining with item intervals[END_REF] the authors consider that when two consecutive items in a sequence are separated by a time gap bigger than a predefined threshold, they are temporally too distant to represent an association that makes sense. In the same context, [START_REF] Sato | Text mining using prefixspan constrained by item interval and item attribute[END_REF] discards uninteresting patterns by introducing an interval constraint between items.

Sequential Patterns Mining in EDM

Sequential Pattern Mining has been extensively used in Educational Data Mining. They are mainly used to identify frequent patterns of students' activities [START_REF] Gutierrez-Santos | Sequence detection for adaptive feedback generation in an exploratory environment for mathematical generalisation[END_REF][START_REF] Poon | Mining sequential patterns of students' access on learning management system[END_REF], including those that maximize the student learning performance [START_REF] Doroudi | Sequence matters, but how exactly? A method for evaluating activity sequences from data[END_REF]. In [START_REF] Kinnebrew | A contextualized, differential sequence mining method to derive students' learning behavior patterns[END_REF] SPM is used to study the differences in students' productive and unproductive learning behaviors and thus identify high versus low performing students. A similar objective has been studied on group work systems to understand the success factors in groups behavior [START_REF] Perera | Clustering and sequential pattern mining of online collaborative learning data[END_REF][START_REF] Maldonado | Analysing frequent sequential patterns of collaborative learning activity around an interactive tabletop[END_REF]. SPM is also used to detect learning problems early, such as in [START_REF] Kay | Mining patterns of events in students' teamwork data[END_REF] where frequent sequential patterns and flag interaction sequences that are indicative of problems are mined.

One step further, SPM can act as a first step in decision making. In [START_REF] Chen | Discovering prerequisite structure of skills through probabilistic association rules mining[END_REF], the prerequisite structure of skills is find out, by identifying relations between variables from data. The algorithms developed in [START_REF] Poon | Mining sequential patterns of students' access on learning management system[END_REF][START_REF] Tarus | A hybrid knowledge-based recommender system for e-learning based on ontology and sequential pattern mining[END_REF][START_REF] Fatahi | An empirical study of using sequential behavior pattern mining approach to predict learning styles[END_REF] provide students with personalized recommendations of learning resources according to their current activity or their learning style. A complete view of various approaches used in educational data mining is presented in [START_REF] Anjum | A study of different techniques in educational data mining[END_REF].

Temporal Pattern Mining

Temporal information appears to be fundamental in many contexts, hence the number of works interested in the mining of patterns that contain temporal information. Here again, time information can be used in several ways and for different goals: gaps, duration, intervals, etc. Time information is often used as a gap between events of a pattern. For example in [START_REF] Yoshida | Mining sequential patterns including time intervals[END_REF], the author considers that each occurrence of a sequential pattern may have different, but close, temporal elements. So, they propose to associate each pair of events of a pattern with a minimal, a mean and a maximal gap values between these events. The resulting model is made of sequential patterns enriched with temporal information, called delta patterns. Similarly, [START_REF] Giannotti | Mining sequences with temporal annotations[END_REF] proposes to add temporal information to each pair of events in a sequential pattern. This information, referred to as an annotation, represents a typical gap value between each pair of events of a pattern. In this work, the acceptance of the variation around this typical gap value is automatically evaluated. At the opposite of the previous works, [START_REF] Vautier | An inductive database for mining temporal patterns in event sequences[END_REF] predefines a maximal gap value between events of a pattern, which results in temporal patterns called chronicles. [START_REF] Kitakami | Modified prefixspan method for motif discovery in sequence databases[END_REF] introduces an even more constraining frame, the exact gap interval value is imposed. This approach results in a decrease in the support of each pattern. Thus, the number of extracted patterns decreases. In addition to the gap value, [START_REF] Guyet | Mining temporal patterns with quantitative intervals[END_REF] exploits the duration of events. Each element of a pattern is composed of the event, associated with its begin and end timestamps. They propose an Apriori-like algorithm, that uses a hypercube representation of temporal sequences. More recently, [START_REF] Fricker | Sequential pattern mining of multimodal data streams in dyadic interactions[END_REF] introduces an Apriori-like temporal pattern mining algorithm on multi-modal data streams. At the opposite of the previous works, they do not only use the time gap between events (that represents the duration of the event), but also use the exact starting time of each event.

In line with the works presented above, [START_REF] Chen | Discovering time-interval sequential patterns in sequence databases[END_REF] also manages gap values between events, that are grouped into intervals. At the opposite of other works, the intervals of gap values are predefined, and form "time-interval sequential patterns". A time-interval sequence is defined as:

α = E1τ1E2τ2 . . . τ l-1 E l
where Ei ∈ E is the set of events for 1 ≤ i ≤ l and τi ∈ T I the set of time-intervals. The sequence α is a timeinterval pattern if supp(α) ≥ δ. We note T P the set of frequent time-interval patterns of a database D. In their article, the authors propose two algorithms called I-Apriori and I-prefixSpan, and results show that I-PrefixSpan outperforms I-Apriori both in computing time and scalability. The pseudo-code of the I-PrefixSpan algorithm is presented in Section 2.

A few years later, [START_REF] Hu | On mining multi-time-interval sequential patterns[END_REF] pointed out that most algorithms of the literature use time information only as a time constraint or to represent the time-interval between successive items [START_REF] Chen | Discovering time-interval sequential patterns in sequence databases[END_REF]. The novelty of this work is that not only the delay between successive items is taken into account, but also between distant items. The "multi time-interval (MI) sequential pattern" models the time-intervals between all pairs of items within a pattern. Two algorithms have been proposed, Discovering time-interval patterns has attracted considerable efforts, due to its widespread applications. However, several challenges remain, such as the definition of the adequate set of intervals (whether manual or automatic), including the problem of the granularity of the intervals.

Temporal Granularities

As soon as intervals are introduced, an issue arises: how to choose these intervals? [START_REF] Bettini | Discovering frequent event patterns with multiple granularities in time sequences[END_REF] proposes to manage different temporal granularities. An algorithm composed of Timed Automata with Granularities (TAGs), associated with heuristics is proposed. TAGs test whether a candidate time pattern appears frequently in a time sequence. The heuristic allows to reduce the number of candidates. [START_REF] Rashidi | Keeping the resident in the loop: Adapting the smart home to the user[END_REF] focuses on mining periodic patterns, where interesting periods cannot be defined in advance. Two temporal granules are proposed: a fine-grained granule for hourly periods and a coarse-grained granule for daily periods. The time distribution of different time granularities is then estimated by using a combination of Gaussian distribution.

Temporal Data Mining in EDM

To the best of our knowledge, little use has been made of Temporal Pattern Mining in the EDM field. [START_REF] Krauss | The timeliness deviation: A novel approach to evaluate educational recommender systems for closed-courses[END_REF] takes time into account by evaluating the rate at which students change the learning resources of interest. They progressively improve "when" resources have to be recommended to the student. In a learning context, where students can choose both which and when courses and exams to take, the research work presented in [START_REF] Campagni | Data mining models for student careers[END_REF] uses time information that corresponds either to the "semester in which the exam was taken" or to the "delay with which it was taken". Using this time information, they then study the course and exam schedule that the students take and understand better students' behaviors. Using clustering and comparison, they are then able to suggest improvements to the scheduling of courses and exams of students.

DEFINITIONS AND METHODOLOGY

The previous literature review highlights that time-intervals are mainly adopted to model temporal patterns. The algorithm proposed in [START_REF] Chen | Discovering time-interval sequential patterns in sequence databases[END_REF], I-PrefixSpan, has the main advantage to consider intervals as a core element of the patterns and the mining process. Intervals are considered as a constraint about the patterns, not as supplementary information about the patterns. It is the main reason why we choose to adopt this algorithm in our work. We start by introducing definitions that will be used in the following methodology and in the experiments.

Definitions

Let p = E1τ1E2 . . . τn-1En and p = E 1 τ 1 E 2 . . . τ m-1 E m be two ti-patterns and s = E 1 E 2 . . . E l be a s-pattern, with n (resp. m and l), the length of the pattern p (resp. p and s). Given these patterns, we put the following definitions. Recall that T P is the set of frequent ti-patterns and SP the set of s-patterns. Definition 3.1. ti -form of an s-pattern p is a ti-form of s, denoted by isf orm(s, p) if and only if:

(n = m) ∧ (Ei = E i), ∀i ∈ [1 : n]. ti-form(s) is the set of ti-forms, in T P , of s. Definition 3.2. s-form of a ti -pattern s is a s-form of p if and only if: (n = s) ∧ (Ei = E i), ∀i ∈ [1 : n]. s-form(p) is the (unique) frequent s-form, in SP , of p.
s-form(P) is the set of frequent s-forms (in SP ) of the set of ti-patterns p ∈ P . Definition 3.3. s-equivalence of ti -patterns p and p are s-equivalent, denoted s-eq(p, p ) if and only if:

(n = m) ∧ (Ei = E i), ∀i ∈ [1 : n].
In other words, s-form(p)=s-form(p ). Definition 3.4. Prefix of a ti -pattern p is a prefix of p if and only if:

(m < n) ∧ (Ei = E i) ∧ (τi = τ i ), ∀i ∈ [1 : m].
Definition 3.5. Extension of a pattern p is an extension of p if p is a prefix of p . We note ext(p) the set of extensions of p that belong to T P . A similar definition can be put for s-patterns. Definition 3.6. Extended part of a pattern Let p be an extension of p. The extended part of p, with respect to p , is the pattern p , where concat(p, p ) = p . Thus, p = Enτ n E n+1 . . . τ m-1 E m . We note extP art(p) the set of extended parts of p, i.e. the set of patterns that, when concatenated with p, result in a pattern that belongs to T P . A similar definition can be given for s-patterns. Example: Let p = e1I1e0 , and p = e1I1e0I2e1 be two ti-patterns. p = e0I2e1 is an extended part of p. Definition 3.7. Pseudo-equivalence of ti -patterns p and p are said to be pseudo-equivalent, if and only if:

s -eq(p, p ) ∧ (τn = τ n ) ∧ (τi = τ i ), ∀i ∈ [1 : n -1]
., i.e. they differ only in their last time-interval.

Methodology

To support our hypothesis and identify the actual value of a ti-pattern model, we define a methodology. More precisely, this methodology aims at identifying if there actually are temporal regularities between students' activities, if managing temporal activities allows to have a better view of students' future activities, and concretely what type of activities are mined. Recall that mining ti-patterns is quite new in educational data mining.

We intend to mine ti-patterns in a temporal database D, which is a database made up of temporal sequences. A temporal sequence is an ordered list of events (concretely a list of resources students interacted with) and their associated timestamp. Each temporal sequence represents one student's temporal activities and each student is represented by a unique (and long) sequence. Our methodology relies on four steps, described hereafter.

Determining the set of time-intervals

Recall that although timestamps are discrete values, their precision is so high that relying on time-point (or gap) patterns will probably only lead to infrequent patterns. For example, two sequences that only differ by one second: (0, E1) (3, E2) and (0, E1)(4, E2) will correspond to two different patterns. Grouping gaps to form ti-patterns, will increase the support of patterns. In addition, if the intervals are appropriate, the loss of precision about temporal activities will be limited. So, before assessing the relevance of mining ti-patterns, we have to choose the adequate set of time-intervals. Indeed, this set influences the information conveyed.

Let T I = {I0, I1, ...It} be a set of time-intervals, where Ij = [gapminj; gapmaxj[ is an interval that contains all gap values between gapminj and gapmaxj. Notice that the set of intervals should represent a continuum of gap values from gapmin0 to gapmaxt.

We propose to evaluate the quality of a set of intervals T I with 3 criteria: The fitting ratio. It is the ratio between the number of non-empty intervals and the total number of intervals. A non-empty interval is an interval that is part of frequent patterns. The higher the ratio, the better the set of intervals, as the number of "useless" intervals is low. The number of intervals. On the one hand, the more intervals, the higher the potential of the model. Notice that when T I = {I0} = [0 : +∞[, it comes down to PrefixSpan. On the other hand, using too many intervals increases the complexity of the model. In addition, as there are many intervals, the ti-patterns discovered will probably be infrequent. Thus, a good set is a set that has an in-between number of patterns. The horizon. It is represented by T I, the upper bound of the last interval (the maximal time value of the set of intervals). The larger the horizon, the more complete the model, as it is able to represent long-term recurrences.

From our point of view, the best set of intervals is the one that maximizes the fitting ratio while having a large horizon, with a limited number of intervals.

Comparing sets of s-patterns and ti-patterns

After having fixed the set of intervals, the set T P of tipatterns can be mined. In this second step, we aim at comparing the set T P with the set SP set of s-patterns, and propose some measures to perform this comparison.

First of all, we propose to study the number of patterns, and their average length, to get a coarse-grained view of the set of patterns. Of course, this measure cannot be used alone, as the goal is definitely not to mine the highest number of patterns. Second, we study the correspondence between both sets of patterns. Let us start by noticing that the number of tipatterns cannot be deduced (not even approximately) from the number of s-patterns. A brief explanation follows.

Let s be a frequent s-pattern and ti -cand(s) = {ts1, ts2, . . . , ts k } the set of the candidate-ti-forms of s. Note that tsi may be infrequent. Two cases arise: -|ti -cand(s)| = 1. This case occurs when all the occurrences of s have the same candidate ti-form ts. Here, supp(ts) = supp(s), thus ts is also frequent. The corresponding set of s-patterns is noted S 1 .

-|ti -cand(s)| > 1. This case occurs when some occurrences of s have different candidate ti-forms. As a consequence, ∀i, (supp(tsi) < supp(s))∧ ( k i=1 supp(tsi) = supp(s)). Here, come three possibilities: -tsi, supp(tsi) > δ: there exists no frequent ti-form of s, thus the number of frequent patterns decreases. The associated set of patterns is noted

S 0 . -∃! tsi, supp(tsi) > δ, thus: ∀j|{(1 ≤ j ≤ |ti -seq(s)|) ∧ (j = i)}, supp(tsj) < δ.
In this case, there exists a unique frequent ti-form of s, the number of patterns remains stable. -∃(i, j), (i = j) ∧ (supp(tsi) > δ) ∧ (supp(tsj) > δ). In this case, there exist several frequent ti-forms of s, the number of patterns increases. The set of patterns associated with both last cases is noted S 1+ . Based on this, we first introduce the pattern loss measure, that represents the ratio of s-patterns that have no ti-form in T P (s ∈ S 0 ).

pLoss(SP ) = |SP | -| p∈T P s-form(p)| |SP | (1)
To complete the pattern loss measure, we define the support loss measure, which applies for any s-pattern that has at least one frequent ti-pattern (s ∈ S 1+ ). The support loss measure evaluates the proportion of "lost" occurrences of s, i.e. that have no correspondence in T P .

Let s be a s-pattern and P = {p1, p2, • • • , p k } be a set of ti-patterns, where isf orm(s, pi), ∀pi ∈ P . The support loss of s is defined in equation ( 2).

sLoss(s) = supp(s) -supp * (P ) supp(s) (2)
where supp * (•) is the support of a set of patterns, defined in equation [START_REF] Bettini | Discovering frequent event patterns with multiple granularities in time sequences[END_REF].

supp * (P ) = | p∈P Seq id(p)| ≤ p∈P |supp(p)| (3)
where Seq id(p) is the set of sequence ids in D, where p is a subsequence. We can see that the support of P is not defined as the sum of the supports of the patterns in P . To explain this, let us consider P = {p1, p2}, with p1 and p2 two s-equivalent ti-patterns.

By definition, the s-form of p1 (which is the same as the s-form of p2) occurs at most once in each sequence of D.

Similarly, p1 and p2 occur at most once in each sequence, but both can occur in the same sequence. As a consequence, the support of P may be lower than the sum of the supports of p1 and p2.

The support loss defined above applies for a s-pattern. If the support loss has to be evaluated on a set of patterns, the average support loss and the associated standard deviation can be used.

Evaluating the impact of time on the set of possible future activities of students

In the following third and fourth steps, we aim to evaluate the benefit brought by time in patterns (through ti-patterns) about the possible future activities of students. To perform this evaluation, we adopt a two-stage approach.

Let p be a ti-pattern and extP art(p) the set of extended parts of p (see Def. 3.6). From the educational point of view, the set of extended parts of a ti-pattern p represents the ti-activities that students frequently do after p.

In this third step, we aim at discovering if managing time allows to reduce the uncertainty about the future activities of students. We compare the set of extended parts of spatterns and the set of extended parts of their ti-forms.

To conduct this comparison, we propose to use the wellknown entropy measure. The entropy of a pattern p represents the "degree of disorder" of the set of its extended parts.

From the educational view, given an activity performed by students, the entropy measures the uncertainty of its following activities. The higher the entropy, the more uncertain the following activities. Relying on the entropy is not new in the educational field [START_REF] Zhu | Uncovering the sequential patterns in transformative and non-transformative discourse during collaborative inquiry learning[END_REF]. Equation (4) presents the way the entropy of a ti-pattern p is evaluated.

Ent(p) = - m j=1 prob(pj)log2(prob(pj)), (4)
with prob(pj) = supp(p j ) m k=1 supp(p k ) and pj is one of the m extended parts of p. The same equation stands for s-patterns. Given a s-pattern s, we thus propose to evaluate the benefit of considering time-intervals in this pattern, by evaluating the entropy loss (see Equation 5). Entropy loss of an s-pattern s considers the entropy of s (Ent(s)) and the maximum entropy of its ti-form.

eLoss(s) = Ent(s) -max p∈ti-form(s) {Ent(p)} Ent(s) (5)
Several cases may arise. First, eLoss = 1. This represents the best case: each of the ti-forms of s has exactly one extension. This means that when managing time in patterns, the future activities are totally certain. Second, eLoss = 0.0. This case represents one of the worst cases: at least one ti-form of s has the same entropy as s.

Here, we cannot say that managing time makes the possible future activity less uncertain.

Last, eLoss < 0.0. This case represents the other worst case: all the ti-form of s has an entropy higher than s. In this case, considering time decreases the quality of the model. Notice here that the term Loss is a misnomer as it may theoretically be < 0.0. However, this term has been chosen to be coherent with previous measures. As a consequence, the higher the entropy loss ratio the more managing time in patterns contributes to better estimate students' future activities.

3.2.4

Evaluating the impact of a specific time-interval on students' future activities

This fourth and last step is dedicated to the evaluation of the impact of a specific time-interval of a ti-pattern on its extended parts. More precisely, we are interested in the impact of the last time-interval of a pattern. We focus on the following situation: given two pseudo-equivalent ti-patterns (cf., 3.7), to what extent do their set of extended parts differ?

This evaluation allows to study to what extent two students, who perform the same temporal activity, except about the time of their last activity, do have identical future activities.

In other words, is a temporal difference between two activities an indicator of activities that are beginning to diverge?

To perform this evaluation, we first evaluate the proportion of identical ti-patterns between pairs of sets of extended parts, as defined in Equation [START_REF] Chen | Discovering time-interval sequential patterns in sequence databases[END_REF].

idExt(P Q) = (P,Q)∈P Q |P ∩Q| |P ∪Q| |P Q| (6)
with P Q = {(extP art(p), extP art(q))}|psd-eq(p, q) the pairs of sets of extended parts of all pseudo-equivalent pairs of tipatterns. The higher this proportion, the lower the impact of the last time-interval.

Second, we rely on the proportion of s-equivalent extended parts. This measure also evaluates the impact of the last time-interval on the set of extended parts, but by considering only their sequential nature. The proportion of s-equivalent extended parts is defined in Equation [START_REF] Chen | Discovering prerequisite structure of skills through probabilistic association rules mining[END_REF].

sidExt(P Q) = (P,Q)∈P Q | s-form(P ) ∩ s-form(Q)| | s-form(P ) ∪ s-form(Q)| | s-form(P, Q)| (7)
This proportion represents if students tend to share their following sequential activities, even though they differ in their last time-interval. Here also, the higher this proportion, the lower the impact of the time-interval.

Notice that for reasons of readability, s-form(•) is used here to represent the sequential form of a set of ti-patterns and a set of pairs of ti-patterns.

EXPERIMENTS

We apply the methodology described in the previous section to evaluate to what extent mining ti-patterns increases the knowledge about students' activities. We first present the dataset on which the experiments are conducted, then use the 4 steps of the methodology and draw conclusions for each of them. Finally, some mined ti-patterns are displayed.

Dataset overview and implementation

We collected data from 113 first-year university students, enrolled in a Mathematics and Computer Science Bachelor program and who interact with learning resources on their LMS. We focus on one specific course: algorithms and programming from the Fall semester in 2018. This course is a core course of this program. Diverse online materials are available: slides, exercises for lab sessions, tests, etc. Most of the students own a personal computer, so they can access the course both during teaching hours (lectures or lab sessions) and after official teaching hours. The set of events E is made of 35 learning resources, that students can consult. About 50% of these resources are studied during the teaching hours (lectures or lab). The dataset is made up of about 6,300 actions and each student sees on average 56 resources. The dataset spans almost one year, as it includes actions performed not only during the teaching period but also during revisions for the final examination and actions conducted for the retake examination (for the subset of students who failed the final examination).

In the experiments conducted, we use a relative minimum support δ = 0.1. Two algorithms are studied: PrefixSpan, to mine sequential patterns and I-PrefixSpan, to mine tipatterns. The source code used for I-PrefixSpan algorithm is the one available in [START_REF] Fournier-Viger | The SPMF open-source data mining library version 2[END_REF] (we have slightly adapted the code to our needs). The source code used for the classical PrefixSpan algorithm is the one proposed by Gao [START_REF] Gao | Prefixspan-py[END_REF].

Determining the set of time-intervals

We propose to study two types of intervals: Linear intervals, where each interval has an equal duration, and granular intervals, where the duration of intervals grows with the gap value.

Table 1 presents various sets of intervals studied. For each of them, the number of intervals, the maximal horizon, the fitting of the set, the frequency of each frequent interval, as well as the number of frequent patterns, are displayed. To avoid an artificially high fitting value, we consider that an interval is frequent if its frequency is no less than 10. The frequency of an interval is evaluated as the number of times the interval is used in the frequent patterns.

Before going into the details of the analysis of the set of intervals, we would like to mention that the sets do not all have the same number of intervals, so these values in Table 1 are not directly comparable. In addition, two contiguous granular intervals represent a totally different duration (for example up to 1 hour and up to 1 day), the frequencies are therefore not comparable. Last, notice that the total number of patterns in one set of intervals cannot be explained by the number of patterns of another set. Let us for example consider two sets of intervals and their associated number of patterns. Suppose that the first interval has an average duration twice longer than the second one. A pattern that is frequent in the first set may correspond to either two frequent patterns in the second set, or only one frequent pattern, or no frequent pattern at all (see section 3.2.2).

Let us first consider the three sets of linear intervals. For the two first sets (30 min and 1 hour), the fitting measure is quite low: 8%, which means that the vast majority of in- tervals are not found in frequent patterns. For example, in "30 min", only the first interval (between 0 and 30 minutes) and the last interval (more than 12 hours) are not empty. We can conclude that both sets of intervals are not good candidates. Caution must be exercised in interpreting this result. It might mean that students do not regularly switch from one resource to another, with a time gap between 30 minutes and 12 hours. It can also mean that the 30 min. time-interval is not relevant. Despite the lack of relevance of these intervals, the number of patterns discovered is important. As only two interval patterns are used, we can consider that I-PrefixSpan behaves here almost as PrefixSpan. The fitting value of the "1 day" set is quite larger: 72%, which means that most of the 25 intervals are frequent. However, the total number of frequent patterns in this set is highly decreased, compared to the "30 min" and "1h" sets (by about 10 times). In addition, many interval frequencies are not so high, some of them being close to the minimal threshold, except the first and last one. This tends to mean that many intervals are not that representative of the data. Moreover, although the number of intervals is quite large [START_REF] Maldonado | Analysing frequent sequential patterns of collaborative learning activity around an interactive tabletop[END_REF], the maximal horizon represented by this set remains limited (all together, except the last one, represent a horizon of smaller than a month). Recall that the dataset spans almost one year. Obviously, the horizon can be extended, but it will be at the cost of an even larger number of intervals, as well as an increase in the space and computation time. These results tend to suggest that the set of intervals should contain small intervals for close events (such as suggested by the frequency of I0 in the 30 min set), and larger intervals for furthest gaps (such as suggested by the frequency of I24 in the 1 day set). Thus, a granular set of intervals should better fit the dataset.

We propose to study now two sets of granular intervals. In the first set, the duration of intervals grows exponentially: the duration of an interval is twice larger than the duration of the preceding interval. The fitting of this set is greater than for the two first ones, but smaller than the third one. Nevertheless, the horizon is larger than for all the previous ones (about 4 months), and the number of intervals is decreased. The empty intervals (from I2 to I7) tend to represent a gap between 20 min and 10 hours 40 min.

The second set of granular intervals is referred to as "human", the intervals are designed to represent the human natural time: minute, hour, day, week, etc. This set of intervals has a maximal fitting (100%). At the opposite of the "1 day" intervals, that has the highest fitting value till then, the frequency of each interval is quite large (greater than 1,600) and the number of intervals is reduced (only 6 intervals). Besides, the total number of patterns is larger than both the "1 day" and the "exponential" sets.

All these elements contribute to consider the "human" set as the best set of intervals. In this set, time is represented by the {minute, hour, day, week, month, year} intervals. This set has a maximal fitting (100%), covers a large horizon (till a year, which corresponds to the span of the dataset), with a limited number of intervals (6 intervals) and provides a quite large number of frequent temporal patterns. Therefore, in the following experiments, this set of intervals will be used.

Given these elements, we would like to highlight that this set of intervals intrinsically represents the classical rhythm of courses, for example one lecture (or one lab session) is planned each week. The human set of intervals thus allows to mine patterns that represent natural students temporal activities: some students tend to work immediately following a lab session (or a lecture) represented by I0 or I1; other students wait for some hours in the same 24h, and others work during the week, or even the week after (before the next session) represented by I4. It is typically the type of information that we expect to get when we aim at modeling students' activities.

Comparing s-patterns and ti-patterns

This second experiment aims at comparing sets of s-patterns and ti-patterns. Table 2 presents both sets of patterns, associated with measures introduced in the methodology. Let us first focus on the number of frequent patterns (line 1). The total number of frequent ti-patterns is dramatically smaller than the number of frequent s-patterns. The pattern loss is larger than 0.99. This means that the great majority of s-patterns has no frequent ti-forms, probably due to the spread of occurrences of s-patterns over numerous tipatterns. These findings are in line with [START_REF] Kitakami | Modified prefixspan method for motif discovery in sequence databases[END_REF]. In addition,

δ = 0.1(=
) s = e31, e29 ts1 = e31I0e29 , supp(ts1) = 1 ∧{ p | (isf orm(s, p) ∧ f requEnt(p))} supp(s) = 26 ts2 = e31 I1 e29 , supp(ts2) = 10 ts3 = e31 I2 e29 , supp(ts3) = 6 ts4 = e31 I3 e29 , supp(ts4) = 3 ts5 = e31 I4 e29 , supp(ts5) = 4 ts6 = e31 I5 e29 , supp(ts6) = 7 Example of pattern s, f requEnt(s) s = e22, e33 p1 = e22I0e33 , supp(p1) = 25 ∧{∃(pi, pj)|(f requEnt(pi) ∧ f requEnt(pj))} supp(s) = 53 p2 = e22I1e33 , supp(p2) = 22
Support loss sLoss(SP 1+ )= 0.33 ; std(sLoss(SP 1+ ))=0.10

Table 2: Comparison of sets of patterns mined with PrefixSpan and I-PrefixSpan we can see in Line 2 that the average length of ti-patterns is about twice smaller than the length of s-patterns, the same for their maximal length. A first conclusion that can be drawn here is that most of frequent sequential patterns have no recurrences in their time-intervals. This means that students tend to have numerous recurrent sequential activities, and quite less recurrent time-interval activities. However, even though the average length of patterns is divided by 2, ti-patterns have a significant length, which means that they do represent a meaningful students' activities. Moreover, a tens of thousands s-patterns (about 34,000) have one or more frequent ti-forms (about 51,000). This means that for these sequential activities, there are actually temporal regularities. These activities will be studied in more detail in the following section.

Lines 4 and 5 in Table 2 illustrate some examples of spatterns and their candidate or frequent ti-forms. Line 4 presents one of the 99.8% s-patterns that has no ti-form (thus, from SP 0 ). This pattern (s = e31, e29 ) has 6 candidate ti-forms, but none of them is frequent. We can conclude that no obvious time-interval regularity is observed for this activity. Thus, this activity does not seem to be guided by temporal constraints. We can also observe here that the sum of the support of the ti-patterns is greater than the support of their s-form s. This was mentioned in section 3.2.2.

In the remaining sequential patterns (SP 1+ ) made up of about 34,000 s-patterns, 65% of the s-patterns have exactly 1 frequent ti-form and 91% have 1 or 2 frequent ti-forms.

The highest number of frequent ti-forms of an s-pattern is 9, which is quite high. Let us now consider line 5 in Table 2, that presents a s-pattern that has several frequent ti-forms. This s-pattern (s = e22, e33 ) has a support equal to 53 and two frequent ti-forms. Such a pattern occurs with two temporal recurrences, and most of its occurrences have a time gap between 1 minute and 1 day. Such patterns are highly interesting and will also be further studied.

Based on these findings, it is legitimate to ask whether a tipattern-based model can replace a s-pattern-based model. Line 1 gives first indications. Many sequential patterns "disappear" with such a model (more than 99% of sequential patterns have no frequent ti-pattern). If the objective is to replace traditional s-patterns by ti-patterns, a problem of coverage of the model arises. However, if the goal is to iden-tify which activities (sequential) have temporal regularities, ti-patterns are of the highest interest.

Let us now focus on the support loss associated with the complete set SP 1+ of s-patterns that have at least one frequent ti-form. sLoss(SP 1+ ) = 0.33, with a standard deviation equal to 0.1. This means that on average 1/3 of the occurrences of an s-pattern "disappear", i.e. they do not belong to any frequent ti-form. We can conclude that among patterns with identified temporal regularities, 33% of the occurrences do not follow this regularity, which may be high.

Evaluating the impact of time on the set of possible future activities of students

Following our methodology, we evaluate now if ti-patterns carry more information than s-patterns about future activities of students. As a preliminary remark, we would like to mention that s, eLoss(s) ≤ 0. We mentioned previously that this case would occur rarely, in practice here it does not occur.

In the set S 1+ , 71% of the patterns have at least one extension in SP (see Def. 3.5). Let us first consider the 66% of these patterns that have a unique extension. By definition for these patterns, Ent(s) = 0 and Ent(p) ≥ 0, ∀p ∈ ti-form(s). The first Line of Table 3 is an example of such a case. The s-pattern e24 e27 e14 has only one extended part, so its entropy equals zero. It has three ti-forms, but only one has an extended part. So, all these ti-forms have an entropy equals to zero. In this case, even if the entropy loss is null, the information about the future activities of students is increased, as only one ti-pattern has a frequent extended part.

Let us now consider the 34% remaining patterns, which have more than one extension in SP . The average entropy is 0.84 with a maximal entropy of 7.71. When focusing on the set of their ti-forms, the average entropy is 0.35 and the maximal entropy is 6.22. To make entropies as comparable as possible, the average entropy for s-patterns has been evaluated only on the set of s-patterns that have at least one ti-form.

We can first notice that entropy of s-patterns is globally higher than the one of ti-patterns (for both maximal and average values). More precisely, the average entropy of spatterns is 2. patterns allows to decrease the uncertainty of students' future activities.

We will now compare the entropy of each s-pattern, with the entropy of its ti-forms (through eLoss). In 68% of the cases, the entropy loss between the s-patterns and their tiforms is higher than 0. This means that when considering a temporal student activity, in 2 cases out of 3, the future activity of this student is less uncertain than when managing his/her sequential activity. These 68% are divided into 51% with a loss equal to 1, which means that future activities become certain. 17% of the cases have a loss between 0 and 1. The average entropy loss on all s-patterns is quite high: eLoss = 0.4. Roughly speaking, the future activities of students are on average 40% less uncertain when managing time in patterns, which is highly promising.

Thanks to these experiments, we confirm that managing time-interval patterns allows, in most cases, to have a better view of the following activities of students. In addition, for a significant number of activities, future activities are now totally certain.

Let us now focus on an example presented in the second Line of Table 3. The s-pattern s = e1e10e12 has many extensions in SP and many ti-forms, among which many of them have extensions. Notice that although the entropy loss is low (the maximal entropy of the ti-forms is 4.55), on average it is significantly lower (0.78). In this specific case, eLoss measure is not that representative of the difference in entropy, the entropy decrease is probably higher than the eLoss value.

Evaluating the impact of a specific timeinterval on students' future activities

The experiments conducted here fall within the scope of the last step of our methodology. They aim at evaluating to what extent two students who perform a similar activity (both in terms of resources and time-interval) and who only differ in their last time-interval, have the same future activities. In the experiments conducted, we will only focus on patterns made up of at least 3 events (and 2 time-intervals) to ensure that the patterns can be considered as activities.

In the set P Q composed of |P Q| = 9, 510 of pseudo-equivalent pairs of patterns (cf., Definition 3.7), 25% of the extended parts of a pattern of any pair are also part of the extended parts of the other pattern (sequentially and temporally identical). 11% additional pairs have sequentially identical extended parts. This highlights that even when two ti-patterns differ in their last time-interval only, this small difference leads to a significant difference in their sets of extended parts. In terms of students' activities, this means that when two students make exactly the same activity, except on the last time-interval, their following activities mainly differ: not only in terms of temporal activities but also in terms of their sequential activities. We can conclude that the last timeinterval highly influences students' future activities and that it may be viewed as an indicator of activities that are beginning to diverge.

Experiments conducted in both previous sections confirm that ti-patterns contribute to the increase of the information about students' future activities whereby the uncertainty of this future is reduced. As a consequence, we can say that time is an important information in students' activities.

Interpretation of ti-patterns

In this section, we present examples of frequent ti-patterns, in an understandable format to better analyze and understand students' activities.

The events ids in patterns are replaced by their type and an id. Lecn will refer to the slides associated with the n th lecture; Glosn will be the n th glossary resource; Stxn a syntax resource; Sumn will be a summary resource ; Labn a resource that contains exercises that are studied during lab sessions (exercise sheets); F An are facultative additional exercises; finally Ad is the advise resource. The time-intervals are noted (Is, Imn, I h , I d , Iw, Imt), which refer to seconds, minutes, hours, days, weeks and months.

Given that the longer an activity, the more information it contains, we will preferably focus on the longest ti-patterns.

Activities made up of temporally close events

Let us start by studying activities that contain only the "seconds" time-interval (i.e. events with a maximal gap of 1 minute). This will allow us to have a better view of the type of activities that are performed on the spot. First, the corresponding activities tend to be made up of specific types of events: they are a mix of glossary, syntax, advertisement and lab resources. Second, the maximum length here is 7, which means that there are actually long recurrent "quick" activities made by students. Third, when analyzing the activities, we can remark that they all have a similar skeleton: students generally start by looking at the following resources (in any order): {Sum5, Stx3, Glos3}, then study one or more Lab exercises and finally consult an advice page. Let us for example present a ti-pattern of length 7: Sum3 Is Stx3 Is Glos3 Is Lab1 Is Lab2 Is Lab3 Is Ad Such patterns can be interpreted as follows: they represent typical activities performed when preparing an exam. Not only several Lab sheets are studied, but also before these resources, students have a quick look at the syntax, glossary, and summary of the lectures. They finally consult the advice page. In such patterns, students interact with resources within a short time, including with the lab sheets.

Activities made up of I h time-intervals

Let us now focus on patterns that use the "hours" timeinterval, where patterns are made up of events with a gap value between 1 hour and 1 day. Here again, we identify a skeleton shared by most of the ti-patterns: Lab {1,3} I h Lab {2,3} I h Lab {2,3,4} , where Lab {1,3} means either Lab1 or Lab3. These patterns highlight that some students tend to work sequentially on several exercise sheets. The gap being between 1 hour and 1 day, tends to mean that students dig deep into their works: they spend some hours to perform each exercise sheet.

Other intervals

We have performed a similar study on other time-intervals.

For each of them, we also identify skeletons shared by almost all the patterns.

An interesting conclusion that can be drawn from these findings is that for any given time-interval, typical long activities are made by students, that do all have the same skeleton. More importantly, when comparing skeletons between timeintervals, they are totally different. We can thus conclude that the type of activity performed is strongly linked with the "rhythm" of the activity. Here, "rhythm" means a timeinterval granularity shared by all gap between all events of the activity. Last, when studying the timestamps associated with each occurrence of the activities presented above, there is no specific period associated: they are performed at any moment in the semester. For example, when considering the first example given, that is mainly related to the 3 rd lecture, we found similar patterns for the 1 st , 2 nd , etc. lecture resources.

DISCUSSION

While traditional studies emphasize that students have typical sequential learning behaviors (identified by frequent sequential patterns), this study further emphasizes that for specific activities students work with temporal regularities.

Based on the experiments conducted in the previous sections, we initiate a discussion.

The results have highlighted that among the sets of intervals tested (linear and granular), the one that represents the human natural time is the most relevant one, at least for the dataset used in the experiments (see section 4.2). In addition to outperforming other sets of intervals according to predefined measures, this set conforms to the scope of application: the duration of most of the lectures or lab sessions is about one hour, two successive lectures tend to occur each week, etc. So, the interpretation of the discovered patterns is enhanced. Of course, many other sets of intervals remain untested and may be more adequate. Besides, an automatic approach that learns the optimal set of intervals could be tested, as in [START_REF] Mahajan | An approach to optimize fuzzy time-interval sequential patterns using multi-objective genetic algorithm[END_REF]. However, this would be at a significant additional computational cost, without any guarantee of applicative interpretability of these intervals.

As expected, a high number of sequential patterns have no frequent ti-form. In the experiments conducted, we have even highlighted that most of the sequential activities have no temporal regularities. This results in a high number of "lost" patterns, which can be problematic, in case we are interested in both frequent ti-patterns and s-patterns. A solution could manage both types of patterns: sequential students' activities mixed with temporal students' activities. This solution would not only maintain the coverage of the model, thanks to sequential patterns but also manage time, thanks to temporal patterns, when suitable. Here is an example of such a pattern: E2 E27 I1 E13 . This pattern means that many students consult E2 then E27 (whatever is the time-interval), then between 1 minute and 1 hour later they do consult E13.

Focusing on s-patterns and their various frequent ti-forms can help to highlight different learning approaches adopted by students. For example, an activity done with a gap lower than 1 minute between its events may represent the fact that the associated students are used to first download all the resources and then work offline. The same activity with a time gap between 1 minute and 1 hour may reflect that students do work online, they do not access a resource before finishing the previous one. So, in addition to highlighting the diversity of activities of students, ti-patterns are also a way to identify students' learning practices. One can foresee that these patterns could be used as input information for many works such as those that focus on students' engagement.

CONCLUSION AND FUTURE WORKS

The study presented in this paper highlights the relevance of using time information when mining patterns of students' activities. A time-interval pattern mining approach, through the I-PrefixSpan state-of-the-art algorithm, has been adopted to conduct this study.

The experiments conducted have pointed out that the nature of the set of intervals used highly impacts the representativity of the model and that the set of intervals that represents the human natural time is adequate. We also found that most of the sequential students' activities do not correspond to any time-interval activity. However, for other cases, managing this time-interval provides a better view of the future possible students' activities, thanks to temporal indicators. Moreover, results show that a single time-interval difference between two events of two patterns sequentially equivalent results in significantly different subsequent activities. We thus confirm our hypothesis: temporal information is highly promising for a more precise modeling of students' activities. One additional experiment has illustrated some frequent students' activities both temporal and sequential. It has put forward that, by looking at some specific timeintervals, we can understand what activities students often perform instantly or throughout a longer period. The work we have conducted provides a first step towards longer-term research. One of our future goals is to provide students with recommendations of educational resources. By relying on ti-patterns, we are confident that not only the accuracy of the recommendations provided to students will be increased but also that these patterns will give indications about the right time to propose recommendations to students.

  To understand what a frequent sequential pattern is, let us first define what a sub-sequence is. α = E1 . . . En is a sub-sequence of β = E 1 . . . E m if: ∃[1 ≤ j1 ≤ . . . ≤ jn ≤ m]|{E1 = E j 1 , . . . , En = E jn }.

Table 1 :

 1 Fitting, examples of intervals and number of patterns for several sets of intervals

	Type	Duration	Number of intervals Horizon Fitting	Used intervals	#
						& associated frequency	patterns
	Linear	30 min	25	12h.	8%	I0 : 350, 000 ; I24 : 1, 770, 000	550, 000
	Linear	1 hour	25	12h.	8%	I0 : 549, 380 ; I24 : 1, 843, 660	356, 811
						I0 : 90, 976 ; I1 : 42 ; I2 : 25 ;
						I3 : 54 ; I4 : 119 ; I5 : 36 ;
	Linear	1 day	25	24d.	72%	I6 : 239 ; I7 : 189 ; I8 : 17; I11 : 14 ; I12 : 44 ; I13 : 80;	37, 764
						I14 : 17 ; I17 : 14 ; I18 : 13 ;
						I20 : 36 ; I21 : 45 ; I24 : 33, 298
	Granular	expon.	16 I0 = [0 sec. ; 10mn.[	8mt.	56%	I0 : 17, 739 ; I1 : 79 ; I8 : 82 ; I9 : 269 ; I13 : 3, 693 ; I14 : 1, 159 I10 : 4, 278 ; I11 : 5, 126 ; I12 : 6, 403 ;	15, 754
	Granular	human	6 I0 = [0 sec. ; 1 mn.[	1y.	100%	I0(sec) : 7, 706 ; I1(min) : 10, 551 ; I4(week) : 68, 614 ; I3(month) : 22, 479 I2(hour) : 1, 615 ; I3(day) : 30, 925;	51, 025

  [START_REF] Fatahi | An empirical study of using sequential behavior pattern mining approach to predict learning styles[END_REF]

		PrefixSpan	I-PrefixSpan
	Number of patterns	|SP | = 12, 826, 760	|T P | = 51, 025 -pLoss(SP ) = 0.998
	Average	8	3.8
	Max. length	17	8
	Example of pattern s, f requEnt(s

Table 3 :

 3 [START_REF] Campagni | Data mining models for student careers[END_REF] times bigger than the one of ti-patterns. We can thus draw a first global conclusion: managing time in Examples s-patterns with ti-forms, extended parts and entropy values. ( * ) The corresponding extension pattern is p = e24 I3 e27 I3 e14 I3 e12 .

	s-pattern Examples of Ent(s)	Examples of	nbExt(p) Examples of	max-mean-
		extPart(s)		p ∈ ti -form(s)		extPart(p)	Ent(p) Ent(p)
	e24 e27 e14	e12	0.0	e24 I2 e27 I3 e14 e24 I3 e27 I3 e14	0 1	e14 I3 e12 ( * )	0.0	0.0
	e1 e10 e12	e {3,13} e19 e {3,22} e12 e {12,19}	5.49	e1 I5 e10 I1 e12 e1 I2 e10 I2 e12 e1 I5 e10 I0 e12 e1 I5 e10 I2 e12	0 24 1 13	e12 I5 e {19,13,15} e12 I4 e3 e12 I5 e {19,13,12}	4.55	0.78


Benjamin Gras
email: benjamin.gras@loria.fr

Armelle Brun
email: armelle.brun@loria.fr

Anne Boyer
email: anne.boyer@loria.fr

For and by Student Dashboards Design to Address Dropout

Keywords: Student dashboard, student dropout, higher education. 1

This paper focuses on the design process of a student dashboard, in the frame of a Learning Analytics project. The dashboard is intended to reduce dropout of first-year University students. The strong points of this dashboard are three-fold: 1) the involvement of students in the whole design process, 2) the possibility of students to personalize the dashboard, 3) the possibility of indicating personalized goals.

INTRODUCTION

The French project EOLE (Engagement to Open Education -http://www.dune-eole.fr) aims at designing a different approach of education at the University, both in its modalities and in the enlargement of its target audience. This major challenge is fundamental to develop the contribution of universities in the learning sector, in both a citizen and a competitive approach. One of the numerous goals in EOLE is to address students' dropout problem, mainly for first-year students, who are known to massively dropout.

Two achieve this goal, EOLE relies on two main hypotheses: 1) most of the students who dropout lack information about their learning behaviour: the way they learn, compared to others or not, about their progress related to the expected output, etc. So, if students can get more information, dropout will decrease; and 2) if students can feel under control of their learning process or feel heard, by informing their expectations, receiving advice, etc., dropout will also decrease.

Based on these hypotheses, EOLE proposes students to access course-level personalised dashboards. To ensure a high quality dashboard, a multi-profile team has been set up: teachers, students, vice-rectors, computer scientists, researchers, among others. For the sake of representativeness, teachers and students from a great diversity of disciplines are involved in the team. This multi-profile team and the involvement of students in all steps of the project are one strong point of this project.

The dashboard, and the associated features and indicators, are further presented and explained below. A study of the impact of the dashboard on student dropout will be conducted in the following weeks.

RELATED WORKS

In the literature, self-regulated learning (SRL) can be defined as being "an active, constructive process, where learners define their learning objectives and try to supervise, regulate and control their cognition, motivation behaviours, guided and constrained by their objectives and characteristics related to the environment" [START_REF] Pintrich | The role of goal orientation in self-regulated learning[END_REF]. Zimmerman [START_REF] Zimmerman | Becoming a self-regulated learner: An overview[END_REF], also explains that the differences in learning success are mostly attributed to the self-regulation ability of learning, which are relevant to the initiation and maintenance of the learning process. In addition, a recent study by Aljohani [START_REF] Aljohani | An integrated framework for course adapted student learning analytics dashboard[END_REF] shows that student-centred dashboards [START_REF] Govaerts | The student activity meter for awareness and self-reflection[END_REF][START_REF] Odriozola | Empowering students to reflect on their activity with StepUp!: Two case studies with engineering students[END_REF] increase student engagement (investment in time, etc.) more than teacher-centred dashboards [START_REF] Guo | MyCOS Intelligent Teaching Assistant[END_REF] (in the latter case student engagement could be increased through the interaction between students and teachers). In this latter study, students can consult a dashboard giving them statistical, graphical and textual feedback about their learning. The use of this dashboard by students has been tracked and an analysis shows that students who use the dashboard are significantly more engaged (i.e. spend more time on the platform and have more activities on the forums).

BUILDING A STUDENT-CENTERED DASHBOARD

As highlighted in the literature, dashboards are a way to support students in the selfregulation of their learning. EOLE assumes that it can also be a way to address student dropout, and thus proposes to design a dashboard that is targeting students. The key point in this design is that students are at its core.

In a preliminary phase, students have been invited to share their needs, in terms of features/functionalities of a dashboard. A needs analysis has been conducted with about 100 first-year students. Below are the most recurrent needs that students expressed. Notice that some of them have been highlighted in a similar study [START_REF] Schumacher | Features students really expect from learning analytics[END_REF].

• Indicators should be sufficiently diversified so that every student can find those corresponding to his/her wishes.

• Obsession with indicators should be avoided.

• Indicators must be beneficial and their reading must be easy.

• Indicators should value the advice between peers. Senior students should volunteer to mentor junior students.

• Advice about the methodology of academic work (organization, work methods) is welcome, not just help about course content.

This needs analysis resulted in the design of a first prototype of the student dashboard. It has then been presented to other first-year students to obtain their feedback about the features and the indicators proposed. In all, more than 300 first-year students, spread over several iterations, gave their opinion during the iterative and incremental co-design of the dashboard.

A strong assumption on which EOLE relies is that a dashboard is a tool that should be made for the students' own interest, and that it should not be intended to constrain students. To ensure that, at each iteration of the dashboard design, students' opinions were collected through a questionnaire. The first version of the dashboard (from the first iteration) was presented to 88 students, along with questions about the features they think they would use if these features were made available to them. The results of this questionnaire are presented in Table 1. Although the literature highlights the comparison with peers, especially in higher education, only 56% of the students are in favour of this feature, i.e. nearly half of the students do not wish to compare themselves with their classmates. In addition, 5 students (6%) expressed their fears and apprehensions about the impact of comparing themselves with peers on their personal well-being. It has thus been decided to display the peer comparison feature only on demand of the student. Thus, students who want this feature have to explicitly tick the appropriate answer. Other less requested features were not explicitly criticized by students, so it has been decided to keep them on the dashboard. The final version of the dashboard (presented in Figure 1) has been obtained after three iterations. The dashboard displays indicators about the activity of a specific student in his/her Algorithms and programming course. It is divided into two parts. This final version is available at the following link. Since this is an interactive and customizable dashboard, the best way to understand it is to interact with it directly online.

"My activity" part of the dashboard

The left part of the dashboard, named "My activity" (Figure 1) displays raw indicators of the student's activity. This part includes the most awaited functionality by the students: the individual performance (99% of students have declared to want it). Three types of raw indicators have been selected:

• Activity indicators (blue ones): number of submitted works, quiz scores, number of resources viewed and the total number of actions on the course.

• Student engagement indicators (red ones): number of active days, weekly regularity and number of completed automatic advice. The weekly regularity of a student has been adapted from [START_REF] Boroujeni | How to quantify student's regularity[END_REF], which shows not only that students easily understand the meaning of this indicator but also that they are interested in discovering if they are working less than in previous weeks. A correlation of 0.28 is observed between the weekly regularity that we have adapted and the course score. This correlation is calculated from the students' traces of activity enrolled in the course during the previous years and the final results of the students. With a p-value of 0.002, we can conclude that there is a significant link between the weekly regularity of student work and their academic performance.

• Collaboration indicators (green ones): number of created topics on the forum, number of answers, number of times the student asked for help.

Figure 1: Final version of the dashboard

Besides, during the co-design iterations, most of the students have put forward the fact that learning traces collected by the system only represent a partial view of their activity. Based on this feedback, it has been decided to add an edit function to allow students to modify the indicators displayed on the dashboard. Thus, these user-modified indicators should better reflect students' actual learning activity. More importantly, students stay in control of their personalised dashboard.

Last, an additional indicator requested by students and proposed in this dashboard, is the student's overall performance (lower section of the left part). To make this indicator possible, students are proposed to provide, on the dashboard, the score that they would appreciate to achieve on the final exam (the personal goal). In Figure 1, the expected score given by the student is 10 (out of 20). In this case, the associated overall performance indicator is 61.23%. This indicator is evaluated as the odds percentage that a student achieves his/her personal goal. As students directly set their personal goal, the student's overall performance is directly influenced. Let two students have the same value on two indicators, the one who foxed his/her personal goal to 18 will not have the same odds percentage as a student who fixed his/her personal goal to 10.

Many indicators that are proposed in the dashboard, such as the indicator of future success, which depends on students' personal goal, but also the fact that students can modify the content of the indicators, have not only been designed to increase the information students can access, but also to increase their feeling of being heard and understood. The expected effect is a decrease of student dropout, especially for students who may lack of confidence in themselves.

"My follow-up" part of the dashboard

The right part of the dashboard (Figure 1) is divided into three elements.

• At the top, the evolution of the student's performance over time is displayed, in the form of a line chart. This is where the student can choose to display the average performance of his/her classmates.

• In the middle of this part, personalized advice is provided to the student to help him/her improve his/her performance (orange rectangle). The student can follow or not the advice, depending on his/her goodwill.

• At the bottom, two action buttons allow the student to ask for help. The first one is dedicated to receiving help from the teacher. By clicking on this button, the student also accepts to share the data displayed in his/her dashboard. The second one, labelled "send this dashboard", only shares a capture at time t of the dashboard. The last button is a notification queue to manage the actions of the first two buttons.

The highly personalisable side of this dashboard is intended to make students feel understood and may access the dashboard more often, which could reduce dropout, as students may feel less isolated.

4

USABILITY

The usability of the dashboard proposed here has been evaluated with the System Usability Scale (SUS) [START_REF] Bangor | An empirical evaluation of the system usability scale[END_REF]. Although this scale does not allow to strictly quantify the usability, the score obtained (between 0 and 100) allows to locate the perceived usability of the dashboard by the student. 127 students took this well-known test of the user experience literature. The results obtained are presented in Figure 2.

Figure 2: SUS results repartition

We observe the 1st quartile at 65, the median at 75 and the third quartile at 85. The average score given is 74.12, the minimum 27.5 and the maximum 100. In UX Design methods [START_REF] Lallemand | Méthodes de design UX: 30 méthodes fondamentales pour concevoir et évaluer les systèmes interactifs[END_REF], the authors propose an interpretation scale of the SUS score. Figure 3 presents the associated interpretation scale.

Figure 3: Interpretation scale of SUS score

With an average score of 74.12, the dashboard proposed is between "Good" (73) and Excellent (86), which is rather promising for our imminent 1st live study.

CONCLUSION

The work presented in this paper focuses on a student dashboard design process, adopted by the team of the EOLE project. This dashboard is aimed, among others, to decrease student dropout. Its design involves a multi-profile team, including students, who are the recipients of the dashboard. The futher step is the test of this dashboard and its actual impact on dropout.

Table 1 : What feature students wishes in a dashboard?

 1

		Percentage of students who
	Feature	would use the feature
		(out of 88)
	Individual performance	99%
	Peers comparison	56%
	Automatic advice	52%
	Help other students	48%
	Ask for advice	38%