Nicholas D Kullman

Aurelien Froger

Jorge E Mendoza

Justin C Goodson

frvcpy: An Open-Source Solver for the Fixed Route Vehicle Charging Problem

Electric vehicles offer a pathway to more sustainable transportation, but their adoption entails new challenges not faced by their petroleum-based counterparts. A difficult task in vehicle routing problems addressing these challenges is determining how to make good charging decisions for an electric vehicle traveling a given route. This is known as the fixed route vehicle charging problem.

An exact and efficient algorithm for this task exists, but its implementation is sufficiently complex to deter researchers from adopting it. In this work we introduce frvcpy, an open-source Python package implementing this algorithm. Our aim with the package is to make it easier for researchers to solve electric vehicle routing problems, facilitating the development of optimization tools that may ultimately enable the mass adoption of electric vehicles.

Introduction

Governmental regulations as well as a growing population of environmentally conscious consumers have led to increased pressure for firms to act sustainably. This pressure is particularly high in the logistics domain, which accounts for about one third of emissions in the United States (Office of Transportation and Air Quality 2019). Electric vehicles (EVs) offer a means to more sustainable transportation; however, they present technical challenges to which their conventional (i.e., internal-combustion engine) vehicle (CV) counterparts are immune. For example, because the distance EVs can travel on a single charge is often less than the distance an equivalent CV can travel on a tank of gas, EVs may demand more frequent recharging operations. This difficulty is compounded by the sparseness of EV recharging infrastructure relative to the network of refueling stations available to CVs, potentially forcing EVs to perform longer detours to recharge their batteries. Further, despite recent improvements to battery and charging station (CS) technology, recharging an EV still requires orders of magnitude more time than refueling a CV. The time required to recharge is also nonlinear with respect to the EV's state of charge (SoC), the relative amount of energy left in its battery, posing yet another challenge not applicable to CVs [START_REF] Uhrig | E-mobility in car parks-guidelines for charging infrastructure expansion planning and operation based on stochastic simulations[END_REF]. Companies choosing to adopt EVs require optimization tools capable of handling these additional challenges.

The development of such optimization tools for conventional vehicles has commanded significant attention from the operations research (OR) community in the study of vehicle routing problems (VRPs).

The incorporation of additional constraints that address the challenges posed by EVs has marked a new family of problems within VRPs known as electric vehicle routing problems (E-VRPs). One of the primary tasks in solving E-VRPs is making good charging decisions -namely, where to recharge and how long to do so. This is the crux of the fixed route vehicle charging problem (FRVCP), in which charging operations must be inserted into a fixed sequence of customers being visited by an EV so as to minimize the time for the EV to reach the end of the sequence in an energy-feasible manner. The FRVCP naturally arises as a subproblem in many E-VRPs, since its solution is required in order to determine the true duration or cost of a given route. Having a capable solution method for the FRVCP is thus crucial to the advancement of E-VRP research.

While FRVCP solution methods exist, they tend to suffer from one or more of the following issues: inexactness (e.g., heuristic methods that provide suboptimal solutions), inefficiency (e.g., mixed-integer programs (MIPs) whose solutions require significant run time), or a lack of robustness (e.g., exact algorithms that are limited to simplified versions of the FRVCP). With this work, we offer an implementation of a solution method that suffers from none of these issues. It is based on the labeling algorithm proposed in [START_REF] Froger | Improved formulations and algorithmic components for the electric vehicle routing problem with nonlinear charging functions[END_REF], which, though capable, is notoriously difficult to implement.

In an attempt to remove the burden of implementation for future E-VRP researchers, we offer our implementation in an open-source Python package, frvcpy (a portmanteau of FRVCP and Python; Kullman et al. 2020a). frvcpy is designed to be easily embedded in more complex solution schemes for E-VRPs (such as in a (meta)heuristic or Benders decomposition): it requires minimal dependencies and inputs, can be accessed either via the command line or a Python API, and includes a translator to generate the required inputs from a common instance format in the VRP community (VRP-REP [START_REF] Mendoza | VRP-REP: the vehicle routing community repository[END_REF])). Our aim with frvcpy is to make it easier to solve E-VRPs, thereby stimulating additional research in this field which promises to bring about more sustainable practices in logistics.

The remainder of the paper is organized as follows. We first define the FRVCP in §2, then discuss some of the previous work on FRVCPs in §3. In §4 we give an overview of the algorithm implemented in frvcpy, then describe the package itself in §5. We conclude with brief comments in §6.

Defining the FRVCP

We consider an EV with a fixed route Π = (π 1 , . . . , π R ) that begins at some node π 1 (usually the depot), has a sequence of stops at other nodes (π i ) R-1 i=2 (customers to visit), and terminates at some node π R (also usually the depot). The vehicle begins at π 1 with its battery at some initial energy level q 0 , often taken to be equal to its maximum battery capacity Q. Let the set of nodes in the route be I = {π 1 , . . . , π R }.

We also consider a set of charging stations C at which the EV may recharge between stops in Π. Each CS c ∈ C has some charging type (e.g., fast, slow) associated with a strictly increasing piecewise-linear concave charging function Φ c (t) specifying, for an empty battery, the resulting energy after charging for time t at CS c (see Figure 3). We denote its inverse by Φ -1 c (q), equal to the time required to charge from empty to energy level q, and we define the related function ūc (q 1 , q 2 ) = Φ -1 c (q 2 ) -Φ -1 c (q 1 ) to be the time required to charge from q 1 to q 2 at CS c. Let the set of breakpoints defining the charging function of

CS c be B c , where a breakpoint b i ∈ B c is a (time, charge) pair: (b t i , b q i ).
When the EV travels between nodes i, j ∈ I ∪ C, it incurs some known travel time t ij and energy consumption e ij (we assume the triangle inequality holds for both). At stops in Π, the EV may also incur some processing time (e.g., waiting). The objective of the FRVCP is to determine charging decisions -how much to recharge, at which CSs, between which stops in Π -that minimize the total time for the EV to traverse the route,

Related Literature

FRVCPs fall under the category of EV routing problems, which are themselves part of a larger body of research on VRPs. We focus our review here solely on FRVCPs. For an overview of E-VRPs we refer the reader to [START_REF] Pelletier | 50th anniversary invited article-goods distribution with electric vehicles: review and research perspectives[END_REF], and similarly to [START_REF] Braekers | The vehicle routing problem: State of the art classification and review[END_REF] for an overview of VRPs. [START_REF] Montoya | A multi-space sampling heuristic for the green vehicle routing problem[END_REF] encounter an FRVCP in their work on a green vehicle routing problem. The FRVCP they consider assumes that vehicles may only visit one CS between stops, that they always fully restore their energy when recharging (that is, they follow a "full recharging strategy"), and that doing so requires constant time. To solve this FRVCP, they offer an exact algorithm. [START_REF] Roberti | The electric traveling salesman problem with time windows[END_REF] address an FRVCP in their work on the E-VRP with time windows (E-VRP-TW) and also offer an algorithm that solves this FRVCP exactly. Their solution accommodates a partial recharging policy, assuming that the time required to recharge is linear with the amount of energy. However, unlike in [START_REF] Montoya | A multi-space sampling heuristic for the green vehicle routing problem[END_REF], they assume that the network of CSs is homogeneous; that is, that all CSs have the same charging technology. The FRVCP again arises in related works by [START_REF] Hiermann | The electric fleet size and mix vehicle routing problem with time windows and recharging stations[END_REF][START_REF] Schiffer | An adaptive large neighborhood search for the location-routing problem with intra-route facilities[END_REF][START_REF] Hiermann | Routing a mix of conventional, plug-in hybrid, and electric vehicles[END_REF]. These studies offer exact algorithms for the FRVCP under the assumption that at most one CS may be visited between stops, that the CSs are homogeneous, and that recharging requires linear time. [START_REF] Hiermann | The electric fleet size and mix vehicle routing problem with time windows and recharging stations[END_REF] additionally assume a full recharging strategy while [START_REF] Schiffer | An adaptive large neighborhood search for the location-routing problem with intra-route facilities[END_REF] and Hiermann et al. ( 2019) allow partial recharging. [START_REF] Montoya | The electric vehicle routing problem with nonlinear charging function[END_REF] then consider the first FRVCP that accommodates realistic (nonlinear) recharging times. In the study, they also demonstrate that the assumption of linear recharging times can lead to infeasible or suboptimal solutions. Their FRVCP allows for partial recharging and heterogeneous CSs but assumes that at most one CS may be inserted between stops. To solve their FRVCP, Montoya et al.

offer both a heuristic and a MIP formulation. [START_REF] Koç | The electric vehicle routing problem with shared charging stations[END_REF] adopt the heuristic and MIP formulations from [START_REF] Montoya | The electric vehicle routing problem with nonlinear charging function[END_REF] to solve a similar FRVCP that arises in their work on the E-VRP with shared CSs and nonlinear charging. [START_REF] Baum | Shortest feasible paths with charging stops for battery electric vehicles[END_REF] then offer a labeling algorithm to solve an FRVCP on real road networks that also accommodates realistic recharging times and allows for multiple CS insertions, although it is restricted to the special case where the route length is two (an origin-destination (OD) pair).

Finally, [START_REF] Froger | Improved formulations and algorithmic components for the electric vehicle routing problem with nonlinear charging functions[END_REF] propose an exact labeling algorithm to solve the FRVCP from [START_REF] Montoya | The electric vehicle routing problem with nonlinear charging function[END_REF]. Their algorithm is not restricted to OD pairs, and it additionally allows the EV to visit multiple CSs between stops in the route, making theirs the richest of the aforementioned FRVCP variants.

Over a testbed of nearly 30,000 instances, they compare their labeling algorithm against a heuristic and a commercial solver for a MIP formulation. They find that the labeling algorithm matches the optimality of the MIP with a runtime comparable to the heuristic. The algorithm is thus state of the art for solving FRVCPs. However, the authors note that its performance is not without cost. They state that E-VRP researchers may ultimately prefer to adopt the heuristic solution, despite its inferior performance, given the complexity of implementing the labeling algorithm. Here, we offer an implementation of the algorithm in frvcpy in an attempt to ensure that its complexity does not prevent its adoption.

The algorithm from [START_REF] Froger | Improved formulations and algorithmic components for the electric vehicle routing problem with nonlinear charging functions[END_REF] has also been adapted to accommodate FRVCPs with additional constraints. For example, in work on a stochastic E-VRP with public CSs, [START_REF] Nicholas D Kullman | Electric Vehicle Routing with Public Charging Stations[END_REF] adapt the algorithm to accommodate an FRVCP with discrete charging decisions and time-dependent waiting times at CSs. Similarly, Kullman et al. (2020b) adapt an early version of frvcpy to accommo- date an FRVCP with customer time windows. In both cases, the algorithm's speed and exactness were required as it was called repeatedly to solve a subproblem in a Benders-based branch-and-cut procedure.

Overview of Labeling Algorithm from Froger et al. (2019)

Given the algorithm's complexity, we provide here a cursory overview and refer the reader to Froger et al.

(2019) for additional details (see, in particular, their discussion of Algorithm 3 in §5.3 and Appendix E).

To find the optimal charging decisions for a given route Π, the FRVCP is reformulated as a resourceconstrained shortest path problem. The algorithm then works by setting labels at nodes on a modified graph reflecting the vehicle's possible movements along Π (Figure 1, Inset 1). Labels are defined by SoC functions -piecewise-linear functions comprised of supporting points z = (z t , z q ) that describe a state of departure from a node in terms of time z t and charge (SoC) z q .

During the algorithm's execution, labels are extended along nodes in the graph. When a label is extended to a CS node c, we create new supporting points for each breakpoint in B c to which we could charge (that is, breakpoints with a higher energy than that with which we arrived). Consider Figure 2 Figure 2 shows this explicitly for the new supporting point z 4 , corresponding to the decision to recharge to the breakpoint b 4 (more specifically, to b q 4 ). We continue to extend labels along nodes in the graph until the destination node π R is reached, whereat the algorithm returns the time of the first supporting point in the label's SoC function. Bounds on energy and time are established in pre-processing and are used alongside dominance rules during the algorithm's execution to improve its efficiency. a black square for the initial supporting point (z t 1 , z q 1 ). We create additional supporting points (z 2 , z 3 , z 4 ) for each breakpoint to which we could charge, shown by circles b 2 , b 3 , and b 4 in the CS's charging function (left). Axis labels for new supporting point z 4 (right) detail its creation from the decision to recharge to breakpoint b 4 .

The frvcpy Package

frvcpy is an open-source Python-based implementation of the labeling algorithm from [START_REF] Froger | Improved formulations and algorithmic components for the electric vehicle routing problem with nonlinear charging functions[END_REF] for solving the FRVCP. In this section we give an overview of its structure ( §5.1), demonstrate its usage ( §5.2), and briefly comment on its performance ( §5.3). The code accompanying this section can be found at Kullman et al. (2020a).

Structure

frvcpy is a small package (approximately 1000 lines of code) built in the Python 3 programming language; it is available on the Python Package Index and can be installed via "pip install frvcpy." It is comprised of three primary modules: core.py, solver.py, and algorithm.py. The core.py module consists of class definitions for ancillary objects required in the algorithm's execution such as nodes, labels, and the FRVCP problem instance. Module solver.py defines the user-facing Solver class which is responsible for pre-processing, calling the algorithm, and writing solutions to file. The algorithm itself and its accompanying functions are contained in algorithm.py. Additionally, the package contains the module translator.py, which provides an instance translator that allows for interoperability between frvcpy and VRP-REP instances. VRP-REP is a community-driven repository for vehicle routing problem data files; see [START_REF] Mendoza | VRP-REP: the vehicle routing community repository[END_REF] for more details.

Input/output. Users can interact with frvcpy using a Python API or via the command-line interface (CLI). As input, frvcpy requires a compatible instance, the fixed route for the EV to travel, and the EV's initial energy. Compatible instances are JSON files (or equivalent Python dictionaries) following the schema available in Kullman et al. (2020a). After execution, the algorithm returns the optimal energy-feasible route and its duration. The returned route is a list of tuples indicating stops' node IDs and the amount of energy to be recharged there (the latter given by the keyword 'None' for non-CS nodes; see Listing 1, Line 24).

Testing the installation. frvcpy provides simple testing to determine if its installation was successful. From the command line, users can execute the command frvcpy-test to run a suite of tests that performs an instance translation and solves 134 FRVCPs from the [START_REF] Froger | Improved formulations and algorithmic components for the electric vehicle routing problem with nonlinear charging functions[END_REF]

Example Usage

We provide an example demonstrating the use of frvcpy through the Python API. Consider a user with the VRP-REP-compliant instance "vrprep-instance.xml," depicted in Figure 4. The instance contains "fast," "normal," and "slow" CSs whose charging functions are shown in Figure 3. An EV, which begins at the depot with full battery, has been assigned the fixed route Π = (0, 40,12,33,38,16,0), depicted by the gray arrows in Figure 4. Because the EV does not have sufficient energy to traverse Π without recharging, we solve an FRVCP to determine the optimal insertion of charging operations. We can do this using frvcpy as follows: [0,40,12,33,38,16,0] # route to make energy feasible The solution to the FRVCP instructs the EV to recharge at CS 48 between customers 33 and 38, as by the black arrows in Figure 4 and the printed output on line 24 in Listing 1. This results in a total route duration of about 7.34 hours. We note that detouring to CS 48 actually requires more travel time than detouring to CS 41; however, given that CS 48 offers a faster charging speed, it is ultimately preferred over CS 41 (recharging at CS 41 instead of 48 results in an objective of 7.44 hrs).

The above example would be accomplished with the CLI via

Performance

We test the performance of frvcpy over the nearly 30,000 instances comprising the testbed from [START_REF] Froger | Improved formulations and algorithmic components for the electric vehicle routing problem with nonlinear charging functions[END_REF]. These instances have a median route length of 10 stops and a median of 18 CSs that may be inserted. Over the tests, the algorithm has an average run time of 5.6 ms. In addition, in the same tests we find that our translator can translate instances from VRP-REP format in an average of 0.1 s.

These results suggest that frvcpy requires sufficiently low runtime so as to be included in larger solution schemes for E-VRPs.

Conclusion

We introduced frvcpy, a Python-based open-source implementation of the labeling algorithm from [START_REF] Froger | Improved formulations and algorithmic components for the electric vehicle routing problem with nonlinear charging functions[END_REF] for the fixed route vehicle charging problem. The algorithm and our implementation are flexible, able to accommodate realistic problem features such as non-linear recharging times, partial charging decisions, and heterogeneous charging station technologies. Because FRVCPs are often encountered as subproblems of more general EV routing problems, we designed frvcpy to be easily embedded in larger solution schemes. To that end, the package offers two modes of interaction, has minimal requirements, and is computationally efficient. Our hope is that frvcpy facilitates the solution of E-VRPs, lowering the barrier to entry in this field, and ultimately helping bring about a faster transition to more sustainable transportation practices.

Figure 1 :

 1 Figure 1: Instance excerpt depicting a fixed route in the original problem graph (left) and the modified problem graph for the FRVCP (Inset 1), which includes dummy nodes for CS insertions.

  , which depicts this process when extending a label along the edge from customer 33 to CS 48. When it arrives at CS 48, its SoC function has only one supporting point z 1 (assuming the EV has not yet stopped to recharge) depicted by the black square in the right graph of Figure 2. Then for each breakpoint in the CS's charging function to which the EV could recharge (b 2 , b 3 , b 4 ), we add a supporting point to the label's SoC function (z 2 , z 3 , z 4 ) whose time and charge reflect the decision to charge to that breakpoint.

Figure 2 :

 2 Figure 2: Depicting the creation of new supporting points at CS nodes for the case of CS 48 between customers 33 and 38 in Figure 1. Right shows the SoC function of the label extended to CS 48, with

8Figure 3 :

 3 Figure 3: Piecewise linear charging functions for example instance "vrprep-instance.xml."

Listing 2 :

 2 Example frvcpy usage with CLI # translate existing VRP-REP instance, write it to file frvcpy-translate ./instances/vrprep-instance.xml new-frvcp-instance.json frvcpy --instance=new-frvcp-instance.json --route=0,40,12,33,38,16,0 --qinit=16000 --output=my-solution.xml # Duration: 7.339 # Energy-feasible route: # [(0, None), (40, None), (12, None), (33, None), (48, 6673.379615520617), (38, None), (16, None), (0, None)]

Figure 4 :

 4 Figure 4: Depiction of example instance "vrprep-instance.xml." We consider an EV given the route shown by the gray arrows. The solution to the FRVCP for this route instructs the EV to recharge at CS 48 between customers 33 and 38 (black arrows).

  testbed. The same test suite can also be run in Python via

	import frvcpy.test
	frvcpy.test.runAll()

Acknowledgements

Nicholas Kullman is grateful for support from the Société des Professeurs Français et Francophones d'Amérique. This research was also partly funded by the French Agence Nationale de la Recherche through project e-VRO (ANR-15-CE22-0005-01). Justin Goodson wishes to express appreciation for the support from the Center for Supply Chain Excellence at the Richard A. Chaifetz School of Business.


Zhao Zhang

Armelle Brun

Anne Boyer

New Measures for Offline Evaluation of Learning Path Recommenders

Keywords:

Recommending students useful and effective learning paths is highly valuable to improve their learning experience. The evaluation of the effectiveness of this recommendation is a challenging task that can be performed online or offline. Online evaluation is highly popular but it relies on actual path recommendations to students, which may have dramatic implications. Offline evaluation relies on static datasets of students' learning activities and simulates paths recommendations. Although easier to run, it is difficult to accurately evaluate offline the effectiveness of a learning path recommendation. To tackle this issue, this work proposes simple offline evaluation measures. We show that they actually allow to characterise and differentiate the algorithms.

Introduction

Over the last decade, there has been an increasing adoption of technology in our everyday life. Nevertheless, this general adoption faces a main limit: students can be drowned in the large number of learning resources they can access. Educational recommender systems, that aim at selecting and recommending students learning resources, rely on the mining of the traces of interaction of students [START_REF] Venant | Using sequential pattern mining to explore learners behaviors and evaluate their correlation with performance in inquiry-based learning[END_REF][START_REF] Léonard | Patterns and loops: Early computational thinking[END_REF], and are one solution to this problem. The goal of recommender systems can be classified into two time horizons: shortterm goal and long-term goal recommender systems. Short-term goal recommenders suggest users a set of resources to consult next. They are traditional recommenders and are still highly studied. In education, they recommend the next-step learning resource. Long-term goal recommenders recommend users sequences of resources. More and more attention is being paid to these recommenders, that are also called sequence-aware or path recommender systems. In education, recommending a sequence of learning resources is referred to as a Learning Path (LP) recommendation [START_REF] Liu | Exploiting cognitive structure for adaptive learning[END_REF][START_REF] Nabizadeh | Adaptive lp recommender approach using auxiliary learning objects[END_REF][START_REF] Zhu | A multi-constraint learning path recommendation algorithm based on knowledge map[END_REF]. When students adopt a recommended LP, they may improve their level of knowledge, increase their chance of success at exams, maintain their motivation, etc. Obviously, managing the accuracy of such a recommendation is of the highest importance. However, it remains relatively unstudied. In line with traditional evaluation methodologies, the evaluation of the recommendation of a sequence or a LP can be performed either online or offline. Online evaluation is used in a live environment and focuses on the impact of the recommendations on the users. Although highly informative, online evaluation is highly time consuming, requires the availability of real users and is often not completely reproducible. At the opposite, offline evaluation focuses on the accuracy of the recommendations by relying on existing offline datasets. It is thus less costly, which justifies its popularity, but provides a limited estimation of the accuracy of the recommendation model, it is thus more challenging. Although offline evaluation has been highly studied for traditional recommender systems, few evaluation measures and frameworks have been proposed for the evaluation of sequences of recommendations, especially for offline evaluation [START_REF] Monti | Sequeval: An offline evaluation framework for sequence-based rs[END_REF]. The educational domain is highly sensitive when conducting online evaluation, as it relies on real recommendations to students. Indeed, low quality recommendations may negatively impact the students' learning experience, especially for LP recommendations, and can even ruin students' whole academic year. To the best of our knowledge, although offline evaluation of LP recommendations is attracting more and more attention, it still remains a challenge [START_REF] Zhou | Personalized learning full-path recommendation model based on lstm neural networks[END_REF] and there is no commonly adopted evaluation measures. Several measures are simple adaptations of those used in traditional recommender systems, but they do not totally fit the characteristics of the educational context. Other measures, specifically designed for LP, tend to be highly complex and rely on specific data. From this observation, this paper proposes to supplement the set of LP offline evaluation measures, to contribute to a more complete set of measures. These measures are designed to be simple, generic and easy to implement. This paper is organised as follows. Section 2 discusses related work. Section 3 proposes new offline evaluation measures for LP recommendations, which are assessed in section 4. Finally, section 5 concludes this work.

Related Work

Learning Path Recommender Systems

Recommender systems are traditionally designed to recommend a user, referred to as the target user, one resource or a set of resources that fits his/her profile, preferences, etc. These short-term goal recommenders usually focus on the immediate value of the recommendations. Recently, the problem of recommending personalised sequences of resources to reach a long-term goal has emerged, but remains less widespread [START_REF] Nabizadeh | Long term goal oriented recommender systems[END_REF][START_REF] Quadrana | Sequence-aware recommender systems[END_REF]. Sequence recommendation is highly different from traditional recommendation [START_REF] Liu | Exploiting cognitive structure for adaptive learning[END_REF] as it does not only exploit the temporal ordering for the generation of sequences of resources, but also contribute to reach a goal along the sequence.

In education, LP recommenders aim at identifying the most appropriate sequence of learning resources that fits the target student's preferences, capability or increases the chance of reaching a long-term goal (success at exam) [START_REF] Durand | Graph theory based model for learning path recommendation[END_REF][START_REF] Zhou | Personalized learning full-path recommendation model based on lstm neural networks[END_REF].

Several approaches have been proposed for sequence recommendations. For example, [START_REF] Feng | Personalized ranking metric embedding for next new poi recommendation[END_REF] relies on Markov chains to manage transition probabilities between and generate a sequence. However, this may not guarantee to reach the long-term goal, as the LP is made up of a series of "local" recommendations.

In the educational context, several approaches have been proposed, that differ not only in the data and information they exploit, but also in the way the paths are formed. Recently, [START_REF] Zhou | Personalized learning full-path recommendation model based on lstm neural networks[END_REF] relied on deep learning, precisely LSTM associated with clustering. [START_REF] Dwivedi | Learning path reco. based on modified variable length genetic algorithm[END_REF] focused on students learning styles and knowledge levels with a variable length genetic algorithm to recommend personalised LPs. Few years ago, Bayes theorem, Association Link Network, Item Response Theory, graph theory were studied to form and recommend LPs.

When developing a recommender system, an important dimension lies in its evaluation. As previously mentioned, two scenarios are highlighted: online and offline evaluation. The two following sections introduce some works conducted for both settings for LP recommendations.

Online Evaluation of LP Recommendations

Generally, online evaluation aims at quantifying the actual effect of the recommended LP on the students learning experience. Concretely, such an evaluation exploits the students' level of knowledge: results at exams, tests, quizzes, etc. at different times. Notice that commitment, motivation, etc. are rarely considered.

[10] proposed to use A/B testing, one of the most popular and simple online evaluation form. Students are divided into two groups: one group gets personalised LP recommendations, while the other group gets non personalised LP recommendations. After getting recommendations, the students in each group take the final exam. The average score in each group are compared. The main advantage of this approach is that it is easy to implement and the measure is very simple as it does not require additional elements However, from our point of view, the fact that the second group gets recommendations (non personalised) introduces a bias in the evaluation. [START_REF] Xie | Discover learning path for group users: A profile-based approach[END_REF] went a step further in the evaluation approach. The students' performance is not only compared between groups, but their performance before and after adopting a LP is also considered. Concretely, a single test is used, which is taken twice by the students. A pre-test score is associated to the first time the test is taken and a post-test score is associated to the second time the test is taken, i.e. after adopting a LP. The students are randomly divided into three groups: those who follow a self-decided LP, those who follow a non personalised LP, and those who follow a personalised recommended LP. Self-decided LPs are determined by students themselves (obtained by questionnaires for example) as proposed in [START_REF] Zhu | A multi-constraint learning path recommendation algorithm based on knowledge map[END_REF]. In our view, the evaluation conducted here is more fine grained.

Although online evaluation is accurate, we can see from these works that conducting such an evaluation is costly, mainly in terms of time required and stakeholders availability (including students availability). We identify two additional limitations. First, if the recommendation model is not accurate, the recommendations proposed to students may negatively impact their learning experience.

Second, it cannot guarantee student equity, especially when students are split into groups and get different types of recommendations. This is the reason why offline evaluation is generally used before conducting online evaluation.

Offline Evaluation of LP Recommendations

Offline evaluation relies on predefined static datasets. In education, these datasets are mainly made up of students' activities, i.e. students' LP. At the opposite of online evaluation, no real LP recommendations and no test can be actually proposed to students. So, to evaluate the accuracy of LP recommendations, the ground-truth is thus required. This ground-truth can be inferred from the data, obtained from experts, or estimated from external knowledge [START_REF] Rossetti | Contrasting offline and online results when evaluating recommendation algorithms[END_REF]. Inferring ground-truth from the data is the most simple technique as it requires only data. Nevertheless, as it relies on data obtained independently of the recommender system under evaluation, it is always uncertain.

The literature assumes that for each target student one ground-truth LP exists, i.e. a LP that conducts to the expected goal, for example success at exams. The challenge is thus twofold: (1) identify the ground-truth LP, that is supposed to exist in the dataset, (2) compare this LP to the recommended LP. The measures proposed in the literature differ in these two elements.

However, although many measures have been proposed for traditional recommender systems, evaluating LP is far less popular. We can still identify two types of measures: those commonly used in many other domains and adapted to LP recommendations, and those specifically designed for LP recommendations.

Measures Used in Traditional Settings Precision is the most commonly used evaluation measure in information retrieval, classification, prediction and recommendation [START_REF] Quadrana | Sequence-aware recommender systems[END_REF]. It represents the fraction of recommendations that are relevant. In the frame of LP recommendations, precision has been adapted to fit the complex nature of the LPs. It is often assumed that one LP is a series of resource recommendations, where each resource recommendation is viewed as either a good or a bad resource. Precision identifies the resources in the estimated LP that are also part of the reference LP: one recommended resource that is part of the reference LP is a good resource [START_REF] Hsieh | A mining-based approach on discovering courses pattern for constructing suitable learning path[END_REF]. One main drawback of this technique is that it does consider not consider the order between the resources.

Conversely, [START_REF] Su | Designing and developing a novel hybrid adaptive learning path recommendation system (alprs) for gamification mathematics geometry course[END_REF][START_REF] Zhou | Personalized learning full-path recommendation model based on lstm neural networks[END_REF] considered a LP as a whole: good LP or a bad LP. A LP is considered as good if its estimated learning effect on the target student is greater than the expected student's learning effect when no recommendation is proposed. The associated precision measure thus represents the proportion of good LPs. In this work, the evaluation of the effect relies on both student and resource descriptions, as well as on results at tests. The main disadvantage of this measure lies in its complexity: it requires much information and the estimation of several elements: each resource degree of difficulty, each student's learning ability, the effect of a resource on a student, and the effect of a LP on a student. [START_REF] Monti | Sequeval: An offline evaluation framework for sequence-based rs[END_REF] has recently proposed a framework for offline evaluation of sequence-aware recommenders, that relies on the combination of 8 traditional measures used in recommendation, among which precision, coverage, diversity.

Another part of the literature proposes to rely on the evaluation of the distance between the recommended LP and the ground-truth LP. Concretely, the well-known Edit Distance (ED) is the most commonly used measure [START_REF] Zhu | A multi-constraint learning path recommendation algorithm based on knowledge map[END_REF]. ED simply counts the minimum number of operations required to transform one of the sequences into the other. In [START_REF] Zhu | A multi-constraint learning path recommendation algorithm based on knowledge map[END_REF], ground-truth LPs are determined by asking the students self-organized LP (through questionnaires), which is the only interaction with students. One main drawback in this work lies in the fact that students have to be able to self-determine their LP.

Measures Designed for LP The measures presented above are not specifically designed for sequence or path evaluation, nor for the educational domain. We will focus now on some of the rare measures recently proposed in the specific frame of LP recommendations.

[8] proposed two measures: promotion and logicality. Promotion represents the increase in the knowledge of students. This measure is close to the one presented in [START_REF] Xie | Discover learning path for group users: A profile-based approach[END_REF][START_REF] Nabizadeh | Adaptive lp recommender approach using auxiliary learning objects[END_REF], mentioned in the online evaluation section. The second measure proposed is more original, it represents the logicality of a LP, with respect to the knowledge structure in the recommended LP. This evaluation is not only performed automatically, by relying on the knowledge structure of the resources, but more originally, it is also performed by experts in education. [START_REF] Nabizadeh | Estimating time and score uncertainty in generating successful learning paths under time constraints[END_REF] proposed to automatically associate each resource with a score and a learning time, that can be estimated by several approaches such as the well known item response theory (IRT) [START_REF] Embretson | Item response theory[END_REF]. Given a recommended LP, its accuracy is estimated by using the traditional Mean Absolute Error (MAE) between the real LP score and time, and the recommended path's estimated score and time.

New Measures for Offline Evaluation of LP

The previous section has highlighted that the literature lacks of well-adopted and well-defined offline evaluation measures for LP recommendations. In addition, we have seen that most of them requires additional information. For example, they require either expert knowledge, learning resource information: content, difficulty level, structure, or student knowledge: learning ability, level of knowledge, etc. They are thus highly complex to implement and may not be adapted to many contexts. This may partially justify their lack of success.

In this section, we propose new measures that cope with these limits. They are designed to be simple so that they can be easily and widely adopted, whatever are the contexts, datasets, even goals. Concretely, they rely only on students' traces of activities (their LP) and students' performance (results at exams or quiz for example). No content information or difficulty level about learning resources or about students is required. In line with the literature, they rely on a groundtruth LP. These measures do not require that this ground-truth LP is defined by experts. It is automatically inferred from the dataset. They vary not only in the way the ground-truth LP is defined, but also in the way the link between the recommended LP and the ground-truth LP is used.

As for online evaluation, two focus elements have guided the design of these measures. First, a recommended LP should be in line with the target student's learning behavior. Second, a recommended LP should improve the student's level of knowledge. A downside of the simplicity of the measures is that they do not fit all the students' level. Thus, for each measure, we will indicate the student level(s) it is mainly dedicated to. In addition, these measures may be combined during evaluation.

Problem definition

The problem can be defined as follows. Let t be a target student, known by [START_REF] Durand | A learning design recommendation system based on markov decision processes[END_REF] the sequence of resources he has consulted so far: his starting LP (LP start (t)), (2) his estimated level knowledge, from the grades obtained for the evaluations he took (level(t)). RLP (t) is the recommended LP, i.e. the estimated LP that t should adopt as his ending LP (LP end (t)).

An evaluation measure evaluates the accuracy of the LP recommended to t (RLP (t)). To reach this goal, it relies on a set of students S from the dataset and the associated traces of activity (their LP) (LP (s), s ∈ S), as well as their estimated level (level(s), s ∈ S).

The problem is thus: how to determine the ground-truth LP from the dataset and how to compare it to RLP (t)?

Mentor Students Based Measures

The first two measures introduced in this section rely on the hypothesis that students who have a high level of knowledge (i.e. get good grades at exams, quizzes, etc.) have an adequate learning behavior, i.e. they adopt an adequate LP. Their ending LP (LP end (s), s ∈ S) is thus considered as the ground-truth LP. These students are considered as "mentor" students.

The exact way mentor students are chosen and the way the comparison of both LPs is taken into account varies between measures.

The GSLP Measure The Good Students LP based measure (GSLP) simply defines a mentor student m as a student with a high level of knowledge (level(m)) (i.e. has good grades), whatever is his LP. The set of mentor students in the set S of students (m(S)) is thus shared by all target students and can be defined beforehand the evaluation of GSLP. GSLP considers that if the recommended LP of a target student (RLP (t)) is close to the LP of a mentor student (LP end (s), s ∈ S), the recommendation is accurate.

The main focus element of GSLP is thus the increase in knowledge of the students. Indeed, considering that the paths adopted by good students lead to a high level of knowledge (as the students who adopt them get good grades at exams), if the recommended LP is close to the one adopted by a good student, we can say that this path leads to an increase in the student's level of knowledge. The recommendation is thus accurate. GSLP represents the proportion of target students with an accurate recommendation, as presented in Equation [START_REF] Durand | A learning design recommendation system based on markov decision processes[END_REF].

GSLP = |t, ∃m ∈ m(S) ∧ d(RLP (t), LP end (m)) < c| |S| ( 1
)
where m is a mentor student, d(L 1 , L 2 ) is a distance measure between two LPs L 1 and L 2 and c is a maximal distance threshold. The larger GSLP, the the recommender system. Notice that the distance measure can simply rely on the number of resources shared in both LP (as in traditional metrics), or on the traditional Edit Distance.

Obviously, GSLP has several limits due to its simplicity. First, it does not consider the fact that several mentor students may have a LP close to the recommended LP of a target user, which could improve the accuracy of the measure. In addition, as in the traditional precision measure, the distance value is not explicitly taken into account in the evaluation of GSLP.

Notice that GSLP has not been designed for students with a low level(t) value. Indeed, it may not be adequate to consider that the LP adopted by good students is a good LP for other students, due to the large difference in their levels of knowledge, their LPs and their learning ability. Thus the values of GSLP for students with a low level(t) value may not make sense.

The SSLP Measure The Similar Students LP based measure (SSLP) refines GSLP. A mentor student m is first of all a student who adopted a LP (LP start (m)) close to the one of the target student (LP start (t)). Second, a mentor student should be a successful student (as for GSLP).

At the opposite of GSLP, SSLP ensures that a mentor student (m(t)) has a learning behavior (LP) close to the one of the target student, so the main focus element of this measure is the learning behavior of the target student. The improvement in level of knowledge ranks second. Another difference with GSLP is that SSLP uses the distance value between the recommended LP of a target student (RLP (t)) and the actual LP adopted by his mentor student (LP end (m(t))), not only the fact that a close path exists.

SSLP represents the average distance between the recommended LPs and the ground-truth LPs. SSLP is evaluated as defined in Equation ( 2). The smaller SSLP, the better the recommender system.

SSLP = |S| t=1 d(RLP (t), LP end (m(t))) |S| (2)
As for GSLP, several distance measures can be adopted, as ED.

Difference in LP Based Measure (DLP)

The DLP measure addresses the limitation highlighted in GSLP: it is probably not adequate to consider that the recommended LP of a student who does not comply with the expected level (a promising student) should be close to the one adopted by a student who does (a good student). Intuitively, it should even be a highly different LP. DLP is in line with this intuition: it evaluates to what extent the recommended LP of a target student, especially a promising student, is different from the ones adopted by good students. To avoid that a non-sense recommended LP is considered as a good recommended path (as it is highly different from any LP adopted by good students), this distance is also compared to the one with average students (with level(s) close to the average level). Assuming that average students tend to have LPs less different from the ones of promising students, the paths recommended to average students should be closer to the ones recommended to promising students. Concretely, DLP represents to what extent the distance between the recommended LP of a target student and the actual LP of a good student is larger than the distance between the recommended LP of a target student and the actual LP of an average student. The highest this value, the better the recommendation.

The main focus element of DLP is thus the increase in the level of knowledge and at the second rank the learning behavior. Equation (3) presents the exact way this measure is evaluated.

DLP = |Sp∪Sa| t=1 d(LP end (m g (t)), RLP (t)) -d(LP end (m a (t)), RLP (t)) |S p ∪ S a | (3)
where S p and S a are respectively the set of promising and average students, m g (t)) is the mentor student of t in the set of good students and m a (t)) is the mentor student of t in the set of average students. A mentor student (m(t)) is chosen as the student with the closest starting LP with t. At the opposite of the previous measures, DLP is dedicated to promising and average students.

Performance in LP based Measure (PLP)

As for the three first measures, PLP considers the ending LP of a mentor student as the ground-truth. PLP requires that at least two temporally distant evaluations have been taken by students, noted E 1 and E 2 (E 1 being taken before E 2 ). Given a target student t, PLP relies on the following assumption: if there exists a student m with a level of performance associated to E 1 similar to the one of t, and with a level of performance associated to E 2 higher than the one of t, this means that the LP adopted by m is more adequate than the one adopted by t (as it leads to a higher increase in performance). m can thus be considered as the mentor of t. A similar assumption is made about m's ending LP. As a consequence, the closer the recommended LP of t to the ending LP of m, the better the recommendation. m is chosen as the student that meets a trade off between minimum distance in the levels of knowledge for E 1 and maximal distance in the levels of knowledge for E 2 .

The main focus element of PLP is thus the increase in the level of knowledge of students, the learning behavior of t is not considered at all.

P LP = |S| t=1 d(RLP (t), LP end (m(t)) |S| with level(m(t), E 1 ) = level(t, E 1 ) ∧ level(m(t), E 2 ) > level(t, E 2 ) (4)
level(t, E 1 ) represents the grade obtained by t at the evaluation E 1 . Notice that at the opposite of the previous measures, P LP can be used for each student, from promising to good students.

PLP and SSLP both evaluate the average distance between target students and their mentor student. They differ in the way their mentor student is selected: SSLP focuses on the learning behavior of students (LP), and PLP focuses on their level of knowledge.

EVALUATION

In this section we describe the experiments we conducted to assess the evaluation measures that we introduced into the previous section.

Dataset and Experimental Setup

The dataset used is made up of the traces of interactions of students with learning resources on their LMS. These students are first-year university students and are enrolled in a Mathematics and Computer Science Bachelor program. The traces represent the interactions for one specific course: "algorithms and programming" from the Fall Semester in 2018, which is a core course of this program. The students' LP are made up of the sequence of all accessed resources. Diverse learning materials are available: slides, exercises for lab sessions, quizzes and final exam. Each learning record represents one student activity, and includes (1) student ID, (2) timestamp of the interaction (3) type of access (view, submit), (4) resource ID ( 5) score (for quizzes and exam).

In this dataset, we identify two periods. The course period, where students access resources after they are uploaded by the teacher. Some activities are imposed by the teachers, other activities are optional and are mainly adopted on the students' initiative. The review period, where all learning resources are available and students review on their own initiative for the final exam.

Before the course starts, each student's level of knowledge is assessed on an initial quiz. At the end of the semester, each student takes a final exam. Besides, an optional mid-term quiz is proposed to students in the middle of the course period. Some of the students do not take the initial or mid-term quiz, but everyone took the final exam.

To perform a detailed analysis of the measures, we propose to divide students into three groups, based on their estimated level of knowledge. This level of knowledge is estimated with a weighted average of the quizzes and exam scores. The three groups are good students (GS), average students (AS) and promising students (PS). The split criterion between groups has been determined by the teacher of the course, on the basis of his personal experience. As a result, GS represents the top 30% students, AS represents the 40% following students, and the remaining students are PS. As the number of students in each group is almost similar, we can consider that the dataset is well-balanced and that students' level will not bias the models trained. Table 1 presents an overview of the dataset. First of all, we can see that this dataset is not a large dataset, which will be considered in the experiments. We can also notice that students work more during the course period than during the review period, about 3.5 times more for each group. This can be due to the duration of the periods (course is 1.5 times longer than review), but it also means that students work less by themselves, which justifies the need of recommendations. In addition, good students tend to have a larger amount of activity than other students, but they do not tend to review neither more, nor less than promising students, w.r.t. their initial amount of activity. To refine this analysis, Figure 1, displays the number of activities (learning records) for each group of students. Students are sorted in descending order of their number of activities (1 point every 3 students). This figure confirms that the amount of activity of students differs between groups. Although we can see that the students who work the more during the course period are good students, we cannot say that students that have a low activity are promising students. GS and AS tend to have a similar number of activities during the review period. Since the dataset is quite small in terms of number of students, number of resources and number of activities, we propose to adopt the leaving one out cross-validation method. One student (and his associated activities, whether the course of review period) is used as test (called the target student) and the remaining students form the training set. The traces of activities of this target student are split into two: course and review periods. The traces from the course period will be used for computing the recommended LP. The traces from the review period will not be used at all. Let us notice here that we decide to not compare the recommended LP to the actual LP of the students, as from our view this does not make sense for PS, and only a little bit for AS.

As mentioned in [START_REF] Liu | Exploiting cognitive structure for adaptive learning[END_REF], the length of the RLP is required by any algorithm. In accordance with the literature, it will be set as the median length of the students' LP in the review period. It is thus fixed to 10 (see Table 1).

We propose to rely on two well-adopted LP recommendation models: Sequential Pattern Mining (SPM) [START_REF] Su | Learning portfolio analysis and mining for scorm compliant environment[END_REF][START_REF] Vesin | Applying recommender systems and adaptive hypermedia for e-learning personalizatio[END_REF] and Markov Chain (MC) [START_REF] Taraghi | Markov chain and classification of difficulty levels enhances the learning path in one digit multiplication[END_REF]. Similarly to [START_REF] Vesin | Applying recommender systems and adaptive hypermedia for e-learning personalizatio[END_REF], the recommended LP of the target student is formed by exploiting the patterns from the model, that fit his traces during the course period. The MC algorithm considers the recommendation as a decision making problem, and forms the recommended LP by exploiting Markovian properties [START_REF] Durand | A learning design recommendation system based on markov decision processes[END_REF]. The recommended LPs are formed by a naive optimal policy calculated by MC.

Recall that the goal of the evaluation is not to identify the best recommendation algorithm, but to study the evaluation measures. That is the reason why we did not choose the most evolved recommendation algorithms, but simple algorithms that we can thoroughly understand. As mainly proposed by the literature, the distance measure we use is the simple Edit Distance.

Experimental Results and Analysis

Each recommendation algorithm (MC and SPM) is trained and used to recommend each target student, in turn, a LP. As previously mentioned, to perform a detailed analysis, the set of target students is split into the three previously introduced groups (GS, AS, PS) and the four measures are evaluated for each group. Table 2 presents the associated values. GSLP measures to what extent the students' recommended path is close to at least one (ending) LP of good students. As GSLP does not perform any strict selection of mentor students (except the fact that they are good students), it can be seen as the upper bound of the accuracy. It is expected that the values for AS and GS are high, for both algorithms. It is actually the case: for AS and GS, both algorithms tend to recommend LPs adopted by good students. The highest values are obtained for GS (0.7 and 0.6), which means that from 60% to 70% of good students receive a LP that is a LP close to the one adopted by good students. Notice that we expected that this value would be higher. This may be due to the fact the training set is made up of traces of all students' levels and that both MC and SPM do not consider the students' level during training. So, AS and PS influence the recommended LP. A simple experiment on GS only (for train and test) shows a small improvement (> 0.8), which confirms the impact of promising students. SPM has a lower value than MC, whatever is the group. This was expected as SPM may suffer from overfitting in case of lack of data, which is the case of the dataset used. Notice that the values for PS are the lowest ones. As mentioned in section 3.2, this value may have non sense as recommending a promising student the path adopted by good students may not be adequate. We confirm that GSLP is an interesting measure. It highlights differences between both algorithms, and shows that MC tends to recommend paths close to those adopted by good students. SSLP focuses on students' learning behavior, it evaluates to what extent the recommended LPs are close to the ones adopted by students with a similar learning behavior, and who also succeed in their exam. Obviously, none of both recommendation algorithms recommend LPs with this explicit purpose. First, we can see that SPM recommends LPs that better fit the students' learning behavior, which can be explained by the fact that SPM manages long patterns that better reflect students' learning characteristics. Average students are those who get recommendations that respect more their learning behavior, for both algorithms. This is probably explained by the number of possible students who adopt a close LP (some may be part of GS and others of PS). We confirm that SSLP is also an interesting measure, that shows the ability of the algorithms to recommend LP that are in accordance with students' LP. Here also, a difference between both algorithms is highlighted, for SSLP it is in favor of SPM.

PLP focuses on the increase in the level of knowledge of students, while respecting the students' learning behavior. As for SSLP, this information is not explicitly considered by both algorithms. First, we can see that, at the opposite of SSLP, MC is the most adequate recommendation algorithm in the sense of PLP: the paths recommended by MC are closer to the paths adopted by the students who increase more their level. For both algorithms, PS tend to get recommendations that actually correspond to an increase in their level, which can be justified by the fact that the training set is mainly made up of traces of activity of students that have better grades. AS for SSLP, the values associated to GS are not that good. They tend to not get recommendations that increase their level. A thorough study explains that this is due to a lack of 'candidate' mentors for good students. An additional experiment confirms that promising students negatively influence the recommendations made to good students.

DLP is totally different from the other three measures and is used for promising students only. It identifies to what extent the recommendation algorithms tend to recommend PS LPs that are more different from the ones adopted by good students than the ones adopted by average students. As previously mentioned, all groups are almost equally represented in the training dataset, which guarantees that there is no bias due to their weight. For both algorithms, the DLP value is positive, it confirms that the paths recommended to PS are closer to those adopted by AS, which is a positive finding. As mentioned in section 3.2, promising students should not receive LPs that are too similar to those adopted by good students. In addition, MC has the largest value, which shows that the recommendations proposed by MC to promising students are highly different from the ones adopted by good students, so they may fit the target students better.

Throughout these experiments we confirmed the usefulness of the measures proposed. They contribute to highlight some specificities of each of the recommendations algorithm studied. As mentioned previously, the algorithms used are simple algorithms, but we can analyse their recommendations and assess the relevance of the measures. Thanks to these measures, we can say that MC tends to recommend paths that are close to the ones adopted by average students, and that it recommends LPs that increase the level of knowledge of students. As for SPM, we can conclude that it is more adequate to recommend LPs that fit students' learning behavior. Let us notice that if only one recommendation algorithm is used, the values of the measures, especially SSLP and PLP can be compared to a baseline that represents the average distance between the recommended LP and the actual ending LP of the target students.

Conclusion

Based on the fact that the literature lacks of offline evaluation measures for LP recommendation algorithms, this work has proposed new measures to contribute to such an evaluation. These measures are designed to be simple measures. First, they rely on few information: only students traces of activity and results at exams, so that they can be used in many datasets. Second, they are designed to fit the educational domain, at the opposite of several measures of the literature. Indeed, they are designed to represent elements such as the increase in knowledge level and the fitting to students' learning behavior. These measures have been assessed on recommendation algorithms of the literature. These algorithms have been chosen for their simplicity and the exact understanding of the recommendations performed.

In a future work, we plan to conduct a thorough analysis on more and larger datasets, as well as a more evolved distance measure to draw stronger conclusions. We also plan to propose additional measures to conduct a thorough analysis of the recommendation algorithms of the literature.

Fig. 1 :

 1 Fig. 1: Students' activities by group during both course and review periods

Table 1 :

 1 Overview of dataset

		Indicators	Values
		Number of students/resources	104/39
	Course period	Number of learning records	3,279
		Median number of students' learning records	46
		GS/AS/PS average number of learning records	60/49/35
	Review period	Number of learning records	693
		Median number of students' learning records	10
		GS/AS/PS average number of learning records	17/13/11

Table 2 :

 2 Evaluation measures for each recommendation algorithm and group

	Evaluation Measure Student group		Recommendation
			MC	SPM
	GSLP	PS / AS / GS 0.5 / 0.6 / 0.7 0.4 / 0.5 / 0.6
	SSLP	PS / AS / GS 6.9 / 7.3 / 7.9 6.8 / 5.4 / 5.8
	PLP	PS / AS / GS 5.4 / 5.6 / 6.8 5.7 / 6.3 / 7.9
	DLP	PS	3.5	1.4