Franco Marchesoni-Acland

Philippe Lauret

Alvaro Gómez

Rodrigo Alonso-Suárez

Analysis of ARMA Solar Forecasting Models Using Ground Measurements and Satellite Images

Keywords: Forecasting, solar irradiance, adaptive filters, satellite images

As the solar photovoltaic (PV) share in the electricity grid is growing year by year, solar irradiance forecasting is becoming increasingly important. In this work the performance of a recursive formulation of ARMA models suitable for operational context using the Pampa Húmeda region as a case study is analyzed. Results are promising, as this simple adaptive algorithm does not require historical data and outperform persistence at all lead times. The improvement produced by adding satellite cloudiness data and short-term local variability as exogenous inputs is also evaluated. It is found that the spatially averaged satellite albedo is a useful input variable, improving the forecast performance, while the introduction of short-term variability produce negligible performance changes under this kind of models.

I. INTRODUCTION

Solar energy is being increasingly used as a means of producing electricity [START_REF]Renewables 2018: Global status report[END_REF]. As an example, in 2017, the milestone of having more than 400 GW worldwide installed solar photovoltaic (PV) capacity was achieved. Due to the intermittent character of solar energy, increasing the solar energy share into the electricity grid pose a challenge for its stability. Grid utilities need accurate forecast at different granularities and time horizons for an efficient electricity dispatch [START_REF] Mathiesen | Chapter 14 -case studies of solar forecasting with the weather research and forecasting model at glgarrad hassan[END_REF]. Predicting the solar irradiance at ground level is the first step for solar PV power forecasting.

There are four main techniques for addressing solar energy forecast [START_REF] Coimbra | Chapter 8 -overview of solar-forecasting methods and a metric for accuracy evaluation[END_REF]- [START_REF] Diagne | Review of solar irradiance forecasting methods and a proposition for small-scale insular grids[END_REF]: Numerical Weather Prediction (NWP) models [START_REF] Perez | Forecasting solar radiation -Preliminary evaluation of an approach based upon the national forecast database[END_REF]- [START_REF] Mathiesen | Evaluation of numerical weather prediction for intra-day solar forecasting in the continental united states[END_REF], satellite forecast by means of Cloud Motion Field (CMF) estimation [START_REF] Lorenz | Short term forecasting of solar radiation based on satellite data[END_REF], all-sky camera forecast that track clouds in the very short term [START_REF] Marquez | Intra-hour DNI forecasting based on cloud tracking image analysis[END_REF] and statistical and machine learning tools, commonly referred as time series forecast [START_REF] Reikard | Predicting solar radiation at high resolutions: A comparison of time series forecasts[END_REF]- [START_REF] Lauret | A benchmarking of machine learning techniques for solar radiation forecasting in an insular context[END_REF]. In this context, several authors have proposed forecasting models, such as Neural Networks (NN) [START_REF] Mellit | A 24-h forecast of solar irradiance using artificial neural network: Application for performance prediction of a grid-connected pv plant at trieste, italy[END_REF]- [START_REF] Lima | Forecast for surface solar irradiance at the brazilian northeastern region using nwp model and artificial neural networks[END_REF] or different kinds of Auto-Regressive Moving Average (ARMA) models [START_REF] David | Probabilistic forecasting of the solar irradiance with recursive ARMA and GARCH models[END_REF]- [START_REF] Huang | Forecasting solar radiation on an hourly time scale using a Coupled AutoRegressive and Dynamical System (CARDS) model[END_REF], being the latter the focus of this work.

ARMA models are a tool to describe stationary processes originally proposed by Whittle [START_REF] Whittle | Hypothesis testing in times series analysis[END_REF] and popularized by Box and Jenkins [25]. These models are widely used in economics and other fields where time series analysis and modeling are an unavoidable task. General ARMA models can provide multivariate predictions by modeling each observation as a vector or an univariate prediction for an unique target variable.

The prediction may have also extra inputs variables, known as exogenous variables. In order to produce a forecast, the ARMA models linearly combines past observations and residuals of both endogenous and exogenous variables. Despite the strong hypothesis that the formal theory holds behind, such as stationarity and linear relations between the variables, they proved to be a good forecasting technique in several fields [START_REF] Valipour | Comparison of the arma, arima, and the autoregressive artificial neural network models in forecasting the monthly inflow of dez dam reservoir[END_REF]- [START_REF]Short-term load forecasting via arma model identification including non-gaussian process considerations[END_REF]. Solar irradiation forecasting have not been an exception. ARMA models have been used for Global Horizontal Irradiance (GHI) prediction [START_REF] Coimbra | Chapter 8 -overview of solar-forecasting methods and a metric for accuracy evaluation[END_REF], [START_REF] Bacher | Online short-term solar power forecasting[END_REF], [START_REF] David | Probabilistic forecasting of the solar irradiance with recursive ARMA and GARCH models[END_REF] and consistently proved to obtain reduced estimation uncertainty when compared with persistence and climatology forecasting. They also proved to be competitive versus NN approaches [START_REF] Boland | Short term solar radiation forecasting: Island versus continental sites[END_REF], [START_REF] Benmouiza | Small-scale solar radiation forecasting using arma and nonlinear autoregressive neural network models[END_REF]. An ARMA model can be formulated in many different ways: as a fixed digital filter, as a Kalman filter, as an adaptive filter or in a recursive manner.

A comprehensive study of the forecast errors when using an ARMA model alongside a Generalized Autoregressive Conditional Heteroskedasticity (GARCH) model is presented in [START_REF] David | Probabilistic forecasting of the solar irradiance with recursive ARMA and GARCH models[END_REF]. Only the solar radiation time-series is used as input variable. The ARMA models inspected are framed in the Recursive Least Squares (RLS) technique, which is the same that we use in this work. A specific model order was selected for each place and lead time, and the uncertainty of this approach for each location is reported. In [START_REF] Mazorra-Aguiar | Solar radiation forecasting with statistical models[END_REF] ARMA models framed in a train-test approach are compared to other models. In particular, ARMA models are compared to a NN that includes solar satellite estimates retrieved from the Helioclim-3 database (version HC3v5). The solar estimates available in this platform are based on the Heliosat-2 model [START_REF] Rigollier | The method Heliosat-2 for deriving shortwave solar radiation from satellite images[END_REF] and images from the MSG (Meteosat Second Generation) geostationary satellite administrated by ESA (European Space Administration). The advantage of including solar satellite estimates becomes explicit. This is also shown in a similar but previous work over Gran Canaria Island [START_REF] Aguiar | Use of satellite data to improve solar radiation forecasting with bayesian artificial neural networks[END_REF].

In this work, the introduction of two exogenous variables in ARMA models in a RLS framework for forecasting ground level solar irradiation at an horizontal plane (GHI), using the Pampa Húmeda region (south-east of South America) as a case study, is explored. In particular, the effect of adding a short-term variability index, calculated from recent past ground measurements values, and satellite cloudiness information, obtained from the GOES-East satellite administrated by NOAA (National Oceanic and Atmospheric Ad-ministration), is tested. In this case, satellite cloudiness is included directly without being processed by a solar resource assessment model. It is found that including cloudiness satellite information significantly improves the performance of the ARMA-RLS forecast, while short-term local variability has a negligible influence. A discussion regarding the order selection of the ARMA filters for GHI forecasting in the region is provided. It is shown that a fixed order can be used for all lead times without any relevant performance loss in comparison to an optimal filter. This reduces the problem to an appropriate choice of exogenous inputs. These observations may be extrapolated to others regions or particular sites, an analysis that, to the best of our knowledge, is not found in the literature. Finally, the work also seeks to provide a first in-detail assessment of the forecasting performance of these kind of methods for the region.

The work is organized as follows: in Section II the data is presented by describing the equipment, the stations' characteristics and the exogenous variables being used. In Section III the ARMAX model and the RLS algorithm are presented, with a description of the advantages of this approach. In Section IV the forecast performance metrics to be used in evaluation are introduced. A discussion of the order selection for the ARMA models based only on past ground measurements follows in Section V. In Section VI the results including the exogenous variables in the ARMAX model are exhibited. Finally, the work is concluded in Section VII. The code used for this work is available at https://github.com/ franchesoni/ARMAXrls-solar-irradiance-forecasting. II. DATA GHI data recorded in six measuring stations in Uruguay is used. These stations are representative of the subtropical climate of the south-east part of South America, known as Pampa Húmeda. The sites' locations are presented in Table I. One of these sites (LE) is located at the Solar Energy Laboratory (LES, http://les.edu.uy) experimental facility, where a Kipp & Zonen Solys2 ground station records the three main irradiance components (global, direct and diffuse). The GHI in this station is recorded using a Kipp & Zonen CMP10 Secondary Standard. The other four stations are located on field in semi rural environments. They are equipped with First Class or better pyranometers for the GHI measurement and a data logger with remote communication. These pyranometers are calibrated every two years against a Kipp & Zonen CMP22 Secondary Standard that is kept with traceability to the World Radiometric Reference in the World Radiation Center, Davos, Switzerland. At least two years data at a 10 minutes granularity from the period 2015-2017 is considered for each station.

In this work, the effect of adding short-term local variability and satellite information to a RLS auto-regressive point forecast is inspected. The variability index (σ) is computed from the past ground measurements as the standard deviation of the last five clear-sky index changes,

σ(t) = 1 M M -1 i=0 ∆k c (t -i) -∆k c 2 , (1)
with M = 5, where k c is the ratio between the GHI measurements and the clear sky GHI as estimated by a clear sky model, ∆k c (t) = k c (t) -k c (t -1) are the changes in this variable and ∆k c is the average of the M considered changes. The McClear model [START_REF] Lefèvre | McClear: a new model estimating downwelling solar radiation at ground level in clear-sky conditions[END_REF], publicly available at the SoDa platform (http://www.soda-pro.com), is used for calculating the clear-sky index from the GHI time series. In order to avoid variability issues related to the interface between two consecutive days, the variability is set to zero at the beginning of a day and only current day samples are used for its calculation. Satellite images are also required. GOES-East's satellite Earth Albedo (ρ p ) averaged in a 10 min × 10 min latitudelongitude cell is used. The spatial average of the satellite information is intended to include near future information of the cloudiness, as it is not only taking into account the cloudiness over the specific site, but also the surroundings. The GOES13 satellite operated in the GOES-East position during the period 2015-2017, providing an irregular time-slot acquisition for South America. Satellite images for this region are usually available at a rate of two images per hour, but periods with hourly or tri-hourly images may occur. The 10minutes time resolution for the satellite albedo was obtained via a linear interpolation. Satellite gaps of more than two consecutive hours are not interpolated and are removed from the data set. III. ARMAX FILTERING Auto-Regressive (AR) and Moving-Average (MA) models with Exogenous Variables (ARMAX) describe a process as a linear combination of past measurements (X t-i ), past errors ( t-i ) and exogenous variables (E t , a vector which may include past values). If p and q are the orders of the AR and the MA terms respectively, we have,

X t = p i=1 α i X t-i + q i=1 β i t-i + γ T E t + t , (2)
where t is a deviation from the model at time t, assumed to be white Gaussian noise. For later forecasting use, the set of parameters α, β and γ that best fit the data need to be found. For this the RLS algorithm is used, explained in the following. The predicted variable is the clear-sky index (X t = k c (t)) which is then converted to a GHI point forecast using the McClear clear-sky model. The signals σ(t) and ρ p (t) are the variables inspected for E t .

Recursive Least Squares (RLS) is an optimization algorithm that solves recursively the minimization of a cost function depending on the weights w n (or α, β, γ, in this case),

C(w n ) = n i=0 λ n-i e 2 (i), (3)
where e(i) is the forecasting error of observation i. Note that if the lead time is h, then e(i) = w T n z i-h -X i , being z i-h a vector including all input variables. The factor λ is called forgetting factor and when it is near 1, it resembles Least Squares Minimization while allowing the weights to adapt to the statistical changes of the k c time series. A sketch of the RLS algorithm is provided in Algorithm 1, where a big initialization parameter b denotes uncertainty in the first estimate.

Algorithm 1 Recursive Least Squares Algorithm

h : forecast horizon p : AR order q : MA order X i : measurement at time i E i : exogenous variables at time i b = 1000 : initialization parameter λ = 0.999 : forgetting factor

Initialization ŵ0 ← 0 X0 ← 0 P 0 ← bI When new data arrives X n ← (X n , X n-1 , ..., X n-p+1 ) T e n ← Xn -X n z n ← X T n , e T n , E T n T K n ← Pn-1z n-h λ+z T n-h Pn-1z n-h P n ← 1 λ P n-1 -K n z T n-h P n-1 ŵn ← ŵn-1 + K n X n -z T n-h ŵn-1 Xn+h ← z T n ŵn
As historical data is not necessary, this adaptive approach is useful for operational context. It also avoids fixing the weights (and the optimization challenge this brings). Furthermore, statistical properties varies between seasons and even days (i.e. cloudy and clear-sky days), causing short-term adaptability to be a desirable property. A deduction of RLS algorithm for an arbitrary lead time is provided in the Appendix A, which is not easily found in the bibliography.

IV. PERFORMANCE METRICS AND EVALUATION

Results are presented in terms of the standard Mean Bias Deviation (MBD) and Root Mean Squared Deviation (RMSD) metrics, as well as the Forecasting Skill (FS) metric [START_REF] Coimbra | Chapter 8 -overview of solar-forecasting methods and a metric for accuracy evaluation[END_REF]. The MBD and RMSD definitions are,

MBD h = 1 N i ŷh (i) -y ref (i + h) , (4)
RMSD h = 1 N i [ŷ h (i) -y ref (i + h)] 2 , ( 5
)
where ŷh is the GHI forecast, y ref is GHI measurement's and h is the forecast horizon. Their relative values, rMBD and rRMSD respectively, are expressed as percentage of the average irradiance value (see last column of Table I).

The MBD definition is such that a positive value means a forecasting overestimation and a negative value means a forecasting subestimation. The forecasting skill represents the gain of the forecasting RMSD with respect to the persistence procedure, and it is defined as,

FS = 1 - RMSD m RMSD p , (6)
where the subscripts 'm' and 'p' refers to the model and persistence respectively. The persistence is calculated by setting kc (t + h) = k c (t), for every h ≥ 1, where kc (t + h) is the forecast of k c (t + h). As explained before, the corresponding GHI forecast is inferred by using the clear sky model estimates.

The results averaged over all the sites, which act as the mean performance over the region, are presented here. Note that this is not an assessment in which it is intended to predict the joint resource for all sites, known as regional forecast, but rather is the average of the sites performances. The complete results (the performance metrics) for each station, model and lead time can be found in the following link: http://les.edu.uy/pub/ARMAX-RLS.csv. As the filters are adaptive, no train set is needed. Nevertheless, the first 400 forecasts were ignored in order to avoid including convergence and initialization discussions that are not relevant to operational performance.

V. ARMA ORDER SELECTION

The orders for the ARMA model have to be selected (p and q) for each lead time. In this Section, the order selection is done using only the past ground measurements. ARIMA models (models that work with the time series derivative) were also considered, yielding worse results in all the cases, so the differentiation term (known as d) is ignored in the following analysis. Information criteria do not always lead to an acceptable model selection, as was found out in some tests.

There is a qualitative approach based on Auto-Correlation Function (ACF) and Partial ACF plots, but it proved not suitable for working with these solar data sets (although the approach is widely used in economics [START_REF] Nau | Introduction to linear regression analysis[END_REF]). However, a grid-search model selection method provided one important insight: if the irradiance time series is long enough, then different orders for p ≥ 1 yield a difference always smaller than 1 % in terms of rRMSD for our data set, as observed in Fig. 1 (orders for p ≤ 10 are plotted). In Fig. 1 the performance of different (p, q) ARMA models is shown as a function of the time horizon and it is observed that the curves with worse performance correspond to those models with p = 0. For p > 0 small performance changes are observed for the different model's orders. Also, the rRMSD span is reduced with the time horizon. The rRMSD span is of 0.9% for 10 minutes ahead, 0.8% for 90 minutes ahead and less than 0.4% for 240 minutes ahead. Most of these differences are concentrated between the p = 1 models and the others, being the rRMSD surfaces more flat for higher p orders. This implies that, in practice, the selection of the model order (both p and q) is not a crucial issue for p ≥ 2. In Fig. 2 a simple AR model, with p = 5 AR lags (as it was found as a good trade off filter between accuracy and simplicity for all lead times), applied to each lead time and location, is compared to the error found with the optimum (p, q) for each lead time and location. The difference is negligible.

VI. RESULTS WITH EXOGENOUS VARIABLES

Aside from reasonable order selection, the most important performance changes come from an appropriate choice of exogenous variables. Intuitively, including the local shortterm variability of the time series would add information about the current state of the atmosphere; i.e., if the last conditions were clear/cloudy sky or partially cloudy (which will exhibit higher variability). Cloudiness measured via satellite is strongly (negatively) correlated with irradiance measurements. Assuming an ergodic hypothesis, the average in space of the satellite cloudiness includes information of the near future irradiation over the site. Given the above, we implement and compare the following models:

1) Model PGM: is an ARMAX-RLS model that only uses Past Ground Measurements (PGM).

2) Model PGM-S: is the PGM model including current time satellite albedo (ρ p ) as exogenous variable.

3) Model PGM-V: is the PGM model including local short-term variability (σ) as exogenous variable.

4) Model PGM-SV: is the PGM model including both exogenous variables (satellite and variability).

The results were obtained for the PGM, PGM-S, PGM-V and PGM-SV models using a (5, 0) order for the ground measurements part. Exogenous variables are included at time t, without neither past values or past errors. Table II presents the average performance over the region for each model and the persistence as a function of the time horizon. It shall be noted that all the models present similar and small bias (rMBD) and outperform the persistence procedure in terms of RMSD, yielding positive forecasting skills. The trend of the RMSD against the forecast horizon is illustrated in Fig. 3.

It is clear that the introduction of the spatially averaged satellite albedo as an exogenous variable improves the forecast performance. For instance, the PGM model achieves a minimum forecasting skill of FS = 7.8 % (h = 10 minutes) and a maximum forecasting skill of FS = 14.8 % (h = 240 minutes) while the PGM-S model obtains a FS = 16.3 % and FS = 18.2 % for the same time horizons, respectively. Also, the performance of the PGM-S model peaks at FS = 24.1 % (for h = 30 minutes). The characterization of the impact of the spatial averaging cell size on the forecasting performance will be part of future work.

On the other hand, including short-term variability has almost no effect in the model performance, as also shown in Fig. 3. Thus, variability is not a suitable exogenous variable to introduce in the ARMAX-RLS models: linearly combining this variable does not improve the forecast performance. This does not mean that this variable it is not suitable for any model, as it may be useful for others machine learning approaches for point or probabilistic forecast.

VII. CONCLUSIONS

The results obtained for GHI forecasting in the Pampa Húmeda region (south east part of South America) using an ARMAX Recursive Least Squares technique with past ground measurements and two different exogenous variables (satellite albedo and a local short-term variability index) as input were presented in this work. We found that all the models tested, whether they include or not exogenous variables, outperform the classical persistence procedure for the region. A discussion for the ARMA order selection (p and q) was provided, and it was found that for p ≥ 2 all the orders yield to similar performance. Also, it was observed that a fixed filter for all sites and time horizons can be used in the region without sacrificing performance. The importance of including satellite data as predictor is revealed, as it greatly improves the forecasting performance. However, variability information was found to be irrelevant to these kind of models, since negligible changes were found in the model's performance due to its introduction. The recommended model for the region from the ones inspected here is the PGM-S model.

The results presented here comprise a solely homogeneous subtropical climatic area and should not be incautiously extrapolated to other sites. In future research these ideas will continue to be explored over other parts of the world, covering other climates with different solar resource variability. Also, as the site's satellite information prove to be an improvement, adding the satellite cloudiness in the surroundings of the site is a promising future work for improving solar forecast. Combining information and predictors correctly is needed to improve solar forecasting accuracy and recursive ARMA models can serve as a good framework for this objective.

⇒ n i=0 λ n-i (w T n z i-h -X i )z i-h = 0 ⇒ n i=0 λ n-i (w T n z i-h )z i-h = n i=0 λ n-i X i z i-h
that can be shown equivalent to:

      n i=0 λ n-i z i-h z T i-h Θ(n)       w n = n i=0 λ n-i X i z i-h r(n)
, identifying Θ(n) the weighted sample covariance matrix for z and being r(n) the estimate of the cross-covariance between X n and z n-h . So we have: Θ(n)w n = r(n). Now note that we could write Θ as a recursion:

Θ(n) = z n-h z T n-h + λ n-1 i=0 λ n-1-i z i-h z T i-h Θ(n-1)
.

Remembering the Woodbury Matrix Identity:

(A + U CV ) -1 = A -1 -A -1 U C -1 + V A -1 U -1 V A -1 ,
and identifying right hand side of Eq. ( 7) as A + U CV

A -1 = 1 λ Θ -1 (n-1), U = z n-h , C = I, V = z T n-h ,
we can invert Θ(n), obtaining,

Θ -1 (n) = λΘ(n -1) + z n-h z T n-h -1 = (A + U CV ) -1 = A -1 -A -1 U C -1 + V A -1 U -1 V A -1 = 1 λ       P n-1 - P n-1 z n-h λ + z T n-h P n-1 z n-h Kn z T n-h P n-1      
, where P n-1 = Θ -1 (n -1) and P n = Θ -1 (n). So we have:

P n = 1 λ P n-1 -K n z T n-h P n-1 .
Playing with K n :

K n = λ -1 P n-1 z n-h 1 + λ -1 z T n-h P n-1 z n-h ⇒ K n 1 + λ -1 z T n-h P n-1 z n-h = λ -1 P n-1 z n-h ⇒ K n = λ -1 P n-1 -K n z T n-h P n-1 z n-h = P n z n-h
From the definition of r(n):

r(n) = n i=0 λ n-i X i z i-h = X n z n-h +λ n-1 i=0 λ n-1-i X i z i-h
= λr(n -1) + X n z n-h

Recalling Θ(n)w n = r(n) and replacing r(n):

ŵn = P n r(n)
⇒ ŵn = λP n r(n -1) + P n X n z n-h = P n-1 r(n -1) -K n z T n-h P n-1 r(n -1) + X n P n z n-h

= ŵn-1 -K n z T n-h ŵn-1 + X n K n = ŵn-1 + K n X n -z T n-h ŵn-1
That is the coefficient update summarized in Algorithm 1.

Figure 1 .

 1 Figure 1. Performance analysis for different (p, q) ARMA models. Curves for p ≤ 10 are plotted.

Figure 2 .

 2 Figure 2. Performance comparison between a (5, 0) model and an optimum model which uses the best performing p and q values for each time horizon and location.

Figure 3 .

 3 Figure 3. rRMSD metric: average performance over all Uruguayan sites.

Table I LOCATION

 I OF THE GROUND MEASUREMENT STATIONS.

	station	station	lat.	lon.	alt.	GHI
	name	code	(deg)	(deg)	(m) (W/m 2 )
	LES facility	LE	-31.28 -57.92	56	460
	Artigas	AR	-30.40 -56.51 136	451
	Las Brujas	LB	-34.67 -56.34	38	440
	Tacuarembó	TA	-31.71 -55.83 142	444
	Rocha	RO	-34.49 -54.31	20	428
	La Estanzuela	ZU	-34.34 -57.69	70	442
		all sites measurements average	444

Table II PERFORMANCE

 II METRICS FOR THE ARMAX-RLS ALGORITHM AVERAGED OVER THE FIVE STATIONS.

		Persistence		PGM			PGM-S			PGM-V			PGM-SV
	time	MBD RMSD	MBD RMSD	FS	MBD RMSD	FS	MBD RMSD	FS	MBD RMSD	FS
	horizon	(%)	(%)	(%)	(%)	(%)	(%)	(%)	(%)	(%)	(%)	(%)	(%)	(%)	(%)
	10 minutes	0.1	20.8	0.2	19.2	7.8	0.3	17.3	16.6	0.2	19.2	7.7	0.3	17.3	16.5
	20 minutes	0.1	24.7	0.4	22.3	9.9	0.5	19.1	22.9	0.4	22.3	9.8	0.5	19.1	22.8
	30 minutes	0.2	26.7	0.5	24.0	10.2	0.6	20.3	24.1	0.5	24.0	10.1	0.6	20.3	24.0
	40 minutes	0.2	28.2	0.6	25.3	10.2	0.7	21.6	23.6	0.6	25.3	10.1	0.7	21.6	23.5
	50 minutes	0.3	29.4	0.7	26.4	10.2	0.7	22.8	22.7	0.7	26.5	10.1	0.8	22.8	22.6
	60 minutes	0.4	30.5	0.8	27.4	10.1	0.8	23.9	21.7	0.8	27.5	10.0	0.8	23.9	21.6
	120 minutes	0.8	36.2	1.2	32.3	10.6	0.9	29.4	18.7	1.2	32.4	10.5	0.9	29.4	18.6
	180 minutes	1.2	41.2	1.5	36.1	12.4	0.9	33.7	18.3	1.5	36.1	12.3	0.9	33.7	18.2
	240 minutes	1.4	45.7	1.6	38.9	14.8	1.0	37.1	18.8	1.6	39.0	14.7	1.1	37.1	18.8

ACKNOWLEDGMENTS

The authors gratefully acknowledge the financial support given by Uruguay's National Research and Innovation Agency (ANII) under the FSE-ANII-2016-131799 grant.

APPENDIX A DEDUCTION OF RECURSIVE LEAST SQUARES ALGORITHM FOR ARBITRARY LEAD TIME

Remember the cost function to minimize Eq. ( 3),

being e(i) the forecasting error of observation i. Note that if the lead time is h, then e(i) = w T n z i-h -X i , being z i-h a vector including all input variables.

Differentiating and equaling to zero:

2λ n-i e(i) de(i) dw n = i=0 n2λ n-i e(i)z i-h = 0


Mohamed Mouaici

Laurence Vignollet

Mael Etienne

Christine Galez

Detection of learners at risk of failure in online professional training

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Detection of learners at risk of failure in online professional training

Mohamed Mouaici 1,2 , Laurence Vignollet 1 , Mael Etienne 2 , Christine Galez 1 1: Savoie Mont Blanc University,France {mohamed.mouaici,laurence.vignollet,France {mohamed.mouaici,mael.etienne}@logipro.com Track: Practitioner track (LA implementation) and Industrial track (contexts practices).

Purpose

Our research is conducted in the field of Learning Analytics (LA) in a context of formal online professional training and in collaboration with a French organization providing e-learning trainings through a learning management system (LMS). The objective is twofold: detect learners likely to fail their training based on a set of calculated indicators from the learning traces collected by the LMS; provide explanations about the predictions in order to facilitate intervention of the pedagogical team to help learners. From this dataset, we calculate first the correlation factor between each indicator and the failure in order to check whether an indicator alone can be a strong predictor of failure. Afterwards, we test three predictive models for the detection of failure. We favoured supervised machine learning methods that allow us to explore the importance of each indicator in the final predictive model in order to provide an explanation for predictions. To train and test these models we combine repeated cross validation while always respecting the stratification of each fold at each iteration. This process is applied to the initial dataset with and without over-sampling.

Over-sampling allows to balance non-failure and failure observations. Finally we evaluate the stability of the best predictive model based on a new dataset collected for the two types of training between February and March 2019. We evaluate also the relevance of the provided explanations with the pedagogical team.

Results

The correlation factors obtained between each indicator and the failure are relatively low for both types of training which means that no single indicator can be a strong predictor of failure.

The results obtained by the three predictive models demonstrate the good performance of random forest in detecting the failure (Table 1). In addition, the over-sampling had a positive impact on the performance of the three predictive models particularly in terms of recall (probability of failure correctly classified by models). We thus retained the model based on random forest.

Table 1: results obtained by the three predictive models using repeated cross validation and stratified sampling for the initial dataset. O.S: Over Sampling. We report the average performance across all iterations: Accu.: accuracy. Prec.: precision. Reca. recall.

To explain the predictions, we extract the importance of each indicator from the final random forest predictive model. The most important indicators are displayed in the main page of a dashboard in order for the pedagogical team to know why a learner has been classified as atrisk of failure. A textual explanation is also provided based on the actual value of each indicator compared to the split value used by the predictive model in the classification.

By using a new dataset in the evaluation stage we show that the random forest predictive model remains stable since it detects the majority of failure cases. The explanations provided also receive a positive evaluation by the pedagogical team.

Implications

We believe that this solution could help the pedagogical team to identify more easily the learners at risk of failure, to analyse the possible reasons of this risk and thus to intervene effectively to help them. In addition, it is expected that the support provided with this solution will increase the satisfaction of learners and their success in training. The impact of this solution on the pedagogical team and the learners needs to be evaluated yet.

  to learners' behaviour regarding their e-learning activities for two types of training (T1 and T2) are used, based on learning traces collected by the LMS: total time spent, total number of visualisations of the content, number of active days on the activity, average time by content and by active day, number of attempts, average time by attempt, total time spent and score obtained in the evaluation, number of active days on the platform. In both types of training, the failure in an e-learning activity involves the failure in the training. However, the failure in the T1 training activities is based on the score obtained on the evaluation, while the failure in the T2 training activities is based on the total time spent by learners and the target skills of the activities. This difference is due to administrative constraints imposed by the professional training funder.We use an initial dataset collected for both types of training over two years(2017 and 2018) with a failure rate of 32.1% on T1 training and 34.52% on T2 training.