Zuokun Ouyang
email: zuokun.ouyang@univ-orleans.fr

Meryem Jabloun
email: meryem.jabloun@univ-orleans.fr

Philippe Ravier
email: philippe.ravier@univ-orleans.fr

Rankformer: Leveraging Rank Correlation for Transformer-based Time Series Forecasting

Keywords: Time Series, Forecasting, Transformer, Rank Correlation, Nonlinear Dependencies

Long-term forecasting problem for time series has been actively studied during the last several years, and preceding Transformer-based models have exploited various self-attention mechanisms to discover the long-range dependencies. However, the hidden dependencies required by the forecasting task are not always appropriately extracted, especially the nonlinear serial dependencies in some datasets. In this paper, we propose a novel Transformer-based model, namely Rankformer, leveraging the rank correlation function and decomposition architecture for long-term time series forecasting tasks. Rankformer outperforms four state-of-the-art Transformer-based models and two RNNbased models for different forecasting horizons on different datasets on which extensive experiments were conducted.

I. INTRODUCTION

Time series forecasting is a process for predicting the future values of a given time series based on its historical behavior by developing a model describing its underlying characteristics and extrapolating into the future. It has been widely used in many applications, such as weather forecasting [START_REF] Shi | Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting[END_REF], GDP prediction [START_REF] Longo | A neural network ensemble approach for GDP forecasting[END_REF], traffic forecasting [START_REF] Zuo | Graph convolutional networks for traffic forecasting with missing values[END_REF], and energy consumption prediction [START_REF] Yaslan | Empirical mode decomposition based denoising method with support vector regression for time series prediction: A case study for electricity load forecasting[END_REF]. It has been dominated for a few decades by statistical methods such as AutoRegression Integrated Moving Average (ARIMA), ExponenTial Smoothing (ETS), and Theta method [START_REF] Box | Time Series Analysis: Forecasting and Control[END_REF]- [START_REF] Ouyang | STL Decomposition of Time Series Can Benefit Forecasting Done by Statistical Methods but Not by Machine Learning Ones[END_REF]. In the past few years, deep learning has been applied to time series forecasting and achieved great success [START_REF] Guennec | Data Augmentation for Time Series Classification using Convolutional Neural Networks[END_REF]- [START_REF] Hewamalage | Recurrent Neural Networks for Time Series Forecasting: Current status and future directions[END_REF]. The most popular deep learning models for time series forecasting include Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Transformer models. CNNs and RNNs have been widely exploited in forecasting tasks due to their ability to capture sequential/temporal dependencies in the time series [START_REF] Ouyang | Are Deep Learning Models Practically Good as Promised? A Strategic Comparison of Deep Learning Models for Time Series Forecasting[END_REF]. Some representative works include LSTNet [START_REF] Lai | Modeling Long-and Short-Term Temporal Patterns with Deep Neural Networks[END_REF], DeepAR [START_REF] Salinas | DeepAR: Probabilistic forecasting with autoregressive recurrent networks[END_REF], and TCN [START_REF] Bai | An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling[END_REF].

Since its first birth in 2017, Transformer models have become increasingly popular and applied successfully in various fields, including machine translation, computer vision, and text generation, to list a few [START_REF] Vaswani | Attention is All you Need[END_REF]- [START_REF] Brown | Language Models are Few-Shot Learners[END_REF]. In the time series domain, Informer [START_REF] Zhou | Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting[END_REF] was the first work that introduced Transformer for time series forecasting with a ProbSparse self-attention calculation and a self-attention distilling mechanism to handle the quadratic computational complexity. Autoformer [START_REF] Wu | Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting[END_REF] substitutes the self-attention block with an AutoCorrelation mechanism to discover the period-based dependencies and adopts a decomposition structure to separate the long-term stationary trend and the seasonal patterns. Other Transformer models were also applied to time series forecasting tasks, such as Reformer [START_REF] Kitaev | Reformer: The Efficient Transformer[END_REF], which employed locally sensitive hashing selfattention, and LogTrans [START_REF] Li | Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting[END_REF], which uses a heuristic method to reduce the complexity of the self-attention mechanism.

Nevertheless, the formerly mentioned Transformer models have not been able to exploit the long-range dependencies in time series fully, especially the nonlinear serial dependencies. Informer applied the ProbSparse self-attention mechanism to reduce the computational complexity, but the hidden longrange dependency was not extracted properly. Autoformer used the AutoCorrelation mechanism to discover the period-based dependencies. However, the AutoCorrelation used in the model is based on the Pearson correlation function, which supports only linear correlation even on an NN basis, while in some time series, the long-term dependencies are nonlinear.

In this paper, we propose Rankformer, a novel Transformer model for long-term forecasting tasks, leveraging the rank correlation function and a decomposition architecture for time series forecasting tasks. Rankformer outperforms other Transformer-based models in extensive experiments on four forecasting benchmark datasets for four forecasting horizons.

The rest of the paper is organized as follows. Sec. II introduces the proposed Rankformer model. We then present the experimental setups and configurations in Sec. III. The comparison and discussions based on the results are given in Sec. IV. Finally, Sec. V concludes the paper.

II. METHODS

In this section, we introduce the architecture of Rankformer and its two key components, i.e., the Rank Correlation and Multi-Level Decomposition modules.

A. Rankformer Architecture

As shown in Fig. 1, Rankformer has an encoder-decoder architecture per Autoformer [START_REF] Wu | Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting[END_REF]. The encoder is composed of a stack of N identical layers, each containing one multihead Rank Correlation (RankCorr) block, two Multi-Level Decomposition (MLDecomp) blocks, and one Feed-Forward (FF) block. The decoder is a stack of M identical layers, each of which is composed of two multi-head RankCorr blocks, three MLDecomp blocks, and one FF block. Combining the outputs of the last MLDecomp block and the refined trendcyclical part in the decoder composes the final prediction. The architecture of Rankformer is detailed in the following contents. 1) Encoder: With the RankCorr and MLDecomp blocks, the encoder decomposes the series into seasonal and trendcyclical parts. With the latter being neglected during the modeling process, the encoder mainly models the seasonal component. The output of the l-th encoder layer can be summarized as X l en = Encoder(X l-1 en ) and the process in one encoder layer is expressed as follows:

S l,1 en , = MLDecomp RankCorr(X l-1 en ) + X l-1 en , S l,2 en , = MLDecomp FF(S l,1 en ) + S l,1 en , (1)
where S l,i en denotes the seasonal component after the i-th MLDecomp block and X l en = S l,2 en , l ∈ {1, 2, ..., N }. 2) Decoder: The decoder in Rankformer has two streams, i.e., the trend-cyclical stream and the seasonal stream. While the seasonal stream continuously refines the seasonal part of the time series, the trend-cyclical stream focuses on modeling the trend-cyclical component. With a similar notation as per encoder, we can summarize the process in one decoder layer as X l de , T l de = Decoder(X l-1 de , T l-1 de ) and formalize it as follows:

The outputs of the l-th decoder layer are two fold: the refined seasonal patterns X l de = S l,3 de , and the multiple level trendcyclical patterns T l de , where l ∈ {1, 2, ..., M }.

3) Model Inputs and Outputs:

We denote the input length as I, the output length as O, and the model dimension as d. There are three inputs for Rankformer:

• The encoder input are the last I time steps in the time series: X en ∈ R I×d . • The seasonal stream input concatenates the latter half of the encoder's decomposed input and a length-O placeholder with zeros: X de,S = concat(X en,S , X 0 ) ∈ R ( I 2 +O)×d . • The trend-cyclical stream input also consists of the latter half of the decomposed X en and a placeholder filled by the average of X en : X de,T = concat(X en,T , X avg ) ∈ R ( I 2 +O)×d . The relationship between the inputs can be formalized as follows:

X en,S , X en,T = MLDecomp X en I 2 : I , X de,S = concat(X en,S , X 0 ), X de,T = concat(X en,T , X avg ). (3)
The final output of the model is a combination of the seasonal and the trend-cyclical streams in the decoder: W S X M de + T M de , where W S is a trainable weight to project X M de into the target dimension.

B. RankCorrelation Block

The Pearson correlation coefficient, also known as Pearson's ρ, is widely used to measure the linear correlation between two variables. Given two random variables X and Y , Pearson's ρ defined as follows:

ρ p (X, Y ) = cov(X, Y ) σ(X)σ(Y ) . (4)
The AutoCorrelation Function (ACF) adopts the Pearson correlation function to measure the correlation between two distant time points in a stationary time series y t :

ACF(k) = ρ p (y t-k , y t ) = cov(y t-k , y t ) σ(y t-k )σ(y t ) , k = 0, 1, 2, ..., ∀t.
(5) However, in some time series, the long-term dependencies are not linear. In this case, the nonlinearity can cause a low Pearson's ρ and thus result in an erroneous dependencies measurement. To address this issue, we propose to use the Rank Correlation Function (RCF), more generally known as Spearman's ρ [START_REF] Spearman | The Proof and Measurement of Association between Two Things[END_REF], to measure the nonlinear correlation. Spearman's ρ is defined as follows:

ρ s (X, Y ) = cov(R(X), R(Y )) σ(R(X))σ(R(Y )) , (6)
where R(X) and R(Y ) are the ranks of X and Y . ρ denotes the usual Pearson correlation coefficient but is applied to the rank variables, which is leveraged to compute the Ranked ACF (RACF). ρ s is defined in [-1, 1], where -1 indicates a perfect negative monotonic relationship, 0 indicates no monotonic relationship, and 1 indicates a perfect positive monotonic relationship. ρ s is invariant to monotonic transformations of the variables and is robust to outliers. Therefore, it is more suitable for stationary time series with nonlinear serial dependencies.

Our RACF is defined as:

RACF(k) = ρ s (y t-k , y t ), k = 0, 1, 2, ..., ∀t. (7)
In our implementation, the RACF is computed by exploiting the FFT, which accelerates the Fourier transform to O(N log N ), and the Wiener-Khinchin theorem, which states that the ACF of a stationary time series can be computed by the Fourier transform of its power spectrum. The ranking procedure is supported by the torchsort 1 library, which offers an efficient O(N log N ) sorting operator [START_REF] Blondel | Fast Differentiable Sorting and Ranking[END_REF]. Thus, the RACF is computed by ranking the time series and then computing the ACF of the ranked time series. The total computational complexity of calculating the RACF is thus O(N log N ).

C. Multi-Level Decomposition Block

We adopted a multi-level decomposition block to decompose the input time series into the seasonal and trend-cyclical components. The block consists of Multiple Moving Average (MMA) filters with varying kernel sizes to yield different trendcyclical components, rather than just one fix-length MA filter in Autoformer. The MLDecomp block is formalized as:

X trend-cyclical = K k=1 W decomp,k • MMA(X input , k), ( 8
)
where K is a set of kernel sizes, W decomp,k is a trainable weight tensor, and MMA are multiple moving average filters. The output of the MLDecomp block is a weighted sum of the trend-cyclical components.

III. EXPERIMENTS

This section presents our experimental settings and results.

A. Datasets

Rankformer was tested with other state-of-the-art methods on four well-known datasets:

• Electricity Transformer Temperature (ETT) [START_REF] Zhou | Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting[END_REF] [START_REF] Wu | Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting[END_REF] and Informer [START_REF] Zhou | Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting[END_REF]. The datasets' statistics are listed in the first four rows of Tab. I.

We also evaluated the significance of the nonlinearity in the serial dependencies by performing Engle's Lagrange Multiplier Test [START_REF] Engle | Autoregressive Conditional Heteroscedasticity with Estimates of the Variance of United Kingdom Inflation[END_REF] on the four datasets. It assesses the significance of autoregressive conditional heteroskedasticity (ARCH) effects in a time series. A significant result reveals nonlinear serial dependencies in the series. The test results are listed in the last two rows of Tab. I.

B. Experimental Settings

We kept the same number of encoder-decoder layers as Autoformer: two encoder layers and one decoder layer. Rankformer was trained using the Mean Square Error loss and the Adam optimizer [START_REF] Kingma | Adam: A Method for Stochastic Optimization[END_REF] with an initial learning rate of 10 -4 . The batch size was set to 32. The model was trained for ten epochs with a learning rate scheduler that reduces the learning rate by a factor of 0.5 when the validation loss plateaus. The model was implemented in PyTorch [START_REF] Paszke | PyTorch: An Imperative Style, High-Performance Deep Learning Library[END_REF] and trained on a single NVIDIA Tesla V100 GPU.

IV. RESULTS AND DISCUSSION

We compared Rankformer with the following state-of-theart methods: Autoformer [START_REF] Wu | Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting[END_REF], Informer [START_REF] Zhou | Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting[END_REF], LogTrans [START_REF] Li | Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting[END_REF], Reformer [START_REF] Kitaev | Reformer: The Efficient Transformer[END_REF], LSTNet [START_REF] Lai | Modeling Long-and Short-Term Temporal Patterns with Deep Neural Networks[END_REF], and LSTM [START_REF] Hochreiter | Long short-term memory[END_REF]. We used the Mean Square Error (MSE) and the Mean Absolute Error (MAE) as the evaluation metrics, and we fixed the input length to 36 for ILI and 96 for others as per Autoformer. The results are presented in Tab. II. The best results are highlighted in bold, and the second-best results are highlighted with underscores.

Overall, Rankformer outperforms the other methods on the ETT, Exchange-Rage, and Weather datasets and is slightly weaker than Autoformer on the ILI dataset. Particularly, under the Input-96-Output-96 setting, Rankformer yields 13.3%

Models

Rankformer Autoformer [START_REF] Wu | Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting[END_REF] Informer [START_REF] Zhou | Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting[END_REF] LogTrans [START_REF] Li | Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting[END_REF] Reformer [START_REF] Kitaev | Reformer: The Efficient Transformer[END_REF] LSTNet [START_REF] Lai | Modeling Long-and Short-Term Temporal Patterns with Deep Neural Networks[END_REF] LSTM [START_REF] Hochreiter | Long short-term memory[END_REF] Metric The results on the Exchange-Rate dataset are particularly impressive. Despite the fact that Exchange-Rate is a very challenging dataset without any notable periodicity, Rankformer still gives the best improvement over Autoformer. We attribute this to the nonlinear serial dependencies in the dataset being captured more properly by Rankformer than by Autoformer.

On the contrary, due to the high linear correlation in the ILI dataset, Rankformer is not able to outperform Autoformer. In fact, the p-value of Engle's Lagrange Multiplier test of the ILI dataset is 0.8126 (≫ 0.05), which means that there are statistically significant linear serial dependencies inside the ILI series that can be handled more appropriately by Autoformer.

On the other hand, thanks to the FFT and Wiener-Khinchin theorem, Rankformer achieves an O(N log N ) complexity. It is not only a huge advantage in the computing speed compared to the original Transformer's O(N 2 ) complexity but also brings the convenience of nonlinear serial dependencies measurement to Autoformer without augmenting the time complexity. This makes Rankformer much more efficient than the Transformer, especially when the input sequence is long, and also more appropriate for forecasting time series with nonlinear serial dependencies.

Rankformer and Autoformer are very similar in terms of their measurement of correlation. The only difference is that Rankformer uses a rank-based ACF, i.e., the RACF, while Autoformer uses a value-based ACF, which means, with an optimized sorting and ranking operator, the RACF can be easily integrated into Autoformer and enables it for nonlinear-time-dependencies-measurement. Nonetheless, a small difference in the dependencies measurement can leverage a decent improvement in the forecasting performance, especially for the datasets with nonlinear serial dependencies.

V. CONCLUSION

In this paper, we propose a novel method, Rankformer, for forecasting time series, especially those with nonlinear serial dependencies. Rankformer is based on the Transformer architecture and uses a rank-based ACF to measure the nonlinear serial dependencies. We show that Rankformer outperforms Autoformer, a state-of-the-art method with linear serial dependencies measurement, on three real-world datasets. We also show that Rankformer is more efficient than the original Transformer in terms of both computing speed and memory usage. Furthermore, we show that the RACF used by Rankformer can be easily integrated into Autoformer to improve its performance on datasets with nonlinear serial dependencies. In fact, in most nonlinear serial dependencies cases, the series shows an ARCH effect, which can be captured by the RACF. Therefore, we believe that Rankformer can be a good alternative to Autoformer for forecasting time series with nonlinear serial dependencies. In the future, we plan to investigate: 1) the robustness of Rankformer in the presence of perturbation; 2) the possibility of integrating other nonlinear serial dependencies measurement methods and the general ARCH (GARCH) characteristics to further improve the performance of Rankformer.

Fig. 1 .

 1 Fig. 1. The architecture of Rankformer.

TABLE II FORECASTING

 II RESULTS FOR DIFFERENT MODELS ON DIFFERENT FORECAST HORIZONS

S l,1 de , T l,1 de = MLDecomp RankCorr(X l-1 de ) + X l-1 de , S l,2 de , T l,2 de = MLDecomp RankCorr(S l,1 de , X N en ) + S l,1 de , S l,3 de , T l,3 de = MLDecomp FF(S l,2 de ) + S l,2 de ,T l de = T l-1 de + W l,1 T l,1 de + W l,2 T l,2 de + W l,3 T l,3 de ,(2)where S l,i en and T l,i de are the seasonal and trend-cyclical components respectively, and W l,1 , W l,2 , W l,3 are trainable weights.

https://gis.cdc.gov/grasp/fluview/fluportaldashboard.html

ACKNOWLEDGMENT

This research was partly supported by Association Nationale de la Recherche et de la Technologie (CIFRE 2019/0551).


Evismar Andrade

Leo R Quinlan

Richard Harte

Bronwyn Reid-Mcdermott

Frank Kirrane

Enda Fallon

Martina Kelly

Tony Hall

Michael Scully

John Laffey

Patrick Pladys

Ethel Ryan

Dara Byrne

Gearóid Ólaighin
email: gearoid.olaighin@universityofgalway.ie

Dr P R Pennathur

The development and preliminary evaluation of a clinician e-learning training platform for a neonatal sepsis risk monitor for use in ICU settings

Keywords:

niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

The importance of patient monitoring system training

Patient monitoring systems (PMS) require their intended users to receive training on those systems for them to be used safely and effectively. International regulatory medical device standards, such as IEC-62366-1, consider training as a central part of the usability engineering process for a medical device (IEC, 2015). In intensive care units (ICU) inappropriate, insufficient, or lack of training on PMS can lead not only to frustration when using the devices but, more seriously, can lead to patient harm due to use errors. This is evidenced by medical device guidance documents, such as IEC/TR 62366-2, which advise manufacturers of medical devices that insufficient training is one of the factors that contribute to user error (IEC, 2016).

In the literature, we can identify several examples that demonstrate the effect of PMS training or its absence on patient care. For instance, it has been reported that inadequate training with infusion pumps has been associated with use errors using these critical care devices, including errors configuring the infusion pump (e.g. the delay and alarm functions of these devices) [START_REF] Keers | Causes of medication administration errors in hospitals: a systematic review of quantitative and qualitative evidence[END_REF] (Doyle and Agrawal, 2016). Studies focusing on the problem of alarm fatigue, one of the major problems in critical care worldwide, suggest that clinicians' productivity and response to alarms may be considerably improved if adequate training on the relevant device is provided [START_REF] Graham | Monitor alarm fatigue: standardizing use of physiological monitors and decreasing nuisance alarms[END_REF] [START_REF] Sowan | Improving the safety, effectiveness, and efficiency of clinical alarm systems: simulation-based usability testing of physiologic monitors[END_REF] (Borowski et al., 2011) [START_REF] Brantley | Clinical trial of an educational program to decrease monitor alarms in a medical intensive care unit[END_REF]. In studies on the usability of physiological monitors (PM), where participants' skills with the PM were measured, training deficiencies with these devices were associated with lack of confidence and knowledge relating to PM functionalities [START_REF] Fidler | Human factors approach to evaluate the user interface of physiologic monitoring[END_REF] [START_REF] Wung | Critical care nurses' cognitive ergonomics related to medical device alarms[END_REF] [START_REF] Sowan | Nurse competence on physiologic monitors use: toward eliminating alarm fatigue in intensive care units[END_REF]. These studies point to the importance of providing comprehensive PMS training to critical care clinicians, to improve, not only the user experience with the device, but patient care overall.

Challenges in providing PMS training

Appropriate training on PMS typically requires ICU staff to either attend training sessions with representatives of the medical system supplier or in-house training sessions provided by the hospital (Doyle and Agrawal, 2016). For example, initial training may be delivered by the system supplier on-site, whenever a new PMS is introduced or when an older model of device is replaced with a new model. Depending on the hospital's training policy, clinicians may be provided with further continuous or refresher training in the hospital by a staff educator which may include comprehension of knowledge/skills verification. In either case, clinicians in these scenarios must be available to physically attend those sessions in person to receive training on the PMS functionalities and, most importantly, to practice using the system.

Nonetheless, delivering this level of training to critical care nurses and doctors can be very challenging for several reasons, including the difficulty of releasing busy staff to attend the training sessions, the availability of trainers, the costs associated with the training delivery, and synchronising the availability of several staff members and the training team for face-to-face training (Borowski, Görges, Fried, Such, Wrede, Imhoff) [START_REF] Brand | Attendance at NHS mandatory training sessions[END_REF] [START_REF] Brand | Just a piece of equipment? The importance of medical device education[END_REF].

The evidence is that in some cases clinicians are not receiving adequate training. A survey conducted by [START_REF] Ewertsson | Use of technical skills and medical devices among new registered nurses: a questionnaire study[END_REF] with 113 newly registered nurses (up to 1 year of work), revealed that although 76% of the nurses stated a need for continued practical training with the PMS used in their units, less than half of them (48%) had received training [START_REF] Ewertsson | Use of technical skills and medical devices among new registered nurses: a questionnaire study[END_REF]. The findings of an unpublished survey conducted by our research team in 2018/2019 on the usability of PMs (N = 98), revealed that almost half of the participants reported not having received any training to use the PM used in their unit. Similar findings were reported in other studies conducted with ICU clinicians who interact with the PM on a daily basis [START_REF] Fidler | Human factors approach to evaluate the user interface of physiologic monitoring[END_REF]) [START_REF] Wung | Critical care nurses' cognitive ergonomics related to medical device alarms[END_REF] [START_REF] Sowan | Nurse competence on physiologic monitors use: toward eliminating alarm fatigue in intensive care units[END_REF].

It is also important to consider that, in some cases, the gap in time between when the initial training is received and when the actual use of the system occurs in clinical practice can be weeks or even months. Clinicians therefore may forget or not properly recall portions of what they learned about using the system during training (IEC, 2016).

E-learning for PMS training

A possible way of addressing the challenges in providing PMS training is to deliver training using online platforms (e-learning), which can provide asynchronous/on-demand access. E-learning has the potential to mitigate barriers associated with time restrictions of trainers and trainees and tends to be cost-effective, flexible, and convenient for learners in healthcare settings [START_REF] Brand | Just a piece of equipment? The importance of medical device education[END_REF], [START_REF] Walsh | Cost and value in e-learning: the perspective of the learner[END_REF]. The literature has reported that e-learning for clinical staff can be at least as effective as face-to-face training for a variety of applications [START_REF] Lahti | Impact of e-learning on nurses' and student nurses knowledge, skills, and satisfaction: a systematic review and meta-analysis[END_REF], [START_REF] Sinclair | The effectiveness of Internetbased e-learning on clinician behaviour and patient outcomes: a systematic review[END_REF]. In the case of PMS, however, hands-on training sessions are an essential element of training to enhance the learning process. In these sessions, the learners can test the knowledge they have acquired during the training, by interacting with the PMS, thus simulating real practice. For this reason, the hands-on element is also important for competency assessment [START_REF] Scalese | Simulation technology for skills training and competency assessment in medical education[END_REF], [START_REF] Sep | The assessment of clinical skills/competence/performance[END_REF].

Although e-learning cannot completely replace hands-on training, given that replicating the physical experiences (e.g., physically touching buttons and connecting cables to the device) is not possible with elearning, it could be used to supplement this training. Recent advancements in the area of web development make it possible for software developers to create interactive interfaces that can be used to replicate the PMS interface and make the interface easily available to the learner for familiarisation and interaction [START_REF] Challapalli | Web development and performance comparison of web development technologies in Node.js and Python[END_REF]. E-learners can use this technology to learn about the system's functionalities and interact with the simulated interface.

A potential use case for this approach can be seen in Fig. 1, where the learners can be initially trained individually using e-learning before taking the traditional face-to-face training. With this approach, face-toface training may become more productive, allowing the learners to query more advanced aspects of the system's use during face-to-face training.

Another potential use case for e-learning is as supplementary training for critical care clinicians who feel they need refresher training, but do not have the opportunity to attend an additional face-to-face training session. As a result, blended learning, combining hands-on training sessions with the use of an e-learning training platform is proposed as a potential solution to the aforementioned PMS training needs [START_REF] Vallée | Blended learning compared to traditional learning in medical education: systematic review and meta-analysis[END_REF][START_REF] Lawn | An integrative review of e-learning in the delivery of self-management support training for health professionals[END_REF].

A systematic literature review of the facilitators and barriers to elearning in health sciences practice [START_REF] Regmi | A systematic review of the factorsenablers and barriersaffecting e-learning in health sciences education[END_REF]) highlighted the potential of innovative online educational technology but also reported that "further studies are needed to ensure rigorous study design to deliver quality and effective e-learning". This research highlighted the importance of bespoke e-learning design, to meet the specific, situated needs of medical professionals working in diverse, complex settings.

Digi-NewB, a non-invasive Health Decision Support System (HDSS), was developed by an international research consortium, under the auspices of the Horizon 2020 program, to support NICU nurses and doctors to make an early sepsis diagnosis in the preterm infant [START_REF] Värri | The Digi-NewB project for preterm infant sepsis risk and maturity analysis[END_REF]) (Digi-NewB, 2022). Since the IEC/TR 62366-2 guidance document advises medical device manufacturers to consider training early in the development process of a medical device, e-learning was considered as a training tool during the development of Digi-NewB. Nonetheless, evidence remains unclear regarding the best e-learning style and format for teaching healthcare professionals, since this is context-dependent [START_REF] Lawn | An integrative review of e-learning in the delivery of self-management support training for health professionals[END_REF]. However, research on a variety of e-learning training platforms designed for healthcare professionals, provides exemplars of good practice that could potentially be implemented to enhance the proposed Digi-NewB training approach. Recommendations for e-learning in healthcare identified from the literature were: R1.1. Flexible training platform, allowing learners to access training whenever they want, to be able to record and save their training progress and navigate back and forth to different stages of the training [START_REF] Lahti | Impact of e-learning on nurses' and student nurses knowledge, skills, and satisfaction: a systematic review and meta-analysis[END_REF] [START_REF] Lawn | An integrative review of e-learning in the delivery of self-management support training for health professionals[END_REF] [START_REF] Regmi | A systematic review of the factorsenablers and barriersaffecting e-learning in health sciences education[END_REF]. R1.2. Short modules of 20 min or less with a clear format (e.g. introduction, problem presentation, possible solutions, practical tasks etc.) [START_REF] Lawn | An integrative review of e-learning in the delivery of self-management support training for health professionals[END_REF]. R1.3. Accommodate learners with low technological affinity and be designed in an easy-to-follow format to avoid frustration from such learners [START_REF] Regmi | A systematic review of the factorsenablers and barriersaffecting e-learning in health sciences education[END_REF] [START_REF] Grundgeiger | Training students to use syringe pumps: an experimental comparison of e-learning and classroom training[END_REF]. R1.4. Use authentic scenarios that reflect 'real world' examples. These can be presented using short videos including clinician/patient interactions, as this was identified as a compelling method of fostering learning [START_REF] Lahti | Impact of e-learning on nurses' and student nurses knowledge, skills, and satisfaction: a systematic review and meta-analysis[END_REF] [START_REF] Lawn | An integrative review of e-learning in the delivery of self-management support training for health professionals[END_REF] [START_REF] Jang | Use of online clinical videos for clinical skills training for medical students: benefits and challenges[END_REF] [START_REF] Regmi | A systematic review of the factorsenablers and barriersaffecting e-learning in health sciences education[END_REF] [START_REF] Grundgeiger | Training students to use syringe pumps: an experimental comparison of e-learning and classroom training[END_REF]. R1.5. Incorporate interactivity is a key element in the learning process. Allowing learners to interact with the system interface to gain and retain knowledge and apply it in the work setting [START_REF] Lahti | Impact of e-learning on nurses' and student nurses knowledge, skills, and satisfaction: a systematic review and meta-analysis[END_REF] [START_REF] Lawn | An integrative review of e-learning in the delivery of self-management support training for health professionals[END_REF] [START_REF] Grundgeiger | Training students to use syringe pumps: an experimental comparison of e-learning and classroom training[END_REF]. R1.6. Incorporate clinical problem-solving activities where learners can use the learned content to solve simulated problems associated with the use of the PMS [START_REF] Lawn | An integrative review of e-learning in the delivery of self-management support training for health professionals[END_REF]. R1.7. Build in learner self-assessment activities (formative assessment), which provide immediate feedback to learners [START_REF] Lawn | An integrative review of e-learning in the delivery of self-management support training for health professionals[END_REF].

The goal of this study was to develop an e-learning platform that supports training clinicians on the use of the Digi-NewB user interface, at a time and place of their convenience and which augments the hands-on training that the clinicians would typically receive following the use of the e-learning training platform. The e-learning training platform would include online formative assessments to track the progress of the clinician with the online training. The final summative assessment of whether the clinician is sufficiently trained to use the patient monitoring system in clinical practice with patients would be made in the clinical setting.

This paper will describe in detail the development of this e-learning platform based on the recommendations for e-learning in healthcare R1.1-R1.7. It will also present the outcomes of a preliminary evaluation of the e-learning platform, based on the performance of NICU nurses during a usability evaluation of the system interface. This evaluation was carried out using a high-fidelity simulation; interviews with the simulation participants post-simulation; an interview with a NICU key opinion leader (KOL) (who observed all simulation sessions) and a postevaluation survey sent to the NICU nurses who participated in the testing.

Design of an E-learning platform for Dig-NewB

Design philosophy

It was proposed to adopt the recommendations for e-learning in healthcare, presented in the Introduction section, for the design of the Digi-NewB e-learning platform. It was proposed to implement these recommendations by incorporating specific features into the platform. The specific features proposed for Digi-NewB are listed as follows and reference is made to the relevant recommendation: F2.1. Learners should be able to access the e-learning platform remotely via commonly used web browsers from any computer or tablet to facilitate access (R1.1). F2.2. Learners should be able to pause the training at the end of any session and continue whenever they wanted to from the point where they left off (R1.1). F2.3. The training should be divided into short modules, with maximum duration of 20 min each, and each module should be divided into short sessions (approximately 3 min each) covering specific topics to avoid learners getting distracted (R1.2). F2.4. The application should be designed to be easy to use, with clear instructions provided at each step. Simple controls to be used to navigate the training (e.g., back and next buttons). The controls to be carefully designed to avoid misuse, for example, the next button would only be enabled once the session is finished, to incorporate best practice for interface design and be compliant to Nielsen's heuristics (R1.3). F2.5. To increase the opportunity of gaining the attention of the learners, instead of pictures or slides, the instructors and scenarios were presented in videos, using animated characters moving, talking to the learner and each other, and interacting with the NICU devices (R1.4). F2.6. The learners should be able to interact with a replica of the Digi-NewB system interface on multiple occasions throughout the training. In these simulated sessions, learners should complete the tasks they would be expected to do with the system interface under different circumstances (R1.5). F2.7. If the learner is required to carry out a task using the Digi-NewB interface, they must be informed immediately if the task was not performed correctly and should be asked to do the task again until they have completed it correctly (R1.6, R1.7). F2.8. Formative assessment must be included in the form of quizzes at the end of each module. If the learner answers a question incorrectly, they will be required to watch a recap video on the topic and have the opportunity to re-take the quiz until all questions are answered correctly (R1.7).

The training programme was developed by a research team composed of engineers, medical educators, an educational technologist, human factors specialists, and medical consultants. Based on feature F2.3, the training programme is composed of 3 short modules, lasting approximately 20 min each (1 h in total) with the overall structure of the training presented in Fig. 2.

Structure of the E-learning platform

Fig. 2 presents an overview of the developed training programme, which contains 3 modules. Each module is composed of three key components: Instruction videos, interactive sessions and an assessment session composed of quizzes. The components were designed to be short and dynamic to retain the attention of the learners (to meet requirement R1.2). For example, an instruction video on a specific functionality of the Digi-NewB system would take no more than 3 min, followed by an interactive section on the same functionality.

Module 1 introduces the Digi-NewB system to the learners, showing how the training platform works and teaching the learners how to interact with the simulated interface. In modules 2 and 3, learners interact with the system interface in different scenarios. In these scenarios, how the interface supports the decision-making process of the clinicians (characters in the scenario) is explained to the learners, using the sepsis risk score and clinical observations. For example, a change in the sepsis risk score from low risk to medium risk would lead clinicians to take a different decision, depending on the presence of clear clinical signs of deterioration (as in module 2) or on no clear evidence of clinical signs of deterioration (as in module 3).

Instruction videos

Based on feature F2.5, instruction videos were created using animated characters that would introduce the system concept, explain the different functionalities of the system and present scenarios describing the intended use cases of the system (Fig. 3).

Interactive sessions

After the instruction videos, learners are presented with scenarios in which a task must be performed using the Digi-NewB user interface and they have an opportunity to interact with this interface during an elearning interactive session (based on feature F2.6). For example, the instruction video would explain what the Digi-NewB clinical observations are, then a scenario would be presented in which the clinical observations of a baby in a deteriorating condition have to be updated. Then, the learner is asked to use the Digi-NewB interface to update the clinical observations using the data in the baby's chart, for example belly distension, temperature and type of catheter used (urinary or venous catheter).

The interface presented in the e-learning training looks and functions exactly like the actual Digi-NewB prototype (Fig. 4).

Learners are not able to skip the interactive session before completing all the required tasks correctly. In the e-learning interactive sessions (feature F2.4), when a task is completed incorrectly, learners are immediately informed of the mistake, so they can perform the task again (feature F2.7, Fig. 6).

Quizzes

A quiz is presented at the end of each training module (formative assessment based on feature F2.8). Learners are asked several multiplechoice questions related to the topics covered in each session (Fig. 6 (a)). At the end of the quiz, the learners are provided with the results of the quiz (Fig. 6 (b)). If a learner answered one or more questions incorrectly, they are required to watch a recap video on these specific topics and are then provided with the opportunity to answer the question(s) again. Learners are only able to complete the training session after answering all questions correctly.

The training platform was developed using ReactJS, an open-source JavaScript library used for building web user interfaces. This library was chosen since it allowed the building of rich user interfaces that can run fast in commonly used web browsers. A cloud-based platform (Google's Firebase Firestore Database) was used to store the progress of each learner during training. For this research project, the database was designed so that no personal information from the learners was stored. However, a product version of the training platform would need to store personnel data to generate a certificate of completion.

Preliminary evaluation of the E-learning training

Performance criteria of the E-learning training

The preliminary evaluation of the developed e-learning training platform was conducted in parallel with the summative usability evaluation of the Digi-NewB user interface. In this evaluation, a simulation was designed to take place in a high-fidelity simulation centre. Participants were asked to monitor a simulation mannikin of a new-born baby while performing a series of scripted tasks using the Digi-NewB user interface, mimicking the tasks they would normally do in the NICU. While this evaluation method is effective in assessing the learner's competency in using the Digi-NewB interface, it cannot be the sole mechanism to assess the quality of the e-learning training platform given the confounding effect of the Digi-NewB interface design on the training effect. For instance, a well-designed and simple-to-use interface may require minimal training while a poorly designed interface may require extensive training to be used effectively.

Therefore, in addition to the simulation analysis, it was important to seek feedback from the participants on their perceptions of the effectiveness, efficiency, and user experience with the e-learning training platform itself. A multi-modal qualitative research approach was used, including the analysis of three qualitative data sources:

1. Audio and video recordings of the summative usability testing sessions, conducted in a high-fidelity simulation setting. 2. Interviews with the simulation participants and with a NICU KOL who observed all simulation sessions. 3. A survey was distributed to the simulation participants after the simulation session.

Outcome measures associated with effectiveness, efficiency, and user experience with the e-learning training platform were assessed using a combination of the three qualitative data sources.

Outcome measures associated with effectiveness:

• In addition to these outcome measures, participants were asked about their perceptions of using a similar e-learning training platform to enhance their skills with other PMSs.

Participants

Participants were recruited over a period of 2 weeks using convenience sampling. The requirements for the participants were that they should be nurses that (i) had experience in the NICU (2 years or more) and (ii) were available and willing to participate in the training, usability testing and interview sessions. In addition, the presence of a NICU key opinion leader (KOL) observed all simulations and provide feedback regarding the usability testing and the system itself. The requirements for the KOL were that they should have a senior leadership position in the NICU, with over 10 years of experience. Ethical approval for the study was obtained from the Galway Clinical Research Ethics Committee (Ireland, C.A. 2480, PI: Prof. Gearoid ÓLaighin).

After giving consent to participate in the study, participants were asked to complete the three Digi-NewB online training modules. Participants were provided with the link to the training platform and asked to log in using their initials. The training progress of participants was monitored to ensure all participants completed the three training modules prior to the simulation. Upon completion of the three online training modules, participants were emailed an acknowledgement that they had completed the training and to schedule the simulation date and time.

Simulation settings and analysis

The usability testing of the Digi-NewB system user interface was designed to be conducted in a high-fidelity simulation centre using a 30week premature infant patient simulator (Premie HAL S2209, Gaumard) to simulate the new-born baby. The new-born's vital signs were generated by the infant patient simulator and presented on a display at the bedside. The Digi-NewB interface was presented using a Microsoft Surface Pro 4 beside the vital signs display. In addition, one member of the research team acted as a NICU consultant and one as a novice NICU nurse, these are referred to as the confederate consultant and the confederate nurse in this article.

Throughout the experiment, the participant nurse and the confederate nurse stayed in the simulation room looking after the simulated baby. The confederate consultant was in the control room, and available to enter the simulation room if needed. The technicians were also in the control room, to ensure that all simulation devices worked properly. The remaining faculty present were able to watch the simulation on a screen in an adjoining room, from where they took notes of the simulations. The screen in the adjoining room displayed real-time video and audio from 2 cameras in the simulation room (Fig. 7 (a) and (b)), the output of the Digi-NewB display Fig. 7 (c) and the output of the simulated physiological monitor in the unit (Fig. 7 (d)).

Participants were asked to perform a series of scripted tasks during the simulation, mimicking the tasks they would normally do in the NICU (e.g., monitor the baby's vital signs, take clinical observations, report signs of abnormalities to the consultant etc.). In addition to those tasks, participants were asked to perform specific tasks related to the Digi-NewB interface, including: Fig. 4. In this part of the training, learners have the chance to interact with a simulation of the Digi-NewB system using their own devices to interact with it. They are asked to 'care for a simulated new-born baby', feed the simulated system with clinical observations and acknowledge the sepsis risk score. The learner needs to update the clinical observations for the simulated patient by clicking on the clinical observation buttons at the bottom of the page as they become out of date (see Fig. 5). Similarly, if the note sepsis risk score needs to be acknowledged (button changing colour from blue to orange), the learner needs to click on it to acknowledge the sepsis risk score. (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.) Fig. 5. After entering the incorrect value for the weight parameter, the learner would be presented with this error message and hear the voice of the instructor informing them they would need to try again.

1. Monitor the sepsis risk score (SRS) on the Digi-NewB screen and acknowledge the SRS on the system when required. 2. Update the clinical observations (CO) when they expired or when they noticed a mismatch between the COs on the system and the baby (e.g., when the baby's temperature changed). 3. Optionally use the trends tab to investigate the presence of cardiorespiratory events in the previous hours and correlate these with the SRS trend. While doing this, they may also change the trends timescale, to visualize the trends for a longer or shorter time-period.

The simulation sessions were video, and audio recorded with the objective of further analysing participants' interaction with and behaviour relating to the Digi-NewB system interface. During the video analysis, the research team focused on the behaviour of the participant towards the Digi-NewB system interface, including the following analysis:

1. Did the participant complete all of the scenario-related tasks, and the required tasks with the Digi-NewB interface? 2. Did the participant make any mistakes during the simulation? 3. How long did it take to complete the tasks associated with Digi-NewB interactions? 4. Did the participant need any assistance to use the Digi-NewB system interface during the simulation? 5. Could the participant answers questions regarding the Digi-NewB system interface from the confederate nurse and doctor? 6. What did the participant say regarding Digi-NewB during the simulation?

This information provided evidence on the effectiveness and efficiency of the Digi-NewB UI, but also on the effectiveness of the Digi-NewB training.

Interviews and survey

The interviews were conducted in a semi-structured format with four main open-ended questions asked to all participants:

1. What is your overall impression of the Digi-NewB online training? 2. What things did you like about the Digi-NewB online training? 3. What things did you not like about the Digi-NewB online training? 4. How useful do you think and online training like that would be for other medical devices?

These questions were used to guide participants to talk about the training and their experience with the simulation and follow-up questions were asked over the course of the interview depending on the participant's answers. The same questions were posed to the KOL. The interview data were analysed using thematic analysis (TA), a data analysis method "for identifying, analyzing and reporting patterns (themes) within data" (Braun and Clarke, 2006:79) [START_REF] Braun | Using thematic analysis in psychology[END_REF].

After completing the thematic analysis and reviewing the qualitative data, the research team surveyed participants on some specific aspects of the training platform to generate quantitative data using Likert Scale questions. The questions focused on the quality of the training, training time and suitability of using the training platform for medical device training. Participants were sent a Survey Monkey link to anonymously answer the survey questions and eight out of nine participants completed the survey.

Results

Participants

After obtaining ethical approval from the Galway Clinical Research Ethics Committee (Ireland), 9 nurses who worked in the UHG NICU were recruited using convenience sampling. All participants were working in UHG NICU at the time and had at least 2 years of experience in the specialism of neonatology. In addition, a senior NICU consultant was invited as key opinion leader (KOL) to participate. The KOL was asked to complete the e-learning training programme and to observe all simulations with the research team. After each simulation, the KOL was asked for her feedback on the Digi-NewB system interface and training platform.

Simulation analysis

All nine participants completed the entire scenario proposed for the simulation, including all the required tasks with the Digi-NewB interface and simulation baby. The median simulation time was 19 min and 20 s, with the shortest simulation taking 17 min and 44 s and the longest simulation taking 22 min and 47 s. Participants were in general able to interact with the Digi-NewB interface without any assistance during the Fig. 6. At the end of each module, learners are asked to answer questions related to the content covered and are provided with the results of this quiz. In (a), the learner would need to select option 3 "It indicates that the clinical observation needs to be updated" to answer correctly. If the learner answers a question incorrectly, they will have the chance to re-watch a video explaining the related topic and answer the question again. As an example, in (b), the learner answered one of the questions related to the PMS context of use incorrectly so they would need to re-watch the video which explains the topic and answer the question again. simulation, except for 1 participant, who was unclear on how to access the trends tab. During the sepsis assessment, participants performed the usual tasks they would normally do in their unit such as checking the vital signs in the physiological monitor, conducting clinical observations, reviewing lab results etc.

During the simulation, the confederate nurse and doctor asked participants questions regarding the meaning of Digi-NewB visual elements. For example, when participants called the confederate doctor to inform them that the sepsis risk increased, the confederate doctor would say "I missed the Digi-NewB training this morning, what does this yellow risk score mean?" or the confederate nurse would ask "Sorry, I started here last week. What do you use the trends tab for?". Eight (out of the nine participants) could correctly answer the questions asked demonstrating confidence in their answers, which indicates that participants acquired a good knowledge of the system functionalities during the training. Only one participant hesitated to answer a question about the meaning of the sepsis risk score.

Two participants were observed encountering difficulty with the touchscreen when entering the PIN number. For example, some participants touched the palm of their hands on the screen while typing the number or pressed the gap between numbers, which delayed the task and required extra screen touches. When this happened, participants usually tried to use the pen to interact with the screen, which facilitated the interaction. This issue did not occur during the training, as participants used devices they were familiar with, which highlights the need for face-to-face training to achieve a comprehensive evaluation of the user experience with the device.

Interview and survey analysis

This section presents the outcomes of the thematic analysis of the interviews conducted post-simulation combined with the outcomes of the survey, sent to participants after the interview sections. The outcomes were divided into 8 themes presented below:

Overall satisfaction with the training

All survey participants reported being either satisfied (62.5%) or very satisfied (37.5%) with the online training platform overall. They were also asked about their satisfaction with each element of the training, and the results are presented in Fig. 8. Participants were satisfied with the different elements of the training, except one participant who reported being dissatisfied with the interaction with the screen prototype and the quizzes at the end of the training sections.

Knowledge and know how

The video analysis provided a good indication that the Digi-NewB training was successful in preparing participants to interact with the system interface. Nonetheless, it was decided to further investigate participants' perceptions of their learning experience in the survey. Fig. 9 shows that all participants agreed or strongly agreed that the training was successful in explaining to them how the system interface worked and preparing them to interact with it.

Only one participant was not sure whether they were able to remember all the Digi-NewB material relevant to the simulation during the simulation day. When asked approximately what percentage of the relevant learning material from the online training they were able to remember during the simulation, 3 participants reported between 60% and 80%, 2 between 80% and 90% and 3 participants reported between 90% and 100%.

Participants were asked in the survey what they thought about the effectiveness of the training overall and its individual components. As can be seen in Fig. 10, all survey participants reported being satisfied or very satisfied with the effectiveness of the training platform.

Training location and device used for training

Given the flexibility of an online training platform, participants could do the training at any location. Four survey participants reported they did the full training at work, 2 did the full training at home and 2 participants did part of the training partially at work and partially at home. While most participants reported using a laptop/desktop to do the training, some participants reported having tried to use their mobile phones to do the training. Although the training platform could be accessed by any device, participants using smartphones had eventually to switch to a laptop/desktop or a tablet to be able to interact properly with the Digi-NewB interface, given the screen size limitation of the smartphone. Seven out of 8 survey participants reported they would prefer doing the training using a laptop and 1 preferred a PC. No participant chose the mobile alternative (a Tablet).

Training time

The training time was a recurring theme in the interview sessions. The nurses mentioned that, before starting the training, they were concerned it would take too much of their time. Nonetheless, participants considered it to be shorter than they expected, which for them was a positive aspect: Fig. 7. In (a) and (b) the participant nurse is interacting with the Digi-NewB interface, entering the required clinical observation, while the confederate nurse is checking the baby. Image (c) shows the Digi-NewB interface as presented to the participant and image (d) shows the physiological monitor displaying the current vital signs. Although the Digi-NewB system takes data from the monitor, the physiological monitor used in the simulation was an interface designed for training purposes (not a real physiological monitor). Technicians controlled both the physiological monitor and Digi-NewB interface to behave as required for the simulation. P4: "I thought it was very good. Short. The training sections were not overly long and explanatory. They're very to the point."

In the survey, participants were asked what they thought about the duration of each element of the Digi-NewB training. As can be seen in Fig. 11, all participants judged the overall training time and duration of each training element appropriate, except one participant who considered that the interaction time with the prototype was too short.

Training scenarios

The different Digi-NewB use cases were presented to the participants using cartoon characters in different scenarios. Participants reported that the scenarios were a key learning element for them to understand how Digi-NewB worked, which helped them during the simulation: P7: "Once you go through the scenarios, by the end of it, you know everything."

Interface interaction

Another key training element was the interface interaction. Participants were able to interact with the Digi-NewB interface in different scenarios during the training. It was expected that this allowed them to "play" with the interface during the learning process without any real consequences for the patients. Participants reported that this interactivity was extremely important for their learning: Participants also reported that the company representants normally leave manuals to be used, but commented that this format can be tedious: KOL: "We usually get reps to talk us through the new equipment and manual and then the manual can't be found, or it is online, but it's just very tedious. Sometimes reading through 25 pages."

Updating traditional medical device training

During the interviews, participants were also asked if they thought that a similar training structure used in the Digi-NewB training could be used for other medical devices. The general response was that it would be very useful. The main reason for this can be attributed to the interactive element of the training: P4: "It would because you would practice, you would actually be doing it. For me now, I'm not reading a book (manual). I have to practice doing it, to get it into my head. A lot of people are like that, especially nurses and doctors, they are probably more practical in some respects."

Another reason why online training would be useful relates to the availability of the learners (nurses and doctors) and instructors. Healthcare professionals are usually very busy and do not always have the availability to attend a face-to-face training session with the medical device supplier: P5: "When we get a new device in the unit, the rep needs to come back so many times. I might be working for 5 days in a row and miss it. They will never be able to capture everybody. So, it could be useful to just get everyone to do something online. Just to make sure everyone gets it done." Some participants commented that an online training platform could be also useful after the system is already in place to update their skills and refresh their knowledge of the system (especially some features that are not used often): P5: "I just think to keep being able to dip into that (the training). Just keep refreshing it because you might not be using something all the time.

In the survey, all participants agreed that this format of training could be used for refresher training or skill maintenance for medical devices as well as initial training on these devices. As it can be seen in Fig. 12, participants were not unanimous regarding using the online training as a replacement for the medical device method in place, which would suggest that this type of training works best as supplemental training.

Discussion and conclusions

This paper described the development of an e-learning training platform for a sepsis risk user interface for use by neonatal critical care clinicians and presented the results of a preliminary evaluation of the platform using a multi-modal qualitative research case study approach. The e-learning training platform was developed to train NICU nurses to use the Digi-NewB user interface prior to participating in usability testing of the interface, which was conducted in a high-fidelity clinical simulation centre. The evaluation of the e-learning training platform against the performance criteria, defined in section 3.1, was conducted using video/audio analysis of the simulation sessions, interviews with simulation participants and a KOL and carrying out a survey of simulation participants.

The video/audio analysis of the simulations revealed that participants were able to use and interpret the Digi-NewB interface successfully. In general, participants were able to use the Digi-NewB interface appropriately, performing all the tasks required without the need for assistance. Similarly, participants in general were able to answer questions about the meaning and functionalities of different aspects of the Digi-NewB interface during the simulation. These findings coupled with feedback from the participants on the e-learning training platform (acquired during interviews and surveys) indicate that the platform was effective in teaching the intended users about the different aspects/ functionalities of the Digi-NewB user interface and in enabling them to interact with and interpret the interface as intended.

The outcomes from the thematic analysis of the interview transcripts and the feedback extracted from the post-simulation survey revealed that participants were satisfied with the duration of the training. Participants reported doing the training from work (4), home (2) or both (2), which indicates that the possibility of doing PMS training from various locations may be an important capability for some learners. Similarly, the possibility of using different devices to do the training may be an important feature for some learners as participants had different device preferences. However, while some participants would have liked to do all training on a smartphone, it is not recommended that smartphones be allowed for this type of training, as the small screen size is not conducive to effective e-learning on a medical device user interface. Our experience with this issue prompted us to propose a new requirement for the training platform prohibiting the use of smartphones for the PMS elearning training.

The literature shows that distributed learning (spacing learning periods) provides a mnemonic advantage over massed learning (learning in one long session without breaks), helping to consolidate learning, especially for long-term retention [START_REF] Son | Distributed learning: data, metacognition, and educational implications[END_REF] [START_REF] Litman | Distributed learning enhances relational memory consolidation[END_REF]. For this reason, the e-learning training for Digi-NewB was originally designed so that each module had to be completed on different days and it would be our intention that this approach would be adopted in the final system. However, during this case study evaluation of the platform, it was not feasible to use distributed learning as the clinician volunteers requested that they be facilitated to complete all of the training on the same day, without enforced breaks. It was observed that all participants chose to complete the three training sessions on the same day. It is possible that this change in the training consumption had a negative impact on the effectiveness of the training.

Another important aspect of training to consider is the gap in time between receiving the training and the actual use of the system. Standard IEC 62366-1:2015 notes that "an appropriate wait time might be needed between the training and the rest of the summative evaluation to allow for representative learning decay." IEC, 2015). Nevertheless, given the constraints associated with the availability of the case study participants, it was not feasible to apply a gap in time between training and the actual use of the system. Most participants chose to do the training one day before or on the same day as the simulation, which is likely to have helped them remember the information necessary to complete the tasks during the simulation session. One participant, who did the training 2 days prior simulation, reported that they forgot some aspects of the training when they were asked by the confederate doctor during the simulation.

The challenges with traditional PMS training, as discussed in the introduction, are mostly related to efficiency and resource management (e.g., the need to be available to attend face-to-face/hands-on training and the costs associated with this training approach). This case study proposes that e-learning platforms have the potential to address such issues by providing asynchronous training, accessible from any location at any time. However, PMS e-learning training also has limitations as clinicians need to be able to physically interact with the device during hand-on training and then to use the device with the patient in situ (e.g., attaching sensors and administering medication) to be to be fully trained on its use. Also, the traditional training allows learners to work in teams, which can be very important depending on the PMS, especially in complex use cases. In the context of e-learning, team training can be very limited and may not be feasible at all.

It is expected that, by adopting a blended leaning approach for the Digi-NewB training, combining e-learning, hands-on sessions and onthe-job training, such challenges will be minimized as the learner will have the opportunity to learn about and practice with the Digi-NewB interface in different circumstances, having the opportunity to access to the e-learning platform when necessary to refresh their training. Regarding the applicability of an e-learning training platform for other medical devices used in the NICU/ICU, the overall perception from participants was that this format of online training would be appropriate and desirable as supplementary training, augmenting current training methodologies.

Finally, the applicability of e-learning as part of PMS training will depend on several factors such as: the type and complexity of the medical device, how much training will be provided, how many sessions are needed and the preference of the end users. For this reason, the adoption of e-learning for PMS training should be analysed case-by-case.

Limitations

The number of participants was a limitation in this study. The number of participants was sufficient for a case study design, to conduct a preliminary investigation of the feasibility of using an online training platform to deliver training on a medical device (Digi-NewB). However, more participants would be required to comprehensively investigate the use of the platform.

The nurse participants were invited to participate in the study by a member of the research team (the KOL, who worked in the same hospital) based on their availability. Since convenience sampling was used in the case study, participants were not chosen at random. Therefore, we recognize the inherent bias that the sample is unlikely to be representative of the population being studied.

Eight (out of the nine participants) reported completing the training on the day before, or the same day of the simulation, which is likely to have distorted the memorability observed during the simulation session. It is possible that, had the gap between training and simulation been bigger, the memorability would have been reduced.

Considering the limitations in this case study, future work should compare blended learning (combing e-learning and hands-on session) with hands-on training only in terms of effectiveness, efficiency, and user experience. This should also include enough participants with a clinical background, randomised between control and experimental groups, allowing for qualitative as well as quantitative (statistical) analysis.

Fig. 1 .

 1 Fig. 1. Potential use case of e-learning as supplement training for critical care PMS.

Fig. 2 .

 2 Fig. 2. Overview of the Digi-NewB training. Each module is composed of 3 key elements: instruction videos (black), interactive sessions (green) and a quiz (purple.). (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.)

Fig. 3 .

 3 Fig.3. The nurse, Joanna (on the left image), instructs the learners about one element of the Digi-NewB interface and explains how it can be used in critical care. On the right side, the learner can see a video which explains the "traffic-light" concept of the sepsis risk score, which indicates the three sepsis risk levels (low, medium and high risk).

Fig. 8 .

 8 Fig. 8. Participants' satisfaction with the main features of the Digi-NewB training.

Fig. 9 .

 9 Fig. 9. Participant's level of agreement with statements related to the learnability and memorability of the Digi-NewB features.

Fig. 10 .

 10 Fig. 10. Survey participants' perceptions on the effectiveness of the online Digi-NewB training.

Fig. 11 .

 11 Fig. 11. Survey participants' perceptions of the duration of the training overall and its subcomponents.

Fig. 12 .

 12 Fig. 12. Participants' perceptions of the use of an online training platform for other medical devices used in critical care.

Acknowledgements

This research was funded as part of the Digi-NewB project by the European Union (EU) H2020 research and innovation program (GA # 689260).

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.