Mobyen Uddin Ahmed
email: mobyen.ahmed@mdh.se

Shahina Begum
email: shahina.begum@mdh.se

Rikard Gestlöf

Hamidur Rahman
email: hamidur.rahman@mdh.se

Johannes Sörman

Machine Learning for Cognitive Load Classification -A Case Study on Contact-Free Approach

Keywords: Cognitive Load (CL), contact-free approach, k-nearest-neighbor (k-NN), Support Vector Machines (SVM), Machine Learning (ML)

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Driving is a complex task that requires high concentration with simultaneous skills and abilities [START_REF] Kim | Time domain EEG analysis for evaluating the effects of driver's mental work load during simulated driving[END_REF]. Research shows that, most of the traffic accidents are caused by the human error. [START_REF] Kim | Time domain EEG analysis for evaluating the effects of driver's mental work load during simulated driving[END_REF]. Most of the cases a road accident occurs when a driver loose concentration on driving due to look outside or thinking anything rather than driving task. Therefore, there is a connection between low concentration level and traffic accidents has been identified in [START_REF] Miyaji | Study on effect of adding pupil diameter as recognition features for driver's cognitive distraction detection[END_REF]. Research shows that the number of road deaths increased by 28% in Sweden, by 14% in the Czech Republic and by 11% in the Netherlands in 2018 compared to 2017 [START_REF]Road Safety Annual Report 2019[END_REF]. Over 20 % of those 90 % are due to fatigue or CL (e.g. use of invehicle devices which can lead to cognitive overload for the driver) [START_REF] Sena | Studying the influence of cognitive load on driver's performances by a Fuzzy analysis of Lane Keeping in a drive simulation[END_REF]. CL is also known as cognitive workload, which is a concept that describes the relationship between the cognitive demands from a task and the environment that influence the user's cognitive resources [START_REF] Palinko | Estimating cognitive load using remote eye tracking in a driving simulator[END_REF][START_REF] Kumar | Measurement of Cognitive Load in HCI Systems Using EEG Power Spectrum: An Experimental Study[END_REF].

To identify CL, psycho-physiological measures are mainly used, as they assess interactions between physiological and psychological states of a human. The most common psychophysiological measurement to detect CL is EEG, because it objectively identifies the cognitive cost of performing tasks [START_REF] Kumar | Measurement of Cognitive Load in HCI Systems Using EEG Power Spectrum: An Experimental Study[END_REF]. Other parameters are also used to detect CL, such as: pupil dilation, HRV, galvanic skin response (GSR), and EEG etc. [START_REF] Reimer | An on-road assessment of the impact of cognitive workload on physiological arousal in young adult drivers[END_REF]. There are disadvantages with these contact-based approaches as most of them used electrodes and they should be connected to the human body, which can be cumbersome to use in situations such as driving. In many occasions the measurement signals are affected due to movements and artifacts [START_REF] Zander | Evaluation of a Dry EEG System for Application of Passive Brain-Computer Interfaces in Autonomous Driving[END_REF]. That is why a contact-free way of measuring CL could be ideal for these situations. On the contrary, the contact-free approach e.g. camera systems have been incessantly growing in the research community [START_REF] Rahman | Vision-Based Remote Heart Rate Variability Monitoring using Camera[END_REF]. Here, the camera system have developed based on color schemas such as RGB and the Lab color space to extract heart rate (HR) and IBI by detecting variation in facial skincolor caused by cardiac pulse [START_REF] Rahman | Vision-Based Remote Heart Rate Variability Monitoring using Camera[END_REF]. McDuff et. al [START_REF] Mcduff | Remote measurement of cognitive stress via heart rate variability[END_REF] has built a remote person independent classifier and predict CL with 85% classification accuracy, here, a contact-free camera-based system was used in order to measure HR, HRV and breathing rate (BR).

In order to detect or classify CL, HRV features are used, since the HR fluctuates with varying levels of CLs which can be detected with HRV metrics [START_REF] Sahadat | Wireless ambulatory ECG signal capture for HRV and cognitive load study using the NeuroMonitor platform[END_REF]. In medical diagnoses and stress detection HRV is a commonly used parameter, there are many different features in HRV that can be calculated and can be used as a parameter to estimate CL. There are three established ways to calculate the HRV, either by time-domain, frequency-domain or non-linear measurements [START_REF] Shaffer | An Overview of Heart Rate Variability Metrics and Norms[END_REF]. CL detection is a very inefficient for humans to manually review large data sets because of the factors that occur naturally in everyday life situations [START_REF] Hussain | Classification of Cognitive Load from Task Performance & Multichannel Physiology during Affective Changes[END_REF]. This is why the ML algorithms are an essential factor in classifying CL.

The data collection for cognitive load research are generally conducted either in indoor environment using simulator driving or in outdoor environment using real road driving scenario [START_REF] Solovey | Classifying driver workload using physiological and driving performance data: Two field studies[END_REF][START_REF] Engström | Effects of visual and cognitive load in real and simulated motorway driving[END_REF][START_REF] Reimer | The Impact of Cognitive Workload on Physiological Arousal in Young Adult Drivers: A Field Study and Simulation Validation[END_REF]. The authors in [START_REF] Liu | Towards Practical Driver Cognitive Load Detection Based on Visual Attention Information[END_REF] have focused on the feasibility of using visual attention features to classify CL in three different states i.e., low, medium and high cognitive load. Four different classification models such as k-NN, logistic regression, SVM, adaptive boosting and random forest have used and the k-NN received the highest accuracy of 81 %. A study performed by Wang et. al. [START_REF] Wang | A data-driven framework for learners' cognitive load detection using ECG-PPG physiological feature fusion and XGBoost classification[END_REF] has received a 97.2 % of classification accuracy in detecting high CL.

This paper presents a case study on contact-free approach for CL classification based on the features extracted through HRV analysis and using on ML. Here, the goal of the study is to compare the contact-free approach i.e., camera-based system with the contact-based approach i.e., a Shimmer GSR+ system to determine how well the contact-free approach performs in detecting CL. To classify the CL, two different ML algorithms, mainly, SVM and k-NN are used. This study includes a collection of data for both the systems in a controlled environment. Thirteen HRV features were extracted to determine three states of CL based on the gathered IBI values from both the systems. The three levels are, S0: reference point for low CL, S1: normal CL where the test subjects performed easy puzzles and drove normally in a video game and S2: high CL where the test subjects completed hard puzzles and drove on the hardest course of a video game while answering math questions [START_REF] Sörman | CONTACT-FREE COGNITIVE LOAD CLASSIFICATION BASED ON PSYCHO-PHYSIOLOGICAL PARAMETERS[END_REF]. To get the best classification accuracy with the ML algorithms, different optimizations such as kernel functions i.e., radial basis function (RBF), and sigmoid function were chosen with different feature matrixes both for binary and combined class classifications.

Materials and Methods

In the contact-free approach, a Logitech HD Webcam C615 camera with the resolution of 1920x1080, and a frame rate of 30fps equipped with auto focus was used to record faces of the test subjects. The recorded videos were saved in MP4 format. The camera was adjusted according to the height of the head of the test subject so that it can capture the facial image properly. The test subjects were sat roughly 70cm in front of the camera during the different phases of the controlled experiment, thus the desired ROI of the test subjects and facial features were extracted [START_REF] Rahman | Vision-Based Remote Heart Rate Variability Monitoring using Camera[END_REF]. During the data collection, the test subjects were asked to move their heads as normal but not too much. In the contact-based approach, a Shimmer GSR+ system was wired up to the subjects' body through a clamp to the earlobe, which provided the corresponding IBI values. The data recorded by the shimmer sensor was connected to a laptop via Bluetooth. The standard settings of Consensus were recorded as Unix_Timestamps, PPG_A13 as IBI and PPGtoHR. The IBI value was recorded by the shimmer system and saved in Excel sheets through a software called Consensus to monitor and manage the data gathered by the shimmersensing 1 . The shimmer had a sampling rate of roughly 40000 after a 5-minute test run. Here the frequency of the sensor reading was 133 Hz. An overview of the workflow used in this study is presented in Figure, that describes the different parts i.e., Data collection, Data Processing, Data Set Preparation, Classification Learner, ML Algorithms, Model Classifier, Classification Results.

Data collection

This study includes 11 test subjects with age between 21 and 33 years. Among them 9 were men with average age of 26 and two were women with average age of 27. Again, four of them had lowest education in math, three of them had highest math education and the rest had average math courses. The mean value of how good they see themselves in mental arithmetic were 5,9 in a scale of 10. The mean value of time spent playing digital games per week was 13 hours, but the time variation is high between the test subjects. The one with the highest hours per week had forty hours, and the lowest had one hour. Two of the eleven test subjects only played on mobile, and the rest played on console/computer.

The data was collected through a controlled study consists of seven phases [START_REF] Sörman | CONTACT-FREE COGNITIVE LOAD CLASSIFICATION BASED ON PSYCHO-PHYSIOLOGICAL PARAMETERS[END_REF], between phases there were a minor pause to set the next phase up and in the fifth phase there were a 15-minute break. The 1 st phase of this experiment was a five-minute rest while sitting normal and watching TV silently. The 2 nd Phase was to complete the two different easy metal puzzles while sitting normal. Here, the test subject played an easy puzzle first, and when they have dissembled the first one, they started over the next one. When both the puzzles were dissembled the test subject would start to reassemble both the puzzles in the same order. This process was repeated for five minutes. The 3 rd Phase was about to play both the metal puzzles with a higher difficulty. The 4 th phase was about the test subject rested for 15 minutes and were offered coffee and snacks. The 5 th phase was about to sit a normal watching TV silently same as in phase one, for five minutes. In the 6 th phase the test subjects play a game 'Mario Kart Double Dash' on the 150cc course BOWER´S CASTLE competing against seven bots. The time of this phase was five minutes; however, the time was varied depending on how well the test subjects performed. Finally, the 7 th phase is about to play the 'Mario Kart Double Dash' 150cc once again, but on RAINBOW ROAD, which is the final and the hardest course of the game. Here, instead of competing against seven bots the test subjects had to answer math questions while playing. The math questions had to be answered before the next question was asked, the questions were asked in every 30 seconds and five questions in total for each time. Table 1 illustrates the summary of the phases.

Table 1. Summary of each phase during the data collection

Data Pre-processing

For each phase of every test subject, the sample size was 40 000 on an average. A python program with the library OpenPyxl was created in order to handle samples, the program both transformed the UNIX time to 'date time' and removed the samples where the IBI was -1.0. Thus, a smaller sample size is reduced to 250-500 depending on the recorded time as well as the precision of the device. The reason of this huge decrease was that the IBI was calculated approximately once per second, since the HR was around 60-80 BPM for all the test subjects. This resulted in roughly 300 recorded IBI's for each phase of every test subject, since the length of each experiment phase was five minutes long.

For the camera data, IBI was extracted from the recorded video and saved in a separate file. From each of the recorded video, first, the face of the test subject was detected and then a region of interest (ROI) was selected from the detected face [START_REF] Rahman | Real Time Heart Rate Monitoring From Facial RGB Color Video Using Webcam[END_REF]. Each ROI was converted into three color frames which are red, green and blue which are transferred into Lab color space or Lab signal. Three signal processing algorithm First Fourier Transform (FFT) [START_REF] Buijs | Implementation of a fast Fourier transform (FFT) for image processing applications[END_REF], Independent Component Analysis (ICA) [START_REF] Comon | Independent Component Analysis, a new Concept[END_REF] and Principle Component Analysis (PCA) [START_REF] Kramer | Nonlinear principal component analysis using autoassociative neural networks[END_REF] were used to extract IBI from the Lab signal by applying a band pass filter of 40-120 Hz [START_REF] Rahman | Non-contact Heart Rate Monitoring Using Lab Color Space[END_REF]. An overview of the camera system that extracts IBI from a video recording is presented in Figure 2.

Feature extraction

Once the collected data were structured in a manner that would make it easier for a script to extract features from the recorded data the implementation of the feature extraction session started with the so-called trial and error phase in order to find a way of extracting all the HRV features properly. The purpose of the script was to reduce the total duration of the recorded time into three minutes and only consider the middle of the recording and remove the data from the beginning and end of the recording to avoid faulty recording. These three minutes were then be further divided into three parts of one minute each. Then for each minute of these three minutes data, the feature function in MATLAB was used to calculate the 13 features where six time domain features and seven frequency domain features as shown in Table 2 and Table 3 respectively. Then the features for each phase was calculated for each test subject. This was resulted in a 3* 14 matrix linked to one class (in the controlled experiment phase), where SittingNormal = S0, Driving 1 or Puzzle 1 = S1, Driving 2 or Puzzle 2= S2.

Classification

For the classification using k-NN and SVM, the whole data sets were divided into four groups both for shimmer and camera data separately as follows: First, 80% of the data for each group of data set was randomly selected for training and 20% for data was selected for testing. Binary classification was conducted for this paper works and two ML algorithms e.g. k-NN and SVM were considered an instance of supervised learning in which an algorithm learns to classify new observations from examples of labeled data [START_REF] Alpaydin | Introduction to Machine Learning[END_REF]. SVM constructs a hyperplane or a set of hyperplane to analyses data to identify pattern and commonly used for classification and regression analysis [START_REF] Cortes | Support-vector networks[END_REF]. K-NN is a nearest-neighbor classification model in which it can be altered both the distance metric and the number of nearest neighbors. Because a k-NN classifier stores training data, the model is used to compute re-substitution predictions. Alternatively, this model is used to classify new observations using the predict method [START_REF] Tamrakar | Integration of lazy learning associative classification with k-NN algorithm[END_REF]. Different kernel functions were investigated both for k-NN and SVM which were weighted, fine and medium kernel for k-NN and cubic, fine and medium kernel for SVM. The best cross-validation k-fold value was 10 for both k-NN and SVM. The binary SVM classifier was implemented according to [eq. 1], the fitcsvm2 is a built-in function in MATLAB that returns a trained SVM model, based on the features of the training set 𝑥 and the list with the corresponding class 𝑦. The parameters ′𝐾𝑒𝑟𝑛𝑒𝑙𝐹𝑢𝑛𝑐𝑡𝑖𝑜𝑛′ indicates that the kernel used in this model should be the variable 𝑘. The parameter ′𝐾𝑒𝑟𝑛𝑒𝑙𝑆𝑐𝑎𝑙𝑒′ indicates that the kernel scale used in this model should be the variable 𝑠. ′𝐵𝑜𝑥𝐶𝑜𝑛𝑠𝑡𝑟𝑎𝑖𝑛𝑡′ is the parameter indicating that the boxconstraint of the model should be the variable 𝑏.

 S0
𝑡𝑟𝑎𝑖𝑛𝑒𝑑𝑆𝑉𝑀=𝑓𝑖𝑡𝑐𝑠(𝑥,𝑦′,𝐾𝑒𝑟𝑛𝑒𝑙𝐹𝑢𝑛𝑐𝑡𝑖𝑜𝑛′,𝑘',𝐾𝑒𝑟𝑛𝑒𝑙𝑆𝑐𝑎𝑙𝑒′,𝑠',𝐵𝑜𝑥𝐶𝑜𝑛𝑠𝑡𝑟𝑎𝑖𝑛𝑡′,𝑏)

For k-NN model, [eq. 2] was implemented which is also a MATLAB built-in function.

The function fitck-NN returns a trained K-NN model and based on the features of the training set 𝑥 and the list with the corresponding class 𝑦. The parameter 𝑁𝑢𝑚𝑁𝑒𝑖𝑔ℎ𝑏𝑜𝑟𝑠′ flags that the variable 𝑘 should be used as the number of neighbors for this K-NN model. ′𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒𝑊𝑒𝑖𝑔ℎ𝑡′ is a parameter in the fitck-NN that indicates that the variable 𝑤 should be used as the distance weight.

𝑡𝑟𝑎𝑖𝑛𝑒𝑑𝐾𝑁𝑁 = 𝑓𝑖𝑡𝑐𝑘(𝑥, 𝑦,′ 𝑁𝑢𝑚𝑁𝑒𝑖𝑔ℎ𝑏𝑜𝑟𝑠′, 𝑘,′ 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒𝑊𝑒𝑖𝑔ℎ𝑡′, 𝑤) (2)
The optimized classifiers were implemented for each training set of the class combinations based on the feature matrixes. The trained models were then given the features of the test sets and based on how they predicted the class of the features, the accuracy of each model could determine by how many correct class predictions it achieved. Each model executed 20 times for each data set and mean value of the result was considered.

Experimental Results

The classification learner revealed that the two best classification algorithms for the collected data were k-NN and SVM. Classification accuracy for the binary data sets (S0 vs S1), (S0 vs S2), (S1 vs S2) and (S0 vs S1andS2) and class classification accuracy for the combined data sets (S0, S1, S2) are calculated using ML algorithms i.e., k-NN and SVM. Also, F1-score is calculated for each of the data set using the ML algorithms. A summary of the classification accuracy and F1-score for the shimmer data is presented in Table 3 and Table 4 presents the classification accuracy for the camera data. The highest value of F1 -score both for the shimmer and camera are highlighted in gray color in the table. For k-NN and SVM, a graphical representation of average F1-score between shimmer and camera is presented in Figure 3 and Figure 4. From the figures it can be seen that F1-score for camera is larger than Shimmer both for k-NN and SVM considering binary and combined class classification. The experimental result shows that highest accuracy has been achieved in a controlled environment using the camera recording during the driving phase. Also, this shows that binary classification using the k-NN algorithm with the class combination of S0 vs S2 achieves an accuracy of 81% which was higher than the expected accuracy from the classification learner by one percent. With the same class combination and setup recorded with the Shimmer sensor system only got 78% accuracy, which is higher than the expected accuracy 70%. For the class combination S0 vs (S1 and S2), the highest F1 score 84% was achieved with medium gaussian SVM using the feature matrix recorded from the Camera system.

The results also demonstrates that the ML algorithms could classify CL in Driving using video games with equal or even higher accuracy than a traditional way (i.e., using metal puzzles). This is significant because controlled experiments that involve actual video games have not been extensively used in the state-of-the-art research. This research also reveals that the accuracy in classifying CL between driving normal and driving with distractions was greater than the accuracy between the Easy puzzle and Hard puzzle. The reason for this might have been that the difficulty between the easy puzzle and hard puzzle was too small. If math questions had been added to the hard puzzle phase of the controlled experiment the results could have been different. The best improvement was achieved from the testing with an increase of the accuracy with 3% based on the focus to feature selection. The MATLAB function sequentialfs (Sequential Feature selection) was used to determine the best features among the extracted features. This shows that the NFFT feature, was not necessary and therefore NFFT was removed from the feature sets. After analyzing more into the features, the MeanNN, RMSSD, VLF and TotalPower were found the most robust features while considering stable accuracies. The data set S2 is provided more accuracy compare to S0 using contact-free approach since it was the most difficult and the hardest difficulty, the driver usually ended up so far behind the bots on the course that they were not disturbed by the bots in the game. The math questions were used to make distraction of the driver which means there should have difference level of cognitive load; however, practically it was not found the difference between cognitive load levels due to less distraction of the drivers by math questions. The choice of an easier track could have made it possible to have the data differentiate even more than it did. Also, the time determined when a math question would be asked, every 30 seconds. This made most of the questions to be asked when the driver had a straight line without any worries and could therefore answer the question more easily. Also, more questions could have been held at hand in order for the questions to stretch across the whole course.

When conducting the controlled experiment, the two recording systems (i.e., Shimmer and Camera) had to be started at the same time. The challenge with this was that a button had to be pressed on each system in order to start the recordings. This made that the two system recordings had a minor time difference of a few ms for each recording for each of the test subject. Also, when cutting the recordings from five minutes into three minutes the implementation cut at the one-minute mark. The same thing occurred when the implementation cut the last minute from the recording at the three-minute mark. Because of this cutting the time gap was small enough that it should not affect the result.

With these findings it seems plausible that the contact-free system can be used reliably in real driving scenario. However, it would also be interesting to investigate if there is something that increase the CL more than other things and try to categorize how much it increased the CL. Also, do more experiments with the game over a longer period of time, to see how different test subjects will adapt to the Mario Kart game. Here, only the frequency and the time-domain features were considered. So, it would also be interesting to involve the non-linear HRV features in the future.

Fig. 1 .

 1 Fig. 1. Overview of the workflow for measuring CL

Fig. 2 .

 2 Fig. 2. Different steps of a camera system to extract IBI from camera data

  vs S1, Normal sitting vs Driving normal  S0 vs S2, Normal sitting vs Driving with distractions  S1 vs S2, Driving normal vs Driving with distractions  S0-S1-S2, Combined classification

Fig. 3 .

 3 Fig. 3. Graphical representation of F 1 -score between Shimmer and Camera data considering binary classification.

Fig. 4 .

 4 Fig. 4. Graphical representation of F 1 -score between shimmer and camera data considering classification of combined class

Table 2 .

 2 List of different time-domain HRV Features

	Feature	Description

Table 3 .

 3 List of different frequency-domain HRV Features

	Feature	Description	Unit
	VLF	Absolute power of very-low-frequency (0.003-0.04 Hz)	𝑚𝑠	2

Table 4 .

 4 Classification results using Shimmer system

	Classifica-	Data		Shimmer
	tion	Set	k-NN		SVM
			Mean accuracy with	F 1	Mean accu-	F 1
			20 runs		racy with 20
					runs
	Binary	S0 vs S1	0.74	0.7	0.74	0.78
		S0 vs S2	0.78	0.74	0.66	0.79
		S1vs S2	0.58	0.41	0.58	0.65
	Comibanation	S0 vs	0.72	0.86	0.68	0.85
		S1and S2

Table 5 .

 5 Classification results using Camera system

	Classification	Data		Camera
		Set	k-NN		SVM
			Mean Accu-	F 1	Mean Accu-	F 1
			racy with 20		racy with 20
			runs		runs
	Binary	S0 vs S1	0.71	0.71	0.70	0.68
		S0 vs S2	0.81	0.84	0.74	0.74
		S1 vs S2	0.67	0.57	0.65	0.66

http://www.shimmersensing.com/products/consensys

https://se.mathworks.com/help/stats/fitcsvm.html

Acknowledgment. The authors would like to acknowledge Embedded Sensor Systems for Health Plus (ESS-H+) for their support of the research projects. The authors would also like to acknowledge all the participants for supporting in this study.


Georgios Barlas
email: barlasgeorgios@gmail.com

Efstathios Stamatatos
email: stamatatos@aegean.gr

Cross-domain Authorship Attribution Using Pre-trained Language Models

Keywords: Authorship Attribution, Neural Network Language Models, Pre-trained Language Models

Authorship attribution attempts to identify the authors behind texts and has important applications mainly in cyber-security, digital humanities and social media analytics. An especially challenging but very realistic scenario is cross-domain attribution where texts of known authorship (training set) differ from texts of disputed authorship (test set) in topic or genre. In this paper, we modify a successful authorship verification approach based on a multi-headed neural network language model and combine it with pre-trained language models. Based on experiments on a controlled corpus covering several text genres where topic and genre is specifically controlled, we demonstrate that the proposed approach achieves very promising results. We also demonstrate the crucial effect of the normalization corpus in cross-domain attribution.

Introduction

Authorship Attribution (AA) is a very active area of research dealing with the identification of persons who wrote specific texts [START_REF] Neal | Surveying stylometry techniques and applications[END_REF][START_REF] Stamatatos | A survey of modern authorship attribution methods[END_REF]. Typically, there is a list of suspects and a number of texts of known authorship by each suspect and the task is to assign texts of disputed authorship to one of the suspects. The basic forms of AA are closed-set attribution (where the list of suspects necessarily includes the true author), open-set attribution (where the true author could be excluded from the list of suspects), and author verification (where there is only one candidate author). The main applications of this technology are in digital forensics, cyber-security, digital humanities, and social media analytics [START_REF] Kestemont | Authenticating the writings of Julius Caesar[END_REF][START_REF] Rocha | Authorship attribution for social media forensics[END_REF].

In real life scenarios the known and the unknown texts may not share the same properties. The topic of the texts may differ but also the genre (e.g., essay, email, chat). Cross-domain AA examines those cases where the texts of known authorship (training set) differ with respect to the texts of unknown authorship (test set) in topic (cross-topic AA) or in genre (cross-genre AA) [START_REF] Sapkota | Cross-topic authorship attribution: Will out-of-topic data help?[END_REF][START_REF] Stamatatos | Masking topic-related information to enhance authorship attribution[END_REF]. The main challenge here is to avoid the use of information related to topic or genre of documents and focus only on stylistic properties of texts related to the personal style of authors.

Recently, the use of pre-trained language models (e.g., BERT, ELMo, ULM-FiT, GPT-2) has been demonstrated to obtain significant gains in several text classification tasks including sentiment analysis, emotion classification, and topic classification [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] Howard | Universal language model fine-tuning for text classification[END_REF][START_REF] Peters | Deep contextualized word representations[END_REF][START_REF] Radford | Language models are unsupervised multitask learners[END_REF]. However, it is not yet clear whether they can be equally useful for style-based text categorization tasks. Especially, in cross-topic AA, information about the topic of texts can be misleading.

An approach based on neural network language models achieved top performance in recent shared tasks on authorship verification and authorship clustering (i.e., grouping documents by authorship) [START_REF] Rosso | Overview of pan'16[END_REF][START_REF] Stamatatos | Overview of the pan/clef 2015 evaluation lab[END_REF]. This method is based on a character-level recurrent (RNN) neural network language model and a multiheaded classifier (MHC) [START_REF] Bagnall | Author identification using multi-headed recurrent neural networks[END_REF]. So far, this model has not been tested in closed-set attribution which is the most popular scenario in relevant literature. In this paper, we adopt this approach for the task of closed-set AA and more specifically the challenging cases of cross-topic and cross-genre AA.

We examine the use of pre-trained language models (e.g., BERT, ELMo, ULMFiT, GPT-2) in AA and the potentials of MHC. We also demonstrate that in cross-domain AA conditions, the effect of an appropriate normalization corpus is crucial.

Previous Work

The vast majority of previous work in AA focus on the closed-set attribution scenario. The main issues is to define appropriate stylometric measures to quantify the personal style of authors and the use of effective classification methods [START_REF] Neal | Surveying stylometry techniques and applications[END_REF][START_REF] Stamatatos | A survey of modern authorship attribution methods[END_REF].

A relatively small number of previous studies examine the case of cross-topic AA. In early approaches, features like function words or part-of-speech n-grams have been suggested as less likely to correlate with topic of documents [START_REF] Madigan | Author identification on the large scale[END_REF][START_REF] Menon | Domain independent authorship attribution without domain adaptation[END_REF]. However, one main finding of several studies is that low-level features, like character n-grams, can be quite effective in this challenging task [START_REF] Sapkota | Cross-topic authorship attribution: Will out-of-topic data help?[END_REF][START_REF] Stamatatos | On the robustness of authorship attribution based on character n-gram features[END_REF]. Typed character n-grams provide a means for focusing on specific aspects of texts [START_REF] Sapkota | Not all character n-grams are created equal: A study in authorship attribution[END_REF]. Interestingly, character n-grams associated with word affixes and punctuation marks seem to be the most useful ones in cross-topic AA. Another interesting idea is to apply structural correspondence learning using punctuation-based character n-gram as pivot features [START_REF] Sapkota | Domain adaptation for authorship attribution: Improved structural correspondence learning[END_REF]. Recently, a text distortion method has been proposed as a pre-processing step to mask topic-related information in documents while keeping the text structure (i.e., use of function words and punctuation marks) intact [START_REF] Stamatatos | Masking topic-related information to enhance authorship attribution[END_REF].

There have been attempts to use language modeling for AA including traditional n-gram based models as well as neural network-based models [START_REF] Bagnall | Author identification using multi-headed recurrent neural networks[END_REF][START_REF] Fourkioti | Language models and fusion for authorship attribution[END_REF][START_REF] Ge | Authorship attribution using a neural network language model[END_REF]. The latter is closely related to representation learning approaches that use deep learning methods to generate distributed text representations [START_REF] Ding | Learning stylometric representations for authorship analysis[END_REF][START_REF] Kocher | Distributed language representation for authorship attribution[END_REF]. In all these cases, the language models are extracted from the texts of known authorship.

As a result, they heavily depend on the size of the training set per candidate author.

The Proposed Method

An AA task can be expressed as a tuple (A, K, U ) where A is the set of candidate authors (suspects), K is the set of known authorship documents (for each a ∈ A there is a K a ⊂ K) and U is the set of unknown authorship documents. In closedset AA, each d ∈ U should be attributed to exactly one a ∈ A. In cross-topic AA, the topic of documents in U is distinct with respect to the topics found in K, while in cross-genre AA, the genre of documents in U is distinct with respect to the genres found in K.

Bagnall introduced an AA method1 [START_REF] Bagnall | Author identification using multi-headed recurrent neural networks[END_REF] and obtained top positions in shared tasks in authorship verification and authorship clustering [START_REF] Rosso | Overview of pan'16[END_REF][START_REF] Stamatatos | Overview of the pan/clef 2015 evaluation lab[END_REF]. The main idea is that a character-level RNN is produced using all available texts by the candidate authors while a separate output is built for each author (MHC). Thus, the recurrent layer models the language as a whole while each output of MHC focuses on the texts of a particular candidate author. To reduce the vocabulary size, a simple pre-processing step is performed (i.e., uppercase letters are transformed to lowercase plus a symbol, punctuation marks and digits are replaced by specific symbols) [START_REF] Bagnall | Author identification using multi-headed recurrent neural networks[END_REF].

Text

MHC LM

Selector Fig. 1: The proposed model consists of two parts, the language model (LM) and the multi-headed classifier (MHC). The DEMUX layer in MHC part functions as a demultiplexer, its state is defined by the selector. During training phase the selector is defined by the author of the input text and during calculation of normalization vector or test phase the input of DEMUX is connected to all its outputs.

The model, as shown in Figure 1, consists of two parts, LM and MHC. LM consists of a tokenization layer and the pre-trained language model. MHC comprises a demultiplexer which helps to select the desirable classifier and a set of |A| classifiers, where |A| is the number of candidate authors. Each classifier has N inputs, where N is the dimensionality of the LM's representation, and V outputs, where V is the size of the vocabulary. The vocabulary is created using the most frequent tokens. The output of LM is a representation of each token in text. If the token exists in vocabulary its representation propagates to MHC, otherwise is ignored (despite the fact that the representation is not further useful, the calculations that took place in LM to produce the representation are mandatory to update the hidden states of the pre-trained language model). If the sequence of input tokens is modified, the representation is also affected.

The function of LM remains the same during training, calculation of normalization vector n and test phase. The MHC layer during training propagates the LM's representations only to the classifier of the author a which is the author of the given text. Then the cross-entropy error is back-propagated to train MHC. During the test phase (as well as the calculation of normalization vector n explained below) the LM's representation is propagated to all classifiers.

The MHC calculates the cross-entropy H(d, K a ) for each input text d and the training texts of each candidate author K a . The lower cross-entropy is, the more likely for author a to write document d. However, the scores obtained for different candidate authors are not directly comparable due to different bias at each head of MHC. To handle this problem, a normalization vector n is used which is equal to zero-centered relative entropies produced by using an unlabeled normalization corpus C [START_REF] Bagnall | Author identification using multi-headed recurrent neural networks[END_REF]:

n = 1 |C| d i ∈C H(d i , K a ) ( 1
)
where |C| is the size of the normalization corpus. Note that in cross-domain conditions it is very important for documents in C to include documents belonging to the domain of d. Then, the most likely author a for a document d ∈ U is found using the following criterion:

arg min a (H(d, K a ) -n) (2)
In this paper, we extended Bagnall's model in order to accept tokens as input and we propose the use of a pre-trained language model to replace RNN in the aforementioned AA method. The RNN proposed by Bagnall [START_REF] Bagnall | Author identification using multi-headed recurrent neural networks[END_REF] is trained using a small set of documents (K for closed-set AA). In contrast, pre-trained language models have been trained using millions of documents in the same language. Moreover, RNN is a character-level model while the pre-trained models used in this study are token-level approaches. More, specifically, the following models are considered:

-Universal Language Model Fine-Tuning (ULMFiT) : It provides a contextual token representation obtained from a general domain corpus of millions of unlabeled documents [START_REF] Howard | Universal language model fine-tuning for text classification[END_REF]. It adopts left-to-right and right-to left language modeling in separate networks and follows auto-encoder objectives.

-Embeddings from Language Models (ELMo): It extracts context-sensitive features using a left-to-right and a right-to-left language modeling [START_REF] Peters | Deep contextualized word representations[END_REF]. Then, the representation of each token is a linear combination of the representation of each layer. -Generative Pretrained Transformer 2 (GPT-2): It is based on a multi-layer unidirectional Transformer decoder [START_REF] Vaswani | Attention is all you need[END_REF]. It applies a multi-headed self-attention operation over the input tokens followed by position-wise feed-forward layers [START_REF] Radford | Language models are unsupervised multitask learners[END_REF]. -Bidirectional Encoder Representations from Transformer (BERT): It is based on a bidirectional Transformer architecture that can better exploit contextual information [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF]. It masks a percentage of randomly-selected tokens which the language model is trained to predict.

Experiments

Corpus

We use the CMCC corpus introduced in [START_REF] Goldstein-Stewart | Person identification from text and speech genre samples[END_REF] and also used in previous crossdomain AA works [START_REF] Sapkota | Cross-topic authorship attribution: Will out-of-topic data help?[END_REF][START_REF] Stamatatos | Masking topic-related information to enhance authorship attribution[END_REF]. CMCC is a controlled corpus in terms of genre, topic and demographics of subjects. It includes samples by 21 undergraduate students as candidates authors (A), covering six genres (blog, email, essay, chat, discussion, and interview) and six topics (catholic church, gay marriage, privacy rights, legalization of marijuana, war in Iraq, gender discrimination) in English. To ensure that the same specific aspect of the topic is followed, a short question was given to subjects (e.g., Do you think the Catholic Church needs to change its ways to adapt to life in the 21th Century?). In two genres (discussion and interview) the samples were audio recordings and they have been transcribed into text as accurately as possible maintaining information about pauses, laughs etc. For each subject, there is exactly one sample for each combination of genre and topic. More details about the construction of this corpus are provided in [START_REF] Goldstein-Stewart | Person identification from text and speech genre samples[END_REF].

Experimental Setup

In this study, our focus is on cross-topic and cross-genre AA. In cross-topic, we assume that the topic of training texts (K) is different from the topic of test texts (U ) while all texts (both K and U ) belong in the same genre. Similar to [START_REF] Stamatatos | Masking topic-related information to enhance authorship attribution[END_REF] and [START_REF] Sapkota | Cross-topic authorship attribution: Will out-of-topic data help?[END_REF], we perform leave-one-topic-out cross-validation where all texts on a specific topic (within a certain genre) are included in the test corpus and all remaining texts on the remaining topics (in that genre) are included in the training corpus. This is repeated six times so that all available topics to serve exactly once as the test topic. Mean classification accuracy over all topics is reported. Similar to cross-topic, in cross-genre we perform leave-one-genre-out crossvalidation as in [START_REF] Stamatatos | Masking topic-related information to enhance authorship attribution[END_REF], where all texts on a specific genre (within a certain topic) are included in the test corpus and all remaining texts on the remaining genres (in that topic) are included in the training corpus. The number of available genres is also six like topics, and though we repeat the leave-one-genre-out cross-validation six times and report the mean classification accuracy. In both scenarios, crosstopic and cross-genre, the candidates authors set A consists of 21 undergratuate students as mentioned in section 4.1.

All the examined models use a MHC on top of a language modeling method. First, we study the original Bagnall's approach where a character-level RNN is trained over K. Then, each one of the pre-trained language models described in previous section. In our experiments, all of the pre-trained LMs was fine-tuned for the specific AA task with MHC as classifier without further training the language model , since our goal is to explore the potential of pre-trained models obtained from general domain corpora.

In MHC, each author corresponds to a separate classifier with N inputs and M outputs, where N is the dimensionality of text representation, Table 1, and M is equal to vocabulary size V . During training, each classification layer is trained only with the documents of the corresponding author. The vocabulary is defined as the most frequent tokens in the corpus. These are less likely to be affected by topic shifts and the reduced input size increases the efficiency of our approach. The selected values of V are 100, 500, 1k, 2k and 5k. Each model used its own tokenization stage except from ELMo (where ULMFiT's tokenization was used). Note that RNN is a character-level model while all pre-trained models are token-based.

Since RNN is trained from scratch for a corpus of small size, it is considerably affected by initialization. As a result, there is significant variance when it is applied several times to the same corpus. To compensate this, we report average performance results for 10 repetitions. Regarding the training phase of each method, we use 100 epochs for RNN and examine four cases for the pre-trained models: the minimal training of 1 epoch and the cases of 5, 10 and 20 epochs of training.

Results on Cross-topic AA

Table 2 presents the leave-one-topic-out cross-validation accuracy results for each one of the six available genres as well as the average performance over all genres for each method. Two cases are examined: one using the (unlabeled) training texts as normalization corpus (C = K) and another where the (unlabeled) test texts are used as normalization corpus (C = U ). The former means that C includes documents with distinct topics with respect to the document of unknown authorship while the latter ensures that there is perfect thematic similarity. As can be seen, the use of a suitable normalization corpus is crucial to enhance the performance of the examined methods.

As concerns individual pre-trained language models, BERT and ELMo are better able to surpass the RNN baseline while ULMFiT and GPT-2 are not that competitive. In addition, BERT and ELMo methods need small number of training epochs while ULMFiT and GPT-2 improve with increased number of epochs.

Table 2 also shows the corresponding results from previous studies on crosstopic AA using exactly the same experimental setup. These baselines are based on character 3-grams features and a SVM classifier (C3G-SVM) [START_REF] Sapkota | Cross-topic authorship attribution: Will out-of-topic data help?[END_REF], a compressionbased method (PPM5) [START_REF] Stamatatos | Masking topic-related information to enhance authorship attribution[END_REF], and a method using text distortion to mask thematic information (DV-MA) [START_REF] Stamatatos | Masking topic-related information to enhance authorship attribution[END_REF]. As can be seen, when C = U all of the examined methods surpass the best baseline in average performance and the improvement is high in all genres. It is remarkable that all models except ULMFiT achieve to surpass the baselines (in average performance) even when C = K. From the aspect of vocabulary size, in contradiction to the state of the art [START_REF] Stamatatos | Masking topic-related information to enhance authorship attribution[END_REF], where the best results achieved for vocabularies that consisted of less than 1k words (most frequent), in our set up the most appropriate value seems to be above 2k. Despite the gap between 2k and 5k words in vocabulary size BERT and ELMo have minor difference in accuracy indicating that above 2k words the affect of vocabulary size is minor. GPT-2 continues to increment the accuracy and ULMFiT started to decrement for values above 1k words, Table 2. Experiments with values over 5k were prohibitive due to runtime of training, with 5k words the runtime was approximately 4 days for each model running on GPU.

From the aspect of training epochs, BERT and ELMo achieved their best performance in C = K cace with minimal training. In C = U case their performance is slightly affected by the number of training epochs. This behavior raises the question of over-fitting. As mentioned in section 3 the selection criterion Equation 2, is based on the cross-entropy of each text. MHC is trained on predicting the text flow and thus the cross-entropy decreases after each epoch of training. Having in mind the cross-entropy, if we have a second look on Figure 2, the case of over-fitting is rejected since the behavior of accuracy in relevance with the number of training epochs (indicated by the shape of point) do not have the characteristics of over-fitting (increment of training epochs decrements the accuracy).

Results on Cross-Genre AA

The experiments on cross-genre performed on the same set up as in cross-topic. Table 3 presents the accuracy results on leave-one-genre-out cross-validation for each one of the six available topics and the average performance over all topics, similar to Table 2. Based on the results of Section 4.3 the most reasonable value of V in order to check the performance of each method is V = 2k. The case of V = 5k is very time consuming without offering valuable gain and below 1k the performance is not remarkable. For the experiments on cross-genre the values of 1k and 2k were selected for V . Comparing the two cases, the results with V = 2k surpass in all experiments the results with V = 1k and thus we selected to present only the case of V = 2k on Table 3.

BERT and ELMo achieved high results as expected from their performance on cross-topic, with ELMo achieving the highest accuracy result. Unexpectedly, ULMFiT which had the worst performance in cross-topic achieved the second best performance. GPT-2 performed lower than RNN baseline in both cases of C = K and C = U . Comparing Table 2 and Table 3 is noticeable that ELMo and BERT are more stable in performance than GPT-2 and ULMFiT. The main diference between the former and the latter is the directionality, the former two are bidirectional while the latter are unidirectiocal, we suspect that this is the main reason that affects the stability in performance.

Conclusions

In this paper, we explore the usefulness of pre-trained language models in crossdomain AA. Based on Bagnall's model [START_REF] Bagnall | Author identification using multi-headed recurrent neural networks[END_REF], originally proposed for authorship verification, we compare the performance when we use either the original characterlevel RNN trained from scratch in the small-size AA corpus or pre-trained tokenbased language models obtained from general-domain corpora. We demonstrate that BERT and ELMo pre-trained models achieve the best results while being the most stable approaches with respect to the results in both scenarios. -PPM5 [START_REF] Stamatatos | Masking topic-related information to enhance authorship attribution[END_REF] -60.00 DV-MA [START_REF] Stamatatos | Masking topic-related information to enhance authorship attribution[END_REF] -33.00

A crucial factor to enhance performance is the normalization corpus used in the MHC. In cross-domain AA, it is very important for the normalization corpus to have exactly the same properties with the documents of unknown authorship. In our experiments, using a controlled corpus, it is possible to ensure a perfect match in both genre and topic. In practice, this is not always feasible. A future work direction is to explore how one can build an appropriate normalization corpus for a given document of unknown authorship. Other interesting extensions of this work is to study the effect of extending fine-tuning to language model layers and focus on the different layers of the language modeling representation.

Figure 2

 2 presents the mean classification accuracy with respect to vocabulary size on cross-topic AA. Each sub-figure correspond to different LM. The type of the line indicates the normalization corpus, dashed line indicates the use of training texts (C = K) as normalization corpus, while the continues line indicates the use of test texts (C = U ), as noted in section 3 in both cases the texts are unlabeled. The shape of each point correspond to epochs of training, 1, 5, 10 and 20 for circle, triangle, square and x-mark respectively.

Fig. 2 :

 2 Fig. 2: Accuracy results for vocabulary sizes (V ) 100, 500, 1k, 2k and 5k for each pre-trained model. Colored symbols blue circle, red triangle, yellow square and green x-mark correspond to 1, 5, 10 and 20 training epochs, respectively. The type of the line indicates the normalization corpus, a dashed line indicates the use of training texts (C = K) as normalization corpus, while a solid line indicates the use of test texts (C = U ).

Table 1 :

 1 Dimensionality of representation (N ) for each language model in this study.

	Model RNN BERT ELMo GPT-2 ULMFiT
	N	149 768 1024 768	400

Table 2 :

 2 Accuracy results (%) on Cross-Topic AA. The reported performance of baseline models is taken from the corresponding publications. ‡ LM V epochs Blog Email Essay Chat Disc. Interv. Avg.

		RNN	-	100 47.94 44.37 41.00 73.41 75.71 72.54 59.16
		BERT	2k	1	57.14 49.21 60.32 84.92 79.37 80.16 68.52
		BERT	5k	1	53.97 52.38 58.73 86.51 77.78 78.57 67.99
	K	ELMo	2k	1	56.35 55.56 56.35 80.95 72.22 76.98 66.40
	C =	ELMo GPT-2	5k 2k	1 20	55.56 53.17 57.14 82.54 70.64 76.19 65.87 60.32 57.94 54.76 76.98 63.49 79.37 65.48
		GPT-2	5k	20	58.73 59.52 61.11 84.13 63.49 76.98 67.33
		ULMFiT	2k	10	50.00 43.65 52.38 79.37 72.22 71.43 61.51
		ULMFiT	5k	20	46.83 40.48 50.79 80.16 69.84 70.64 59.79
		RNN	-	100 61.67 56.43 68.36 81.27 86.90 84.52 73.19
		BERT	2k	5	72.22 64.29 76.98 90.48 84.13 90.48 79.76
		BERT	5k	5	73.81 61.11 77.78 92.86 84.13 90.48 80.03
	U	ELMo	2k	10	72.22 65.08 75.40 89.68 76.19 91.27 78.31
	C =	ELMo GPT-2	5k 2k	10 20	72.22 67.46 76.98 88.10 76.98 89.68 78.57 72.22 64.29 73.02 80.16 67.46 82.54 73.28
		GPT-2	5k	20	69.84 65.87 69.84 84.13 73.81 85.71 74.87
		ULMFiT	1k	10	64.29 57.94 73.02 87.30 80.16 88.89 75.26
		ULMFiT	2k	10	64.29 54.76 73.81 88.89 78.57 88.10 74.74
		ULMFiT	5k	20	58.73 54.76 75.40 88.89 75.40 84.13 72.88
		C3G-SVM [19] -	-	33.41 36.53 36.66 57.46 49.91 56.35 45.05
		PPM5 [22]	-	-	52.38 39.68 50.00 57.94 36.51 47.62 47.35
		DV-MA [22]	-	-	43.65 65.87 60.32 71.43 80.16 67.46 64.81

Table 3 :

 3 Accuracy results (%) on cross-genre AA for vocabulary size 2k (V = 2k) and each topic (Church (C), Gay Marriage (G), War in Iraq (I), Legalization of Marijuana (M), Privacy Rights (P), Gender Discrimination (S)). The reported performance of the baseline models (only available in average across all topics) is taken from the corresponding publications. .30 88.10 82.54 77.78 78.57 83.07 ELMo 20 87.30 88.89 88.89 83.33 76.98 81.75 84.52 GPT-2 20 69.84 76.98 74.60 67.46 61.11 72.22 70.37 ULMFiT 10 88.10 89.68 85.71 82.54 77.78 79.37 83.86 C3G-SVM [19]

	‡ LM	epochs C	G	I	M	P	S Avg.
		RNN	100 58.89 68.33 71.59 60.24 50.40 62.22 61.94
	K	BERT	10	70.63 77.78 83.33 73.81 62.70 76.98 74.21
	C =	ELMo GPT-2	10 20	68.25 78.57 78.57 71.43 55.56 65.08 69.58 52.38 67.46 61.11 57.94 50.79 53.17 57.14
		ULMFiT	20	72.22 77.78 79.37 70.63 61.11 68.25 71.56
		RNN	100 75.32 75.95 86.11 79.52 69.37 74.21 76.75
	U	BERT	5	84.13 87
	C =

https://github.com/pan-webis-de/caravel


D Panteleris

N Michel

K Pantzou

Dimitrios Argyros
email: argyros@ics.forth.gr

Kosmopoulos

G Styliaras

C Constantinopoulos

P Panteleris

D Michel

N Pantzou

K Papavasileiou

K Tzortzi

A Argyros

D Kosmopoulos

The MuseLearn Platform: Personalized Content for Museum Visitors Assisted by Vision-Based Recognition and 3D Pose Estimation of Exhibits

Keywords: Museum Guide System, Recommender System, Pose estimation, Content Management System

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Museums are perhaps the most important institutions for the preservation and promotion of the world's cultural heritage and act as powerful learning environments as well, contributing to social and economic development. Traditionally, museums used linear, non-interactive ways of presenting exhibits and guide material, e.g. audio tours or QR codes that offer static information. Μultimedia technologies have recently begun to be exploited, but they require considerable investment on the part of museums and often have limitations on the extent of the presented material but also on the number of people who can use them. Also, a key concern for museums has been the evaluation of their exhibitions and activities, as it is critical to understand how satisfied visitors are about the exhibits, the additional digital content and the way guidance material is presented.

In this paper, we present the initial results of the project MuseLearn, the main objective of which is the development of an innovative guide platform in a museum for mobile devices that will provide personalized additional multimedia content by using exhibit detection, as a basis for providing recommendations to visitors. A key aim of the project is to increase the number of museum visitors and their satisfaction.

The project is implemented in the Herakleidon Museum, Athens [1], which is a museum organising temporary exhibitions with focus on ancient Greek technology. The current exhibitions (2019) deal with the themes of ancient automata (such as the Antikythera Mechanism, and ancient war technology (such as a battering ram). The exhibits are of educational character, in the sense that they take the form of informative material, representations and reconstructions based on original artifacts.

Regarding the presentation of the initial results of MuseLearn, we first present the developed in Section 2. In Section 3 we discuss existing approaches and work related to these components. Section 4 presents the management of additional multimedia content that is offered to visitors. Section 5 describes the proposed recommender system along with visitor tracking techniques. Section 6 details the exhibit detection technique used in the project. Section 7 describes the implementation of pilot content and initial evaluation results in all components of the project. Finally, Section 8 discusses the key conclusions of the paper and presents plans for future work.

Platform architecture

The MuseLearn platform (see Figure 1) consists of the following basic systems that are presented in the following sections:

• A content management system that stores multiply structured multimedia material that is offered to visitors through augmented reality for better understanding the exhibits of the museum. • A recommender system that allows adaptation of the information to the needs and requirements of visitors. • An exhibit detection and pose estimation system that can detect the exhibit the visitor is interested in and provides identification (exhibit name/id) and orientation information (3D pose) using the camera of a mobile device (phone/tablet). According to the typical usage scenario, the visitor sees the exhibit via a portable device. The exhibit is then detected by the pose estimation system (based on the images of stored exhibits) and the recommender system suggests appropriate digital material stored in the CMS to the visitor according to their profile. At the same time, visitor choices are recorded for further refining their profile.

3

Related work

Content management systems (CMS): Regarding content management systems (CMS) for museums should offer administrators the ability to structure content in various multimedia formats for artifacts, concepts and supplementary material that is necessary for their presentation [START_REF] Kosmopoulos | A Survey on Developing Personalized Content Services in Museums[END_REF]. A research of commercial and research CMS that are mainly or indirectly targeted for museums is found in [START_REF] Lanir | Visualizing museum visitors' behavior: Where do they go and what do they do there?[END_REF]. Argus [START_REF] Lucidea | Argus, make your collection more visible and accessible than ever before[END_REF] is a web-based Collections Management Solution (CMS) that can be used for managing and presenting artifacts and objects. Argus provides a tabular visual interface organized in rows for accessing and editing content. The retail oriented RetailPro platform [START_REF] Retailpro | [END_REF] supports responsive design, portable content, reports, transformation of content parameters, reorganization of collections and organization of virtual collections. Content is mainly presented in semantically colored tables. The Museum System (TMS) [START_REF]GallerySystems, Collection management software, museum and art collections[END_REF] is a CMS that supports planning and managing of exhibitions and generates reports; it contains a digital asset management module and offers administrative support. MuseumPlus [START_REF] Zetcom | Museum plus[END_REF] supports managing collections and exhibitions and is in use in over 900 museum sites worldwide. Unlimited images and other multimedia content can be linked to objects, artists, addresses and other record types. CollectiveAccess [START_REF]Collective Access[END_REF] is a web-based open source CMS that embraces Dublin Core, PBCore, VRA Core etc. The embedded media viewer allows to enlarge and inspect uploaded images, video and audio playback with time-based annotations and PDF viewers. Museum Anywhere [START_REF] Anywhere | [END_REF] offers collection management and tile-based presentation through mobile devices and supports integrating content from other museum management software. The proposed CMS aims to overcome limitations of existing museum CMS by introducing a simple but flexible and expandable structure for organizing content about exhibits and concepts, which can be visualized by using modern web technologies and exploiting the spatial and interface capabilities of mobile devices.

Museum recommendation systems: So far, museum recommendation systems are mainly used for supporting personalized museum tours. There are examples of recommendation systems in museums that offer unique tours based on specific interests [START_REF] Keller | Recommender Systems for Museums: Evaluation on a Real Dataset[END_REF][START_REF] Bohnert | Non-intrusive Personalisation of the Museum Experience[END_REF]. The traditional recommendation approaches are (a) Content-based filtering (e.g., [START_REF] Pazzani | Content-Based Recommendation Systems[END_REF]) gives recommendations based on similar content that was of interest to the user in the past and (b) Collaborative filtering (e.g., [START_REF] Herlocker | An algorithmic framework for performing collaborative filtering[END_REF]) that are based on content that was of interest to the users that are most similar to the current one based on their selections. Combination of these leads to hybrid approaches. The latent factor models (e.g. [START_REF] Koren | Matrix Factorization Techniques for Recommender Systems[END_REF]) are more recent and try to factorize the matrix that associates users and items to be selected via SVD, or sparse methods. Therefore, hidden relations between users and items may be identified. The Herakleidon Museum can provide a great deal of information to the visitors, in a variety of formats, such as texts, pictures, sounds, videos, games, etc., instead of a comprehensive description of the exhibits. Thus, arises the need for filtering, hierarchy and effective delivery of this information. Our goal is to create a recommender system to manage the large amount of multimedia information accompanying the exhibits of and to recommend visitors to explore the museum on the basis of their interest. Exhibit detection and 3D pose estimation: Regarding exhibit detection and 3D pose estimation modules, there is a large number of relevant works. Some recent relevant publications address some of the requirements of the MuseLearn pose estimation module. A very significant building block of the overall approach is the capability to localize a camera relative to the scene. Kinect Fusion [START_REF] Newcombe | KinectFusion: Real-time dense surface mapping and tracking[END_REF] proposed by Newcomb et al, fuses all the data from a depth camera into a voxel space, that represents a dense 3D model of the scene to be reconstructed. ICP is used with it to track the camera position. More elaborated versions [START_REF] Whelan | Real-time largescale dense RGB-D SLAM with volumetric fusion[END_REF] allows it to operate in larger scenes. Although promising, the method requires depth input to operate and can be sensitive to outliers and specular surfaces. These requirements make it a bad fit for the MuseLearn scenarios. Mur-Artal et al [START_REF] Mur-Artal | ORB-SLAM: a Versatile and Accurate Monocular SLAM System[END_REF][START_REF] Mur-Artal | ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras[END_REF] proposed the ORBSlam method. OrbSlam2 performs simultaneous localization and mapping of the environment. It tracks the camera position and builds a map of the environment based on ORB features. The map contains a sparse 3D point cloud, each point being associated with an ORB descriptor, and some geometric information about the view corresponding to it. A set of keyframes (i.e., representative frames) is also maintained in time, and each converted to a bag of words (BoW) descriptor [START_REF] Galvez-Lopez | Bags of binary words for fast place recognition in image sequences[END_REF]. This approach operates with RGBD or monocular/stereo RGB input, can make use of a static map, and works in real time, making it a possible base solution for our problem. However, the method assumes a static scene and is not robust to moving objects and large numbers of outliers. More recently, deep-learning based methods [START_REF] Tekin | Real-time seamless single shot 6d object pose prediction[END_REF][START_REF] Peng | Pvnet: Pixel-wise voting network for 6dof pose estimation[END_REF] solve the object pose estimation problem from RGB input using Convolutional Neural Networks (CNNs). The networks are trained to predict the 2D coordinates of the projections of known 3D landmarks on the objects of interest. Subsequently, they solve the PnP problem to acquire the 3D object pose. Although promising, these approaches do not scale well with multiple exhibits and suffer in dynamic environments and in the case of partially occluded exhibits. This renders them unsuitable for the MuseLearn use case. FORTH's proposed method for exhibit identification and pose estimation borrows many ideas from [START_REF] Mur-Artal | ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras[END_REF], especially for the training phase as it will be detailed in later sections. Lourakis and Zabulis [START_REF] Lourakis | Model-Based Pose Estimation for Rigid Objects[END_REF] detect the pose of known rigid objects in monocular RGB images, using a 3D model for each object. The model consists of 3D points each associated with a SIFT descriptor. A handheld camera is used to capture the appearance of the object from all view angles. Then, the camera motion together with the 3D point cloud describing the object is recovered using "Shape from Motion" techniques similar to [START_REF] Snavely | Photo Tourism: Exploring Photo Collections in 3D[END_REF]. The authors report a run time of 0.6 s per frame. While the work of Lourakis et al is a good candidate for solving our problem, the dense object representation makes it too computationally expensive, and the lack of use of background features makes it harder to detect featureless objects. However, aspects of this work such as the F2P features matching strategy and the pose estimation method (posest library) are used in the proposed method.

Content management system

Modern web technologies have made possible the visualization of content on a spatial area. For example, Scalable Vector Graphics (SVG) and Cascading Style Sheets (CSS) may be employed for implementing vector shapes and animations that are presented uniformly in devices of different sizes and interface options. On the other hand, in a smartphone world, content-editing operations should resemble natural operations by introducing spatial interfaces and gestures and not relying on providing a desktop experience on a smaller screen. The Content Management System (CMS) developed for MuseLearn embraces these ideas for displaying and interacting with content spatially. Two on-site surveys have been conducted in order to record the information provided by the museum and all kinds of exhibits and concepts. Information about exhibits and their related multimedia informative material has provided the basis for populating the system's database. The database of the CMS stores all information about the museum's exhibits and concepts that they may be assigned to. Database design is flexible enough so exhibits may be assigned to multiple concepts along with other multimedia material linked to them. More specifically, the main entities in the DB are:

• Concepts: The concepts that museum exhibits may be assigned to. For example, the exhibit "battering ram" is assigned to the concept "combative weaponry". Concepts also describe all kinds of informative material within the museum. • Exhibits: The main table holding information about every exhibit, such a title and description about the exhibit, the exhibit code within the museum, the museum room where the exhibit lays and the exhibit types. • Links: This table is responsible for defining all possible relations among exhibits and concepts. There may be relations between exhibits; between concepts for defining the concepts hierarchy. • Multimedia: Multimedia content that is linked to an exhibit or a concept.

The database has been implemented in MySQL and content has been inserted about the presentational needs of exhibits and their supplementary content for both museum buildings along with the hierarchical structure of concepts for the certain museum and all kinds of relations among them. More specifically, the database currently stores 98 concepts of all hierarchy levels; 82 exhibits; 152 links of all types and 180 multimedia items for concepts and exhibits.

PHP code has been employed for the hierarchical visualization of museum content starting from the museum level, continuing with the two main collections ("Automata" and "Ancient War Technology") and finishing at single concept and exhibit level. For every concept, its description and multimedia content are displayed along with links to subordinate concepts that analyze it and exhibits that are related directly to the concept. Visualization is both textual, as in a subject catalog, and graph-based. The latter is implemented in SVG, so that it is accessible by all device types. Treant.js [22] free to use, graph creation library has been used for drawing content hierarchies.

Recommender system

For the recommender system a trial mode is scheduled to initiate a few users. Gradually and based on data from more users, the system will be retrained. The preferences will arise indirectly from the viewing time of the accompanying material. Also, when entering the museum, the user will have a profile that matches his/her demographics and/or short quote when signing up. Progressively the profile will be completed during the visit and as long as its preferences are known. We currently use alternatively latent factor approaches. For a new item or a new visitor, the new object has no score and the new user has no history, which is known as the cold-start problem. So any recommendation result could be doubtful. We handle it by using demographic data, which has been an effective method for cold-start in the past [START_REF] Solanki | Recommender system using collaborative filtering and demographic characteristics of users[END_REF]. Different versions of the same content are associated with some initial and grossly-defined profiles based on a few deterministic rules, which are tested against a questionnaire as described in the following subsection. After some views the system is able to fine-tune the user profile.

The question as to how to identify and distinguish visitor profiles has been a key issue in both the museum studies (e.g., [START_REF] Serrell | Paying attention: Visitors and museum exhibitions[END_REF][START_REF] Falk | Identity and the museum visitor experience[END_REF]), and information systems literatures (see [START_REF] Kosmopoulos | A Survey on Developing Personalized Content Services in Museums[END_REF]). For the pilot system, it was decided to create and organize the content, addressing three broad requirements related to visitors' prior knowledge, experience and expectations about the exhibition: 'general interest', 'specialized knowledge', and 'enthusiasm about new technology'. To identify these broad requirements, a group of basic questions forms the starting section of the pilot system. These focus on visitors' demographic profile (sex, age range, highest education level), and their familiarity both with the subject of the exhibition and the use of technology. A sixth key question asks for the driving motivation behind the visit. This question draws on fundamental distinction of five identity-related categories of visitors ('explorers', 'facilitators', 'professional/hobbyists', 'experience seekers', and 'rechargers') [START_REF] Falk | Identity and the museum visitor experience[END_REF]. At the pilot stage of the project, visitors' responses to these questions will offer, in combinations, 'personalized' content at the most basic level. For example, a visitor aged 17 and familiar with new technologies will be offered 'general interest' content, enhanced by interactive educational games. Visitors' responses will provide the foundations on which to fine-tune later the recommender system. These six questions are also used for the primary evaluation procedure, as described in Section 7.2.

Pose estimation system

The developed exhibit localization and pose estimation method is tailored to the specialized requirements of MuseLearn. Specifically, the inference is performed using RGB input; the solution is scalable to large exhibitions; it supports multiple levels of pose estimation detail; is associated with small (re)configuration overhead; and achieves real time performance. The method is using a "features and reference poses" database that is generated offline, in order to identify the pose of exhibits in query images captured by mobile devices.

Training phase

Creating the features and reference poses database is a critical step for the correct operation of the pose estimation pipeline. In order to maximize the accuracy, we employed the use of RGB-D cameras in the training phase. This type of sensors (i.e. MS Kinect2, Asus Xtion, ORBEC) provide apart from the usual red, green and blue channels an additional channel for depth. This channel provides distance information for each imaged point of the scene.

Training sequence acquisition: For each exhibit we capture an RGB-D video sequence. The goal is to cover a wide range of possible visitor viewing directions.

Feature and pose extraction:

We apply a SLAM method to extract features and key frames and camera poses. For the purposes of MuseLearn we chose ORBSlam2 [START_REF] Mur-Artal | ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras[END_REF] as the SLAM method and the ORB [START_REF] Rublee | ORB: an efficient alternative to SIFT or SURF[END_REF] features and descriptor for feature extraction. Exhibit annotation: An annotator selects a reference frame, pose and bounding volume for each exhibit in the museum coordinate frame. The common coordinate frame is required in order to report pose information to the other MuseLearn modules.

Exhibit DB: In the final step we automatically select frames from the training set where each exhibit is fully visible. The ORB features from these "keyframes" are converted into a bag of words (BoW) representation and are inserted in the exhibit database.

Fig. 2:

The pipeline of the pose estimation method during inference. From left to right: (i) the query image captured using the camera on the mobile device, (ii) ORB features are detected on the query image and their descriptors are computed, (iii) The BoW representation of the ORB features is used to find the most similar (high similarity score) keyframes in the exhibit db, (iv,v) Pose refinement using exhibit frames and matching ORB features from the query image, (vi) the exhibit pose with the highest score is returned.

Exhibit identification and pose estimation

The first step in the pose inference pipeline is to compute the 2D features O(I) in the input image I. From these, a BoW representation q is created. We compute the similarity score S between the BoW of the query image and all the keyframes K in the exhibit database. The set C of keyframes that have a similarity score above a threshold Si (q,Ki )>Ts are the initial "coarse" estimations. If the query from the mobile device requires only exhibit identification (no pose estimation), the exhibit ID of the keyframe in C with the highest similarity score is returned. The pipeline is depicted in Fig. 2.

The pose estimation is a refinement step over C using the features O(I). Initially we compute C' that is the subset of keyframes in C that belong to the identified exhibit. Using the ORB descriptors and matching described in [START_REF] Rublee | ORB: an efficient alternative to SIFT or SURF[END_REF] we find the best correspondences M(O(I),Kj) between O(I)and the ORB features of each Kj keyframe in C'. Posest [START_REF] Lourakis | Model-Based Pose Estimation for Rigid Objects[END_REF] is applied to all matches were |M|<Tm. Tm is the minimum number of features on the keyframe that belong to the exhibit (i.e., not in the background) and have a good match in O(I). Posest uses a RANSAC scheme to iteratively select a subset of M and compute the rigid transformation that best explains the camera motion with respect to a given keyframe pose. The quality of the transformation is measured using the reprojection error of the matched ORB features from the query frame to the keyframe. The pose with the highest score is selected as the camera pose with respect to the exhibit.

7

Pilot implementation and primary evaluation procedure and results

During the pilot implementation, we had the chance to test the various systems of the MuseLearn platform in the Herakleidon museum. Pilot multimedia-rich content has been developed for certain exhibits. When these exhibits are identified by the pose estimation system through visitors' mobile devices, the additional material will be displayed on visitors' devices based on the suggestions of the recommender system.

Content for pilot implementation

We used the CMS described in Section 4. First, a pilot extended scenario of 75 screens for a specific section of the exhibition "EUREKA Science, Art and Technology of the Ancient Greeks" was drafted. Focus was centered on exhibits related to armors and weapons from the Mycenean to the Hellenistic Period. We decided to fully deploy content for five exhibits belonging to the "Warrior and Armor/Equipment" section of the exhibition and for three assigned categories of users/testers (see Section 5.1). The five exhibits were selected on the basis of their associative and historical value, as well as their potency to enhance the storytelling, increase visitor engagement and unfold stories about individual artifacts or specific object groups.

Content creation decisions were taken by accounting for users of different needs, interests and knowledge (see [START_REF] Raptis | Context-based design of mobile applications for museums: a survey of existing practices[END_REF]). Hence, two levels of content exist for each exhibit: specialized and basic content, enhanced occasionally by interactive applications. This content includes texts, images, videos, hyperlinks and games. More precisely, as in most guide systems, the user can access descriptions of the exhibits, information about their function and the historical and the social context of their use and production from a variety of sources, as well various interpretations and representations, both scientific and popular (see [START_REF] Kosmopoulos | A Survey on Developing Personalized Content Services in Museums[END_REF]) by using the content gathered as described in Section 4. The system also allows users/testers to zoom in and observe object details, consult the glossary for unknown scientific terms, place names and archaeological periods, and search for related content from the museum collection or from other physical or virtual spaces with the help of hyperlinks and object recommendation. Finally, interactive applications through matching, ordering and sequence games and object identification provide users with the opportunity to test their knowledge and acquire new, get acquainted with typologies, explore objects' correlation/interconnection and interact in a playful way.

Primary evaluation procedure and results for visitor satisfaction

Nowadays, evaluation construes integral component of museum practice and research (i.e. [28. 29, 30]), but also of mobile application designing and multimedia development (i.e. [START_REF] Economou | Evaluation strategies in the cultural sector: the case of the Kelvingrove Museum and Art Gallery in Glasgow[END_REF][START_REF] Rasimah | Evaluation of user acceptance of mixed reality technology[END_REF][START_REF] Tost | Exploring the suitability of Virtual Reality interactivity for exhibitions through an integrated evaluation: the case of the Ename Museum[END_REF]). To be beneficial, evaluation should be approached as "an ongoing process" [START_REF] Economou | Evaluation strategies in the cultural sector: the case of the Kelvingrove Museum and Art Gallery in Glasgow[END_REF]. The primary evaluation procedure was carried out by a group of six undergraduate students from the University of Patras. At this preliminary stage, both qualitative and quantitative methodologies were employed. First, a questionnaire was drafted both in English and Greek consisting of 15 questions and organized in two sections. The aim of this survey is to assess visitors' motivation, familiarity with the exhibition topic as well as information and communications technology, their level of satisfaction with information provided through various interpretive means, as well as their potential interest in using a mobile application during their visit.

The questionnaire is self-administered. Therefore, an introductory note informed the participants about the goals and context of this survey. As expected, the last section collected information about the age, gender, higher level of education and place of origin of the respondents. Initial feedback was encouraging. Visitors expressed their strong interest in using a mobile application to navigate around the exhibition space.

The primary evaluation procedure was complemented by observation of visitor behavior, which has become a key element in feedback studies on museum performance (see for example [START_REF] Serrell | Paying attention: Visitors and museum exhibitions[END_REF][START_REF] Tzortzi | Museum space: Where architecture meets museology[END_REF]34,[START_REF] Yalowitz | Timing and Tracking: Unlocking Visitor Behavior[END_REF]). The aim of this research was to observe how visitors move, explore and use space and display in the exhibition 'Technology of War in Ancient Greece', without the use of the mobile guide system. These data will then be used for a comparative study after the implementation of the platform and potentially lead to a deeper understanding of how the platform impacts on visitor experience.

Visitor behavior, in particular their patterns of moving and viewing, is analyzed using established techniques. First, the arrangement of the display is recorded on the building layout as the basis for designing the observation record sheet for mapping visitors' movement and interactions with the displays. Traces of the paths of visitors, who are randomly selected, spread across time periods and have consented to take part in the research, are recorded for their whole visit to the exhibition. When the visitor stops to look at a work, read a text, or watch a video, a stopping point is recorded on the plan of the exhibition by the observer. Other symbols are used to clarify where a visitor stops for longer periods of time. The tracking data are complemented by the recording of the total time visitors spent in the exhibition. The resulting ratings table which associates users with exhibits and their preferences will be used to initialize the system profiles and thus to address the cold-start problem in our system.

Primary evaluation results for the pose estimation system

Regarding the pose estimation system, we evaluated our baseline method using data acquired on two sites. (i) The AMI facility: The ambient intelligence (AMI) facility of FORTH hosts exhibition and demonstration areas. We trained our system on six (6) exhibits selected for their size and placement to be compatible with typical exhibits in a museum. Data were acquired with low-and high-quality cameras to investigate performance with different sensors. A training set and two test sets were acquired. The first test set uses the low quality RGB sensor of an Asus Xtion. The second uses a highquality camera similar to the cameras found in modern phones and tablets. The exhibit locations and reference poses were annotated in the training dataset as well as in a test dataset. (ii) Herakleidon museum: We recorded a preliminary dataset of six exhibits. In both cases the exhibit locations/reference poses were annotated in the training dataset. The preliminary version of the pose estimation method focused on the exhibit identification and coarse pose estimation. Specifically, the system must correctly identify the quadrant from which the visitor is approaching the exhibit: Left, Right Front, Back. The results of our experiments on both sites are given in Table 1 andTable 2

Primary evaluation results for the recommender system

We started to evaluate different approaches using public data, until we collected our proprietary ones for the Herakleidon museum. More specifically we evaluated the data provided in [START_REF] Bohnert | Non-intrusive Personalisation of the Museum Experience[END_REF]. According to this, a dataset of visitors' times spent in each thematically organised exhibit area is created, which were recorded and collected by computer-supported methodology. The recommendations concern specific thematic areas. In order to create appropriate recommendations, the exhibits were grouped into semantic and spatially coherent areas. With this in mind, 126 thematic and naturally organised areas were created at the museum's facilities. The time spent by 158 visitors on these areas of the museum was recorded and expressed implicitly the rating. The viewing times are transformed to logarithmic with values between 1 and 8. We applied two methods: a simple collaborative filtering with KNN and a latent factor method with SVD. The root mean square error is calculated by ten-fold cross validation. With both methods the mean RMSE is around 1 with standard deviation between 0.02 and 0.03. There are similarities and differences with our system. We similarly use the viewing times as an implicit rating mechanism; the target users are museum visitors as well. However, we provide recommendations on supplementary digital material associated to museum artifacts and not directly on museum artifacts.

Future work & Conclusions

We presented the MuseLearn platform that is constituted of three innovative systems (artifact detection, content, recommendation,). Some initial evaluation results were presented after the implementation of the platform for Herakleidon Museum.

In the future, subsystems of MuseLearn will be extended both in functionality and content for covering all exhibits of Herakleidon Museum and other museums. Contentwise, further support will be implemented for inserting and linking multimedia content, such as 3D representations and virtual reality material. Towards this direction, refined visitor profiles will be developed on the basis of the empirical findings, and their preference for certain themes, in conjunction with demographic data. We also plan the creation of more educational applications based on content that will have been prepared for the presentation of exhibits.

Fig. 1 .

 1 Fig. 1. Outline of the MuseLearn platform. The visitor can see the exhibit through a mobile device. The exhibit is then recognized by the system (based on the acquired image and the stored exhibits) and the appropriate digital material is suggested based on the user's profile. Concurrently the users' choices are recorded. Asynchronously, the curator may analyze the exhibition statistics.

Table 1 .

 1 . Average Precision and Recall on exhibit classification accuracy.

		Herakleidon	AMI (low quality)	AMI (high quality)
	Average Precision	0.947	0.980	0.971
	Recall	0.771	0.844	0.810

Table 2 .

 2 The coarse (quadrant) pose estimation accuracy. Two metrics are shown. Left is the error in degrees relative to the ground truth. On the right we quantize the same error using 3 thresholds. The quadrant approach corresponds to "<45 degrees".

	Error in degrees on the yaw axis	% of frames within correct quadrant
	avg error (std dev)	median error	<22.5 degrees	<45 degrees <90 degrees
	8.98 (+/-7.94)	6.73	93.0%	99.8%	100%

Acknowledgement: Co-financed by the EU and Greek national funds through the Operational Program Competitiveness, Entrepreneurship and Innovation, under the call RESEARCH-CREATE-INNOVATE (project: T1EDK-00502 -MuseLearn).