mounir.hafsa Mounir Hafsa

A Multi-Objective E-learning Recommender System at Mandarine Academy *

Keywords: CCS Concepts:, Applied computing → Multi-criterion optimization and decision-making, E-learning, • Information systems → Retrieval models and ranking Recommender Systems, Multi-Objective Optimization, Evolutionary Algorithms, E-Learning, MOOC, Corporate

Recommender systems are quickly becoming a part of our daily digital life. Mainly found in applications such as e-commerce, social media, and online entertainment services. They help users overcome the information overload problem by improving the browsing and consumption experience. Mandarine Academy is an Ed-Tech company that operates more than a hundred online e-learning platforms. They creates online pedagogical content (videos, quizzes, documents, etc.) on daily basis to support the digitization of work environments and to keep up with current trends. Suggesting items that are relevant to both users and visitors is challenging, the company is looking for ways to improve the learning experience by providing content that adheres to specific conflicting requirements.

These requirements include similarity with user profile, the novelty of proposed content, and diversity of recommendations. Mandarine Academy is looking to implement an approach that can handle multiple conflicting goals with the possibility to adjust which to use in each browsing scenario. In this article, we propose a solution for Mandarine Academy Recommendation System (𝑀𝐴𝑅𝑆) problem by using Evolutionary Algorithms based on the concept of Pareto Ranking. After modeling objectives (Similarity, Diversity, Novelty, RMSE, and nDCG@5) as an optimization problem, we compared different algorithms (NSGA II, NSGAIII, IBEA, SPEA2, and MOEAD) to study their performance under different test settings. Extended data analysis of real-world user interactions showed drawbacks of many graphical issues that prevented users from learning efficiently and we proposed enhancements to the overall user experience and interface. We discuss initial findings under various objectives which show promising results considering production mode scenarios. A proposed custom mutation operator was able to outperform the classical swap mutation. A multi-Criteria Decision-Making phase that uses by default pseudo weight is responsible for providing results for end users after training our model.

INTRODUCTION

Mandarine Academy is an Ed-tech company that supports the digital transformation of work environments by facilitating the handling and use of new technologies by all employees. It offers a new way of training more effectively in terms of skills, capacity, time, and budget due to an exclusive approach that combines both digital learning and personalized support. With over half million users and more than 100 platforms, the company operates several products for multiple partners in varying sectors. Mandarine Academy saw how Massive Open Online Course (MOOC) impacted the traditional higher education market as well as the increasing rate of industry digitization and proposed custom MOOCs focused on the corporate sector to save employees from obsolescence.

One of their most popular products is the Mooc-office365-training 1 , a public bilingual (French & English) MOOC destined for learning Microsoft office 365 tools and soft skills related to workplaces. Having around four thousand monthly active users and more than hundred and thirty thousand registered users, the platform includes different types of learning materials such as:

• Resources: Tutorials & use cases (short format videos), quizzes, documents, recorded live conferences, SCORM (Shareable Content Object Reference Model).

• Courses: A collection of unordered learning resources.

• Learning Paths: A predefined set of courses to master certain skill/job/specialization.

In this work, we focus on video resources (tutorials & use cases) because they make up the majority of the Mooc-office365-training's catalog. The company is providing up-to-date content that matches changes in work environments and current trends. Unfortunately, this impacts users as they have to spend more time selecting the appropriate learning material which can lead to known problems such as information overload, distraction, disorientation, lack of motivation [START_REF] Debbah | Solving the curriculum sequencing problem with dna computing approach[END_REF], among other identified issues:

• New subscribers/visitors may have difficulty selecting the appropriate content, to begin with, depending on their needs (learn a new skill or career change).

• Watch next: After finishing viewing a video, users are not given a playlist of what to watch next. This can cause frustration and an increase in dropout rates.

• One size fits all: A lack of personalization as the same content is displayed for all users and not tailored to their specific interests.

To reduce the time spent searching for relevant information, scientific literature proposed multiple approaches that matches users with relevant content through the use of recommender systems. Unfortunately, most of the works deal with academic recommender systems which is different from corporate objectives.

In this paper, we investigate and solve problems encountered in a public e-learning platform operated by Mandarine Academy. After reviewing the literature and analysing user behavior by collecting real-world interactions (explicit and implicit), we proceed to mathematically represent company objectives and constraints into an optimization problem. In addition, we perform a critical analysis of the user interface and experience to understand its impact on a user's learning journey. Our approach combines both traditional recommender system techniques with metaheuristics in order to recommend relevant content to users. The established experimental protocol aims to compare the performance of many evolutionary algorithms and provides an in-depth analysis of their results. Finally we give insights about production mode findings and future research directions. The rest of the paper is organized as follows: Section 2 provides an overview of related works. In section 3, we discuss the findings of data analysis done on Mooc-office365-training and list different graphical issues. In section 4, we showcase our proposed approach. Section 5 provides experimentation design and results analysis. Section 6 concludes the paper and gives directions for future work.

STATE OF THE ART: RECOMMENDER SYSTEMS AND E-LEARNING

In this section we give a brief description of Multi-Objective Optimization (MOO) and common recommendation system techniques before diving into related scientific studies.

Real-world optimization problems are rarely mono objective, instead we end up with many conflicting goals. This can be defined as optimizing 𝐹 (𝑥) = (𝑓 1 (𝑥), 𝑓 2 (𝑥), ..., 𝑓 𝑛 (𝑥)) with 𝑥 ∈ 𝐹 𝑠𝑜𝑙 , 𝑛 is number of objectives (𝑛 ≥ 2), 𝑥 being a vector of decision variables, 𝐹 𝑠𝑜𝑙 a set of feasible solutions and 𝑓 𝑖 (𝑥) represents each objective that we want to minimize or maximize [START_REF] Coello | Multi-objective combinatorial optimization: Problematic and context[END_REF]. Unlike mono-objective optimization, results are not a single solution, but a Pareto set [26] of optimal solutions where no improvement can be found for an objective without degrading another objective value.

Metaheuristics represent a family of approximate optimization strategies that offer acceptable solutions to complex problems in reasonable time. Unlike exact optimization algorithms, metaheuristics do not guarantee that the results are optimal [START_REF] Hertz | Guidelines for the use of meta-heuristics in combinatorial optimization[END_REF]. Evolutionary techniques such as Genetic Algorithms (GAs) have been extensively used to solve complex optimization problems GAs were developed by J. Holland in the 1970s [START_REF] John | Genetic algorithms and the optimal allocation of trials[END_REF] to mimic the adaptive processes of natural systems.

Before we can assess "how good" any single run of a Multi-Objective Evolutionary Algorithm (MOEA), we must first grasp two concepts. The first is convergence, which indicates how "close" we come to find the best solution. The second metric is diversity which measures if solutions are fully spread throughout the set or are clustered together. The following quantitative metrics can provide an evaluation mechanism for both convergence and diversity [START_REF] Zitzler | Performance assessment of multiobjective optimizers: An analysis and review[END_REF]: Hypervolume (𝐻𝑉 ) (Maximize), Generational Distance (𝐺𝐷) (Minimize), Inverse Generational Distance (𝐼𝐺𝐷) (Minimize), Epsilon-Indicator (𝜖) (Minimize).

Recommender Systems (RS) are algorithms aimed at suggesting relevant items to users. This is made possible by filtering massive amounts of data that can be obtained from users, content, or other sources (e.g. context).

Recommender systems are becoming a part of our daily digital life. Mainly found in online entertainment services, e-commerce, and social networks [START_REF] Esteban | A Hybrid Multi-Criteria Approach Using a Genetic Algorithm for Recommending Courses to University Students[END_REF][START_REF] Esteban | Helping university students to choose elective courses by using a hybrid multi-criteria recommendation system with genetic optimization[END_REF][START_REF] Wang | Multi-objective optimization for long tail recommendation[END_REF], but other sectors are adopting this technology. From the user's perspective, recommenders reduce the time it takes to find appropriate items, increasing brand satisfaction, loyalty, and familiarity.

From the standpoint of a business owner, this provides information about what users like without requiring additional marketing/support efforts.

Generally, the most used types of recommender systems are: Collaborative Filtering (CF) and Content-Based filtering (CBF). The main difference between these two techniques is the type of data employed. The CBF approach requires items metadata to recommend content having similar attributes with the user profile [START_REF] Michael | Content-based recommendation systems[END_REF]. CF approaches on the side, leverages user ratings (explicit/implicit) to predict the likelihood of a user's liking an item. Two different approaches can be used, Model-Based (Machine/Deep learning models) and Memory Based (User/Item-based).

We found that existing recommendation algorithms (CF and CBF) do not perform well when evaluated in terms of accuracy, novelty, and diversity. Approaches that exploit the combination of such recommendation algorithms are known as Hybrid Recommenders [START_REF] Burke | Hybrid recommender systems: Survey and experiments[END_REF]. The advantage of this approach is that it limits the cons of each method used in the hybridization process all while inheriting their advantages. Other approaches that exploit the Pareto efficiency concept in order to combine such recommendation algorithms in a way that a particular objective is maximized without significantly hurting the other objectives. Recommender systems and the use of metaheuristics have been the focus of many academic researchers [START_REF] Chai | Recommendation system based on singular value decomposition and multi-objective immune optimization[END_REF][START_REF] Goldberg | Eigentaste: A constant time collaborative filtering algorithm[END_REF][START_REF] Wang | A multiobjective genetic algorithm based hybrid recommendation approach[END_REF][START_REF] Wang | Decomposition based multiobjective evolutionary algorithm for collaborative filtering recommender systems[END_REF]. We critique a few works that take a similar approach in the following section.

Xie et al. [START_REF] Xie | Personalized approximate pareto-efficient recommendation[END_REF] integrated a personalized approximate Pareto-efficient recommendation on the WeChat Top Stories section for millions of users. Their approach used reinforcement learning to find objective weights for the target user using list representation. Five online metrics (click through rate, dwell-time scores for both system and item, has-click rate and diversity) were used to evaluate models. Fortes et al. [START_REF] Silva Fortes | Individualized extreme dominance (IndED): A new preference-based method for multi-objective recommender systems[END_REF] also adopted a similar technique, relying on user preferences concerning objectives weights during both the decision-making and optimization phases.

Zuo et al. [START_REF] Zuo | Personalized recommendation based on evolutionary multi-objective optimization [research frontier[END_REF] proposed a multi-objective personalized recommendations using clustering to improve the computational efficiency. Their approach optimises both accuracy and diversity using NSGA-II algorithm [START_REF] Deb | A fast elitist non-dominated sorting genetic algorithm for multi-objective optimization: NSGA-II[END_REF]. Another work [START_REF] Jugovac | Efficient optimization of multiple recommendation quality factors according to individual user tendencies[END_REF] that collects the tendencies of users, based on their past behavior, to provide a personalized recommendations list Manuscript submitted to ACM that adhere to the defined goals. Using a greedy re-ranking technique to match items with user profiles. The use of multiple recommendations engines is developed in the work of Ribeiro et al. [START_REF] Tulio Ribeiro | Multiobjective pareto-efficient approaches for recommender systems[END_REF]. A Pareto-efficient recommendation approach optimize weights of associated engines to provide items that are accurate, novel, and diverse.

Our work addresses different aspects that are missing or under-exploited in the aforementioned works:

• The use of multiple recommendation engines to initialise our solution population and provide more diversity.

• Working with real-world implicit ratings to train our model.

• We propose a customized mutation operator to improve diversity of recommended items.

• Performance comparison of various Multi-Objective Evolutionary Algorithms (MOEA).

• Optimizing five conflicting objectives in the context of a real-world problem.

• The use of parameter tuning to optimize algorithms depending on user behaviour and selected objectives.

• Work is being integrated in production-ready environment.

UNDERSTANDING USER BEHAVIOR

The more you know about your users, the better equipped you'll be to make informed decisions about your service. In order to gain a richer understanding of how users interact with content, events are used to independently track users journeys. A typical method of providing feedback is in the form of rating methods that captures users preferences in explicit ways (like button, social sharing, course/learning path registration and bookmarks). The disadvantage is that users tend to avoid the burden of explicitly stating their preferences. To overcome the shortage of explicit ratings, platforms tend to collect users behavior through multiple ways (page views, percentage of videos watched, etc). This is called implicit feedback. The advantage is that users can trigger a lot of actions when using a service. This generates a lot of data that can be significant in some cases but shows a major inconvenient, which is not having a ground truth.

When running short on ratings (explicit or implicit), content descriptors like (subtitles, title, description, number of views, duration, etc.) are used as additional input to recommender systems.

We conduct our data analysis using the Mooc-office365-training platform (French version) with the following catalog:

• 41 Learning Paths.

• 142 Courses.

• 1294 Tutorials and 113 Use Cases.

Collected data was captured from early 2018 to late 2020. Both Table 1 and Table 2 shows available user events (explicit and implicit). Starting with column % of users which indicates the percentage of users that used the feature at-least once. The difference between explicit and implicit ratings is distinguishable. Only about 1% of users have explicitly indicated their feedback, with social sharing being the most used. When we look at implicit interactions, we see a different story, as more significant users are interacting with content.

Same behaviour applies with column % of content as explicit interactions have a smaller number of involved content compared to implicit ones. Furthermore, investigating implicit ratings reveals that approximately 7% of pages have never been visited and approximately 9% of videos have never been watched. These findings are alarming, specially when a part of the catalog is hidden from public view.

Finally, column sparsity score defines the ratio of unspecified ratings to the total number of entries in the user-item matrix and calculated as follows

𝑠𝑝𝑎𝑟𝑠𝑖𝑡𝑦 = 1 - |𝑅 | |𝑈 |𝑥 |𝐼 | .
Observations taken from Table 1 and 2 not only show a high sparsity score which is normal for real-world data but also a usage gap between both implicit and explicit interactions. Since explicit ratings are visible to users, we suppose Manuscript submitted to ACM that other reasons beside avoiding to express their opinion might be possible. To confirm our hypothesis we observed the graphical interface available for both registered users and visitors. We list below our findings per page.

(1) Home page: The current homepage offers a list of newest courses and tutorials. Users / visitors are limited if they are looking to learn about certain tools, required skills for specific jobs or certification. What we propose for visitors is a list of items (courses and resources) with options to select popular or newer items. Furthermore, categories (skills, jobs, certificates) should be shown in top of the page to guide visitors efficiently. For registered users, a multiple personalized lists of recommended items provided by our approach and others (CF, CBF) will help users find relevant content easier.

(2) Content page: Learning paths, courses and videos (tutorials and use cases) are presented to both users and visitors without similar items, visible interactions or feedback options. The like and share buttons are provided without text, only a small icon. In case a video doesn't correspond to a user's needs, they must go back to the previous page and spend additional time looking for another one. We propose for both users and visitors a more appealing interface with visible interactions (like, dislike, social share), addition of a "save to watch later (Bookmark)" and "feedback" options. Multiple recommended lists (CF, CBF, Popularity) of similar items to minimise the burden of content search and provide guidance.

Initial propositions address the way content as well as interactions are displayed and insists on improving both visibility and readability for users. One of the proposed features (Bookmark) has already been integrated into the platform and its shown in Table . 1, since it's relatively new, additional data need to be collected before assessing its significance. Furthermore, the search process is upgraded with more filters to empower users looking for specific information. Finally, we highlight recommended content by improving its graphical positioning for users [START_REF] Chen | Users' decision behavior in recommender interfaces: Impact of layout design[END_REF]. Overall, the process aims to make the platform more accessible and easy to use by reassembling certain graphical elements (video player, item sliders, and search bar) to match common online services (entertainment and e-commerce websites).

The above propositions aim to reduce cognitive overload caused by clumsy and unfamiliar browsing experiences, which users may encounter on occasion [START_REF] Waldner | Emphasize, don't filter! displaying recommendations in twitter timelines[END_REF]. Not only that but to make the exploration process easier and more productive.

Further AB testing campaigns are planned across these pages to measure the effects on user/visitor behavior and satisfaction levels.

Manuscript submitted to ACM

MARS: MANDARINE ACADEMY RECOMMENDER SYSTEM

The performance evaluation of our approach is highly dependent on the data it processes and the task it has to perform.

In our particular context, this task is complex since it must satisfy different goals that are far from being complementary.

We had to find a compromise satisfying both the need to match user taste, highlight diverse content and also focus on unpopular items. These goals represent what the company aspires to achieve. These requirements were updated with two additional objectives widely used in the literature and considered as a standard in recommender systems evaluation metrics.

(Objective 1) Maximize similarity with user profile. This is done by calculating the overall cosine score between items in user's profile and items in the proposed recommendation.

𝑃𝑠𝑖𝑚 = 𝑖=0 𝐿 𝑗=0 𝑁 𝑐𝑠𝑖𝑚(𝑟 𝑖 , 𝑢 𝑗 ) 𝑆 𝐿 ( 1
)
Where 𝐿 is the recommended list (solution) and 𝑟 𝑖 is the item number 𝑖 from 𝐿. The user profile is expressed as 𝑁 where 𝑢 𝑗 is the item number 𝑗 from 𝑁 . 𝑐𝑠𝑖𝑚 is the item-item cosine distance score. 𝑆 𝐿 is the length of the solution.

(Objective 2) Maximize diversity which is responsible for how dissimilar items are in the solution. This can be achieved by using the Intra-List Similarity metric (ILS) [START_REF] Vargas | Novelty and diversity enhancement and evaluation in recommender systems and information retrieval[END_REF]. It uses the same logic behind Objective 1 by calculating the average cosine similarity of all items in a list of recommendations. Note that this objective is conflicting with the first objective. As the first looks for similar items to the user profile, the second looks for more diversified items within the proposed list itself.

𝑅𝑑𝑖𝑣 = 𝑖=0 𝐿-1 𝑗=1 𝐿 𝑐𝑠𝑖𝑚(𝑟 𝑖 , 𝑢 𝑗 ) 𝑡𝑐 (2)
Where 𝐿 is the recommended list (solution) and 𝑟 𝑖 is the item number 𝑖 from 𝐿. 𝑐𝑠𝑖𝑚 is the item-item cosine distance matrix. 𝑡𝑐 is the item pairs count.

(Objective 3) Maximize novelty. In this objective, we aim to recommend less popular items and focus on content having both low number of views and recently added to the catalog. A scoring function that sums the number of views and number of days since release, returns the median. The smaller the median, the more novel the items are.

𝑅𝑛𝑜𝑣 = 𝑖=0 𝐿 𝑛𝑠 (𝑟 𝑖 ) 𝐿 (3)
Where 𝐿 is the recommended list (solution) and 𝑟 𝑖 is the item number 𝑖 from 𝐿. 𝑛𝑠 is novelty scoring function.

To add more flexibility to the proposed recommender system, two additional objectives were added. Root Mean Square Error 𝑅𝑀𝑆𝐸 and the Normalized Discounted Cumulative Gain 𝑛𝐷𝐶𝐺. Both are well known metric and widely used in recommender systems research field.

(Objective 4) Minimize 𝑅𝑀𝑆𝐸. What this metric does essentially is finding difference between a predicted rating and a real rating. Having lower error value, means our model is predicting ratings similar to what the user gave.

𝑟𝑚𝑠𝑒 = ( 1
𝑛 ) 𝑛 ∑ 𝑖=1 (𝑦 𝑖 -𝑥 𝑖 ) 2 (4)
Where 𝑦 𝑖 is the actual rating, 𝑥 𝑖 is the predicted rating and 𝑛 is the number of ratings.

(Objective 5) Maximize 𝑛𝐷𝐶𝐺. This is a measure of ranking quality where highly relevant items are more useful when ranked first. Also this metrics follow the assumption that highly relevant items are more useful than marginally Manuscript submitted to ACM relevant items, which are in turn more useful than non-relevant items. We will be using nDCG@5 which corresponds to the number of relevant results among the first 5 recommended items.

𝑛𝐷𝐶𝐺 𝑝 = 𝐷𝐶𝐺 𝑝 𝐼𝐷𝐶𝐺 𝑝 (5)
With 𝐼 𝐷𝐶𝐺 𝑝 = 𝑖+1) . Where 𝑅𝐸𝐿 𝑝 is a list of top 𝑝 relevant items (ordered by relevance). 𝑟𝑒𝑙 𝑖 is the graded relevance of the result at position 𝑖.

|𝑅𝐸𝐿 𝑝 | 𝑖=1 𝑟𝑒𝑙 𝑖 log 2 (𝑖+1) and 𝐷𝐶𝐺 𝑝 = 𝑝 𝑖=1 2 𝑟𝑒𝑙 𝑖 -1 log 2 (
Even though both metrics reflect relevance, 𝑛𝐷𝐶𝐺 is crucial for ranking problems while 𝑅𝑀𝑆𝐸 isn't relevant for that case. The intuition behind adding these objectives is to provide the ability for platform managers more options to tune from selecting only a single objective to combining many. For example, a platform wants to highlight on content similarity to users profile can use both (Objective 1) and (Objective 4).

Note that initial objectives don't capture learning objectives of users. This is due to available events (explicit and implicit). Work is being conducted to incorporate learning performance tracking events to better suggest items for users with specific learning goals (job requirements or mastery of certain tools).

Since we are working on an optimization problem we must define our constraints in order to determine if a solution is feasible or not. Constraints can be defined as conditions that a solution must satisfy in order to be feasible. Two constraints are considered in this work:

• Recommended list 𝐿 must be unique and contain no duplicates.

• Recommended items must not exceed the fixed length 𝐾.

To define our problem, objectives and constraints, we chose JMetalPy [START_REF] Benitez-Hidalgo | jMetalPy: A Python framework for multi-objective optimization with metaheuristics[END_REF], a python framework for solving multi and many-objective optimization problems. JMetalPy offers both parallel computing capabilities and a rich set of features such as real-time interactive visualization of the pareto front. We will be using popular multi-objective genetic algorithms found in the literature due to their proven efficiency handling complex problems [START_REF] Hafsa | A Multi-Objective Evolutionary Approach to Professional Course Timetabling: A Real-World Case Study[END_REF][START_REF] Ishibuchi | Performance comparison of NSGA-II and NSGA-III on various many-objective test problems[END_REF]. They offer a better exploration of the search space and diversified solutions:

• Non-dominated Sorting Genetic Algorithm 2 (NSGA-II) [START_REF] Deb | A fast elitist non-dominated sorting genetic algorithm for multi-objective optimization: NSGA-II[END_REF],

• Non-dominated Sorting Genetic Algorithm 3 (NSGA-III) [START_REF] Deb | An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, part I: solving problems with box constraints[END_REF],

• Indicator-based evolutionary algorithm (IBEA) [START_REF] Zitzler | Indicator-based selection in multiobjective search[END_REF],

• Multi Objective Evolutionary Algorithms by Decomposition (MOEA/D) [START_REF] Zhang | MOEA/D: A multiobjective evolutionary algorithm based on decomposition[END_REF],

• Strength Pareto evolutionary algorithm (SPEA 2). [START_REF] Zitzler | SPEA2: Improving the strength Pareto evolutionary algorithm[END_REF]

Solution encoding

Genetic algorithms begins with the choice of the chromosome encoding (solution representation). This depends on the problem in-hand. Following the works of [START_REF] Alhijawi | A collaborative filtering recommender system using genetic algorithm[END_REF][START_REF] Chai | Recommendation system based on singular value decomposition and multi-objective immune optimization[END_REF][START_REF] Soleimani Neysiani | Improve performance of association rule-based collaborative filtering recommendation systems using genetic algorithm[END_REF] we define a solution as a list of unique item identifiers. The list will have a fixed length 𝐾, contain unique elements specific and relevant for each user profile.

Initial Population

After defining a structure that represents our solutions, we proceed to generate an initial set or population of solutions as input for our approach. Generally initial populations can be produced through many approaches (random, heuristics, etc.) [START_REF] Sousa | Evaluation of different initial solution algorithms to be used in the heuristics optimization to solve the energy resource scheduling in smart grids[END_REF]. In order to generate good initial solutions that contain relevant results for each user we took advantage Manuscript submitted to ACM of available data (interactions and content descriptors) to create multiple recommendation engines. The following approaches are used:

• Random.

• Content Based Filtering (CBF).

• Collaborative Filtering (CF) -Item Based.

• Collaborative Filtering (CF) -User Based.

• Collaborative Filtering (CF) -Model Based (SVD++).

• Collaborative Filtering (CF) -Model Based (ALS).

• Association Rules (FP-Growth).

We try to get a recommendation from each approach listed above for each user we test. In case an algorithm is unable to provide personalized recommendations, we have defined a fallback method, which is the "Random" approach that provides a list of randomly selected items. To implement most of these algorithms, the python library "Surprise" [START_REF] Hug | Surprise: A Python library for recommender systems[END_REF] was used. This library gives access to baseline algorithms, neighborhood methods, matrix factorization-based approaches and various similarity scores (Cosine, Mean Squared Difference, Pearson, etc) and evaluation metrics (𝑅𝑀𝑆𝐸, Fraction of Concordant Pairs (𝐹𝐶𝑃), etc).

Crossover

Genetic algorithms usually applies a crossover operation to two solutions in order to produce new chromosomes (solutions) called "children chromosomes". Operators like One-Point and Two-Point crossover are popular in the literature [START_REF] Alhijawi | A collaborative filtering recommender system using genetic algorithm[END_REF][START_REF] Chai | Recommendation system based on singular value decomposition and multi-objective immune optimization[END_REF][START_REF] Huang | Recommender engine for continuous-time quantum Monte Carlo methods[END_REF][START_REF] Soleimani Neysiani | Improve performance of association rule-based collaborative filtering recommendation systems using genetic algorithm[END_REF]. The idea behind such operators is simple, chose a random one or two items from parent 1 and interchange their places with parent 2. These operators are responsible for the order of which elements are shown.

Despite its simplicity, such operators can render a solution invalid. Consider that items from parent 2 were placed in parent 1. These unique items in parent 2 can be redundant in parent 1, thus applying a repairing mechanism is a must. This repair function changes the redundant items in one of the parent solutions by using the random approach and validate the solution.

Mutation

Mutation is a genetic operation used to maintain genetic diversity. It introduces changes inside solutions in an attempt to avoid local minima. Random mutation is frequently found in the literature [START_REF] Soleimani Neysiani | Improve performance of association rule-based collaborative filtering recommendation systems using genetic algorithm[END_REF], [START_REF] Chai | Recommendation system based on singular value decomposition and multi-objective immune optimization[END_REF] along with 1-point mutation [START_REF] Alhijawi | A collaborative filtering recommender system using genetic algorithm[END_REF], 2-point mutation [START_REF] Wang | A multiobjective genetic algorithm based hybrid recommendation approach[END_REF] and Uniform mutation [START_REF] Alhijawi | A collaborative filtering recommender system using genetic algorithm[END_REF]. We propose a custom mutation operator named 𝑀𝐴𝑅𝑆 𝑚𝑜 to be compared to classical operators. The concept behind 𝑀𝐴𝑅𝑆 𝑚𝑜 is to choose 𝑁 , with (1 ≤ 𝑁 ≤ 𝑘/2), elements from a candidate solution. The selected elements are then randomly swapped with either (1) Similar items (using Content-Based or Item-Based approaches), ( 2) Random (3) Novel (Recently added to the catalog) items. The full pseudo code is demonstrated in 1.

To make use of initial population generators, platform managers will be able to create these models using an administration dashboard. We identified a total of three model types each with different associated algorithms:

• Popularity: Provide popular content based on either view portions (default) or number of page visits.

• Personalized: Based upon one method (User Based CF, Item-Based CF, Content-Based (default), Model-Based CF, Metaheuristic, FP-Growth).

• Similar Items: Based upon one method (Item-Based CF, Content Based (default), FP-Growth). This dashboard is still in work and will also provide monitoring features to track performance of each model in real-time by using metrics such as click-rate (𝐶𝑅), watch time and click through rate (𝐶𝑇 𝑅).

Manuscript submitted to ACM

EXPERIMENTS AND TESTS

Along with the implementation of the metaheuristic and different solution generators, we conducted a series of tests in a more experimental setting in order to compare different approaches for solving the recommendation problem with different objectives combinations.

Technically, platform managers can select any combination of objectives they desire. However, in our test settings we tested a subset of possible combinations to observe the impact of having multiple objectives on performance.

Each test setting has a parameter tuning phase to ensure each algorithm is using the best configuration for the task.

The data in this experiment was collected from real-world users and were adapted to our approach.

Dataset

Moving on to initial experiments where the first step was selecting the right data to work with. We have seen that explicit ratings in Table 1 suffer from low user engagement compared to implicit ratings in Table .2. Unfortunately for page views, there isn't a ground truth that indicates if viewing a content page multiple times leads to increased user satisfaction. However, view portions (Watch time) might be a good fit for our approach as it can be used to measure user interest. Table . 3 details the different attributes and values present in view portions.

The dataset has a sparsity of 99.312%. With a total of 822 users, 776 items, and 3699 ratings. The company uses the following scale to describe viewing events:

• (1) Not considered: viewings from 0% to 10% of the video.

• (2) In progress: viewings from 11% to 69% are considered equal.

Manuscript submitted to ACM • (3) Finished: viewings from 70% to 100% consider the user has finished watching the item.

We believe that the degree of viewing has an impact on the overall impression and that the previous scoring function does not reflect user interest. When plotting the old scoring scale shown in Fig. 1, the majority of users are in the "In progress" state followed by "Finished". This implies that users are still learning or have completed their videos. A new scoring system is proposed. It builds on the previous approach by incorporating more levels of appreciation. The assumption is that longer viewing times indicate a higher user interest and satisfaction. Considering the following scale:

• (1) No interest: viewings from 0% to 20% of the video.

• (2) Small interest: viewings from 21% to 40% of the video.

• (3) Medium interest: viewings from 41% to 60% of the video.

• (4) High interest: viewings from 61% to 80% of the video.

• (5) Finished: viewings from 81% to 100% of the video.

This introduces 5 different levels that have varying degrees of importance and reassembles the classical 5-star rating system. When plotting the new scoring system, a different narrative is found. The majority of users fall into the "No interest" category, followed by "Finished" and "Small interest". Different assumptions are made here, starting with the highest group count "No interest" which indicates that users watched a maximum of 2 seconds on a scale of 10. This can be interpreted as users stumbling into wrong content and returning to search for something more suitable. Perhaps the title wasn't clean enough since descriptions aren't always provided, or the video content was advanced for user's skills.

Additional insights are gathered after further dataset analysis. When grouping seen elements per user we obtained an average of 5 items. With 80% of users have seen less than or equal to the average. This might indicate that most users have abandoned their learning path or are unable to locate appropriate content. We only consider videos with over 50% of watch time in users profile. The assumption is that longer watch times indicate a higher user interest and satisfaction.

Parameter Tuning

Instead of choosing fixed parameters empirically and applying them to all algorithms indifferently, another protocol is used to provide a fair performance comparison. It uses the irace package [START_REF] López-Ibánez | The irace package: Iterated racing for automatic algorithm configuration[END_REF] which implements an iterated racing approach to automatically find optimal settings. The library focus on improving optimization algorithms and machine learning models. For each parameter a list with possible values is shown in Table 4. A fixed computing time limit of one hour is defined as a stopping criteria for our experiment. Concerning results, 𝐾 was set to 10 items (max number of recommended items).

Manuscript submitted to ACM Elite configurations are returned based on their average best Hypervolume (𝐻𝑉 ) metric [START_REF] Zitzler | Performance assessment of multiobjective optimizers: An analysis and review[END_REF] across different test instances. The 𝐻𝑉 metric is capable of measuring both convergence and diversity of our solutions. The higher 𝐻𝑉 value, the better our solutions are. Note that we will be comparing the proposed custom mutation operator 𝑀𝐴𝑅𝑆 𝑚𝑜 with the classical 𝑆𝑤𝑎𝑝 mutation.

Our first test experiment will focus on the three initial objectives (𝑃𝑠𝑖𝑚, 𝑅𝑑𝑖𝑣, 𝑅𝑛𝑜𝑣) which were proposed initially by the company. The second test experiment will add both 𝑅𝑀𝑆𝐸 and 𝑛𝐷𝐶𝐺@5 to the other three objectives. This will make it a many-objective optimization task and will test how the parameters and performance adapts. Note that both 𝑅𝑀𝑆𝐸 and 𝑛𝐷𝐶𝐺@5 requires a portion of user history to validate predictions. Since 80% of users have less than or equal to 5 items in their history we will be using the remaining 20% of users as they have more watched items. Both test experiments are samples of the possible choices that platform managers can select. For example another setting can be focused on relevance by having three different objectives (𝑃𝑠𝑖𝑚, 𝑅𝑀𝑆𝐸 and 𝑛𝐷𝐶𝐺@𝐾).

Starting with Table 5 which shows the elites configurations provided by irace for each algorithm (NSGA2, NSGA3, SPEA2, MOEA/D, and IBEA) using implicit interactions for three objectives (𝑃𝑠𝑖𝑚, 𝑅𝑑𝑖𝑣, 𝑅𝑛𝑜𝑣). Observations indicate that 1 -𝑃𝑜𝑖𝑛𝑡 crossover operator has been chosen over the 2 -𝑃𝑜𝑖𝑛𝑡 crossover operator by the majority of algorithms.

This can be explained by the fact that the crossover operator in this test setting does not impact objectives performance, so selecting a simpler operator could be the reason. Only 𝑀𝑂𝐸𝐴/𝐷 chose 𝑀𝐴𝑅𝑆 𝑚𝑜 as the mutation operator, while the rest of the algorithms used the random mutation operator. This can be attributed to a variety of factors, including the allowed computing time and length of solutions 𝐾 aside from the number of objectives. When looking at Table 6 for the many-objectives irace runs, most elite configurations have chosen the 2 -𝑃𝑜𝑖𝑛𝑡 crossover operator over 1 -𝑃𝑜𝑖𝑛𝑡.

This confirms our previous assumption, that crossover operators, are chosen depending on their role in improving the objectives. Since the additional objectives in this experiment have an interest in item ordering, a change in elite configuration was anticipated. Similarly, all algorithms selected 𝑀𝐴𝑅𝑆 𝑚𝑜 as a mutation operator, indicating that this operator has superior performance, particularly in complex settings.

Results and Performance Analysis

The following experiments are based on the results provided by irace. For each algorithm, 30 independent executions are launched using the same settings as seen in parameter tuning: stopping criteria of one hour, objectives (3𝑂𝐵𝐽 &5𝑂𝐵𝐽 )

and 𝐾 = 10 recommended items). We used these metrics discussed in Section 2 to measure the quality of our solutions:

𝐻𝑉 , 𝐺𝐷, 𝐼𝐺𝐷 and 𝜖. However, the latter metrics do not guarantee better recommendation results for the end user, for

Manuscript submitted to ACM continued to perform marginally better in which make them more fit for our future experiments.

Moving on to discuss the evolution of the hypervolume 𝐻𝑉 indicator for each algorithm. We start with 3𝑂𝐵𝐽 performance charts shown in Fig. 3a. Each algorithm 𝐼𝐵𝐸𝐴, 𝑀𝑂𝐸𝐴𝐷, 𝑁 𝑆𝐺𝐴3, 𝑁 𝑆𝐺𝐴2, and 𝑆𝑃𝐸𝐴 has its respective colors (Red, Grey, Blue, Black, Green). Same as our findings in Table 7, 𝑁 𝑆𝐺𝐴3, 𝐼𝐵𝐸𝐴, and 𝑆𝑃𝐸𝐴2 are in the lead when looking at the end of the graph. 𝑁 𝑆𝐺𝐴3 was able to maintain its superiority from the beginning, while both 𝐼𝐵𝐸𝐴 and 𝑆𝑃𝐸𝐴2 lacked behind in the first third of the experiments. This behavior changes when looking at 5𝑂𝐵𝐽 graph in Fig. 3b. 𝑆𝑃𝐸𝐴2 keeps the lead from the start compared to other algorithms. From both experiments, we can see that most algorithms are not improving at the same level as the beginning of the experiments. This indicates a stagnation state and algorithms aren't likely to improve considerably.

Considering production scenarios and after examining Fig. 3 Selecting the best solution can be complicated specially since these objectives are conflicting with each other. Multi-Criteria Decision Making (MCDM) deals with such decision problems. Among many methods we selected Pseudo Weights (PS) which calculates the normalized distance to the worst solution regarding each objective 𝑖 [START_REF] Kalyanmoy | Multi-objective optimisation using evolutionary algorithms: an introduction[END_REF]. The following equation provides the pseudo weight 𝑤 𝑖 for the 𝑖-ith objective.

𝑤 𝑖 = (f 𝑚𝑎𝑥 𝑖 -𝑓 𝑖 (𝑥))/(f 𝑚𝑎𝑥 𝑖 -f 𝑚𝑖𝑛 𝑖 ) 𝑀 𝑚=1 (f 𝑚𝑎𝑥 𝑚 -𝑓 𝑚 (𝑥))/(f 𝑚𝑎𝑥 𝑚 -f 𝑚𝑖𝑛 𝑚 ) (6)
The steps are rather simple, first we get nadir (ideal) points from Pareto front. We proceed to calculate the normalized distance to the worst solution for each objective 𝑤 𝑖 . Finally we find the closest solution to the normalized distance. Other methods aside from Pseudo Weights can be used, such as the high trade-off method [START_REF] Weck | Multiobjective optimization: History and promise[END_REF] and Compromise Programming [START_REF] Lindahl | Strategic, Tactical and Operational University Timetabling[END_REF].

Platform managers are provided a graphical interface to either selected already defined profiles (balanced objectives, high relevance, etc) or tune the weights assigned to each selected objective using a simple slider. The end-user will receive the first solution returned after the MCDM phase.

CONCLUSION AND FUTURE WORKS

In this article, we solve a many-objective recommendation problem at Mandarine Academy. The approach is applied on e-learning platforms and took advantage of real-world interactions to better understand user behavior and identify key points for user interface/experience improvement.

Studying related works and following company guidelines, we mathematically formulated our objectives into a Multi-Objective Combinatorial Optimization Problem (MOCOP). With five objectives (𝑆𝑖𝑚𝑖𝑙𝑎𝑟𝑖𝑡𝑦, 𝐷𝑖𝑣𝑒𝑟𝑠𝑖𝑡𝑦, 𝑁𝑜𝑣𝑒𝑙𝑡𝑦, 𝑅𝑀𝑆𝐸, and 𝑛𝐷𝐶𝐺@5). Our proposed approach focuses on personalization of recommendations by providing each user

Manuscript submitted to ACM the items that match their profile and ratings while emphasizing on novelty and diversity. By evaluating different evolutionary algorithms using real-world user data we were able to find best performing approaches considering different test settings.

Existing users may benefit from models that generates diversified or novel items to further explore the catalog, while new users may receive recommendations created by emphasizing ratings and ranking. The freedom of selecting what goals to prioritize is provided for platform managers in a user-friendly interface. Future graphical improvements insists on the same principles of readability, ease of use and availability of interactions.

The most time-consuming part of our approach is performed entirely offline, where chosen objectives are trained on user data and served online through Application Programming Interfaces (APIs). One major drawback is the scalability Manuscript submitted to ACM of the proposed system when the user base and catalog are expanding, training times can be significantly affected.

Also, exploring the possibility for users to indicate their objective preferences which will be taken into account when updating the model is considered in future work.

The integration of Mandarine Academy Recommender System 𝑀𝐴𝑅𝑆 is currently underway and will include an administration dashboard specific to each platform owner for managing 𝑀𝐴𝑅𝑆 and monitoring performance.

( a )Fig. 1 .

 a1 Fig. 1. Count of implicit interactions (View Portions) per user.

Fig. 2 .

 2 Fig. 2. Box-plot of 𝐻𝑉 indicator for both 3𝑂𝐵𝐽 and 5𝑂𝐵𝐽 experiments using all algorithms (30 Executions).

Fig. 3 .

 3 Fig. 3. Evolution of the 𝐻𝑉 indicator (Y-axis) for both 3𝑂𝐵𝐽 and 5𝑂𝐵𝐽 over 1 hour of computing time (X-axis) using all algorithms (30 Executions).

Table 1 .

 1 Explicit interactions captured from Mooc-Office365 (French) starting 2018 to late 2020.

	Interaction	% of users % of content Number of entries Sparsity
	Likes/Dislikes	0.02%	0.9%	28	0.830%
	Social Shares	0.66%	58.11%	2179	0.997%
	Learning Path/ Course Registration	0.439%	40%	1202	0.996%
	Bookmarks	-	-	-	-

Table 2 .

 2 Implicit interactions captured from Mooc-Office365 (French) starting 2018 to late 2020.

	Interaction	% of users % of content Number of entries Sparsity
	Page View	21.86%	93.08%	610,956	0.985%
	View Portion (Resources)	8.26%	91.57%	68,894	0.993%

Table 3 .

 3 Algorithm 1 Pseudo-code of 𝑀𝐴𝑅𝑆 custom mutation operator 𝑀𝐴𝑅𝑆 𝑚𝑜 Implicit Interactions (View Portions) from Mooc-office-365 (French) starting 2018 to late 2020.

	Require: 𝐷𝑒 𝑓 𝑖𝑛𝑒𝑑 𝑃𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦, 𝐿
	𝑅𝑒𝑝𝑙𝑎𝑐𝑒𝑚𝑒𝑛𝑡 𝑀𝑒𝑡ℎ𝑜𝑑 ← 𝑟𝑎𝑛𝑑𝑜𝑚 𝑚𝑒𝑡ℎ𝑜𝑑 ∈ (𝑆𝑖𝑚𝑖𝑙𝑎𝑟𝑖𝑡𝑦, 𝑅𝑎𝑛𝑑𝑜𝑚, 𝑁𝑜𝑣𝑒𝑙𝑡𝑦)
	𝑅𝑒𝑝𝑙𝑎𝑐𝑒𝑑 𝐼𝑡𝑒𝑚𝑠 ← 𝑟𝑎𝑛𝑑𝑜𝑚 𝑖𝑡𝑒𝑚𝑠 ∈ 𝐿
	𝑀𝑢𝑡𝑎𝑡𝑖𝑜𝑛 𝑃𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 ← 𝑟𝑎𝑛𝑑𝑜𝑚 ∈ [0.0, 1.0]
	if 𝑀𝑢𝑡𝑎𝑡𝑖𝑜𝑛 𝑃𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 ≤ 𝐷𝑒 𝑓 𝑖𝑛𝑒𝑑 𝑃𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 then
	for each Item ∈ Replaced Items do
	𝐼𝑡𝑒𝑚 ← 𝑅𝑒𝑝𝑙𝑎𝑐𝑒𝑚𝑒𝑛𝑡 𝑀𝑒𝑡ℎ𝑜𝑑 (𝐼𝑡𝑒𝑚)
	end if
	User ID Item ID View Portion	State	Date
	792	995	7%	Not considered 2018-01-21
	3475	516	30%	In progress	2018-05-25
	...	...	...	...	...
	687	520	80%	Finished	2020-08-14
	542	498	45%	In progress	2020-09-16

Additional advanced settings can be fine tuned for example, selecting objectives associated to each model and what interactions to use. This makes it possible to experiment with different parameters across different interface placements.

Table 4 .

 4 Parameters settings considered for tuning phase.

	Parameter	Description	Possible Values
	𝑃𝑜𝑝	Population size at each generation.	10, 50, 100, 200, 500, 1000
	𝐶𝑥	Crossover Genetic Operator	1-Point, 2-Point
	𝐶𝑥 𝑝	Crossover Probability	0.1 -1.0
	𝑀𝑥	Mutation Genetic Operator	Random, 𝑀𝐴𝑅𝑆 𝑚𝑜
	𝑀𝑥 𝑝	Mutation Probability	0.1 -1.0
	𝐾𝑝	Kappa (IBEA)	0.1 -1.0
	𝑁 𝑆𝑃	Neighbourhood Selection Probability (MOEAD) 0.1 -1.0
	𝑀𝑁 𝑅𝑆	Max Number of Replaced Solutions (MOEAD) 10, 50, 100, 200, 500, 1000
	𝑁 𝑆	Neighbor Size (MOEAD)	10, 50, 100, 200, 500, 1000

Table 5 .

 5 Elites configurations provided by i-race using 3 objectives on implicit dataset.

	Parameter NSGA2 NSGA3	SPEA2	MOEAD	IBEA
	𝑃𝑜𝑝	10	10	10	500	10
	𝐶𝑥	1-Point	1-Point	1-Point	2-Point	1-Point
	𝐶𝑥 𝑝	0.3	1.0	0.9	0.7	0.1
	𝑀𝑥	Random Random Random 𝑀𝐴𝑅𝑆 𝑚𝑜 Random
	𝑀𝑥 𝑝	1.0	0.6	1.0	0.9	0.9
	𝐾𝑝	-	-	-	-	0.2
	𝑁 𝑆𝑃	-	-	-	1.0	-
	𝑀𝑁 𝑅𝑆	-	-	-	1000	-
	𝑁 𝑆	-	-	-	500	-

Table 6 .

 6 Elites configurations provided by i-race using 5 objectives on implicit dataset.Starting with 𝐻𝑉 3𝑂𝐵𝐽 column found in Table7, results show that 𝑁 𝑆𝐺𝐴3 has a maximum score of (0.91). In second place 𝐼𝐵𝐸𝐴 followed by 𝑆𝑃𝐸𝐴2 with a score of (0.85) and (0.80) respectfully. The previous findings are compared with the results of 5 Objectives experiments. Since objectives 𝑃𝑠𝑖𝑚, 𝑅𝑑𝑖𝑣 and 𝑅𝑛𝑜𝑣 are already included, we aggregate their values. This is indicated by 𝐻𝑉 3𝑂𝐵𝐽 column in Table8. Both 𝑆𝑃𝐸𝐴2 (0.85) and 𝑁 𝑆𝐺𝐴3 (0.84) kept a robust performance with 𝑁 𝑆𝐺𝐴2 (0.84) outperforming its previous 𝐻𝑉 score. 𝐼𝐵𝐸𝐴 didn't perform well compared to the 𝐻𝑉 3𝑂𝐵𝐽 experiment. Shifting our focus to 5 Objectives results shown in 𝐻𝑉 5𝑂𝐵𝐽 column. 𝑆𝑃𝐸𝐴2 and 𝑁 𝑆𝐺𝐴3 achieved good scores of (0.81) and (0.79) respectively.Taking into account other performance indicators (𝐺𝐷, 𝐼𝐺𝐷, 𝜖) which must be minimized and starting with 𝐺𝐷 column shown in Table7. Most algorithms obtained similar scores with 𝐼𝐵𝐸𝐴 achieving the lowest score (1.22). However, looking at 𝐼𝐺𝐷 column, 𝑆𝑃𝐸𝐴2 was able to obtain a value of (1.002) and creating a gap with the rest of algorithms.Note however that both 𝐺𝐷 and 𝐼𝐺𝐷 are easier metrics to meet compared to 𝜖. For which, 𝐼𝐵𝐸𝐴 was able to achieve the lowest 𝜖 score with (0.093) followed by both 𝑁 𝑆𝐺𝐴3 and 𝑆𝑃𝐸𝐴2. Discussing the same metrics for 5𝑂𝐵𝐽 experiments shown in Table8. 𝑁 𝑆𝐺𝐴2 obtained the lowest 𝐺𝐷 score (1.41) not far from other algorithms. Surprisingly thought, 𝑁 𝑆𝐺𝐴2 was also able to obtain the lowest 𝐼𝐺𝐷 score of (1.06) followed by 𝑁 𝑆𝐺𝐴3. When considering the 𝜖 column, 𝑆𝑃𝐸𝐴2 achieved a performance similar to 𝑁 𝑆𝐺𝐴2 with scores of (0.06) and (0.07) respectfully. Setting aside the 𝐻𝑉 5𝑂𝐵𝐽 Manuscript submitted to ACM results, 𝑁 𝑆𝐺𝐴2 have shown good results considering the many-objective problem setting. But, 𝑆𝑃𝐸𝐴2 and 𝑁 𝑆𝐺𝐴3

	Parameter NSGA2	NSGA3	SPEA2	MOEAD	IBEA
	𝑃𝑜𝑝	10	10	10	100	100
	𝐶𝑥	1-Point	2-Point	2-Point	2-Point	2-Point
	𝐶𝑥 𝑝	0.1	0.1	0.6	0.3	0.6
	𝑀𝑥	𝑀𝐴𝑅𝑆 𝑚𝑜 𝑀𝐴𝑅𝑆 𝑚𝑜 𝑀𝐴𝑅𝑆 𝑚𝑜 𝑀𝐴𝑅𝑆 𝑚𝑜 𝑀𝐴𝑅𝑆 𝑚𝑜
	𝑀𝑥 𝑝	1.0	0.8	1.0	1.0	0.9
	𝐾𝑝	-	-	-	-	1.0
	𝑁 𝑆𝑃	-	-	-	0.8	-
	𝑀𝑁 𝑅𝑆	-	-	-	500	-
	𝑁 𝑆	-	-	-	100	-
	this we will be comparing the recommended items after the Multi-Criteria Decision Making (MCDM) phase but as
	we seen in Section 4, metrics like the click-rate (𝐶𝑅), watch time and click through rate (𝐶𝑇 𝑅) will be implemented to
	measure how users are handling recommended items.

  findings show that around five minutes of computing time, good results are obtained. While indeed most approaches continue to improve after that period of time but it doesn't justify the additional computing. The company can quickly update the recommendations for users. However, one major drawback of our approach is the fact that modeling solutions as lists of recommendations for each user does not guarantee that our model will always converge in five minutes with the increase of users or items in the catalog. This scalability issue can be improved by clustering users and providing recommendations per groups instead of individual users, this is considered in future work. At last, these initial findings indicate that for both experiments

(3𝑂𝐵𝐽 & 5𝑂𝐵𝐽 ), two algorithms 𝑁 𝑆𝐺𝐴3 and 𝑆𝑃𝐸𝐴2 possess a robust performance and are the best fit for this task compared to the rest.

Table 7 .

 7 Performance comparison (average best value and the standard deviation) using implicit interactions and 3 Objectives for over 30 independent runs.

	Algorithm 𝐻𝑉 3𝑂𝐵𝐽	𝐺𝐷	𝐼𝐺𝐷	𝐸𝑃
	NSGA2	0.77 0.05 1.23 0.002 1.019 0.016 0.147 0.024
	NSGA3	0.91 0.04 1.24 0.001 1.018 0.018 0.101 0.025
	SPEA2	0.80 0.03 1.24 0.0007 1.002 0.035 0.146 0.025
	MOEAD	0.68 0.07 1.23 0.013 1.076 0.059 0.233 0.042
	IBEA	0.85 0.06 1.22 0.014 1.070 0.021 0.093 0.037

Table 8 .

 8 Performance comparison (average best value and the standard deviation) using implicit interactions and 5 Objectives for over 30 independent runs. .05 0.84 0.034 1.41 0.003 1.06 0.03 0.07 0.04 NSGA3 0.79 0.03 0.84 0.037 1.44 0.0006 1.07 0.03 0.10 0.08 SPEA2 0.81 0.06 0.85 0.026 1.43 0.004 1.12 0.02 0.06 0.01 MOEAD 0.47 0.02 0.57 0.029 1.45 0.03 1.31 0.07 0.29 0.02 IBEA 0.68 0.04 0.74 0.018 1.46 0.01 1.17 0.01 0.15 0.0008

	Algorithm 𝐻𝑉 5𝑂𝐵𝐽	𝐻𝑉 3𝑂𝐵𝐽	𝐺𝐷	𝐼𝐺𝐷	𝐸𝑃
	NSGA2	0.74 0

(a) 𝐻𝑉 3𝑂𝐵 𝐽 (b) 𝐻𝑉 5𝑂𝐵 𝐽

https://mooc.office365-training.com/ Manuscript submitted to ACM


Robert Blasiak
email: robert.blasiak@su.se

Jean-Baptiste Jouffray

Diva J Amon

Joachim Claudet

Paul Dunshirn

Peter Søgaard Jørgensen

Agnes Pranindita

Colette C C Wabnitz

Erik Zhivkoplias

Henrik Österblom

Making marine biotechnology work for people and nature

Despite this great potential, marine biotechnology has been almost exclusively driven by highly-industrialized countries and remains misaligned with blue economy principles of equity and inclusivity [START_REF] Blasiak | Corporate control and global governance of marine genetic resources[END_REF] . Efforts to eliminate such inequities have relied on protracted international negotiations that have yielded mixed outcomes, not least due to rapid advances in biotechnology that dramatically outpace the development of appropriate regulatory frameworks [START_REF] Wynberg | Fast Science and Sluggish Policy: The Herculean Task of Regulating Biodiscovery[END_REF] . Here we argue that a singular focus on regulatory solutions could result in the marine biotechnology industry remaining misaligned with the aspirations of a blue economy, and that a more holistic approach for people and nature is crucial.

Complexities and tradeoffs in marine biotechnology

Marine biotechnology has frequently resulted in innovations that can contribute to achieving sustainability goals, for example improvements in aquaculture production. Genetically-engineered salmon grows twice as quickly and can thrive in near-freezing conditions due to insertion of genes from two other fish species [START_REF] Blasiak | The ocean genome and future prospects for conservation and equity[END_REF] . While such advances are improving food yield, they can also encourage over-reliance on monocultures, and spatial expansion of salmon pens into more coastal areas, potentially harming local and Indigenous communities who rely on integrity of wild salmon populations [START_REF] Blasiak | The ocean genome and future prospects for conservation and equity[END_REF] . Another example of the complex trade-offs arising from marine biotechnology is the recent development of transgenic canola plants with genes from a variety of marine and freshwater algae [START_REF] Stuchtey | Ocean solutions that benefit people, nature and the economy[END_REF] . Transgenic canola produces high levels of omega-3 fatty acids and could become a key agrofeed ingredient, potentially reducing the need for fishmeal production and relieving fishing pressure in low-income coastal regions, where fish is nutritionally vital for local communities. Yet a trade-off would remain if increased demand results in further conversion of land for monoculture canola production.

A regulatory landscape struggling to keep up

Substantial effort has been focused on regulatory instruments to address sustainability and equity issues. Perhaps the most significant milestone for the biotechnology community was the entry into force of the Nagoya Protocol in 2014, intended to eliminate inequitable and unethical practices. These include biopiracy, which involves the appropriation of genetic resources (and often associated traditional knowledge) from Indigenous peoples and local communities and subsequent commercialization without sharing of benefits. The Nagoya Protocol established a framework for "source" and "user" countries to regulate access to genetic resources and subsequent benefit sharing according to mutually agreed terms. The Nagoya Protocol, however, follows an overall pattern of policymaking being far outpaced by scientific and technological advances [START_REF] Blasiak | Corporate control and global governance of marine genetic resources[END_REF] . During the 12 years that it was being negotiated, for instance, the first synthetic life form was created, the CRISPR gene editing technique was introduced, and cultivation of genetically modified organisms had spread to over 10% of the world's farmland [START_REF] Blasiak | Corporate control and global governance of marine genetic resources[END_REF] . As a tool most effective at regulating the movement of physical samples across national boundaries, the Nagoya Protocol came into effect at the same time that the industry was growing less reliant on physical samples and increasingly working directly with genetic sequence data. Today, the industry applies a growing suite of bioinformatics and omics technologies to analyze vast databases such as the GenBank Sequence Read Archive. Since 1982, this database has been doubling in size roughly every 18 months 10 as a result of the average cost of sequencing a base pair of DNA falling by six orders of magnitude within two decades from over USD 6,000 in 2001 to less than USD 0.01 in 2020 [START_REF] Blasiak | The ocean genome and future prospects for conservation and equity[END_REF] . Multiple negotiations are currently underway to better regulate access, use and transparency requirements associated with marine genetic resources and genetic sequence data, including in areas beyond national jurisdiction [START_REF] Blasiak | Scientists Should Disclose Origin in Marine Gene Patents[END_REF] . While progressive regulatory frameworks can help to clarify and level the playing field for all, sluggish regulatory responses can result in a widening gap between those with the capacity to engage in marine biotechnology and everyone else.

Bringing marine biotechnology into the blue economy

We contend that successfully transforming marine biotechnology into an element of the blue economy will depend on coordinated actions by diverse actors, including scientists, local communities, and companies. We suggest four interrelated pathways to accelerate this transformation.

Strengthen capacity in lower income countries

An aspiration of the blue economy is that it can drive greater equity and inclusivity in the ocean economy [START_REF] Stuchtey | Ocean solutions that benefit people, nature and the economy[END_REF] . However, just ten countries account for 98% of filed patent sequences from marine life [START_REF] Blasiak | Corporate control and global governance of marine genetic resources[END_REF] . This relates to capacity limitations, which are a particular barrier in the case of the most lucrative biotechnology products. For example, it costs an estimated USD 1 billion [START_REF] Blasiak | The ocean genome and future prospects for conservation and equity[END_REF] to bring a new drug from development to market, and all marine drugs that have been brought to market were developed by companies in Europe, Japan and North America. For low and middle-income countries to fully benefit from marine biotechnology, efforts at multiple levels are needed to develop capacity and close resource gaps. First, research groups can contribute with efforts to strengthen human and technical capacity, and to provide research and product development infrastructure that builds lasting ability to develop solutions and foster change that aligns with national interests and priorities. Second, the handful of companies driving innovations in the marine biotechnology sector [START_REF] Blasiak | Corporate control and global governance of marine genetic resources[END_REF] , which are currently benefitting most from the ocean's genetic resources, should play a similarly disproportionate role in ensuring that global sustainability goals and ocean equity is achieved by advancing capacity building and transfer of marine technology. Third, providers of development finance and philanthropies should dedicate more resources to drive equitable outcomes, noting that SDG14 ("Life Below Water"), which includes Target 14.8 on increasing scientific knowledge, research and technology, receives the least development funding of any of the Goals 10 . Finally, the international community should elevate capacity building and the transfer of marine technology within the UN Decade of Ocean Science for Sustainable Development 2021-2030 as well as the negotiations on a treaty for biodiversity in areas beyond national jurisdiction to generate further institutional support.

Equitable and inclusive research collaborations

Although industry is seen as the main commercial driver of marine biotechnology, academic institutions are central players [START_REF] Blasiak | Scientists Should Disclose Origin in Marine Gene Patents[END_REF] . Not only are they driving the exponential growth of databases like GenBank by depositing sequence data from research expeditions, but many universities also own and operate their own commercialization centers -private companies established to monetize university research [START_REF] Blasiak | Scientists Should Disclose Origin in Marine Gene Patents[END_REF] . Nearly one-third of patent applications associated with marine genetic resources have been filed by universities or their commercialization centers [START_REF] Blasiak | Scientists Should Disclose Origin in Marine Gene Patents[END_REF] . The transboundary nature of many marine science topics has spurred international collaboration, but may be strengthening imbalances rather than challenging them [START_REF] Tolochko | Selective world-building: Collaboration and regional specificities in the marine biodiversity field[END_REF] unless these collaborations ensure representative inclusion, rely on working openly and in a transparent manner with a broad range of stakeholders in developing solutions, and ensure that stakeholders from low and middle income countries have an active voice and role in study design [START_REF] Faure | Mapping experiences and perspectives of equity in international health collaborations: a scoping review[END_REF] . More explicit requirements from research funders and scientific journals to disclose sample origin and acknowledge collaborators may help shift existing norms [START_REF] Anon | Nature addresses helicopter research and ethics dumping[END_REF] toward more equitable collaborations.

Commit to responsible data sharing

While a broad landscape of environmental and genetic sequence databases already exists, interoperability and access issues limit the potential for diverse groups to fully utilize these resources. The UN Decade of Ocean Science for Sustainable Development aims to address this issue, and acknowledges that data infrastructures need to be co-designed with stakeholders to achieve the desired social and political impact [START_REF] Muller-Karger | Marine Life 2030: building global knowledge of marine life for local action in the Ocean Decade[END_REF] . Local authorities as well as private companies can play crucial roles in supporting the flow of data through targeted partnerships and investment focused on ensuring data accessibility and development of technical capacity. Protracted and unresolved negotiations under the auspices of the World Trade Organization have focused on new regulatory obligations to disclose the origin of genetic samples being commercialized, which would add a layer of transparency and accountability to such activities. Scientists involved in filing marine biotechnology patents could advance best practices by disclosing origin of marine genetic resources throughout all academic and commercial activities [START_REF] Blasiak | Scientists Should Disclose Origin in Marine Gene Patents[END_REF][START_REF] Blasiak | International regulatory changes poised to reshape access to marine genes[END_REF] . While commitments to sharing marine genetic sequence and origin data are crucial for transparency and can facilitate access and engagement by as broad a set of constituents as possible, there is a tension between such commitments and the protection of Indigenous rights and data sovereignty. In such cases, attention should be paid to the International Indigenous Data Sovereignty Interest Group who developed the 'CARE Principles for Indigenous Data Governance' [START_REF] Carroll | The CARE Principles for Indigenous Data Governance[END_REF] based around the principles of Collective Benefit, Authority to Control, Responsibility, and Ethics. These people-and purpose-oriented principles build on earlier data-centered work represented in the 'FAIR Guiding Principles for scientific data management and stewardship' (Findable, Accessible, Interoperable, Reusable) and represent a cornerstone of responsible data sharing highly relevant in the context of genetic resources and marine biotechnology.

Connect marine biotechnology to marine conservation

Marine biotechnology can support local to global conservation efforts in multiple ways. Examples include bioremediation and ecosystem monitoring, where the collection and sequencing of samples from marine ecosystems can provide a baseline for taxonomic and conservation efforts [START_REF] Blasiak | The ocean genome and future prospects for conservation and equity[END_REF] . Biotechnology and genomic research are increasingly being used to design ecosystem adaptation strategies, most prominently perhaps in efforts to create bionic corals, including through CRISPR gene editing [START_REF] Van De Water | Coral holobionts and biotechnology: from Blue Economy to coral reef conservation[END_REF] . Better connecting biotechnology to local conservation planning and outcomes can broaden the range of beneficiaries from this industry and highlight the existential reliance of the marine biotechnology community on intact and functioning marine ecosystems. While the environmental impacts of sample collection can be minimal [START_REF] Sigwart | Unlocking the potential of marine biodiscovery[END_REF] -marine natural products are increasingly sampled through analytical chemistry rather than collection of physical samples -an analysis of genetic sequences referenced in patents identified multiple endangered and critically endangered marine species (Figure 2). Importantly, the vast majority of species associated with marine biotechnology have not been assessed by the IUCN (1,191 of 1,488 species). Responsible practice within such contexts of uncertainty requires precautionary efforts, and provides further incentive to conserve entire ecosystems, which can result in living repositories of genetic information with potential future biotechnological potential.

Conclusion

While marine biotechnology has resulted in diverse benefits, more effort is needed to ensure this sector does not perpetuate systemic injustices through stark discrepancies in access, capacity and opportunities. The four interrelated pathways highlighted in this commentary present opportunities to build lasting capacity where it is needed, to leverage existing advances, and to accelerate progress towards ensuring the marine biotechnology sector espouses the principles at the core of the blue economy. Crucially, these pathways are viable irrespective of whether key international negotiations have stalled, or result in regulatory frameworks that are easily sidestepped by bad actors. A status quo approach may result in a marine biotechnology industry that remains profitable and continues to deliver impressive scientific advances that benefit human well-being, but at its core, it would be at odds with the larger aspirations of equity, sustainability and inclusivity inherent to a blue economy, to the UN Decade of Ocean Science for Sustainable Development, and to the Sustainable Development Agenda. Acknowledgments: RB, AP and EZ are funded by FORMAS, project number 2020-01048. AP is funded by FORMAS, project number 2019-01220.

Author contributions statement

All authors contributed to the writing and revision of the manuscript.

Competing interests statement

The authors declare no competing interests.

FIGURE 1 :

 1 FIGURE 1: Commercialization of marine genetic resources (A) Bacillus jeotgali was originally isolated from traditional Korean seafood jeotgal, and an associated strain has been identified as a potent bioremediating agent for polluted waterways 7 . (B) Halomonas titanicae was isolated from rusticles collected from the wreck of the Titanic and has been identified for its predicted functionality as a neurotransmitter to treat mental disorders 8 , (C) Ecteinascidia turbinata, a sea squirt found on mangrove roots in the Caribbean, is the source organism for the active ingredient Ecteinascidin-743 in the drug Yondelis, a treatment for advanced soft tissue sarcoma 3 . (D) Tachypleus tridentatus, a horse-show crab assessed as "Endangered" in the IUCN Red List, is a source of in vitro diagnostic reagent for invasive fungal infections prevalent in tropical regions. Photo credits: (A) [CC0 Public Domain]; (B) Lori Johnston [NOAA-Public Domain], (C) Pauline Walsh Jacobson [CC BY 4.0]; (D) [CC0 Public Domain].

Figure 2 :

 2 Figure 2: Marine biotechnology and conservation. Updating and expanding a previous analysis (see Supplementary Information), we identified sequences from 1,488 marine species referenced in patent filings. A total of 279 of the species have been assessed by the International Union for Conservation of Nature (IUCN), including species that are Near Threatened (21), Vulnerable (23), Endangered (12) andCritically Endangered (3). The latter category includes the (A) European eel (Anguilla anguilla), (B) West Indian Ocean coelacanth (Latimeria chalumnae) and (C) vaquita (Phocoena sinus). Photo credits: (A) Lara Maleen Beckmann [CC BY 4.0]; (B) Bruce Henderson [CC BY 4.0]; (C) Paula Olson [NOAA-Public Domain]


Ugur Soysal

Emmanuelle Algré

Pierre Didier
email: pierre.didier@u-pec.fr

Charles Motzkus

Uğur Soysal

Evelyne Géhin

Development of MEMS type sensors dedicated to real-time measurement of mass concentration in aerosols

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Commonly used systems are limitated because of their large size, their significant sampling times or their costs. Thus, one of the major ways of improvement of these systems is towards their miniaturization.

This study couples an airborne particle sampling method with a sub-µm air gap resonant bulk mode MEMS mass sensor as an appropriate option for a real-time aerosol monitoring system.

Airborne particle sampling design

Measurement principle

Conclusions

• The impactor and the MEMS-type mass sensors have been successfully designed, fabricated, characterized and the sensors integrated into the impactor developed.

• The mass of the impacted particles is estimated by shifting the resonance frequency of the MEMS. The sensitivity is estimated at (53±33) Hz/ng which agrees with the theoretical sensitivity of (44 Hz/ng).

• The proposed solution aims to obtain a micro-sensor allowing real-time monitoring while respecting regulatory directives and standards.

Future works

• Improvement of the sensitivity by modifying the surface structure.

• Development of the electrical measurement allowing the real-time monitoring • Integration of the electrical part into the impactor system, in order to increase the reproducibility of the measurement.

Mass calibration experiment

Frequency shit

  Calculated mass from particle counting (ng) Best-fit line defines the experimental sensitivity of the sensors as (53±33)Hz/ng Impaction of fluorescent particles (aerodynamical diameter 1µm) the International Agency for Research on Cancer has classified atmospheric pollution and the particles that compose it as compounds "definitely carcinogenic for humans" which can have harmful short and long-term effects on the body