Agathe Hubau
email: a.hubau@brgm.fr

Alexandre Chagnes

Michel Minier

Anne-Gwenaëlle Guezennec

Metal recovery from printed circuit boards (PCBs) by acidophilic bioleaching

Keywords: Bioleaching, Printed circuit boards, Acidophilic, Recovery, WEEE

During bioleaching, the use of bacteria allows the dissolution of metals with markedly reduced amount of reagents and mild pressure and temperature conditions, compared to usual operating conditions. In our study, acidophilic bioleaching was applied to comminuted spent printed circuit boards (PCBs). These wastes contain various metals, including precious and base metals, with concentrations that are generally higher than those found in primary resources. Some studies already demonstrated the efficiency of PCBs bioleaching but significant variations are observed between reported bioleaching rates. In the present study, a double-stage bioreactor has been designed and used in order to bioleach comminuted PCBs in continuous mode. The first stage of the continuous bioreactor was a bubble column in which the acidophilic consortium BRGM-KCC mainly composed of Leptospirillum ferriphilum and Sulfobacillus benefaciens was used to oxidize Fe(II) into Fe(III). The resulting leaching solution was employed to leach the metals contained in PCBs under mechanical stirring in the second stage of the bioreactor. Such a configuration was particularly relevant to reduce the impact of PCBs toxicity on bacterial growth and to maximize bioleaching rates. The culture was able to slowly adapt to PCBs (1% w/v). Consequently, with 48 hours residence time, main metals were dissolved with high yields : 96% Cu, 85% Zn, 73% Ni and 93% Co. When the solid concentration was increased up to 1.8%(w/v), it resulted in higher dissolution rates.

Introduction

Spent printed circuit boards (PCBs) are raising attention because they contain almost 35% of metals including precious and strategic metals even at greater concentration than in primary resources [START_REF] Tuncuk | Aqueous metal recovery techniques from e-scrap: Hydrometallurgy in recycling[END_REF]. Consequently, spent PCBs are becoming a valuable resource. Today, high-grade PCBs are treated by pyrometallurgy to recover precious metals but many strategic metals are lost in the slag during this operation and the energy-cost of such processes is more and more disadvantageous.

Therefore, the design of energy-efficient and cost-effective new processes capable to perform efficient metal recovery from PCBs is particularly important. Emerging techniques based on mechanical processes and hydrometallurgy appear as alternative solutions. In particular, biohydrometallurgy could be very promising. In the literature, few studies deal with the use of bioleaching for the treatment of spent PCBs by means of acidophilic microorganisms. Following main reactions may occur:

2Fe 3+ + M 0  2Fe 2+ + M 2+ (chemical) (1)
4Fe 2+ + O2 + 4H +  4Fe 3+ + 2H2O (biological) (2)
In the literature, most studies dedicated to acidophilic bioleaching of spent PCBs were performed at the lab scale in shake flasks and several parameters were studied. Staggering the addition of PCBs from the inoculation appeared particularly important, as PCBs may be responsible for inhibition phenomena towards the growth and activity of microorganisms [START_REF] Liang | Optimizing mixed culture of two acidophiles to improve copper recovery from printed circuit boards (PCBs)[END_REF]. In order to avoid such limitations, a double-stage continuous bioreactor was designed. In the first stage, ferrous iron is biologically oxidized while in the second stage, comminuted spent PCBs are leached, using the biogenic lixiviant solution produced in the first stage. The present paper focuses on the results obtained with this double-stage continuous bioreactor.

Materials and methods

2-1. Growth medium and bacterial culture

The inoculum used for continuous cultures was obtained from the BRGM-KCC microbial consortium subcultured on pyrite tailings. This consortium has already been fully described [START_REF] Guezennec | Influence of dissolved oxygen on the bioleaching efficiency under oxygen enriched atmosphere[END_REF]. The predominant organisms in the culture are affiliated to the genera Leptospirillum, Acidithiobacillus and Sulfobacillus. The culture medium, adjusted to pH 1.1, consisted of 0.40 g.L -1 (NH4)2SO4, 0.81 g.L -1 H3PO4 85%, 0.48 g.L -1 KOH, 0.52 g.L -1 MgSO4.7H2O and 1 g.L -1 Fe(II) as FeSO4.7H2O.

2-2. Printed Circuit Boards

A sample of 526 kg of medium-grade spent PCBs from the Small Waste Electrical and Electronic Equipment category was collected. PCBs were shredded to 750 µm using a shear shredder and a laboratory knife mill and were successively divided using quartering methodology [START_REF] Hubau | Recyclingoriented methodology to sample and characterize the metal composition of waste Printed Circuit Boards[END_REF]. To determine metal concentrations, a triplicate of 40 g-samples of shredded PCBs was leached at reflux during 2 hours with 450 mL of aqua regia (HNO3:HCl 1:3). After filtration, leachates were analyzed to determine metal concentrations, which are given in Table 1. This bubble column, fully described elsewhere [START_REF] Hubau | Continuous production of a biogenic ferric iron lixiviant for the bioleaching of printed circuit boards (PCBs)[END_REF], was immersed in a water bath at 35 °C, aerated with air flow between 20 and 30 L.h -1 . Air was enriched with 1% CO2 and hydraulic residence time set to 3.2 h. The second stage was a stirred tank reactor (STR) of 2.25 L continuously fed by the leaching solution produced in the bubble column. The hydraulic residence time reached 48 h in this reactor, maintained at 35 °C. Comminuted spent PCBs were continuously added with a dosing screw.

The solid concentration was set to 1%(w/v) and then 1.8% (w/v). The bubble column was inoculated with the BRGM-KCC culture at 10%v/v and was first operated in batch mode before switching into continuous mode when the redox potential reached 900 mV vs SHE.

Results

3-1. Bioleaching kinetics at 1%(w/v) PCBs concentration

At the beginning of the study, the PCBs concentration was set to 1%(w/v), to evaluate the ability of bacteria to dissolve metals and to grow in the second stage (STR reactor). The microbial concentration in the liquid phase was correlated to Fe(III) concentration in the STR. Both increased over time, to reach a permanent regime after 1600 h (see Figure 1). Dissolution kinetics followed the same trend with a slow increase over time, to reach high kinetics after 1600 h (see Table 2). rates (mg.L -1 .h -1 ) 28.33 12.17 6.04 2.71 0.40 0.28 0.45 0.067 Dissolution yields (%) 96% 59% 49% 85% 11% 12% 73% 93% 3-2. Bioleaching kinetics at higher solid concentration (1.8%(w/v))

After the increase of solid concentration, metals dissolution rates increased gradually and stabilized after 1000 hours (see Table 3). Higher dissolution yields were reached. For Cu, dissolution yield was above 100%. This could reflect a problem during the determination of Cu concentration in the initial sample of PCBs. The redox potential was higher than 880 mV vs SHE for both solid concentrations, which showed that all Fe(II) produced by the dissolution of metals was oxidized. Concerning biomass concentration, the total cell concentration reached 2.10 8 cells/mL at 1.8%(w/v), while it reached 5.10 7 cells/mL at 1%(w/v).

Discussion

At the beginning of the study of PCBs bioleaching in continuous mode with 1%(w/v) PCBs, slow increase in metals dissolution yields was observed and can be attributed to cells adaptation phases.

Initially, bacteria entering the second stage (STR) were not able to grow or to oxidize Fe(II) noticeably. Metals dissolution was mainly related to chemical leaching by incoming Fe(III) (1 g.L -1 ) and sulfuric acid. Then, metal dissolution rates increased but no Fe(III) is detected, since its consumption for metal leaching is faster than its regeneration. Finally, the number of cells exponentially increased in the liquid phase. The microbial culture regenerated Fe(III) much faster than the consumption of Fe(III) by the metal dissolution reactions, so that only Fe(III) was detected. High dissolution kinetics were obtained. Microbial adaptation was also noticed when the solid concentration was increased to 1.8%(w/v). This concentration did not inhibit the bacterial growth and activity. The higher concentration of substrate favored the growth of micro-organisms in the second stage.

Conclusion

A continuous double stage bioreactor was used to carry out the bioleaching of spent PCBs. The adaptation of microbial culture to PCBs with solid concentrations up to 1.8%(w/v) led to high dissolution kinetics for most metals (Cu, Ni, Zn, Fe and Co).

Figure 1 :

 1 Figure 1: Biomass and Fe(III) concentration in the 2 nd stage of the continuous bioreactor

Table 1 :

 1 Description of experimental setup during continuous experimentsContinuous culture experiments were carried out in a two-stage bioreactor. The first stage was a bubble column of 200 mL filled with 150 mL of liquid and 15 g of activated charcoal as solid support.

	Cu	Fe	Al	Zn	Sn	Pb	Ni	Co
	Concentration (%) 14.58% 12.23% 6.04% 1.67% 1.67% 1.17% 0.34% 0.036%

Major metal concentrations (in % weight) in 40 g-samples of PCBs.

Table 2 :

 2 Dissolution rates and yields during continuous bioleaching at 1%(w/v) of PCBs.

	Cu	Fe	Al	Zn	Sn	Pb	Ni	Co
	Dissolution

Table 3 :

 3 Dissolution rates and yields during continuous bioleaching at 1.8%(w/v) of PCBs.

		Cu	Fe	Al	Zn	Sn	Pb	Ni	Co
	Dissolution rates (mg.L -1 .h -1 ) 62.50 34.48 11.67 5.63 0.69 0.34 1.08 0.117
	Dissolution yields (%)	115% 76% 52% 91% 11% 8%	86% 87%

Acknowledgement:

This work was supported by the Chair "Mines Urbaines" from ParisTech foundation supported by Eco-systèmes.


Bimba Andrew Thomas

Norisma Idris

Ahmed A Al-Hunaiyyan

Rohana Binti Mahmud

Liyana Bt

Mohd Shuib

Design of an Algebraic Concept Operator for Adaptive Feedback in Physics

Keywords: OAR Model, Knowledge Base, Pedagogy, Learning Environment, Student, Problem Solving

In an adaptive learning environment, the feedback provided during problem-solving requires a means, target, goal, and strategy. One of the challenges of representing feedback to meet these criteria, is the representation of the effect of multiple concepts on a single concept. Currently, most of the methods (linguistic knowledge base, expert knowledge base, and ontology) used in representing knowledge in an adaptive learning environment only provide relationships between a pair of concept. However, a cognitive knowledge base which represents a concept as an object, attribute, and relations (OAR) model, provides a means to determine the effect of multiple concepts on a single concept. Using the OAR model, the relationships between multiple pedagogical, domain, and student attributes are represented for providing adaptive feedback. Most researchers have proposed adaptive feedback methods that are not fully grounded in pedagogical principles. In addition, the three knowledge components of the learning environment (pedagogical, domain and student models) are mostly treated in isolation. A reason for this could be the complex nature of representing multiple adaptive feedback characteristics across the main components of a learning environment. Thus, there is a need to design a concept operator that can relate the three facets of knowledge in an adaptive learning environment. Using the algebraic concept operator R in i , the effect of multiple attributes of the three knowledge components on the student's performance is represented. The algebraic concept operator introduced in this article will allow teachers and pedagogy experts to understand and utilize a variety of effective feedback approaches.

Introduction

In an adaptive learning environment, the feedback provided during problemsolving requires a means, target, goal, and strategy [START_REF] Bimba | Adaptive feedback in computer-based learning environments: a review[END_REF]. One of the challenges of representing feedback to meet these criteria, is the representation of the effect of multiple concepts on a single concept. Currently, most of the methods (linguistic knowledge base, expert knowledge base, and ontology) used in representing knowledge in an adaptive learning environment only provide relationships between a pair of concept [START_REF] Bimba | A cognitive knowledgebased framework for adaptive feedback[END_REF]. However, a cognitive knowledge base which represents a concept as an object, attribute, and relations (OAR) model, provides a means to determine the effect of multiple concepts on a single concept. Using the OAR model, the relationships between multiple adaptive feedback means, target, goal, and strategy are represented for providing adaptive feedback by showing the effect of these characteristics on the performance of the student. The proposed concept operator is initialized using randomly pre-configured relationships between the characteristics of the pedagogical, domain and student model. During the process of learning, these relationships are automatically updated to suggest the appropriate combination of adaptive feedback means, target, goal, and strategy which can lead to optimal performance for different students.

Knowledge in a adaptive learning environment can be represented in the form models. The three most important models are the pedagogical model, domain model, and the student model. The pedagogical model represents the knowledge and technique of teaching. Specific knowledge represented in the pedagogical model is based on a pedagogical principle or learning theory. This principle or theory determines the effective teaching methods, instructional methods, sequence of activities, feedback types, and assessments modeled by the learning environment. On the other hand, the domain model, is a facet of the pedagogical model, which represents the knowledge of the subject been learned. The domain model represents concepts, learning materials, facts, problems, solutions, feedback, rules, equations etc. Finally, the student model represents information about the student's knowledge of the domain, learning style, interactions with the system, response to feedback, emotional state, performance etc. These information determines the characteristics of the student during problem-solving.

[10] defines a problem as an obstacle between a present state and a goal and it is not immediately obvious how to get around the obstacle (p. 365). Two types of problems identified by Psychologists are well-structured and ill-structured problems. The well-structured, are problems that by applying certain procedures will lead to a correct answer, while ill-defined problems do not usually have a single correct answer and have an unclear process to arriving at a solution [START_REF] Shekoyan | Introducing illâĂŘstructured problems in introductory physics recitations[END_REF][START_REF] Shin | Predictors of wellâĂŘstructured and il-lâĂŘstructured problem solving in an astronomy simulation[END_REF]. Depending on the type of problem considered, the Novice tends to be more accurate when the problem at hand is well-structured, but in cases where the problems are ill-structured, the Novice tends to perform poorly [START_REF] Chi | Categorization and representation of physics problems by experts and novices[END_REF][START_REF] Hardiman | The relation between problem categorization and problem solving among experts and novices[END_REF][START_REF] Shin | Predictors of wellâĂŘstructured and il-lâĂŘstructured problem solving in an astronomy simulation[END_REF][START_REF] Singh | Categorization of problems to assess and improve proficiency as teachers and learners[END_REF][START_REF] Soong | StudentsâĂŹ difficulties when solving physics problems: Results from an ict-infused revision intervention[END_REF][START_REF] Goldstein | Cognitive psychology: Connecting mind, research and everyday experience[END_REF][START_REF] Docktor | Impact of a short intervention on novicesâĂŹ categorization criteria[END_REF]. This could be related to the fact that the Novice has little knowledge of the deep structure of the problem, thereby not understanding the under-lining principles and concepts related to such problems. Physics instructions are designed to help students improve their problem solving expertise [START_REF] Soong | StudentsâĂŹ difficulties when solving physics problems: Results from an ict-infused revision intervention[END_REF]. This is why some physics problems are ill-structured, similar to most real-life and professional problems.

Physics is more closely associated to everyday experiences. Thus, having proficiency in physics is a relevant skill, which can be extended to solving everyday problems. Physics is generally regarded as a difficult subject. Factors contributing to this difficulty include: 1) the presence of misconceptions; 2) degree of logical precision and reasoning required; 3) mathematical skills required; 4) misinterpretation of question posed; 5) concept gaps; and 6) knowledge gaps [START_REF] Soong | StudentsâĂŹ difficulties when solving physics problems: Results from an ict-infused revision intervention[END_REF][START_REF] Clement | StudentsâĂŹ preconceptions in introductory mechanics[END_REF]. During physics problem-solving in an adaptive learning environment, feedback is provided to assist the student in achieving a set objective.

Most researchers have proposed adaptive feedback methods that are not fully grounded in pedagogical principles [START_REF] Gerdes | An interactive functional programming tutor[END_REF][START_REF] Rivers | Data-driven hint generation in vast solution spaces: a self-improving python programming tutor[END_REF]. In addition, the three main knowledge components of the learning environment (pedagogical, domain and student model) are mostly treated in isolation. A reason for this could be the complex nature of representing multiple adaptive feedback characteristics across the main components of a learning environment. Thus, there is a need to design a concept operator that can relate the three facets of knowledge in an adaptive learning environment.

The article is structured into 5 sections. First, feedback in an adaptive learning environment is introduced in section 1 and a brief background on pedagogy and student modeling is presented in section 2. In section 3, we explain the design of an algebraic concept operator for adaptive feedback. Preliminary results, on the implementation of the algebraic concept operator is presented in section 4. And finally, in section 5 we conclude the article and highlight the future works.

Background

Educational institutions are relatively successful in arranging and conveying large forms of knowledge and standard pedagogical practices [START_REF] Collins | Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. Knowing, learning, and instruction: Essays in honor of Robert Glaser[END_REF][START_REF] Paul | Helping students to become capable learners[END_REF]. However, little attention is paid to the strategies experts use to acquire knowledge in solving complex task. The emphasis, in educational institutions is mostly on formulaic methods for solving textbook problems. While few resources are devoted to high-order problem-solving which requires students to actively integrate and apply conceptual knowledge. In order to make a positive impact on student's skill acquisition, there is a need to understand how experts acquire skills and to formulate different methods that are appropriate to learning these skills [START_REF] Collins | Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. Knowing, learning, and instruction: Essays in honor of Robert Glaser[END_REF]. Therefore, more effort should be made on recognizing cognitive and metacognitive strategies rather than low-level sub-skills and factual knowledge. To support student learning, teachers and educationists regularly use a wide range of educational theories and teaching strategies [START_REF] Tompkins | Application of cognitive apprenticeship model (ca) to library instruction[END_REF]. Over the past decades, researchers have developed various learning theories in an attempt to explain how knowledge is acquired. These include theories such as behaviorism, cognitivism, and constructivism. The cognitive approach places the learner and the internal mental processes at the center of teaching. Because cognitivism focuses on revealing the various processes involved in knowledge acquisition, thereby providing strategies that support students learning, tutors can utilize this in their effort to aid students in attaining their goals. The cognitive apprenticeship (CA) instructional approach utilizes cognitive and meta-cognitive skills and processes to guide learning [START_REF] Dennen | The cognitive apprenticeship model in educational practice[END_REF]. The CA concept is defined by [START_REF] Collins | Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. Knowing, learning, and instruction: Essays in honor of Robert Glaser[END_REF] as learning through guided experience, cognitive and meta-cognitive skills rather than physical skills and processes as in traditional apprenticeship. In CA learning environment, students cannot engage in the apprenticeship process alone, rather it involves modeling (demonstration by expert) and coaching in the initial stage of learning [START_REF] Dennen | The cognitive apprenticeship model in educational practice[END_REF]. The CA model for designing learning environments has four dimensions: i) content or domain (the type of knowledge required for expertise), ii) method (teaching method to promote student development), iii) sequencing (ordering learning activities), and iv) sociology (social characteristics of the learning environment) [START_REF] Collins | Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. Knowing, learning, and instruction: Essays in honor of Robert Glaser[END_REF][START_REF] Polo | Using the cognitive apprenticeship model to develop educational learning modules: An example from statics[END_REF].

The selection of appropriate student's characteristics is significant in the early stages of developing a student model. It is important to consider what aspect of the student characteristics is to be modeled according to the type of system developed [START_REF] Gonzãąlez | A qualitative comparison of techniques for student modeling in intelligent tutoring systems[END_REF]. In order to implement adaptation in a learning environment it is necessary to model the flexibly stable, dynamic and domain dependent characteristics of the student [START_REF] Chrysafiadi | Student modeling approaches: A literature review for the last decade[END_REF]. The flexibly stable characteristics of the student includes cognitive styles, student's demographics, learning objectives and goals. The dynamic parameters are concerned with the students interactions and experience with the learning environment. These characteristics include the student's errors and mis-conceptions, motivations, attitude, collaboration, learning actions and behaviors. The domain dependent characteristics of the student involves the student's knowledge level and interest in a specific topic.

The development of an adaptive feedback environment requires the combination of pedagogical principles, domain attributes and various student's characteristics. This research emphasizes on the cognitive apprenticeship principle. The attributes of the students considered for adaptive feedback are the cognitive learning style, student's performance, and knowledge level. The cognitive apprenticeship approach to learning involves the guidance of a student to achieve cognitive and meta-cognitive skills with the help of an expert. This principle was selected for this research because it clearly supports feedback, thus providing an avenue to demonstrate the use of a dynamic knowledge base approach to adaptive feedback.

Algebraic Concept Relations for Adaptive Feedback in Physics

In this article, we focus on the cognitive apprenticeship (CA) pedagogical principle and the physics domain. In a CA principle, the main aim of designing teaching methods is to help students acquire and use cognitive and meta-cognitive strategies for discovering, utilizing and managing knowledge [START_REF] Collins | Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. Knowing, learning, and instruction: Essays in honor of Robert Glaser[END_REF]. The acquisition and use of these strategies solely depends on the interaction between the student's current knowledge, the social and physical environment of the problem-solving and the intricacies of the problem-solving. The challenge is to represent the attributes of the CA principle that affect the type, timing, goal, and sequence of feedback, relating it to the characteristics of the domain and student models.

The representation of the characteristics of all 3 models relating to adaptive feedback is shown in An algebraic concept relation which provides adaptive feedback in physics is proposed. The main objective is to observe the relationship between the performance of a student based on the adaptive feedback provided. However, the adaptive feedback provided depends on the adaptive feedback means, goal, target, and strategy. These involves the 3 facet of the adaptive learning environment of pedagogy, domain, and student models. The effect of multiple concepts on a single concept is defined in Eq. 1.

R in i ⊆ |Ri| i=1 R c i (1)
The effect of a concept is represented as the union between all its related external concepts as shown in Eq. 1.

where -R c i is the internal relationships that exist between multiple concept that have a relation or effect on the current concept concerned.

For example, in an adaptive learning environment as shown in Figure 1 the effect of the attributes of concepts within the pedagogy, domain, and student models on the performance of the student is represented in Eq. 2.

R in p ⊆ R c CA ∪ R c CA d ∪ R c S (2)
where

-R c CA ⊆ (CA s ∪ CA sq ∪ CA im )
is the union of active sets of attributes within the pedagogical model which are related to the performance of a student at a given time.

-R c CA d ⊆ (D c ∪ C f ∪ C p )
is the union of active sets of attributes within the domain model which are related to the performance of a student at a given time.

-R c S ⊆ (S cls ∪ S kl ) is the union of active sets of attributes within the student model which are related to the performance of a student at a given time.

By expanding Eq. 2, all possible relationships between the performance of a student and the attributes of pedagogical, domain, and student model is presented in Eq. 3.

(3)

R in sp ⊆ {{S ec } ∪ {{SQ ic , SQ gl } ∪ {IM m , IM c , IM s , IM r , IM a , IM e }} ∪ {{F ts , F h , F c , F n } ∪ {D c } ∪ {P sim , P a , P d , P vd }} ∪ {{CLS f d , CLS f i } ∪ {KL nl , KL il , KL l , KL e }}}
During the learning process, not all attributes are activated at the same time. For example, a student cannot be a FI and FD learner at the same time. Therefore, at any given time (t), the effect of multiple concepts on the performance of a student can be represented as shown in Eq. 4.

R in sp ⊆ {{SQ ic , IM m } ∪ {F ts , D c , P sim } ∪ {CLS f d , KL nl , KL il , KL l , KL e }} (4) R in sp ⊆ {SQ ic , IM m , F ts , D c , P sim , CLS f d , KL nl } (5)
Using Eq. 5, the performance of a novice student KL nl with a field Dependant cognitive learning style CLS f d , solving a simple problem P sim within a domain concept D c , provided with an increasing complexity of feedback SQ ic in a modeling stage IM m can be easily represented. Thus, the system can automatically analyze the right combination of characteristics of the pedagogy, domain, and student models which provides the best performance for different type of students at different knowledge levels. In an adaptive feedback perspective, this implies an automatic update of the knowledge base, based on the best performance of different students solving different problems and receiving different feedbacks according to their individual characteristics and knowledge level on a specific concept. The proposed adaptive feedback model allows the knowledge base to make an automatic decision on the function, time, schedule, and type of feedback provided. At an initial state of building the knowledge base the knowledge bonding process selects random combinations of the required attributes needed to provide adaptive feedback to the student. As more students use the system, the knowledge base learns and updates its optimum combination of concepts to achieve an effective feedback to the student.

Preliminary Results

The provision of feedback based on the cognitive apprenticeship principle involves the instructional method, sequencing, sociology, and domain. When providing feedback in this context, a decision has to be made on what instructional method to be applied, the sequencing, and the application of a sociological component. In Table 1, a summary of the different forms of feedback provided based on the CA principle is presented. Certain interventions can be provided to the student before and during problem solving in a specific sequence. As seen in Table 1, there are 17 possible forms of feedbacks to be provided based on the feedback model presented in Fig. 1. The feedback attributes as shown in Table 1 are the characteristics of the CA pedagogical principle that determine the type of feedback that will be provided before and during problem-solving.

In the process of solving a problem there are different scenarios based on the domain attributes such as topic and problem and also the characteristics of the students which includes cognitive learning style and knowledge level. Based on these characteristics a total of 408 problem solving states are possible. In table 2, a summary of these problem solving states are presented. As students solve problems, they receive feedback based on the pedagogical principles and their performance is feedback to the knowledge base according to the current problem solving state. These allow the knowledge base to be dynamic and the choice of feedback is based on the algebraic concept operation R in p defined.

Conclusion

The aim of this article is to present the design of an algebraic concept operator which represents the relationship between student's performance and multiple attributes in the pedagogy, domain and student models. Fig. 1, represents the relationship between the 3 facets (pedagogy, domain, and student models) of an adaptive learning environment. The model shows how full adaptive feedback characteristics can be modeled in the physics domain. It shows how concept algebra is applied to designing a concept association operator which defines the effect of multiple concepts on a single concept. Based on the preliminary results, about 17 adaptive feedback scenarios based on the cognitive apprenticeship principle are defined. Using the algebraic concept operator R in p , the effect of multiple attributes of the pedagogy, domain, and student models on the student's performance is represented. These allows a selection of a effective adaptive feedback to be provided to the student during problem-solving. The algebraic concept operator introduced in this article will provide developers of adaptive learning environments with a means capturing effective relationships between multiple attributes in the pedagogy, domain, and student models. In addition it allows teachers and pedagogy experts to understand and utilize a variety of effective feedback approaches. In the future, a learning algorithm will be included to allow adjustments in the weights between the related concepts in the pedagogy, domain, and student models.

Fig. 1 .

 1 The knowledge in the proposed pedagogy, domain, learner -cognitive knowledge base (PDL-CKB) model is based on the OAR model. The nodes of the concept network in Fig.1, represents concepts in the pedagogy, domain and student model in an adaptive learning environment. The arcs represent the relationship between concepts and their attributes.

Fig. 1 .

 1 Fig. 1. PDL-CKB Model for Adaptive Feedback in Physics

Table 1 .

 1 Type of Feedback based on the Cognitive Apprenticeship Principle.

	S/N Feedback ID Feedback Attributes	Before Problem-solving	During Problem-solving	Timing
	1	FT1	exploiting competition, global to local state, modelling	student's worked-example	student's solution	immediate, after first error
	2	FT2	exploiting competition, modelling, increase complexity	student's worked-example	hints, clues, student's explanation	immediate, after each error
	3	FT3	modelling,global to local state tutor's solution tutor's worked-example immediate, after first error
	4	FT4	increase complexity, modelling tutor's solution	hints, cues, tutor's explanation	immediate, after each error
	5	FT5	global to local state, scaffolding non	tutor's partial worked-example	immediate, after more than one error. Provide feedback not more than twice.
	6	FT6	increase complexity, scaffolding	non	hints, cues, tutor partial explanation	immediate, after each error. Provide feedback not more than twice.
	7	FT7	exploiting competition, global to local state, scaffolding	non	student's partial worked-example	immediate, after first error. Provide feedback not more than twice.
	8	FT8	exploiting competition, increase complexity, scaffolding	non	hints, cues, student's partial explanation	immediate, after each error. Provide feedback not more than twice.
	9	FT9	exploiting competition, global to local state, coaching	non	student's worked-example	immediate, after first error
	10 FT10	exploiting competition, increase complexity, coaching	non	hints, cues, student's explanation	immediate, after each error
	11 FT11	global to local state, coaching	non	tutor's worked-example	immediate, after first error
	12 FT12	increase complexity, coaching	non	hints, cues, tutor's explanation	immediate, after first error
			increase complexity,
	13 FT13	exploiting competition,	non	student's explanation	at the end of problem-solving
			articulation
	14 FT14	increase complexity, articulation	non	tutor's explanation	at the end of problem-solving
	15 FT15	global to local state, reflection	non	tutor's worked-solution	at the end of problem-solving
	16 FT16	global to local state, exploiting competition, reflection	non	student's worked-solution	at the end of problem-solving
	17 FT17	exploration	non	non	result at the end of problem-solving

Table 2 .

 2 Relationship between students performance and pedagogy, domain, and student attributes

											effect of
											PM, DM,
	S/N	Problem Solving State	Pedagogical Model (PM)	Domain Model (PM)	Student Model (SM)	& SM on Student's
											Performance
											(R in p )
			global to local state	modelling	exploiting competition	FT1 average	linear motion	field independent	novice
	1	pss1	0.5	0.5	NA	0.5	0.5	0.5	0.5	0.5	0.5
			global to local state	coaching	exploiting competition	FT9 difficult	rotational motion	field independent	intermediate
	2	pss2	0.32	0.32	0.32	0.32 0.32	0.32	0.32	0.32	0.32
			global to local state	coaching	exploiting competition	FT9 simple	rotational motion	field independent	expert
	3	pss3	0.43	0.43	NA	0.43 0.43 gravitation	0.43	0.43	0.43
			global to local state	coaching	exploiting competition	FT9 average	static	field independent	intermediate
	4	pss4	0.5	0.5	NA	0.5	0.5	0.5	0.5	0.5	0.5
			global to local state	coaching	exploiting competition	FT9 average	linear motion	field independent	novice
	5	pss5	0.27	0.27	0.27	0.27 0.27	0.27	0.27	0.27	0.27
			global to local state	coaching	exploiting competition	FT9 average	static	field dependent	expert
	6	pss6	0.18	0.18	0.18	0.18 0.18	linear motion	field dependent	0.18	0.18

Grant [RP040B-15AET, 2015].


Noureddine Chikouche

Abderrahmen Ghadbane

Performance Evaluation of Post-Quantum Public-Key Cryptography in Smart Mobile Devices

Keywords: Post-quantum cryptography, public-key encryption, publickey signature, performance, mobile devices

The classical public-key schemes are based on number theory, such as integer factorization and discrete logarithm. In 1994, P.W. Shor proposed an algorithm to solve these problems in polynomial time using quantum computers. Recent advancements in quantum computing open the door to the possibility of developing quantum computers sophisticated enough to solve these problems. Post-quantum cryptography (PQC) is resistant against quantum attacks. The aim of this paper is to evaluate the performance of different post-quantum public-key schemes for constrained-resources smart mobile devices; and to give a comparison between the studied post-quantum schemes in terms of computational time, required memory, and power consumption.

Introduction

The classical public-key algorithms used today to secure user data and networking communications (e.g. Internet, mobile, etc.) are based on number theory. For example, the RSA cryptosystem is based on integer factorization problem, and the Diffie-Hellman scheme is based on discrete logarithm problem. In 1994, P.W. Shor [START_REF] Shor | Polynomial-time algorithm for prime factorization and discrete logarithms on a quantum computer[END_REF] proposed an algorithm to solve these problems in polynomial time using quantum computers.

In 2015, the National Security Agency (NSA) [27] announced that it is working with several partners to develop quantum-resistant encryption algorithms. In 2016, the NIST (National Institute of Standards and Technology) [START_REF]Post-Quantum Cryptography Standardization[END_REF] has started the process of developing, evaluating, and standardizing one or more public-key post-quantum cryptographic algorithms. It's crucial to re-evaluate the existing cryptographic schemes which are used to protect information, and to improve quantum-safe cryptography.

The quantum computing is not a future project but it already exists in the real-world. The D-Wave Systems Inc. [START_REF]The D-Wave 2000Q Quantum Computer: Technology Overview[END_REF] was the first company that commercialises products based on quantum computing principles that are being used by some of the important advanced organizations, including Google and NASA Ames. In 2017, this company developed the D-Wave 2000Q system which is a quantum annealer that has up to 2048 qubits and 5600 couplers. It can solve larger problems in various areas, such as machine learning, financial analysis, optimization and security. In security, this quantum computer can carry out factoring integers, detects computer viruses and network intrusion.

Nowadays, there are various classes of post-quantum public-key cryptography, including:

-Code-based cryptography: The classic example is McEliece cryptosystem based on Goppa codes [START_REF] Mceliece | A public-key system based on algebraic coding theory[END_REF]. It uses error correcting codes to generate publickey from private matrices with intentionally added errors. It is employed in the construction of diverse cryptographic schemes and it does not need any cryptographic processor. -Lattice-based cryptography: The most basic lattice problem is the shortest vector problem (SVP), given an arbitrary basis of lattice, the goal is to find the shortest nonzero vector in it. NTRU scheme [START_REF] Hoffstein | NTRU: a ring-based public key cryptosystem[END_REF] is one of the most interesting lattice-based variants. -Multivariate-based cryptography: It uses a set of multivariate polynomial equations that are based on the multivariate quadratic (MQ) Problem. One of its many interesting schemes is Rainbow public-key signature scheme which was proposed by J. Ding and D. Schmidt [START_REF] Ding | Rainbow, a new multivariable polynomial signature scheme[END_REF] in 2005. -Hash-based cryptography: It is based on so-named one-time signature (OTS), a single key pair must only be used once. it requires a cryptographic hash function to create a public-key signature. For example, Winternitz onetime signature (W-OTS) [START_REF] Dods | Hash based digital signature schemes[END_REF] that relies on collision resistance, which means that using the same private-key to sign multiple documents will not yield a similar signature. -Isogeny-based cryptography: This category was introduced as a solution to breaking elliptic curve cryptography by Shor's algorithm. Isogeny problem is to find the isogeny mapping between two elliptic curves with the same number of points. NIST [START_REF] Chen | Report on post-quantum cryptography[END_REF] stated that not enough analysis has been done prove the claimed security.

Post-quantum cryptography (PQC) is resistant against quantum attacks and its computational complexity is of type NP-hard problem. Recently, there are several security protocols based on PQC have been presented, such as [START_REF] Li | A new self-certified signature scheme based on NTRUSing for smart mobile communications[END_REF][START_REF] Chikouche | RFID authentication protocols based on error-correcting codes: A survey[END_REF][START_REF] Cho | A mceliece-based key exchange protocol for optical communication systems[END_REF][START_REF] Chen | A novel NTRU-based handover authentication scheme for wireless networks[END_REF]. In the context of smart mobile technology, to secure user data and mobile communication, it is crucial to implement efficient cryptographic primitives. In the other hand, the most important problem is the limitation of resources, storage, processing, and power.

The aim of this paper is to survey the post-quantum public-key algorithms in regards to their efficient in smart mobiles. We evaluate their performance in terms of computational time, required memory, and power consumption. Moreover, we compare the different PQC in terms of performance and security.

The rest of this paper is structured as follows: Section 2 presents related works. Section 3 presents post-quantum encryption schemes. Section 4 presents post-quantum signature schemes. The experimental results is detailed in Section 5. We discuss of obtained results in Section 6. Finally, conclusion has been presented in Section 7 .

Related works

An important number of studies have been realised in order to present efficient implementation of post-quantum cryptography in constrained resource devices. In the rest of this section, we introduce some research efforts in this area.

Eisenbarth et al. [START_REF] Eisenbarth | MicroEliece: McEliece for embedded devices[END_REF] investigated the efficient software implementation of McEliece scheme on embedded systems, low-cost 8-bit AVR microprocessor and a Xilinx Spartan-3AN FPGA. Hayes [START_REF] Heyse | Post Quantum Cryptography: Implementing Alternative Public Key Schemes On Embedded Devices[END_REF] evaluated different implementation possibilities for McEliece, Niederreiter, and their variants. In addition, He evaluated the performance of the schemes using various types od codes on smartcard class microcontrollers and a range of FPGAs. Wang et al. [START_REF] Wang | FPGA-based Niederreiter cryptosystem using binary goppa codes[END_REF] improved the previous implementations of Niederreiter encryption scheme in terms of efficiency and security level by presenting a new implementation using binary Goppa codes in FPGA.

About the implementation of PQC on mobiles devices, Tayoub et al. [START_REF] Tayoub | Implementation of public-key cryptographic systems on embedded devices (case: Computation speed)[END_REF] implemented NTRU scheme and other classical public-key schemes on Android mobiles, and evaluated their performances in terms of timing and memory occupation. In 2016, A. Boledovic and J. Varga [START_REF] Boledovic | Practical implementation of McEliece cryptosystem on Android[END_REF] implemented McEliece encryption scheme in messenger application of Android operating system by using Bouncy castle provider.

Post-quantum encryption schemes

Post-quantum encryption scheme is used to safeguard the confidentiality of stored and exchanged information. It consists of three processes: key generation, encryption, and decryption. The key generation process creates a key pair consisting of a public and a private key. The public-key is used to encrypt a plaintext and the private-key, to decrypt a ciphertext. Various post-quantum encryption schemes subsequently designed are presented below.

McEliece scheme

In 1978, Robert J. McEliece [START_REF] Mceliece | A public-key system based on algebraic coding theory[END_REF] introduced the first public key cryptosystem (PKC) based on coding theory. The security of McEliece scheme is based on the problem of computational dual decoding syndrome. Let C[n, k, t] be a binary linear code, where n is length, k is dimension which stands as a generator matrix G. C can correct up to t errors.

The McEliece encryption scheme is defined as follows:

Key Generation:

-Generate three private matrices, a generator matrix G ∈ F k×n 2 of a binary Goppa code C, a permutation matrix P ∈ F n×n 2 and an invertible matrix S ∈ F k×k 2 , -Compute the public-key matrix G = S G P , which is another valid generator matrix, -The private-key is (S , G , P, A(.)), where A(.) is a polynomial-time decoding algorithm, -The public-key is (G, t).

Encryption: To encrypt a message m ∈ F k 2 -Generate an error vector e ∈ F n 2 of weight wt(e) ≤ t, -Compute the codeword c ∈ F n 2 where c is mG and the plaintext is m ∈ F k 2 , -The cryptogram c = c ⊕ e. Decryption: To decrypt a cryptogram c -Compute z = c P -1 , -y = A(z), -Output m = yS -1 .

Niederreiter scheme

Niederreiter encryption scheme [START_REF] Niederreiter | Knapsack-type cryptosystems and algebraic coding theory[END_REF] introduced the dual version of McEliece encryption scheme. This variant is based on the syndrome decoding SD-problem using the parity check matrix. The important advantage of this scheme compared to McEliece is reduction of the public-key size from k × n to n × (n -k).

The Niederreiter encryption scheme is defined as follows:

Key generation:

-Parameters: n, t ∈ N, where t < n -Generate a parity check matrix

H ∈ F (n-k)×n 2 of a binary linear C, -Generate a permutation matrix P ∈ F n×n 2 , -Generate an invertible matrix Q ∈ F (n-k)×(n-k) 2 , -private-key: (Q, H , P, A(.)) with A(.) a decoding algorithm until d 2 errors, -public-key: H ∈ F (n-k)×n 2 := QH P and t integer < d 2 . Encryption: To encrypt message m -decode m to error vector e ∈ F n 2 with wt(e) = t, -c := H T e, -output ciphertext c . Decryption: To decrypt cryptogram c -Q -1 c := Q -1 QH (P e), -compute P -1 (P e),
encode e into message m.

McE Kobara-Imai scheme

Kobara and Imai [START_REF] Kobara | Semantically secure McEliece public-key cryptosystemsconversions for mceliece pkc[END_REF] proposed modified versions of McEliece scheme that can be proven to be semantically secure against adaptive chosen-ciphertext attacks (CCA2). In addition, their conversion γ (Algorithm below) uses the entropy in the error vector, to decrease the overhead of data further.

c := y5 c , -y3 := Decrypt M celeice (c ), -y3G ⊕ y0, -y4 := Conv -1 (z), -(y2 y1) := (y5 y4 y3), -r := y2 ⊕ Hash(y1), -(x Const ) := y1 ⊕ Gen(r), -if Const == Const return x := x,
else return ⊥.

NTRU encryption scheme

The NTRU cryptosystem was published by Jeffrey Hoffstein and Jill Pipher and Joseph Silverman in 1998 [START_REF] Hoffstein | NTRU: a ring-based public key cryptosystem[END_REF] and it was standardised by IEEE in 2008. NTRU uses a public parameter N to specifies the size of the polynomials used, a large modulus q and a smaller modulus p. The sender creates a pair of a public and a private key by generating two polynomials f and g,where f is invertible modulo both p and q.

The NTRU encryption scheme is defined as follows:

Key Generation: -Parameters: N ,q,p, where N and p prime, gcd(p, q) = gcd(N , q) = 1 randomly choose two private polynomials f and g in the ring -Private key: consists of the polynomials f and f q := f -1 mod p -Public key: h := p * f q * g mod q where f q := f -1 mod q Encryption: To encrypt a message m generate a random polynomial r The digital signature scheme is a cryptographic primitive that provides publickey message authentication. It consists of three processes, key generation, signature generation, and signature verification. The key generation process creates a key pair consisting of a public and a private key. The private-key is used to sign a document and to generate a signature when the public-key is used to verify this signature. In this section, we present four important post-quantum digital signatures: Niederreiter CFS, NTRUSign, Rainbow, and XMSS.

Niederreiter-CFS signature scheme

A signature scheme based on the Niederreiter encryption scheme was introduced by Courtois, Finiasz and Sendrier in [START_REF] Courtois | How to achieve a McEliece-based digital signature scheme[END_REF]. The idea of the Niederreiter-CFS scheme is to frequently hash the message, randomized by a counter of bit-length i, until the output is a ciphertext that can be decrypted. To determine the errorvector, the signer uses his corresponding private key, and with the current value of the counter, the error vector will then serve as a signature. This signature scheme has the following parts:

The Niederreiter-CFS signature scheme is defined as follows:

Signature: To sign the document d

1. i ← i + 1, 2. s := A(Q -1 h(h(d) i)), 3
. if no x was found go to 1 else output (i, x P ). Verification: To verify the (i, x P )

-s = Hx T , -s = h(h(d) i),
-If s and s equals, then the signature is valid.

NTRU signature scheme

The NTRU Signature scheme, also known as NTRUSign was presented in [START_REF] Hoffstein | NTRUSign: digital signatures using the ntru lattice[END_REF], it's based on the GGH signature scheme. NTRUSign includes mapping a message to a random point in 2N-dimensional space, with N being a parameter, and solving the closest vector problem (CVP) in a lattice which is related to the NTRUEncrypt lattice problem.

The NTRU signature scheme is defined as follows:

Key Generation:

pick two short polynomials h and f in ring R, find (F, G) with f * g -g * F = q, -Private key: (f, g, F, G), -Public key: h := g * f -1 mod q with f is invertible in R q . Signature: To sing the document d

-t := 0 -Repeat 1. t := t + 1, 2. µ := Hash(d t) ∈ R q , 3. (x, y) := (0, µ) G F -G f , 4. s := -x * f -y * g, -until (s, s * h -d) ≤ N , -return (s, t). Verification: To verify (d, s, t) -µ := Hash(d t), -if (s, s * h -d) ≤
N then the signature is valid.

Rainbow signature scheme

In 2005, J. Ding and D. Schmidt [START_REF] Ding | Rainbow, a new multivariable polynomial signature scheme[END_REF] proposed a public-key signature scheme named Rainbow, which is based on the idea of Oil and Vinegar variables. The idea of Oil and Vinegar variables is one way to create easily invertible multivariate quadratic systems. The Rainbow signature scheme is defined as follows:

Key Generation: -Private key: Two invertible affine maps L 1 and L 2 and the map F = (f v1+1 (x), ..., f n (x)). The number of components of F is m = n -v1, -Public key: The composed map

P (x) = L 1 • F • L 2 . Signature: To sign a document d -h := hash(d), -x := L -1 1 (h), -y := F -1 (x) where F -1 (x) means finding one pre-image of x, -z := L -1
2 (y), output: the signature z. Verification: To verify z h := P (z), h := hash(d), if h = h then the signature is valid.

XMSS Scheme

The eXtended Merkle Signature Scheme (XMSS) was proposed by Buchmann et al. [START_REF] Buchmann | XMSS -a practical forward secure signature scheme based on minimal security assumptions[END_REF] in 2011. It is a hash-based digital signature system that is a variant of the Merkle tree scheme and it is forward secure. Recently, IETF has published XMSS as informational RFC (RFC 8391).

The XMSS signature scheme is summarized as follows:

Key Generation:

-Private key: It consists of a cryptographic seed for a pseudorandom function, P RN G. Using P RN G function, create the WOTS Key pair and the leaf index i corresponding to the next W-OTS keys to be used, -Public key: It contains bitmasks and the root node value used in the transitional levels of the hash tree. Signature:

input: message M , the private-key sk and the index i, use the ith W-OTS key pair to sign ith message, -The signature (σ, i, Auth) consists of the W-OTS signature σ, index i, and the leaf node authentication path Auth, the authentication path contains the hash values of H different nodes in the XMSS tree, the contained leaf index i in the XMSS private-key is updated. Verification:

input: a signature (σ, i, Auth), a message M and the XMSS public-key, verify the W-OTS signature σ using the corresponding W-OTS publickey, verify the authentication path by traversing the tree using Auth to obtain P H , -If P H is equal to the root node value in the public-key, the signature is valid.

Performance Evaluation

In this section, we present the developing environment of different post-quantum schemes in smartphone and the obtained experimental results.

Developing environment

Android smartphones are the most used mobiles. Android is an operating system based on Linux kernel and other open-source software; written in Java, C and other programming languages. It was developed by Google for mobile devices. In this work, we develop an PQC benchmark set with the studied schemes by using two cryptographic providers: FlexiProvider [START_REF]FlexiProvider Homepage[END_REF] and Spongy Castle [START_REF]Spongy Castle Homepage[END_REF] which is a repackage of Bouncy Castle for Android platform. This performance evaluation is based on the running time, required memory, and power consumption for each studied scheme.

For the evaluation of the timing and the memory usage, we used the methods of java.lang.System and java.lang.Runtime classes, respectively. To measure the energy consumption of the three processes of each scheme, we used Batterystats which is a tool included in the Android framework that collects battery data on smartphones.

We benchmarked the performance of the previously studied schemes on smartphone Samsung Galaxy A5, model SM-A500H which is equipped with Exynos 7880 Octa-core 1.9 GHz processor, an internal memory with 3072 MB capacity, and a battery with a capacity of 3000 mAh. The Android version installed was 6.0.1 Marshmallow.

Experimental results

NIST [START_REF] Chen | Report on post-quantum cryptography[END_REF] recommends that all data that has to stay secure for more than 10 years should use a minimum of 128-bits security level. Table 1 describes different parameters used for each scheme with 128 security. Table 2 presents the running time (key generation, encryption and decryption) and energy consumption of different encryption schemes. We mention that the measure of consumption energy is for all parties of each scheme. For each post-quantum scheme, the required space in terms of pair key, ciphertext (in encryption schemes) or signature (in signature schemes) and memory usage (RAM) is showed in Table 4. The Niederreiter-CFS signature was the worst performing scheme on the signing process, it used a lot of computational resources and therefore is not suitable for mobile devices.

In the other hand, code-based schemes requires large key sizes especially the McEliece scheme with an astounding 3.8 MB for a key pair. To avoid this limitation, quasi-cyclic and quasi-dyadic variants [START_REF] Berger | Reducing Key Length of the McEliece Cryptosystem[END_REF][START_REF] Misoczki | Compact McEliece Keys from Goppa Codes[END_REF] were designed to offer much lower key sizes. In addition, the generation of pair key is slower than NTRU-Encry. We notice that the code-based cryptography has been recommended by Post-Quantum Crypto Project of Europe because it "has been studied since 1978 and has withstood attacks very well" [START_REF] Daniel | Initial recommendations of long-term secure postquantum systems[END_REF].

When we compared the code-based schemes to the NTRUEncrypt scheme we found that NTRUEncrypt offers a balanced execution speeds for encryption and decryption processes and requires the lowest amount of memory in RAM when performing the operations of encryption and decryption. Moreover, the key length of NTRUEncrypt is smaller, which is suitable with limited resources of mobiles phones. In addition, NTRUEncrypt scheme uses the least amount of energy. NTRUSign scheme is fast in signature generation process and consumes less energy compared to other post-quantum signature schemes. However, NTRUSign is slow in signature verification and it occupies the biggest space of RAM memory (49MB).

The NTRU cryptosystem has been broken by recent attacks that use special structures of the rings used in those schemes. Recently, Bernstein et al. [START_REF] Bernstein | NTRU Prime: Reducing attack surface at low cost[END_REF] proposed a new variant of NTRU, called "NTRU Prime" to avoid the weaknesses of NTRU, and proved that it is IND-CCA2 secure. They submitted this work to NIST's "Post-Quantum Cryptography Standardization Project".

The Rainbow signature scheme provided the shortest signature, 7 times smaller than the NTRUSign and 31 times smaller than the XMSS scheme. It is fast and fairly similar to the NTRUSign scheme in signing operation. Unfortunately, it is the worst scheme in terms of key pair generation time, key pair size, and consumption energy. In security, Rainbow scheme is still uncertain.

The verification speed of XMSS was good. It has small private-and publickey and consumes less energy. However, it provides the biggest signature among post-quantum signature schemes and its signing speed is very slow. Concerning the security of XMSS, Hülsing et al. [START_REF] Hülsing | Mitigating multi-target attacks in hash-based signatures[END_REF] presented a multi-target attack against hash-based signature schemes like SPHINCS and XMSS.

Based on our experimental results and discussion, we summarized the advantages and the limitations of each tested scheme in 5.

Conclusion

In this work, we have presented results of performance benchmarks of implemented different post-quantum schemes on smart mobile devices. Based on the experimental results, we have compared the studied schemes from the viewpoints of computational time, required memory, and power consumption.

In code-based cryptography, the encryption operation is faster than other PQC and it seems to have the most important security, but it uses large keys. Recently, it was improved to generate small key sizes. The optimisation of de- NTRUEncrypt and NTRUSign schemes gave the best results compared to other schemes, it should be noted that NTRUSign uses some improvement for memory consumption when generating digital signatures. The Rainbow scheme is very fast and it generates the lowest signature sizes, but it takes too long to generate its key pair which lead to great energy consumptions. The XMSS scheme has lower key sizes, and acceptable signature generation speeds, however the generated signature is too big and could use some optimizations in that regard. The most posed problem in NTRU, Rainbow, and XMSS schemes is the security, there are various detected attacks in these schemes in the last years.

Finally, we mention that the selection of the post-quantum scheme depends on its performance, its security, and the context in which it will be used.

  The McE Kobara-Imai scheme (Kobara-Imai conversion γ for the McEliece scheme)is defined as follows: Encryption: To encrypt a message m r := Rand, -y1 := Gen(r)⊕(m Const) where Gen(.) is a random number generator and Const is a public constant, -y2 := r ⊕ Hash(y1), -(y5 y4 y3) := (y2 y1), z ← Conv(y4) where Conv(.) a constant weight encoding function, -Output the cryptogram c := y5 y2G ⊕ z. Decryption: To decrypt a cryptogram c -

  c := p * r * h + m mod q -The cryptogram is the polynomial c Decryption: To decrypt a cryptogram c a := f * c where the coefficients of a lie between q/2 and q/2 b := a mod p m := f p * b mod p output m 4 Post-quantum signature schemes

Table 5 .---XMSS--

 5 SchemeAdvantages Disadvantages McEliece-Very fast encryption process -Very large key sizes -High memory usage and energy consumption

Table 1 .

 1 Parameters for post-quantum schemes at the 128 bits security level

	Scheme	Category	Parameter	Reference
	McEliece
	McE Kobara-Imai Niederreiter	Code-based	(n=4096,k=3604, t=41)	[2]
	Niederreiter CFS
	NTRU (Encry and Sign)	Lattice-based	APR2011 439	[22]
	Rainbow	Multivariate-based v1= 36, o1=21, o2=22	[30]
	XMSS	Hash-based	xmss-sha2 10 256	[20]

Table 2 .

 2 Computation speed and consumption energy in encryption schemes

	Scheme	Timing (ms) KeyGen Encryption Decryption	energy (mAh)
	McEliece	320313	11	364	18.3
	McE Kobara-Imai 64478	134	463	6.3
	Niederreiter	27046	157	475	0.6
	NTRUEncry	4052	47	70	0.3

Table 3

 3

	presents the running time (key generation, signature generation
	and signature verification) and energy consumption of different digital signa-
	ture schemes.

Table 3 .

 3 Computation speed and consumption energy in signature schemes

	Scheme	Timing (ms) KeyGen Signature Verification	Energy (mAh)
	Niederreiter-CFS 63115	>1h	>1h	-
	NTRUSign	4052	262	491	6.3
	Rainbow	855157	276	77	42
	XMSS	352714	486	199	18

Table 4 .

 4 Required space for each schemeIn this work, we evaluate three variants of schemes based on coding theory, McEliece,McE Kobara-Imai, Niederreiter, and Niederreiter CFS. McEliece encryption scheme has high-speed encryption process compared to other postquantum schemes. Moreover, it is easy to implement as it does not need any cryptographic processor. However, McEliece scheme consumes an important amount of energy (18.3 mAh). McE Kobara-Imai which is the CCA2 secure conversion of McEliece scheme, led to slower results because of the added steps to the encryption procedure. Niederreiter scheme performs better than the McEliece one when it comes to the timing of the decryption process. It generates the lowest ciphertext sizes, 8 times smaller than any other encryption scheme, which make it the best scheme when storing or transmitting the ciphertext. Moreover, McE Kobara-Imai scheme and Niederreiter scheme consume less amount of energy.

		Scheme	pair key(kB) private key public key signature (Byte) usage (MB) ciphertext or memory
		McEliece	1851.61	1802.05	512	33.57
	Encryption	McE Kobara-Imai 256.29 Niederreiter 40.074	218.264 216.744	512 62	9.20 2.76
		NTRUEncry	0.67	0.59	604	0.22
		Niederreiter-CFS	40.07	216.74	-	-
	Signature	NTRUSign Rainbow	0.67 156.24	0.59 136.359	604 82	49.01 4.82
		XMSS	2.264	0.063	2499	5.61
	6 Discussion


Sami Hyrynsalmi
email: sami.hyrynsalmi@tut.fi

Eriks Klotins
email: eriks.klotins@bth.se

Michael Unterkalmsteiner

Tony Gorschek

Nirnaya Tripathi
email: nirnaya.tripathi@oulu.fi

Leandro Bento Pompermaier
email: leandro.pompermaier@pucrs.br

Rafael Prikladnicki
email: rafael.prikladnicki@pucrs.br

What is a Minimum Viable (Video) Game? Towards a research agenda

Keywords: Minimum viable product, Minimum viable game, Game business

The concept of Minimum Viable Product (i.e., MVP) is largely adapted in the software industry as well as in academia. MVPs are used to test hypotheses regarding the target audience, save resources from unnecessary development work and guide a company towards a stable business model. As the game industry is becoming an important business domain, it is not surprise that the concept has been adopted also in the game development. This study surveys how a Minimum Viable Game (MVG) is defined, what is reported in extant literature as well as present results from a small case study survey done to nine game development companies. The study shows that despite popularity of MVG in the industrial fora, the presented views on the concept are diverged and there is lack of practical guidelines and research supporting game companies. This study points out research gaps in the area as well as calls for actions to further develop the concept and to define guidelines.

Introduction

A key characteristic of the modern business world is speed [START_REF] Davis | Blur: The Speed of Change in the Connected Economy[END_REF]. An often repeated argument claims that the pace of business is going to even increase in the future [START_REF] Bosch | Speed, Data, and Ecosystems: Excelling in a Software-Driven World[END_REF][START_REF] Järvinen | From agile software development to mercury business[END_REF]. The requirement for speeding up is visible also for start-ups, which aim to quickly find a repeatable and scalable business model [START_REF] Blank | The Startup Owner's Manual: The Step-By-Step Guide for Building a Great Company[END_REF]. As it is frequently stated, start-ups are lacking time and resources [START_REF] Giardino | What do we know about software development in startups[END_REF], and therefore, it is crucial for them to find quickly whether their product or service is suitable for markets.

In the beginning of the 2010s, the so-called Lean Startup methodology (LSU, hereafter), by Eric Ries [START_REF] Ries | The lean startup : how constant innovation creates radically successful businesses[END_REF], started to gain popularity. The methodology aims to through continues hypothesis-testing to validate whether there are business potential in the product or service. One of the methodology's cornerstones is the concept of Minimum Viable Product (MVP, hereafter). The concept was first defined already in 2001 by Frank Robinson (c.f. [START_REF] Lenarduzzi | MVP explained: A systematic mapping study on the definitions of minimal viable product[END_REF]), yet it become widely known with the raising popularity of the LSU [START_REF] Blank | Why the lean start-up changes everything[END_REF]. According to Ries 1 , "the minimum viable product is that version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort. " 2

The MVP concept can be seen as continuum of Blank's [START_REF] Blank | The Four Steps to the Epiphany: Successful Strategies for Products that Win[END_REF] Customer development methodology. In this methodology, Blank's key advice, for new entrepreneurs, is to face potential customers early upon-informally referred as the 'Get out of the building' principle [START_REF] Blank | The Startup Owner's Manual: The Step-By-Step Guide for Building a Great Company[END_REF][START_REF] Hokkanen | From Minimum Viable to Maximum Lovable: Developing a User Experience Strategy Model for Software Startups[END_REF]-to collect valuable information on what customers wants and is willingly to pay for. As the objective of the LSU is to avoid of building a complete product before learning that it is not valuable for the customers [START_REF] Ries | The lean startup : how constant innovation creates radically successful businesses[END_REF], the concept of a minimum viable product fits well to test the suitability of product before it has been finished. In addition, it allows a company to redirect its product and strategy to areas that are the most fruitfull for the company.

As the global video game industry is estimated to reach the total revenue of over USD 80 billion 3 by the year of 2018, it is not surprise that also game development companies and advocates have been interested towards the MVP. For example, a senior manager [START_REF] Blank | The Startup Owner's Manual: The Step-By-Step Guide for Building a Great Company[END_REF] of Zynga Inc. revealed in an interview that FarmVille , a hugely popular Facebook game, was initially launched as a 'minimum viable product'. In addition, the popular Pokémon Go mobile game, by Niantic Inc., is in the grey literature often claimed 5 to be an example of a successful MVP usage in the video game industry.

However, there are a few remarkable differences between utility and game software that should be acknowledged when the concept of MVP is adapted to new domains. First, while utility software solutions can be used for several years or even decades without major changes, the expected life-cycle of a computer game is much shorter. For example, the average length to complete the ten most popular games in the HowLongToBeat 6 service range from around 20 hours to a bit over 100 hours.

While there are of course games which core game play can be repeated ad infinitum , games are, on average, disposable products that have to impress gamers quickly. Second, whereas the utility software-such as a spreadsheet programme or a fitness tracker-should aim to help users in all possible ways, the games, instead, should aim to challenge the users [39]. Yet, too challenging as well as too helpful games create frustration [START_REF] Smed | Towards a definition of a computer game[END_REF]. Thus, the games aim to reach a flow experience [START_REF] Csikszentmihalyi | Flow: The Psychology of Optimal Experience[END_REF]. However, this should be done with little or no delay as a gamer might got annoyed and switch game to another. Thirdly, the user experience and playability-i.e. overall quality of gameplay-is everything. Whereas small irritating mistakes in a utility software can be tolerated for years, these can be fatal in a video game.

These contradictories between video games and utility software turns our focus on the use of the minimum viable product concept in the game industry. The MVP was initially promoted for utility software solutions where adding a new feature or pivoting product's strategy would be more feasible approach than in the game industry. While MVPs in the software industry have been previously addressed (e.g. [START_REF] Hokkanen | From Minimum Viable to Maximum Lovable: Developing a User Experience Strategy Model for Software Startups[END_REF][START_REF] Lenarduzzi | MVP explained: A systematic mapping study on the definitions of minimal viable product[END_REF][START_REF] Nguyen-Duc | Minimum viable product or multiple facet product? the role of mvp in software startups[END_REF]), to the best of authors' knowledge, only a little of using MVPs-or Minimum Viable Games (MVGs, hereafter)-in the game industry discussed in the extant literature (e.g. [START_REF] Järvi | Game development accelerator -initial design and research approach[END_REF][START_REF] Rosenfield Boeira | Mvps: Do we really need them? In: Lean Game Development: Apply Lean Frameworks to the Process of Game Development[END_REF]).

The objective of this paper is to study how computer game start-ups perceive and use MVPs in their businesses. The study's approach is explorative, aiming to shed light on the current practices and extant knowledge. We use a recent structured review as a starting point and present an unstructured literature review on the area. As empirical data, we use a large global survey done to software start-ups (c.f. [START_REF] Klotins | Software engineering in start-up companies: An analysis of 88 experience reports[END_REF]). In total, the survey received 86 start-up stories and more than 23 000 data-points. In this study, we focus on the subset of the start-up companies identified to be video game companies. empirical observations from using MVPs in the game industry. Based on the problems arisen from the empirical material as well as from the extant literature, we discuss needed future research actions and lay some building blocks for further inquires.

The remaining of this paper is organised as follows. Section 2 presents a review of literature as well as positions our view of a MVP. It is followed by a description of empirical research process used as well as an overview of the result in Section 3. Discussion is presented in Section 4 and finally, Section 5 concludes the study.

Review of Literature

According to the recent systematic mapping study on the evolution of the minimum viable product concept by Lenarduzzi and Taibi [START_REF] Lenarduzzi | MVP explained: A systematic mapping study on the definitions of minimal viable product[END_REF], the concept was first discussed by Frank Robinson as early as 2001 and then brought forth and popularised by Eric Ries and Steve Blank. Altogether, Lenarduzzi and Taibi [START_REF] Lenarduzzi | MVP explained: A systematic mapping study on the definitions of minimal viable product[END_REF] found 22 different definitions for a MVP which were mostly published between 2012 and 2015. However, they also pointed out that most of those definitions were not referred or used by other than by the authors themselves.

In this study, we follow the market-oriented line of definitions, e.g., those given by Ries as a part of the LSU methodology. For instance, in his book, Ries [31, p. 93] elaborated MVP's concept more by stating that it "[i]s not necessarily the smallest product imaginable; it is simply the fastest way to get through the Build-Measure-Learn feedback loop with the minimum amount of effort." Also, Moogk [START_REF] Moogk | Minimum viable product and the importance of experimentation in technology startups[END_REF] emphasise the value of a MVP to a new venture for engaging new customers and testing the product's value proposition.

In addition, it is worth to note that we draw a distinction between a prototype and a MVP. In our use, the former refers a product version that is used to test the product inside the organisation. The latter refers to a product version that is used to collected validated learning from potential customers and users. That is, in our use, an MVP aims to maximise learning from real customers-and therefore, from the market-with the minimum amount of effort needed. However, in the extant literature, differences and similarities between these two concepts are not always clear (c.f. [START_REF] Lenarduzzi | MVP explained: A systematic mapping study on the definitions of minimal viable product[END_REF]).

In the extant literature, several studies have focused on different aspects of minimum viable products. For instance, Hokkanen [START_REF] Hokkanen | From Minimum Viable to Maximum Lovable: Developing a User Experience Strategy Model for Software Startups[END_REF] studied interweaving user experience design to the development of a MVP. Hokkanen, Kuusinen and Väänänen [START_REF] Hokkanen | Minimum viable user experience: A framework for supporting product design in startups[END_REF] presented a framework for start-up companies for design Minimum Viable User Experience (MVUX). In addition, Münch et al. [START_REF] Münch | Creating minimum viable products in industry-academia collaborations[END_REF] studied the creation of MVPs in industry-academia collaboration and Bosch et al. [START_REF] Bosch | The early stage software startup development model: A framework for operationalizing lean principles in software startups[END_REF] as well as Björk et al. [START_REF] Björk | Lean product development in early stage startups[END_REF] have discussed on the usage of MVP as a part of software development model in early-stage start-ups. Furthermore, in a large survey to more than 4,000 companies, Wang et al. [START_REF] Wang | Key challenges in software startups across life cycle stages[END_REF] found that early-stage start-ups list MVP as one of the key challenges.

However, a number of empirical studies reporting the use of MVPs in the industry is small. For example, Ngueyn-Duc and Abrahamsson [START_REF] Nguyen-Duc | Minimum viable product or multiple facet product? the role of mvp in software startups[END_REF] studied the usage of MVPs in five software start-ups and showed that the concept seems to be useful as a design artefact, a boundary spanning artefact as well as a reusable artefact. Khanna, Ngueyn-Duc and Wang [START_REF] Khanna | From mvps to pivots: a hypothesisdriven journey of two software startups[END_REF] reported a case study on how two start-ups selected hypotheses they wished to test with their MVPs. The study showed that the relationships between business goals and hypotheses tested in MVPs are complex in their cases.

When the focus is narrowed down to the game industry and in the domain of computer games, the research in new companies and product launches are remarkable scarcer. This is a clear contradictory to the practice as games form a major part of, e.g., most installed and sought mobile applications [START_REF] Hyrynsalmi | Letters from the War of Ecosystems -An Analysis of Independent Software Vendors in Mobile Application Marketplaces[END_REF] as well as are located among the most used mobile applications [START_REF] Böhmer | Falling asleep with Angry Birds, Facebook and Kindle: A large scale study on mobile application usage[END_REF]. Furthermore, there is lack of academic research about using MVPs in the game development. However, there is a series of work in grey literature devoted to characterise and guide on using MVPs.

For example, Tyler York 7 discusses on applicability of LSU methodology and MVPs in the game industry. In addition, some companies has reported their company's approach on using a MVP 8 ,

9

. However, also alternative views and critique have been presented -the director of developer relations of Chillingo 10 , the publisher of mobile games such as Angry Birds and Cut the Rope, stated that it is disrespectful from game developers to publish unfinished products and force gamers to wait for major features. In addition, while Pokémon Go has been praised as an example of publishing a MVP game in the grey literature, it has also been labelled to be a launch failure due to technical problems and lack of content early upon 11 . In the extant literature, only Järvi et al. [START_REF] Järvi | Game development accelerator -initial design and research approach[END_REF] and Rosenfield Boeira [START_REF] Rosenfield Boeira | Mvps: Do we really need them? In: Lean Game Development: Apply Lean Frameworks to the Process of Game Development[END_REF] have discussed on the use of MVPs in the game development. Järvi et al. [START_REF] Järvi | Game development accelerator -initial design and research approach[END_REF] presents a conceptual design for video game start-up accelerator and discusses on central concepts related to the accelerator program. They define a minimum viable game as a product version "that implements the core game mechanics leaving out everything else. In addition, they state that "games are holistic products and it is not trivial to know what contributes to the players' experience and what can be left out of the game". Finally, they also emphasise the difference between the MVG and MVP concepts as adding and removing new features in the latter is easier than in the former. In the case of MVG, the authors underlines the importance of keeping the gameplay in balance with every changes made.

Rosenfield Boeira [START_REF] Rosenfield Boeira | Mvps: Do we really need them? In: Lean Game Development: Apply Lean Frameworks to the Process of Game Development[END_REF] discusses how MVPs can be used in the gaming industry and gives advice for the practitioners. The study notes the concepts of MVP and MVG, yet it presents a minimum viable game as a synonym for a MVP in the video game industry. Furthermore, Rosenfield Boeira [START_REF] Rosenfield Boeira | Mvps: Do we really need them? In: Lean Game Development: Apply Lean Frameworks to the Process of Game Development[END_REF] draws a line between a MVG and a prototype, stating that MVG aims to deliver value to the client whereas a prototype delivers to the development company. That is, the prototype focuses on technical and the MVG to the commercial viability. For the sake of simplicity, we follow this divisions in the remaining of this study.

A classic example, often seen in the grey literature, is to consider the most minimum viable game of Super Mario Bros . Its minimum viable version, according to e.g. [START_REF] Rosenfield Boeira | Mvps: Do we really need them? In: Lean Game Development: Apply Lean Frameworks to the Process of Game Development[END_REF], would include only abilities for walking and jumping over the holes. While the basic game mechanism can be tried with this kind of a version, the question remains what kind of validated learning a game company would be able collect. The skeleton, left after stripping off everything extra, would be a yet another sidescrolling platform video game and we argue that testing it with customers would not bring much new information. To depart from the existing side-scrollers, the minimum viable version of a new game should contain enough content to validate whether the 'core fun' is exiting and enticing. Similarly for a totally new kind of a game, there should be enough content to test whether the hypothesis on the core gameplay mechanism hold.

To summarise our review of extant literature, it can be stated that an important question remains mainly unanswered. That is, what would be a minimum viable game for gathering the maximum amount of validated learning with minimum amount of extra effort? For example, music and audio are an important part of any 11 Derek Anderson. [START_REF] Blank | Why the lean start-up changes everything[END_REF] Steps for Creating a Minimum Viable Product (and the case of the Pokémon GO app). https://sphereinc.com/5-steps-for-creating-a-minimum-viable-productand-the-caseof-the-pokemon-go-app/. Last accessed June 26, 2018.

game experience, yet they might not be an essential part for validating the core mechanisms and fun factors of a game. As our review of extant literature show, not much has been done regarding this area. Therefore, the remaining of this article will first focus an empirical survey on whether video game start-ups are utilising MVGs in their work. Finally, we identify and present research gaps regarding the concept of MVG in the game industry.

Empirical Research

Research Process

This study uses a case survey method as a tool for data collection [START_REF] Larsson | Case survey methodology: Quantitative analysis of patterns across case studies[END_REF]. The case survey method combines case study with a survey [START_REF] Klotins | Using the case survey method to explore engineering practices in software start-ups[END_REF][START_REF] Petersen | Choosing component origins for software intensive systems: In-house, cots, OSS or outsourcing? -A case survey[END_REF], thus allowing an in-depth analysis of a larger number of cases. For this study, we used a large questionnaire designed to survey development practices in start-up companies. During the design phase, the survey was externally reviewed by 10 researchers and it was tested with four software start-ups [40].

The final questionnaire contains 85 questions in 10 sections 12 . The questionnaire was globally marketed to former and current software startups in social media, industrial events and by personal contacts. In addition, respondents were recruited with the help of, e.g., Software Start-up Research Network 13 . The data was collected from December 2016 to June 2017. In total, the questionnaire received 86 usable responses from Europe, North-and South-America as well as from Asia. Some initial results of the survey on the technical debt's impact on software start-ups have been reported in [40].

For this study, all responses of the survey were went through by two researchers independently. From the set, they selected game companies for further analysis. A game company is defined as an organisation which main product or service is a digital game. The game can be for, e.g., educational or purely a leisure purpose. Finally, the selected case companies and their approaches to the MVGs were analysed.

12 http://startupcontextmap.org/expsurvey/woifenw2 13 https://www.softwarestartups.org/

Companies

Out of the all response cases, nine were categorised to work in the field of game development. In Table 1, a short description of each company is given. In the following, each company's development approach is briefly described.

Company A produces animations and related educational games for pre-school aged children. They have been able to stabilise their product and market size. Their products were mainly based on own ideas as well as analysis of similar educational games. The company followed a Waterfall-like development process and acquired most of the software development work from subcontractors. The company does not report of using prototypes or any kind of MVGs.

Also Company B works in the educational game sector; however, their focus is on school-aged children. They have not published their product yet to the market. Requirements ideas are based on market research, literature review and user studies. Customer interviews were video recorded and used for requirements validation. The company uses user-testing for its prototypes and an agile development model, but they did not have formal plans or actions for any kinds of minimum viable versions.

Company C develops mobile games for modern smart phone platforms. Ideas for their own games are gathered by following market trends and recent developments. In addition, the company does work-for-hire game development for other companies. Feedback is gathered with face-to-face interviews with players. The company uses 'MVPs' for collecting feedback from related partners such as publishers and partners. Company C acknowledges the need for using 'MVPs' to test fun and core game mechanisms of the game.

Company D works also with casual mobile games. Their main product has already reached a stable market position. Also they gather ideas for their own games by following market trends and development. The company uses prototypes but does not actively utilise customers in requirements validation; however, they use a selection of modern mobile game metrics to measure the game.

Also Company E develops mobile games and they, similarly, report of using market trends and previous experience as a source for the requirements ideas. The company users 'MVP' approach in its development, yet they the scope of the 'MVP' is based on a gut feeling. The company uses A/B testing for fine-tuning KPIs while the product is measured based on profit metrics (such as average revenue per users).

Company F develops a game for easy and fast gambling. On the contrary to the other cases, they did not report of using similar products as a starting point for requirements engineering. However, the company extracted requirements from brainstorming sessions as well as observations, crafted a MVG and tested the game. In addition, the company reports of using customer interviews for feature prioritisation.

Company G develops games for a smart phone platform, focusing on people, who want games that requires ability to succeed, but who does not have time. They have already published two titles. The company also uses similar products as a baseline for requirements gathering. They also report of using Lean start-up and Agile methods. However, while the company demonstrate prototypes to customers, they do not report of using actively any kinds of MVGs.

Company H builds an educational game, for a mobile platform, targeted to children aged from 7 to 12. They have not yet published the product. The company uses benchmarking against published solutions and brainstorming for creating requirements. The idea is tested with a MVG and target audience. The company reports prioritising game-play features and using MVGs to verify that core of the games has been captured successfully.

Company I focuses on development of video game to casual and hardcore players. The company uses a wide range of techniques to gather ideas and document requirements. In addition, they are verified with internal prototyping and using A/B testing when features can be verified with customers.

Analysis

There are some observations that should be noted from the case companies. Firstly, all of the companies reported that they had at least adequate experience working with similar kinds of products. For example, Company C reported that all members had worked earlier in different game development projects. Thus the case selection can be considered consisting of experienced developers and possible deviations due to inexperienced developers should not be present.

Secondly, only two companies ( F and H ) reported explicitly that they had used minimum viable products to test the game mechanisms and flow with the target audience. In addition, Company E and Company I are using a A/B testing to fine-tune different aspects of a mature product, such as e.g. monetisation. While A/B testing can be used as a part of a MVG, in these cases the testing seems to be used in a mature product; thus, it seems not be used to guide the development of the product.

However, several companies reported that they had created prototypes as a part of their development process, yet those prototypes where used mainly for in-house testing and testing with "a small group of friends from the industry" as reported by Company C. Nevertheless, MVGs, as vehicles of gathering validated information from the target audience, are infrequently utilised by the sample companies.

Furthermore, CompanyC and CompanyE reported that they had developed an 'MVP', but only for in-house use. Thus, this is more closely related to a in-house prototype than a product version used to gather validated information from the target audience. Nevertheless, this emphasises the confounded views on the concept in the field.

Thirdly, the few companies, that used metrics to measure their products, reported that they measured the successfulness of by revenue and number of downloads. This is in line with findings by Koskenvoima and Mäntymäki [START_REF] Koskenvoima | Why do small and medium-size freemium game developers use game analytics?[END_REF] who found that game start-ups do not use analytics as a source of competitive advantage but rather a risk management tool. However, it also shows that the case game companies do not heavily rely on the LSU methodology, as the use of using analytic tools together with MVP listed as the corner stones of the methodology. In addition, to fully utilise the potential of MVGs, the companies should measure and analyse different aspects of customer behaviour.

Discussion

Key observations

We recapitulate our key observations from the study in the following:

-Review of literature shows that there is disputed view on the concept of 'minimum viable product'. On one hand, MVP is used to refer to any kind of a prototype. On the other hand, the concept refer to a product version used by the target audience in order to gather validated learning. -Similarly, in the context of game industry, grey literature often emphasise a MVP-or a MVG-as the most simplistic version of the game. That is, this reflects more a technical prototype used in-house to test the game mechanism than a version offered to the target audience. -There seems to be lack of discussion of minimum viable games and their development in the computer game field. However, there is a series of work devoted to game design [START_REF] Rouse | Game Design Theory and Practice[END_REF][START_REF] Salen | Rules of Play: Game Design Fundamentals[END_REF][START_REF] Schell | The Art of Game Design: A book of lenses[END_REF]]; yet, the focus is on designer's perspective and often omits, e.g., financial aspects. -The empirical inquiry revealed that although a few of the case companies were indeed using a MVG-like approach with the target audience, a clear majority of game companies were not. Instead, they were using more traditional style leaded by development of a technical prototype and its in-house testing.

To summarise the observations, there is surprisingly little done for the defining and guiding of designing MVGs in the game industry. Furthermore, the empirical inquiry shows that case companies are not fully utilising MVGs in their game development.

Towards a formation of a research agenda

Our key argument is that game development companies and the game industry would benefit from the usage of MVGs for gaining validated learning with minimum amount of effort. This would allow saving resources from unnecessary development tasks. Yet, there is a little of academic work done; thus, this area offers clear research gaps and fruitful questions for the researchers. In addition, while there is a growing interest towards software development practices in software development companies (c.f. [START_REF] Giardino | Software development in startup companies: The greenfield startup model[END_REF][START_REF] Unterkalmsteiner | Software startups -a research agenda[END_REF]), there is hardly any work done on investing software development practices in start-up game companies.

The game industry's special characteristics make the field challenging. For example, video game development can be seen as a cooperation of three different viewpoints [START_REF] Hakonen | Learning to make computer games: An academic approach[END_REF]: design or humanistic perspective, business perspective, and constructive or software engineering perspective. Therefore, it differs remarkably from regular software engineering projects and, for example, favours 'cowboy coders' more than traditional software engineering projects [START_REF] Murphy-Hill | Cowboys, ankle sprains, and keepers of quality: How is video game development different from software development?[END_REF]. It is also worthy to note that a 'minimum viable game' needs to intertwine all these three perspectives together.

First, there are existing literature of game design patterns and best practices [START_REF] Bjork | Patterns in Game Design. Charles River Media game development series[END_REF], yet testing the game mechanisms with the target audience is an important step [START_REF] Schell | The Art of Game Design: A book of lenses[END_REF]. As discussed above, the game cannot be too challenging or too easy, and it should aim to reach the flow experience. In addition, the core game mechanism should lure players. While the basic game mechanisms can be tested with, e.g., pen and paper, defining what is needed for a minimum viable game is much more complex task. For example, should music and audio be included for evaluating the flow of a MVG?

Second, the game should also be financially viable; it should be able to earn enough profit to cover its development costs and provide a stable revenue stream to the development company. During the last years, different kinds of freemium models have gained popularity among the mobile game developers [START_REF] Hyrynsalmi | Revenue models of application developers in Android Market ecosystem[END_REF]. In addition of testing playability, a MVG should also help to address different financial aspects of the game. For example, is there a market for this kind of a game? Third, construction perspective involves questions from the software engineering field and research of software start-ups (c.f. [START_REF] Unterkalmsteiner | Software startups -a research agenda[END_REF]). For example, how to build a MVG efficiently with existing components? However, as discussed in Section 2, even the concept of MVP is still a infrequently addressed theme in the extant literature and further work is needed for understanding how a MVG should be effientcly created.

Finally, this study request formation of a research agenda to address the different facts of minimum viable games as well as briefly discusses on a few starting point for this work. This study is naturally limited with a small sample of case study companies found as well as the use of unstructured literature review. Nevertheless, this study shows that not much academic work have been done in this domain.

Conclusion

This study surveyed extant academic and grey literature on the concept of a minimum viable product in the game industry, that is minimum viable games. The results show that the concept is used in various ways from a synonym to a in-house prototype to a vehicle to test customer hypotheses. In addition, the results of a case study survey on nine start-ups were used to show that also new game companies are infrequently using MVGs in their work development work. Yet, a few companies were successfully utilising MVGs,

indicating that the concept could be useful also for a larger number of companies.

Thus, this study request a research agenda to be formed to further advance this research area.