Senay Sadic
email: senay.sadic@antalya.edu.tr

Jorge Pinho De Sousa

José Crispim

ICT Platform Design for SME Collaboration

Keywords: SME Collaboration, ICT Platform, Process Flow

Collaboration is frequently used both in literature and in practice as a sustainability and survival strategy for SMEs. In this study, we propose an ICT Platform to support SME Collaboration in discrete complex manufacturing industries. The proposed ICT Platform is defined by a conceptual platform and functional and process flows. Initially an SME network vision is set with three dimensions; sustainability, growth, and survival. And then, a Balanced Scorecard application has been performed to translate the SME network strategy to operational level ICT initiatives. Finally based on the guidance received from the literature and the established ICT initiatives, a set of ICT tools were created for the business model. These tools include a conceptual framework and the characterization of functional, informational and process flows to support the business model.

Introduction

Small and Medium Enterprises (SMEs) represent a high percentage of the world's economic power. Forming collaborative networks is frequently addressed as a survival and sustainability tool for SMEs in the global markets [START_REF] Camarinha-Matos | Collaborative networked organizations -Concepts and practice in manufacturing enterprises[END_REF], [START_REF] Carneiro | Reference model for collaborative manufacturing of customised products: applications in the fashion industry[END_REF]. By joining their resources and competencies through networked manufacturing, SMEs can reach a larger dimension, access global markets, share risks, and nurture innovation through collaborative product development [START_REF] Camarinha-Matos | Collaborative networked organizations: Status and trends in manufacturing[END_REF]- [START_REF] Afsarmanesh | On management of 2nd generation Virtual Organizations Breeding Environments[END_REF].

The need to align business strategy with ICT strategy and development was highlighted frequently in the literature [START_REF] Coronado | A framework to enhance manufacturing agility using information systems in SMEs[END_REF], [START_REF] Gunasekaran | Information systems in supply chain integration and management[END_REF]. While ICT development starts from the operational level and builds through tactical and strategical levels, strategy setting starts from the strategic level and is translated to tactical and operational levels [START_REF] Gunasekaran | Agile manufacturing: A taxonomy of strategic and technological imperatives[END_REF]. In this context, in order to develop efficient operational level ICT tools, it is therefore required to clearly translate strategic objectives into implementable operational initiatives.

In an extensive literature review, we have not come across any studies, which relate the company's business strategy with operational level ICT initiatives. Some of the reviewed papers work on the integration of the operational level, with no evidence of strategy concerns or they rather follow an incremental approach, where they initially develop business architecture and then create more focused decision support tools.

Our main observation is that, while the theoretical literature continuously repeats the need for strategic and operational alignment and for business strategy and ICT strategy alignment, in practice and in general, applications are very limited and deceiving. On the other hand, the literature on Collaborative Networks mainly covers research to guide real life applications and focuses on developing practical tools to support inter organizational collaboration. Organizations are looking for methodologies to support a high level of integration and since collaboration brings many immediate benefits to all partners, the development of a long-term vision and of a strategy has been ignored.

In this work, we have presented the methodology that supports the design of ICT platform to support a Collaborative network. The methodology consists of three main steps: Initially the SME network vision was set as sustainability, survival, and growth. Later by implementing a balanced scorecard approach, the vision was translated into ICT initiatives, which create the base for a conceptual framework. Finally, the functional and process flows of the ICT Platform are designed and presented.

2

The Proposed Business Model

The proposed collaborative business model, designed to support SMEs functioning in discrete complex manufacturing industries, is composed of two organizational layers: SME Network and Dynamic Manufacturing Network (DMN).

While an SME network is the first organizational layer (strategic network), the DMN constitutes the second layer (operational net-work). SME networks are strategic partnerships of autonomous SMEs that collaborate to reach joint goals and they precede DMN formation. SME Networks provide long-term integration between network members, support their healthy operation, maintain trust and fairness between members, and develop strategies to manage the operational level decisions. The DMN, as the second layer of the business model, is defined as a temporary or long term collaborative network, that counts on joint manufacturing efforts of geographically dispersed SMEs and/or OEMs [START_REF] Markaki | Benefits and Risks in Dynamic Manufacturing Networks[END_REF], [START_REF] Viswanadham | Partner Selection and Synchronized Planning in Dynamic Manufacturing Networks[END_REF]. DMNs are formed to satisfy specific business opportunities (either one time or repetitive) and dissolve once the orders are delivered.

Figure 1 presents the proposed business model, which functions as an intermediary between the customer and the manufacturer sides of the industry. The customer side is integrated through an e-commerce module, and a sell side marketplace is developed for customer communication.

On the manufacturing side, DMN formation and operational planning require integrated business processes and an automated, collaborative ICT platform. This collaborative platform needs to be built in order to assist the DMN life cycle, to support SME network decision making, and to monitor order processing. The collaborative platform can be used simultaneously by several DMNs that are designed to fulfill different business opportunities.

The proposed business model requires automated processes to assist the DMN life cycle, and the business functions to support SME net-work decisions. A DMN works at the operational level and requires detailed focused decision support tools to enable and optimize its operations. In this context, an ICT platform should both support the back end and the front end of the whole supply chain, should facilitate interoperability among autonomous members, should enable communication flows within the network, and should assist business processes through the DMN life cycle [START_REF] Liu | A case study of an inter-enterprise workflow-supported supply chain management system[END_REF].

Fig. 1. SME Network and DMN Business Model

Moreover, the development of ICT tools is necessary to decrease decision making time and increase operational efficiency. ERP applications provide control over shop floor operations but they do not provide a means to link the autonomous network members.

From Strategy to ICT Initiatives: A Balanced Scorecard Application

Several researchers have discussed the need for alignment of the business strategy and the ICT strategy. It is claimed that identifying a sound business strategy is the key for business process agility [START_REF] Coronado | A framework to enhance manufacturing agility using information systems in SMEs[END_REF]. To successfully support business processes, the ICT strategy also needs to be aligned with the business strategy. An automated network needs to initially define its business requirements which will lead to a business architecture to be further supported by an ICT infrastructure [START_REF] Gunasekaran | Information systems in supply chain integration and management[END_REF]. Business model development and goal setting are clearly the basis for developing a correct information technology infrastructure.

While strategy development needs to start at the strategic level, by SME network goal setting and strategy setting, process integration needs to start at the operational level by developing a set of automated collaborative processes. In terms of process integration, as we go from bottom to top, the level of integration decreases and tools move from detailed mathematical decision support systems to conceptual frameworks or reference models. On the other hand, in terms of strategy setting, decision makers need to first decide the strategy of the SME network and later, develop ICT tools at the operational level, by translating that strategy to operational goals. In order to create successful collaborative networks, the business strategy should be integrated into the development of ICT tools and decision-making methodologies.

In order to align ICT design with business strategy, we defined the SME Network vision, and translated it into ICT strategies. The SME network vision is initially grounded on three components: sustainability, survival, and growth. While survival is the act of standing against economic crisis and other disturbances in the system, sustainability stands for withstanding internal organizational challenges. Growth, on the other hand, stands for the expansion of the SME network along time. This vision is then translated into operational level ICT initiatives through the Balanced Scorecard methodology.

The Balanced Scorecard (BS) framework has been widely used as a strategic management tool. Since it was proposed in the 1990s, it is has been used to measure and manage four aspects of organizational performance: Financial, Customers, Internal Business Processes, and Learning and Growth.

BS allows decision makers to extend their myopic, only financially focusedperspective, to other decision dimensions and stakeholders. In BS development, all four perspectives are guided through four major steps:

1. objectives clarify the company vision and translate it into a strategy; 2. measures provide quantitative indicators for each objective, and allow decision makers to link objectives with results; 3. targets allow decision makers to set specific goals, through long term or short term quantitative or qualitative goals; 4. initiatives recommend some actions that can be taken in order to reach identified targets for each objective

We have adapted the BS methodology by focusing on our three different vision components: sustainability; growth; and survival. We have connected each vision to one or more balanced scorecard perspective as follows: Sustainability to internal business processes perspective; growth to customers and financial perspectives; and survival to learning and improvement perspectives. Due to page limitations only sustainability balance scorecard will be presented in this section.

Table 1 presents the developed Sustainability Balanced Scorecard. In order to maintain the group cohesion and harmony required to sustain the collaborative network, the following sustainability objectives were developed: supporting conflict resolution between members; establishing high trust value, establishing high reliabil-ity value, establishing high fairness value and providing a membership management function.

Table 1. Sustainability Balanced Scorecard

These objectives have guided us to identify the following ICT initiatives: set and track measures for each group cohesion component; provide reporting for network decisions and actions; develop pre-membership agreements; develop fair sharing mechanisms; develop member performance module; and develop a member association/dissociation DSS. The ICT initiatives will later be classified in a conceptual framework, which will support the design of the ICT platform. A conceptual framework draws the outline for business models by defining a number of sub-models, collections of templates, procedures, methods, rules and tools.

OBJECTIVES MEASURES TARGETS IT INITIATIVES ICT TOOLS

The outputs of the Balanced Scorecards (from the previous section) have been grouped in order to create the conceptual framework. The developed framework covers three main functions: SME network support functions, e-commerce functions and DMN support functions. Due to page limitations, conceptual framework is only briefly mentioned.

ICT Tools

Based on the ICT initiatives, conceptual framework and literature review findings, we have developed a set of ICT tools to assist the business model. We propose here an organization of the functional flows as follows: Order Promising; DMN Life Cycle Management; Customer Relations; Membership Management; and Group Cohesion Management. Fig. 2. shows the functional flows of the ICT platform and Fig. 3. presents the process flows of the system. The overall process of operational in an SME network starts with a customer interaction through the e-marketplace. The production system operates under an Available to Process (ATP) strategy. Once the e-marketplace receives a new customer order, the order-promising module will be triggered, in order to check order feasibility both in terms of available capacity and required competencies. Online partner and order information will be extracted via the DMN Collaborative Platform. After the Order Acceptance submodule confirms acceptance of an order, this order will be combined with other orders for classification and prioritization.

The Order Prioritization submodule will compute order priorities, via a multicriteria decision making tool. Order priorities will be utilized in the DMN Creation submodule, so that orders that are more valuable are processed first. On the other hand, the Order Classification submodule will compute order classes through data mining and data science approaches. Order classes can be used in strategy and promotion development for different order classes. These modules will be fed with information on order characteristics (due date, volume, processing time, etc.) and on customer characteristics.

In the DMN Creation submodule of DMN Life Cycle Management module, a multi-objective mathematical model is employed to decide DMN configuration and to compute the production and transportation lot sizes. The model will use several objectives such as cost, flexibility, partner reliability, order priority or operational risk and will take into account partner capacities, capabilities, order priorities, and costs. The order priorities generated by the Order Prioritization submodule and customer priorities calculated by the Customer Prioritization submodule, will also be considered in the DMN formation process.

Fig. 2. Functional Flows

Since DMNs typically serve to a group of distinct customers, it is a good strategy to take into account customer characteristics during DMN formation. In order to enable the formation of customer and order driven DMNs, the Customer Relations module will provide its input on customer priorities and customer segments. At this stage, the DMN Risk Management submodule of DMN Crisis Management uses mathematical tools to predict operational risks related to DMN processes, and integrates the results to the DMN creation process. Once the DMN configuration and operational plans are set, job orders will electronically be transmitted to selected partners. In order to maintain visibility within the network, all the partners of the SME network will receive a report stating the DMN configuration and plans.

In the DMN tracking phase, if a deviation from the initial plan is detected, the DMN Event Management submodule of the DMN Crisis Management submodule will trigger an action. It may either reschedule production among current DMN partners, or include new partners to the DMN in order to assign them the failed operations. Once the operations are performed, the DMN Performance Measurement submodule assesses the performance of each partner. Moreover, DMN partners will also evaluate their trust towards the SME network and the other partners. DMN performance assessments will be stored in the Collaborative Platform database for future tracking purposes. While failing in one DMN is probably acceptable for a partner, failing frequently is an important problem that requires further attention. Finally, the DMN Sharing module will employ decision-making mechanisms to partition joint costs and benefits among partners, by taking into account their performances within the DMN.

The Customer Relations module analyzes customer data, and consists of three distinct submodules: Customer Prioritization; Customer Segmentation; and Customer Tracking. Initially, the Customer Prioritization submodule feeds the DMN Creation submodule with values for customer priorities. The Customer Segmentation submodule then creates customer segments, again based on past customer information, thus providing information that can be utilized to develop strategies and promotions for similar customers. On the other hand, the Customer Tracking submodule calculates customer preference patterns, in order to support product development.

Conclusions

In this work, we have designed a set of ICT tools to support a business model based on two organizational layers: SME Networks and Dynamic Manufacturing Networks. Initially, we have identified three components of the SME network vision: Sustainability; Survival; and Growth. Later, we have implemented a Balanced Scorecard approach to translate the SME network vision into operational level ICT initiatives. These ICT initiatives, along with comprehensive literature review findings, provided a basis to design an ICT Platform. Two layers of ICT Tools were designed for the business model: a conceptual framework to support SME Network functions; and functional and process flows for the business model. These instruments are expected to adequately guide the development of focused decision support tools.

Nowadays, Collaborative Networks are highly dependent on ICT platforms and automated processes. Developing such integrated tools by following a well-defined methodology will have several benefits. Since partners get involved in these collaborative networks mostly for long-term advantages, developing a long-term vision and aligning strategy with action improves the credibility of the Collaborative Network in the partners' perspective. Moreover, it broadens the short term oriented, financial benefits-focused perspective into longer-term objectives, such as growth, sustainability and survival. Developing a clear vision and implementing it into operations increases the resilience of organizations in today's turbulent markets. Moreover, automated processes working with real time data significantly shorten the decision making time and make the operational execution much easier.

Fig. 3 .

 3 Fig. 3. Process Flows


Marie Gribouval

Davy Monticolo

Cecilia Zanni-Merk

Dimitri Sydor-Vienne

Assessment Methods to ICT Platform Used in Educational Guidance: Application to an Immersive World

School counselling is a decision-making process in which high school students have to decide what higher education course they will register for. Many digital devices support high school students in this counselling process. Our objective is to evaluate the impact of those digital devices. The research literature offers indicators to evaluate digital devices, however, evaluation protocols are very contextdependent and there are many types of digital devices. We want to determine how to measure the impact of these digital devices in the orientation process. This document presents an evaluation protocol and its application with an immersive world (VirBELA).

Introduction

In 2017, 3,40035 people registered for university in France for the first time [START_REF] Depp | Les étudiants : sur les enseignements, la formation et la recherche[END_REF]. Among these new students, some, at the end of their first year of higher education, changed their course of study or stopped studying. Reorientation rate is important. In 2018, 10% of new entrants reoriented their studies in the first year and 20% stopped their studying [START_REF] Depp | La formation continue : sur les enseignements, la formation et la recherche[END_REF]. These statistics show that it is important to take into account the orientation process and to analyze the impact of new devices dedicated to orientation.

In school counselling, various tools support the high school student orientation. We focus here on the study of the analysis of digital tools dedicated to orientation. Indeed, existing digital tools designed to help high school students with their orientation, have different objectives and formats such as immersive worlds, websites, mobile applications, etc. Some of them aim to inform, others to answer questions or, to give ideas for careers, etc. To make the most of digital devices in this context, it is important to be able to evaluate them. It is also important to understand which ones are the most appropriate for orientation and for individual issues. That evaluation raises a question:

• How to identify indicators to measure the impact of a digital device in the orientation process (ease of use, visual attractiveness, etc.)?

• How to measure impact and motivation degree while using digital devices? In this paper, we propose to analyze and identify from the existing research work the relevant indicators to measure the impact of a digital device in the orientation process by firstly describing a concise state of the art evaluation of digital devices. Then by presenting an experiment about the evaluation of an immersive world by using the relevant indicators. (This immersive world was used to introduce the different courses and training programs of the University of Lorraine to high school students.) And finally by discussing the results and the limits of our experimentation.

Previous work

To understand how to evaluate digital devices, we studied the indicators for measuring digital innovation. The definitions of innovation used in the context of a particular community or market are focused on "new products" [START_REF] Lyytinen | Digital product innovation within four classes of innovation networks[END_REF] and "services to satisfy human needs" [START_REF] Stosic | Selected indicators for evaluation of eco-innovation projects[END_REF]. The term "needs satisfaction" is present in the definition of eco-innovation [START_REF] Stosic | Selected indicators for evaluation of eco-innovation projects[END_REF]. Moreover, the goal of digital devices in the guidance process is to help people, so we will also use this term for our definition. Moreover, the authors of [START_REF] Rivas | Canonical correlation of technological innovation and performance in sheep's dairy farms: Selection of a set of indicators[END_REF] use the term "improve" for technological innovation in farms. The improvement of high school student counselling is a point to be considered in the evaluation of digital devices. These definitions show the importance of the user in considering innovation with the needs satisfaction and the novelty aspect according to the user.

Impact of user behavior on the diffusion and adoption of an innovation

Jahanmir et al. present the problem of late users regarding to the use of search engines [START_REF] Jahanmir | Factors affecting late adoption of digital innovations[END_REF]. Late adopters include laggards and the late majority of users. Laggards are the last to adopt an innovation, they are attached to traditions, while the late majority is rather skeptical. Late adopters are defined as users who are slow to adopt a new product or are reluctant to buy it. Five variables are presented to analyze this behavior [START_REF] Jahanmir | Factors affecting late adoption of digital innovations[END_REF]. These variables are attitude toward technology, negative word of mouth about the technology, global brand image, consumer innovativeness ("the force behind innovative behavior which results in the ability to generate new ideas"), and lead-user profile.

To go further into this issue, we have to consider the willingness of late users to continue using a product/service. Jahanmir et al. present two categories of factors; the perception and the personal category [START_REF] Jahanmir | Determinants of users' continuance intention toward digital innovations: are late adopters different?[END_REF]. The three perception factors measure user satisfaction. To do this, the authors study: • the "perceived radicalness", that is the perceived novelty of the solution, the market value and the concepts,

• the "perceived system quality", is about technical quality, • the "expectation fulfillment", this factor measures the satisfaction of the user's needs and expectations. The category of personal factors includes two factors, skepticism which is defined as a cautious attitude towards innovations, and slow adoption, a slower adoption attitude tends to show more loyalty. These users are furthermore impacted by the users that preceded them [START_REF] Tidd | Managing innovation: integrating technological, market and organizational change[END_REF]. Long-term adoption is also studied in the context of connected autonomous vehicles with the notion of a social network [START_REF] Talebian | Predicting the adoption of connected autonomous vehicles: A new approach based on the theory of diffusion of innovations[END_REF]. This social network symbolizes the communication of individuals with each other and leads to changes in the users' perception of innovation.

This axis allows us to question the position of digital innovations in connexion with these variables. Late adopters represent about half of the potential users [START_REF] Jahanmir | Factors affecting late adoption of digital innovations[END_REF].

In addition to the characteristics of the user, the diffusion of the innovation depends on its characteristics and the communication process used [START_REF] Jahanmir | Factors affecting late adoption of digital innovations[END_REF].

According to these previous research works, we have to consider how the evaluation of digital devices has already been conducted in the literature.

Evaluation of digital devices

There are some articles on the evaluation of digital devices, but their evaluation protocols are very context-dependent.

Karacapilidis seeks to evaluate and compare online school test interfaces [START_REF] Karacapilidis | Solutions and Innovations in Web-Based Technologies for Augmented Learning: Improved Platforms, Tools[END_REF]. They carry out their evaluation according to three categories, an educational dimension, an economic category, and a technical category. Each of these categories contains several indicators, for instance, regarding to the education they look at content, presentation, sequencing of test questions, and feedback on answers.

Other papers focus on users to evaluate their devices [START_REF] Zhang | Consumer innovativeness, product innovation and smart toys[END_REF], [START_REF] Lessiter | Evaluating Immersive User Experience and Audience Impact[END_REF]. Zhang et al. investigate the willingness to buy a digital product, especially a smart toy [START_REF] Zhang | Consumer innovativeness, product innovation and smart toys[END_REF]. To do this, the authors observe categories of indicators about users. They ask them about demographic information, perceived product innovation, perceived product value, consumer innovation, and willingness to pay. Perceived value is divided into sub-categories of performance, emotional value, monetary value, and social value. Consumer innovation has also been discussed [START_REF] Jahanmir | Factors affecting late adoption of digital innovations[END_REF], [START_REF] Benson | Digital innovation evaluation: user perceptions of innovation readiness, digital confidence, innovation adoption, user experience and behaviour change[END_REF], it is the user's initial attitude towards the technology.

An article also aims to evaluate a digital device, this time an immersive content and more specifically a virtual reality device [START_REF] Lessiter | Evaluating Immersive User Experience and Audience Impact[END_REF]. This assessment is carried out to determine possible user interest, cultural and commercial impact. Their evaluation is conducted in three stages, with questions asked to users before, just after, and 14 or 21 days after using virtual reality. Some common indicators reappear about what has been presented, such as users' behavior towards the technology beforehand, but others have not been presented until now. Many indicators are present in this study. The authors divide the indicators into 9 categories (audience quality of experience, audience attitudes towards content & tech, creator's intended impacts, audience characteristics, interaction affordances, economic impacts, audience behaviors, audience traits, stakeholder target impacts).

In the field of health, the question of the evaluation of digital innovations has also been raised. Benson question the ways and reasons for the diffusion of these devices and new models of care [START_REF] Benson | Digital innovation evaluation: user perceptions of innovation readiness, digital confidence, innovation adoption, user experience and behaviour change[END_REF]. In medicine, digital innovations can for example be, as discussed in the article [START_REF] Benson | Digital innovation evaluation: user perceptions of innovation readiness, digital confidence, innovation adoption, user experience and behaviour change[END_REF], a digital coach for patient self-management of diabetes. To evaluate these devices, five categories are studied: • Innovation Readiness: the user's earliness to adopt new ideas compared to others, • Digital Confidence: personal and peer use of digital devices, • Innovation Adoption: the work of medical staff before, during, and after using the tool, • User Satisfaction: the usefulness of the tool, its ease of use and overall satisfaction, • Behavior Change: capability (e.g. physical), opportunity, and motivation to use the device for the user.

This state of the art study shows the diversity of evaluations of a digital device and the need to adapt the evaluation to the context. As a result of it, two user-related themes emerge. The information related only to the user and his/her relationship with the digital devices and the theme of the digital device to be evaluated as seen by the user.

In the following section, we propose to categorize relevant indicators to evaluate a digital device while considering the context of school guidance.

Research method

In this paper we seek to evaluate the degree of innovation of digital tools that is the perceived novelty, the satisfaction of needs, and the improvement that the device provides. However, the state of the art studies shows that these three axes are not the only user-level issues to be considered. It is necessary to consider the overall user experience as well as the overall relationship with digital tools.

Furthermore, as we are looking for indicators to assess the degree of innovation of digital devices, we need to define the term indicator. We consider the definition presented by Dziallas et al.; an indicator is a value measured to provide information [START_REF] Dziallas | Innovation indicators throughout the innovation process: An extensive literature analysis[END_REF]. This value respects three assumptions stated by Miremadi et al., that are: ease and simplicity of understanding, the existence of data, and relevance [START_REF] Miremadi | Assessing the performance of energy innovation systems: Towards an established set of indicators[END_REF].

We present in the next sections a proposal of an indicators list that enables measuring the degree of innovation of a digital tool.

Categories of indicators

Following this state of the art studies, we have identified the user-related indicators of interest in our problem. To do this, we cross-referenced the indicators and their categories present in the different articles. This combination allowed us to distinguish two themes about the users, user-specific indicators and indicators from the product evaluated according to the user. Indicators are present for each category, and indicators may be added or removed as this work evolves. In addition, indicators may be included in several categories.

In the user-specific indicators we find three categories:

• the social information, • consumer innovativeness (in our context it is digital devices),

• the user's behavior towards the information domain delivered or used by the innovation (in our context it is orientation).

For the indicators from the user-assessed product, there are eight categories: • the perception of the overall innovation [START_REF] Zhang | Consumer innovativeness, product innovation and smart toys[END_REF][START_REF] Lessiter | Evaluating Immersive User Experience and Audience Impact[END_REF][START_REF] Benson | Digital innovation evaluation: user perceptions of innovation readiness, digital confidence, innovation adoption, user experience and behaviour change[END_REF]: wanting to know if the user sees it as a new tool if the overall image is positive, • the perceived value of the innovation [START_REF] Zhang | Consumer innovativeness, product innovation and smart toys[END_REF]: wanting to find out if, according to the user, the use of this tool has a value in this context,

• the emotional value caused by the innovation [START_REF] Zhang | Consumer innovativeness, product innovation and smart toys[END_REF][START_REF] Lessiter | Evaluating Immersive User Experience and Audience Impact[END_REF]: the different emotions caused by the use of the digital device, • the social value induced by the innovation [START_REF] Karacapilidis | Solutions and Innovations in Web-Based Technologies for Augmented Learning: Improved Platforms, Tools[END_REF][START_REF] Zhang | Consumer innovativeness, product innovation and smart toys[END_REF]: wanting to know if the use of the tool has impacts on the user's view of himself and others on the user, • the content [START_REF] Karacapilidis | Solutions and Innovations in Web-Based Technologies for Augmented Learning: Improved Platforms, Tools[END_REF]: wanting to know what the user has learned and if they have learned anything. These things are to be defined according to the context, • the use of the tool [START_REF] Karacapilidis | Solutions and Innovations in Web-Based Technologies for Augmented Learning: Improved Platforms, Tools[END_REF][START_REF] Benson | Digital innovation evaluation: user perceptions of innovation readiness, digital confidence, innovation adoption, user experience and behaviour change[END_REF]: its ease and fluidity of use, • the sensory [START_REF] Karacapilidis | Solutions and Innovations in Web-Based Technologies for Augmented Learning: Improved Platforms, Tools[END_REF]: how the user feels according to the senses affected by the innovation (for example, hearing and sight),

• the future behavior [START_REF] Lessiter | Evaluating Immersive User Experience and Audience Impact[END_REF]: includes repeating the experience, remembering it after several days/weeks, and wanting to talk about it to relatives.

This classification of indicators aims to determine the user's vision and behavior about innovation, considering his or her personal preferences.

Users objectives

In addition to these indicators, it is necessary to consider the objectives of the different actors concerned with digital devices [START_REF] Lessiter | Evaluating Immersive User Experience and Audience Impact[END_REF][START_REF] Benson | Digital innovation evaluation: user perceptions of innovation readiness, digital confidence, innovation adoption, user experience and behaviour change[END_REF]. Indeed, we would like to check that the digital device satisfies users' expectations in terms of use. This requires identifying the different types of actors. For each category of actor, we identify objectives, and we check that they have achieved their objectives at the end of their use.

Application to an immersive world

Context of the study

On 16 March 2021, the CapSup 4.0 day was organized as part of the Printemps de l'orientation. It is developed for high school students in the 10 th grade (UK year 11) and 11 th grade (UK year 12). That corresponds respectively to seconde générale et technologique and première generale ou technologique in the French education system. This period of several days aims to promote actions (with contacts, resources) to a better harmonization between the three years before graduation (high school USA or upper secondary school UK) and the three years after graduation. The CapSup day is an event that regroups higher education courses. It aims to introduce high school students to the various courses available in higher education and provide guidance counselling. The previous editions were hosted in high schools. But this year, due to the pandemic, the digital tool VirBELA was used [START_REF] Virbela | Ready for Virtually Anything[END_REF]. The SOIP (Service d'Orientation et d'Insertion Professionnelle) organized it in partnership with the regional education authority. The SOIP is a guidance service available to students in each university. The regional education authority of Nancy-Metz provided information and educational resources to the high schools and also mobilized "national education psychologists", civil servants employed in guidance and counselling.

VirBELA platform (VP) is an immersive 3D world, each user is represented by a character. The user can move his character in the virtual world. It is possible to hear and talk to other people, and the spatiality of the voice is preserved.

Our challenge is to evaluate a digital platform in the context of CapSup with our research evaluation model. For this purpose, we prepared a research protocol before, during, and after the event.

Protocol

The protocol is divided into three stages, before, during, and after the experiment.

• Collection of speakers' objectives (staff presenting training course(s)), before the experiment: a first questionnaire was sent on 8 March to 108 speakers, to find out in which areas they would like high school students to acquire information during this day.

• Collection of feedback from high schools from the VP, during and after the experiment: we distributed an online questionnaire to high school students on the VP several times during the day on 16 March (at 11:15, 14:30, and 16:30). The questionnaire aimed to obtain a quantitative view of the use of the VP. On 23 March, this questionnaire was sent to 2967 high school students (these are the students who logged on to the platform on 16 March) by e-mail to obtain more answers.

• Observation of the use of the platform with a pilot class, during the experiment: in addition to our quantitative analysis by questionnaire, we carried out a qualitative analysis with a pilot class of high school students. The qualitative analysis aimed to identify the uses and behaviors of high school students when using VP using an observation approach supplemented by exchanges.

• Analysis of the data and comparison with the identified evaluation indicators, after the experimentation: An analysis of the data is made as well as a comparison of the objectives achieved by the students with the objectives of the speakers and the SOIP's expectations.

For the implementation of this protocol, we had to consider the fact that we were interviewing high school students. To obtain as many answers as possible, we chose to ask few questions (7 questions) in the online questionnaire. This small number of questions meant that not all the identified indicator categories could be worked with. We use six categories of indicators, the perception of the overall innovation, the emotional value caused by the innovation, the content, the use of the tool, the sensory, the future behavior.

To reinforce the data obtained in this questionnaire we observed and exchanged it with a pilot class. This class came to the ERPI laboratory to use the VP. We asked more qualitative questions to this class compared to those on the online questionnaire.

Survey population

649 students have filled the online questionnaire. 6,000 high school students were expected to be on VP on March 16th and about 2,960 were logged on. We noticed that a higher percentage of women (65.2%) responded to the questionnaire. In addition, it should be noted that 79% of the respondents to the online questionnaire were in the second year of high school.

A pilot class composed by 35 high school students from the high school "Chopin" in Nancy came to use the VP platform at the ERPI laboratory during the CapSup day on 16 March. That class was divided into two groups; the morning group with 17 high school students and the afternoon group with 15.

Analysis of its use by high school students

In this section, we analyze the use of the VP with the indicators identified in the literature.

Perception of the overall innovation.

During our discussions with the pilot class and by combining the answers to the questionnaire sent to all the high school students, we aimed to analyze the degree of innovation of the immersive world during this orientation information process. We measure the quality of the overall experience [START_REF] Lyytinen | Digital product innovation within four classes of innovation networks[END_REF]. The answers from the questionnaire show that the experience in the immersive world was beneficial for the students as 49% mentioned a score of 4 or 5 (1 for very unpleasant to 5 for very pleasant). These results are in line with those obtained from data collected from another questionnaire (SOIP survey) with the question "How do you find this day? (I love it / It's nice / I don't like it at all)", as out of 850 respondents, 89% indicated "I love it" or "it's nice".

Emotional value caused by the innovation.

Zhang et al. indicate that there is a significant connection between perceived emotional value and willingness to pay (the article refers to a smart toy), we inferred that this emotional value also has impacts on their use of the platform studied here [START_REF] Zhang | Consumer innovativeness, product innovation and smart toys[END_REF]. As emotion is not something easy to measure in an online questionnaire, we just asked them about their boredom. 13% said that the event bored them, however in the "other" answers, we find 4 high school students who said they found it "funny" or "amusing". In addition, we wanted to know whether the students present enjoyed the tool and whether the virtual aspect made them feel more relaxed. On the whole, this was the case; the word "fun" came up again in the morning group and they told us they found it "original". The afternoon group were more, concerned about orientation came up more, with 60% saying they were worried about it. They felt periodic pressure from the school on this subject. Even if the virtual platform did not seem to alleviate their worries, it allowed them to ask questions more easily than in person and reduced their shyness.

Content.

One of the objectives of CapSup was to inform high school students about higher education. We wanted to evaluate the accessibility of information for high school students. 52% of the high school students stated that they found the information they were looking for. We can assume that the high school students who did not respond positively were not looking for information as they were not looking by themselves. To go further, we asked the high school students present at the ERPI laboratory if they had got answers to their questions about orientation. When they arrived at the ERPI, the students did not have any particular questions. It was therefore not easy to determine whether they had learned anything or not. However, we did get some positive responses, for example, a student who wanted to do a BTEC Higher National Diploma after the baccalaureate was helped by the corresponding conference. And, more generally, in the morning group, 12 people told us that they had learned something.

In the online questionnaire, a few remarks were made about the content (about 12 out of 145 remarks). Some of them were concerned about what the speakers said and the form they had to fill. For example, some students thought that speakers should "Prepare the speeches more so that they are more interesting, many of us learned nothing". For others, the problem was with the workshops offered, some of them said that "It's a shame that there are no jobs related to the fauna and flora apart from the earth and life sciences workshop". Concerning the ease of understanding, the noise was an issue, because of some disruptive high school students.

We also asked what they had learned to compare it with what the speakers were trying to convey. We can see that the priorities are not the same (this may be due to a low number of responses from the speakers), even if two identical trends emerge. The choice of option for the final year of secondary school and student life is less of a priority in the context of CapSup' for both the high school students and the stakeholders.

Use of the tool.

A digital platform is necessarily impacted by the effort the user has to make to use it [START_REF] Karacapilidis | Solutions and Innovations in Web-Based Technologies for Augmented Learning: Improved Platforms, Tools[END_REF]. In our case, we observed the interaction among high school students and the VP interface.

The questionnaire sent to all students allowed us to analyze the degree of ease of moving in the virtual world. The students indicated the degree of ease on a scale of 1 to 5. The number 5 symbolizes a "very easy" use, and the number 1 a "very difficult" use. The majority of students found it easy or very easy to use, as 65% answered 4 and 5. On the contrary, 12% found it difficult (values 1 and 2). The latter percentage can be explained by the difficulties that some students had in orienting themselves with their avatars in the virtual world.

These results were confirmed by the observation of the pilot class in the laboratory, as all the students present described the use of VP as intuitive. Only one person out of the 32 presents said that they had some difficulties in getting used to it. One of the difficulties the students had was navigating the VP. Comments on this subject were added to the questionnaire ("We don't know where to go"). In addition, several students present asked us questions at the beginning of the session about changing rooms or finding the schedule.

Sensory.

In this section, we seek to find out what students thought of the graphics and sounds of the interface, as this has an impact on the attractiveness of the interface [START_REF] Karacapilidis | Solutions and Innovations in Web-Based Technologies for Augmented Learning: Improved Platforms, Tools[END_REF].

Over the day, 12 students told us that they found the graphics pleasing and that they found it attractive, particularly through the use of avatars. This last remark was repeated in the morning and afternoon.

As far as the sound is concerned, opinions were more mixed, and it emerged from the high school students present that it was difficult to understand the people, mainly because of the behavior of some. This remark was also made in the online questionnaire. Concerning the music and sound effects, in the morning, they found it good and it accentuated the realism. In the afternoon, four people said they liked it, while others found it disturbing, especially the footsteps in the halls. 4.3.6. The future behavior. The future behavior studied here, with the willingness to repeat the experience and to talk about it [START_REF] Lessiter | Evaluating Immersive User Experience and Audience Impact[END_REF], also shows the appeal of the platform. Most of the students present said that they would like to have access to the platform again in their final year. They would also recommend it to first-year students, and probably at the end of the second year. Nevertheless, they think that changes should be made according to the user's school level, but also according to whether or not they know what they want to do after the graduation baccalauréat (for example, a whole day to attend all the workshops for those who do not know what they want to do).

Analysis of its use for guidance

This platform meets the objectives of SOIP, as it allows the adapted information on higher education courses to spread. With its 3D virtual world interface and the use of an avatar, it is an "easy-to-use and dynamic software" for high school students. The acquisition of information by the high-school students is mixed. We see that the law workshop, which was the most attended, was attended by 150 of the 649 students who responded to the questionnaire. The objective of providing information on student life was not achieved to any great extent, since 15 of the 649 students said they had been to the university life meeting point.

Discussion and limits

In this article, we wanted to identify indicators to measure the impact of a digital device in high school counselling. Categories of indicators have been put in place to assess the use of digital devices while considering the user himself. With the experimentation, we see that these categories allow us to know the user's opinion from different viewpoints. Our second scientific interest was to measure impact and motivation degree while using digital devices. To do this, we have established a protocol. Some of the results of the indicator categories were easily analyzed as use of the tool. However, for the content category, to evaluate the VP, we would have needed additional data. It would have been interesting to observe the interactions in the workshops, to know how long high school students stayed in the conference. This analysis focused on the user at a given moment; a digital device will not be seen in the same way by the student depending on the stage of the decision process in which he is. To solve this problem, we need to consider the digital device and the guidance in greater depth.

Conclusion

The purpose of this article is to evaluate the impact of digital devices in the context of school counselling. To do this, we consider in particular users with two points of view (user-specific indicators and indicators from the product evaluated according to the user). A first experiment was conducted with immersive world VP in the context of CapSup. That experiment has allowed us to use our indicators and to take into account new ideas to consider for future experiments.

In future work, we want to focus on the characterization of digital devices. As the first work has been done on their impacts, we want to know more about the interest of each tool regarding the temporality of the guidance process.