ELLIOT -Experiential Living Lab for the Internet Of Things

Thus this third deliverable in the D4.3 series integrates the updates on:

i|) The Living Lab use-cases

ii) Progress through Living Lab cycles

iii) The evolution of the operational deployment of the KSB Model iv) Insights into different ways of using the KSB Model

The introduction section provides an outline of the scope of this deliverable; the document concludes with a summary of the achievements and generalise-able results across the three usecases.

Introduction

The main goal of the ELLIOT (Experiential Living Labs for the Internet Of Things) project is to develop IoT technologies and Ambient Intelligence (AmI) services by and for users/citizens through the design of a set of Knowledge-Social-Business (KSB) Experience Models and their implementation in an innovative ELLIOT Experiential Platform operating as a knowledge and experience-gathering environment. The early involvement of users/citizens, as recommended in the ICT Work-programme, is conducted according to the precepts of the Open User-Centred

Innovation paradigm and through co-creation and experimentation using the Living Lab approach which aims to involve users/citizens in research and innovation pathways. This combination of market pull and technology push is expected to have a positive impact on the development and adoption of IoT technologies and innovative services.

Purpose, Intended Audience and Scope

The purpose of this deliverable is to document the final results of WP4 in order to describe and disseminate the research and development that has been carried out within and beyond the ELLIOT project. Within the project, this deliverable is of particular relevance for Work Packages 1, 2, and 5. For Work Package 1, this deliverable provides feedback regarding the revised ELLIOT KSB model. For Work Package 2, this deliverable provides additional feedback regarding the technical ELLIOT experiential platform with a view to possible future improvements. For Work Package 5, the deliverable provides input for dissemination activities and for exploitation planning.

The near-term intended audience of this deliverable is the ELLIOT project Partner community, where in particular Partners involved in the aforementioned work packages are expected to benefit when continuing work based on ELLIOT beyond the end of this project. The mid-to-long term intended audience of this deliverable furthermore extends to Living Labs using ELLIOT project outcomes in the field; for them, this document should convey ways in which ELLIOT can be applied as well as lessons learned that may help them in avoiding potential problems when applying ELLIOT in practice.

Document Outline

The content of this deliverable D4.3.3 follows a similar organisational structure as the previous deliverables D4. 3.2 and D4. Each use-case is presented in an independent chapter; all use-case chapters follow the same overall structuring in sections and subsections with a few minor variations in order to account for differences between the three use-cases.

All three use-case Partners provide additional information on privacy protection measures in place in their respective use-cases in order to provide explicit descriptions concerning how this subject matter is addressed within the scope of WP4.

A brief conclusions section completes the deliverable. Additional project outcomes related to the project outcomes reported on in this deliverable can be found in the relevant final deliverables of WP1 and WP4.

Applicable Documents

The following ELLIOT deliverables in particular apply and are of importance for this deliverable D4. The logistics use-case, set up within the BIBA and LogDynamics Living Lab (LL) environment, follows an iterative improvement approach in the domain of intra-logistics. From discussion with experts, a specific focus on intra-logistics processes has been derived for ELLIOT. Intra-logistics deal with the logistics processes between companies (e.g. warehouse logistics, material logistics/production supply; etc.). The main goal of the use-case is to create a service that identifies risk situations along an intra-logistics process chain and encourages stakeholders to freely explore alternatives for increasing safety and security for operators and goods; by increase safety and security, intra-logistics processes will thus be improved.

The Living Lab process and its development were described throughout WP4 in deliverables D4. 3. 1 and D.4.3.2. Since then, no major changes have been made regarding the Living Lab process and workshops. However, the main focus for the period covered by this final report was on the experimentation workshops, in which services were tested on the BIBA shop floor area.

As presented in D4.2.1, a Serious Game (SG) and a sensor tool kit support a discussion and translation process which leads from idea creation (co-creation) to an IoT service (exploration and experimentation); the process is monitored and guided by continuous evaluations. The experimentation activity is carried out using a risk monitoring application (see Figure 1); a simpler GUI for the forklift driver exists (less distracting, fewer information and configuration opportunities), but since the development is performed within the risk monitoring application configuration GUI, the adoption to this GUI was relatively high and the experiment was also run with this GUI. In addition to the focus on experimentation, a Web Service to access the IoT toolkit remotely over the Internet was finalised for the final project period (see Section 3.1.4).

During the experimentation, it is observed whether the created services reach a number of predefined goals which are related to the KSB model. Reduced product damage due to temperature changes for instance can indicate an increased property of "cognitive artefacts" since participants have become more aware of temperature-related issues due to the developed IoT service and its support of the working environment. This would as well be one important factor in the user experience of the service.

Experiments were conducted on May 3 rd and June 18 th . The experiments covered two services each. These services had the same functional goals and were only changed slightly in order to investigate design changes relevant for the user experience. The services were: a) a product temperature service, observing product and outside temperatures, and b) a forklift angle service, observing the forklift angle in order to indicate risk of goods toppling to the driver and/or the surroundings. Specifically, changes were made to the form of risk representation through the use of actuators. For the forklift driver, risks are indicated mainly via a rotating warning light and a traffic signal light.

KSB Instantiation

The use of the KSB Model and the selected KSB Properties have been described in D4. Compared to the Table of D4.3.2, a few properties were dropped and elements were changed to some extent. Former K4.1 "Human computer interaction", S7.2 "Appealingness", B2.1 "Reliability" and B4.4 "Loyalty" were removed from the KSB framework or changed as a result of the KSB model refinement. For example, K3.2 "Cognitive Artefacts" was previously denoted as K4.1. Furthermore, B4.5 "(Intention of) Loyalty" was changed to B3.1 "Ergonomic Quality", based on the Sofia workshop and in order to streamline investigated properties with Partners (this property is also used by FCSR).

The previously used business property B5.2 "Availability" was discussed with the other pilots and was found to be lacking the potential to indicate a positive or negative KSB value. Therefore, this indicator is not considered in the KSB evaluation here (for more details, please refer to Section 3.2.3).

Connection to ELLIOT Platform

The raw sensor data from the logistics use-case sensor toolkit (as well as some qualified sensor data) is pulled in by the Data Provider located on the BIBA site. This pull is performed by the Data Provider Client that is part of the ELLIOT platform. Data is transferred through the ELLIOT middleware which then delivers this data to the ELLIOT core platform for storage, representation and computation of KSB analyses. At the BIBA site, a database which the Data Provider accesses on request from the Data Provider Client is provided. The anonymisation module and the rule engine of the ELLIOT middleware are not currently in use in the logistics use-case (for more details on privacy protection and anonymisation, see Section 3.4.3).

Since database changes due to new requirements and improvements at pilot case level are required, the effort to update the database model at platform level rises. To mitigate this need, database views which contain some pre-processed data (i.e. aggregated data from sensor logs) are provided.

Once the data is available on the ELLIOT platform, Key Performance Indicators (KPIs) are defined based on the KSB instantiation and data which is collected, as well as depending on the quality of data aggregation (qualification) by Web Services. As described in the relevant documentation, the platform is used to select KSB properties and to prepare the rules, the scale and if applicable the weight for each KSB property.

ELLIOT Web App

The ELLIOT Logistics web app provides access to relevant information of the Logistics toolkit (and services built upon it) and is accessible from any web-enabled device such as a PC, smartphone or tablet. Access to the web app is permitted only to registered users of the Logistics toolkit. A user account is generated and reveals user-specific details only. As all created risk situations are saved within the user ID, no data created by other users can be viewed.

Consisting of three views next to the home screen, the Logistics web app provides different levels of details concerning information gathered with the Logistics toolkit. In the first layer, all sessions of the logged -in user are listed ("session list"). When choosing a session for closer examination, the layer of the "session details" is viewed. Details for each risk are displayed in the third layer -

the "risk details" layer. The Logistics web app can be reached at http://134.102.95.221/. The home screen, as displayed in Figure 3, appears when this website is opened. A user ID is required in order to log in. When logged in, a "session list" (see Figure 4) is displayed. It gives an overview of the latest 20 sessions held. The name of the session is shown in bold letters at the top of each session box.

Additional information about the start as well as the end date and time of the session can be found in the box, too. The traffic light indicates whether the session is running (green) and still active or finished (red). Clicking on one of the sessions displayed in the "session list" enters the "session details" view (Figure 5). There, determined risks are listed with their current risk level. Each risk can be chosen for a detailed look into the specifications and current states. In the "risk details" layer, the current risk level is displayed again; here, a coloured bar indicates the risk level and the risk threshold defined in the Logistics toolkit is also displayed. This bar graph is connected to the traffic light placed on the upper right corner of the display. When the risk threshold is reached, the light turns from green to yellow. It turns red when the risk limit is crossed.

In addition, the connected sensors are listed in this layer and show the currently measured values.

Below, the date and the time of the last limit and threshold crossing are shown. The ten latest threshold crossings are also displayed graphically in the box at the bottom of the display. Again, the color of the graphs is related to the risk level.

Experiment Lessons Learned und Updates Relative to D4.3.2

The following improvements and changes have been made relative to D4.3.2 (for details see the following sections below):

 The general metrics which form the basis for the indicators used to measure the KSB properties were refined based on findings and decisions made during the property definition process. This is related to the refinement of the KSB property and indicator selection. Furthermore, some adjustments of the living lab workshops have been made in order to improve their outcome.  In order to improve the data collection, the toolkit software was adjusted and the data processing web services were implemented at platform level. Finally, the web interface for remote access of the IoT services was finalised (see Section 3.1.4).

Design Changes

Metrics

KSB properties and KPI selections in ELLIOT have been undergoing a constant refinement process. This has proceeded alongside a refined view on data analysis (cf. D4. 3.2). Since the data analysis is intended run mainly automatically, supported by the ELLIOT platform, general metrics have been developed. These metrics have been refined during the final adjustments of the data analysis. The set of metrics was slightly changed since 'coverage' was not used in the end. Therefore, the remaining set was 'efficiency', 'efficacy', 'time spent in' and 'interaction'. However, since they are used in different KSB properties, the metrics were adjusted accordingly.

Efficiencythe level of efficiency achieved in addressing the use-case needs is computed as the achieved percentage of the theoretical 100% full satisfaction of the workshop objectives. In this case it has to be possible to have results greater than 100% to accommodate the case where participants exceeded the defined maximal goals that the workshop planned to obtain.

Because comparisons with previous service setups are performed on platform level, the efficiency is calculated only for one setup at a time. Therefore, the formula changed. Efficacythe level of accuracy and effectiveness of the proposed solutions when compared with alternative solutions. This metric should be computed as the ratio between "% of success achieved with IoT" to "% of success achieved without IoT".

 Example: defined goal for the service improvement, i.e.:

o # 𝑜𝑓 𝑡𝑜𝑝𝑝𝑙𝑖𝑛𝑔: 4, 𝑟𝑒𝑎𝑐ℎ𝑒𝑑: 6 o 4 6 = 66,67% 𝑒𝑓𝑓𝑖𝑐𝑎𝑐𝑦 Time spent in (design, implementation, testing, etc.)objective measure of the time spent in the service as expected and as actually determined.

Interactionthis metric is composed of two measured elements: a measure of the overall interaction and a measure of the ratio between achieved interaction levels to the expected one.

The level of interaction should be expressed qualitatively as: absent, very low, low, medium, high, very high. So far this metric is captured as the level of interaction (number of participants involved) over time and an average can then be calculated. Initially, the expected level is always "high" which equals the number of participants minus 1; a high interaction is expected, needed and aimed at with a Living Lab approach. This metric is captured manually by observation.

Adjustment of Living Lab iterations

Findings from the first Living Lab workshops and the evaluation of their results encouraged an approach in which parts of a Living Lab cycle (co-creation, exploration, experimentation) are carried out instead of a full cycle every time a Living Lab iteration starts. Instead, the order of workshops and Living Lab steps is based on the stage of development of the service. For this reporting period this is in particular relevant, since the focus was placed on experimentation and the relevant data collection and less on the co-creation and exploration.

Questionnaire

To gain more knowledge about the Living Lab workshop participants, their expectations, knowledge about IoT and opinion about IoT, a questionnaire with self-assessment questions was used during co-creation and exploration. Previous workshops were evaluated and reported in D4.3.2, therefore only additional data is analysed and new findings are discussed in the context of previous findings. Since the questionnaire remained unchanged, please refer to D4.3.2, Section 3.2.2 for details.

Data Collection Changes

Since the last report about IOT Living Labs Continuous Exploration and Evaluation, changes were qualitative rather than quantitative in terms of reduction or changes of KSB elements/properties and according data collection (indicators). Before, findings from the initial report and analysis (D4. 3.1 and D4.3.2) had led to a better understanding of the usage of the model which led to a reduction of observed properties. During the final period, the experiences gained helped to improve the data collection and analysis regarding selected KSB properties.

The final list of KSB properties is presented in Section 3.1.2, Table 1.

The final selection of KSB properties is based on the following goals:

(1) To cover all perspectives of the model: K, S and B

(2) To work with properties which are clear and reasonable for this particular use-case

(3) To find interdependencies with other use-cases (cf. property B3.1)

In terms of sensors for data collection, the types of sensors remained unchanged since D4.3.2.

The toolkit software, i.e. the related database, was adjusted to improve the database structure, the representation of data gathered and the pre-processing of data. These improvements were made until June 2013. Due to lab-internal technical issues which caused successive errors the comparison of experiment data is limited (see Section 3.5.1).

The pre-processing of data is performed at database level, where raw data from sensors is computed together with configuration settings or predefined sets (like target values). This preprocessed and structured data is represented in a database view which is published to the ELLIOT platform.

Data Analysis Changes

During the reporting period, minor changes were made regarding the KSB properties and related indicators (KPIs); see above, Section 3.1.2.

Table 3 shows the final KSB properties from Section 3.1.2 and the related indicators. The processing and calculation of the properties is explained below. Information about the calculations of the named KSB properties out of selected KPIs is also provided below, representing the selected approach for calculation of KPIs/KSB.

K2.2 -Attunement of Affordances

This indicator was not set up in experiment practice, but an exemplary calculation and involvement into the KSB model will be shown in the data analysis section. The indicator used (duration of threshold) should indicate a positive or negative development, especially in comparison with previous/later experiments. In the application case, it is relevant to identify the service configuration which supports positively the reduction of potential risk situations.

Indicators:

o Duration of threshold crossings Calculation: 𝑎𝑣𝑒𝑟𝑎𝑔𝑒 𝑜𝑓 𝑑𝑢𝑟𝑎𝑡𝑖𝑜𝑛 (𝑐𝑟𝑜𝑠𝑠𝑖𝑛𝑔𝑠) 𝑡𝑎𝑟𝑔𝑒𝑡 𝑣𝑎𝑙𝑢𝑒 )

K3.2 Cognitive Artefacts

This KSB property refers to the property K4.1 in D4. Calculation:

1 + (1 - 𝑡𝑎𝑟𝑔𝑒𝑡 (𝑡ℎ𝑒𝑠ℎ𝑜𝑙𝑑) # 𝑜𝑓 𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 𝑐𝑟𝑜𝑠𝑠𝑖𝑛𝑔𝑠 ) 𝐼𝐹 ⌈𝑥⌉ > 1 → [𝑥] = 1 𝐼𝐹 ⌈𝑥⌉ ≤ 0 → [𝑥] = 0

S5.1 Attractiveness

The KSB property Attractiveness has been reviewed. The indicators are still the duration [time] and the number of complaints during this time. It was also possible to define a formula which helps express the Attractiveness:

Indicators:

(i) = duration (time spent) compared to 6h in % (max 100%)

(ii) = complaints during this session (duration) in # Calculation:

(𝑖): ∑ 𝑑𝑢𝑟𝑎𝑡𝑖𝑜𝑛 𝑥 𝑛 𝑥=0 6 ℎ𝑜𝑢𝑟𝑠 [%] (𝑖𝑖): 1 - 𝑥 𝑚𝑎𝑥 𝑥 [(𝑖) * 0,5] + [(𝑖𝑖) * 0,5]
However, due to the fact that users were using the configuration interface for observation during experimentation (therefore a different GUI was meant to be used), the complaint button did not come into use. Therefore, the Attractiveness is based on the duration only.

B1.2 -Performance Level

Previously, the Performance Level was intended to compare directly two different consecutive time periods. Now, the comparison with previous time periods is carried out via the platform. A change in KSB properties indicated positive/negative influence of setup changes to the overall user experience.

Calculation:

𝑃𝑒𝑟𝑓𝑜𝑟𝑚𝑎𝑛𝑐𝑒 𝐿𝑒𝑣𝑒𝑙 = 1 + (1 - # 𝑟𝑖𝑠𝑘𝑠 𝑖𝑛 𝑡 1 𝑔𝑜𝑎𝑙 # )
Indicators:

D4. o Efficacy (# of risks in observation period vs. target value [#] of maximum risks)

B4.5 (Intention of) Loyalty

Indicators:

o Duration: [time spent in service]

Calculation:

The calculation is done with the time spent compared to 6h during shift [in%]

(𝑖):

∑ 𝑑𝑢𝑟𝑎𝑡𝑖𝑜𝑛 𝑥 𝑛 𝑥=0 6 ℎ𝑜𝑢𝑟𝑠 [%] 𝐼𝐹 (𝑖) > 1 → (𝑖) = 1

B5.1 Accessibility

Indicators:

-(i): [# of configured risks in a session] -(ii): [# of public risks] • Configuration session: checkbox [public/private] Calculation: 𝐴𝑐𝑐𝑒𝑠𝑠𝑖𝑏𝑖𝑙𝑖𝑡𝑦 = (𝑖) (𝑖𝑖)

B5.2 -Availability

This indicator was discussed with the other use-case Partners and was found to be lacking the potential to indicate a positive or negative KSB value. This is due to the fact that it was difficult to identify the number below/above which the result would be rated as negative/positive.

Furthermore, it was unclear whether this indicator is measured per session or over all sessions; or whether it would even measure the potentially achievable number of those platforms able to connect to the service. It was agreed that this indicator needs further improvement before it is used in the pilot.

The reference values used (expected duration [hours], expected crossings [#]; etc.) should be adjusted through the platform.

D4.

Participants

Table 4 shows the Living Lab workshops which have been carried out since the report on IoT

Living Labs Continuous Exploration and Evaluation, D4.3.1. One additional co-creation workshop was conducted and the results are discussed in this final deliverable. Furthermore, additional results from the questionnaire analysis were reported in D3.4, since they were related to the toolkit, and an additional analysis is presented below as well.

The number of participants was usually 5 plus one moderator in each workshop. As described in D4.3.2, this number is mainly affected by the Serious Game which is used during the co-creation; the game supports five players. The group of participants in the 4 th co-creation workshop is shown in Table 5. As previously, assumptions about their experience were checked by using questionnaires. ratio of 3:2 (as here) might provide a suitable sample. Nevertheless, even with the best possible mix based on the available players, a sample ratio 3:2 is sufficient for providing a non-gendered result.

Data Collection

Collected and Qualified Risk Data

The sensor data is collected based on defined time intervals which depend on the type of sensor.

This raw data from the sensors is stored in the data base. Furthermore, the database log collects data about sessions:

 durations,  sensors (name/ID),  defined thresholds and  defined risk limits (namely when risk take place)

 time stamps for events, action, etc.  Configurations (e.g. of actuators)

The most relevant data for the KSB evaluation are gathered in the risk_data_detail table of the pilot database. However, this table reveals the relevant information in order to elaborate the following KPIs, which are extracted from the data base in order to provide qualified data.

The qualified data contains  threshold and limit crossing_IDs o needed to elaborate number of threshold/limit crossings  number of threshold crossings  number of limit crossings o These IDs level interruptions in sensor data streams and identify related sensor values for a crossing; this levelling ensures that the same crossing is not identified as two independent ones.  time stamps for crossings o needed to elaborate durations threshold/limit crossings  duration of threshold crossings  duration of limit crossings

 duration of sessions

More elaborated data (number of crossings, durations) can either be acquired by using Web Services from the ELLIOT platform or by accordingly configured data base views.

Observation and Survey Data

Observation of Living Lab workshops

In D4.3.2, an overview of the level of participation during the Living Lab activities deducted from workshops observations was presented. Since then, an additional co-creation workshop was carried out; therefore, the overview in Figure 6 focuses on data from co-creation. The analysis of the Living Lab workshop, especially the exploration, regarding the Arduino IoT tool kit as a tool for IoT-oriented user co-creation, was carried out within the scope of WP3 and can be found in D3.4.

Questionnaires from Co-Creation and Exploration Sessions

Questionnaires provide information about the participants, their expectations, and their workshops. The relevant questionnaire analysis with a focus on the additional co-creation workshop will be discussed in the following section. A more detailed analysis about previous cocreation and exploration surveys can be found in D.4.3.2.

Privacy Protection

The privacy protection in the logistics pilot case is taken care of from various perspectives.

(1) The tool kit which is used for the service design (hardware) and configuration (GUI, software) allows marking risk data assessment as private. This marker limits the availability of risk assessment to the local Living Lab (or place of service implementation); marked risk situation-related data is not available for online access via the web app (see Section 3.1.4).

(2) Moreover, access to the web app is permitted only to registered users of the ELLIOT toolkit app. As all created risk situations are saved within the user ID, no data created by other users can be viewed without knowing the particular access details.

(3) The tool kit (or service) does not require personal identification yet. This means that while no services are designed which rely on the data of individuals/specific persons, the service can be used by a universal alias (e.g. for users of the living lab manager account). This lowers the potential of data abuse. In case that a service with personal identification were to be developed, users would have to agree to that usage and the privacy option (see above, (1)) applies.

(4) The data finally provided to the ELLIOT platform does not contain any kind of user specific identifier. The control over individual data remains local and is secured by passwords.

(5) Furthermore, the overall living lab process does not necessarily require data which has to be connected to a single individual. However, two exceptions exist. First, the toolkit has users which are logging in for the service development (configuration during exploration), their profiles are connected to configurations stored earlier. Data collected during exploration could then be connected to users, but not necessarily the users who are logged where naturally more participants are present. Secondly, in case workshops or other living lab activities are documented (video streams, photos, etc.) for research purposes, the participants are informed and have to agree to the usage of such personal data for specified fields (research, presentations etc.), otherwise no such data will be recorded.

3.5

Data Pre-processing and Data Analysis

Sensor Data Analysis

The data collected from sensors is processed by data analysis components, either by Web Services or at the database level, resulting in database views (see Section 3.1.3). How the data is processed is defined by the needs of KPIs (indicators). For more details about KPIs and indicatorproperty relations, see Section 3.2 and Deliverable D4.3.2

The following is an example for data processing based on the expressed KPI requirements. The computation was carried out at the living lab site.

From the living lab database, session data from two experiments was analysed. The goal was to identify relevant configurations or design changes which have a positive influence on the user experience. However, difficulties with the database lead to results which are difficult to compare.

Nevertheless, the data is analysed in terms of the potential of the KSB model as an indicator for user experience and accordingly relevant aspects from the KSB perspectives (knowledge, social, and business).

Each experiment covered two services, the "forklift angle" and the "product temperature".

Indicators in Table 6 were elaborated based on raw sensor data and the applicable data processing.

Problems with data collection

The accelerometer attached to the forks broke down during the initialisation of the service, and a spare part was not available at that time. Therefore, only 26 Minutes of service activity were collected (Table 6, column 2). Furthermore, due to a bug in the toolkit which collects the data and stores it in the database, the number of threshold crossings was not recorded correctly. Therefore, the "product temperature" risk service collected 880 threshold crossings. This error in the data base could not be resolved after the experiment. For that reason, the threshold crossing target value was moved up in order to reach a "relative value".

Explanations

Table 6 contains the processed indicators including duration, number of crossings and the number of available services. Furthermore, for the threshold and limit crossings, the relevant target values are given as well. The missing target value for the duration is 6 hours (see D4.3.2), the target value for publicly available services is the number of risk services covered by one service configuration, in this case there are two risks, therefore the target value is 2.

Analysis

The analysis of the KSB properties for Experiment I following the approach presented in Section 3.2.2 results in the following percentage values for KSB properties (Table 7). The indicators elaborated for the second experiment are shown in Table 8.

Problems with data collection

Previous data collection problems were resolved, but sensor hardware issues affected the temperature sensors this time. Although the sensors were not damaged, they could not be used for data collection during this session. This results in only 55 minutes of data from the product temperature service.

Overall, the forklift angle service operated for 3:42 hours, even though an expected duration would be at least 6 hours (for a shift duration of 8 hours it was estimated that around 75% of the shift the forklift would be in active use, these 75% equal 6 hours)., However, due to the shortened time, the influence of this on the B3.1 Ergonomic Quality property can be observed.

Explanations

Table 8 contains the processed indicators including duration, number of crossings and the number of services available for Experiment II. Furthermore, for the threshold and limit crossings, the relevant target values are given as well. The missing target value for the duration is 6 hours (cf.

D4.3.2), the target value for public available services is the number of risk services covered by one service configuration, in this case there are two risks, therefore the target value is 2.

Analysis

The analysis of the KSB properties for Experiment I following the presented approach in section 3.2.2 results in the following percentage values for KSB properties (Table 9). The 200% for K3.2 and B1.2 in the product temperature service can be explained by the fact that no crossings during the 55 minutes of running service were recorded. For such a short duration, another target value would usually be applied, but the value would reach the maximum of 200%.

Additionally, for the property K2.2 Attunement of Affordances, the threshold crossing durations for the risk "forklift angle" from the final experiment were elaborated (see Table 10).

The eleven threshold crossings which were identified ranged from 1 millisecond to a maximum of 1:45:41 minutes. In average a threshold crossing duration lasted 19:54 seconds. With an assumed target value of 3 seconds (time until the occurred risk "disappears"), this property would, for example, provide a strong indicator for improvement of signaling or training. A deeper analysis of the data reveals that only a few of the occurred threshold crossings were above the target value. Therefore those situations in particular would have to be analysed with the forklift driver in more detail. In general, this indicator would especially help in the comparison of different service configurations and in an observation over time where e.g. learning effects might also influence the results. Since not enough data for the other experiments could be derived, this indicator is not included in the KSB modelling. which then remained between average and high (3 -4) until it rose up to a very high level. This is a distinct development at the end of this workshop in comparison to the other workshops held. The LL IV co-creation workshop shows a unique situation in the beginning where the activity level started higher than before and rose quickly up to 4 participants. However, the activity level decreased shortly after and levelled off at around 3 participants. Furthermore, this workshop finished quicker than the others before (1.5 hours instead of 2 hours).

Based on experiences from previous workshops, this workshop had more pre-set boundaries for the ideation phase. Even though the duration decreased to 90 minutes, the results were relevant for service development and merged into a satellite transport service (this service could not be used for experimentation, but was discussed with industry and business representatives, results

were published and are discussed in Section 3.6.1.7)

Compared with earlier workshops, two major findings are:

(1) A workshop duration of around 80-90 minutes seems appropriate. Workshops should not last too long because the activity level decreases over time in most cases (alternatively a forced break after approx. 90 minutes seems helpful; cf. D4.3.2)

(2) In order to achieve active participation and motivation, the workshop has to take the participants into account and should define necessary boundaries in order to focus accordingly. Furthermore, participants should be prepared in advance; in addition, the workshop facilitator should have the necessary experience in such kind of workshops (cocreation, logistics, IoT, etc.)

Questionnaire

In this section, the main results of the questionnaire analysis and interpretations of the collected data are presented. The mayor part of this analysis was already covered by D.4.3.2; here especially the additional co-creation workshop is taken into account. This section does not include a detailed analysis of responses to individual questionnaire items. Instead, all questionnaires and questionnaire items are presented through an overview of questionnaire sections.

Co-creation Questionnaire Data Analysis

This analysis includes the co-creation workshop from Living Lab group IV (additional data from Living Lab IV is marked with the colour orange and highlighted in bold typeface in the tables).

Personal expectations for the workshop:

The average of the responses by participants to this set of questionnaire items was moderately positive. Average responses to every questionnaire item ranged between slightly and highly positive.

The first co-creation workshop was very positive in terms of participant responses. The majority of participant responses were highly positively regarding their expectation of actively taking part in the workshop. The second co-creation workshop saw higher variance in terms of responses, while the third co-creation workshop had less of a spread of response averages with a majority of slightly negative and slightly positive answers.

In comparison to the former workshops, Living Lab IV reached an average and above responses and also the score regarding question 5 is very high: this supports the previously found commonality between all workshops that found a high expectation of participants 'to actively take part' in the workshop (question 5). Experience with IoT: The average of participant responses was moderately positive. However, there was a distinct difference between the average responses to specific questions. Overall, the responses frequently varied from highly positive to highly negative.

The first co-creation phase saw some participants with significant experience with the application and development of IOT services; however, a high variance in responses was found. The second co-creation phase saw slightly more positive responses than the first phase. But still the questionnaire data revealed a distinct gap between the participants in terms of experience. The third co-creation phase showed more positive results with participants having 'sound knowledge' about IoT services. The spread of responses was narrower. The fourth co-creation workshop fits into the former results, showing a high variance in responses and experience gaps of the participants. Especially the application and development is showing variances whereas an understanding of IoT seems to exist, however not remarkably high. Co-

Creation IV 1 0 -1 0 2 0 1 2 3 1 2 3 4 -3 -3 1 5 2 2 3
Personal attitude regarding IoT: The overall response of the participants was highly positive in the first three workshops. With a very low spread of data, all responses were positive to all the questionnaire items in this section.

Results from the fourth workshop fit quite well with the results of previous tests. No response lies outside previous responses (see Table 13; numbers of answers from LL IV and changed sums are marked bold). IoT can generate services for logistics.

2

(1/0/0/1)

11

(2/3/4/2)

5

(2/2/0/1)

IoT services can increase the safety in logistics.

2

(0/0/1/1)

11

(4/2/3/2)

5

(1/3/0/1)

IoT services can offer relieve for employees in logistics.

1

(0/1/0/0) 8 (3/2/1/2)

7

(1/1/3/2)

2

(1/1/0/0)

Additional Findings from Exploration Questionnaire Data Analysis

Since the BIBA Logistics Pilot focuses on co-creation and understanding of how to involve users / novices into activities, the question of knowledge about IoT is important and hence covered by the questionnaires used in the Living Lab. Based on the exploration workshops, the introduction of the toolkit and the forklift model, additional questions arose e.g. regarding the learning and acceptance support by the toolkit concerning IoT. Therefore, additional findings from the exploration workshop questionnaire regarding the latter aspects are discussed.

When comparing the results of Section B about the personal experience with the workshop (Figure 7), it is noticeable that Group III had the highest self-assessed activity level (B5) as well as the highest score on influencing the decision making process actively (B6). When looking at the other questions in this section, it becomes obvious that, though a very high motivation of the participants of group III is existent, it appears incongruent with the experience level as negative scores could be listed on questions B2 (get an insight into IoT) and B3 (gain new skills).

Therefore, a learning effect on IoT insights as well as new skills is not necessarily dependent on the level of activity of the participants, as suggested by the examination of the results of groups I and II might suggest. A comparison of the average activity level in each group (B5 from section B, see above) with the average results of Section C "Experience with IoT" in each group does not show clear consequences (Figure 8). An increasing level of activity can be observed among the groups, which could be caused by the previously described learning effect of hosting the workshop and resulting increased understanding of the workshop process. However, no congruent results can be found Again, a comparison between the average activity level of the participants and another question is examined, see Figure 9. Here it is proposed to show a dependence between the understandability of the functions of the toolkit and the activity level. As Figure 9 shows, the results are not providing clear evidence. The examination indicates that higher understanding could be related to higher activity of the participants. The difference between the understandability of Group I and II with 1.1 stands out and could again be related to the learning effect of hosting the workshop (see above). In order to find better support for the indicated effects of the learning effect, it would be necessary to carry out further iterations of the respective Living Lab phase.

Usage of ELLIOT Data Analysis Features

For the data analysis in terms of K, S, and B, Web Services were applied at platform level. Those services are applied to the data received from the living lab in order to extract indicators which can be used for property measurement.

The Web Services developed are able to calculate the number of threshold crossings during a shift/session, the number of risk crossings and durations of sessions (based on selected time frames for analysis). Based on these results, the web services provide the metrics Efficiency, Efficacy and "time spend in" (duration) (see Section 3.2.1.1).

Metrics for further KSB analysis are provided at platform level.

Data-KSB Mapping

Elaborated values from the properties mentioned above can either be represented as a percentage terms or in a binary manner: achieved versus not achieved. In the deliverable D.4.3.2, the latter approach was explained. Here, the approach by percentage is applied (cf. Section 3.5.1), as described in D4.3.2.

Finally, with KSB properties all defined and calculated as described, it is necessary to compute them into one final KSB perspective. Therefore, for each KSB perspective the selected properties have to be weighted. In the case that only one property per perspective is selected, the weight per perspective equals the weight of the respective property. Otherwise, weights are applied as described in Table 14. As a summary of this KSB result, the following analysis was obtained (taking into account the limitations of underlying data as explained earlier in this deliverable).

Experiment I -Product temperature service

General summary:

This service shows weakness from the Knowledge perspective.

The KSB results indicate that users become aware of the risks the service indicates, though it is strongly recommended to increase the knowledge content and engagement of the service in order to improve the user experience. Furthermore, a long term observation of this property is necessary in order to see whether the service provides a lasting positive user experience. For the social perspective, the result relates to the S5.1 Attractiveness property (limitations explained before apply). For the first time in use, the achieved duration of the service in use is a good result and indicates that it causes some attractiveness. However, in particular the duration of use requires a long-term observation in order to identify variances which might indicate (decreasing) curiosity due to something "new" rather than the necessary attraction on a long term basis which is aimed-at in this case.

In business terms, the service is quite successful, fulfilling the expectations. However, in the midto long-term perspective it needs be ensured that the service is not overly complicated to use. As a summary for this KSB analysis, the following result was obtained. Even though the data for this experiment is better than for Experiment I, the limitations of underlying data have to be considered:

Experiment II -Product Temperature Service

General summary:

The service shows weakness from the Social perspective

The KSB results indicate that users are learning about the risk situations while interacting with the service, though it is strongly recommended to increase the knowledge content and engagement of the service in order to improve the user experience.

For the social perspective, the results seem to indicate that users are not attracted to the service.

In this particular case, a closer investigation of the data allows the conclusion that due to technical issues the service went down after 55 minutes and therefore the duration of usage was very low.

Thus, for a better knowledge about the social perspective, the stability of the service has to be improved first. However, the model shows the influence of this indicator and therefore makes it possible to identify aspects of the service which need deeper analysis.

Regarding the business perspective, the service supports the expectations in terms of risk reduction; the service seems to perform well. However, the results from the social perspective are also relevant in this context since such a short duration would most probably always result in a positive performance level (no risks occurred). Furthermore, the influence on the Ergonomic Quality property is quite high (in this case negatively) which reduced the outcome at the same time.

D4. As a summary to this KSB result, the following analysis can be provided. In Experiment I, the Forklift Angle service could not be analysed. Although the data for Experiment II in general is better than the data for Experiment I (for details please refer to Section 3.5.1), limitations of the underlying data have to be considered:

Experiment II -Forklift Angle Service

General summary:

The service shows extreme weakness from the Knowledge perspective.

The KSB results indicate that the service does not provide users with the opportunity of learning about the risk situations (Knowledge perspective: 0%) A more detailed view on the data shows that the defined threshold was crossed almost 3 times more often than expected. Therefore, a deeper analysis with feedback from the users is necessary to understand this deviation better.

For the social perspective, the results are positive, however they can still be improved. Particularly in this case, since the duration of the forklift angle services showed weakness in the first experiment and therefore additional experiments are necessary for a more convincing analysis.

Regarding the business perspective, the service supports the expectations in terms of risk reduction, the service seems to perform well, though some improvement is still necessary to create a more complete user experience.

3.6

Results and Inferences

Experiment Outcomes

In this section, the outcomes of the experiment described in the previous subsections are interpreted . Furthermore, an overview of the Living Lab activities in terms of results and inferences is given. This includes a short explanation of industrial involvement and feedback which has uncovered remaining challenges as well as opportunities.

Furthermore, the contribution to the co-creation of services in the physical world in particular is outlined (Section 3.6.1.7).

Attractiveness

Overall, the Living Lab and its toolkit were rated positively by workshop participants and resulted in much interest in discussions with experts. However, the acceptance of the services by the workforce / forklift drivers was mentioned as a critical point to observe. It can be assumed that the service solution design will have a high impact on this aspect.

More insights into service attractiveness for industry are discussed in Section 3.6.1.7 below.

Benefits

On a more granular level, the workshop results were positive, some new ideas were developed and new aspects concerning the application fields or ways of observation were found by the participants. Participants also provided valuable insights into the Living Lab which increased the value of results further. In this context, changes that were made to workshop and Living Lab design were evaluated and adjusted, including the KSB and KPI refinement which was discussed in D4.3.2 as well as throughout this deliverable (see also Section 3.6.2). This work also led to a better understanding of the user experience from a granular view of indicators and properties. The application of the KSB graph provides a useful macro perspective view. Furthermore, the examination of the KSB model itself supported the understanding of goals the developed services (should) address in terms of user experience.

Strengths

The Toolkit is seen as a strong point in the Logistics Living Lab, including its recent extension with demonstration artefacts. In deliverable D4.3.1, an extension of the toolkit by way of physical artefacts had been suggested. This idea was put into practice by placing a model of a forklift (1:6) and suitable pallets and packages into the Living Lab. This model is mainly used throughout the Exploration phase. Participants use it to explain risk situations and to test sensors (D4.3.2). These demonstration artefacts improve the Exploration workshops and help the participants to better understand circumstances and dependencies as well as to reach a common understanding of the toolkit (see also opportunities below).

Weaknesses

Even though important aspects have already been considered, some weaknesses are still remaining. On the one hand, the model was not fully applied, the majority of properties were left out. This was helpful and necessary in order to understand how to apply the model, but future work may benefit from a broader view on the model; covering more properties by different usecases in order to understand to what extent it could be applied and how the application could be improved. Furthermore, the individual properties are not yet completely measurable by the sensor data and measuring approaches available in the Living Lab.

Moreover, the current development status of individual services (from Living Lab groups) and the integrated solution cannot yet be considered to be a "final product". The case-specific characteristics imply that a final solution is not available from the start. This means that more time is necessary for the development of integrated solutions for industry. It is still a great challenge to develop solutions for the physical world (i.e. not software) in a Living Lab or even to apply such solutions successfully.

Opportunities

The results of the questionnaire data analysis indicate that some challenges lie ahead regarding the management of expectations (what to expect from the workshops), taking into consideration the realistically attainable learning experience during these co-creation and exploration workshops. However, the exploration workshop in particular seems to lead to the workshop participants gaining knowledge about and understanding IoT technologies and their potential.

Thus, the co-creation experience in particular should be focused on when redesigning the workshops; expectations could be more closely connected with the follow-up Living Lab workshops (e.g. exploration) to increase their appeal and prevent a loss of motivation during cocreation. The serious game could be used to support this strategy, for instance with extended content aiming at IoT exploration and activities.

Overall, experiences and analysis of the Living Lab workshops, groups and results indicate potential for IoT co-creation in highly specialised application areas. This includes working together with trained personnel in the field of sensible goods transportation. Showing these experts the functionalities and related potential of IoT technology should increase their involvement in the development of new products and services.

The observations made on the toolkit use, including the added artefacts, provide evidence that this approach might be helpful to support IoT-oriented user co-creation. Observations have led to the conclusion that combining the toolkit with artefacts helps in discussing and understanding challenges of the specific use-cases and risk scenarios. However, attention that would ideally be focused on the IoT usage may be diverted to the specifics of the toolkit and its artefacts. It was observed that the toolkit artefacts can distract the participants, for example when participants start to discuss where and how to assemble the sensors. Such discussions can easily move away from the initial goal to discuss the sensor and actuator setup of the service solution. This challenge needs further attention to select an accordingly adjusted workshop design. Some related aspects were discussed in WP3 deliverable D3.4.

Threats

Already in D4.3.2, results and discussions from the Living Lab workshops indicated one important aspect relevant for the utilisation of an IoT toolkit for the development of (IoT)

services. The current prototype model of the toolkit is subject to relatively frequent technical issues. Such failures cause disappointment and can lead to a lack of interest in workshops/cases by participants. Using a system that is still at the development stage would quite likely lead to disappointment and reduced adoption rates if it was used for a service development in an industrial environment. Additionally, when developing services with non-experts / end-users, these groups of users will have to deal with "daily business" as well as with participating in workshops. Technical failures causing delays would then be likely to lead to dwindling acceptance for user co-creation. This problem remains a difficult aspect; however, it has to be seen in the context of co-creation which was investigated by the Logistics use-case Living Lab.

Contribution to Living Lab (CoC) Research and Industry

In dealing with the challenge of providing an environment which enables novices, field experts and technology experts to participate equally in co-creation activities, a mayor question is whether or not the environment and the applied tools and methods could fulfil this task in different group formations and the corresponding tasks (also related to deliverable D4.2.2). The core tool which was used during exploration and experimentations is the Arduino IoT Toolkit, introduced to the LL in order to support learning and training of the IoT service development (in the co-creation phase the Serious Game is the mayor tool; WP3). Workshops were carried out to develop the service prototypes.

The toolkit was evaluated regularly based on user feedback and questionnaires (see deliverable D3.4). As an important result, it was found that the toolkit as the core element of the exploration phases in the BIBA ELLIOT Living Lab had a knowledge impact regarding IoT. Furthermore, the way the toolkit supports co-creation activities and enables participants to physically create

IoT services in the real world, rather than creating virtual solutions via platforms, provided important findings for the understanding of co-creation; these are relevant contributions for the co-creation research especially at the transfer between virtual and physical solutions (cf. [START_REF] Kalverkamp | Logistics IoT services development with a sensor toolkit in an experiential training environment[END_REF]Baalsrud Hauge and Kalverkamp 2013).

Beyond the findings about co-creation, IoT solutions were developed in several Living Lab groups with the help of the toolkit. One Living Lab group in particular developed an IoT service supporting the safety of satellite transports. The service concept was tested at the BIBA Living Lab and three field experts were interviewed for feedback. It could clearly be derived from interviews that IoT services, which enable a real-time assessment and analysis of sensor data, could significantly improve transport processes, especially for highly sensible goods. Two experts from the space industries confirm that current usage of offline sensor systems and data logging units requires time-consuming (and expensive) readouts and data analysis to extract the needed information. This information can then be used to identify actions to be taken and to improve the process in the future. An IoT system, which provides relevant information to involved personnel and/or keeps other stakeholders informed about potential risks arising risks in real-time, is seen as a valuable tool to improve such transports. However, the industrial implementation still has to be proven (e.g. costs vs. increased service quality (i.e. better transparency)).

The expert feedback shows a clear need for IoT systems and the Living Lab approach together with the IoT toolkit could be used to develop a service concept covering the whole Living Lab cycle. Additionally, other industry contacts (e.g. at Hannover Messe 2013) showed interest in the modular approach realised by the toolkit and GUI (cf. D5.3); and additional events are planned to further attract industry (e.g. at a DHL Think-Tank event 3 rd /4 th July 2013).

KSB Analysis

In the deliverable D4. The City of the Future Living Lab is involved in the ELLIOT Project by using different scenarios.

Each scenario has a unique set of goals in terms Knowledge-Social-Business expected benefits. The "Media Scenario" is intended to provide a simple service to make children's hospitalisation at the OSR paediatric department more tolerable; this is done by way of leisure activities and motivational support during the hospitalisation period through an interactive totem able to provide applications including meal ordering, leisure services and gaming to hospitalised children. The application use has been monitored through various types of data logs, making it possible to collect data on frequency of attendance and quality of usage. The activity in the Media scenario experimentation phase consisted of the investigation of the outcomes gained from the co-creation phase and the subsequent development into prototypes and mock-ups for validation. Data was collected in this phase, and KPIs for the KSB model were also identified. In continuous exploration of the Living Lab process of the scenarios, the issues and considerations emerging from the experimentation phase were discussed regarding the way data was collected, as well as The Tourism Service Scenario aims to promote physical activity as a well-being tool for health, rehabilitation, leisure and tourism practices. This is made possible by providing a service based on a wearable monitoring device, a smart-phone app and a web portal. This service is evaluated involving volunteer participants into an (at least) 3-day usage experience where they wear the monitoring device and use the related smartphone application and web portal. In this way they are able to monitor their body performance indicators during daily activities. At the end of the experience, a questionnaire based on KSB aspects was given to each participant to explore the scenario from the consumer point of view.

Personalised Service and Public Transport Scenarios were set up in a temporary store where data were collected and analysed together.

This temporary store in the commercial area of the San Raffaele Hospital, was able to:

-dispense beverages and food;

-help user to become more aware regarding nutritional behaviours (i.e. showing nutrient data and healthy diets);

-provide mobility information regarding the hospital private automatic metro line and the public transportation system of Milan.

To achieve these objectives, two innovative vending machine and one coffee vending machine were made available to the public; they have been attended by various type of users, such as patients, occasional visitors, returning visitors, clinicians, employees and others. Users are involved in an indirect way here: KSB model analysis have been carried out based on data acquired from vending machine (such as products or bounce rate).

KSB Instantiation

Media Scenario K

K3.2 Mental mapping

Use of educational application of Totem that improve conceptual mapping and makes it possible to link different mental models and their meanings.

K4.1

Human computer interaction

Use of any application of Totem that implies human-machine interfaces and cognitive artefacts. Access to new functionalities such as meal ordering app and welcome videos.

B4.1 Usefulness

Use of whatever application of Totem that implies high levels satisfaction for the service provided.

K.5.2 Selfconsciousness

Experience feelings; wakefulness; having a sense of selfhood; or the executive control system of the mind. It creates new connections with mental models.

S

S1.1 Social Networking

Ability to establish positive social (interpersonal) ties as information carrying connections among people (social networking).

B

B3.1 Ergonomic quality

The degree to which the design optimizes human well-being and the overall system performance.

B.3.2

Ease of use The level of usability.

B.3.3 Learnability

Learnability of a human-made objects or artefacts. Easier to learn as operations can be learned by observing the object.

B4.2 Emotional Connection

The affinity the user feels for an object that appeals to her/him, due to the formation of an emotional connection with the object.

B.4.3 Hedonic quality

Subjective evaluation of the experience positiveness in terms of pleasure, fun, cool, originality, innovativeness, interesting, engaging, appealing, desirability, comfort and attractiveness. Table 17: As part of the instantiation of four scenarios and their evaluation through the use of the KSB model ultimately developed a tool able to evaluate the impact on the user experience in an objective manner and to communicate the results to the public in a quick and comparable manner. Moreover, the data collected are unfiltered and so it has been possible to gain direct and objective access to the user experience and to uncover insight from users that other methodologies were unable to disclose.

The ELLIOT platform has allowed the City of the Future Living Lab to involve users in an ubiquitous and continuous innovation process; experimentation has also confirmed the ability for the system to maintain user privacy and protection, but at the same time allow the research team to collect a large amount of data from a vast number of receptors located over a wide area in a fast and efficient manner.

Once the ELLIOT platform has been implemented and results have been acquired, activities such as published articles, participation to international conferences and Summer School increased which in turn have augmented competitiveness of the City of the Future in Italy and in Europe.

Design Changes

Media Scenario

The Media Scenario in this project has been primarily used to understand how to acquire data and how to send them to the Elliot platform. So the evaluation of this scenario through KSB model acts as the first attempt to value what KSB elements are the most meaningful for an automatic data logs analysis coming from complex data sources (such as semantic analysis of video coming from a webcam to calculate the number of interactions with a totem).

To enable the KSB evaluation of the service provided in the Media Scenario, a totem has been placed in the paediatric department at OSR. This totem is focused on children, so games provided are developed for entertainment but also for education activities. They are:

 "Blinken"  "TuxPaint"  "I giochi dello Zecchino d'oro"  "Letters and Numbers"  "Sudoku"  "Gcompris" (an educational software suite comprising numerous activities for children aged 2 to 10)  "Potato Guy" Moreover, a welcome video explains activities of the department and try to involve children and help them during their hospitalisation period.

The innovative service provided also contains a meal ordering application accessible by children themselves so they can select the meal supported by a decision support system able to help them to compose and order their meal (for example with more vegetables) in an nutrition-aware and entertaining way.

The usage of the Totem has been acquired using data-logging by the operating system; in this way it has been possible to analyse data of what application is selected, when it has been started and when it has been closed. Moreover, the same data logging has been used to collect data coming from a webcam to count the attendances; in this way is possible to identify and delete

Tourism Scenario

The Tourism Scenario was very important for the ELLIOT project, because through its evaluation it has been possible to understand how to instantiate the KSB model inside the User eXperience evaluation. Here we provide the explanation of the wearable monitoring device service provided to evaluate the Tourism Scenario.

The tests related to the Tourism Scenario try to evaluate the usability of a wearable sensor used to monitor physiological parameters to maintain and improve the health status of users and their awareness about this issue, motivating them to undertake physical activity in their spare time.

This device is able to collect data related to various physiological parameters and to communicate them to a smartphone that will deal with a first exposure to data of the user and then communicate them to a web platform.

The parameters that can be monitored by the user are the following:

-HR, heart rate (in rpm); The test campaign covered various elements:

-a wearable monitoring device of body parameters;

-a smartphone application developed to allow the end-user to monitor (and send to the webportal) data acquired from the sensor through a smart-phone device with the Android operating system;

-a web portal used to monitor activities daily, weekly and monthly and to access personal profiles, regarding habits and lifestyle. The material available for each participant was as follows:

-The sensor SPMHBGV3 developed by STMicroelectronics based on our requirements to evaluate the Scenario (complete of platform for re-charge, and adapter for wall outlets Ltype at 10 Ampere);

-Some patches Patch B-B-V3 made by Spes Medica to apply the sensor (and the electrodes)

to the chest;

-A smartphone Samsung Galaxy ACE (GT-S5830) or (depending on availability and type of activity carried out by participants) a tablet Samsung Galaxy Tab (GT-P1000) or an ASUS Transformer Prime TF201, all equipped with the Android operating system, with the ability to connect via BlueTooth v2.1 and WiFi 802.11b/g/n; all devices were provided with a specific smartphone application that was developed to display the data coming from the device SPMHBGV3 and so send them to the web portal. A first evaluation test was performed with 10 participants selected from the City of the Future Living Lab at the Hospital San Raffaele, who were asked to follow these steps:

-Register themselves on the web portal; -Complete the questionnaires on lifestyle and personal profile;

-Use the service for at least 3 days during which they would be wearing the sensor (and patches) for at least 12 hours a day.

In conjunction with the supply of material, a brief explanation of the service and a set of instructions to access and use the service in proper manner were also provided.

The test began in November 2012 and was completed in December 2012.

After a 3-day evaluation, each participant was asked to complete a questionnaire.

This questionnaire was divided into three distinct parts: one on the sensor and its patches, one on the applications installed on an Android device and one regarding the web portal. All question contained are strictly related to a specific KSB measurement presented in Chapter 4.1.2. In the second evaluation, Data was automatically collected and stored for the elaboration. This was very important in order to understand how to instantiate the KSB Model/Triangle for the first time.

The KSB triangle was instantiated in a Ternary Plot (or de Finetti diagram). This is a barycentric plot on three variables which sum to a constant. It graphically depicts the ratios of the three variables as positions in an equilateral triangle. The advantage of using a ternary plot for depicting compositions is that three variables can be conveniently plotted in a two-dimensional graph.

So in our ternary plot, the proportions of the three perspective must sum to some constant. To do that the three variable to be used are not the absolute values of the three perspectives but are the percentages of the single perspectives characterizing the entire service. In this way it is possible to explain where the barycenter of the observed service is and whether it tends toward a specific perspective. So these relative values that rely on Relative Perspective Indexes can be easily obtained just knowing the Absolute Perspective Indexes (PIK, PIS and PIB stand for Knowledge Perspective Index, Social Perspective Index, Business Perspective Index) that range from 1 to 5 and divide it with the sum of the three absolute perspective indexes.

So:

𝑅𝑃𝐾 = 𝑃𝐼𝐾 ∑ 𝑃𝐼 (%) 𝑅𝑃𝑆 = 𝑃𝐼𝑆 ∑ 𝑃𝐼 (%) 𝑅𝑃𝐵 = 𝑃𝐼𝐵 ∑ 𝑃𝐼 (%)
In this way is possible to give a value for every KSB property, in a scale that is used to evaluate the parameter service provided: all these KSB properties can be inserted in the right perspective (Knowledge, Social and Business) and the right experience type ("Sensory, Perspective, Appreciation,…") so as described in the KSB instantiation chapter.

After the elaboration of the questionnaire we found a KSB actualisation of our services (Figure 29) and we understood how to use and apply the KSB Model in our service evaluation.

Data Collection Changes

As mentioned before, the most important changes in Data Collection have been applied in the Tourism Scenario, where, in order to understand how to instantiate the KSB Model, we applied a manual evaluation of the KSB Model.

After the 3-day experience with our patch, each participant was asked to complete a questionnaire.

This was divided into three distinct parts: one on the sensor and its patches, one on the applications installed on the Android device and one regarding the web portal. All question are explained in Chapter 4.4.2.2; they are strictly related to a specific KSB measurement presented in Chapter 4.4.2.2. Device log data have been also collected in order to apply the automatic IoT-based evaluation.

The questionnaire was managed in the following way: all participants were asked to evaluate the wearable monitoring device and its patches (the first section of the questionnaire). Half of the participants were also asked to evaluate the smartphone application and the web portal (second and third sections).

For each phase, a different KSB triangle was elaborated in order to understand the difference between user experiences related to different parts of the service.

The automatic IoT-based data collection for each scenario did not undergo any change and it is possible to refer to deliverables D4.3.1 and D4.3.2.

Data Analysis Changes

The data analysis was performed for each scenario in the traditional and the "Elliot" way in order to better understand the potential and issues of this new evaluation approach.

After the KSB instantiation we also attempted to transfer Elliot results into human readablemessages trying to associate sentences in respect to the actualisation of the UX inside the KSB Triangle. In our opinion this could be grounds for further research in fields such as semantics and human computer interaction. In other cases and when necessary, the products and services are exposed to a much more restricted number of users, as with for example the media scenario where only the patients of a single ward are involved, and these are subsequently filtered according to their age, abilities and by other specifications required by the scenario. In the case of Media Scenario the participants involved in the experimentation are hospitalised children in a paediatric department (about 40 of them per week and who change from week to week). They had free access to the device called "totem".

We collected data from different numbers of participants in order to understand the participation in the experimentation phase of all our scenarios that are summarised in the following figure:

Data Collection

As previously mentioned, data related to each scenario will be generated by two observation steps: the first step is the co-creation phase, the second step is the real evaluation of scenarios that can be done through sensor data analysis and observation/survey data analysis.

Sensor Data

Media Scenario

Data collected in this scenario comes from a data logging application developed from our team.

Here is an example of data logs created by the application: In the first column it is possible to see the number of children in front of the Totem, then it is possible to see the time and date of execution and the application launched by the children.

…

Personalized Service and Public Transport Scenario

Data collected by the innovative vending machines regard customer purchases and their presence in front of the vending machines. Some examples of data collected are in the deliverable D4.3.2.

The following table illustrates all the data collected via the IOT system for the vending machine scenario, derived from the touch screen, video camera, proximity sensor, food dispenser, and smart card reader, as well as from the activities registered by the software as system logs pertinent to the Graphic User Interface.

Co-creation Data

Regarding the first step, it is possible to map the flow of people accessing the Gaggiots web site (http://www.gaggiots.com), where a set of serious games were provided to engage people in the co-creation process; during this co-creation phase a large set of data regarding serious game participants and their answers has been collected.

Participants were asked to play to earn points depending on their answers and how other participants appreciate their answers; in this way a huge set of ideas to empower the service presented in serious games has been collected. The following questions were provided in serious games to enable the Co-Creation process regarding all our Scenarios. The totem in question has a touchscreen, a video camera, microphone and speaker, an internet connection and an RFID reader. This gaggiot is about pushing the boundaries of traditional automatic distributors and imagine a world were vending machines could interact with their users and offer them something more than just fast products"

And with a video (screen shots are below): This gaggiot is about pushing the boundaries of personal healthcare through the introduction of wearable sensors and a service that can help users to take action and know more about their health."

The questions provided were:

Q1: Imagine that Smart Patch, could sense proximity; weather; touch. The test began on Friday, November 9, 2012 and was completed on Monday, November 25, 2012:

after each evaluation (3 days at least), each participant was asked to complete a questionnaire.

This questionnaire was divided into three distinct parts: one regarding the sensor and its patches, one regarding the applications installed on Android device and one regarding the web portal. All question contained are strictly related to a specific KSB measurement presented in chapter 4.1.2.

As explained before, the questionnaire can be divided into three parts: each question refers to a specific KSB property, the KSB property acronym has been appended with letter "p", "a" or "w" depending on whether it refers to patch, mobile application or web portal KSB evaluation. This is specifically explained in Chapter 4.2.3.1 "Scenario observation and survey data".

Regarding the sensor and related patches, participants were asked to evaluate the following statements (from No. 1 to No. 9) with a rating from 1 to 5 in which the value 5 indicate perfect agreement with the phrase reported, the value 1 instead indicate completely disagreement. Regarding the smartphone application, participants were asked to evaluate the following statements (from No. 12 to No. 20) with a rating from 1 to 5 in which the value 5 indicate perfect agreement with the phrase reported, the value 1 instead indicates completely disagreement.

12. The text in the app was easy to read (B3.1a ergonomic quality) 13. The language and the graphic of the app were easy to understand (B3.1a ergonomic quality) 14. The navigation of the app was easy to understand (B3.1a ergonomic quality) 15. The app was easy to use (B3.2a ease of use) 16. It was easy to learn how to use the app (B3.3a learnability) 17. The app was innovative and original (B4.3a hedonic quality) 18. The app helped me create new social relations (S1.1a social networking) 19. When I saw the app I was interested in using it (K2.4a appealingness) 20. The more I used the app, the more I wanted to use it (B4.2a attractiveness)

This part of the questionnaire ended with two open-ended questions: 21. How did you feel whilst using the app? (K5.2a self-consciousness) 22. Did using the app change you in any way? (K5.1a self-examination)

Regarding the web portal, participants were asked to evaluate the following statements (from No. 23 to No. 31) with a rating from 1 to 5 in which the value 5 indicates perfect agreement with the phrase reported, the value 1 instead indicates complete disagreement.

23. The text in the portal was easy to read (B3.1w ergonomic quality) 24. The language and the graphics of the portal were easy to understand (B3.1w ergonomic quality) 25. The navigation of the portal was easy to understand (B3.1w ergonomic quality) 26. The portal was easy to use (B3.2w ease of use) 27. It was easy to learn how to use the portal (B3.3w learnability) 28. The portal was innovative and original (B4.3w hedonic quality) 29. The portal helped me create new social relations (S1.1w social networking) 30. When I saw the portal I was interested in using it (K2.4w appealingness) 31. The more I used the portal, the more I wanted to use it (B4.2w attractiveness)

This part of the questionnaire ended with two open-ended questions: 32. How did you feel whilst using the portal? (K5.2w self-consciousness) 33. Did using the portal change you in anyway? (K5.1w self-examination)

The questionnaire was managed in the following way: all 10 participants were asked to evaluate the wearable monitoring device and its patches (the first section of the questionnaire). 5 participants were asked to also evaluate the smartphone application and the web portal (second and third sections).

Data Collection and Privacy Issues

The data collection during each phases of the Living Lab process inside the City of the Future

Living Lab makes it important to adequately address the relevant issues regarding user privacy.

Each scenario has tackled this constraint by acquiring and processing data in an anonymous way.

So it has been possible to collect a huge dataset without any reference to specific person but only with data regarding anonymised choices or activities and their data and time (see Figure 33Error! Reference source not found. for an example in the Media Scenario).

In particular, video streams are destroyed in real time, just after processing that permits face counting, gender/age recognition, and other algorithms that provide -always anonymousclassification of the user.

Furthermore, to keep the users and participants of living lab activities informed, posters have been placed inside the Living Lab facilities. For example in the temporary store a poster explains terms regarding privacy protection and how data are acquired and used, for example data regarding number of accesses inside the store come from images that once processed to extract anonymous data are immediately destroyed.

.1 Media Scenario

To enable the KSB scenario evaluation, the post processing of logs and sensor data extracts different KPIs from data logs presented in Figure 33; these KPIs are extracted for 3 different groups of children ("prescolare", "primario", "secondario", based on their age) and for the entire population of the experimentation.

The KPIs extracted are:  KPI1, Total running time of all application (all kinds of application so it comprise games, welcome videos and meal ordering).  KPI2, Total running time (it starts when sensor detect the presence of a person in front of the Totem and it stops when the sensor detect that no one is standing in front of the totem).  KPI3, Total running time of educational games (the games are Blinken, Letters and Numbers and GCompris gameset).  KPI4, Total time with 2 or more children standing in front of the totem and using an educational game.  KPI5, Total time with 2 or more children standing in front of the totem and using whatever application.  KPI6, Total time with 3 or more children standing in front of the totem and using whatever application.  KPI7, Total time using meal ordering application or welcoming video

Personalised Service and Public Transport Scenario

Data coming from vending machines are automatically analysed to provide a set of KPIs to describe the service provided, the typology of product purchased, the frequency and duration of purchasing session, the bounce rate and so on. Moreover, a complete set of indicator have been extracted for each category and macro-category of products, also to describe the user experience. In the same way a complete set of KPIs can be used to generate a KSB model; moreover, all of these values can be generated for a specific time. For our purpose, two time samples will be used, October 2012 and February 2013; during these months there was a huge re-do of the user interface of the vending machine, so the difference are relevant:

Some remarks:

-A "session" is launched with the proximity sensor activation and is stopped by a purchase or by the proximity sensor deactivation. -A "passage" is counted when the proximity sensor is activate and then deactivate within 4 seconds. -A "Menu" is a mix of 2 or more products purchased in one interaction.

-"Contiguous sessions" are sessions that have less than 10 seconds of time distance between each other. -"Automatic info pages" are: product basic info and pills of nutritional knowledge; they are displayed automatically during the product purchasing. -"Non-Automatic info pages" are: nutritional info, Mediterranean diet info and infomobility. They are displayed only if customer clicks on related icons. -"Healthy products" are those products purchased by vending machines that can be defined as "healthy" based on FSA (Food Standard Agency) traffic lights limits (see Figure 40). For our specific purpose, products defined "healthy" are those products which have fat, saturates fat, salt and sugar below the high limit, so they could be associated with green and yellow labels.

Observation and Questionnaire Data Analysis

Co-creation serious game data

As previously mentioned, data related to each scenario will be generated by 2 observation steps: the first step is the co-creation phase, the second step is the real evaluation of the scenario.

Regarding the first step is possible to map the flow of people accessing the Gaggiots web site (http://www.gaggiots.com), where a set of serious games where provided to improve involvement of potential users in co-creation processes; During this co-creation process a large set of data regarding serious game participants and their answers has been collected.

Media Scenario

The serious game provided to assess the Service of Media scenario is called "Totem for Children".

The following plots shows data regarding attendance: The following graphs depict the answers to the questions proposed during the serious game:

Q1: Imagine that totem for children, could sense emotions; physiological data; presence.

How could this new sense change the totem for children, adding value to what it offers its users?

Figure 44: number of choices by item regarding Q1.

In the previous graph the type of sense chosen by participants and how many appreciations ("likes") each sense has got can be viewed: it can be highlighted that participants focused on a service that can understand how a child feels and can act in an appropriate manner.

Q2: Think of best friend. What's so special about this figure? How does it distinguish itself from others? Take these qualities and apply them to the totem for children and describe how it could evolve if it had these same qualities. Be as creative and blue-sky as possible! Have fun!

For question Q2 it is impossible to select a specific answer, so no data analysis could be performed but it is important that all the answers explain that even if the totem could feel emotions participants do not believe that it could replace a real best friend.

Q3: Think of a WII console. What's so special about this product? How does it differ from other products? Take these qualities and apply them to the totem for children and describe how it could evolve if it had these same qualities. Be as creative and blue-sky as possible! Have fun! Also for question Q3 it is impossible to select a specific answer so no data analysis could be

made. The answer that received more "likes" suggests that if the totem had the same capability as the WII it could become an entertainment tool not only for a child, but also for her/him together with her/his family. In the previous graph it can be seen that all three verbs have the same importance, but participants seem to find the teaching aspect inside a hospital structure seems very interesting.

Q4:

Q5) what if the totem for children was used mainly by adolescent?

In what way could it respond and react in this user? What could it do or offer that is unique to this user?

All answers state that adolescents will probably not be interested in Totem usage.

Personalised Service and Public Transport Scenario

This scenario is presented in the Gaggiots portal through 2 serious games: "Innovative Vending Machines and "Mobility Service". Due to a low participation in the second serious game and due to more compliance of the first game to the service provided by real vending machines, only the results from "Innovative Vending Machine" serious game will be shown. Many of the ideas focused on time spent waiting after the had been product purchased: it could be used with activity such as games, network oriented information and discount (on healthy service or personal care activities) based on the chosen product or other healthy activities.

Q2) Think of an iPhone. What's so special about this product? How does it differ from other products? Take these qualities and apply them to the Innovative Vending Machineand describe how it could evolve if it had these same qualities. Be as creative and blue-sky as possible! Have fun!

Answers try to explain how the service can be "cool", for example with smooth sounds and movements. Suggestions include trying to change the appearance or offer cool moments to involve the costumer. It is also important to develop the vending machine in such a way that it will not be useable only for a restricted number of user (such as only for users of Apple products).

Q3) What if the Innovative Vending Machine was used in a school. In what way could it respond and react to this context? How could it evolve to fit in this new context?

Participants propose a vending machine able to inform children about their habits and able to involve them in educative games. Also, innovative vending machines could be programmed to recognise children and prohibit them from purchasing products during lesson time.

Q4) What if the Innovative Vending Machine was used mainly at night time. In what way could it respond and react in this timeframe?

Many participants focused on the feeling of costumers. On one hand the vending machine can provide extras such as virtual flowers or presents to customers; on the other hand it could provide safety and security services such as the closing time of some services, help buttons, or automatic alcohol tests; it could also be connected to the CCTV system.

Q5) Think of a doctor. What's so special about this figure? How does it distinguish itself from others? Take these qualities and apply them to the Innovative Vending Machineand describe how it could evolve if it had these same qualities. Be as creative and blue-sky as possible! Have fun!

All participants focused on trustworthiness aspects: vending machines could promote correct choices to improve quality of life based on specific person characteristics. Also people feel that they can trust the fairness of the service and rely on it. The co-creation process focused on the Tourism Scenario is provided through a serious game based on a "smart patch", here the data collected related to participants involved in this serious game can be viewed. In the graph above it can be seen what type of sensor is chosen by participants and how many appreciations ("likes") each sensor has received.

Q2: In What way could you imagine improving the flexibility of the Smart Patch? Think of practical examples and try to describe them in as much detail as possible.

For question Q2 it is impossible to select a specific answer, so no data analysis could be carried out. The only data that could be extracted is the total number of responses: 6. Moreover no "likes" were added by participants.

Q3: What if the Smart Patch was used mainly by senior citizens. In what way could it respond and react in this user? What could it do or offer that is unique to this user?

Also for question Q3 it is impossible to select a specific answer, so no data analysis could be carried out. The only data that could be extracted is the total number of responses: 6. Only 2 appreciation where related to this answer: "The patch could have a panic button (slap the patch to call for assistance) or sense if the senior citizen has fallen on the floor and cannot get up." In the graph above is shown which verb has been chosen by participants and how many "likes" each verb has received.

Q5: Think of LinkedIn. What's so special about this service? How does it differ from other services? Take these qualities and apply them to the Smart Patchand describe how it could evolve if it had these same qualities. Be as creative and blue-sky as possible! Have fun!

For question Q5 it is impossible to select a specific answer, so no data analysis could be carried out. The only data that could be extracted is the total number of responses: 7. Moreover, 6 "likes" were added, equally distributed for 6 different answers.

Scenario evaluation data

Tourism Scenario

All data collected from this scenario are coming from questionnaires provided to 10 participants as presented in Chapter 4.4.2: they were employees and researchers from Ospedale San Raffaele City of the Future Living Lab. Participants were equally distributed by sex and are from 25 to 34 years with an average age of 29 years old.

After having worn the device for at least 3 days and for at least 12 hours per day, it was possible to value usability in different situations: working activities, household activities, transportation, sport activities and sleeping. Below (Table 20) the average value provided by participants during the survey after the 3 day evaluation activity can be viewed. The table show only answers to closed questions so from #1 to #9 regarding wearable monitoring device and patches, from #12 to #20 regarding the smartphone application and from #23 to #31 regarding the web portal.

The analysis of the open-ended responses given by participants (questions #10 and #11, #21 and #22, #32 and #33) allows a further enrichment of the evaluation.

Concerning the sensor and related patches (questions #10 and #11) a general commitment by the user to check that the sensor has been turned on or whether low battery has been indicated. 4 participants indicated a slight feeling of itchy skin but no one has reported a skin irritation. 3 out of 10 have shown that wearing the sensor has positively affected daily physical activity (physical activity both as an incentive and as a tool to be more aware of the physical effort). Three persons have instead indicated that the presence of the sensor was perceived during a particular body movement or activity. Two persons have shown a slight feeling of detachment of the patch especially by the end of the day.

The responses relating to the application for smartphones (questions #21 and #22) have shown a strong interest in the data, but at the same time a series of negative assessments of some deficiencies in the application or some graphical presentation of data-related failures between the application and the web portal. In addition, two people have explicitly emphasised the discomfort in having to maintain a Bluetooth connection between the sensor and Android smartphones. A commitment on the part of the user to check that the sensor worked properly (checking for right charts of BR, HR, ECG and METs) was also shown.

Regarding the portal (questions #32 and #33), the evaluations tend to emphasize the absence of any other relevant information on the data submitted with the smartphone app. It was also highlighted that the lack of such as time reference to be able to recognize different activities carried out during the sampling, the need to be able to actually change the type of activity associated with each registration and lack of confirmation messages during saving or other data post-processing were considered negatively.

Data-KSB Mapping

In order to obtain a proper KSB mapping, all data recorded during the evaluation process must be post-processed. This is done by composing the values of K, S and B perspective properties starting from collected data coming from data logs, sensors and questionnaires.

Media Scenario

The KSB mapping of previously collected KPIs refer to this table: here KPIs are used to create a set of KSB properties. These KSB properties can also be computed as a percentage value or with a scale from 1 (worst value) to 5 (best value).They are collected with different relative weight to obtain a KSB perspective value.

Tourism Scenario

As explained before a KSB mapping can be created for each of the three components of the Transport Scenario (wearable monitoring device, mobile app and web portal). Regarding properties B3.1 it can be noticed that value refers to three answers (Q1, Q2 and Q3 for B3.1p, Q12, Q13 and Q14 for B3.1a and Q23, Q24 and Q25 for B3.1w). To measure the value for B3.1 property (Ergonomic quality) the same weight has been selected for each of the three questions (to create a simple mean).

The following table show the procedure to obtain the B3.1 value for the wearable monitoring device.

D4. Regarding K5.1 and K5.2, to obtain a value starting from an opened-ended question the following steps have been followed:

1. For each participants and for each question identify every item explained inside the answer.

2. For each answer identify the total number of items explained by all the participants.

3. For each participant and for each question the related value is just the number of item explained by participants divided by total number of items explained for that question (this value must be multiplied by 5 to obtain a value comparable with other KSB properties).

4. The average value of the property can be obtained through the average value for each participant.

The following shows an example of how to evaluate the K5.2 property from an open ended question:

Q10 answer by User #1:

"It was not very comfortable, especially when I was doing some activity, and because it seems as to fall off and because I always have to bring back the tablet/smart-phone.

I often check if the device was on and if it was properly communicating the data because several times it does not communicate to smart-phone.

I felt itching in the area of the patch.

I really liked the device and I think it give up me to be more active! The only problem is that the device is very big and visible.

Trying to make some swing during golf activity I noticed that it was quite uncomfortable to wear while playing golf!"

As reported into the KSB experience model, the K5.2 property can be evaluated following the related description ("Experience feelings; wakefulness; having a sense of selfhood; or the executive control") and evaluation type ("Amount of established connections"). So reading the text written by user #1, it can be noted that there are five established connections from causes to consequences.

The same procedure can be executed for every participant/user. After that all connections (excluding redundant ones) can be collected. For a wearable device, the total number of connections established is 12. So the valuation of K5.2 regarding user #1 is 2,92, while the mean value is 2,08.

Following these steps the average values for each KSB property are the following: As can be observed, values of all KSB properties value vary between 1 and 5 (as answers extracted from questionnaires). So all these KSB properties can be clustered to develop an overall KSB model able to measure the knowledge, social and business aspects of the IoT services. That is possible, because KSB properties refer to a specific K, S or B perspective and contribute to value it from 1 to 5 depending on value and relative weight.

Results and Inferences

Experiment Outcomes

Media Scenario

The KSB perspective value depends on the importance of each property. The table below explains the weight for each KSB property. Based on these values, a KSB ternary plot can be created for each category. As previously explained in 4.2.3 "Data Analysis Changes", after setting the "Perspective Discovery" related to each KSB property, it is possible to obtain the Perspective Knowing of each perspective. In the Media Scenario, thanks to the Elliot analysis is possible to determine the 60% of the Knowledge Perspective, 75% of the Social Perspective and 80% of the Business Perspective. Due to the high quantity of data acquired, Perspective Knowing is set to be a constant value through each data set ("prescolare", "primario", "secondario" and "total").

These are the KSB perspective values obtained from KSB ternary plot: As can be seen in the ternary plot, the Perspective Knowing is intended to be the same for all the four KSB evaluations. As can be seen from the ternary plots presented above, the perception of service by hospitalised children is quite balanced between perspectives. Some differences can be highlighted focusing on each age cluster: "primario" and "secondario" groups are quite unbalanced on the Knowledge perspective and in the meanwhile express a lack on the Business one. The analysis based on the "prescolare" group highlights an unbalancing in the Social perspective.

The "Total" group analysis finally confirms a weakness on the Business perspective (present on all the analyses made). This confirms the objective of the Totem that is mainly focused on socialisation and educational/entertainment interaction. Post-processing of data logging makes it groups). So the way chosen to enhance the totem service after the KSB evaluation is a complete redesign of the user interface in order to facilitate the activities but also to provide different games for older children. Moreover, the lack of appreciation for older children can be explained due to dimensions and ergonomics of the Totem that is focused on younger patients.

Personalised Service and Public Transport Scenario

The table below explains the weight for each KSB property used to evaluate the service provided in the Personalised Service and Public Transport Scenario. The same weights have been used for datasets of October 2012 and February 2013. As can be seen by the ternary plot, the Knowledge perspective is intended to be the same for all of the four KSB evaluations. The ternary plots of the Personalised Service and Public Transport scenarios clearly explain the differences between the first and the second step. While in the first case, there is a huge lack in the business perspective, the innovation added in the second case results in a more balanced service, even while there is a constant lack on social perspective. At the same time, the knowledge perspective is still very important in this service, due to the automatic and non-automatic information providing nutritional aspects. So it can be asserted that user interface improvements from October to February were very effective for every kind of attendance. More than 50.000 people were detected accessing the temporary store and 21.000 interactions with vending machines from June 2012 to March 2013 highlight the attractiveness of the service; moreover, the Transport Scenario can be analysed thanks to more than 500 accesses to the infomobility service provided by the innovative vending machines. properties have equivalent weights (20%) generating the overall B perspective value (from 1 to 5). In the same way all other perspectives value can be obtained (same weights for each KSB properties): the value regarding K perspective is 2,73 while the value of S perspective is 1,1. In this case, the Perspective discovery percentage results are very low in respect to the other scenarios. This is simply because here the service is composed of three elements. Hence, evaluating the Tourism scenario using only data coming from patch questionnaires could be very poor in information; evaluating the service with all the three components (patch, mobile app and web portal) could be very useful to get an overview. As can be seen, the Perspective Knowledge in the first case is very low (it is possible to measure only 30% of perspective K, only 20% of perspective S and only 25% of perspective B). The other KSB elements introduced by the mobile app and web portal increment the Perspective Discovery rate at 90%, 60% and 75% (under the assumption that each KSB property for each of the three elements contribute in the same way, as for the Absolute Perspective Value).

It is useful to evaluate the service as if it was composed of only the wearable monitoring device or as if it was composed of all the three components.

Moreover it is useful to develop a KSB model showing relative values of the K, S and B perspective for the observed service or product instead of an absolute value (from 1 to 5): this value is called the Relative Perspective Index. To obtain the Relative Perspective Index, each perspective absolute value has to be divided by the sum of all the three perspective absolute value and expressed as a percentage (%). The previous two processes can be replicated in order to evaluate the entire service. Also in this case, each property of each perspective has the same relative weight. Because the service has been developed primarily for body performances analysis, the social aspect is non-existent: this is confirmed by the KSB analysis. This lack of a social aspect could be changed through the introduction of social network aspects into the web portal.

Moreover, the KSB analysis shows how there is no balance between the remaining perspectives, which is because the 3-day evaluation period highlights bugs and inefficiencies on the service, which have been discovered thanks to participant reports on questionnaires. This implies that the data acquisition step for KSB models can also directly help to enhance the service.

The two KSB actualisations are quite similar because of this: bugs and errors cause participants to not appreciate the difference between a stand-alone wearable monitoring device and a complete service where the wearable monitoring device can communicate with mobiles and web portals and where the user can easily access comprehensive information regarding their body performances and their activities.

KSB Analysis

The most important innovation in ELLIOT is the introduction of a unique way to evaluate the service provided and the related User eXperience. This can be achieved through the KSB ternary plot and its indicators. They provide a general overview thanks to an absolute Perspective Index (from 1 to 5) for each perspective, but also with a barycentric analysis of the system through the Relative Perspective Index that highlight whether the service provided is balanced or not. The evaluation system developed also makes it possible to evaluate the completeness of the evaluation of the system through the Elliot platform (this issue allows us to highlight that the knowledge of the Perspective depends on how many relevant KSB elements are covered from the User Experience analysis).

Thanks to these improvements it is possible to report some short findings regarding each scenario.

Media Scenario

It has been evaluated that there is a constant leakage in the Business perspective regarding the other perspectives: to avoid this weakness, a new user interface has been developed and installed onto the Totem.

A very important improvement made on account of Elliot is the enhancement of the collaboration between children using games and applications provided by the totem (it can be seen in K6.1 "Team cognitive process" and S2.2 "Collaboration" properties) for every age cluster.

Personalised Service and Public Transport Scenario

The redesign of the user interface has resulted in improvements of the service: this is highlighted in property B3.2 "Ease of use" that has been developed from 1.3 to 3.72. The redesign of the interface and of the management system also provide a generalised improvement of all three perspectives. Moreover, the second interface provides a more balanced service than using the first edition of interface (October 2012).

The evaluation in B1.1 "New functionalities" has been evaluated through purchases of new products such as menus. This is achieved by comparing the number of menus purchased compared with the total number of products purchased. So the evaluation in B1.1 can be explained as: the users have strong preference to select only one product at time and not a complete menu from the vending machine.

Tourism Scenario

The analysis of this service highlights that it was developed without a social perspective. This is because the service and related products are developed to motivate the user to enjoy sports during her or his spare time but do not consider networking or community support to reach the objectives as factors.

The analysis of service has been carried out in two different ways: evaluating only the patch and evaluating the entire service. The results are quite the same just because the questionnaires were provided after the 3-day test where the users had access to the entire service (wear the patch but also use the mobile app and access to the web portal). Adding mobile applications and a web portal to the service, the KSB ternary plot shows a more balanced service, but the business perspective lacks some decimals primarily due to mobile apps and web portal bugs.

The two ternary plots also show the importance of knowledge of the perspective: using only the patch, the knowledge of the system is very low but makes it possible to understand the service primarily regarding the social perspective leak but also giving a first point of view regarding the others perspectives. In the next In this deliverable, we report on two experiments of MyGreenServices which are issued partially from the co-creation workshops [START_REF] Negri | Ideation of IoT services with citizen: coupling GenIoT and AloHa! Methods[END_REF] for the experimentation and evaluation steps in our Living Lab process

Experiment Description

Two experiments are reported on in this deliverable. Both of them are related to the use and the evaluation of MyGreenServices portal (versions v1 and v2).

Participants were initially divided into two groups: producers and consumers. However, taking into consideration the low number of sensors available for the experiments and in order to avoid frustration concerning the user experience, it was decided to focus on producer recruitment.

Consumers recruited are most often the husband or wife of producers.

Producers had an IoT device (fixed: Polluxand AxIS Box in the second experimentationor mobile: Azimut) and their task was to charge the device regularly to ensure proper operation of the device and fill out questionnaires as part of a longitudinal study (detailed information gathered is showed in Section 5.4. while responding to questionnaires.

This deliverable reports on two new experiments:

 Experiment 1: the experiment with MyGreenServices portal v1 was carried out between the 5 th of February and the 21 th of February. In this experiment 13 participants were recruited, with 9 producers and 4 consumers.  Experiment 2: the experiment with MyGreenServices portal v2 was carried out between the 5 th of June and the 19 th of June. In this experiment 8 participants were recruited, with 6 producers and 2 consumers.

In this deliverable, we focus on the experimentations related to the discovery and usage of the IoT citizen sensors as well as the portal gathering and visualising IoT data: the MyGreenServices portal.

The experiment covers both -setup and usage of citizen sensors at participant premises -usage of the citizen measurements via MyGreenServices (map visualisation, data history download as well as SMS/email geo-localised pollution alerts).

The aim of the experiments is to assess the user experience and experiential learning related to MyGreeenServices; this includes experience related to the IoT devices, to the measures and services as well as air quality awareness and behaviour changes monitoring.

On top of the exploratory nature of the experiment, one of the hypotheses tested is that the user experience, the participation in the study and the co-creation would be higher for the participants who have participated in the co-creation workshop and/or hosted an IoT station than for the citizen who just used the portal measure and contributed to the MyGreenServices forum.

MyGreenServices Description

MyGreenServices provides access to citizen measures (stations and electric vehicles) for any registered user. Moreover, citizens who host a station can trace the time history of the data sensed.

The priority was to give back data of each user. Two ways to represent data have been chosen:

-The use of maps with measures coming from environmental sensors and based on a colour scale indication.

-The pollution curves that support the cartography and allow the access to the detailed data for the user. -The second allow the user to define an area to follow and the user will be advised of pollution alerts for the area by email or text message.

KSB Instantiation

In accordance with the overall objective of MyGreenServices, the UX measurement focused on the change of behaviour (in a wide sense), the ease of use and diffusion aspects (as being a tool provided to the citizen). Indeed, KSB elements chosen to describe the user experience cover mainly:

 K elements for the experiential learning aspects (specifically the "cognitive" elements),

 S elements for the usage of the IoT system as a persuasive and dissemination tool (specifically the "reciprocal" elements),  B elements for the new technological functionality and ease of use. 1. Green Services use-case data that could be imported into the Elliot Platform using a comma-separated value file.

2. Connection to the Elliot Platform using the LinkSmart middleware has to be completed but the Green Services side is near to be ready. For computing indicators related to KSB properties for our use-case experiments, both generic and specific Web Services were used (see Appendix).

Experiment Lessons Learned und Updates Relative to D4.3.2

In terms of methodology, as the diary study was very effective and efficient (see deliverables D4.2.2 and D4.3.2) related to the "fake green watch" experiment, the ICT Usage Living Lab researchers decided to maintain this approach for the two MyGreenServices experiments.

However, it was enhanced by the possibilities offered by real-data and real-time sensing for gathering more reliable data.

More lessons were also learned from internal tests aimed at installing the IoT devices in a "live setting".

Such tests allowed the iterative writing of an IoT device setting guide to be provided to the participants for installing IoT devices at their premises. They also enhanced the list of requirements and warnings to be presented to the participants volunteering for hosting IoT devices (battery charge for instance requires participants to really take care of the IoT device, but protection of the sensor was also an issue).

Design Changes

Various design changes have been done:

 User manuals: The user manuals for IoT devices have been improved between the experimentation 1 and the experimentation 2. Such manuals were also included in the user account on the MyGreenServices [MGS] portal during the experimentation 2.  Way of sending questionnaires: As this experimentation runs with "real" IoT devices, it is then possible to monitor the data sensed and to send contextual questionnaires when predefined conditions are met. As a consequence, the questionnaires are not sent via the Sphinx component of FocusLab but directly from the MyGreenServices portal.  Improvement of Questionnaires: Questions have been added in questionnaires in order to add another property to experiment 2 in our use-case (B7.1 Data protection).  Improvement of MyGreenServices (v1  v2): MyGreenServices v1 has been improved

Data Collection Changes

The following changes were carried out:  Manufacturing a new sensor AxISBox: A new sensor AxisBox has been manufactured by INRIA (with a hardware cost less than € 100) in order to increase the number of citizen sensors due to the non-delivery of some sensors. This prototype has been tested during the second experiment (June) and has met the project needs concerning fixed stations.  IoT data quality: The quality of geo-location is not guaranteed, in fact we can have up to 10 meters offset with a real GPS sensor, more with a 3G system (sometimes the signal is on the 3G Antenna) and sometimes no signal. The big problem encountered in our region is that some data was geo-located in the sea. We have sensors from different companies and handle this problem in two different ways: one consists of sending (0;0) for (latitude; longitude) when we have very poor precision (the case of the equipped electric car), another one is to send a precision value with the latitude and longitude (the case of the mobile citizen sensors Azimut). For the second experiment, it was decided to visualise

IoT data only if sensors provide good precision values in addition to their location.

Data Analysis Changes

Various data analysis changes have been carried out:

 New types of UX data: Various types of UX were considered (see Table below). Indeed, as mentioned in D1.5 (Section 6.3), UX measured could either be cumulative, episodic or momentary depending on the moment of the measurement and the object of the measure.

For MyGreenServices, ICT Usage Researchers opted for a composed selection of UX spans measurement. Moreover, some UX elements were generici.e. related to the overall serviceswhile others applied to a specific service such as the forum or the alerting services.

MyGreenServices Alert Forum Maps/ Measures

Momentary UX {Q3} Let us note that, due to the low number of citizen sensors, Q3 has not been addressed.

Episodic UX Q1.1, Q1.2 Q1.2 Q2 Q2 Cumulative UX Q1.2, Q4 Q1.2

Adding IoT Data Synthesis

Taking into account the large amount of IoT data and user feedback in terms of IoT data synthesis needs after the first experiment of MyGreenServices, two functions have been added: IoT data synthesis by day (to be chosen by the user) and IoT data synthesis by hour for the experiment 1 (whatever the day). To achieve these new functions, ICT Usage Lab researchers started by dividing the city of Nice into a grid. Then a new function was provided in the MyGreenServices portal: the user can choose a day and the algorithm calculates the daily average hourly maxima for each area (i.e. box) where data (at least 10 values) are available. The returned data is then displayed on a map with the colours corresponding to the indices (cf. Figure 72).

Adding a statistical tool for Living lab manager

In order to make the analysis of MyGreenServices and in particular the user experience easier, ICT Usage Lab researchers have created MGS_Stat, allowing the quick visualisation of statistics.

This interface is only available to ICT Usage Lab researchers. This interface allows researchers to analyse the navigation and the use of MyGreenServices.

Participants

As described below, participants were planned to be placed in two main groups; moreover, another relevant criteria considered was the previous engagement in the green services cocreation process:  "Producers": this group gathers all participants who volunteered for hosting/wearing an IoT device. Some of them also took part in the co-creation workshops or the "fake watch" experiment, i.e. were previously involved in the green services co-creation process.  "Consumers": this group gathers all participants who both registered to MyGreenServices and volunteered for taking part in the user experience/usage inquiry (contextual and diary questionnaires, on top of the automatic traces). Some of these participants were also previously involved in the green services co-creation process.

As already mentioned in 5.1.1, due to the low number of citizen sensors, and in order to avoid the creation of a bias in our experiments, mainly producers were recruited.

All participants were screened for meeting mixed knowledge and attitude related to ICT and sustainability. They were asked whether they live in Nice and use a smartphone (in order to be eligible for the contextual diary study). They all answered a common profile survey (see the Appendix in Section 8 for results).

The statistical sample does not allow us to draw conclusions. The criteria of representativeness would require 10% of the reference population. In addition, our sample does not reproduce sufficiently the characteristics of the reference populationcity of Nice.

Nevertheless we must qualify these aspects in different ways. Indeed, the experiment can be seen as a pre-test. As in pre-investigation, the need for representativeness is less. Furthermore, when users (producers and consumers) were recruited, care was taken that they were potentially interested in this service. Thus, discriminator criteria were used in the profile questionnaire (such as sensitivity to sustainable development, the proximity to professional fields related to the environment and health, respiratory diseases, sport, etc.).

Data Collection

Sensor Data

The Sensor data gathered (see Figure below) is used for several purposes:

-For providing "green services": for providing the citizen with open measures via

MyGreenServices, but also for alerting citizen in case of pollution.

o This aspect is therefore a prerequisite for running the experimentation. IoT data collected for this purposed is defined below.

o This aspect is also very important a posteriori for qualifying the context in which the user experienced the green services. For this purpose summarised measures are enough (as showed in 5.5.1 IoT data collected were summarised thanks to the FocusLab Web Service "REGLO")

-For triggering the user experience data gathering in specific contexts. Specific questionnaires were planned to be sent to participants in specific contexts (a high pollution or not) detected AND in the case of smartphone configuration enabling geo-localisation.

Let us note that it was not possible to address this feature in our experiments, as predicted, due to the low number of participants able to participate in geo-localisation.

Usage Data of IoT based system

A log system redrawing the journeys of each user and the information restituted has been developed. It enables the analysis of the user interaction with the components of the portal in order to enhance the future versions. We see in the following table four alerts were raised and sent to 3 participants (97, 54, and 5) at different times and logged in our database.

Observation and Survey Data

All observations and survey data were interpreted in light of the objective data gathered via traces/logs regarding the usage of the technical system and in the light of the summarised IoT data regarding the physical context of the participant.

User experience and behaviour changes were assessed via recurrent, mid/end of study and/or contextual questionnaires (+ a retrospective interview based on usage data):

-Recurrent episodic questionnaires (Q1.1 and Q1.2), triggered every 4 days in order to measure change in behaviour and opinion.

-One shot cumulative questionnaires at mid (Q2) and end (Q4) of the experiment in order to gather the holistic UX and prepare the retrospective interview.

Attitude changes were also assessed by measuring the pre/post experimentation (profiling questionnaire and Q4 final questionnaire) delta to some of the questions of the screening survey that were asked before the final interview (when relevant, they were also mentioned in the interview).

Ten participants among thirteen have fulfilled the various questionnaires and are considered for the KSB analysis.

Privacy Protection

The MyGreenServices portal stores and links personal data from participants and sensors data.

First, to ensure the security of personal data, a user authentication to access real-time data is Next, only users that produce data can see the detailed data of others producers. A simple user called consumer can access only a "summary" of the data.

Finally, for improving security, it was decided to use two separate databases:

a) one for sensors data and geo-location (the data store), b) another for other data (users and portal).

The data store ingests a value of any geo-located sensor which is identified by a unique sensor Id.

This sensor Id is connected to a station in the other database. This station is connected to a user.

Insofar as the

MyGreenServices portal provides open data, some elements of geo-localisation are visible. However, these data are only available for the participants of the experiment. Participants have signed a loan agreement for materials containing a clause on the provision in the context of their journey (mobile stations) and location of their home (fixed stations).

Article 5.6 states that: " [Participant] agrees that the data collected by the station are recovered on the MyGreenServices platform for the purpose of the experiment (via mapping Nice measures presented in MyGreenServices.inria.fr) measures remain anonymous."

5.5

Data Pre-processing and Data Analysis

Sensor Data Analysis

A study of the application of one generic FocusLab web service and one k-means clustering method on the IoT data was carried out. The results of the study can be found in Appendix 8.2:

-The REGLO method on the evolution of one Pollux station. We believe that summarising

IoT data is important to build individual contextual data and like this to help the living lab manager to better interpret user behaviour and user experience. An analysis of participant profiles is reported in the Appendix (cf. Appendix 8.4).

A classification of the profiles has been performed using MND, but no relevant indicators related to environment, health and mobility topics have been identified due to the size of the sample.

The MGS_Stats tool

In order to ensure a proper data analysis, log and usage analytics were structured and gathered in an admin tool designed by the AxIS team at Inria. This tool is a component of the MyGreenServices portal. Log data has been used for analysis presented in this report and also in order to prepare the final interview. The screen shots below depict the tool and its functionalities.

Data-KSB Mapping

As per the objectives of MyGreenServices, the cognitive artefact, mental mapping as well as ease of use were the most relevant KSB elements; as a consequence, these KSB elements were mapped with multiple data (several UX indicators, on top of the usage data) in order to provide a complete picture of the user experience. Each element of the KSB was assessed both by one or more dedicated questions and usage data (log, data input, etc.). In the simple cases, where one KSB element matches one question (+ log), the KSB value takes the value of the scale used four points. It is important to highlight that the usage data may inform about the context in which the UX was formed (e.g. intensity of usage of the service); however it is not possible to downgrade for instance the reliability of the KSB elements on the basis of the usage data. Indeed, even if participants did not try out a function, what matters is what they feel as being their user experience and not how deeply they interact with the IoT based setting.

Below, a snapshot on the number of input used to qualify each KSB element is provided. It also gives hints on the relevant log for each KSB element.

KSB rules

All rules allow the measurement of KSB properties per user. They are written and used in order to test the ELLIOT approach to measuring User Experience as per Green services Use-case mission. In this document, these rules are also used to measure the overall service experience.

K2.2 Sensing and attunement of affordance rule

Sensing affordances and attunement of affordances are calculated considering two questions: the rate of users who declare a change of perception to air quality and noise pollution. This question is correlated with questions coming from the contextual questionnaire, questioning about the user's perception of health, well-being.

-If % users declaring a change of perception is < α then Sensing and attunement of affordance is low.

-If % users declaring a change of perception (including mental awareness) is > α and < β then Sensing and attunement of affordance is medium.

-If % users declaring a change of perception (including mental awareness) is > β then Sensing and attunement of affordance is high.

For these aspects, α = 20% and β = 40% were chosen. For these aspects, α = 80% and β = 50% were chosen.

B4.1 Usefulness rule

Usefulness is calculated by aggregating the analysis of two questions related to a change of behaviours during and/or after the experiment in terms of: transportation, aeration, outgoing, sport, aeration or others and with the support of log data (alert service). We use two web services:

one Focuslab web service called MNDClustering_Sequence and another one, GreenServices_B4.1 (cf. sections 5.6.2.8 and 5.5.4), to classify the answers to these questions:

-If % users declaring a change of behaviour > α then high -If % users declaring a change of behaviour < α and > β then medium -If % users declaring a change of behaviour < β then low For these aspects, α = 5% and β = 1% and were chosen. Note that other questions related to the usefulness of some MyGreenServices functionalities (alerts, forum, data synthesis, etc.) might be integrated in a more global rule.

5.5.3.1.9 B4.2 Hedonic quality rule Hedonic quality is calculated merging three questions: the opinion on the MGS portal, the usability of the alert service and the aesthetic aspect of the forum. To conclude, different values to the questions were adopted.

-If % users "very satisfied by the services" > α then Hedonic quality is high.

-If % users "very satisfied by the services" > β and < α then Hedonic quality is medium.

-If % users "very satisfied by the services" < β then Hedonic quality is low.

For these aspects, α = 50% and β = 20% were chosen.

B4.5 Loyalty rule

Loyalty is the result of the analysis of two questions and of the data log: intention to use the MGS portal after the experimentation and frequency of connections (perceived). In the first question, it is clear that the estimation concerns more the intention of loyalty than the loyalty itself. Data log concerning frequency of connection, duration, number of clicks are used to calculate the indicator.

-If % users planning to use the MGS portal after the experimentation is > α then loyalty is high -If % users planning to use the MGS portal after the experimentation is > β and < α then loyalty is medium -If % users planning to use the MGS portal after the experimentation is < β then loyalty is low For these aspects, α = 80% and β = 50% were chosen. -If % users declare themselves confident concerning data protection is > α then Data protection is high.

-If % users declare themselves confident concerning data protection is > β and < α then Data protection is medium.

-If % users declare themselves confident concerning data protection is < β then Data protection is low.

For these aspects, α = 80% and β = 50% were chosen.

Usage of ELLIOT Data Analysis Features

In addition to the IoT Data analysis with FocusLab methods previously explained, the Green services Use-case uses 10 Web Services for the UX data analysis:

 one generic advanced web service from Focuslab server (MNDClustering_Sequence Web Service) to address user behaviour change or user experience evolution,  9 simple web services (including one used for 4 KSB properties).

The data analysis process is composed of two main steps, which are outlined below.

First step, Usage of Focuslab server

A new advanced Focuslab Web Service MNDCluster_Sequence implementing the MND method 2 has been created linked to user behaviours evolution. This Web Service is available via the following URL: http://focuslab.inria.fr/focuslab/MNDCluster_Sequence_process.jsp 2 Verde, R., De Carvalho, F.A.T., Lechevallier, Y. (2000) : A Dynamical Clustering Algorithm for P.J.F. Groenen and M. Schader (Eds.): Data Analysis,Classification,and Related Methods,Heidelberg,[387][388][389][390][391][392][393][394] The result of the K2.2 web service call is an XML file:

<?xml version="1.0" encoding="UTF-8"?> <xml> <perCentOfUserChangePositiv>38.0</perCentOfUserChangePositiv> </xml>

5.6

Results and Inferences

Experiment Outcomes

Unlike in previous experiments, participants in the two experiments of MyGreenServices were interacting with real sensors, providing actual measurements of air quality. Therefore, their involvement in the process cannot be compared with previous experiments.

It is important to note that the results from the two experiments of MyGreenServices are good and globally similar. For a comparison on quantitative values of KSB properties for MyGreenServices v2, see the conclusion of our experiments (5.6.3).

In these two experiments, data producers have experienced the insertion of an IoT device in their daily lives as positive. Fear of permanent geo-localisation for participants holding a mobile station during the experiment as expressed in the green fake watch experiment (reported in the D4.3.2),

was not ultimately confirmed. Instead, participants were very interested in and motivated by the possibility to learn about their pollution data during their daily journey. The proposed service must remain free and accessible to all citizens. "The information and all information must flow freely; we must tell people and stop lying to them (e.g. Chernobyl and others). With the information we can do something. "

The weaknesses were coming from technical problems with the stationsinterruption in the data transmission to the server and station failuresand the small number of sensors in a relatively large area (71.92 square kilometres for the city of Nice). It should therefore come to get a better coverage of the country in order to provide more reliable and representative data.

Second, even if he KSB UX model is similar, some improvements on two KSB properties can be noted. Indeed, some improvements to the MyGreenServices platform when creating a second version (v2) have been carried out between the two experiments based on user feedback from experiment 1. Let us note one variation in the KSB instantiation in terms of value obtained (high, respond more quickly to questions on the forum which helped to improve the user experience of the MyGreenServices community space.

KSB Analysis (MyGreenServices v1).

Here, we report on the interpretation of the data from the KSB-guided user experience analysis related to the participation to theMyGreenServices experiment v1 reported above.

Based on the computed indicators (KPIs) (cf. Section 5.5. Using the FCSR KSB modelling tool, a KSB graph was generated which is shown in Figure 87.

The graph is based on the element values (see the last column in the previous table) which confirms good results for the K, S and B dimensions. To compute this property, we use the user answers to one indirect question from the questionnaires (see below).

 Profiling/Q2 and Q4/Q1: What is your level of knowledge in the field of air quality?

In relation to the KSB instantiation, given that the low coverage of citizen measures considering the area of Nice and the fact that participants do not select the geo-localisation, no questionnaire Q3 was sent during the experiments. Overall, participants emphasise that the study has led to a growing awareness about the issue of pollution. In terms of exposure, they are mostly positively surprised by the level of pollution in their immediate environment. Here, we have to add contextual data. The experimentation has been carried out in February and this period is known as being less polluted. This contextual factor plays on the perception of pollution.

To understand this result, it is also important to consider the profile. Participants recruited are generally sensitive to sustainable development. IoT devices are equipped with a temperature sensor, many participants used these to monitor the temperature inside their home or to identify during which slots their terrace was the sunniest.

They also tried to identify the best time slots to carry out the ventilation of their homes.

Few comparisons have been made to the extent that there are actually other which are relevant or equivalent. To compute this property, we use the user answers to three indirect questions from questionnaires and three from the final interview (see below). Users are divided on the ease of access to information on the forum. The final interviews confirm this result. A participant focuses on the fact that he has not posted because the forum lacked thematic organisation. To compute this property we use the user answers to three indirect questions from questionnaires (see below). Starting from one connection per day of experimentation, we consider that the user is connecting at least "often enough". Data usages coming from logs are used to confirm the user's perception of the use of the MGS portal. Taking into account the different questions we get the following result: There has been no testing of the new service based on green measures available on the portal.

However, some participants exchanged ideas and posted on the forum. Only a few users can be considered as active. In our case it was proposed to declare the user active when she/he posted an idea in the forum. We have 5 users in this category. For this indicator we can use the PEW pyramid (PEW/Internet, Generation Online, 2009).

Five participants posted an idea on the forum, and 3 of these 5 posted an idea for a new service (identification of improvement or new service). A few participants did not use the forum to post their idea of new service but use the final interview to express it. These users were not interested in sharing information in the forum and frequently expressed that they were not usually using forums to communicate.

B7.1 Data protection: High (cf. rule 5.5.3.1.12)

To compute this property, user answers to two indirect questions from questionnaires were used (see below). All users declare having confidence in the protection of personal data MyGreenServices portal. Section 5.6.2.3) there is an increase of the percentage of users (57% instead of 50% in the experiment 1) who posted an idea (pro-active user) in the MGS forum.

Let us remember that the two use-case scenariosthe mobility scenario and the health/well-being scenario has been addressed together with the development of MyGreenServices. Some user More citizen sensors are required in order to cover in a relevant way a large territory such as Nice and to better promote some existing interesting functionalities such as alert management.

An important result is the ability to use an advanced Focuslab method in the ELLIOT platform that provides new data to be added via the middleware. In this way, the Elliot platform allows the Living Lab manager to play with data by applying advanced data analysis methods (such as those provided by the FocusLab server), to choose the newly generated data to be added via the middleware, and finally to process the KSB UX measurements in a multi-step process (from raw data to the computation of indicators used for the KSB property rule).

Another result is also the generation of a pollution database from mobile and fixed IoT sensors with around 4 million pollution measures through the ICT Usage lab, and also other valuable user contributions (usage scenarios and ideas during the workshops, posts on the forum): As noted at the outset of this report, the three deliverables in the 4.3 series are complementary and all three deliverables taken together provide a complete report on the WP4 Living Lab activities throughout the project.

In comparison to D4.3.2, the Living Lab activities reported here involve a larger number of IoT sensors and devices, and their output is used more extensively in the computation of KSB model Below, the IoT Sensor Data is reported on collected by the Green Services data server. A REST API has been implemented to save data in the green services data server.

In addition to the IoT data collection, other developments related to the user needs in terms of visualisation of IoT data (maps), alerts, user preferences and so on are in progress. Different user interfaces for the green services use-case are in development (for computer and for smart phone, an additional administration interface for green services is also implemented). A specific interface for the administrator of green services is also implemented. The two planned versions of green services (v1 and v2) will be reported in the upcoming deliverable D4.3.3 with the results of our two next experiments (v1, v2) based on green services. Guynemer (Antibes) NO2

IoT connectivity

Air PACA will post one file with air quality measures by station to the Air PACA server every hour. The green services application reads these files hourly and stores data updates in the "raw data DB" on a secured Inria server.

IoT connectivity

The performance of the green watch (user alerting, type(s) of measure(s) displayed/used for events, collection/transmission of data etc.) is configured via a programmable automat that requires the specification of usage scenarios. The green watch user interface will be composed of a screen (96x96 px, monochrome), four buttons and an amplificatory (noisy BIP). The green watch has 4 LEDs visible on the side of the watch; the LEDs can be used for instance to visually alert the user in case of pollution. User interactions with green watches as well as measured IoT data will be logged.

The various modes between the green watch and the supervision server called M2M are managed by the applicative software of the green watch via the WLM Wireless Module. The communication uses a wireless communication network (GPRS). An API is implemented to manage WLM. o "I would like to check whether the network coverage is fine".

IoT Connectivity

Each car is equipped with an embedded computer called "vubox". Based on ARM processor technology and with machine-to-machine communication (GPRS) and positioning (GPS) capabilities. In this way, car data are accessible 24/7.

Custom software developed for the Elliot project communicates with the smart sensor through a serial link RS232. This sensor realises the measure: analogue acquisition, digitalise, filtering, buffering. The vubox controls the rate of acquisition and of transfer to the server using GPRS technology.

The ELLIOT Web Service for electric cars is a SOAP implementation of Web Services. The WSDL descriptor is available at the same address of the general server; access is restricted with a security code (KEY). -The service GetTracerSerie(KEY, tracerId, day) gives the sensors data of a given car and a

given date & day.

For the first vehicle, a prototype system is installed on the roof of the car. Each vehicle provides data under the same structure: Date (UTC), location (WGS84), sensor data (0...N). The sensor data field contains the type and the measure itself.

Each sensor has a meta-description including its name, the unit of the measure and an identifier.

A general structure has been developed to enlarge the scope of the software to any kind of environmental measurement.

The first sensor, which was installed at the end of August, has a 5 minutes transmission rate. This has to be tuned to take into account the cost of the transmission. First data available are:

-Temperature (+/-0,4°) -Noise level (+/-0,1 dBA)

-Humidity (+/-3%) -Pollution, measure by a mix of O3 and N02 gas (ppm) The current VU Log data server uses the following format:

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlnssi="http://www.w3.org/2001/XMLSchema-instance" xmlnssd="http://www.w3.org/2001/XMLSchema"> <soap:Body> <GetTracerSerieResponse xmlns="http://VU Log.com/"> <GetTracerSerieResult tracerid="VM000"> <metadata> <sensor-metadata id="gaz" name="Gaz sensor"> <variables> <variable id="O3NO2" name="Concentration O2+N03" unit="PPM"/> <variable id="TEMP" name="Temperature" unit="°C"/> <variable id="HYGRO" name="Hygrometrie" unit="%"/> </variables> </sensor-metadata> <sensor-metadata id="noise" name="Noise sensor"> <variables> <variable id="DBA" name="DBA measure" unit="DB"/> </variables> </sensor-metadata> </metadata> <measure> <date>2012-09-06T00:00:00</date> <location lat="43.67505666666667" lon="7.2285733333333333"/> <sensor type="gaz" values="70.00 24.20 56.00"/> <sensor type="noise" values="54.40"/> </measure> <measure> <date>2012-09-06T00:01:00</date> <location lat="43.67505666666667" lon="7.2285733333333333"/> <sensor type="gaz" values="90.00 24.20 55.81"/> <sensor type="noise" values="53.35"/> </measure> ...... <measure> <date>2012-09-06T06:20:00</date> <location lat="43.675361666666667" lon="7.228995"/> <sensor type="gaz" values="82.00 21.62 58.29"/> <sensor type="noise" values="61.12"/> </measure> <measure> <date>2012-09-06T06:21:00</date> <location lat="43.675361666666667" lon="7.228995"/> <sensor type="gaz" values="72.00 21.63 57.74"/> <sensor type="noise" values="59.81"/> </measure> </GetTracerSerieResult> </GetTracerSerieResponse> </soap:Body> </soap:Envelope>

Air/Noise POLLUX stations: FING (resp) + Inria

Finally we got 8 Pollux air/noise stations instead of 10 initially planned: 1 (FING) and 7 (Inria, out of ELLIOT budget)

Sensors

Due to higher acquisition costs (sensor change) from the manufacturer and due to a low precision of the CO sensor (0-300ppm ±0.25 ppm) and NO2 sensor (0-50ppm ±0.05 ppm) on the POLLUX station, Inria decided to order only PM10 sensor plus temperature and noise and has ordered 7 stations instead of 10 initially. This is the same for FING: 3 stations instead of 7. The data store extensions are used for the data processing, mainly for sending the results to the data aggregation platform (in our case it is the green services or local services (such as CSV data collection). These extensions are simple scripts written in Python (with some constraints regarding usable language features), one method for calling Pollux'NZ City application and a dictionary containing the default configurations.

Green services (v1):

A python script (embedded in the Pollux station) pushes data to the green services (v1) raw data server using HTTP. Tests of this implementation are planned for the second week of September 2012. -The Arduino board connected to Raspberry PI on USB and get PM10 Data from the dust sensor.

-The dust sensor.

-The WiFi dongle to connect to the Internet. This solution costs less than 100€ in material costs excluding labour for assembling the system.

IoT Connectivity

INRIA developed two programs:

-One program that, every minute, records the micro-particle value into a file from the micro-particle PM10 sensor connected to Arduino board.

-Another one that, every 5 minutes, recovers all the PM10 values recorded since the last data send then sends to MyGreenServices server. Clusters 1 and 2 seem to be pollution clusters and cluster 5 seems to be a cluster with low O3-NO2 but high temperature.

Applying REGLO, a FocusLab generic web service on one Pollux IoT data

This section shows the results of the application of the REGLO (with the method of middle) web service on IoT data issued of the Pollux station of one participant, from February 4th to March 5th. Our motivation is to summarize these data in order to have a pollution context for this user which may be useful for uUX interpretation. In the figure above we can see that these summaries are not far from the initial curve. The goal now is to carry out an analysis of these summaries to automatically determine the characteristics of the curve.

We select only segments. For each segment we calculate four variables that characterize it:

1. The slope of the segment 2. The midpoint of the segment (average of this segment)

3. The length of the segment 4. The duration of the segment (the time interval between the start time and the end time of the segment)

From these four values we can achieve an interpretation of the previous curve, taking into account only two variables and constructing a 2D representation.. The following curve allows to visualize three periods. To what extent do you consider that you work for Sustainable Development? What is your use of ICT? In which occupational group do you fall? Inria extracted classical statistics fromthe data issued from profile questionnaires using the Sphinx tool and also applied the MND clustering method in order to identify profiles clusters. Clustering results were ultimately not relevant for UX analysis due to the size of the sample.

Generic questions

What is your main mode of transportation? You can check up to three. What is your level of knowledge in the field of air quality? To what extent do you consider that you work for Sustainable Development?

Your relation to information and communication technology (ICT)

Please indicate if you have the following equipment What is your use of ICT? Type of participant?

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (This deliverable reports on the final iterations and achievements of the ELLIOT use-cases and scenarios in the logistics, city of the future and green services application domains. As the final deliverable in the D4.3 deliverable series, D4.3.3 follows up on work done as reported in deliverables D4.3.1 and D4.3.2. This deliverable focuses on providing information on the iterations and achievements by the use-case Partners after the completion of D4.3.2; information from D4.3.1 and D4.3.2 is not repeated in this deliverable unless deemed necessary for reporting on the final iterations and achievements. For each of the represented usage scenarios, the responsible use-case Partners have reported on  the overall experiments carried out  lessons learned and updates relative to previous work as reported in D4.3.2,  participants in the experiment  data collection  data pre-processing and data analysis  results and inferences.

D4. 3 . 3 -Figure 1 :

 331 Figure 1: The toolkit GUI for configuration (used also during experiments)

Figure 2 :

 2 Figure 2: Logistics use-case connection to ELLIOT platform via the LinkSmart ELLIOT middleware

Figure 3 :

 3 Figure 3: Opening Screen Logistics web app

Figure 4 :Figure 5 :

 45 Figure 4: Session List Logistics web app

o

  Example1: # 𝑜𝑓 𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 𝑐𝑟𝑜𝑠𝑠𝑖𝑛𝑔𝑠 (𝑡𝑎𝑟𝑔𝑒𝑡1): 15, 𝑟𝑒𝑎𝑐ℎ𝑒𝑑:

Figure 6 :

 6 Figure 6: Level of participation in different Living Lab co-creation workshops

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (knowledge. Questions aim to understand what the participants expect to learn or do in the workshop (e.g. try out own ideas; gain new skills, take an active role), what their level of knowledge about IoT is (e.g. experience about applications and services) and their attitude regarding IoT (e.g. can IoT make life easier, can IoT generate logistic services). These insights are especially relevant for a better understanding of the individual Living Lab phases and

Figure 7 :

 7 Figure 7: Average personal experience in the workshop Groups I to III

Figure 8 :

 8 Figure 8: Comparison of average activity level and average results of section C

Figure 9

 9 Figure 9 Average results of activity level and understandability of the toolkit's functions

Figure 10 :

 10 Figure 10: Experiment I -Product temperature service

Figure 11 :

 11 Figure 11: Experiment II -Product Temperature Service

Figure 12 :

 12 Figure 12: Experiment II -Forklift Angle Service

  3.1, the definition and instantiation of KSB properties and related KPIs was extensively elaborated on. Throughout the following project months, described in D.4.3.2 and during discussions with WP4 Partners, the instantiation of the KSB model for the logistics usecase was adjusted. Therefore, the number of initially investigated KSB properties was reduced significantly and the number of indicators (and later KPIs) was reduced accordingly. This was a necessary step in order to make applying the model to the use-case easier. After these simplifications, the modelling of the selected KSB properties is considered positive and "applicable" to the logistics use-case by the Living Lab. The application of the presented KSB property selection into the full model view (KSB triangles, Section 3.5.4) led to the expected outcome in terms of a measurement regarding the Knowledge, Social and Business aspects of the developed IoT service. However, such measurements are still an early step in understanding complex relationships underlying successful solution development by user co-creation. D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (

Figure 13 :

 13 Figure 13: City of the Future use-case scenarios.

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (proposed to address these. Thanks to the implementation of a serious game for the co-creation phase (Gaggiots) carried out in WP3, it has been possible to apply serious gaming in a new co-creation phase of this scenario, with interesting output coming directly from new users involved in the game.

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (application by more than one child at the same time allows a distributed cognitive process. application by more than one children at the same time.S4.1Relationship enhancement Access to any application by more than two children at the same time.S7.2 AppealingnessUse of whatever application of Totem that implies high levels of curiosity and desire.

  Set of K, S and B perspectives and related properties for Tourism Scenario 4.1.3 Connection to ELLIOT Platform In order to collect data coming from different IoT systems involved in the City of the Future scenarios, a server has been installed within the eServices for Life and Health research unit able to collect data, direct it to a repository and allow the ELLIOT/Hydra Middleware client to send it via push or pull to the ELLIOT Core Platform. FCSR developed different analytics and preanalytics tools in order to support the use-case activities.

Figure 14 :

 14 Figure 14: Overall view of the way in which our scenarios communicate with the ELLIOT Platform

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (

Figure 16 :

 16 Figure 16: pictures of real Totem usage inside the OSR paediatric department.

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (select or compose various kind of data such as the bounce rate of the service.

Figure

  Figure 17: Blinken game.Figure18: GCompris game set

Figure 20 :

 20 Figure 20: Meal Ordering application

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (breath rate (in bpm); -ECG, electrocardiogram tracing (in mV); -metabolic cost of physical activity, expressed in METs (Metabolic Equivalent of Task);All participants were able to see their physical parameters in real-time, but they must ensure that sensor and smartphone were always connected via wireless connection to allow proper data transmission and storage.

Figure 21 :

 21 Figure 21: Data flow explanation starting from Body Sensor Network and ending to the web-portal and web-storage devices.

D4. 3 . 3 -

 33 Figure 22: web portal homepage.

D4. 3 . 3 -

 33 Figure 23: SPMHBGV3 wearable monitoring device and related B-Patch B-V3 patches.

Figure 24 :

 24 Figure 24: the smartphone Samsung Galaxy ACE (GT-S5830).

Figure 25 :Figure 26 :

 2526 Figure 25: example view of graph provided by mobile app

Figure 27 :

 27 Figure 27: ternary plot indexes distribution applied to KSB model.

Figure 29 :

 29 Figure 29: screenshot of the ternary plot.

Figure 30 :

 30 Figure 30: example of human readable interpretation of the KSB instantiation of the service.

Figure 31 :

 31 Figure 31: An image illustrating the ecosystem of City of the Future Living Lab

Figure 32 :

 32 Figure 32: ELLIOT Scenarios Living Lab's numbers

Figure 33 :

 33 Figure 33: screen shot of data logs coming from Totem OS.

Figure 34 :

 34 Figure 34: screenshot of web portal providing the serious game of City of the Future Living Lab.

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (2.1.1 Media Scenario The co-creation process focused on Media Scenario is provided through a serious game based on the same totem and exposed in the OSR Paediatric Department. The game was presented with the following sentences: " How could ICT and IoT enabled totem change children's experience during their stay in hospital?

Figure 35 :

 35 Figure 35: video contained into the serous game "Innovative vending machines".

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (How could this new sense change the Smart Patch, adding value to what it offers to its users. Q2: In what way could you imagine improving the flexibility of the Smart Patch? Think of practical examples and try to describe them in as much detail as possible.

1.

  The patch was comfortable to wear (B3.1p ergonomic quality) 2. It was easy to position the patch in the correct position (B3.1p ergonomic quality) 3. It was easy to charge the patch (B3.1p ergonomic quality) 4. It was easy to learn how to use the patch (B3.3p learnability) D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (patch was easy to use (B3.2p ease of use) 6. The patch was innovative and original (B4.3p hedonic quality) 7. The patch helped me create new social relations (S1.1p social networking) 8. When I saw the patch I was interested in trying it (K2.4p appealingness) 9. The more time passes, the more I liked to wear the patch (B4.2p attractiveness)This part of the questionnaire ended with two open-ended questions: 10. How did you feel whilst wearing the patch? (K5.2p self-consciousness) 11. Did wearing the patch change you in any way? (K5.1p self-examination)

Figure 36 :Figure 37 :

 3637 Figure 36: temporary store entrance point of view with highlighted the privacy protection poster.

D4. 3 . 3 -Figure 38 :

 3338 Figure 38: Report on user experience in terms of sessions

D4. 3 . 3 -

 33 Figure 40: FSA traffic lights table

D4. 3 . 3 -Figure 41 :

 3341 Figure 41: visit per day graph regarding the Totem serious game for a total of 134 visits.

FigureFigure 43 :

 43 Figure 42: number of users subdivided per ages.

Figure 46 :Figure 47 :

 4647 Figure 46: visits per day graph regarding the innovative vending machine serious game (in logarithmic scale) for a total of 226.

Figure 48 :

 48 Figure 48: percentage of users by sex.

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (

Figure 49 :

 49 Figure 49: visit per day graph regarding the smart patch serious game for a total of 132 visits.

Figure 50 :

 50 Figure 50: number of users subdivided per ages.

Q4:Figure 53 :

 53 Figure 53: number of choices per item for Q4.

D4. 3 . 3 -Figure 55 :Figure 56 :Figure 57 :

 33555657 Figure 54: KSB evaluation, the yellow star indicate UX actualization of service as valued only with "prescolare" group (children younger than 6 years old).

Figure 58 :Figure 59 :

 5859 Figure 58: KSB evaluation of the service provided in PS and PT Scenario through the vending machines, during October 2012.

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (Regarding the KSB model of the wearable monitoring device, the value of the Business perspective depends on 4 properties (B3.1, B3.2, B3.3, B4.2, B4.3) with values (from 1 to 5) coming from questionnaire answers as explained before. It has been supposed that all four KSB

Figure 61 :

 61 Figure 61: KSB evaluation, the yellow star indicate the KSB actualization of the complete service as if it is during the 3day evaluation experience.

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (Green Services Evolution process shows below, the Green Services Use-case has achieved phase 2 where IoT-based services are implemented, tested and then used.

Figure 62 :

 62 Figure 62: Green services evolution process

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (service has been created considering two points of view: -The first consists of localising a person and indicating via email or text message the passage through a polluted area.

Figure 63 :Figure 64 :

 6364 Figure 63: MyGreenServices "Citizen Measures" page.

Figure 66 :

 66 Figure 66: Types of Green badges used in the forum

D4. 3 . 3 -

 33 Figure 68: Green Sensors used

5. 1 . 4

 14 Figure below, two approaches for the connection to the Elliot platform have been prepared:

Figure 69 :

 69 Figure 69: Green Service Use-case ELLIOT integration schema.

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (after the experiment 1 related to the ergonomic aspects and the addition of new functionalities (more IoT data access and visualisation, IoT data synthesis). The addition of a new sensor AxISBox (see below) on the interface was also taken into account.  Re-Calibration of citizen sensors: Pollux sensors have been re-calibrated after the experiment because the margin between two sensors was too large.

Figure 70 :

 70 Figure 70: AxISBox prototype.

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (

Figure 71 :

 71 Figure 71: Creation of the grid for the city of Nice.

Figure 72 :

 72 Figure 72: Pollution IoT Daily Synthesis

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (

Figure 73 :

 73 Figure 73: MGS_Stat tool.

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (

  Every user interaction with the interface of MyGreenServices and the display of the services are stored into a MySQL database containing more than tables: for the user request 3 tables User, Session, Selection and three tables for the output of MyGreenServices, TableResult TableResultLine TableResultLineCouple,

Figure 74 :

 74 Figure 74: Relation between IoT, session and selection

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (registration to MyGreenServices requires completing procedures rules such as captcha validation and email validation. An SSL certificate has been added to ensure nonvisibility when transferring data between client and MyGreenServices server.

-

  A k-means clustering method has been used to classify different quarters of the city based on their IoT data (Azimut data O3-NO2) for each hour/day in order to provide a new citizen function in a future version (v3) of MyGreenServices. D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (

Figure 75 :

 75 Figure 75: MGS_Stats -Connexions and sessions statistics

Figure

  Figure 82: Idea Analytics

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (Cognitive artefact is evaluated by merging four questions and the data log: the capacity of the user to remember the last value seen on the MyGreenServices portal, their perception of behaviour change, the comparison made between data provided by the MGS portal and other sources and the utilisation of the downloading functionality. -If % users able to self-assess their environment is < α then Cognitive artefact is low -If % users able to self-assess their environment is >= α and < β then Cognitive artefact is medium -If % users able to self-assess their environment is > β then Cognitive artefact is high For these aspects, α = 20% and β = 40% was chosen. See Figure 83 below for an illustration of the definition of this rule via the ELLIOT platform.

Figure 83 :

 83 Figure 83: Definition of the K3.2 rule

D4. 3 . 3 --

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (If % users inactive > α then Social networking and openness is low -If % users proactive >= β the Social networking and openness is high -Medium for the rest. α = 90% and β = 2% were chosen based on the common rule 1 .5.5.3.1.4 S2.1 Communication ruleCommunication is evaluated by one question: the rate of users who have reported on the experiment involving the MyGreenServices portal, the IoT device or talking about them.-If % users reporting about the experimentation is < α then Communication is low -If % users reporting about the experimentation is > α and < β then Communication is medium -If % users reporting about the experimentation is > β then Communication is high For these aspects, α = 10% and β = 30% were chosen.5.5.3.1.5 S5.1 Attractiveness ruleAttractiveness results for the correlation between two questions (Q7, Q9) supported by data logs.Q7 and Q9 concern respectively recommendation intentions on the MyGreenServices portal and effective recommendations.

  by one question: the judgment concerning data sensors provided by the MGS portal in terms of: reliability, understandability, usability, suitability, consistence and coming from trusted sources, actors and validated by log data (frequency of connections).-If % users considering data reliable > α then Reliability is high -If % users considering data reliable > β and < α then Reliability is medium -If % users considering data reliable < β then Reliability is low For these aspects, α = 70% and β = 50% were chosen.5.5.3.1.7 B3.1 Ergonomic quality ruleErgonomic quality is evaluated merging three questions and data log: the findability on the forum, the intuitive aspect of the alert service. Different values are affected to the questions in order to have a global indicator.-If % users finding MGS portal and his components intuitive > α then Ergonomic quality is high -If % users finding MGS portal and his components intuitive > β and < α then Ergonomic quality is medium -If % users finding MGS portal and his components intuitive < β then Ergonomic quality is low

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (

  User ideas are evaluated by the number of new services identified by the users during the experiment. Participants were invited to post these ideas on the forum. Data log can be used to D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (of the new service (number of comments, number of votes). -If number of new services > α then User Ideas is high -If number of new services > β and < α then User Ideas is medium -If number of new services < β then User Ideas is lowFor these aspects, α = 8 and β = 5 were chosen.5.5.3.1.12 B7.1 Data protection ruleData protection is evaluated by two questions: the user perception about the feeling concerning the protection of identities and of private data on the MGS portal.

D4. 3 . 3 -Figure 85 :

 3385 Figure 85: Frequentation of MyGreenServices portal number of connexions per day and number of sessions

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (), and better results for two properties.Indeed the variation in terms of value occurs in B3.1 Ergonomic Quality (High for experiment 2 instead of Medium in experiment 1). The percentage of users finding MGS portal and his components intuitive rose between MGS v1 and MGS v2 (respectively 57.3% in experiment 1 and 83% in experiment 2). The questions used in order to calculate this property concerned the intuitive aspect of MyGreenServices portal (overall and functionalities such as alert service and forum). Those aspects are very relevant in the understanding of User Experience.Concerning B2.1 Reliability, users felt more confident in the data provided by MyGreenServices portal (100% instead of 75% in experiment 1). After experiment 1, some improvements regarding data consultation and visualisation were made:-Extended data access: possibility to view all stations -Improvement of the user interface: unification of menus -Assessment by participant/station -Data sensor summaries by hourly periods Regarding S1.1 Social Networking and Openness, there is an increase of the percentage of users (57% instead of 50% in the experiment 1) who have posted an idea (pro-active user) in the MGS forum. ICT Usage Lab Researchers, based on the results of the first experiment, decided to be more reactive on the forum related to the participants. ICT Usage Lab researchers sought to

Figure 87 :

 87 Figure 87: Triangle KSB for MyGreenServices

Figure 88 :

 88 Figure 88: The attractiveness for sharing information on MGS forum

  portal has not always been shown by participants to others, they have still shown their object or IoT, spoke about the experiment and the existence of the platform. During an interview, a participant specified that he did not show the portal to anyone since data were only available to users who have an account with login and password.The communication aspect cannot be reduced to a single demonstration of MyGreenServices portal. Always when this question was asked during the interview, the participant stated that the fact of not having shown might be a bit simplistic into considering the communication aspect of the experiment. Indeed, they have communicated through other means, showing either their IoT object (fix or mobile), or simply by talking informally about the MGS portal and/or the experimentation in which they participated.

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (coupling of quantitative and qualitative data can provide a high value to the KPI Communication.To compute this property, we use the user answers to two indirect questions (Q7, Q9) from questionnaire Q4 (see below).

  The overall attractiveness of the service is high. The participants expressed their intention to recommend the portal. Producers of the experiment 1 helped recruit other participants for future experiments.

Figure 89 :

 89 Figure 89: Recommendations of MGS by users

  these consultations did you find the information you were looking for?

D4. 3 . 3 -Figure 90 :

 3390 Figure 90: Findability in MGS forum

  the beginning of the experiment, to what extent did you change your habits in the following areas? (0 for no change, and 4 for a significant change).  I9: Comparing with your first use of MGS, how have you changed your habits transport, physical activity (health, fitness, etc.), aeration home / office? MND clustering of all answers into three classes is carried out elaborating the sequence of answers by the users in terms of classes. Considering classes c_1/3, c_2/3 and c_3/3, then the number of users with class changes is counted.

  the last four days, what were your reactions to alerts received?  Q2/Q7: How do you find MyGreenServices the community space / Forum?

:

  I8Have you posted an idea or experienced a new service based on green measures available on the MGS portal? If yes, which?

  behaviour changes are related to mobility aspects, others to Health & Wellbeing. The second version of MyGreenServices modified based on user feedback has been perceived as a real improvement compared to the first one (more data access, more curves, more IoT synthesis, a better community management).

  outcomes. The reports by the Living Lab Partners show that the integration of IoT sensors and actuators into Living Lab environments remains challenging, due to: a) Specific requirements in different phases of a Living Lab lifecycle that may make it difficult to use IoT devices, in particular during co-creation b) Technical difficulties starting from the development and sourcing of suitable IoT devices to deal with technical faults and failures c) Expectations of Living Lab participants concerning the maturity of development of hardware and software components as deployed in Living Labs that may vary significantly depending on the target user group (cf. the willingness of participants to interact with a mock-up green watch in the Green Services use case vs. the reluctance of potential industrial partners in the Logistics use case to invest efforts when a proposed environment is still in an early stage of a development cycle). With respect to KSB indicators, two of the Living Lab partners have further focused on a small set of KSB indicators used in their Living Lab activities, while the Green Services use case has retained a comparatively wide range of indicators that are determined and evaluated. This distinction reflects the complexity of measuring user experience in the Green Services use case, which incorporates a number of distinct relevant elements and a range of measurement instruments from a large number of heterogeneous data sources, whereas the two other use cases act in more focused contexts within the project. The three Living Lab Partners have used an enhanced visualisation model for the instantiation of the KSB model results as presented in this deliverable; this model incorporates uncertainty concerning evaluation outcomes explicitly as part of the output visualisation and could be D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (future update of the overall Elliot KSB model instance visualisation approach. The visualisation was proposed and developed by FCSR; the underlying principle of it is described in Section 4.2.1.2. This third deliverable in the D4.3 series has presented an updated account and analysis of the evolutionary development of the Living Lab use-cases; the progress through Living Lab cycles, and the evolution in the usage of the KSB Model. The deliverables has presented useful insights into different ways of deploying the KSB Model and should provide future users with first-hand accounts of helpful deliberations and applications by the three use-case Partners concerning the KSB Model. D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (This appendix from D4.3.2 has been updated by Inria and Fingin some sections in order to represent the latest state of the use-case. green services are planned related to our Living Lab evolution process. Our first version of green services (v1, being implemented at the time of writing) will be improved at the end of November 2012 using the user feedback. As already mentioned, phase 2 has been begun with the development of the version 1 of green services based on the results of previous steps (cocreation and exploration steps).

Figure 92

 92 Figure 92 presents a user-centered overview of the green services use-case with the three different roles represented in our use-case.

Figure 92 :

 92 Figure 92: Green services applications

8. 1 . 2

 12 Green watches: Fing (resp) + InriaFing is involved in a Consortium with other Partners aiming to design the second version of the green watch and a new Citypulse platform. Six green watches provided by Fing were expected to be available for the ELLIOT project (Inria ordered 10 additional green watches out of the ELLIOT budget). After having specifying the interaction with the four buttons (Fing + Inria) around summer 2012, many delays have been noted due to technical problems. In December 2012, Inria received one green watch prototype which was not fully functional: this confirmed Fing and Inria that green watches could not be used for the first experiment of MyGreenServices planned in February 2013 (FING,/Podium). Eventually, at the end of May 2013, FING had received only one new prototype of the second version of the green watch, with several problems unsolved like GPS tracking, calibration of the gas sensors, and many software bugs to fix. Despite the fact that a lot of work had been carried out, (see below) regarding data exchange with the green services platform or settings of the experiments, it was decided to cancel the tests previously scheduled with the green watches, and proposed to the people who were involved in the second experiment of MyGreenServices to use the others sensors set up by the team, such as Pollux stations and AxISbox (Arduino-based sensors).

Figure 93 :

 93 Figure 93: green watch (2 nd version)

D4. 3 . 3 -Figure 94 :Figure 95 :

 339495 Figure 94 : Data transmission between green watch and data server

D4. 3 . 3 -

 33 Figure 96: EV-based sensors (equipped MIA)

D4. 3 . 3 --

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (The service GetTracerIds(string KEY) gives the real time list of vehicles equipped with sensor.

  is not yet available. Data collected by VU LOG server are structured in a hierarchical manner: starting in the root folder, data are organised by year, month and day. Each day folder may contain as many "vml" data files as there are active tracers. Green services (v1): the data collection from VU LOG data server by GS v1 has been implemented and tested on the 27 th of February. Due to the change of sensors by VU Log and a new specification of the VU LOG data server, Inria has updated this implementation and has retested the data collection in real-time with the first equipped vehicle at the beginning of September.

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (

D4. 3 . 3 -Figure 97 :

 3397 Figure 97: Pollux green station (gateway module and sensors)

  number of sensors and get more data during the second experiment of MyGreenServices, AxIS team decided to build a microparticules (PM10) low cost sensor. This solution uses very cheap elements and was easy to re-use. We used the same sensor than the one used in Pollux station.

Figure 98 :

 98 Figure 98: AxISbox

  Figure 99: AxISbox data flows

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (Applying a K-means clustering method for quarters clustering according to PollutionIn this study, the Nice Côte d'Azur territory was discretised into small areas. The temporal and spatial units were clustered into 5 and then into 6 clusters. The partition into 5 clusters was selected, then the temporal units for each area were counted. For the partition in 5 clusters, for each area the percent of each cluster was counted. Around 30 areas with more than 10 temporal units were found.

Figure

  Figure 100: Cluster Description

Figure 101 :

 101 Figure 101: REGLO application of Holy's micro-particles station

Figure 102 :

 102 Figure 102: Beginning of the curve plotted with REGLO (by the method of middle).

D4. 3 . 3 -

 33 Report on IOT Living Labs Continuous Exploration and Evaluation (

Figure 103 :

 103 Figure 103: Curve with the average values and the duration of each segment.

D4. 3 . 3 -

 33 Figure 104: Curve with the slope and the duration of each segment.

Figure 107 :

 107 Figure 107: Implication in Sustainable Development

Figure 108 :

 108 Figure 108: Involvement in an association or sport club

Figure 109 :

 109 Figure 109: Practice of a sport outside

Figure

  Figure 110: Frequency of practice

Figure 111 :

 111 Figure 111: Job related to environment or sustainable development

Figure 3

 3 Figure 112: Attitudes regarding sustainable development

Figure 113 :Figure 114 :

 113114 Figure 113: User ICT equipment

Figure 117 :

 117 Figure 117: Use of ICT

Figure

  Figure 118: Activities on internet

Figure 119 :

 119 Figure 119: Cardio respiratory diseases in participant relations

Figure

  Figure 120: Employment in healthcare

Figure 121 :

 121 Figure 121: Participants cardio respiratory diseases

Figure 122 :

 122 Figure 122: Importance of real time information on air quality

Figure 123 :

 123 Figure 123: Knowledge of services providing information on air quality or noise

FigureFigure

  Figure 124: Knowledge of Air PACA

Figure 128 :

 128 Figure 128: Participants with children

Figure

  Figure 129: City of residence

Figure 130 :

 130 Figure 130: City of work

Figure

  Figure 131: Occupational group

Figure 132 :

 132 Figure 132: Type of participant

Figure

  Figure 133: IoT devices

Figure 134 :

 134 Figure 134: Main mode of transportation

Figure 135 :

 135 Figure 135: Air quality level of knowledge

Figure 136 :

 136 Figure 136: Implication in Sustainable Development

Figure 137 :

 137 Figure 137: Involvement in an association or sport club

Figure 138 :

 138 Figure 138: Practice of a sport outside

Figure

  Figure 139: Frequency of practice

Figure 140 :

 140 Figure 140: Job related to environment or sustainable development

Figure 141 :

 141 Figure 141: Attitudes regarding sustainable development

Figure 142 :Figure 143 :

 142143 Figure 142: User ICT equipment

Figure

  Figure 145: Use of contactless payment

Figure 146 :

 146 Figure 146: Use of ICT

FigureFigure 148 :

 148 Figure 147: Activities on internet

Figure

  Figure 149: Job related to health

Figure 150 :

 150 Figure 150: Participants cardio respiratory diseases

Figure 151 :Figure 152 :

 151152 Figure 151: Importance of real time information on air quality

FigureFigure

  Figure 153: Knowledge of Air PACA

FigureFigure 159 ;

 159 Figure 157: Participants with children

Figure

  Figure 160: Occupational group

Figure 161 :

 161 Figure 161: Type of participant

Figure 162 :

 162 Figure 162: IoT devices

235 3 Logistics 3.1 Experiment Overview 3.1.1 Experiment Description

3.3: AD(1). D1.1 KSB Experience Model Overall Framework AD(2). D1.5.1 KSB Experience Model Evaluation and Refinement Report AD(3). D2.1.2 User Requirements and Architectural Design (final) AD(4). D2.4 Final Version of the ELLIOT Platform AD(5). D3.4 Report on Verification and Validation AD(6). D4.1 Specification of the IoT Use-cases AD(7). D4.2.2 Report on IoT Living Labs Methodology and Tools (final) AD(8). D4.3.1 Report on IoT Living Labs Continuous Exploration and Evaluation (initial) AD(9). D4.3.2 Report on IoT Living Labs Continuous Exploration and Evaluation (interim) AD(10). D6.1 Project Handbook and Quality Plan D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final) Date 19.09.2013 ELLIOT Consortium Dissemination: Public 8/

  3.2. Dueto the ongoing refinement process and the adjustments of the KSB model itself, the selected KSB

Properties in D4.3.2 have changed slightly. The adjustments to the table for D4.3.3 were made together with the project Consortium after careful consideration (significantly during the WP4/7 D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final) Date 19.09.2013 ELLIOT Consortium Dissemination: Public 10/235 use-case Partner workshop in Sofia, February 2013).

Table 1 : Final Table of KSB Properties

 1

Table 2 : Levels of Detail of the Logistics web app

 2

	Home
	> session list
	> session details
	> risk details

Table 3 : KSB properties related to KPIs

 3

	KSB

  3.2. The value of Cognitive Artefacts is still calculated by the measured efficiency, but the formula has been changed. A change in efficiency indicates the potential "learning result" of a user.

Indicator/s: o Efficiacy (target number of threshold crossings vs. the acutal number of crossings) D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final) Date 19.09.2013 ELLIOT Consortium Dissemination: Public 18/235

Table 4 : ELLIOT logistics Living Lab workshops since D4.3.1

 4

	Location	Date	Living Lab Phase	Duration	LL no. / Iteration
				(hh:mm:ss)
	Bremen (BIBA)	25.06.2012	Co-Creation	03:05:00	I / 1 st
	Bremen (BIBA)	27.06.2012	Exploration	03:05:00	I / 1 st
	Bremen (BIBA)	05.07.2012	Co-Creation	02:40:00	II / 1 st
	Bremen (BIBA)	06.07.2012	Exploration	02:15:00	II / 1 st
	Bremen (BIBA)	11.07.2012	Co-Creation	02:45:00	III / 2 nd (I)
	Bremen (BIBA)	27.07.2012	Co-Creation	02:15:00	IV / 1 st

Table 5 : General information about participants of the logistics use-case Living Lab IV -Co-Creation Workshop General Information about Living Lab participants

 5

	Co-Creation	(27.7.2012)	Age 21 29 28 25	Sex f m m f	Profession Student Student Professional Professional	Expertise History Production Logistics Industrial Engineering	Experience 0 0 3 1
			22	m	Student	Production	0
	It has already been discussed in D4.3.2 that the number of participants attending a single
	workshop or a couple of workshops are not likely to reach a representative sample either for

Bremen or for Bremen University or for the number of students in engineering at the University of Bremen. However, taking into account that the majority of engineering students are male, a D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final) Date 19.09.2013 ELLIOT Consortium Dissemination: Public 21/235

Table 6 : Indicator and target values for experiment I

 6

		Duration Threshold	Target	Risk	Target	Public
			crossings	threshold	crossings	Risk
				crossing		crossing
	Product temperature	08:14:28	880	650	8	6	1
	Forklift angle	0:26:05	5	3	5	5	1

D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final) Date 19.09.2013 ELLIOT Consortium Dissemination: Public 26/235

Table 7 : KSB values (in percentage terms) for the two evaluated services (Experiment I) KSB Property Product temperature Forklift angle K3.2 Cognitive

 7

		Artefacts	64,62%	33,00%
	S5.1 Attractiveness	137,35%	7,25%
	B1.2	Performance Level (IoT)	66,67%	100,00
	B3.1	Ergonomic Quality	137,35%	7,25%
	B5.1	Accessibility	100,00%	100,00%

S5.1 and B3.1 have the same value since the complaint button was not used during the experiment.

Values over 100% indicate that the target value was exceeded; meaning that during the service D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final) Date 19.09.2013 ELLIOT Consortium Dissemination: Public 27/235 usage the number of crossings was lower than expected. If the data is consistent, this would indicate positive influence on the user experience of the services.

Table 8 : Indicator and target values for experiment II

 8

		Duration Threshold	Target	Risk	Target	Public
			crossings	Threshold	crossings	Risk
				crossing		crossing
	Product temperature	00:55:31	0	10	0	6	1
	Forklift angle	03:42:00	11	3	7	5	1

Table 9 : KSB values (in percentage terms) for the two evaluated services (Experiment II)

 9

	KSB Property	Product temperature	Forklift angle
	K3.2 Cognitive Artefacts	200,00%	0,00%
	S5.1 Attractiveness	15,42%	61,67%
	B1.2	Performance Level (IoT)	200,00%	60,00%
	B3.1	Ergonomic Quality	15,42%	61,67%
	B5.1	Accessibility	100,00%	100,00%

S5.1 and B3.1 have the same value since the complaint button was not used so far during the experiment. Values over 100% indicate that the target value was "overfulfilled", meaning that during service usage the number of crossings was lower than expected. Assuming the data is consistent, this would indicate a positive influence of the services on the user experience, as implied by the difference between the property values B1.2 from Experiment I and II; however, in this particular case it can only be used as an example for the KSB model application. The K3.2 for the forklift angle is not achieved, because the number of threshold crossings exceeds the target value almost threefold.

Table 10 : Threshold Crossings for Experiment II (Risk: forklift angle)

 10

	threshold_id	total
		duration
	2720	0:00:01
	2721	0:00:07
	2722	00:00:02
	2723	00:00:01
	2724	00:00:01
	2986	00:00:01
	2988	1:45:41
	3410	00:01:25
	3429	00:30:13
	3502	0:00:03
	3720	1:21:21

3.5.2 Observation and Questionnaire Data Analysis 3.5.2.1 Observation The

observation of Living Lab workshops focuses on co-creation and exploration. The main reason is that during experimentation, usually only one forklift driver is active (where activity equals the running time of the service). In D.4.3.2, the activity level of two workshops each for co-creation and exploration were analysed. Results from that analysis are referred to here while taking into consideration an additional fourth co-creation workshop (cf. Section 3.4.2.1). The co-creation workshop in this Living Lab is based on a serious game; the time spent playing the game is the analysed duration.

The observation of the interaction and activity of the co-creation workshop participants shows differing results. Co-creation II started smoothly, then reached a very high level of participation quickly and maintained a high level until the end of the workshop. The co-creation workshop in Living Lab III started at a slightly higher level of activity, reached a high participation level D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final) Date 19.09.2013 ELLIOT Consortium Dissemination: Public 30/235

Table 11 : Average of responses to questionnaire items regarding expectations in four workshops Question: Workshop 1 try out my own ideas. 2 get an insight into Internet of Things (IoT) 3 gain new skills 4 change something 5 take an active role in the events

 11

	I	2	1.8	1.6	1.4	2.4
	II	-0.4	1.2	0.2	0.8	2.2
	III	1	1.5	0	0.25	1.5
	IV	1	1.2	1.4	1	2.4

Table 12 : Answers to questions regarding the experience with IoT (workshop group IV) Workshop Group Participant no. Q1 I have experience with the application of IoT services. Q2 I have experience with the development of IoT services. Q3 I know and understand the technical background (sensors, actuators, logic) of an IoT service.

 12

Table 13 : Answers to questions regarding "attitude regarding IoT" in co-creation phase (all groups; answers are indicated overall and in brackets per workshop)

 13

	---(-3)	--(-2)	-(-1)	-/+ (0)	+ (1)	++ (2)	+++ (3)
	IoT services can make					14	4
	day to day life easier.					(3/3/4/4)	(0/2/2/0)

Table 14 : Weights for the KSB Measurement KSB KSB name weight

 14

	K2.2	Attunement of affordances	[none]
	K3.2	Cognitive Artefacts	1.0
	S5.1	Attractivenes	1.0
	B1.2	Performance Level (IoT)	0.4
	B3.1	Ergonomic Quality	0.4
	B5.1	Accessibility	0.2
	By applying the property values from experiments and the respective weights to the KSB model,
	the following results can be obtained.
	The graphs were generated using an FCSR KSB modelling tool and an additional ellipse
	indicating outcome uncertainty was added into it in order to visualise to which extent the applied
	KSB properties were measured through the Living Lab.

Table 15 : Set of K, S and B perspectives and related properties for Media Scenario. Personalized Service & Public Transport Scenario

 15

		K2.4	Conation	Level of appreciation on buying healthy products.
			Human
	K	K4.1	computer
			interaction
		S7.2	Appealingness	Entity of purchasing over total products visualizations
			New
		B1.1	functionalities	Appreciation of new products for vending machine such as menu.
			(IoT)
	B	B.1.2	Performance level (IoT)	Entity of purchasing over total products visualizations
		B.3.2	Ease of use	User sessions with a purchase and with a time length that implies a good user interface.

User sessions with a time length that implies a good user interface.

K.4.3

Cognitive artefacts Efficiency of providing additional information un purchasing products.

S

S2.1

Communication Entity of interactions between two or more other persons near the vending machine.

S7.1 Attractiveness

Effective sessions started by users regarding total access inside the temporary store.

Table 16 : Set of K, S and B perspectives and related properties for PT & PS Scenario. Tourism Scenario

 16

		K2.4	Conation	Conation drives how one acts on those thoughts and feelings.
	K	K5.1	Self-examination	Introspection on self-actions via experiences (thoughts).

Table 18 : Data collected via the IOT system for each vending machine.

 18

	Log Id Date	Message

4.4.1.3 Tourism Scenario

Data collected by the wearable device and used by the related service are:

-HR, heart rate (in rpm); -BR, breath rate (in bpm); -ECG, electrocardiogram tracing (in mV); -metabolic cost of physical activity, expressed in METs (Metabolic Equivalent of Task); 4.

4.2 Observation and Survey Data

  In what way could you imagine improving the accessibility of the mobility service? Think of practical examples and try to describe them in as much detail as possible. Q3) In what way could you imagine improving the socialization of the mobility service? Think of practical examples and try to describe them in as much detail as possible.

	Q2) Think of a iPhone. What's so special about this product? How does it differ from other
	products? Take these qualities and apply them to the Innovative Vending Machine -and describe
	how it could evolve if it had these same qualities. Be as creative and blue-sky as possible! Have
	fun!
	inclined to use public means of transportation more frequently? And how could this service
	communicate the benefits of travelling through the public network such as time that can be used
	for other things rather than sitting in a car stuck in traffic?
	This gaggiot would like to explore further the theme of public transportation and find new ways

D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final) Date 19.09.2013 ELLIOT Consortium Dissemination: Public 72/235 The question provided were: Q1) In what way could you imagine improving the rewardship of the Innovative Vending Machine? Think of practical examples and try to describe them in as much detail as possible. Q3) What if the Innovative Vending Machine was used in a school. In what way could it respond and react to this context? How could it evolve to fit in this new context? Q4) What if the Innovative Vending Machine was used mainly at night time. In what way could it respond and react in this timeframe? Q5) Think of a doctor. What's so special about this figure? How does it distinguish itself from others? Take these qualities and apply them to the Innovative Vending Machineand describe how it could evolve if it had these same qualities. Be as creative and blue-sky as possible! Have fun! The second game (Mobility Service) was presented in the following way: "How could ICT and IoT service improve the way we move around our cities and incentivize us to use public means of transportation more? Many of us use a number of different means of public transportation every day, to go to work, to go shopping, to go study. Nevertheless, there are still a great number of people who rely on their cars for weekend trips, or sporadic trips when they do not know how to get to their final destination. How can a mobility service help users feel more confident and therefore more to engage users so that they can truly appreciate the benefits of travelling on public means of transportation, in terms of quality of time and impact on the environment." D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final) Date 19.09.2013 ELLIOT Consortium Dissemination: Public 73/235 The questions were: Q1) Look at these verbs (award, coach, operate) . Imagine that the mobility service could do one of these thingswhat would you chose it to do? How would the mobility service evolve if it could do this one thing? Q2) Q4) What if the mobility service was used in a hospital. In what way could it respond and react to this context? How could it evolve to fit in this new context? Q5) Think of Groupon. What's so special about this service? How does it differ from other services? Take these qualities and apply them to the mobility serviceand describe how it could evolve if it had these same qualities. Be as creative and blue-sky as possible! Have fun! 4.4.2.1.3 Tourism Scenario The co-creation process focused on the Tourism Scenario was provided through a serious game based on a "Smart patch". "How could a wearable sensor-embedded Patch change the way we monitor and improve our health? In what way could a wearable sensor-embedded patch help us to learn more about our health and become more conscious about our well-being? In what way could this product support us when we fall ill? And how could it support both beginners as well as professional athletes in monitoring their biological vital signs? This wearable sensor-embedded patch can measure a person's ECG and bioimpedence, and also contains accelerometers.

  Q3: What if the Smart Patch was used mainly by senior citizens. In what way could it respond and react in this user? What could it do or offer that is unique to this user? Q4: Look at these verbs: motivate, control, alert. Imagine that the Smart Patch could do one of these thingswhat would you chose it to do? How would the Smart Patch evolve if it could do this one thing?

Q5: Think of Linked-in. What's so special about this service? How does it differ from other services? Take these qualities and apply them to the Smart Patch and describe how it could evolve if it had these same qualities. Be as creative and blue-sky as possible! Have fun!

4.4.2.2 Scenarios observation and survey data

4.4.2.2.1 Tourism Scenario

  D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation

		(final)	Date	19.09.2013
	ELLIOT Consortium	Dissemination: Public		76/235

11 10/11 11/11 12/11 13/11 14/11 15/11 16/11 17/11 18/11 19/11 20/11 21/11 22/11 23/11 24/11 25/11 26/11 user #1

  D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation

							(final)			Date	19.09.2013
		Android device 9/Samsung Galaxy
		ACE								x	x	x	x	x
		Samsung Galaxy
	user #2	ACE	x	x	x	x
		Samsung Galaxy
	user #3	ACE								x	x	x
		Samsung Galaxy
	user #4	ACE	x				x	x	x
		Samsung Galaxy
	user #5	ACE								x	x	x	x
		Samsung Galaxy
	user #6	ACE								x	x	x
		Samsung Galaxy
	user #7	ACE				x	x	x	x	x	x	x
		Samsung Galaxy
	user #8	ACE								x	x	x	x	x
	user #9	Samsung Galaxy Tab								x	x	x
	user #10 ASUS Transf. Prime					x	x	x	x
		ELLIOT Consortium				Dissemination: Public	77/235

Table 19 : time scheduling of body sensor network evaluation phase (x show the day when body sensor network and related service have been used)

 19

D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final) Date 19.09.2013 ELLIOT Consortium Dissemination: Public 78/235

Figure 52: number of choices by item regarding Q1.

						D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final) D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final)	Date Date	19.09.2013 19.09.2013
							Percentage of Users by sex
	18 20						Number of visits per day 50% 50%			male female
	16
	14
	12
	10
	8
	4 6																					Visits/day
	2
	0
	07/11/2012	09/11/2012	11/11/2012	13/11/2012	15/11/2012	17/11/2012	19/11/2012	21/11/2012	23/11/2012	25/11/2012	27/11/2012	29/11/2012	01/12/2012	03/12/2012	05/12/2012	07/12/2012	09/12/2012	11/12/2012	13/12/2012	15/12/2012	17/12/2012
										Users per ages
		8
		7
		6
		5
		4
		3																			# Users
		2
		1
		0
	ELLIOT Consortium ELLIOT Consortium						Dissemination: Public Dissemination: Public								91/235 92/235

Figure 51: percentage of users by sex. Below are the questions proposed during the serious game: participants can answers by selecting a specific tag and improving the answer with added text. Moreover, each users can indicate what answer he/she likes given by other users. Q1: Imagine that Smart Patch, could sense proximity; weather; touch. How could this new sense change the Smart Patch, adding value to what it offers to its users.

Table 20 : answers analysis for questions on device, app and web-portal. It's reported average value and standard deviation (for all participants).

 20

	D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final)	Date	19.09.2013

Table 21 : KSB Mapping Media Scenario

 21

		K3.2	Mental mapping KPI3/KPI1
			Human
	K	K4.1	computer	KPI1/KPI2
			interaction
		K.6.1	Team cognitive process	KPI4/KPI3
		S2.2	Collaboration	KPI5/KPI1
	S	S4.1	Relationship enhancement	KPI6/KPI5
		S7.2	Appealingness	KPI1/KPI2
			New
	B	B1.1	(IoT) functionalities	KPI7/KPI1
		B4.1	Usefulness	KPI1/KPI2

Table 22 : KSB Property Weights Media Scenario

 22

			Prescolare		Primario		Secondario		Total
			percentage	1to 5	percentage	1to 5	percentage 1to 5 percentage	1to 5
	K3.2	Mental mapping	43,86%	2,75	10,20%	1,41	35,51%	2,42	32,73%	2,31
	K
		Human
	K4.1	computer interaction	2,69%	1,11	28,63%	2,14	7,19%	1,28	6,56%	1,26

4.5.3.2 Personalised Service and Public Transport Scenario

The KSB mapping of previously collected KPIs refer to this table: here KPIs are used to create a set of KSB properties to evaluate the service provided in Personalised Service and Public Transport Scenario.

Table 23 : KSB Mapping Personalised Service and Public Transport Scenario

 23

		K 2.4	Conation (Desire)	KPI7 / KPI11
			Human
	K	K 4.1	computer	KPI4 / KPI3
			interaction
		K 4.3	Cognitive artefacts	KPI14 / KPI13+KPI14
		S 2.1 Communication KPI11 / KPI3
	S	S 7.1 Attractiveness	KPI3 / KPI3 + KPI10
		S 7.2 Appealingness	KPI1 / KPI2
			New
		B 1.1	functionalities	KPI7 / KPI1
	B	B 1.2	(IoT) Performance level (IoT)	KPI1 / KPI2
		B 3.2 Ease of use	KPI6 / KPI5

The KSB properties obtained by these KPIs can also be computed as percentage values or on a scale from 1 (worst value) to 5 (best value). The following table explains KSB property values D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final) Date 19.09.2013 ELLIOT Consortium Dissemination: Public 98/235 for October 2012 and February 2013.

Table 24 : KSB Property Weights Personalised Service and Public Transport Scenario

 24

	Oct. 2012		Feb. 2013
	percentage	1to 5	percentage	1to 5

Table 25 : KSB properties and related survey questions regarding the wearable monitoring device.

 25

		K2.4p Conation	Q8. When I saw the patch I was interested in trying it
	K	K5.1p Self-examination	Q11. Did wearing the patch change you in any way?
		K.5.2p Self-Consciousness	Q10. How did you feel whilst wearing the patch?
	S	S1.1p	Social Networking	Q7. The patch helped me create new social relations

Table 26 : KSB properties and related survey questions regarding the smart-phone application.

 26

		K2.4a	Conation	Q19. When I saw the patch I was interested in trying it
	K	K5.1a	Self-examination	Q22. Did using the app change you in any way?
		K.5.2a	Self-Consciousness	Q21. How did you feel whilst using the app?
	S	S1.1a	Social Networking	Q18. The app helped me create new social relations
				Q12. The text in the portal was easy to read (B3.1 ergonomic
				quality)
		B3.1a	Ergonomic quality	Q13. The language and the graphic of the app were easy to
				understand
				Q14. The navigation of the app was easy to understand
	B
		B.3.2a Ease of use	Q15. The app was easy to use
		B.3.3a Learnability	Q16. It was easy to learn how to use the app
		B4.2a	Emotional Connection	Q20. The more time passes, the more I liked to wear the patch

B.4.3a Hedonic quality Q17. The app was innovative and original D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final) Date 19.09.2013 ELLIOT Consortium Dissemination: Public 100/235

Table 27 : KSB properties and related survey questions regarding the web portal.

 27

		K2.4w	Conation	Q30. When I saw the patch I was interested in trying it
	K	K5.1w	Self-examination	Q33. Did using the portal change you in anyway?
		K.5.2w	Self-Consciousness	Q32. How did you feel whilst using the portal?
	S	S1.1w	Social Networking	Q29. The portal helped me create new social relations
				Q23. The text in the portal was easy to read
		B3.1w	Ergonomic quality	Q24. The language and the graphics of the portal were easy to understand
				Q25. The navigation of the portal was easy to understand
	B
		B.3.2w Ease of use	Q26. The portal was easy to use
		B.3.3w Learnability	Q27. It was easy to learn how to use the portal
		B4.2w	Emotional Connection	Q31. The more time passes, the more I liked to wear the patch
		B.4.3w	Hedonic quality	Q28. The portal was innovative and original
	Excluding B3.1, K5.1 and K5.2, all KSB properties value can be directly obtained from survey

results: the value of KSB properties is the average value of the related answer for all participants.

Table 28 : Example of B3.1 property valuation starting from survey answers weighted together.

 28

	1p value

Question

Average values for all participants Relative weight (%) B3.

Table 29 : KSB properties regarding wearable monitoring device.

 29

		K2.4p Conation	4,4
	K	K5.1p Self-examination	1,71
		K.5.2p Self-Consciousness	2,08
	S	S1.1p	Social Networking	1,1
		B3.1p	Ergonomic quality	4,06
		B.3.2p Ease of use	3,6
	B	B.3.3p Learnability	3,5
		B4.2p	Emotional Connection	2,5
		B.4.3p Hedonic quality	3,5

Table 30 : KSB properties regarding smart-phone application.

 30

		K2.4a	Conation	4,4
	K	K5.1a	Self-examination	2
		K.5.2a Self-Consciousness	2,09
	S	S1.1a	Social Networking	1
		B3.1a	Ergonomic quality	4,33
	B
		B.3.2a Ease of use

Table 32 : KSB Weights Media Scenario

 32

		Property index	Property name	Weight (%)	Perspective Discovery (%)
		K3.2	Mental mapping	33	20
	K	K4.1	Human computer interaction	33	40
		K.6.1	Team cognitive process	33	20
		S2.2	Collaboration	40	30
	S	S4.1	Relationship enhancement	20	5
		S7.2	Appealingness	40	40
			New
	B	B1.1	(IoT) functionalities	50	40
		B4.1	Usefulness	50	40

Table 33 : KSB Perspective Values Media Scenario

 33

		prescolare		primario		secondario		total
		Absolute	Relative	Absolute	Relative	Absolute	Relative	Absolute	Relative
		Perspective	Perspective	Perspective	Perspective	Perspective	Perspective	Perspective	Perspective
		value	Index (%)	value	Index (%)	value	Index (%)	value	Index (%)
	K	2.24	28.78	1.69	46.52	2.58	43.96	2.14	36.84
	S	3.04	47.53	1.47	31.67	2.29	36.25	2.25	40
	B	2.02	23.67	1.33	21.8	1.71	19.8	1.73	23.16

  possible to evaluate the high rate of service requests and the user acceptance (more than 2000 children played with the totem and it received more than 200 meal reservations from June 2011

to May 2012). From the previous table it is possible to see that only the "prescolare" group of children appreciate the Totem (K, S and B absolute perspective index is higher than the others D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final) Date 19.09.2013 ELLIOT Consortium Dissemination: Public 109/235

Table 34 : KSB Property Weights Personalised Service and Public Transport Scenario

 34 Based on these values, a KSB ternary plot can be created for each category. As for the Media Scenario it is possible to obtain the Perspective Knowing of each perspective of Personalised Service and Public Transport Scenario: thanks to the Elliot analysis is possible to determine about 60% of the Knowledge Perspective, 80% of the Social Perspective and 90% of the Business

		Property index	Property name	Weight (%)	Perspective Discovery (%)
		K 2.4	Conation (Desire)	33	20
	K	K 4.1	Human computer interaction	33	20
		K 4.3	Cognitive artefacts	33	20
		S 2.1	Communication	20	20
	S	S 7.1	Attractiveness	40	30
		S 7.2	Appealingness	40	30
			New
		B 1.1	functionalities	30	30
	B	B 1.2	(IoT) Performance level (IoT)	35	30
		B 3.2	Ease of use	35	30
	Perspective.
	Each dataset contains an equivalent quantity of data so it is possible to set the Knowledge
	perspective to behave as a constant through Oct 2012 and Feb 2013.
	These are the KSB perspective values obtained from the KSB ternary plot.

D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final) Date 19.09.2013 ELLIOT Consortium Dissemination: Public 110/235

Table 35 : KSB Perspective Values Personalised Service and Public Transport Scenario

 35

		Oct 2012		Febr 2013
		Absolute	Relative	Absolute	Relative
		Perspective	Perspective	Perspective	Perspective
		value	Index (%)	value	Index (%)
	K	2.73	55.86%	2.67	42.9%
	S	1.94	30.41%	2.06	27.39%
	B	1.41	13.72%	2.16	29.69%

Table 36 : KSB Property Weights Tourism Scenario

 36

		Property index	Property name	Weight (%)	Perspective Discovery (%)
		K2.4	Conation	33	10
	K	K5.1	Self-examination	33	10
		K.5.2	Self-Consciousness	33	10
	S	S1.1	Collaboration	100	20
		B3.1	Ergonomic quality	20	5
		B3.2	Ease of use	20	5
	B	B3.3	Learnability	20	5
		B4.2	Emotional Connection	20	5
		B4.3	Hedonic quality	20	5

Table 37 : KSB model with perspective valuation (absolute and relative) regarding Tourism service composed of only the wearable device.

 37

	Perspective	Property	Property	Absolute Perspective value	Relative Perspective Index
			value
		K2.4p	4.4
	K	K5.1p	1,71	2.73	37.25%
		K5.2p	2,08
	S	S1.1p	1,1	1.1	2.14%
		B3.1p	4,07
		B3.2p	3,6
	B	B3.3p	3,5	3.81	60.59%
		B4.2p	4.4
		B4.3p	3,5

Table 38 : KSB model with perspective valuation (absolute and relative) regarding the complete service.

 38

	Perspective	Property	Property	Absolute Perspective value	Relative Perspective Index
			value
		K2.4p	4.4
		K2.4a	4.4
		K2.4w	4
		K5.1p	1.71
	K	K5.1a	2	2.71	41.14%
		K5.1w	1.5
		K5.2p	2.08
		K5.2a	2.09
		K5.2w	2.19
		S1.1p	1.1
	S	S1.1a	1	1.03	0.8%
		S1.1w	1
		B3.1p	4,07
		B3.1a	4.33
		B3.1w	4
		B3.2p	3,6
		B3.2a	4
		B3.2w	3.75
	B	B3.3p B3.3a	3.5 4.4	3.41	58.06%
		B3.3w	4.25
		B4.2p	2.5
		B4.2a	2.6
		B4.2w	2.5
		B4.3p	3.5
		B4.3a	2.4
		B4.3w	1.75

Figure 60: KSB evaluation, the yellow star indicate the KSB actualization of service as if composed only by wearable monitoring device

		D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final)	Date	19.09.2013
	ELLIOT Consortium	Dissemination: Public		116/235

Table 39 : Participants involved in Green Services Use-case

 39 Table we present the participants involved in our use-case since the co-creation workshops. We reach a total of 48 participants of which 44 are different.

		Co-creation / Exploration	Experimentation
		Workshop	Workshop	Green fake	Experiment	Experiment	Total
		Mobility	Health	watch	1	2
	New	13	8	6	11	6	44
	participant
	Previous				2	2	4
	recruitment
	Total	13	8	6	13	8	48

  ELLIOT -Experiential Living Lab for the Internet Of Things

			Project N.	258666
		D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final)	Date	19.09.2013
	ELLIOT Consortium	Dissemination: Public		126/235

Table 40 : Green Services KSB Instantiation

 40

	K elements	Perceptual	K2.2	Sensing affordances
		Cognitive	K3.2	Cognitive artefact
		Social Ties		Social networking and
		(Connectivity) S1.1	openness
	S elements	Interaction
		(Interactivity) S2.1	Communication
		Emotional
		Connection
		(Affectivity)	S5.1	Attractiveness
		Performance
		(conformability) B2.1	Reliability
		Friendliness	B3.1	Ergonomic quality
		(usability)	B4.1	Usefulness
	B elements	Satisfaction	B4.2	Hedonic quality
		(Favourability)	B4.5	Loyalty
		Ownership
		(Recognition)	B6.1	User ideas
		Privacy
		(Protection)	B7.1	Data protection

Table 41 : UX type and target assessed via online questionnaires/diaries in the MyGreenServices experiment

 41

Table 42 : Extraction of alerts tracks

 42

Table 43 : Green Services KSB instantiation approach

 43

	Direct questions	Indirect questions	No
					question
	Without log	With log Without log With log
	S2.1	S1.1	K2.2	K3.2	B4.5
		B6.1	B4.2	S5.1
		B2.1	B7.1	B3.1
				B4.1

Table 44 : Data KSB mapping for Experimentation

 44

		Ref	KSB	Input	Indicators with weighting
			Properties
	Knowledge K2.2	Sensing	Questionnaires	Air quality knowledge
			affordances	(profiling and
				Q4)
		K3.2	Cognitive	Questionnaire	Environment evaluation (0.2), Change in
			artefact	(Q4), Log,	activity (0.4), IoT Data comparison (0.4)
				Interview
	Social	S1.1	Social	Questionnaire	Level of activity, Frequency of connections,
			Networking	(Q2), Log	Forum statistics
			and openness
		S2.1	Communication Interview	Number of MGS demonstrations

Table 45 : Attractiveness rule

 45

			Q7
			Yes	No
	Q9	Yes	high	medium
		No	medium	low

  -Report on IOT Living Labs Continuous Exploration and Evaluation It takes the following data table as input:

		(final)	Date	19.09.2013
	ELLIOT Consortium	Dissemination: Public		151/235

Table 46 : Input for Usefulness

 46 Using the Focuslab server of the Elliot platform, this data table is analysed by calling the MNDCluster_sequence Web Service which generates the best partition composed of three clusters and then a sequence of clusters for each user (cf. Table47).The Output is a csv data file such as the following data table output (used in the B4.1 Web Service) when Focuslab is accessed interactively. In our case, the Elliot platform updates our dataset using FocusLab, which is one important role of FocusLab methods and the natural integration of

	User	Session S1 Transport S1 Sport S1 Sorties S1 Aeration S1 … Sorties S5 Aeration S5 Autre S5
	User 1	Non	Non	Non	Non	Non	… Non	Non	Non
	User 3	Oui	Non	Non	Non	Oui	… Non	Oui	Oui
	User 6	Non	Non	Non	Non	Non	… Non	Oui	Non
	User 2	Non	Non	Non	Non	Non	… Non	Non	Non
	User 8	Non	Non	Non	Non	Non	… Non	Non	Non
	User 9	Non	Non	Non	Non	Non	… Non	Non	Non
	User 4	Oui	Oui	Oui	Oui	Non	… Non	Oui	Oui
	User 7	Oui	Non	Non	Non	Oui	… Non	Non	Non
	User 10 Non	Non	Non	Non	Non	… Non	Non	Non
	User 5	Oui	Non	Non	Non	Non	… Non	Oui	Oui
	…	…	…	…	…	…	… …	…	…
	FocusLab server.

Figure 84: List of Web Services for Green Services Use-case

  We illustrate here these web services with two examples K2.2 and K4.1. For more information see the appendix 8.5. Our web services are available using URLs such as: For K2.2 http://focuslab.inria.fr/focuslab/mygreenservices_usecase/inria_greenservices_k2.2.jsp

		D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final)	Date	19.09.2013
	or
	for B4.1 http://focuslab.inria.fr/focuslab/mygreenservices_usecase/inria_greenservices_b4.1.jsp
	The same implementation model is used than the ones defined for FocusLab server (RESTful
	Web Services), as described in Appendix 8.5.
	For B4.1, the data file in input looks like the previous data table obtained with
	MNDCluster_sequence. The result of the B4.1 web service call is an XML file:
	<?xml version="1.0" encoding="UTF-8"?>
	<xml>
	<perCentOfUserChange>50.0</perCentOfUserChange>
	</xml>
	For K2.2, the data file in input is given in section 5.6.2.1:
	ELLIOT Consortium	Dissemination: Public		154/235

Table 49 : Users change in air quality knowledge

 49

	User	Profiling	Q4
	User 1	Low	Rather low
	User 2	Rather low Rather low
	User 3	Low	Rather low
	User 4	Rather low Rather
			good
	User 5	Rather low Good
	User 6	Rather low Rather low
	User 7	Rather low No data
	User 8	Rather low Rather low
	User 9	Rather low Rather low
	User 10	Rather low Rather
			good

Table 53 : Percentage of users who made a comparison between data coming from MGS portal and other sources

 53

	Comparison
	Yes	No
	66,70%	33,30%

Table 54 : Cognitive artefact balancing

 54

		Cognitive artefact
		Yes	No
	% users	50,44%	47.56%
	5.6

.2.3 S1.1 Social Networking and Openness: High (cf. rule5.5.3.1.3)

  To compute this property, we use the user answers to one indirect question Q8 from questionnaire Q2 and log data (see below).

		Q2/

Q8: Did MyGreenServices Forum encourages you to participate and share information?

		D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final)	Date	19.09.2013
	ELLIOT Consortium	Dissemination: Public		162/235

Table 55 : User profile on MGS forum

 55 To compute this property, we use the user answers to one indirect question from the final interview and log data (see below).

		Activity
		Pro Active	Active	Inactive
	# users	3	5	5
	% users	23% 38,50%	38,50%
	5.6.2.4 S2.1 Communication: High (cf. rule 5.5.3.1.4)

Table 57 : Recommendations

 57

		Yes	No
	Yes	77,80% medium
	high
	No	22,20%	low
	medium

Table 58 : The reliability of MGS data

 58

			Reliability
		Yes indeed Rather yes Rather no No, not at all Total
	%	5%	70%	15%	10%	100%
	users

Table 59 : Reliability resumed

 59

		Reliability
		Yes	No	Total
	% users	75%	25%	100%
	5.6

.2.7 B3.1 Ergonomic quality: Medium (cf. rule 5.5.3.1.7)

Table 60 : User perception against the intuitivity of the alert service Intuitiveness alerte service

 60

		Yes, indeed Rather yes		Rather no	No, not at all
	Nb users	0	5	2	4
	% users	45,45%	18,18%	36,36%

Table 61 : Findability on MGS portal

 61

			Findability portal
		Yes,	Rather	Rather no No, not at
		indeed	yes	all
	Nb users	2	8	3	0
	% users	15,38%	61,54%	23,08%

Table 62 : Ergonomic quality property resumed

 62

		Ergonomic quality
		Yes	No
	% users	57,3%	38,62%
	5.6.2.8 B4.1 Usefulness: High (cf. rule 5.5.3.1.8)

To compute this property, we use the user answers to two indirect questions (Q1, Q9) from episodic questionnaire Q1.2 and one from the final interview (see below).

Table 63 : Output from MNDCluster_Sequence classification

 63

	VARIABLE	Part_3
	Session	S1	S2	S3	S4	S5
	User 1	C_3/3	C_3/3	C_3/3	C_3/3	C_3/3
	User 3	C_2/3	C_2/3	C_2/3	C_2/3	C_2/3
	User 6	C_3/3	C_3/3	C_3/3	C_3/3	C_1/3
	User 2	C_3/3	C_3/3	C_3/3	C_3/3	C_3/3
	User 8	C_3/3	C_3/3	C_3/3	Null	C_3/3
	User 9	C_3/3	C_3/3	C_3/3	Null	C_3/3
	User 4	C_1/3	C_1/3	C_1/3	C_1/3	C_1/3
	User 7	C_2/3	C_3/3	C_3/3	Null	Null
	User 10	C_3/3	C_3/3	C_3/3	C_3/3	C_3/3
	User 5	C_3/3	C_3/3	C_2/3	Null	C_1/3

Table 64 : Percentage of users declaring a change in their habits

 64

	Change in habits
	Yes	No
	# users	5	5
	% users	50%	50%

5.6.2.9 B4.2 Hedonic quality: Medium (cf. rule 5.5.3.1.9)

Table 65 : Opinion on MGS portal

 65

			Opinion MGS portail
		Very satisfied Satisfied Unsatisfied Very unsatisfied Total
	Nb users	4	7	0	0	11
	% users	36,4%	63,6%		100

Table 66 : Opinion on the alert service

 66 More than half of the users expressed satisfaction with the aesthetic aspect of the forum.Nevertheless the fact that aesthetics should not prevail is highlighted in answers to open questions.

			Opinion alerte service
		Very satisfied Satisfied Unsatisfied Very unsatisfied	Total
	Nb users	3	6	0	2	11
	% users	27,3%	54,5%	0,00	18,2%	100

"By cons, pretty or ugly, I do not know, I think it is very subjective as question the practicality should come before."

Table 67 : Aesthetic aspect of the MGS forum Aesthetic forum

 67

		Pretty	Rather	Rather Ugly	Ugly	Total
			pretty
	Nb users	1	3	3	0	10
	% users	10,00%	30,00%	30,00%	0,00%	100

Table 68 : Hedonic quality resumed

 68

		Hedonic quality
	Very	Satisfied Unsatisfied Very
	satisfied	unsatisfied
	% users	27,50% 52,90%	7,50%	4,50%

Table 71 : Statistics from user session

 71

	User	#Number	#Duration	Average	Number	Conclusion
		of sessions		session	of days
				/day
	User 1	22	424	1,375	16	Yes
	User 2	5	173	0,3125	16	Yes
	User 3	14	69	0,875	16	Yes
	User 4	27	138	1,6875	16	No
	User 5	65	553	4,0625	16	Yes
	User 6	16	185	1	16	No
	User 7	2	5	0,125	16	Yes
	User 8	19	132	1,1875	11	No
	User 9	4	21	0,25	16	Yes
	User 10	52	251	3,25	16	Yes
		17,92	156,54

Table 72 : User perception of MGS frequentation matched with log data

 72

	Yes	No		Total
	#users	7	3	10
	%users	70%	30%	100%

Table 73 : Loyalty property resumed

 73

		Yes	No
	% users	77,5%	22,5%
	In our case and considering the results shown above, loyalty is high.

5.6.2.11 B6.1 User Ideas and created content: Medium (cf. rule 5.5.3.1.11)

To compute this property, we use the user answers to one indirect question from questionnaires D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (see below).

Table 74 : Number of ideas posted on MGS forum

 74

	User	#ideas
	User 1	9
	User 2	2
	User 3	1
	User 4	14
	User 5	8
	User 6	0
	User 7	0
	User 8	0
	User 9	0

Do you feel that your personal data are protected by MyGreenServices?

		D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final)	Date	19.09.2013
		Q4/Q15: Please specify
		ELLIOT Consortium	Dissemination: Public	171/235



Q4/Q14:

Table 75 : User perception of data protection on MGS portal

 75

	Data protection
	Yes	No
	# users	6	0
	% users	100%	0%
	5.6.

3 Conclusions of experiments Figure 91: Quantitative values of KSB properties.

  instead of Medium in experiment 1). The percentage of users finding MGS portal and his components intuitive rose between MGS v1 and MGS v2 (respectively 57.3% in experiment 1 and 83% in experiment 2. The first experiment related to MyGreenServices clearly indicates good results in terms of user experience. The new data issued from the second experiment has confirmed the results of the first one. Let us note one variation in the KSB instantiation in terms of value obtained (high, medium, low) for the B3.1 property, and better results for KPIS used for two properties (B2.1 and S1.1). These differences are due to the improvement of

		D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final)	Date	19.09.2013
	MyGreenServices (v2) and better community management.
	Concerning B2.1 reliability (cf. Section 5.6.2.6), participants of the second experiment are more
	120.00% confident in the measures provided by MyGreenServices than those in the first experiment (100%
	100.00% instead of 75% in the experiment 1). Regarding S1.1 Social Networking and Openness (cf.
	80.00%
	60.00%
	40.00%
	20.00%			Expe1
	0.00%			Expe2
	The first experiment related to MyGreenServices clearly indicates good results in terms of user
	experience. The new data issued from the second experiment has confirmed the results of the first
	one. Let us note one variation in the KSB instantiation in terms of value obtained (high, medium,
	low) for the B3.1 property, and better results for KPIS used for five properties (K3.2, S1.1, S2.1,
	B2.1, and B4.2). These differences are due to the improvement of MyGreenServices (v2) and
	ELLIOT Consortium	Dissemination: Public		172/235

better community management. The decrease of attractiveness is due to more missing answers. Indeed, the variation in terms of value occurs in B3.1 Ergonomic Quality (High for experiment 2

Table 76 : Collected citizen IoT Data

 76 This deliverable D4.3.3 reports on the progress achieved arising from the Living Lab experiments performed from the date of the delivery of ELLIOT deliverable D4.3.2 right up to the end of the project duration.

	Name	Nb measures	DateMin	DateMax
	AxISbox	324340	26/02/2013 12/07/2013
	(PM10)
	Pollux (PM10)	43708	26/01/2013 28/06/2013
	Azimut	3457602	26/01/2013 12/07/2013
	(O3No2)

8.1.1 Air quality Air PACA fixed sensors: Inria (resp) 8.1.1.1 Sensors Five fixed air stations in Nice and 2 in Antibes was

intended used. Observed data were to be collected by Airpaca every hour. Such real-time data was not finally provided in time for our experiments.

Table 77 : Sensors involved in the green watch Locality Sensors

 77

	Nice Airport (Nice) NO2, O3, SO2 et PM10
	West botanic (Nice) O3
	Lenval (Nice)	NO2 et PM10
	Pellos (Nice)	NO2 et SO2
	Paillon (Nice)	PM10
	Moulin (Antibes)	NO2, O3 et PM10

Table 78 : Sensors involved in the green watch

 78

	03	ppb, +/-20 ppb ( 20-200 ppb)
	NO2	Ppb, +/-20 ppb (20-200 ppb)
	noise	dBA, +/-3dB
	temperature	t°
	GPS	Mercator transformed in WGS84

Table 81 : Sensor involved in AxIbox

 81

	PM10, Dust	PM10 0-05 mg/m3

Table 83 : Cluster size and WSS

 83

	Clusters	5
	Cluster Description	Size	WSS
	cluster n°1 c_kmeans_1	262 820,9292
	cluster n°2 c_kmeans_2	151 690,8069
	cluster n°3 c_kmeans_3	610 1473,4363

Table 84 : Cluster centroids (partition into 5 classes) Attribute Cluster n°1 Cluster n°2 Cluster n°3 Cluster n°4 Cluster n°5

 84

	min_oz	67,007634 132,092715 26,442623 22,019370 17,895442
	moy_oz	74,949230 134,360756 33,298374 33,262322 23,847718
	max_oz	83,931298 136,556291 41,419672 46,566586 32,742627
	min_temp	13,993932 12,906835 10,317812	6,668108 22,581107
	moy_temp 14,388958 12,922044 10,510046	6,866689 22,750133
	max_temp 14,812272 12,970190 10,721937	7,109940 22,900516
	min_hygro 26,227903 39,474221 40,213306 62,545584 28,993578
	moy_hygro 27,119017 39,571993 41,034563 63,293836 29,419781
	max_hygro 28,071536 39,631205 41,869595 64,241953 29,810476

http://en.wikipedia.org/wiki/1%25_rule

D4.3.3 -Report on IOT Living Labs Continuous Exploration and Evaluation (final) Date 19.09.2013 ELLIOT Consortium Dissemination: Public 176/235

Those data are added to the qualified dataset of our use-case.

Second step, applying a specific web service

Our data set is now completed via the use of the FocusLab Web Service MNDCluster_sequence.

To compute KSB indicators, Inria has implemented nine other simple Web Services (including 1 used both for 4 KSB indicators) for the Green Services use-case. Participants were asked in the ultime questionnaire to remember the last value seen on the MGS portal. Then their answer has been checked with data log.

Modification activity Yes

No 44,40% 55,60%

Overall, it appears from the qualitative interviews that it is difficult to intervene massively in activities. On the other hand, participants declared changes in activities relating to leisure practices, or on the choice of mode of transportation. It joined the co-creation workshops where participants reported on the fact that the margin of change is not huge. All participants intend to use the portal after the experiment. In this question, it is clear that the estimation concerns more the intention of loyalty than the loyalty itself.

Sensors

There are important delays in selecting the sensors and to equip electric vehicles. In June, VU Log changed the air quality sensor. Mid-August, one vehicle was equipped and will be tested in September. VU Log announced that 1 to 4 additional vehicles would be equipped with sensors later. What is your level of knowledge in the field of air quality?

FocusLab generic Web Service "MNDCluster_Sequence"

This web service uses the MND clustering method described in Deliverable D2.4 and computes the best partition based on all data for each (user, timestamp). Then it builds for each "user" the sequence of clusters taking into account the user timestamp (five in our case). The resulting sequences are then added for each "user" as new qualified data in the dataset of Green Services.

This service needs a csv file and a valid key (to authenticate the call) as input. These inputs can be given via the simple interface of the FocusLab server or via the ELLIOT platform that calls the Web Service. The ELLIOT platform calls this Web Service on all the data (around 119 lines of data (extracted from the middleware)) corresponding to the answers of the same question at different times by a user: four timestamps (S1..S4) during the experiment of MyGreenServices and one more during the interview (S5). A partition into three clusters (C1/3, C2/3 and C3/3) has been identified as the best one; then, the user answers for each session are clustered; the output of this service call is a csv stream as given in the following example: This output has been added in the middleware and used as input for a specific GreenServices Web Service for B4.1 indicator. In this way, the Elliot platform allows the living lab manager to play with data using the FocusLab server and to process data an interactive way until the computation of the indicators required for the KSB property rule.

Calling other Web Services for the Green Services KSB Model

Here are the URLs for calling the 9 Web Services to be applied on the extracted data. servletUrl = "http://focuslab.Inria.fr/focuslab/mygreenservices_usecase/Inria_greenservices_k2.2.jsp"; servletUrl ="http://focuslab.Inria.fr/focuslab/mygreenservices_usecase/Inria_greenservices_b2.1.jsp"; servletUrl ="http://focuslab.Inria.fr/focuslab/mygreenservices_usecase/Inria_greenservices_b3.1.jsp"; servletUrl ="http://focuslab.Inria.fr/focuslab/mygreenservices_usecase/Inria_greenservices_b4.1.jsp"; servletUrl ="http://focuslab.Inria.fr/focuslab/mygreenservices_usecase/Inria_greenservices_b4.2.jsp" servletUrl ="http://focuslab.Inria.fr/focuslab/mygreenservices_usecase/Inria_greenservices_b5.1.jsp"; servletUrl ="http://focuslab.Inria.fr/focuslab/mygreenservices_usecase/Inria_greenservices_b6.1.jsp"; servletUrl ="http://focuslab.Inria.fr/focuslab/mygreenservices_usecase/Inria_greenservices_s1.1.jsp"; servletUrl ="http://focuslab.Inria.fr/focuslab/mygreenservices_usecase/Inria_greenservices_generic_percentOfYes. jsp".