Salvatore Andolina

Paolo Ariano

Davide Brunetti

Nicolò Celadon

Guido Coppo

Alain Favetto

Cristina Gena

Sebastiano Giordano

Fabiana Vernero

Introducing gestural interaction on the shop floor: empirical evaluations

Keywords: Gestural interaction, Empirical evaluation, Industry 4.0

This paper describes two empirical evaluations we carried out in the context of an Industry 4.0 project, where we explored the use of touchless gestural interaction on the shop floor.

Introduction

In the "smart factory" vision, industries are production-oriented cyberphysical systems [START_REF] Riedl | Cyber-physical systems alter automation architectures[END_REF] able to exploit emerging information technologies to guarantee flexible, efficient, environment-friendly, high-quality and low-cost production processes [START_REF] Wang | Implementing smart factory of industrie 4.0: An outlook[END_REF]. In this context, however, designing appropriate modalities to communicate, analyze and interact with all the information related to production processes can be challenging, especially in consideration of its many environmental and safety-related constraints. For example, keyboards or touch devices may be inconvenient for workers on the shop floor, who must wear protective gloves, while voice-based interaction may be impractical due to the background noise.

Guided by the requirements expressed by our industrial partners, in the HOME regional project 4 on Industry 4.0 we explored the use of large displays in combination with touchless gestural interaction. Based on previous work in the neuromotor rehabilitation field [START_REF] Di Girolamo | Emgbased biofeedback system for motor rehabilitation: A pilot study[END_REF], the project consortium developed a smart armband which allows to detect gestures from movement and muscle biosignals, while a machine learning library allows to calibrate and recognize task-specific gestures. Notice that this ad hoc solution was preferred to the use of motion sensing input devices, in that these can raise a number of issues related to privacy and perceived surveillance. The definition of an appropriate set of gestures has undergone several steps, including a guessability study [START_REF] Andolina | Experimenting with large displays and gestural interaction in the smart factory[END_REF], in an attempt to balance the sometimes competing needs for gesture intuitiveness (end users) and recognizability (project consortium), combined with a preference for minimum calibration, due to the fact that the armband can be shared by several workers.

In this paper, we describe our final empirical evaluations with our industrial partner Galeasso s.r.l. 5 , which specializes in sheet metal fabrication and is going to adopt the armband on its shop floor.

Empirical evaluations

Two evaluations were carried out with the future users of the armband to assess its ease of use and elicit comments and suggestions on gestural interaction at the workplace. We focused on a specific scenario suggested by our partner Galeasso s.r.l., where welders frequently switch between their workbench and a nearby desktop computer to browse the tasks they have been assigned and visualize 3D models of the final product. Thus, we developed a prototype web application (Figure 1) consisting in: 1) a grid-like interface to simulate selection tasks and 2) a CAD-like environment showing 3D objects. The application can be controlled through the armband and is to be shown on large displays installed on the shop floor. The main actions allowed by the application are point and select, which can be enacted through arm movements and the "close fist" gesture, respectively.

First empirical evaluation

The first empirical evaluation was carried out in June 2019. Participants were two skilled welders. One of them (P1, 32 y/o) used computers and other digital devices for more than 14 hours per week and was already familiar with gestural interfaces (Wii, X-Box), while the other (P2, 47 y/o) was less technology-savvy. Participants were asked to perform two tasks:

1. Task 1 (grid-like interface): select a series of tiles. Each user was assigned three sequences of four tiles each.

2. Task 2 (CAD-like interface): view a cube with differently-coloured facets; select the cube and rotate it, so as to make the blue facet visible.

We manually recorded the number of errors made by participants, as well as their comments, elicited through the thinking aloud protocol. After each task, participants were also invited to assess their user experience through a short survey, consisting of 4 questions which could be answered through a 1-4 scale6 .

Results. In the first task, both participants made two errors each, which consisted in selecting a certain tile twice. Such errors were probably due to the sensitivity setting of the armband, which might have caused small involuntary movements to be treated as selection gestures. Figure 2 reports our results for user experience evaluations. While P1, the participant having previous experience with gestural interaction, consistently expressed positive evaluations for both tasks, P2 was less positive, especially for the second task, where he had experienced most difficulties (see the free comments section for details). Free comments. P1 experienced no particular difficulties and only suggested that the application should cope with involuntary movements such as little tremors. On the whole, he looked forward to using gestural interaction to carry out his everyday activities at the workplace. P2 found the task of rotating objects especially difficult and pointed out some additional problems due to the specific implementation of the test interface. He was quite satisfied about both the comfort and the sensitivity settings of the armband, but he also observed that arms might get tired after prolonged use. On the whole, he was convinced that using gestural interaction might significantly improve his work experience.

Second empirical evaluation

Based on the results of the first evaluation, the project consortium worked on an improved version of the armband, focusing in particular on its sensitivity setting. Taking into account the delays caused by the COVID-19 pandemic, a second and final empirical evaluation was carried out in October 2020, where we used the same procedure, tasks and measures as in the previous evaluation. The participant (P3, 30 y/o) was an accomplished 3D designer, with some welding skills and similar demographics as P1 and P2 (notice that we were unable to recruit the very same participants due to shift management issues). In particular, similarly to P1, he was also an heavy user of digital devices and had had previous experience with gestural interaction (Kinect in particular).

Results. User experience evaluations are reported in Figure 2. Although he made no errors, P3 expressed a slightly less positive evaluation for the first task, in comparison with P1 and P2. In particular, he complained that the interaction was too slow. His evaluation, however, was notably higher for the second task.

Free comments. P3 observed that he had had some difficulties in pointing to the desired part of the interface and in having his gestures recognized, which was probably due both to his initial lack of familiarity with the armband and to persisting problems in sensitivity settings. Coherently with his low evaluation for the "speed" dimension in the first task, he pointed out that his experience would have been more satisfying, had the application been more responsive.

Conclusion

Our results offer some insight on the adoption of gestural interaction in smart factories. Most importantly, all the participants were favourable to our solution and willing to use it in their everyday work activities. User experience evaluations were generally positive, even if free comments highlighted the importance of fine tuning sensitivity settings: in the second evaluation, there were no errors due to involuntary movements, but the application appeared less responsive.

An open issue is the evaluation of the long-term user experience, which might be negatively affected by a natural decrease in motivation, as the initial enthusiasm starts to fade, as well as by the insurgence of arm fatigue, as hypothesized by one of the participants. While we are quite confident that the identified gestures avoid highly fatiguing arm positions which might cause the well-known "gorilla arm syndrome" [START_REF] Hansberger | Dispelling the gorilla arm syndrome: The viability of prolonged gesture interactions[END_REF], these aspects should be assessed empirically in future studies.

Finally, we must point out that one of the limitations of the here presented work is the small number of participants. On the other hand, however, we were able to involve actual representatives of our end users and to carry out the evaluations in a natural setting, i.e., on the shop floor, thus gaining good ecological validity.

Fig. 1 .

 1 Fig. 1. The grid-like (a) and CAD-like (b) interfaces.

Fig. 2 .

 2 Fig. 2. User experience assessments for the first (a) and second (b) task.

This work was supported by HOME (Hierarchical Open Manufacturing Europe) Regional Project (POR FESR, code 319-50).[START_REF] Riedl | Cyber-physical systems alter automation architectures[END_REF] https://www.home-opensystem.org/

https://www.galeasso.com/

While quantitative measures are not significant due to the small number of participants, they were deemed to be useful to better interpret and understand free comments.


José Luís Silva
email: jose.luis.silva@iscte-iul.pt

Regenerative Swiping: a Hybrid Vision for Improved Sustainability with "Free" Energy Harvesting

Keywords: Human-centered Computing, Human computer interaction, Interaction Techniques, Interaction Devices, Self-powered Users, "Free" Energy Harvesting, Interaction-powered Devices

In a world facing climate change emergency, energy harvesting must be improved. Future interactive devices (with new materials) and a move of energy harvesting from devices to users can trigger this improvement. This paper presents a vision on how (many) future interactive devices should be powered. Beyond the benefits of a self-powered and ultra-low power interactive devices vision, a complementary one with self-powered users and "free" energy harvesting is essential. For example, harvesting user's energy (e.g. heart rate pulsations) and/or enabling him/her to produce (e.g. kinetic/inertial energy harvesting) and store (e.g. wearables) energy for future interactions. Self-powered users can then perform interactions with devices (that require only power during interaction or extra power during interaction) powering them through direct contact interaction. This will allow the removal of built-in batteries on these devices and a global reduction of batteries. The proposed hybrid vision combines self-powered devices/users and "free" energy harvesting.

Introduction

The world desperately needs an energy paradigm shift for an improved sustainability. The energy used by interactive devices is electricity, considered clean energy if obtained exclusively from renewable sources. However, electricity is still highly reliant on burning fossil fuels [START_REF]Biomass processing technologies[END_REF] and/or nuclear activity. There is a pressing need to increase energy harvesting from renewable sources to improve sustainability.

In chemistry, the well-known law of conservation of mass1 discovered by Lavoisier states that "nothing is lost, nothing is created, everything is transformed". Similarly, the conservation of energy 2 law state that energy can neither be created nor destroyed; rather, it can only be transformed or transferred from one form to another. Therefore, we still have nowadays an immense waste of energy because most of them is not transferred where needed.

For instance, focusing on interactive devices, when users interact with them (e.g. pressing a button or swiping) there is "free" energy (from pressure and/or friction) that could be transferred back to the device.

The trend to an increasing number of electronic devices demands novel directions such as ultra-low power devices, interaction, and ambient energy harvesting but also novel low maintenance power schemes to decrease the burden of maintaining this huge number of devices.

The Vision

To achieve a successful paradigm shift, a hybrid approach should be followed. This means that efforts should be made in different directions. First, the number of batteries should be reduced to diminish the burden of maintaining a huge and increasing number of devices. This can be accomplished moving batteries from devices to users [START_REF] Zhang | TouchPower: Interaction-based power transfer for power-as-needed devices[END_REF] (see Fig. 1a). Users with (wearable) batteries enable additional energy harvesting that can be obtained from users and their activities (e.g. Energy Harvesting Footwear [START_REF] Akay | [END_REF]). Second, ultra-low power devices associated with ambient energy harvesting (e.g. some wristwatches are powered by kinetic energy) will tend to make devices self-sustainable. And, devices' functionalities or states with additional energy requirements could be powered with energy from self-powered users while interacting with the device. Finally, energy from the interaction should also be harvested. For example, when a user is interacting with a touchscreen the frictional energy resulting from the interaction should be transferred back to the device or to the user for future interactions (regenerative swiping).

Fig. 1a.

A user wears the user-end of TouchPower and uses a mouse and TV remote controller without batteries [START_REF] Zhang | TouchPower: Interaction-based power transfer for power-as-needed devices[END_REF]. Fig. 1b. Paper Generator harvests energy from rubbing and activates the e-paper display, revealing the word "hello" [START_REF] Karagozler | Paper generators: harvesting energy from touching, rubbing and sliding[END_REF].

According to Karagozler et al. [START_REF] Karagozler | Paper generators: harvesting energy from touching, rubbing and sliding[END_REF] these interactions are in a microjoule and milliwatt order of magnitude. There are few thousand of Joules in a smartphone battery however, as billions of users interact daily with touch screens (according to the study of the dScout company our screen phones are touched more than 2 thousand of time a day [START_REF] Nelson | We touch our phones 2,617 times a day, says study[END_REF]) this simple additional energy harvesting could globally save an immense number of Joules. Furthermore, recent advances in hypersurfaces 3 enable the conversion of any material of any shape into an intelligent surface that will recognize gestures. Each gesture on the surface creates a distinctive vibration pattern that is identified and converted into a command. This leads to believe that touch interactions will not decrease in near future. Therefore, friction/ pressure energy harvesting will have a huge impact.

The move of battery from devices to user (as proposed in the work of Zhang et al. [START_REF] Zhang | TouchPower: Interaction-based power transfer for power-as-needed devices[END_REF]) enables a lower maintenance power scheme and an increasing potential source of energy to harvest available. Reducing battery per user to one and being able to harvest energy from both users (their activities and interactions) and environment are advantages that became possible.

Self-powered users will foster power-as-needed devices (i.e., devices that only require power during interaction) and the removal of built-in batteries, but other devices will continue to be required. Therefore, self-powered and sustainable devices must also be fostered. However, some of these devices might require more energy while being interacted that might not be available. This falls in the benefits of this proposed hybrid vision (see Fig. 2) that merges both:

1. Self-powered devices -able to manage power requirements of devices while "idle" (nonbeing interacted); 2. Self-powered users -provide power to power-as-needed devices and extra power to other devices to deal with additional power requirements associated with interactions; 3. "Free" energy harvesting -provides a source of energy to self-powered users. Fig. 2. Proposed hybrid vision -regenerative swiping with self-powered users, "free" energy harvesting and interaction-powered Devices. Some authors [START_REF] Gorlatova | Movers and shakers: Kinetic energy harvesting for the internet of things[END_REF][START_REF] Gorlatova | Movers and shakers: Kinetic energy harvesting for the internet of things[END_REF][START_REF] De Winkel | Battery-free game boy[END_REF] have started to harvest "free" energy from common human activities. For example, De Winkel et al. [START_REF] De Winkel | Battery-free game boy[END_REF] recently developed an interactive battery-free device (i.e., a Battery-Free Game Boy) that harvests energy from gamer button presses and sunlight. This energy together with a move to lower-power devices pave the way to self-powered and sustainable users fueled with "free" energy from their common activities (e.g., walking, opening a door, breathing). Furthermore, interaction-powered interactive devices can complement the stated vision. For instance, a touchscreen built with energy harvesting materials (e.g., Paper Generator [START_REF] Karagozler | Paper generators: harvesting energy from touching, rubbing and sliding[END_REF] see Fig. 1b) and being able to harvest energy from interaction (e.g., scrolling) can increase the usage efficiency of "free" energy.

Discussion and Conclusions

The proposed vision presents potential benefits in terms of sustainability however, with potential implications in terms of User Experience. For instance, current ultralow energy devices provide less attractive interfaces [START_REF] Grosse-Puppendahl | Exploring the design space for energy-harvesting situated displays[END_REF][START_REF] Jensen | TransPrint: A Method for Fabricating Flexible Transparent Free-Form Displays[END_REF]. Furthermore, some devices might require more pressure to harvest more energy from the interaction and/or might be unable to provide a colorful interface. Another disadvantage of the proposed vision is the required use of a glove by self-powered users to transfer energy to devices. However, as stated by Zhang et al. [START_REF] Zhang | TouchPower: Interaction-based power transfer for power-as-needed devices[END_REF] some techniques such as skin electronics [START_REF] Weigel | Iskin: flexible, stretchable and visually customizable on-body touch sensors for mobile computing[END_REF][START_REF] Kawahara | Digital fabrication technologies for on-skin electronics[END_REF] and intrabody [START_REF] Partridge | Empirical measurements of intrabody communication performance under varied physical configurations[END_REF] can help get rid of the glove. Future research should focus on harvesting more "free" energy from user interaction. The effectiveness of the presented vision might imply changes in users' interactions and behavior. For example, more vigorous interactions might be needed to harvest enough energy or a need for a more active users' lifestyle. This might be mitigated by designing solutions focusing on users' needs instead of users' wants.

This paper presents a hybrid vision aiming to improve energy sustainability. The approach focusses on the intersection of three points:

1. powering users; 2. harvesting "free" energy; 3. low powered, self/interaction-powered and sustainable devices.

https://en.wikipedia.org/wiki/Conservation_of_mass (last accessed June 1st,

2021). 2 https://en.wikipedia.org/wiki/Conservation_of_energy (last accessed June 1st, 2021).

https://www.hypersurfaces.com (last accessed June 1st, 2021)

Acknowledgements

This work was supported from Fundação para a Ciência e a Tecnologia (FCT, Portugal), through LARSyS (project UIDB/50009/2020) and through projects UIDB/04466/2020 and UIDP/04466/2020.


Carmelo Ardito

Eugenio Di Sciascio

Fatemeh Nazary

Gianluca Sapienza
email: gianluca.sapienza@e-distribuzione.com

Keywords: Self-healing system, Interpretability, Fault prediction

This paper reports ongoing research for the definition of a data-driven self-healing system using machine learning (ML) techniques that can perform automatic and timely detection of fault types and locations. Specifically, the proposed method makes use of spectrogram-based CNN modeling of the 3-phase voltage signals. Furthermore, to keep human operators informed about why certain decisions were made, i.e., to facilitate the interpretability of the black-box ML model, we propose a novel explanation approach that highlight regions in the input spectrogram that contributed the most for the prediction task at hand (e.g., fault type or location) -or visual explanation.

ISCADA: Towards a Framework for Interpretable Fault Prediction in Smart Electrical Grids 1 Introduction and Context

Electrical grids are susceptible to a variety of electrical abnormalities, failures, and security threats, which if not tackled abruptly, in some cases they can leave a devastating impact on lives and the country's critical industries, leading to a national dilemma as the result of a cascading effect [START_REF] Burpee | Us-canada power system outage task force: final report on the implementation of task force recommendations[END_REF][START_REF] Majidi | A sparse-data-driven approach for fault location in transmission networks[END_REF]. Over the last decade, the problem of automatic fault detection has been studied from various viewing angles using a combination of tools and techniques from computer science, electrical engineering, statistics, and artificial intelligence (AI) and in particular using automatic data-driven machine learning (ML) algorithms [START_REF] Ardito | Revisiting security threat on smart grids: Accurate and interpretable fault location prediction and type classification[END_REF][START_REF] Raja | Multi-stage fault warning for large electric grids using anomaly detection and machine learning[END_REF]. From a software point of view, the use of SCADA (Supervisory Control And Data Acquisition) has dominated the electrical industry. SCADA is a control system architecture composed of computers, networked data communications, and graphical user interfaces (GUIs) for high-level process supervisory management tasks [START_REF] Boyer | SCADA: supervisory control and data acquisition[END_REF]. The users of the SCADA, however, can be diverse ranging from electrical engineers to expert data-scientists, and lower professional-level workers. 2 Overview of the self-healing management system of an electricity provider workflow

We model the self-healing management system of an electricity distributor as shown in Figure 1 according to hierarchical architecture, where the lowest layer is the closest level to the input data (i.e., raw electrical signals), and the highest level is semantic data that are supposed to be understandable to the human observer, thus bottom (data)→top (user). We describe the functionality of each layer below:

1. Data-collection level: The main role of the data collection layer is the acquisition of electrical, and logical data e.g., whether the fault breaker status is close (0) or open [START_REF] Ardito | Interacting with features: Visual inspection of black-box fault type classification systems in electrical grids[END_REF]. The data collection task by an electricity provider is mainly managed by the SCADA (short for Supervisory control and data acquisition) system, which is a standard approach to different types of input data. 2. Automation levels: This level is responsible for automating the decisionmaking by the system. Fault classification is the functionality that is inside the operative center, however, traditionally it relies on the knowledge of specialists that analyze fault to understand if it is necessary to inspect the feeder. Attack detection (remote control part of smart grid), is mainly managed by the telecommunication network. The workforce management unit relates to the crew who directly goes to check the problem. Service impact analysis deals with regulatory and quality of service. We have added an accurate data-driven location prediction system (where the latter is shown with red color) for isolating faulty zones or their precise location. We believe this can be a crucial part of modernized self-healing SGs since it expedites the dispatch of the crew members to the targeted (faulty) areas. 3. Information level: This level is responsible for reporting the monitoring status and fault decision outcomes to the user/operator. Practically, this is implemented in electricity providers using SCADA dashboard and data dashboard that can summarize data or give some detailed information to the user, depending on the type level of analysis he is interested in.

In the next section, we elaborate our contribution to the existing management self-healing system of the electricity provider.

Our proposed contribution

We propose ISCADA, interpretable SCADA that aims to augment fault prediction results in SGs with information that facilitates interpreting them to both experts and non-experts operators or users. To this end, we have built a deep-learning-based fault prediction system3 that can provide accurate and interpretable fault type and location with relatively high precision. ISCADA extends the traditional SCADA by adding other components in the specific layers, as shown with red colors in Figure 1. They include a fault location detection (automation layer) and interpretable fault prediction visualization (data layer). We now describe the main idea and the functioning mechanism behind the proposed work.

As shown in Figure 2, starting from the raw 3-phase voltage signal collected from a hypothetical faulty area (zone A), we transform the data into a spectrogram representation, which will serve as a visual time-frequency representation of the original signal [START_REF] Birajdar | Speech and music classification using spectrogram based statistical descriptors and extreme learning machine[END_REF]. This representation is not only an efficient representation of the signal as it contains both time and frequency representation, but also a suitable technique for visual inspection of the fault prediction results according to time and frequency changes in the signal. For the fault prediction system, we build a data-driven classification system utilizing a powerful deep convolutional neural network (CNN). Furthermore, an approach based on gradient-weighted class activation mapping (Grad-CAM) [START_REF] Selvaraju | Gradcam: Visual explanations from deep networks via gradient-based localization[END_REF] technique is utilized to highlight important regions in the spectrogram that can explain fault type and location, or visual explanation. The result of the explanation is a heatmap visualizing where on the spectrogram images the network was paying attention to. These can provide insightful visual cues to the human operator about the characteristics of the faults and can keep them in the loop.

A hypothetical extension of these results would involve how to turn such interpretable outcomes [START_REF] Ardito | Interacting with features: Visual inspection of black-box fault type classification systems in electrical grids[END_REF][START_REF] Ardito | Revisiting security threat on smart grids: Accurate and interpretable fault location prediction and type classification[END_REF] into an explanation for non-expert users. In our

Data transformation

Interpretable fault type and location Prediction (Multimodal explanation) Fig. 2. Illustration of the principles behind the proposed intereptable fault prediction system scenario, we can use the result of fault prediction as an input to a knowledge graph or other semantic repository of knowledge that can be also found freely on the web. An example of such freely available resource we can name of DBPedia, as shown in Figure 2. The results are presented to the user in terms of multimodal explanation using natural language (results returned from the KG) and visual outcome (the result of the automated fault prediction system shown with the red zone over the spectrograms).

Conclusion

In this paper, we have reported the results of an ongoing research work aiming at the definition of an interpretable and accurate data-driven self-healing system using machine learning (ML). The end goal on the one hand is to perform automatic and timely detection of fault types and locations; on the other hand, we aim to make the prediction results interpretable to the user. We discussed briefly the main insights behind our proposed CNN-based modelling of fault spectrograms and the Grad-CAM technique, and how the latter can be used to make the decisions made by the system more transparent.

Fig. 1 .

 1 Fig. 1. The 3-layered management architecture of an electricity provider network. The red modules show our contribution.

Fault

  There is a short circuit fault of type ABC on Zone A. It is a low-impedance fault and can cause damage due to excessive current flow. In the Grad-CAM, we can see this fault and location activates the middle-range frequencies and lower time instances.

Distribution Network Communication Data collection Information Layer Automation Layer Fault detection and type Fault location Map Bari Scada dashboard Service Impact Analysis Event List User notification Work-force managemen Attack Detection SCADA-based data collection voltage and current Voltage and current breaker Data dashboard Interpretable prediction

This work is currently under review at the IEEE Transaction of smart grids (SG)

Acknowledgments

This work has been partially funded by e-distribuzione S.p.A company, Italy, through a PhD scholarship granted to Fatemeh Nazary.