Hassan Rabah

Olivier Sentieys

Thibaut Pascal

Pierre Le Coz

Nassim Abderrahmane

Yasmina Zaky

Lyes Khacef

Marino Adrien Russo

Bettys Jordan

Alexis Arcaya Diaz

Lila Arcaya

Keywords: Objets connectés portés, modélisation au niveau système, consommation énergétique, autonomie, performance, qualité du service, lunettes connectées, microcontrôleur, capteurs, état de charge des batteries Wearables, system-level modeling, energy consumption, autonomy, performance, quality of service, smart connected glasses, microcontroller, sensors

A growing number of physical objects are being connected to the Internet at an unprecedented rate realizing the idea of the Internet of Things (IoT). Wearables are a class of IoT devices worn on the body that can detect and process physiological signals. A wearable can then give bio-feedback to the wearer or send information to the cloud for further analysis. In this context, the Ellcie-Healthy start-up is developing an intelligent and connected eyewear solution with the aim at improving the user's quality of life and to prevent risks for their health and safety. To do so, several sensors are integrated into the wearable eyeglass to collect and send physical, physiological and environmental information towards a gateway, typically a smartphone. Designing wearable devices deals with numerous challenges such as the energy consumption, the quality of service (QoS) and the scalability, to cite only a few.

Several design constraints, such as the autonomy and QoS, must be addressed early in the design flow to ensure the wearable device will meet the user requirements.

In this PhD thesis, we propose a simulation flow based on a system-level modeling approach to help designers making the best choices of hardware/software settings early in the design phase. To do so, we have first developed analytical power models for the Ellcie Healthy smart glasses. These models, built from data measured on the real platform, allow estimating the average power consumption of the system, and so the battery lifetime, for different application scenarios. Then, performance constraints have also been considered in our system-level approach. The drowsiness detection application, developed at Ellcie Healthy, has been profiled to identify the hardware and software configurations impacting the most the energy consumption and the application QoS, i.e. the detection performance. A solution based on a Convolutional Neural Network (CNN) has been proposed to further improve the drowsiness detection QoS. We have also demonstrated that it is possible to execute this CNN at the edge (i.e. on a micro-controller). A battery testbench has also been developed to evaluate battery safety conditions as well as the aging phenomenon.

Using this experimental platform, a significant amount of information on the battery charge/discharge process has been collected. A data-driven battery model has been proposed and validated. All these models have been integrated into a simulation framework which allows modeling and simulating existing or future versions of the smart connected glasses. Using this framework, designers can rapidly investigate trade-offs between hardware and/or software settings, as well as study system autonomy and application QoS. In addition, this simulation flow can be used to evaluate power management strategies from a high-level point of view.

v Résumé De nos jours, de plus en plus d'objets se connectent à Internet formant l'Internet des Objets (IoT). Les wearables, une classe de dispositifs constituant l'IoT, sont des équipements portés par un utilisateur qui permettent de détecter et traiter des signaux physiologiques. Le porteur de l'équipements peut ainsi recevoir directement des informations qualitatives sur son état de santé par exemple. Les données collectées pouvant par ailleurs être transmises sur le cloud pour une analyse plus approfondie. C'est dans ce contexte que la start-up Ellcie-Healthy développe des lunettes connectées intelligentes. L'objectif est d'améliorer la qualité de vie des utilisateurs et d'éviter certains risques pour leur santé et leur sécurité (e.g. la détection des chutes). Pour cela, de nombreux capteurs ont été intégrés dans ces lunettes connectées afin de collecter et envoyer des données physiques, physiologiques et environnementales vers une passerelle, typiquement un smartphone. Or, concevoir des dispositifs wearables comporte de nombreux challenges tels que la consommation énergétique, la qualité de service (QoS) ou leur miniaturisation. Ces contraintes de conception, telles que l'autonomie ou la QoS, doivent être adressées tôt dans la phase de conception afin de garantir un respect des contraintes et des besoins utilisateurs. Dans cette thèse, nous proposons ainsi une méthodologie basée sur une approche de modélisation de haut niveau. L'objectif est d'aider les concepteurs, tôt dans le flot de conception, à effectuer les meilleurs choix des configurations matérielles et logicielles. Dans ce but, nous avons tout d'abord proposé des modèles analytiques de consommation de puissance pour les différents modules composant les lunettes connectées. Ces modèles, élaborés à partir de données mesurées directement sur des prototypes matériels, permettent pour différents scénarios applicatifs d'estimer la puissance consommée moyenne du système, et donc son autonomie.

Les contraintes de performances (ou QoS) ont également été considérées dans notre approche de niveau système. L'application de détection de l'endormissement au volant, développée par Ellcie-Healthy, a été étudiée afin d'identifier les configurations matérielles et logicielles impactant le plus la consommation énergétique et la performance (i.e. la QoS liée à la détection des clignements des yeux). Une solution, basée sur un algorithme de réseaux de neurones convolutifs (CNN), a été proposée afin d'améliorer les performances de l'application. Bien que plus contraignantes en termes de temps d'exécution et d'empreinte mémoire, nous avons montré qu'il est possible d'exécuter ce type d'algorithme CNN at the edge (i.e. sur un microcontrôleur). Durant cette thèse, un banc de test a également été développé afin d'évaluer les conditions de sécurité et le phénomène de vieillissement des batteries. L'utilisation de cette plateforme expérimentale a permis de collecter de nombreuses informations liées au processus de charge/décharge de diverses batteries utilisées par vi l'entreprise. Différents modèles de batterie data-driven ont ainsi été proposés et validés pour différents cas d'usage. L'ensemble de ces modèles ont été intégrés dans un environnement de simulation, permettant ainsi de modéliser et simuler les versions actuelles ou futures des lunettes connectées intelligentes. Les concepteurs peuvent alors rapidement évaluer l'impact de différentes configurations matérielles et logicielles ou de politiques de gestion de l'énergie sur l'autonomie et les performances du système. Cette méthodologie générique de modélisation et simulation permet également de considérer d'autres types de dispositifs wearables.

List of Tables

2.1 Karolinska Sleeping Scale (KSS levels [START_REF] Kaida | Validation of the karolinska sleepiness scale against performance and eeg variables[END_REF]) vs Drowsiness Levels of Smart Connected Glasses (SCG levels) . . . . . . . . . . . . . . . . . . .

Chapter 1

General introduction 1.1 Context and problem statement

Over the last few years, more and more applications based on latest technological advances have been continuously added to the world of connected objects, better known nowadays as the Internet of Things (see Fig. 1.1). The Internet has become a daily life basic need for the whole world, evolving year after year to deliver better bandwidth, better performance and improved user experience. This evolution of the Internet has given space to the development of the Internet of Things or IoT, which is considered as an ecosystem of devices and digital machines spread around the world that exchange data between them over the Internet. As long as an object is connected and sharing data through the Internet, it is part of the Internet of Things. Source: [1] Since the beginning of the 21st century, forecasts of worldwide technology statistics providers have been focused on the growth of the Internet of Things market in terms of the number of connected objects spread around the world, as well as the worldwide investment. Certainly, the number of IoT-based applications has strongly grown year over year in the last two decades. In 2019, forecasts given by Gartner, Inc. [2] pointed towards an IoT market growth of 21% in 2020, compared Source: [4] and [5] to the number of active connected objects spread throughout the world in 2019. Furthermore, the expected revenue from these connected objects would be $389 billion globally. In June 2020, the International Data Corporation (IDC) [3] published a new update of the Worldwide Internet of Things Spending Guide showing that the 2020 IoT spending growth is at 8.2%, even though a growth of 14.9% had been expected from previous forecasts. Worldwide IoT spending has been significantly impacted by the economic effects of the COVID-19 pandemic in 2020. Nevertheless, IDC forecasts that IoT spending will achieve a compound annual growth rate (CAGR) of 11.3% during the next four years. IoT Analytics [4] and Strategy Analytics [5], two providers of IoT market insights, have estimated in 2018 and 2019 respectively, the number of active IoT devices worldwide already deployed and the number of devices to be deployed in the next five years. The numbers obtained by these two sources are a bit different, but the growth pace is quite similar. A synthesis of these two sources is depicted in Fig. 1.2, where around 36 billion objects will be constituting the Internet of Things in 2025.

.3 -IoT architecture Source: [6] Whatever the source, it is admitted that the number of connected objects and the investments in the IoT market will not cease to increase in the coming years. This continuous IoT-based applications increase concerns extremely varied fields such as medical, personal assistance (e-health), agriculture, environment, industries, transportation, smart home or smart city, to cite only a few of them. The IoT architecture (independent on the field), can be represented by the example of Fig. 1.3, where the LORA RF protocol is used for wireless communication between end devices (i.e.

sensor nodes) and gateways (GW). As can be seen in this figure, there are connected objects in charge of collecting data from their environment (according to their field) through sensors. Data exchange is then done using a gateway interfacing these objects with the cloud through the Internet. This access to the cloud allows the execution of very powerful data processing algorithms on the application servers, capable of processing massive amount of data. It is worth noticing that between the gateways and the application servers, one or several network servers must be connected to a complete IoT architecture, thus allowing to interface the IoT domain with the Internet. Embedded systems can take multiple different forms such as electric vehicles, satellites, robotic systems for robotically-assisted surgery, real-time cameras, smartphones, environmental wireless sensors or wearables.

The deployment of these embedded systems for IoT-based applications deals with numerous challenges [7] such as unique objects identifiers, the massive amount of data collected and aggregated via connected objects (i.e. Big Data), the energy consumption, the heterogeneity needed to address one application, the data transmission media, the security, the quality of service (QoS), the human interaction and the scalability. To deal with these challenges, the trend is to bring intelligence as close as possible to the sensors (i.e. Edge Computing). Edge intelligence or Edge AI has indeed many advantages compared to a cloud-based approach. Some of them are: limiting bandwidth, better latency, energy consumption optimization and better security due to the reduction of wireless data transmission. However, this edge AI trend poses many challenges related to the embedded system constraints, mainly the execution time, the energy consumption and the memory footprint.

These embedded system constraints have a direct impact on the QoS of an IoT-based device. Defining the QoS is not an easy task since it is highly dependent on both the kind of device and the application. To define QoS requirements, we need to know the main characteristics of the device to be designed and the application to be addressed. Let a wireless sensor be the IoT-based device, and the temperature sensing be the application. Constraints in terms of system autonomy, temperature measurements accuracy, data sensing latency, data transmission packet error rate and data transmission latency, can define the QoS for this application. It is worth noting that energy efficient methods intending to increase the battery lifetime, so the autonomy related QoS, are becoming essential.

Technology advances in terms of miniaturization of microprocessors, have allowed an increase in computing power and memory capacity, while keeping the same area (from some MHz to several GHz in the last 25 years). Thanks to these enhancements, the QoS of embedded applications has been considerably increased in last years, thus improving the user experience. This evolution has also accelerated the integration of Machine Learning (ML) and Deep Learning (DL) high-performance algorithms. This is specially true for powerful embedded systems such as electric vehicles, aircraft, tablets and smartphones, but also for some wireless sensors.

Wearable devices belong to a class of embedded systems where constraints are even more restricted due to the size and weight of the device that must be respected. Despite autonomy and computational resources have been improved in recent years on wearable devices, it has been effective on a smaller scale. For example, a common smartwatch is in most cases based on a low-power microcontroller composed of a single microprocessor core and scarce memory capacity for the execution of wearable applications. This kind of microcontrollers has nevertheless seen a progressive enhancement year after year in terms of memory capacity and clock speed, allowing the execution of more complex but still lightweight algorithms.

Wearable applications represent an emerging field in the IoT market. Manasi Mishra, in the article Rise of Wearables and future of Wearable technology [11], affirms that "In today's busy world, people want to track their daily activities to improve their lives". Additionally, the author expresses that "From physical fitness to fashion Industry, the use of wearable devices and wearable apps is increasing day by day leveraging wearable applications development services". Some examples of the most common devices in the today's wearable market, are presented in Fig. 1.5.

FIGURE 1.5 -Diversity of wearable devices in the IoT

Source: [11] It is in this context that the French start-up Ellcie-Healthy, founded in May 2016, is developing an intelligent and connected eyewear solution (see Fig. 1.6). The objectives are to improve users quality of life and to prevent risks for their health and safety. To do so, Ellcie-Healthy proposes to integrate several sensors into this wearable eyeglass frame to collect and send physical, physiological and environmental information towards a gateway, typically a smartphone. Sensor data, processed using embedded algorithms, are used to provide information and/or predictions related to the health and/or safety of the user (but also to the close relatives and/or medical staff if necessary). Ellcie-Healthy is thus betting on the fast-growing e-health field through an accessory fully adopted by the public (eyeglass frames). About 60% of the world's population wears glasses with corrective lenses (representing more than 4 billion people). Moreover, a large part of the population wears glasses during outdoor activities [12]. The objective of the company is then to take a part of this market by offering smart connected glasses to the public at large.

As for all the embedded systems, different challenges are faced during the development of the smart glasses. Figure 1.7 shows the main phases of a product design flow, going from the requirement phase where technical specifications are given, passing through the design, development and validation phases, and finishing in the production phase where the wearable is ready to be deployed. A successful product deployment requires indeed detailed analysis of embedded constraints specially at the design level. The size of the printed circuit boards (PCB) and integrated circuits (IC), the available energy, the computing power and the memory, are the main technical constraints to be considered in an embedded system design flow. Designing embedded systems is then a key issue in the field of connected objects applications.

Studying these aspects early in the design flow can indeed allow the wearable to offer good autonomy while delivering the desired QoS, thus covering the user experience expectations. In this context, an undeniable need for studying, during the design of the smart connected glasses, the energy consumption dependencies as well as the QoS constraints, has become the main concern of this thesis. For example, the first prototype of Ellcie-Healthy smart glasses presented in Fig. 1.8

showed that a major portion of the final product cost is due to batteries. Due to the constraints of miniaturization and cost, the energy storage capacity that can be embedded in the glasses frames is very limited. Yet, Ellcie-Healthy glasses must guarantee a minimum autonomy and QoS with the available amount of energy (e.g. the battery capacity of the current smart glasses industrialized version is 95 mAh), which is both a technical and a technological challenge. The choice of embedded available energy, was indeed made according to the manufacturer's power consumption data for the various sensors, the System-on-Chip (SoC) integrating an ARM Cortex-M4

processor and a Bluetooth Low Energy (BLE) transceiver. The sizing of the energy storage system was therefore the result of a static analysis based on the basic functional sensors characteristics (e.g. sampling rate, low-power modes usage), microcontroller characteristics (e.g. CPU frequency) as well as BLE device characteristics (e.g. theoretical estimation of the amount of data to be exchanged at a fixed transmission power). To reduce the energy consumption (thus improving the autonomy) optimizations must be performed. In this thesis, the QoS requirements of the smart connected glasses are defined according to the detection of events such as eye blinks (for the driver drowsiness detection application) or falls (for the fall detection application). As it will be seen later in this document, accuracy, sensitivity and speci f icity metrics will be used to evaluate event detection. Moreover, the feedback from the user experience in terms of autonomy, is also considered as a QoS requirement for this kind of devices.

Energy consumption optimization of connected objects has been widely studied in recent years within the EDGE (Edge computing and DiGital systEms) team of the LEAT laboratory. The research, usually carried out in the field of wireless sensor networks, has mainly allowed to propose system-level modeling/simulation approaches [13,14], new power management algorithms [15,16,17,18,19,20,21] and low-power network protocols [22,23], to mention a few.

The expected benefits of this thesis work mainly concern the smart connected glasses energy consumption modeling. Additionally, it is also expected that this work will allow the company to better estimate, control and so optimize the overall energy consumption, and so the system autonomy, according to the application requirements. Finally, the objective is also to propose relevant solutions regarding the design of incoming Ellcie-Healthy products. In consequence, the scientific challenge of this thesis is to propose an approach allowing to consider early in the design flow the issues related to power consumption, but also to model and quantify the QoS. This approach will offer to designers the resources to evaluate different alternatives (configuration of system parameters, power management strategies, battery sizing, etc.)

to control and optimize the energy consumption, while guaranteeing a satisfactory QoS.

Objectives

Considering the context and problem statement previously introduced, the general objective of this thesis is to propose a system-level modeling methodology to rapidly estimate and then optimize the energy consumption on smart connected glasses, while considering the QoS (i.e. performance). To do so, this problem has been broken down into six sub-problems:

-Perform a preliminary characterization in terms of performance and energy consumption on a prototype of connected glasses developed by the Ellcie-Healthy company.

-Identify the set of parameters having the biggest impact on the performance and the overall energy consumption.

-Propose a high-level model of functional (e.g. application tasks) and nonfunctional (e.g. energy consumption) embedded system elements to provide accurate energy consumption and system autonomy estimations.

-Propose a high-level model for the battery SOC prediction, while considering the impact of the aging phenomenon. Our aim is also to consider battery sizing for next connected glasses generations using the proposed methodology.

The goal is to help designers to determine the suitable battery sizing, i.e. the battery capacity that maximizes the system autonomy while guaranteeing a satisfactory QoS.

-Integrate the different models into an overall system-level modeling methodology.

-Propose and evaluate power management policies using the proposed methodology, considering autonomy and QoS constraints related to different application cases.

Contributions and thesis outline

The present chapter has served as general introduction and context presentation of this research work. In this section, we present the different contributions to achieve the objective of proposing a system-level modeling methodology for the energy consumption study on smart connected glasses, considering QoS constraints. In addition, the thesis outline is detailed during the presentation of every contribution.

The general contribution of this thesis is a system-level modeling methodology described in Fig. 1.9. This methodology allows running scenarios to rapidly obtain diverse insights needed for optimizing energy consumption on the smart connected glasses. This methodology will be presented in chapter 2. First of all, a detailed presentation of the smart connected glasses, their applications as well as their positioning according to the state-of-the-art for similar wearable applications, will be addressed in chapter 2. Then, constraints in terms of autonomy and performance (i.e. QoS), as well as an introduction of existing high-level frameworks for embedded systems hardware design, will be presented. Finally, the aspects of the proposed methodology intended to facilitate the modeling, simulation and design of smart connected glasses, will be introduced. Hereafter, an enumerated description of all the contributions related to this methodology, is presented.

1.

A system-level power consumption modeling approach based on analytical power models of different system activities (i.e. application tasks), is proposed. Analytical power models are based on the electrical characteristics provided by manufacturers of different modules involved in the execution of every application task. This approach allows us to estimate an accurate battery lifetime by scenario, while remaining at a high level of abstraction. This power modeling approach will be presented in chapter 3. In this chapter, every hardware module is detailed according to the electrical characteristics provided by manufacturers. Then, a high-level description of main functional characteristics by module will be given. Furthermore, this analytical modeling approach will be compared to state-of-the-art techniques for modeling the power consumption from a high-level point of view. To evaluate the precision of our models, average power consumption and autonomy estimations will be then compared to real measurements. This contribution has been valued through the publication of an article at an IEEE international conference in the first quarter of 2019 [24].

2. Another contribution of this thesis consists in integrating the performance constraints into the system-level power consumption modeling approach, to study the hardware settings impacting the most the smart glasses energy consumption and the application's performance. This contribution will be presented in chapter 4. First, an in-depth study about the performance of the drowsiness detection application is proposed. Then, different performance metrics (accuracy, sensitivity and speci f icity) are analyzed to identify which one offers the best performance description. Hardware and software dependencies are evaluated afterwards to measure the impact of system settings on both the energy and the performance. In addition, a comparison between this performance evaluation and those presented in the state-of-the-art for wearable detection systems, will be given. The results of the system settings assessment will be finally presented. This contribution has been valued through the publication of an article at an IEEE international conference in the first quarter of 2020 [25]. the feasibility of implementing this deep learning approach at the edge. This contribution will be detailed in chapter 4 and has been published in an IEEE international journal [26].

4.

A battery testbench has been developed to evaluate both battery safety conditions and the aging phenomenon. This battery testbench allows voltage, current, number of cycles and SOC to be collected during the charge/discharge process. The design and development of this battery testbench will be detailed in chapter 5.

5.

A data-driven battery model using the battery testbench to perform automatic battery SOC predictions during the execution of smart glasses scenarios, will also be detailed in chapter 5. First, the creation of a dataset for implementing the data-driven battery model will be explained. Then, a state-of-theart battery modeling study will be presented. The data-driven battery model for battery SOC prediction will be finally presented.

6. As a last contribution, a framework integrating all the blocks presented in Fig. 1.9 has been developed, to model and simulate current or new versions of smart connected glasses. Chapter 6 will be dedicated to the presentation of this framework. First, the software architecture and implementation choices of this framework will be presented. Then, several scenarios are simulated using this framework to get estimations based on the modeling approach depicted in previous chapters. Finally, these estimations are compared to real measurements obtained through the execution of the same scenarios using the smart glasses. Finally, in chapter 7, the conclusions and perspectives of this research work will be presented.

Chapter 2

Smart Connected Glasses: Applications and Design challenges

Introduction

Wearable devices running real-time sensor-based applications for human health, wellness and safety monitoring, are increasingly useful because of the improvement in their reliability. For example, fitness trackers through smartwatches is becoming a very competitive market as expressed by Cat Ellis in [1]. Technology advances in this kind of connected objects evolve day after day, offering a large range of choices that help users to better understand their health. In the last few years, researchers have proposed a large number of wearable applications leveraging embedded detection systems such as fall detection [2,3], gait phase detection [4], fatigue detection [? ],

drowsiness detection [5] or epileptic seizure detection [6,7], to name a few. In that context, the Ellcie-Healthy startup develops smart connected glasses. This multiplepurpose wearable device has been designed for e-health and road safety applications such as driver drowsiness detection, fall detection for elderly people or human activity recognition to prevent a fall. The Ellcie-Healthy smart connected glasses [8] shown in Fig. 2.1 contain infrared (IR) proximity sensors embedded inside the rims to measure eyes movements 1 . Several other sensors (e.g. barometer, temperature)

as well as a 32-bit microcontroller (MCU) and an inertial measurement unit (I MU)

are placed in the hinges of the frame. Additionally, a BLE chip is also integrated into the glasses to ensure a wireless communication with a gateway (typically a smartphone). Finally, a 95 mAh Lithium Polymer (LiPo) battery placed on the left hinge of the frame allows powering the whole system. Embedded algorithms can thus be 1. Considering the risks of using IR light toward the eyes, an important assessment to quantify the degree of infrared hazard has been performed on the connected glasses, thus following EN 62471:2008 [9] and IEC/TR 62778:2014 [10] standard documents. The results of this assessment demonstrated that there are no risks with infrared radiation coming from the connected glasses.

Chapter 2. Smart Connected Glasses: Applications and Design challenges executed directly on the glasses to provide healthy and/or security information to users and generate alerts when a risk situation (e.g. driver drowsiness) is detected. In section 2.2 the two main applications addressed with the smart connected glasses are presented. For every application, a detailed state-of-the-art is presented. Then, the positioning of the smart connected glasses compared to the state-of-the-art is discussed. In section 2.3 the main challenges in terms of autonomy (i.e. battery lifetime constraints) and QoS (i.e. performance constraints) for designing wearable devices leveraging a single MCU are discussed. In section 2.4 the proposed system-level modeling methodology intended to execute smart glasses scenarios is presented in detail. Finally in section 2.5, the conclusions of this chapter are presented.

Smart Connected Glasses: Applications

As mentioned in chapter 1, about 60 % of the world's population wears glasses with corrective lenses (representing more than 4 billion people) and 100 % could wear them during outdoor activities (more than 7 billion people). Based on this fact, smart connected glasses represent a suitable, non-intrusive and efficient way to deal with risk detection applications like for example, driver drowsiness detection and fall detection (Figure 2.2). Ellcie-Healthy is also working on the development of activity recognition algorithms intended to identify behavioral patterns of users, and prevent falls. To do so, activities of daily living (ADL) such as walking, sitting down, and standing up are analyzed with the smart connected glasses, as shown in [11].

Driver drowsiness detection

Recent road safety statistics [12,13] have identified the driver fatigue and drowsy driving as one of the major causes of fatal road crashes. Automatic driver drowsiness detection systems has begun to be recognized as a promising solution, thus receiving growing attention from industry and academics. Those systems are useful for warning or assisting drivers and alerting them about their drowsiness state. To evaluate the level of driver drowsiness, algorithms are mostly based on data collected either from vehicle-based measures, behavioral measures or physiological measures In many cases, existing solutions focus on camera-based systems [15,16,17,18,19], which have the advantage of non-intrusiveness. However, these solutions require a lot of power and expensive equipment compared to sensor-based systems. Wearable brain computer interfaces (BCI) have also been used to classify signs of drowsiness, through either electrooculography (EOG) or electroencephalography (EEG) [5,20,21]. EEG signals are known to be reliable indicators of fatigue and drowsiness. BCI sensor-based devices offer both high detection accuracy and energy efficiency. Nevertheless, electrodes are not suitable for everyday use, and therefore represent a limitation for portability. The authors in [22,23], have proposed similar and attractive driver drowsiness detection solutions based on IR sensors. An IR sensorbased solution uses photosensor oculography [24], an eye-tracking technique based on photosensors to measure the amount of reflected light from eyes. This technique can be implemented using proximity sensors on wearable devices through eyelids reflected infrared light measurements (see Figure 2.3). IR sensor-based solutions offer an ultra low-power sensing architecture, low hardware cost, as well as real portability and non-intrusiveness. Additionally, such devices offer a high detection accuracy while respecting real-time constraints. In [25,26] the authors demonstrated the feasibility of driver drowsiness detection systems based on information coming from an I MU. Results in terms of detection accuracy are interesting and take part from now on, in the wearable driver drowsiness detection state-of-the-art.

Driver drowsiness detection: State-of-the-art

In [15], You et al. proposed a camera-based system that computes the Eyes Aspect Ratio (EAR) parameter every frame at over 20 fps, to estimate whether drivers are in a drowsy condition. In this work, the face of drivers is first detected every frame through a Deep Cascaded Convolutional Neural Network (DCCNN). Then, a support vector machine (SV M) classifier is trained with two different EAR states: closed In [19], Sharan et al. developed a driver fatigue system based on eye states. A CNN is also used for binary classification (i.e. open or closed eye). As in [15], the authors evaluate driver drowsiness through PERCLOS. After a training phase, the system is able to distinguish open and closed eyes as drowsiness indicator. In this work, a Raspberry Pi board is used to implement the detection system and to raise alerts.

It is worth mentioning that for these camera-based solutions, training and inference phases are carried out either in simulation platforms or on a PC. In all those cases, the drowsiness detection systems are not executed in embedded platforms or wearables.

In [5], Kartsch et al. presented an alternative approach based on both physiological (i.e. spectral power of alpha waves) and behavioral (i.e. blink duration) features from EEG using a BCI. This work describes a MCU-based wearable system capable of detecting three different drowsiness levels through EEG signals, and raising alerts when some levels are reached. To offer an user interface, this wearable communicates with a mobile application running on a smartphone through a Bluetooth interface. Accuracy (85%) and battery lifetime (7.1 h for a battery capacity of 200 mAh) results demonstrate that a MCU-based wearable BCI using EEG signals is suitable for driver drowsiness detection. Similarly, Li et al. proposed in [20], a wireless wearable MCU-based BCI system for driver drowsiness that combines EEG and gyroscope signals. The proposed system is composed of two main parts: a wearable BCI system and a wearable smartwatch. The communication between them is established via a BLE interface. The BCI system is in charge of sensing and extracting EEG and gyroscope features. These information are then transmitted toward the MCU-based smartwatch for processing and classification of drowsiness states. The classification on the smartwatch is done by a SV M model previously trained. These BCI-based approaches have promising results for a wearable drowsiness detection application. However, its limitation lies in wearing the BCI interface, which can be considered as not user-friendly for daily use.

In [21] In [22], Chang et al. proposed a smart glasses prototype leveraging an IR light sensor for driver drowsiness and fatigue detection through eyelid closure (i.e. blink duration). The definition of blink duration thresholds allows the system to determine whether drivers are in awake, drowsiness or fatigue state. In this prototype, a MCU is used to execute algorithms, thus offering lightweight wearable smart glasses.

In [23], He et al. demonstrated the feasibility of using the Google Glass for driver drowsiness detection. To do so, IR proximity sensors embedded on the glasses are used to compute eye blink frequency. A threshold-based blink detection algorithm has been implemented on the embedded system. Driving performance and eye blink frequency were evaluated together during simulations based on the detection of two states: alertness and drowsiness when driving. Results showed that drowsy drivers increased blink frequency and lane deviation, while producing longer braking response, compared to alert drivers. Additionally, IR proximity sensors have demonstrated to be suitable for wearable devices because of their low cost in terms of energy consumption and signal processing.

In [25], Warwick et al. used the BioHarness 3 wearable biosensor device produced by Zephyr Technology to collect drivers' physiological data. These data have been analyzed to find key parameters, useful for designing a drowsiness detection algorithm and to alert drivers through a mobile application. This wearable biosensor outputs several individual physiological data such as 3-axis acceleration, electrocardiogram (ECG), heart rate, breathing rate or posture. The biosensor, placed close to driver's chest, collects physiological parameters and transmits them via wireless to a smartphone that finally executes a drowsiness detection mobile application. Heart and breathing rates seem to be good indicators of drowsiness. However, this study lacks for accuracy and battery lifetime information to evaluate its pertinence.

In [26], Lee and Chung proposed a real-time and standalone wearable system for driver drowsiness detection using commercial smartwatches. Detection is performed by computing motion data from an I MU. In this system, drowsiness state is mainly determined according to the driver behavior that is extracted from the motion data collected from accelerometer and gyroscope sensors. Several extracted features are used as input to a SV M classifier.

For all the previously mentioned works, drowsiness detection accuracy has been evaluated through comparisons with sleepiness scales (also known as sleepiness surveys or questionnaires) intended to assign levels of drowsiness according to the feeling of users tiredness. Drowsiness level predictions from automatic driver drowsiness detection systems are then compared to one of those sleepiness scales in order to estimate a percentage of accuracy. A typical and well-known sleepiness scale is the Karolinska Sleepiness Scale (KSS) [START_REF] Kaida | Validation of the karolinska sleepiness scale against performance and eeg variables[END_REF], presented in Table 2.1. As can be seen,

Driver drowsiness detection using the connected glasses

In the Ellcie-Healthy smart connected glasses, the driver drowsiness detection algorithm is based on blinking signal time series coming from the infrared proximity sensors (IR sensors). As shown in Fig. 2.3, a blinking signal is obtained by illuminating both eyes using infrared light-emitting diodes (IR LED) and measuring the reflected light using IR photodetectors.

The raw data coming from both IR photodetectors are then merged to obtain a single blinking signal shown in Fig. 2.4. The drowsiness evaluation is performed through three successive phases: 1) eye blink detection, 2) metric computation, and 3) drowsiness index estimation. Besides blinks, several other events can occur. As can be seen in Fig. 2.4, a driver can also perform look events (e.g. look down) during a driving period. The challenge is therefore to detect as many blinks as possible while ignoring other events. As soon as some events are detected as blinks, three metrics are computed:

the average amplitude, the duration and the rate of blinks, over a period of time (e.g. 60 s). Drowsiness is then estimated using a five-point composite scale based on the abovementioned metrics, reflecting the variability in blinking duration. The KSS sleepiness scale presented in Table 2.1 has been adopted for this purpose. As can be seen in this table, and for the sake of simplicity, a translation from the original nine-level scale to a five-level scale has been done. A continuous analysis of blinking variability allows measuring the progressive evolution of the driver drowsiness condition. The performance of the drowsiness prediction strongly depends on the aforementioned metrics and therefore on the blink detection accuracy. In consequence, the performance needs to be assessed from the eye blink detection phase.

This performance analysis will be presented in details in chapter 4.

The Ellcie-Healthy smart glasses also propose a fall detection application. Automatic fall detection systems represent nowadays a topic of interest for industrial and research communities.

Fall detection

About one third of elderly people (i.e. over 65 years of age) suffer a fall each year, in most cases being alone at home [START_REF] Kangas | Development of accelerometry-based fall detection[END_REF][START_REF] Li | Accurate, fast fall detection using gyroscopes and accelerometer-derived posture information[END_REF][START_REF] Cola | Fall detection using a headworn barometer[END_REF]. The time it takes for people to receive medical assistance is directly proportional to the consequences of their fall. The consequences go from contusions or serious fractures to death in cases where medical assistance takes too much time (i.e. more than 1 hour). Consequences of falls on elderly people are often irreversible, representing a major cause of death in the elderly population. In that context, automatic fall detection systems must ensure a very high detection rate level and good latency to guarantee a rapid medical assistance, thus avoiding fatal consequences. A study of the state-of-the-art of technical approaches adopted in the literature is addressed hereafter.

Fall detection: State-of-the-art

As mentioned in [START_REF] Lim | jour-nal=Journal of Applied Mathematics. Fall-detection algorithm using 3-axis acceleration: Combination with simple threshold and hidden markov model[END_REF], fall detection systems can be classified into context-aware systems and wearable systems. Context-aware systems are based on sensors to be deployed in the environment such as cameras, microphones, infrared sensors, floor sensors or pressure sensors. On the other hand, wearable systems are based on embedded sensors worn by the user, intended to detect body movements and postures.

This state-of-the-art study focuses on wearable systems to make fair comparisons between the smart connected glasses and devices with similar characteristics. Wearable fall detection systems are mainly based on information from an I MU. I MUbased wearable devices have proven to be very useful for detecting falls of elderly people [START_REF] Kangas | Development of accelerometry-based fall detection[END_REF][START_REF] Li | Accurate, fast fall detection using gyroscopes and accelerometer-derived posture information[END_REF][START_REF] Cola | Fall detection using a headworn barometer[END_REF][START_REF] Lim | jour-nal=Journal of Applied Mathematics. Fall-detection algorithm using 3-axis acceleration: Combination with simple threshold and hidden markov model[END_REF][START_REF] Kostopoulos | F2d: A fall detection system tested with real data from daily life of elderly people[END_REF][START_REF] Wu | Development of a wearable-sensorbased fall detection system[END_REF][START_REF] Qu | Evaluation of a low-complexity fall detection algorithm on wearable sensor towards falls and fall-alike activities[END_REF][START_REF] Chen | Self-adaptive fall-detection apparatus embedded in glasses[END_REF][START_REF] Bianchi | Barometric pressure and triaxial accelerometry-based falls event detection[END_REF][START_REF] Lu | Low-power operation of a barometric pressure sensor for use in an automatic fall detector[END_REF][START_REF] Wang | A low-power fall detector balancing sensitivity and false alarm rate[END_REF]. Because fall events are very rare to be produced, activity recognition is important to determine whether the user is executing a motion or a stationary activity. Typical I MU-based wearable systems are based on a single 3-axis accelerometer. According to the features that can be obtained from accelerometric data, a fall detection algorithm can be either threshold-based or MLbased [START_REF] Wu | Development of a wearable-sensorbased fall detection system[END_REF]. A threshold-based method needs a previous handcrafted feature extraction phase as well as an heuristic choice of thresholds according to every feature to be processed. A threshold-based algorithm offers low latency during data processing, while consuming low computational resources. On the other hand, a ML-based algorithm may enhance the robustness and reliability of the system, with automatic feature extraction (using Artificial Neural Networks or ANN), but at the expense of a high resource consumption. As a 3-axis accelerometer provides effective information, threshold-based methods are suitable for wearable devices where energy and computational resources are very limited. Incorporating data from other wearable sensors such as gyroscope, magnetometer and/or barometer to a threshold-based solution, can offer a composite index, useful for a better fall detection robustness.

A complete background about the characteristics of falls as well as the methods to evaluate the fall detection performance, is presented in [START_REF] Kangas | Development of accelerometry-based fall detection[END_REF]. The author of this work suggests that a fall is a cascade of phases: 1) a normal phase, where subjects perform ADLs such as walking, sitting down, standing up, etc; 2) a critical or pre-impact phase, which regroups two body movements prior to a fall such as sudden free fall towards the ground and impact to the ground; 3) a post-fall phase, where the person is lying on the ground; and 4) a recovery phase, if the person is able to get up and move after a fall. The performance evaluation of fall detection systems is often based on four metrics:

-The number of detected falls (i.e. True Positives or TP).

-The number of not detected falls (i.e. False Negatives or FN). -The number of ADL not detected as falls (i.e. True Negatives or TN).

Based on those metrics the sensitivity, the specificity and the accuracy of a fall detection system can be calculated, as shown below.

Sensitivity = TP TP + FN × 100% (2.1)
The sensitivity is the fraction of TP over the total number of falls, expressed in percentage as shown in equation 2.1. Maximizing the sensitivity is important for increasing the detection of true events.

Speci f icity = TN TN + FP × 100% (2.
2)

The specificity is the fraction of TN over the total number of events that are not falls (ADL), expressed in percentage as shown in equation 2.2. Maximizing the specificity can be useful to reduce the amount of false detection. However, increasing the specificity could lead to a decrease of the detector sensitivity.

Accuracy = TP + TN TP + TN + FP + FN × 100% (2.
3)

The accuracy is the fraction of detected falls and ADL not detected as falls, over the total number of events (falls + ADL), expressed in percentage as shown in equation 2.3. The accuracy brings a general performance evaluation, but in the case of fall detection systems, both the sensitivity and the specificity offer a better performance characterization.

This state-of-the-art study evaluate the relationship between the performance of the fall detection expressed in terms of sensitivity and specificity, and several embedded system aspects such as the used sensors, the wearable location, the detection method and the algorithm. In some works, the battery lifetime of such devices is also indicated, offering another important characteristic for comparison.

In [START_REF] Kangas | Development of accelerometry-based fall detection[END_REF], Kangas has studied methods for fall detection to be adapted for real-life applications in the older population. This study confirmed that body-worn accelerometers can be used for fall detection. It suggests that acceleration signals measured from the head or waist offer high fall detection reliability with simple thresholdbased fall detection algorithms. At the opposite, the author demonstrated that measures from wrist do not offer a good detection rate, being not suitable for fall detection. To arrive at this conclusion, an evaluation of different low-complexity fall detection algorithms using a single 3-axis accelerometer attached to the waist, wrist, and head, has been performed. The fall data were obtained from standardized types of intentional falls (e.g. forward, backward, and lateral) in three middle-aged subjects. Data from ADL were used as reference. Three different detection algorithms with increasing complexity were investigated using two or more of the following phases of a fall event: beginning of the fall, falling velocity, fall impact, and posture after the fall.

At the opposite, Kostopoulos et al. proved in [START_REF] Kostopoulos | F2d: A fall detection system tested with real data from daily life of elderly people[END_REF] that an accelerometer-based wearable fall detection system placed at the wrist can offer high reliability. This fall detection system is embedded on a smartwatch. A threshold-based fall detection algorithm is implemented to distinguish ADL from falls. This algorithm analyses linear acceleration information from the accelerometer. Rebound, residual movement and position of the body, are the three references observed from linear acceleration measurements. When a critical situation is detected, an alarm is triggered to inform the caretakers. However, there is no information about the wireless communication system used in this approach.

In [START_REF] Wu | Development of a wearable-sensorbased fall detection system[END_REF], Wu et al. developed a similar fall detection wearable system based on a single 3-axis accelerometer and a GPS module. The wearable device is to be worn on the waist. The system can detect falls by analyzing accelerometric measurements interpreted as linear movements (e.g. displacement, velocity and acceleration) and angular movements (e.g. angle rotation, angular velocity and angular acceleration).

Angular movements are computed from accelerometric measurements (i.e. without gyroscope). Once a fall is detected, the system is able to get the geographic position of the user and to send fall alarm short messages (SMS) to caregivers. To do so, both a GPS and a GSM modules are embedded together with the 3-axis accelerometer and a microcontroller unit. Considering the hardware design of this wearable device, the authors estimated a battery lifetime of around 2 days using a 1200 mAh 3V7 lithium polymer battery.

Another body location used for fall detection is the chest. Qu et al. [START_REF] Qu | Evaluation of a low-complexity fall detection algorithm on wearable sensor towards falls and fall-alike activities[END_REF] presented a low-complexity threshold-based fall detection algorithm based on a single 3-axis accelerometer. This study focused on evaluating the fall detection performance of wearing the device on the chest, while considering a set of fall-alike activities that introduce false positives. Some of these fall-alike activities are: sitting on the couch, standing from the couch, running upstairs/downstairs, jumping into the bed and lying in bed after jumping. Experimental results show a 90 % fall detection sensitivity, with a high confusion between falls and several fall-alike activities, thus introducing a considerable number of false positives. This study demonstrated that a thresholdbased algorithm using a single 3-axis accelerometer placed at the chest offers a good fall detection sensitivity but a low speci f icity.

Hybrid methods intending to improve the performance have been also proposed in the literature. For example in [START_REF] Lim | jour-nal=Journal of Applied Mathematics. Fall-detection algorithm using 3-axis acceleration: Combination with simple threshold and hidden markov model[END_REF], Lim et al. proposed a fall detection algorithm that combines a threshold-based method and a Hidden Markov Model (HMM) using a single 3-axis accelerometer. This wearable device has been fixed on the subject chest. Several fall features of 3-axis acceleration are introduced and applied to the threshold-based method. In this study the combination of both methods has proved to offer high reliability. Events detected as possible falls by the thresholdbased method are chosen and applied to the HMM method to distinguish between a fall and ADL, thus improving the performance of the system. However a computational overhead (in consequence, the energy consumption) derives from this performance improvement.

Another way to improve the performance of wearable fall detection systems while maintaining low complexity and low power consumption is through the use of other sensor-based measurements. In [START_REF] Li | Accurate, fast fall detection using gyroscopes and accelerometer-derived posture information[END_REF], Li et al. used an industrial sensor node (i.e. TEMPO 3.0) that integrates both a 3-axis accelerometer and a 3-axis gyroscope. In this approach, two nodes are used for fall detection: one is placed at the chest while the other is located on the right thigh. Using two nodes in separated body locations, the system is able to recognize static postures (e.g. standing, bending, sitting and lying) as well as motions between these static postures (considered as dynamic transitions). By coupling accelerometer and gyroscope measurements, the fall detection algorithm can reduce both false positives (e.g. sitting down fast) and false negatives (e.g. falling on stairs). However, this method has some confusion between jumping into bed and falling against a wall in a seated posture. The authors proposed as perspectives the exploitation of context information (environmental/physiological) to better discriminate these two events. Even though the hardware architecture of the TEMPO 3.0 node is described in this work, no information about the energy consumption and the battery lifetime is given by the authors.

In [START_REF] Chen | Self-adaptive fall-detection apparatus embedded in glasses[END_REF], Chen and Kuo proposed a self-adaptive threshold-based fall detection system composed of a 3-axis accelerometer, a 3-axis gyroscope and a 3-axis magnetometer embedded on glasses. This kind of system provides convenient, comfortable and non-intrusive wearing. The magnetometer helps to discard normal events such as head rotations. Then, through acceleration measurements, falls are identified. A selfadaptive mechanism is employed to dynamically update the acceleration threshold.

Finally, fall direction can be determined by using the gyroscope. Even though this 9-axis I MU allows a high sensitivity, the speci f icity remains relatively low. In this system, a GSM module is integrated to raise alarms in case of falls. Despite the authors provide information about the hardware architecture, there is no information about the autonomy of the system to evaluate the energy-efficiency of this approach.

Another combination of sensors integrated in a wearable fall detection system consists in a 3-axis accelerometer and a barometer. In [START_REF] Bianchi | Barometric pressure and triaxial accelerometry-based falls event detection[END_REF], Bianchi et al. investigated the incorporation of a barometric pressure sensor, as a surrogate measure of altitude to assist in discriminating real fall events from normal ADL, thus reducing the number of false positives. The acceleration and air pressure data were recorded using a wearable device attached to the subject's waist and analyzed offline. The study incorporated several protocols including simulated falls and ADL, performed by 20 young healthy volunteers. Results demonstrated that using the barometric pressure sensor together with a 3-axis accelerometer significantly improves the performance obtained only using the accelerometer. This improvement goes from 75% to 97.5% of sensitivity, and from 91.5% to 96/5% of specificity. This work presents nevertheless some limitations. First, the fall detection algorithm is executed offline, meaning that the wearable device is only in charge of collecting and transmitting data to a workstation, through a Bluetooth module. The system also seems to be very energy greedy, offering an autonomy of 24 hours when powered by a rechargeable battery of 30 Ah.

Cola et al. [START_REF] Cola | Fall detection using a headworn barometer[END_REF] also studied the combination of accelerometric and barometric information by placing the sensors at the head of users (through glasses or headwear).

The barometer is useful for measuring the pressure variation associated to a movement measured by the accelerometer. In fact, in the presence of a slow fall (so a small acceleration), a simple threshold-based algorithm may not detect the fall event. With the incorporation of the barometric measurement, this movement is identified as a fall because of its pressure variation, thus improving the sensitivity. It has proven to be an efficient combination that leads to a reliable discrimination between slow falls and ADL movements (e.g. stand-to-sit). In this work, only two events have been evaluated to generate performance metrics (slow falls and stand-to-sit). The authors claimed that a sensitivity and a speci f icity of 100 % have been obtained during their tests, thus demonstrating a perfect discrimination between these two events.

In [START_REF] Lu | Low-power operation of a barometric pressure sensor for use in an automatic fall detector[END_REF], Lu et al. implemented an ultra low-power fall detector (LPFD) to be attached to a neck-worn lanyard. This LPFD integrates a 3-axis accelerometer and a barometer that collect information to be processed by an algorithm based on a binary decision-tree classifier able to discriminate between falls and ADL. The novelty of this approach is the integration of a semi-permeable membrane (SPM) installed across a window in a waterproof enclosure to delay the equilibrium between internal and external pressure after a change in altitude. This delay between internal and external pressure allows the barometer in the LPFD to be switched off until it is activated by an acceleration interrupt indicating a potential fall. The system is therefore able to capture enough information about the pre-fall pressure to compute the differential pressure that determines whether or not the acceleration interrupt corresponds to a fall. This approach allows to optimize the overall power consumption of the LPFD, thus its battery lifetime. The authors claim to reach an autonomy of 996 days with a battery capacity of 450 mAh. Despite these results are interesting, there is a lack of information about the hardware architecture and if the algorithm is executed online or offline. In case of an online implementation, there is no information about the target platform that has been used.

Similarly, Wang et al. proposed in [START_REF] Wang | A low-power fall detector balancing sensitivity and false alarm rate[END_REF], a wearable fall detection device (NEON) to The obtained results showed a sensitivity of 91 %, an average power consumption of 111 µW, and an estimated autonomy of 1125 days for a battery capacity of 1 Ah. These interesting results in terms of battery lifetime are however dependent on the activity performed by the user, so the scenario. The longer the user remains in a motion state, the less autonomy of the system.

The fall detection performance of a wearable device is typically evaluated using datasets containing simulated falls and ADL, from young and healthy people. However, real-world fall datasets from elderly people are important to get real signal signatures and to study activities preceding the fall. It is also worth noting that existing works evaluate the performance through a specific protocol for the acquisition of simulated data. Yet, fair comparisons are not possible because every protocol differs from each other. A summary of the existing fall detection wearable systems leveraging threshold-based algorithms is given in Table 2.3. Based on this study, a fall detection algorithm has been implemented on the Ellcie-Healthy connected glasses. This solution offers 97% of sensitivity and 99% of specificity and more than 30 hours of battery lifetime. The performance has been estimated through the simulation of 300 fall events from 10 healthy volunteers.

Fall detection using the Ellcie-Healthy smart glasses

The fall detection algorithm embedded on the Ellcie-Healthy smart connected glasses integrates three sensors: a 3-axis accelerometer, a barometer and a IR proximity sensor. Accelerometric measurements are used to identify both the head inclination and the fall impact through a threshold-based solution. To do so, it is important to know the axis orientation within the connected glasses (see Fig. 2

.5).

As can be seen, the gravity axis changes between X, Y and Z, depending on the head inclination. However, the Z axis is in most cases in charge of carrying the gravity acceleration when glasses are worn in normal conditions (e.g. in a stationary or motion state such as activities of daily living ADL). Since the orientation of the head during and after a fall is rarely in a vertical position (i.e. with the Z axis as the gravity In the meantime, using barometric measurements the system is able to compute the differential pressure due to changes in the altitude. In fact, the barometer embedded on the connected glasses provides accurate atmospheric pressure measurements that allow to detect altitude changes even from a few centimeters (i.e. 30 cm). As mentioned in the state-of-the-art study, integrating a 3-axis accelerometer and a barometer on a wearable device has proven to offer high reliability. Additionally, an IR proximity sensor is used to enhance this reliability. Using this sensor, the system is indeed able to analyze the eyelid activity. It is useful for both determining whether the connected glasses are worn, and identifying whether or not the user is in a consciousness state after a fall. Once a fall has been detected, data collected from these three sensors are indeed used to detect the recovery of the user. If no recovery is detected within 30 seconds after the fall, the system notifies relatives and medical care using a smartphone paired with the connected glasses through BLE.

Smart Connected Glasses: Design challenges

The design of the current industrialized version of smart connected glasses has been done from scratch in multiple aspects. The hardware architecture has been the result of a complete product market study that considered requirements in terms of product shape, material, color, finishing touch and fashion trending, as well as the maximum size and weight. In addition to this, the objective was to design a multiplepurpose wearable device capable of addressing different daily life issues (e.g. driver drowsiness detection, fall detection, activity tracking) using the same hardware architecture. According to these requirements, the Ellcie-Healthy R&D team has designed and assembled an electronic system shown in Fig. 2.7, to be integrated into the hinges and the front of a frame. As can be seen, the electronic system is composed of three main boards embedding hardware modules. First, the PMU board embeds the battery management system. This board is located close to the battery in order to monitor and control the battery state of charge as well as the battery safety operation. Then, the MCU board that embeds a 32-bit MCU, a BLE device, an accelerometer and several others environmental sensors. To power up the MCU board, a flexible PCB (FLEX) is connected between the boards. Power supply routes as well as communication routes (using I2C and SPI protocols) embedded on the FLEX, allow the connection and data exchange between the boards. Finally, on each side of the FLEX, IR sensors used to monitor eyelids activities, are placed. Even though this hardware architecture satisfied several constraints in terms of size, weight, frame design, comfort and user safety, some constraints such as the battery lifetime and the performance were more difficult to consider. As mentioned in chapter 1, apart from the size criteria, the battery capacity has been chosen according to a static analysis based on the electrical information given by manufacturers of hardware modules. The basic functional characteristics of sensors (e.g. sampling rate, low-power modes usage), as well as the characteristics of the microcontroller (e.g. CPU frequency) and the BLE device (e.g. theoretical estimation of the amount of data to be transmitted, fixed transmission power) have been considered to estimate the best and worst cases in terms of battery lifetime. However, several aspects such as other sources of power consumption (e.g. modules that compose the battery management system) or the battery aging, have not been considered during the design phase, thus significantly impacting the system autonomy. Furthermore, dynamic behaviors from the application execution such as the connected glasses states and the transitions between states have not been analyzed. However, these behaviors may impact both the power consumption and the performance of the application. In consequence, online optimizations are needed to offer the best compromise between autonomy and performance, in other words the best user experience. To do so, energy consumption (so battery lifetime) and performance constraints related to every application use case, must be studied.

Battery lifetime constraints

According to the social legislation related to road transport on the official journal of the European Union and in compliance with Regulation (EC) No 561/2006 [39], the daily driving time shall not exceed 9 hours. The first objective of the Ellcie-Healthy company was therefore to ensure a system autonomy greater than 9 hours when the driver drowsiness detection (DDD) is activated. On the other hand, an automatic and wearable fall detection system must offer an autonomy of at least 16 hours. In fact, the system must ensure a continuous monitoring of the user during daily activities. Assuming that the user has regular 8-hour nights of sleep (taking advantage of this time for charging the connected glasses), the second objective was to ensure a system autonomy greater than 16 hours when the fall detection (FALL) is activated. In that context, several optimization rules (e.g. switching to MCU lowpower mode when glasses are not used, choosing lower consuming sensor settings) have been carried out to improve the autonomy of the system for both applications (DDD and FALL). Leveraging these optimizations, the current firmware version of the connected glasses allows a battery lifetime of more than 13 hours for DDD and more than 30 hours for FALL.

However, from a common constructive feedback from beta-testers it was clear that it was necessary to further improve the battery lifetime. Indeed, if people are used to recharge their smartphone on a daily basis, wearable users are not willing to recharge their device with such a period. This fact directly affects the user experience of wearable devices as well as their adoption. The objective of this thesis is therefore to provide solutions to notably improve the autonomy of the smart connected glasses, so the user experience.

Performance constraints

The Ellcie-Healthy company aims at delivering as fast as possible both applications (DDD and FALL) to the market. To do so, a Continuous Integration / Continuous Delivery (CI/CD) pipeline approach [START_REF]Ci/cd pipeline: A gentle introduction[END_REF] has been adopted by the R&D team. The objective of this approach is to deliver as soon as possible, stable versions of the applications. The benefit of this approach is to quickly be aware about the feedback of users to continuously adapt the product. Both the autonomy of the system and the performance of applications have been improved through this methodology. However, important decisions that slow down the optimization process, have been taken. To ensure acceptable performance, static system settings have for instance been adopted to rapidly deliver new applications versions. However, switching from static to dynamic system settings is not a trivial task. These changes must indeed be done according to several criteria such as the battery state of charge or the improvement of the quality of service in situations of danger. In consequence, performance optimizations have been carried out by analyzing the impact of the chosen static settings on both the sensitivity and the speci f icity rates. The DDD performance has been estimated by computing the sensitivity and the speci f icity rates of an eye blink detection algorithm, reaching up to 81 % and 87 %, respectively. Since the estimation of the driver drowsiness index strongly depends on the eye blink detection algorithm, further improvements are needed to avoid an overestimation or an underestimation of this drowsiness index. On the other hand, the thresholdbased fall detection algorithm has been proven to be very efficient from the beginning. This algorithm offers indeed a sensitivity and a speci f icity higher than 95 % for different types of falls. Whatever the application use case, the challenge remains the same: improving the autonomy while controlling the performance degradation, or improving the performance while controlling the autonomy degradation.

High-level hardware description

High-level hardware description environments are more and more used to model embedded systems, being able to simulate and optimize them early in the design flow. A few examples of these environments are:

-SystemC/TLM [START_REF]Systemc[END_REF]: This is a popular open-source class library within C++ that provides the advantage of modeling both hardware and software, to design both the architecture and the behavior of a hardware component [START_REF] Barnes | Verification and validation of wireless sensor network protocol properties through the system's emulation[END_REF].

The SystemC/TLM library includes a simulation kernel allowing the execution of source code intended to be implemented into the modeled platform.

The main purpose of this open-source implementation is to execute both transaction level and register transfer level (RTL) simulations at a higher abstraction level compared to VHDL and Verilog (which are specialized hardware description languages). This modeling abstraction allows for faster development of models while maintaining a high accuracy in the simulation results. When a precise simulation of both the hardware and the source code (i.e. software) is required, this language is perfectly adapted.

-Ptolemy II [START_REF] Ptolemaeus | System Design, Modeling, and Simulation using Ptolemy II. Ptolemy.org[END_REF]: This is an open-source project developed in Java and conducted in the Department of Electrical Engineering and Computer Sciences of the University of California at Berkeley. Ptolemy studies modeling, simulation, and design of concurrent and real-time embedded systems. This opensource project allows the implementation of distributed simulations for heterogeneous embedded systems running complex applications. To do so, this framework is based on actor oriented design modeling that deal with parallel computing and concurrency problems. The main benefit of Ptolemy is the possibility of integrating multi-task and heterogeneous computational models together, in order to accurately simulate the behavioral aspect of complex computational environments such as wireless sensor networks [START_REF] Brito | Development and evaluation of distributed simulation of embedded systems using ptolemy and hla[END_REF].

-SysML [START_REF]SysML. Sysml open source project -what is sysml? who created it? URL SysML[END_REF]: This is a general-purpose architecture modeling language for system engineering applications. SysML supports the specification, analysis, design, verification and validation of a broad range of systems (e.g. hardware, software, information, processes, personnel, and facilities) and systemsof-systems (SoS). This framework is highly adapted to model-based systems engineering (MBSE) as shown in [START_REF] Steimer | Model-based design process for the early phases of manufacturing system planning using sysml[END_REF], where the authors modeled the processes related to the early phases of a MBSE-based multidisciplinary manufacturing system planning. In [START_REF] Steimer | Model-based design process for the early phases of manufacturing system planning using sysml[END_REF], SysML provides a formalized and visually interpretable description of the manufacturing processes, and allows developers to easily understand it from an abstraction level. This language is more and more used for the design phase and the performance validation of real-time embedded systems, as explained in [START_REF] Nikolaidou | Simulating sysml models: Overview and challenges[END_REF]. The authors in this work presented an overview of several simulation tools based on SysML models, such as MARTE, TTool and Modelica. These tools enable the transformation of SysML models to executable simulation code. SysML is nevertheless a complex methodology intended to modeling the processes of a system composed of multiple domains or disciplines through the description of requirements, system's structure and behavior.

-Wisebatt [START_REF] Wisebatt | Wisebatt | where do you want to start[END_REF]: This is an online simulation tool for designing connected devices and having rapid access to approximated results in terms of battery lifetime, performance and manufacturing cost. Before becoming a commercial tool, this methodology had been presented in [START_REF] Bramas | Wisebat: accurate energy benchmarking of wireless sensor networks[END_REF]. In this work, the authors proposed a battery lifetime estimation module called WiSeBat (Wireless Sensor Battery), integrated in an existing simulation tool (WSNet). The WiSeBat module performs battery lifetime estimations for wireless sensor networks integrating both a power consumption model and a battery model. The power consumption model considers the power consumed by individual components.

The authors of this work validated its approach against measurements on a real sensor device. They demonstrated that their models offer accurate battery lifetime estimations for duty-cycled scenarios and give more realistic results than the default energy model of the WSNet simulator.

Several other methodologies for system-level modeling from the design phase of embedded system architectures intended to simulate and characterize the application's behavior (i.e. performance), have been also proposed by academics in the last years [START_REF] Erbas | A framework for system-level modeling and simulation of embedded systems architectures[END_REF][START_REF] Oletic | System-level power consumption analysis of the wearable asthmatic wheeze quantification[END_REF][START_REF] Fallahzadeh | Trading off power consumption and prediction performance in wearable motion sensors: An optimal and real-time approach[END_REF]. One example of them is a modeling and simulation environment called Sesame [START_REF] Erbas | A framework for system-level modeling and simulation of embedded systems architectures[END_REF], that facilitates performance analysis of embedded multimedia systems architectures. To do so, Sesame decouples the application from the architecture by modeling at first, the functional behavior (i.e. application) separated from the architectural issues (e.g. timing characteristics, resources usage, bandwidth constraints, etc.). Then, a platform architecture model is implemented by defining computational resources and performance constraints. Finally, a co-simulation of both models is possible through a mapping step, thus allowing the exploration of a wide range of design choices as well as the performance evaluation of the application. Using this modeling and simulation tool facilitates, for example, the choice of how many processor cores are needed to execute a specific multimedia application composed of a predefined number of parallel processes.

Through those frameworks, high-level hardware modeling represents nowadays an accurate solution to rapidly validate embedded systems designs, thus optimizing costs, energy consumption and applications performance. Models help working at a higher level of abstraction and therefore removing the complexity of the system to evaluate specific aspects. Since the objective of this thesis is to propose a systemlevel modeling methodology to rapidly estimate and then optimize the energy consumption on smart connected glasses, so the aspect to be evaluated is the power consumed by hardware modules. It is worth noting that modeling the application's behavior of this real-time embedded system is not mandatory for characterizing and then modeling the power consumption. In consequence, frameworks intended to offer accurate behavioral models such as SystemC/TLM, Ptolemy II and SysML, are considered as complex and therefore do not fit into this thesis objective. The behavioral aspect must however be considered as it is required to describe the scenario to be executed by the glasses, useful for the simulation of the power consumption. At the opposite, hardware models in the Wisebatt environment make a higher abstraction while considering only the energy consumption aspect of an embedded system, and therefore are more adapted to our work. In that context, a test license of the Wisebatt tool has been requested, followed by a modeling trial of several hardware modules embedded on the smart glasses. Even if handling this tool was simple and fast, several aspects have made us decide not to use it as part of the methodology to be proposed. First, around 50 % of the hardware modules embedded on the smart glasses were not implemented on this tool at the beginning of this thesis and those already implemented had missing features. We had the possibility to request the inclusion of the missing modules and the completion of the existing modules, but without having access to the source code. This tool has been initially designed for modeling the power consumption of embedded systems on the wireless sensor area, not having enough maturity for wearable devices as the smart glasses in 2017 and 2018. Finally, another aspect that has prevented the use of this tool has been the cost of a license to opt for a private project and protect the intellectual property of the company. However, important insights have been obtained from this framework.

On the one hand, as mentioned by [START_REF] Bramas | Wisebat: accurate energy benchmarking of wireless sensor networks[END_REF], the estimation of the battery lifetime of a device requires both the power consumption of the device and its battery to be accurately modeled. On the other hand, the information given by manufacturers of hardware modules is accurate enough for bringing precise power consumption estimations. For this reason, we decided to propose a modeling methodology based on analytical information and composed of different modeling blocks (e.g. power models and a battery model) presented hereafter.

System-level modeling methodology

Data exchange between the different blocks interacting with the smart connected glasses is depicted in Fig. 2.8. Interactions with the environment and users are done by measuring environmental and behavioral data through sensors. Another way to interact with users is through a mobile application typically running on a smartphone. In this case, users can either receive information from glasses, or set different configurations to glasses. For that, glasses must be paired with a smartphone using a BLE communication protocol. An Internet link between the smartphone and servers allows storing data related to users, glasses and historical records. As mentioned in section 1.1, the common limitation of wearable detection systems is the scarce amount of computational and energy resources available to offer both high reliability and autonomy. To deal with these limitations, hardware and software dependencies must be studied together. Algorithms for health and safety applications embedded on wearable devices must be reliable. These algorithms are then usually very energy consuming [START_REF] Chia Bejarano | A novel adaptive, real-time algorithm to detect gait events from wearable sensors[END_REF][START_REF] Varatharajan | Wearable sensor devices for early detection of alzheimer disease using dynamic time warping algorithm[END_REF][START_REF] Yan | An emerging technology -wearable wireless sensor networks with applications in human health condition monitoring[END_REF], thus impacting significantly the autonomy of the system. Modeling power consumption early in the design flow is useful for optimizing energy consumption, so the battery lifetime. Additionally, characterizing the performance of embedded algorithms is useful for quantifying the QoS degradation when optimizing the energy consumption. Moreover, a high-level battery model able to predict its SOC, can allow recognizing system settings that cause significantly SOC losses. For all these reasons, a system-level modeling methodology is proposed. The objective is to perform simulations of scenarios with the current as well as future versions of Ellcie-Healthy smart connected glasses. Moreover, designing a generic methodology that can be used to model and simulate a variety of wearable devices apart from the smart glasses, is another important objective to be accomplished with this system-level modeling approach. An overview of this methodology is presented in Fig. 2.9.

Based on system-level power models, designers can explore the solutions space and find the best hardware/software settings. Moreover, using battery SOC predictions, battery lifetime can be evaluated and compared to real measurements. Using this methodology, scenarios representing worst or corner cases can be easily simulated to get information that is difficult to retrieve in real tests. Additionally, power management strategies can be evaluated to improve autonomy while measuring the QoS degradation. More functionalities can be provided using the proposed system-level modeling methodology. One of them is battery sizing for the next generations of smart connected glasses. Based on the user experience, a defined autonomy can be used as input of scenario simulations to determine an appropriate battery size. As can be seen in Fig. 2.9, the system-level modeling methodology is separated into three different stages: Input, modeling, and output.

Input stage

The input stage consists in defining the smart glasses use cases. In this thesis, two different use cases are considered: the drowsiness detection and the fall detection. A scenario is represented by a finite state machine (FSM) using a time period to control the scenario execution. Initial conditions as well as the QoS and the autonomy constraints, are defined during the input stage. However, and for the sake of visibility, this information is presented at the modeling and optimization stages on Fig. 2.9, respectively.

Modeling stage

The modeling stage is the heart of the proposed system-level modeling methodology. In this stage, a high-level description of the smart connected glasses is provided. This description considers functional (e.g. application tasks) and non-functional (e.g. energy consumption) embedded features to offer accurate energy consumption and system autonomy estimations. Three system elements are modeled: the battery SOC, the average power consumption of application tasks, and a characterization of the performance.

Battery model

Modeling the battery of a wearable device such as the connected glasses implies taking into account three important factors:

a) The remaining capacity.

b) The ongoing number of charge/discharge cycles.

c) The total number of charge/discharge cycles the battery can support before reaching 80 % of its capacity (aging).

It is worth mentioning that the battery end of life is typically defined as the point where the battery only provides 80% of its maximum capacity [16]. This information, combined with the average power consumption of the system during a discharge phase, allows estimating the battery lifetime. The battery model will be detailed in chapter 5.

Power consumption model

To estimate the average power consumption required for an application, analytical power models have been used. The development of these models requires a highlevel hardware description of the system. The hardware architecture shown in Fig. This system level representation illustrates the hardware modules of the system, as well as the power domain of each module. The battery directly supplies two modules: the power management integrated circuit (PMIC) and the fuel gauge. The PMIC realises the charge of the battery whenever an external supply source is connected, as well as the supply voltage regulation for the rest of the system through DC/DC converters. Two DC/DC converters are used to separate the system into two power domains (i.e. V1 and V2). The MCU, FLASH memory, BLE chip, I MU, IR sensors and environmental sensors (e.g. humidity, temperature, pressure), take part on the V1 power domain. On the other hand, actuators (e.g. buzzer and RGB leds) as well as IR leds, belong to the V2 power domain. Using the supply voltage of each power domain as well as the theoretical current consumption data provided by manufacturers on their datasheets, the power consumption (in Watts) of each module can be estimated.

The overall energy consumption of the system (E SYS ) can be considered as the addition of the instantaneous power consumption (P SYS ) over time as shown in Equation 2.4.

E SYS (t) = T t=0 P SYS (t) dt (2.4)
The required average power to operate the device PSYS can be divided into application tasks, as shown in Equation 2.5.

PSYS = n ∑ i=1 Pi → ∀n = 1, 2, 3... (2.5)
Where n is the number of application tasks. These tasks derive from the application executed with the Ellcie-Healthy smart glasses. Hardware modules impacted by every task must be identified to estimate the average power consumption of the system. The analytical power consumption models will be detailed in chapter 3.

Characterization of the performance

Whatever the targeted scenario, information about the quality of a detection system are needed to characterize the performance. Regarding the drowsiness detection, the main event to be detected is the blink. In that case, studying the blinks detection performance provides a first information about the algorithm efficiency, as well as the impact of different system configurations. Regarding the fall detection, the event to detect is obviously the fall. Studying the quality of fall detection helps understanding the information required to characterize the performance of this application, taking into account the configuration of the system. For the human activity recognition, different activities are considered (e.g. walk, going up/down stairs, sit-to-stand). In consequence, an important number of metrics must be correlated for this application, making the performance study more complex. It is worth noting that several algorithms are currently in development phase and the definition of performance metrics is still in study. The characterization of the performance will be detailed in chapter 4.

Output stage

The output stage allows to display information related to the autonomy of the system, the discharge rate of the battery or the QoS degradation, for a scenario that simulates worst, corner or typical cases. This simulated outputs provide insights for the development of an optimization system intended to maximize the autonomy while guaranteeing an acceptable QoS. Our goal is therefore to help designers finding the best tradeoff between energy and performance early in the design flow. An optimization system can indeed be developed in simulation and then be embedded on the smart connected glasses to enable a dynamic decision making optimizer dealing with online autonomy and performance constraints. This optimizer module can finally determine whether the constraints are satisfied through the verification of an objective function, considering the system settings and the battery SOC.

Conclusion

In this chapter, we have first introduced the smart connected glasses developed by Ellcie Healthy, presenting an overview of their hardware architecture and detailing the wearable applications addressed nowadays with them. We have later exposed the context of this PhD thesis by introducing the challenges to be faced when designing this kind of wearable devices. The state-of-the-art of system-level modeling methodologies intended to facilitate the design of efficient embedded systems has been then discussed to support our choice of the methodology to be adopted. Finally, a presentation of the adopted methodology has been done, identifying the topics that have been explored during the three years of this thesis. A power consumption characterization has marked the beginning of this work, thus allowing to propose a system-level power modeling approach to be detailed in the next chapter.

Chapter 3

Energy consumption modeling: A high level approach

Introduction

Wearable devices are intended to continuously monitor different human data without interruption [1]. However, their battery lifetime is critical because of the limited area for embedding energy storage devices. In consequence, the energy consumption must be meticulously analyzed to design wearable devices that meet both autonomy (i.e. battery lifetime) and performance (i.e. QoS) constraints. Modeling energy consumption early in the design flow, from a high level of abstraction, can be useful for choosing the best hardware and software architecture considering the aforementioned constraints. Moreover, a high-level modeling approach can also be useful for energy consumption optimizations and battery lifetime estimations during the development phase. Optimizations or predictions can indeed be easily performed for different hardware and software settings, thus avoiding the tedious task of measuring the power consumption.

The smart connected glasses collect behavioral and environmental information using data sensors. These collected data are the input for embedded applications such as driver drowsiness detection, fall detection for elderly people or activity recognition, as mentioned in Chapter 2. According to a certain number of predefined QoS rules, the collected data must respect several requirements (e.g. minimum signal amplitude), to ensure a complete and correct execution of the embedded algorithms, as well as reliable detection applications. The system's energy consumption is impacted by these QoS requirements because improvements in the quality of the sensor's signal lead to an augmentation of the sensor's power consumption. In that context, hardware and software settings must be carefully defined to improve battery lifetime, while being sure that signals always meet the QoS requirements.

In this chapter, a state-of-the-art for energy consumption modeling on wearable devices is presented. Then, the adopted energy consumption modeling approach for the smart connected glasses, is described. To do so, a high-level representation of the smart connected glasses hardware architecture is detailed. This high-level modeling approach is based on power consumption models of different application tasks: Data sensing (DST), data processing (DPT), data transmission (DTT), event notification (ENT) and battery management (BMT). Hardware modules taking part in the execution of a task, are then presented. After that, first results about analytical average power consumption estimation, are given. Finally, the conclusion of this high-level power consumption modeling approach is presented.

Energy consumption modeling: state-of-the-art

The analysis of the energy consumption on wearable devices using a system-level modeling approach, has been already addressed in multiple researches [2,3,4,5,6,7]. In those works, a common strategy consists in collecting the information related to the power required by each hardware module and using it to create energy consumption models. This information is either from datasheets of manufacturers (i.e.

theoretical data) or from real measurements (i.e. experimental data). In fact, systemlevel modeling approaches based on theoretical data have proven to have lower complexity and good accuracy. As mentioned in section 2.3.3, it has been demonstrated in [3] that the power consumption information coming from the datasheet of hardware manufacturers is accurate enough for bringing precise power consumption estimations. For instance, Wisebatt [8], a commercial simulation tool provides rapid approximated results of battery lifetime for wireless sensor networks devices based on theoretical power consumption data.

The power consumption of each hardware module can also be predicted from the total power consumption of the wearable. In [2], the authors studied the power consumption on a group of smartwatches that support the open source Android Wear OS. They developed a linear regression model for analyzing the contribution of several hardware components on the total power consumption of each smartwatch. The model was trained with physical measurements on smartwatches. Using this linear regression model, the hardware components that impact the most the total average power consumption can be identified. Moreover, using the coefficients of correlation of this linear regression model, each hardware component power consumption can be predicted with less than 16.8% of median relative absolute error. The authors considered the following components: Screen, CPU, SD Card and BLE. Including sensors and actuators, as well as components for the battery management system may lead to a more complex modeling and less accurate estimations. Moreover, the relationship between the predicted power consumption and the configuration of each hardware component is not explicit. Despite the linear regression model provides good predictions, this approach is not suitable for modeling the Ellcie-Healthy smart glasses since much more hardware components and settings details must be considered.

Another way to model power consumption from a high abstraction level is dividing the execution of a wearable application into different application activities (taskbased system-level modeling approach). In [4], Oletic and Bilas carried out a systemlevel power consumption analysis on a wearable wireless acoustic sensor used for asthmatic wheezing quantification. The objective of this study was to determine whether a local algorithm execution (i.e. data acquisition + processing) is more power efficient than a remote algorithm execution (i.e. data acquisition + raw data streaming). Results demonstrated that the local execution featuring the lowest communication cost enables the lowest total sensor system power consumption. The power consumption of the wearable sensor was estimated through the theoretical information of the selected hardware components in active state. Similarly, Fallahzadeh and Ghasemzadeh proposed a real-time energy consumption optimization algorithm for activity recognition (AR) on wearable sensing nodes [5]. To do so, an energy consumption model was defined. The authors identified the data sensing, the data processing and the wireless communication as the main sources of energy consumption on a wearable AR algorithm. The power required to operate in active mode the hardware components associated to each source, was computed to estimate the total average power consumption that provides insights for the real-time energy consumption optimization algorithm. Similarly, in [6,7] the authors describe the embedded systems power consumption by modeling the different subsystems to make the models description easier. In [7] a methodology for studying and analyzing the energy consumption of IoT applications is proposed. In all these works, the implementation of power models has been simple and efficient for bringing accurate and rapid energy consumption estimations of wearable applications. In addition to this, the analytical power models obtained from [6,7] can be easily validated with physical measurements of the wearable system total average power consumption.

Scenario definition for the smart glasses

Before tackling the issue of the power consumption modeling approach adopted in this thesis, it is necessary to define the notion of scenario in the context of the smart glasses. As mentioned in section 2.4.1, a scenario is represented by a finite state machine (FSM) using a time period (i.e. a tick) to control the scenario execution.

This time period represents the system period (T SYS ) in which, the proposed power modeling methodology will be executed. The global FSM illustrated on -PDN: In this state the whole system is in shutdown mode. To go to this state, the glasses must be turned off by the user or the battery capacity must go below its minimum limit. This state can be reached from all the other states.

-ACT: This is a transitory state in which the MCU is in active mode, the BLE device is not yet paired with a smartphone, the accelerometer, pressure, temperature and humidity sensors are in active mode and the rest of sensors and actuators are in shutdown mode. To go to this state, several options are possible:

1. From PDN state: the glasses must be turned on by the user.

2. From SLP state: the inactivity of the glasses must be interrupted either by the user or by a periodic event.

3. From DDD state: the user must deactivate driver drowsiness detection by tapping the glasses with the right sequence.

4. From FALL state: the user must deactivate fall detection by tapping the glasses with the right sequence.

5. From CONN state: the BLE connection between the glasses and a smartphone has been recently lost.

-CONN: This is a transitory state in which the MCU is in active mode, the BLE device is paired with a smartphone, the accelerometer, pressure, temperature and humidity sensors are in active mode and the rest of sensors and actuators are in shutdown mode. To go to this state, several options are possible:

1. From ACT state: the user must establish a BLE connection between the glasses and a smartphone.

2. From SLPC state: the inactivity of the glasses must be interrupted either by the user or by a periodic event.

3. From DDDC state: the user must deactivate driver drowsiness detection by sending the right BLE command through the mobile application.

4. From FALLC state: the user must deactivate fall detection by sending the right BLE command through the mobile application.

-SLP: In this state the MCU is in low power mode (i.e. CPU is not running)

and the rest of the system is as described in ACT state. To go to this state, the glasses must be initially in ACT state and then stay inactive for at least 20 seconds.

-SLPC: In this state the MCU is in low power mode (i.e. CPU is not running) and the rest of the system is as described in CONN state. To go to this state, the glasses must be initially in CONN state and then stay inactive for at least 20 seconds.

-DDD: In this state the MCU is in active mode, both IR sensors are in active mode and the rest of the system is as described in ACT state. To go to this state, the glasses must be initially in ACT state and then the user must activate driver drowsiness detection by tapping the glasses with the right sequence.

-DDDC: In this state the MCU is in active mode, both IR sensors are in active mode and the rest of the system is as described in CONN state. To go to this state, the glasses must be initially in CONN state and then the user must activate driver drowsiness detection either by tapping the glasses with the right sequence or by sending the right BLE command through the mobile application.

-FALL: In this state the MCU is in active mode, an IR sensor is in active mode and the rest of the system is as described in ACT state. To go to this state, the glasses must be initially in ACT state and then the user must activate fall detection by tapping the glasses with the right sequence.

-FALLC: In this state the MCU is in active mode, an IR sensor is in active mode and the rest of the system is as described in CONN state. To go to this state, the glasses must be initially in CONN state and then the user must activate fall detection either by tapping the glasses with the right sequence or by sending the right BLE command through the mobile application.

A typical scenario with the Ellcie-Healthy smart connected glasses is depicted in Fig. 3.2. The execution of this scenario is controlled by a time period (for example: T SYS = 60 s). In this example, we assume that the glasses are always paired via BLE with a smartphone. The scenario starts in sleep mode (state = SLPC). After The objective of the proposed system-level power modeling approach is therefore to accurately estimate the energy consumption of the system for scenarios such as the one presented above, and then simulate the discharge process of the smart glasses.

It is worth noting that smart glasses typical scenarios can last from a few hours to several days. However, our objective is to get energy consumption estimations only in few seconds or minutes. This requirement justifies the use of a high-level system modeling approach.

Energy consumption modeling for smart glasses

A task-based system-level modeling approach has been adopted for the smart glasses analytical power models. In this section, the proposed analytical power models are detailed. Models that depend on the system activities are proposed to estimate the power required by the different parts of the system to execute an application or scenario. These models allow estimating the total average power consumption of the system and so the battery lifetime. Analytical power models have been validated

with real measurements of the overall average power delivered by the battery while executing the driver drowsiness detection and the fall detection applications. A discharge process scenario has also been executed with the smart glasses to compute the battery lifetime. Measures of autonomy are compared with power models estimations. Obtained results will be presented at the end of this chapter.

As mentioned in chapter 2, The overall energy consumption of the system (E SYS ) can be considered as the addition of the instantaneous power consumption (P SYS ) over time as shown in Equation 3.1.

E SYS (t) = T t=0 P SYS (t) dt (3.1)
The power models of the smart glasses consider the average power consumption of the system PSYS , as the sum of the average power consumed by the application tasks, for a particular scenario. For this reason, the modeling process needs first, the definition of both the application and the scenario to be simulated. Then, tasks involved in the execution of the application can be identified, so modules or hardware components impacted by every task. The required average power to operate the device is described in Equation 3. (Task_5) Battery management (BMT): this task controls the system's power supply through the PMIC and the fuel gauge.

Therefore, using these five tasks, the Equation 3.3 that derives from the Equation 3.2, allows to compute the system's average power consumption:

PSYS = PDST + PDPT + PDTT + PENT + PBMT (3.3)
Hereafter, these tasks as well as each hardware component are detailed. For this, the main functional (i.e. what is it for?) and non functional (i.e. power consumption) characteristics of each hardware component are presented.

Data sensing task (DST)

Data sensing task involves smart glasses data acquisition during the runtime process, using sensors. As shown on the Fig. Where Psensor (j) is the average power consumption of a sensor and S is the number of sensors used by the system. To estimate Psensor , the hardware settings (e.g. sampling frequency and operation mode) and electrical characteristics (e.g. voltage supply and current consumption) of the component must be studied. For example, in Fig. 3.4, the main parameters needed to compute the average power consumption of a sensor, are detailed.

PSENSOR = V 1 * (DC ACTIVE * I ACTIVE + DC SLEEP * I SLEEP ) (3.5)
The Equation 3.5 represents therefore the basis of the system-level power modeling methodology used in our approach.

IMU: Inertial Measurement Unit

The glasses integrated I MU is the LSM6DS3 inertial module [9] that features 3-axis digital accelerometer and 3-axis digital gyroscope in a single package. The I MU allows to detect orientation and gestures with high sensitivity, green tech and low noise interference. It can be configured to different sensitivity levels of acceleration and different angular rate measurement ranges. This product can be used for applications such as robotics and IoT devices, by computing tilt, motion and tap sensing.

The information available (given by the manufacturer) to model this device is synthesized in Table 3.1. Both the accelerometer and the gyroscope can be configured in four different operation modes (power down, low power, normal and high performance), as shown in

IR Proximity Sensor

An IR proximity sensor is able to detect the presence of objects without any physical contact, using both infrared photo-diodes (IR LED) and photo-detectors. As According to this, the algorithm 2 represents the implementation of the proposed As can be seen, using this information, duty cycles of different IR sensor states (e.g.

F base_acc ← 1600 16: Pacc (s) ← V 1 * I acc (s) * F acc (
analytical
sleep, active, measurement) and the IR LED duty cycle, can be estimated. Once having these duty cycles, the average power consumption of an IR sensor ( Pir_sensor ) can be computed. It is worth mentioning that two IR sensors are embedded on the smart glasses, and every IR sensor pilots one IR LED. For this reason, two instances of this model are required ( Pir_sensor_1 and Pir_sensor_2 ).

Pressure Sensor

An environmental data to be collected by the glasses is the atmospheric pressure.

For this purpose the LPS22HB pressure sensor [11] is used. It is an ultra-compact portable device used in altimeters, barometers, GPS applications, weather station equipment and sport watches. In operating mode, the host communicates with the sensor through the I 2 C device registers, thus making the device particularly suitable for direct interfacing with a microcontroller. The power consumption directly depends on both the operating mode (e.g. low-power and normal mode) and the measurement rate (i.e. sampling frequency), which can be configured between 1 and 75 Hz. Table 3.2 shows the sensor theoretical current consumption given by the manufacturer datasheet.

Humidity/Temperature Sensor

Another important environmental variables to consider are the relative humidity and the medium temperature. For that, a capacitive digital sensor for relative humidity and temperature called HTS221 [12] is also integrated into the glasses. It is an ultra-compact sensor which includes a sensing element and a mixed signal ASIC to provide the measurement information through digital serial interfaces. The power consumption directly depends on both the operating mode (e.g. power down and normal power modes) and the measurement rate (i.e. sampling frequency), which can be configured between 1 and 12.5 Hz. Table 3.3 shows the theoretical current consumption given by the manufacturer in the sensor datasheet. The sensor models presented above allow the average power consumption of the data sensing task ( PDST ) to be estimated using the equation 3.6.

Algorithm 3 : Generic Sensor (Pressure, Humidity and Temperature)

(s) ← V 1 V base * Fsensor (s)
F base * I low_power Where F base and V base are given in tables 3.2 and 3.3

5:

else 6:

Psensor (s) ← V 1
V base * Fsensor (s)

F base * I normal_power
Where F base and V base are given in tables 3.2 and 3.3

7:

else 8:

Psensor (s) ← V 1 V base * I sleep
Where V base is given in tables 3.2 and 3.3

9:

return Psensor (s)

PDST = Pimu + Pir_sensor_1 + Pir_sensor_2 + Ppress_sensor + Phum_sensor (3.6)
Where Pimu , Pir_sensor_1 , Pir_sensor_2 , Ppress_sensor and Phum_sensor are the average power consumption of the inertial measurement unit, both infrared proximity sensors, the pressure sensor and the humidity/temperature sensor, respectively.

Data processing task (DPT)

Data processing task involves the computation and storage of data measured by sensors through embedded algorithms. Two devices are thus considered in this task: the MCU and a FLASH memory. As for data sensing, the average power consumption of the data processing task ( PDPT ) depends on the state of the smart glasses as well as the settings of the MCU and FLASH hardware modules. Modeling the power of these modules allows estimating PDPT , as shown in the equation 3.7.

PDPT = Pmcu + Pf lash (3.7)

MCU: Microcontroller Unit

The central unit for control and computation on the smart connected glasses is the STM32L451xx SoC [13]. This SoC integrates a microprocessor and several internal peripherals to communicate with the I MU, the BLE device as well as the sensors and actuators. This ultra-low-power microcontroller is adapted to high-performance embedded applications. It is based on a 32-bit ARM Cortex-M4 core operating at a frequency up to 80 MHz. Regarding electrical characteristics, this microcontroller offers ten different operation modes, according to both the CPU frequency and the embedded linear voltage regulator that controls the power supply. Two voltage regulators are available: the main regulator (MR) and the low-power regulator (LPR).

In run and low-power run modes, voltage regulators can adjust power consumption ranges according to the CPU maximum operating frequency.

-Range 1 run mode using MR: Users may set a CPU frequency between 26

MHz and 80 MHz. 16:

Pmcu (s) ← V 1 * I stop_0 17: case stop_1 18: Pmcu (s) ← V 1 * I stop_1 19: case stop_2 20: Pmcu (s) ← V 1 * I stop_2

21:

case standby

22:

Pmcu (s) ← V 1 * I standby

23:

return Pmcu (s)

According to this, and based on the electrical information obtained from the manufacturer's datasheet, theoretical power consumption estimations can be performed.

FLASH Memory

Apart from the ROM memory integrated into the MCU, the smart glasses embed an external non-volatile memory offering more capacity for data storage. This is a MX25R1635F 16Mb bits Serial NOR FLASH memory [14]. Table 3.5 contains the technical specifications to be considered into the average power consumption model for this FLASH memory ( Pflash ). Values of typical current consumption are obtained from the manufacturer's datasheet.

According to the glasses state, so the application, the system reads from and writes to the FLASH at a predefined period of time (i.e. T read and T write ). The inputs of this model are therefore the configurable parameters of the system such as the operation mode (mode f lash ), the frequency and the number of inputs/outputs for reading (F read and I/O read ), both read and write periods (T read and T write ), and the system period (T SYS ). It is worth recalling that T SYS has been introduced in section 3.3.

According to these parameters and information of Table 3.5, the FLASH average power consumption ( Pflash ) can be estimated. The algorithm 5 allows modeling this hardware component. First, the different current consumption values can be selected using the mode f lash , F read and I/O read values. Duty cycles for reading, erasing and writing can be then calculated using T SYS , T read , T erase and T write values. Finally, the Pflash estimation can be obtained by computing these values and considering the FLASH voltage (V 1 ). return Pf lash (s)

Data transmission task (DTT)

Data transmission task refers to the wireless communication between the smart connected glasses and a gateway (GW), typically a smartphone. In the smart glasses environment, this wireless communication is done using the Bluetooth protocol (BLE).

The microchip used for this purpose is the BlueNRG-MS [15] proposed by ST Microelectronics. This device contains a BLE processor based on an ARM Cortex-M0 core. In the smart glasses, the communication between the BLE chip and the MCU is done via a SPI interface. As mentioned in [16], BLE technology provides two modes Connection. During the first phase, the slave executes advertising with a predefined interval (T ADV ) [18]. As soon as the end user chooses to connect both the slave and the master, the connection phase starts. In this phase, the master initiates regular connections every connection interval (T CON ). The connection is a sequence of data exchanged between the slave and the master [19]. Following this protocol, the Equation 3.8 defines the average power consumption required for a wireless communication.

PDTT = PBLE = PADV + PCON + PSLEEP (3.8)
Where PADV , PCON and PSLEEP are the average power consumption in advertising, connection and sleep phase, respectively. Either in advertising or in connection phase, the BLE device goes to sleep mode as soon as an interval of activity is finished. In the connection phase, the slave decides whether it has data to send. If there is no data to be sent, the connection is executed without payload. Otherwise, one packet with a payload length up to 20 bytes is sent every data interval (T DATA ).

As it can be seen in Fig. 3.6, connections with payload data are triggered as soon as the end user starts an application (e.g. driver drowsiness detection). This payload gathers for example data from sensors. To model the BLE power consumption, we only consider the case where the slave sends data to the master. The reason is that BLE commands sent from the master to the slave have a negligible impact on the BLE Device power consumption. T DATA can be seen as a multiple of T CON , as shown in Equation 3.9. Current consumption information has been obtained using an estimation tool [20] provided by ST-Microelectronics. Using this tool, the average current consumption of the BLE device during the advertising phase as well as the connection phase, with and without payload, has been estimated. These estimations have been obtained for a range of hardware settings summarized in Table 3.6. As it can be seen, the current depends both on the BLE operation mode and the transmit power. In order to better understand the different active phases of the BLE device, experimental power consumption measurements have been performed using an oscilloscope. These measurements have also allowed assessing the accuracy of the estimation tool mentioned before. In Fig. In consequence, to model the average power consumption of the data transmission task ( PDDT ) defined in Equation 3.8, the state of the smart glasses as well as the hardware settings of the BLE module, must be considered as inputs. The algorithm 6 summarizes the BLE power consumption model. As can be seen, when the BLE device is not in power down mode, the average power consumption of every phase is estimated according to the selected current consumption (that depends on mode BLE (s) and TX POWER (s)), and the estimated duty cycles (that depends on the application, thus the glasses state). Duty cycles are estimated using the algorithm 7.

T DATA = m * T CON → ∀m N + (3.
As can be seen, when the glasses are in the advertising phase (i.e. disconnected from the smartphone), the BLE device spends a part of the system period (T SYS ) in a advertising active phase (DC ADV ) and the rest of the time in sleep phase (DC SLEEP ).

At the opposite, when the glasses are in the connection phase, the BLE device spends its time in connection intervals without payload (DC CON_NP ), as well as connection intervals for raw data streaming (DC DATA STR ), variable updates (DC DATA VAR ) and event notifications (DC DATA EVT ).

Event notification task (ENT)

Events are generated in runtime when anomalies or risk situations related to both the user and the smart glasses are detected. Such events are: falls and recovery after falls when executing the fall detection application, or drowsiness levels when executing the driver drowsiness detection application. Another kind of events are raised when some thresholds are exceeded. Such thresholds are: low or high temperature of the smart glasses that can put in danger the battery safety (so the user), and battery SOC thresholds to indicate the approximated amount of energy available in the battery. Apart from using the mobile application to notify the user about these events via wireless communication, the smart glasses embed actuators such as a BUZZER and two RGB LEDs (Red, Green, Blue) to raise sound and light alerts. The event notification task can therefore be modeled according to the sequences to be performed by the actuators, as well as their duration and frequency. PENT = 2 * PRGB_LED + PBUZZER (3.12)

RGB LED

The RGB LED used on the smart glasses for light notification is the HSMF-C116

[21] ultra-small chip-LED. This LED is designed for wearable applications due to its small size. Two HSMF-C116 are embedded on the smart glasses, each one controlled by its own LED driver (the LP5562 LED driver from Texas Instruments [22]). This LED driver is designed to produce variety of lighting effects in a more efficient way than using only the RGB LED. An I 2 C communication is established between the MCU and the LED driver. The RGB LED is piloted by the driver through a PWM (Pulse-width modulation) signal. The voltage supply of the RGB LED (power domain V1, see Fig. 3.3) differs from the one for the LED driver (power domain V2).

According to this, the average power consumption of a RGB LED system ( PLED RGB ) can be considered as the sum of the average power consumption of the LED driver ( PDRIVER LED ) and the RGB LED ( PLED ), as shown in the Equation 3.13.

PLED RGB = PDRIVER LED + PLED (3.13)

BUZZER

Sound notification is done using a BUZZER integrated on the smart glasses. This BUZZER is a piezo audio transducer KMTG0902 from Kingstate Electronics Corp [23]. This piezo audio transducer is suitable for wearable devices due to its low power consumption and its small size. This module is piloted by the MCU using a PW M signal. The only configurable parameter is the BUZZER frequency used to regulate the sound pressure level. In table 3.7 a summary of the electrical information given by the manufacturers of the BUZZER, the RGB LED and the LED Driver, is given.

V = 3 V 5 mA F = 4 kHz
This information is used to model the power consumption of the event notification task. The sequence of events to be played by the actuators always involve the BUZZER and both RGB LEDs simultaneously. These three modules are therefore in active mode when an event is produced, and they stay in power-down mode otherwise. Each event has its own sequence that involves aspects such as the duration of the sequence (t sequence i ) and the period in which this sequence is reproduced (T sequence i ). Those events are obviously asynchronous. Based on the feedback from the user experience, we were able to assign a period for each event. Moreover, hardware settings such as the intensity of each RGB LED color (i.e. instantaneous current consumption by color), the frequency of the BUZZER or the operation mode of the components, also depend on the sequence of each event. According to this information, the RGB LED and BUZZER power model is described in the Algorithm 8.

if mode DRIVER LED (s) is power_down then 3: PRGB_LED (s) ← V 1 * I SLEEP DRIVER LED 4: PBUZZER (s) ← 0 5: else 6: PRGB_LED (s) ← ∑ n i=1 PRGB_LEDsequence i (s)
where n is the number of sequences 7:

PBUZZER (s) ← ∑ n i=1 PBUZZERsequence i (s) 8: return PRGB_LED (s), PBUZZER (s)
As can be seen, the average power consumption of the actuators depends on the estimated average power consumption of every sequence which is performed by the smart glasses to notify an event. The average power consumption of every sequence is obtained using the model presented in the Algorithm 9. As can be seen in this model, using the sequence duration and period as well as the hardware settings by glasses state (s), the average power consumption of the BUZZER and RGB LED can be computed for every sequence.

Algorithm 9 : RGB LED and BUZZER power modeling for sequence i

Battery management task (BMT)

The smart glasses power supply is provided by a LiPo rechargeable battery which is connected to both a charger, two DC/DC regulators and a fuel gauge. The charger executes battery recharges when an external power supply is connected to it. During a discharge process, two DC/DC regulators control the power supply of the system by using the energy available in the battery to deliver a constant voltage to the load (which consists in the MCU, FLASH, BLE device, sensors and actuators). Every DC/DC regulator supplies a group of hardware modules with a specific voltage supply, thus creating two different power domains as previously shown in Fig. 3.3.

In order to be aware of the remaining battery capacity, the MCU gets periodic battery SOC values using the fuel gauge. When the battery capacity goes below a predefined threshold, the DC/DC regulators are switched off to avoid a complete discharge of the battery. Modeling the power consumption of the battery management task involves therefore the analysis of the average power consumption of these three modules, as shown in Equation 3.14.

PBMT = PCHARGER + PDC/DC REG + PGAUGE (3.14)

Fuel Gauge

The smart glasses embed the MAX17055 Fuel Gauge from Maxim Integrated Products [24]. This microchip provides a large variety of battery information such as cell voltage, charge and discharge current, cell temperature and remaining capacity, to mention a few. Using an I 2 C interface, the MCU periodically obtains fuel gauge information to monitor battery conditions in runtime.

Charger and DC/DC regulators

The smart glasses use a PMIC that includes the charger and DC/DC regulator functionalities in the same microchip. This microchip is the ultra-low power PMIC MAX77650 from Maxim Integrated Products [25], that provides highly-integrated battery charging and power supply solutions for low-power wearable applications where size and efficiency are critical. One of the main advantages of this device is that it properly allows charging the battery and powering the system simultaneously. Using the electrical information given by the manufacturer (see Fig. For the experimental characterization, we decided to subtract the power consumption of the charger and the fuel gauge, thus allowing to model the power of the DC/DC regulators, as shown in Fig. 3.8. Where PSYS can be seen as the total average power consumption of the system, and PDC/DC REG derives from PLOAD according to a linear coefficient (β). The experimental characterization depicted in Table 3.9 allows estimating a value of β for both power domains (V1 = 1.8V, and V2 = 3.9 V).

According to this experimentation, β is equal to 36% ± 4.5%.

Having this information, the power of the battery management task can be modeled as shown in Algorithms 10 and 11.

Experimental results: analytical vs measured autonomy

The driver drowsiness detection application has been used to validate the accuracy of the proposed system-level power consumption modeling approach. To do so, two different system configurations have been used, as shown in Table 3.10. As can be seen, only three hardware modules have been modified to have two different system's power consumption values. Hardware settings for the rest of the smart glasses architecture have been considered identical for both configurations. This study allows analyzing the impact of a few system parameters on the power consumption of the smart glasses. These system parameters are: the CPU frequency of the MCU, the sampling frequency and the activation time of the IR sensors and IR LEDs. As can be seen in this table, the configuration number 2 requires a lower average power consumption ( PHW ) because both the CPU frequency and the sampling frequency decrease. The difference between the average power consumption of the IR LED for both configurations is caused by its activation time (that provokes an increase of the intensity of IR light) as well as its frequency. Using the power models presented in this chapter, the average power consumption of the system ( PSYS ) can be estimated for both configurations. Then, the autonomy can also be predicted by dividing the total energy available on the battery (E BATT with PSYS , as shown in Equation 3.17).

Autonomy = E BATT PSYS (3.17 To validate our models, real autonomy measurements of the smart glasses have been performed through the execution of a discharge process for both configurations, from 100 % of battery SOC to 0 %. In Fig. 3.9 a comparison between experimental and analytical results is presented. The discharge process was configured to operate from 4.2 V (SOC = 100%) to 3.3 V (SOC = 0%), as can be seen in the long dotted line curves. Experimental autonomy results (short dotted lines) have been obtained by measuring the SOC every 10 seconds using the fuel gauge. Analytical autonomy estimations (denoted by the continuous lines) have been obtained using the Equation 3.17. As can be seen in Table 3.11, the estimation error between the measured and estimated autonomy is 3.3% for the configuration # 1, and 4.5% for the configuration # 2, meaning that the accuracy of estimations is higher than 95% in both cases. It is worth noticing that our models are between 3% and 5% pessimistic for these two discharge processes, meaning that power consumption are overestimated compared to real measurements.

Conclusions

In this chapter, we have proposed a system-level power consumption modeling approach based on analytical information (i.e. manufacturer datasheets), to rapidly evaluate the autonomy of the smart glasses for different system settings. For two different system configurations, we have been able to predict the autonomy with only 5% error, thus demonstrating the relevance of our approach. This efficient modeling approach is useful for designers to help them rapidly finding the best hardware settings to improve the autonomy of the smart glasses. Moreover, this approach can be helpful to easily identifying the system parameters that impact the most the average power consumption, by performing estimations rather than tedious measuring tasks. In addition to this, designers can also take advantage of this proposed method to predict the system's autonomy of future smart connected glasses versions, by adding new hardware modules, new wearable applications or even a different battery capacity. Finally, confronting applications performance with the estimated autonomy using this approach, is useful to improve energy efficiency.

For example, increasing the power of the infrared LED leads to a better signal-tonoise ratio for blinking measurements, but a lower autonomy. A parameter can therefore impact the performance of the application, and in the meantime the autonomy of the system. In the next chapter, a characterization of the performance for the driver drowsiness detection application will be proposed, focusing on parameters that directly impact both the autonomy of the system and its performance.

(i.e. on the MCU), thus offering better results in terms of performance while respecting the constraints of the wearable device. Finally, the conclusion of this chapter is presented.

Performance characterization

In an embedded detection system as the one proposed with the smart glasses, the study of the application performance represents a key issue for providing reliability. However, maximizing performance must be carefully undertaken to avoid an important system's autonomy degradation. For this reason, hardware and software dependencies must be studied together to improve the performance while being aware of the impact on the autonomy. In this thesis, we propose to assess the tradeoff between the detection performance and the autonomy of the system for an eye blink detection algorithm. This approach can then be replicated for different algorithms to improve their performance.

In the literature, many researches have been carried out to propose methodologies for optimizing the performance of embedded detection systems [1,2,3,4,5,6,7,8].

In most works, maximizing the performance is the main purpose, but at the expense of the energy consumption [1,2,3,4,6]. The main reason is that constraints in terms of computational and energy resources are often not considered. The reliability is indeed placed as the priority. However, optimizing both the performance and the energy consumption of a detection system, as done in [7,8], can offer a better user experience.

Evaluating performance or QoS for binary detection systems as the one proposed with the eye blink detection algorithm, is often performed through metrics such as the sensitivity, the specificity and the accuracy [7,8,4,5,6]. In most cases [4,5,6],

the focus is on algorithms used in wearable Fall Detection Systems (FDS). In those works, the choice of metrics is taken according to several relevant criteria exposed in the literature. In our work, this choice is based on the comparison between sensitivity, specificity and accuracy rates. The sensitivity describes the capability of a system to detect an event whenever it occurs, while the specificity represents the ability of a system to moderate false detection occurrences. In that case, the goal is then to maximize the sensitivity to ensure that no fall remains unnoticed, even when the specificity is compromised. In other cases, FDS performance is assessed by measuring the percentage of good decisions (i.e. its accuracy). It is worth noting that the accuracy metric is useful for binary detection systems where events to detect are balanced. This is the case of blink detection used for estimating the drowsy state of drivers. In such a case, the number of blinks over a period of time is often proportional to the number of other eye's events (e.g. looking up, down, left or right). In this work, the performance of blink detection is evaluated through the analysis of sensitivity, specificity and accuracy metrics. To do so, we realises a data collection phase, followed by a labeling and a processing phases, to generate all the important metrics for different system configurations.

Performance characterization for an eye blink detection algorithm

The driver drowsiness detection application proposed with the Ellcie-Healthy smart glasses is mainly based on measurements of eyelid movements using the IR proximity sensors. These measurements are obtained from an eye blink detection algorithm in charge of computing the raw data (i.e. time series coming from IR sensors) to determine whether a segment of a time series corresponds to an eye blink. This time series, also called blinking signal, is obtained by illuminating both eyes using infrared light emitting diodes (IR LEDs) and measuring the reflected light using IR photodetectors (i.e. photosensor oculography, as mentioned in section 2.2.1), as illustrated in The drowsiness evaluation is performed through three successive phases: 1) eye blink detection, 2) metric computation, 3) and drowsiness index estimation. Besides blinks, several other events can occur. As can be seen in Fig. 4.1, a driver can also perform look events (e.g. look down) during a driving period. The challenge is therefore to detect as many blinks as possible while ignoring other events. As soon as some events are detected as blinks, three metrics are computed over a period of time (e.g. 60 s):

-the average amplitude, -the duration, -and the rate of blinks.

Drowsiness is then estimated using a five-point composite scale based on the abovementioned metrics (as mentioned in section 2.2.1). A continuous analysis of blinking variability allows measuring the progressive evolution of the driver drowsiness condition. The performance of the drowsiness prediction strongly depends on the aforementioned metrics and therefore on the eye blink detection accuracy. In consequence, the performance needs to be assessed from the eye blink detection phase.

For this reason, in this chapter we focus only on blink detection to propose a performance evaluation method.

Eye blink detection: a threshold-based algorithm

The threshold-based eye blink detection algorithm aims at determining whether there are blinks in a blinking signal. To do so, a baseline is first computed, as shown in Fig. 4.1. This baseline is used for several reasons such as identifying a level of IR data (i.e. amplitude) that corresponds to open eyes, and setting a blink detection threshold at the beginning of the IR data processing. This threshold is set empirically and has two main objectives: ignore noise peaks that frequently occur at the baseline level, and guarantee that the initial blinks are correctly detected. As illustrated in Fig. 4.1, as soon as N blinks are detected (e.g., N=10), the threshold is dynamically updated according to the average amplitude of those detected blinks.

This technique is necessary to dynamically adapt the detection process to the variations of blink amplitude over time, as well as to ignore false events. It is worth pointing out that a pre-processing filtering phase is also implemented to disable eye blink detection when some external perturbations occur, thus avoiding false results.

Performance evaluation for eye blink detection

The performance evaluation is done either through the relationship between the sensitivity and specificity, or through the detection accuracy. The goal is to compare several system configurations to identify those offering the best autonomy/performance trade-off. To do so, an experimental study has been carried out. This study consists in three stages: 1) data collection and labeling, 2) creation of a dataset for autonomy/performance analysis, and 3) performance metrics analysis.

Data collection and labeling

The accuracy of the blink detection is estimated from real blinking data collected using the smart glasses. Raw data from IR sensors are streamed in real-time toward a smartphone, then stored on a server to be processed afterwards. During data collection, the gaze events shown in Fig. 4.1 (e.g. look left/right, look up/down, blink) are labeled to get the ground truth of the experimentation. The data collection process has been performed under ideal conditions (dark room, static body and head posture, same glasses position and acquisition protocol) to reduce noise and perturbations as much as possible. We can then compare the impact of different system configurations on the blinking signal in an ideal scenario. To study the impact of ambient light perturbations on the performance, a random noise is added to the

Creation of a dataset for autonomy/performance analysis

A data collection using different system configurations of the IR sensors has permitted to create a dataset for this experimental study. Three hardware parameters have been varied: the sampling frequency of IR sensors (F ir_sensor ), the power supply of the IR LEDs (P ir_led ), and their activation time (t ir_led ). Each combination of these three parameters has an impact on both the SNR and the blink detection performance. For example, by augmenting the power supply of LEDs the SNR is improved, but at the expense of a power consumption increase. The different possible values of the parameters used in our study are shown below. Combining each parameter leads to a total of 120 possible configurations. It is worth noting that the remaining system parameters are not modified during the experimentation. The goal is therefore to study these 120 possible combinations and assess, for each of them, the impact on both the detection performance and the autonomy. The Table 4.1 shows an extract of the results obtained through this experimentation. For the sake of visibility, only 8 configurations randomly selected, are detailed in this table.

Performance metrics analysis

As can be seen, the three first columns represent the system configurations. The fourth column corresponds to the obtained blink detection threshold that is set at the beginning of the IR data processing for each configuration. Columns five, six and seven show the performance metrics to be analyzed. Finally, the eighth column shows the estimated autonomy obtained for each configuration. It is worth mentioning that the autonomy has been computed using the power models presented in the previous chapter.

The blink detection threshold is used during the blinking signal processing to detect the presence of eye blinks. The purpose is to determine whether the IR signal increases and crosses the threshold, stays over it for a lapse of time and decreases until it crosses the same threshold again. This threshold is dynamically adapted as the processing progresses, to cope with different signal and noise levels, and to avoid the detection of spurious (look) events such as: look at up, look at down, look at left or look at right (see Fig. we can identify in green circles, the system configurations that meet this constraint.

We can easily observe that the autonomy is inversely proportional to F ir_sensor , P ir_led and t ir_led . For example, when t ir_led is set at 400µs, we can then observe that both F ir_sensor and P ir_led must be reduced at their minimum to maintain an autonomy of 12h.

Regarding the performance constraints, during this experimentation, around a half of system configurations have permitted us to obtain a percentage of detection higher than 80% for the three performance metrics (accuracy, sensitivity and specificity).

Only a few (7 configurations), were over 85% for all the metrics. Based on it, for this experimentation we have decided to select 80% as the constraint for each metric. The configurations that meet this constraint are also identified by green circles in their respective plot. From the accuracy and sensitivity plots shown in figures 4.3b and 4.4a respectively, we can observe that the detection performance is dominated by P ir_led and t ir_led . As an example, an accuracy of at least 80% can be achieved using either the maximum value of P ir_led (450 mW) or the two highest t ir_led values (200µs and 400µs). In this case, the detection performance is maximized because increasing P ir_led and t ir_led leads to a higher received infrared power at the sensor, thus improving the SNR.

At the opposite, the specificity plotted in Fig. 4.4b shows that the best operating points can be obtained in the lowest P ir_led and t ir_led areas, while presenting an homogeneous distribution regarding F ir_sensor . Decreasing the signal strength indeed reduces the probability of detecting spurious events. The reason is that the eye blink detection algorithm stops the detection process when the SNR is lower (mostly at high F ir_sensor and at low P ir_led and t ir_led ). In those cases, despite a good specificity rate, the probability of non detection of blinks tends to increase, thus reducing the sensitivity. Finally, autonomy and performance constraints have been conjointly evaluated (see Fig. 4.5). As can be seen, the objective is to analyze the system configurations that help to maximize the autonomy while offering more than 80% of detection accuracy, sensitivity or specificity. Configurations identified with triangles, squares and stars are among those offering an autonomy higher than 12h. Moreover, configurations in triangle shape offer an accuracy and sensitivity higher than 80%, those in square shape offer an accuracy and specificity higher than 80%, and those in star shape have more than 80% for all these metrics. Evaluating the blinking detection performance through the accuracy, cannot ensure that both the sensitivity and the specificity constraints are met. To propose an eye blink detection system sufficiently robust for neglecting most of spurious events (i.e. specificity constraint respected), while considering as much blinks as possible (i.e. sensitivity constraint respected), the best way is the analysis of both metrics together (sensitivity and specificity).

So among the configurations represented by a star (those with a sensitivity and specificity higher than 80%), in this experimentation, the configuration that maximizes the autonomy is the one with F ir_sensor = 25Hz, t ir_led = 50µs and P ir_led = 150mW. Using this system configuration, we have obtained the best trade-off between autonomy and performance.

The user experience feedback has been taken into account to set performance constraints for the threshold-based eye blink detection algorithm. An overestimation of drowsiness levels appeared to be a main issue. To overcome this overestimation, we found that both the sensitivity and specificity must be higher than 90%.

Significant performance improvements have been achieved by optimizing the blinking data processing using a variety of filtering techniques. However, we have only reached up to 81% and 87% of sensitivity and specificity, respectively. This level of performance represents therefore an important limitation for the threshold-based algorithm, thus making necessary, the exploration of new approaches for an eye blink detection optimization.

Eye blink detection: performance optimization

In this section, we propose to assess the benefits of DL-based solutions for eye blink detection in a wearable device (i.e., smart glasses). CNN have indeed proved to be very effective in time series classification for a wide range of applications [10,11]. CNNs are very noise-resistant and able to extract features extremely well, thus avoiding a tricky human-based feature extraction process. In addition, CNNs are among the AI algorithms supported by the ST X-CUBE-AI library to run optimized inferences on MCUs. Our objective is two-fold: first, the eye blink detection performance must be noticeably improved to worth the effort of implementing a CNN-based algorithm at the edge. On the other hand, the CNN implementation at the edge must meet several embedded constraints of the smart glasses: a maximum 10% decrease of the battery lifetime (with respect to the current threshold-based solution) and ROM and RAM usage lower than 90 kB and 32 kB, respectively. In the following, CNN-based solutions are described and then evaluated using a dataset of samples collected from different subjects wearing the smart glasses. To do so, a CNN hyperparameter algorithmic study is performed. Then, both algorithmic approaches, threshold and CNN, are compared through a fair evaluation of the performance, battery lifetime and memory footprint.

CNN-based algorithm

A current limitation with the threshold-based eye blink detection algorithm presented with the smart glasses, is the non-negligible number of look events detected as blinks (i.e., False Positives). This issue is also illustrated in Fig. 4.1, where both look left and look down events are detected as blinks. In most cases false positives lead to an increase in the average blink duration, thus causing a drowsiness overestimation. Despite various performance improvement efforts, we came to the conclusion that the threshold-based algorithm strongly depends on the morphology of individual users that provoke sharp differences in the resulting blinking signals (e.g. blink amplitudes, baseline perturbations). This assertion makes essential the exploration of more generalized models, able to extract features and find patterns regardless of the blinking signals characteristics.

Implementing CNN-based prediction models at the edge is very challenging due to embedded system constraints. Important contributions have recently been made to improve energy efficiency and execution time for embedded systems such as smartphones [12,13,14]. In [12], Abderrahmane et al. studied Spiking CNNs for classification tasks. The authors proposed a methodology for efficiently designing neuromorphic embedded chips adapted to specific applications. In [13], Motamedi et al. introduced a framework called Cappuccino that synthesizes software for SoCs on mobile devices with the objective of executing a CNN efficiently. In [14], Meloni et al. introduced NEURAGHE, a customizable hardware architecture for CNN execution at the edge. The undeniable classification performance offered by CNN algorithms, justify the effort made by the community. However, running a CNN inference is even more challenging for the Ellcie-Healthy smart glasses, a wearable device having a single 32-bit MCU. In the literature, some studies demonstrate the feasibility of implementing Neural Networks algorithms on a MCU [5,15]. In [10] and [11] authors evaluated DL architectures for time series classification addressing different classification problems. In this work, we propose a CNN architecture inspired from [11] as the baseline model of our study (see Fig. 4.7). An evaluation of this baseline model in terms of accuracy for a blink detection problem has been carried out with a binary blinking signal dataset. The accuracy has been then improved through a CNN hyperparameters algorithmic study. Finally, the baseline as well as the best CNN models in terms of accuracy have been embedded on the glasses to evaluate metrics such as the memory footprint, the inference execution time and energy consumption.

Datasets for training and testing

Data were collected from five subjects wearing the smart glasses. Raw data were collected from IR sensors at a 100 Hz sampling frequency. Then, 2600 blink or nonblink input vectors were extracted, normalized and labeled. An input vector is then composed of 64 samples (640 ms) and each sample is normalized to one byte (i.e., ranging from 0 to 255) from a raw data sample initially coded as a 32-bit integer.

Using CNN, the pre-processing filtering phase is no more needed as eye blink de- samples where no events are present (i.e. other in Table 4.2). So, all these patterns are included in the dataset to train the CNN model. Table 4.2 details the percentage of events within the dataset considered as non-blink.

tection

CNN hyperparameter algorithmic study

To make the eye blink detection algorithm lightweight, an exploration phase of hyperparameters has been performed to find the CNN configuration that maximizes the accuracy and minimizes ROM memory footprint, as well as power consumption and execution time. To do so, the exploration study has been carried out for 880 different combinations of hyperparameters using the same training and test datasets.

As shown in Fig. 4.7, the baseline CNN architecture is composed of two 1-D convolution layers with 6 and 12 filters, respectively, and a filter size (fs) of 7 for both layers. Moreover, every 1-D convolution layer is followed by a mean-pooling layer that performed down-sampling with a fs of 3. Finally, a classifier (fully connected layer) is applied to obtain the predicted class as output. The classifier consists in two consecutive 32-neuron hidden layers followed by one output binary neuron.

The hidden layers uses a rectified linear activation function (Relu), while a sigmoid activation function is used at the output layer.

The model is optimized using the binary cross-entropy loss function and the efficient Adam version of gradient descent. Model C in Table 4.

and a batch size of 10, an accuracy validation using the test dataset was performed.

The mean accuracy obtained for 5 iterations was equal to 98.2% ± 0.8%. In order to improve the accuracy, the number of convolutional layers, the number of filters per layer, the size of the filters as well as the type of down-sampling operation were varied. No change was made to the fully connected layers of the classifier. The best models obtained from this algorithmic study are shown in Table 4.3. Apart from the baseline model (C), six models having 1 to 3 convolution layers are shown. These models offer the best accuracy for the considered type of CNN topology.

CNN-based algorithm: Experimental results

In this work, the STM32Cube.AI ecosystem was used to run optimized inferences on an MCU. The X-CUBE-AI library offers the capability of converting pre-trained neural network models into embedded C source code. The target MCU integrated in the glasses (STM32L451xx family) is designed around a 32-bit ARM Cortex-M4 processor running at a frequency of up to 80 MHz (in this work, the processor is cadenced at 40 MHz), a ROM size of 512 kB and 160 kB of embedded RAM, as presented in chapter 3. As mentioned before, the ROM memory available for this algorithm is limited to 90 kB. However, the X-CUBE-AI library comes with an overhead of 47 kB, thus restricting the maximum allowed CNN model size to 43 kB. This limit mainly impacts the maximum number of trainable parameters of the CNN architecture. The choice of the best blink detection model does not only depend on its performance.

The execution time, battery lifetime and memory footprint must also be considered for implementation at the edge. Therefore, in the following we evaluate the performance of CNN-based solutions to verify whether those constraints are met. Then, through a fair comparison we assess whether it is worth the effort to implement a CNN-based solution rather than the existing threshold-based method. In Tables 4.4

and 4.5, the ROM and RAM memory footprints, the execution time and the average MCU power consumption as well as the performance metrics are given for each CNN configuration.

Threshold-based vs CNN-based: Real-time constraints Execution time

To reproduce the real-time conditions in connected glasses, the algorithms used 600 new vectors as a continuous data flow. As can be seen in Table 4.4, the inference time of all CNN models exceeds the 10 ms sampling period. As inferences must be executed at a lower rate, samples need to be temporarily saved in a buffer. We decided to perform an inference every 10 samples (i.e. every 100 ms), giving an acceptable margin for a real-time execution. In terms of execution time, the threshold-based algorithm only takes 2.6 ms, while it takes around 54 ms to execute a CNN inference

Battery lifetime

Using active and sleep power consumption values from the MCU datasheet, the average power consumption for both algorithms was computed and is shown in Table 4.4. As it can be observed, the threshold-based algorithm requires around 30% less power than the CNN-based models. However, as shown in Fig. 4.8, the battery lifetime of the system, estimated following the method presented in chapter 3, only decreases by half an hour (around 5%) for a battery capacity of 95 mAh. The reason is that most of the energy consumption is due to the IR sensors, not the execution of the eye blink detection algorithm.

Memory footprint

As can be seen in

Threshold-based vs CNN-based: Performance

The performance of a detection system is typically evaluated through the sensitivity, specificity and accuracy metrics [16]. On the left side of Fig. 4.8, the accuracy achieved with both algorithms, is presented. As can be seen, and despite a lower inference rate, CNN models provide a better accuracy than the threshold-based solution. A CNN is indeed able to efficiently discriminate blinks from non-blinks. Fig. 4.9 shows Class Activation Maps (CAM) obtained for blinks and look-down events, considering activation of the last convolutional layer plotted as a heatmap, scaled between 0 (lowest activation) and 1 (highest activation). Even though the sensitivity and the overall accuracy are better, we can observe in Table 4.5 that 1-convolution layer CNN models (i.e. models A and B) do not improve

Conclusions

In this chapter, we have proposed a comprehensive study about the performance of an eye blink detection algorithm used for estimating the driver drowsiness condition with the smart glasses. The performance of a threshold-based eye blink detection algorithm has been assessed for 120 system configurations using different metrics (e.g. accuracy, sensitivity and specificity). Obtained results have shown that those performance metrics offer important details about different aspects of the algorithm such as their relevance for discriminating spurious events from blinks to avoid false positives. This study provides a step-by-step procedure to assess the performance of other algorithms embedded on the smart glasses, such as the fall detection.

Additionally, the consideration of performance constraints imposed by designers according to the user experience feedback has motivated the exploration of a CNNbased eye blink detection algorithm, based on IR sensor signals and implemented at the edge. Using convolutional layers allows us to obtain a more general model compared to traditional threshold-based algorithms. Results showed that all the metrics can be significantly improved. For application to driver drowsiness, it means a better user experience for people wearing the connected glasses. We also evaluated overheads of this solution in terms of memory and battery lifetime. Our study showed that the proposed solution meets the specified requirements, thus demonstrating the feasibility of a CNN-based implementation at the edge for drowsiness detection. The table that resumes the drowsiness detection state of the art presented in chapter 2, is updated with this contribution and presented in Table 4.6. As can be seen, this work addresses an existing gap to enable eye blink detection based on IR sensor signals on a wearable MCU-based embedded system.

Finally, this performance study has been useful for demonstrating the relevance of the proposed system-level power modeling approach. A lot of different combinations of system parameters have indeed been used during this study. Using the power models has allowed us to rapidly evaluate the impact of system parameters on the autonomy. To go further in the evaluation of the smart glasses applications, a data-driven battery modeling is detailed in the next chapter, with the purpose of having a complete modeling methodology for smart glasses simulations.

during the discharge process, is presented. In section 5.4, the different battery SOC prediction approaches studied in this work, are detailed. Then, the proposed analytical system-level battery modeling approach is presented. In this section, results in terms of accuracy of predictions are analyzed. Finally, in section 5.5, the conclusion of this chapter is presented.

Battery SOC prediction: state of the art

The description of the battery remaining capacity in an embedded system, is based on the estimation of SOC. Since SOC cannot be directly observed from batteries be- The state of the art for rechargeable battery modeling is closely related to the development of efficient battery management systems (BMS) for a wide range of applications, from smartphones to electric vehicles. Diverse types of battery models intended to predict SOC have been proposed in last decades. However, the choice of the best battery model to monitor conditions and states of battery packs, remains a source of discussion. Comparisons are often based on SOC predictions accuracy as well as the implementation complexity of battery models [1]. Depending on SOC predictions accuracy, a BMS is able to ensure battery safety operation, improve the operational lifespan, prevent overcharge or discharge and control power consumption strategies to save as much energy as possible [2]. On the other hand, implementation complexity refers to the effort made during model's design and development, as well as its portability on embedded systems for instance. To address those two challenges, researchers have proposed numerous types of models over the last few years [2,3,4,5,6,7,8,9,10,11,12,13,14]. The study of these contributions that propose different battery modeling methods to estimate the SOC in real time, has allowed us to divide them in two main groups: direct and indirect methods. Direct methods consist in low level battery description models, using equivalent physic representations of the battery cell to determine its remaining electrical capacity. On the other hand, indirect methods consist in higher level battery representations that allow the estimation of the remaining electrical capacity in a battery cell through its behavioral aspect.

Direct SOC estimation methods

Among the direct SOC estimation methods, there are models that describe the electrochemical processes in a battery cell, and others that describe only its electrical circuit. As mentioned in [4], electrochemical models are often accurate but require a complex configuration of parameters as well as the deployment of complex differential equations. The electrical models, on the other hand, represent the battery as equivalent electrical circuits. In [5], a survey on equivalent electrical circuit models for different battery types (e.g. Li-ion and LiPo), is presented. It has been demonstrated that using a common model (i.e. Thevenin model) for all the battery types, leads to good generalization and a simpler model implementation compared to electrochemical models. However, electrical models still demand a low level description and remain quite complex compared to a more abstracted representation of the battery, based on indirect estimation methods that would be accurate enough.

Indirect SOC estimation methods

Regarding the indirect SOC estimation methods, a review of the different categories has been carried out in [6]. This work presents a classification of SOC estimation methods into four categories: measurements, book-keeping estimation, adaptive systems and hybrid.

-Measurement methods use observable variables such as terminal voltage and impedance to describe the relationship with the battery SOC. The most common measurement method is Open Circuit Voltage (OCV). This method computes the difference of electrical potential between two terminals of a device when disconnected from any circuit. Then, a linear approximation between these voltage measurements and the battery remaining capacity (SOC), can be done [7]. The main issue of OCV is the long duration the battery needs to be relaxed to reach its equilibrium before performing measurements. This method is therefore not suitable for online applications [2].

-Book-keeping estimation methods use the discharging current to estimate battery SOC. The most common book-keeping method is Coulomb Counting (CC), that measures the discharging current and then integrates it over time to estimate SOC [8]. CC offers an excellent linearity and short-term performance, thus being suitable for real-time SOC estimations. However, its main limitation is the accumulation of measurement errors that has a non-negligible impact on the prediction accuracy.

-Adaptive systems methods are considered as data-driven mathematical SOC estimation methods. In this category, several artificial intelligence AI algorithms take part, SV M and ANN [3]. The main advantage of AI methods is their adaptability to non-linear conditions. Nevertheless, AI techniques intended to be implemented in embedded systems are limited by the large number of data required to train a model as well as the important amount of required computational resources [9]. Apart from common representations of ANN such as the well-known multi-layer perceptron (MLP), recurrent neural network (RNN) approaches have proven to be useful for SOC predictions [3,10,11,12]. RNN have indeed the ability to represent time series and sequences in general. Long short-term memory neural networks (LSTM), a more sophisticated form of RNN, has demonstrated to be very efficient in learning the long-term aging characteristics of rechargeable batteries [13,14].

-Hybrid models are intended to combine the benefits of two methods, thus obtaining an accuracy improvement in battery SOC estimations. It is worth pointing out that the fuel gauge embedded on the smart glasses [15], implements an algorithm based on an hybrid battery SOC prediction model. This algorithm combines the benefits of both OCV and CC methods to provide accurate SOC estimations in real time.

The choice on the method that is best suited to our needs, strongly depends on the precision to reproduce the discharge process of the battery embedded on the smart glasses. Moreover, the chosen method must offer a low implementation complexity suited to our a system-level modeling methodology and allowing simple and rapid simulations of the smart glasses to be performed. In consequence, we consider that adaptive systems or data-driven methods fit both requirements, reproducing the discharge process of the smart glasses and maintaining a high level of abstraction. In that context, we propose to assess the benefits of implementing data-driven regression models through an experimental exploration, to find a model able to predict accurate battery SOCs. To do so, a data collection process has been carried out with the glasses to train regression models. An analytical system-level regression model based on the collected data has been developed to perform SOC predictions. This analytical model outcomes a group of regression models based on machine learning that have been tested during the experimental exploration. The novelty of our approach consists in an analytical SOC prediction model for the simulation of discharge processes, reproducing the embedded real-time conditions and considering the battery aging phenomenon.

Battery testbench

The data collection process has been performed using a battery testbench. This battery testbench has been designed with the purpose of executing automatic and continuous charge/discharge processes. To do so, the battery testbench shown in Fig. 5.1 has been developed. On the left of the figure, we can observe the schematic of the battery testbench, whereas on the right, a picture of the real implementation using five prototypes of smart glasses is shown.

Experimental setup

As can be seen in the schematic on the left, five pair of glasses have been used (G1 to G5), each of them being connected to a charger at one of the ends (C1 to C5), for charging the battery when empty. At the other end of each pair of glasses, an It is worth noting that a rate of 1 h or less is considered as very stressful conditions for LiPo batteries (as the one used with the smart glasses), thus impacting the safety operation and accelerating their aging. For the sake of visibility, the Fig. 5.2 shows the measured charge and discharge rates by testbench glasses. As can be seen, the time spent in every charge and discharge processes is slightly higher than the configured rates presented in Table 5.1. The reason is that the smart glasses the power consumption is not constant over the time. Instead, in the charge process there is a period of relaxation provoked by the charger embedded on the glasses when the battery reaches a certain threshold (e.g. 90%). At this point, the charging current is decreased to reach a full charge (i..e 100% SOC) at a lower pace, thus increasing the battery lifespan. Regarding the discharge process, the rates have been configured through fixing hardware settings on the glasses. Each pair of glasses has a specific hardware configuration that allows a fixed power to be consumed from the beginning until the end of a discharge process. Configured settings include the MCU at a specific CPU frequency, the RGB LEDs at a specific light intensity, and several sensors at specific sampling frequencies. The power consumption of each configuration has been measured to validate that the nominal rates presented in Table 5.1, are well estimated.

In Fig. 5.2, the charge and discharge duration values correspond to the first cycle of every pair of glasses. In this figure we can also observe the duration time of two other pair of glasses, one of them configured to execute the driver drowsiness detection application (DDDC), and the other one configured for the fall detection application (FALLC). It is worth noticing that both DDDC and FALLC charge duration times are higher than those configured for the glasses TB-1 to TB-4. Regarding the discharge duration time, it is also higher than those configured for all the glasses testbench. This comparison aims at demonstrating that the real usage of the smart glasses is less stressful than the experimental conditions carried out with the battery testbench.

Battery testbench results

The battery testbench has been executed for around five continuous months, the number of cycles performed by every pair of glasses being presented in Table 5.1.

Through the execution of these experimental charge/discharge processes, a dataset has been created. An extract of this dataset is presented in Fig. 5.3. As can be seen, after 500 charge/discharge cycles, the maximum capacity available in the battery is around 80% of its initial capacity (while it is only 60% for TB-1 and TB-3 glasses).

It is worth pointing out that the chosen one-cell pack LiPo battery to be embedded in the smart glasses, has a nominal capacity C BATT equal to 95 mAh, delivering a 3V7 nominal voltage. The information provided by the battery constructor lacks of details about battery lifespan and aging. The unique information is that in the nominal charge/discharge range (between 0.1C and 0.5C), the battery performs 500 charge/discharge cycles before reaching 80% of its nominal capacity. We can therefore conclude that the obtained aging results using the battery testbench match the information given by the manufacturer when using charge and discharge rates between the nominal range. As shown in Fig. 5.2, both applications addressed with the smart glasses (DDDC and FALLC) follow the aforementioned nominal range.

In consequence, the battery aging of the smart glasses can be modeled using the average aging curve (moderate stress group) presented in Fig. 5.4.

Battery SOC prediction: data-driven regression models

To perform SOC predictions during discharge simulations of the smart glasses and using analytical power models presented in chapter 3, a data-driven battery model is required. The average power consumption of the system PSYS is the unique available information provided by the analytical power models. So the battery SOC predictions will only be based on PSYS . However, and as presented in the previous section, PSYS is obtained by calculating the average power over the total discharge duration time. In addition to this, the state of charge at the previous timestamp (SOC (t-1) ) has also been added to this modified dataset. The objective is then to find a regression model that learns to predict SOC (t) every 30 s (T SYS ), while considering the aging aspect and having as inputs the number of cycles, SOC (t-1) and PSYS . T SYS being the period of the system that controls the execution of a SOC prediction.

Machine learning models

An exploration study of three machine learning models has been carried out to find a battery regression model that fits the requirements explained above. These three The precision of SOC predictions is evaluated using the Mean Absolute Error (MAE).

This metric is a measure of errors between the estimated values and the actual values. MAE uses the same scale as the data being measured, thus making easier the interpretation of results. In the context of the proposed system-level modeling methodology, the SOC prediction must be performed at every system period (T SYS ).

Let suppose the simulation of a discharge process of the smart glasses. At the beginning of the simulation (t = 0), the SOC prediction battery model receives as input the average power consumption of the system ( PSYS ), the number of cycles already executed by the battery as well as the initial battery capacity (SOC (t) ). Once the first period has elapsed, the battery model must forecast the new battery capacity SOC (t+T SYS ) , as shown in the algorithm 12. As mentioned in [16], "forecasting is a sub-discipline of prediction in which we are making predictions about the future on the basis of time series data". In other words, forecasting predicts the future value for the time series, looking at its unique trends. For example, predicting next year revenues for a company, based on the revenues (i.e. time series data) from 10 years prior.

In the context of the time series data from the battery discharge process, the SOC prediction takes the test dataset and performs a prediction for each input vector (every input vector is composed of a value of PSYS , number of cycles and SOC (t-1) ). Then, the predicted output is compared to the actual output for each input vector. At the contrary, the SOC forecasting takes only the first input vector ( PSYS = 25mW, number of cycles = 250, SOC (t=0) = 100%) to predict SOC (T SYS ) , SOC (2 * T SYS ) , SOC (3 * T SYS ) ,..., SOC (n * T SYS ) , where n is the number of forecasts that must be performed to reach a battery SOC of 0%.

In supervised learning, a prediction concerns the estimation of the target variable for unseen data. To do so, a regression model is fitted to a training dataset, thus allowing estimations for new samples. At the opposite, forecasting only concerns a single prediction about the future after considering a certain number of previous events. The goal in the context of the simulation is to continuously forecast SOC by using the previous forecast as the input for the next one, until the end of the simulation (e.g. SOC MI N = 0%), as shown in the algorithm 12.

Algorithm 12 : SOC forecasting As a battery discharge is represented by a negative slope from 100% to 0% of SOC with respect to time, forecasting SOC can be seen as a sequence. According to this, the precision of the studied regression models is evaluated for both a prediction and a forecasting scenario.

Linear Regression model

Linear regression allows modeling the relationship between a dependent variable and one or several independent variables. In our case, the dependent variable is represented by the SOC to be predicted, whereas PSYS , the number of cycles and SOC (t-1) are considered as the independent variables. Using a linear regression allows estimating a linear correlation of each independent variable and the dependent one. As can be observed, the precision decreases (MAE = 14.34%) with respect to SOC prediction results. The SOC forecast sequence does not maintain the same pace with the actual sequence.

MLP model

A multi layer perceptron or MLP is the simplest class of feed forward artificial neural networks (ANN). As can be seen in Fig. 5.9, MLP consists in fully connected neuron layers (an input layer, one or several hidden layers and an output layer). By modifying weights between neurons according to a back-propagation algorithm, MLP learns solving complex classification and regression tasks.

As for linear regression, a MLP regression model has been trained using the dataset from the battery testbench. The MLP architecture is composed of one-neuron input layer, two 32-neuron fully connected hidden layers and one-neuron output layer.

LSTM model

Recurrent neural networks (RNN) are a type of ANN specially designed for sequence problems. A RNN uses the output of the network as an input with the next input vector, as shown in Fig. 5.11 (X t being the input vector at the time t and h t being the output). As can be seen, the prediction of h t also considers the previous prediction (h t-1 ). For an unseen discharge process, LSTM is capable of predicting the SOC with a MAE error = 0.47%, while the MAE error = 7.03% for the SOC forecasting. However, compared to Linear Regression and MLP models, LSTM exhibits a significant improvement in predictions and forecasting accuracy, thus confirming the relevance of LSTM for time series. However, even if the forecasting accuracy has been improved using LSTM, it remains low with respect to the prediction accuracy. This is likely due to the average power PSYS used during training, which is higher than the average power used for testing the model. Another possible reason is that PSYS could have a low impact on both the SOC and the aging. Regarding the forecasting execution time and as shown in Table 5.2, LSTM can provoke an important latency on the simulation of a scenario (i.e. 65 seconds).

Discussion of results

There is still much to explore for SOC forecasting to allow us to propose a mature ML-based battery model offering the expected results in terms of accuracy and latency. In the context of this thesis, we have finally decided to propose an analytical battery model based on the nominal capacity of the battery and using the testbench collected data only for the aging prediction. This choice is motivated by our willingness to obtain a better accuracy results for battery SOC forecasting.

Proposed analytical approach for SOC forecasting

To estimate the duration of a discharge process using a simple modeling approach, we can compute the energy consumption at every time interval T SYS , and obtain the new available battery energy capacity (E (t+T SYS ) ) by subtracting the previous available energy (E t ) with the new energy consumption (E T SYS ). The Equations 5.2 and 5.3 resume this approach.

E (t+T SYS ) = E t -E T SYS (5.
2)

E T SYS = PSYS * T SYS (5.3)
However, in this work, from experimental practices we obtain discharge processes that are expressed in percentage of battery SOC. For this reason, we have decided to propose an analytical battery modeling approach based on the prediction of SOC.

The battery SOC can be defined as the ratio between the current available electrical capacity C (t) and the maximum nominal capacity (C n ) [6], as shown in equation 5.4. ). Throughout the execution of this scenario (from SOC MAX = 100% to SOC MI N = 0%), we have the same power consumption PSYS . In this case, the elapsed time to get an empty battery (t cuto f f ) is equal to the scenario duration (t END ), and can be estimated using the Equation 5.5.

t cuto f f = t END = C n PSYS (5.5)
As mentioned at the beginning of this section, the simulation of a scenario requires the estimation of the battery SOC at every system interval (T SYS ). So, the estimation As can be seen, this is an example of a discharge process estimation using the proposed model. Every point of the discharge curve (from SOC MAX = 100% to SOC MI N = 0%) corresponds to an estimation of SOC (t+T SYS ) . To do so, we first use the Equation 5.5 to determine the time to fully discharge the battery (i.e. time-to-cutoff).

Using this information, we can then compute the reference time using the Equation 5.7 to know the required time to decrease the SOC by 1% SOC. As illustrated, we can compare this reference to the duration interval T SYS . In this example, T SYS is equivalent to a half of t 1% . Using this relationship, we can then estimate the percentage of SOC T SYS that is lost every duration interval. As can be observed, the battery decreases by 0.5% of its capacity every T SYS . Finally, using this information we can compute the Equation 5.6. This procedure can be applied in the algorithm 12, by replacing the line 9 with these four equations. It results in the algorithm 13.

Algorithm 13 : Analytical SOC forecasting The proposed analytical model is therefore based on the nominal electrical capacity of the battery C n , the average power consumption of the system PSYS and the duration interval T SYS .

Executing SOC forecasting using this analytical model differs from forecasting performed by the ML-based models presented before. The difference is that the MLbased models have seen the training data, thus automatically learning the aging phenomenon present in the dataset. In the case of the analytical model, the SOC forecasting is only based on the nominal electrical capacity, meaning that the aging is not presented to this model. As can be seen in Fig. 5.15, the SOC forecasting of the discharge process in the test dataset is done using this analytical model. However, the obtained result (MAE = 8.59%) is quite different from the actual measures. The reason is that aging (i.e. 250 cycles) is not considered.

As mentioned in section 5.3.2, the battery aging behavior obtained from the testbench can be used for modeling this phenomenon. An exploration of three different models has been carried out on the average aging considering glasses TB-2, TB-4 and TB-5 (i.e. the red continuous line plotted in Fig. 5.4).

The first model consists in a linear function that represents the aging evolution as 1% of aging every 25 charge/discharge cycles, as shown in the Equation 5.9. The aging modeling results obtained using those models are presented in Fig. 5.16.

As can be seen, the quadratic function matches the best the real aging curve. Estimations of the mean absolute error (presented in Table 5.3) have been computed in order to obtain a more formal comparison. As can be observed, the model based on quadratic function allows obtaining the best accuracy, even though batteries suffer from aging in a more important pace (between 0 and 25 cycles). The integration of the three studied aging models into the analytical SOC forecasting model presented in the algorithm 13, has allowed a considerable improvement on results accuracy. In Fig. 5.17, we can observe that the gap between the actual SOC and the predicted one using the analytical model (presented in Fig. 5.15) has decreased. The reason is that the battery aging (250 charge/discharge cycles) has been taken into account in the SOC forecasting process. In addition to this, the proposed analytical methodology exhibits around 4% of improvement compared to ML-based results, where the best accuracy can be obtained using the LSTM model (MAE = 7.03%). Finally, in terms of forecasting execution time, the proposed analytical model takes less than 1 s, thus not causing any latency on simulations.

Conclusion

In this chapter, we have presented a complete analysis of the battery discharge behavior on the smart glasses. Through this analysis, we have been able to propose

an analytical system-level battery modeling methodology for SOC forecasting. This data-driven method allows considering the aging phenomenon of the battery. Using a battery testbench has allowed us to study the aging of the battery integrated into the smart glasses. Moreover, collected data have served as a training dataset for exploring machine learning regression models to learn the characteristics of the battery discharging process, and therefore estimate the SOC at every system period.

Even though a lot of time has been invested in this exploration, results in terms of accuracy and inference time were not satisfactory. We therefore proposed an analytical system-level battery model. Experiments performed using this approach showed that a better accuracy can be achieved compared to the machine learning methods, thus proving the relevance of our model. The SOC forecasting model together with the analytical power consumption models presented in chapter 3, have been implemented in a simulation environment to simulate typical scenarios of the smart connected glasses. This simulation environment as well as the results obtained for three different scenarios are presented in the next chapter.

Chapter 6

System-level simulations for smart connected glasses

Introduction

A framework has been developed to perform high-level simulations of scenarios.

This simulation flow is based on the analytical power models as well as the battery SOC models presented in chapter 3 and chapter 5, respectively. The analytical power models of the different modules (e.g. microcontroller, RF device, sensors) involved in a defined use case are used to estimate the corresponding overall system average power consumption ( PSYS ). This estimation can also be obtained for a specific parameters configuration. The objective of this framework is to rapidly get useful metrics (such as the autonomy or the suitable battery capacity) for typical Ellcie-Healthy smart connected glasses use cases.

In the next sections, the software architecture of the simulation framework as well as the implementation choices are first presented. Then, it is explained how to describe and run scenarios using configuration files and the simulator graphical user interface (GU I). We will also pay a specific attention to the simulation time management.

In the section 6.3, three scenarios lasting for 2,17 hours, almost 2 days and more than 10 days respectively, are executed to get an estimation of the remaining SOC.

Predictions are then compared with real measurements obtained using the smart connected glasses for the same scenario. This chapter ends with some discussions on the achieved estimation results.

Software Architecture and Implementation choices

The simulator has been implemented using the Python programming language. The software hierarchical organization of the simulator is shown in Fig. 6.1.

As can be seen, the code is organized in five main software modules. Each module is in charge of executing a specific functionality of the simulator. These modules are listed below: -scenario_controller.py : Create, schedule and execute the selected scenario.

-\U I : The code that implements the GU I is stored in this directory. Note that the Qt Designer tool has been used to create the user interface.

-\power_models : This directory contains the different power models described in the Chapter 3. These power models are implemented as independent software modules (e.g. sensing_module). During the execution of a scenario, the power_models_controller.py module is in charge to provide power estimations to the simulation_controller for each system state.

-\con f ig\scenarios.yml : This file contains a description of the different scenarios in yaml markup language as it will be explained in section 6.3.

Users can interact with the simulator through the GUI. Several parameters can be easily modified through the graphical user interface such as the scenarios to be simulated or the battery settings. The GUI will be described in details later in the chapter.

Scenario description using Finite State Machines

Another input of the simulator is represented by the scenario description. The user can indeed define a scenario as a FSM. We have decided to use the pytransition / transitions python library to define and execute FSM. A scenario, written using the yaml language in our simulator, is a high-level description of the behavior of the smart connected glasses listing the different states through which the system passes.

It is worth noticing that any scenario of the smart connected glasses can be described using only 7 system states (i.e. SLP, SLPC, DDD, DDDC, FALL, FALLC and PDN). An example of a typical scenario is shown in Fig. 6.2. As can be seen, the yaml description is composed of four main parts:

-The name of the scenario. This name will be automatically displayed on the GU I in the list of available scenarios.

-The list of all states defined for a scenario. The different system states that can be modeled have been presented in the Chapter 3. Each state has a name and a duration. The state duration is expressed in seconds using the tags mechanism of the pytransition/transitions python library. For instance, the state DDDC_1 has a duration of 3600 seconds. It is worth noticing that each state must have a unique name (it is an implementation constraint). Therefore, when the system enters in the same state several times, the name of that state must be postfixed with the corresponding increment. This is what happens with the states FALLC, SLPC and DDDC of the scenario defined in Fig. 6.2. For example, as the FALLC state is executed five times during that scenario, we have to define FALLC_1 to FALLC_5 states. The tags mechanism is used to indicate to the simulator that those post-fixed state names correspond, in fact, to the same system state.

-In transitions, the user can list all the transitions between states. It is worth noting that the pytransition/transitions library allows using a wildcard (denoted as * ) to indicate a transition from, or to, all the defined states. As an example, the SOC_min transition indicates that there is a transition to the PND state (i.e.

Power Down) from all the states. To simplify the transitions between states, we decided to implement ordered FSM of the pytransition/transitions library. In -The last part, initial, allows the user to indicate the initial state of the defined scenario (e.g. FALL_1 for that scenario).

This yaml description is directly interpreted in the scenario_controller.py module to construct and then execute the FSM. The FSM corresponding to the scenario defined in Fig. 6.2 is partially displayed in the Fig. 6.3. Note that displaying the FSM within the simulator graphical user interface has not been implemented yet.

Scenario execution is managed by a software module, called scenario controller. The main purpose of this module is to schedule the execution of the FSM described in the scenario configuration file. Since our simulator allows modeling timed FSM, we have developed a strategy to manage simulation time. We use a fixed quanta of time, called tick, that represents the minimum amount of time that can be modeled in our simulator. In other words, the system cannot remain in a given state for less than the duration of a tick. Tick duration can be selected, from the GUI, at the beginning of the simulation. We offer the choice between two tick values: 5 or 10 minutes. Despite these duration could be considered quite long, this choice has been motivated by the fact that our goal is to simulate the system for several hours or even several days. Obviously, for a given scenario duration, the simulation time to execute this scenario increases when the tick duration decreases. As an example, to simulate a 1h scenario, the simulation controller must wait for the iteration of 6 ticks of 10 minutes or 12 ticks of 5 minutes.

To manage FSM scenario execution we use the Advanced Python Scheduler (AP-Scheduler) library. This library allows Python code to be executed periodically, and provides an API to manage job execution (start, stop, pause and resume). The core of the simulator is run by APScheduler at a fixed time interval. At each iteration we increment the internal simulator time by a tick, and we then reschedule (if needed) the next state of the FSM. This fixed time interval is currently set to a value of the order of milliseconds, leading to a speed-up factor of several thousands compared to the simulated time. According to the time spent in each state, and thanks to the power and SOC models, we can then compute the energy consumed by the system and the SOC, as we are going to discuss in the next section.

Running the scenario under the simulator

The user interface provides control and visualization of the simulation. The different GUI components, highlighted in Fig. 6.4, are described below.

1. The Scenario Settings provides pulldown menus to select the scenario and the tick that will be used to run the simulation. It is also possible to define an end simulation time (lower than the scenario duration). In this case, the simulator will stop when the simulated time exceeds the time given in the menu.

2. The Battery Settings menu, provides the control over the battery parameters.

The User can specify the battery nominal capacity, the aging (number of charging/discharging cycles), the state of charge of the battery at the beginning of the simulation and the battery minimum SOC. Note that the simulation will stop as soon as the current SOC reaches the minimum value.

3. The simulation execution is controlled through four buttons. As can be seen, it is also possible to pause and resume the simulation.

4. A progress bar shows the progression of the simulation. Text output, such as the current state and other simulation information, is displayed on the console underneath the progress bar.

5. A series of four buttons allows the user to clear the simulation output, export or save the simulation results and quit the simulator.

6. Simulation output is visualized through a series of plots. We used the pyqtgraph library that provides an API to plot data in real-time during the simulation. As can be seen from Fig. 6.4, the SOC, the instantaneous power and the current state of the system are displayed.

In the next section we will show some examples of simulated scenarios and compare our results with experimental data. The SOC evolution is shown on Fig. 6.8. As it can be observed, the experimental and simulated results are well aligned, even if the autonomy is slightly under-evaluated in the simulation. We measured a real autonomy of 44h20, while in simulation the battery time-to-cut-off is estimated at 40h15 hours.

This difference of a little more than 4 hours between the measure and the simulation of the SOC corresponds to an error of 9.2%, which can be considered as acceptable for such a scenario duration. As the estimated lifetime is lower than the real one, it means that our power models are pessimistic. In fact, the SOC of the battery is estimated by subtracting at each simulation period (T SYS = 5 min), the value of SOC that is lost during this time interval. This method leads to an accumulation of errors coming from both the power models and the battery model. For the first scenario (i.e.

25 min in SLPC and 1h45 in DDDC), we have demonstrated that the power models are optimistic. In consequence, the main source of the accumulation of errors could come from the fall detection states (FALL and FALLC). This represents therefore an important output for future optimizations.

Scenario 3: Drowsiness detection (10 days)

A typical use of the drowsiness detection application is when a person goes to, and then comes back from his workplace every day. In between the driving periods, the drowsiness application is stopped and the smart glasses enter in sleep mode to save battery. We modeled this kind of scenario for a period of almost 10 days, where the drowsiness detection application is activated twice a day for around 40 minutes.

The scenario starts with a fully charged battery and ends when the SOC reaches 0% (i.e. the minimal threshold). In Fig. 6.9 we can observe the associated scenario description.

The comparison between experimental and simulation results is shown in Fig. 6.10.

As can be observed, the two SOC curves have overall a similar behavior. We can also see that the simulations still underestimate the battery time-to-cut-off, meaning that our models are pessimistic. We measured experimentally an autonomy of 255h30 (10.6 days), while an autonomy of 233h30 (9.7 days) is estimated. The simulation error is therefore equal to 8.6%. This error can also be considered as acceptable since we are estimating the SOC on a scenario that lasts for more than ten days. Looking at the curves of Fig. 6.10, we can observe that from tick 500 to almost tick 2000 (i.e.

2 days duration), the simulations are overestimating the battery discharge. During this interval, the system is in sleep mode. So, it probably means that we overestimate the power consumption of the sleep mode. In this state, the power consumption is very low, and thus difficult to characterize with a high level of precision. However, we can expect to further improve the simulation accuracy with a more accurate power model in sleep state. Table 6.1 summarises the results obtained experimentally and using the simulator.

We have also indicated the required simulation time to execute each scenario. Note that the simulation is performed on a laptop PC equipped with an Intel Core i7 and 16GB of RAM. As it can observed, it takes only 9 seconds and less than 15 minutes to simulate scenarios that last for 2 hours and more than 10 days, respectively.

Comparing the simulation time with respect to the scenario duration, we observe a speed-up factor of almost 1000. Moreover, from the error between measurements and simulation we can infer that the accumulation of errors due to the estimation of battery SOC every T SYS , is more important when the fall detection application takes part in the scenario. These results deserve further study in the future.

Conclusion

The system level simulation tool developed during this thesis and discussed in this chapter can be used by a system designer to get many insights during the early stages of the system design. It is well known that product development is subject to stringent time-to-market constraints. Designers must be able to study early in the design phase the technical trade-offs before entering into the industrialization phases. Using our simulator, a designer will be able to estimate the autonomy for a

given system configuration, explore the effect of different battery parameters such as the capacity or the aging, and much more. Moreover, as we have shown in Tab.

6.1, the tool allows long scenarios to be rapidly simulated. It takes only 14 minutes to simulate a 10-day scenario.

Unfortunately, we did not have enough time to propose and implement power management strategies. As future works, we propose to implement different power management policies that can be selected through the GUI like other system parameters. For example, the detection algorithm can be selected according to the required autonomy: the smart glasses can start the drowsiness application using the CNN algorithm, and then switch (when needed) to the less consuming threshold-based detection algorithm. We also let the implementation of the performance and QoS models as a future work. This could be done by giving the user the choice to select different HW configurations (e.g. IR LED currents) or different SW modules (e.g. the type of blink detection algorithm).

Chapter 7

Conclusion and perspectives

Conclusion

During the last three years of this thesis, we have addressed a complex problem which is the joint analysis of energy consumption and performance for smart connected glasses. Wearable systems are, as expressed throughout this manuscript, very constraining systems with harsh limitations in terms of size, available energy, computational resources and memory capacity. Taking this into account, we have proposed a methodology to facilitate the analysis of this problem early in the design flow. We have first proposed a modeling approach for the energy consumption of the smart glasses (in chapter 3). We have then characterized the performance of these smart glasses (in chapter 4) to evaluate the impact of system parameters on the QoS for the Driver Drowsiness Detection application. This system-level approach makes as much abstraction as possible of details that are not closely related to the behavior and the energy consumption of the system. We have finally presented a system-level battery modeling approach (in chapter 5) to rapidly estimate the SOC of the battery after a certain period of time.

This approach has been consolidated in a global methodology that allows modeling and simulating the smart glasses. This methodology is now available to the Ellcie-Healthy company through the implementation of a simulation environment (in chapter 6). The execution of typical scenarios using this environment has proven the relevance of the proposed approach. Only a few minutes are indeed needed to get precise results about the autonomy of the system for scenarios that can last for days. To summarize, this work has resulted in the following contributions:

1. A system-level power consumption modeling approach based on analytical power estimations from the electrical characteristics provided by manufacturers of different hardware modules. This work has been published in an IEEE international conference [1].

2. The integration of performance constraints into the system-level power consumption modeling approach, to study hardware settings impacting the most the smart glasses energy consumption and performance. This work has resulted in a second publication in an IEEE international conference [2].

3. The implementation of a CNN-based eye blink detection algorithm to improve driver drowsiness detection user experience. We have demonstrated the feasibility of implementing a deep learning approach at the edge, by dealing with the imposed embedded system constraints. This work has been published in an IEEE international journal [3].

4.

A system-level battery modeling approach for SOC estimations based on a data-driven method for modeling the aging phenomenon. The aging has been modeled from data collected using a battery testbench that automatically performs charge and discharge processes.

5. Finally, a simulation environment integrating these different models has been developed. The execution of scenarios with this simulator has shown that the autonomy of the system can be estimated with high accuracy, thus proving the relevance of the proposed methodology.

Through these contributions, the company has now at its disposal a tool to explore various aspects for its smart glasses. Designers have the possibility to evaluate existing or future systems leveraging different parameters such as the battery capacity or/and aging, scenarios, simulation period, or hardware modules. This evaluation concerns for example:

-The study of system's hardware settings that allow the autonomy constraints defined by project managers to be respected.

-The analysis of the impact of a new battery capacity. It is for instance possible to estimate the best suited battery capacity, i.e. the one having the minimum size while respecting autonomy constraints.

-The analysis of atypical scenarios (e.g. corner cases or long-lasting cases).

-The study of the contribution of each application task or hardware module on the energy consumption.

The proposed system-level methodology is generic as it could be quite easily extended to different types of wearable systems. A wearable can indeed be defined as a system composed of at least a microcontroller, a RF interface and several sensors.

In our approach we make abstraction of many important states participating in the execution of an application with the smart glasses. For example, transition states used (internally) by the BLE device to transmit a packet have been ignored. Our motivation was to rapidly describe and execute scenarios that can last for days. To do so, we chose a granularity adapted to this modeling approach. Adopting such a system-level abstraction can indeed provides accurate estimations and low simulation times. With the proposed methodology, a 10-day scenario can be described using only 8 macro states (DDD, DDDC, SLP, SLPC, FALL, FALLC, ACT, CONN).

To model another wearable system (e.g. a smartwatch), new macro states should be defined according to the application. Then, application tasks to be executed during the simulation of each macro state must be identified and associated to hardware components. Finally, the model of each component must be defined according to its datasheet. Obviously, the SOC forecasting model should also be adapted if the system is equipped with a new type of battery (e.g. Lithium-ion).

Our approach can still be improved on different aspects. This work thus opens up several perspectives in the short, medium and long term. The next section is dedicated to the description of some perspectives.

Perspectives

As a short-term perspective, our approach could be extended to the point of view of the performance. So far, our simulation environment does not provide any ability to jointly explore the autonomy and the QoS. For instance, the choice of the drowsiness detection algorithm is statically chosen at the beginning of the simulation. Scenarios presented in the chapter 6) have been executed using only the threshold-based eye blink detection algorithm. However, it would be interesting to evaluate the impact on energy consumption (i.e. autonomy) of the CNN-based algorithm, that we know it provides better performance but requires more energy. Therefore, we propose to extend our simulation environment to allow users to select the type of algorithm. We could also add performance constraints (e.g. a required average detection accuracy)

to allow an automatic evaluation of the system performance. Similarly, our simulation environment could be extended to allow users to manually choose among different hardware settings such as the RF transmit power, IR LED power supply, etc. Finally, we could extend much more our simulation tool by adding the generation of pie charts to help designers performs Pareto comparisons in terms of power consumption contributions, by application tasks and by hardware component.

As a medium-term perspective, we plan to perform online optimizations of the smart glasses settings. To do so, a power manager needs to be defined and implemented. This power manager could involve different strategies. We can indeed imagine having different policies that favor either the autonomy, or the performance, or both of them. To do so, the power manager has to determine the right settings for both the hardware and the software.

-Hardware settings: The objective will be to take into account different hardware settings according to their influence on the performance and the autonomy. The system configuration could then be dynamically adapted to meet defined performance constraints, thus controlling the QoS during the execution of a scenario.

-Software settings: An application can be composed of different versions of algorithm, each of them having a different impact on the energy consumption and the QoS. So, the power manager will dynamically decide to execute an algorithm or another one according to the objective (or constraints) in terms of autonomy and QoS.

As a medium-term perspective, the battery modeling approach also requires further investigation. First, more heterogeneous types of battery must be considered in the battery testbench. The objective is indeed to propose a better and more general model of battery. To do so, the execution of new charge/discharge protocols is also needed. These new protocols can be richer in terms of collected variables and the range of selected charge/discharge current consumption, as well as the capacity and the type of battery. This can in fact, allow us to:

-Validate the proposed approach for a range of LiPo battery capacity (e.g. from 80mAh to 200mAh).

-Extend this model to other types of rechargeable batteries commonly found in wearable systems (e.g. Lithium-ion).

-Leveraging machine learning and deep learning approaches to improve the model accuracy. The objective will be to get a model as general as possible for the battery discharge process and considering the aging.

As a long-term perspective, we plan to automate some tasks related to the description of scenarios. Even though the task related to the description of a scenario is not so long and painful, it would be nice to define scenarios using a more friendly graphical interface (rather than the yaml language). This interface would allow users to construct their own scenario FSM using a list of available states and transitions (for a type of application). Note that displaying the finite state machine of scenarios is also a nice-to-have feature that we would like to add in the future to the simulation environment. Finally, we could also imagine to design a wearable by assembling different hardware and software modules available from the user interface.

2. 2

 2 State-of-the-art: Drowsiness Detection Systems . . . . . . . . . . . . . . 2.3 State-of-the-art: Fall Detection Systems . . . . . . . . . . . . . . . . . . 3.1 IMU: power consumption information provided in the manufacturer datasheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Pressure sensor: Power consumption information provided in the manufacturer datasheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Humidity and Temperature sensor: Power consumption information provided in the manufacturer datasheet . . . . . . . . . . . . . . . . . . 3.4 MCU: power consumption information provided in the manufacturer datasheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 FLASH: power consumption information provided in the manufacturer datasheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 BLE connection interval current consumption without payload I CON_NP (t active CON_NP = 1.35 ms) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7 Current consumption given by manufacturers of the BUZZER, RGB LED and LED Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8 Current consumption given by manufacturers of the CHARGER and FUEL GAUGE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.9 DC/DC Regulators: power consumption characterization (using V BATT = 3.7 V) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.10 System configuration by discharge process . . . . . . . . . . . . . . . . 3.11 Autonomy results by configuration . . . . . . . . . . . . . . . . . . . . . 4.1 Performance and Autonomy by System Configuration . . . . . . . . . . 4.2 Considered events in training dataset . . . . . . . . . . . . . . . . . . . 4.3 Architecture and Accuracy of best CNN models . . . . . . . . . . . . . 4.4 Model size, ROM footprint, execution time, power consumption and autonomy by blink detection algorithm . . . . . . . . . . . . . . . . . . 4.5 Performance metrics by blink detection algorithm . . . . . . . . . . . . 4.6 State-of-the-art: Drowsiness Detection Systems . . . . . . . . . . . . . . 5.1 Battery testbench: charge/discharge profiles configuration . . . . . . . xviii 5.2 SOC prediction vs forecasting using machine learning . . . . . . . . . . 5.3 Aging modeling using 3 different regression models . . . . . . . . . . . 5.4 SOC forecasting using the analytical model . . . . . . . . . . . . . . . . 6.1 Comparison: Experimental vs Simulation results . . . . . . . . . . . . .

FIGURE 1 . 1 -

 11 FIGURE 1.1 -IoTSource:[1]

FIGURE 1 . 2 -

 12 FIGURE 1.2 -Number of active IoT devices WorldwideSource:[4] and[5]

FIGURE 1 . 4 -

 14 FIGURE 1.4 -Moore's Law: Number of transistors on ICsSource:[8]

FIGURE 1 . 6 -

 16 FIGURE 1.6 -Ellcie-Healthy Smart Connected Glasses

FIGURE 1 . 7 -

 17 FIGURE 1.7 -Phases of a product design flow

FIGURE 1 . 8 -

 18 FIGURE 1.8 -First prototype of Smart Glasses

FIGURE 1 . 9 -

 19 FIGURE 1.9 -System-level modeling methodology

3 .

 3 The study of a Convolutional Neural Network (CNN)-based eye blink detection algorithm to improve driver drowsiness detection user experience, is also proposed. A detailed study of the eye blink detection performance that directly impact the driver drowsiness detection QoS, has allowed to identify an important confusion between blinks and look down events. This confusion provokes false blink detection (i.e. False Positives), causing an overestimation of the driver drowsiness level (and so a bad user experience). The CNN-based eye blink detection has removed most of those false positives, thus reducing the aforementioned overestimation. Additionally, it has been demonstrated

FIGURE 2 . 1 -

 21 FIGURE 2.1 -Ellcie-Healthy Smart Connected Glasses

FIGURE 2 . 2 -

 22 FIGURE 2.2 -Ellcie-Healthy Smart Connected Glasses Applications

FIGURE 2 . 3 -

 23 FIGURE 2.3 -Driver eyelids behavior measurements

  , Rohit et al. explored the feasibility of a driver drowsiness detection system, using a commercial wearable EEG headset called MUSE. Feature extraction from EEG signals spectral analysis is handcrafted done to train both a SV M model and a linear discriminant analysis (LDA) model, for alert and drowsy states classification. Results demonstrate that SV M offers a better accuracy than LDA. Additional exploration has been carried out by analysing blink duration from EEG signals. This study showed that it is a relevant indicator of drowsiness. In this work, experiments have been only performed in simulation. Moreover, feature extraction and classification algorithms have not been executed on embedded systems.

FIGURE 2 . 4 -

 24 FIGURE 2.4 -Events in a blinking signal

Chapter 2 .

 2 Smart Connected Glasses: Applications and Design challenges -The number of ADL detected as falls (i.e. False Positives or FP).

  be placed at the chest and based on a 3-axis accelerometer and a barometer. There are two 3-axis accelerometers within the NEON device, one of them for detecting fall events and the other one used to determine whether the user is in a stationary state or in a motion state. The fall detection algorithm is then executed whenever a motion state is detected. During the experimentation, 2 different trials were conducted. The first one was a simulated fall trial with 20 young healthy volunteers. Several falls were simulated while wearing the NEON device. 200 fall events were collected. A fall event is produced when signals of the 3-axis accelerometer and the barometer contain three classic kinematic characteristics of a fall model: Weightless falling, impact and post-fall inactivity. The second trial was a free-living trial with 10 young healthy volunteers. They continuously worn for a day the NEON-SD, a version integrating a SD memory for logging data. No fall occurred during the trial.

FIGURE 2 .

 2 FIGURE 2.5 -3-axis accelerometer orientation

FIGURE 2 . 6 -

 26 FIGURE 2.6 -Fall detection using accelerometer X-Y axis

FIGURE 2 . 7 -

 27 FIGURE 2.7 -Design of connected glasses hardware architecture

FIGURE 2 . 8 -

 28 FIGURE 2.8 -Global data flow system description

FIGURE 2 . 9 -

 29 FIGURE 2.9 -Smart glasses system-level modeling methodology

  2.10, represents the current version of the Ellcie-Healthy smart glasses.

FIGURE 2 .

 2 FIGURE 2.10 -System level hardware architecture

Figure 3 . 1 defines

 31 the different states that the smart glasses can adopt to deliver a service. These states are: power down (PDN), active (ACT), connected (CONN), sleep (SLP), sleep connected (SLPC), driver drowsiness detection (DDD), driver drowsiness detection connected (DDDC), fall detection (FALL) and fall detection connected (FALLC).

FIGURE 3 . 1 -

 31 FIGURE 3.1 -Smart glasses global finite state machine (FSM)

FIGURE 3 . 2 -

 32 FIGURE 3.2 -Example of smart glasses typical scenario

2 )FIGURE 3 . 3 -

 233 FIGURE 3.3 -Hardware architecture with application tasks

  2.1 and Fig.2.7, infrared proximity sensors are embedded inside the rims of the glasses while remaining sensors (e.g. accelerometer, temperature, humidity, etc.) as well as the 32-bit micro-controller are placed in the hinges of the frame. Depending on the state of the smart glasses (e.g. DDDC), sensor modules are either in power down, sleep, or active mode. In addition to the operation mode, some other settings specific to each sensor, determine the average power consumption of the data sensing task PDST . Each sensor module must be therefore modeled to estimate PDST (See equation 3.

FIGURE 3 . 4 -

 34 FIGURE 3.4 -Synchronous measurements executed by a sensor

  other. In the current version of the Ellcie-Healthy smart glasses, the gyroscope is always disabled (i.e. in power down). So far, the algorithms executed with the glasses are indeed only based on accelerometer measurements. The 3-axis accelerometer orientation embedded in the glasses is shown on Fig.2.5. For the sake of simplicity, the analytical I MU power model only considers the accelerometer as the main source of power consumption. However, the average power consumption of the gyroscope in power down mode is taken into account in our model. The sampling frequency range of the accelerometer goes from 12.5 Hz to 1.6 kHz. The choice of the operation mode depends on the desired sampling frequency. The I MU model receives therefore as input:1. the glasses state (e.g. DDDC), 2. the voltage supply, 3. the selected accelerometer operation mode. 4. the selected accelerometer sampling frequency, Using these inputs, the I MU model computes the average power consumption following the algorithm 1. First, the average power of the gyroscope ( Pgyr ) in power down mode is estimated. The estimation of the accelerometer average power ( Pacc ) is then performed depending on the settings (e.g. sampling frequency, operation mode). Finally, the I MU average power ( Pimu ) is obtained by adding Pacc and Pgyr .

s)/F base_acc 17 : 18 :

 1718 Pimu (s) ← Pacc (s) + Pgyr (s) return Pimu (s) illustrated on Figure 2.3, an IR LED is piloted by the sensor to emit signals that are then captured by a photo-detector by refraction. Two Si1153 IR proximity sensors [10] are embedded on the smart glasses for this purpose. Si1153 is an active optical reflectance proximity detector with an incorporated ambient light sensor, whose operational state is controlled by registers accessible through the I 2 C interface. Sending commands from the host (the I 2 C master, usually a microcontroller), allows the IR sensor to perform proximity measurements. Sampling frequency is set up by the firmware, and it defines the period of time to execute one single proximity measurement. Every measurement is divided into three states in function of the time, as shown in Fig. 3.5. The manufacturer provides the time and the current consumption by state, needed to perform a measurement. Using this information, the average current consumption by measurement (i.e. every sampling period) can be estimated.

FIGURE 3 . 5 -

 35 FIGURE 3.5 -IR sensor measurement time description

TABLE 3 . 3 -= 1

 331 Humidity and Temperature sensor: Power consumption information provided in the manufacturer datasheet Hz and V base = 2.5 V 2 µA Power down V base = 2.5 V I sleep = 0.5 µA According to this, the algorithm 3 represents the implementation of an analytical average power consumption model for both the pressure and the humidity and temperature sensors. Data provided by the manufacturer are: V base , F base , I sleep , I low_power , and I normal_power . Note that the temperature and humidity sensor does not have a low-power mode. Additionally, configurable parameters for the sensors are: The voltage supply (V 1 ) and the sampling frequency (F sensor ). Having this settings information, the estimation of the average power consumption for each instance of this generic sensor model (e.g. pressure sensor Ppress_sensor , humidity/temperature sensor Phum_sensor ) can be computed.

of communication: 1 )

 1 connection-based and 2) broadcast-based. The first mode requires the connection of two devices before exchanging data, while the second one allows broadcaster to simultaneously send data to several observers without connection [17]. A connection-based mode is used between the BLE chip integrated into the smart glasses (i.e. communication slave) and a smartphone (i.e. communication master).

FIGURE 3 . 6 -

 36 FIGURE 3.6 -Advertising and Connection phases of the BLE chip

9 )

 9 Different types of data can be sent from the smart glasses such as raw data from sensors (i.e. data streaming) DATA STR , update of variables DATA VAR (e.g. battery SOC, temperature and humidity) or events notification DATA EVT (e.g. falls, recovery from falls, drowsiness levels and temperature thresholds exceeded). The Equation 3.10 allows therefore to estimate the average power consumption required for the connection phase ( PCON ).PCON = PCON_NP + D ∑ k=1 PDATA (k) → ∀D = 1, 2,3... (3.10) Where PCON_NP is the average power consumption of connection intervals without payload. As mentioned before, connection intervals with payload depend on the data to be sent from the smart glasses to the smartphone. PDATA (k) is therefore the average power consumption of a specific data forwarding type and D the number of transmitted data. Using the Equation 3.10 and considering the different types of data transmission from the smart glasses, the Equation 3.11 allows to estimate PCON . PCON = PCON_NP + PDATA STR + PDATA VAR + PDATA EVT (3.11)

Algorithm 8 :

 8 RGB LED and BUZZER power modeling 1: procedure GET_RGB_POWER(mode DRIVER LED (s))

FIGURE 3 . 8 -

 38 FIGURE 3.8 -DC/DC Regulators: power consumption modeling

  the Equation 3.3 we can define PLOAD as the sum of the average power consumption of the four application tasks (DST, DPT, DTT, ENT), as shown in the Equation 3.16. PLOAD = PDST + PDPT + PDTT + PENT (3.16)

  )

FIGURE 3 . 9 -

 39 FIGURE 3.9 -Modeling validation: analytical vs measured autonomy

Figure 2 . 3 .

 23 Figure 2.3. The raw data coming from both eyes are then merged to obtain a single blinking signal (see Fig. 4.1), thus allowing to analysis the movements of both eyes together.

FIGURE 4 . 1 -

 41 FIGURE 4.1 -Blinking signal for eye blink detection

  4.1). The adaptive procedure uses the signal amplitude of the previous blinks to increase or decrease the detection threshold. In fact, the threshold can be set to a value ranging from 35% to 95% of the average amplitude of the previously detected blinks. As mentioned before, the current blink detection threshold used in the smart glasses has been empirically set, according to the analysis of real data from beta-testers. However, a change in the system configuration results in a change of the detection threshold. To better justify the choice of this software parameter, a study based on a Receiver Operating Characteristic ROC curve[9], has been carried out. In Fig.4.2 we detail the selection process of the blink detection threshold for the eight random configurations presented in Table4.1.

FIGURE 4 . 2 -

 42 FIGURE 4.2 -ROC curve: detection threshold by configuration

FIGURE 4 . 4 -

 44 FIGURE 4.4 -Sensitivity and Specificity by Configuration

FIGURE 4 . 5 -

 45 FIGURE 4.5 -Autonomy/Performance trade-off by Configuration

  FIGURE 4.6 -Dataset: (A) Label 1: blinks; (B) Label 0: non-blinks

  3 corresponds to the CNN baseline architecture presented in Fig. 4.7. After training model C with 30 epochs

FIGURE 4 . 7 -

 47 FIGURE 4.7 -CNN baseline architecture

FIGURE 4 . 8 -

 48 FIGURE 4.8 -Accuracy, ROM memory footprint and autonomy by algorithm

Fig. 4 . 8 ,

 48 the threshold-based algorithm requires only 10.7 kB of ROM, while the smallest CNN model (F) needs almost 59 kB (Model ROM + 47 kB). Nevertheless, all CNN-based models, except model B, respect the maximum ROM constraint of 90 kB. It is worth noting that the number of trainable parameters of a CNN model strongly depends on the number of downsampling filters in each convolutional layer: the fewer convolutional layers, the more trainable parameters.

FIGURE 4 . 9 -

 49 FIGURE 4.9 -Blinks and look down events: Class activation maps (CAM)

FIGURE 5 . 1 -

 51 FIGURE 5.1 -Battery testbench design and development

FIGURE 5 . 2 -

 52 FIGURE 5.2 -Charge and Discharge time

FIGURE 5 . 3 - 1 )

 531 FIGURE 5.3 -Raw dataset from battery testbench discharging cycles

FIGURE 5 . 4 -

 54 FIGURE 5.4 -Battery aging observed in the discharge process

  providing the number of charge/discharge cycles already executed by the smart glasses allows modeling battery aging. In that context, a modified dataset containing only the average power consumption ( PSYS ) and the number of cycles as input has been created. An extract of this dataset is shown in Fig.5.5.

FIGURE 5 . 5 -

 55 FIGURE 5.5 -Modified dataset for regression models

  different approaches are: linear regression (LR), MLP and LSTM. Data collected by glasses TB-2, TB-4 and TB-5 have been used for training these machine learning models. For the sake of simplicity, an unique discharge process performed by a different pair of glasses is used for testing the precision of every model. An extract of this discharge process is shown in Fig.5.6. This test dataset (Fig.5.6) corresponds to a discharge process of the smart glasses running the driver drowsiness detection application (DDDC) from 100% to 0% of

FIGURE 5 . 6 -

 56 FIGURE 5.6 -DDDC discharge process with 250 cycles of aging

Fig. 5 .

 5 7 shows an example of a linear regression model. Blue points represent the dependent variable data, while the red line determines the line which best fits the data. Training a linear regression model with the battery testbench data can allow us to find a linear representation of the discharge process on the smart glasses. The objective is first to obtain a SOC prediction curve as close as possible of the SOC behavior presented in Fig. 5.6. The obtained SOC prediction is shown in Fig. 5.8a.

FIGURE 5 . 7 -

 57 FIGURE 5.7 -Linear Regression modelSource:[17]

FIGURE 5 .

 5 FIGURE 5.9 -MLP modelSource:[18]

FIGURE 5 .

 5 FIGURE 5.11 -LSTM modelSource:[19]

Fig. 5 .FIGURE 5 . 13 -

 5513 Fig. 5.13 illustrates the relationship between the electrical capacity and the battery SOC. A fully charged battery has a SOC of 100% whereas a fully discharged battery has a SOC of 0% of its nominal capacity C n .

Fig. 5 .

 5 Fig. 5.14 illustrates the proposed analytical battery SOC model.

FIGURE 5 . 14 -

 514 FIGURE 5.14 -Formulation of the proposed analytical model

FIGURE 5 .FIGURE 5 . 16 -

 5516 FIGURE 5.15 -SOC forecast using mathematical model without aging modeling

FIGURE 5 .

 5 FIGURE 5.17 -SOC forecast using the analytical model with aging modeling

FIGURE 6 . 1 -

 61 FIGURE 6.1 -Simulation software organization

FIGURE 6 . 2 -

 62 FIGURE 6.2 -An example of scenario description

FIGURE 6 . 3 -

 63 FIGURE 6.3 -Scenario Finite State Machine

FIGURE 6 . 4 -

 64 FIGURE 6.4 -Simulator GUI

FIGURE 6 . 6 - 6 . 5 . 2 Scenario 2 :

 666522 FIGURE 6.6 -Scenario 1: experimental and simulated SOC

FIGURE 6 . 7 -

 67 FIGURE 6.7 -Scenario 2 (fall detection) FSM

FIGURE 6 . 8 -

 68 FIGURE 6.8 -Scenario 2: experimental and simulated SOC

FIGURE 6 . 9 -

 69 FIGURE 6.9 -Scenario 3 (drowsiness detection): yml description

TABLE 2 .

 2 1 -Karolinska Sleeping Scale (KSS levels[START_REF] Kaida | Validation of the karolinska sleepiness scale against performance and eeg variables[END_REF]) vs Drowsiness Levels of Smart Connected Glasses (SCG levels)

	KSS levels	KSS level description	SCG levels
	1 2	Extremely alert Very alert	Level 1
	3 4	Alert Rather alert	Level 2
	5 6	Neither alert nor sleepy Some signs of sleepiness	Level 3
	7 8	Sleepy, but not difficulty remaining awake Sleepy, some effort to keep alert	Level 4
	9	Extremely sleepy, fighting sleep	Level 5

TABLE 2 .

 2

	2 -State-of-the-art: Drowsiness Detection Systems

Driver Drowsiness Characteristics of embedded systems Detection studies Sensing Classification Embedded Method Algorithm Classification

	You et al. [15]	Camera	ML-based	None
	Xu et al. [16]	Camera	DL-based	None
	Kim et al. [17]	Camera	DL-based	None
	Nikolskaia et al. [18]	Camera	DL-based	None
	Sharan et al. [19]	Camera	DL-based	Raspberry Pi
	Kartsch et al. [5]	BCI sensor	Threshold based	Wearable MCU
	Li et al. [20]	BCI sensor	ML-based	Wearable MCU
	Rohit et al. [21]	BCI sensor	ML-based	None
	Chang et al. [22]	IR sensor	Threshold based	Wearable MCU
	He et al. [23]	IR sensor	Threshold based	Wearable MCU
	Warwick et al. [25]	Multi-variable sensors	Threshold based	Smartphone
	Lee et al. [26]	Hand motion sensors	ML-based	Wearable Platform
	Ellcie-Healthy Smart Glasses	IR sensors	Threshold based	Wearable MCU
	a KSS number from 1 to 9 is attributed to each KSS level (extremely alert being the
	level 1, while extremely sleepy being the level 9).
	Table 2.2 summarizes the main characteristics of the aforementioned solutions for
	driver drowsiness detection. Concerning classification algorithms, a close relation-
	ship between camera-based approaches and algorithms leveraging ML or DL can
	be observed. It is worth noticing that, in most cases, ML or DL algorithms are not
	executed in MCU-based embedded systems. On the other hand, threshold-based
	algorithms are mostly handled by MCUs.

TABLE 2 .

 2

	3 -State-of-the-art: Fall Detection Systems

Wearable Embedded System Characteristics Classification Performance Fall Detection Battery Systems Body-worn Sensor Detection Algorithm Sensitivity Specificity Lifetime Sensor Position Method

	Lu et al. [37]		Cola et al. [30]			Bianchi et al. [36]			Chen et al. [35]			Li et al. [29]		Lim et al. [31]			Qu et al. [34]			Wu et al. [33]			kostopoulos et al. [32]			kangas et al. [28]
	Barometer	Accelerometer	Barometer	Accelerometer		Barometer	Accelerometer	Gyroscope	Accelerometer	Magnetometer		Gyroscope	Accelerometer	Accelerometer			Accelerometer			GPS	Accelerometer		Accelerometer			Accelerometer
	(neck-worn	Chest	Head			Waist			Head			Thigh	Chest	Chest			Chest			Waist			Wrist		Waist	Head	Writs
	Threshold-based		Threshold-based			Decision Tree Classifier	Threshold-based		Threshold-based	Self-adaptive		Threshold-based		Hidden Markov Model	Threshold-based		Threshold-based			Threshold-based			Threshold-based			Threshold-based
	Posture,	Impact, Inactivity,	Pressure variation	Acceleration	Differential Pressure	Posture,	Impact, Post-fall activity,	Fall direction	Impact,	Head rotations,	Transition	Posture,	Activity intensity,	Post-fall activity	Impact,	Posture	Post-fall activity,	Impact,	Posture	Impact,	Fall beginning,	Posture	Post-fall activity,	Rebound,	Posture	Velocity, Impact,	Fall beginning,
	94 %		100 %			97.5 %			98.7 %			91 %		99.17 %			90 %			91.6 %			93.48 %			98 %
	90 %		100 %			96.5 %			81.7 %			92 %		99.69 %			-			88.7 %			98.54 %			100 %
	for a C BATT	996 days	-		of 30 Ah	for a C BATT	1 day		-			-		-			-		of 1200 mAh	for a C BATT	2 days		-			-

TABLE 3 .

 3 1 -IMU: power consumption information provided in the manufacturer datasheet

	Operation mode Accelerometer	Operation mode Gyroscope	Sampling Frequency Configuration	I typ
	High performance High performance	>= 1.6 kHz	1.25 mA
	Normal	Normal	208 Hz	0.9 mA
	Low-power	Low-power	12.5 Hz	0.42 mA
	Power down	Power down	-	6 µA
	High performance	Power down	>= 1.6 kHz	240 µA
	Normal	Power down	104 Hz	70 µA
	Low-power	Power down	12.5 Hz	24 µA

Table 3 .

 3 1. It is possible to disable both, enable both or enable one while disabling the

  Algorithm 1 : I MU (Average Power Consumption by State) 1: procedure GET_IMU_POWER(F acc (s), mode acc (s))

			s = glasses state of the scenario
	2:	Pgyr (s) ← V 1 * I power_down_gyr
	3: 4:	if mode acc (s) is power_down then Pacc (s) ← V 1 * I power_down_acc
	5:	else
	6:	switch mode acc (s) do
	7:	case low_power	In low power mode, F base_acc = 12.5Hz
	8:	I acc (s) ← I low_power_acc
	9:	F base_acc ← 12.5
	10:	case normal_power	In normal power mode, F base_acc = 104Hz
	11:	I acc (s) ← I normal_power_acc
	12:	F base_acc ← 104
	13:	case high_power	In high power mode, F base_acc = 1.6kHz
	14:	I acc (s) ← I high_power_acc
	15:

  average power consumption model. Data provided by the manufacturer are: t active , t setup , and t measurement , as well as I active , I measurement , and I sleep . Additionally, configurable parameters of the IR sensor are: The sampling frequency of the sensor (F ir_sensor ), thus its sampling period (T ir_sensor ), as well as the IR LED current (I ir_led ), and the IR LED activation time (t ir_led ).

	Algorithm 2 : IR Sensor (Average Power Consumption by State)
		DC active (s) ← 0
	9:	DC measurement (s) ← 0
	10:	DC sleep (s) ← 1
	11:	DC ir_led (s) ← 0
	12:

1: procedure GET_IR_SENSOR_POWER(F ir_sensor (s), t ir_led (s), I ir_led (s)) s = glasses state of the scenario 2:

if F ir_sensor (s) = 0 then 3:

DC active (s) ← t active /T ir_sensor (s) 4: DC measurement (s) ← 2 * (t setup + t measurement )/T ir_sensor (s) 5: DC sleep (s) ← 1 -DC active (s) -DC measurement (s) 6: DC ir_led (s) ← t ir_led (s)/T ir_sensor (s) 7:

else 8: Pir_rx (s) ← V 1 * (I active * DC active (s) + I suspend * DC measurement (s)) 13: Psleep (s) ← V 1 * I sleep * DC sleep (s) 14: Pir_tx (s) ← V 2 * I ir_led (s) * DC ir_led (s) 15: Pir_sensor (s) ← Pir_rx (s) + Psleep (s) + Pir_tx (s) 16: return Pir_sensor (s)

TABLE 3 .

 3

	2 -Pressure sensor: Power consumption information pro-
	vided in the manufacturer datasheet
	Sensor operation mode	Measurement conditions	I typ
	Normal power	F sensor = 1 Hz, and V base = 1.8 V	12 µA
	Low power	F sensor = 1 Hz, and V base = 1.8 V	3 µA
	Power down	V base = 1.8 V	I sleep = 1 µA

  procedure GET_SENSOR_POWER(F sensor (s))

		s = glasses state of the scenario
	2:	if F sensor (s) = 0 then
	3:	if mode(s) is low_power then
	4:	Psensor

1:

TABLE 3 .

 3 4 -MCU: power consumption information provided in the manufacturer datasheet

	MCU operation mode	Voltage regulator	I typ	F exe	I avg
	RUN	MR RANGE 1 94 µA/MHz 80 MHz 7.52 mA
	RUN	MR RANGE 2 85 µA/MHz 26 MHz 2.21 mA
	LP RUN	LPR	95 µA/MHz	2 MHz	190 µA
	SLEEP	MR RANGE 1 27 µA/MHz 80 MHz 2.16 mA
	SLEEP	MR RANGE 2 27 µA/MHz 26 MHz	700 µA
	LP SLEEP	LPR	38 µA/MHz	2 MHz	76 µA
	STOP 0	LPR	-	-	125 µA
	STOP 1	LPR	-	-	10.5 µA
	STOP 2	LPR	-	-	2.3 µA
	STANDBY	LPR	-	2 MHz	0.35 µA
	case run_main_regulator_1
	4:
	case run_main_regulator_2
	6:
	case run_low_power_regulator
	8:
	case sleep_main_regulator_1
	10:
	case sleep_main_regulator_2
	12:
	case sleep_low_power_regulator
	14:

-Range 2 run mode using MR: Users may set a CPU frequency between 2 MHz and 26 MHz.

-Low-power run mode using LPR: CPU runs up to 2 MHz frequency. Algorithm 4 : MCU (Average Power Consumption by State) 1: procedure GET_MCU_POWER(F mcu (s), mode mcu (s)) s = glasses state of the scenario 2: switch mode mcu (s) do 3:

Pmcu (s) ← V 1 * I run_main_regulator_1 * F mcu (s) 5: Pmcu (s) ← V 1 * I run_main_regulator_2 * F mcu (s) 7: Pmcu (s) ← V 1 * I run_low_power_regulator * F mcu (s) 9: Pmcu (s) ← V 1 * I sleep_main_regulator_1 * F mcu (s) 11: Pmcu (s) ← V 1 * I sleep_main_regulator_2 * F mcu (s) 13: Pmcu (s) ← V 1 * I sleep_low_power_regulator * F mcu (s) 15: case stop_0

Table 3 .

 3 4 contains the microcontroller technical specifications given by the manu-

facturer, regarding power consumption. The algorithm 4 resumes how the MCU average power consumption ( Pmcu ) is modeled, considering the smart glasses state and the MCU settings.

TABLE 3 .

 3

		5 -FLASH: power consumption information provided in the
		manufacturer datasheet
	FLASH state	FLASH operation mode Measurement conditions Execution time	I typ
	Power Down	-	-	-	7 nA
	Standby	Low Power High Performance	--	--	5 µA 9 µA
			F = 8MHz & I/O = 2	-	1.9 mA
	Read	Low Power	F = 8MHz & I/O = 4 F = 16MHz & I/O = 4	--	2.2 mA 2.8 mA
			F = 33MHz & I/O = 1	-	2.2 mA
			F = 33MHz & I/O = 4	-	4.2 mA
	read	High Performance	F = 80MHz & I/O = 1 F = 80MHz & I/O = 2	--	3.8 mA 4.2 mA
			F = 80MHz & I/O = 4	-	6.5 mA
	Program	Low Power High Performance	--	3.2 ms 8.5 ms	3.5 mA 5.8 mA
	Write Register	Low Power High Performance	--	40 µs 32 µs	3.5 mA 3.5 mA
	Erase Sector/Block	Low Power High Performance	--	58 ms 40 ms	3.1 mA 3.5 mA
	Erase Chip	Low Power High Performance	--	30 s 13 s	3.1 mA 4 mA

  Algorithm 5 : Flash Memory (Average Power Consumption by State) 1: procedure GET_FLASH_POWER(mode f lash (s), F read (s), I/O read (s), T read (s), T write (s), T SYS ) 2: DC standby ← 1 -DC read -DC erase -DC write

		T erase (s) ← T write (s)
	3:	if mode f lash (s) is power_down then
	4:	Pf lash (s) ← V 1 * I power_down
	5:	else
	6:	DC read ← read_time/(T read (s) * T SYS )
	7:	DC erase ← erase_time/(T erase (s) * T SYS )
	8:	DC write ← write_time/(T write (s) * T SYS )
	9:
	10: 11: 12: 13:	Pread (s) ← V 1 * I read * DC read Perase (s) ← V 1 * I erase * DC erase Pwrite (s) ← V 1 * I write * DC write Pstandby (s) ← V 1 * I standby * DC standby
	14:	Pf lash (s) ← Pread (s) + Perase (s) + Pwrite (s) + Pstandby (s)
	15:

TABLE 3 .

 3 6 -BLE connection interval current consumption without payload I CON_NP (t active CON_NP = 1.35 ms)

	BLE operation mode	Tx Power	Active phase current
		0 → -18dBm	6.12 mA
		1 → -15dBm	6.13 mA
		2 → -12dBm	6.15 mA
	Standard	3 → -9dBm 4 → -6dBm	6.17 mA 6.21 mA
		5 → -2dBm	6.26 mA
		6 → 0dBm	6.36 mA
		7 → 5dBm	6.63 mA
		0 → -14dBm	6.28 mA
		1 → -11dBm	6.30 mA
		2 → -8dBm	6.33 mA
	High Power	3 → -5dBm 4 → -2dBm	6.39 mA 6.44 mA
		5 → 2dBm	6.51 mA
		6 → 4dBm	6.70 mA
		7 → 8dBm	7.12 mA
	For the sake of simplicity, only estimations for connection intervals without pay-
	load are presented in this table. Whatever the hardware settings, the average time
	needed to execute a connection interval without payload is 1.35 ms. Likewise, the
	advertising interval and the connection interval with a payload of 20 Bytes last for
	3.68 ms and 1.56 ms, respectively. Furthermore, we can observe in Table 3.6 that two
	BLE operation modes (mode BLE ) are possible: standard and high-power. For each of
	them, eight different transmission power (TX POWER ) values can be selected, cover-
	ing a range of TX POWER from -18 dBm to 8 dBm. So the average power consumption
	of the active phase for advertising, connection without payload or connection with
	payload, depends on these two hardware settings (mode BLE and TX POWER ).

  BLE average current consumption during a connection interval without payload. In this example, the mode BLE and TX POWER have been configured in high-power and 8 dBm, respectively. STR (s)← V 1 * I DATA STR * DC DATA STR (s) 8: PDATA VAR (s) ← V 1 * I DATA VAR * DC DATA VAR (s) 9: PDATA EVT (s) ← V 1 *I DATA EVT * DC DATA EVT (s) PADV (s) + PCON_NP (s) + PDATA STR (s) + PDATA VAR (s) + PDATA EVT (s) + PSLEEP (s)

	Algorithm 6 : Bluetooth Low Energy (Average Power Consumption by State)
	1: procedure GET_BLE_POWER(mode BLE (s), TX POWER (s))	s = glasses state of the scenario
	2: 3:	if mode BLE (s) is power_down then PBLE (s) ← V 1 * I SLEEP
	4: 5: 6:	else PADV (s) ← V 1 * I ADV * DC ADV (s) PCON_NP (s) ← V 1 * I CON_NP * DC CON_NP (s)
	7:
	10:	PSLEEP (s) ← V 1 * I SLEEP * DC SLEEP (s)
	11:
	12:	return PBLE (s)

3.7, we can observe a synthesized comparison between a measurement (at the top) and an estimation (at the bottom) of the PDATA PBLE (s) ←

  Algorithm 7 : Bluetooth Low Energy (Duty Cycles by State) 1: procedure GET_DC(T DATA STR (s), T DATA VAR (s), T DATA EVT (s), T SYS )

				s = glasses state of the scenario
	2:	if BLE(s) is in advertising then	Glasses in advertising phase
	3:	N ADV ← T SYS /T ADV		Number of advertising intervals
	4: 5:	DC ADV (s) ← N ADV *	t active ADV T ADV	(s)

DC CON_NP (s) ← 0 FIGURE 3.7 -BLE Current consumption: estimated vs measured

TABLE 3

 3

	.7 -Current consumption given by manufacturers of the
	BUZZER, RGB LED and LED Driver
	Hardware component Operation mode	Conditions	Current I typ
	LED Driver LP5562	Sleep mode Active mode	V = 1.8 V	2.4 µA 1 mA
			Red color	25.5 mA
	RGB LED	Active mode	Green color	25.5 mA
			Blue color	25.5 mA
	BUZZER	Active mode

  ← V 2 * (I RED + I GREEN + I BLUE ) PBUZZERsequence i (s) ← DC sequence i (s) * V 2 * I ACTIVE BUZZER

	1: procedure GET_RGB_LED_POWER(T sequence i (s), t sequence i ) 2: if T sequence i (s) = 0 then 3: DC sequence i (s) ← t sequence i /T sequence i (s) 4: else	s = glasses state
	5:	DC sequence i (s) ← 0
	6:	PDRIVER LED (s) ← V 1 * I ACTIVE DRIVER LED
	7: 8:	PRGB_LEDsequence i	(s) ← DC sequence i (s) * ( PDRIVER LED (s) + PLED (s))
	9:
	10:

PLED (s))

return PRGB_LEDsequence i (s), PBUZZERsequence i (s)

TABLE 3

 3

	.8 -Current consumption given by manufacturers of the
		CHARGER and FUEL GAUGE
	Hardware component Operation mode	Conditions	Current I typ
		Power down	None of Regulators activated	1 µA
	CHARGER	Low power	Both Regulator activated (no load)	5.6 µA
		Normal power	Both Regulator activated (no load)	40 µA
		Power down	-	0.5 µA
	FUEL GAUGE	Low power	-	7 µA
		Normal power	-	18 µA

TABLE 3 .

 3

	V LOAD	R LOAD	I SYS	I LOAD	P SYS	P LOAD	P DC/DC REG	Coefficient β
		2000 ohm	0.6 mA	0.9 mA	2.2 mW	1.6 mW	0.6 mW	37 %
		1000 ohm	1.1 mA	1.8 mA	4.1 mW	3.2 mW	0.8 mW	26 %
		500 ohm	2.5 mA	3.6 mA	9.1 mW	6.5 mW	2.6 mW	40 %
	1.8 V	400 ohm 300 ohm	3.0 mA 4.0 mA	4.5 mA 6.0 mA	11.1 mW 14.7 mW	8.1 mW 10.8 mW	3.0 mW 3.9 mW	37 % 36 %
		150 ohm	7.9 mA	12.0 mA	29.2 mW	21.6 mW	7.6 mW	35 %
		100 ohm	11.4 mA	18.0 mA	42.2 mW	32.4 mW	9.8 mW	30 %
		50 ohm	23.0 mA	36.0 mA	85.1 mW	64.8 mW	20.3 mW	31 %
		20 ohm	57.6 mA	90.0 mA	213.1 mW	162.0 mW	51.1 mW	32 %
		2000 ohm	3.0 mA	2.0 mA	11.0 mW	7.6 mW	3.3 mW	44 %
		1000 ohm	5.8 mA	3.9 mA	21.3 mW	15.2 mW	6.1 mW	40 %
		500 ohm	11.4 mA	7.8 mA	42.2 mW	30.4 mW	11.8 mW	39 %
	3.9 V	400 ohm 300 ohm	13.85 mA 19.29 mA	9.8 mA 13.0 mA	51.2 mW 71.4 mW	38.0 mW 50.7 mW	13.2 mW 20.7 mW	35 % 41 %
		150 ohm	37.3 mA	26.0 mA	138.0 mW	101.4 mW	36.6 mW	36 %
		100 ohm	58.1 mA	39.0 mA	215.0 mW	152.1 mW	62.9 mW	41 %
		50 ohm	112.5 mA	78.0 mA	416.3 mW	304.2 mW	112.1 mW	37 %
		20 ohm	278.8 mA 195.0 mA 1031.6 mW 760.5 mW	271.1 mW	36 %
				Percentage of P DC/DC REG over P LOAD in average:	36 % ± 4.5 %

9

-DC/DC Regulators: power consumption characterization (using V BATT = 3.7 V)

TABLE 3 .

 3 10 -System configuration by discharge process

	Hardware		Configuration 1			Configuration 2
	Module	F HW	T ACTIVE	DC HW	PHW	F HW	T ACTIVE	DC HW	PHW
	MCU	80 MHz	-	100 %	14 mW	16 MHz	-	100 %	2.8 mW
	IR Sensor	100 Hz	1.9 ms	19 %	360 µW	50 Hz	600 µs	3 %	90 µW
	IR LED	100 Hz	800 µs	8 %	15.6 mW	50 Hz	220 µs	1 %	6.7 mW

TABLE 3 .

 3 11 -Autonomy results by configuration

	Configuration	Autonomy Measured Estimated	Estimation Error
	# 1	05h08m20s 04h58m20s	3.3 %
	# 2	11h08m10s 10h37m50s	4.5 %

TABLE 4 .

 4 1 -Performance and Autonomy by System Configuration

	System configuration		Performance and Autonomy metrics
	P (mW)	F (Hz)	Time (µs)	Threshold	Sensitivity	Specificity	Accuracy	Autonomy
	75	75	400	90 %	84.9 %	87.7 %	86.4 %	9.7 h
	75	100	400	90 %	80.6 %	88.5 %	84.8 %	8.6 h
	150	75	50	75 %	43.7 %	91.7 %	67.3 %	13.6 h
	150	125	100	85 %	18.2 %	94.8 %	58.2 %	12.4 h
	225	50	50	80 %	59.7 %	68.4 %	64.1 %	13.6 h
	375	25	100	90 %	72.3 %	75.2 %	73.9 %	12.6 h
	375	50	200	90 %	83 %	87.7 %	85.5 %	7.8 h
	375	100	200	90 %	74.5 %	86.1 %	80.7 %	5.2 h
	signals in the post-processing phase to simulate different levels of signal-to-noise ra-

tio (SNR). The noise samples are generated from a uniform probability distribution with zero mean and sigma variance (σ). Using generated noisy signals, the blink detection performance can be evaluated in more realistic conditions.

TABLE 4 .

 4 2 -Considered events in training dataset

	Training Dataset	Class "1" = Blink	Class "0" =

Non-blink Look down Look up Look at left Look at right Other

	Percentage	45 %	15 %	7.5 %	7.5 %	7.5 %	17.5 %

TABLE 4 .

 4 3 -Architecture and Accuracy of best CNN models

CNN Number Number of Size of Size of Mean model of layers conv. filters conv. filters pool. filters Accuracy

	A	1	8	7	2	98.6 %
	B	1	16	3	3	98.1 %
	C	2	6-12	7-7	3-3	98.2 %
	D	2	12-12	7-7	3-3	99.2 %
	E	2	16-24	5-7	3-3	99.4 %
	F	3	12-12-12

TABLE 4 .

 4 4 -Model size, ROM footprint, execution time, power consumption and autonomy by blink detection algorithm

	Blink Detec.	Trainable	ROM	RAM	Execution	Avg MCU	Battery
	Algorithm	Params.	Usage	Usage	Time	Power	Lifetime
	THR-based	-		10.7 kB	1.8 kB	2.6 ms	3.5 mW	14 h
	CNN A	8609	81.4 kB	5.0 kB	54.2 ms	5.0 mW	13.4 h
	CNN B	11425	92.7 kB	5.4 kB	-	-	-
	CNN C	3221	59.9 kB	4.6 kB	53.9 ms	4.9 mW	13.4 h
	CNN D	3773	62.1 kB	5.1 kB	53.9 ms	4.9 mW	13.4 h
	CNN E	7001	75.0 kB	5.7 kB	54.1 ms	5.0 mW	13.4 h
	CNN F	2921	58.7 kB	5.1 kB	53.9 ms	4.9 mW	13.4 h
	CNN G	6481	72.9 kB	5.6 kB	54.0 ms	5.0 mW	13.4 h
	TABLE 4.5 -Performance metrics by blink detection algorithm
	Blink Detec.	Mean	Mean	Mean	Mean	Mean	Mean	Mean
	Algorithm	TP	FN	FP	TN	Sensitivity	Specificity	Accuracy
	THR-based	214	51	45	290	80.8 %	86.6 %	84.0 %
	CNN A	243	22	53	282	91.6 %	84.3 %	87.5 %
	CNN B	246	19	56	279	92.8 %	83.1 %	87.4 %
	CNN C	244	21	43	292	91.9 %	87.2 %	89.3 %
	CNN D	240	25	38	297	90.5 %	88.7 %	89.5 %
	CNN E	243	22	39	296	91.8 %	88.3 %	89.8 %
	CNN F	240	25	31	304	90.8 %	90.8 %	90.8 %
	CNN G	241	24	33	302	91.0 %	90.0 %	90.5 %

Table 5 .

 5 1 (where the glasses_id acronym 'TB' is for testbench).

TABLE 5 .

 5

	1 -Battery testbench: charge/discharge profiles configura-
	tion

Glasses_id Charge Rate Discharge Rate Charge/Discharge Cycles

	TB-1	1C (1 h)	0.67C (1.5 h)	800 cycles
	TB-2	0.5C (2 h)	0.4C (2.5 h)	600 cycles
	TB-3	1C (1 h)	0.1C (11 h)	600 cycles
	TB-4	0.5C (2 h)	0.1C (11 h)	400 cycles
	TB-5	0.17C (6 h)	0.1C (11 h)	250 cycles
	As can be seen in this table, two main groups can be observed, according to the
	charge/discharge rate. A group of moderate stress (i.e. charge and discharge rates
	over 2 h) composed by the glasses TB-2, TB-4 and TB-5, and a group of high stress
	(i.e. charge rate at 1C) composed by the glasses TB-1 and TB-3. A 1C rate means that
	the charge (or the discharge) current will charge the battery in 1 hour. For a battery
	with a capacity of 100 mAh, it implies a charge current of 100 mA. A 2C rate would
	be 200 mA, and a 0.5C rate would be 50 mA.

  procedure FORECAST_SOC(Cycle # , PSYS , SOC MAX , SOC MI N , T SYS ) SOC MI N ) do Stop loop when SOC = SOC MI N SOC (t+T SYS ) ← get_regression_model_ f orecast(Cycle # , PSYS , SOC t )

		function for forecasting
	2:
	3:	t ← 0
	4:	SOC t ← SOC MAX
	5:	array f orecast ← [ ][ ]
	6:	array f orecast ← [t][SOC t ]
	7:
	8:
	9:
	10:	array f orecast ← [(t + T SYS )][SOC (t+T SYS ) ]
	11:	t ← t + T SYS
	12:
	13:	return array f orecast

1:

while (SOC t =

Table 5 .

 5 2 resumes the results of the exploration of machine learning approaches for SOC estimation. In this table we can see the predictions and forecasts accuracy for

	each model. Additionally, we can observe that the time required to get forecasting
	results (for a full battery discharge) is 5, 25 and 65 seconds for linear regression, MLP
	and LSTM, respectively. The forecasts have been executed on a laptop equipped
	with an Intel i5 processor and 8 GB of RAM.

TABLE 5 .

 5 2 -SOC prediction vs forecasting using machine learning

	Machine Learning SOC Prediction SOC Forecasting Forecasting
	Approach	MAE	MAE	Time
	Linear Regression	6.89 %	14.34 %	5 s
	MLP	3.08 %	22.73 %	25 s
	LSTM	0.47 %	7.03 %	65 s

As can be seen, LSTM outperforms the Linear Regression and MLP models regarding both prediction and forecast accuracy. It is worth pointing out that the training dataset used for Linear Regression and MLP models includes the SOC (t-1) . At the contrary, LSTM is able to learn the sequence of the observations without this information. LSTM is therefore more appropriate for regression in a sequence problem.

  of the battery SOC after a certain elapsed time (SOC (t+T SYS ) ) depends on the previous SOC (SOC t ) and the percentage of SOC that has been lost during the considered elapsed time (SOC T SYS ), as shown in the Equation 5.6.SOC (t+T SYS ) = SOC t -SOC T SYS (5.6) SOC T SYS is therefore considered as the SOC loss at every T SYS . Even if the scenario is composed of only one state (DDDC in this case), SOC (t+T SYS ) must be estimated at regular intervals to evaluate the progress of the discharge process. So the proposed analytical model computes SOC T SYS every T SYS , even if the scenario state has not changed from the previous interval. To estimate SOC T SYS , we first calculate how long it takes to decrease the SOC by 1% considering an average power consumption of PSYS (as shown in the Equation 5.7).

	t 1% =	t cuto f f 100	(5.7)
	SOC T SYS =	T SYS t 1%	(5.8)

t 1% serves as a duration of reference to estimate SOC T SYS at every interval T SYS . Using the Equation

5

.8, we obtain SOC T SYS .

  procedure FORECAST_SOC(C n , PSYS , SOC MAX , SOC MI N , T SYS ) SOC MI N ) do Stop loop when SOC = SOC MI N

		function for forecasting
	2:
	3:	t ← 0
	4:	SOC t ← SOC MAX
	5:	array f orecast ← [ ][ ]
	6:	array f orecast ← [t][SOC t ]
	7:
	8:
	9:	t cuto f f ← Cn PSYS
	10: 11: 12:	t cuto f f 100 SOC T SYS ← T SYS t 1% ← t 1%
	15:
	16:	return array f orecast

1:

while (SOC t = SOC (t+T SYS ) ← SOC t -SOC T SYS 13: array f orecast ← [(t + T SYS )][SOC (t+T SYS ) ] 14: t ← t + T SYS

TABLE 5 .

 5

	3 -Aging modeling using 3 different regression models
	Aging Model	(MAE)
	Straight-line Regression (SLR)	1.36 %
	Linear Regression (LR)	0.76 %
	Quadratic Regression (QR)	0.22 %

TABLE 5 .

 5 This improvement is confirmed by the results presented in Table5.4. As can be seen, the three aging models provide a better SOC forecasting accuracy. The mean absolute error decreases by approximately 5% with respect to the SOC forecasting without aging model, thus proving the relevance of considering aging during SOC predictions.

	4 -SOC forecasting using the analytical model
	SOC Forecasting Model	Aging Model	MAE
	Analytical Model	-Straight-line Regression (SLR) 3.67 % 8.59 % Linear Regression (LR) 3.67 %
		Quadratic Regression (QR)	2.95 %

TABLE 6 .

 6 1 -Comparison: Experimental vs Simulation results

	Scenario	Autonomy (Exp) [min.] Autonomy (Sim) [min.] Error [%] Simulation time
	Scenario 1	130	140	7.7%	9sec
	Scenario 2	2660	2415	9.2%	2min21sec
	Scenario 3	15330	14010	8.6%	14min8sec

Acknowledgements

List of Abbreviations

29:

Performance characterization and optimization

Introduction

In a wearable system, a common limitation is the scarce amount of computational and energy resources available to offer high reliability and autonomy. We have seen in chapter 3 that modeling the power consumption early in the design flow, using a system-level approach, is a good way to accurately anticipate the system's autonomy according to the hardware settings. Designers must base their choices not only on the autonomy of the system, but also on its performance, to ensure therefore energy efficiency (i.e. the best autonomy/performance trade-off, in this case). For this, the constraints of the system in terms of autonomy and performance must be defined according to the application use cases. Moreover, the hardware and software parameters impacting energy efficiency must be carefully studied to meet those constraints. In this chapter we propose to assess the performance of a threshold-based eye blink detection algorithm (used in the driver drowsiness detection application), using the main hardware and software parameters to evaluate their impact on the system's energy efficiency. To do so, we compare two performance metrics related to detection: the accuracy and the relationship between the sensitivity and specificity.

We will see why the second metric is suitable for evaluating the eye blink detection performance.

In this chapter, the performance characterization of a threshold-based eye blink detection algorithm is detailed. An optimization of the performance for this algorithm is then presented, considering the impact on the energy consumption, thus the autonomy. After that, a new eye blink detection algorithm based on a CNN model is proposed. For that algorithm, the trade-off between performance and autonomy, as well as the overhead in terms of memory footprint and execution time is studied. Then, a comparison between the threshold-based and CNN-based algorithms is presented. It is shown that the CNN-based approach can be executed at the Edge

Where:

-True Positives or TP: number of detected blinks.

-False Negatives or FN: number of not detected blinks.

-False Positives or FP: number of look events detected as blinks.

-True Negatives or TN: number of look events not detected as blinks.

The ROC curve serves to visualize the evolution of the performance detection for every threshold. The selection of the best threshold by system configuration has been done then through the accuracy resulting from computing the TP, FN, FP and TN values, as shown in the equation 2.3 (presented in chapter 2). Apart from computing the accuracy, based on the ROC curve we can visually determine the best threshold by configuration. To do so, TPR must be as higher as possible while FPR must be as close as possible from 0%. For example, considering the system configuration composed of P ir_led , F ir_sensor and t ir_led at 75mW, 100Hz and 400µs respectively, we can see that the best threshold is represented by the point at 90%.

Experimental results: autonomy/performance trade-off

The conjoint autonomy and performance evaluation has been based on constraints fixed according to several insights. Using figures 4. the specificity offered by the threshold-based algorithm. Specificity is improved with CNN models that have at least two convolutional layers (models C, D, E, F and G).

However, only models F and G respect both the sensitivity and specificity minimum constraint of 90%. Finally, it is worth noting that a 10% drop in accuracy can be observed in Table 4.5 with respect to the results obtained in Table 4.3. The reason is that in real conditions, blinks are not necessarily in the middle of the 64-sample window as in the training condition. Moreover, performing an inference every 10 samples exacerbates this problem.

To conclude, the results obtained show that the model F offers the best compromise in terms of performance, battery lifetime, memory footprint and execution time. In comparison with the current threshold-based algorithm used on connected glasses, this model improves the sensitivity, the specificity and the accuracy by 10%, more than 4% and almost 7%, respectively. This performance improvement comes with an overhead in terms of battery lifetime and memory footprint (ROM and RAM usage). However, the CNN implementation fully meets the initial constraints defined in section 4.3.

Chapter 5

Battery lifetime modeling

Introduction

Lithium-ion (Li-ion) and Lithium Polymer (LiPo) rechargeable batteries have became essential in many applications such as electric vehicles, smartphones, wearable devices, and the IoT area in general. These applications strongly rely on precise battery SOC predictions to determine the available battery capacity and to implement efficient power management policies. Since battery SOC cannot be directly measured, most of the SOC prediction algorithms are based on measurements of physical variables (e.g. current, voltage, temperature) during the dynamic charge/discharge processes. In the Ellcie-Healthy smart connected glasses, the battery SOC plays a fundamental role in the runtime process (i.e. battery discharge process). On the one hand, it is useful for informing users about the remaining available battery capacity of the glasses. On the other hand, important decisions could be made according to predefined battery SOC thresholds such as data storage before system shutdown, entering lower power consumption modes or switching from greedy system configurations to less consuming ones, to increase the battery lifetime (i.e. autonomy).

In this research work, a battery model for SOC prediction, thus simulating the discharge process of the smart glasses has been developed. The proposed systemlevel modeling methodology that includes the analytical power models presented in chapter 3, needs to get battery SOC predictions during the execution of a scenario to study the battery discharge behavior (e.g. the discharge rate). Using simulations, we can quickly estimate battery lifetime of the smart glasses or the remaining battery capacity at the end of a scenario. The accuracy of the simulations is therefore highly dependent on the precision of SOC predictions performed with the battery model. To provide precise SOC estimations, the aging phenomenon must also be considered in this modeling approach.

The organization of this chapter is as follow. In section 5.2, a state of the art for battery modeling is presented to identify latest approaches as well as to align our approach regarding existing methods. In section 5.3, a battery testbench for analyzing battery safety conditions and collecting data for modeling the battery SOC

Comparing Experimental and Simulated results

In order to validate our approach, we have modeled and simulated three real life scenarios of the smart glasses. The simulated scenarios have also been played with the smart glasses to collect the battery SOC and measure the autonomy. During the field test, the battery SOC is measured through the fuel gauge integrated in the PMU board. This component estimates the battery SOC with a resolution of 10%. It is worth noticing that, in simulation, we can estimate the SOC with a full resolution. This allows us to compare our simulation results with real data, and validate the effectiveness of our power and battery models. The three scenarios last for approximately 2 hours, 2 days and 10 days respectively. Each scenario mimics a real life utilization of the smart glasses, as we will discuss below.

Scenario 1: Drowsiness detection (2 hours)

In the first scenario, that lasts for 2 hours and 10 minutes, a trivial use-case of the drowsiness detection application is modeled. As shown in Fig. 6.5, the scenario FSM is composed of three states. At the beginning the smart glasses stay in sleep mode for 25 minutes. Then, the drowsiness detection application is started and executed for almost 2 hours. Finally, the system goes into the power down mode. We can observe that the simulated battery SOC at the end of the scenario (SOC MI N = 60%) is reached after the measured battery SOC (10 min). It means that the power models are optimistic for this scenario, corresponding to an error of 7.7%. The limited resolution of the battery gauge does not allows us to visualize the detailed progress of the SOC during the whole scenario execution. On the other hand, the simulated SOC allows us to observe that the battery is first discharged at a low rate during 25 minutes (i.e. Sleep state), where the power consumption is low. Then, the SOC drops quickly as soon as the smart glasses enter into the drowsiness detection state. In the next two sub-sections we will simulate longer scenarios of two and ten days respectively. We will also simulate a full discharge of the battery.