Lucas Magnana
email: lucas.magnana@gmail.com

Hervé Rivano

Nicolas Chiabaut

A DRL solution to help reduce the cost in waiting time of securing a traffic light for cyclists

Keywords: Deep reinforcement learning, traffic light, cyclists, waiting time, 3DQN, actuated, bike counts, car counts

Cyclists prefer to use infrastructure that separates them from motorized traffic. Using a traffic light to segregate car and bike flows, with the addition of bike-specific green phases, is a lightweight and cheap solution that can be deployed dynamically to assess the opportunity of a heavier infrastructure such as a separate bike lane. To compensate for the increased waiting time induced by these new phases, we introduce in this paper a deep reinforcement learning solution that adapts the green phase cycle of a traffic light to the traffic. Vehicle counter data are used to compare the DRL approach with the actuated traffic light control algorithm over whole days. Results show that DRL achieves better minimization of vehicle waiting time at almost all hours. Our DRL approach is also robust to moderate changes in bike traffic. The code of this paper is available at https://github.com/LucasMagnana/A-DRL-solution-to-help-reduce-the -cost-in-waiting-time-of-securing-a-traffic-light-for-cyclists..

Introduction

Promoting cycling as a mode of transport is prevalent in urban policies worldwide, especially to reduce CO2 emissions of transportation [START_REF] Mizdrak | Potential of active transport to improve health, reduce healthcare costs, and reduce greenhouse gas emissions: A modelling study[END_REF]. Cycling also saves residents time, money and improves their health [START_REF] Oja | Health benefits of cycling: a systematic review: Cycling and health[END_REF]. However, the cyclability of a city depends on space intensive infrastructures, such as bike lanes separated from the flow of cars, the feeling of safety being a major criterion for cyclists [START_REF] Adam | Conducting Interviews with Maps and Videos to Capture Cyclists' Skills and Expertise[END_REF][START_REF] Cervero | Bike-and-ride: Build it and they will come[END_REF]. The development of innovative, space-saving infrastructures is therefore becoming a necessity. In particular, traffic lights could be used to separate the flows of bikes and cars, hence securing the former. Each green phase of a traffic light allows certain vehicles that are not in conflict to pass through the intersection. Unfortunately, the green phase cycle is often independent on the traffic situation, making them unfit for bike isolation. However, recent advances in artificial intelligence for decision-making [START_REF] Openai | Dota 2 with large scale deep reinforcement learning[END_REF] and image recognition [START_REF] Naranjo-Torres | A review of convolutional neural network applied to fruit image processing[END_REF], among others, can enable traffic lights to adapt to the traffic [START_REF] Genders | An open-source framework for adaptive traffic signal control[END_REF].

Cyclists and traffic lights

Cyclists are known not to always respect red lights. The proportion of cyclists observed running a red light varies from study to study, ranging from 40% [START_REF] Schleinitz | e-)cyclists running the red light -the influence of bicycle type and infrastructure characteristics on red light violations[END_REF] to 60% [START_REF] Richardson | Investigating traffic light violations by cyclists in dublin city centre[END_REF]. [START_REF] Johnson | Why do cyclists infringe at red lights? an investigation of australian cyclists' reasons for red light infringement[END_REF] showed that in Australia, where people drive on the left, cyclists are more willing to infringe a red light when they want to turn left. The authors conclude that they cannot demonstrate that running a red light increases the likelihood of an accident. The red light infringements did not result in any risk to the safety of cyclists during their observations. [START_REF] Hollingworth | Risk factors for cycling accident related injury: The UK cycling for health survey[END_REF] showed a small increase in risk of accident-related injuries for cyclists infringing red lights. They note, however, that this increase could be caused by the generally riskier behavior of cyclists running red lights rather than actually running them. Traffic light-controlled intersections are nevertheless still dangerous places for cyclists. [START_REF] Miranda-Moreno | Disaggregate exposure measures and injury frequency models of cyclist safety at signalized intersections[END_REF] studied the cyclist injury occurrence at traffic lights, and their results suggest that cyclists safety at traffic lights is significantly affected by cyclist volumes and traffic flows. The conflicts between motorized vehicles and cyclists, especially when it comes to right-turns, seem to significantly increase the risk of collisions in Montreal, Canada, where people drive on the right. Whether in Canada or Australia, dangerous behavior performed or experienced by cyclists increases in the case of trajectories that do not involve crossing other lanes [START_REF] Johnson | Why do cyclists infringe at red lights? an investigation of australian cyclists' reasons for red light infringement[END_REF][START_REF] Miranda-Moreno | Disaggregate exposure measures and injury frequency models of cyclist safety at signalized intersections[END_REF]. To address this issue in France, M12 signs indicate that cyclists may cross the intersection in specified directions when the light is red, with priority to vehicles with green lights.

The safety of cyclists at traffic lights is an issue. Some experiments try to help cyclists reach traffic lights when they are green. [START_REF] Andres | co-riding with my eBike to get green lights[END_REF]) created e-bikes designed to make cyclists catch a green wave. When the cyclist crosses the first green light, the e-bike adapts its assistance to make the cyclist go at the most adapted speed for the green wave. Similarly, [START_REF] Fröhlich | BikeNow: a pervasive application for crowdsourcing bicycle traffic data[END_REF] developed a smartphone application suggesting a range of speed allowing the cyclist to reach the next traffic light during a green phase. Other studies try to modify intersections for cyclists, which has the advantage of benefiting all cyclists and not just those with the right equipment. De Angelis et al. ( 2019) asked cyclists to rate several interfaces at traffic lights, indicating whether cyclists are on time for a green wave. [START_REF] Anagnostopoulos | Cyclist-aware traffic lights through distributed smartphone sensing[END_REF] proposed traffic lights that prioritizes cyclists by detecting their smartphones. They however did not evaluate the impact of such a system on motorized traffic.

DRL for traffic light control

Deep reinforcement learning (DRL) has been used to adapt the behavior of traffic lights to the current traffic conditions and optimize the performance of the intersection. DRL is based on reinforcement learning (RL), an area of machine learning in which an agent develops its behavior through experience. The agent evolves in its environment and have the possibility to perform actions which modify it. At each step t, the agent receives the state of its environment s t ∈ S and chooses an action a t ∈ A with S the set of all possible states and A the set of possible actions per state. Once the action executed, the environment sends its new state s t+1 ∈ S and a reward r t to the agent. The reward is a numerical value indicating how good or bad the action was. The goal of the agent is to develop a policy π which maps an action to a state π(s) = a as to maximize the cumulative reward T t=0 γ t r t with γ ∈ [0, 1) the discount-rate weighting the distant future events. DRL algorithms are RL algorithms in which the agent uses deep learning to make decisions (further explanations are given in Section 2). Some studies use DRL to modify a traffic light's pre-defined behavior. [START_REF] Li | Traffic signal timing via deep reinforcement learning[END_REF], [START_REF] Tan | Deep reinforcement learning for adaptive traffic signal control[END_REF] as well as [START_REF] Wei | IntelliLight: A reinforcement learning approach for intelligent traffic light control[END_REF] used traffic lights with a static cycle and applied DRL to optimize the changing phase timing. The agent chooses whether the light switches to the next phase or remains at the current one, with a minimum time between two phase changes. Genders and Razavi (2019a) used a traffic light with a static cycle and an initial duration for each phase. The DRL agent can increase or decrease the duration of a phase and has to find the optimum duration for each phase.

Several studies used DRL to learn a dynamic cycle. In these, the DRL agent chooses at regular intervals which phase is the best. It selects not only the timing of phase changes, but also the order in which they take place. Some authors compare their DRL approaches to a deep learning approach [START_REF] Genders | Using a deep reinforcement learning agent for traffic signal control[END_REF] or to other DRL approaches [START_REF] Mousavi | Traffic light control using deep policygradient and value-function-based reinforcement learning[END_REF]. S. [START_REF] Wang | Deep reinforcement learning-based traffic signal control using high-resolution event-based data[END_REF] compared their DRL approach to one static and one dynamic traffic light control method on simulations with traffic demand evolving arbitrarily. Genders and Razavi (2019a) compared their DRL approach to the same methods, but simulated peak hours to test its robustness in a more realistic setup.

Positioning and contributions

Cyclists are known to prefer infrastructure that allows them to ride away from cars [START_REF] Caulfield | Determining bicycle infrastructure preferences -a case study of dublin[END_REF][START_REF] Tilahun | Trails, lanes, or traffic: Valuing bicycle facilities with an adaptive stated preference survey[END_REF]). That's part of the reason some experimentation are planned in France to allow bikes to set off earlier in order to regain sufficient speed before the departure of other vehicles. The idea behind this work is to extend the latter concept by creating specific green phases for cyclists. This type of space-saving infrastructure would make it possible to separate bike and car flows just as well as conventional dedicated lanes, but at the expense of waiting time. Indeed, more green phases means a longer cycle, and therefore a longer waiting time between two green phases for all lanes. In this paper, we propose a DRL based green phase selection method, allowing the creation of specific green phases for cyclists with a limited impact on the waiting times at the intersection. The agent controls the order and the timing of phase changes. We use real life counts data to compare our approach to existing traffic light control methods with realistic traffic. We hope that an infrastructure of this type with a sufficiently low impact on waiting time at the intersection would foster a modal shift toward cycling, thereby further increasing cyclists' safety [START_REF] Elvik | The non-linearity of risk and the promotion of environmentally sustainable transport[END_REF]. The contributions of this paper can be summarized as :

• We propose a traffic light system that is safer for cyclists as it includes specific green phases for them. • We design a phase-change method using DRL to reduce the waiting time increase caused by such an infrastructure, using and improving existing designs. • We use real life counts data to test our approach on a daily scale.

• We compare our approach to a dynamic one already deployed in order to demon-strate the relevance of using a DRL based solution.

Deep reinforcement learning

To limit the increase in waiting time caused by the addition of green phases for cyclists, a DRL based phase-change method is proposed. This solution uses the Double Dueling Deep Q-Network (3DQN) algorithm, which is detailed in this section.

Deep Q-Network

In 2015, [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] developed an algorithm called Deep Q-Network (DQN) capable of learning human level policies. DQN is based on a reinforcement learning algorithm called Q-learning [START_REF] Watkins | Q-learning[END_REF]. A function Q : S × A → R calculates the quality of a state-action combination. Every time the agent chooses an action a t , Q(s t , a t ) is updated using the Bellman equation. The final policy chooses the action with the best Q-value π(s) = max a Q(s, a). In DQN, a deep neural network called Q-network approximates Q and is noted Q(s, a; θ) where θ represents the parameters (i.e. the weights) of the neural network. The Q-network is trained by minimizing the loss function L defined as :

L(θ) = (Y DQN t -Q(s, a; θ)) 2 Y DQN t = r t + γ max a Q(s t+1 , a ; θ )) Y DQN t
is called the target value and θ represents the parameters of the target network, a second neural network with the same architecture as the Q-network. The target network is only used to compute Y t and is updated towards the Q-network during training.

Double Deep Q-Network

In 2016, [START_REF] Van Hasselt | Deep reinforcement learning with double q-learning[END_REF] have shown that DQN has an overestimation bias for Q-values and proposed a solution inspired by the double Q-learning algorithm [START_REF] Hasselt | Double q-learning[END_REF] called Double Deep Q-network (DDQN). DDQN is very similar to DQN but calculates the target value a little differently:

Y DDQN t = r t + γQ(s t+1 , arg max a Q(s t+1 , a ; θ); θ )
In DQN, the action a used to calculate the target value is chosen and evaluated by the target network where in DDQN, the action is chosen by the Q-network and evaluated by the target network. This reduces the overestimation of Q-values, thus increasing the quality of the policies produced.

Dueling Deep Q-Network

The advantage function is defined as A(s, a) = Q(s, a) -V (s) where V (s) represents the expected long term reward for being in the state s. From this equation, Q can be decomposed as the sum of V (s) and A(s, a). The idea behind Dueling Deep Q-Network developed by Z. [START_REF] Wang | Dueling network architectures for deep reinforcement learning[END_REF] is to decompose the Q-network in two streams : V (s; θ) which approximates V (s) and A(s, a; θ) which approximates A(s, a). The Q-network and the target network are therefore defined as :

Q(s, a; θ) = V (s; θ) + (A(s, a; θ) - 1 |A| a A(s, a ; θ)) Q(s, a; θ ) = V (s; θ ) + (A(s, a; θ ) - 1 |A| a A(s, a ; θ ))
with θ the parameters of the Q-network and θ the parameters of the target network. The mean of the approximations of advantages is subtracted from the approximations of A(s, a) to increase learning stability and performance.

Double Dueling Deep Q-Network (3DQN)

The Double DQN and the Dueling DQN can be combined to obtain the Double Dueling Deep Q-Network (3DQN). 3DQN shows better learning stability and performance than DQN or DDQN in general. In 3DQN and all algorithms derived from DQN, the agent doesn't learn after every action it performs. Instead, it has a memory buffer in which it stores all the transitions (s t , a t , s t+1 , r t , d). This transition means that during the step t, the agent received s t , chose the action a t which was rewarded by r t and made the environment in the state s t+1 . d contains the information whether s t+1 is a final state or not. The memory buffer has a finite size and if a new transition needs to be stored once it's full, the oldest is replaced by the new one. At each learning phase, the agent randomly fills a batch with transitions contained in the memory buffer and computes their mean loss. The mean loss is backpropagated to modify the weights of the Qnetwork. In our implementation, the weights of the target network are periodically replaced by the weights of the Q-network. Finally, a -greedy policy is used during training. When the agent needs to choose an action, it computes the Q-values using the Q-network. A random number r ∈ [0, 1] is generated. If r < , the action is chosen randomly. Otherwise, the action with the highest Q-value is chosen. is set to 1 at the beginning of the training and decreases as training progresses, in order to have a lot of exploration at the beginning and a lot of exploitation at the end.

3DQN approach

This section presents all the components the 3DQN agent will need to approximate an optimal policy, as well as how it is trained. First, the type of environment in which the agent will evolve is detailed. Then, the type of states that the environment will send to the agent is explained. The actions the agent will be able to perform in the environment, as well as the function rewarding them are defined. Finally, the training process and all the implementation choices made are explained.

Environment

As explained in Section 1.3, the traffic light has a green phase for each incoming car lanes but also for each incoming bike lanes. The environment in which the agent will evolve in is thus an intersection, made up of several intersecting axes, and controlled by a traffic light. For the sake of simplicity, each axis is assumed to be two-way, with a bike lane in each direction. If the car lanes on a given axis all have a green light at the same time (i.e. there are no specific green phases for turning cars), the light at an intersection of n axes will have 2n different green phases. For example, the set of green lanes on an intersection of 2 axes will be G = {g ax1 car , g ax1 bike , g ax2 car , g ax2 bike } with g axx t meaning that the vehicle type t on the axis x has the green light. A graphical example of a two axes intersection is shown in Figure 1, with ax 1 being the North-South (N-S) axis and ax 2 the East-West (E-W) axis.

States

The states sent to the agent need to condense the useful information about the environment. We identified two type of information about the vehicles at the intersection that the agent needs to be informed of. First, the position of the vehicles. To make the best decision, the agent needs to know how many vehicles arrive at the intersection and on which lane they are. The second one is the speed. The agent needs to make the difference between the vehicles that are waiting and the one that are moving. The more vehicles waiting in a lane, the more it is necessary to change phase to let them pass. As in the work of [START_REF] Liang | A deep reinforcement learning network for traffic light cycle control[END_REF], the states are two matrixes of same dimensions. They are named respectively the position matrix and the speed matrix.

The environment is divided in squares of 5-meters long. Only the squares belonging to the lanes arriving at the intersection are put in these matrices. The other ones can not contain useful information. The matrixes are thus smaller than those of [START_REF] Liang | A deep reinforcement learning network for traffic light cycle control[END_REF], as their dimensions are N × P with N the number of lanes arriving at the intersection and P the number of squares each lane contains. The position matrix contains the number of vehicles that are in each square, and the speed matrix contains the mean speed of the vehicles in each square. These matrixes could be reconstructed with cameras pointed at the lanes arriving at the intersection, and recent methods for estimating vehicle position and speed [START_REF] Gunawan | Detection of vehicle position and speed using camera calibration and image projection methods[END_REF]. A graphical example of a position matrix is shown in Figure 1.

Actions

The actions performed by the agent need to modify the behavior of the traffic light at an intersection. As it has been done several times (Genders & Razavi, 2019a;[START_REF] Mousavi | Traffic light control using deep policygradient and value-function-based reinforcement learning[END_REF][START_REF] Wang | Deep reinforcement learning-based traffic signal control using high-resolution event-based data[END_REF], the set of green phases G is used as the action space (see Section 3.1). Once a green phase start, 10s pass before the agent chooses the future green phase. If the chosen green phase is different from the actual one, a 4s orange light phase is triggered for the lanes that had green light until the decision. After this orange phase, or if the chosen green phase is the same as the actual one, a new 10s period is started before the agent chooses again. Waiting 10s after the start of a green phase avoids sudden changes that could surprise vehicles, and increases the stability of the agent's learning.

Rewards

Genders and Razavi (2019a) used the same action space and developed a reward function working with it. The rewards they used are adapted as explained below.

Reward function

The reward function is defined as :

r t = -(w b + w c ) 2
with w b and w c being respectively the number of waiting bikes and the number of waiting cars. A vehicle is considered to be waiting when its speed is less than 0.5 km/h. Note that the reward can only be negative, and that the more vehicles waiting, the more negative the reward. The agent must minimize the number of waiting vehicles in order to maximize the reward. The sum of w b and w c is squared to to discriminate more strongly against bad decisions and facilitate the start of the training.

Scaling factor

However, the large negative values that the reward function can take, especially at the start of training, may hamper the agent's convergence. Genders and Razavi (2019a) coped with this issue by dividing the rewards by r max , the biggest reward calculated. In our case, this normalization allowed the agent to converge, but has not led to the creation of effective policies. r mean , the mean of all calculated rewards, is therefore used instead. All the calculated rewards as well as the number of actions performed by the agent during training are stored, and r mean is updated at the end of each training episodes (which are detailed in Section 3.5). Using r mean as a scaling factor incites the agent to perform actions that are better on average than those it has performed so far.As the agent improves, the average rewards will increase, pushing the agent ever further to become better. This allows the agent to finish training with a highperformance policy.

Training

Q-network architecture

A state is made up of two matrixes that can be seen as a two channels image condensing the useful information of traffic at the intersection. Thus, the Q-network needs to be able to find patterns in an image in order to correctly process the states it receives. Convolutional layers extract features from images to lower dimensions without loosing their characteristics. A convolutional layer is composed of kernels, which are matrixes of small dimensions. Each kernel has its own weights in order to find a specific type of pattern in the image. The kernels slide along the image, and a multiplication is performed between their weights and the image values for each sub-area they cover. The Q-network is composed of two convolutional layers containing 16 kernels of dimension 2x2. These layers are followed by two fully connected layers of 128 nodes each. Then comes two output layers for the value function and the advantage function (see Section 2.3).The ReLU activation function is used between all layers in order to provide non-linear properties to the Q-network. Figure 2 summarizes the architecture of the Q-network.

Hyperparameters

The values of the hyperparameters used during training as well as all the variables used in this paper are shown in Table A1 in Appendix A. The agent is driven in episodes. During each episode, vehicles can spawn during 6 simulated hours, one step per second.An episode ends when all vehicles have spawned and no vehicle remain in the simulation. A vehicle disappears once it has reached its destination, which is the end of one of the lanes leaving the intersection. After acting pt times, the agent learns at the end of each episode. Training stops when the agent has made f actions. decreases linearly each time the agent chooses an action, and reaches its ending value at the f th action. Finally, the target network is updated by replacing its weights with those of the Q-network every υ actions performed by the agent.

Experimental setup

In this section, the simulated environment(cyclists, cars, and traffic lights) is presented. Then the synthesis of the trafficbased on real count data is detailed. Finally, our performance evaluation methodology is explained.

Simulated environmnet

SUMO (Simulation of Urban MObility

) is used to simulate the environment. SUMO is a tool that allows different actors to interact on a road graph. The behavior of the different actors can be changed in real time. SUMO is commonly used in traffic light control studies using simulation. Figure 3 shows a screenshot of the environment. An intersection with two axes crossing (NS for North-South and EW for East-West) is managed by a traffic light. Each road arriving at the intersection is 150m long and has two lanes, one for bikes and one for cars. The traffic light has four green phases G = {g N S car , g N S bike , g EW car , g EW bike }. g N S bike is activated on Figure 3. The vehicles spawn at the edge of a road and have the edge of another road as their destinations. When a vehicle is on green and wants to turn left, it must give way to oncoming traffic before crossing the intersection. This adds a waiting time that does not depend on the green phase of the traffic light. Moreover, when possible, vehicles tend to position themselves in the middle of the intersection when waiting to turn left to let vehicles behind them pass. Unfortunately, SUMO doesn't allow this behavior, making everyone wait behind it when a vehicle wanting to turn left is waiting to do so. This adds even more waiting time not dependent on the state of the traffic light. Vehicles are therefore prohibited from turning left. Vehicles have equal probabilities of having as destination one of the two roads they can access without turning left.

Vehicle counts and traffic synthesis

The city of Paris makes automatic vehicle counter data available on its open-data website1 . Various temporal aggregations are available, from the year to the hour. Hourly aggregation is used here as it is the most precise. The data of two unidirectional car counters and one bidirectional bike counter are collected. The counters are located on boulevard Montparnasse and are close to each other. The boulevard Montparnasse is two-way, with two car lanes and a bike lane in each direction. The number of counted cars is halved, since our simulation has a single car lane in each direction. The data are from June 20, 2023, a Tuesday with good weather.

Figure 4 shows the sum of the number of vehicles counted in both directions per hour. Differences between car and bike distribution are observable. There are hardly any bikes counted at night, a relatively stable number of bikes from 10:00 to 16:00 and two huge peaks at 08:00 and 18:00, the usual commuting hours. For the cars, the number decreases during all the night before increasing again at 06:00. The counted cars then reach a plateau that lasts until 17:00. Then comes a small increase with a peak at 19:00 before a decrease that will last until late at night. These differences between the vehicles distribution shows the importance of using real data on a daily scale. The 3DQN approach must be able to adapt its decisions to changes in both car and bike traffic in order to be efficient on a daily scale.

In order to simulate the traffic at the scale of each vehicle, we assume that the number of vehicle arriving at lane l each second follows a Poisson process p l . The intensity λ p l (t) of this process at time t is considered fixed during each hour and follows the aggregated count data : extrapolating the hourly aggregation of real vehicles counts.

λ p l (t) = c l h(t)

Performance evaluation methodology

In our settings, the traffic is never saturated at the exit of the intersection. The performance of a solution is therefore the time spend by vehicles before they reach the intersection and get a green light. After training, whole days (3600×24 = 86400 steps) are simulated and the mean waiting time of vehicles cars is calculatedfor each hour. The 3DQN approach is compared to the following traffic light control mechanisms.

• static unsecured : The first approach compared, named unsecured in the following, serves as a baseline to quantify the addition of waiting times when securing the traffic light for cyclists. This is a classic static traffic light, with only one green phase per axis. The bikes and the cars on the same axis cross the intersection at the same time. • static secured : This approach is a naive one. It simply consists in preventing bikes from passing during the existing green phases and adding a green phase for bikes to all axes containing at least one bike lane. In our case, the traffic light has four green phases when using this approach, each lasting 40s. This behavior shows the huge increase in waiting time for all vehicles if the bike safety system is naively implemented. • actuated : The static secured approach serves mainly to demonstrate the importance of a dynamic phase-change method when implementing specific green phases for bikes. Comparing the 3DQN approach which is highly-dynamic only to a static approach would not be fair. Thus, actuated is used. actuated is a dynamic phase-change method commonly implemented in Germany [START_REF] Brilon | Priority for public transit in germany[END_REF]. A traffic light in actuated mode has vehicle detectors on each of its incoming lane, approximately 50m ahead. The traffic light has a duration parameter, and each green phase has a minimum duration minDur and a maximum duration maxDur. When a green phase start, the traffic light waits minDur seconds before starting a counter of duration seconds. Once the counter reaches zero, the traffic light switches to the next phase of its cycle. If one of the detector on the lanes with the green light detects a vehicle before the counter reaches zero, the counter is reset. If a green phase reaches a duration of maxDur, the traffic light switches to the next phase, regardless of the counter's status. In the implementation of actuated used, all green phases have a minDur of 10s, a maxDur of 40s and the duration parameter is set to 5s. actuated is commonly used to test the performance of DRL approaches doing traffic light control (Genders & Razavi, 2019a;[START_REF] Tan | Deep reinforcement learning for adaptive traffic signal control[END_REF][START_REF] Wang | Deep reinforcement learning-based traffic signal control using high-resolution event-based data[END_REF].

Results

The results are in two parts. The first ones are on an hourly scale for a simulation of one day. An initial simulation with a traffic light controlled by the 3DQN agent is carried out, with our random traffic synthesis. The times and places vehicles appear during this simulation are recorded. Three further simulations, one for each other control mechanisms, are then carried out with the traffic trace of the first simulation. This allows a comparison under perfectly identical conditions. The second part of the results is at a larger scale. The traffic demand of bikes is changed, and five different initial simulations are made for each bike traffic demand. The actuated approach is then used for each initial simulation in the same way as explained above.

Hourly results

Figure 5 shows the results produced by a simulation of one day. The x-axis shows the hours of the day, from 0 to 23, and the y-axis shows the number of vehicles on Figure 5a and the mean waiting time of the vehicles on Figure 5b. The vehicle distribution curves have the same shape as those shown in Figure 4 without being perfectly identical. This is due to the randomness of the Poisson processes.

As expected, the unsecured approach does better than all the other ones. Unlike other curves, the unsecured 's one is flat and changes little with traffic. The traffic is indeed not strong enough to saturate the road graph in this configuration. All vehicles waiting at a red phase are able to cross the intersection on the first green phase they are granted. Adding green phases dedicated to cyclists increases the occupancy rate of lanes since car lanes and bike lanes on the same axis are then emptied successively.

As expected, the worst approach is the static secured one. Doubling the duration of the traffic light's cycle results in an explosion of the waiting time, as each lane has to wait longer between two green phases. Reducing green phase durations to 20s increases the average waiting time even further, with lanes becoming more saturated because they don't have enough time to empty during their green phases.

actuated does much better than the static secured approach whatever the time of the day, and handles the double car-bike peak at 19:00 with much greater ease. As the green phases duration are adapted to the traffic situation, the lanes are much less saturated. The green light time wasted for empty lanes is greatly reduced. The 3DQN approach does even better than actuated, with a lower mean waiting time at almost every hour. The gain in waiting time is significant during low-traffic hours. This is logical, as actuated is less accurate when lanes are empty or almost empty. Indeed, an actuated traffic light must wait a minimum of 15s for each green phase and follows a static cycle, regardless of traffic conditions. 3DQN is able to detect vehicles and change the green phase accordingly. But as lanes fill up, the cycle becomes less important than the phase change timing. The 3DQN appoach being able to do both, it is still slightly better than actuated during high-traffic hours. The performance of the 3DQN approach is closest to that of the unsecured one. It's worth noting, however, that the less traffic there is, the better the 3DQN approach performs.

On the scale of the day, adding specific green phases for cyclists multiplies the mean waiting time by 4.35 when using the 40s naive approach, by 1.69 when using the actuated approach and by 1.55 when using the 3DQN approach. Working with higher traffic would certainly allow us to reach the saturation rate of the unsecured method, where it would be less effective, but this saturation would be all the more noticeable with the addition of green lanes for cyclists. This would further degrade the performance of the other approaches, and possibly even prevent the agent's learning process.

Robustness to changes in bike traffic

The 3DQN approach limits the increase of the mean vehicle waiting times on a day with traffic similar to that used during training. However, the choice of whether to cycle is strongly correlated with the weather. As the traffic demands are calculated on the basis of data from a sunny day, the number of cyclists is likely to be lower on bad weather days. On the other hand, the aim of the secured intersection is to provide safe passage for cyclists, and the deployment of such infrastructure could attract cyclists, leading to an increase in bike traffic. The robustness of a 3DQN approach to changes in bike traffic therefore appears to be an important point to check. A multiplying coefficient which goes from 0.5 to 1.5 in steps of 0.1 is set. The number of bikes counted each hour is multiplied by this coefficient. That varies the bike traffic linearly from 50% to 150% of the observed one in the count data. Five days are simulated for each new spawn rate. actuated being the best comparison approach on a secured traffic light, it is used to evaluate the performance of 3DQN. Since night-time hours are not very relevant due to the absence of traffic, the results shown in Figure 6 and 7 focus on the hours between 6h and 20h.

Figure 6 shows the number of vehicles (6a) and the sum of all the waiting times (6b) for each multiplying coefficient. Logically, the number of cars per simulation is stable and the number of bikes is increasing linearly. The sum of vehicle waiting times generated by 3DQN starts out lower than the one generated by actuated. The two increase progressively, finally coming together when the coefficient reaches 1.5. When the coefficient is 1.5, 3DQN makes vehicles wait longer than actuated. This is consistent with hour-by-hour observations. The fewer vehicles there are at the intersection, the higher the probability that actuated will leave an empty lane green, due to its fixed cycle and minimum green phase time. This logically favors 3DQN, which is by nature more dynamic. It still shows that 3DQN is able to adapt to a decrease in bike traffic without any significant impact on its decision-making performance. This is probably due to the high variations in both car and bike traffic that the agent faces during training, with nighttime hours having very low traffic levels. 3DQN's performance is also fairly stable as the number of bikes increases. The sum of waiting times logically increases, as more bikes means more green light time is needed to clear the bike lanes, which impacts the waiting times of all vehicles at the intersection. However, the difference in performance between the two approaches diminishes as the coefficient increases, until it reaches 1.5. When the multiplying coefficient reaches 1.5, not only does 3DQN make vehicles wait longer on average than actuated, but the discrepancy in performance between simulations also increases significantly. The first reason for this is that, as explained above, the more vehicles there are, the better the actuated performance. But it's also probably due to less relevant decisions made by 3DQN. The further the bike traffic moves away from the one used during training, the greater the probability that the agent will receive a state it is not accustomed to handling. This situation may lead the agent not to make the best possible decision, resulting in more saturated lanes that it is no longer able to manage properly. The 3DQN agent therefore appears to be robust with bike traffic ranging from 50% to 140% of the traffic used during training.

To go into more details, Figure 7 shows the average waiting time for bikes (7a) and cars (7b). For actuated, mean waiting times evolve in a fairly similar way for both types of vehicle, with a slight linear increase. 3DQN is different. For cars, there is also a linear increase, but much more pronounced to the point where the average waiting time for cars observed with 3DQN exceeds that of actuated when the multiplier coefficient reaches 1.1. On the other hand, the average waiting time for bikes is stable and even decreases until the coefficient reaches 1.1, before rising slightly. This is a surprise, as we expected the mean waiting times for the two types of vehicles to evolve in the same way as the sum of the waiting times in Figure 6. Instead, the average waiting time for cars increases more sharply, allowing bikes to wait less. The agent seems to wait for a lane to reach a certain occupancy rate before turning it green, thus favoring cyclists in the experiment. Indeed, as the number of cyclists spawning in the simulations increases, the bike lanes reach this occupancy rate more quickly, prompting the agent to turn them green more often. As a result, the waiting time for bikes does not increase despite their higher numbers, but cars are given the green light less often. That's why the mean waiting time for cars increases this way. The sum of mean waiting times observed with 3DQN ends up exceeding that of actuated, because by giving preference to bikes in this way, the mean waiting time for cars ends up being too great for the agent to be as efficient as actuated.

Conclusion

In this paper, we proposed a traffic light allowing cyclists to cross an intersection safely during dedicated green phases. Results show that adapting the behavior of a traffic light to the traffic situation using DRL can reduce the cost in waiting time induced by securing the passage of cyclists with dedicated green phases. A 3DQN agent is capable of controlling this type of secure traffic light with different levels of traffic, enabling it to absorb fluctuations in traffic on a typical day. Performance remains relatively stable with moderate deviations from training conditions in terms of bike traffic. However, even though real count data are used to simulate traffic, it is modelled with Poisson processes, which is an unrealistic simplification of traffic demand. In the same spirit of realism, the vehicles in the simulations use SUMO's default behavior, which perfectly respects the rules of the road. Experiments with traffic demand and individual behaviors closer to reality must be carried out before such an infrastructure can be deployed. If these experiments prove conclusive, an interesting extension of our work would be to observe the distribution of vehicles exiting the intersection and to train another DRL agent with these distributions, with the aim of creating DRL driven green waves along a path with several intersections. We would also like to point out that an agent has been trained using another type of (policy-based) DRL algorithm called Proximal Policy Optimization (PPO). Although this agent has converged, its final policy performs less well than that of the 3DQN agent, but we don't know whether this is due to the nature of PPO or to our implementation. We are making available the code containing both algorithms for future works.

Figure 1 .

 1 Figure 1. Diagram showing the construction of the position matrix from an image of an intersection.

Figure 2 .

 2 Figure 2. Diagram showing the structure of the Q-network.

Figure 3 .

 3 Figure 3. Screenshot of the environment simulated by SUMO.

  the number of vehicles (bikes or cars) counted at hour h(t) on lane l. To summarize, the environment simulates the crossing of two Montparnasse boulevards during one day, each with only one car lane, and with traffic synthesized by

  (a) Hourly bike count. (b) Hourly car count.

Figure 4 .

 4 Figure 4. Average number of vehicles counted per hour in both directions on boulevard Montparnasse on June 20, 2023.

  (a) Hourly number of vehicles. (b) Hourly mean waiting time.

Figure 5 .

 5 Figure 5. Hourly number of vehicles and mean waiting time for a simulation of one day.

  (a) Number of vehicles. (b) Sum of waiting times.

Figure 6 .

 6 Figure 6. Number of vehicles and sum of waiting times with respect to bike traffic changes (from 6h to 20h).

  (a) Mean waiting times of bikes. (b) Mean waiting times of cars.

Figure 7 .

 7 Figure 7. Mean waiting times of vehicles with respect to bike traffic changes (from 6h to 20h).

https://opendata.paris.fr/pages/home/

Disclosure statement

The authors declare that they have no relevant or material financial interests that relate to the research described in this paper.

Appendix A. Variables used


Emeric Perez
email: emeric.perez@cea.fr

Carlos Augusto

Yasser Moursy

Bruno Allard

Sami Oukassi

Gaël Pillonnet
email: gael.pillonnet@cea.fr

Ultra-Low Frequency DC-DC Converters Using Switched Batteries

Keywords: Switching converter, battery, switched capacitor converter, DC-DC converter

To address ultra-low frequency DC-DC converters, we explore the concept of the switched battery converter. It is mainly derived from the switched capacitor converter topology replacing the flying capacitors by flying batteries. The best-inclass energy density of batteries compared to inductors and capacitors reduces drastically the operating frequency at low output power delivery while maintaining low output ripple. An experimental evaluation using mm 3 -scale off-the-shelf NiMH battery is described in this paper to present a first proof-ofconcept of switched-battery converter. The characterized 2:1 DC-DC converter achieves 25mW at 1.2V output voltage in 74mm 3 battery operating as low as 0.1 Hz operating frequency while maintaining less than 15% output voltage ripple without output filtering.

I. INTRODUCTION

In the context of fully integrated voltage regulators (FIVR), the footprint of DC-DC converters is mainly limited by the passive components (capacitors, inductors). The switched capacitors converters (SCC) have been considered as good candidates thanks to high power density and integrability in CMOS technology compared to inductors below the cm 3 scale [1] - [START_REF] May | Analysis of soft charging switched capacitor power converters[END_REF]. The SCC output impedance is inversely proportional to the product of the capacitor value C f ly and the operating frequency f sw [START_REF] Seeman | A comparative analysis of Switched-Capacitor and inductor-based DC-DC conversion technologies[END_REF]. Hence, increasing the operating frequency by 10× reduces more or less the footprint of the passive component with a similar factor. However, the switching losses are increased by 10× thus reducing the power efficiency. At low power -e.g. mW scale and below -as illustrated in Fig. 1 the switching losses limit the achievable operating frequency if maintaining an efficiency larger than 90%. As the operating frequency is limited, capacitors with higher capacitance density are an alternative to increase the power density (W/mm 3 ) [4]- [START_REF] Andersen | A 10 W On-Chip Switched Capacitor Voltage Regulator With Feedforward Regulation Capability for Granular Microprocessor Power Delivery[END_REF]. Nevertheless, the increase in the capacitance density is limited up to few 200 nF/mm 2 [START_REF] Chang | A fully-integrated switched-capacitor 2:1 voltage converter with regulation capability and 90% efficiency at 2.3A/mm2[END_REF] (deep-trench capacitors), failing to maintain non-negligible switching loss at low power. Alternatively to capacitor and inductor, a third energy Batteries exhibit more than three order of magnitude higher energy density compared to traditional passives thanks to a volumetric based energy storage of ions. Solid-state batteries prior art proved the feasibility of high energy density even at mm 3 -scale µ-batteries [START_REF] Oukassi | Millimeter scale thin film batteries for integrated high energy density storage[END_REF] and open the possibility of smart integration in a single chip. Based on this excellent energy property, we propose to replace capacitors by batteries in a new converter concept called switched battery converters (SBC). Employing batteries allow to work at ultra-low operating frequencies (Hz range) and maintain low output ripple, thus reduces the switching losses at low output power delivery. High efficiency at low power is essential for IoT applications, especially to power always-on IPs (Integrated Processors) such oscillators or wake-up radio. Nevertheless, batteries suffer from limited power capability and life cycle if large depth of discharge is experienced. Maximum power achievable for the battery comes from the current limit. This paper explores the capability of mm 3 -scale NiMH battery as a candidate for DC-DC power conversion at low power (in mW range). Firstly the key enabler of batteries compared to passives is emphasized, then we show the battery response imposed by the DC- DC converter mission profile. The battery is finally included in an step-down SBC to give some experimental results on achievable performances. A comparison to prior art is given to provide figures of merit. Furthermore, a discussion section underlines both passives inherent limitations and batteries possible optimization for SBC usage.

II. BATTERY: AN ALTERNATIVE?

To understand the fundamental differences between batteries and traditional passives, we have to look at the mechanisms of energy storage. Traditional passives rely on storage of charges on a surface (capacitor) or magnetic field in a volume (inductor), while battery stores energy inside a volumetric crystalline structure. Besides electrostatic and electromagnetic processes are fast compared to chemical reactions involved in batteries thus providing higher power density to capacitors and inductors.

In regards to [START_REF] Seeman | A comparative analysis of Switched-Capacitor and inductor-based DC-DC conversion technologies[END_REF] the power and energy density strongly depend on the passive component size. Tab. I compares three selected components wih similar volume, i.e. mm 3 -scale, corresponding to typical coin battery size. They share low voltage/current ratings -lower than 6.3V and 0.145A to compare similar devices. Battery energy density is 1,000× higher compared to their counterparts. The maximum battery power density is 10,000× lower than for the one of capacitor and inductor because the amount of power delivered by the battery is independent from the self-resonance frequency (SRF) but limited by the maximal allowed current. If high efficiency at low power level is targeted, the operating frequency is limited to few Hz as mentioned in [START_REF] Paidimarri | 10.8 A Buck converter with 240pW quiescent power, 92% peak efficiency and a 2×106 dynamic range[END_REF] and shown in the schematic of Fig. 1. Power density of the battery at 1 Hz is hence similar to the one of capacitor and significantly higher than the one of inductor.

III. BATTERY µ-CYCLE

As illustrated in Tab. I batteries are competitive in the Hertz range compared to other passives. SBC operation mode requires the battery to experience radically different usage profile in comparison with large State of Charge (SoC) utilization in hours scale. SBC operation creates a new usage profile for batteries which has not been explored yet in prior art. A battery under test (BUT) has been then introduced to depict their innovative ability to be cycled in µ-Coulomb-scale to milli-Coulomb-scale at second time scale for mimicking SBC operation mode. Such usage profile involves only a small-displacement Q E -much less than a percent of battery capacity -around a bias point as illustrated in (V,Q) characteristic in Fig. 2(b). This bias point Q DC is chosen at 50% State of Charge (SoC) point as the battery resistance is here minimal. In the following, the SBC mission profile of the battery is called µ-cycle. Aiming to explore off-the-shelf battery, we chose a 74 mm 3 NiMH battery model (V6HR) of 1.2 V nominal voltage (V bat ) within the power range batteries. The nominal capacity of our BUT is 6 mAh and the manufacturer DC current limit I lim is fixed at 3C (C-rate, equivalent to 18 mA). Battery DC resistance (R int ) is given by the manufacturer to be 6 Ω. BUT has been then chosen to meet the milliwatt power delivery initial requirement.

Considering the SBC power density to be inherently limited by the impedance of the BUT, a small-signal impedance Z bat against frequency has been acquired experimentally with Electrochemical Impedance Spectroscopy (EIS) around the 50% SoC point. As shown in Fig. 2(a) the real part of the impedance is dominant below the kHz and roughly equal to the battery DC resistance. Impedance magnitude varies around the value of 6 Ω with only a factor of 3 over four decades of frequency. The battery can be used up to ten of kHz, with a resonance frequency around the hundred of kHz. As a first approach, the maximum battery output power level P max is limited to V bat 2 /4Z bat which gives 60 mW. Nevertheless several electrochemical limitations are not assessed with small-signal analysis as it implies currents far lower than 1C order of magnitude. A quasi-static current limit (I lim ) is indeed fixed by the electrochemical stability potential window of the battery electrolyte limiting the maximal output power level [START_REF] Méry | A Critical Review for an Accurate Electrochemical Stability Window Measurement of Solid Polymer and Composite Electrolytes[END_REF] [START_REF] Lee | Power Capability Analysis of Lithium Battery and Supercapacitor by Pulse Duration[END_REF] in full charging/discharging conditions. In practice the achievable battery quasi-static power is given by V bat × I lim , here 22 mW which is lower than the calculated P max . BUT power indicated in Tab. I is a little bit higher as it corresponds to tested BUT power under µ-cycle.

The battery impedance also depends on the current level because of limited rate for ion transfer under large operating currents, i.e. above 1C [START_REF] Barai | A study of the influence of measurement timescale on internal resistance characterisation methodologies for lithium-ion cells[END_REF]. Consequently the small-signal analysis has to be complemented with a large-signal characterization of the BUT.

To assess large-signal characterisation, a rectangular current wave of amplitude I bat,in (mA range) and frequency f sw (Hz range) with zero mean value has been applied to our BUT to reproduce SBC µ-cycle operation. Fig. 2(b) and Fig. 2(c) respectively SBC µ-cycle operation and the resulting voltage response acquisition. Two major voltage components -representing the large signal battery behavior -in BUT response has been identified in Fig. 2(c): i) IR drop V IR due to the internal resistance ii) redox voltage V redox related to redox reactions occuring upon µ-cycling. As expected, the IR drop is linearly correlated to current as shown in Fig. 3(a). The resistance extracted from the slope in Fig. 3(a) is R µc and gives 7Ω +/-15% over three frequency decades. R µc is slightly different from R int and previously obtained small-signal impedance highlighting the interest of large-signal characterization. A small frequency dependance is observed in IR drop as it has been previously observed in small-signal analysis in Fig. 2(a). Redox voltage follows a logarithmic frequency law which denotes a superposition of chemical mechanisms detailed in [START_REF] Barai | A study of the influence of measurement timescale on internal resistance characterisation methodologies for lithium-ion cells[END_REF]: i) a ion diffusion phenomenon, chemically defined as function of the square root of time ; ii) a charge transfer process, corresponding to a small-displacement on the (V,Q) characteristic. Moreover, it appears that redox voltage evolves more or less linearly with respect to current (R 2 =0.99, @ 10 Hz) as shown with gray arrow in Fig. 3(b). To summarize, ionic voltage decreases with frequency while both redox voltage and IR drop increases with current.

We observed that the battery average voltage (V bat,avg ) is close to the open-circuit relaxed potential (OCRV) under rectangular current wave input, in steady-state conditions. This value is independent from the rectangular current level as well as the frequency as shown in Fig. 3(c) what constitutes a new result. Moreover, V bat,avg is stable over 10 7 µ-cycles, while operating at the battery current limit under rectangular current wave. It indicates an extended cycle life of batteries under µcycles as compared to usual cycle life of NiMH batteries (10 3 cycles rated by the manufacturer) in accordance to [START_REF] Evans | Multicycle Testing of Commercial Coin Cells for Buffering of Harvested Energy for the IoT[END_REF].

IV. SWITCHED BATTERY CONVERTERS

The BUT is incorporated in an widely adopted DC-DC power stage called SBC inspired from SCC. Our implementation is a 2:1 step-down converter as shown in Fig. 4a. More optimal power stage could be be proposed but for clarity purposes, we have adopted the most classical topology for easier comparison with capacitive-based converterters. In open-circuit condition the output voltage is half of the input voltage. The SBC is composed of four switches (0.25 Ω on-state resistance) and a single flying battery. One cycle consists of two equal duration phases. During the first phase ϕ 1 the battery is connected in series with the input voltage source (V in ) and the output voltage (V out ). The battery receives a charge Q 1 equal to I out t ϕ1 . During a second phase ϕ 2 the battery is partially discharged through the load. The battery releases Q 2 charges (I out t ϕ2 ).

In SBC, the amount of charge transfer into the battery Q E is far from Q DC . The battery experiences hence very small SoC variation (<1%). If perfect charge balancing is assumed the battery returns to its initial state of charge Q DC . If perfect charge balancing is not respected, a battery monitoring can be implemented. Thanks to expected low SoC drift (Q E /Q DC ≪ 1) monitoring would be activated intermittently. Battery and output voltage experimental waveforms are given in Fig. 4b when the output current and operating frequency are 6 mA and 1Hz, respectively. The battery voltage follows the same shape as introduced in Fig. 2(c).

Aforementioned, the average battery voltage V bat,avg is independent from I out but its value is dependent on the input voltage. Under long-time scale -e.g much greater than hour, to let chemical species concentrations reach their new equilibrium point -V bat,avg will tend to the value of V in /2 denoted V bat,st . It demonstrates that battery in SBC doesn't behave anymore as a constant voltage source but as an adaptative voltage source to the input. Careful design considerations must be hence taken to ensure that the value of V in /2 doesn't exceed the maximum battery voltage rating.

The average output voltage V out,avg of SBC can be expressed as [START_REF] Mathúna | Review of Integrated Magnetics for Power Supply on Chip (PwrSoC)[END_REF].

|V out,avg | = | R load R load + Z out | V in 2 (1)
Where R load is the load resistance and Z out the SBC output impedance.

The output impedance is derived from the series combination of on-switch resistance and R µc assuming hence negligible charge-sharing losses compared to SCC.

The output voltage ripple shown in Fig. 4b is due to the redox voltage component of the battery as defined in Section III. If input voltage level changes abruptly an extra-ripple component equal to V in -2V bat,avg appears on the output while V bat,avg is different from V bat,st . Hence, battery high energy density becomes a penalty in the case of fast SBC dynamic responses.

Conduction effciency η c (e.g without switch drive loss) of SBC is depicted in Fig. 5(a) and can be roughly estimated using :

η c = R load R load + ℜ(Z out ) (2)
SBC features 14.3 mW @ 90% conduction efficiency at only 0.1 Hz. SBC output power varies slightly over three decades of operating frequency (0.1 Hz to 10 Hz) showing frequency independance. A SCC featuring a passive of mm 3 scale (respectively 74mm 3 and 59mm 3 for SBC and SCC) operates at 10× higher operating frequency (100 Hz) compared to SBC. SCC fails to provide the same output power level as SBC below 100 Hz showing large contribution of charge sharing losses at such low frequency. High induced charge sharing losses limit drastically the power capability of capacitors components as introduced in Section II. Fig. 5(b) shows the depth of discharge (DoD) against output power. The DoD is defined by Q E /Q T where Q T is the nominal capacity. In SBC the DoD is lower than 10 4 ppm at maximal output power level and can be linearly scaled down with frequency. This low DoD allows extended battery life-cycle as reported in [START_REF] Evans | Multicycle Testing of Commercial Coin Cells for Buffering of Harvested Energy for the IoT[END_REF], enabling battery life-time to be compatible with continous switching operation. Fig. 5(c) explores the relative output voltage ripple (∆V out /V out,avg ) against output power. V in in SBC has been set to twice the average battery voltage to minimize the ripple (under short-time scale). Even at 0.1 Hz and without any output decoupling capacitors the ripple is limited to a moderate 15% @ 25mW. The maximal ripple is 4× higher using SCC operating at 1,000× higher frequency.

A second battery used in SBC as an output filter seems to be not beneficial as the battery sees the switching edges and responds to it with limited dynamics i.e. voltage peaks. Nevertheless, a decoupling capacitor can be added to the output to further reduce the moderate output voltage ripple without significant extra area cost.

V. PERFORMANCE COMPARISON

The proposed SBC is compared to existing DC-DC converters providing the same output power level. Fig. 6(a) illustrates converter energy density versus converter power density both estimated from litterature data. A declination of the proposed SBC has been achieved with another battery model called V150H (NiMH). V150H battery features one order higher in size scale compared to V6HR battery. Converter energy density is defined as the energy stored in the converter considering the passive or battery volume. For SCC the flying silicon capacitor area is estimated from reference data and we assumed 1 mm of die thickness. For inductive converters the off-chip inductor volume is estimated from off-the-shelf CMOS inductors.

In Fig. 6(a) the converter power density is the maximum output power density at peak efficiency. The SBC energy density is far higher than prior art but enhances a lower power capability. These considerations confirm the fundamental differences between batteries and passives introduced in Section II. Even if power density is one keypoint for converter design, energy density is also a key enabler to provide low output ripple at low operating frequency.

1/M E = ∆E trans E store (3)
Where ∆E trans is the energy transfered to the output during a converter period, E store is the energy continuously stored in the converter. ∆E trans is calculated using the output power at peak efficiency considering maximal reported operating frequency (f sw,max ). E store is estimated from an average bias current or bias voltage of the passives.

As mentionned in the paper motivation, low operating frequency leverages efficiency at low power. Higher M F shows better ability to fulfill this target.

Even if SBC operates at operating frequency four order of magnitude lower, SBC features high M E meaning ability to provide low output ripple without output filtering. To counteract the low M E (high passive energy utilization) in reported litterature an output capacitor or interleaved scheme may be added to maintain acceptable output ripple. This strategy leads to additional passive components thus reducing the overall power density.

VI. DISCUSSION

Passive energy density is inherently limited because of technological feasibility on the dielectric and magnetic materials. According to [START_REF] Johari | High-Density Embedded Deep Trench Capacitors in Silicon With Enhanced Breakdown Voltage[END_REF] the product of the dielectric permitivity ε r and the maximum electric field in CMOS capacitors dielectric (CM ) is empirically limited to approximatelly 400 V 2 .cm -2 . Based on this value, maximal capacitor energy density is then bounded by 10 -2 J/mm 3 using the following formulae : (1/2)ε 0 CM . Magnetic energy density is defined as the product of the magnetic flux density B and the magnetic field density H. Currently [START_REF] Sugimoto | Current status and recent topics of rare-earth permanent magnets[END_REF] gives a magnetic cores maximal energy density to be equal to 59 MGoe (Mega Gauss Oersteds) at room temperature (Nd-Fe-B material). Taking half of this product gives the inductor energy density limit which is roughly equal to 10 -4 J/mm -3 . These results are shown in Fig. 6(a) and they are in accordance with [START_REF] Kyaw | Fundamental Examination of Multiple Potential Passive Component Technologies for Future Power Electronics[END_REF]. Passives devices in general cannot reach the level of V6HR and V150H selected batteries as shown in Fig. 6(a). Derating in converter energy density is observed in Fig. 6(a) as a consequence of design margins, packaging size, manufacturing constraints. Blue dotted arrow in Fig. 6(b) denotes the displacement of one litterature reference if operating frequency were tuned (M E × M F = constant). Considering the same M E ratio, batteries would feature always either a lower size or lower operating frequency compared to other passive devices.

Solid-state batteries may be considered to envisage a fully integrated SBC version. Using solid-state batteries manufacturing techniques open the way to a fine-grain co-optimization of battery power and energy density. As the energy density gap between batteries and other passive devices is high, a slight decrease in battery energy density to gain power delivery capability by co-optimization is tolerable.

Operating at relatively low-power levels i.e. inside the battery safe operating current limit prevents thermal management issues. Battery internal resistance power losses can be then dissipated on the relatively large considered battery surface (0.7 mW/mm 2 ).

Battery ageing concerns under SBC operation hasn't been deeply investigated in this study due to timing constraints. Nervertheless, [START_REF] Grillon | Cycle life and statistical predictive reliability model for all-solid-state thin film microbatteries[END_REF] suggests that spending time within high battery charging / discharging states (respectively high and low SoC) is a key contributor to battery ageing. Hence, using a 50% SoC biasing point seems to be a preventive action to counteract battery ageing. Three months of "intensive" experiments -e.g. 500 hours of experiment with accumulated 1 billion µ-cycles -using a single battery sample showed no noticeable battery voltage drifts nor internal resistance shifts suggesting not accelerated battery ageing under SBC operation mode.

V6HR and V150H batteries are NiMH based batteries but Li-ion batteries have been also successfully tested upon µ-cycle operation in a SBC. Consequently, SBC is not limited to NiMH technology but further investigation has to be made to explore the advantages and disdavantages of each battery technology for SBC applications. This paper firstly introduced the SBC concept in open-loop but future works are required to explore close-loop structure to regulate the output voltage. SCC usually uses frequency-based control which is not envisageable for SBC as their output impedance is almost frequency independent at first order. A two stages regulation scheme could be adopted: a coarse-grain voltage regulation is insured by SBC while a LDO [START_REF] Quelen | LDO-Assisted Voltage Selector Over 0.5-to-1V VDD Range for Fine Grained DVS in FDSOI 28nm with 200ns/V Controlled Transition[END_REF] can be placed at the SBC output to fine-grain regulate output voltage considering low differential voltage and hence high LDO efficiency.

VII. CONCLUSION

The presented work deals with an innovative architecture of µ-power converter based on a battery (SBC). Thanks to best-in-class energy density, batteries allow ultra-low operating frequency and low output ripple. A first presented prototype achieves 340µW/mm 3 with 25 mW maximal tested power while maintaining 84% conduction efficiency using off-theshelf battery (V6HR). The output voltage ripple is limited to 1.2% at 1/10 of maximal power capability even if the operating frequency is reduced to 0.1 Hz.

Fig. 1 :

 1 Fig. 1: Losses in a converter against frequency [schematic example]: (a) Low power case (b) High power case

Fig. 2 :Fig. 3 :

 23 Fig. 2: (a) Experimental small-signal impedance of the battery (b) (V,Q) characteristic of the battery (c) Schematic battery voltage response to microcyles

Fig. 4 :

 4 Fig. 4: (a) Electrical schematic of a 2:1 SBC and (b) Experimental voltage curves of battery and output in a SBC (without output capacitor) in steadystate

Fig. 5 :

 5 Fig. 5: (a) Conduction efficiency (without switch drive loss) against Pout (b) Relative charge storage utilization (c) Relative output voltage ripple

Fig. 6 (

 6 Fig. 6(b) illustrates this statement by introducing a Figureof-Merit based on the passive energy utilization rate against the inverse of the maximal operating frequency by using M F = 1/f sw,max . To assess the ability of a converter to achieve low output ripple without output filtering, M E is introduced and defined by:

Fig. 6 :

 6 Fig. 6: Converter energy versus power (a) FoM graph (b)

TABLE I :

 I OFF-THE-SHELF DEVICES SAMPLE WITH MM 3 VOLUME

	Passive	Metric	Energy density [J.mm -3 ]	Power density [W.mm -3 ]	Power density [µW.mm -3 ]	Voltage/current rating	ESR [Ω]	Volume [mm 3 ]
		Frequency	-	@ SRF/10	@ 1 Hz
	Battery (V6HR)	3.6×10 -1	N.A	360	0.018 A	6	74
	Capacitor [7]	5×10 -3	2	50	6.3 Vdc	15×10 -3	60
	Inductor [8]	7×10 -8	0.01	0.07	0.145 A	2.9	69