Andreas Metzger

Clément Quinton
email: clement.quinton@univ-lille.fr

Zoltán Ádám Mann

Luciano Baresi
email: luciano.baresi@polimi.it

Klaus Pohl

Feature Model-Guided Online Reinforcement Learning for Self-Adaptive Services

Keywords: Adaptation, reinforcement learning, feature model, cloud service

Abstract. A self-adaptive service can maintain its QoS requirements in the presence of dynamic environment changes. To develop a self-adaptive service, service engineers have to create self-adaptation logic encoding when the service should execute which adaptation actions. However, developing self-adaptation logic may be difficult due to design time uncertainty; e.g., anticipating all potential environment changes at design time is in most cases infeasible. Online reinforcement learning addresses design time uncertainty by learning suitable adaptation actions through interactions with the environment at runtime. To learn more about its environment, reinforcement learning has to select actions that were not selected before, which is known as exploration. How exploration happens has an impact on the performance of the learning process. We focus on two problems related to how a service's adaptation actions are explored: [START_REF] Acher | Feature model differences[END_REF] Existing solutions randomly explore adaptation actions and thus may exhibit slow learning if there are many possible adaptation actions to choose from. [START_REF] Arabnejad | A comparison of reinforcement learning techniques for fuzzy cloud auto-scaling[END_REF] Existing solutions are unaware of service evolution, and thus may explore new adaptation actions introduced during such evolution rather late. We propose novel exploration strategies that use feature models (from software product line engineering) to guide exploration in the presence of many adaptation actions and in the presence of service evolution. Experimental results for a self-adaptive cloud management service indicate an average speed-up of the learning process of 58.8% in the presence of many adaptation actions, and of 61.3% in the presence of service evolution. The improved learning performance in turn led to an average QoS improvement of 7.8% and 23.7% respectively.

Introduction

A self-adaptive service is capable of modifying its own structure and behavior at runtime based on its perception of the environment, of itself and of its require-ments [START_REF] De Lemos | Software Engineering for Self-Adaptive Systems: A Second Research Roadmap[END_REF][START_REF] Salehie | Self-adaptive software: Landscape and research challenges[END_REF][START_REF] Metzger | Addressing highly dynamic changes in service-oriented systems: Towards agile evolution and adaptation[END_REF]. As an example, take a self-adaptive web service. Faced with a sudden increase in workload, the web service may reconfigure itself by deactivating optional system features. An online store, for instance, may deactivate its resource-intensive recommender engine in the presence of a high workload. By adapting itself at runtime, the web service is able to maintain its QoS requirements (here: performance) under changing workloads.

To develop a self-adaptive service, service engineers have to develop selfadaptation logic that encodes when and how the service should adapt itself. Among other concerns, this requires anticipating the potential environment states the service may encounter at runtime to define when the service should adapt itself. However, anticipating all potential environment states at design time is in most cases infeasible due to design time uncertainty [START_REF] Chen | Self-adaptive and online QoS modeling for cloud-based software services[END_REF][START_REF] D'ippolito | Hope for the best, prepare for the worst: multi-tier control for adaptive systems[END_REF]. In addition, due to simplified design assumptions, the precise effect of an adaptation action may not be known and thus accurately determining how the service should adapt itself is difficult [START_REF] D'ippolito | Hope for the best, prepare for the worst: multi-tier control for adaptive systems[END_REF]. As an example, while service engineers may know in principle that activating more features will have a negative impact on performance, exactly determining the performance impact is more challenging [START_REF] Siegmund | Performance-influence Models for Highly Configurable Systems[END_REF].

Online reinforcement learning (RL) is an emerging approach to address design time uncertainty of self-adaptive services by employing RL at runtime (see existing solutions discussed in Sec. 6). In general, RL aims to learn suitable actions via an agent's interactions with its environment [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]. The agent receives a reward for executing an action. The reward expresses how suitable that action was. The goal of RL is to optimize cumulative rewards.

Problem Statement

RL faces the exploration-exploitation dilemma [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]. To optimize cumulative rewards, actions should be selected that have shown to be suitable, which is known as exploitation. However, to discover such actions in the first place, actions that were not selected before should be selected, which is known as exploration. How exploration happens has an impact on the performance of the learning process [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF][START_REF] Bu | Coordinated self-configuration of virtual machines and appliances using a model-free learning approach[END_REF][START_REF] Filho | Defining emergent software using continuous selfassembly, perception, and learning[END_REF]. We focus on two problems related to how a service's set of possible adaptation actions, i.e., its adaptation space, is explored. Random exploration of adaptation space. Existing online RL solutions for self-adaptive services propose randomly selecting adaptation actions for exploration (see Sec. 6). The effectiveness of exploration therefore directly depends on the size of the adaptation space, because each adaptation action has an equal chance of being selected. Some RL algorithms can cope with a large space of actions, but require that the space of actions is continuous in order to generalize over unseen actions [START_REF] Nachum | Bridging the gap between value and policy based reinforcement learning[END_REF]. Self-adaptive services may have large, discrete adaptation spaces; e.g., if their adaptation actions entail changes of service compositions [START_REF] Moustafa | Learning efficient compositions for QoS-aware service provisioning[END_REF] or reconfigurations of service features [START_REF] Metzger | Coordinated run-time adaptation of variability-intensive systems: An application in cloud computing[END_REF]. A simple example is a service composition consisting of eight abstract services that may allow dynamically binding 2 concrete services each. Assuming no temporal or logical constraints on adaptation, this constitutes 2 8 = 256 possible adaptation actions.

In the presence of such large, discrete adaptation space, random exploration thus may lead to slow learning at runtime [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF][START_REF] Bu | Coordinated self-configuration of virtual machines and appliances using a model-free learning approach[END_REF][START_REF] Filho | Defining emergent software using continuous selfassembly, perception, and learning[END_REF]. Evolution-unaware exploration of adaptation space. Existing online RL solutions are unaware of service evolution [START_REF] Kinneer | Managing uncertainty in self-adaptive systems with plan reuse and stochastic search[END_REF][START_REF] Sharifloo | Learning and evolution in dynamic software product lines[END_REF]. They do not consider that a self-adaptive service -like any service -may undergo evolution [START_REF] Papazoglou | The challenges of service evolution[END_REF]. In contrast to self-adaptation, which refers to the automatic modification of the service by itself, evolution refers to the modification of the service by humans [START_REF] Metzger | Addressing highly dynamic changes in service-oriented systems: Towards agile evolution and adaptation[END_REF]. During evolution, service engineers may modify the service to correct bugs, remove no longer used features, or introduce new features. Service evolution means that the adaptation space may change, e.g., existing adaptation actions may be removed or new adaptation actions may be added. Some RL algorithms can cope with environments that change over time, so called non-stationary environments [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF][START_REF] Nachum | Bridging the gap between value and policy based reinforcement learning[END_REF]. However, a change of the adaptation space cannot be determined by observing the environment, as the adaptation space is an intrinsic property of the RL agent. As a result, existing solutions may explore new adaptation actions only with low probability (as all adaptation actions have an equal chance of being selected). It may thus take quite long until the new adaptation actions have been explored.

Contributions

We introduce exploration strategies for online RL that address (1) a service's potentially large adaptation space, and (2) changes of its adaptation space due to evolution. Our exploration strategies use feature models [START_REF] Metzger | Software product line engineering and variability management: Achievements and challenges[END_REF] to give structure to the service's adaptation space and thereby leverage additional information to guide exploration. A feature model is a tree or a directed acyclic graph of features, organized hierarchically. An adaptation action is represented by a valid feature combination specifying the target run-time configuration of the service.

Our strategies traverse the feature model to select the next adaptation action to be explored. By leveraging the structure of the feature model, our strategies guide the exploration process. In addition, our strategies detect added and removed adaptation actions by analyzing the change of the feature model due to evolution. Adaptation actions removed as a result of evolution are no longer explored, while added adaptation actions are explored first.

We implement our strategies as part of the Q-Learning RL algorithm [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF] widely used in the related work (see Sec. 6). We experimentally assess our strategies using an actual cloud resource management service and compare the learning performance with that of the widely used -greedy random exploration strategy.

In what follows, Sec. 2 explains fundamentals and a running example. Sec. 3 describes our exploration strategies and how they are integrated with RL algorithms. Sec. 4 presents the design and results of our experiments. Sec. 5 provides a critical discussion. Sec. 6 analyzes related work.

Fundamentals

Feature Models and Self-adaptation. A feature model is a tree of features organized hierarchically [START_REF] Metzger | Software product line engineering and variability management: Achievements and challenges[END_REF]. A feature can be decomposed into mandatory, op-tional or alternative sub-features. A mandatory sub-feature has to be activated if its parent feature is activated. While an optional sub-feature may or may not be activated, at least one of the alternative sub-features has to be activated if their parent feature is activated. Additional constraints, such as "excludes" or "requires", express inter-feature dependencies. Thereby, a feature model describes the possible and allowed feature combinations.

Feature models are traditionally used in software product line engineering to define the set of system variants at design time [START_REF] Metzger | Software product line engineering and variability management: Achievements and challenges[END_REF]. Dynamic software product lines extend the use of feature models to describe possible run-time configurations of a system [START_REF] Hinchey | Building dynamic software product lines[END_REF]. A feature model thereby can be used to define a selfadaptive system's adaptation space, where each adaptation action is expressed in terms of a possible runtime configuration, i.e., feature combination [START_REF] Esfahani | A Learning-Based Framework for Engineering Feature-Oriented Self-Adaptive Software Systems[END_REF].

Fig. 1 shows the feature model of a self-adaptive web service as an example. The DataLogging feature is mandatory (which means it is always active), while the ContentDiscovery feature is optional. The DataLogging feature has three alternative sub-features, i.e., at least one data logging sub-feature must be active: Min, Medium or Max. The ContentDiscovery feature has two optional sub-features Search and Recommendation. The constraint Recommendation ⇒ Max ∨ Medium specifies that a sufficient level of data logging is required to collect enough information about the web service's users and transactions to make good recommendations. Let us consider the above web service should adapt to changing number of concurrent users to keep its response time below 500ms. A service developer may express an adaptation rule for the web service such that it turns off some of the features in the presence of more users, thereby reducing the resource needs of the service. The right-hand side of Fig. 1 shows a concrete example for such an adaptation. If the service faces an environment state of more than 1000 concurrent users, the service self-adapts by deactivating the Search feature.

Reinforcement Learning (RL). RL aims to learn suitable actions via an agent's interactions with its environment [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]. At a given time step t, the agent selects an action a (from its adaptation space) to be executed in environment state s (see Fig. 2). As a result, the environment transitions to s at time step t + 1 and the agent receives a reward r for executing the action. The reward r together with the information about the next state s are used to update the knowledge of the agent. The goal of RL is to optimize cumulative As mentioned in Sect. 1, a trade-off between exploitation (using current knowledge) and exploitation (gathering new knowledge) must be made. For a self-adaptive service, "agent" refers to the self-adaptation logic of the service and "action" refers to an adaptation action [START_REF] Palm | Online reinforcement learning for self-adaptive information systems[END_REF].

Environment

Feature-Model-Guided Exploration

As motivated in Sec. 1, our exploration strategies use feature models (FM) to guide the exploration process. We first explain how our FM-guided exploration strategies can be integrated into an existing RL algorithm and then introduce the realization of these strategies.

Integration into Reinforcement Learning

Algorithm 1 shows how our FM-guided strategies can be integrated into RL by using the well-known Q-Learning algorithm as basis. We chose Q-Learning because it is the most widely used algorithm in the related work (see Sec. 6). Adapt service to configuration a; Observe reward r; Observe new state s ; 7: return a; 22: end function Q-Learning employs a value function for representing the learned knowledge. The value function Q(s, a) gives the expected cumulative reward when performing a particular action a in a given state s [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]. Q-Learning offers two hyperparameters: the learning rate α, which defines to what extent newly acquired knowledge overwrites old knowledge, and the discount factor γ, which defines the relevance of future rewards (see knowledge update in line 7).

Algorithm 1 Q-Learning with FM-guided Exploration

Q(s, a) ← Q(s, a) + α[r + γmax a ∈A Q(s , a ) -Q(s,
Our strategies are integrated into RL within the getNextAction function, which selects the next adaptation action while trading off exploration and exploitation. To make this trade-off we use the -greedy strategy as a baseline, as it is a standard action selection strategy in reinforcement learning and the most widely used strategy in the related work (see Sec. 6). With probability 1 -, -greedy exploits existing knowledge by selecting the action a that has the highest Q value and thus highest expected reward (line 13). With probability , -greedy randomly explores a new action. In contrast to this random exploration, we use our FM-guided exploration strategies by calling the getNextConfiguration function (line 18). The different realizations of getNextConfiguration are explained below. To prevent FM-guided exploration from prematurely converging to a local minimum, we follow the literature and use a small amount of randomness [START_REF] Plappert | Parameter space noise for exploration[END_REF], i.e., we perform random exploration with probability δ • .

Leveraging the Feature Model Structure for Exploration

Incremental Exploration Strategy. This strategy takes advantage of the semantics typically encoded in the structure of feature models. Non-leaf features in a feature model are usually abstract features used to better structure variability [START_REF] Thüm | Abstract features in feature modeling[END_REF]. These abstract features often do not have an impact at implementation level, but delegate their realization to their sub-features. Sub-features thus may offer different realizations of their abstract parent feature. The sub-features of a common parent feature, i.e., sibling features, can thus be considered semantically connected. In the example from Sec. 2, the ContentDiscovery feature has two sub-features Search and Recommendation offering different concrete ways how a user may discover online content. The idea behind the Incremental strategy is to exploit the information about these potentially semantically connected sibling features and explore them first before exploring other features. Note that this entails a random selection of the order of sub-features. Table 1 shows an excerpt of a typical exploration sequence of the Incremental strategy with the step-wise exploration of sibling features highlighted in gray.

Logging Min Medium Max Content Disc. Search Recommend. 1 ✓ ✓ ✓ ✓ 2 ✓ ✓ ✓ ✓ 3 ✓ ✓ ✓ ✓ ✓ 4 ✓ ✓ ✓ ✓ ✓ 5 ✓ ✓ ✓ ✓ 6 ✓ ✓ ✓ ✓ 7 ... … … … … ... … Table 1: Example exploration via Incremental strategy (excerpt)
The Incremental strategy is realized by Algorithm 2, which starts by randomly selecting an arbitrary leaf feature f (i.e., a feature with no sub-features) among all leaf features that are part of the current configuration (lines 5-6). Then, the set of configurations C f containing feature f is computed, while the sibling features of feature f are gathered into a dedicated siblings set (line 7). While C f is non-empty, the strategy explores one randomly selected configuration from C f and removes the selected configuration from C f (lines 11-13). If C f is empty, then a new set of configurations containing a sibling feature of f is randomly explored, provided such sibling feature exists (lines [START_REF] Jamshidi | Transfer learning for improving model predictions in highly configurable software[END_REF][START_REF] Kinneer | Managing uncertainty in self-adaptive systems with plan reuse and stochastic search[END_REF][START_REF] Mann | Interplay of virtual machine selection and virtual machine placement[END_REF]. If no configuration containing f or a sibling feature of f is found, then the strategy moves on to the parent feature of f , which is repeated until a configuration is found (line 13) or the root feature is reached (line 22).

Algorithm 2 Incremental Strategy

Feature Degree Exploration Strategy. Even though the Incremental strategy makes use of the structure of the feature model, it still randomly determines the order in which leaf and sibling features are explored. To better guide the decision about which of these features to explore, we make use of the concept of feature degree. We define the feature degree for a given feature f as the number of configurations that contain f . The intuition here is that there may be a higher probability of finding a suitable configuration when considering features with high feature degrees, as they are present in more configurations.

In our example, the feature degree of Search is 5, while of Recommendation it is only 4 (due to the constraint requiring at least the Medium logging level). The Feature Degree strategy thus first explores all configurations involving the Search feature before exploring other configurations. Table 2 shows an excerpt of a typical exploration sequence of the Feature Degree strategy (the exploration of the sibling feature with highest feature degree highlighted in gray).

The Feature Degree strategy is realized by modifying Algorithm 2 to make use of the feature degree as shown in Algorithm 3. On the one hand, the feature if siblings = ∅ then 17:

Logging Min Medium Max Content Disc. Search Recommend. 1 ✓ ✓ ✓ ✓ 2 ✓ ✓ ✓ ✓ ✓ 3 ✓ ✓ ✓ ✓ ✓ 4 ✓ ✓ ✓ ✓ 5 ✓ ✓ ✓ ✓ 6 … … … … … … …
f ← argmax f ∈siblings (featureDeg(f ));

Leveraging Feature Model Differences for Exploration

To capture changes in the service's adaptation space due to evolution, we propose analyzing the differences in feature models before (M) and after (M ) an evolution step. Following the product line literature, we consider two main types of changes of feature models [START_REF] Thüm | Reasoning About Edits to Feature Models[END_REF]:

Added configurations (feature model generalization). New configurations may be added to the adaptation space by (i ) introducing new features to M , or (ii ) removing or relaxing existing constraints (e.g., by changing a sub-feature from mandatory to optional, or by removing "requires" or "excludes" constraints). In the example from Sec. 2, a new sub-feature Optimized might be added to the DataLogging feature, providing a more resource efficient logging implementation. Thereby, new configurations are added to the adaptation space, such as {DataLogging, Optimized, ContentDiscovery, Search}. As another example, the Recommendation implementation may have been improved and it now can work with the Min logging feature. This removes the constraint shown in Fig. 1, and adds new configurations such as {DataLogging, Min, ContentDiscovery, Recommendation}.

Removed configurations (feature model specialization). Symmetrical to above, configurations may be removed from the adaptation space by (i ) removing features from M, or (ii ) by adding or tightening constraints in M .

To determine these changes of feature models, we compute a set-theoretic difference between valid configurations expressed by feature model M and feature model M . Detailed descriptions of feature model differencing as well as efficient tool support can be found in [START_REF] Acher | Feature model differences[END_REF][START_REF] Bürdek | Reasoning about product-line evolution using complex feature model differences[END_REF]. The feature model differences provide us with adaptation actions added to the adaptation space (M \ M), as well as adaptation actions removed from the adaptation space (M \ M ).

Our evolution-aware strategies thus first explore the configurations that were added to the adaptation space, and then explore the remaining configurations if needed. The rationale is that added configurations might offer new opportunities for finding suitable adaptation actions and thus should be explored first. Configurations that were removed are no longer executed and thus the learning knowledge can be pruned accordingly. In the Q-Learning realization (Sec. 3.1), we remove all tuples (s, a) from Q, where a represents a removed configuration.

Such evolution-aware exploration can also be introduced to -greedy. Instead of randomly exploring the whole new adaptation space, exploration is limited to first randomly exploring the set of new configurations.

Experiments

We experimentally assess our FM-guided exploration strategies and compare them with -greedy as the strategy used in the related work (see Sec. 6). Research Questions. We aim to answer the following research questions:

RQ1: How does learning performance using FM-guided exploration compare to using -greedy and how does it impact on QoS?

RQ2: How does learning performance using evolution-aware exploration strategies compare to evolution-unaware exploration and how does it impact on QoS? Experiment Setup. We use a self-adaptive cloud resource management service, CloudRM4 , as subject system [START_REF] Mann | Interplay of virtual machine selection and virtual machine placement[END_REF]. CloudRM controls the allocation of computational tasks to virtual machines (VMs) and the allocation of virtual machines to physical machines in a cloud data center. CloudRM can be adapted by reconfiguring it to use different allocation algorithms, and the algorithms can be adapted by using different sets of parameters. We implemented a separate adaption logic for CloudRM by using the extended Q-Learning algorithm as introduced in Sec. 3.1. In total, CloudRM provides 344 possible adaptation actions. These are structured in a feature model that is four levels deep and includes 65 different features. The feature model together with the code of our algorithms and the data of our experiments are available online 5 .

Our experiments are based on a real-world workload trace with 10,000 tasks, in total spanning over a time frame of 29 days [START_REF] Mann | Resource optimization across the cloud stack[END_REF]. The CloudRM algorithms decide on the placement of new tasks whenever they are entered into the system (as driven by the workload trace). To allow sufficient time in the experiment to observe the impact of an adaptation, CloudRM is allowed to run one hour before the next adaptation action is executed. For RQ2, the same workload was replayed after each evolution step to ensure consistency among the results.

We define the reward function for online RL as r = -(ρ • e + (1 -ρ) • m), with e being the energy consumption and m being the number of VM manipulations (i.e., migrations and launches), each normalized to be on the same scale. We use ρ = 0.8, meaning we give priority to reducing energy consumption, while still maintaining a low number of VM manipulations. If several adaptation actions show similar energy consumption, the one that achieves this with less VM manipulations receives a higher reward.

To determine suitable hyper-parameter values (see Sec. 3.1), we performed hyper-parameter tuning (via grid search). We used the best performing learning rate α = 0.85 and discount factor γ = 0.2 for -greedy and used this also for our FM-guided strategies. To facilitate convergence of the learning process, we used an decay approach. This is a typical approach in RL, meaning that starts at 1 and diminishes with a predefined rate after each time step. We used an decay rate of 0.97 (i.e., < 1% after time step 150), as this led to fastest convergence with highest asymptotic rewards for -greedy. For the FM-guided strategies we used a δ decay rate of 0.9 (i.e., δ < 1% after time step 45). Due to the stochastic nature of the learning strategies (both -greedy and to a lesser degree our strategies involve random decisions), we repeated the experiment 100 times and averaged results. Results for RQ1. Fig. 3 visualizes the learning process for the different exploration strategies by showing how rewards develop over time. As visible, the FMguided exploration strategies (Incremental and Feature Degree) more quickly reach maximum rewards than -greedy (our baseline). Table 3 characterizes the learning process of the different strategies by using the metrics presented in [START_REF] Taylor | Transfer learning for reinforcement learning domains: A survey[END_REF]: Asymptotic performance (maximum reward at end of learning process, here: average rewards of time steps 200-250), total reward (area between reward curve and asymptotic reward), time to threshold (time step when x% of asymptotic reward is reached for first time, here: x = 90), jumpstart (rewards at beginning of learning process, here: at time step 10). In addition, the table shows how the learning performance of the different strategies impact on the QoS characteristics of CloudRM.

Results indicate that our FM-guided exploration strategies lead to a consistent improvement of the learning process. In addition, the Feature Degree strategy performs better than the Incremental strategy, suggesting that considering additional information about the service's features has an effect. Our FM-guided strategies perform better when compared with -greedy wrt. total reward (58.8% on average), time to threshold (48.6% on average), and jumpstart (1.3% on average), while performing comparably wrt. asymptotic performance. Considering the impact on QoS, FM-guided learning consistently leads to less VM manipulations and sightly lower energy consumption. While savings in energy are rather small (less than 1%), FM-guided learning reduces the number of virtual machine manipulations by 7.8% on average. This is caused by the different placement algorithms having a rather small difference wrt. energy optimization, but having a much larger difference wrt. optimizing the number of virtual machine manipulations. Results for RQ2. We compare three evolution-aware strategies (evolutionaware -greedy, evolution-aware Incremental, and evolution-aware Feature Degree) with their respective evolution-unaware counterparts (i.e., the strategies used for RQ1). It should be noted that even though we provide the evolutionunaware strategies with the information about the changed adaptation space (so they can fully explore it), we have not modified them such as to differentiate between old and new adaptation actions. We use a 3-step evolution scenario incrementally adding features and thus adaptation actions to CloudRM. Initially, CloudRM offers the Simple placement feature (creating a dedicated virtual machine for each task) and Multiple placement features (allowing a given number of tasks to be deployed on a virtual machine), offering 26 adaptation actions. In evolution step #1, the Maxsize placement feature is added, which creates virtual machines of a fixed capacity and selects virtual machines using the First-Fit (FF) heuristic, adding 30 adaptation actions. In evolution step #2, the Maxsize placement feature is enhanced by allowing different VM capacities and adding two new virtual machine selection heuristics: Best-Fit (BF) and Worst-Fit (WF), adding 72 adaptation actions. In evolution step#3, the Consolidation Friendly placement feature is added, which selects a physical machine that can accommodate the given task, and then selects a virtual machine hosted on the physical machine, adding 216 adaptation actions.

Like for RQ1, Fig. 4 visualizes the learning process for the different exploration strategies. After each evolution step, we observe the learning process for 250 time steps, before moving to the next step of the evolution scenario. Table 4 shows the results of learning performance and QoS impact across all three evolution steps. We computed the metrics separately for each of the evolution steps and report their averages. The evolution-aware strategies consistently perform better than their evolutionunaware counterparts wrt. total reward (61.3% on average), time to threshold (51.0% on average), jumpstart (5.1% on average), and asymptotic performance (0.4% on average). With respect to the impact on QoS, the evolution-aware strategies reduce the number of virtual machine manipulations by 23.7% on average, while keeping energy consumption around the same as the non-evolutionaware strategies. As can be seen, the evolution-unaware FM-guided strategies (from RQ1) may perform much worse than any of the evolution-aware ones. This is because they again explore old adaptation actions, many of which were not suitable. Finally, it can be observed that evolution-aware -greedy may even outperform the other evolution-aware strategies. This suggests that, during evo-lution, considering the changes of the adaptation space has a much larger effect than considering the structure of the adaptation space.

Discussion

Validity Risks. We used an actual cloud resource management service and a real-world workload trace to measure learning performance and the impact of the different exploration strategies on QoS characteristics. Still results are only for a single system, which limits generalizability.

We purposefully chose -greedy as a baseline, because it was the exploration strategy used in existing online RL approaches for self-adaptive services (see Sec. 6). Alternative exploration strategies were proposed in the field of machine learning. Examples include Boltzmann exploration, where actions with a higher expected reward (e.g., Q value) have a higher chance of being explored, or UCB action selection, where actions are favored that have been less frequently explored [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]. Another alternative is to use policy-based RL, which in contrast to value-based RL such as Q-Learning, directly represents the policy as a neural network, and thus intrinsically exhibits stochastic action selection behavior [START_REF] Palm | Online reinforcement learning for self-adaptive information systems[END_REF]. A comparison among those alternatives is beyond the scope of the current paper, because a fair comparison would require the careful variation and analysis of a range of many additional hyper-parameters.

We focused on evolution steps that increase the size of the adaptation space to assess to what our strategies are able to capture adaptation spaces of increasingly larger size. Our experiments may be complemented by analyzing how the different strategies compare to each other when the size of the adaptation space is reduced. Even though in an adaptation space of reduced size, fewer configurations have to be explored -thus leading in principle to faster learning -there still may be differences in the way these fewer configurations are explored. Limitations and Assumptions. We assume that feature models are complete with respect to the coverage of the adaptation space and that during an evolution step they are always consistent and up to date. A further possible change during service evolution can be the modification of a feature's implementation, which is currently not visible in the feature models. Encoding such kind of modification thus could further improve our online learning strategies.

One aspect that impacts FM-guided exploration is the depth of the feature models. On the one hand, if the feature model has only few levels, the FM-guided exploration strategies behave very similar to random exploration, because such models do not provide enough structure. On the other hand, based on initial experiments with the RL approach in [START_REF] Palm | Online reinforcement learning for self-adaptive information systems[END_REF], providing an RL agent with too structured knowledge might in fact hinder learning an optimal policy. How to define feature models at the right level of detail thus deserves further investigation.

In the realization of the exploration strategies (both -greedy and FM-guided), we assumed we can always switch from a configuration to any other possible configuration. We were not concerned with the technicalities of how to reconfigure the running service (which, for example, is addressed in [START_REF] Chen | Self-adaptation through incremental generative model transformations at runtime[END_REF]). We also did consider constraints concerning the order of adaptations. In practice, only certain paths may be permissible to reach a configuration from the current one. To consider such paths, online RL may be enhanced by building on work such as [START_REF] Ramirez | Automatically generating adaptive logic to balance non-functional tradeoffs during reconfiguration[END_REF].

Related Work

The following authors applied online RL to self-adaptive services and considered different approaches to improve the performance of the learning process. Yet, they did not consider large adaptation spaces nor service evolution. Tesauro et al. Q-Learning for autonomic resource allocation in data centers [START_REF] Tesauro | On the use of hybrid reinforcement learning for autonomic resource allocation[END_REF]. Xu et al. employ Q-Learning (with -greedy) for the automatic configuration of cloud virtual machines and applications [START_REF] Xu | URL: A unified reinforcement learning approach for autonomic cloud management[END_REF]. Both suggest offline learning to increase the jumpstart at runtime. Barrett et al. propose using Q-Learning with -greedy for autonomic cloud resource allocation [START_REF] Barrett | Applying reinforcement learning towards automating resource allocation and application scalability in the cloud[END_REF]. They propose parallel learning to speed up the learning process. Caporuscio et al. propose using two-layer hierarchical RL for multi-agent service assembly [START_REF] Caporuscio | Reinforcement learning techniques for decentralized self-adaptive service assembly[END_REF]. They observe that by sharing monitoring information, learning happens faster than when learning in isolation. Arabnejad et al. apply fuzzy RL with -greedy to learn fuzzy adaptation rules [START_REF] Arabnejad | A comparison of reinforcement learning techniques for fuzzy cloud auto-scaling[END_REF]. They also demonstrate that transfer learning may speed up learning [START_REF] Jamshidi | Transfer learning for improving model predictions in highly configurable software[END_REF]. Wang et al. use Q-Learning (using -greedy) together with function approximation. They use neural networks to generalize over unseen environment states and thereby facilitate learning in the presence of many environment states, i.e., they address large state spaces but not large action spaces [START_REF] Wang | Large-scale and adaptive service composition using deep reinforcement learning[END_REF]. Moustafa and Zhang propose multi-agent Q-Learning with -greedy for adaptive service compositions [START_REF] Moustafa | Learning efficient compositions for QoS-aware service provisioning[END_REF]. To speed up learning, they use collaborative learning, where multiple systems simultaneously explore the set of concrete services to be composed. Zhao et al. propose using RL (using -greedy) combined with case-based reasoning to generate and update adaptation rules for web applications [START_REF] Zhao | A reinforcement learning-based framework for the generation and evolution of adaptation rules[END_REF]. Their approach may take as long to converge as learning from scratch, but it may offer a higher jumpstart.

Bu et al. explicitly consider large adaptation spaces [START_REF] Bu | Coordinated self-configuration of virtual machines and appliances using a model-free learning approach[END_REF]. They employ Q-Learning (using -greedy) for self-configuring cloud virtual machines and applications. They reduce the size of the adaptation space by splitting it into coarse-grained sub-sets for each of which they find a representative adaptation action using the simplex method. Their experiments indicate that their approach indeed can speed up learning. Yet, they do not consider service evolution.

Dutreilh et al. explicitly consider service evolution [START_REF] Dutreilh | Using reinforcement learning for autonomic resource allocation in clouds: towards a fully automated workflow[END_REF]. They employ Q-Learning for autonomic cloud resource management and speed up learning by providing a good initial estimate for the Q-function, as well as by using statistical estimates about the environment behavior. They indicate that system evolution may imply a change of system performance and sketch an idea on how to detect such drifts in system performance. Yet, they do not consider that evolution may also introduce or remove adaptation actions. As explained in Sec.1, such a change in the adaptation space cannot be determined by observing the environment, as the adaptation space is an intrinsic property of the RL agent.

In our previous work, we used online RL for a self-adaptive cloud service [START_REF] Palm | Online reinforcement learning for self-adaptive information systems[END_REF]. We addressed the problem of large environment spaces (similar to Wang et al.) but did neither consider large action spaces nor service evolution. In earlier work, we sketched the principal dependencies between learning and evolution, but did not provide concrete technical solutions [START_REF] Sharifloo | Learning and evolution in dynamic software product lines[END_REF].

A different line of work uses supervised machine learning to reduce the size of the adaptation space. As an example, Van Der Donckt et al. use deep learning to determine a representative and much smaller subset of the adaptation space [START_REF] Van Der Donckt | Applying deep learning to reduce large adaptation spaces of self-adaptive systems with multiple types of goals[END_REF]. Supervised learning requires sufficient labeled training data representative of the service's environment, which may be challenging due to design time uncertainty.

Conclusion

We introduced feature-model-guided exploration strategies for online reinforcement learning that address potentially large adaptation spaces and the change of the adaptation space due to service evolution. Experimental results for an adaptive cloud management service indicate a speed up of the learning process and an improvement of QoS characteristics.

As part of our future work, we will perform additional experiments, considering further types of services and the comparison with alternative exploration strategies. We also aim to integrate our strategies with more advanced reinforcement learning algorithms, such as policy-based reinforcement learning. In addition, we aim to address the current limitations of our strategies and will, for instance, also consider feature modifications during evolution.

Fig. 1 :

 1 Fig. 1: Feature model and adaptation of example web service

Fig. 2 :

 2 Fig. 2: RL concept

1 :: repeat 5 :

 15 function FMQ-Learning(FeatureModel M; Double α, γ, , δ) 2: Initialize Q(s, a) for all s ∈ S (state space) and a ∈ A (adaptation space); 3: Determine current state s; 4Set<Feature> a = getNextAction(M, s); // Action Selection 6:

Fig. 3 :

 3 Fig. 3: Learning curves (RQ1)

  Evo.-aware -greedy blue: Evo.-aware Incremental green: Evo.-aware Feature Degree brown: Evo.-unaware -greedy black: Evo.-unaware Incremental cyan: Evo.-unaware Feature Degree

Fig. 4 :

 4 Fig. 4: Learning curves across evolution steps (RQ2)

Table 2 :

 2 Example exploration via Feature Degree strategy (excerpt) degree is used to determine which leaf feature to start exploring from. Instead of randomly selecting a leaf feature as done in Algorithm 2 (line 6), the Feature Degree strategy selects a leaf feature with the highest feature degree. On the other hand, instead of randomly choosing sibling features as done in Algorithm 2 (line 16), the Feature Degree strategy explores the sibling in descending order of their feature degrees. To realize the featureDeg function, existing feature model analysis tools, such as[START_REF] Thüm | Fea-tureIDE: An extensible framework for feature-oriented software development[END_REF], can be used to efficiently compute the number of possible configurations containing f .

	Algorithm 3 Feature Degree Strategy
	5:	leaves ← getLeaves(currentConfiguration);
	6:	f ← argmax f ∈leaves (featureDeg(f ));
	[...]
	16:

Table 3 :

 3 Comparison of exploration strategies (RQ1)

Table 4 :

 4 Comparison of exploration strategies across evolution steps (RQ2)

https://sourceforge.net/p/vm-alloc/task_vm_pm

https://git.uni-due.de/online-reinforcement-learning/icsoc-2020-artefacts

Acknowledgments. We cordially thank Amir Molzam Sharifloo for constructive discussions during the conception of initial ideas, as well as Alexander Palm for his comments on earlier drafts. Our research received funding from the EU's Horizon 2020 R&I programme under grants 780351 (ENACT) and 871525 (FogProtect). This work was also partially funded by the ANR-19-CE25-0003-01 KOALA project.


MD Pierre-Clément Thiebaud
email: pierre-clement.thiebaud@aphp.fr

Camille Martin

Diane Naouri

Alexandre Le Joncour

Jennifer Truchot

Youri Yordanov

Alcohol consumption among French physicians: a cross-sectional study

Keywords: Alcohol, Alcohol-Related Disorders, AUDIT score, Physicians, Doctors, Occupational Health

Highlights

 Prevalence of hazardous alcohol consumption among French physicians is 12.6%.

 Drugs use and fixed term contract are associated with hazardous alcohol consumption.  Risky alcohol consumption seems similar between physicians and general population.

Introduction

Alcohol consumption has a major health impact worldwide with 58 million hazardous alcohol users and about 195,000 deaths each year in the European Union [START_REF] Anderson | Alcohol in Europe: a public health perspective[END_REF]. The rate of hazardous consumers was 7% of the French general population in 2016 according the World Health Organization (WHO) and 41,000 deaths were attributable to alcohol in 2015 [START_REF] Bonaldi | La mortalité attribuable à l'alcool en France en 2015 [Alcoholattributable mortality in France in 2015[END_REF]WHO, 2018). Physicians play an important role in the prevention, screening and management of patients with alcohol disorders [START_REF] Aubin | Mésusage de l'alcool : dépistage, diagnostic et traitement. Recommandation de bonne pratique[END_REF].

However, recent international studies found a prevalence of hazardous alcohol consumption among physicians varying between 2.5% and 30% [START_REF] Joos | Alcohol use and hazardous drinking among medical specialists[END_REF][START_REF] Kenna | Alcohol use by healthcare professionals[END_REF][START_REF] Oreskovich | Prevalence of alcohol use disorders among American surgeons[END_REF][START_REF] Romero-Rodríguez | Alcohol consumption in Spanish primary health care providers: a national, cross-sectional study[END_REF][START_REF] Rosta | Hazardous alcohol use among hospital doctors in Germany[END_REF][START_REF] Rosta | Drinking Patterns of Doctors: a comparison between Aarhus in Denmark and Mainz in Germany[END_REF][START_REF] Sebo | Use of tobacco and alcohol by Swiss primary care physicians: a cross-sectional survey[END_REF][START_REF] Sørensen | Alcohol and drug use among Danish physicians. A nationwide cross-sectional study in 2014[END_REF][START_REF] Voigt | Consumption of alcohol, cigarettes and illegal substances among physicians and medical students in Brandenburg and Saxony (Germany)[END_REF][START_REF] Wurst | Estimating the prevalence of drinking problems among physicians[END_REF]. The different measurement tools and different cut-offs could explain this wide range.

Data regarding French physicians' alcohol behaviours are scarce. Most studies address this issue within the population of either medical students or residents, who have been shown to drink more alcohol than licensed physicians. In Germany, it has been shown that the average amount of alcohol consumed among the medical students was significantly higher than among physicians.

In Norway, the prevalence of hazardous drinking was reduced by half from medical school to 10 years after graduation (Mahmood et al, 2015).

In France, 21% to 31.9% of medical students and more than a third of residents report hazardous alcohol consumption [START_REF] Chiriaco | Consommation de substances psychoactives chez les internes en médecine : revue de la littérature et enquête auprès des internes parisiens[END_REF][START_REF] Fond | Venus and Mars on the benches of the faculty: Influence of gender on mental health and behavior of medical students. Results from the BOURBON national study[END_REF][START_REF] Gignon | Alcohol, cigarette, and illegal substance consumption among medical students: a cross-sectional survey[END_REF][START_REF] Herault | Consommation de substances psychoactives des internes en médecine [Use of psychoactive drugs by medical trainees[END_REF][START_REF] Houdoux | Santé psychique chez les internes de l'université de médecine de l[END_REF][START_REF] Vaysse | Alcool, tabac, cannabis, anxiété et dépression des étudiants en 2e année de médecine. Repérer pour agir [Alcohol, tobacco, cannabis, anxiety and depression among second-year medical students. Identify in order to act][END_REF]. Residents report drinking more alcohol and more frequent binge drinking episodes than people from the same age group [START_REF] Herault | Consommation de substances psychoactives des internes en médecine [Use of psychoactive drugs by medical trainees[END_REF]. A third of young doctors declare a daily or almost daily alcohol consumption (Conseil National de l'Ordre des Médecins, 2016). A survey among general practitioners showed that whitin the seven days preceding their interview: 51.1% had consumed alcohol for one to two days, 30.3% more than three days and 17% daily [START_REF] Gautier | Baromètre santé, médecins-pharmaciens 2003[END_REF]. Another study among general practitioners identified punctual risk consumption in 21% to 31% and chronic risk in 6% to 11% [START_REF] Desprès | Santé physique et psychique des médecins généralistes[END_REF].

Data regarding hospital doctor's alcohol behaviours are scarce. [START_REF] Beaujouan | Prévalence et facteurs de risque de l'addiction aux substances psychoactives en milieu anesthésique: résultats de l'enquête nationale [Prevalence and risk factors for substance abuse and dependence among anaesthetists: a national survey[END_REF] reports that 6.5% of anaesthesiologists were abusers or dependent on alcohol. To our knowledge, there is no data on the prevalence of hazardous alcohol consumption among French licensed physicians, regardless of their specialty.

We aimed to describe the alcohol consumption of general practitioners and hospital doctors, to assess the prevalence of hazardous alcohol consumption and to identify the associated risk factors.

Methods

Participants

This study was a regional, cross-sectional, survey. The recruitment of the survey population followed two modalities: 1/Parisian general practitioners were drawn from the Yellow Pages professional directory (one in four in alphabetical order) and contacted by phone to obtain their email address. Physicians who did not practice general medicine, were retired or refused to participate were excluded. 2/ Physicians working in the Est Parisien university hospital group (Assistance Publique -Hôpitaux de Paris) received solicitation to participate from the institutional mailing list.

Data collection

The survey was conducted from March 19 to April 22, 2018 using an online questionnaire (Google Form®). All participants received an email explaining the study with a link to the questionnaire. Three reminders were sent out, during a five weeks period. The questionnaire took 5 minutes to complete. Participation was voluntary and the data anonymized. Data were extracted from the online questionnaire and processed using Microsoft Excel®. This study has been registered with the French Data Protection Authority (CNIL).

Questionnaire

The questionnaire consisted of 28 items divided into 3 sections. The first section included items on sociodemographic characteristics (age, sex, marital status, children), occupational information (medical specialty, professional status and hospital contractual status, weekly working time, oncall service) and health behaviour (smoking, illegal substances use). Specialties were categorised into six groups according to their similarities: general practice, intensive care (emergency, anaesthesia, intensive care), medicine (internal medicine, hepato-gastro-enterology or cardiology for example), surgery (orthopaedic or otorhinolaryngology for example), paediatrics and others (radiology and biology for example).

The second section assessed alcohol consumption by the Alcohol Use Disorders Identification Test (AUDIT), a WHO 10-item screening test, translated and validated in many languages including French [START_REF] Babor | AUDIT: The Alcohol Use Disorders Identification Test: guidelines for use in primary care[END_REF][START_REF] Gache | The Alcohol Use Disorders Identification Test (AUDIT) as a screening tool for excessive drinking in primary care: reliability and validity of a French version[END_REF][START_REF] Saunders | Development of the Alcohol Use Disorders Identification Test (AUDIT): WHO Collaborative Project on Early Detection of Persons with Harmful Alcohol Consumption--II[END_REF]. Response options for each item range from 0 to 4 points, resulting in a total score of 0 to 40. A score < 8 indicates non-harmful alcohol consumption. A score ≥ 8 indicates hazardous alcohol consumption and a score ≥ 16 illustrates a problematic alcohol consumption and/or dependence. The first three items of AUDIT constitute the AUDIT-C, developed as a brief and easy-to-use screening measure [START_REF] Bush | The AUDIT alcohol consumption questions (AUDIT-C): an effective brief screening test for problem drinking. Ambulatory Care Quality Improvement Project (ACQUIP). Alcohol Use Disorders Identification Test[END_REF]. Cut-offs used to define risky alcohol consumption vary between 3 points and 6 points. A score ≥ 5 for men and ≥ 4 for women seems to be the most consistent according to the literature and has been used in similar studies [START_REF] Oreskovich | Prevalence of alcohol use disorders among American surgeons[END_REF][START_REF] Reinert | The alcohol use disorders identification test: an update of research findings[END_REF][START_REF] Sebo | Use of tobacco and alcohol by Swiss primary care physicians: a cross-sectional survey[END_REF]. Two other similar studies used a score of 5 points for both sex (Rostal, 2008;[START_REF] Wurst | Estimating the prevalence of drinking problems among physicians[END_REF]. We decided to use both AUDIT and AUDIT-C for two reasons. First, some studies showed very different results depending on the score used. Secondly, to make our results comparable with past or future studies using only AUDIT-C, therefore enhancing the external validity of our results to other researchers.

The third section of our survey included questions on the reasons for alcohol consumption, a selfassessment of physical and mental health, an auto-evaluation of the risk associated with their consumption ("Do you think you have a risky alcohol consumption?") and of the potential impact on patients' care ("Do you think your alcohol consumption has an impact on your patients' care?").

Statistical analysis

Continuous variables are presented as median and interquartile range; categorical variables are presented as count and percent. Statistical analysis was conducted in 2020 using SAS/STAT 2002-2003 (SAS Institute Inc., Cary, NC, USA). Chi-square test (or exact Fischer test, as appropriate) was used to compare characteristics between: 1/ general practitioners and hospital doctors, 2/ hazardous (AUDIT ≥8) and non-harmful (AUDIT <8) alcohol consumers. To analyse factors associated with AUDIT ≥ 8, adjusted odds ratios (aORs) and their 95% confidence intervals (95% CIs) were estimated from a logistic regression model. Variables which were statistically significant on univariate analysis at p <0.20 were included in the model, excepted two items: « consumption considered at risk » because it was too strongly correlated to hazardous alcohol consumption and "potential impact on patients' care" because the number of yes respondents was too low.

Results

One thousand and eight emails were sent. Five hundred fifteen (51.1%) physicians completed the survey and were included in the study: 108 were general practitioners and 407 hospital physicians. Table 1 summarizes participants' sociodemographic and occupational characteristics.

The median age was 40 years 32-55 and 59% were women. Twenty-five percent were smokers, 6% and 4% cannabis and other illegal drugs users respectively. A majority worked more than 50 hours a week and half did on-call shifts. The main reasons for alcohol consumption were celebrations or festive situations (76%), taste (65%), to relax after work (36%) and to fit in socially (17%). Ten percent thought they had risky alcohol use and 1.6% that it could have an impact on their patients' care. They considered their physical and mental health as average or bad in 10% and 8% of cases, respectively. Compared to hospital doctors, general practitioners were more often men (52.8% vs 37.6%, p=0.004), reported having shorter weekly working hours (less than 50h for 58.3% vs 33%, p<0.001), participated less frequently to on-call service (17.6% vs 54.3%, p<0.001) and were older, even though the difference was not statistically significant (56.6% vs 47.2%, p=0.08).

The median AUDIT score was 3 2-5, the extreme ranged from 0 to 26 points. The prevalence of hazardous alcohol consumption (score ≥ 8) among physicians was 12.6 % and problematic consumption or dependence (score ≥ 16) affected 1.2% of them. Five percent were abstinent. The median AUDIT-C score was 3 2-4. When using the AUDIT-C instead of the AUDIT, the rate of hazardous alcohol rose to 35.7% (score ≥ 4 for women and ≥ 5 for men) or 23.3% (score ≥ 5 for both sex). Table 2 shows the results for the 10 questions. Daily alcohol consumption was found for 15% of physicians, more frequent in those over 40 than in those under 40 years old (19% vs 12%, p=0.03). The majority (75%) consumed one or two drinks on a typical day. During the past year, 45% reported at least one event involving the consumption of six or more drinks; this was more frequent in those under 40 than those over 40 years old (63% vs 27%, p<0.001). At least once in the past year, 13% declared they were not able to stop drinking once they started, 10% failed to do what was normally expected of them because drinking, 22% had had remorse or feeling of guilt after drinking and 9% were unable to remember what happened because they drank alcohol. There was no difference between general practitioners and hospital doctors concerning the AUDIT, AUDIT-C results and each individual question.

The univariate analysis identified an association between hazardous alcohol consumption (AUDIT ≥8) and age, marital status, childless, fixed-term hospital contract, smoking or illegal substance consumption, average or bad physical or mental health self-assessment, selfrecognition of risky alcohol consumption and thinking it may influence patients' care (Table 3).

There was no significant difference according to gender, medical specialty, working time or participation to on-call shifts. Regarding specialties, classification from the largest to the lowest rate of hazardous alcohol consumption was surgery (16.2%), intensive care (16.0%), general practice (13.9%), medicine (13.2%), paediatrics (10.4%) and others (6.3%). Among the 65 physicians with hazardous alcohol consumption, 27 (41.5%) did not considered it to be "at risk" and four (6.2%) mentioned a potentially negative impact on their patients' care. Reasons for alcohol consumption among hazardous consumers compared to safe drinkers were more frequently to relax after work (70% vs 33%, p<0.001), recreational (42% vs 9%, p<0.001), to fit in socially (28% vs 16%, p=0.02) and to fight again anxiety, stress, depression (17% vs 3%, p<0.001). In the multivariate analysis, the factors independently and significantly associated with hazardous alcohol consumption were cannabis and other illegal drugs use ) and fixed term contract for hospital doctors ).

Discussion

To our knowledge, this is the first study to explore the issue of alcohol consumption among multi specialty French physicians. We found with the AUDIT score that 12.6% of them showed hazardous alcohol consumption, among which 41.5% did not consider it as such. However, they more often considered their physical and mental health as average or bad. Illegal substance users and fix-termed contract hospital doctors were more likely to have risky alcohol consumption. We did not show any difference between the different medical specialties or other occupational factors.

Our results are consistent with similar studies among physicians in Western Europe: according to the AUDIT, hazardous alcohol consumption was found to vary from 13.4% to 18.9% [START_REF] Joos | Alcohol use and hazardous drinking among medical specialists[END_REF][START_REF] Sørensen | Alcohol and drug use among Danish physicians. A nationwide cross-sectional study in 2014[END_REF][START_REF] Wurst | Estimating the prevalence of drinking problems among physicians[END_REF]. Using the AUDIT-C, it ranged from 19.8% to 27.4% (cut-off ≥ 5 for both sex) and from 30% to 32% (cut-off ≥ 5 for men and ≥ 4 for women) [START_REF] Romero-Rodríguez | Alcohol consumption in Spanish primary health care providers: a national, cross-sectional study[END_REF][START_REF] Rosta | Hazardous alcohol use among hospital doctors in Germany[END_REF][START_REF] Sebo | Use of tobacco and alcohol by Swiss primary care physicians: a cross-sectional survey[END_REF][START_REF] Wurst | Estimating the prevalence of drinking problems among physicians[END_REF]. In our study, we found a twofold prevalence of hazardous alcohol consumption with the AUDIT-C when compared to the AUDIT. This result is consistent with a previous study using the same methodology [START_REF] Wurst | Estimating the prevalence of drinking problems among physicians[END_REF]. Indeed, the prevalence of risky alcohol consumption seems to be overestimated by AUDIT-C compared to the original AUDIT, the latter is the only assessment tool validated by the WHO. Regarding the third question of AUDIT examining binge drinking (consumption of 6 drinks or more in one occasion), it was reported at least once a month for 13.5% to 19.5% of European physicians [START_REF] Joos | Alcohol use and hazardous drinking among medical specialists[END_REF][START_REF] Romero-Rodríguez | Alcohol consumption in Spanish primary health care providers: a national, cross-sectional study[END_REF][START_REF] Rosta | Hazardous alcohol use among hospital doctors in Germany[END_REF]. Factors associated with hazardous alcohol consumption in these studies were male, single status, childless and older age. Risky alcohol consumption appeared more frequent among anaesthesiologists and surgeons and less frequent among paediatricians, as our data suggest.

Lastly, the reasons for alcohol consumption reported in only one Danish study were to enjoy the taste (74%), to get in a good mood (55%) and to relax after work (55%) [START_REF] Sørensen | Alcohol and drug use among Danish physicians. A nationwide cross-sectional study in 2014[END_REF].

Comparison between physicians and the general population is difficult because the data are not measured with the same method or at the same time. The most recent data in the general population are issued from the Public Health France survey on alcohol consumption in adults in 2017 [START_REF] Richard | La consommation d'alcool chez les adultes en France en 2017 [Alcohol consumption in adults in France in 2017[END_REF]. The prevalence of hazardous alcohol consumption was not reported but some trends can been drawn in drinking habits. Physicians seemed to drink more frequently compared to the general population with fewer abstainers (5.1% vs 13.5%) and occasional users (15.5% vs 27.2% "less than once a month"), the rate of regular users was similar (15.5% vs 15.1% "more than 4 days a week"). Regularly binge drinking was similar (16.5% vs 16.2% "monthly" and 4.3% vs 4.6% "weekly") but physicians were more likely to have occasional binge drinking (44.5% vs 35.2% "at least once a year"). In both studies, a risky consumption was mainly found in men, the frequency of alcohol consumption increased with age while the quantity consumed on one occasion was more common among young people. The largest cohort from the French general population is the "CONSTANCES" cohort examining 33.992 currently working participants between 2012 and 2016. Among them, 14% showed hazardous alcohol consumption according to the AUDIT score ≥ 8 and 49.1% had binge drinking episodes at least once a year [START_REF] Airagnes | Job exposure to the public in relation with alcohol, tobacco and cannabis use: Findings from the CONSTANCES cohort study[END_REF]. In this study, men with daily occupational exposure to the public had a higher risk of chronic, hazardous alcohol consumption, and binge drinking episodes. Stressful exposure was associated with an increased risk of binge drinking episodes in men and of chronic and hazardous alcohol consumption in women. Physicians have a regular daily stressful occupational exposure to the public (patients) and therefore belong to these risk groups. This was also shown for the specific population of anaesthesiologists [START_REF] Beaujouan | Prévalence et facteurs de risque de l'addiction aux substances psychoactives en milieu anesthésique: résultats de l'enquête nationale [Prevalence and risk factors for substance abuse and dependence among anaesthetists: a national survey[END_REF]. Our results highlight that physicians practicing in front line specialties (surgery, emergency medicine, general practice) seem to be at a greater risk for alcohol misuse. This has also been shown in studies exploring physicians 'wellbeing and burnout [START_REF] Shanafelt | Burnout and satisfaction with work-life balance among US physicians relative to the general US population[END_REF]. Among a large sample of surgeons, a study identified that burnout and depression were predisposing factors for high alcohol consumption [START_REF] Oreskovich | Prevalence of alcohol use disorders among American surgeons[END_REF]. These results suggest the potential benefit of multimodal prevention programs on specific high-risk population of physicians. Our study was not designed to compare physicians to the general population but the prevalence of hazardous consumption in our survey seemed to be similar to the general population.

There is a growing body of evidence suggesting that health care professionals' individual health behaviours may influence the general population's behaviours (Frank et al, 2015). Frank et al (2000) reported that physicians with a healthy lifestyle would be more likely to counsel their patients: those who had a healthy diet were better at screening their patients for cholesterol, nonsmokers more often advised smoking cessation and those who drank less were more likely to address the issue of alcohol use. Therefore, physicians' alcohol consumption could have an impact on the care of their patients. One study showed that surgeons reporting a major medical error in the previous 3 months were more likely to have alcohol abuse or dependence [START_REF] Oreskovich | Prevalence of alcohol use disorders among American surgeons[END_REF]. In our survey, four physicians with hazardous alcohol consumption reported that it could have an impact on the management of their patients. Some authors suggest that medical schools and hospitals lack the proper health educational programs, which could prevent hazardous alcohol consumptions among physicians [START_REF] Aach | Alcohol and other substance abuse and impairment among physicians in residency training[END_REF]. Training physicians on the dangers of alcohol use could prevent risky consumption and improve screening and treatment for their patients. Indeed, a study reported that the cut-off to define hazardous alcohol consumption for patients is overestimated by doctors who consume more alcohol [START_REF] Geirsson | Attitudes of Swedish general practitioners and nurses to working with lifestyle change, with special reference to alcohol consumption[END_REF]. Additional studies are needed to investigate the impact of specific interventions on health behaviour of health care professionals and explore the consequences on their patient's management.

For instance, an individual brief intervention protocol implementation for physicians could generate positive benefits and also increase awareness towards patients' screening. Brief interventions (BI) consist of the assessment of alcohol consumption, providing information on the related risks, and suggesting moderation goals. They have been shown to be effective to reduce alcohol dependence in numerous studies and meta-analyzes [START_REF] Kaner | Effectiveness of brief alcohol interventions in primary care populations[END_REF][START_REF] Moyer | Brief interventions for alcohol problems: a meta-analytic review of controlled investigations in treatment-seeking and nontreatment-seeking populations[END_REF]. For hazardous alcohol consumptions, BI have been associated with a significant reduction in average alcohol consumption, a decrease in the proportion and frequency of binge drinking episodes, and a decrease of somatics issues related to alcohol [START_REF] Wutzke | The long-term effectiveness of brief interventions for unsafe alcohol consumption: a 10-year follow-up[END_REF]. These improvements have been observed in different clinical environments, however this has never been assessed for the specific population of physicians. BI are based on the concept of motivational interviewing, involving empathy and listening for the caregiver. The caregiver's degree of empathy and his practice of motivational interviewing are efficacy criteria highlighted by randomized studies [START_REF] Hettema | Motivational interviewing[END_REF][START_REF] Miller | Focused versus broad-spectrum behavior therapy for problem drinkers[END_REF]. For all these reasons, offering the possibility of a specific, anonymous, BI program for physicians with hazardous alcohol consumption could also improve the quality of care and awareness towards these specific issues.

This hypothesis could inspire a future interventional research project.

Limitations

This study has several limitations. First, the response rate of 51.1% could lead to selection bias, even if it compares favourably to similar studies involving healthcare professionals, which ranged from 6.1 to 68.7% [START_REF] Beaujouan | Prévalence et facteurs de risque de l'addiction aux substances psychoactives en milieu anesthésique: résultats de l'enquête nationale [Prevalence and risk factors for substance abuse and dependence among anaesthetists: a national survey[END_REF][START_REF] Joos | Alcohol use and hazardous drinking among medical specialists[END_REF][START_REF] Kenna | Alcohol use by healthcare professionals[END_REF][START_REF] Oreskovich | Prevalence of alcohol use disorders among American surgeons[END_REF][START_REF] Romero-Rodríguez | Alcohol consumption in Spanish primary health care providers: a national, cross-sectional study[END_REF][START_REF] Rosta | Hazardous alcohol use among hospital doctors in Germany[END_REF][START_REF] Rosta | Drinking Patterns of Doctors: a comparison between Aarhus in Denmark and Mainz in Germany[END_REF][START_REF] Sebo | Use of tobacco and alcohol by Swiss primary care physicians: a cross-sectional survey[END_REF][START_REF] Sørensen | Alcohol and drug use among Danish physicians. A nationwide cross-sectional study in 2014[END_REF][START_REF] Voigt | Consumption of alcohol, cigarettes and illegal substances among physicians and medical students in Brandenburg and Saxony (Germany)[END_REF][START_REF] Wurst | Estimating the prevalence of drinking problems among physicians[END_REF]. Some physicians may not have responded due to the sensitive nature of the topic, especially those with hazardous alcohol consumption. An international review estimated underreporting between 40 and 60% in studies on alcohol consumption [START_REF] Midanik | The validity of self-reported alcohol consumption and alcohol problems: a literature review[END_REF]. In France, survey data compared to alcohol sales data suggests an underestimation of alcohol consumption of 63% [START_REF] Arwidson | Consommation annuelle d'alcool déclarée[END_REF]. This indicates that our study may underestimate the prevalence of hazardous alcohol consumption among physicians.

Second, we cannot determine cause and effect between risky alcohol use and associated factors in this cross-sectional study. Third, there are potential confounders due to the non-exhaustive nature of variables influencing alcohol consumption, including psychological distress and medication use. Fourth, a lack of statistical power may prevent the demonstration of an association between hazardous alcohol consumption and other factors, such as age, sex or occupational factors including medical specialties. A large-scale national study is necessary to better identify these associated factors. Lastly, the sample may not be representative of French physicians as alcohol consumption differs by region [START_REF] Desprès | Santé physique et psychique des médecins généralistes[END_REF]. The region studied (Ile-de-France) is the one where the daily consumption of alcohol in the general population is the lowest (7.1% for a national average of 10%) (Santé Publique France, 2020).

Conclusions

The prevalence of hazardous alcohol consumption among French physicians was 12.6%, consistent with results from studies on the general French population and European physicians.

There was no difference regarding alcohol behaviour between general practitioners and hospital doctors. Illegal substance users and fix-termed contract hospital doctors were more likely to have risky alcohol consumption. Hazardous consumers more often assessed their physical and mental health as average or bad, but 41.5% of them did not consider their consumption as risky. A largescale national study would confirm the factors associated with hazardous alcohol consumption among physicians and could explore the efficacy of preventive measures to insure the safety and wellbeing of physicians and their patients. No = 0 point 499 (96.9) Yes, but not in the past year = 2 points 15 (2.9) Yes, during the past year = 4 points 1 (0.2) Q10 -Has a relative or friend, doctor or other health worker been concerned about your drinking or suggested you cut down? No = 0 point 480 (93.2) Yes, but not in the past year = 2 points 15 (2.9) Yes, during the past year = 4 points 20 (3.9)

  often do you have six or more drinks on one occasion? Daily or almost daily = 4 points 0 (0) Q8 -During the past year, how often have you been unable to remember what happened the night before because you had been drinking? you or someone else been injured as a result of your drinking?

Table 1 -Socio-demographic and occupational characteristics of respondents

 1

		Hospital	General
	All	physicians	practitioners	p-value
	n=515	n=407	n=108

Table 2 -AUDIT score, results for each question

 2

Table 3 -Factors associated with hazardous alcohol consumption Univariate analysis

 3

	Multivariate analysis

AUTHOR DISCLOSURES

Role of Funding Source: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Contributors: PCT: conceptualization, methodology, investigation, supervision, writingoriginal draft, review and editing; CM: methodology, investigation, formal analysis, writingreview and editing; DN: formal analysis, writing -review and editing; ALJ: writing -original draft, review and editing; JT: conceptualization, methodology, writing -original draft, review and editing; YY: methodology, writing -review and editing. All authors have approved the final article.

Conflicts of interest: none.