Anne Kieffer
email: anne.kieffer@insa-rennes.fr

Patrick Maillé
email: patrick.maille@imt.fr

Bruno Tuffin
email: bruno.tuffin@inria.fr

The network neutrality debate has been raging worldwide for around fifteen years now. Our goal in this paper is to model and discuss a quite recent option which could be seen as a trade-off between neutrality and differentiation operated by Internet service providers (ISPs), and satisfy both ends of the world: differentiation potentially chosen by end users. By using a model from the literature, we compare the outcomes of three scenarios: neutrality, non-neutrality with differentiation decided by ISPs, and non-neutrality decided by users. We illustrate that, depending on network parameters, letting end users decide may end up as a fair and viable solution, and that non-neutrality imposed by ISPs is not necessarily bad for all actors.

Introduction

The success of the Internet is based on the notion of packets treated equally and routed through the network in a "best-effort" way. The network neutrality debate [START_REF] Wu | Network neutrality, broadband discrimination[END_REF] started with Internet service providers (ISPs) claiming that they are supporting the infrastructure development and maintenance while content providers (CPs) send an increasing traffic and get an increasing share of the revenue generated by the network activity. ISPs were asking for the possibility to request payments for services and to differentiate traffic. This raised complaints from user associations and CPs arguing among other things that it would prevent innovation [START_REF]Net Neutrality or Net Neutering: Should Broadband Internet Services be Regulated[END_REF][START_REF] Maillé | Internet governance and economics of network neutrality[END_REF][START_REF] Maillé | Telecommunication Network Economics: From Theory to Applications[END_REF][START_REF] Maillé | Toward a net neutrality debate that conforms to the 2010s[END_REF][START_REF] Odlyzko | Network neutrality, search neutrality, and the never-ending conflict between efficiency and fairness in markets[END_REF]. The debate has 1 been raging, with neutrality rules defined and imposed by regulators worldwide (see as examples [START_REF]Protecting and promoting the open internet[END_REF] for the FCC in the USA and [START_REF] Berec | Berec guidelines on the implementation by national regulators of european net neutrality rules[END_REF] for BEREC in Europe). The debate is even exacerbated with the recent decision to repeal neutrality in the USA [START_REF]Restoring internet freedom[END_REF][START_REF] Maillé | Neutral and Non-Neutral Countries in a Global Internet: What Does it Imply?[END_REF].

There have been many models to analyze the pros and cons of neutrality and/or service differentiation; see among others [START_REF] Altman | Network non-neutrality debate: An economic analysis[END_REF][START_REF] Coucheney | Network neutrality debate and ISP inter-relations: Traffic exchange, revenue sharing, and disconnection threat[END_REF][START_REF] Maillé | Telecommunication Network Economics: From Theory to Applications[END_REF][START_REF] Njoroge | Investment in two sided markets and the net neutrality debate[END_REF] and the references therein. The goal is in general to discuss (using tools from game theory) whether service differentiation can be beneficial or hurtful to the society, and whether introducing regulation would improve the outcomes.

It would indeed be interesting to find a trade-off satisfying all actors, CPs, users and ISPs. Quoting [START_REF] Schulzrinne | Network neutrality is about money, not packets[END_REF]:

Rather than focusing on network behavior only, it may be more helpful to consider end-user choice as the principle for deciding whether a particular traffic management or other policy is reasonable if it cannot be readily justified as protecting the network against attack or abuse.

In other words, i) service differentiation is often seen as a restraint to innovation, but users should be able to choose to favor new services if they find them relevant, and innovative CPs could also gain from this; ii) with user-driven differentiation, the ISP stays "neutral" in the sense that it does not choose to differentiate; iii) such an approach would allow ISPs to operate some differentiation, release the necessity to invest on capacity and get a reasonable share of revenue.

The purpose of this paper is to apply a mathematical model to investigate whether the option of allowing differentiation, operated by users, is worthwhile with respect to neutrality or differentiation operated by the ISP. This is studied in the context of actors (here users, an ISP and CPs) making decisions to optimize their own interest. (Therefore the mathematical framework is that of non-cooperative game theory [START_REF] Maillé | Telecommunication Network Economics: From Theory to Applications[END_REF][START_REF] Osborne | A Course in Game theory[END_REF].) We propose to use the model and results in [START_REF] Mandjes | Pricing Strategies under Heterogeneous Service Requirements[END_REF] where users want to access two types of services, video (it was voice in [START_REF] Mandjes | Pricing Strategies under Heterogeneous Service Requirements[END_REF] but we "update" it) and data, through a network modeled by an M/M/1 queue. We assume an infinity of potential infinitesimal users sensitive to price and delay and asking for service as soon as they get a non-negative utility: the user equilibrium notion is then called Wardrop equilibrium [START_REF] Wardrop | Some theoretical aspects of road traffic research[END_REF]. The three scenarios we compare are: a unique class of service (neutral scenario) for which packets are served in a first-come-first-served manner; a situation with two priority classes but for which the ISP decides which class should get priority, and the same two-priority classes but for which each user decides its service class. On top of that, the ISP optimizes its price(s) for service, anticipating the users' response, for the three scenarios. Our comparison of the outcomes shows that letting users decide the differentiation is a viable and balanced solution, since it allows both CPs to be served, and may also satisfy all actors with respect to a neutral situation. Similarly, letting the ISP differentiate may actually be beneficial for users, a conclusion consistent with other analyses in standard revenue management theory.

The remaining of the paper is organized as follows. Section 2 presents the basic model we are taking from [START_REF] Mandjes | Pricing Strategies under Heterogeneous Service Requirements[END_REF], the relevant results therein, and the scenarios with the structure of the game that is played. Section 3 compares the outputs of the three scenarios to discuss if results are surprising, and conclusions and recommendations are made in Section 4.

Model

Basic model

We recall in this subsection the model and results first introduced in [START_REF] Mandjes | Pricing Strategies under Heterogeneous Service Requirements[END_REF].

The model considers two classes of (infinitesimal) customers, say video (indexed by v) and data (indexed by d) users, each one generating packets with an average rate λ d (resp. λ v ) per unit of mass of customers. If p is the per-packet price charged to users, the respective (per-packet) utilities of type-d and type-v users are

U d (E[D]) = u d (E[D]) -p with u d (y) = y -α d U v (E[D]) = u v (E[D]) -p with u v (y) = y -αv ,
where u i (E[D]) for i ∈ {d, v} is the willingness to pay for a packet transmission in the network if the expected delay is E[D]. It is assumed that 0 < α d < α v , meaning that video users are more sensitive to congestion than data users. The curves intersect to highlight that voice users give more value to small delays.

Let N d (resp. N v ) be the number (or more precisely, mass since users are assumed infinitesimally small) of data (resp. video) users. It is assumed that there is a potential unlimited number of video and data customers coming in as soon as their utility is positive, or leaving if negative. The equilibrium notion in terms of the actual mass of customers is the Wardrop equilibrium [START_REF] Wardrop | Some theoretical aspects of road traffic research[END_REF], where users of a given type either do not use at all a class of service because of a non-positive utility, or use a class of service but have a zero utility (otherwise new users would enter or leave).

The network is represented by its bottleneck modeled as an M/M/1 queue. In the neutral case without differentiation,

E[D] = 1 µ -(λ d N d + λ v N v )
where µ is the network service rate. In the case of two priority classes H for high and L for low, with packet rates (to be more clearly defined later) λ H and λ L ,

E[D H ] = 1 µ -λ H E[D L ] = 1 (µ -λ H )(1 -(λ H + λ L )/µ)
.

See [START_REF] Kleinrock | Queueing Systems[END_REF] if a proof of those formulas is needed.

Scenarios and goal

We will consider three different scenarios:

• The neutral scenario for which the ISP proposes a single class of service;

• The non-neutral scenario for which the ISP decides which type of users will use which priority class;

• The non-neutral scenario for which each infinitesimal user will selfishly decide its priority class.

In each case, the decisions are taken in the following order:

1. The ISP first determines price(s) maximizing its revenue.

2. Customers use the service or not: given the price(s) and the service policy, the mass of users of each type asking for service satisfies the Wardrop principle.

Even if the ISP plays first, it plays anticipatively, taking into account what the reaction of users will be.

We will compare the output of the game for the three types of actors: users, CPs and the ISP, in order to determine if a strategy should be favored by regulators, in particular the recent proposition to let users decide their service class. This type of comparison was not the purpose in [START_REF] Mandjes | Pricing Strategies under Heterogeneous Service Requirements[END_REF]. More precisely, we are going to compare for the three scenarios: i) the ISP revenue to see if and why the ISP is pushing for a solution; ii) the CPs individual revenue a j λ j N j , j ∈ {v, d} (a j being a perunit-of-volume advertisement revenue), and cumulated revenue a d λ d N d + a v λ v N v to evaluate if differentiation necessarily means a loss for CPs and the type of preferred differentiation; iii) user satisfaction/demand N j and total demand N v + N d .

In the next subsections, we describe the user equilibria for our three scenarios that were computed in [START_REF] Mandjes | Pricing Strategies under Heterogeneous Service Requirements[END_REF] but called differently and for a different analysis.

User equilibrium from the literature

No differentiation/neutrality

The Wardrop equilibrium in the case with no differentiation is such that: 3. If p = 1, there is an infinite number of equilibria, type-d and v users having the same sensitivity to price, but we will consider in that case that only type-v users will be present.

The optimal revenue in the neutral case is therefore

Π (n) = max p d µ α d +1 , p v µ αv+1 with p d = µα d α d +1 α d and p v = µαv αv+1 αv .
The type of users present at the revenue-maximizing price therefore depends on the value of the service rate µ. More specifically, there is a threshold

µ * = α d α d + 1 α d α v + 1 α v αv α v + 1 α d + 1 1 αv -α d , (1)
such that if µ < µ * , we are in the case with type-d users only, whereas if µ ≥ µ * we are in the scenario with only type-v users.

Dedicated classes/ISP deciding differentiation

The case when the ISP decides which type of users gets which priority class is called the dedicated class scenario in [START_REF] Mandjes | Pricing Strategies under Heterogeneous Service Requirements[END_REF]. Define p H and p L as the respective prices for the high and low priority service classes. The output of the case with dedicated classes (with type-v users being assigned the high-priority class) is:

1. The mass of class-H is N H = N v (p H , p L ) = µ-αv √ p H λv
if p H < µ αv and 0 otherwise.

The mass of class-L is

N L = N d (p H , p L ) =    λ -1 d ( αv √ p H -µ α d √ p L / αv √ p H ) if p L ≤ p 2α d /αv H /µ α d and p H ≤ µ αv λ -1 d (µ -α d √ p L ) if p L < µ α d and p H > µ αv 0 otherwise.
Price optimization is tricky, with non-informative formulas provided in [START_REF] Mandjes | Pricing Strategies under Heterogeneous Service Requirements[END_REF], hence we will perform a numerical optimization.

Open classes/user-defined differentiation

= N v = µ-αv √ p H λv + and N L = N d (p H , p L ) =    λ -1 d ( αv √ p H -µ α d √ p L / αv √ p H ) if p L ≤ p 2α d /αv H /µ α d and p H ≤ µ αv λ -1 d (µ -α d √ p L ) if p L < µ α d and p H > µ αv 0 otherwise.
Similarly to the case with dedicated classes, choosing the prices optimizing the ISP revenue will be performed numerically.

Numerical results

We now compute and compare the revenues and demand in the three scenarios. To start, Figure 1

a v = 1, α d = 1, α v = 1.5, λ d = 1, λ v = 5
there are only type-v users when p > 1. We can check that there is an optimal price to be charged if the ISP wishes to maximize its revenue. As could be expected, that price does not correspond to an optimum for users or CPs.

The outputs at prices optimizing the ISP revenue for the three scenarios are displayed in Figure 2 (optimal price), Figure 3 It can be readily checked in Figure 2 that prices increase with the service rate µ;

in other words, a scarce resource does not correspond to a price increase. When µ is small, the price p L when the ISP decides the classes is equal to the optimal neutral price, but that class is not served. If users decide differentiation, p L is smaller. The price p H of the high priority class is larger if users decide than if the ISP does when µ is large, and, it may be counter-intuitive, for the largest values of µ, the neutral price is larger than the high-class price when ISP decides differentiation; it is due to type-d users in service in the former case and type-v in the latter.

a v = 1, α d = 1, α v = 1.5, λ d = 1, λ v = 5
On Figure 3, the ISP revenue is as expected larger if it decides differentiation, but letting users decide is a better solution for the ISP than a fully neutral scenario. The larger the network capacity, the larger the revenue differences. Now let us look at CP revenues in Figure 4 and demands N d , N v in Figure 5: note that they do not always increase with µ. Remark also that letting users decide differentiation is the only case when both types of service will be active in the network for a range of service rates; in this sense it is an interesting and fair scenario.

Looking at total demand in Figure 6, letting users decide is the best option when µ is small, and is always better than the neutral case. As µ increases, differentiation decided by the ISP is better, but it hides that only one type of service is used, which is again not the case when users decide for a range of values.

Conclusions

As the neutrality debate is still raging, we have discussed in this paper the option of letting users choose a service class for each application. This way, innovation would not be slowed down with users asking for good quality for new and good quality applications; it would also reduce the load on ISPs, with no requirement to provide a good quality for all services. The network would stay neutral because not interfering (except for the price decision) on service class choices.

We have used a model from the literature to discuss and compare the output with a neutral situation and a fully non-neutral one where the ISP decides how to differentiate traffic. According to our results, letting users decide seems a nice tradeoff in terms of demand and revenues for CPs and ISPs.

As future works, we would like to analyze other types of models and check whether they corroborate the results we have obtained here. Another issue is how to implement this promising option in practice, from a technical point of view.

1 . 2 .

 12 If p > 1, only type-v users join, with a mass N v (p) = µ-αv √ p λv . If p < 1, only type-d users join, N d (p) = µ-α d √ p λ d and the ISP revenue is Π(p) = λ d N d (p).

Figure 1 :

 1 Figure 1: Revenues and demand in the neutral case when µ = 4, a d = 0.7,a v = 1, α d = 1, α v = 1.5, λ d = 1, λ v = 5

Figure 2 :

 2 Figure 2: Optimal prices in terms of µ when α d = 1, α v = 1.5, λ d = 1, λ v = 5

Figure 3 :

 3 Figure 3: Optimal ISP revenues in terms of µ when a d = 0.7,a v = 1, α d = 1, α v = 1.5, λ d = 1, λ v = 5

Figure 4 :Figure 5 :Figure 6 :

 456 Figure 4: Optimal CP revenues in terms of µ when a d = 0.7,a v = 1, α d = 1, α v = 1.5, λ d = 1, λ v = 5


C Thellier

Damien Subtil

D Pelletier De Chambure

Bruno Grandbastien

C Catteau

A Beaugendre

D Poitrenaud

Anne Prevotat

Pierre Richart

K Faure

Rémi Le Guern
email: remi.leguern@chru-lille.fr

An educational

Keywords: educational intervention, penicillin allergy, clindamycin, cefazolin

published or not. The documents may come

INTRODUCTION

Worldwide, infection is one of the leading causes of both maternal and neonatal mortality [START_REF] Sachs | Hemorrhage, infection, toxemia, and cardiac disease, 1954-85: causes for their declining role in maternal mortality[END_REF] . Antibiotics have made maternal sepsis related mortality a rare event, and the frequency of maternal-fetal infections has fallen substantially [START_REF]WHO Recommendations for Prevention and Treatment of Maternal Peripartum Infections[END_REF] . Overall, 15% of pregnant women receive antibiotics during pregnancy [START_REF] Jacob | Prevalence and predictors of prescription of antibiotics in pregnant women treated by gynecologists in Germany[END_REF] and 55% at delivery [START_REF] Desai | Morbidity in Pregnant Women Associated with Unverified Penicillin Allergies, Antibiotic Use, and Group B Streptococcus Infections[END_REF] . β-lactams are among the most useful antibiotics, because of their efficacy, tolerability, and safety for use during pregnancy. Streptococcus agalactiae is responsible for more than half of maternal-fetal infections [START_REF] Baveux | Prevention of early-onset group B streptococcus neonatal diseases. The 2005 experience of the Lille University Health Center[END_REF] , and almost all isolates are highly susceptible to penicillins [START_REF] Hays | Changing Epidemiology of Group B Streptococcus Susceptibility to Fluoroquinolones and Aminoglycosides in France[END_REF] . Around 5 to 10% of adults report a history of penicillin allergy [START_REF] Immunology | Drug allergy: an updated practice parameter[END_REF] ; the frequency among pregnant women is similar [START_REF] Van Dyke | Evaluation of universal antenatal screening for group B streptococcus[END_REF][START_REF] Wendel | Penicillin allergy and desensitization in serious infections during pregnancy[END_REF] . Penicillin-allergic patients often receive macrolides, which can result in lower efficacy, especially as resistance is present in 20 to 30% of cases for S. agalactiae [START_REF] Six | Molecular Characterization of Nonhemolytic and Nonpigmented Group B Streptococci Responsible for Human Invasive Infections[END_REF] . However, between 80 and 90% of women who self-report a history of penicillin allergy able are actually able to tolerate penicillin and could be treated with a β-lactam [START_REF] Blumenthal | Impact of a clinical guideline for prescribing antibiotics to inpatients reporting penicillin or cephalosporin allergy[END_REF] . Decision algorithms have been developed to guide selection of appropriate antibiotic in patients with suspected penicillin allergy [START_REF] Vorobeichik | Misconceptions Surrounding Penicillin Allergy: Implications for Anesthesiologists[END_REF][START_REF] Hermanides | Presumed beta-Lactam Allergy and Cross-reactivity in the Operating Theater: A Practical Approach[END_REF] . In pregnant women, specific guidelines were issued for the prevention of S. agalactiae perinatal infection [START_REF] Verani | Prevention of perinatal group B streptococcal disease--revised guidelines from CDC, 2010. MMWR Recommendations and reports : Morbidity and mortality weekly report Recommendations and reports[END_REF] . According to these guidelines, women are at high risk of anaphylaxis if they report at least one of the following four clinical responses following administration of a penicillin or a cephalosporin: anaphylaxis, angioedema, respiratory distress, or urticaria. Among those who selfreported penicillin allergy but did not present these symptoms, treatment by cefazolin (a firstgeneration cephalosporin) is recommended. However, in clinical practice, only 14% to 44% of the women at low risk for anaphylaxis received cefazolin for the prevention of S. agalactiae perinatal infection [START_REF] Van Dyke | Evaluation of universal antenatal screening for group B streptococcus[END_REF][START_REF] Briody | Use of Cefazolin for Group B Streptococci Prophylaxis in Women Reporting a Penicillin Allergy Without Anaphylaxis[END_REF] . Another frequent cause of antibiotic prescription during pregnancy is prophylaxis for cesarean delivery. First-line treatment is cefazolin, except for women at high risk of anaphylaxis following administration of penicillins or cephalosporins [START_REF] College Of | ACOG Practice Bulletin No. 120: Use of prophylactic antibiotics in labor and delivery[END_REF] . However, in a recent study more than 50% of the women that received alternative antibiotics were not at high risk of anaphylaxis and should have received cefazoline [START_REF] Kawakita | Choice of Prophylactic Antibiotics and Surgical Site Infections After Cesarean Delivery[END_REF] . Thus, there can be major discrepancies between guidelines and clinical practice in pregnant women. The objective of our study was to optimize the prescription of antibiotics for pregnant women self-reporting penicillin allergy, through an educational intervention about classification of penicillin allergies targeting gynecologists, anesthesiologists and midwives.

METHODS

This quasi-experimental study intended to assess the effect of an educational intervention about penicillin allergy classification. The target population of the intervention was the hospital staff potentially prescribing antibiotics for pregnant women (gynecologists, anesthesiologists, midwives). It took place in a single university hospital. Approval of the study was obtained from the French Data Protection Authority (CNIL) under number DEC2015-86.

One year before the beginning of the study, we proposed a standardized protocol of penicillin allergy classification based on allergy history. This classification was validated by interdisciplinary consensus between the allergy, anesthesiology, bacteriology, infectious diseases, gynecology and obstetrics teams of our hospital. Three levels of risk were defined, associated with recommendations of antibiotic therapy (Table 1). Level 0 is defined by symptoms not usually associated with penicillin allergies. In this situation, treatment by penicillin is recommended if indicated. At level 1, the existence of a severe allergic reaction is considered improbable; penicillins are contraindicated but cefazolin is recommended. At level 2, the probability of a severe allergic reaction is judged high and all β-lactams are contraindicated. The penicillin allergy classification was available in local prescribing resources from June 2014.

We conducted an educational intervention to bring this classification to the staff's attention. For 6 months (from July 2015 to December 2015), a combination of two strategies were used: dissemination of printed educational materials and group education. Posters of the classification were displayed in the prenatal and anesthesia examining rooms as well as in the delivery rooms. It was included in the pocketbook given to medical residents on their first day of arrival. The classification was formally presented to all the hospital staff potentially prescribing antibiotics for pregnant women. During weekdays morning reports, on which complicated deliveries from the last 24 hours are discussed, each time penicillin allergy was reported the penicillin allergy classification was reminded and feedback was provided. After the intervention period, the penicillin allergy classification continued to be presented to the new staff joining (mainly medical residents), but there was no specific focus during weekday morning reports.

To measure the effect of the educational intervention, two periods were studied. The preintervention period took place from March 1 to May 29, 2015. The postintervention period began from August 1 to October 26, 2016. Consecutive women having given birth during these periods were eligible to inclusion, even if the pregnancy had resulted in termination or an in-utero death. Only women with multiple pregnancies were excluded. Maternal and neonatal data were collected from the medical files: social and demographic characteristics, obstetric history, data related to pregnancy, delivery, and the newborn. All the antibiotic prescriptions during pregnancy or delivery were also collected for all women who reported penicillin allergy.

The antibiotic therapy was considered appropriate when the allergy was classified according to the three predefined levels of risk, and the recommended antibiotic therapy was prescribed; it was considered incorrect in all other cases, including when the allergic risk level was not reported in the medical file. When penicillin was indicated as a first-line treatment (e.g. prevention of S. agalactiae neonatal disease), the recommended antibiotic therapy was penicillin for risk level 0, cefazolin for risk level 1, non-β-lactams for risk level 2. When cefazolin was indicated as a first-line treatment (e.g. antibiotic prophylaxis for cesarean section), the recommended antibiotic therapy was cefazolin for risk level 0 and 1, non-β-lactams for risk level 2.

The statistical analysis was performed with R Software [START_REF] Core | R: A language and environment for statistical computing[END_REF] . Qualitative variables were compared with Fisher's exact test or the Chi-2 test, and continuous variables with Student's t-test. Percentages are reported in parentheses, and means are reported with the standard deviation of the distribution. The principal endpoint was the appropriateness of the antibiotic therapy prescribed to women having given birth, defined in advance for each level of allergic risk. Secondary study outcome was the percentage of self-reported penicillin allergies correctly classified in the medical records. A P-value less than 0.05 was considered statistically significant.

RESULTS

There were 998 deliveries during the preintervention period and 971 deliveries in the postintervention period (Figure 1). After exclusion of multiple pregnancies and of records that could not be located, 903 and 893 women, respectively, were included.

The women in both study periods were representative of the hospital's patient population. The two groups were comparable for almost all their general and obstetric characteristics (Table 2), except for a slight increase in body mass index (BMI) and a reduction in the preterm delivery rate. Reported allergies to penicillin were stable (6.8 vs. 5.4%, P= 0.24).

Among the women who reported penicillin allergy, the penicillin allergy classification was more frequently used in the postintervention period (68.9 vs. 100%, P<0.001) (Table 3). Around half the women who reported penicillin allergy received at least one antibiotic during pregnancy or delivery, this percentage did not vary significantly between the study periods (47.5 vs. 56.3%, P=0.44).

Following the educational intervention, the rate of appropriate antibiotic treatment increased from 17.2 to 66.7% (P<0.001) (Table 4). In particular, the rate of cefazolin prescription increased from 7.1 to 38.5% for the women at a low-risk level, though it was not statistically significant (P=0.07). Antibiotics were prescribed principally at delivery. No anaphylactic reactions were reported. Table 5 presents the classes of antibiotics administered during each period, according to the women's allergy risk level. In women for whom a serious allergy was considered improbable (risk level 1), cefazolin use increased between the two periods but this was not significant (7.1% before vs. 38.5% after, P=0.07).

DISCUSSION

We investigated the impact of an educational intervention on selection of appropriate antibiotic therapy among women who reported penicillin allergy. Selection of appropriate of antibiotic therapy increased significantly between the two periods (from 17.2% to 66.7%). Our study suggests that an educational intervention could increase the use of a penicillin allergy classification among gynecologists, anesthesiologists, and midwives.

Educational interventions about penicillin allergy have been previously described in the literature [START_REF] Krishna | Enhancing antibiotic stewardship by tackling "spurious" penicillin allergy[END_REF][START_REF] Blumenthal | Effect of a drug allergy educational program and antibiotic prescribing guideline on inpatient clinical providers' antibiotic prescribing knowledge[END_REF] . An electronic survey of 323 inpatient providers showed baseline knowledge deficiencies, as 42% reported no prior education in penicillin allergy [START_REF] Blumenthal | Effect of a drug allergy educational program and antibiotic prescribing guideline on inpatient clinical providers' antibiotic prescribing knowledge[END_REF] . After a 10-min educational presentation introducing a penicillin allergy clinical guideline, preparedness to determine if an allergy was severe improved from 77% to 92% (P=0.03). Preparedness to prescribe antibiotics to patients with penicillin allergy slightly improved from 66% to 76%, but this was not statistically significant (P=0.13) [START_REF] Blumenthal | Effect of a drug allergy educational program and antibiotic prescribing guideline on inpatient clinical providers' antibiotic prescribing knowledge[END_REF] . Another study proposed the use of a computerized algorithm to help healthcare professionals to identify low-risk patients with an inaccurate label of penicillin allergy [START_REF] Krishna | Enhancing antibiotic stewardship by tackling "spurious" penicillin allergy[END_REF] . However, as this study only demonstrated the feasibility of implementing a computerized algorithm, its effect in clinical practice remains unknown.

The proportion of pregnant women in our study who reported penicillin allergy was around 6% during both periods. This proportion is similar to the general adult population, ranging from 5 to 10% [START_REF] Borch | The prevalence of suspected and challenge-verified penicillin allergy in a university hospital population[END_REF][START_REF]Elective penicillin skin testing and amoxicillin challenge: effect on outpatient antibiotic use, cost, and clinical outcomes[END_REF][START_REF] Solensky | Clinical approach to penicillin-allergic patients: a survey[END_REF] . The prevalence of penicillin allergies reported among pregnant women was recently assessed at 8.4% among 170,000 women in California [START_REF] Desai | Morbidity in Pregnant Women Associated with Unverified Penicillin Allergies, Antibiotic Use, and Group B Streptococcus Infections[END_REF] . The prevalence of anaphylaxis during pregnancy is much lower, estimated at 1.6 per 100 000 maternities, and is caused by antibiotics in 49% of cases [START_REF] Mccall | The incidence, characteristics, management and outcomes of anaphylaxis in pregnancy: a population-based descriptive study[END_REF] . In the UK, the primary antibiotics associated with perioperative anaphylaxis were co-amoxiclav (a β-lactam) and teicoplanin [START_REF] Harper | Anaesthesia, surgery, and lifethreatening allergic reactions: epidemiology and clinical features of perioperative anaphylaxis in the 6th National Audit Project (NAP6)[END_REF] . In particular, teicoplanin was 17fold more likely to cause anaphylaxis than other antibiotics. Therefore, avoiding penicillin in patients who are not truly penicillin-allergic can paradoxically cause more anaphylactic reactions if teicoplanin is used as an alternative.

Among pregnant women, antibiotics are often prescribed during delivery, especially to prevent perinatal S. agalactiae [START_REF] Desai | Morbidity in Pregnant Women Associated with Unverified Penicillin Allergies, Antibiotic Use, and Group B Streptococcus Infections[END_REF] . The Centers for Disease Control (CDC) issued specific guidelines in 1996, and updated them in 2002 and 2010 [START_REF] Verani | Prevention of perinatal group B streptococcal disease--revised guidelines from CDC, 2010. MMWR Recommendations and reports : Morbidity and mortality weekly report Recommendations and reports[END_REF] . The current guidelines call for pregnant women who are colonized by S. agalactiae at the end of the third trimester to be questioned about the specific types of allergic reaction. The use of penicillin or cephalosporins is contraindicated among women who reported severe reactions such as anaphylaxis, angioedema, respiratory distress, or urticaria following administration of penicillins or cephalosporins. These women should receive clindamycin or vancomycin during labor, depending on the antimicrobial susceptibility of the S. agalactiae strain. Among pregnant women self-reporting penicillin allergy but without these severe symptoms, treatment by cefazolin (a first-generation cephalosporin) is recommended.

Compared with the CDC classification, our local policy classification was used for this study and covers a broader range of indications for antibiotic therapy during pregnancy and delivery. It also introduces a level 0 of risk for women with symptoms not associated with penicillin allergy (mycosis, fever, vertigo, diarrhea). Isolated diarrhea following β-lactams administration was not considered to be associated with allergy in our classification. Diarrhea might be observed in allergies, but is rarely the only presenting sign [START_REF] Ellis | Diagnosis and management of anaphylaxis[END_REF] . Moreover, diarrhea occurs in 10% of patients treated by amoxicillin-clavulanic acid [START_REF] Bush | Ureidopenicillins and beta-lactam/beta-lactamase inhibitor combinations[END_REF] , leading us to include it in risk level 0 (probably not an allergy) [START_REF] Hermanides | Presumed beta-Lactam Allergy and Cross-reactivity in the Operating Theater: A Practical Approach[END_REF] . Risk level 0 covered 12.5% of the women during our postintervention period. Such women should be able to receive penicillin, as recommended during labor for colonization by S. agalactiae.

We selected cefazolin for women who presented with delayed urticaria after penicillin administration, although the CDC guidelines from 2010 recommend not using β-lactams in this situation. A recent decision algorithm for presumed β-lactam allergy recommended cefazolin for mild symptoms after exposure to penicillin, that included: delayed urticaria and any skin rash occurring > 2h after exposure [START_REF] Hermanides | Presumed beta-Lactam Allergy and Cross-reactivity in the Operating Theater: A Practical Approach[END_REF] . Indeed, the most common reaction is the type IV allergic reaction, which is T-cell-mediated and occurs from 2h to several days after the last drug administration. These nonimmediate reactions usually include skin reactions like delayed urticaria or maculopapular exanthema [START_REF] Cernadas | General considerations on rapid desensitization for drug hypersensitivity -a consensus statement[END_REF] and pose no risk of anaphylaxis [START_REF] Hermanides | Presumed beta-Lactam Allergy and Cross-reactivity in the Operating Theater: A Practical Approach[END_REF] .

Cross-reactions between penicillins and cephalosporins are rare [START_REF] Campagna | The use of cephalosporins in penicillin-allergic patients: a literature review[END_REF] , and 90% of them are due to a similarity in the R1 side chain, target of an IgE-mediated immune-allergic reaction [START_REF] Baldo | Detection and side-chain specificity of IgE antibodies to flucloxacillin in allergic subjects[END_REF] . Cefazolin does not share this side chain with the penicillins, nor do the third and fourth generation cephalosporins [START_REF] Zagursky | Cross-reactivity in beta-Lactam Allergy[END_REF] . In two retrospective studies of penicillin-allergic patients treated by cefazolin (n=128 and n=413), only one patient presented an anaphylactic reaction [START_REF] Romano | Crossreactivity and tolerability of cephalosporins in patients with immediate hypersensitivity to penicillins[END_REF][START_REF] Goodman | Cephalosporins can be given to penicillin-allergic patients who do not exhibit an anaphylactic response[END_REF] . In another study based on electronic health records, out of 65,915 patients with a history of penicillin allergy, only 3 cases of cephalosporin-associated anaphylaxis (0.004%) were documented [START_REF] Macy | Adverse reactions associated with oral and parenteral use of cephalosporins: A retrospective population-based analysis[END_REF] . Therefore, cefazolin can be safely administered to most of the patients with a history of penicillin allergy, in case of mild symptoms occurring > 2h after exposure to penicillin.

The quasi-experimental design of our study is one of its limitations. Indeed, the improved appropriateness of antibiotic therapy observed over the two periods might be associated with other factors than the educational intervention. This hypothesis nonetheless appears improbable given the short interval between the two periods and the extent of the difference observed (a year apart and an increase in the appropriateness of antibiotic therapy from 17.2 to 66.7%). Moreover, no major modification of our management protocols took place between the two periods. Patients characteristics were similar, except for a moderate rise in BMI and a reduction in the preterm delivery rate. An increase in BMI was observed among pregnant women throughout France during the same period [START_REF] Blondel | Trends in perinatal health in metropolitan France from 1995 to 2016: Results from the French National Perinatal Surveys[END_REF] . However, the pre-term birth rate slightly increased in France between 2010 (6.5%) and 2016 (7.5%) [START_REF] Blondel | Trends in perinatal health in metropolitan France from 1995 to 2016: Results from the French National Perinatal Surveys[END_REF] . Another limitation of this study is its monocentric design. This may limit the generalization of our findings.

In conclusion, this study showed that an educational intervention about penicillin allergy classification was associated with an improvement in the selection of appropriate antibiotic therapy in women who reported penicillin allergy. Further studies are warranted to identify the most effective educational strategies to improve the management of patients with suspected penicillin-allergy.

Figure 1 :

 1 Figure 1: Flow chart of the study.

Table 1 .

 1 Clinical classification of penicillin allergy depending on the allergy history.

	Self-reported symptoms of penicillin	Risk level
	allergy
	Diarrhea, nausea, vomiting, abdominal	Level 0. Probably no allergy
	pain	-No contraindication to penicillins
	Vertigo	-Use of penicillin desirable if indicated
	Fever
	Mycosis
	Cutaneous eruption	Level 1. Severe allergy improbable
	Delayed urticaria	-Contraindication to penicillins
	Doesn't know	-Cephalosporins desirable
		if β-lactams are indicated a
	Hospitalization for allergies	Level 2. High risk of serious allergy
	Anaphylactic shock	-Contraindication to all β-lactams
	Malaise with loss of consciousness
	Facial edema, angioedema
	Asthma, respiratory problems, feeling of
	suffocation

a Prefer cefazolin or third-or fourth-generation cephalosporins.

Table 2 .

 2 Women's general and obstetric characteristics before or after the educational intervention.

	Preintervention	Postintervention	P
	n= 903	n= 892

a Values for age, body mass index, gestational age at delivery and birth weight are mean ± standard deviation.

b NICU = Neonatal Intensive Care Unit.

Table 3 .

 3 Classification of allergies according to allergy history and prescription of antibiotic therapy.

	Preintervention	Postintervention	P
	n= 61	n= 48

Table 4 .

 4 Antibiotic therapy during pregnancy or delivery among women who self-reported penicillin allergy.

	Preintervention	Postintervention	P
	n= 29	n= 27

Table 5 .

 5 Type of the antibiotics administered depending on the risk level of penicillin allergy.The antibiotic class in bold is recommended for the corresponding risk level. a 2 women received fluoroquinolones, which were more appropriate for the infection (according to the antibiotic susceptibility testing); this antibiotic therapy was considered correct.

	Risk level	Preintervention	Antibiotic therapy	Postintervention	p
		n= 29		n=27
		n=2		n=3
	0	1 (50.0%)	Penicillins	2 (66.7%)	>0.99
	Probably no allergy	-	Cephalosporin	1 (33.3%)
		1 (50.0%)	Macrolides	-
		n=15		n=13
	1	1 (7.1%)	Cephalosporin	5 (38.5%)
	Severe allergy	3 (14.3%)	Penicillins	-	0.07
	improbable	11 (78.6%)	Macrolides	6 (46.2%)
		-	Other*	2 (15.4%)
	2	n=3 3 (100%)	Macrolides	n=11 11 (100%)
	High risk of serious	-	Penicillins	-	> 0.99
	allergy	-	Cephalosporin	-
	Unclassified	n=9 1 (11.1%)	Penicillins	n=0 -
		1 (11.1%)	Cephalosporin	-	-
		7 (77.8%)	Macrolides	-
	Total	7 (24.1%) 5 (17.2%)	β-lactams -Penicillins	8 (29.6%) 2 (7.4%)
		2 (6.9%)	-Cephalosporins	6 (22.2%)	0.56 b
		22 (75.9%)	Macrolides	17 (63.0%)
		-	Other a	2 (7.4%)

b β-lactams vs. other.

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.


Alexandre Benoit

Laurent Bonnaud

Alice Caplier

Phillipe Ngo

Lionel Lawson

Daniela G Trevisan

Vjekoslav Levacic

Céline Mancas

Guillaume Chanel

Multimodal Focus Attention Detection in an Augmented Driver Simulator

Keywords: driver simulator, facial movements analysis, physiological signals, stress, attention level, data fusion, fission, OpenInterface

This project proposes to develop a driver simulator, which takes into account information about the user state of mind (level of attention, fatigue state, stress state). The user's state of mind analysis is based on video data and physiological signals. Facial movements such as eyes blinking, yawning, head rotations… are detected on video data: they are used in order to evaluate the fatigue and attention level of the driver. The user's electrocardiogram and galvanic skin response are recorded and analyzed in order to evaluate the stress level of the driver. A driver simulator software is modified in order to be able to appropriately react to these critical situations of fatigue and stress: some visual messages are sent to the driver, wheel vibrations are generated and the driver is supposed to react to the alertness messages. A flexible and efficient multi threaded server architecture is proposed to support multi messages sent by different modalities. Strategies for data fusion and fission are also provided. Some of these components are integrated within the first prototype of OpenInterface (the Multimodal Similar platform).

I. INTRODUCTION

T he main goal of this project is to use multimodal signals processing to provide an augmented user's interface for driving. The term augmented here can be understood as an attentive interface supporting the user interaction. So far at most basic level, the system should contain at least four components:

1. sensors for determining user state of mind; 2. an inference engine feature extractor to evaluate incoming sensor information 3. an adaptive user interface based on the results of step 2 4. an underlying computational architecture to integrate these components.

In fact a fully functioning system would have many more components, but the previous components are the most critical for inclusion in an augmented cognition system and they are covered in the project implementation.

Basically to provide such multimodal application, we address the following issues: which driver simulator to use? How to characterize a user's state of fatigue or stress? Which biological and/or physiological signals to take into account? What kind of alarm to send to the user? How to integrate all these pieces -data fusion and fission mechanism? Which software architecture is more appropriate to support such kind of integration? A software architecture supporting real time processing is the first requirement of the project because the system has to be interactive. A distributed approach supporting multi thread server can address such needs. The choice of the driver simulator has to take into account some features such as: open source software, "First person view": (i.e. cockpit view with wheel) and dashboard, source code easy to modify and possible use of a vibration feedback wheel. We consider two user's states to be detected: stress and fatigue. The detection of these states is based on video information and/or on biological information. From video data we extract relevant information to detect fatigue state while the biological signals provide data for stress detection. Physiological signals could be associated to video data in order to detect fatigue but in the context of the driver simulator used in this project, a real fatigue state is very difficult to obtain. It is possible to simulate fatigue on video data (by closing the eyes or by yawning for example). On the contrary, such kind of simulation is not possible on physiological signals.

The third step is to define what kind of alarms to provide to the user. Textual messages and force feedback are considered to alert the user.

An other challenge of this project is to provide a concrete example in order to test OpenInterface, the multimodal Similar platform.

The rest of the paper is organized as follow: section II present the global architecture of the demonstrator, section III describes how we detect driver's hypo-vigilance states by the analysis of video data, section IV presents how to detect driver's stress states by the analysis of some biological signal, sections V and VI describe the data fusion and fission strategies and section VII gives details about the demonstrator implementation.

II. CONCEPTUAL ARCHITECTURE

The diagram of Figure 1 presents the conceptual architecture of our attentive driver simulator. We propose a distributed approach to integrate our components. On one PC under Linux we have integrated all video data based detection and analysis as well as the fusion and fission components. An other PC under Windows is used to run the driver simulator and a third PC is used for biological signals acquisition and analysis. Communication between all the PCs is done exchanging XML messages. For that the Dialog Controller included in the driver Simulator software should be able to receive multi messages (i.e. from biological signals station and from video based station). In this case a multi thread server approach is developed and included in the driver simulator. The state of hypo-vigilance (either related to fatigue or inattention) is detected by the analysis of video data. The required sensor is a camera facing the driver. In this project, three indices are considered as hypo-vigilance signs: yawning, head rotations and eyes closing for more than 1s.

A. Face detection

Face detection is the first and maybe the most crucial step of the image processing phase. The face detector should be robust (no error in face localization) and should work in real time. The chosen face detector is the free toolbox MPT [START_REF]Machine Perception Toolbox[END_REF]. This face detector extracts a square-bounding box around each face in the processed image. Face detection is done for each image of the sequence without any face tracking. The advantage is that the head is not lost because of tracking error propagation. The main drawback is the decrease of the frame rate even though MPT works nearly in real time for pictures of size (320x200 pixels), which is not the case of other face detectors such as OpenCV [13] for example. Whichever face detector you use, the extracted face bounding box is not exactly the same from frame to frame so that we use a temporal median filter with temporal adaptive position mean to make the spatial localization of the face temporally stable (note that the size of the bounding box of the face is supposed to be constant during an experiment: in a car the driver face distance w.r.t the camera is stable if the driver stays on his seat).

B. Head motion analysis

Once a bounding box around the driver face has been detected, head motion such as head rotations, eyes closing and yawning are detected by using an algorithm working in a way close to the human visual system. In a first step, a filter coming from the modeling of the human retina is applied. This filter enhances moving contours and cancel static ones. In a second step, the FFT of the filtered image is computed in the log polar domain as a modeling of the primary visual cortex. Figure 2 gives a general overview of the algorithm.

Figure 2: Algorithm for hypo-vigilance features extraction from video data

The first step consists in an efficient prefiltering [START_REF] Beaudot | The neural information processing in the vertebrate retina: A melting pot of ideas for artifficial vision[END_REF]: the retina OPL (Outer Plexiform Layer) that enhances all contours by attenuating spatio-temporal noise, correcting luminance and whitening the spectrum (see Figure 3) . The IPL filter (Inner Plexiform Layer) [START_REF] Beaudot | The neural information processing in the vertebrate retina: A melting pot of ideas for artifficial vision[END_REF] removes the static contours and extracts moving ones. The second step consists in a frequency analysis of the spectrum of the OPL and IPL filters outputs in each region of interest of the face: global head, eyes and mouth (see section C for the description of eyes and mouth region of interest extraction).

Figure 3 : OPL filtering results

In order to estimate the rigid head rotations [START_REF] Benoit | Head nods analysis : interpretation of non verbal communication gestures[END_REF], the proposed method analyses the spectrum of the IPL filter output in the log polar domain. It detects head motion events and is able to extract its orientation. Cortical optical flow filters [START_REF] Torralba | An efficient neuromorphic analog network for motion estimation[END_REF] are oriented filters that compute the optical flow of the global head to extract the motion direction. Optical flow is computed only when motion is detected.

For the detection of yawning or eyes closing, three identical processes are done independently [START_REF] Benoit | Hypovigilence Analysis: Open or Closed Eye or Mouth ? Blinking or Yawning Frequency ?[END_REF]. On each region of interest (each eye and the mouth), a spectrum analysis of the OPL and IPL filters output is done for motion event detection: we are looking for vertical motion related to eyes closing or to yawning.

C. Eyes and mouth detection

The mouth can be easily extracted in the lower half of the detected bounding box of the head. The detection algorithm will work even if the mouth is not perfectly centered in the area because we analyze the spectrum energy instead of spatial features, which is more robust. Moreover, there are no disturbing contours in that area that could generate false detections. Concerning the eyes, the spectrum analysis in the region of interest is accurate only if each eye is correctly localized. Indeed around the eyes, several vertical or horizontal contours can generate false detection (hair boundary for example). The MPT toolbox proposes an eye detector but it requires too much computing time so that it is not compliant with real time constraint. We use another solution: eye region is supposed to be the area in which there is the most energized contours. To do so, assuming that the eyes are localized in the 2 upper quarters of the detected face, we use the retina output. The retina output gives the contours in these areas and due to the fact that the eye region (containing iris and eyelid) is the only area in which there are horizontal and vertical contours, the eye detection can be achieved easily. We use two oriented low pass filters: one horizontal low pass filter and a vertical low pass filter and we multiply their response. The maximum of the result is obtained in the area in which there are the most horizontal and vertical contours that is an eye region. To make the eye areas temporally stable, their position is smoothed from frame to frame using adaptive mean positions. This eye detection takes about 6 operations per pixel for each search area (i.e. each upper quarter of the face bounding box). Figure 4 gives an example of the input picture of the eye detector; bright areas are the most important contours. Figure 5 shows the output of the horizontal vertical filters.

D. Hypo-vigilance alarms generation

• We generate an alarm when both eyes are closed longer than a specific time period (1 second for example). • We detect mouth yawning: when a yawn occurs, the mouth is wide open, then, this generates a very highenergy increase on the spectrum that can be easily extracted.

• The global head motion events are detected with the global head spectrum analysis. We only extract the fact that a head motion has occurred. The proposed algorithms are able to extract the motion direction with the cortical optical flow algorithm, but it is not yet integrated in the fusion system.

E. Fusion strategy

After the video analysis, Boolean information about yawning or not, about eyes closing or not and about head moving or not are available. A very simple and easy to compute fusion strategy based on the three index is proposed: if head motion is detected send an alarm to the user hypo-vigilance value=100 else if both eyes are closed during 1s send an alarm to the user hypo-vigilance value = 50 if the driver is yawning send an alarm to the user hypo-vigilance value = 50+hypovigilence value end The variable hypo-vigilance associated to each index is set to 50 or 100. The highest the value, the highest the hypovigilance. Note that in this very simple fusion strategy, information about head motion kind of rotation is not taken into account. A more sophisticated fusion strategy has been tested and is described in section V.

IV. STRESS DETECTION BASED ON BIOLOGICAL SIGNALS ANALYSIS Physiological signals are used in order to detect stress situation. ECG (Electrocardiogram) and GSR (Galvanic Skin Response) are announced by literature as very promising to detect driver stress in real situations [START_REF] Lal | Driver fatigue: electroencephalography and psychological assessment[END_REF][START_REF] Healey | Quantifying driver stress: developing a system for collecting and processing bio-metric signals in natural situation[END_REF]. In a stressful time, the GSR signal and the heart rate signal (extracted from the ECG) are supposed to increase. Two different experiments have been considered; they aim at detecting either driver stress over a long time period or punctual driver stress. In this experiment, we use the Biopac system MP30B-CE for ECG and GSR acquisition. The main drawback of the data acquisition system is that for the moment, on line analysis is not possible. For that reason, the study on biological signal for stress detection has not been implemented in the final demonstrator.

A. ECG signals analysis 1) Prefiltering

Since we are analysing the stress state of a driver, the ECG can be disturbed by several muscle artefacts that generally come from hands or arms movements (see Figure 7). This is why it is necessary to pre-filter signals. The pre-filtering is based on the characteristics of the ECG peaks: we observed that these peaks contain energy in the frequency band 10-35 Hz. As we do not need other components of the signal we choose to band-pass the signal using this interval and a Butterworth IIR filter with 8 coefficients. Figure 8 shows the original signal and the results after filtering.

2) Heart rate computation

The first thing to do, before computing heart rate, is to identify peaks in the filtered signal. For this, we use a reference record (ECG when the subject is supposed to be relaxed) as a baseline to identify the general height of the peaks depending on the subject. We define the general height of peaks as one third (chosen empirically) of the maximum value. In order to improve the peak detection, we also use a priori information: we consider that the heart rate cannot exceed 180 BPM (Beats Per Minute). If two peaks are too close, so that they do not validate this assumption, we keep only the one with the maximum value. Finally, the heart rate is computed by evaluating the number n of samples between two peaks and by using this simple formula:

HR = 60/ (n*(1/f e ))
Where f e is the sampling rate.

3) Stress level assessments

In case of unexpected or stressful events, the heart rate does not increase as generally assumed, but one can observe a raise in its variation. In order to determine this variation, we use the absolute value of the first derivative of the signal. After smoothing the result by a Gaussian filter we obtain what we call the stress level (see Figure 9).

1) Global stress detection

We can easily measure the minimum and maximum of the user's GSR level. By normalizing the signal to analyze with these values as usual:

) _ min( ) _ max( ) _ min( _ _ _ rest GSR rest GSR rest GSR analyse to level GSR - -
where GSR_rest correspond to the values of the GSR when the user is supposed to be relaxed. We can then define a score to know the global stress level.

Global stress can occur after a very difficult day of work or when the traffic jam is increasing for example. Global stress is related to slow but constant increase of the GSR. The global stress level is used to know the initial state of the driver or with a sliding window, to know the global state of the driver in a certain amount of time.

2) Local stress detection

Local stress is supposed to be related to punctual and unforeseen events as it could occur on roads such as a pedestrian crossing and so on. Local stress detection can be modelled as high peaks in GSR signals. GSR signals have the property to react quite quickly to an event but to have a decreasing response to go back to a calm situation very slowly. We used this particular property as a priori information in our algorithm. First of all, we detect local maxima using the watershed algorithm. Local maxima correspond to watershed pixels. Once maxima are detected, we keep only those, which have a difference of 1 unit with the previous maximum. This threshold of 1 unit is based on the correlation of punctual events, precisely recorded, and GSR signals. Then, we remove all maxima, which are too close to each other by keeping only the highest one. This rule is based on the assumption that if two maxima are too close to each other (inferior to 5 seconds), they belong to the same event. Figure 10 presents an example of GSR local increasing detection: each peak on the bottom curve corresponds to a stress alert. In this section, we describe and test a data fusion based on Bayesian Network. It is used for the purpose of hypovigilance detection but it also represents a global fusion method for the integration of additional information in the detection process. Note that this fusion process is not integrated in the final demonstrator for the moment due to the lack of significant data. Both fusion strategies are implemented in the demonstrator but for the moment, only the simplest one described in III.D. is used by default for computational cost reduction. Human fatigue generation is a very complicated process. Several uncertainties may be present in this process. First, fatigue is not observable and it can only be inferred from the available information. In fact, fatigue can be regarded as the result of many contextual variables such as working environments, health and sleep history. Also, it is the cause of many symptoms, e.g. the visual cues, such as irregular eyelid movements, yawning and frequent head tilts. Second, human's visual characteristics vary significantly with age, height, health and shape of face. To effectively monitor fatigue, a system that integrates evidences from multiple sources into one representative format is needed. Naturally, a Bayesian Networks (BN) model is a good option to deal with such an issue.

Figure 11: Fusion strategy based on a Bayesian Network

A BN provides a mechanism for graphical representation of uncertain knowledge and for inferring high-level activities from the observed data. Specifically, a BN consists of nodes and arcs connected together forming a directed acyclic graph. Each node can be viewed as a domain variable that can take a set of discrete values or a continuous value. An arc represents a probabilistic dependency between the parent node and the child node. Some contextual information such as temperature, time of day, sleep history, etc can be used to build a prior probability for the fatigue node. For that we use the parameters proposed in [START_REF] Ji | Real-Time Nonintrusive Monitoring and Prediction of Driver Fatigue[END_REF]. For the face data fusion we have considered a very preliminary version where the network evidences change when: eyes closed more than 1 sec; yawning occurs; down head motion are detected simultaneously or not. As result we got the level of fatigue, which is sent to the data fission component.

VI. FISSION STRATEGY

Data fission duty is to collect the data from data fusion and to generate an alert XML message that is sent to the driver simulator. Data fission function is called at the rate the driver state detection is progressing. Generated messages are in XML format. We decided for XML because it is extendable and messages are sent only when the driver state changes. Driver state may be defined by a fatigue value (either coming from the Bayesian Network result or from the simple fusion process) that is an output variable of data fusion. For example, we can set the range of values for fatigue level that determine the driver state. For those range of values we can define different screen messages and wheel shaking power. Table 1 and Table 2 present the fusion strategy for the simple method and for the Bayesian network based method respectively. In that way the XML message does not need to be sent to the driver simulator after each call of the data fission function.

Fatigue

Once the alert message has been sent, the driver is supposed to acknowledge to the system that the message has been understood. For example, in the case of the simple fusion process, each time an alert is detected, wheel vibrations are triggered. The driver has to stop these vibrations by pushing a button. The reaction time is also recorded, this time being correlated with the hypo-vigilance or fatigue user state.

VII. DEMONSTRATOR

A. Overview of the global system

The developed demonstrator is made of

• 2 PCs: one under Windows for the driver simulator and one under Linux for hypo-vigilance states detection

• 1 SONY digital camera • 1 LOGITECH force feed back wheel • 1 projection screen • 1 video-projector • 2 loudspeakers

Figure 12: Global views of the demonstrator

On the used computer (Pentium 4 2.4Ghz), the frame rate is about 5 frames per second but it could be increased up to 8 frames per second thanks to some MPT optimization.

B. Driver Simulator

Around ten driver simulators have been studied. The chosen driver simulator is TORCS [START_REF]TORCS[END_REF] because it is a well architectured GPL program with well structured source code and a well designed user interface. This simulator is working under Linux and windows platforms. The main sources are written in C++ with the OpenGL library. The graphics quality of the simulator is correct and it has a first person view. Figure 13 presents an illustration of TORCS simulator.

Figure 13: Torcs driver simulator illustration

We integrate an interaction from the Data Analysis Kernel to our driving Simulator.

The main work consisted in -Allowing a Text Message to be displayed within the game graphical interface.

-Creating a multi-threaded Server within the application whose purpose is accepting different clients connexions.

-Integrating a force Feedback wheel in order to warn the user with an other modality than the visual one.

-Allowing the user to make a feedback on the message displayed by stopping it.

-Parsing XML messages from the multimodal analysis of the driver. Indeed, it is possible to change the color, the string of the sent message and the feedback power.

C. Implementation of hypo-vigilance detection

Due to the fact that ECG and GSR signals cannot be processed on line with the data acquisition station we used, the detection of stress state has not been implemented in real time. Only the detection of hypo-vigilance state based on video data is available at the moment.

1) Face detection algorithm modifications

For face detection, we use the Matlab implementation of mpiSearch function belonging to the MPT library, which receives a RGB or Gray level frame as input. Outputs of the function are the bounding box coordinates of the detected face. Due to the relative slowness of the mismatch function developed under Matlab a fine study of the algorithm has been done in order to increase the computational rate. We managed to figure out how the algorithm behaves in dynamic environment when the video is acquired with the help of DirectInput library. While streaming, mpiSearch uses a special object that caches the detected face-bounding box. The trick is to modify the mpiSearch Mex function and to put this object as a global DLL variable. Global DLL variables are preserved in Matlab memory space after the DLL is first accessed by Matlab. In that way after each Matlab call of the mpisearch function caching state is preserved. With using the Microsoft VS.NET 2003, we additionally increase the performances by compiling the code for new Pentium 4 generation of processors. With all these modifications, we achieve about 2.7 times speed increase. In order to increase the frame rate again, some of timeconsuming parts of the mpisearch code may be written in assembler language. This is beyond of scope of the project.

D. Alert message generation

1) The Display Changes Three different messages depending on the index of hypovigilance can be displayed on the first view of the driver simulator (see Table 1 for the different considered messages and Figure 14 for an example of message incrustation during the game). In order to show the message, we need to change all the graphical classes within the source code. The communication between the main program and all the libraries is created by using some global variables and also by creating new links between several libraries. 2) Force Feedback implementation Force Feedback is used to make the wheel shaking when hypo-vigilance is detected. Shaking power is defined by XML message of data fission. Shaking becomes stronger and gradually reaches its maximum value. Force Feedback uses the DirectInput library, a part of the Microsoft DirectX SDK. The library is based on "The Force Feedback Direct Input Library (DIL)" made by Bryan Warren and Alex Koch. This library can be loaded at [START_REF]Force Feedback Direct Input[END_REF]. In our project, the library has been altered from functions to class. In class we can set the time period that shaking needs to reach the maximum vibration time, vibration activation threshold and to modify the shaking of the wheel dynamically. We also use the class to check whether the button is pressed, and if pressed stop the wheel shaking.

3) Message parsing and controlling the input devices of driver simulator

After the XML message arrives through the socket connection it is parsed. We use Microsoft XML parser to parse the message. You can download the MSXML parser from Microsoft web site. After the parsing, controller class activates the screen display message or starts the wheel shaking.

4) The Server Side:

We choose to implement a Server side for the interaction between the multimodal devices and the user. The network protocol used is TCP/IP. We implement this socket by using threads. Those threads access global variables under mutual exclusion. We use a "GPL" library called Openthreads for this implementation.

E. Openinterface integration 1) OpenInterface: a short presentation

OpenInterface is the Similar Software Platform that includes software components dedicated to multimodal interaction and multimodal data fusion. OpenInterface integrates results from the Human-Computer Interaction (HCI) community as well as from the Signal Processing community. Each component is registered into OpenInterface Platform using the Component Interface Description Language (CIDL described in XML). The registered components properties are retrieved by the Graphic Editor (Java). Using the editor the user can edit the components properties and compose the execution pipeline of the multimodal application. This execution pipeline is sent to the OpenInterface Kernel (C/C++) to run the application. The OpenInterface Tutorial can be found on the Similar web site [10].

2) OpenInterface implementation of the demonstrator

Currently, OpenInterface is in its early development stage and this driver simulator project is used as a test bed for the working prototype. The latter provides several services and also has limitations.

The services provided by this first OpenInterface prototype are:

• Interface Description Language for the specification of reusable software code. • Seamless integration of reusable heterogeneous software (C/C++, Java, Matlab). Some limitations are:

• The description of a software interface is currently not automated and has to be written by hand. This can be cumbersome as the language syntax is very strict and the validity of a description is not heavily enforced besides DTD checking. This lack of robustness might lead to an inappropriate binding and therefore to unexplained application crash. • At the beginning of the project, it was impossible to perform two ways communication with any Matlab script. The latter could only be called by OpenInterface component. Several issues have been encountered during the integration phase:

• First of all as the current prototype is only running under Linux, we had to reduce the set of software to those that could be run under Linux. It has been decided that the communication with non-compliant Linux software will be done through network communication. Thus, only the following software code have been integrated into OpenInterface: firewire camera driver, face detection, head motion analysis, fusion for hypo-vigilance detection, fission strategy for alerting the user and a Java GUI to start the application. Figure 15 shows how these components are connected into OpenInterface. • After deciding which software to use as component into OpenInterface, the main problem we had to face has been to make those components reusable. Indeed, in their first version, the software was designed for a particular goal and could not be reused by a thirdpart (e.g. OpenInterface) as tools. Thus, we redesigned the components interaction interfaces. To avoid this redesigning step in the future, modular programming habit should be enforced from the beginning among components programmer.

• From this step we have pulled out that a two-way communication with Matlab script is mandatory to prevent heavy code rewriting. Indeed the camera driver has to be called by the face detection component, not the contrary. Therefore, due to the Matlab Engine API limitations, the only way to allow Matlab script to interact with component already running into OpenInterface is to perform communication through UNIX socket. Of course this is transparent to the user.

• The third integration step was the description, in CIDL, of all components interface. This step was straightforward and we did not find any difficulties in expressing the interfaces in the integration platform's CIDL.

• Finally we started the integration of the components.

It is performed in a gradual way so that we were able to test each component inside OpenInterface and point out the incompatibility. Some problems arose when we put two Matlab components in the same execution pipeline. It turned out that due to another undocumented Matlab Engine API limitation, we could not call another m-script while another is running. The lack of time directed us to fix the problem by having one Matlab engine instance per Matlab script taking part in a pipeline. One would think that this solution would drastically slow down the computer but that did not happen. A Matlab process actually uses resource proportional to the computation done by the running script.

Figure 15: Openinterface components coming from the project

As a conclusion of this integration phase using OpenInterface platform, we can say that thanks to this first project integration a lot of feedback has been collected to improve the current prototype of the OpenInterface tool in order to provide another improved version as soon as possible. The future engineering work on OpenInterface will be narrowed to improve the integration of Matlab components and also to provide a user with a friendly integration interface. This would allow distributing the platform and letting people test it on their own.

VIII. FUTURE WORKS AND CONCLUSION

During the project, we have developed an augmented driver simulator based on video analysis for driver's attention controlling. First promising studies about physiological data have to be improved and integrated in the global system. This will induce the development of an appropriate data fusion method in order to control both the driver's attention level and the driver's stress.

Once the driver has been alerted, it will be necessary to perform some specific tests in order to control that driver's stress or fatigue has actually decreased.

For the moment, the global system is running almost 10 frames per second. It will be necessary to optimize video data analysis algorithms in order to speed up the frame rate.

Figure 1 :

 1 Figure 1: Overview of the system architecture

Figure 4 :Figure 5 :

 45 Figure 4: Input picture for eye detection : one of the 2 upper quarters of the face-bounding box

Figure 6 :

 6 Figure 6: On the left, ECG devices and on the right, GSR device

Figure 7 :

 7 Figure 7: Driver with electrodes on the wrists for ECG measure

Figure 8 :

 8 Figure 8: ECG before (first line) and after pre-filtering (second line); computed heart rate from ECG (third line, red signal)

Figure 9 :

 9 Figure 9 : stress level (bottom) computed from heart rate (top) for exciting stimuli B. GSR signal analysis Due to the chosen sampling rate (200 samples per second) and the apparatus, some artefacts occur in the initial signals and a filtering is also required. After trying several smoothing filters and because of the large variability in the conductivity of each user, we opt for a multiscale median filtering. Four successive filters are used with a decreasing window size (100, 50, 30 and 20 samples).

Figure 10 :

 10 Figure 10: GSR record and punctual stress event detection

Figure 14 :

 14 Figure 14: alert message in case of long eyes closing

Table 1 : fission strategy with the simple fusion process

 1

	range	50	50	100
	Message	Open the eyes	Yawning: be careful	Stop moving the head
	Shaking power	'100'	'100'	'100'
	Fatigue range	[0,33]	[33,66]	[66,100]
	Message	''	'Tired'	'Asleep'
	Message color	''	'Green'	'Red'
	Shaking power	'0'	'0'	'100'

Table 2 : fission strategy with the BN based method

 2

Data fission only creates the message if the driver state has changed and is different than the previous driver state. If the user state is the same as in previous call, data fission generates 'NOT_CHANGED' message.

This report, as well as the source code for the software developed during the project, is available online from the eNTERFACE'05 web site: www.enterface.net.

ACKNOWLEDGMENT

This work was supported by the Similar network of excellence [10]

Phillipe Ngo was born in 1980 in France, He is still undergraduated from the UTBM (university of technology of Belfort Montbéliard (France)). His major is computer science with specialization in Picture, interaction and virtual reality. He is currently working for the LIS (INPG Grenoble France) for its last internship of computer science. His work is focused on human and computer interactions within a virtual reality environment. Daniela G. Trevisan was born in Santa Maria, Brazil, on 1974. She graduated in Informatic at the University Federal of Santa Maria, Brazil (UFSM) in 1997. She obtained Master degree in Computer Science from University Federal of Rio Grande do Sul, Brazil (UFRGS) in 2000. Currently she is PhD student at the Université Catholique de Louvain (UCL) at the Communication and Remote Sensing Laboratory (TELE) and she is also member of the Belgium Computer-Human Interaction Laboratory (BCHI). Her research topics are focused on human-computer interaction (HCI) field such as modelling multimodal interfaces, model based-approach, augmented and mixed reality and multimodal interfaces for image-guided surgery.