Battery Monitoring System using switching battery cells

G. Despesse, S. Lechat Sanjuan, S. Gery CEA-LETI, CEA-LITEN, Minatec Campus, 17 rue des Martyrs -38054 Grenoble Cedex 9, France ABSTRACT As opposed to the first battery technologies (Pb, Ni-Mh…), whose protection and management required simple electronic circuits, today's Li-ion batteries need to be precisely supervised during all their working cycles to limit their charges and discharges, to ensure their security and improve their lifetime. Battery management is currently performed by monitoring battery cell voltages: charging is interrupted when the voltage of the first cell of the serial arrangement exceeds a threshold value (charging then goes forward with a lower current for the remaining cells). Battery discharge is managed similarly and is interrupted when the voltage of first battery cell falls below a threshold value. The remaining energy in the other cells of the serial arrangement is not used. Hence, the energy stored into the battery is not fully exploited, which limits the autonomy of the supplied system. Non-dissipative solutions to exchange energy between cells so as to maintain the cells balanced have been reported; yet, they require a large number of DC/DC converters with associated size, weight and price drawbacks. To overcome these limitations and to remove the battery charger and output converter (DC/DC or DC/AC converter to supply the load), we propose a full converter based on battery cell switching, to adjust the battery pack output voltage to the effective need in real time and maintain the battery cells balanced. The architecture we developed consists in connecting or disconnecting in real time each cell placed in the serial arrangement. In the case of an electrical vehicle with 30kWh on board, by using battery cells of 3.3V, the output voltage can be adjusted by steps of 3.3V from -60V to +60V. We show that it is possible to directly drive a threephase electric motor with an efficiency up to 98% from few Hz to 400Hz, while maintaining the cells balanced. Moreover, this is achieved with a transistor cost lower than 300€ (consumer prices, expectedly 10 times lower for large quantities). For the transistor driving part, a distributed electronic is placed in front of each cell of the serial arrangement. Each distributed electronic circuit is able to drive up to 14 transistors, to carry out local measurements (temperature, current, voltage) and to calculate the estimated State of Charge of the local cell. Distributed electronic circuits communicate with an electronic advisor by using galvanically isolated a common serial communication bus.

SWITCHED BATTERY STRUCTURE

Standard battery architectures for electric vehicles are based on a battery arrangement of s battery levels connected in series, and battery levels are composed of p cells connected in parallel. This battery pack is connected on one side to a charger (to recharge the battery on the electrical network) and on the other side to the electric motor by an inverter (DC/DC or DC/AC electrical converter depending on the type of the motor to supply).

Figure 1 : Standard architecture around the battery pack

In typical applications, the number s of cells placed in series ranges from 15 (≈48V on DC bus) to few hundreds (up to 900 V on DC bus).

In this architecture, if one battery fails, the complete battery pack capability to provide a current is significantly reduced. To overcome this, one solution is to divide the battery pack into a few electrically connectable/disconnectable modules. If one cell fails, a single module of the battery pack is isolated, enabling the system to continue its operation. This is illustrated in Fig. 2.

Adjustable DC bus

Charger Inverter

Battery monitoring system

Battery monitoring system Battery monitoring system most suitable option. At constant price and constant losses, it is interesting to decrease the transistor size and increase the number of transistors, which in addition multiplies the physical dissipating points and decreases the losses on each point, making it easier to maintain the temperature of the system without any costly and heavy radiators. Multiplying the number of transistors also ensures redundancy, which increases security and shortens electrical connections (one does not need to concentrate the whole current in one transistor and redistribute it afterwards). Transistor multiplication can be implemented as represented on the upper module of Figure 2. This enables module division into several commutable columns and single module subpart isolation.

The number of serial cells to be placed in each module is mainly related to the voltages that the transistors can support. Transistor cost is proportional to its maximum voltage. It is thus suitable to divide the battery pack into many small voltage modules in order to improve the battery granularity. An optimum module voltage is around 25V, since below this limit, the individual transistor cost is no longer proportional to the voltage and becomes constant. 25V are equivalent to 4 or 5 serial battery cells depending on their maximum voltage. This module voltage is close to the standard maximum MOSFET gate-source voltage Vgs and hence easily drivable. Furthermore, small voltage modules enable to reduce voltage slew rates and corresponding electromagnetic perturbations during transistor transitions.

With the optimal module size in terms of transistor cost sorted, the system now requires a Battery Monitoring System (BMS) in each module (to ensure cell balancing inside each module), a charger and an inverter. The cell balancing circuit can be replaced by single-serial-cell modules, cell balancing thus proceeding by using more often the modules having higher States of Charge and less often the modules having lower States of Charge. With such modules, the output voltage can be adjusted by steps of a single battery cell voltage (3.2V/3.6V for Li-ion battery). By dynamically switching the modules, it becomes possible to produce a near-perfect sinusoidal waveform that can directly drive an electrical motor. As the output voltage step is lower than the resistive voltage drop in classical electric chains, one can also directly recharge the battery pack on the electrical network without any or with a very low size inductor. In conclusion, dynamic switching of the modules would enable to get rid of the costly inverter and its cooling system, the costly and heavy capacitor placed before the inverter to filter the current, the costly and heavy charger and the costly converter placed in the rapid charging stations. Furthermore, as the battery cell voltages tend to increase with the new battery technologies and the power transistor voltages tend to decrease with the transistors gate length reduction, there is a good chance that in the near future, optimal cost architectures will indeed tend towards modules of only one serial cell.

The objectives of this work are (i) to evaluate the feasibility of completely replacing the charger, the inverter and the battery cell balancing by module dynamic switching, (ii) to estimate the effects on the global structure and (iii) to identify the main challenges of implementing such a solution. Two prototypes have been fabricated to validate this principle, as described below.

FIRST PROTOTYPE DESIGN

Our first prototype is based on 18 modules having only one serial cell (s=1) and four parallel cells (p=4) as presented in figure 3: Each module has its own electronic circuit which aims at:  Sensing the current, voltage and temperature of each battery cell  Computing the state of charge and the state of health of each battery cell  Driving the power transistors through orders sent out by a master system  Sending the measurement data to the master system via an insulated serial bus  Ensuring security by opening the transistors in case of short circuit detection, too high temperature measurement or computational bug

The "Motor controller" block of the master system calculates in real time the number n of battery cells to place in series to reach the voltage required to drive a three-phase electric motor. The "Cell selector" block then selects n modules to use depending on their state of charge (for maintaining cell balancing), their current capability and their temperature. The motor controller works like multi-level inverters that are well-known in the state of the art [START_REF] Cortés | Model Predictive, Control of Multilevel Cascaded H-Bridge Inverters[END_REF][START_REF] Palanivel | Multicarrier Pulse Width Modulation Methods Based Three Phase Cascaded Multilevel Inverter Including Over Modulation and Low Modulation Indices[END_REF][START_REF] Karuppanan | FPGA based Cascaded Multilevel Pulse Width Modulation for Single Phase Inverter[END_REF]; figure 4 shows one output signal example that can be produced by such a device: The main advantages of these multilevel inverters are:

-Low switching losses due to the lower voltages to switch (losses proportional to f*C*V²) -Lower voltage transistors -Limited EMC emissions and improvement of the voltage signal applied to the motor (no full voltage commutation at high frequency like in a classical inverter) -Low size or no inductor to filter the current In our case, the output voltage quantification level is so low (few volts) that it is not required to carry out pulse width modulation between two voltage levels unless in a very low frequency range (< 5Hz). Furthermore, in many cases, no inductor needs to be added to filter the current (for example to recharge the battery pack from the electrical network).

Cell selector

The "Cell selector" block collects data from each module, such as their state of charge, state of health, temperature and remaining current capability, by using a serial insulated bus common to all modules. This data provides the ranking priority order of modules to connect or to disconnect.

This block is mainly aimed at:

 balancing the state of charge of modules  allowing the continuity of operation in case of faulty modules  continuing to exploit the weakened modules when the current value enables it  taking the modules' characteristics into account (all modules are not necessarily similar; they could be of different capacities or technologies)

Module

Each module embeds an electronic circuit powered by the energy available in the module and composed of:

 a micro-controller to decode orders from the master system, synthesize measurements, compute the states of charge, arbitrate the cells to use in the module, ensure security with extreme temperatures, currents and voltages  a serial communication interface to communicate with the master system: the estimation of the module states is performed within the module and transmitted to the master system, which then computes the future configuration of the module (connection or disconnection) and transmits the order to it  a measurement interface to sense the current, temperature and voltage of each cell  a discreet logic circuit to open the shunt or serial transistors in case of short circuit or transistor failure detection  a power driver interface to drive the 8 or 10 transistors of the module; this driver is able to command each transistor with 20V in less than 300ns

Power transistor driver

To drive the transistors, we have chosen to charge an inductor under 3.3V and to discharge it in the gate of the power NMOS to command. Figure 5 shows the corresponding electronic circuit and the experimental waveforms:

Imax

If the NMOS command has to be maintained on its "on" state for more than 1ms, then a new energy pulse is sent to the NMOS gate every 1ms.

The NMOS is open by opening T1 and closing T3 which discharges the NMOS gate.

PROTOTYPE IMPLEMENTATION

The first prototype is composed of an electronic card designed to be screwed on the battery cells; we aim for as few cables as possible. The power transistors are placed to make the power path as short as possible. Figure 6 shows the first 9 modules management card of our first prototype. The next 9 modules are not represented for clarity but are just in the extension of the first card.

EXPERIMENTAL RESULTS

The first prototype was tested on a low power 180 W ABB electric motor (for security reasons). As the prototype can only produce 1 phase, the other two were produced by using two four-quadrant converters driven by the same "Motor controller" block. Figure 7 shows the voltage and current waveforms applied to the electric motor. The efficiency of the electrical conversion is higher than 99% in low power demand (< 2 kW) and decreases to 98% when the current reaches 160 A (nominal regime -9.5 kW). This efficiency is significantly higher than that of standard embeddable inverters, which is no more than 90%, with moreover an induced high frequency voltage in the electrical motor. As a result, no cooling system is required to evacuate the losses, which limits the weight and cost.

The output inverter used to reverse the output voltage in our prototype does not induce many losses because it switches only twice per period and on 0V output voltage (at voltage inversion). Nevertheless, this inverter necessitates a lot of costly high voltage transistors. Furthermore, these transistors are dependent on the number of modules, which impedes the flexibility in terms of adding or removing of modules.

To overcome this limitation, we designed and fabricated a second prototype that duplicates the inverter on each module. The latter still works on a low voltage (<20 V) independently of the module number. This second prototype is presented in details in the following part.

Phase1 (our card) Phase2 Phase3

Curent of phase1

SECOND PROTOTYPE ARCHITECTURE

The objectives of the second prototype are:

 to place an output voltage inverter on each module in order to only switch low voltages (no need of high voltage transistors even in the presence of a large number of modules)  to enable a serial/parallel commutation of cells within the modules, according to 4 configurations: all parallel, parallel-serial-parallel, parallel-serial-serial, all serial.

The module architecture is presented in Fig. 8: This kind of module can be inserted in a serial arrangement to reach a large global output voltage and/or to store a large amount of energy. The configuration of each module is sorted in real time by a master system in order to provide the instantaneous required output voltage/current to drive, for example, an electric motor (Fig. 9).

SECOND PROTOTYPE IMPLEMENTATION

The second prototype was also designed to manage 40 A.h battery cells; a picture of this prototype is presented in figure 10. As the battery cells can be placed in parallel, the efficiency can significantly be increased in case of low output voltages compared to the first architecture; and much more compared to a standard inverter.

CONCLUSIONS

We have shown that it is possible to directly drive an electric motor by dynamically switching batteries with an efficiency up to 98%. Losses are very low and spatially distributed, which prevents the use of a cooling system. Although this solution requires numerous transistors, the latter operate in low voltage and hence are relatively cheap (less than 1$). Overall, this solution is thus relatively cost-efficient (around 300 € for transistors for 3 columns of 9.5 kWh) and the cost is bound to be significantly reduced through large scale production. This solution was validated with an electric motor; yet, it can also replace the normal and the fast battery charger without adding cost and weight to the vehicle. This solution can also significantly increase the security by withdrawing the failing cells, cutting the power path in many points in case of vehicle crash, flood, short circuit or simply during maintenance. The main remaining challenge is to increase the data bus speed for driving more modules and for producing an output wave at more than 500Hz (currently limited to 400 Hz).

Figure 2 :

 2 Figure 2 : Battery pack composed of connectable/disconnectable modulesThe current through the battery pack in electrical vehicles can exceed 100 A (depending on the continuous bus voltage and the embedded power level) and the transistor cost increases as a function of the current level that the transistor can withstand. As a result, a two-switch-per-module configuration may not be the

Figure 3 :

 3 Figure 3 : Schematic of the first prototypeThis prototype is designed to manage 18*4=72 battery cells of 40A.h, representing 9.5kWh (considering 3.3V battery cells). Chosen transistors are MOSFETs (IRFH5250DTR2PBF) and allow a continuous current of 40A without any thermal radiator. An inverter is placed at the output to invert or to reproduce the output voltage. This inverter is driven by the upper and the lower modules.

Figure 4 :

 4 Figure 4 : Output voltage example of a multilevel generator [1]

Figure 5 :

 5 Figure 5 : Power transistor driver principleWhen closing T1 and T2, the current in the inductor L increases as IL=UL/L*t; this is verified experimentally (i.e. Figure5: red curve). The final value of the energy stored in the inductor is ½*L*Imax². This energy is then transferred to the gate of the power NMOS by opening T2, and becomes ½*CGS*VGS². The energy stored in the inductor and then the gate voltage can be tuned by adjusting the inductor charging time. With a 47 nH inductor, the charging time to reach 20 V on the power NMOS gate is only about 300ns, which is negligible compared to the computation time of the motor controller.

Figure 5 :

 5 red curve). The final value of the energy stored in the inductor is ½*L*Imax². This energy is then transferred to the gate of the power NMOS by opening T2, and becomes ½*CGS*VGS². The energy stored in the inductor and then the gate voltage can be tuned by adjusting the inductor charging time. With a 47 nH inductor, the charging time to reach 20 V on the power NMOS gate is only about 300ns, which is negligible compared to the computation time of the motor controller.

Figure 6 :

 6 Figure 6 : Partial view of the first prototypeThe inverter transistors are rather large and in significant number because they have to support the full voltage and are more resistive compared to the module's transistors which are nearly invisible on the picture.

Figure 7 :

 7 Figure 7 : Generated voltage and current waveforms Figure 7 shows that the generated voltage is close to a pure sinus and that the current is particularly filtered by the inductance of the motor windings.

Figure 8 :

 8 Figure 8 : Second prototype architecture This module can produce a voltage from -4Vcell to +4Vcell per step of Vcell , where Vcell is the cell voltage.

Figure 9 :

 9 Figure 9 : Serial arrangement of modules of prototype 2

Figure 10 :

 10 Figure 10 : Picture of the second prototype switched module


Suzanne Kieffer
email: kieffer@isys.ucl.ac.be

Noëlle Carbonell
email: noelle.carbonell@loria.fr

How really effective are Multimodal Hints in enhancing Visual Target Spotting? Some evidence from a usability study

Keywords: H.5.2 [User Interfaces]: Ergonomics, Evaluation/Methodology, Graphical User Interfaces (GUI), Natural Language, Voice I/O. I.3.6 [Methodology and Techniques]: Interaction Techniques Performance, Design, Experimentation, Human Factors Visual search. Multimodal system messages. Speech and graphics. Usability study. Experimental evaluation. Visual target spotting

The main aim of the work presented here is to contribute to computer science advances in the multimodal usability area, in-as-much as it addresses one of the major issues relating to the generation of effective oral system messages: how to design messages which effectively help users to locate specific graphical objects in information visualisations? An experimental study was carried out to determine whether oral messages including coarse information on the locations of graphical objects on the current display may facilitate target detection tasks sufficiently for making it worth while to integrate such messages in GUIs. The display spatial layout varied in order to test the influence of visual presentation structure on the contribution of these messages to facilitating visual search on crowded displays. Finally, three levels of task difficulty were defined, based mainly on the target visual complexity and the number of distractors in the scene. The findings suggest that spatial information messages improve participants' visual search performances significantly; they are more appropriate to radial structures than to matrix, random and elleptic structures; and, they are particularly useful for performing difficult visual search tasks.

Context and motivation

In the 90s, numerous interactive information visualisation techniques were invented and commercialized, mainly in order to facilitate user access to graphical information and multimedia. Numerous visualisation techniques have been propounded, such as, zoom views, multiscale interfaces and hierarchical views (see, [START_REF] Shneiderman | The eyes have it: A task by data type taxonomy for information visualizations[END_REF] for a general overview of visualisation techniques and their use). Few ergonomic studies have been published about the effectiveness of visual search in such spatial organisations.

At the same time multimodal interaction appeared. Numerous forms of speech-based input multimodality have been proposed, implemented and tested. Combinations of speech with gestural modalities have been studied extensively, especially combinations of speech with modalities exploiting new input media, such as touch screens, pens, data gloves, haptic devices. Both usability and implementation issues have been considered; see, among others, [START_REF] Oviatt | Integration and Synchronisation of Input Modes during Multimodal Human-Computer Interaction[END_REF] on speech and pen for the first category of issues, [START_REF] Nigay | A design space for multimodal systems: concurrent processing and data fusion[END_REF] for the second category. Contrastingly, speech combined with text and graphics has only motivated a few studies. As an output modality, speech is mostly used either as a substitute for standard visual presentation modes (cf. phone services) or for supplementing deficiencies in visual exchange channels.

In spite of that, most information visualisations now combine both images and spoken language. One essential question remains. Do verbal messages provide a specific and significant contribution to human-computer interaction for visual search on complex displays? Information visualisations convey more indications to users than other media, even when combined [START_REF] Ware | Information Visualization. Perception for design[END_REF]. Consequently, it is not obvious that the combination of spoken information and visual information will improve and facilitate visual activites within information visualisations. Moreover, it is justifiable to ask whether such output multimodality will not result in a cognitive overload.

Current study objectives

The main aim of the work presented here is to contribute to computer science advances in the multimodal usability area, in-as-much as it addresses one of the major issues relating to the generation of effective oral system messages: how to design messages which effectively help users to locate specific graphical objects in information visualisations? An experimental study was carried out to determine whether oral messages including coarse information on the locations of graphical objects on the current display may facilitate target detection tasks sufficiently for making it worth while to integrate such messages in GUIs. In addition, the display spatial layout varied in order to test the influence of visual presentation structure on the contribution of these messages to facilitating visual search on crowded displays. Finally, three levels of task difficulty were defined, based mainly on the target visual complexity and the number of distractors in the scene [START_REF] Chelazzi | Serial attention mechanisms in visual search: a critical look at the evidence[END_REF].

Target detection was selected as the experimental task for the following reasons. First, it is one of the few human activites, besides reading, that have motivated a significant amount of psychological research, cf. [START_REF] Chelazzi | Serial attention mechanisms in visual search: a critical look at the evidence[END_REF]. Second, the design of numerous computer applications may benefit from a better knowledge of this activity such as: online help for current interactive application software, geographical applications and navigation systems in vehicles, visualisations of very large data sets, navigation in hierarchical views of personal data sets.

Methodology

Twenty-four volunteers participated to this study. This gender-balanced group of participants was composed of experienced computer users, ranged in age from 24 to 29 years and with normal eyesight (assessed using the Bioptor test kit). Thus, all participants were expert mouse users with alike quick motor reactions and they were experienced in visual search activities. Target selection time and spotting accuracy were likely to reflect visual search performance reliably, and task learning effects were prevented.

Procedure

First, participants were given an explanation of the research study. Prior to the experiment, they performed the Bioptor eye test, filled in demographic questionnaire and performed some training target selection tasks. Finally, participants performed the series of target mouse selections. Following the computer-based tasks, participants were asked to fill in second questionnaire and were debriefed.

Experimental Design

The usability study employed a 4x3x10 factorial design, with 2 modality conditions: 120 scenes were used in each modality condition, each scene including 30 photographs, organized along one out of four standard symmetrical structures (see figure 1). Three levels of task complexity were used (easy, difficult, and very difficult). Levels of task difficulty were distributed among the four structures. In short, the usability study employed 10 scenes by level of difficulty (3) and structure (4) in each modality condition.

Participants were asked to retrieve and select, with the mouse, as fast as they could, a previewed photograph in each scene according to 2 modality conditions: the VP condition (target visual presentation) and the MP condition (target multimodal presentation). In the VP condition, the isolated target was displayed in the centre of the screen during 3 seconds. In the MP condition, a short oral message containing information on the target location was played simultaneously with the target visual presentation.

Messages were composed of one or two short spatial phrases, for instance, "On the left (of the screen)" or "At the bottom (of the screen), on the right". Following target presentation, participants had to click on a button in the centre of the screen for launching the scene display. Thus, the position of the mouse at the beginning of the search was identical for all tasks. The order of modality conditions was counterbalanced between participants so as to neutralize possible task learning effects.

The dependent variables used to assess participant task performance were target selection time (from scene display onset until first mouse click), and accuracy (i.e., mouse click on the target vs elsewhere).

Results

Analyses Methodology

Analyses of variance (ANOVA) were used to examine the presence of significant differences in task performance, as measured by both selection times and error numbers, according to: (i) the target presentation mode; (ii) the scene structure; (iii) the task difficulty. If significant differences were revealed by the ANOVA procedure, then paired t-tests were also performed to further highlight any difference. The following analyses were computed over all subjects (24): selection times and errors per presentation mode; selection times per presentation mode and structure; selection times per presentation mode and task difficulty.

The results of the ANOVA procedure are presented in tables 1 and 2. Tables 3, 4 and5 present a summary of participants' performance analyses respectively per presentation mode, per presentation mode and structure, and, per presentation mode and task difficulty. In tables 4 and 5, values preceded by "-" or "+" are respectively inferior or superior to the corresponding average values per presentation mode reported in table 2.

Target presentation mode and task difficulty are highly significant factors

The results from table 1 show that target presentation mode, scene structure and task difficulty are significant factors. First, they validate our classification of scenes into three levels of difficulty (40 scenes per target presentation mode and level). Secondly, they suggest that target presentation mode and task difficulty have more influence on results than scene structure. In order to elicit the specific influence of scene structure on participants' selection times with and without spatial information messages -respectively, in the MP condition vs. in the VP condition-a complementary ANOVA procedure was computed. Results are reported in table 2 together with results concerning task difficulty.

Spatial information messages match radial structures best

Radial structure improves participants' visual search performances -speed and accuracyin the MP condition (table 4). Since scene structure is not a significant factor in the MP condition, no paired t-test was computed. However, compared to matrix, random and elliptic structures, the radial structure leads to the fastest and most accurate target detection. This result may be explained by the following reason: spatial information messages like "On the left (of the screen)" or "At the bottom (of the screen), on the right" are more appropriate to radial structures than matrix, elliptic and random structures. This interpretation is supported by some spontaneous comments collected during the debriefing interviews: the radial structure was preferred by 11 out of 24 participants, that is to say almost half of them.

In the VP condition, selection time differences between the radial and elliptic structures, the radial and matrix structures, the elliptic and matrix structures are statistically significant (respectively, t=3.64 p=0.0003, t=1.25 p=0.0024, t=2.18 p=0.0296). The four spatial structures can be ordered as follows according to increasing averaged selection times: radial, random, matrix, elliptic. These results are somewhat unexpected, since participants were experienced computer users, and the use of 2D arrays is currently prevailing for displaying pictures. Participants' subjective judgments were at variance with their performances: more than half expressed a marked preference for elliptic layouts compared to the other structures, and two thirds of them judged either the matrix or the radial structure the most inefficient layout. Participants' performances and subjective judgments concerning the matrix structure in the VP condition are in accordance with the results presented in [START_REF] Shneiderman | The eyes have it: A task by data type taxonomy for information visualizations[END_REF].

Influence of messages depends on the task complexity

Spatial information messages are particularly useful for performing difficult visual search tasks. Acually, a careful analysis of participants' performances shows that average selection times increase from level 1 (easy) to level 3 (very difficult) less rapidly in the MP condition (25%) than in the VP condition (35%) (See table 5). Therefore, it seems worth while to assist users in difficult visual search activities through spatial information messages. As such short oral messages will be well accepted by potential users, or so it seems according to participants' subjective judgments, their use for helping users to carry out easy visual search tasks may also be considered.

In both conditions, averaged selection times increase noticeably from level 1 to level 3.

For the VP condition, the difference between any pair of levels is statistically significant, the difference between levels 1 and 3 being highly significant (t=-6.40; p<.0001). For the MP condition, differences between level 1 and 3, and 2 and 3, are highly significant (t=-5.29; p<.0001 and t=-5.33; p<.0001 respectively), while the difference between levels 1 and 2 do not reach significance.

Conclusion and discussion

The usability study reported here aims at assessing the actual contribution of voice system messages to visual serach efficiency and comfort. The experimental task used was target detection which is representative of visual search tasks since it is commonly used in current GUI environments. Oral messages comprised one or two spatial phrases conveying coarse information on the target location on the display. Participants carried out visual search tasks in two conditions differing from each other in initial target presentation only: visual presentation of the target versus multimodal presentation, that is, visual presentation of the target simultaneously with oral indications on its location on the screen.

Spatial information messages improve participants' visual search performances significantly. In addition, regarding participants' speed and accuracy, they match radial structures best compared to matrix, random and elleptic structures. Finally, spatial information messages are particularly useful for performing difficult visual search tasks.

According to subjective judgments, oral messages were well accepted, and multimodal target presentations were preferred to visual presentations by a majority of participants.

Consequently, designers of graphical user interfaces might consider resorting to short oral messages including coarse spatial information for drawing users' attention to some displayed object. As such messages are likely to be well accepted by users, they may provide designers of advanced conversational user interfaces with a useful substitute "pointing" technique for any visual enhancement method in interaction contexts where gaze activity is intense and there is a risk of visual attention overload and eyestrain.

However, these empirical results need to be consolidated and further refined before reliable recommendations inferred from them can be proposed to designers. Indeed, the target detection task proposed to participants during the experiment remains a laboratory task, even though it is realistic. Our further research needs to focus on more elaborate tasks than locate and identify, such as compare, associate, distinguish, rank, cluster, correlate or categorize. In particular, a case study exhibiting effective and beneficial integration of multimodal messages to a domain-oriented interactive system would be extremely convincing for potential adopters who are left wondering what performance improvement would be achieved with the multimodal interface. These multimodal usability challenges are comparable with those expressed within the Information Visualization community [START_REF] Plaisant | The challenge of information visualization evaluation[END_REF].

Prospective applications of multimodal messages

The main difficulty for applying our spatial information multimodal messages to target detection within present GUIs lies in that the system is supposed to know exactly where the potential targets are located. Nevertheless, the following case study would offer the opportunity to successfully test the technical faisability of this new form of humancomputer interaction: combination of speech and visual presentation as an output style of multimodality. It deals with the valuation of the potential benefits from speech, as a graphical expression mode in a context of image retrieval [START_REF] Descampe | Coarse-to-fine textures retrieval in the JPEG 2000 compressed domain for fast browsing of large image databases[END_REF]. In their work, Descampe et al. use a coarse-to-fine classification process in order to retrieve similar textures in megaimages (JPEG 2000), that is, textures are similar to a sample entered by the user. Spatial information multimodal messages could be used to improve the whole retrieval process by helping the user to retrieve collections of similartextures within mega-images. The long-term view of this application is to take advantage from HCI and Signal Processing knowledge to design, implement and evaluate a multimodal user interface.

Figure 1 :

 1 Figure 1: Matrix, Random, Elliptic and Radial structures.

Table 1 .

 1 ANOVA Procedure. Factors: target presentation mode, scene structure, task difficulty.

	Factors	Selection times	Error numbers
	Presentation (i)	t=1202.98; p<.0001	t=23.18; p<.0001
	Structure (ii)	t=6.26; p=0.0003	t=2.58; p=0.05
	Difficulty (iii)	t=32.49; p<.0001	t=7.59; p=0.0005
	The results from table 2 reveal that spatial information messages suppress scene structure
	influence on participants' speed (MP: t=2.2602; p=0.0899). On the other hand, task
	difficulty still has an effect on participants' speed with spatial information messages as
	well as without any message (VP: t=22.72, p<.0001; MP: t=21.50, p<.0001).

Table 2 .

 2 Compementary ANOVA Procedure. Factors: scene structure, task difficulty.

	Factors	Presentation Selection times
	Structure	VP	t=4.89; p=0.0022
		MP	t=2.2602; p=0.0899
	Difficulty	VP	t=22.72; p<.0001

them (18), target spotting was easier (less hesitations) in the MP condition than in the VP condition. Most participants mentioned that they had experienced some strain and visual fatigue during the VP condition whereas they had felt perfectly comfortable during the MP condition. All participants considered that oral messages including coarse information on target location could provide efficient support to visual search activities, and two thirds (16) expressed a marked preference for the MP condition.

Table 3 .

 3 Participants' selection times and errors per target presentation mode.

	Presentation	Avg ST	Std Dev	Nb	%	Nb Obs
	mode	ms	ms	Errors	Errors
	VP	5674	5985	150	5.2%	2880
	MP	1747	1552	79	2.7%	2880

Table 4 .

 4 Participants' selection times and errors per structure.

	Structure Presentation Avg ST ms Std Dev ms
	Radial	VP	-5081	-5565
		MP	-1640	-1256
	Random	VP	-5626	-5819
		MP	-1737	-1437
	Matrix	VP	+5738	-5879
		MP	+1763	+1819
	Elliptic	VP	+6250	+6585
		MP	+1851	+1633

Table 5 .

 5 Participants' selection times and errors per target presentation mode and task difficulty.

	Difficulty	Presentation Avg ST	Std Dev ms Actual	%
			ms		Errors	Errors
	Easy	VP	-4919	-5011	17	18%
		MP	-1611	-1387	10	20%
	Difficult	VP	-5439	-5879	33	34%
		MP	-1620	-1272	15	31%
	Very	VP	+6663	+6801	46	48%
	difficult	MP	+2012	+1893	24	49%


D Jeambrun

Y Moreau

J L Costaz

J P Tourret

P Jouanna

G Lecoy

Simulation of the propagation of a cylindrical shear wave : non-linear and dissipative modelling

The simulation of a wave propagation caused by seismic stimulation allows to study the behaviour of the environment and to evaluate the consequences. The model involves the wave equation with a hysteresis loop in the stress-strain relationship. This induces non-linearities and, at the vertices of the loop, non-differentiable mathematical operators. This paper offers a numerical process which works out this simulation.

Introduction

Protection against earthquakes is an increasingly important objective for civil engineering, especially for those buildings in areas of high seismic risk [START_REF] Plested | In situ investigation of shear waves in soil media[END_REF] [START_REF] Mabssout | Introduction à la reconnaissance dynamique de sols in situ par système harmonique sismique[END_REF]. Of course, the problem is to know how a structure is going to respond under earthquake loading. But, before seismic motion reaches the structure, it must pass through the various soil layers underlying the site. Depending on the behaviour of the soils, the structure may be quite safe, or doomed to collapse. For this reason, the engineers look with interest upon the problem of soil response under dynamic loading.

The soil characterisation needs an in situ experimental device. The elaboration of the device requires a theoretical study before it is manufactured, and some parameters seem hard to evaluate directly from the measures. This is especially the case when considering the thermal dissipation. A link between experimental results and simulation results should allow us to identify the values of soil parameters. Furthermore, an experimentation with a single well for both stimulation and response may be designed [START_REF] Jeambrun | Propagation d'une onde de cisaillement en milieu non-linéaire dissipatif[END_REF], much cheaper than the usual cross holes system.

The simulation is based on the propagation of cylindrical shear waves. This configuration corresponds to the experimental system project. Several simulations have been performed through linear models which consider the soil as elastic. A linear model requires the determination of the shear modulus G 0 as well as the soil density  to compute the propagation of the shear wave. In the real case of important seismic stress, elastic behaviours can no longer be assumed.

The first improvement is to consider a non-linear phenomenon, introducing an internal damping parameter  (saturation effect of the stress for the high level of strain) added to the geometrical damping (dispersal of the energy density when the distance from the source is increased; obviously due to the volume augmentation in correspondence with the distance from the source). This is described by a pure non-linear model. Moreover, when the strain is important, a part of mechanical energy is converted thermal energy. This induces a delay on the propagation signals. In one space dimension, this can be modelled by a hysteresis cycle between the stress and the strain (Figure 1). Simple non-linear modelling enables to adjust parameters, used with dissipative models. The difficulty is to obtain the computation stability and convergence in spite of acceleration discontinuities. These discontinuities follow the non-differentiable mathematical operators due to the elasto-plastic nature of the non-linear dissipative model [START_REF] Achenbach | Wave propagation in elastic solids[END_REF].

Configuration, Assumptions and Models

Figure 2 shows, schematically, the experimental device (designed for Electricité de France [START_REF] Jeambrun | Propagation d'une onde de cisaillement en milieu non-linéaire dissipatif[END_REF]). One expects this device will allow the measures of the displacement and force at the well edge (A). The experimental system might also give the soil accelerations in the well vicinity (B and C).

The soil is assumed to be homogenous and isotropic. The probe is assumed to have an infinite length. Therefore, the wave propagation can be considered as radial and the displacement purely vertical, enabling the use of one-dimensional model.

The macroscopic soil models are based on the momentum balance relationship. In one radial dimension, it is described by the equation ( 1) where (r,t) is the vertical shear stress in the soil, w(r,t) is the vertical material displacement of the soil, r is the horizontal distance from the axis, t is the time and  is the specific mass of the soil. To complete the models, behaviour laws are added to this equation. The simplest is the elastic behaviour (2) where (r,t) is the vertical shear strain of the soil and G 0 is the shear wave modulus at low strain. This behaviour is, in general, valid for a strain less than 10 -5 and allows the computation with the well-know analytic solutions [START_REF] Hardin | Shear Modulus and Damping in Soils: Design Equations and Curves[END_REF].

0 ) , ( ) , ( ) , ( 2 2    t t r w r t r r t r        (1) ) , ( ) , ( 0 t r G t r    (2)
) , ( 1) , ( ) , (

0 t r t r G t r       (3)      ) ( ) , ( 2
) ( 2 ) ( ) , ( ) ( 2 ) , ( 0 0 0 0 0 r t r r r t r r G t r              (4)
For a higher strain, and before irreversible effects (which occur when the strain is approximately greater than 10 -4 ), the damping phenomenon must be taken into account. The nonlinear behaviour law of Hardin and Drewich [START_REF] Hardin | Shear Modulus and Damping in Soils: Design Equations and Curves[END_REF][START_REF] Engquist | Absorbing Boundary Conditions for the Numerical Simulation of Waves[END_REF] (3) (drawn with thin lines in Figure 1), where  is the damping parameter, has this particularity. Moreover, to introduce the dissipative effect, a nonlinear and hysteresis cycle behaviour law (4) [START_REF] Mabssout | Introduction à la reconnaissance dynamique de sols in situ par système harmonique sismique[END_REF] (drawn with thick lines in Figure 1) can be used. The loop surface is proportional to the thermal energy dispersed in soil. The law (4) depends on two soil parameters: the linear parameter G 0 and the dissipative parameter . A  coefficient makes possible the loading when  = 1 and unloading when  = -1.  0 and  0 are the values of the vertex of the hysteresis loop (seen in Figure 1). For the determination of the last three parameters ,  0 and  0 we use a pre-computation with the law (3). The law (4) induces discontinuities at the times when the stress reaches the vertices of the hysteresis loop [START_REF] Achenbach | Wave propagation in elastic solids[END_REF].

Method of Integration, Stability and Convergence

To achieve the computation, conditions must be found on the space domain boundary. The displacement w at the well edge is the system input. Far away from the well, an absorbing boundary condition (5) [START_REF] Barhe | Stability and accuracy analysis of direct integration methods[END_REF] is used, where r b is the distance from the well axis and c s is the shear wave celerity. The boundary must be in the elastic behaviour area, in this study the boundary was taken at 10 m. The non-linearity and the dissipative effect are major near the well. The space steps must be small in this zone, but the computation must be made far from the well to be allowed to use the absorbing boundary equation [START_REF] Hardin | Shear Modulus and Damping in Soils: Design Equations and Curves[END_REF]. Therefore, the finite difference method with variable steps is used for space. With this method, space steps can be thin in the well vicinity, whereas, far from the well, the space steps can be larger. Here the space steps are equal to 0.1 mm at the well edge with a 5% progression at every step.

The time evolution is insured by a Newmark-Wilson method [START_REF] Newmark | A method of computation for structural dynamics[END_REF] [10]. This method allows the stabilisation of the computation when the Wilson parameter is greater than one. The simplest methods tested are unstable with this non-linear dissipative model. The instabilities are caused by the discontinuities of the acceleration.

At each time step, the convergence of the non-linear equations is achieved with the help of the classic Newton-Raphson process. The Newton-Raphson iteration itself requires successive solutions of linear tri-diagonal systems.

The Newmark parameters a and b are taken such as a = b = 1. Although this condition is more restrictive than the stability criterion in the linear case, we didn't note the overall stability in the cases we tried. The Wilson parameter  behaves as a low pass frequency filter. Therefore, when the space steps are small enough, this parameter ensures the stability in the well vicinity in spite of the acceleration discontinuities. The display of the acceleration close to the well allows the checking of the stability.

When the non-linear parameter  is equal to zero, the comparison with the linear analytic computation makes errors less than 1%. The non-linear model is computed in the first half of the periods, the non-linear dissipative model is used in the last half of the periods.

Results and Discussion

We Figure 3 shows the acceleration at the well edge (r = 0.1 m) and in the well vicinity at r = 0.2 m, r = 0.3 m and r = 0.5 m from the axis. The discontinuities can be seen in the last three distances. The harmonic modules versus the distance from the well edge are shown in figure 4. The even harmonics are negligible. One sees that the harmonics are generated in the first 20 cm. The figures 5 and 6 show hysteretic cycles respectively at different distances from the well axis when  = 2000 and for some other values of . This shows the decrease of the loop's surface with respect to the distance from the source and with respect to the  decrease.

To have a fine knowledge of the mechanic properties of soils, civil engineers need to identify the parameters G 0 and . The elasticity parameter G 0 has a well-known influence on the measurable variables. As soon as the stress is strong enough and before the soil ruptures, previous works [START_REF] Plested | In situ investigation of shear waves in soil media[END_REF] and this paper show that the plasticity parameter  also influences these variables. It can be checked by the generation of the odd harmonics on accelerations within the non-linear area close to the well versus . Thus, two kinds of methods could be applied for this parameter identification with in situ measures. On the one hand, both parameters can be found by a disconsolation process such as the non-linear chi-squared minimisation. On the other hand, the G 0 parameter can be worked out with low stress measures where the influence of  is negligible. When G 0 is known,  can be identified with the help of the computed odd harmonic accelerations.

Conclusions

The computer code based on pure non-linear elastic and non-linear dissipative models using adapted algorithms, time integration of Newmark-Wilson, spatial mesh with variable steps and a Newton-Raphson process for the non-linear computation, overcomes the stability difficulties related to acceleration discontinuities present in the case of high seismic stress. The influence of the dissipative model has been shown on the acceleration spectrum and on the hysteresis loop. These simulations would allow the identification of soil parameters by comparison with in situ measures.

Figure 2 :

 2 Figure 1 : stress-strain diagrams

Figure 3 : 9 Figure 4 :

 394 Figure 3 : Acceleration vs. distance

Figure 5 :

 5 Figure 5 : Hysteretic loop vs. various distances

Figure 6 :

 6 Figure 6 : Hysteretic loop for various 


Daniel Gepner
email: daniel.gepner@loria.fr

Jérôme Simonin
email: jerome.simonin@loria.fr

Noëlle Carbonell
email: noelle.carbonell@loria.fr

Gaze as a Supplementary Modality for Interacting with Ambient Intelligence Environments

Keywords: gaze tracking, gaze pointing, gaze-contingent displays, gaze and speech human-computer interaction, multimodal human-computer interaction, ambient intelligence

We present our current research on the implementation of gaze as an efficient and usable pointing modality supplementary to speech, for interacting with augmented objects in our daily environment or large displays, especially immersive virtual reality environments, such as reality centres and caves. We are also addressing issues relating to the use of gaze as the main interaction input modality. We have designed and developed two operational user interfaces: one for providing motor-disabled users with easy gaze-based access to map applications and graphical software; the other for iteratively testing and improving the usability of gaze-contingent displays.

Context and motivation

Thanks to recent advances in gaze tracking techniques, it is now possible to consider the integration of gaze as an input modality into user interfaces. Eye trackers have become much less intrusive while maintaining satisfactory accuracy (0.5° visual angle). Users have not any more to keep their head and body perfectly still like in the nineties; they can move them with sufficient amplitude to feel comfortable while sitting at desktop PCs. In addition, remote eye trackers can be embedded in screen frames and coupled with standard video cameras; instant vision-based detection of the user's actual gaze direction, head position and orientation is used to correct raw point of gaze measurements and to adjust the angle of the eye tracking infrared camera(s). As for head-mounted eye trackers, which place no constraint on people's movements and mobility, they have become much less intrusive: their size and weight have decreased so much that they can be attached to spectacles. Finally, calibration has become much simpler and quicker. So, it is reasonable to expect that, in a few years, user gaze direction will be reliably detected and followed through "invisible" wearable devices. Vision-based gaze detection and tracking, a truly non intrusive approach, appears as an appealing solution in the long term. However, considerable research efforts and scientific advances are still needed to achieve appropriate accuracy despite steady significant advances [START_REF] Beymer | Eye Gaze Tracking Using an Active Stereo Head[END_REF], [START_REF] Kang | Vision-based Facial and Eye Gaze Tracking[END_REF].

While technical feasibility is in the process of being achieved, usability issues are yet to be further investigated. Gaze is obviously a useful human-computer interaction (HCI) modality for computer users with severe motor impairments [START_REF] Bates | Towards eye based virtual environment interaction for users with high-level motor disabilities[END_REF], or for users who are engaged in demanding manual activities, such as pilots in cockpits; see the early study of gaze used to designate objects referred to in speech commands [START_REF] Hatfield | Principles and guidelines for the design of eye/voice interaction dialogs[END_REF].

Contrastingly, the utility and usability of gaze as a supplementary input modality for able-bodied users has motivated much fewer studies and remains to be demonstrated. Gaze cannot of course supersede the mouse for pointing at, and selecting, graphical objects displayed on the screen of standard desktop PCs. As a pointing modality it can neither outmatch finger designation on a touch screen in efficiency and usability. However, its superiority to hand/arm gestures for remote pointing at large displays (e.g., "walls", reality centres or caves) is beyond doubt. Gaze pointing is faster, more precise and less tiring than gestures; hands are free and can be used to carry out other tasks, which may prove to be an invaluable advantage in some interaction contexts; in addition, it is more natural to use. Gaze, coupled with speech for expressing depth, can designate positions in a 3D virtual space more precisely and easily than 3D gestures; in particular, users have to mentally apply a homothetic transformation to their hand gestures and body moves to anticipate the actual span of these movements in a 3D virtual world, while planning them. Gaze pointing complemented with spoken phrases including action verbs is nevertheless inappropriate for manipulating virtual objects (e.g., on a Workbench screen), especially for shaping and deforming them. Therefore, gaze coupled with speech seems well suited to immersive virtual world exploration, but ill suited to virtual 3D object manipulation, as this form of input multimodality can be used for moving virtual objects but not for transforming them.

Research on the implementation of such concepts as 'Ambient Intelligence' and 'Ubiquitous/Pervasive Computing' is fast developing in the HCI scientific community. This evolution is likely to stimulate research on the integration of gaze into user interfaces as a remote pointing input modality. Ambient Intelligence (AmI), which subsumes Ubiquitous and Pervasive Computing, is a vision where humans are surrounded by computing and networking technology unobtrusively embedded in their surroundings. AmI lays emphasis on user friendliness, efficient and distributed services support, user empowerment, and advanced support to human interactions. This vision assumes a shift away from PCs to a variety of interactive "intelligent" agents which are unobtrusively embedded in our environment; see [START_REF] Carbonell | Ambient Multimodality: towards Advancing Computer Accessibility and Assisted Living[END_REF] for information on recent research advances on the implementation of AmI principles. Definitions of Ubiquitous and Pervasive Computing, [19], and [START_REF] Want | Guest Editors' Introduction: Energy Harvesting and Conservation[END_REF], also put emphasis on "the integration of computation into our daily work practice to support our activities without being noticed" or even "visible"; which implies distributing interaction facilities in the environment and embedding them in everyday objects. So, future users of AmI and Ubiquitous/Pervasive Computing environments will be naturally induced to perform computer tasks while engaged in other non computing activities, at home or on the move. These parallel activities will sensibly reduce the perceptual and motor resources they will be able to invest in computing tasks. So, they will be confronted with similar computer access and interaction difficulties to those encountered by people with disabilities. Therefore, advances in research on Ambient Intelligence and progress in implementing this concept in software agents meant to support the general public in their daily activities are likely to significantly contribute to advancing the inclusion of people with disabilities in the fast developing Information Society, mainly by increasing computer access flexibility. They will also contribute to simplifying and unifying user-centred interface design, in-as-much as user interfaces meant to satisfy the needs and expectations of able-bodied users in a wide range of contexts of use will be easy to tailor to suit the specific needs of various communities of users with physical disabilities. The development of AmI in society will stimulate the progress of Universal Access.

These observations explain our interest for speech and gaze as a substitute multimodal command "language" for direct manipulation in some contexts of use, such as interaction with AmI artefacts or immersive virtual reality environments.

Our work on speech and gaze as a promising multimodal command "language" for interacting with augmented objects and large displays is presented in the following sections. We first specify our objectives. Then, the software tools that make up our experimental platform are described. Present research progress and outcomes are presented next. Future research directions are sketched in the conclusion.

Scientific objectives

Our main objective is to design and implement an appropriate speech-and gaze-based command language for interacting with large displays. Design requirements include the following major constraints. This multimodal language should be:

• robust, that is, tolerant to ambient noise and atypical or faulty utterances,

• flexible to changes of context of use and inter-individual expression differences;

• effective and efficient, and capable of outmatching existing interaction facilities;

• usable: in particular, easy to use and to learn (i.e., transparent). To be usable, gaze pointing, as an input modality supplementary to speech, should be implemented in compliance with the following specific requirements:

•

The role of gaze should be limited to solving deictic and underspecified reference phrases included in speech commands. For instance, the pronouns "that" and "there" in Bolt's famous example "Put that there." are deictics which refer to objects or positions on the user's screen [START_REF] Bolt | Put-that-there: Voice and gesture at the graphics interface[END_REF].

•

Interpretation of deictic phrases should be mainly inferred from the analysis of gaze fixation positions on the display. Minimum constraints should be placed on users' eye movements. Of course, inference of referents matching deictics from spontaneous eye movements would be an ideal solution; it represents our ultimate goal. But present knowledge and modelling of gaze activity during humancomputer tasks is insufficient for interpreting spontaneous gaze movements accurately and reliably. Therefore, at least for the time being, controlled gaze pointing is mandatory; see section 4 for a presentation of our current work on this issue. Nevertheless, implementation of this constraint should interfere as little as possible with users' spontaneous visual exploration strategies. So, users might be asked to "look once at the objects and/or locations on the display referred to ambiguously in the current command"; yet, they should be free to choose when to look at referents (i.e., before, during and/or after the current verbal command), and for how long. In particular, dwells which have to last between 600 ms and 1 sec. to be reliably recognized in typing tasks on virtual keyboards [START_REF] Marajanta | Twenty years of eye typing: systems and design issues[END_REF] ought not to be required from users. Research on speech-and gaze-based interaction is still in its early stages. Few studies have been published yet on the implementation and evaluation of gaze as a pointing modality: see, for instance, [START_REF] Tanriverdi | Interacting with Eye Movements in Virtual Environments[END_REF] and [START_REF] Cournia | Gaze vs. Hand-Based Pointing in Virtual Environments[END_REF] for an assessment of pointing by gaze compared to standard pointing techniques, [START_REF] Oyekoya | Eye Tracking as a New Interface for Image Retrieval[END_REF] for the use of this modality in pervasive computing environments, and [START_REF] Smith | Use of Eye Movements for Video Game Control[END_REF] for its use in games. Multimodal interaction involving gaze has motivated fewer published studies; see, for instance, [START_REF] Zhang | A Gaze and Speech Multimodal Interface[END_REF] on mutual disambiguation in gaze-and speech-based interaction.

One of the main difficulties is the well known "Midas touch" effect [START_REF] Jacob | Eye tracking in advanced interface design[END_REF] which stems from the double origin of eye movement control: depending on the context and activity, gaze direction is either controlled by low-level perception mechanisms activated by the visual properties of objects in the environment (bottom-up control) or by intentional top-down processes. Fixations resulting from the latter processes are those which have the potential to yield information on the user's current intention. A way to avoid the Midas touch effect in gaze-based interaction is to avoid providing users with a visual feedback to their eye movements.

Our work on gaze as a supplementary input modality to speech is grounded on a main working hypothesis which is essential to obtain reliable and robust gaze and speech controlled user interfaces. We expect that users will behave as follows when using speech for controlling graphical software applications. There is some chance that they will glance at the graphical object on which the current oral command is designed to operate while planning the action they intend to perform on this object, and/or during the utterance of the command itself, and there is every chance that they will look at it once they have finished speaking, to check whether its execution by the system meets their expectations. If this assumption proves to be valid, it will be possible to distinguish between fixations driven by visual stimuli and those controlled by intents. Analysis of fixations occurring just before, somewhat after, and during the utterance of speech commands, is then likely to provide useful information on the object or action specified in the command.

Software tools

Numerous advanced software tools are necessary for studying the potential of gaze as a pointing modality. We have developed a software platform comprising a set of components with the following functionalities:

• Robust real time computation of gaze fixations using a dispersion-based algorithm. This component is necessary for implementing gaze as an input modality. See [START_REF] Salvucci | Identifying fixations and saccades in Eye-tracking protocols[END_REF] for descriptions of the main fixation computation algorithms and comparisons of their performances. We use a head mounted eye tracker (ASL-501) at two sampling rates, 60 Hz or 240 Hz. Results of real time fixation computation are identical to those of the ASL Eyenal batch processing software.

•

Recording of rich traces of user interactions with any Windows application software. Time stamped traces include:

system and user generated events; -screen displays (from 4 to 8 screen copies per second) and mouse phantom (or cursor) positions on the screen; -coordinates of the user's gaze points on the screen, and speech from both user and system; plug-ins can be added to process other modalities.

•

Synchronised play back of multimodal interactions from files of recorded traces. Eye fixations appear superimposed on the displays; optional visual markers display qualitative information on pupil diameter, fixation duration and size (i.e., spatial dispersion of samples included in each fixation).

•

Assistance to manual annotation of displayed traces so as to facilitate their qualitative and quantitative analysis. Main annotation facilities include:

-Manual or semi-automatic segmentation and labelling of visual and audio traces so as to define meaningful sequences of interactions. For instance, replacing all occurrences of a phrase in a text by another phrase represents a meaningful sequence of interactions with a word processor. -Manual or automatic definition of interest zones on the displays so as to determine what graphical objects are most frequently looked at by users during the execution of a given task.

•

Implementation of the Wizard of Oz paradigm, an efficient rapid prototyping technique for simulating functionalities of innovative user interfaces which aim at reproducing human expertises too costly or difficult to implement appropriately, such as natural language understanding. Traces of users' interactions are simultaneously recorded on the user PC and sent to the wizard PC via local network. It is possible to filter the data sent to the wizard to avoid overcrowding his/her screen. Wizard-to-user information is also conveyed through the local network. An application dependent plug-in meant to assist the wizard in simulation of the user interface can be added to the software component running on the wizard's PC. A version of this distributed platform has been developed under Linux with the aim of obtaining appropriate observation tools for studying gaze-and speech-based interaction with 3D virtual environments. The overall functionalities of this platform are identical to those of the Windows tools described above. Main specific features originate from the chosen 3D development environment, a multi-agent system (oRis, ENIB, France) which represents each 3D object in the animated scene as an agent. This feature impacts on the platform functionalities as follows:

• Implementation of the Wizard of Oz technique: 3D animations generated by oRis run on both workstations; this strategy makes it possible to limit wizard-to-user data exchanges to the oRis commands necessary for simulating the execution of the user's multimodal requests, and thus, to easily obtain smooth evolution of the animation on the user's screen.

•

The play back includes the names of the objects gazed at by the user during fixations, since agents can be labelled. In addition, the play back displays pupil diameter evolution, phonetic and orthographic transcripts of the user's oral commands together with a graphical representation of the speech signal. Transcripts are either done by human experts or generated by acoustic-phonetic decoders and speech recognition systems.

Eye tracking contribution to HCI research is twofold. First, data on eye movements are most useful and are increasingly used for assessing the ergonomic quality of advanced graphical user interfaces and interactive visualisations of very large data sets. We have used the software platform presented here for eliciting the possible influence of display spatial layout on the efficiency and comfort of visual search [START_REF] Simonin | Effects of display layout on gaze activity during visual search[END_REF].

Secondly, eye tracking data analysis is quite obviously an essential source of knowledge for implementing gaze as a pointing modality. The next section presents our scientific contribution to this emerging research area, and demonstrates the usefulness of the software tools presented above.

Gaze, a promising pointing modality

Our work on the implementation of gaze as an appropriate substitute modality for mouse pointing and selection in contexts where users interact with large displays, such as image walls or immersive virtual reality environments, is currently developing in three directions which are detailed next: -prototyping of gaze controlled map-based applications, -development of flexible gaze-contingent displays, -design and development of "natural", gaze and speech command languages.

Gaze control of map-based applications

We have developed a prototype gaze controlled user interface for interacting with standard map applications. All the interaction facilities provided by applications on the Web, such as Mappy, can be activated using gaze sampled at a 60 Hz rate. This interface is presented in Figure 1. Looking at one of the four buttons surrounding the zoom-in window (on the right of the screen) moves the focus of the zoom to the right (left, top or bottom) by a fixed step the length of which can be adjusted by the user. The zoom-in factor can be specified using the seven buttons on the top left of the screen. We are currently considering implementation of continuous zoom-in.

Looking at a point on the overall map (at the bottom of the screen on the left) causes the current content of the zoom-in window to be replaced by a portion of the map the contours of which are represented on the overall map by a black rectangular line; the centre of this rectangle coincides with the averaged positions of the samples included in the fixation; its size varies according to the zoom-in factor. To avoid the Midas touch effect without constraining the user's eye movements, we have proceeded like this: (i) when a fixation on the overall map is detected, the system waits for about 170 ms from the start of the fixation; then, (ii) it displays a blue circle around the averaged x, y coordinates of the 10 first samples in the fixation to warn the user that it will change the content of the zoom-in window about 200 ms later; (iii) if the user goes on fixating the same position on the overall map for about 200 ms (i.e., 12 samples), the system replaces the content of the zoom-in window with the portion of the overall map surrounding the averaged coordinates of the 22 first samples in the fixation. These thresholds have been determined from the data collected in [START_REF] Simonin | Effects of display layout on gaze activity during visual search[END_REF].

This prototype which has been developed to test the usability of gaze pointing as an input modality has been evaluated by a few users who have judged it intuitive and comfortable to use. Potential applications include accessible graphical software for motor disabled people, and interaction with large graphical displays, for instance, digital wall maps in lecture rooms.

Gaze-contingent displays

A gaze-contingent display is a display that responds to where an observer looks within it. The most informative details are generated at the current point of gaze, and the display resolution is degraded at the periphery of the foveal or parafoveal field of vision. Research on the implementation of gaze-contingent displays, an emerging area, is fast developing; see [START_REF] Duchowski | Gaze-Contingent Displays: a Review[END_REF] for a review. Potential application areas include:

• Interaction with complex distant visualisations or animations in cases when image compression techniques are insufficient for achieving satisfactory display speed and reactivity; for instance, remote surgery through Internet is a promising application area.

•

Interaction with virtual reality applications, especially in immersive environments such as reality centres or caves, where gaze represents a more efficient and usable pointing modality than hand or finger gestures.

We have developed a robust gaze-contingent prototype which functions as a gazecontrolled lens: the display is blurred outside a circle the centre of which coincides with the averaged coordinates of the 4 first gaze samples (i.e., 70 ms or so) included in the current fixation. See Figure 2. The display resolution is highest inside this circle the diameter of which corresponds to the standard size of the foveal field of vision, that is, 5° visual angle. This prototype is operational, and its accuracy has been successfully tested by 5 participants who used it for exploring complex realistic displays. However, participants complained that the "lens" followed their eye movements too slowly, which obliged them to sensibly reduce their natural pace of visual exploration in an irritating way. A 70 ms lag between fixation start and "lens" move is indeed too long, but it is necessary for achieving reliable fixation detection. The decision of restricting the high resolution area on the display to the span of the foveal vision field was motivated by the will to test the actual accuracy of our prototype. Obviously, this area should be extended so as to also include the parafoveal vision field, which might reduce users' frustrations significantly. The same motive explains the absence of smooth resolution degradation. The study described in [START_REF] Ashmore | Efficient Eye Pointing with a Fisheye Lens[END_REF] differs from ours in that the gaze-contingent lens creates a fisheye deformation, and the task amounts to target search and selection in a grid of 9x9 boxes.

We are currently investigating two research directions in parallel so as to overcome the main usability weakness of contingent displays, that is, their insufficient reactivity by: (i) increasing the sampling frequency rate, from 60 Hz to 240 Hz, with a view to shortening fixation detection time; (ii) achieving a satisfactory trade-off between accuracy and time delay by analysing saccade velocity variations to predict next fixation approximate landing. An approach combining both strategies will also be tested. Finally, to further improve the usability of gaze-contingent displays, we shall test whether smooth degradation of resolution is actually more comfortable visually than stepwise reduction.

Gaze, a supplementary modality to speech

We have collected a corpus of realistic speech and gaze interactions with various 3D environments to acquire the necessary knowledge for implementing a "natural" style of input multimodality, using the Wizard of Oz technique for interpreting users' eye movements and oral commands. See Figure 3. 5 participants performed 5 scenarios lasting 4 min. each. They carried out two types of interactive tasks, manipulation of virtual objects and interaction with a reactive game. They had to comply with a few expression constraints only:

the vocabulary they could use was restricted to a predefined list of words; -during the last scenario only, they had to designate each object they had to move using a deictic and "looking at" the object they wanted to move, then at the new position it should occupy; during the previous 4 scenarios they could move their eyes freely. Participants' speech commands have been segmented into words and phones by an expert phonetician; time-stamped orthographic and phonetic transcripts of the speech corpus are now available. Gaze fixations are currently analysed to determine which ones correspond to moments when the user's visual attention is focused on the 3D object that the current 1 command aims to act upon. Then, various algorithms will be designed to filter controlled and spontaneous eye movements with a view to identifying fixations on task objects alluded to in the current command, especially by underspecified reference phrases, thus contributing to reduce the ambiguity and imprecision of speech commands. These algorithms will be trained, tested and evaluated on the corpus of multimodal interactions presented in this section.

Conclusion

We have presented our work on the implementation of gaze as an appropriate input modality for pointing at virtual interactive objects in various contexts of use: user interfaces with graphical applications for people with motor disabilities, gazecontingent displays, especially on large screens and multimodal, speech-and gazedbased interaction with immersive virtual environments and augmented reality interfaces. In the short term, our research on the possible contribution of gaze to enriching HCI will be focused on the following goals: -Improve gaze tracking accuracy by taking account of pursuit movements the occurrence of which is very frequent during interaction with animated scenes. -Increase the reactivity of gaze-contingent displays by experimenting with various approaches. -Assess the effective contribution of spontaneous or loosely controlled gaze to disambiguating imprecise references in speech commands, interpreting deictic phrases, and improving speech recognition accuracy (see [START_REF] Oviatt | Mutual disambiguation of recognition errors in a multimodal architecture[END_REF]).

Fig. 1 .

 1 Fig. 1. Gaze-controlled user interface of the map-based application. Control buttons are in grey.

Fig. 2 . 3 .

 23 Fig. 2. Play back: gaze-controlled lens. Three Fig. 3. Play back: Wizard of Oz simulation of fixations are visible; the one on the right is starting. gaze-and speech-based interaction.

The oral command may be in progress or forthcoming or just completed.


Nadine Herry-Benit

Roussi Nikolov

Anne Tortel

POSITIONAL DETERMINATION OF THE QUALITY OF SCHWA IN ENGLISH

Keywords: acoustic phonetics, schwa vowel, context-driven quality

The prototypical articulatory-acoustic properties of schwa being normally thought of as definitional, this vowel sound is typically described in terms of tongue position as a mid-central, neutral vowel sound. At the same time, instrumental analysis shows that schwa in English is context-driven to a much larger extent than any other vowel; articulatorily, it is in fact "inherently unspecified for tongue position", and acoustically, a comparison of the experimental data with this first approximation model does not show a good agreement, hence the interest of a systematic contextual study of schwa.

The prototypical articulatory-acoustic properties of schwa being normally thought of as definitional, this vowel sound (occurring in unstressed syllables in English) is typically described in terms of tongue position as a mid-central, neutral vowel sound, corresponding to "the single uniform tube acoustic model of the vocal tract" (Rosner & Pickering 1994: 34). At the same time, instrumental analysis shows that schwa in English is context-driven -much more than any other vowel -, to the extent of being considered by some, articulatorily, as "inherently unspecified for tongue position" (Bates 1995: 266-267). Not surprisingly, therefore, the experimental acoustic data are hardly in line with such a simple first approximation assumption of constant phonetic features, hence the interest of a systematic contextual study of schwa. Furthermore, like other monophthongs, more or less gliding in English -and unlike in French, for example, in which "each oral vowel can by fully specified by a single spectrum" (Vaissière 2007: 59) -, schwa retains rather rarely the same quality throughout its duration (which is explainable by its relatively low, only occasional informative value1 , related to the prosodic features in short unstressed syllables): thus the spectral variability of this [ə] vowel manifests itself both as a discrete (contextual-dependent) and continuous (time-dependent or dynamic) phenomenon, and hence readily describable in the F2/F1 acoustic plane using the software tool SaRP.

Observing the patterns of the positionally determined variability of this vowel was indeed greatly facilitated in this study by the use of a new software tool called SaRP -Speech Analyzer Rapid Plot (http://web.uni-plovdiv.bg/rousni/sarp/). This add-on for Speech Analyzer was designed in view of the frequent need of language researchers and students for accessing, collecting, manipulating and representing acoustic data, in particular in order to create vowel charts. F2/F1 vowel space mapping is essential for the acoustic description of the vowel system of any language. Vowel plots for a given language or dialect (or individual speaker) being easily interpretable in terms of articulatory characteristics, these graphical representations are also valuable in the field of language teaching, learning and assessment or self-assessment. The SaRP tool, which is an extension to the programme Speech Analyzer version 3 or later, (http://www.sil.org/computing/sa/index.htm) allows creating informative charts in an easy and interactive manner.

As far as vowel quality is concerned, which is represented well enough for our purpose by F1/F2 measurements, the contextual dispersion of schwa is, in fact, so severe and pervasive that one may even think of a "vowel triangle" formed by joining the extreme peripheral markers' positions in a scatter chart built from a set of schwa realisations plotted on the F2/F1 vowel space (see Fig. 1). This term of vowel triangle relates here to the notion of dispersion, in contrast with the well-known inter-vowel triangle or quadrilateral. Conceptually, shape and area of such a triangle may be relevant as parameters, evaluating among others, the degree and nature of coarticulation for particular speech styles. Southern British English 1 -This gave the boy so much pleasure that a few days later he tried exactly the same trick again 2 -ten minutes later 3 -the motto of the organisation Such a large dispersion -symbolised here by a sort of uni-vowel triangle which is commensurable with the cardinal vowels diagram -is only partially due to inter-speaker variability. Our systematic instrumental observations of schwa productions by native speakers of Standard Southern British English (carried out so far for about ten different male natives, all of them speakers on the BBC) indicate that the acoustic, and therefore articulatory nature of this so-called "neutral vowel [which] can be considered as a kind of homebase to which the tongue returns frequently in the course of speech" (Kühnert & Fougeron 2004: 1) is rather governed by a double functional principle: the principle of economy: "languages tend to get rid of anything that is superfluous", and the principle of emphasis: "languages tend to stress or exaggerate anything that is necessary" (Passy 1891 : 227). Paul Boersma disentangles Passy's two principles into "a more fundamental speaker-oriented principle of minimization of articulatory effort and an equally basic listener-oriented principle of minimization of perceptual confusion" (Boersma 1998: 2). Since a particular contextual condition favours the one or the other tendency, this especially flexible vowel is often produced with spectral characteristics more or less close to those of some non-neutral vowel. In view of this, acoustic data don't support the conception -such as that put forward by [START_REF] Kühnert & Fougeron ; Kühnert | Schwa-productions and hesitations by native French speakers during English L2 performance[END_REF] -that the centrality of schwa is a most relevant feature. Figure 2 visualises a "schwa-triangle" established by plotting, in the F2/F1 vowel space, all three occurrences of schwa in the utterance If you keep him talking, I'll go on his computer and get rid of it. In Fig. 2, the position of the marker 3 (the /ə/ in rid of it) is obviously inadequate when compared with that of the labelled marker /I/, which is due to the fact that the labelled markers here correspond to another speaker's "canonical" pronunciation of the vowels. Knowing this, the background vowel chart must only be viewed as a rough reference. What is relevant here is the relative (and rather extreme) positions of the schwa vowels in the same utterance, giving a good representation of observed patterns of phonetic variability of this phoneme.

According to Barry Heselwood (Heselwood 2007: 148), "two types of function are identified for schwa: anaptyctic (e.g. today, abbot), and positional (e.g. support, lotus) [in opposition to sport and lotes, respectively]" In the word computer, the first occurrence of schwa is anaptyctic and the second positional since a consonant sequence such as /k_mp/ is unpronounceable and therefore needs the insertion of a support vowel (a manifestation of the phenomena of epenthesis), whereas the final vowel segment in computer (this word is followed here by a pause) functions, phonologically, as a distinctive segment (in opposition to the constitutive function of a segment) by virtue of the opposition segment position / empty position. The last schwa (… and get rid of it) is also anaptyctic, since it is inserted into the sequence as a vowel sound, qualitatively assimilated to neighbouring sounds and for that reason different from theoretical canonical schwa, mainly in order to facilitate pronunciation.

The phonological distinction between anaptyctic and positional schwas could be usefully transposed to the study of the positionally determined articulatory and acoustic features of this vowel, in so far as, logically, anaptyctic schwas are expected to be influenced by the phonetic context to a more considerable extent than positional schwas, all other things being equal. The vowel quality (and quantity) of a positional schwa should tend, in general, to be syntagmatically more contrastive. In our example (I'll go on his computer # and…), schwa is quite open, long and intense (i.e. typically vocalic) and the following long pause eliminates the possibility to explain it only by reference to the following vowel in the word and. We rather think that the influence is progressive; here it is not a matter of vowel harmony, but inversely of an emphasis of the contrast between the two contiguous vowels, as shown in the Fig. 3. Fig. 3 shows in a condensed form the results obtained in our experiments3 . One can see that substantial variation occurs in the F1 and F2 values of the so-called mid central vowel in English. These results confirm in a tangible way the well-known (but nonetheless arguable) 'targetlessness' of schwa: theoretically, "schwa does have a target lingual gesture but it can be completely overridden by the gestures of neighbouring segments." (Browman & Goldstein 1992: 26) What is not confirmed, on the other hand, is our hypothesis that a distinctive schwa may resist coarticulation better than a constitutive (or support) schwa. Thus, the sequence of two grammatical words and, at the same time, minimal pair 'in an' (Mark wrote bad things about Peter in an email) presents a curious pronunciation "inversion" (undoubtedly due to the gestures of neighbouring segments), since the two consecutive vowels /I/ and /ə/ are respectively produced as /ə/ and /I/ (see Fig. 4). It is clear that exploiting global high level pragmatic, semantic and syntactic information contributes more than exploiting low level spectral contrasts between /ə/ and any other vowel in English to enhance speech comprehension, a nice example of top-down information process. Contextual factors in schwa productions do not limit themselves to the gestures of neighbouring segments, but the following point of view is even more reductive: "The actual least-effort position of the jaw and other articulators will depend on the neighbouring consonants, so that the actual realization of the reduced vowel varies according to the phonetic context" (Boersma 1998: 215). In fact, the actual realisation of schwa also depends on the prosody of the sentence and, through vowel harmony, on the neighbouring vowels. Finally, with respect to inter-speaker variability, positional determination of the quality of schwa for each individual speaker should be studied by reference to the vowel quadrilateral of the same speaker. It's because "a listener relates every vowel he hears to what he has perceived as the speaker's personal range of vowel variation, rather than to any absolute formant values"4 .

From this perspective, it is necessary to have a convenient software tool at hand that is capable of reliably retrieving and efficiently managing formant values for vowels in large corpus of linguistic data. Thus, for each short audio text used to retrieve and represent acoustic data about particular vowel productions, the analysis is done by referring the measurement results to a vowel chart established ad hoc -on the same basis: same text and same speaker) -as an idiolectal model describing, numerically and graphically, "the speaker's personal range of vowel variation". Our software tool SaRP (developed in LAPS Scientific Team in collaboration with the University of Plovdiv and described in greater detail in Nikolov, Dommergues & Ryst 2007: 88-95) allows managing an unlimited number of sound libraries and sound collections of analysed speech utterances, as well as an unlimited number of (easy editable) vowel quadrilaterals: vowel charts established each of them as a set of reference vowels. At the end of an intonation unit (i.e. before a pause, including a psychological one, where delimitation is only based on prosodic and/or spectral contrasts), schwa tends to be pronounced rather as a near-open front vowel, to such an extent that it practically coincides with the canonical pronunciation of /ae/ for the same speaker. These conclusions are confirmed as well, in the same text and the same phonetic contexts, by the multiple instances of /ə/ in lexical words such as liar, later, after, burger, either, Milner, brother, etc. By analogy with the so-called target-undershoot model, we may speak here of an articulatory overshoot in such contexts.

Conversely, one can see Fig. 5 that inside expressions or phrases such as computer screen, computer terminal and the computer in front of him, schwa behaviour at end of word does not significantly differ from that in middle of word. Variability of schwa in middle of word mainly concerns the vowel aperture and, in consequence, the F2 value. The average

Fig. 1 :

 1 Fig. 1: Markers 1, 2 and 3 refer to three vocalic samples from an audio text (Beginner's Luck by Chris Rose, British Council Podcasts 2 ) read by a native speaker of Standard Southern British English 1 -This gave the boy so much pleasure that a few days later he tried exactly the same trick again 2 -ten minutes later 3 -the motto of the organisation

Fig. 2 :

 2 Fig. 2: A vowel triangle formed by joining the positions of the three schwa-productions in the utterance If you keep him talking, I'll go on his computer and get rid of it, and showing that the centrality of schwa is only a theoretical constant.

Fig. 3 :

 3 Fig. 3: The "positional", distinctive schwa in computer (marker 2), coinciding with the prototypical /ae/, versus the following emphasised /ae/ (marker 4) in the word and.

Fig. 4 :

 4 Fig. 4: The two consecutive vowels /I/ and /ə/ in the sequence 'in an' are respectively produced as /ə/ and /I/.

Fig. 5

 5 Fig.5shows the F2/F1 set of marks of all the twelve schwa productions for a same speaker and a same word (computer, a statistical key word) in a same audio text (Beginner's Luck by Chris Rose, British Council Podcasts 5 ).

Fig. 5 :

 5 Fig. 5: The F2/F1 set of marks for the second /ə/ in computer -twelve tokens

For example in accept, by opposition with except.

<http://www.britishcouncil.org/learnenglish-podcasts-stories-beginners-luck.htm>

Being an integral part of the SaRP data, the entire sound library named UK -schwa, with multiple sound collections, is downloadable therefore all acoustic analyses are reproducible.

<http://www.phon.ucl.ac.uk/home/wells/formants/preldis.htm> Wells wrote in, A study of the formants of the pure vowels of British English "...it seems to have been conclusively shown byLadefoged and Broadbent [26] that a listener relates every vowel he hears to what he has perceived as the speaker's personal range of vowel variation, rather than to any absolute formant values. An /e/ is identified as such not because of a particular ratio of F1 to F2 or of mel-scale F1/F3 to F2/F3, but because its formants lie at a particular point in the interval between the observed values of the extremes manifested in /i/ and /A/. This is only to be expected in view of the fact that it is a linguistic activity that is under investigation, and all linguistic activities are relative rather than absolute."

Ladefoged & Broadbent 1975. Information carried by vowel, Journal of the Acoustical Society ofAmerica, 29,

second formant frequency then depends progressively and/or regressively on the neighbouring vowels (as in the previous examples: a few days later he..., the motto of the organisation, get rid of it) or the neighbouring consonants (as in computer screen and computer terminal, with a rather front vowel, in contrast with the computer in front of him, with a rather back vowel). Schwa is rather back when followed by nasal consonants /m/ or /n/, apart from consonant /r/.

Often enough, particular realisations of schwa (especially its quality) will adapt and vary under consonantal influence throughout its duration, as is the case with the first /ə/ in the word computer under consideration here (see Fig. 6). The first /ə/ in computer adapts itself to the following consonant /m/ by a falling transition of the F2 from a mean frequency of 1900 Hz to a frequency of 1400 Hz, the latter being characteristic of /m/ as defining one of the power consonant frequency ranges (by analogy with vowel formants). The transitional nature of interconsonantal schwa is best shown in Fig. 7. Following Catford (Catford 1985: 340) and Messum (Messum 2002: 24), interconsonantal schwa can be labelled "an 'open transition' between the two consonants involved, meaning that the first consonant is produced and that when this is complete we follow it with the second. This contrasts with a 'closed transition' when the articulations of the two consonants overlap".