Zhiguo Song

Caroline Lelandais-Perrault

Daniel Poulton

Philippe Bénabes

Adaptive Equalization for Calibration of Subband Hybrid Filter Banks A/D Converters

Hybrid Filter Banks (HFB) A/D converters (ADC) are attractive to software-defined radio application, however their high sensitivity to analog imperfections is still a bottleneck for their realization. In this paper, adaptive equalization is applied for calibrating a subband HFB ADC. Thus, the synthesis filter coefficients are iteratively adjusted for compensating the mismatches between the analog part and the digital part. Simulations show that the robustness of subband HFB ADC is therefore improved.

I. INTRODUCTION

In the context of cognitive radio, Hybrid Filter Banks (HFB) A/D converters (ADC) may be an attractive solution to meet the future A/D conversion system requirements [START_REF] Lelandais-Perrault | Wide-band, band pass and versatile Hybrid Filter Bank A/D, conversion for software radio[END_REF]. HFB are parallel architectures that consist in an analog frequency band decomposition (analysis bank), a conversion-stage (ADC bank) and a digital reconstruction (synthesis bank). From a given sampling rate on each channel, classical HFBs enlarge the conversion bandwidth [START_REF] Petraglia | High speed A/D conversion incorporating a QMF bank[END_REF], [START_REF] Velazquez | Design of hybrid filter bank for analog/digital conversion[END_REF], [START_REF] Löwenborg | A class of twochannel hybrid analog/digital filter banks[END_REF] with a resolution related to the digital reconstruction complexity. Also, it is possible to improve the resolution in a narrow band by concentrating the digital part effort on the reconstruction of the band of interest. This can be achieved by using a subband HFB (Figure 1) for which only the digital part differ from the classical HFB. This architecture has been proposed in [START_REF] Asemani | Subband architecture for Hybrid Filter Bank A/D converters[END_REF] and make it possible to reconstruct one or several subbands of the wideband signal. Thus, it is possible to adapt by software the conversion characteristics (bandwidth and resolution) to the required specifications.

However, the imperfections, such as the analog realization errors, degrade the performances of the above-mentioned HFB ADCs dramatically [START_REF] Asemani | Subband architecture for Hybrid Filter Bank A/D converters[END_REF], [START_REF] Petrescu | Sensitivity of hybrid filter banks A/D converters to analog realisation errors and finite word length[END_REF]. The manufacturing of high precision or tunable analog components is so costly that the adjustment of synthesis filter coefficients after the analog filter implementation may be an appropriate solution. Such a method should make the HFB recover the expected reconstruction quality. In [START_REF] Sanada | Decorrelating compensation scheme for coefficient errors of a filter bank parallel A/D converter[END_REF], a calibration method using decorrelation technique can be applied to a multi-channel IF receiver to suppress the interference among the subband signals, whereas it is necessary to estimate the values describing the analog filters. Furthermore, a complex cross-correlation matrix needs be inverted, which is a critical point for realization. [START_REF] Tg | System of high performances digitalization -Study of the solutions to hybrid filter banks -Characterization and optimization[END_REF] proposes a method that uses a known signal at the input of HFB to estimate the analog filter frequency responses. The synthesis filter coefficients are then calculated from the measurement of the analog filter frequency responses. This method offers good performance but such a calibration cannot be implemented in a final product because it does require a too complex calculation (a large matrix inversion).

Subband M-1 ADC ADC M T M T M T ADC Hm(s) + + + HM-1(s) x(t) H0(s) Subband 0 Subband m ŝ0(n) ŝM-1(n) ŝm(n) Fm,m(z) Fm,0(z) x0(n) xm(n) xM-1(n) F0,0(z) F0,m(z) F0,M-1(z) Fm,M-1(z) FM-1,0(z) FM-1,m(z) FM-1,M-1(z)

Synthesis bank Analysis bank

H(s) F (z) dm(n) ŝm(n) x(t) xm(n) s(t) em(n)
Fig. 2. Phase 1: calibration of HFB ADC using adaptive channel equalization; phase 2: conversion of the extern wideband signal x(t).

In this paper, channel adaptive equalization is proposed to calibrate a subband HFB. With this method, widely used in many applications [START_REF] Haykin | Adaptive Filter Theory[END_REF], the synthesis filter coefficients should be iteratively adjusted to minimize an error signal that should represent the reconstruction quality. In Figure 2, the phase 1 shows the proposed adaptive equalization principle in the HFB context. The signal generator provides two linked signals. s(t) is a wide-band test signal for the HFB input. d m (n) is the desired signal at the output of the HFB. It corresponds to the selection of the subband of interest signal of the signal s(t). The error signal e(n) is used to adjust the synthesis filter coefficients. The HFB calibration will be achieved once the mean-square error (MSE) of e(n) signal is low enough to meet the reconstruction accuracy requirements. Thus, fixing the syntheis filter coefficients achieved in the phase 1 and switching to the phase 2, the subband HFB can convert the extern wideband signal x(t). The quantization errors are neglected here.

In section II, the proposed method is first theoretically justified upon a subband HFB ADC discrete-time model. In section III, the method is extended to a continuous-time subband HFB. An eight-channel HFB with initial good performance and then affected by analog errors is simulated in time domain. The adaptive equalization calibration is simulated and discussed.

II. ADAPTIVE CHANNEL EQUALIZATION FOR A DISCRETE-TIME SUBBAND HFB ADC MODEL A. Perfect Reconstruction (PR) conditions of a subband HFB ADC

Figure 3 shows a theoretical model of a subband HFB in discrete-time domain. The input signal x(t) is assumed to be band-limited that it could be sampled at 1 T without aliasing. This model, first proposed in [START_REF] Asemani | Subband architecture for Hybrid Filter Bank A/D converters[END_REF], involves ideal filters G m (z) that select each subband (Figure 4). The matrix H(z) reconstructs the real signals x m (n) (ADC output signal) from the ideal subband signals. The synthesis filter matrix F(z) reconstructs an estimation of each subband signal. The matrix H(z) is a multiple-input multiple-output (MIMO) model consisting in M -by-M FIR filters. The outputs of the analysis bank can be presented in frequency domain by

X i (e jω ) = M -1 m=0 H i,m (e jω )S m (e jω ), (1)
where H i,m (z) is the element of branch i, subband m in H(z) matrix. Likewise, the outputs ŝm (n) of the synthesis filter banks are given by

Ŝm (e jω ) = M -1 i=0 F m,i (e jω )X i (e jω ), (2)
where F m,i (z) is the element of subband m, branch i in F(z) matrix. Substituting the term X i (e jω ) in ( 2) with (1), the relation between the estimated output ŝm (n) and the inputs subband signals is presented in frequency domain by

Ŝm (e jω ) = M -1 i,j=0
F m,i (e jω )H i,j (e jω )S j (e jω )

For the subband m, let's define

T † m,m (e jω ) = M -1 i=0 F m,i (e jω )H i,m (e jω ) (4)
T † m,j (e jω ) = M -1 i,j=0 j =m F m,i (e jω )H i,j (e jω ) (5)
x(t) x(n ′ ) GM-1(z) s0(n) sm(n) sM-1(n) H(z) xM-1(n) x0(n) xm(n) ŝ0(n) F(z) ŝm(n) ŝM-1(n) T Gm(z) G0(z) ↓ M ↓ M ↓ M M -
1 T et 1 M T . π π M (M -1) π M (m + 1) π M m π M ω 0 1 GM-1(e jω )
Gm(e jω ) G0(e jω ) Fig. 4. Ideal filters for selecting the subbands

[m π M , (m + 1) π M ], m ∈ [0, M -1].
T † m,m (e jω ) is usually called the distortion function, which is expected to be a pure delay n d and scaled by a factor c. T † m,j (e jω ), j ∈ [0, M -1] \ {m}, are the aliasing functions, which are desired to be 0. The PR conditions for the subband HFB ADC are defined here by

T † m,m (e jω ) = e -jωn d T † m,j (e jω ) = 0 (j ∈ [0, M -1] \ {m}) (6)
Hence, the outputs for the subband m of HFB ADC can be presented as follows Ŝm (e jω ) = T † m,m (e jω )S m (e jω )

Distortion term

+ M -1 j=0;j =m T † m,j (e jω )S j (e jω )
Aliasing terms [START_REF] Sanada | Decorrelating compensation scheme for coefficient errors of a filter bank parallel A/D converter[END_REF] Assuming that the synthesis filters are L-length FIR, their optimal coefficient values can be achieved by solving an overdetermined linear system [START_REF] Lelandais-Perrault | Wide-band, band pass and versatile Hybrid Filter Bank A/D, conversion for software radio[END_REF], [START_REF] Asemani | Subband architecture for Hybrid Filter Bank A/D converters[END_REF]. To do that, ( 6) is written for a set of N discrete frequencies, with N larger than L.

B. Adaptive channel equalization for subband HFB ADC calibration

Because the problem is similar for all subbands, we can just focus on the reconstruction of one subband signal s m (n). Due to analog realization errors, the theoretical original response H i,m (e jω ) are changed to Hi,m (e jω ); the corresponding ADC output signals are xi (n). The PR conditions [START_REF] Petrescu | Sensitivity of hybrid filter banks A/D converters to analog realisation errors and finite word length[END_REF] require that F m,i (e jω ) should be accordingly modified to keep the channels equalized. During the proposed calibration phase, the synthesis coefficient vectors noted fm,i = [ fm,i (0), fm,i (1), • • • , fm,i (L -1)] are adaptively equalized in order to minimize the reconstruction error (Figure 5). Let fm denote the synthesis filter coefficients in the subband m

fm = [ fm,0 , fm,1 , • • • , fm,M-1 ] (8)
Let x(n) denote the inputs of the synthesis filters in subband m,

x(n) = [x 0 (n), x1 (n), • • • , xM-1 (n)] T , (9)
where v T means the transpose of the vector v, and xi (n) is denoted by

xi (n) = [x i (n), xi (n -1), • • • , xi (n -L + 1)] (10)
Let e m (n) denote the difference between the desired signal and the estimated subband signal ŝm (n). ŝm and e m (n) are defined by

ŝm (n) = fm x(n) (11) e m (n) = d m (n) -ŝm (n) (12)
The chosen cost function is the MSE J m defined by

J m = E(|e m (n)| 2 ) = E(|d m (n)| 2 ) -2 fm E(x(n)d m (n)) + fm E x(n) T x(n) f T m (13)
The expectation E(x(n) T x(n)) is equal to the autocorrelation of the vector x(n), noted R xx ; the expectation E(x(n)d m (n)) is equal to the cross-correlation between x(n) and d m (n), noted p m . So, (13) can be written

J m = σ 2 dm -2 fm p m + fm R xx f T m (14)
Hence, the optimal coefficients f opt m achieved by the Wiener-Hopf method to minimize J m [START_REF] Haykin | Adaptive Filter Theory[END_REF] is

f opt m = R -1 xx p m (15)
The calculation of f opt m with (15) is obviously too costly. Here, the LMS algorithm is chosen for its simplicity. It consists of updating iteratively the synthesis filter coefficients f opt m with the formula

fm (n + 1) = fm (n) + µe m (n)x(n), ( 16
)
where µ is named step-size. The condition of stability of long LMS FIR filters is shown below [START_REF] Haykin | Adaptive Filter Theory[END_REF]

0 < µ < 2 LX max , ( 17
)
where X max is the maximum value of the power spectral density of the inputs xm (n). Therefore, the adaptive equalization seems theoretically suitable to the discrete-time model of subband HFB. In the next section, this method is extended to the continuous-time HFB architecture presented in Figure 1.

+ Branch 0 + + + Branch m Branch M-1 H(z) MIMO matrix Z -∆ GM-1(z) ↓ M s0(n) sM-1(n) sm(n) ↓ M ↓ M Subband m F(z) MIMO matrix + xM-1(n) xm(n) x0(n) x(n ′ ) Gm(z) G0(z)

Synthesis part Analysis part

s(t) x(t) dm(n) em(n) ŝm(n) Fm,0(z) Fm,M-1(z) Fm,1(z)

Analysis bank

Fig. 6. Continuous-time adaptive subband M -channel HFB ADC architecture: calibration of the subband m before converting a wide-band signal x(t).

III. ADAPTIVE CHANNEL EQUALIZATION FOR A CONTINUOUS-TIME HFB ADC ARCHITECTURE

Figure 6 shows the adaptive equalization for the continuoustime HFB ADC calibration. Hm (s) denotes the analog filters transfer function H m (s) with errors. The signal generator provides two signals s(t) and d m (n) for calibration purpose. s(t) is a wide-band signal. To make sure that the calibration is efficient, this signal should be spectrally rich enough. The desired signal d m (n) is the signal s(t) filtered in the subband of interest. The calibration is achieved once the MSE is below a predetermined threshold. Then the digital coefficients are fixed. The HFB is ready to convert other signals. In order to measure and compare the performances of HFBs in different cases in time-domain, the distortion-to-aliasing ratio (DAR dB ) is defined in (18). ŝm dis (n) denotes the distortion term, that is to say the useful signal at the HFB output, ŝm alia (n) is the sum of the aliasing terms of the subband m. K is the number of the considered samples.

DAR dB = 10 * log       K-1 k=0 |ŝ m dis (n -k)| 2 K-1 k=0 |ŝ m alia (n -k)| 2       (18)

A. Simulation conditions

Considering the frequency band [800MHz 1.2GHz], an eight-channel subband HFB has been simulated in time domain with Matlab/Simulink. The adaptive equalization is shown in Figure 6 has been simulated for the calibration of the first subband [800 MHz 850MHz]. The analysis filter bank consists in eight pure Gm-LC resonators [START_REF] Lelandais-Perrault | Wide-band, band pass and versatile Hybrid Filter Bank A/D, conversion for software radio[END_REF] whose central frequencies are 2πΩ i and the quality factor, Q i . The filter central frequencies are assumed to be affected by a frequency offset of 2π∆Ω . The transfer function of each analysis filter with analog errors is then

Hi (s) = Ωi+∆Ω Qi s s 2 + Ωi+∆Ω Qi s + (Ω i + ∆Ω ) 2 (19)
Synthesis filters are 128-length FIR. The test signal and the desired signal are obtained thanks to a pseudo-random sequence and several filters. The pseudom-random signal is filtered through a filter bank, each filter selecting one subband. The desired signal is the one corresponding to the band to reconstruct and the test signal is the sum of the signals of all subbands. Practically, the test signal and the desired signal samples could be calculated only once and stored in a memory in the device. The test signal should be converted into an analog signal by a DAC, which should not be a limitation because, often, for the same technology, DAC offers a much better speed performance than ADC.

In order to reduce the aliasing terms, it is efficient to allocate a small part of frequency borders (low and high frequencies) at each subband spectrum as guard band (GB) [START_REF] Asemani | Subband architecture for Hybrid Filter Bank A/D converters[END_REF], which is equal to 7% of the subband bandwidth in this paper.

B. Simulation results

First, the ideal synthesis filters f goal m corresponding to the analysis bank with errors Hi (s) (∆Ω = 1% * π M ) are calculed in the frequency domain with the least mean square global approximation synthesis method [START_REF] Lelandais-Perrault | Wide-band, band pass and versatile Hybrid Filter Bank A/D, conversion for software radio[END_REF]. As expected, this subband HFB ( Hi (s) with f goal m ) achieves a DAR of 96 dB. This performance can be considered as the goal of the calibration.

Then, the synthesis filters are set to the theoretical values f th m corresponding to the analysis bank without errors. The DAR goes down to 53 dB. The adaptive equalization is started and processed with a variable step-size beginning at 0.3 and ending at 1.8 * 10 -3 . Figure 7 shows the convergence of two selected synthesis filter coefficients and the DAR. After 10 6 iterations which corresponds to a duration of 10ms in real-time, a stable DAR of 87 dB is achieved which is sufficient for high resolution telecommunication applications (typical resolution of 14 bit). The corresponding synthesis filter coefficient set is called f opt m . The convergence time could be improved with a more advanced adaptive algorithm.

Last, to verify the performance of the subband HFB with f opt m (phase 2 in Figure 2), a realistic wide-band signal is generated for the input. The results show also rather good performance of 86 dB. Another simulation is performed for an analog error of 5%. All the simulation results are included in Table I.

IV. CONCLUSION In this paper, an adaptive subband HFB ADC architecture is proposed. An optimal coefficient set can be approached iteratively to compensate the analog errors by applying the LMS algorithm. Such a calibration can be easily implementable in a digital circuit unlike other calibration techniques that require a large matrix inversion. Simulation results show that a signalto-aliasing ratio can be achieved to 87 dB.

Fig. 1 .

 1 Fig. 1. Continuous-time M -channel subband HFB ADC, T denotes the Nyquist sampling period of input wide-band signal, m ∈ [0, M -1].

Fig. 3 .

 3 Fig. 3. Discrete-time subband M -channel HFB ADC model, n ′ et n denote represent respectively the discrete-time indices associated the sampling rate

-Fig. 5 .

 5 Fig. 5. Adaptive equalization of a subband HFB ADC discrete-time model.

Fig. 7 .

 7 Fig. 7. Above: the convergence of synthesis filter coefficients. ∆Ω = 1% * π M . Below: the convergence of DAR dB VS the iteration number.

TABLE I SUBBAND

 I HFB ADC CALIBRATION PERFORMANCES (DAR dB ), (∆Ω = err * π M ).

			DAR dB
	Analog	Goal	Before	After	Wide-band
	Error	of calibration	calibration	calibration	signal
	(err)	(f goal m )	(f th m )	(f opt m )	(f opt m )
	1%	96	53	87	86
	5%	96	39	83	79


Denis Besnard

Damien Fabre

Wim Van

Eduardo F A Runte

An account of scientific transfer to the industry: the co-development of an incident analysis tool

This paper is an account of scientific transfer to, and co-development with, a commercial organization. The context is that of safety management of an energy distribution network. A company called EnergyCo approached the authors about their need of an incident analysis method. The company wanted a tool that would help them better capture the causes of incidents during maintenance operations on their network. The expertise they particularly asked for was that of human factors; an angle to work analysis they valued but had not been able to deploy at the time.

Beyond the mere technical contents of the tool that was developed, we wish to provide an account of the various transactions, exchanges and collaborations that took place during the development of this method. We will highlight a particular angle of scientific transfer: co-development. In doing so, we will try to convey the message that scientific transfer goes beyond a mere "push" model. Indeed, we are of the opinion that in the case of the development of a method for the industry, transfer can be turned very productively into a codevelopment collaboration.

Introduction

The authors have been approached in the recent past by a French energy distribution company:

EnergyCo. This company had just set a new safety target related to the operation of their distribution network. Within the company, top managers rapidly acknowledged that their ongoing approach to safety (essentially based on a technical analysis of incidents) would not allow them to meet the target. This opened the way for a drastic cultural shift: the integration of human factors (HF) in the analysis of safety performance. Within this shift, our mission would be to integrate HF within an incident analysis tool in order to facilitate organizational learning across the entire company.

Because this tool was meant to be used by operational team managers without HF training, the challenge would essentially be that of transferring en encapsulating our skills under some usable form for the final users. This is essentially what the paper will describe. We will go through the process by which academia and industry collaborated with each other in order to fulfill a need that implied using science for operational non-specialists. In doing so, we will set the focus on the collaborative aspects more than on the scientific contents of the method.

The paper begins with a brief overview of EnergyCo's operations, followed by a presentation of the scientific standpoint proposed by the authors and accepted by the company. The paper then reports on the process of understanding user needs and requirements and moves on to a description of the development of the tool. Emphasis is given to the adjustments and compromises made to meet the needs of the company. The main features of the tool are then presented and a case study is discussed.

As a concluding remark, the authors comment on the experience of transferring academic knowledge to a practical environment with the assistance of those who are the ultimate beneficiaries of that knowledge.

EnergyCo operations

EnergyCo is a nationwide energy distributor in France. Within the largest branch of the company, several thousands of people are involved in maintaining the distribution network, a task that basically obeys two main scenarios:

• planned operations, whereby works are taking place under controlled conditions (time, allocated resources, schedule, etc.);

• emergency operations, whereby an unplanned event is happening on the network, interrupting the delivery of energy to consumers, whose number ranges from a dozen to several thousands.

For workers, a significant portion of the work time is spent on excavations, at the direct contact with network infrastructures buried into the ground. The role of team managers is that of coordinators, especially when critical works on the network implies interacting with public authorities, safety professionals, the public, the media and so on. Although the activity of workers and managers is significantly different by nature, all contribute to a mission that is very clearly identified by all: continuity of service. Indeed, energy is a service that is meant to reach customers without interruption. It follows that any energy loss in the network is dealt with carefully, with particular attention paid to how many customers would be cut off in the case where the network had to be disconnected for repairs.

Another particularity of energy infrastructures is that they carry a service to physical habitations. In dense urban areas, this is challenging working environment since the energy network cohabits with other infrastructures (water, telecommunications, sewers, etc). What is more, this mesh of disparate infrastructures is buried underneath roads or pavements whose topography changes along with urban development. This causes updates in the mapping of the network difficult, a point that is a source of problems.

Finally, as an energy supplier whose network is several thousands of kilometers long in densely populated areas, EnergyCo cannot monitor all the parties working at the vicinity of their infrastructures, not check the qualifications of every worker from each subcontracting company. Socalled third parties are a source of mishap to EnergyCo managers since they do not always comply with company regulations and legal requirements.

The three features above, and their implications in terms of EnergyCo workers, subcontractors and the public, entice EnergyCo to thoroughly manage the operations that take place on their network.

Historically, this management relied (at least in part) on recording factual data on incidents, and discuss potential causes within units, during informal "on the fly" meetings 1 . We now turn to a more precise description of the safety practices within EnergyCo units.

EnergyCo safety approach

At EnergyCo, at the time when the research team was contacted, a number of practices were in place that impacted on how precisely incidents were analyzed, how thoroughly the contributing human factors were captured, and how the gathered information could feed organizational learning:

• Generally speaking, human and organizational factors had not permeated the safety culture of site managers at the time, thereby causing incident analysis to focus on the technical dimension alone;

• Minor incidents were discussed verbally during informal meetings within local teams scattered over the entire country;

1 A serious accident would trigger a formal procedure whereby a safety expert would be sent to the unit to analyse the case.

• Tools that were used to analyze minor incidents were essentially computer-supported data collection sheets focusing on factual information about the case;

• The collection sheets were shared digitally across the company but several tools existed, that were home-made and did not contribute to the company databases;

• Only major incidents were analyzed formally with the assistance of a company expert, but relied on traditional tools such as root cause analysis.

EnergyCo's safety approach is at a point of change with the incorporation of human and organizational factors in their incident analysis. At the mean time and as a result of this shift,

EnergyCo is also modifying their process of learning from experience.

Learning from experience

Because analyzing incidents generally aims at preventing future occurrences, a mere repetitive analysis of incidents is not a productive answer. Instead, an organizational learning mechanism has to be put in place in order to systematically capitalise experience, communicate the lessons learned and influence future practices. This is the context within which the need of EnergyCo, in terms of incident analysis, took place. Organizational learning by a learning from experience process is composed of 4 steps: detection and information collecting (knowing), analysis (understanding), formalization and capitalization (stocking) and sharing and reutilizing (learning) [START_REF] Van Wassenhove | Retour d'expérience et prévention des risques : Principes et méthodes. Series "Sciences du Risque et du Danger[END_REF][START_REF] Gauthey | État des pratiques industrielles de REX. Series[END_REF].

Organizational learning (learning from experience) is mostly implemented in an organization by the means of a tool. Such a tool must be thoroughly conceived with a clear definition of the objectives of the process in mind. If not, there is a possibility that the data collected could not contain the information needed to satisfy the general objectives and questioning. Too often in industry an learning from experience process is implemented by merely collecting facts with little concern for analysis and interpretation of collected information.

The conception of learning from experience process consists of the integration and articulation of the four major steps of the process. We were approached by the company to develop an incident analysis tool, that is, step two in the organizational learning process ("understanding: analysis of the informations"). Furthermore, the tool that was to be developed for EnergyCo was a "second level" analysis to investigate the human and organizational factors of some of the incidents or accidents.

The first level analysis corresponds to a systematic technical analysis already in place at EnergyCo.

So EnergyCo was changing a fair amount of their organizational learning structure. Capturing the causes of incidents in a systematic way made sense, especially when the latter was meant to be deployed nationwide across the entire company. It was just difficult, to us researchers, to integrate one part in a process without having a general view on the objectives and the design of the whole process.

The company had acknowledged the need to know what to do with the data before collecting it. On that front, a computer-based structure existed that would accommodate the data that were to be collected thanks to our tool 2 . Our job would then be to design a collection and analysis tool that would help EnergyCo capture human dimensions of incidents during operations on their network.

General approach and scientific standpoint

Understanding human performance

Our general approach to human performance (whether safety-related or not) is to understand work and to pinpoint the conditions under which operators performed their task. The assumption behind this approach ties back to the Common Performance Conditions (CPC; [START_REF] Hollnagel | Cognitive reliability and error analysis method[END_REF], a set of criteria that influence human performance. The idea was not totally new at the time: a similar concept had been proposed by [START_REF] Swain | Handbook of human reliability analysis with emphasis on nuclear power plant applications[END_REF] and coined performance shaping factors.

The philosophy behind the CPC is what underpinned the second generation accident analysis methods: human performance cannot be reduced to an equation relying on a nominal failure rate 3 .

Instead, it is the context in which people work that determines for the most part how well or badly they are going to perform on a given task.

Table 1: Overview of the 9 Common Performance Conditions

2 What this computer-based structure was composed of falls outside of our scope and will not be discussed here. However, we will touch upon the organisational learning loop and discuss the fit between the latter and the incident analysis tool. 3 The CPC are actually only one sub-set of an accident analysis and risk assessment method authored by Hollnagel:

CREAM.

Conditions Description

Adequacy of organization

The quality of the roles and responsibilities of team members, additional support, communication systems, Safety Management System, instructions and guidelines for externally oriented activities, role of external agencies, etc.

Working conditions

The nature of the physical working conditions such as ambient lighting, glare on screens, noise from alarms, interruptions from the tasks, etc.

Adequacy of MMI and operational support

The Man-Machine Interface in general, including the information available on control panels, computerised workstations, and operational support provided by specifically designed decision aids.

Availability of procedures and plans

Procedures and plans include operating and emergency procedures, familiar patterns of response heuristics, routines, etc.

Number of simultaneous goals

The number of tasks a person is required to pursue or attend to at the same time (i.e. evaluating the effects of actions, sampling new information, assessing multiple goals, etc.).

Available time

The time available to carry out a task and corresponds to how well the task execution is synchronised to the process dynamics.

Time of day (circadian rhythm)

The time of day ( or night) describes the time at which the task is carried out, in particular whether or not the person is adjusted to the current time (circadian rhythm). Typical examples are the effects of shift work. It is a well established fact that the time of day has an effect on the quality of the work, and that performance is less efficient if the normal circadian rhythm is disrupted.

Adequacy of training and experience

The level and quality of training provided to operators as familiarization to new technology, refreshing old skills, etc. It also refers to the level of operational experience.

Crew collaboration quality

The quality of the collaboration between crew members, including the overlap between the official and unofficial structure, the level of trust, and the general social climate among crew members.

The principle behind the CPC is simple. They are used as a set of criteria to assess the conditions in which people perform their task. This assessment is performed against qualitative scales: very efficient, efficient, inefficient, deficient.

In the case of EnergyCo, we suggested this approach since it matched their needs of understanding the so-called human factors at play during maintenance operations. The CPC also had the advantage of being designed as a set of basic work analysis criteria. This would ease their translation into a tool, a possibility that EnergyCo saw as an asset.

Incident analysis and safety management

Typically, at EnergyCo, an incident is an event that:

• produces unwanted outcomes;

• causes injuries (to EnergyCo employees or the public) and/or loss of assets;

• disrupts the continuity of service to an unacceptable extent.

In dealing with incidents, top managers at EnergyCo took the decision to manage human factors along with the traditional set of technical causes. The high-level objective was to capitalise on causes of incidents, both on the technical and human factors sides. In doing so, managers were of the opinion that widening the spectrum of the search for causes would also widen the spectrum of the cures to incidents. This is consistent with the following safety proverb: plateauing for several years, only a significant paradigm shift would allow them to further improve safety performance. This situation is the very same one that commercial aviation is facing in western countries: mastering the technical design of aircraft and introducing crew resource management programmes in the 1990s lowered accident rates to some unprecedented level . However, statistics now show that the number of aviation accidents, although at an all-time low, is now asymptotic (Boeing, 2005), revealing that the philosophies and actions deployed over the past decades have come to their limits in terms of improving safety records.

Accompanying EnergyCo through their paradigm shift was our mission. This is where our human factors standpoint and the company's objective met. Concretely, this translated into turning the CPC into a tool; a development that was undertaken under a collaborative model. This is what the following sections will describe.

Analysis of requirements and early specifications

This section is not a step-by-step account of a typical development project. Instead, we will focus on the phases were co-development was the driving force. Our intention is to highlight the contribution of both parties (EnergyCo and our team) to the method. Another interesting aspect is that any one party would have been able to produce some version of the final tool on their own. However, the codevelopment strategy revealed fruitful in:

• saving development time, each party bringing their expertise into the development process;

• making sure the scientific assumptions behind the use of the CPC were understood and shared;

• capturing the reality of the work of maintenance operators through the involvement of EnergyCo former unit operational managers.

Understanding the need

In order to understand the needs and requirements, as well as the constraints, of the company, the research team conducted an assessment. This assessment involved:

• visiting a couple of operational sites and workshops;

• speaking to several people within the organization, including a number of local managers.

As we did this, we made an effort to explain our approach, i.e., the importance of understanding why people do not perform as expected. Our assumption, as explained previously, is that the causes are to be found essentially in the conditions of work.

It became clear to us that the tool should be able to quickly assist in the identification of good practices and failures. In addition, the tool should serve as a first level of incident analysis, and therefore assist in the identification of incidents that merited deeper investigation.

Understanding the context: factors of adoption and rejection

Another requirement is that the method has to be simple because it is was meant to be used by non-

specialists. An important factor of acceptation of a new tool are the benefits that the tool gives to the persons who use it:

• gain of time/effort to do the incident analysis compared to other (existing) methods or tools;

• the information obtained from the use of the tool should be useful, i.e., lead to an improvement in safety;

• the tool should add value to the job and be a factor of motivation for the user;

The co-development is a way to assure that the management of the company obtains a tool that fits their needs (and a tool that is scientifically based), but attention must be given to the final use of the tool. The feedback of the learning process to the operational managers and field operators is an important step in a successful organizational learning process; without it the efforts of the persons who collect (and analyze) the information aren't rewarded and the long term success of the process can't be guaranteed.

Early specifications

From the 2 subsections above, we came to the conclusion that a CPC-driven method would match the above requirements best. Indeed, it would bind together several essential points:

• a tool that would make the analysis of incidents more systematic;

• an approach that would focus on human and organizational factors;

• a philosophy that would be based on the conditions of work;

• a product that would be simple and match as many as the acceptance/rejections factors as possible.

Other points mentioned by the company were:

• to introduce dynamics in the organizational learning process and to avoid methods that are too complex to use

• share information and experience efficiently

• a simple and fast method: no special skills are needed and no more than 90 minutes are needed to analyze an event

• the method must be compatible with the "level one" analysis that is based on technical factual data (see section 1.3);

As a technical answer to the identified requirements, the specifications were as follows:

• a set of analysis criteria derived from Hollnagel's (1998) CPC;

• a portable, cross-platform spreadsheet-based analysis grid;

• a simple quantitative assessment system for the analysis criteria.

A tool can satisfy only two of the following characteristics at a time: simplicity, generality and precision [START_REF] Weick | Sensemaking in organizations[END_REF]. In this case, the tool had to remain relatively simple and precise. The tool is targeted at the detection of human/organizational factors for the context of the working conditions of EnergyCo. The general approach of CREAM, which is rather complex to implement, had to be transformed into CREAM-light, a simple and specific tool to the context of EnergyCo.

Co-development of the method

The co-development team

On the academic side of the project, 2 senior and 3 junior researchers were involved. The company participated with 1 engineer from EnergyCo's Research & Development Department, and 1 former operational manager who served as project lead. Although the company engineer was for all purposes responsible for the project, the presence of a person with field experience was essential, not only because he knew what would work and what would not, but also because he was able to open the doors of the company to the academic team. In that sense, he performed the role of the "gatekeeper" described by [START_REF] Lofland | Analyzing social settings -a guide to qualitative observation and analysis[END_REF].

Scientific foundations

Hollnagel's (1998) initial set of 9 CPC (see section 2.1) stemmed from a cognitive view of human activities, based on generic classical psychological dimensions. This was both the strength and the weakness of CPC. Indeed, CPC were designed as generic and versatile indicators, and offered a wide range of potential fields of application. However, this also meant that before being deployed, CPC would need some tuning to mirror the exact type of activity performed by EnergyCo. Therefore, the following modifications occurred:

• Renaming conditions. This was done to increase the ease of understanding of non-specialists.

• Redistributing conditions. Some of the conditions were not easily transposable to EnergyCo since they were heavily focussing on a very specific cognitive dimension (e.g. Number of simultaneous goals). The condition was dropped, its underlying contents were renamed and eventually redistributed to other conditions.

• Adding conditions. The new set comprised 12 indicators instead of 9. The added indicators were Maps, Communication, and Third parties. This addition was very important to capture the specificity of operators' work since EnergyCo is constantly using maps to locate their installations, communicates a lot between operational managers and on-terrain operators, and also deals with third parties who work on, or in the vicinity of their network.

• Increasing the level of detail. The new set of 12 indicators was given 44 subcriteria altogether, in order to adopt a finer granularity in the analysis of performance conditions, guide the assessment, and dampen the effects of individual interpretation.

The 12 indicators 4 and their subcriteria 5 . are described in Table 2 These conditions and subcriteria 4 From here onwards, we will use the term indicator to avoid any confusion with Hollnagel's seminal CPC. 5 The precise contents of each subcriteria are not described here in order to keep the table readable. A more precise description exists in the final tool, screenshots of which will be presented in further sections of the paper.

Prototyping

Numerous iterations of the method, many discussions to adjust contents, adapt wording to end users.

Because time constraints were tight, conducting an intensive field study to capture the reality of the activity was not an option. Instead, a former operational manager worked with us in order to bring the context into the development cycle. Although not ideal, this strategy ensured that an acceptable compromise between time and ecological validity could be achieved.

We developed the tool with people who belonged to R&D and management. The latter personnel used to work within network management units and knew the job very well. That the co-developers knew the job was essential to producing a method that would be used. Because they were people who A particular challenge was to make sure that the scientific principles behind the tool were respected.

This is because the development team, on EnergyCo's side, had a limited grasp of Human Factors approaches. Thus, the process of educating company personnel on HF was taking place simultaneously with the development of the tool. Conversely, the academic side of the development team had to learn about the specific characteristics of EnergeyCo's business, and in particular, of the risks its infrastructures faced and the means available to keep them at bay. The learning process took the shape of multiple in-loco and remote iterations, with both sides presenting their views and aiming at reaching satisfactory compromises when questions arose.

Keeping requirements in mind

Throughout the development of the tool, the team constantly went back to the list of requirements identified. Simplicity of use was a major requirement because we were well aware that the final users would have limited opportunity to learn about the principles of human and organizational factors behind the tool. This simplicity was to be reflected both on the design of the tool, i.e., what the tool would in effect do for the user, and on its interface.

The incident analysis tool

Presentation

The version of the incident analysis tool (IAT) that is presented here is the generic version. The final version is owned by EnergyCo and will not be presented here. The main differences between the two versions are about the exact subcriteria used to to assess indicators and the use of professional terminology.

Figure 1: Screenshot of the layout of an indicator with its subcriteria and comments boxes (computer window and menu bars not displayed in this graph)

IAT is supported by a spreadsheet that hosts the 12 indicators, their subcriteria, a textual guide for the subcriteria, description boxes, assessment bar (this latter point will be described in the next figure), and comments boxes. Each indicator has a spreadsheet page of its own. As described in Figure 1, the page includes the indicator label, its subcriteria and a textual description of each subcriteria, a description box and an assessment bar. The spreadsheet is therefore composed of 12 pages of indicators, each broken down into further subcriteria (see table 2).

This paper does not aim at describing the method in detail. Instead, it focuses on co-development.

Therefore, it might suffice to say that each subcriteria is aimed at receiving a qualitative assessment, and comments if need be. The indicator itself (Working conditions, in our example) can also receive extra open comments in a box located at the bottom of the page.

Following with our example indicator, the task of the incident analyst is then to assess the contribution of each subcriteria to Working conditions. To do so, the analyst selects an assessment option (Working environment in our example in Figure 2) by clicking in the assessment bar, and then selecting the desired option from the pop-down menu. Each assessment option corresponds to an individual score. Once all subcriteria have been assessed, the spreasheet then calculates an average score for this indicator, and then converts it to a centesimal scale6 . The incident analysis process is a mere 12-fold repetition of the short procedure described above.

In order to provide the analyst with more than a series of scores for each indicator, the latter are displayed as a radar graph on a separate page of the spreadsheet. This graph is composed of 12 branches, one per indicator. Each branch comprises 10 notches where the indicator score is plotted (see Figure 3). Last, a summary page that compiles all comments entered during the analysis and displays color codes that mirror the average mark received.

Strengths and weaknesses

At the conclusion of the project, a final version of the tool was delivered to EnergyCo. Below, the strengths and weaknesses of this tool are addressed:

• Co-Development: Co-development of a custom-built tool, as opposed to one just off-theshelf, helps ensure the the product is well adapted to EnergyCo's needs. In addition, because company personnel was directly involved in this process, a sense of ownership is created, which would otherwise be absent;

• Domain jargon: The language used in the tool -in the menus, in the help boxes, but most essentially, in the revised CPC table -is compatible with EnergyCo's lingo. This reduces the user's effort to understand "what the computer means," and ensures that the analyses done with the tool are understood at all levels of the organization;

• Takes into consideration human and organizational factors: the tool incorporates human and organizational factors into data collection and analysis of incidents, which until then had been overlooked by EnergyCo. The tool, if effectively deployed and used, will assist EnergyCo in determining in what areas it must concentrate its efforts in order to improve safety;

• Easy to learn, easy to use: the tool is built upon a software platform already used by EnergyCo. This ensures that from a "computer literacy" point of view, users will find it extremely easy to find their way around when using the tool. A particular effort was made to ensure that the criteria presented for assessment are explained in detail, so as to prevent confusion;

• Serves as an "entry point" for a more detailed investigation: large-scale events are often unique in nature and require the use of more powerful analysis tools than the one presented in this paper. On the other hand, not all small-and medium-scale events are the same. Events of a repetitive nature may receive a faster, simpler treatment. Because the IAT is intended to feed a database, it serves an added function of helping identify repetitive cases as well as unique cases. The latter can then be deepened with other auxiliary analysis tools;

Nevertheless, we must also note that the tool developed has some weaknesses, to which we now turn:

• The simplicity/generality/precision compromise: given the requirements of the project, the development team prioritized simplicity and precision over generality. This means that the tool was custom-built to address only a specific type of event, namely, incidents affecting the company's distribution network. It is not in the scope of the tool to handle incidents taking place at EnergyCo's administrative buildings and workshops, for example;

• Cost in time: although the tool is quite simple to use, time will be needed to train users to exploit the IAT to its full capacity. In addition, some time will be required until sufficient experience with the tool exists for a discussion of the analyses to emerge. Finally, users are "practical people" who may quickly abandon the tool if they see it as extra paperwork added to their workload without any visible benefits. EnergyCo is advised to ensure that the use of the tool is rewarded with prompt action;

• Incident feedback: likewise, the research team has no information on incidents investigated, so that the actual benefits of using the tool, compared to EnergyCo's previous incident investigation tool, remain to be evaluated. As mentioned above, it is nevertheless important that EnergyCo makes sure that incidents analyzed with the aid of our tool result in visible benefits for the users;

• The analysis is meant to be done as a group (during debriefing meetings) but this can introduce biases on the assessments left:

• It is possible to fill in the spreadsheet even if the data is incomplete or even missing: the major risk here is the temptation to complete only the easier sections of the spreadsheet and to disregard those which require more effort. The company will be able to detect instances of such behavior, but if it wishes to make sure that investigation meets high standards, it must provide workers with the resources needed;

• the tool only analyses human/organizational factors, no technical analysis: even though this was what we had been asked to do, we list it as a weakness. It would have been desirable to promote an integrated approach that covered technical, human and organizational factors of incidents. However, the company believes it already has the methods and tools in place to handle technical factors.

Training, evolution and deployment

A small number of training sessions were organized for some pre-selected teams. Feedback was then gathered internally and some amendments were done to the tool. Namely, the data collection sheets that EnergyCo used for technical analysis ("level one") were integrated to the tool. The idea was to enrich the factual and technical description of an incident. Also these collection sheets have been used for several years already. Therefore, it allows one to ensure some continuity between the data that used to be collected in the past, and the data that is handled by the new tool. The feedback we were able to gather indicates that users like the tool and are eager to give it a chance. From the perspective of the research team, this type of feedback is probably the most interesting output of the co-development process.

Lessons learned and recommendations

At the end of this account of scientific transfer, we wish to highlight what we have learned, and what our recommendations might be for researchers who wish to embark on developing methods and tools for the industry. We wish to do so to share our experience in priority with:

• academics, in order to describe what interacting with the industry might look like;

• researchers from R&D departments, who are meant to work with operational units from their own company.

Changing the method to accommodate operational needs

To the authors, the most obvious trait of EnergyCo's position during the co-development was their great effort to tune the method to their practices and existing tools. Some of their concerns were about the psychological terminology used, which in places had to be made less technical. Examples include the words cognitive, heuristics, interface. Also, some indicators of IAT were not understood as conditions of performance but as criteria of compliance to regulations. An instance of the latter was the Individual protection equipment. The latter was not seen as having an influence on the way work is done (e.g. gloves can prevent one from getting a good grasp on tools or objects) but was used to assess whether the equipment was worn fully or not. For this subcriteria, the assessment options retained by EnergyCo were a mere yes or no. Such a view can be a problem since it will not leave room to capture the reason why the equipment was not worn entirely (e.g. clumsy, uncomfortable, too hot, lack of feedback from objects, etc.). In the long run, this compliance-driven approach might become a factor of lack of safety, thereby leaving behind a latent condition waiting for future mishaps. This clearly linked to top management decisions regarding the orientation of the final method. Wearing protective equipment, when not complied to, was interpreted as a failure in itself and was simply not tolerated, although the idea that such violations have causes too was accepted (see Alper & Karsh, 2009).

Trade-off between scientific grounding and industrial wishes

From the section above, one might get the impression that scientific transfer towards the industry is sometimes in danger of being more about transfer than science. It is not exactly so. Indeed, when the industry adopts a scientifically-informed method or tool, they still have to reconcile constraints. To them (it certainly was the case for IAT), the choice is between a less-than-perfect method that everybody understands, against a perfect one that only a few people can deploy. Because of this tension, the temptation is great for the industry to adapt the method beyond what it can reasonably accommodate. The opposite temptation exists for researchers to keep the method unchanged because of all the carefully chosen pieces of underlying research. In hindsight, we stood as moderators. That is, we did not try to defend perfection but promoted informed adoption instead. We assisted the company in finding a reasonable compromise between a method whose roots would still be in contact with some scientific ground, and yet allows its final users to easily achieve what the company wanted initially.

CPC had to be adapted to EnergyCo operations, which require intense participation of EnergyCo personnel. While the research team is knowledgeable about principles of human and organizational factors at work, its members are not specialists in EnergyCo's domain of activity. Transfer is therefore a matter of contextual awareness and transformation of knowledge. This did not imply any scientific breakthrough from us (as the scope of the project was transfer, not research) but needed a fair amount of explanation to get the concept of CPCs through, and demonstrate the usefulness of using working conditions as analysis criteria.

Conditions of performance were a totally new approach to EnergyCo operational managers and although it remains to be seen whether we were able to effectively "convert" them to the HF approach, we are confident that we have at least raised their awareness of the subject.

Conclusion

Co-development can be fruitful for both researchers and the industry. Indeed, the industry is often pulled forward by market-driven operations and ventures. In terms of reflection, this can be detrimental to long-term performance since little time and resources are set aside for the integration of new views and philosophies. In this paper, we focused on a new set of indicators for incident analysis at EnergyCo but this rush-forward phenomenon is widespread in the industry and potentially touches any domain of activity. A striking aspect to this situation is that academia has the exact opposite problem: performance is assessed from publications, an exercise that focuses on new ideas first and overlooks industrial needs, and even potential application.

Our position is that the industry and academia have a lot of opportunities to collaborate that are yet to be discovered. On the one hand, industry has very interesting problems waiting to be solved, with limited resources to allocate. On the other hand, academia is a wonderful think tank whose outputs are vastly under-used. From this point of view, transfer is an intention to make these two worlds come together to produce a piece of knowledge or engineering that none of these two worlds could have produced alone as effectively as through collaboration.

One might now wonder why transfer might be such a good thing to do, after all. A simple and immediate answer is that it can be good for society. In the case of our interaction with EnergyCo, it will bring a new way to look at incidents and hopefully decrease the occurrence of serious events.

Beyond this potentially naive and maybe oversimplified view, transfer gives industry the opportunity to try new ideas without necessarily have in-house research force. For academia, transfer is a chance to see how research ideas and prototype methods get tuned and adapted and tested though real-world applications.

  were the first concrete outcome of the co-development process. It probably also was the most important since it touched on the very heart of what aspects of work would be captured in the final incident analysis method. The fist phase of the co-development of the indicatorsTable 2: Overview of the 12 indicators and their subcriteria

  could have been the method users themselves, the development team had a very precise idea of what would be understood and what would not, what would pose a problem and what would not.

Figure 2 :

 2 Figure 2: Screenshot of an indicator tab with a pop-down menu showing assessment options (shaded area added for readability purposes)

Figure 3 :

 3 Figure 3: Screenshot of a radar chart after evaluation of the 12 criteria

  the European Sociology Association, Lisbon, 02-05 Sept. 2009

Figure 4 :

 4 Figure 4: Screenshot of a section of the summary page.

  What you look for is what you find, and what you find is what you fix. Indeed, to EnergyCo top managers, it was clear that concentrating on the technical dimension of incidents would only bring up more instances of known technical causes without increasing safety records. And because the company safety record was

9th conference of the European Sociology Association, Lisbon, 02-05 Sept. 2009

This latter scale is used across the 12 indicators of the spreadsheet in order to obtain comparable scores even when the number of assessments in not equal.

SUMMARY

Unfavourable

No effect Favourable

WORKING CONDITIONS

Summary of comments 0 0 0 The connector's nuts were too small to be handled with gloves on. 0 0 0 0 0 0

TOOLS

Summary of comments

The operator had to crouch in a very narrow space to reach the underside of the connector.

The lighting within the trench was insufficient but the operator knew the connector's position and layout very well.


Laëtitia Matignon
email: laetitia.matignon@ens2m.fr

Guillaume J Laurent
email: guillaume.laurent@ens2m.fr

Nadine Le Fort-Piat

Reward function and initial values : Better choices for accelerated Goal-directed Reinforcement Learning

An important issue in Reinforcement Learning (RL) is to accelerate or improve the learning process. In this paper, we study the influence of some RL parameters over the learning speed. Indeed, although RL convergence properties have been widely studied, no precise rules exist to correctly choose the reward function and initial Q-values. Our method helps the choice of these RL parameters within the context of reaching a goal in a minimal time. We develop a theoretical study and also provide experimental justifications for choosing on the one hand the reward function, and on the other hand particular initial Q-values based on a goal bias function.

Introduction

The reinforcement learning (RL) paradigm [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF] provides techniques in which an agent can optimize environmental payoff for the autonomous resolution of tasks. A RL agent tries to learn a policy, i.e. it learns by trial-and-error to select actions that maximize its expected discounted future rewards for state-action pairs, represented by the action values. Q-learning [START_REF] Watkins | Learning from Delayed Rewards[END_REF] is a commonly form of RL where the optimal policy is learned implicitly in the form of a Q-function.

One of the main limitations of RL is the slowness in convergence. Thus, several methods have been proposed to speed up RL. They involve the incorporation of prior knowledge or bias into RL. [START_REF] Mataric | Reward functions for accelerated learning[END_REF] proposed a methodology for designing reward functions that take advantage of implicit domain knowledge. It involves the use of continuous reward functions and progress estimators. Likewise, with reward shaping, the rewards from the environment are augmented with additional rewards [START_REF] Ng | Policy invariance under reward transformations: theory and application to reward shaping[END_REF]. However, reward shaping can lead the agent into learning suboptimal policies and so, traps the system. [START_REF] Wiewiora | Potential-based shaping and Q-value initialization are equivalent[END_REF] completed the reward shaping study and moreover, proved certain similarities between potential-based shaping and initial Q-values. Indeed, the most elementary method for biasing learning is to choose the initial Q-values [START_REF] Hailu | On amount and quality of bias in reinforcement learning[END_REF]. So [START_REF] Koenig | The effect of representation and knowledge on goaldirected exploration with reinforcement-learning algorithms[END_REF] studied various representations of reward functions and the complexity of Q-learning methods depending on the choice of RL representation. Finally, concerning imitative reinforcement, [START_REF] Behnke | Learning to play soccer using imitative reinforcement[END_REF] proposed to give the learning agent access to the Q-values of the experienced agent. Thus, reward function and initial Q-values play an important part in RL. Nevertheless, although RL has been studied extensively and its convergence properties are well known, in practice, people often choose reward function on one's intuition and initial Q-values arbitrarily [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]. In this paper, we discuss the effects of RL parameters on the policy in order to suggest a generic analysis. We validate our analysis with Q-learning algorithm. The main issue is to shed light on how to correctly initialize RL parameters in order to obtain the desired optimal behavior in a minimal time within the context of a goal directed task.

Reinforcement Learning

The framework of most of RL algorithms is a Markov Decision Process (MDP), defined as a finite set of states, S, a finite set of actions, A, and a transition function T : S × A × S → [0; 1] giving for each state and action a probability distribution over states. R : S × A × S → R is a reward function giving the expected immediate reward or reinforcement received under each transition. The goal is to learn a mapping from states to actions, called a policy, π.

In this work, we have validated our analysis with Q-learning [START_REF] Watkins | Learning from Delayed Rewards[END_REF] algorithm. In Q-learning, an action-value function Q π (s, a) is estimated over the learning process and stored in a tabular representation. An action-value represents the expected sum of rewards1 the agent expects to receive by executing the action a from state s and following the policy π . The optimal action-value function Q * is known to be the unique solution to the Bellman equation,

Q * (s, a) = ∀s ∈S T (s, a, s ) R(s, a, s ) + γ max a Q * (s , a ) . (1)
Q-learning is an off-policy method and its updating rule is :

Q(s, a) ← Q(s, a) + α r + γ max a Q(s , a ) -Q(s, a) (2)
where r is the reward received for the transition from the state s to the new state s by executing the action a. α ∈]0; 1] is the learning-rate parameter and γ ∈ [0; 1[ the discount factor. Under some cases [START_REF] Watkins | Technical note: Q-learning[END_REF], Q-learning algorithm is guaranteed to converge to the optimal value function. The Q-Learning algorithm chooses the action according to an exploration/exploitation criteria. We used the -greedy method, in which the probability of taking a random action is and, otherwise, the selected action is the one with the largest Q-value in the current state2 .

Choice of uniform initial Q-values with binary rewards

We make the assumption that two general trends stand out : the global policy and a specific behavior at the beginning of the learning process. In this section, we are going to set out that these both tendencies depend on the shape of the reward function and on the initialization of the action-value function. We first considered a binary reward function which has the advantage to include a lot of cases and it is possible to extrapolate.

Optimal policy

The binary reward function is such as the reward received is always r ∞ except if the new state is the goal state and then, the reward is r g . It's given by :

∀s ∈ S ∀a ∈ A, R(s, a, s ) = r g if s = s g r ∞ else ( 3
)
where s is the state obtained by executing the action a from s, and s g the goal state. In case of all rewards are identical (r g = r ∞ ), the solution of the Bellman equation ( 1) is a constant noted Q ∞ ,

∀s ∀a Q * (s, a) = Q ∞ = r ∞ 1 -γ . ( 4
)
That is to say that during the learning process, Q-values for all state-action values converge to Q ∞ . Nevertheless, if r g = r ∞ , Q ∞ is the limit of the action-value function Q * (s, a) when the distance between s and s g tends toward infinity. So, toward the goal, states are more and more or less and less attractive depending on r g and Q ∞ . On the one hand, if r g > Q ∞ , the Q-value of state-action pairs moving to the goal state will be more and more attractive than Q ∞ . So the global optimal policy is the shortest way toward s g . On the other hand, if r g < Q ∞ , the optimal policy is random everywhere except a local repulsion of s g . As a matter of course, the shortest way toward the goal is the sought optimal policy concerning goal-directed tasks. So r g must always be superior to Q ∞ .

Behavior at the beginning of the learning process

As well as the reward function, the initial value Q i of the action-value function has an effect on the policy, but only at the beginning of the learning process. We believe that a global trend can be underscored during the first trials of the learning process.

Let's examine what the Q-values are expected to be at the beginning. If we calculate the first updating of a state-action pair (s, a) thanks to (2), such as the next state s is not the goal state and has not been updated, we have :

Q(s, a) ← Q i + α [r ∞ + (γ -1)Q i ] ← Q i + α(1 -γ)(Q ∞ -Q i ) . (5)
So the discriminating value of Q i is also Q ∞ . According to the value of Q i compared to Q ∞ , states already visited will be more or less attractive as long as the agent has not reached plenty of times the goal state.

-If Q i > Q ∞ : states which have already been visited will have a value lower than the value of states which haven't yet been visited (Q(s, a) < Q i ). In other words, states which haven't yet been visited will be more attractive.

It induces the agent to explore more systematically at the beginning of the learning than a random exploration. We called it systematic exploration behavior.

-If Q i < Q ∞ : states which have already been visited will have a value superior to the value of states which haven't yet been visited (Q(s, a) > Q i ).

That's to say states which have already been visited will be more attractive.

It leads the agent into less exploration at the beginning, which considerably slows down the learning. We named this behavior "moving round in circles".

Of course, it's better to avoid it. -If Q i = Q ∞ : states which have already been visited will have the same value than states which haven't yet been visited (Q(s, a) = Q i ). So we obtain at the beginning a pure random behavior.

Gridworld experiments

We have discussed how traditional reward functions and arbitrary initial Qvalues may slow down the learning of an interesting policy. At this point, we are going to validate this previous analysis and we have chosen for simplicity and clarity to use at first a non-deterministic gridworld domain to demonstrate how the behavior is influenced by using binary rewards and different initial Q-values.

Benchmark. The system is represented by a mouse going around a maze (Fig. 1a). Each mouse's position is a discrete state. When the mouse reaches the goal state, the trial ends. The mouse chooses from four actions, representing an intention to move in one of the four cardinal directions (N,E,S,W). An action that would move the mouse in a wall instead leaves the mouse in its current position. Any movement moves the mouse in the intended direction with probability 0.6, and otherwise in a random state of the four neighboring states of the expected state. All trials use Q-learning with a learning-rate α of 0.1, a discount factor γ of 0.9, a tabular Q-table initialized uniformly Q i and follow a policy where the greedy action is taken with a probability 0.9 ( = 0.1).

Binary reward function. Our reward function replicates the function given in (3), with r g = 1 and r ∞ = 0. With such a reward function, the optimal policy is the shortest way toward the goal and the discriminating value of Q i is 0 (Q ∞ =0). Fig. 1b illustrates our previous analysis. As can be readily seen, using the systematic exploration behavior helped speed up learning during the first trials. The agent visited every nook and cranny of the complex maze and the goal state s g was discovered faster than in case of a random exploration. But given that the agent was always spurred on to explore, it always took more steps to reach the goal after some trials. Besides, we used different values of Q i for the systematic exploration and we notice the more Q i was superior to Q ∞ , the more the agent explored. So, the more the difference between Q i and Q ∞ is important, the more the general behavior is underlined.

Concerning the moving round in circles behavior (Q i < 0), we do not submit any experiments because the agent took too much time to reach the goal. Anyway, this behavior has to be avoided.

Conclusion

This shed light on the importance of initial Q-values. The choice of Q i is not trivial and must be done according to the desired behavior. When the system naturally goes away from the goal, a systematic exploration should be preferred in order to speed up learning at the beginning. Indeed, systematic exploration forces the controller to explore unvisited states, and so to approach the goal.

Choice of continuous reward function and heterogeneous initial Q-values

In order to broaden the scope of our study, we propose henceforth to use first two different continuous reward functions with uniform initial Q-values, and secondly to initialize the action value function with a goal bias function.

Reward function using progress estimators

First, we propose to study the use of a continuous reward function instead of binary rewards. Thus, some authors introduce reward functions by using progress estimators [START_REF] Mataric | Reward functions for accelerated learning[END_REF] or reward shaping [START_REF] Ng | Policy invariance under reward transformations: theory and application to reward shaping[END_REF]. Progress estimators provide a measure of improvement relative to an objective. They do not supply a complete information but only partial, goal-specific "advice". For instance, concerning the gridworld, the progress estimator can be an assessment of the expected number of steps needed to get to the goal from the new state s , defined as ϕ(s , a) = d(s , s g ). d is the manhattan distance between the new state s and s g . The aim of the agent in the gridworld is to minimize this function and the parameters could be then : r(s, a, s ) = -ϕ 2 = -d 2 (s , s g )

Q i = 0 (6)
This way, the agent is less and less punished by approaching the goal. The global policy is the shortest way toward the goal. Given our maze, this reinforcement is spurious as there are plenty of walls between the initial state and the goal. In particular, it entails an unlearning phenomenon after few trials. Indeed, if the agent was taken off toward a dead end (that moves the agent closer to the goal), it would get out of the trap only thanks to exploration because states are more and more attractive toward the goal. At the beginning, Q-values are near 0 so systematic exploration is strong : going out of the trap is possible. But after few trials, turning back is tantamount to choosing a less attractive Q-value and will happen only if several exploration actions follow one another, i.e. seldom. So, both reward shaping and progress estimators are risky. It's better to ban these approaches insofar as they may lead to a pernicious behavior.

Continuous reward function inspired by gaussian function

Consequently, we propose a continuous reward function such that on the one hand, r is uniform for some states far from the goal in order to avoid the unlearning phenomenon, and on the other hand, there is a reward gradient in a zone around the goal. We suggest the reward function inspired by the gaussian function :

r(s, a, s ) = βe -d(s ,sg ) 2 2σ 2 . ( 7
)
Q i values are uniform, β adjusts the amplitude of the function and σ, the standard deviation, specifies the reward gradient influence area. As a matter of course, "moving round in circles" behavior must be avoided, i.e. Q i ≥ β 1-γ . For the gridworld task, we have chosen Q i = 100 and β = 10. Fig. 2a shows the unlearning phenomenon as from 80 trials with σ = 3.53 . Indeed, the reward gradient influence area is too large and includes few dead ends. On the contrary, if the reward gradient influence area is only 6 steps around the goal (σ = 2), there won't be any unlearning phenomenons and the learning process is accelerated.

Such a continuous reward function is adjustable so as to avoid a harmful behavior. Anyway, the best approach would be to influence fleetingly the learning.

Goal Bias

In view of the importance of the action-value function initialization, we propose to be inspired by progress estimators in order to initialize the action-value func- tion with more precise information. In this section, the reward function is the binary one given by (3) with r ∞ = 0 and r g = 1.

We are going to settle a correct goal bias function thanks to our previous analysis. An interesting bias shall achieve an adjustable state gradient and in addition, must avoid the "moving round in circles" behavior. We suggest for instance a gaussian goal bias function :

Q i (s, a) = βe -d(s,sg ) 2 2σ 2 + δ + Q ∞ . (8)
δ fixes the level of systematic exploration far away from the goal, β the amplitude of the bias and σ the bias influence area. Concerning the gridworld, the bias is such that states near the goal are more and more interesting a priori than states far away from the goal. So δ and β must be chosen very small compared to one (in order to avoid too much systematic exploration). Fig. 2b presents goal bias results on the previous gridworld which are unambiguous. The goal bias leads to a much faster learning process. It is worth noticing that there is no problem concerning dead ends even if the bias is wrong. Contrary to Sect. 4.2, the effect of the goal bias function is transient. It advises the agent only at the beginning of the learning process.

Conclusion

Both reward shaping and progress estimators methods must be used cautiously to design a continuous reward function. Consequently, we have proposed a continuous reward function inspired by a gaussian function and whose reward gradient influence area is adjustable in order to deal with risks. Anyway, the best solution is to choose a suitable goal bias function that does not lead to any problems. Our analysis helps the choice of a correct goal bias function.

Experiments with the pendulum swing-up problem

Last of all, we validate our results on the continuous space control task of a pendulum swinging upwards with limited torque [START_REF] Doya | Reinforcement learning in continuous time and space[END_REF] (Fig. 3a). The control of this one degree of freedom system is non-trivial if the maximal output torque u max is smaller than the maximal load torque mgl. The controller has to swing the pendulum several times to build up enough momentum to bring it upright and has to decelerate the pendulum early enough to prevent it from falling over.

We have chosen a two-dimensional state space x = (θ, ω). 30 × 30 × 9 bases were used for the state-action space (θ, ω, u). Each trial was started from an initial state x(0) = (π, 0.1) and lasted 20 seconds. The sample time is 0.03 seconds. As a measure of the swing-up performance, we have chosen t up as the time in which the pendulum stayed up (|θ| < π/4). A trial was regarded as "successful" when t up is superior to the t up average of the 1000 last trials.

We tested the performance of the Q-Learning algorithm depending on the shape of the reward function and initial Q-values (Fig. 3b). In all cases, the t up average after 10000 trials is around 14 seconds.

We tested first the binary reward function

R(x, u, x ) = 1 if |θ | < π/4 0 otherwise (9)
with different uniform initial Q-values. With Q i = 0, the task was really difficult to learn (bar1) because the behavior far from the goal is random. A better performance concerning the binary reward and uniform Q i was observed with Q i > 0 (bar2), i.e. the followed behavior when |θ| > π/4 is systematic exploration. The policy drives the agent to unexplored areas which are assigned higher Q-values, i.e. the controller is spurred on to swing the pendulum upwards. Systematic exploration is the best strategy in this specific case.

Then, we tested goal bias with binary rewards : Q i (x) = βe -θ 2 2σ 2 +δ. Given our previous results, it is obvious that the goal bias function shall favor systematic exploration when |θ| > π/4, so we chose δ = 0.1, β = 1 and σ = 0.25 (bar3). Goal bias does not improve obviously the learning.

The system is a continuous space control task so the classical reward [START_REF] Doya | Reinforcement learning in continuous time and space[END_REF] is given by the height of the tip of the pendulum, i.e. R(x, u, x ) = cos(θ ) and the arbitrary Q i value is 0 (bar4). Thus, the pendulum moves round in circles when |θ| < π/2 and explores systematically when |θ| > π/2 at the beginning. The result is disappointing.

We applied our continuous reward function [START_REF] Koenig | The effect of representation and knowledge on goaldirected exploration with reinforcement-learning algorithms[END_REF] with Q i = 10 and β = 1. The distance is defined as d(x', x up ) = θ with x' the new state and x up the goal state. So the continuous reward function is Results (bar5) are near the case of a binary reward and systematic exploration. Lastly, we tried goal bias with continuous reward function. The reward function [START_REF] Doya | Reinforcement learning in continuous time and space[END_REF] is the continuous equivalent to the binary one so we have kept this one. The reward function is continuous, it is the same for Q i which must be higher than R(x) 1-γ . So goal bias is Q i (x) = β(1 + 1 1-γ )e -d 2 2σ 2 + δ. We have kept previous choices: δ = 0.1, β = 1 and σ = 0.25. This last simulation (bar6) is the better performance concerning the pendulum.

R(x, u, x ) = e -θ 2 2σ 2 . (10)

Conclusion

In this paper, we have called into question the arbitrary choice of the reward function and initial Q-values within the context of goal directed tasks. Our analysis has resulted in rules to correctly evaluate rewards and initial Q-values according to the desired behavior. Notably, some values of Q i lead to a detrimental behavior that must be avoided. Thanks to our experiments, we have confirmed the presence of bounds which mark out diverse behaviors. It is worth noticing that the farther Q i is from the bounds, the more the characteristic behaviors are distinguished.

Moreover, we advise to be wary of reward shaping or progressive estimators that may entail pernicious behavior. A safer adjustable continuous reward function is also suggested. At last, thanks to our conditions on the initial Qvalues, we developed a generic goal bias function, whose main feature is to be transient. This method turns out to be an effective way to improve the learning performance of goal-directed tasks. Table 1 recapitulates the better choices.

Fig. 1 .

 1 Fig. 1. On the left, non-deterministic 20 × 20 gridworld with a single start state (state [2, 2]) and a goal state (cheese) (state [14, 14]). On the right, gridworld experiments with different Qi averaged over 50 independent runs. Sreps to goal vs. trial number.

Fig. 2 . 2 .

 22 Fig. 2. Gridworld experiments averaged over 20 independent runs. Steps to goal vs. trial number. On the left, reward function inspired by gaussian function . Qi = 100 ; r(s, a, s ) = 10e -d(s ,sg ) 2 2σ 2 . On the right, goal bias function with binary rewards. Random test is Qi = 0. Goal bias function is Qi(s, a) = 0.001e -d(s,sg ) 2 2×13 2 .

Fig. 3 .

 3 Fig. 3. On the right, comparison of number of trials before one successful trial. The simulation lasted 10000 trials averaged over 10 independent runs. bar1: {binary reward ; Qi(x) = 0} bar2: {binary reward ; Qi(x) = 0.1} bar3: {binary reward and goal bias } bar4: {R(x, u, x ) = cos(θ ) ; Qi(x) = 0} bar5: {continuous reward and Qi(x) = 10 } bar6: {continuous reward and goal bias}

The rewards are discounted by a discount factor γ that controls the balance between the significance of immediate rewards and future rewards.

If several values are identical, the choice will be random among greedy actions.

i.e. states 10 steps away from the goal have uniform r.

Choice of goal biased initial Q-values :

Choice of goal biased initial Q-values :

+ δ δ ≥ 0 fixes the level of systematic exploration far away from the goal, β > 0 adjusts the amplitude of the gradient, σ specifies the gradient influence area and γ is the discount factor s is the previous state, s the new state, sg the goal state