Zwi Altman

Soumaya Sallem

Ridha Nasri

Berna Sayrac

Maurice Clerc

Particle Swarm Optimization for Mobility Load Balancing SON in LTE Networks

Keywords: Mobility Load Balancing, SON, Self-Organizing Networks, handover margin, LTE, Particle Swarm Optimization I

This paper presents a self-optimizing solution for Mobility Load Balancing (MLB). The MLB-SON is performed in two phases. In the first, a MLB controller is designed using Multi-Objective Particle Swarm Optimization (MO-PSO) which incorporates a priori expert knowledge to considerably reduce the search space and optimization time. The dynamicity of the optimization phase is addressed. In the second phase, the controller is pushed into the base stations to implement the MLB SON. The method is applied to dynamically adapt Handover Margin parameters of a large scale LTE network in order to balance traffic of the network eNodeBs. Numerical results illustrate the benefits of the proposed solution.

INTRODUCTION

The Self-Organizing Network (SON) technology has been receiving a growing interest for two main reasons. The Radio Access Networks (RAN) landscape is becoming increasingly complex and heterogeneous with coexisting and co-operating technologies. In parallel, network operators experience growing pressure to reduce operational expenditure (OPEX). The SON technology has been introduced as a means to manage complexity, to reduce cost of operation, and to enhance performance and profitability of the network. Self-organizing networks aim at autonomously configure newly deployed network nodes (self-configuration), tune parameters to improve Key Performance Indicators (KPIs) (self-optimization) and perform diagnostic and reparation of faulty network nodes (self-healing) [START_REF]Evolved Universal Terrestrial Radio Access Network (E-UTRAN); Self configuration and self-optimization network use cases and solutions[END_REF].

Self-optimization mechanisms are control loops that can be deployed in the management-or the control-plane. Control plane SON can be implemented in the eNodeBs (eNB) and benefit from high reactivity to traffic variations. For this reason, these SON solutions are sometime denoted on-line SON. On the other hand, management-plane (or off-line) SON solutions are deployed in the Network Management System (NMS), and are interfaced to the operator Operation and Maintenance Center (OMC) by means of APIs. Off-line SON solutions benefit from abundant data (metrics and KPIs) and computational means necessary for processing and running powerful optimization methods. The main drawback of the centralized approach is related to the long time scale that is typically used, in the order of an hour and more.

Mobility Load Balancing (MLB) aims at balancing network load by adapting mobility parameters. MLB has been mainly studied in the context of control-plane SON [START_REF] Rodriguez | Load balancing in a realistic urban scenario for LTE networks[END_REF][START_REF] Nasri | Handover adaptation for dynamic load balancing in 3GPP long term evolution systems[END_REF][START_REF] Combes | Self-organization in wireless network: a flow-level perspective[END_REF], and only a few contributions have addressed the management-plane MLB problem [START_REF] Tiwana | Troubleshooting of 3G LTE mobility parameters using iterative statistical model refinement[END_REF]. The aim of this paper is to study a hybrid approach for MLB SON that is performed in two phases. In the first, expert knowledge is combined with the Multi-Objective Particle Swarm Optimization (MO-PSO). The former allows to derive a parameterized form of the SON function (or the controller) which is then optimized by the MO-PSO in the management-plane. In the second phase, the controller is pushed into the control-plane, e.g. within eNBs, for on-line implementation. The advantage of such a solution is that it benefits from both computation resources available at the management-plane, and from high reactivity when implemented in the control-plane. It is noted that a two phase approach for self-optimization load balancing has been considered in the framework of Fuzzy Q-Learning (FQL) [START_REF] Muñoz | Optimization of a fuzzy logic controller for handover-based load balancing[END_REF][START_REF] Nasri | Fuzzy Q-learning based autonomic management of macrodiversity algorithms in UMTS networks[END_REF]. This approach can be viewed as a fully control-plane approach, where both learning and control (exploration and exploitation in the learning nomenclature) are performed on-line. It is noted that a hybrid learning approach could also be implemented. The utilization of expert (or a priori) knowledge has also an equivalence in the Reinforcement Learning (RL), namely the definition of a parameterized form of a policy which is then optimized during the learning process (e.g. Policy Gradient RL [START_REF] Combes | Self-Organizing relays: dimensioning, self-optimization and learning[END_REF]).

The paper is organized as follows: Section II introduces the system model. Section III presents the optimization approach for the static and dynamic cases, including the formulation of the MO-PSO, and the incorporation of expert knowledge to the solution. Simulation results for the MLB SON are presented in Section IV followed by concluding remarks in Section V.

II. SYSTEM MODEL

This section describes the LTE interference and Signal-to-Interference-plus-Noise Ratio (SINR) model used in this work. Similar modelling has been used in other contributions (see for example [START_REF] Nasri | Handover adaptation for dynamic load balancing in 3GPP long term evolution systems[END_REF]).

A. Interference and SINR model

Let I k,m denote the average downlink interference perceived by a mobile m connected to eNB k. The interference is generated by neighbouring eNBs utilizing the same frequency. The probability of eNB i to transmit on the same sub-carrier utilised by eNB k is equal to the load of eNB i, L i , defined as the ratio of the number of used Physical Resource Blocks (PRBs) to the total number of available PRBs. I k,m can then be written as [START_REF] Nasri | Handover adaptation for dynamic load balancing in 3GPP long term evolution systems[END_REF]:

( ) ∑ ≠ = k i m , i m , i i i m , k Q G P L i , k I Λ (1)
where Λ stands for the interference matrix and the matrix element Λ(k,i) equals one if cells k and i utilize the same frequency band and zero otherwise; P i denotes the downlink transmitted power per sub-carrier of eNB i; G i,m and Q i,m are the antenna gain and the path loss between eNB i and the mobile m, respectively. It is noted that for data services, when there is a single mobile in a cell, certain technological implementations allow to allocate the entire eNB frequency bandwidth (i.e. all available PRBs) to the mobile. In this case P i is the maximum eNB power and L i -the proportion of transmission time. The SINR of mobile m attached to eNB k is derived from (1) as follows [START_REF] Nasri | Handover adaptation for dynamic load balancing in 3GPP long term evolution systems[END_REF]:

( ) th m , k m , k m , k k m , k N I Q G P SINR + = (2)
N th being the thermal noise per sub-carrier. The adaptive modulation and coding scheme is determined from the SINR value through the perceived BLoc Error Rate (BLER). In the network simulator, the throughput per PRB for each user is determined as a function of the SINR via link level curves. It is assumed here that the user physical throughput equals N m times the throughput per PRB, where N m is the number of PRBs allocated to user m.

B.

Handover model 3GPP standard has specified hard handover for mobility in the LTE network however the implementation of the handover algorithm is not specified in the standard. A GSM-like hard handover algorithm is proposed. A handover condition on the trasmitted pilot power is defined as:

( ) Hysteresis i , k HM P P k i + ≥ - (3)
and i (in dB) respectively and HM(k,i) is the Handover Margin (HM) between eNB k and i. Hysteresis is a counter-measure that prevents ping-pong effect by providing a baseline handover threshold against power fluctuations due to channel variations. For the sake of simplicity, Hysteresis is set to zero and the baseline threshold value is integrated into HM(k,i). In addition to (3), the following two conditions should be verified:

The received power from the target eNB should be higher than a predefined threshold, and Enough resources should be available in the target eNB.

III. HANDOVER OPTIMIZATION

A. Expert knowledge

Expert knowledge refers to a priori knowledge on the optimization problem. It gives rough information (or tendency) on the type of parameter modification that will improve the system performance in different states of the system. Expert knowledge can be used to guide the optimization process and to reduce the search space. It is typically given in the form of a set of rules that relates qualitatively the system state to the parameter x. The system state is defined by a vector

( ) m u , , u u 1 =
of system indicators such as eNB load or interference, and often does not give direct information on the user perceived QoS.

As an example, assume that HM(i,j) depends on the load of station i, L i , and that of its neighbor j, L j . Denote by HM 0 the planning (or default) value used in the network planning process. Then the expert knowledge can be given in the form of the following four rules:

(i) If Low is L i and Low is L j then ( ) 0 HM to j , i HM set (ii) If High is L i and High is L j then ( ) 0 HM to j , i HM set (iii) If Low is L i and High is L j then ( ) value high to j , i HM set (iv) If High is L i and Low is L j then ( ) value low to j , i HM set
Rules (i) and (ii) prevent unnecessary handovers (ping-pong effects). Rule (iii) aims at helping the loaded eNB j by delaying handovers from eNB i. Rule (iv) aims at alleviating the loaded eNB i by advancing handovers towards eNB j.

In the general case, the parameter x is a function of the vector u . For example the parameter x stands for the Handover Margin (HM), and the vector u -for the load vector (L i ,L j ) (Figure 1). We write x as ( )

u surf x =
where surf stands for a multi-dimensional surface (i.e. the control function). Experience in HM optimization shows that the parameter function x varies smoothly with u . The expert knowledge given by a set of rules is used to guess a simple form for the function surf. Our aim is to find a parametric representation of surf, namely to write it as a function of a parameter vector p , ( )

m p , p p 1 =
with a few elements:

( ) p ; u surf x = (4)
Hence the form of the function surf is fully defined by the vector p that is determined via an optimization process. The function surf is denoted hereafter as the parameterization surface.

B. Multi-objective optimization model

Denote by ( )

( ) ( ) ( ) x f ,..., x f x F n 1 =
the objective function to be optimized where ( )

x f i
represents a KPI which depends on the parameter x. It is recalled that the parameter x has a parametric representation given by (4), hence F can be written as a function of p . The element p i is defined within an interval [

max i min i p , p
]. The optimization problem is a constraint multi-objective optimization problem and is written as follows:

( )

( ) ( ) ( ) ( ) [ ] m , , i ; p , p p : to subject x f , ... , x f , x f x F : maximize max i min i i n 2 1 1 = ∈ = (5)
Pareto optimal solutions are sought. A solution is Pareto optimal if the objective vector cannot be improved in any dimension without degradation in another dimension. The solution x dominates the solution x' if and only if

{ } ( ) ( ) { } ( ) ( )      > ∈ ∃ ≥ ∈ ∀ x' f x f n , , j and x' f x f , n , , i j j i i 1 1 (6)
A solution is said to be non-dominated if there exists no solution that dominates it. The set of non-dominated solutions within the entire search space constitute the Pareto optimal front.

C. HM Parameterization Surfaces

This section is devoted to the construction of two parameterization surfaces for the HM parameter. According to (4) we can write:

( ) ( ) p ; L surf j , i HM HM = (7)
where ( )

j i L L L , =
, L i and L j being the load of eNB i and of its neighbor j respectively. According to the expert knowledge and the four rules presented in Section III.A (see Fig. 1), we define the function surf via a linear interpolation of the four extreme points: HM(0,0), HM(1,0), HM(0,1) and HM [START_REF]Evolved Universal Terrestrial Radio Access Network (E-UTRAN); Self configuration and self-optimization network use cases and solutions[END_REF][START_REF]Evolved Universal Terrestrial Radio Access Network (E-UTRAN); Self configuration and self-optimization network use cases and solutions[END_REF]. It is noted that HM(0,0) and HM(1,1) can be chosen as the planning value HM 0 or as a value belonging to a small interval around HM 0 :

( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) j i j i HM L L , HM , HM , HM , HM L , HM , HM L , HM , HM , HM L surf ⋅ ⋅ - - + + ⋅ - + ⋅ - + = 0 1 1 0 1 1 0 0 0 0 1 0 0 0 0 1 0 0 (8)
Eq. ( 8) has the form of a polynomial in the loads L i and L j : ( )

j i j i HM L L b L b L b b p ; L surf 3 2 1 0 + + + = (9) with ( ) 3 2 1 0 b , b , b , b p =
. Figure 1 shows an example of the function

( ) p ; L surf HM
. The vector p is determined via an optimization process.

With the aim of further improving the solution for the HM, an exponential parameterization surface is investigated. This solution accentuates pushing (delaying) mobiles to make handovers for high (absolute value) load differences |L i -L j | between neighboring cells. The exponential type of variation for surf HM as a function of the load difference w,

j i L L w - =
, and is written as:

( ) ( ) ( )        < ⋅ - ≥ ⋅ = - 0

=

. A 3D plot of surf HM in [START_REF] Combes | Self-Organizing relays: dimensioning, self-optimization and learning[END_REF] with b=6, a 1 =20 and a 2 =20 is depicted in Figure 2. Results obtained using the optimized polynomial and exponential surfaces are presented in Section IV.

D. Particle Swarm Optimization

The Particle Swarm Optimization (PSO) method is used here to optimize the parameter vector p that defines the parameterization surface. The PSO method is a robust technique belonging to the category of Swarm Intelligence methods which is inspired by the social behaviour of flocking organisms [START_REF] Kennedy | Swarm Intelligence[END_REF][START_REF] Clerc | Particle Swarm Optimization[END_REF]. It utilizes a population of particles, each of which represents a solution, namely a parameter vector defining the parameterization surface. In the PSO notation, the position of a particle i, i p stands for the parameter we seek to optimize (i.e. the parameter vector defining the parameterization surface in the present work). The particles probe regions in the solution space in a partially random way. The exploration of a particle is described in terms of a velocity i v , which is added to the current position to bring the particle to its next position. Hence the velocity stands for the update brought to a current solution. The velocity of a particle comprises three components which depend on its past best position, i q , on the past best position of its neighbours, i g , and on its own (current) velocity (Figure 2). The optimization process is carried out as an optimization loop. At each iteration, the particles evolve using the following update equation applied to each dimension (element) d of the particle: [START_REF] Clerc | The particle swarm: Explosion, stability, and convergence in a multi-dimensional complex space[END_REF] where rand(.) denotes the random function. It has been shown that c 1 and c max can be derived analytically from a single parameter ϕ [START_REF] Clerc | The particle swarm: Explosion, stability, and convergence in a multi-dimensional complex space[END_REF], which has been set to 4.14 here. The problem at hand is multi-objective, with two objectives to be optimized, ( ) ( ) ( ) ( )

Particle

Following its own speed

Towards past own best performance

Towards past best performance of its neighbours

New position

( )( ) ( )( )      + ← - + - + ← d i d i d i d i d i max d i d i max d i d i v p p p g , rand c p q , rand c v c v 1 0 1 0 1
i i i p f , p f p F 2 1

=

: the network throughput and the access probability defined as the ratio between the number of successful network access attempts to the total number of network access attempts. For each new solution (particle) the network is simulated and ( )

i p F
is evaluated. In this work, the swarm dimension, namely the population size equals 10. The set of neighbors for each particle comprises 3 different particles chosen at random (a particle cannot serve as its own neighbor). Hence the vector ( ) i k g comprises 3 particles.

The pseudo code for the multi-objective PSO algorithm is given below:

Initialization For 1 = i to Population Size Initialize randomly i i q , p
, and list

of neighbours of i p : ( ) i k g , { } 3 2 1 , , k ∈ ; End Repeat for max N iterations For 1 = i to Population Size 1. If ( ) ( ) i i q F p F then i i p q = 2. ( ) i k k i g max g = 3.
For each dimension, update d i p using equation ( 11) End

The term ' ' determines the multi-objective characteristic of the algorithm. We say that the particle i p is better than j p , i.e. ( ) ( )

j i p F p F
, if the following condition is satisfied

( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) j i j i j i j i j i p f p f OR p f p f AND p f p f AND p f p f IF p F p F 2 2 1 1 2 2 1 1 > > ≥ ≥ ( 12
)
The condition [START_REF] Reyes-Sierra | Multi-Objective Particle Swarm Optimizers: A Survey of the State-of-the-Art[END_REF] states that the solution i p is better than j p if at least one of its objectives, f 1 or f 2 , is strictly bigger whereas for the other objective, the relation ' ≥ ' holds. In step 2 of the PSO algorithm, the maximum is calculated with respect to the operator defined in [START_REF] Reyes-Sierra | Multi-Objective Particle Swarm Optimizers: A Survey of the State-of-the-Art[END_REF]. If more than one non-dominated solution is found in a neighbouring set, i.e. for which the relation ( 12) is not verified, the first non-dominated solution encountered is chosen. This last simplification allows to use the multi-objective version of the PSO algorithm written above, which is simpler than the ones described in [START_REF] Reyes-Sierra | Multi-Objective Particle Swarm Optimizers: A Survey of the State-of-the-Art[END_REF].

E. Dynamic optimization

The optimization framework presented above can be directly extended to perform dynamic optimization of the HM parameter, HM(i,j), written hereafter as HM ij . In the dynamic optimization, HM ij becomes a function of time:

( ) ( ) ( ) p ; t L surf t HM HM ij = (13)
and is updated every ∆ t seconds (set here to 5 sec.) according to the load values that are used to sample surf HM .

Each solution for the surface ) ; ) ( ( p t L surf HM is evaluated as in the static optimization case, during a simulation period of 2000 (simulator) seconds, required to achieve convergence of the optimization objectives.

It is noted that the small dimension of the search space considered here allows to use the simple version of the PSO described in Section III.D in a dynamic context. For a bigger search space, more sophisticated "dynamic PSO" algorithms may be necessary with slower convergence properties [START_REF]Multi-Swarm Optimization in dynamic environments applications of evolutionary computing[END_REF].

IV. RESULTS

Consider a LTE network composed of 45 eNBs in a dense urban environment (see Figure 4). A semidynamic simulator is used to simulate the LTE network in the downlink. The simulator performs correlated Monte Carlo snapshots with a time resolution of a second to account for the time evolution of the network. FTP data traffic is considered. The principles of a semi-dynamic simulator are described in [START_REF] Samhat | Semi-dynamic simulator for large scale heterogeneous wireless networks[END_REF]. The Okumura-Hata propagation model is used for the 2GHz band. The path loss at a reference distance of 1 km and the path loss exponent are chosen as -128 dB and 3.76 respectively. Shadowing is modeled as a log-normal random variable with a 6 dB standard deviation. The spectrum efficiency depends on the SINR and on the Adaptive Modulation and Coding scheme used. The mapping between the SINR and the corresponding spectral efficiency is carried out using a quality table incorporated within the simulator. Fast fading is implicitly taken into account by the quality table.

Each eNB has 15 PRBs, corresponding to a 3 MHz bandwidth allocation. A frequency reuse factor of 3 is used. FTP calls are generated using a Poisson process of parameter 5 = λ arrivals per second. The arrivals are uniformly distributed within the network area. The non-uniform eNBs' positions results in highly non-uniform load distribution as can be seen from the histogram of Figure 6b. Each user is allocated between one and four PRBs to download a file of 10 Mbits. The duration of a communication depends on the allocated resources and on the user spectral efficiency (or throughput). The simulation parameters are summarized in Table I.

The HM parameterization surface is optimized using the multi-objective PSO algorithm described in Section III.D. Each particle (solution) corresponds to a distinct parameterization surface and is defined by the parameter vector p defined in Section III.C for the polynomial and the exponential surfaces. Two objectives are used to guide the PSO optimization: the total throughput of all eNBs in the network averaged over the simulation period, Throughput, and the probability of accessing the network, P access . For each particle the network is simulated during a period of T simu = 2000 (simulator) seconds to allow Throughput and P access indicators to converge. The PSO algorithm uses a population size of 10 particles and is repeated 30

= max N
iterations, namely 300 fitness (solution) evaluations are performed. Both static (eq. ( 7)) and dynamic (eq. ( 13)) optimization are performed. In the static optimization, the loads used to sample the optimization surface are considered fixed during the entire optimization process. These load values are the average loads calculated from a single simulation of the network with the planning HM value, 6 0 = HM dB. Figure 5 presents the MO-PSO results in the P access -Throughput plane for both static and dynamic optimization. The solution corresponding to the planning HM 0 value (denoted hereafter as the planning solution) is plotted using a yellow circle. The first Pareto front in red triangles corresponds to the static optimization with the polynomial parameterization surface function [START_REF] Nasri | Fuzzy Q-learning based autonomic management of macrodiversity algorithms in UMTS networks[END_REF]. The results for the Pareto front using the exponential parameterization surface [START_REF] Combes | Self-Organizing relays: dimensioning, self-optimization and learning[END_REF] are shown with blue squares in the Figure . The parameter b in ( 10) is fixed to the planning value of 6 dB and the parameters a 1 and a 2 are optimized by the PSO. The Pareto-front for the exponential solution clearly dominates that of the polynomial solution. The results for the dynamic optimization using the exponential parameterization surface [START_REF]Multi-Swarm Optimization in dynamic environments applications of evolutionary computing[END_REF] are depicted in green diamond in Figure 6. Each eNB adapts the HM ij parameters every five seconds by sampling the exponential parameterization surface at the corresponding load values L i and L j . One can see that dynamic optimization produces a better controller than the static optimization, and the performance gain using (13) improves the throughput and access probability by a few percent with respect to the planning solution. Denote the neighbor with which the eNB has the largest handover traffic exchange as the best neighbour. Figure 6a presents a histogram of HM ij of the eNBs and their corresponding best neighbours for a solution on the Pareto front for the polynomial parameterization surface in Figure 5. One can see that the optimization spreads the histogram to both low and high values. The optimized solution has many eNBs with low HM ij values (i.e. smaller than HM 0 ) allowing to advance the handovers. Several eNBs see their HM ij increase above 7 dB resulting in the delay of handovers towards loaded eNBs.

The load histogram for the planning solution with HM 0 =6 dB (white) and for the same optimized solution is depicted in Figure 6b. One can see that the MO-PSO reduces the number of eNBs with very high/low loads.

V. CONCLUSIONS

This paper has presented an efficient methodology for designing MLB SON controller. The method combines a priori expert knowledge with Multi-Objective Particle Swarm Optimization (MO-PSO), which allows to considerable reduce the search space and the computational time required for designing the MLB SON controller. The a priory knowledge provides the parameterized form of the controller which is optimized by the MO-PSO. It has been shown that dynamic optimization outperforms static optimization, namely produces better MLB-SON controller, which improve the throughput and access probability by a few percent with respect to the planning solution.

Fig. 1 .

 1 Fig. 1. Polynomial parameterization surface of HM in the form of a linear interpolation of the extreme points:HM(0,0), HM(1,0), HM(0,1) and HM(1,1), as a function of the loads of two neighboring eNBs.

Fig. 2 .

 2 Fig. 2. Exponential parameterization surface for HM.

Fig. 3 .

 3 Fig. 3. Update of a particle position in the solution space.

Fig. 4 .

 4 Fig. 4. A LTE network with 45 eNBs .

Fig. 5 .

 5 Fig.5. Pareto-front solutions for constant optimization with polynomial (triangles) and exponential (squares) parameterization surfaces, and for the dynamic optimization using the exponential surface (diamonds) in the Paccess-Throughput plane. The planning solution with HM0 of 6 dB is plotted using a circle.

Fig. 6 .

 6 Fig. 6. HM histogram for a solution on the Pareto front for the polynomial parameterization surface (a), and the corresponding load histogram for the planning (white) and an optimized (black) solutions (b).

TABLE I .

 I SIMULATION PARAMETERS.

	Parameter	Setting
	System bandwidth	3MHz
	Frequency reuse scheme	3
	Cell layout	45 eNBs, sectorized
	Inter-site distance	1.5 to 2 km
	PRB per eNB	15
	PRB assigned per mobile	1 to 4 (first-come first-serve basis)
	PRB transmit power	32 dBm
	Thermal noise density	-174 dBm/Hz
	Traffic type	FTP
	File size	10 Mbits
	Path Loss	128 + 37.6 log 10 (R), R in km
	Shadowing standard deviation	6 dB
	Mobility	40% of users are mobile
	Speed	30 km/h


Haithem Ayari

Florence Azais

Serge Bernard

Mariane Comte

Vincent Kerzérho

Michel Renovell

Enhancing Confidence in Indirect Analog/RF Testing against the Lack of Correlation between Regular Parameters and Indirect Measurements

Keywords: Test; Analog/RF integrated circuits, alternate test, indirect test, prediction error, machine-learning algorithm I

The greedy specification testing remains mandatory for analog and Radio Frequency (RF) integrated circuits because of the accuracy of the sorting based on these measurements. Unfortunately, to be implemented, this kind of testing method often incurs very high costs (expensive instruments, long test time...). A common approach, in the literature, is the so-called indirect/alternate test strategy. This strategy consists in deriving targeted specifications from low-cost Indirect Measurements (IMs). During the industrial test phase, the estimation of regular specifications using IMs is based on a correlation model that has been built previously, during a training phase. Despite the substantial test cost reduction offered by this strategy, its deployment in industry is limited, mainly because of a lack of confidence in the accuracy of estimations made by the correlation model. A solution to increase the confidence in the estimation of specifications using the indirect approach is to implement redundancy in the prediction phase. In this paper, we demonstrate that the redundancy implementation brings more than identifying rare misjudged circuits from a high-correlated model. Indeed redundancy massively increases the accuracy despite of the lack of accurate models that have been assumed in previous implementations of redundant indirect testing. This approach is illustrated on a real case study for which we have experimental measurements on a set of 10,000 devices.

INTRODUCTION

The efficiency of a mixed-signal electronics system relies on the performances of all the components. The global performance of such a system is close to the one of the component in the path with the lowest performance. Electronics system manufacturer also called in the industry Original Equipment Manufacturer (OEM) builds a system on the base of performances given in the datasheet of the components. As a consequence for the analog/RF Integrated Circuit (IC) provider, to ensure the confidence of the OEM in the performances of the components, it is mandatory to test these specifications after manufacturing.

Historically, testing analog/RF integrated circuits aims at measuring the majority or totality of the circuit performances/specifications defined in its datasheet. Thanks to testing, the manufacturer is able to sort the manufactured circuits as good or bad regarding its datasheet. According to 2011 edition of the International Technology Roadmap for Semiconductors (ITRS) report [1] on test and test equipment, the test cost for analog and mixed-signal is a great concern. "These trends of increasing ATE (Automated Test Equipment) instrument channel count, complexity, and performance are expected to continue, but at the same time the cost of test must be driven lower".

In order to lessen the burden of specification testing, a promising solution is the so-called indirect/alternate testing, in which the results of specification testing are derived from a set of few Indirect Measurements (IMs) obtained with low-cost test equipment. The basic principle is, during a first phase, to use a limited set of devices in order to train a non-linear regression algorithm used to learn the mapping between some indirect measurements and the circuit performance parameters. For this first phase, it is mandatory to measure the circuit performance parameters and the IMs. The second phase is called the production testing phase. It aims at measuring the performances of a huge lot of new devices. Then during this phase, only the low-cost IMs are performed and used to feed the previously-trained regression algorithm in order to predict device specification or to classify the devices as good or bad. As a consequence thanks to indirect testing, it is possible to significantly decrease the number and complexity of measurement configurations. Despite the clear advantages of employing such approach and a number of convincing attempts to prove its efficiency [START_REF] Variyam | Enhancing test effectiveness for analog circuits using synthesized measurements[END_REF][START_REF] Variyam | Prediction of analog performance parameters using fast transient testing[END_REF][START_REF] Voorakaranam | Signature Testing of Analog and RF Circuits: Algorithms and Methodology[END_REF][START_REF] Bhattacharya | A built-in loopback test methodology for RF transceiver circuits using embedded sensor circuits[END_REF][START_REF] Haider | Low-cost alternate EVM test for wireless receiver systems[END_REF][START_REF] Bhattacharya | Test time reduction for ACPR measurement of wireless transceivers using periodic bitstream sequences[END_REF][START_REF] Sheng; Kerzérho | Predicting dynamic specifications of ADCs with a low-quality digital input signal[END_REF][START_REF] Han; Chatterjee | Robust Built-In Test of RF ICs Using Envelope Detectors[END_REF][START_REF] Bhattacharya | Use of embedded sensors for built-in-test RF circuits[END_REF][START_REF] Abdallah | Sensors for built-in alternate RF test[END_REF][START_REF] Ellouz | Combining Internal Probing with Artificial Neural Networks for Optimal RFIC Testing[END_REF][START_REF] Barragan | Improving the Accuracy of RF Alternate Test Using Multi-VDD Conditions: Application to Envelope-Based Test of LNAs[END_REF][START_REF] Ayari | Smart selection of indirect parameters for DC-based alternate RF IC testing[END_REF][START_REF] Stratigopoulos | Nonlinear decision boundaries for testing analog circuits[END_REF][START_REF] Stratigopoulos | Non-RF to RF Test Correlation Using Learning Machines: A Case Study[END_REF][START_REF] Stratigopoulos | Defect Filter for Alternate RF Test[END_REF][START_REF] Stratigopoulos | Error Moderation in Low-Cost Machine-Learning-Based Analog/RF Testing[END_REF][START_REF] Stratigopoulos | Enrichment of limited training sets in machine-learning-based analog/RF test[END_REF][START_REF] Kupp | On proving the efficiency of alternative RF tests[END_REF][START_REF] Ayari | On the Use of Redundancy to Reduce Prediction Error in Alternate Analog/RF Test[END_REF], the deployment of the indirect test strategy in industry is limited. A solution to enhance the confidence in the test method is to implement a strengthened strategy based on prediction model redundancy [START_REF] Ayari | On the Use of Redundancy to Reduce Prediction Error in Alternate Analog/RF Test[END_REF]. According to [START_REF] Ayari | On the Use of Redundancy to Reduce Prediction Error in Alternate Analog/RF Test[END_REF], the combination of the predictions of three models with strong correlations between IMs and specifications, allows chasing the rare misjudged circuits. These rare misjudged circuits are commonly highlighted as the source of lack of confidence in the indirect strategy. The purpose of the publication is to demonstrate a new advantage of implementing redundancy into indirect test strategy. Indeed more than chasing rare misjudged circuits, redundancy can be efficient in the case that there are only available weak models in terms of correlation between IMs and specifications. This paper is organized as follows. In section II, the general background of indirect testing is described and previous works are summarized. Section III introduces the problem of the evaluation of indirect test quality and shows that although a good average prediction error can be achieved, the maximal prediction error cannot be guaranteed. Section IV presents the proposed strategy to reduce the maximal prediction error based on the use of redundancy and a two-tier test scheme. This strategy is evaluated in section V on a real case study. Finally, section VI concludes the work.

II. BACKGROUND

A. Indirect/alternate test principle

The purpose of indirect testing is to relax constraints on the number and complexity of industrial test configurations by using low-cost indirect measurements in order to infer the results of specification testing. As presented by figure 1, the underlying idea of indirect testing is that manufacturing process variability that affect the conventional performance parameters of the device also affect non-conventional low-cost indirect parameters. If the correlation between the indirect parameter space and the performance parameter space can be established, then specifications may be verified using only the low-cost indirect signatures. Unfortunately the relation between these two sets of parameters is complex and cannot be simply identified with an analytic function. The solution commonly implemented uses the computing power of machine-learning algorithms. Two main directions have been explored for the implementation of indirect testing, i.e. prediction-oriented strategy [START_REF] Variyam | Enhancing test effectiveness for analog circuits using synthesized measurements[END_REF][START_REF] Variyam | Prediction of analog performance parameters using fast transient testing[END_REF][START_REF] Voorakaranam | Signature Testing of Analog and RF Circuits: Algorithms and Methodology[END_REF][START_REF] Bhattacharya | A built-in loopback test methodology for RF transceiver circuits using embedded sensor circuits[END_REF][START_REF] Haider | Low-cost alternate EVM test for wireless receiver systems[END_REF][START_REF] Bhattacharya | Test time reduction for ACPR measurement of wireless transceivers using periodic bitstream sequences[END_REF][START_REF] Sheng; Kerzérho | Predicting dynamic specifications of ADCs with a low-quality digital input signal[END_REF][START_REF] Han; Chatterjee | Robust Built-In Test of RF ICs Using Envelope Detectors[END_REF][START_REF] Bhattacharya | Use of embedded sensors for built-in-test RF circuits[END_REF][START_REF] Abdallah | Sensors for built-in alternate RF test[END_REF][START_REF] Ellouz | Combining Internal Probing with Artificial Neural Networks for Optimal RFIC Testing[END_REF][START_REF] Barragan | Improving the Accuracy of RF Alternate Test Using Multi-VDD Conditions: Application to Envelope-Based Test of LNAs[END_REF][START_REF] Ayari | Smart selection of indirect parameters for DC-based alternate RF IC testing[END_REF] or classification-oriented strategy [START_REF] Stratigopoulos | Nonlinear decision boundaries for testing analog circuits[END_REF][START_REF] Stratigopoulos | Non-RF to RF Test Correlation Using Learning Machines: A Case Study[END_REF][START_REF] Stratigopoulos | Defect Filter for Alternate RF Test[END_REF][START_REF] Stratigopoulos | Error Moderation in Low-Cost Machine-Learning-Based Analog/RF Testing[END_REF][START_REF] Stratigopoulos | Enrichment of limited training sets in machine-learning-based analog/RF test[END_REF][START_REF] Kupp | On proving the efficiency of alternative RF tests[END_REF] as illustrated in figure 2. In the first direction, the training set is used to derive functions that map the low-cost indirect measurements to the performance parameters (typically using statistical regression models or artificial neural networks). The objective is actually to predict the individual performance parameters of the device; subsequent test decisions can then be taken by comparing predicted values to specification tolerance limits. In the second direction, the training set is used to derive decision boundaries that separate nominal and faulty circuits in the low-cost indirect measurement space (specification tolerance limits are therefore part of the learning algorithm). The objective is actually to perform the classification of each circuit as a good circuit or a faulty circuit, but without predicting its individual performance parameters.

B. Previous works addressing analog/RF indirect testing

The fundamental principle of indirect testing, described in the previous sub-section, is common to all contributions found in the literature and have been applied to various types of devices, i.e. analog, mixed-signal and RF devices. Over this basic principle, there are several aspects of the strategy that have been considered in the literature.

At first the machine-learning algorithm should be defined. Different solutions have been proposed, based either on statistical regression models [START_REF] Variyam | Enhancing test effectiveness for analog circuits using synthesized measurements[END_REF][START_REF] Variyam | Prediction of analog performance parameters using fast transient testing[END_REF][START_REF] Voorakaranam | Signature Testing of Analog and RF Circuits: Algorithms and Methodology[END_REF] or artificial neural networks (ANN) [START_REF] Ellouz | Combining Internal Probing with Artificial Neural Networks for Optimal RFIC Testing[END_REF]. Then in order to feed the machine-learning algorithm there is a need of signatures giving strong information on the process variability. To gather relevant signatures, research developments have been done in order to define and optimize some stimuli: piece-wise linear signals [START_REF] Variyam | Prediction of analog performance parameters using fast transient testing[END_REF][START_REF] Voorakaranam | Signature Testing of Analog and RF Circuits: Algorithms and Methodology[END_REF], multitone signal [START_REF] Haider | Low-cost alternate EVM test for wireless receiver systems[END_REF], digital bitstream [START_REF] Bhattacharya | Test time reduction for ACPR measurement of wireless transceivers using periodic bitstream sequences[END_REF][START_REF] Sheng; Kerzérho | Predicting dynamic specifications of ADCs with a low-quality digital input signal[END_REF]. In case of complex signatures, some authors also propose to process these signatures to extract pertinent information using for instance Fast Fourier Transform (FFT) [START_REF] Voorakaranam | Signature Testing of Analog and RF Circuits: Algorithms and Methodology[END_REF][START_REF] Haider | Low-cost alternate EVM test for wireless receiver systems[END_REF][START_REF] Bhattacharya | Test time reduction for ACPR measurement of wireless transceivers using periodic bitstream sequences[END_REF] or wavelet analysis [START_REF] Han; Chatterjee | Robust Built-In Test of RF ICs Using Envelope Detectors[END_REF]. Finally, another approach to gather pertinent information consists in inserting elements within the circuit such as sensors [START_REF] Bhattacharya | A built-in loopback test methodology for RF transceiver circuits using embedded sensor circuits[END_REF][START_REF] Han; Chatterjee | Robust Built-In Test of RF ICs Using Envelope Detectors[END_REF][START_REF] Bhattacharya | Use of embedded sensors for built-in-test RF circuits[END_REF][START_REF] Abdallah | Sensors for built-in alternate RF test[END_REF] or DC probes associated with an analog test bus [START_REF] Abdallah | Sensors for built-in alternate RF test[END_REF][START_REF] Ellouz | Combining Internal Probing with Artificial Neural Networks for Optimal RFIC Testing[END_REF], or exploiting different test conditions such as multiple Vdd conditions [START_REF] Ellouz | Combining Internal Probing with Artificial Neural Networks for Optimal RFIC Testing[END_REF][START_REF] Barragan | Improving the Accuracy of RF Alternate Test Using Multi-VDD Conditions: Application to Envelope-Based Test of LNAs[END_REF].

Another aspect regarding the practical implementation of indirect testing deals with the selection of the appropriated indirect measurements in case of a large number of indirect measurements is available. This aspect is addressed in [START_REF] Bhattacharya | Use of embedded sensors for built-in-test RF circuits[END_REF][START_REF] Ayari | Smart selection of indirect parameters for DC-based alternate RF IC testing[END_REF] in case of prediction-oriented strategy and in [START_REF] Stratigopoulos | Nonlinear decision boundaries for testing analog circuits[END_REF][START_REF] Stratigopoulos | Non-RF to RF Test Correlation Using Learning Machines: A Case Study[END_REF] in case of classification-oriented strategy.

Specification Tolerance Limits

Finally other essential points for the deployment of indirect testing deal with confidence in test predictions and efficiency evaluation. Regarding prediction confidence, a defect filter in presented in [START_REF] Stratigopoulos | Defect Filter for Alternate RF Test[END_REF] that permits to screen out outliers, i.e. devices not consistent with the statistical distribution of manufactured devices and for which indirect test prediction should not apply.

An interesting approach is also proposed in [START_REF] Stratigopoulos | Error Moderation in Low-Cost Machine-Learning-Based Analog/RF Testing[END_REF] in the context of classification-oriented strategy, that relies on the use of guard bands in the indirect measurement space in order to identify devices suspect to misclassification. Regarding efficiency evaluation, one of the main problems is that only a limited number of circuit instances are usually available for validation.

Recent works have been reported in [START_REF] Stratigopoulos | Enrichment of limited training sets in machine-learning-based analog/RF test[END_REF][START_REF] Kupp | On proving the efficiency of alternative RF tests[END_REF] that permit to generate a large number of synthetic device instances from a comparatively small set of representative devices.

III. ON THE USE OF REDUNDANCY

As previously explained, two main strategies are possible for the implementation of indirect testing: prediction-oriented strategy or classification-oriented strategy. In this paper, we focus on the prediction-oriented strategy. The main advantage of this strategy is that it provides a prediction of the individual performance parameters. This information can then be used to monitor possible shift in process manufacturing, adjust test limits during production phase if necessary, or perform multibinning. Model redundancy is complementary strategy to regular indirect testing technique. It has been firstly introduced in [START_REF] Ayari | On the Use of Redundancy to Reduce Prediction Error in Alternate Analog/RF Test[END_REF].

The problem highlighted in [START_REF] Ayari | On the Use of Redundancy to Reduce Prediction Error in Alternate Analog/RF Test[END_REF] is related to the efficiency evaluation of prediction-oriented indirect test methods. The efficiency is usually expressed in terms of prediction accuracy, and more particularly in terms of average prediction error.

Many of the experiments reported in the literature on various devices demonstrate that very low average prediction error can be achieved, typically 2%. However, two main points limit the credit that can be given to this good accuracy. Firstly, low average prediction error does not guarantee low maximal prediction error, which is of crucial importance regarding the classification step which consists in comparing the predicted values to the specification limits indicated in the datasheet. Secondly, evaluation is usually performed on a small set of validation devices, typically ranging from few hundreds to one thousand instances, while the technique aims at predicting values for a large set of fabricated devices, typically one or several millions of devices. So even if low maximal prediction error can be observed on the small validation set, there is no guarantee that the maximal prediction error will remain in the same order of magnitude when considering the large set of devices under test.

Based on measurments made on 10,000 devices, and using the IMs selection process described in [START_REF] Ayari | Smart selection of indirect parameters for DC-based alternate RF IC testing[END_REF], it has been pointed out [START_REF] Ayari | On the Use of Redundancy to Reduce Prediction Error in Alternate Analog/RF Test[END_REF] an important weakness of the prediction-based indirect test method: although it provides accurate prediction results for most of the devices, aberrant predictions are observed for a very small number of devices. In addition aberrant predictions happen regardless of the training and/or validation set size, and that such errors do not depend on the device itself since the same device may exhibit correct or aberrant prediction depending on the considered regression model. As a result, it seems extremely difficult to foresee such prediction errors. Obviously, this is a serious obstacle for the deployment of the strategy in an industrial context, where one or several millions of devices have to be processed by the test flow.

In this context, the strategy described in [START_REF] Ayari | On the Use of Redundancy to Reduce Prediction Error in Alternate Analog/RF Test[END_REF] consists in introducing an additional step in the indirect test flow in order to provide an indication on the confidence that can be placed on the prediction; if this confidence is low, the device is removed from the indirect test flow for further action to be taken. This approach is similar to the approach suggested in [START_REF] Stratigopoulos | Error Moderation in Low-Cost Machine-Learning-Based Analog/RF Testing[END_REF] in case of classification-oriented strategy, where guard-bands are allocated in the indirect measurement space in order to identify devices for which the indirect test decision is prone to error. Note that this solution does not apply in case of prediction-oriented strategy because test decisions are not taken in the indirect measurement space but in the performance parameter space by comparing predicted values to specification tolerance limits.

As it has been previously explained, different regression models may lead to aberrant predictions for different devices. As a consequence the indication of confidence in the estimation of a parameter for a given device is based on the crosschecking of the predictions obtained with different models. A device whose performance predictions are similar whatever the regression model used is likely to be properly predicted. On the contrary, when different models lead to different performance predictions for the same device, we can suspect that at least one of the models does not predict the performance correctly. Unfortunately, it is impossible to deduce which one of the predictions is correct and which is not. As a consequence the prediction for this device cannot be trusted, the prediction is considered suspect and the device is removed from the indirect test flow. In other words, model redundancy is used to distinguish reliable predictions from suspicious predictions. This strategy, called two-tier indirect test scheme using model redundancy, is illustrated in figure 5. In the classical implementation of the prediction-oriented indirect test, one regression model is built during the training phase for each specification; these models are then used during the testing phase to predict device specification values (see figure 2.a). In the proposed new implementation, 3 regression models that involve different combinations of indirect measurements are built during the training phase, for each specification. During the testing phase for each specification, 3 predicted values are therefore computed using the 3 models derived in the previous phase; prediction confidence is then established by checking the consistency between these 3 predictions. More precisely for each pair of models, the difference between the predicted values is computed and checked against a threshold value ε. If all these differences are inferior to the threshold, it means that there is no discrepancy between the values predicted by the 3 models and the specification prediction is considered reliable. On the contrary, if one (or more) of these differences is superior to ε, the prediction is considered suspect. Two scenarios are then possible:

• Predictions are considered reliable for all specifications. In this case the device is directed to the first tier, where device performances are computed by averaging the 3 predicted values, for each specification. • Prediction is suspect for one (or more) specification. In this case, the device is removed from the indirect test flow and directed to the second tier, where further testing may be applied to characterize the device (for instance standard specification testing).

Note that this strategy based on model redundancy assumes that it is possible to build several accurate models involving different combinations of indirect measurements. Some experiments have been driven on a large number of devices for a given case study in order to demonstrate the efficiency of the model redundancy for indirect testing. Figure 6 and figure 7 present the results for two RF parameters (1dB compression point, CP1 and 3 rd order intercept point, IP3) of the considered power amplifier. For each figure, is presented the scatter plot of the estimated performance versus the true value for the three prediction models and the scatter plot after removing the suspect estimations identified thanks to redundancy. The blue stars are the devices used for training, the red stars are the devices used for validation and the green stars the devices identified as suspect thanks to redundancy.

IV. ACCURATE ESTIMATION USING REDUNDANCY DESPITE OF LOW-CORRELATION IMS

The previous section has introduced the concept of redundancy and demonstrated its efficiency to identify suspicious predictions. The circuits that have a suspicious prediction are not directly sorted as good or bad according to comparison between the value predicted and the test limits. The circuit should go through an additional test phase. This second test phase is not described here because it should be defined according to the targeted yield or the total test cost. The circuits can be dumped or tested using a conventional test methodology. The redundancy applied to indirect testing strategy improves considerably the confidence in the predicted RF specification value. As explained previously the strategy described in [START_REF] Ayari | On the Use of Redundancy to Reduce Prediction Error in Alternate Analog/RF Test[END_REF] and summarized in the previous section, that relies on model redundancy, improves considerably the confidence in the predicted RF specification value, but assumes that it is possible to build several accurate models involving different combinations of indirect measurements. This assumption can be justified by the potential wide range of IMs. Unfortunately there are cases for which the variety of potential IMs and the correlation between these IMs can be limited. Indeed the quality of an RF path is very sensitive to intrusion, and measuring IMs can require probing the RF path. Another reason is the potential additional cost of implementing circuitry for testing purpose. The limited number of IMs induces a lack of correlation between IMs and specifications and consequently models are not accurate enough. In this section, we will demonstrate that redundancy also improve the accuracy and the prediction confidence in the case of non-accurate models. This will give more flexibility in IMs set selection and will give a kind of confidence even though we deal with non-accurate models.

To illustrate these points, we have performed some experiments on a case study for which we have experimental test data on a large number of instances. The case study is a Power Amplifier (PA) fabricated by NXP Semiconductors. The experimental test data, from 10,000 devices, include 37 low-cost Indirect Measurements (IMs) based on standard DC tests and 2 RF performance measurements, namely the 1dB compression point (CP1) and 3 rd order intercept point (IP3). Because it is a case study, there is available a wide variety of IMs. Thanks to this variety, it is possible to study the impact of the range of the variety and the related correlation between IMs and RF specifications to estimate. These data are separate in two sets of 5,000 devices, one that can be used for training and the other for validation. All the models that are used in the following study have been trained using the same training set made of 5,000 devices. In [START_REF] Ayari | On the Use of Redundancy to Reduce Prediction Error in Alternate Analog/RF Test[END_REF] two optimum sets of 3 IMs from a global set of 37 IMs have been used to train the prediction models for the CP1 and the IP3. These two sets are selected using the method described in [START_REF] Ayari | Smart selection of indirect parameters for DC-based alternate RF IC testing[END_REF]. If we look at all possible solution using 3 IMs we will have 7,770 different models. The huge number of combinations of IMs induces a large variety of combinations in terms of prediction accuracy. The Figure 11 shows the histogram of the average prediction error observed on training set for all the 7,770 models. In these two figures, we identify some models having an average prediction error below the targeted value of 1%. In the two cases (i.e. CP1 and IP3) the accuracy of the majority of the models is above 1%. Also we have to notice that the targeted value of the average prediction error observed on training set is not guaranteed on the validation set. So, using models which have an average prediction error above the target is strongly not recommended. In our case we are fortunate to have models that fulfill the required target. But as explained previously, in many cases this could not be the case. For the purpose of this study, we propose to select randomly three models which have an average prediction error observed on training set exceeding 1%. The selected models will be used according to the strategy proposed in the previous section. The same specification will be predicted using these models, the circuits having a difference between the three predictions exceeding a given threshold will be removed from the main test flow. For the other circuits the prediction of the specification will be the average of the three predictions. This operation will be repeated 5,000 times in order to obtain a set of 5,000 combinations of randomly-chosen models. In order to achieve a precise analysis, for every draw, the average prediction error and the maximum prediction error of the "Mean of models" (after removing suspect prediction) and the single models will be compared. Figure 12 It clearly appears that over the 5,000 draws, there are a limited average prediction error observed on validation set when the redundancy strategy has been implemented. On the other side, the average prediction error of the best model is predominantly over 5% value and we can observe a large variance over the 5,000 draws. We can conclude that those "best" models (over the three used for redundancy), which have an average prediction error over 1% during training are not eligible for a regular indirect test because of a lack of correlation between the IMs and the wanted specifications.

To illustrate clearly this fact, we will rearrange the models regarding their ascending order of their average prediction error. The Figure 13 and Figure 14 respectively illustrate the average prediction error and the maximum estimation error of the worst models, the best models (over the three drawn randomly) and the Mean of models over the 5,000 iterations. The figures are zoomed in. This zoom is mandatory because of enormous average prediction error over #4,500 and enormous maximum prediction error over #3,000. As the values are sorted in ascending order, we can expect that values that are not presented are over the maximum value of the ordinate. For the two single models, the average prediction error is usually greater than that observed for the Mean of models after removing suspect predictions. We can observe in the case of the CP1 that only 60% of time, for the best model, the average prediction error on the validation set is less than 2%. If we focus on the result of the Mean of models, we can identify clearly that 100% of time the average prediction error is bellow 2%. The same observation is done with the second specification i.e. IP3. The benefit of the proposed method is clearly highlighted in this case for which the aberrant predictions are efficiently limited. In the case of the CP1, we see that in the best case only 30% of times the maximum prediction error is below 2 dBm.

But we reach easily 99% of times with the "Mean of models". In the case of the IP3 specification the percentage of occurrences for which the maximum prediction error is below 2dBm is around 75% against the 20% in the best case when using only one model.

The implementation of redundancy with accurate models enables to provide strong confidence in accurate estimations. In addition the number of suspicious estimations remains very low. In the case study presented in [START_REF] Ayari | On the Use of Redundancy to Reduce Prediction Error in Alternate Analog/RF Test[END_REF], the number of suspicious estimations using different accurate models was between 0.05 and 0.1%. As demonstrated earlier, redundancy strategy provides the possibility to relax the constraint of cost of IMs to build prediction models with less correlation but with same confidence in the accurate estimations. The main consequence is the increased number of suspicious estimations. Graphic results are provided for CP1 and IP3 parameters respectively by Figure 15 a) and Figure 15 b). Numerical results are stated in Table 1.

(a) Case of CP1 (b) Case of the IP3 Figure 15: number of suspicious estimations over the 5000 iterations Figure 15 presents the number of suspicious estimations for the combinations sorted in ascending order of the averaged estimation error of the Mean of models (cf. blue line of Figure 13). Like in Figure 13 and 14, there is a small gain which is coherent with the constant increase of curves in Figures 13 and14. But according to Figure 15 and to Table 1, there is only a small increase of the number of suspicious estimations compared to the case describes in [START_REF] Ayari | On the Use of Redundancy to Reduce Prediction Error in Alternate Analog/RF Test[END_REF], but it remains low. Indeed the average number of suspicious estimations for CP1 and IP3 are respectively 48.3 and 18.9 over 5,000 DUTs, which corresponds to 0.966%. Moreover for a maximum number of suspicious errors of 173 over 5,000 DUTs, it makes 3.46% of the circuits for which the sorting as good or bad according to the indirect approach cannot be done. A yield loss between 1 and 4% is a value usually accepted in the domain of industrial testing. As explained previously, the objective of the redundant strategy is to sort the estimations as accurate or suspicious.

Conventional sorting of DUTs can be achieved according to the accurate estimation of test parameters. The DUTs, for which the estimation of the test parameters is suspicious, will go through another test or will be dumped. The second step of the whole test strategy is particular for each product. The proposed approach will give great flexibility for test engineer to choose the set of used IMs to implement the indirect test. Moreover the results presented in this publication demonstrate that the redundant strategy makes the indirect test method still applicable to the test case for which there is a limited number of IMs or low correlation between IMs and conventional test parameters.

V. CONCLUSION

In this paper we have enhanced the confidence in using indirect testing combined with model redundancy in the context of a prediction-oriented strategy. It has been demonstrated that this strategy is efficient for any test case, even when the variety of IMs is limited for maintaining the quality of the RF function or simply for economic reasons aiming at limiting the amount of circuitry not dedicated to the core function. Indeed the low variety of IMs is an issue because it induces a lack of correlation between IMs and regular specifications; as a consequence it is impossible to find a combination of IMs enables to train an accurate prediction model. This problem is overcome by implementing model redundancy.

The proposed approach has been validated on a real case study for which we have experimental measurements on a set of 10,000 devices. Results show that most of the devices can be accurately predicted with the indirect testing strategy, with a maximal prediction error in the same range than the one observed on the training set. Only a very small number of the devices are directed to the second tier, therefore incurring very low test cost overhead. First we have used accurate models to implement the proposed method. Then, we have demonstrated that it can works also with less accurate models. This gives more flexibility for the best choice of IMs.

ACKNOWLEGMENTS

Figure 1 .

 1 Figure 1. Underlying idea of indirect/alternate testing.

Figure 2 .

 2 Figure 2. Alternate test principle: prediction-oriented strategy (a) and classification-oriented strategy (b).

Figure 5 .

 5 Figure 5. Flow diagram of the proposed two-tier indirect/alternate test scheme using model redondancy.

Figure 6 .Figure 7 .

 67 Figure 6. Correlation plots for CP1 prediction using 3 different models and after removal of suspect predictions (ε=1dBm)

  (a) case of CP1 (b) case of IP3 Figure 11. Histogram of average prediction error observed on training set

  illustrates the average prediction error observed on validation set for the used model and the "Mean of the models". Here the illustrated single model is the best one: i.e. the model which has the lowest prediction error observed on training set of the three models used for redundancy.

Figure 12 .

 12 Figure 12. Average prediction error on validation set

Figure 13 .

 13 Figure 13. Average prediction error of the worst, best model and mean of models over the 5000 iterations (values sorted in ascending)

Figure 14 .

 14 Figure 14. Maximum prediction error of the worst and best model against Mean of models over the 5000 iterations (values sorted in ascending)

  Table1: Maximum, average and standard deviation of number of suspicious estimations over the 5000

The authors would like to thank C. Kelma from NXP Semiconductors for providing the PA production test data and for fruitful discussions.