Ali Al Ghouwayel

Emmanuel Boutillon
email: emmanuel.boutillon@univ-ubs.fr

A Systolic LLR Generation Architecture For Non-Binary LDPC Decoders

Non-Binary LDPC codes offer higher performances than their binary counterpart but suffer from higher decoding complexity. A solution to reduce the decoding complexity is the use of the Extended Min-Sum algorithm. The first step of this algorithm requires the generation of the first n m largest Log-Likelihood Ratio (LLR), sorted in increasing order, of each received symbol. In the case where GF(q) symbols are transmitted using a BPSK modulation, we propose a simple systolic architecture that generates the sorted list of symbols.

I. INTRODUCTION

Non-binary Low Density Parity Check (NB-LDPC) codes over GF(q), with q > 2, are known to outperform binary LDPC codes for short and medium length [START_REF] Davey | Low density parity check codes over GF(q)[END_REF]. Moreover, for high spectral efficiency modulation, channel symbol can be mapped directly into code symbol, avoiding the information loss generated by the marginalisation process used in the binary LDPC case (gain of 1.6 dB, in the MIMO case, are reported in [START_REF] Pfletschinger | Getting Closer to MIMO Capacity with Non-Binary Codes and Spatial Multiplexing[END_REF]).

These advantages of such a Non-Binary coding scheme come at the expense of increased hardware complexity. However, a recent decoding algorithm called Extended Min-Sum (EMS) Algorithm was proposed by Declercq [START_REF] Declercq | Decoding algorithms for Nonbinary LDPC codes over GF(q)[END_REF] which presents a significant reduction of the computation complexity of the decoding process while keeping a good performance. This algorithm, instead of treating the q LLR values associated to the complete list of GF(q) symbols, considers the sorting of the LLR messages and only treats the n m greatest values and proposes an offset compensating the truncated messages.

One of the key components of the NB-LDPC decoder implementing the EMS algorithm is the LLR generation circuit and its accompanying sorter. This paper considers the design and the hardware implementation of an efficient circuit dedicated to generate the LLR values sorted in increasing/decreasing order.

The rest of this paper proceeds as follows. In section II, we review some basic notions and definitions related to the LLR computation and we define a new simplified formula allowing the generation of the LLR values in an ordered list. In Section III, we discuss the proposed sorting algorithm. Section IV describes the hardware design and gives a complexity evaluation of the proposed LLR generation circuit implemented on Virtex 4 FPGA devices. Finally, a conclusion ends the paper.

II. DEFINITION OF THE LOG-LIKELIHOOD RATIO

This section describes the algorithm implemented in the LLR circuit which provides an ordered list of the smallest n m LLR values and their associated GF(q) symbols from the q binary LLR values received from the channel.

Let X = (x 0 x 1 ... x m-1 ) be a GF(q = 2 m ) symbol composed by m = log 2 (q) binary symbols and let Y = (y 0 y 1 ... y m-1 ) be the symbol received from the channel so that Y = (y i = B(x i ) + w i ) i=0,..,m-1 , where w i is a realization of a white Gaussian noise of variance σ 2 and B(x) = 2x-1 is the BPSK modulation that associates symbol -1 to bit x = 0 and +1 to bit x = 1. For an Additive White Gaussian Noise channel with a noise of variance σ 2 , ln(P (Y |X)) is given by:

ln(P (Y |X)) = ln( m-1 i=0 P (y i |x i )) = m ln( 1 √ 2πσ ) - m-1 i=0 (y i -B(x i )) 2 2σ 2 (1)
Let X be the symbol of GF(q) that maximizes ln(P (Y |X)), i.e. X = {arg max X∈GF (q) , P (Y |X)}. Using equation (1), X is given by X = (x i = D(y i )) i=0..m-1 , where D denotes the Hard Decision on y, i.e. D(y) = 0 if y < 0, D(y) = 1 otherwise.

With the hypothesis that the GF(q) symbols are equiprobable, the reliability L(X) of a symbol X may be defined as the LLR of the probability of X related to the probability of symbol X:

L(X) = ln P (Y |X) P (Y | X) (2)
which, using (1), can be developed as:

L(X) = m-1 i=0 - (y i -B(x i )) 2 2σ 2 + (y i -B(x i )) 2 2σ 2 = - 1 2σ 2 m-1 i=0 2y i (B(x i ) -B(x i )) (3)
By definition of X, L(X) is a negative number. In order to deal with positive numbers, the quantity L(X) = -L(X) will be considered in what follows, as in [START_REF] Sevin | Min-max decoding for non binary LDPC codes[END_REF]. Using (3), L(X) can be written as:

L(X) = 2 σ 2 m-1 i=0 |y i |∆ i (4)
where ∆ i = x i XOR xi , i.e. ∆ i = 0 if x i and xi have the same sign, 1 otherwise. This approach which represents the inverse logic of the conventional LLR computation, associates the lower LLR value to the most reliable GF(q) symbol. Its main advantage is that it avoids the normalization step needed to keep the numerical stability of the extrinsic LLR during the decoding process.

The first step of the EMS algorithms [START_REF] Declercq | Decoding algorithms for Nonbinary LDPC codes over GF(q)[END_REF] requires the generation of the first n m minimum values of L(X) which is not a trivial problem. An elegant algorithm has been proposed by Fossorier et al. in [START_REF] Valembois | An improved method to compute lists of binary vectors that optimize a given weight function with application to soft-decision decoding[END_REF]. Unfortunately, this algorithm is more software oriented, since it builds the LLR list dynamically, and is not adapted to hardware implementation mainly when a parallel approach is considered. Thus, a hardware oriented algorithm has become a necessity when the hardware implementation of the NB-LDPC decoder has become possible.

III. PROPOSED ALGORITHM

Let us tackle the problem of generation of the n m lowest L(X) values and their associated GF(q) symbols sorted in increasing order. In this paper, we propose an iterative construction of the list by considering, at stage c, only the first c coordinates of X (c varying from 1 to m). Let us define L(X c ) = 2 σ 2 c-1 i=0 |y i |∆ i the partial LLR of X using only its first c coordinates. Let λ c be the set of couples (L(X c ), X c ) sorted in increasing order according to the L(X c ) values. For the sake of simplicity, the multiplicative constant factor 2 σ 2 will be omitted in the following. For c = 1, λ 1 is then equal to: λ 1 = {(0, d 0 ), (|y 0 |, d0 )} where d means not(d). At stage c = 2, the list λ 2 can be constructed from λ 1 in two steps. The first step is the expansion of λ 1 into two lists λ 0 2 and λ 1 2 by appending the second coordinate with value (0,d 1 ) and (|y 1 |, d1 ). The obtained lists are respectively

λ 0 2 = {(0, d 0 d 1 ), (|y 0 |, d0 d 1 )} (5) λ 1 2 = {(|y 1 |, d 0 d1 ), (|y 0 | + |y 1 |, d0 d1 )}. (6)
The second step is the merging of λ 0 2 and λ 1 2 to create λ 2 . For example, if

|y 1 | > |y 0 |, then λ 2 will be equal to λ 2 = {(0, d 0 d 1 ), (|y 0 |, d0 d 1 ), (|y 1 |, d 0 d1 ), (|y 0 | + |y 1 |, d0 d1 )}.
In the general case, during the expansion process, the

k th couple (L(X c-1 )(k), X c-1 (k)) of λ c-1 is expanded as: λ 0 c (k) = (L(X c-1 )(k) + 0, X c-1 (k)&d c-1 ) (7) λ 1 c (k) = (L(X c-1 )(k) + |y c-1 |, X c-1 (k)& dc-1 ), (8
) where & stands for the binary append operator. Since both λ 0 c and λ 1 c are sorted, the creation of λ c is simple. These two ordered lists are iteratively compared. At each step, each list sends its smallest remaining LLR value. Those two LLR values are compared and the couple with the smallest LLR is retrieved from the corresponding list and added to the output list λ c . This algorithm can be performed efficiently in a serial pipe-line architecture. One can note that, since only n m lowest values are required by the EMS algorithm, the size s(c) of list λ c is equal to min(2 c , n m ).

Let us consider the following example of LLR generation for m = 4, n m = 10 and Y = (-7, 8, 12, -3). The hard decoding of Y leads to D = (0, 1, 1, 0). Since |y 0 | < |y 1 |, the list λ 2 is equal to λ 2 ={(0, 01), (7, 11), (8, 00), (15, 10)}.

The expansion of λ 2 gives λ 0 3 ={(0, 011), (7, 111), (8, 001), (15, 101)} and λ The direct approach (computation of all q LLRs and sorting process to extract the first n m values) leads to a complexity in q log 2 (q). With the proposed method, the overall complexity is reduced to n m log 2 (q). Moreover, the proposed algorithm can be implemented in a simple systolic hardware architecture.

IV. HARDWARE ARCHITECTURE OF THE LLR CIRCUIT

In this section we describe the systolic hardware architecture serially generating the sorted LLR list. This architecture is composed of m stages working in pipeline mode and each stage consists of one Processing Element (PE). Figure 1-a illustrates the architecture for m = 4. As shown in this Figure, the c th PE receives two inputs: the channel binary observation y c-1 with its corresponding sign and the list λ c-1 from the (c -1) th PE. As an output, the c th PE generates the list λ c to be fed to the (c + 1) th PE of the next stage.

A. Structure of the PE

The PE constitutes the core of the proposed LLR circuit. It consists of two expansion modules, two First-In-First-Out (FIFO) memories and one comparator selecting the minimum of the two FIFO's outputs as shown in Figure 1-b where the third stage PE is illustrated. The input/ouput of this stage represent the intermediate LLR computation of the example described in previous section. The 3 rd PE serially receives (one new couple every clock cycle) the ordered list λ 2 from stage 2. The first step is the expansion of λ 2 into lists λ 0 3 and λ 1 3 as described in the previous section. Once the first elements of λ 0 3 and λ 1 3 are stored in the FIFOs, the merging process can start. At each clock cycle, the two outputs of the FIFOs are compared and the couple having the smaller LLR value is selected to be fed in the list λ 3 and a pull signal is set to 1 to get a new couple at the output of the considered FIFO. Figure 1-c illustrates the temporary contents of the FIFOs during the first six clock cycles of the 3 rd stage processing. The highlighted memory case of the FIFOs represents the content to be selected at each clock cycle. As previously discussed, the FIFO 0 contains the first elements of the LLR list. Thus, it is possible to empty the FIFO 0 and in this case the remaining contents of the FIFO 1 are systematically selected to complete the list λ 3 .

One can note, that, during the first s(c -1) clock cycles, FIFO 0 and FIFO 1 receive one new couple every clock cycle and one couple is output, either from FIFO 0 or FIFO 1. Let f 0 (c) and f 1 (c) be the minimum sizes required by the FIFO 0 and the FIFO 1 of the c th PE respectively. Let us first consider the size f 1 (c) when s(c) is even. In the worst case, the first s(c)/2 couples of λ c are retrieved from FIFO 0. At that point, FIFO 1 contains the s(c)/2 couples sufficient to complete the s(c)/2 remaining elements of λ c . If s(c) is odd, (s(c) -1)/2 elements of λ c at most can come from FIFO 1. Thus, in the general case, f 1 (c) ≤ s(c)/2 , where x represents the greatest integer smaller or equal to x.

The determination of f 0 (c) needs a more careful examination. Let us consider the case where s(c -1) = s(c). In the worst case, the first s(c)/3 couples are retrieved from FIFO 0, then the next s(c)/3 couples are retrieved from FIFO 1. At that point, both FIFOs have already received 2 s(c)/3 couples. The output of FIFO 0 and FIFO 1 are respectively λ 0 c ( s(c)/3 + 1) and λ 1 c ( s(c)/3 + 1). In all cases, the content of FIFO 0 is enough to retrieve the next s(c)/3 values needed to complete the list λ c . If s(c) is not a multiple of 3, s(c) is still equal to s(c)/3 . In this case the (2 s(c)/3 + 1) th element of λ c will be retrieved from FIFO 0, which freed room for the (3 s(c)/3 +1) th element coming from λ c-1 and so on. If s(c -1) = s(c)/2 it can be shown that, in that case, f 0 (c) = s(c)/4. In summary, f 0 (c) varies between f 0 (c) = s(c)/4 and f 0 (c) = s(c)/3.

Figure 2 shows the timing diagram of the LLR computation of the example described in section III. As shown in this Figure, the LLR circuit operates in a synchronous way with a clock signal. The signal startin indicates the start of the LLR computation and the signal load y i indicates the availability of the y i data at the input. The circuit has a Latency of m = 4 cycles, the start of the generation of the output is indicated by signal startout. After n m +4-1 = 13 cycles the (LLR,GF) couples are generated. As previously mentioned, the LLR circuit operates in a pipeline manner where it can starts the processing of the second symbol while the last stage of the circuit is producing the (LLR,GF) couples of the first symbol. Thus as shown in Figure 2, after a delay of one cycle the first (LLR,GF) couple associated to the second symbol is produced. This delay cycle is required to reinitialize the FIFOs.

B. Implementation results

A generic architecture of the proposed systolic architecture has been developed and successfully validated. The architecture accepts any Galois Field size q and any value n m ≤ q. The internal binary word size (n b , n s ) of the (LLR,GF) couple is also parameterized. This architecture has been used in a FPGA based GF(64) NB-LDPC code implementation, with q = 6, n m = 12, n b = 6 and n s = 6. On a Virtex 4 (XC4VLX15), it requires 275 slices and operates at a frequency of 149 MHz.

V. CONCLUSIONS

  This work is supported by INFSCO-ICT-216203 DAVINCI "Design And Versatile Implementation of Non-binary wireless Communications based on Innovative LDPC Codes [www.ict-davinci-code.eu] funded by the European Commission under the Seventh Framework Programme (FP7).

Fig. 1 .

 1 Fig. 1. LLR circuit block diagram designed over GF(16).

Fig. 2 .

 2 Fig. 2. Timing diagram of the LLR computation over GF(16), nm = 10.

We have presented a novel and efficient hardware design of the LLR computation circuit. The proposed circuit is the first of its kind to be designed. It performs the LLR computation in a systolic way and produces an ordered list of (LLR, GF) couples. This architecture has been implemented in a design of a GF(64) NB-LDPC code. It can be also used for other codes, like the soft Reed Solomon decoder described in [6].


F Mauger

A Kamor

C Chandre

T Uzer

Mechanism of delayed double ionization in a strong laser field

Keywords: numbers: 32.80.Rm, 05.45.Ac, 32.80.Fb

When intense laser pulses release correlated electrons, the time delay between the ionizations may last more than one laser cycle. We show that this "Recollision-Excitation with Subsequent Ionization" pathway originates from the inner electron being promoted to a sticky region by a recollision where it is trapped for a long time before ionizing. We identify the mechanism which regulates this region, and predict oscillations in the double ionization yield with laser intensity.

Atoms in strong laser pulses lose their electrons through two ionization channels [START_REF] Becker | [END_REF]: A sequential one (SDI) and its non-sequential counterpart (NSDI). The SDI mechanism consists of successive and independent removals of the electrons. The recollision (or three-step) scenario [3,4] in which a pre-ionized electron returns to the ion core to dislodge a bound electron is the characteristic mechanism for NSDI. It turns out that there is a rich variety of pathways among NSDI processes, also. At first sight, these pathways can be distinguished by the time it takes the second electron to ionize. The common variant involves little, if any, delay between the recollision and ionization. However, this so-called "direct impact ionization" [5] is often accompanied by an alternative (and less straightforward) road to NSDI called Recollision Excitation with Subsequent Ionization (or RESI for short [5][6][7][8][9][10][11][12][13]). Recent experiments have shown that RESI can be the dominant channel for very short pulses [14]. The mechanism for RESI is often attributed to the recollision which excites the parent ion, later ionized by the laser field with a delay sometimes lasting longer than one laser cycle after the recollision and thereby imitating an uncorrelated ionization process.

Here we report results showing that RESI is a purely inner electron process and that the recollision, while helpful for increasing its probability, merely enhances what is already there, namely the ionization process due to extensive state mixing. The reason why the mechanism responsible for RESI remains an open question is due to the spectator role often erroneously assigned to the inner electron. Far from waiting passively for the pre-ionized electron to recollide, the inner electron interacts with the strong field also, and a true nonsequential description of the process reflects the excitation of both electrons. Our view, to be detailed below, is summarized by the cartoon in Fig. 1: Hitting the pool of available initial states, a recollision may send the inner electron into the ionization continuum directly, thereby causing two electrons to emerge with very little time delay (direct impact ionization) or the recollision may drive the bound electron towards the parts of phase space which funnel electrons into the ionization continuum -which, given sufficient [START_REF] Blumel | Chaos in Atomic Physics[END_REF] with px ≤ 0 and py = 0. After a recollision (full line arrows), the inner electron can be directly ionized or promoted to an excited state that is ionized with a delay (dotted arrow). Lower panels: Typical two-electron trajectories for direct impact ionization (left) and RESI (right). Continuous (dashed) curves denote x (y) coordinates.

time, the electrons may have reached without the help of a recollision.

This common-sense distinction between the direct impact ionization and RESI emerges from our classical calculations [15] which allow both electrons to interact with the field and are, therefore, fully nonsequential. Evidence for the veracity of the proposed mechanism comes from two observations: The ionization delays are in harmony with experimental observations, and our treatment confirms that the second ionization in RESI happens at the extrema of the laser field [5,10,[16][17][18].

Classical ensemble methods [19,20] have been remarkably successful in identifying direct impact ionization as well as RESI pathways that lead to double ionization and reproduce the experimental and computational observations closely. Therefore we consider a two active electron atom with soft-Coulomb potentials subjected to an intense and short linearly polarized laser pulse in the dipole approximation [15,[19][20][21][22]. The Hamiltonian is:

H (x 1 , x 2 , p 1 , p 2 , t) = p 1 2 2 + p 2 2 2 - 2 x 1 2 + a 2 - 2 x 2 2 + a 2 + 1 x 1 -x 2 2 + b 2 + (x 1 + x 2 ) • e x E 0 f (t) sin ωt, (1)
where x i is the position vector of the ith electron in d dimensions and p i is its canonically conjugate momentum.

The linearly polarized (along the x-direction with unit vector e x ) laser is characterized by its amplitude E 0 and has a wavelength of 780 nm or 460 nm (ω = 0.0584 or 0.1 a.u. respectively) with a shape f (t) consisting of two-cycle linear ramp-up and six laser cycle constant plateau. The constants a and b are the electron-nucleus and electron-electron softening parameters respectively. For the computations considered in this paper, we choose a = b = 1 [15,[19][20][21][22], even though qualitatively similar results are observed with different softening parameters.

To begin with, we investigate the dynamics when both electrons are confined to a single dimension along the axis of polarization [START_REF]An alternative reduced one-dimensional model has been proposed[END_REF], i.e., d = 1. The motivation for studying the one-dimensional model is twofold: First, the laser field drives the dynamics along the polarization axis by which ionization is naturally expected to happen. Second, we find that this simplified dynamics forms the skeleton of higher-dimensional dynamics, which, however, differs from the single-dimensional dynamics in significant ways, to be specified below. It should be noted that, three-dimensional calculations give similar results to two-dimensional ones due to the cylindrical symmetry of the problem around the polarization axis.

RESI events are described by two-electron trajectories for which, after a recollision, one electron remains bound to the nucleus for a long time before ionizing without any additional recollision, i.e., without the other electron ever returning to the nucleus. We use an energy criterion to designate an electron as ionized or not [15,24]. A recollision is said to have occurred whenever the distance between the two electrons is smaller than some threshold. We consider the NSDI trajectories which double-ionize after a last recollision. Among them, we discriminate RE-SIs by a time delay of at least two laser cycles between the last recollision and the ionization of the remaining FIG. 2: Stroboscopic plots at the maxima of the laser field, for I = 3 × 10 15 W • cm -2 and 780 nm during the plateau of the laser. Upper panel: detected RESI trajectories, after the last recollision, for Hamiltonian [START_REF] Blumel | Chaos in Atomic Physics[END_REF]. Initial conditions are chosen as in Refs. [15,24]. For each RESI, the first point on the section is plotted in light blue while the following ones are in dark blue. Lower panel: stable (light orange dots) and unstable (dark blue dots) manifolds of the periodic orbit O12 of Hamiltonian [START_REF] Becker | [END_REF]. The position of O12 is indicated with a black diamond (see the arrow). The pink area in the panels represents the part of phase space from which the inner electron does not ionize.

ion. Qualitatively similar results are obtained with different (shorter or longer) time delays. In Fig. 2 (upper panel), we display stroboscopic coordinate-momentum plots (taken at the maxima of the field) of detected RE-SIs. In what follows, we explain the rationale for these swirling patterns and connect them to full-dimensional calculations.

RESIs are best understood using a single electron model: A recollision has excited the inner electron while the recolliding electron remains ionized [15], which allows us to neglect the electron-electron interaction in Hamiltonian [START_REF] Blumel | Chaos in Atomic Physics[END_REF]. The resulting reduced-dimensional Hamiltonian reads

H (x, p, t) = p 2 2 - 2 x 2 + a 2 +x • e x E 0 sin(ωt + φ 0 ), (2)
where φ 0 denotes the phase at which the last recollision occurs.

The dynamics given by Hamiltonian (2) turns out to be the key to the patterns formed by the RESI trajectories. Previous studies on Hamiltonian (2) [15,24] with one spatial dimension have identified two qualitatively different kinds of dynamics for the electron driven by the field: In the competition between Coulomb attraction to the nucleus and the laser excitation, either the latter prevails, and the electron is quickly ionized; or the Coulomb attraction manages to maintain the electron trapped near the core [25]. Two areas in phase space emerge from this distinction: A bound region, close to the nucleus (pink area in Fig. 2), where the electron is trapped by the nucleus and cannot ionize; and an unbound region, further away (white area in the same panels), where the electron is quickly ionized by the laser.

A more detailed study shows that the behavior in the unbound region is more complex than anticipated. This is readily apparent in the patterns seen in the unbound region in Fig. 2. A thin transition layer in the unbound region, and located in the area where the Coulomb attraction and the laser excitation compete equally, is responsible for RESI. In practice, we show that this transition region is organized by the main resonances between the free field dynamics [E 0 = 0 in Hamiltonian (2)] and the laser (the electron revolves exactly n times around the nucleus in one laser cycle, and we refer them as 1:n resonances). These resonances give birth to periodic orbits, among which at least one, denoted O n , is unstable (hyperbolic). Other periodic orbits in the vicinity of the bound region merely influence the fine details of the chaotic structure.

Because a periodic orbit corresponds to a recurrent motion, it does not lead to ionization. However, neighboring trajectories may do so after some time, particularly if the periodic orbit is weakly unstable, as it is the case for selectes O n s. The pathways by which the electrons approach or leave the core are quantified by the so-called stable and unstable manifolds [START_REF] Wiggins | Introduction to applied nonlinear dynamical systems and chaos[END_REF] around an unstable periodic orbit. In Fig. 2 (lower panel) we draw the stable and unstable manifolds of O n (light orange and dark blue points respectively) for n = 12. Note the strong similarity between the unstable manifold and the Poincaré sections of RESI trajectories for the two-electron Hamiltonian (1) (upper panel). This similarity confirms the key role played by this one-electron unstable manifold in the RESI process. In addition, we see that the stable and unstable manifolds intersect an infinite number of times, a characteristic feature of a chaotic dynamics [START_REF] Mitchell | [END_REF]. The overlap between the stable and unstable manifolds of O n forms a "sticky" region [START_REF] Wiggins | Introduction to applied nonlinear dynamical systems and chaos[END_REF] that traps trajectories for some time before ionizing. To better understand how RESI electrons find their way to ionization, we display a projection of the unstable manifold of O n in the positionphase of the laser plane (x, φ = ωt + φ 0 ), together with a projection of O n in Fig. 3. Two main branches depart from the central region near x = 0 when the laser phase is φ = π/2 and φ = 3π/2, i.e., near the extrema of the electric field, showing that RESI takes place approximately at the extrema of the electric field [5,17].

The organizing role of resonant periodic orbits can also be found in the relative proportion of RESI to double ionization yields: In Fig. 4, we compute the RESI yields as the intensity is varied. It shows oscillations which are correlated with the intensity range where the 1:n resonance regulates the delayed ionization. The bound region (pink area in Fig. 2) shrinks with increasing laser intensity, and higher-order resonant periodic orbits are drawn to the unbound region. It means that as O n becomes too unstable, the next resonance (associated with periodic orbit O n+1 ) is at play. Note the correlation between the oscillations and the stability index [START_REF] Wiggins | Introduction to applied nonlinear dynamical systems and chaos[END_REF] of the resonant periodic orbits identified from the one-electron model [START_REF] Becker | [END_REF]. The stability index measures the typical time this orbit is expected to influence the neighboring dynamics: The larger the stability index, the sooner a neighboring trajectory diverges from it. Recent experimental results [START_REF] Pfeiffer | Breakdown of the independent electron approximation in sequential double ionization[END_REF], with elliptic polarization, have revealed oscillations in the parallel to anti-parallel double ionization yields analogous to the oscillations observed in the RESI yields of Fig. 4.

Adding a dimension to the one-dimensional calcula- tions provides more scope for the electron dynamics and makes the dynamics more complex. In particular, due to the extra dimension, the dynamics close to the nucleus region becomes partially chaotic both for one-and two-electron Hamiltonians (2) and (1) as it is already seen in Fig. 1. It becomes harder to define a bound and unbound regions for the inner electron. More importantly, the aforementioned 1:n resonances generally enhance (linear) instability in the transverse direction while not affecting the organization of the dynamics in the (polarization) symmetry subspace: At a given intensity more than one resonance drives the RESI dynamics in the sense that their unstable directions participate to the enhancement of delayed ionization. A direct consequence is the smoothing of the oscillations in Fig. 4 for 460 nm wavelength. Increasing the wavelength to 780 nm gives birth to more resonances with a more dense tangle of unstable manifolds so that the oscillations are completely washed out in two-dimensional models whether they are one-or two-electron models, as observed from our classical calculations. In Fig. 4 we notice that the RESI yields are higher for a two-dimensional model than for one dimension. We attribute these higher yields to the aforementioned chaotic dynamics coming from an increase of instability associated with additional resonances in the full-dimensional models. It underscores once more the pivotal role played by the inner electron dynamics [Hamiltonian (2), i.e., without recollision] in these delayed ionizations.

FIG. 1 :

 1 FIG.1: Upper panel: Ionization time for Hamiltonian[START_REF] Becker | [END_REF] with two dimensions for I = 3 × 10 15 W • cm -2 , φ0 = π/2 and 780 nm. Initial conditions are chosen at the energy of the Stark saddle[START_REF] Blumel | Chaos in Atomic Physics[END_REF] with px ≤ 0 and py = 0. After a recollision (full line arrows), the inner electron can be directly ionized or promoted to an excited state that is ionized with a delay (dotted arrow). Lower panels: Typical two-electron trajectories for direct impact ionization (left) and RESI (right). Continuous (dashed) curves denote x (y) coordinates.

FIG. 3 :

 3 FIG. 3: Projection of the unstable manifold of the periodic orbit O12 of Hamiltonian (2) for I = 3 × 10 15 W • cm -2 and 780 nm in the (x, φ) plane. The black curve is a projection of the periodic orbit in the plane (x, φ), and the black squares indicate the position of the saddle point at the maximum of the field.

FIG. 4 :

 4 FIG.4: Normalized RESI yields for 460 nm wavelength laser (markers, left hand y-axis). Red squares (diamonds) label one (two) dimensional one-electron simulations (normalized to the number of non-ionized trajectories). Green dots label two-dimensional two-electron simulations (normalized to the number of double ionization). Curves label the linear stability index[START_REF] Wiggins | Introduction to applied nonlinear dynamical systems and chaos[END_REF] of the resonant periodic orbits On of Hamiltonian (2) (curves, right hand y-axis) as functions of the laser intensity I. Continuous (broken) curves refer to odd (even) 1:n resonances. As intensity increases, the order of the resonance goes from n = 3 to n = 9 as indicated on the curves.

The authors acknowledge useful discussions with C. Cirelli, N. Johnson, R. R. Jones, U. Keller, M. Kling, and A. Pfeiffer. C.C. and F.M. acknowledge financial support from the CNRS. F.M. acknowledges financial support from the Fulbright program. This work is partially funded by NSF.


Yohann Pitrey

Ulrich Engelke

Marcus Barkowsky

Romuald Pépion

Patrick Le Callet

Aligning subjective tests using a low cost common set

Keywords: sujective quality assessment, inter-experiment mapping, scalable video coding, error-concealment

In this paper we use a common set between three subjective tests to build a linear mapping of the results of two tests onto the scale of one test identified as the reference test. We present our low-cost approach for the design of the common set and discuss the choice of the reference test. The mapping is then used to merge the outcomes of the three tests and provide an interesting comparison of the impact of coding artifacts, transmission errors and error-concealment in the context of Scalable Video Coding.

INTRODUCTION

Subjective quality experiments are typically conducted with respect to international recommendations, such as ITU-T Rec. , in order to produce reliable and reproducible outcomes. Despite the strict rules defined in these recommendations, a lot of factors have an impact on the test results which cannot easily be controlled. They lead to a set of context effects that turn each experiment into a non standard environment. As a consequence, special considerations have to be taken into account in order to compare test results between different laboratories or different experiments. Typically, this involves mapping the outcomes of different tests onto a common scale. The common scale is usually built on a subset of conditions shared by all tests, or by groups of experiments. The design of the common set is of great importance for the mapping, to represent precisely the relation between tests.

In this paper, we make use of such a method in order to compare the Mean Opinion Scores (MOS) of three subjective experiments that we conducted in the context of Scalable Video Coding. The originality of our approach is to include the common conditions during the design of the test, so that no extra experiment needs to be conducted in order to build the mapping between tests. After choosing a reference test to map onto, we use a simple linear mapping to derive relationships between the different experiments.

The paper is organized as follows. In Section 2 we briefly introduce the three experiments we conducted. In Section 3, we discuss the mapping of the MOS onto the common scale. Results are presented in Section 4 and conclusions are drawn in Section 5.

DESIGN OF EXPERIMENTS

We conducted three subjective experiments containing various Hypothetical Reference Circuits (HRC), in order to evaluate the impact of video coding artifacts, transmission errors and error-concealment techniques in the context of Scalable Video Coding (SVC). All the tested videos are in VGA (640×480 pixels) and QVGA (320×240 pixels) formats, displayed at 15 or 30 frames per second. Nine video sequences of 12 seconds each were used, representing a good variety of contents and high spatial and temporal activity ranges. The perceived quality was assessed using the Absolute Category Rating (ACR) methodology with 5 levels of quality, and conducted in a subjective test room with standard viewing conditions [START_REF]Rec. Subjective video quality assessment methods for multimedia appl[END_REF].

In the first test (T1), the impact of error-concealment and encoding parameters on two-layer SVC streams is evaluated. We simulate a loss of one second during which no data is received for the highest layer. The visual artifacts induced by the lost data are concealed using two techniques based on upscaling the base layer, which is assumed to be always available. The first technique is referred to as "switched" and consists in replacing the whole distorted frame with an upscaled version of the corresponding frame from the base layer. The second technique is referred to as "patched" and consists in replacing only the distorted area in the frame with the upscaled area from the corresponding frame in the base layer. Two constant bit-rate scenarios are combined with two base-layer temporal frequencies in order to identify the best encoding configuration in such a context. The HRCs in this test are referred to using a structure similar to "120/600kb/s-30Hz-switched", reflecting the bitrate used for the two layers, the frequency of the base layer and the errorconcealment technique (more details about this test can be found in [START_REF] Pitrey | Evaluation of MPEG4-SVC for QoE protection in the context of transmission errors[END_REF]). For reference, one AVC HRC is included in this test under the same conditions of distortion. The artifacts are concealed using a state-of-the-art technique, based on reusing the last non distorted frames and buffer repetition.

In the second test (T2), we evaluate the impact of two-layer SVC coding artifacts on the perceived quality, without transmission errors. We use four 26, 32, 38 and 44 as QP values for each layer, leading to 16 combinations of QP. We also include the 4 versions of the upscaled base layer encoded using the same four QP values. The HRCs from this test are referred to using a structure similar to "QP38/44", 38 being the QP value for the base layer and 44 the QP value for the enhancement layer. The upscaled base layer HRCs are referred to as "QP 38 Upscaled".

In the third test (T3), we evaluate the impact of the distribution of network impairments on a subset of streams from T2. Four factors are varied: the quality of the base layer, the number of impairments, the total length of the impairments and the interval between two impairments. The HRCs from this test are referred to using the values of QP for the two layers and a succession of numbers of frames displayed from each layer in turn. For instance in "44/32 -32-16-32-", the "44/32" part means that the video was encoded with QP 44 for base layer and 32 for enhancement layer. The "-32-16-32-" part means that two impairments of 32 frames are displayed, separated by 16 frames (more details about this test can be found in [START_REF] Pitrey | Subjective quality of SVC-coded videos with different error-patterns concealed using spatial scalability[END_REF]). During the impairments, we use the "switched" error-concealment technique from T1. The positions of the impairments are calculated so that they are globally centered on the middle of the sequence.

We designed the three experiments so that they share a subset of configurations called the common set, used for comparing their respective outcomes. This common set provides a basis to perform fitting operations, in order to compare the results of the three tests. The design of the common set and the fitting process are described in the next section.

METHODOLOGY

During a subjective test, the viewers tend to use the full quality scale, independently from the distribution of the expected quality scores of the presented sequences [START_REF] Garcia | Normalization of subjective video tests results using a reference test and anchor conditions for efficient model development[END_REF]. This phenomenon, known as the corpus effect, forbids direct comparison of the results from different subjective tests. However, presenting a common set of configurations in several tests allows mapping the results from one test onto another, and comparing them as the outcomes of a single test. When trying to compare more than two tests, the use of a reference test has been proposed to facilitate direct comparison between HRCs from different tests [START_REF] Garcia | Normalization of subjective video tests results using a reference test and anchor conditions for efficient model development[END_REF][START_REF] Pinson | An objective method for combining multiple subjective data sets[END_REF]. For this purpose, the conditions contained in the reference test should cover qualities that are evenly spread over a wide range of qualities. To achieve this goal, different methodologies have been deployed in the past.

In [START_REF] Pinson | An objective method for combining multiple subjective data sets[END_REF] the authors create a reference test, which they refer to as a meta-test, from a set of 6 subjective tests. For that purpose, a subset of 185 Processed Video Sequences (PVS) were carefully selected from 479 available PVS, with respect to a uniform quality distribution over the range of the scale. These sequences were then used to conduct another quality test. The MOS from the 6 original tests were then mapped onto the MOS from the meta-test to facilitate comparison between the original MOS. Despite the clear rationale behind this method and the thorough conduction of its implementation, it is apparent that the preparation of an additional test is cumbersome and time consuming. In addition, a new meta-test needs to be created whenever a new test is to be included for comparison.

The work in [START_REF] Garcia | Normalization of subjective video tests results using a reference test and anchor conditions for efficient model development[END_REF] reports on mapping to a reference test focussing on the particular context of IPTV degradations. Here, a different approach has been chosen by preemptively designing a reference test that contains a wide perceptual range of IPTV degradations. Subsequently, four other tests were designed that focus on a selective subset of degradations. This approach is mainly applicable in case where a particular application is considered and the range of degradations can be estimated. Thus, it assumes that the reference test is already designed with regard to the tests that are to be compared. Such foresight, however, is not always given and often it is of interest to compare tests without a reference test being available.

In our work, we therefore take a different approach that avoids conducting additional tests and that also does not expect a reference test to be available for the mapping. When designing a new experiment, we include several configurations in it that come from the existing common set. All our tests share a small amount of configurations, on which we rely to perform mapping operations between experiments. Given a set of available tests, we carefully determine the most suitable test with respect to similar constraints as in the previous works. Hence, this approach is adaptive to the current problem at hand as it relies only on the data available. Four HRCs are shared by T1, T2 and T3. These four conditions contain SVC coding distortions, transmission errors, upscaling artifacts and temporal discontinuities. Moreover, as the T2 and T3 tests are closer to each other in terms of evaluated factors, they share another 9 HRCs, raising the size of the common set to 14 HRCs. These additional HRCs contain a wider variety of coding distortions and transmission errors in order to get a more accurate mapping between the two experiments.

The condition MOS (i.e. average opinion scores on all observers and all source contents) of the three tests are presented in Figure 1 in order of increasing magnitude. It can be seen that all three tests cover a wide range of qualities with test T3 covering the widest range. For this reason, we choose T3 to be the reference test in the scope of this work.

We derived linear mapping functions to map the MOS from tests T1 and T2 onto the scale of test T3 as follows:

yT i = aixT i + bi,
where xT i is the MOS of T1 and T2, and yT i is the mapped MOS on the T3 scale. The parameters ai and bi were determined using linear regression between the condition MOS of the common sets of HRCs respectively from T i and T 3. The mappings are illustrated in Figure 2 and the corresponding mapping parameters are presented in Table 1.

Figure 2 first shows the repartition of the configurations from the common set on the quality scale. This indicates that the MOS of both T1 and T2 are highly correlated to the MOS of T3. In fact, the linear correlation between T1 and T3 is equal to 0.995 and the linear correlation between T2 and T3 is equal to 0.985. It should be noted that for both mappings, the original reference sequence (HRC0) is approximatly at the same location and very close to the diagonal. This indicates that the non-distorted reference sequences were rated very similarly between the three tests.

From Figure 2 one can further see that in both cases the mapping functions are above the diagonal, meaning that the MOS are generally alleviated for both T1 and T2. This is also illustrated in Figure 1, where the mapped MOS are displayed above the MOS scores for T1 and T2. It can be observed that one mapped MOS value in T2 is slightly outside of the scale. This is a result of the mapping between T2 and T3 being considerably above the diagonal and hence, the already high quality reference condition is mapped slightly outside the scale. This might indicate that the scale was compressed at the upper end for this particular test.

After mapping the results of T1 and T2 onto the scale of T3, we consider the outcomes of the three tests as a single experiment. In the next section, we conduct an analysis of a1 b1 a2 b2 0.759 1.212 1.058 0.281 Table 1: Linear mapping parameters from T1 and T2 to T3.

the mapped MOS values, in order to compare the influence of the three types of distortions to each other, namely error concealment, SVC coding artifacts and impairment distributions.

RESULTS AND DISCUSSION

A total of 82 HRCs result from joining T3 and the mappings of T1 and T2 onto T3. This super-set of HRC contains a large amount of different conditions with distinct kinds of distortions. To get a good overview of this large amount of data, Figure 3 compares the different HRCs from the 3 tests after mapping. We display the original test of each HRC as well as a short description following the structures introduced in section 2. On Figure 3, each HRC is represented by a black symbol on the same line as its description. The grey intervals on the upper and lower parts symbolize the HRCs that are statistically equivalent to one given HRC. The statistical equivalence between configurations is determined using the 95% intervals of confidence, calculated on the data after fitting.

As an example of interesting results, we can observe that the reference HRC from T3 (line 6) gets an equivalent quality to (but slightly lower than) the SVC HRCs using a QP of 26 for the enhancement layer (lines 1-5). This might identify a saturation effect at the top of the quality scale, as the viewers fail to give the reference a significantly higher score than the already-high quality HRCs. Alternatively, it couls indicate that a QP of 26 is perceived as lossless by the viewers in our scenario. The corpus effect is also illustrated here, as the high quality SVC HRCs, which only contain limited coding artif and testingacts, are perceived equivalent to the reference in the context of network impairments such as the one in T3.

The mapping of the three tests allows for comparison of HRCs that are located on distinct dimensions of distortion. For instance one can observe that the upscaled SVC base layer encoded with a QP of 26 (line 27) is equivalent to several two-layers SVC streams impaired by different loss patterns (e.g.: lines 23, 26, 28, 31). However, the upscaled version only needs an average bitrate of 0.84 Mb/s to be encoded, which is about half the bitrate needed to transmit one of the equivalent two-layer streams. Therefore, using a good quality video and upscaling may represent an interesting alternative to multiple layers.

One can also draw a parallel between constant bitrate and constant quality configurations. For instance, the 120-600kb/s-30Hz-switch HRC from T1 (line 24) is statistically equivalent in terms of quality to the 44/32 -32-configuration from T3, which has the same network impairment pattern. A correspondence can then be made between QP values and bitrate, which can be useful for the design of adapted bitstreams without using costly bitrate control techniques.

CONCLUSION

In this paper we presented an approach to design subjective tests so that they share a common set of configurations used to align the outcomes of three experiments in a single data set. We used this data set to compare the impact of coding artifacts, transmission errors and error-concealment techniques in the context of Scalable Video Coding. In our future work, we will investigate on statistical tools to help construct the common set, such as determining the minimum number of HRCs to be included in order to reach a sufficient reliability after the fitting. The super-set of data obtained from the three experiments contains high variability in configurations and distortion types, which makes it a valuable resource for data mining and for designing objective quality metrics. This super-set will thus be used to get a better understanding of the factors having an impact on the perceived quality and design quality metrics adapted to the SVC transmission context.

Figure 1 :

 1 Figure 1: Overview of the MOS values for the three tests. From left to right: T1, T2, T3. Lower part of bars (dark-coloured): before mapping, upper part (light-coloured): after mapping on the scale of T3.

Figure 2 :

 2 Figure 2: Linear fitting functions from T1 and T2 onto T3.

Figure 3 :

 3 Figure 3: HRC comparison after mapping the results of T1 and T2 onto T3. Each HRC is marked with a '+' symbol. The '|' symbols denote the statistical equivalence between the MOS values of two HRCs.


Ulrich Engelke

Yohann Pitrey

Patrick Le Callet

Cohen's / Fleiss's K Scott'

Pi Kendall'

TOWARDS A FRAMEWORK OF INTER-OBSERVER ANALYSIS IN MULTIMEDIA QUALITY ASSESSMENT

Keywords: Inter-observer differences, multimedia quality assessment, inter-rater coefficients, data mining. Psychophysical Experiment Intention: What purpose? Experiment design: Assessment method Stimuli preparation Observer recruitment Experiment conduction Pre-processing Cleaning Aggregation Sampling Feature selection Discretisation Transformation Inter-observer analysis Visualisation

In multimedia quality assessment, observer ratings are typically averaged into mean opinion scores (MOS) to obtain a subjective ground truth for a set of stimuli. Valuable information about individual observer rating behaviour and interobserver differences is lost during this process. Such information, however, can be useful to improve subjective experiment procedures and quality of experience prediction models. In this paper, we therefore present an inter-observer analysis framework that tackles the quality assessment problem from an inverse angle, setting the focus on individual observer differences rather than stimuli differences. The framework consists of a set of procedures for inter-observer analysis as well as the necessary considerations during pre-processing and post-processing. The aim of this paper is to raise awareness that sole consideration of MOS simplifies quality assessment too much, especially given the ever increasing complexity of multimedia quality assessment.

INTRODUCTION

Quality of Experience (QoE) has been of major concern for multimedia service providers in recent years to complement more traditional paradigms such as Quality of Service (QoS) [START_REF] Soldani | QoS and QoE Management in UMTS Cellular Systems[END_REF]. The objective is a transition from an application-centered to an user-centered delivery of multimedia services with the main goal being the satisfaction of the users' experience. This task has been considerably complicated through recent advances in coding and networking technology that facilitated ubiquitous usage of a broad range of multimedia applications on a variety of digital devices, ranging from mobile phones over laptop computers to high definition television in the home. For multimedia service providers it is crucial to automatically measure the QoE provided to satisfy the consumers' expectations. Hence, a considerable research effort has been devoted to the development of models that predict human quality perception. However, despite the increased complexity of multimedia quality assessment, subjective experiment outcomes are still reduced to an average vote over all participants, the Fig. 1. Normalised distributions of standard deviations σ for images distorted through Gaussian blur or fast fasting [START_REF] Sheikh | LIVE image quality assessment database release 2[END_REF]. mean opinion score (MOS). The agreement between participants is measured using confidence intervals (CI) and related quantities, such as the standard error of the mean.

The use of MOS and CI may have sufficed in the context of 'simple' assessment tasks, such as in the evaluation of content with single, additive distortions. However, contemporary multimedia applications constitute a considerably more complex scenario, as a multitude of factors impact on the quality, including, source coding, error-prone networks, error concealment, re-scaling, and display devices. With such an increased complexity of the assessment task, the disagreement between observers may increase. This is illustrated in Fig. 1. Here, normalised distributions of standard deviations over all observers are shown for all Gaussian blur and fast fading distorted images in the LIVE image quality database [START_REF] Sheikh | LIVE image quality assessment database release 2[END_REF]. It can be seen, that generally the observer agreement is considerably higher for the Gaussian blur distorted images, as indicated by the lower standard deviations. This can be attributed to the comparably simpler assessment task as the fast fading distorted images exhibit more complex distortion patterns. Not only system related factors, however, but also other internal (subjective) and external (environmental) factors have been found to contribute to observer disagreement. In [START_REF] Engelke | On confidence and response times of human observers in subjective image quality assessment[END_REF], for instance, it was shown that the confidence of human observers into their quality ratings is inversely related to the CI. Different internal and external factors have been profiled in [START_REF] Strohmeier | Profiling experienced quality factors of audiovisual 3D perception[END_REF] in the context of audiovisual 3D perception, revealing individual quality preferences with respect to certain modalities.

To address the shift to user-centric multimedia services, it Fig. 2. Framework for inter-observer analysis in multimedia quality assessment.

is crucial to go beyond considerations of simple MOS and to gain a better understanding of individual observer perception in the context of multimedia services. The importance of evaluating human observer differences has already been recognised in other research disciplines, especially in medical analysis [START_REF] Banerjee | Beyond kappa: A review of interrater agreement measures[END_REF]. Not just the disagreement between observers as such needs to be quantified (as is done with CI) but more insight into rating behaviour needs to be obtained by analysing individual observers and by determining possible observer groups that exhibit similar behaviour. To identify whether differences are due to internal factors (e.g. personal preferences) or external factors (such as environmental influences or artifacts in the experimental design) is one goal of inter-observer analysis. This in turn is instrumental to improve psychophysical experiment procedures and to design elaborate QoE prediction models that more adequately reflect the complexity of modern multimedia applications. To this end, conventional psychophysical assessment strategies and data analysis methodologies deployed by the multimedia quality assessment community may need to be revisited and additional, unconventional procedures need to be established.

Towards a framework of inter-observer analysis

To address the above issues, we present in this paper a framework for assessment of inter-observer variability. The objectives in this context are twofold. Firstly, we aim to raise awareness that inter-observer differences should not be neglected in multimedia quality assessment. This is particularly true in case of multimodal assessment and for novel applications such as 3D video, where observer differences are expected to be large. Secondly, we propose a set of existing and novel techniques to measure inter-observer variations and apply them in a multimedia quality assessment context. The integral parts of the framework are presented in Fig. 2 and are discussed in detail in this paper. Generally, the framework can be divided into three parts; preparation, inter-observer analysis, and data exploitation. To take full advantage of the framework, special considerations should already take place during the psychophysical experiment and the pre-processing of the data. These preparatory issues are explained in Section 2. The focus of this paper is on the heart of the framework, the inter-observer analysis. We consider three different means to quantify inter-observer differences; statistics, inter-rater coefficients, and data mining [START_REF] Tan | Introduction to Data Mining[END_REF], which are discussed in detail in Section 3. The subsequent visualisation and interpretation of the results is essential for a thorough exploitation and comprehension of the inter-observer analysis and is briefly discussed in Section 4, along with some implications on experimental design and predictive modelling. Finally, conclusions are drawn and future directions are given in Section 5.

The ultimate goal of the framework is to answer many important questions, including: What are the underlying causes of observer differences? To what degree do they depend on internal and external factors, as well as the experimental design? Can random variations be distinguished from systematic variations? Are there particular observer groups that can be identified? Can individual observer differences or observer groups be incorporated into predictive quality models to improve their performance? It should be noted that the current state of the framework is by no means considered to be complete. The purpose is rather to initiate re-consideration of conventional quality assessment, discuss representative candidate methodologies for inter-observer assessment, and to pose open questions that need more attention.

PREPARATION

This section discusses special considerations that should be taken into account in the conduction of the experiment and the data pre-processing to maximise the outcomes of the interobserver analysis. A 3D quality experiment [START_REF] Wang | Subjective evaluation of HDTV stereoscopic videos in IPTV scenarios using absolute category rating[END_REF] is briefly introduced that is used throughout the paper as a means to better illustrate the different aspects of the framework.

Psychophysical experiments

Although analysis of inter-observer differences is of great interest for any existing experiment data, one may in fact take it particularly into account already during preparation of future experiments. This applies to all aspects of experiment design and especially to the stimuli preparation, observer recruitment, and the assessment methodology.

The presented stimuli are usually selected with the aim to determine system performance by carefully choosing parameters from a realistic parameter space. Similar considerations can be applied to identify observer differences, where content classes can be carefully chosen such as to identify personal preferences or distortions can be introduced such as to identify thresholds with respect to certain modalities.

The recruitment of observers does not necessarily need to be solely based on the discrimination between experts and non-experts, as is often done. To identify differences in rating behaviour between observers, one may consider further demographic data and observer profiles as a criterion for observer recruitment. For instance, recruitment of observers who have a 3D television and observers who do not may aid in identifying whether there is a difference in the comfort that these observer groups experience when viewing 3D content.

Depending on the application and the intended purpose, data should be recorded during the experiment in addition to the quality ratings. For instance, to identify observer confidence, additional confidence scores and response times have been recorded in [START_REF] Engelke | On confidence and response times of human observers in subjective image quality assessment[END_REF]. To assess the comfort during 3D video viewing, additional comfort scores were collected in [START_REF] Wang | Subjective evaluation of HDTV stereoscopic videos in IPTV scenarios using absolute category rating[END_REF]. Such complementary information aids to gain a deeper insight into the rating behaviour and the perception of the presented stimuli. Questionnaires handed out before and after the experiment can further provide valuable information.

In summary, the increasing complexity of quality assessment in contemporary multimedia applications needs to be reflected already during the conduction of the experiment.

3D video quality experiment

The outcomes of a 3D video quality experiment are used here to illustrate the concepts of the framework. Details of the experiment can be found in [START_REF] Wang | Subjective evaluation of HDTV stereoscopic videos in IPTV scenarios using absolute category rating[END_REF] and are summarised in the following. The experiment was conducted in two laboratories in France and Sweden, however, in this paper we consider only the former experiment. Eleven different 3D video sources (SRC) were used along with 15 hypothetical reference circuits (HRC) to create a total of 165 processed video sequences (PVS). The HRCs consisted of a number of different coding scenarios, including multi view coding and H.264 simulcast, to identify their impact onto perceived quality. The videos were viewed by 22 naive observers. In addition to the quality scores, comfort scores were collected to measure the degree of discomfort due to the 3D experience. Both scores were rated on a 5-point absolute category rating (ACR) scale.

Pre-processing

Pre-processing typically involves detection of observers that exhibit a considerably different rating behaviour compared to the majority of the observer population. These outliers are often removed according to certain criteria such as the ones specified by VQEG [START_REF]Final report from the Video Quality Experts Group on the validation of objective models of multimedia quality assessment, phase I[END_REF]. To identify the causes of the disagreement rather than directly rejecting these observers is one goal of the inter-observer framework.

Inter-observer analysis based on data mining techniques, as discussed in Section 3.3, needs particularly careful preprocessing to prepare the data with the aim to improve the quality of the outcome [START_REF] Tan | Introduction to Data Mining[END_REF]. For instance, dimensionality reduction using principal component analysis (PCA) based methods can be used on high dimensional data to improve comprehension of the results. Furthermore, discretisation of continuous scale data is usually needed for classification algorithms.

To identify certain patterns it can further aid to transform the data into a more suitable format or representation before deploying the data mining algorithm.

INTER-OBSERVER ANALYSIS

In this framework we consider three means of inter-observer analysis, statistics, inter-rater coefficients, and data mining techniques, which are explained in the following sections.

Statistics

Simple statistics such as the mean, standard deviation, skewness, and kurtosis are usually computed over the observer population to quantify the characteristics of the ratings for particular PVS or HRC. Inversely, these simple statistics can be utilised to quantify rating behaviour of each observer over the presented stimuli. Figure 3 shows the mean PVS scores (MPS) for of the comfort versus quality scores for all 22 participants. It can be clearly seen that there is a wide range of different average quality and comfort ratings. To identify whether this 'bias' is due to the observers' different perception of the PVS, due to systematic error in the experiment design, or simply by chance is of great importance.

The scatter plot in Fig. 3 gives an overview of the averaged differences amongst the observers, however, it does not provide detailed information regarding the differences in rating behaviour. These differences are better accounted for in statistics that consider the distribution of the scores, such as the histograms of the co-occurrences of quality and comfort scores illustrated in Fig. 4. Here, one can clearly identify considerable differences in rating behaviour between the four observers. Whereas observer 10 uses both scores in different combinations, observer 17 tends to give the same scores for a particular PVS, meaning, the scores are highly correlated. Observer 18, on the other hand uses a wide range of quality scores but utilises mainly the middle of the comfort scale. Observer 22 focuses mainly on a very narrow range for both scores. These differences in rating behaviour should not be ignored and their underlying cause needs to be investigated.

The difference between any two observers can be quantified using correlations, such as the Pearson linear correlation coefficient illustrated in Fig. 5. The upper figure shows the correlations between all observers for the quality scores (above diagonal) and the comfort scores (below diagonal). The correlations are generally higher for the quality scores, which is supported by the marginal distributions in the lower plot. The lower agreement on comfort can have multiple reasons, including, more difficulty in rating or larger differences in perception, maybe due to the novelty of 3D applications.

In [START_REF] Haber | Coefficients of agreement for fixed observers[END_REF] it is argued that correlation is different from agreement and thus, should not be used to measure inter-observer agreement. This argument is supported by the simple example in Fig. 6. Here, the diagonal represents perfect correlation and perfect agreement. The solid dots, however, represent perfect correlation but poor agreement, due to the distance of the dots from the diagonal. The circles, on the other hand, represent high agreement but zero correlation.

Typically, analysis of variance (ANOVA) is used to measure whether or not differences amongst two or more factors Perfect correlation, low agreement High agreement, zero correlation Fig. 6. Difference between correlation and agreement [START_REF] Haber | Coefficients of agreement for fixed observers[END_REF]. are significant. However, in [START_REF] Haber | Coefficients of agreement for fixed observers[END_REF] it is argued that ANOVA makes two assumptions that not necessarily hold in analysing observer disagreement. Firstly, ANOVA assumes that observers have the same error variance and, secondly, it assumes that correlations between any two observer pairs are the same throughout the population. Especially the latter assumption is particularly questionable in case of subjective experiments with non-expert observers.

Given the above, so called inter-rater coefficients that are widely used in medical research and decision making could constitute a more suitable measure of inter-observer analysis.

Inter-rater coefficients

Inter-rater coefficients take into account that a certain amount of agreement between observers is due to chance alone [START_REF] Banerjee | Beyond kappa: A review of interrater agreement measures[END_REF]. As such, they aim to measure the true disagreement by identifying and segregating the chance agreement. Early coefficients include Kendall's W [10] and Scott's Pi [START_REF] Scott | Reliability of content analysis: The case of nominal scale coding[END_REF]. Cohen's κ [START_REF] Cohen | A coefficient of agreement for nominal scales[END_REF], probably the most widely used inter-rater coefficient, is an extension of Scott's Pi which does not assume that ob-Table 1. Confusion matrix applied to a 5-point scale. servers have the same rating distributions. Fleiss' κ [START_REF] Fleiss | Measuring nominal scale agreement among many raters[END_REF] is an extension of Cohen's κ from two to multiple observers.

Cohen's κ is essentially based on a confusion matrix between two observers, which is given in Table 1 for a 5-point scale. For illustration, the confusion matrices between observers 17 and 18 are given in Fig. 7 for quality and comfort scores. One can see that the agreement between the two observers is higher for the quality scores, as indicated by the high magnitudes around the diagonal.

The κ coefficient is computed as

κ = p o -p e 1 -p e = 1 n ∑ 5 i=1 n ii -1 n 2 ∑ 5 i=1 n i• n •i 1 -1 n 2 ∑ 5 i=1 n i• n •i (1)
with p o and p e denoting the observed and expected agreement, respectively. In [START_REF] Sim | The kappa statistic in reliability studies: Use, interpretation, and sample size requirements[END_REF] it is stated that the conventional κ is only suitable for nominal data and is inappropriate for ordinal data, such as given with ordinal scales in quality assessment. The reason being, that in ordinal scales there are different levels of disagreement and, hence, a weighted κ is recommended that penalises disagreements in correspondence to their severeness as follows

κ w = 1 n ∑ 5 i=1 ∑ 5 j=1 w ij n ij -1 n 2 ∑ 5 i=1 ∑ 5 j=1 w ij n i• n •i 1 -1 n 2 ∑ 5 i=1 ∑ 5 j=1 w ij n i• n •i .
(2) Note that the unweighted κ is a special case of the weighted κ with w ij = 1 for i = j and w ij = 0 for i ̸ = j. Here, we use linear weights w ij = 1 -0.25|i -j|. Both the weighted and unweighted κ are measured on a scale from -1 to 1, with 1 indicating perfect agreement and 0 meaning agreement by chance. Values below zero indicate agreement less than chance and are very rare. To improve interpretation of κ, an ordinal scale was introduced in [START_REF] Landis | The measurement of observer agreement for categorical data[END_REF] as follows: ≤0 = poor, 0.01-0.2 = slight, 0.21-0.4 = fair, 0.41-0.6 = moderate, 0.61-0.8 = substantial, and 0.81-1 = almost perfect. However, the authors themselves admitted that the choice of ranges and labels was somewhat arbitrary, for which reason we do not consider them here.

In Fig. 8, Cohen's weighted κ w between all observers is presented for the quality scores (above diagonal) and the comfort scores (below diagonal). The red crosses additionally mark the κ w where the null hypothesis that agreement is accidental is accepted (based on a p value of 0.05). It can be seen that the κ w are generally lower and the chance agreement is higher for the comfort scores as compared to the quality scores. It is further apparent, that observer 22 not only exhibits chance agreement with the majority of observers for the comfort scores, but also for the quality scores. This disagreement is also indicated in the correlations (see Fig. 5), however, its severeness is more apparent using the κ w coefficient.

Data mining

Within this framework, data mining techniques are instrumental to identify patterns in the experiment data and to determine possible observer groups. These techniques are particularly useful in cases where an extensive amount of data is available and when the dimensionality of the assessment task exceeds easy comprehension of the results. Furthermore, powerful and freely available data mining software, such as the Waikato Environment for Knowledge Analysis (WEKA) [START_REF] Hall | The WEKA data mining software: An update[END_REF] or RapidMiner [START_REF]RapidMiner[END_REF], allow for straightforward application of a wide range of data mining algorithms. In [START_REF] Tan | Introduction to Data Mining[END_REF], data mining techniques are generally separated into four categories, each of which finds their application within inter-observer analysis. Clustering [START_REF] Berkhin | A survey of clustering data mining techniques[END_REF] is considered to be particularly useful to identify user groups with respect to one or multiple different attributes. A range of different algorithms [START_REF] Wu | Top 10 algorithms in data mining[END_REF], such as kmeans, expectation maximisation (EM), and support vector machines, provide powerful tools to identify these patterns. We applied EM clustering with 5 clusters to the MPS presented in Fig. 3. The results are presented in in Fig. 9 with different markers indicating the association of a data point to a cluster. It can be seen, that 5 clusters are created by the EM algorithm, however, the separation between clusters is not always very distinct. Considerations of inter-observer analysis already during the quality experiment (see Section 2.1) may improve cluster separation.

Association analysis can be used to discover strongly associated attributes in the data, which is particularly useful for experiment data where a large number of different factors are considered. Implementations such as the Apriori algorithm facilitate efficient extraction of patterns in such high dimensional experiment data.

Anomaly detection devotes itself to the identification of data that is significantly different from the rest of the data and is thus particularly useful for outlier detection. Various techniques utilising proximity, density, or clustering-based algorithms cover outlier detection for low to high dimensional experiment data.

Predictive modelling is useful to infer models from the experiment data using classification (e.g. decision trees) or regression-based methods for discrete and continuous data, respectively. Typically, predictive modelling is conducted using the MOS, however, more detailed information about individual observer ratings may help to derive models that take into account the complexity of multimedia QoE assessment.

In general, data mining provides powerful tools for interobserver analysis. However, a careful selection of suitable algorithms along with a thorough parameterisation are essential to maximise the usefulness of the output. The pre-processing of the data, as discussed in Section 2.2, is crucial to maximise the performance of the data mining algorithms. Strong integration of data mining techniques into inter-observer analysis is an ultimate objective of the proposed framework.

DATA EXPLOITATION

As with conventional quality assessment, appropriate visualisation of the analysis outcomes is essential for comprehension of the results with respect to the intended goals of the experiment. In case of inter-observer analysis the focus should thus be on emphasising the differences amongst observers rather than different PVS. An appropriate visualisation is the first step towards successful data interpretation and in fact, in some cases visualisation alone can help to identify patterns in the data (often referred to as visual data mining). In case of high-dimensional data, a technique called online analytical processing (OLAP) [START_REF] Tan | Introduction to Data Mining[END_REF] is particularly useful as it aggregates data to aid subsequent analysis. Several of these visualisation techniques have been discussed in this paper, a more comprehensive selection is given in [START_REF] Tan | Introduction to Data Mining[END_REF].

Interpreting the data and making sense of it is possibly the most difficult part of the framework. From the discussion thus far it is obvious that there are indeed strong differences between observers in terms of bias and rating patterns. However, to identify what exactly these differences stem from (internal versus external factors, systematic versus random differences, ...) is a highly difficult task and in fact, the authors believe that data interpretation would benefit from a careful deployment of the entire framework, starting with the experimental design.

Thoroughly interpreted inter-observer analysis outcomes are expected to be beneficial towards the improvement of subjective experiment procedures as well as for the improvement of QoE prediction models. For instance, in [START_REF] Wang | Subjective evaluation of HDTV stereoscopic videos in IPTV scenarios using absolute category rating[END_REF] it was found that there was a bias between the scores that were given in the two experiments, which could be caused by differences in the conduction of the experiment. However, they could also be due to cultural differences between the French and Swedish participants or other factors that yet need to be identified.

CONCLUSIONS

We propose a framework of inter-observer variability analysis in the context of multimedia quality assessment. The paper serves to both, raise awareness to the importance of inter-observer analysis and to provide representative tools to conduct it. The different parts of the framework have been discussed in relation to a 3D video quality experiment. A considerable number of references is provided as a roadmap for the reader to further explore the discussed framework.

The results presented here are only a first step towards a more complete framework and many aspects are yet to be investigated. For instance, other inter-observer coefficients need to be investigated, including bivariate coefficients [START_REF] Shoukri | Bivariate modeling of interobserver agreement coefficients[END_REF] that simultaneously account for two variables (here: quality and comfort). Also, the range of available data mining techniques is vast and yet to be explored to fully exploit their power in identifying patterns in the data. The authors also plan to extend the framework to other domains, such as to the identification in observer differences of gaze patterns recorded through eye tracking [START_REF] Engelke | Analysing interobserver saliency variations in task-free viewing of natural images[END_REF]. Finally, the framework is planned to be applied to exhaustive data sets (such as the VQEG common set) and a dedicated experiment is planned taking the objective of inter-observer analysis into account already during the design of the procedures.

Fig. 3 .

 3 Fig. 3. Quality and comfort MPS including error bars.

Fig. 4 .

 4 Fig. 4. Co-joint quality and comfort ratings for 4 observers.

Fig. 5 .

 5 Fig. 5. Inter-observer correlations for quality scores (above diagonal) and comfort scores (below diagonal) and their marginal distributions.

Fig. 7 .

 7 Fig. 7. Confusion matrices between observers 17 and 18 for (a) quality and (b) comfort scores.

Fig. 8 .

 8 Fig. 8. Cohen's weighted κ w between observers for quality scores (above diagonal) and comfort scores (below diagonal).

Fig. 9 .

 9 Fig. 9. Clustering using EM and 5 clusters.


Yohann Pitrey
email: yohann.pitrey@univ-tes.fr

Marcus Barkowsky
email: marcus.barkowsky@univ-tes.fr

Patrick Le Callet
email: patrick.lecallet@univ-tes.fr

Romuald Pépion
email: romuald.pepion@univ-tes.fr

Subjective Quality Evaluation of H.264 High-Definition Video Coding versus Spatial Up-Scaling and Interlacing

Keywords: HDTV, Subjective Quality Assessment, MPEG-4 AVC/H.264

The upcoming High-Definition format for video display provides high-quality content, especially when displayed on adapted devices. When combined with video coding techniques such as MPEG-4 AVC/H.264, the transmission of High-Definition video content on broadcast networks becomes possible. Nonetheless, transmitting and decoding such video content is a real challenge. Therefore, intermediate formats based on lower frame resolutions or interlaced coding are still provided to address targets with limited resources. Using these formats, the final video quality depends on the postprocessing tools employed at the receiver to upsample and de-interlace these streams. In this paper, we compare the full-HD format to three possible scenarios to generate a full-HD stream from intermediate formats. We present the results of subjective tests that compare the visual quality of each scenario when using the same bitrate. The results show that using the same bitrate, the videos generated from lower-resolution formats reach similar quality compared to the full-HD videos.

INTRODUCTION

Recent technological advances make transmission of High-Definition (HD) streams possible on broadcast networks, reaching a continuously increasing number of users. Combined with continuously growing network transfer capacities, this breakthrough was made possible by the advances in the video coding area. One of the most recent and most efficient is MPEG-4 AVC/H.264 [START_REF] Wiegand | Overview of the H.264/AVC video coding standard[END_REF]. This standard summarizes all the efforts in video compression from the last 20 years, and achieves good compression rates while maintaining good visual quality [START_REF] Ostermann | Video Coding with H.264 / AVC: Tools, Performance & Complexity[END_REF].

Currently, several HD video formats are used by service providers. The so-called full-HD format 1080p50 consists in frames of 1920 × 1080 pixels, displayed at 50 frames per second [START_REF]5 Recommendation. Parameter values for the HDTV standards for production and international programme exchange[END_REF]. This format is quite demanding, because of the large size and high temporal frequency of the images to display. Therefore, intermediate formats are used to address smaller screens, networks with lower transmission rates and devices with limited processing abilities. Some formats use lowerresolution frames such as the 720p50 format (1280 × 720 pixels), or they use interlacing such as the 1080i50 format (1920 × 540 pixels per field). For displaying, the video often needs to be up-scaled and de-interlaced in order to match the full-HD native display resolution. Many techniques exist with variable complexities to perform this type of postprocessing and reduce the visual artifacts. In interlaced mode (in opposition to progressive mode), each actual frame is divided into two fields (the top and the bottom field) containing half the number of lines in the original frame. To get a full-size frame, the two fields are merged and refreshed alternatively, with half the temporal frequency of the original sequence.

In this paper, the performance of the full-HD format is compared with spatially down-scaled and/or interlaced formats, in terms of subjective visual quality. The new MPEG-4 AVC/H.264 video coding standard is used to generate streams that can be transmitted over current broadcasting channels. Using the same bitrate for each format, the impact of the coding artifacts is compared to the impact of spatial up-scaling and/or de-interlacing. Three different bitrate scenarios are proposed to study the performance of each format in low-, medium-and high-bitrate contexts. The influence of the up-scaling techniques on the visual quality is also evaluated by comparing two well-known algorithms called the Lanczos and bilinear up-scalers. Some Standard-Definition (SD) streams are included in our experiment, to show the contribution of HD over SD in a full-HD display context. This paper is organized as follows. Section 2 presents the considered application, and the configurations tested in our experiment. Section 3 describes the subjective quality test methodology used in this experiment. Section 4 presents and discusses the experimental results. Finally, section 5 concludes the paper.

DESIGN OF EXPERIMENT

The purpose of our experiment is to compare different ways of generating a full-HD video stream from various input formats. In the considered scenario, only one version of the video stream would be transmitted on a network channel, followed by postprocessing operations to adapt it to the requirements of a full-HD display device.

Six common video formats are studied with various frame sizes, using interlaced or progressive coding (cf. Table 1).

To ease the reading in the following, we refer to these formats using the abbreviations defined in Table 1. Format 1 is the upcoming full-HD format. No postprocessing operation is needed to display it on a full-HD display. Currently, format 2 is used by many HD-broadcast applications. The interlacing has an impact on the quality of the displayed video, especially for scenes with high motion and high spatial detail. Format 3 is used by many service providers and it is a typical native resolution for HD-ready TV screens.

The spatial up-scaling operation used to display a full-HD stream has an impact on the visual quality, due to the interpolation process. In the case of format 4, the image is only up-scaled horizontally by a ratio of 1.5. For format 5, the images further need de-interlacing combined with a similar up-scaling step. Format 6 is the current Standard Definition (SD) format. The aspect ratio is different from the HD format. The content of the output video thus appears horizontally stretched, which can affect the visual quality.

Five full-HD reference video sequences were processed. Figure 1 displays a snapshot for each one of these videos. They contain a wide variety of contents, including high motion and/or texture information. All the streams were generated from these reference sequences using interlacing and/or spatial down-scaling. The interlacing process is based on a classical convolutional technique. The down-scaling method is based on the 2D-Lanczos filter which calculates the value of a pixel from the values of the neighboring pixels in the input image, using a Mexican-hat shaped function.

The de-interlacing algorithm used is called TDeint. It is based on a motion-adaptive scheme to reconstruct frames from the fields in the interlaced video. It is currently known as one of the best de-interlacing methods. Two up-scaling methods are compared in our experiment. The first method is called bilinear up-scaling, and performs a simple 2D-linear interpolation to generate the output image. The visual performance of this up-scaling method is acceptable, while its processing complexity is comparably low. The second upscaling method uses the Lanczos filter. This up-scaler is known to have good performance, at the cost of a higher complexity in terms of calculation. In the following, these two up-scaling methods are referred to using B and L.

After generating the formats reported in Table 1, each video is encoded using the reference MPEG-4 AVC/H.264 encoder For each condition tested in the experiment, three dimensions have to be specified (bitrate, resolution and up-scaling filter function). We introduce the notation of a specific configuration with the following example: 6.3.L refers to the video encoded at 6 Mb/s using format 3 (720p50) and upscaled by the Lanczos filter.

SUBJECTIVE EXPERIMENT

To evaluate the visual quality of the presented scenarios, we used the Absolute Category Rating (ACR) with 11 levels methodology. The details about this methodology can be found in the IUT-T P-910 Recommendation [START_REF]Rec. Subjective video quality assessment methods for multimedia appl[END_REF]. This methodology uses a single stimulus, in which the viewer is presented one video sequence at a time. For each tested sequence, the viewer gives a score according to the level of perceived quality, located on a scale with 11 levels corresponding to adjectives ranging from bad to excellent quality.

The test was presented to 26 non-expert viewers. Their age ranged from 19 to 50, with an average of 27. Each test session took 45 minutes, starting with a visual acuity and Ishihara color plate test. According to the video contents, frame formats, up-scaling methods and bitrates included in the experiment, 145 videos of 10 seconds each were generated. After a short training, the viewer was asked to rate each video, presented in random order. They were displayed on a 40inch diagonal full-HD reference screen (TVLogic LVM401) and the viewing distance was equal to 3 times the height of the display, according to the ITU recommendations.

Figures 2 to 4 display the Mean Opinion Scores (MOS) obtained on each scenario. Each bar represents the average MOS of a given scenario on the five presented video sequences. The intervals of confidence are displayed as line segments at the top of each bar. They represent the variability of the opinions of the viewers on the quality scale and can be used as a first approximation for the significance of the difference between MOS. If two intervals of confidence overlap, the difference between the corresponding MOS can be considered non-significant. To provide a more precise statistical analysis, Table 2 includes the results of the student t-test, using the introduced notation. The 29 configurations tested on each video content are compared in terms of statistical difference. If two configurations are not statistically different, the corresponding cell in Table 2 contains a '•' symbol. If two configurations i and j are statistically different, it means it is possible to order them in terms of visual quality. If the MOS of configuration i is higher than the MOS of configuration j, the cell located at the i-th line and j-th column in Table 2 contains a '↑' symbol. In the opposite case (the MOS of configuration i is less than the MOS of configuration j), the cell contains a '↓' symbol.

On the three charts in Figures 234, the left-most score corresponds to the non-coded 1080p50 reference. As expected, this scenario obtains the highest score, as the input format does not need de-interlacing nor up-scaling, and there are no coding artifacts. The difference in MOS between the reference and the coded 1080p stream is comparably high at 3 Mb/s. It shows that the coding artifacts have a strong impact on the visual quality at this bitrate. The difference decreases in the 6 Mb/s scenario to almost 2 MOS levels, and even further for the 9 Mb/s scenario. From Table 2, we can further assess that in this scenario, the coded 1080p stream can not be differentiated from the non-coded video.

It can be observed on Figures 2 and3 that the difference between the MOS of the SD and the HD versions is particularly high. One of the main reasons is that the spatial up-scaling does not conserve the aspect ratio, thus deforming the content. In the 3 Mb/s scenario, the average difference between the SD and the HD conditions is about 2 MOS, while in the 6 Mb/s scenario this difference is about 3 MOS. This demonstrates the advantage of HD when compared to SD in terms of visual quality, in particular for higher bitrates. For MPEG-4 AVC, the quality difference between 6 Mb/s and 9 Mb/s for the SD format was considered to be negligible during a pre-test process, therefore the 9 Mb/s SD conditions (i.e.: 9.6.X configurations in our notation) were not included in the experiment.

The 1080p stream coded with MPEG-4 AVC obtains a score equivalent to the scores of the up-sampled format 3 and 5 versions in the three bitrate scenarios. The impact of the coding artifacts on visual quality is thus equivalent or slightly higher than the loss in resolution from the spatial upscaling. This constatation is particularly true in the 3 Mb/s scenario, for which the scores of format 3 are sometimes higher than the scores of format 1. Still, this difference can not be considered as significant from the results of the t-test.

The two up-scaling methods show comparable performance in the three bitrate scenarios and they are not statistically differentiable in 10 out of 11 cases. The scores of the Lanczos method are though slightly higher than the scores of the bilinear method. The bilinear method could be applied in time-critical applications or in contexts where the processing power is limited. A slight advantage can be granted to progressive streams when compared to the interlaced streams. This can be observed between format 1 and format 2, for which the quality change is always statistically significant. An interesting result is that the 1080i50 format encoded at 9 Mb/s is not statistically different from the 1080p50 encoded at 6 Mb/s. This illustrates the interest of progressive coding against interlaced coding in a medium to high bitrate context.

The results show that formats 1 and 3 are equivalent in terms of visual quality for the 3 and 6 MB/s scenarios (this is also valid in the 9 Mb/s scenario with the Lanczos up-scaling method). A possible explanation is that the down-scaling step performed before encoding the video in 720p50 from

↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ 3.1 ↓ ↑ • • • • • • ↑ ↑ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↑ ↑ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ 3.2 ↓ ↓ ↓ ↓ • • ↓ ↓ ↑ ↑ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↑ ↑ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ 3.3.B ↓ • ↑ • ↑ ↑ • • ↑ ↑ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↑ ↑ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ 3.3.L ↓ • ↑ • ↑ ↑ • • ↑ ↑ ↓ ↓ ↓ ↓ • ↓ ↓ ↓ ↑ ↑ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ 3.4.B ↓ • • ↓ ↓ • ↓ ↓ ↑ ↑ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↑ ↑ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ 3.4.L ↓ • • ↓ ↓ • ↓ ↓ ↑ ↑ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↑ ↑ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ 3.5.B ↓ • ↑ • • ↑ ↑ • ↑ ↑ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↑ ↑ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ 3.5.L ↓ • ↑ • • ↑ ↑ • ↑ ↑ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↑ ↑ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ 3.6.B ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ • ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ 3.6.L ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ • ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ 6.1 ↓ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ • • ↑ ↑ • • ↑ ↑ ↓ • • ↓ • • ↓ ↓ 6.2 ↓ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↓ ↓ ↓ • • • ↓ ↑ ↑ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ 6.3.B ↓ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ • ↑ • ↑ ↑ • • ↑ ↑ ↓ • • ↓ • • ↓ ↓ 6.3.L ↓ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ • ↑ • ↑ ↑ ↑ • ↑ ↑ ↓ • • ↓ • • • ↓ 6.4.B ↓ ↑ ↑ ↑ • ↑ ↑ ↑ ↑ ↑ ↑ ↓ • ↓ ↓ • ↓ ↓ ↑ ↑ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ 6.4.L ↓ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↓ • ↓ ↓ • • ↓ ↑ ↑ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ 6.5.B ↓ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ • • • ↓ ↑ • • ↑ ↑ ↓ • ↓ ↓ • • ↓ ↓ 6.5.L ↓ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ • ↑ • • ↑ ↑ • ↑ ↑ ↓ • • ↓ • • ↓ ↓ 6.6.B ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↑ ↑ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ • ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ 6.6.L ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↑ ↑ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ • ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ 9.1 ↓ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ • ↑ ↑ • • 9.2 ↓ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ • ↑ • • ↑ ↑ • • ↑ ↑ ↓ • ↓ • • ↓ ↓ 9.3.B ↓ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ • ↑ • • ↑ ↑ ↑ • ↑ ↑ ↓ • ↓ • • • ↓ 9.3.L ↓ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ • ↑ ↑ ↑ ↑ • • 9.4.B ↓ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ • ↑ • • ↑ ↑ • • ↑ ↑ ↓ • • ↓ • ↓ ↓ 9.4.L ↓ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ • ↑ • • ↑ ↑ • • ↑ ↑ ↓ • • ↓ • ↓ ↓ 9.5.B ↓ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ • ↑ ↑ ↑ ↑ ↑ ↑ • ↑ • • ↑ ↑ • 9.5.L ↓ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ • ↑ ↑ • ↑ ↑ •
1080p50 acts as a low-pass filter on the original video. Small irregularities and noise from the original sequence can thus be smoothed in the down-scaled version, which in some cases eases the encoding. This also holds for the receiver as the up-scaling process might counteract the blocking artifacts.

In our experiment it was observed that the up-scaled video can have a quality that is very close to the 1080p50 video when using a high quality up-scaler such as the bilinear or the Lanczos filter. Thus a transmission in 720p50, 1280×1080p and 1280×1080i formats may be considered. This might be advantageous for the sender or the receiver. Due to the smaller resolution, the computational complexity of the encoding and decoding is reduced. Since the downscaling and up-scaling operations do usually not require an equivalent amount of computation, the overall complexity is reduced. Thus, on the sender side computational resources might be reallocated in order to evaluate more sophisticated video coding options. On the receiver side, the up-scaling is usually performed by hardware, e.g. the graphics card. As a result the reduced computational complexity in the decoder might lead to a longer battery life in the case of a portable device. A disadvantage of decoding a lower resolution video might be the increased memory bandwidth requirement resulting from this two-step approach in which an intermediate image has to be stored.

CONCLUSION

In this paper, we used the results of subjective tests to compare the performance of various High-Definition video formats coded with MPEG-4 AVC/H.264. Spatial upscaling and de-interlacing were performed to display a full-HD stream independently from the encoded format. Using both graphical and statistical analysis, the results show that with equivalent bitrate, the difference in quality between the streams coded directly in full-HD and the interlaced and down-scaled streams is mostly not visually significant.

Figure 1 :

 1 Figure 1: Sample of the High-Definition test sequences used for the experiment.

Figure 2 :Figure 3 :Figure 4 :

 234 Figure 2: Mean Opinion Scores at 3Mb/s.

Table 1 :

 1 Video formats used in the experiment.

	Format	Frame size	Progressive
	abbreviation	(pixels)	/ Interlace
	1	1920 × 1080	p
	2	1920 × 1080	i
	3	1280 × 720	p
	4	1280 × 1080	p
	5	1280 × 1080	i
	6	720 × 576	p

[4]. The encoding parameters were set to typical values, in order to get a good tradeoff between coding efficiency and computational complexity. Three bitrate scenarios were studied : 3 Mb/s, 6 Mb/s and 9 Mb/s, corresponding to low, medium and high quality of service. The non-coded full-HD video was also included in the test as a high quality reference.

Table 2 :

 2 Results of the t-test and comparison of the MOS between the tested configurations.

	Ref	3.1	3.2	3.3.B	3.3.L	3.4.B	3.4.L	3.5.B	3.5.L	3.6.B	3.6.L	6.1	6.2	6.3.B	6.3.L	6.4.B	6.4.L	6.5.B	6.5.L	6.6.B	6.6.L	9.1	9.2	9.3.B	9.3.L	9.4.B	9.4.L	9.5.B	9.5.L
	Ref