I would like to express my deepest thanks and gratitude to my supervisor Mihai MITREA for his continuous support, patience, motivation, and immense knowledge. His precious guidance and fruitful ideas

Table of Contents

List of Figures

List of Tables

 Table A-Les méthodes subjectives d'évaluation de la qualité du contenu visuel servent à quantifier les performances des systèmes multimédias avec des observateurs humains. Afin de garantir la reproductibilité et la comparabilité des résultats, il est indispensable de disposer de conditions d'évaluation bien configurées en se référant, par exemple aux recommandations de l'Union Internationale des Télécommunications (UIT) (UIT-R BT.1788, R BT.500-11, UIT-R BT.500-13, UIT-T P.913,…).

Les spécifications de l'UIT définissent diverses exigences relatives à l'infrastructure, à la conception du groupe d'observateurs (le panel) et à la manière dont les sessions de test doivent être conduites. Tout d'abord, l'infrastructure et les conditions de test sont définies. Par exemple, l'éclairage environnemental, la distance entre l'écran et le spectateur ou l'angle de vue doivent être correctement configurés avant l'évaluation. Ensuite, la taille du panel est dimensionnée en fonction de la sensibilité et de la fiabilité de la procédure de test. Par exemple, il est généralement recommandé d'avoir au moins 15 observateurs, tandis que leur acuité visuelle et leur vision couleur sont testées conformément à des procédures préétablies, comme les diagrammes de Snellen ou Ishihara. Les méthodes d'évaluation et les échelles de notation sont également spécifiées. Les méthodes le plus souvent considérées sont la DSCQS (Double-Stimulus Continuous Quality-scale), DSIS (Double-Stimulus Impairment Scale) ou bien encore SSCQE (Single-Stimulus Continuous Quality Evaluation). Enfin, une fois l'évaluation de l'échelle choisie, les notes attribuées par les observateurs sont enregistrées et ensuite interprétées statistiquement. Par exemple, lorsqu'ils considèrent l'échelle SSCQE à 5 niveaux, les observateurs notent en sélectionnant l'une des étiquettes suivantes: Excellent, Good, Fair, Poor, Bad (dont la traduction française est Excellent, Bon, Satisfaisant, Médiocre, et Mauvais). Par la suite, ces étiquettes sont converties en valeurs entières (Excellent est mappé à 5 tandis que Bad est mappé à 1); les valeurs aberrantes sont éliminées et le MOS (Mean Opinion Scores, c'est-à-dire la moyenne de ces valeurs) est calculé en même temps que ses limites de confiance.

Notre thèse porte sur divers aspects liés à la spécification des échelles de gradation, de leur fondement théorique à leur utilisation pour les vidéos 2D et stéréoscopiques.

Limitations de l'état de l'art

Depuis 100 ans déjà, divers domaines de recherche (psychologie, psychophysique, sociologie, marketing, médecine, ...) ont envisagé l'utilisation d'échelles d'évaluation dans les évaluations subjectives. Malgré cette longue et fructueuse histoire, aucun consensus n'est encore atteint sur l'utilisation d'une échelle spécifique dans un but spécifique, et plusieurs types d'échelles coexistent encore tout se contredisant: échelles étiquetées graphiques / numériques / sémantiques ou continues / discrètes.… De plus, la dynamique des échelles numériques varie avec l'expérience: par exemple, les échelles continues peuvent aller de 0 à 75, à 100, à 120 ou même à 200, tandis que Les échelles discrètes peuvent comporter entre 2 et 11 classes d'évaluation.

Pour l'évaluation du contenu visuel, les Recommandations de l'UIT ont prouvé leur efficacité et sont déjà utilisées de manière intensive dans plusieurs études de recherche visant une grande variété d'applications (évaluation / étalonnage d'appareils, compression, reconstruction d'images en 3D, tatouage, etc.). Certaines études utilisent des échelles à 5 niveaux de qualité tandis que d'autres sur 11 niveaux. Pourtant, aucune réponse sur la manière de choisir ce nombre ou les niveaux de qualité euxmêmes n'est fournie. De plus, la Rec. UIT-T P.913 va au-delà de ces recommandations et fournit une liste de modifications acceptables (telles que l'utilisation d'échelles non étiquetées, l'utilisation d'échelles numérotées non étiquetées, etc.) La relation entre les échelles continue et discrète est abordée de manière récurrente dans les études de recherche. Par exemple, il a été montré que les évaluations de l'échelle discrète présentent le niveau de stabilité le plus élevé, du moins pour l'expérience considérée (une auto-évaluation des connaissances antérieures en statistique). Cette étude soulève également une interrogation quant à la signification même des termes continus et discrets lors des évaluations subjectives.

L'impact des étiquettes sémantiques est discuté et détaillé dans diverses études de recherche. Selon certaines études, les étiquettes UIT adjacentes sont caractérisées par des distances sémantiques non uniformes ; pourtant, un tel comportement n'est pas quantifié. D'autre part, certaines études affirment le contraire, c'est-à-dire que la sémantique des étiquettes UIT adjacentes n'a pas d'impact sur les résultats. Si certains résultats correspondent à des études subjectives effectuées pour différentes langues (japonais, allemand, anglais, français et italien), la Rec. UIT-T P.913 postule explicitement que le MOS est invariant vis-à-vis de la traduction des étiquettes sémantiques, mais ne fournit aucun motif pour cela.

Au-delà des particularités de la stratégie d'évaluation elle-même, des doutes sur le modèle statistique des résultats d'évolution (les scores) surgissent: l'hypothèse implicite de la distribution gaussienne est aussi contredite dans l'interprétation de certains résultats dans UIT-R BT.500 11/13. Enfin, bien que l'UIT demande toujours que le nombre de sujets impliqués dans les évaluations soit supérieur à 15 (par exemple, UIT-R BT.500-13), les études expérimentales mettent en évidence une grande variabilité de ce paramètre ; dans ce contexte, exprimer de manière pragmatique l'influence théorique de la taille du panel d'observateurs quant à l'exactitude des résultats reste une question ouverte.

Objectifs

La présente thèse a pour objectif principal d'analyser les recommandations de l'UIT et d'étudier théoriquement certains de leurs aspects essentiels liés aux échelles d'évaluation. Trois axes principaux de recherche sont pris en compte. D'abord, l'étude sera menée au niveau théorique. À cet égard, la thèse tentera de relier les procédures d'évaluation des échelles continue et discrète et de déterminer si le nombre de classes sur les échelles discrètes est un critère relevant dans les interprétations des résultats ou un simple paramètre. L'étude de l'influence du modèle statistique des scores et de la taille du panel (nombre d'observateurs) dans l'exactitude des résultats entre également dans le cadre de la thèse.

Deuxièmement, au niveau méthodologique, la thèse abordera la question de la quantification du biais induit dans les expériences de qualité vidéo subjective par les étiquettes sémantiques (par exemple, Excellent, Good, Fair, Poor et Bad) généralement associées aux échelles d'évaluation discrètes.

Enfin, d'un point de vue expérimental, les deux directions mentionnées nécessitent des conditions expérimentales capables d'appuyer leur précision et leur pertinence statistique. En conséquence, spécifier et déployer un tel cadre expérimental devient un objectif en soi. De cette manière, une étude sur la variabilité pratique des résultats expérimentaux et méthodologiques avec le type de contenu (vidéo 2D ou stéréoscopique) ou avec sa qualité (évaluée par des métriques objectives) devient également possible.

Contributions théoriques

Il est théoriquement démontré que les scores attribués par les observateurs sur une échelle de notation continue peuvent être mappés a posteriori sur toute échelle de notation discrète, avec une précision statistique contrôlée. À cet égard, en utilisant des transformations de variables aléatoires non-linéaires, la formule reliant les fonctions de densité de probabilité modélisant les échelles de gradation continue et discrète est établie. Les moments du premier et du deuxième ordre (permettant de calculer le MOS et les limites de confiance) sont ensuite déduits.

Ces résultats sont génériques et ne tiennent pas compte de la particularité de la fonction de densité de probabilité d'origine ni du partitionnement sur une échelle discrète (les partitions égales et nonuniformes sont couvertes). Cependant, dans la perspective de transformer ces formules théoriques en outils efficaces d'évaluation de la qualité visuelle, il est nécessaire de disposer d'une description statistique de la densité de probabilité modélisant les scores attribués par l'observateur sur l'échelle continue. Par conséquent, l'étude sous-jacente est également menée. Elle commence par examiner l'hypothèse gaussienne qui est rejetée (sur la base du test de concordance statistique de Chi-2) et propose ensuite des estimations basées sur des mélanges gaussiens. L'impact du nombre d'observateurs humains sur la précision de l'évaluation de la qualité est ensuite évalué

Contributions méthodologiques

Afin de quantifier l'impact psycho-cognitif des étiquettes sémantiques dans l'évaluation de la qualité, la thèse avance une procédure d'investigation interdisciplinaire ; cette procédure combine les formules théoriques mentionnées, l'estimation statistique et les tests statistiques. D'abord, afin de déterminer si un tel impact sémantique existe, une comparaison basée sur le test de Student est effectuée entre les valeurs moyennes (représentant le MOS) correspondant aux échelles continue (non étiquetée) et discrète, sémantiquement étiquetée. Comme les expériences démontrent que l'impact sémantique existe, une procédure pour sa quantification est définie par la suite. La deuxième étape consiste donc à définir une variable aléatoire discrète auxiliaire, caractérisée par une partition inégale mais par des probabilités a posteriori égales. L'impact sémantique est quantifié en définissant un coefficient. Cette variable aléatoire auxiliaire est estimée par des tests binomiaux entre les différences de longueur des classes de partition entre cette variable aléatoire auxiliaire et la variable aléatoire correspondant à l'échelle étiquetée sémantiquement,

Étude expérimentale

L'étude expérimentale est réalisée conformément aux recommandations UIT-R BT.1788, UIT-R BT.500-11, UIT-R BT.500-13 et UIT-T P.913 et prend en compte la méthode SSCQE. Il convient de souligner que la thèse examine la procédure d'évaluation elle-même et n'est pas centrée sur le contenu à évaluer. Par conséquent, le cadre expérimental conçu et déployé dans la thèse devrait adapter / étendre les spécifications UIT de manière à s'adapter à cet objectif.

Une application Android a été développée afin de permettre alternativement l'évaluation sur des échelles continues et discrètes sémantiquement étiquetées. La première échelle se situe entre 0 et 100, avec 10 points équidistants (les scores sont enregistrés avec une précision unitaire). Le deuxième type d'échelle analyse alternativement les évaluations sur 5 et 3 niveaux, avec les libellés suivants: Excellent, Good, Fair, Poor, Bad et respectivement Good, Fair, Bad.

Le contenu à évaluer est représenté par quatre corpora composés par des vidéos de haute et basse qualité (selon des mesures objective a priori de qualité), vidéo 2D et stéréoscopique. Chacun de ces corpus dure environ 20 minutes. Notez que lors de l'étude de la procédure d'évaluation, le contenu luimême ne fait que garantir des conditions réalistes de notation. Au total, 640 observateurs humains participent aux expériences. Ils sont regroupés en quatre panels principaux de 160 observateurs chacun, un panel pour chaque type de contenu. Afin de garantir la répétabilité des résultats, chaque panel principal a été divisé en trois sous-panels, appelés panels de référence (60 observateurs), de validation (50 observateurs) et de contrôle croisé (50 observateurs). Les sujets des panels de référence ont attribué des scores sur les échelles continues. Les sujets des panels de validation et de vérification croisée sont finalement chacun divisés en deux sous-ensembles et ont attribué des notes sur des échelles de notation à 5 et 3 niveaux.,.

La relevance statistique des résultats est assurée en calculant des limites de confiance de 95% pour le MOS et en appliquant les différents tests statistiques (Chi-2, Student et binomial) à un niveau de signifiance statistique de 0,05.

Context

The subjective visual quality assessment methods are essentially used to gauge the performance of multimedia systems with the help of responses obtained from observers who investigate the content displayed by the system under test. Hence, in order to ensure the reproducibility and comparability of the results, well-configured, consensual evaluation conditions are particularly required and the International Telecommunication Union (ITU) Recommendations save as a ground in this respect (ITU-R BT.1788, ITU-R BT.500-11, ITU-R BT.500-13, ITU-T P.913, …).

The ITU specifications define various requirements related to the infrastructure, the design of the panel of observers and the way in which the test sessions should be conducted. First, the test infrastructure and the environment conditions are defined. For instance, the environmental illumination, the distance between the screen and the viewer or the angle of view should be properly set-up prior to the evaluation. Secondly, the panel size is designed depending on the sensitivity and reliability of the test procedure and on the anticipated effect sought. For instance, the number of observers is usually recommended to be at least 15 while their visual acuity and color vision are tested according to preestablished procedures, like the popular Snellen or Ishihara charts. Thirdly, the evaluation methods and the underlying grading scales for subjective quality assessment are specified. Among these, the most popular are the Double-Stimulus Continuous Quality-Scale (DSCQS), the Double-Stimulus Impairment Scale (DSIS) and the Single Stimulus Continuous Quality Evaluation (SSCQE). Finally, once the scale evaluation is chosen, the scores assigned by the observers are recorded and subsequently statistically interpreted. For instance, when considering the 5-levels SSCQE scale, the observers score by selecting one of the following labels: Excellent, Good, Fair, Poor and Bad. Subsequently, these labels are converted into integer values (Excellent is mapped to 5 while Bad is mapped to 1); the outliers are eliminated and the MOS (mean opinion score, i.e. the average of these values) is computed alongside with its confidence limits.

The present thesis focuses on various aspects related to the grading scales specification, from their theoretical basis to their usage for conventional (2D) and stereoscopic video.

State-of-the-art limitations and constraints

For 100 years already, various research fields (psychology, psychophysics, sociology, marketing, medicine, ...) have considered the use of rating scales in subjective evaluations [START_REF] Freyd | The graphic rating scale[END_REF], [START_REF] Froberg | Methodology for measuring health-state preferences-ii: scaling methods[END_REF]. Despite this long and fruitful history, no consensus is reached yet on a usage of a specific scale for a specific purpose, and several scale typologies still coexist and contradict each-other: graphic vs. numerical vs. semantic labeled scales or continuous vs. discrete scales or … Moreover, the dynamics of numerical scales is varying with the experiment: for instance, continuous scales can range from 0 to 75, to 100, to 120 or even to 200 [START_REF] Aitken | Measurement of feelings using visual analogue scales[END_REF], [START_REF] Bond | The use of analogue scales in rating subjective feelings[END_REF], [McG84] while the discrete scales can feature between 2 and 11 evaluation classes [START_REF] Freyd | The graphic rating scale[END_REF].

For the visual content evaluation, the ITU Recommendations have proven their effectiveness and are already intensively used in several research studies aiming at a large variety of applications (device evaluation/calibration, compression, 3D image reconstruction, watermarking, etc.). Some studies report experiments had carried out on 5 quality levels while other on 11 quality levels. Yet, no answer on how to choose either this number or the quality levels themselves is provided. Moreover, the ITU-T P.913 goes beyond such particular evaluation set-ups and provides a list of some acceptable changes (like the use of unlabeled scales, the use of numbered unlabeled scales, etc.) and even of some acceptable yet discouraged changes (like increasing the number of qualitative levels). Here again, all these modifications are left to the experimenter choice and no particular guide is provided.

The relationship between continuous and discrete scales is recurrently addressed in research studies. For instance, [START_REF] Svensson | Comparison of the quality of assessments using continuous and discrete ordinal rating scales[END_REF] shows that assessments on the discrete scale have the highest level of stability, at least for the experiment under consideration (a self-assessment of the previous knowledge in statistics). This study also raises a concern about the very meaning of the continuous and discrete terms during the subjective evaluations.

The impact of semantic labels is discussed and detailed in various research studies. On the one hand, some studies state that adjacent ITU labels are characterized by non-uniform semantic distances [START_REF] Jones | Graphic scaling of qualitative terms[END_REF], [START_REF] Narita | Graphic scaling and validity of Japanese descriptive terms used in subjective evaluation tests[END_REF]; yet, such a behavior is not quantified. On the other hand, some studies [START_REF] Zieliñski | On the use of graphic scales in modern listening tests[END_REF] claim the contrary, i.e. that the semantic of adjacent ITU labels does not impact the results. While some results correspond to subjective studies carried out for different languages (Japanese, German, English, French and Italian), the ITU-T P.913 explicitly postulates that the MOS is invariant with respect to the semantic labels translation, but does not provide any ground for this.

Beyond the peculiarities in the evaluation strategy itself, doubts about the statistical model of the evolution results (i.e. the scores) arise: the implicit assumption of Gaussian distribution [START_REF] Simone | Subjective evaluation of JPEG XR image compression[END_REF], [START_REF] Seshadrinathan | Study of subjective and objective quality assessment of video[END_REF], [START_REF] Winkler | On the properties of subjective ratings in video quality experiments[END_REF] is also implicitly contradicted in some results interpretation in ITU-R BT.500 11/13.

Finally, although the number of subjects involved in evaluations is always requested by ITU to be larger than 15 (e.g. ITU-R BT.500-13), the experimental studies bring to light a large variability of this parameter and pragmatically expressing the theoretical influence of the size of the observers panel in the accuracy of the results remains an open question.

Objectives

The present thesis has as main objective to reconsider the ITU recommendations and to investigate on theoretical basis some of their key aspects related to evaluation scales. Three main research directions are to be considered.

First, the investigation will be carried out at the theoretical level. In this respect, the thesis will try to bridge the continuous and the discrete scale evaluation procedures and to investigate whether the number of the classes on the discrete scales is a criterion meaningful in the results interpretations or just a parameter. Studying the influence of the statistical model of the scores and the size of the panel (number of observers) in the accuracy of the results is also under the scope of the thesis.

Secondly, at the methodological level, the thesis will address the issue of quantifying the bias induced in subjective video quality experiments by the semantic labels (e.g. Excellent, Good, Fair, Poor and Bad) generally associated to the discrete grading scales.

Finally, from the experimental point of view, these above-mentioned two directions require an experimental test-bed able to support their precision and statistical relevance. Hence, specifying and deploying such an experimental test-bed becomes an objective per-se. This way, an investigation on the practical variability of the experimental and methodological results with the type of content (2D or stereoscopic video) or with its quality (as assessed by objective metrics) becomes also possible.

Theoretical contribution

First, it is theoretically demonstrated that the scores assigned by the observers on a continuous grading scale can be a posteriori mapped to any discrete grading scale, with controlled statistical accuracy. In this respect, by using non-linear random variable transformations, the formula connecting the probability density functions modeling the continuous and discrete grading scales is established. The first and second order moments (allowing for the MOS and the confidence limits to be computed) are subsequently derived.

These results are generic and do not take into account either the original probability density function peculiarity or the discrete scale partitioning (both even and uneven partitions are covered). However, in the perspective of turning these theoretical formulae into effective quality evaluation tools, a statistical description for the continuous scale probability density function modeling the observer's scores is required. Hence, the underlying study is also conducted. It starts by investigating the Gaussian hypothesis that is rejected (based on the Chi-square goodness-of-fit test) and follows by suggesting approximations based on Gaussian mixtures. The impact of the number of human observers in the precision of the quality evaluation is subsequently assessed.

Methodological contribution

A cross-disciplinary investigation procedure is defined (combining the above-mentioned theoretical formulae, statistical estimation and statistical tests) to quantify the psycho-cognitive impact of the semantic labels in the quality evaluation.

First, in order to bring to light whether such a semantic impact exist, a comparison (based in the Student's paired test) between the average values (representing the MOS) corresponding to the continuous (unlabeled) and discrete, semantically labeled scales is carried out. As the experiments demonstrate that the semantic impact exists, a procedure for its quantification it is also defined. Hence, the second step is to define an auxiliary discrete random variable, which is characterized by uneven partition but by equal a posteriori probabilities. By comparing the differences in the partition classes length between this auxiliary random variable and the random variable corresponding to the semantically labeled scale, the semantic impact is quantified (by defining an underlying coefficient). This auxiliary random variable is estimated trough repeated binomial tests.

Experimental study

The experimental study is carried out according to the ITU-R BT.1788, ITU-R BT.500-11, ITU-R BT.500-13, ITU-T P.913 recommendations and considers the SSCQE method. It should be emphasize that the thesis investigates the evaluation procedure itself and is not focused on the content to be evaluated. Hence, the test-bed designed and deployed in the thesis should consider the ITU specifications as a backbone and should adapt/extend them so as to fit to the evaluation procedure investigation.

In order to score the content, a versatile Android application has been developed so as to alternatively allow the evaluation on continuous and discrete, semantically labeled scales. The former ranges between 0 and 100, with 10 even marks (the scores are recorded with unit precision). The later alternatively considers 5 levels and 3 levels evaluations, with the following labels: Excellent, Good, Fair, Poor, Bad and Good, Fair and Poor, respectively.

The content to be evaluated is represented by four corpora, representing high and low-quality video (e.g. 35dB < PSNR < 40dB and 25dB<PSNR<30dB), both 2D and stereoscopic video. Each of these corpora is about 20 minutes long and is presented to the observers into downgraded versions obtained through watermarking or compression methods (16 downgraded versions for the stereoscopic contents, 16 downgraded versions for high quality 2D content and 28 downgraded versions for the low-quality 2D content). Note that as the evaluation procedure is investigated, the content itself just ensures realistic conditions for scoring.

A total of 640 human observers are involved in the experiments. They are grouped in four main panels of 160 observers each, on panel for each type of content. In order to grant result repeatability, each main panel was split into three sub-panels, referred to as the reference (60 observers), validation (50 observers) and cross-checking (50 observers) panels. The subjects in the reference panels scores on the continuous scales. The subjects in both the validation and cross-checking panels are finally partitioned into two sets, scoring on 5 level grading scales and on 3 level grading scales, respectively.

The statistical relevance of the results is ensured by computing 95% confidence limits for MOS and by applying the various the statistical tests (Chi-square, Student's and binomial) at 0.05 significance level.

I. Introduction

The present thesis is developed under the framework of subjective visual quality evaluation, a research field for which the International Telecommunication Union (ITU) Recommendations are expected to offer a ground for ensuring the reproducibility and comparability of the results.

Hence, this Introduction chapter succinctly presents the main aspects related to an ITU test session, from the viewing conditions and the panel composition to the post-processing of the scores, passing from the evaluation methodology and the grading scales.

At least three open and surprising issues are thus identified; they relate to: (1) the continuous vs. discrete evaluation scales, (2) the statistical distribution of the scores assigned by the observers and (3) the usage of semantic labels on the grading scales.

I.1. What does image quality mean?

Quality is an ever evolving, yet ever fascinating concept whose meaning goes far beyond the objective of the present thesis.

According to the Oxford English dictionary [WEB01], the first meaning of quality is "The standard of something as measured against other things of a similar kind; the degree of excellence of something." This linguistic definition is very broad, and encompasses various aspects of our daily, professional and private lives. Hence, some 40 years ago, a more pragmatic approach steamed from business and stated that "quality is conformance to requirements, not goodness" [START_REF] Crosby | The puzzlement of quality[END_REF].

It can be considered that a milestone in the societal perception of the quality has been reached some 30 years ago when the International Organization for Standardization (ISO) elaborated a family of standards specifically addressing the quality issues [WEB02]: "ISO 9001:2015 sets out the criteria for a quality management system". This way, quality becomes an abstract concept, rather related to the creation/production process than to the results of such process.

One surprisingly aspect is common to the above-mentioned definitions: the quality is always implicitly assumed to be an a posteriori measurable notion on which we always have some a priori expectations/knowledge. Yet, the way of measuring the quality and expressing with rigor these expectations (where are they come from? how relevant are them? are they individual or shared by various persons?) remain controversial issues.

The earliest mentioning of image quality is credited to date back to the invention of optical instruments, at the beginning of the XVII st century [START_REF] Engeldrum | Image quality modeling: Where are we?[END_REF]. The term became more and more common with the introduction of photography and television, and several definitions coexist today [START_REF] Kundu | Subjective and objective quality evaluation of synthetic and high dynamic range images[END_REF], [START_REF] Marini | Content Based No-Reference Image Quality Metrics[END_REF],

[AVI01], [START_REF] Preiss | Color-Image Quality Assessment: From Metric to Application[END_REF], [START_REF] Chammem | Robust watermarking techniques for stereoscopic video protection[END_REF].

Intuitively, image quality is defined as the difference between a processed image and its original representation, from both visual and cognitive points of views. According to Janssen [START_REF] Janssen | Computational Image Quality[END_REF], in order to bring together these approaches, the image quality should be defined in the context of visuo-cognitive systems, and should relates to the "observer's interaction with his environment"; the notions of usefulness and naturalness are subsequently defined as being the image capacity of being close to both the original representation and to knowledge about the reality of the observer.

The simplest taxonomy in image/video quality evaluation [START_REF] Jahne | Digital Image Processing: Concepts, Algorithms, and Scientific Applications[END_REF] relates to two pragmatic situations.

First, according to what it is to be evaluated, no-reference and reference quality assessments are encountered. A no-reference image quality assessment assumes that solely the content under evaluation is available and that its quality is to be evaluated with respect to some pre-established, commonly (implicitly or explicitly) agreed criteria, derived from some a priori knowledge/expectations. On the contrarily, in reference quality assessment, the content under evaluation is compared to other content whose quality is supposed to serve as reference; actually, such an approach rather evaluates the difference in quality then the quality itself.

Secondly, according to how the evaluation takes place, objective and subjective evaluation procedures are deployed. An objective image quality evaluation solely relays on visual content (the one to be evaluated and, eventually, some reference content) but does not directly and explicitly considers any human observer. This class covers a large variety of image quality metrics, from the popular PSNR (peak signal to noise ratio) to the sophisticated biological-inspired measures, [BEG13]. On the contrarily, subjective evaluation procedures consist in inquiring a panel of human observers about the way they perceive the quality of the content under evaluation (be it presented by itself or in conjunction to some reference content).

Of course, such taxonomy is very broad and, according to the targeted application, one several types of evaluation quality procedures can be deployed. For instance, when considering an image acquisition system (a camera), the image quality intuitively relates to the difference between the digital, captured, image and the pristine (natural) representation of the scene. As the natural representation is not available during the evaluation, a no-reference (subjective or objective) evaluation procedure is likely to be set. However, should we be interested in comparing the performances of two cameras, we may considered the images captured by one as reference and to evaluate the second camera images with respect to this reference. When considering now a compression or watermarking application, the image quality relates to the visual differences between the original and the processed image. In such a case, both original (reference) and processed images can be made available for evaluation and the difference between them can be evaluated either by some objective or by subjective evaluations. Of course, this is not compulsory: watermarking applications can also consider no-reference quality evaluation [START_REF] Cox | Digital Watermarking[END_REF].

The present thesis focuses on the subjective, no-reference evaluation of visual content. In order to ensure the reproducibility and comparability of the results, well-configured, consensual evaluation conditions are particularly required and the International Telecommunication Union (ITU) Recommendations offers a ground in this respect (ITU-R BT.1788, ITU-R BT.500-11, ITU-R BT.500-13, ITU-T P.913, …).

I.2. ITU subjective image quality assessment

The ITU Recommendations provide methodologies for assessing the overall image quality and the overall image impairment of distorted still images and video. The main aspects related to any test session are thus specified:

 the viewing conditions: the luminance and chrominance conditions for the content under evaluation and for the ambient lighting, as well as the relative position of the observer with respect to the display;  the panel composition and size: how many observers in the panel, their typology, the way of testing their visual acuity;  the evaluation methodology and the grading scales: the evaluation/reference content availability, the way to presenting them to the observer, the way the scores are assigned, ….  the content under evaluation: the type of content, its duration, etc.  the scoring and the post-processing of the scores: score gathering, outlier detection, MOS (mean opinion score) computation, …

In the following sections, these aspects will be detailed according to ITU-R BT.1788, ITU-R BT.500-11, ITU-R BT.500-13. These three references are considered as being the widest used and, at the same time, representative for the two types of the targeted content namely the 2D video and the stereoscopic video. Note that ITU also considers possible modifications of the testing session set-up, as described in ITU-T P.913.

I.2.1. Viewing conditions

The ITU provides detailed specifications for the luminance and the chrominance conditions during the tests sessions. Note that such conditions apply both to the content displaying and to ambient lighting. Additionally, the viewing conditions also refer to the position of the observers: the angle of view with respect to the center of the display and the recommended viewing distance between the observer and the display. The latter is computed as the product between the Preferred Viewing Distance -PVD (a standard ratio value for a fixed picture high) and the picture height. Tables I-1 and I-2 present these values according to the ITU-R BT.500-11. No specific mention about the gender involvement is made.

I.2.3. Evaluation methodology and grading scales

The evaluation methodologies described in ITU-R BT.1788 and ITU-R BT.500-11/13 can be structured into three categories, referred to by ITU as double stimulus, comparison and single stimulus methods, as detailed here after. Note that from the conceptual point of view, the double stimulus and comparison methods relate to the reference quality evaluation methods while the single stimulus method to noreference quality evaluation methods.

I.2.3.1. Double stimulus methods

In its broadest usage, this type of methods is meant to provide a measure of the quality of a processed content assuming its original (pristine) content is also available and investigated during the assessment session.

Two main types of methods belong to this class, namely the Double Stimulus Continuous Quality Scale (DSCQS) and the Double Stimulus Impairment Scale Method (DSIS).

According to a DSCQS method, the observers assess the overall content quality of both reference and processed content. The results thus obtained can result in different scores between reference and test content, thus indicating a lost in quality. According to ITU-R BT.500-11/13, this evaluation takes place on scales that "provide a continuous rating system to avoid quantizing errors, but they are divided into five equal lengths which correspond to the normal ITU-R five-point quality scale", as illustrated in In DSIS case, the same reference and processed content are presented to the observer for scoring as in the DSCQS case, but only the impairments (the visual differences) are judged during the evaluation. This time, a " five-grade impairment scale should be used", as illustrated in To conclude with, DSCQS provides absolute quality evaluation for both the content under evaluation and its reference while the DSIS evaluates the impairments between these two types of content.

I.2.3.2. Comparison methods

The stimulus comparison methods consider two visual contents and provide a relative measure of the quality difference between the two of them. While the content is presented to the observer the same way as in double stimulus method, the observer is asked to evaluate the visual impact of the difference between the two types of content. Note that during comparison method, the two content can be of different types (i.e. different natural scenes captured by two different cameras). Yet, in the case in which one the content correspond to a reference and the other to a processed version of that reference, the comparison method becomes equivalent to DSIS. The grading scale is this time a "categorical scale", as illustrated in

I.2.3.3. Single stimulus methods

In single stimulus methods, the subject assesses each content individually, independent with resect to its reference.

The most popular single-stimulus method is Single Stimulus Continuous Quality Evaluation (SSCQE). This method is introduced by ITU in order to emulate the most common situation in practice, namely when the final user watches video on a private, professional uncontrolled set-up (e.g. 2D or stereoscopic TV set). The years following this, the practical relevance of such an approach steadily increased, as the double stimulus method of laboratory testing does not replicate the single stimulus home viewing conditions.

SSCQE can be applied on both quality scaling and impairment scaling using corresponding rating scales, as illustrated in Figure II-5. Of course, the interpretation of the results is different. In the former case, an absolute quality evaluation of the content is obtained, based on its visual appealing. The latter implicitly assumes that the scoring observer has some background knowledge about would-be reference content and consequently, he/she scores such would be impairments.

I.2.4. The content under evaluation

The choices related to the content under evaluation are implicitly or explicitly addressed at various levels of the ITU Recommendations.

First, different ITU specifications relate to different type of content (e.g. 2D video or stereoscopic video). Moreover, upper limits are set for the total duration of the testing sessions (30 min) and for the individual sequence duration (30 s); yet, continuous time evaluation sessions are also included in the ITU specifications.

No clear specifications are set either to the total number of excerpts (still image or video sequence) to be evaluated or to the number of excerpts an individual viewer is recommended to evaluated. However, the total number of evaluations (the sum of the number of excerpts each evaluator scores) should meet (via the confidence limits and the underlying relative error) the precision required by the application.

I.2.5. Scoring and post-processing of the scores

The scoring follows the constraints and procedure inner to the evaluation procedure and the grading scale. In practice both convention (pencil and paper) and software scoring tools are encountered [START_REF]MSU Video Quality Measurement Tool[END_REF].

The first step in the scores post-processing is the observer outliers' detections and ITU recommends in this respect a kurtosis criterion.

Once the outliers eliminated, the mean opinion score MOS is computed:

(I-1)
where are the scores assigned by the observers (after the outliers elimination).

The MOS is expected to be presented alongside with its 95% confidence lower and upper limits and respectively, where is the 95% estimation error:

(I-2)
where S is the unbiased estimator for the standard deviation:

(I-3)

I.2.6. Controversial ITU issues

This walk-through some ITU specifications brings to light that rather than imposing consensual subjective quality evaluation test conditions, the ITU Recommendations let all doors open for an experimenter and that several surprising, controversial aspects are still open to questions and interpretations:

 the terms continuous and discrete can be jointly used for a same grading scale (e.g. DSCQS) and their common, mathematical meanings seem no longer to hold;

 the MOS and its confidence limits can be computed on 15 and less data (assuming outliers are discarded) with traditional formulae which hold (for such few data) only assuming that the scores are Gaussian distributed; moreover, for 15 and less data, the relative estimations error is expected to be somewhat large;

 the grading scales are presented to the observers across with some semantic labels (Excellent, Good, Fair, Poor and Bad) which are subsequently mapped to values between 5 and 1; such an even mapping implicitly assumes that the 5 lables have an equal semantic impact in the evaluation.

The next Chapter presents a state-of-the-art study bringing to light how the research community currently addresses these open issues.

I.3. Thesis structure

This succinct introduction brings to light that rather than imposing consensual subjective quality evaluation test conditions, the ITU Recommendations let all doors open for an experimenter.

Such a situation becomes even more challenging these days when the home video consumption becomes more and more demanding in quality. Hence, a particular attention should be paid on each and every detail involved in the quality assessment and a particular focus should be made on the single stimulus evaluation that better fits the home evaluation conditions, as stated by ITU-R BT.1788, ITU-R BT.500-11, ITU-R BT.500-13, ITU-T P.913, …

In order to encompass issues from ITU recommendations to reliable and precise subjective quality evaluation, the present thesis has the following structure.

Chapter II presents a concise state-of-the-art study, bringing to light both a methodological panorama of subjective visual quality assessment (Section II.1) and some insights on the grading scales for subjective evaluation (Section II.2). Section II.3 presents some current-day interfaces for ITU-based quality evaluations while Section II.4 identifies the main state-of-the-art limitations and précises the challenges taken by the thesis.

Chapter III describes the test-bed designed and deployed in the thesis. It observes to the ITU-R BT.1788, ITU-R BT.500-11, ITU-R BT.500-13, ITU-T P.913 specifications and the SSCQE method. The chapter is structured according to the main aspects in any ITU evaluation session, namely: the viewing conditions, the panel size and composition, the evaluation methodologies and grading scale, and, finally, scoring and post-processing of the scores.

Chapter IV theoretically bridges the continuous and the discrete scale evaluation procedures and investigates whether the number of the classes on the discrete scales is a criterion meaningful in the results interpretations or just a parameter. The instantiations for Gaussian and non-Gaussian models for the scores assigned by the observers on continuous scale are presented in Chapters IV.2 and IV.3, respectively. Both Chapters IV.2 and IV.3 include the theoretical ground for MOS and its confidence limits computation as well as experimental illustrations; studies on the accuracy of the results according to the number of observers in the panel are also presented.

Chapter V starts by demonstrating the existence of the semantic impact (Chapter V.1) and follows by defining a methodological framework for quantifying this semantic impact (Chapter V.2). The methodological approach is presented alongside with the underlying quantitative results.

Conclusions are drawn and perspectives are open in Section VI.

Appendixes A and B complete the quantitative results presented in Chapter IV.

II. State of the art

The state-of-the-art study is structured into two main parts, related to the methodological panorama of subjective visual quality assessment and to a presentation of the insights on the grading scales for subjective evaluation. Specifically, 37 studies, spanning about 100 years of research carried-out in various fields (visual quality, psychology, psychophysics, sociology, marketing, medicine, ...) are investigated. By regrouping their key aspects into three tables and one synoptic representation, it is noticed that the controversial ITU issues identified in the Introduction section (relating to the continuous vs. discrete evaluation scales, to the statistical distribution of the scores assigned by the observers and to the usage of semantic labels on the grading scales) are not yet solved. This way, three objectives are identified for the thesis:

-bridging at the theoretical level the continuous and the discrete scale evaluation procedures and investigating whether the number of the classes on the discrete scales is a criterion meaningful in the results interpretations or just a parameter; studying the theoretical influence of the statistical model of the evolution results and of the size of the panel (number of observers) in the accuracy of the results are also here targeted;

-quantifying the bias induced in subjective video quality experiments by the semantic labels (e.g. Excellent, Good, Fair, Poor, and Bad) generally associated to the discrete grading scales;

-designing and deploying and experimental test-bed able to support the precision and statistical relevance for the targeted results.

The Introduction chapter identifies several surprising, controversial ITU aspects, related to the use of continuous and discrete terms in conjunction with a grading scale, to the statistical model of the scores assigned by the observers as well as to the use of semantic labels for the grading scales. Actually, these aspects echo a longer research effort, carried out even outside the visual quality evaluation.

For 100 years already, various research fields (psychology, psychophysics, sociology, marketing, medicine, ...) consider the use of rating scales in subjective evaluations. For instance, back to 1923, a study published in the Journal of Educational Psychology [START_REF] Freyd | The graphic rating scale[END_REF] raises the need for "constructive effort … towards improving the means whereby ratings are obtained". According to this study, a discrete grading scale has as advantages its simplicity and its self-explanatory character. On the contrarily, on continuous scales, the observer is somehow freed from the constraints imposed by the "quantitative terms". The study also discusses various labeled scales: the number of grading points varies from 2 to 11 while the labels assigned to these scales can be as different as Very impressive, Brilliant, etc. (for a psychology application).

One year before I was born, the study in [START_REF] Froberg | Methodology for measuring health-state preferences-ii: scaling methods[END_REF] discusses the choice of the methodology to measure the health state (labels of the scale, data interpretation). It is brought to light that the choice of the labels depends on the investigator's purpose and recommend limiting the number of the labels to "9 or even fewer attributes".

As a comprehensive state-of-the-art survey in such a broad field is practically impossible to be achieved, the present thesis will illustrate the variety of approaches through 37 studies, structured at two levels:

 methodological panorama of subjective visual quality assessment, see Section II.1 and Table II

[JON86], [NAG93], [TEU96], [WAS98], [WIN03], [ZIE07], [WIN09], [PEC08], [HUY07, [HUY11].
Additionally, the issues connected to the software support for scoring are presented in Section II.3.

II.1. Methodological panorama of subjective visual quality assessment

This section regroups 21 studies related to the subjective quality evaluation for image/video and/or audio content, presented in chronological order (between 2002 and 2015).

For each investigated state-of-the-art study, the main ITU aspects related to a subjective evaluation (as introduced in Section I.2) will be addressed: the viewing conditions, the panel composition, the evaluation methodology and the grading scales, the evaluated content and the scoring and the postprocessing of the scores.

The study [START_REF] Mued | Investigating Interaction of Audio and Video Quality as Perceived in Low-Cost Multimedia[END_REF] considers a video-conferencing application and aims at investigating the variation in subjective quality evaluation introduced by the interaction between the audio and video modalities. Two tests are conducted: a passive test (in which the subject is simply listening to/viewing the content) and an interactive communication (person to person audio/video data exchange). The viewing conditions (in the ITU sense) are not presented. A panel of 20 observers is composed and asked to rate using a 5 point rating scale labeled by Excellent, Good, Fair, Poor and Bad. The results of the study show a strong interaction dependency between audio and video content and discuss a unique benefit on multimedia quality for its psychological effects.

The study reported in [START_REF] Ninassi | Task impact on the visual attention in subjective image quality assessment[END_REF] is centered on the role that the visual attention can play in image quality assessment. The viewing conditions are set according to ITU-R BT.500-10, with subjects positioned at a viewing distance 4 times larger than the height of the picture. The panel is composed by 20 observers. A corpus of 20 images is evaluated: 10 images are considered as undistorted while the rest of 10 have various artifacts. A 5 levels discrete impairment grading scale is considered.

A methodology for subjective assessment of stereo images is presented in [CAM07]. The tests take place in a laboratory environment set according to ITU-R BT.1438. 17 observers are asked to score according to a SAMVIQ (Subjective Assessment of Multimedia Video Quality) method, with a 5-levels discrete labeled scale: Excellent, Good, Fair, Poor and Bad. While the size of the evaluated corpus is not précised, six different distorted contents (JPEG and JPEG2000 compressions blur) are considered.

The aim of [START_REF] Benoit | Quality assessment of stereoscopic images[END_REF] is to introduce an objective quality metric for stereo images quality assessment which relies on both the use of 2D metrics and depth information. The testing environment complies with ITU-R BT.500-11. 17 observers scores 66 video sequences (6 reference and 60 downgraded sequences) according to a SAMVIQ methodology, assigned on a continuous scale ranging from 0 to 100.

The study in [START_REF] Hewage | Prediction of stereoscopic video quality using objective quality models of 2-D video[END_REF] investigates the correlation between subjective and objective assessment of color plus depth map 3D video. The testing conditions are not explicitly stated. 32 observers participated on the experiments using a DSCQS method, with a scale ranging from 1 to 5, where 1 represents Bad image quality/depth perception and 5 represents Excellent image quality/depth perception. The stimulus set contains 13 coded video sequences and the original, uncompressed version of each scene is used as the reference in the evaluation test.

The study in [START_REF] Ninassi | Considering temporal variations of spatial visual distortions in video quality assessment[END_REF] advances a perceptual full reference video quality assessment metric based on the temporal evolutions of the spatial distortions. This metric is compared to a subjective rating in order to investigate its efficiency. The tests are conducted into an environment observing to ITU-R BT.500-10. 36 observers participated on subjective assessment using an impairment discrete scale to rate the impairments induced by the MPEG-4 AVC (H264) codec (used at 5 different encoding configurations) on 10 reference video sequences.

A procedure for subjective evaluation of the JPEG XR codec for compression of still pictures is described in [START_REF] Simone | Subjective evaluation of JPEG XR image compression[END_REF]. The viewing conditions are set according to ITU-R BT.500-11 and are described in detail. A total of 16 subjects took part in experiments and each subject attended four test sessions, scoring a total of 208 test images. A DSCQS method is considered, with a grading scale featuring both continuous markers -form 0 to 100 and the 5 generally considered discrete labels Excellent, Good, Fair, Poor and Bad.

The studies in [GOL10-01] and [GOL10-02] are devoted to the definition of objective stereoscopic video quality metrics. The stereoscopic video database that contains a large variety of scenes captured by using a stereoscopic camera setup consisting of two HD camcorders with different capture parameters. The testing conditions are set according to ITU-R BT.500-11. 20 subjects (6 females, 14 males) are scoring the content; among them, 3 are discarded as outliers. An SSCQS method and a 5 levels discrete grading scale labeled by Excellent, Good, Fair, Poor and Bad are considered. The corpus considers 6 different scenes and, for each scene, 5 different capturing parameters. This study brings to light a particular point of view related to stereoscopic video evaluation: "for non-expert viewers it is quite difficult to distinguish 5 quality levels (excellent, good, fair, poor, bad)" and "3 quality levels (good, fair, bad) may be more appropriate".

This study in [START_REF] Simone | Towards high efficiency video coding: Subjective evaluation of potential coding technologies[END_REF] describes a subjective evaluation framework which is meant to serve as a test-bed for the joint collaborative team video coding (JCT-VC) efforts towards the definition of the HEVC standard. The testing conditions are set according to ITU-R BT.500-11 and are described in detail. A total of 494 naive observers are inquired, among which about 30% are females. Two different evaluation methods are considered:

 the DSIS method, with a joint numbering and semantic labels marks (Imperceptible, Perceptible but non annoying, Slightly annoying, Annoying and Very annoying), as depicted in Figure II-1.a.

 the DSCQS method, with a grading scale featuring both continuous markers -form 0 to 100 and the 5 generally considered discrete labels Excellent, Good, Fair, Poor and Bad, as depicted in  Absolute Categorical Rating (ACR), with Excellent, Good, Fair, Poor and Bad;  Paired Comparisons (PC), according to which the observer selects the content that best matches the targeted criterion; the conversion between the results thus obtained and a virtually continuous scale is discussed.

The study [START_REF] Urvoy | NAMA3DS1-COSPAD1: Subjective video quality assessment database on coding conditions introducing freely available high quality 3D stereoscopic sequences[END_REF] provides the scientific community with a database of high quality Full-HD stereoscopic sequences, shot with a semiprofessional 3D camera. The performances of this database are hinted to by an example of usage for evaluating encoding properties. The underlying subjective video quality evaluation follows ITU-T P.910. 29 observers (12 females, 17 males) performed the ACR-HR (ACR -Hidden Reference) evaluation on a 5-level quality scale labeled by Excellent, Good, Fair, Poor and Bad.

The total evaluated content is composed of 99 sequences of 16s and 11 sequences of 13s.

The study aims at enhancing the understanding of the perception of MPEG-4 AVC compressed stereoscopic 3D videos, in particular spatial video quality, depth quality, visual comfort and overall 3D video quality. Actually, a subjective assessment test was conducted in order to identify whether some of the subjective quality scores can be incorporated into a single stereo 3D quality database or whether separate databases are needed to study different aspects of 3D QoS (Quality of Experience). A SSCQS method, with continuous grading scale between 0 and 100 was considered. 11 video sequences (a 3D pristine video, a 2D pristine video (right view), and nine distorted videos) were shown to the subjects. The number of observers is not presented.

The study aims to explore 3D QoE by constructing the visual experience as a weight sum of image quality, depth quantity and visual comfort. Two experiments in which depth quantity and image quality are varied respectively are designed to validate this model. The test conditions follow ITU-T BT.500-11. In the first experiment, the stimuli consist of three natural scenes and for each scene, four levels of perceived depth variation in terms of depth of focus (0, 0.1, 0.2 and 0.3 diopters) are considered. In the second experiment, five levels of JPEG 2000 compression ratio (0, 50, 100, 175 and 250) are used to represent the image quality variation. 28 observers participated in the test and scored according to a SAMVIQ method labeled with Excellent, Good, Fair, Poor and Bad.

Robust watermarking techniques for stereoscopic video protection are presented in [START_REF] Chammem | Robust watermarking techniques for stereoscopic video protection[END_REF]. The testing conditions are set according to ITU-R BT.500-12. A panel of 15 naïve observers participate in the experiments and is required to evaluate 34 randomly chosen video excerpts of 40 seconds each. The scores are assigned on a 5 levels discrete quality scale with 5 levels with Excellent, Good, Fair, Poor and Bad labels according to the SSCQS method.

[BOS13] presents a study on subjective quality assessment for free-viewpoint video sequences generated from decompressed data. The testing conditions are set according to ITU-T BT.500 and are considered to assess 276 FVV (Free Video Viewpoint) sequences. 27 naïve observers participate in the subjective quality evaluation, according to an ACR-HR method and a 5-level labeled quality scale (Excellent, Good, Fair, Poor and Bad).

The study in [START_REF] Gao | Learning to rank for blind image quality assessment[END_REF] investigates the blind image quality assessment (perceptual image quality scores without access to reference images) and organizes a subjective assessment test where the viewing conditions follow the ITU-R BT.500-13. 24 subjects evaluate distorted images with (JPEG2000 compression, JPEG compression, Gaussian Blur, WN and FF) and assign their scores on a continuous scale from 0 to 100.

The effect the chromatic variations in 3D video have on the perceived visual quality is investigated in [START_REF] Piedade | Chromatic variations on 3d video and qoe[END_REF]. The tests conditions follow ITU-R BT.500-12. 25 subjects participate in the test and assigne their scores on an ACR-HR continuous scale with 11 points. A corpus of 8 video sequences (whose duration is not precised) is evaluated.

The study [START_REF] Hanhart | Benchmarking of objective quality metrics for HDR image quality assessment[END_REF] reports on an extensive benchmarking of objective quality metrics for HDR image quality assessment. In total, 35 objective metrics are benchmarked on a database of 20 HDR contents encoded by 3 compression algorithms at 4 bit rates, leading to a total of 240 compressed HDR images. A subjective assessment is conducted to verify the accuracy of the presented metrics. A total of 24 naïve subjects (12 females and 12 males) took part in the experiments. The test conditions follow the ITU-R BT.2022 and the observers assign their score on double-stimulus impairment scale (DSIS) with 5 Labels according to ITU-R BT.500-13.

A method for no-reference quality assessment of contrast distorted images based on the principle of natural scene statistics (NSS) is presented in [START_REF] Fang | No-reference quality assessment of contrast-distorted images based on natural scene statistics[END_REF]. In order to validate this method, a subjective assessment test following ITU-R BT.500-12 is conducted. 57 subjects participate in the test. A first group of 22 observers assigned their scores on a discrete scale ranging from 1 to 5 and evaluate a database composed of 415 still images. A second group composed of 35 observers considers a discrete scale ranging from 0 to 9 and evaluate of 200 still images. The type of evaluation (single stimulus or double stimulus) is not précised. Although this study brings to light the usage of two different grading scales, it does not make possible any comparison between the 5 levels and the 9 levels scoring (as the two databases are different).

The study in [SU15] considers oriented correlation models of distorted natural images with application to natural stereopair quality evaluation. A subjective assessment test following the ITU-R BT.500-11 is conducted in order to validate those models. The observers (whose number is not presented) assign their score according to the DSIS method, with a joint numbering and semantic labels marks (Imperceptible, Perceptible but non annoying, Slightly annoying, Annoying and Very annoying). The viewing conditions are not presented.

The principles of these various research studies are illustrated in Table II-1 which brings to light that the subjective visual quality evaluations are very heterogeneous in their deployment and that a convergence is still far for being reached. While difference are encountered at practically any level (targeted application, experimental conditions, panels, grading scales and processed content), they are particularly controversial for the grading scales; hence, these aspects will be deeper investigated in Chapter II.2.

II.2. Insight on the grading scales for subjective evaluation

The present section investigates the details related to the grading scales. In this respect, both background studies emerged from various research fields and visual quality assessment are discussed.

II.2.1. Background studies in subjective quality evaluation

Conducted under the applied psychology framework, the study in [START_REF] Matell | Is there an optimal number of alternatives for Likert scale items? Study 1: reliability and validity[END_REF] investigates the reliability of 18 different discrete scales, with the number of categories ranging from 2 to 19. The reliability of the scales is discussed with respect to three factors that are a priori important in determining the number of alternatives to employ: (a) the proportion of the scale which is effectively considered by the subjects when scoring, (b) the duration required for testing, and (c) whether or not an "uncertain" category is provided. 360 students (20 students for each scale) are considered in the scoring sessions. The scale reliability was assessed by an analysis of the variance (second order moment) of the scores; in this respect, the Fisher's test is used. The conclusion is that, at least from this point of view, the scores are largely independent from the number of rating points on the scale: 16 out of the 18 scales examined did not differ significantly. The two exceptions correspond to the 2 level and 3 level grading scales. The results also show that the testing time increases with the number of levels on the scale while the usage of the "uncertain" category decreases as the number of rating steps increases. 7 years later, in the same research field, the study reported in [START_REF] Mckelvie | Graphic rating scales-How many categories?[END_REF] investigates the reliability and validity of the scores assigned on a continuous scale and on discrete scales with 5, 7 and 11 categories. For both continuous and discrete scales, both labeled and unlabeled versions are presented to 30 subjects involved in the experiments. The conclusions are of different types. First, the continuous scale is "most pleasing" to be used. Secondly, the results brought no evidence that the continuous scale would provide either more discrimination or better accuracy than the discrete scales. Concerning the discrete scales, the results brought to light that 5 or 6 categories should be considered for evaluation. It is also stated that even on a continuous scale, the scores assigned by the observers are somewhat clustered into 5 or 6 classes ("subjects using the continuous scale appear to be operating essentially with 5 or 6 categories"). The problem of the optimal number response alternatives on a discrete scale is also raised under the framework of marketing research, [START_REF] Cox | The optimal number of response alternatives for a scale: A review[END_REF]. By using information theory tools, it is considered that the optimal number is the one that maximizes the information provided by respondents in a test, while minimizing the response errors or the likelihood of the random responses. In is thud concluded that using a large number of scale categories (higher than 9 to 12) results in no benefit, while a very small number of categories (less than 5 or 6) could produce a loss in accuracy. In study conducted in academic context, [START_REF] Albaum | Continuous vs. discrete semantic differential rating scales[END_REF] compares a five-category discrete semantic differential scale with the corresponding unlabeled continuous scale. By differential scale it is understood a scale whose extremities are labeled, as for example Friendly vs. Unfriendly or Modern vs. Old fashioned. The discrete scale is presented to the user with intermediate marks but no semantic labels or numerical values are associated to these marks. 176 university students participated in the assessment test. The results are investigated via their mean value and variance, thanks to a paired Student's test. It is thus demonstrated that no large variability among the discrete scales is encountered: only 4 out of the 30 pairs of scale were significantly different. It is also concluded that similar evaluation result can be obtained from the five-category discrete rating scale and from a continuous scale. Yet, it is explicitly stated that "there is a considerable advantage of a continuous rating scale in applications for which individual measurement is important …" Also in an academic context, [SVE00] compares a visual analog scale (similar to a continuous unlabeled scale), a graphic rating scale (similar to the standard ITU labeled continuous scale) and a five-point verbal descriptor scale (similar to the 5-point discrete category scale). 174 students participated in the subjective assessment test relating to dummy application, namely a self-assessment of the previous knowledge in statistics. As a general conclusion, it is stated that assessments on the discrete scale have the highest level of stability; particularly, it is shown that both the verbal descriptor scale and the graphic scale assessments provided a better order consistency compared to the visual analog scale assessment. The results also show that an increased number of possible responses did not guarantee a higher sensitivity of the assessments.

Coming back to the marketing field, [START_REF] Preston | Optimal number of response categories in rating scales: Reliability, validity, discriminating power, and respondent preferences[END_REF] provides to 149 participants a questionnaire concerning the service elements of a recently visited store or restaurant. The questionnaire used scales with a number of judgment category from 2 to 11, and a 101-point scale (from 0 to 100). It is thus shown that the scales that produce the least reliable scores are those with the fewest response categories. However, it is also found that a decrease in reliability is encountered for scales with more than ten response categories. The most reliable scores are found to be those from scales featuring between 7 and 10 response categories.

The principles of these studies are regrouped in Table II

II.2.2. Visual quality assessment

To the best of our knowledge, for visual quality assessment applications, the problem of the semantic impact of the labels in the overall quality evaluation was first raised by [START_REF] Jones | Graphic scaling of qualitative terms[END_REF]. Two panels are considered for experiments: 49 persons compose a panel of English speakers (native USA speakers) while 24 persons compose a panel of Italian speakers. The experiments are conducted in parallel for English and Italian, but they will be illustrated here only for English. During the experiments, a continuous scale featuring no intermediate labels but having its two extremities marked with Best imaginable and Worst imaginable is presented to the panel. The panel members are asked to place, on that continuous scale, according to their own understanding, 15 adjectives (labels): Superior, Ideal, Excellent, Good, Fine, OK, Fair, Passable, Marginal, Not Quite Passable, Poor, Inferior, Bad, Not Usable, Awful. For both languages, the results showed that the ITU labels were not evenly distributed along the graphic scale suggesting a non-uniform semantic distance between adjacent ITU labels. Specifically, a kind of compression at the end points of the scale was identified and explained as a reluctance of the observers in using the continuous scale extremities. The results also show a clustering tendency, with 9 classes (e.g. the adjectives Ideal, Excellent and Superior are very close each-other). These results were later corroborated for other languages, such as Swedish, Dutch and British English, but not for Japanese and German, which exhibited uniformly distributed labels along the scale [NAR93],

[TEU96], [ZIE7].

The semantic impact of the ITU Japanese descriptive terms for quality and impairment is investigated in [START_REF] Narita | Graphic scaling and validity of Japanese descriptive terms used in subjective evaluation tests[END_REF]. The two experiments follow the principles in [START_REF] Jones | Graphic scaling of qualitative terms[END_REF]. A panel of 40 Japanese speakers is inquired. They are asked to alternatively position on a continuous scale with no intermediate but extremities labels either 13 quality terms or 12 impairment terms. It is thus brought to light that the perceived quality intervals are non-uniform. Yet, in Japanese, they are distributed more evenly than in English, French or Italian, following a similar trend as in the German case. The impairment experiment shows the Japanese terms have lower semantic impact than the corresponding terms in other languages. Note that the underlying experiments for German are presented in [START_REF] Teunissen | The validity of CCIR quality indicators along a graphical scale[END_REF].

The two ideas of semantic impact and language dependency related to the ITU labels are considered as a starting point for the research study presented in [START_REF] Watson | Measuring perceived quality of speech and video in multimedia conferencing applications[END_REF]. In order to refine the precision and the stability of the results, the subjective quality evaluation is considered to be a multidimensional process and some means for identifying the different dimensions and the appropriate vocabulary are advanced.

In this respect, the use of unlabeled continuous rating scale is considered as a ground for investigation. A panel of 24 subjects is asked to score audio-visual content on a continuous scale whose extremities are labeled by + and -signs. The results show that such a quality rating procedure is "remarkably consistent", thanks to the fact that the subjects set their own criteria. The results also show that using an unlabeled scale reduces the tendency of subjects to avoid the end points of the scale.

The SSCQE and DSIS methods are compared on the same test material in [START_REF] Winkler | Video quality evaluation for Internet streaming applications[END_REF]. In order to avoid the semantic impact of the Excellent, Good, Fair, Poor and Bad labels, the SSCQE method is presented to the observer as a vertical slider with only the two labels Good and Bad at the top and bottom ends of the slider. On the contrarily, the DSIS scale follows the general ITU recommendations and is presented to the observer on a discrete, 5 levels scale ranging from Imperceptible to Very annoying. Results obtained from 20 observers show that the two methods are highly correlated and produce comparable quality results.

Three subjective audio quality evaluation tests are presented in [START_REF] Zieliñski | On the use of graphic scales in modern listening tests[END_REF]. Each of these three tests considers a different scale: the standard continuous scale with the 5 ITU labels, a 5-point continuous impairment scale, and a label-free continuous scale. The results showed a high similarity between the scores obtained with the ITU labeled scale and the label-free quality scale, with an almost perfect linear regression between them. Hence, this study supports the idea that the ITU quality scale is indeed an equal-interval scale.

The study in [START_REF] Huynh-Thu | Examination of the SAMVIQ methodology for the subjective assessment of multimedia quality[END_REF] investigates the suitability of SAMVIQ assessment methodology; in this respect, two psychovisual experiments are carried out in two different laboratories. The subjective video quality evaluation follows the ITU-T P.910. The observers (whose number is not presented) assign their scores on a continuous 0-100 scale labeled by Excellent, Good, Fair, Poor and Bad. The experimental results indicate that the SAMVIQ methodology provides results comparable to other existing methods, such as the single stimulus ACR methodology.

The ACR and SAMVIQ subjective quality assessment methodologies are compared in [PEC08]. The ACR is presented to the observers with a 5 levels discrete scale associated to the labels Excellent, Good, Fair, Poor and Bad while the SAMVIQ with a continuous scale ranging from 0 to 100, yet featuring the same labels. The viewing conditions are not précised; the number of observers participating in the test is 43.

The results of this study show that the ACR uses 96.3% of the available range while SAMVIQ uses only 82%. It is thus demonstrated that the two assessment methodologies have different behaviors; it is also shown that the relation between their results depends on the evaluated content quality and it is subsequently stated that, for a given number of observers, SAMVIQ is more precise than ACR.

The variability of subjective ratings obtained with different scales (0-100 continuous scale and 5, 9, and 11 discrete scales) is investigated in [START_REF] Winkler | On the properties of subjective ratings in video quality experiments[END_REF]. The study relays on simulated data instead of real experimental data, since it is considered that the differences among experiments available in the literature are too large for reliable direct comparison. It is concluded that although an increased discretization level of the scale leads in theory to an increase of the standard deviation of the scores (and therefore to a decrease of precision), practical proof of this effect remained inconclusive. He also found that the number of subjects may not need to be as high as generally assumed; in fact, the minimum of 15 recommended by ITU appears to be a very reasonable suggestion.

The study [START_REF] Huynh-Thu | Study of rating scales for subjective quality assessment of High-Definition video[END_REF] compares 4 different ITU grading scales with labels: two of them are discrete (with 5 and 9 levels) while the other two are continuous, yet with 11-point and 5-point grades. A subjective assessment test following the ITU-R BT.500-11 is conducted. 92 observers assign their scores according to the ACR method using the 4 different scales. The total evaluated content is composed of 128 sequences of 12s each. The results show that no significant statistical difference is found among subjective results obtained with the different four scales.

The examples of these various studies are regrouped in Table II-3. HDTV content coded at H.264 bitrates

II.3. Graphical user interface for ITU visual quality assessment

In the subjective assessment context, the GUIs are far more appealing versions of the old-fashioned pencil and paper evaluation sheets for score gathering. As their practical usage is traversal with respect to the evaluation procedure itself, they are presented into a separate section in the present thesis.

They are specifically designed so as to meet several convergent yet very different requirements. First, they should accurately reproduce the ITU evaluation conditions. Secondly, they should have virtually a zero impact in the evaluation result itself while allowing a speed-up of the evaluation procedure. Thirdly, they should be simple, versatile and user-friendly.

For instance, the MSU Video Group and Media Lab [MSU13] designed a specific GUI for three subjective assessment method, namely the DSIS, DSCQS and a so-called MSU-CQE (Continuous Quality Evaluation), as illustrated in Figure II-5. For the DSIS method, videos are successively presented (first the reference, then the distorted version). After their playback, the observer is asked to score using impairment scale, Figure II-5.a. In the DSCQS case, videos are played simultaneously in the same window. Each pair is repeated a given amount of times (repetition parameter in the Task Manager). During playback the observer is free to switch between two videos, named Sequence A and Sequence B by pressing Tab button. One of videos is the reference and the other is the impaired, but the observer is not informed about it. After playback the observer is asked to give his opinion about each video sequence, see

II.4. Thesis objectives

The present state-of-the-art study spans about 37 studies (cf. the synoptic representation in Figure II-6) covering about 100 years of research and, surprisingly, brings to light that controversial, open questions still arise about some key aspects related to the grading scales in subjective quality assessment. Actually, a basic taxonomy would consists in continuous vs. discrete and labeled vs. unlabeled categories:

 the a priori most intuitive category of scale, i.e. the continuous unlabeled scales is considered in 3 studies : [START_REF] Benoit | Quality assessment of stereoscopic images[END_REF], [URV12], [START_REF] Gao | Learning to rank for blind image quality assessment[END_REF];

 continuous scales with labels are also considered in 4 studies: [START_REF] Jones | Graphic scaling of qualitative terms[END_REF], [START_REF] Narita | Graphic scaling and validity of Japanese descriptive terms used in subjective evaluation tests[END_REF], [CAM07], [START_REF] Simone | Subjective evaluation of JPEG XR image compression[END_REF]; an interesting point common to these 4 studies is the fact that the while the continuous scales range from 0 to 100, their discrete counter-parts are labeled with discrete terms (Excellent, Good, Fair, Poor and Bad ).

 when considering discrete scales, the general tendency is to use the ITU labels, be them for quality (Excellent, Good, Fair, Poor and Bad in Consequently, the present thesis has as main objective to reconsider the ITU recommendations and to investigate on theoretical basis some of their key aspects related to evaluation scales. In other words, the main objective is to bridge, from both theoretical and methodological points of view, the continuous unlabeled and the discrete labeled realms (see Figure II-6). In this respect, three main research directions are to be considered:

[MAT71], [COX80],[ZIE07],[MUE02], [GOL10-01], [GOL10-02], [SIM11], [BOS11], [URV12], [CHE12-02] [CHA13], [ BOS13
First, the investigation will be carried out at the theoretical level. In this respect, the thesis will try to bridge the continuous and the discrete scale evaluation procedures and to investigate whether the number of the classes on the discrete scales is a criterion meaningful in the results interpretations or just a parameter. Studying the influence of the statistical model of the evolution results and the size of the panel (number of observers) in the accuracy of the results is also targeted.

Secondly, at the methodological level, the thesis will address the issue of quantifying the bias induced in subjective video quality experiments by the semantic labels (e.g. Excellent, Good, Fair, Poor, and Bad) generally associated to the discrete grading scales.

Finally, from the experimental point of view, these above-mentioned two directions require an experimental test-bed able to support their precision and statistical relevance. Hence, specifying and deploying such an experimental test-bed becomes an objective per-se. This way, an investigation on the practical variability of the experimental and methodological results with the type of content (2D or stereoscopic video) or with its quality (as assessed by objective metrics) becomes also possible.

III. Test-bed

The test-bed designed and deployed in the thesis consider the ITU-R BT.1788, ITU-R BT.500-11, ITU-R BT.500-13, ITU-T P.913 specifications and the SSCQE (Single Stimulus Continuous Quality Evaluation) method as a backbone and should adapt/extend it so as to fit to the evaluation procedure peculiarities.

In order to score the content, a versatile Android application is developed so as to alternatively allow the evaluation on continuous and discrete, semantically labeled scales.

The content to be evaluated is represented by four corpora, representing high and low-quality video (as a priori expressed by the PSNR value), both stereoscopic and 2D video. Each of these corpora is about 20 minutes long and is presented to the observers into downgraded versions obtained through watermarking or compression methods. Note that that as the study investigates the very evaluation procedure, the visual content just ensures realistic conditions for evaluation.

A total of 640 human observers are involved in the experiments. They were grouped in four main panels of 160 observers each, on panel for each type of content. In order to grant result repeatability, each main panel was split into three sub-panels, referred to as the reference (60 observers), validation (50 observers) and cross-checking (50 observers) panels. The subjects in the reference panels scores on the continuous scales. The subjects in both the validation and cross-checking panels are finally partitioned into two sets, scoring on 5 level labeled grading scales and on 3 level labeled grading scales, respectively.

III.1. Overview

By its nature, the present thesis belongs to field of random processes and statistics applied to visual quality assessment; hence, the relevance of the targeted results is essentially determined by the processed experimental data. Consequently, the test-bed designed and deployed for the thesis is presented prior to the main theoretical and methodological contributions.

Two criteria are considered in the design of the test-bed:

 the compliance with the most intensively considered ITU specifications;  the possibility of ensuring reliability and relevance for the objectives targeted by the present thesis.

In the present study, the first criterion mainly applies to the viewing conditions and the evaluation methodology while the second criterion imposes strict constraints on the content under evaluation, thus determining the panel composition and size. The post-processing of the scores derives from the two of them.

It should be emphasizeD that the study investigates the evaluation procedure itself and is not focused on the content to be evaluated. Hence, the two above-mentioned criteria should be considered as complementary: while the ITU specifications are the backbone ensuring compatibility with content evaluation studies, they should be adapted/extended so as to fit to the evaluation procedure peculiarities.

The presentation is structured according to the main aspects related to any test session (Section I.2), namely the viewing conditions, the panel composition and size, the evaluation methodology and the grading scales, the content under evaluation, the scoring and the post-processing of the scores.

III.2. Viewing conditions

The evaluation has been conducted at the Advanced Research &Techniques for Multimedia Imaging Systems (ARTEMIS) Department at Telecom SudParis engineering school in France.

The viewing conditions are set in concordance with ITU-R BT.1788, ITU-R BT.500-11, ITU-R BT.500-13, ITU-T P.913, as detailed in Table III-1 A " LG LCD, full HD 3D monitor (1920 x 1080 pixels) and a cd/m² maximum brightness are used in the experiments.

The experiments involve maximum subjects per session.

The subjects are seated at a distance D equal to the height of the screen multiplied by factor F and defined as the Preferred Viewing Distance PVD, see Table I

-2.
The observation angle relative to the normal was kept lower than 30 degrees (although this value is not a must for the LCD monitors).

Test method

After the study and the comparison between different assessment methods seen in the previous chapter, and following the ITU recommendations for evaluation content to be consumed in home environments, a SSCQE (Single Stimulus Continuous Quality Evaluation) method has been adopted.

According to the thesis objectives, this method is considered with two types of scales:



III.3.1.2. Assessment session

Training session

At the beginning of the first session, from 2 to 5 training presentations are introduced to stabilize the observers' opinion. This is a crucial part of method of assessment, since subjects could misunderstand their task; during the training, the observer are explained about what they are going to see, what they have to evaluate and how they will express their opinion. Any question from the subjects is answered in a short and objective way.

If several evaluation sessions are required, only two training presentations are done at the beginning of the next session.

The data issued from these presentations are not taken into account when computing the MOS.

The evaluation session

The evaluation session lasts up to half an hour.

Each observer evaluates video excerpts, extracted from the video corpus (see Chapter III.4 below) and presented in a random order. During the testing session, each sequence is shown once or twice and a break between the presentations is ensured for scoring.

III.4. Content under evaluation

Following the general trend in the state-of-the-art studies, the content under evaluation is selected so as to cater for two particular needs of the thesis, namely to investigate the influence of the type of content and of it's a priori quality in the overall results. Consequently, both stereoscopic and 2D video content are considered, each of which at two quality levels (as a priori expressed by an objective quality metric). Note that in order to ensure realistic conditions, both the stereoscopic and the 2D video content corpora are organized under the framework of French national projects (as subsequently detailed).

III.4.1. Stereoscopic video corpus

The stereoscopic video content processed in the present thesis is produced under the framework of the 3DLive French national project, meant to create expertise in France for shooting and subsequent live TV transmission of 3D stereo contents. The 3DLive corpus sums-up 2 hours, 11 minutes and 24 seconds of stereoscopic video sequences (197000 stereoscopic pairs encoded at 25 frames per second), representing 10 minutes of a rugby match, 10 minutes of a dancing performance, 1 minute of a private gig of rock band "Skip the Use", one hour and 45 minute and 24 seconds of a volley-ball match and 5 minutes of a theater play "Les Fourberies de Scapin" by Molière. These sequences are full HD encoded ( pixels), as illustrated in Figure III-1.

High-quality stereoscopic video corpus

From the 3DLive corpus, 16 sequences with individual durations between 40 sec and 80 sec, summing up about 20 minutes are randomly sampled.

Each of these sequences is subsequently watermarked1 by considering 4 different watermarking methods (namely SS, binary QIM, 5-symbols QIM and IProtect) and, for each of them, 4 different insertion domains (one of the views and 3 different disparity maps, referred to by NTSS, FS-MPEG and 3DV-NTSS). The watermarking parameters are set so as to result into values 35dB < PSNR < 40dB; it is also a posteriori verified that the SSIM (Structural SIMilarity) values are larger than 0.98 while the NCC (Normalized Cross Correlation) values are larger than 0.98.

Thus, a total of 256 (16 original content types x 16 distortion configurations) sequences are obtained. Among these sequences, 1 different type of content for each distortion configuration is randomly selected, thus resulting in 16 sequences summing 20 minutes (each original content type is present once and none of them is distorted the same way). Finally, these 16 sequences are shuffled (randomly reordered) and presented to the observer.

67

All these random sampling and shuffling steps are considered as a possible way to eliminate the influence of the original content and of the distortion types in the final evaluation and to allow the investigation of the grading scales and semantic labels themselves. This principle is kept for the other types of corpora.

Low-quality stereoscopic video corpus

In order to obtain the low-quality stereoscopic video content, the high quality stereoscopic video corpus is compressed (while keeping the frame resolution and rate constant) so as to obtain 25dB<PSNR<30dB.

It should be noticed that the low-quality corpus is downgraded 25db -30dB with respect to the high quality corpus which is, in its turn, downgraded 35db -40db with respect to the original content. It is also a posteriori verified that the values corresponding to the SSIM and NCC (with respect to the high quality corpus) range between 0.97 -0.98 and 0.95-0.97, respectively.

III.4.2. 2D video corpus

High-quality 2D video corpus

The high quality 2D video corpus corresponds to the left view from the high quality stereoscopic video corpus.

Low-quality 2D video corpus

The low-quality 2D video corpus is organized under the framework of the MEDIVALS (waterMarking et Embrouillage pour la DIffusion et les Echanges Vidéos et Audios Legalisés) French national project. The main MEDIEVALS project is to design and deploy an end-to-end security solution for the audio-visual content, ensuring the tracking of the delivered information and of its property rights.

The video content is encoded at 640x480 pixels, 25 fps. An MPEG-4 AVC encoder is considered, with the baseline profile and a 512 kbps rate.

The corpus has a total duration of 1h30 minutes and is composed of 4 types of professional TV content: news, documentary, movies and talk-shows.

In order to obtain the content to be presented to the observers, from each type of content, a sequence with a duration between 50 and 60 sec is randomly extracted. Then, each of this sequence is downgraded with 7 distortion configurations2 . These 28 sequences, summing up to 20 minutes, are then shuffled prior to their presentation to the observer.

III.5. Panel composition and size

A total of 640 human observers are involved in the experiments, as detailed in Table III-3. They are all non-expert viewers with marginal knowledge on the image quality. They speak professional English with a majority of English native speakers. The age distribution ranged from to with an average of . All the subjects were screened for fine and dynamic stereopsis, visual acuity using Snellen chart and color vision using the Ishihara test.

They were grouped in four main panels of 160 observers each, on panel for each type of content. In order to grant result repeatability, each main panel was split into three sub-panels, referred to as the reference (60 observers), validation (50 observers) and cross-checking (50 observers) panels. The subjects in both the validation and cross-checking panels are finally partitioned into two sets, scoring on 5 level grading scales and on 3 level grading scales, respectively.

III.6.1.2. Video control interface

After getting identified, the observer can start loading the videos to be assessed. He/she can ask to play the video and repeat it (the video replay is allowed only twice). Once the video sequence reaches its end, the observer can move to the scoring interface by clicking the "Scoring" button, see Figure III-5.

III.6.1.3. Scoring interface

When the observer clicks on the "Scoring" button from the previous interface, a scoring interface is displayed, Figure III-5.

2D video content scoring

The 2D video content (be it high or low quality) is scored based on the interface presented in The continuous scale ranges between 0 and 100, with numbered marks (with a 10 precision). Note that the dynamic range is chosen according to both the specifications ITU-T P.913 and to several state-of-the art studies [CAM07], [START_REF] Benoit | Quality assessment of stereoscopic images[END_REF], [CHE12-01], [START_REF] Gao | Learning to rank for blind image quality assessment[END_REF].

The 5 levels discrete scale is labeled Excellent, Good, Fair, Poor and Bad; such a scale is in concordance with ITU-R BT.500-11/13 and with the most of the state-of-the-art studies (cf. Section II).

Concerning the 3 levels grading scale, the following labels are considered: Good, Fair and Bad; this choice corresponds to the ITU-T P.913 and to the explicit suggestion in [GOL10-01],

Stereoscopic video content scoring

The stereoscopic video content (both high and low quality) interface is paired design with the 2D evaluation interface

III.6.2. Post-processing of the scores

Outliers' detection

For each assessed video sequence, the scores distribution across observers was tested by calculating the distribution kurtosis coefficient, according to BT.500-11/13.

Type of panels

MOS Computation

Once the outliers eliminated, the mean opinion score MOS is computed according to (I-1).

The MOS is expected to be presented alongside with its 95% confidence lower and upper limits and respectively, where is the 95% estimation error computed according to (I-2).

III.7. Conclusion

The test-bed designed and deployed in the thesis follows two complementary criteria: the compliance with the most intensively used ITU specification and the possibility of ensuring reliability and relevance for the targeted objectives. It has the main particularity that it is not oriented towards content quality assessment but towards the investigation of the evaluation procedure itself.

The viewing conditions are set according to ITU-R BT.500-11/13.

The SSCQE evaluation method is considered, with three types of scales:

 a continuous scale ranges between 0 and 100, with numbered marks (with a 10 precision); this dynamic range is chosen according to both the ITU specifications and to several state-ofthe art studies ITU-T P.913 and [CAM07], [START_REF] Benoit | Quality assessment of stereoscopic images[END_REF], [CHE12-01], [START_REF] Gao | Learning to rank for blind image quality assessment[END_REF] ;

 a 5 levels discrete scale labeled Excellent, Good, Fair, Poor and Bad; such a scale is in concordance with ITU-R BT.500-11/13 and with the most of the state-of-the-art studies (cf. Section II);

 a 3 levels grading scale labeled: Good, Fair and Bad; this choice corresponds to the ITU-T P.913 and the explicit suggestion in [GOL10-01].

The content under evaluation is selected so as to ensure realistic evaluation conditions (according to two French national R&D projects) and should allow precision and reliability in the investigation of the quality evaluation procedure. In this respect, 4 corpora of about 20 minutes each are organized. They represent 2D and stereoscopic video content, at both high and low quality (as a priori expressed by a subjective quality metric).

The panel is composed of 640 human observers who were grouped in four main panels of 160 observers each, one panel for each type of content. In order to grant result repeatability, each main panel was split into three sub-panels, referred to as the reference (60 observers), validation (50 observers) and crosschecking (50 observers) panels. The subjects in both the validation and cross-checking panels are finally partitioned into two sets, scoring on 5 level grading scales and on 3 level grading scales, respectively.

The scoring is achieved by an Android application running on compatible tablets and smartphones.

The post-processing starts by eliminating the outliers then follows by computing the mean opinion score MOS alongside with its confidence limits.

IV. Bridging continuous and discrete unlabeled quality scale evaluation

The present chapter deals with the main challenge of the thesis: it theoretically bridges the continuous and the discrete scale evaluation procedures and investigates whether the number of the classes on the discrete scales is a criterion meaningful in the results interpretations or just a parameter.

In this respect, the non-linear relation between the probability density functions modeling the scores assigned by the observers on continuous and discrete unlabeled scales is established. The instantiations of this formula for two cases of practical relevance, namely the case in which the scores according on a continuous scales are Gaussian distributed and the case in which these scores are not Gaussian (but estimated via a Gaussian mixture) are also presented. In the two cases, it is brought to light that the MOS and the related confidence limits solely depend on the average and variances of the continuous scale models and of the q number of quality levels on the discrete scale. The theoretical results are illustrated through experiments corresponding to the scores assigned by the 4 reference panels (one panel for each type of content -see Chapter III).

The impact of the number of human observers in the precision of the quality evaluation is subsequently assessed. It is demonstrated that for three N values ranging from 15 to 50, the relative errors in MOS estimation are constant with both N and q (differences lower than 0.04 and 0.02 being obtained for Gaussian and non-Gaussian cases, respectively).

Hence, in order to converge to a unique answer to the controversial issue related to the number of quality levels on a stereoscopic video grading scale, it is suggested to perform the evaluation on a continuous grading scale, with no semantic labels associated to the scores and to subsequently map these values on the discrete scales.

IV.1. Continuous vs. discrete unlabeled scale evaluation

Be there a subjective quality evaluation experiment with the r.v. (random variable) theoretically modeling the observer's inner appreciation about the content under evaluation. That is, expresses not only the quality of the content under evaluation but also a large variety of factors related to the observer, but independent with respect to the evaluation procedure itself.

Let assume that is continuously distributed in the interval , according to a probability density function (pdf) . We denote by MOS and the mean value and standard deviation, respectively:

, , (IV-1) where denotes the expectation (average) of a r.v and Var() the variance of a r.v.

Assume now the case in which, during the evaluation procedure, a discrete grading scale with quality levels is imposed to the observers. From the statistical point of view, this means that the observers' scores would be distributed according to a new r.v. . Assuming that the constraint of evaluating on quality levels has no psycho-cognitive impact in the observer's scores, the values taken by can be obtained from the values taken by trough a non-linear mapping function , as illustrated in The mean value of , denoted by represents the mean opinion score corresponding to the evaluation on a of quality level grade scale; it can be computed from (IV-3), by considering the Dirac's Delta function properties:

.

(IV-4)

The standard deviation of , denoted by , can also be computed: To conclude with, equation (IV-3) demonstrates that the r.v. Y is completely determined by the r.v. X and q, irrespective from the pdf of X.

  M X dx x p 0 1 ) (    M X dx x xp X E MOS 0 ) ( ) (     M X MOS dx x p x X Var 0 2 2 ) ( ) (                 q i q M i M x x q M i i x x f y ,..., 2 , 1 where , , 0 ) 1 ( , 0 , 0 ) (       q i q M i q M i X Y dx x p i y y p 1 ) 1 ( ) ( ) ( ) (                      q i q M i q M i X q i q M i q M i X Y q dx x p i dy i y dx x p y dy y yp MOS 1 ) 1 ( 1 ) 1 ( ) ( ) ( ) ( ) (  (
From the quality evaluation point of view, this means that the and corresponding to a continuous scale evaluation would allow to a posteriori obtain on any levels grading scale and to also have information about its statistical precision, by computing the 95% confidence limits according to (I-2), where is given by (IV-5).

Note that an equation (IV-3) is independent with respect to the law of X. Two cases of relevance in quality evaluation are subsequently investigated. First, according to the common trend [START_REF] Simone | Subjective evaluation of JPEG XR image compression[END_REF], [START_REF] Seshadrinathan | Study of subjective and objective quality assessment of video[END_REF], [START_REF] Winkler | On the properties of subjective ratings in video quality experiments[END_REF], X will be considered as a Gaussian law distribution, see Section IV.2. Secondly, as doubts about the possibility of modeling the scores by Gaussian law also arise (ITU-R BT.1788, ITU-R BT.500-11, ITU-R BT.500-13), the X law will be represented as a mixture of Gaussian laws, Section IV.3.

      q i q q M i q M i X q MOS dx x p i 1 2 ) 1 ( 2 ) (  p q 1 2 3 i y P Y (y)    q M i q M i X i dx x p p ) 1 ( ) ( p i p 2 p 1 p 3 0 MOS M P X (x) x x y=f(x)
1 2 i q iM/q (i-1)M/q q IV.2. Gaussian assumption

IV.2.1. MOS and confidence limits computation for q grading scales

Assuming now a Gaussian behavior for the r.v. , equations (IV-3), (IV-4) and (IV-5) can be expressed in an explicit form:

(IV-6) (IV-7) (IV-8)
where It can be concluded that if is Gaussian distributed, the mean value and standard deviation of solely depend on the mean value and standard deviation of , cf. (IV-7) and (IV-8) and on the q number of levels on the grading scale.

From the quality evaluation point of view, this means that the and corresponding to a continuous scale evaluation would allow to a posteriori obtain on any levels grading scale and to also have information about its statistical precision; this property will be investigated in the following section.

IV.2.2. Illustration of continuous to discrete MOS computation

In order to illustrate the way in which the continuous scale parameters can be a posteriori mapped for computing the discrete, unlabeled q levels grading scales, the experiments are structured at two levels.

            q i Y i y q iM erfc q M i erfc y p 1 ) ( ) 2 / ( ) 2 / ) 1 ( ( ) (                q i q q iM erfc q M i erfc i MOS 1 ) 2 / ( ) 2 / ) 1 ( (                q i q q MOS q iM erfc q M i erfc i 1 2 2 ) 2 / ( ) 2 / ) 1 ( (        x dt t x erfc ) exp( 2 ) ( 2  IV.2.2.1.

Continuous scale evaluation

The following experiment is carried out 4 times, once for each type of content described in Section III.4 (2D and stereoscopic video, high and low quality). For each type of content, only the scores assigned by the 60 observers using a continuous scale (between 1 and M) are considered.

For a given type of content, the experiments start by estimating the mean value and the standard deviation (un-biased estimators) of the observers' scores. These estimated values are considered as the theoretical (unknown) parameters and of the r.v. , see (IV-9):

, (IV-9)
We then compute the according to (IV-7), for q=2, 3,…,9 and their confidence limits according to (I-2) and (IV-8). These values will be further considered as theoretical (reference) values for the mean opinion scores which would have been obtained on a discrete grading scale with semantic levels.

IV.2.2.2. Discrete unlabeled scale simulation

The evaluation on the unlabeled, discrete, grading scales are obtained by mapping the data to their corresponding values on a would-be levels grading scale, according to (IV-2), thus obtaining values for the r.v. : . Consequently, the estimated values and the 95% confidence limits can be computed as follows:

, , (IV-10)

IV.2.2.3. Illustrations

Now, a comparison can be achieved between the theoretical values , see Chapter IV.2.2.1 and their confidence limits in Chapter IV.2.2.2. The results corresponding to each of the four types of content, to q=2, 3,…,9 and to N=50, 30 and 15 are detailed in Appendix A. In the sequel of the chapter, only an illustrative selection is considered, as follows:

 Figure IV-2 corresponds to the score assigned for the Image quality of the high-quality stereoscopic video content; three subplots, denoted by (a), (b) and (c) correspond to q = 3, q = 5 and q = 9, respectively. For each subplot, three experimental cases are presented (from left to

   N j j x N MOS 1 1      N j j MOS x N 1 2 ) ( 1 1  [x 1 , x 2 ,..., x N ] [ y 1 = f (x 1 ),y 2 = f (x 2 ),..., y N = f ( y N) ] (MOS q;low ;MOS q;up )    N j j q y N MOS 1 1      N j q j q MOS y N 1 2 ) ( 1 1  N MOS MOS q q low q  96 . 1 ;   N MOS MOS q q up q  96 . 1 ;   ) ; ( ; ; up q low q MOS MOS

IV.2.2.4. Conclusion

This section instantiates the theoretical formulae (IV-3), (IV-4) and (IV-5) for the case in which the scores assigned by the observers follows a Gaussian law. It is thus demonstrated that the MOS and the confidence limits corresponding to a would-be discrete, unlabeled scale with q levels can be theoretically computed starting from the MOS and the standard deviation of the scores assigned on a continuous grading scale.

The illustrations strengthen these theoretical results: each and every time (i.e. for each investigated criterion, each watermarking method, each insertion domain, each and values), the theoretical values belong to the experimental confidence limits:

. As expected, for a given N value, the absolute error is an increasing function of q (see the left-side plots in Figures IV-6, IV-7, IV-8, IV-9, IV-10, IV-11, IV-12 and IV-13). However, it can be noticed that the relative errors are quite constant with q (see the right-side plots in the same 4 figures).

Actually, the experiments bring to light differences in the relative error lower than 0.04, for all the types of content, for N=50, 30 and 15 and q=2,3,..,9.

(IV-12)

where .

IV.3. Beyond Gaussian assumption

The present Chapter IV.3 goes on step further with respect to Chapter IV.2: it first investigates the validity of the Gaussian hypothesis for the scores assigned by the observers on continuous scale then instantiate the theoretical equations (IV-3), (IV-4) and (IV-5) for an arbitrarily (non-Gaussian) case.

IV.3.1. Gaussian mixture estimation

It is commonly assumed that r.v. scores assigned on a continuous scale are Gaussian distributed [SIM09], [START_REF] Seshadrinathan | Study of subjective and objective quality assessment of video[END_REF], [START_REF] Winkler | On the properties of subjective ratings in video quality experiments[END_REF]. However, no firm support is available in this respect and the ITU specification implicitly let the door open for scores that are not Gaussian distributed.

Consequently, the Gaussian behavior of the continuous scale scores is verified by applying the unilateral Chi-square goodness-on-fit tests (with 10 classes, estimated law parameters, and

).

Such tests are applied for each and every 140 investigated score sets (cf. details in Chapter III.4):

 48 score sets for the high quality stereoscopic content, corresponding to the 16 sequences under test and to the 3 evaluation criteria (Image quality, Depth perception and Visual comfort);

 48 score sets for low quality stereoscopic content, corresponding to the same cases as above;  16 score sets for the high quality 2D video content;  28 score sets for low quality 2D video content.

The results of the Chi-square tests demonstrate that the Gaussian hypothesis can be accepted only 15 times while the alternative (non-Gaussian) hypothesis is accepted 125 times. Note that these 15 exceptions are provided by all the four types of content: 3 exceptions for the high quality stereoscopic content, 4 exceptions for low quality stereoscopic content, 5 exceptions for high quality 2D video content and, finally, 3 exceptions for low quality 2D video content. These quantitative values show that the largest relative number of exceptions corresponds to high quality 2D video content, namely 5 out of 16.

Consequently, the study presented in Chapter IV.2 should also be extended with solution for the non-Gaussian models for the scores assigned by the observers. Specifically, the continuous scale scores probability density function is to be estimated. In this respect, a mixture of Gaussian laws combined to an Expectation-Maximization (EM) algorithm [START_REF] Dempster | Maximum Likelihood from Incomplete Data via the EM algorithm[END_REF] is considered and result in a model of the type:

(IV-13) where and are the mean values and the variances of the 5 laws composing the mixture, stands for the underlying weights in the mixture while .

Of course, such a procedure was carried-out 140 times, once for each of the 140 investigated data sets.

X ) (x p X 5  K

IV.3.2. MOS and confidence limits computation for q grading scales

Assuming now the Gaussian mixture model (IV-13) for the r.v. , equations (IV-3), (IV-4) and (IV-5) can be expressed in an explicit form:

(IV-14) (IV-15)
It can be concluded that even if is not Gaussian distributed, the mean value and standard deviation of solely depend on the mean value and standard deviation of the Gaussian pdf composing the mixture, cf. (IV-14) and (IV-15) and on the q number of levels on the grading scale.

Hence, a similar behavior as in the Gaussian case is expected in quality evaluation: the and corresponding to a continuous scale evaluation would allow to a posteriori obtain on any levels grading scale and to also have information about its statistical precision.

IV.3.3. Illustration of continuous to discrete MOS computation

The illustrations in this Chapter are structured the same way as Chapter IV.2.1.

The following experiment is carried out 4 times, once for each type of content described in Section III.4 (2D and stereoscopic video, high and low quality). For each type of content, the scores assigned by the 60 observers using a continuous scale (between 1 and M) are considered. Subsequently, the underlying Gaussian mixture are estimated and, by applying (IV-14) and (IV-15), the can be computed. These values will be further considered as theoretical (reference) values for the mean opinion scores which would have been obtained on a discrete grading scale with semantic levels.

As in Chapter IV.2, the evaluation on the unlabeled, discrete, grading scales are obtained by mapping the data to their corresponding values on a would-be levels grading scale, according to (IV-2), thus obtaining values for the r.v. :

. Consequently, the estimated values and the 95% confidence limits can be computed according to (IV-10).

A comparison can be achieved now between the theoretical values and their confidence limits .

The results corresponding to each of the four types of content, to q=2, 3,…,9 and to N=50, 30 and 15 are detailed in Appendix B. In the sequel of the chapter, the same illustrative selection as in Chapter IV.2 is considered, as follows:

 Figure IV-14 corresponds to the score assigned for the Image quality of the high-quality stereoscopic video content; three subplots, denoted by (a), (b) and (c) correspond to q = 3, q = 5 and q = 9, respectively. For each subplot, three experimental cases are presented (from let to The overall results point to a similar conclusion as in Chapter IV.2.2.4. Even when the scores assigned by the observers on a continuous scale do not follow a Gaussian law and are estimated by a Gaussian mixture, the MOS and the confidence limits corresponding to a would-be discrete, unlabeled scale with q levels can be theoretically computed starting from the MOS and the standard deviations of the Gaussian laws composing the mixture. Beyond its theoretical demonstration, the conclusion is also experimentally supported.

High quality stereoscopic video content

IV.4. Conclusion

By using non-linear random variable transformations principles, Chapter IV.1 establishes the formula connecting the probability density functions modeling the continuous and discrete grading scales scoring. The first and second order moments (allowing for the MOS and its confidence limits to be computed) are subsequently derived. This formula is generic and does not take into account the original probability density function peculiarity. Consequently, Chapters IV.2 and IV.3 consider the instantiations of this formula for two opposite cases of practical relevance, namely the case in which the scores according on a continuous scales are Gaussian distributed and the case in which these scores are not Gaussian (but estimated via a Gaussian mixture). In the two cases, it is brought to light that the MOS and the related confidence limits solely depend on the average and variances of the continuous scale models and of the q number of quality levels on the discrete scale. The theoretical results are illustrated through experiments corresponding to the scores assigned by the 4 reference panels (one panel for each type of content -see Chapter III).

The impact of the number of human observers in the precision of the quality evaluation is subsequently assessed. It is demonstrated that for 3 N values ranging from 15 to 50, the relative errors in MOS estimation are constant with both N and q (differences lower than 0.04 and 0.02 being obtained for Gaussian and non-Gaussian cases, respectively).

To conclude with, in order to converge to a unique answer to the controversial issue related to the number of quality levels on a stereoscopic video grading scale, Chapter IV suggests to perform the evaluation on a continuous grading scale, with no semantic labels associated to the scores and to subsequently map these values on the discrete scales.

Such an approach is of benefit to the evaluation procedure, the tests become more user-friendly and the observer is no longer restricted in his/her choice to some pre-established application-dependent options. Moreover, finer and richer information about the evaluated content can be obtained out of a single experiment. Just for illustration, assume the case of a stereoscopic content provider who would like to perform a single experiment in order to assess the quality of a same content which is to be delivered on both TV (hence, strong quality constraints requiring q=5 or even more) and on some stereoscopic enabled smart-phone (where a coarser evaluation with q=3 is likely to be accepted).

V. Quantifying the impact of the semantic labels

The present chapter addresses the second challenge of the thesis, namely the would-be semantic impact of the ITU labels in the evolution scores: the existence of the semantic impact is first demonstrated, then the possibility of its assessment is addressed.

The existence of the semantic impact is verified by a comparison (based in the Student's paired test) between the average values (representing the MOS) corresponding to the continuous (unlabeled) and discrete, semantically labeled scales.

The main contribution of the chapter is the definition a methodological framework for quantifying this semantic impact. In this respect, an auxiliary discrete random variable is defined: it is characterized by uneven partition but by equal a posteriori probabilities. By comparing (through binomial tests) the differences in the partition class length between this auxiliary random variable and the random variable corresponding to the semantically labeled scale, the semantic impact is quantified (by defining an underlying coefficient).

The experimental results have two-folded outcomes: (1) reference values for the semantic impact (i.e. general values, independent with respect to the observers) are computed for each type of content (stereoscopic and 2D video, high and low quality) and for the two investigated numbers of quality levels (q=3 and q=5); (2) the fact that practically all the labels (and not only Excellent) are involved in the overall semantic impact is demonstrated.

Chapter I.2.6 identified three controversial issues related to the ITU evaluation procedures, related to the use of continuous and discrete terms in conjunction with a grading scale, to the statistical model of the scores assigned by the observers as well as to the use of semantic labels for the grading scales. While the first two of them are dealt with in Section IV, the present Chapter V addresses the last issue, namely the would-be semantic impact of the ITU labels in the evolution scores.

In this respect, the existence of the semantic impact is first investigated (Chapter V.1), then the possibility of its assessment is addressed (Chapter V.2).

V.1. Verifying the existence of the semantic impact of the labels

The semantic impact of the labels is the fact that the labels attached to the discrete evaluation scale stimulate or make the observer more reluctant in assigning the class corresponding to that level, thus modifying the MOS.

Consequently, in order to verify whether such a semantic impact exist, the mean opinion scores corresponding to unlabeled and labeled scales should be compared between them. Of course, should these two types of MOS be identical, no semantic impact is encountered. On the contrarily, differences between these two types of MOS bring to light the existence of the semantic impact.

The un-labeled scores correspond to the scores assigned by the reference panel while the labeled scores correspond to the scores assigned by the validation panel (the 5 level grading scales and 3 levels grading scales are alternatively investigated).

140 data sets are processed (see Chapter IV.3.1 and Chapter III.4):

 48 score sets for the high quality stereoscopic content, corresponding to the 16 sequences under test and to the 3 evaluation criteria (Image quality, Depth perception and Visual Comfort);

 48 score sets for low quality stereoscopic content;  16 score sets for the high quality 2D video content;  28 score sets for low quality 2D video content.

For each of the 140 reference data sets, the unlabeled MOS is computed according to (IV-14) and considered as theoretical values; actually, when applying (IV-14), two q values are alternatively considered, namely q = 5 and q = 3.

The 140 validation scores are considered as experimental data which are compared to these theoretical values by applying an unpaired Student's t-test [START_REF] Walpole | Probability & statistics for engineers and scientists[END_REF]. Such a test is applied with N=25 (the size of validation corpus), and estimated variance.

The experimental results bring to light the number of Student tests which are not passed: 23 when q = 5 and 17 when q = 3. As the proportions of test which are not passed are 0.37 and 0.12 (which are larger than ) we can state that the semantic impact does exist but that it seems to be less important for q = 3 than for q = 5.

V.2. Methodological framework for semantic impact assessment

V.2.1.

Evaluation principle

The principle of the semantic impact evaluation is illustrated in Figure V.1.

In Chapter IV, the r.v. X and Y are introduced so as to model the scores assigned by the observers on a continuous, unlabeled scale and to compute the discrete, unlabeled q level grading scale, (IV-14).

Assume now a discrete r.v. Z modeling the scores assigned by the observers, on a discrete, labeled scale.

In case no semantic impact exists, the Y and Z r.v. should be identical (that is, they are expected to take the same q values with the same probabilities). On the contrarily, the differences between the Y and Z r.v. bring to light the existence of the semantic impact and allow for its assessment.

V.2.2. Methodology presentation

The methodological framework for assessing the semantic impact of the labels is presented in Figure

V-2.
It is composed of three main steps, related to the computation of the Y r.v. by using the results in Chapter IV, to the estimation of the Z r.v. from the scores assigned by the observers (the validation panels), and to the assessment of the semantic impact as the differences between the Y and Z.

Step

1: Continuous & discrete unlabeled scales

Step 1.1 Perform the continuous evaluation experiment

According to this step, the human observers are asked to score the content on a continuous scale, e.g. between 0 and ; the result of this step is the data set , corresponding to the reference panels.

Step 1.2 Estimate the probability density function (pdf)

This step is performed by a Gaussian mixture, whose parameters are estimated under an EM (expectation-maximization) criterion. The result of this step is the pdf.

Step 1.3 Compute the Y random variable, i.e. the discretization of X according to an even partition of the interval This step is performed by applying a non-linear random variable filtering operation , according to (IV-2).

Step 2: Discrete, labeled scale evaluation

Step 2.1 Perform the discrete evaluation experiment According to this step, the human observers are asked to score the content on a levels semanticlabeled discrete scale; this way, the data sets corresponding to the validation panels (cf. Chapter III.5) are obtained.

Step 2.2 Estimate the probability density function

This step can be performed by any discrete pdf estimation method, applied to the data set obtained in the previous step. For instance, in the thesis, a frequency based estimation is considered:

(V-1)
where is the relative frequency of the i th quality class.

Step 3: Unlabeled vs. labeled discrete scales

Step 3.1 Find the identity condition This step searches for the partition ensuring identity between the and random variables. In this respect, the can be considered as theoretical reference and as an experiment outcome to be validated through a goodness-on-fit test (e.g. the binomial test).

Step 3.2 Compute the relative variation of the partition intervals with respect to the uniform partition This step computes the set of coefficients , where:

. (V-2)

A unitary value for such a coefficient demonstrates that the related semantic label does not modify the evaluation -that is, an even partition ensures the identity between Y and Z. A value larger than 1 indicates that the related semantic label makes the observer more likely to score that way while, conversely, a value lower than 1 shows that the related label makes the observers more reluctant in assigning that label when scoring.

Note that this investigation is finer than the one carried out in Chapter V.1: this time the whole pdf is considered while in previous case only its mean value.

V.2.3.

Quantitative results

V.2.3.1.

A finer investigation of the semantic impact existence Chapter V.1 brings to light the existence of the semantic impact in the MOS computation; that is, it was demonstrated there that the r.v. modeling the scores assigned on unlabeled and labeled scales have different mean values. The methodological solution presented in Chapter V.2.2 makes possible a finer analysis: this time, the very pdf modeling the scores assigned on unlabeled and labeled scales can be compared between them.

The same experimental conditions as in Chapter IV and Chapter V.1 are kept. This time, the experiment is carried out separately for the 4 types of content.

The un-labeled scores correspond to the scores assigned by the reference panel while the labeled scores correspond to the scores assigned by the validation panel (the 5 level grading scales and 3 levels grading scales are alternatively investigated). Each time, 140 data sets are processed (see Chapter IV.3.1 and Chapter III.4):

 48 score sets for the high quality stereoscopic content;  48 score sets for low quality stereoscopic content;  16 score sets for the high quality 2D video content;  28 score sets for low quality 2D video content.

Two q values are alternatively considered, namely q = 5 and q = 3. These values will be further considered as reference (theoretical) values.

Let us consider now the validation corpus. The , are estimated as the relative frequencies of the scores assigned on the labeled scales, namely Excellent, Good, Fair, Poor (for q = 5 quality levels evaluation) and Bad or Good, Fair and Bad (for q = 3 quality levels evaluation).

These values will be further considered as experimental values which are compared to the theoretical values according to a binomial test (the test on probability), applied at [START_REF] Walpole | Probability & statistics for engineers and scientists[END_REF].

) (x p X 5  K ] , 0 [ M Y 5 , 4 , 3 , 2 , 1  q ) 1 (  Y P ) 2 (  Y P ) 5 (  Y P           q i y y X y y X Y l l i i dx x p dx x p i l l Y p 1 1 1 ) ( ) ( ) ( ) (  (.)  ) (i p Z   5 , 4 , 3 , 2 , 1  i ) ( i Y P  05 . 0  

123

The experimental results will be subsequently detailed for each type of content.

High quality stereoscopic video content

The number of tests which are not passed (i.e. the number of cases in which statistical significant differences between the probabilities of and are encountered) are presented in Figure V-3, as a function of the investigated value i (equivalent, of the semantic label): the left side correspond to q = 3 while the right side to q = 5.

For q = 3, Figure V-3.a shows that no statistical differences are encountered for i=1 (i.e. for Poor). However, differences are spotted out for i 2,3 , (i.e. for Fair and Good): 4 tests for each class (out of the total of 48) are not passed.

For q = 5, Figure V-3.b shows that no statistical differences are encountered for i 1, 2, 3 , (i.e. for Bad, Poor and Fair). However, differences are spotted out for i 4,5 , i.e. for Good and Excellent): 17 and 13 tests (out of the total of 48) are not passed, respectively. These results reinforce the preliminary investigation in Chapter V.1: Figure V-3 demonstrates that statistical differences between the pdfs of and r.v. exist and that the semantic labels come across with psycho-cognitive side effects in the scores. Moreover, it is thus demonstrated that such semantic impact is not only connected to the Excellent label. Chapter V.3.2.2 will quantify these effects.

Low quality stereoscopic video content

The results corresponding to the evaluation of the low quality stereoscopic video content are presented in Figure V-4, which is organized the same way as Figure V-3. While the semantic impact is also identified, this time it is mainly associated to the Bad and Fair labels for q=3 where no statistical differences are encountered for i=1 (i.e. for Bad). However, differences are spotted out for i 2,3 , (i.e. for Fair and Good): 1 and 3 tests (out of the total of 48) are not passed, respectively.

High quality 2D video content

The results corresponding to the evaluation of the high quality 2D video content are presented in Figure V-5, which is organized the same way as Figure V-3.

The semantic impact is now less important, the number of exceptions being under the limit of statistical significance to which the tests are applied: differences are spotted out for i 2,3 , (i.e. for Fair and Good): 2 and 1 tests (out of the total of 16) are not passed, respectively, see Figure V-5.a.

For q = 5, Figure V-5.b shows that differences are spotted out for i 1,2,3,4,5 , ( i.e. for Bad, Poor, Fair, Good and Excellent): 1, 4, 5, 8 and 7 tests (out of the total of 16) are not passed, respectively.

Here again, Figure V-5 demonstrates that statistical differences between and r.v. exist and reinforce the idea that Excellent is not the single label "guilty" for that. Chapter V.3.2.2 will quantify these effects.

Low quality 2D video content

The results corresponding to the evaluation of the high quality 2D video content are presented in Quantifying the semantic impact

The impact of the semantic labels is quantified by finding the uneven partition which leads to statistical identity between and pdfs and by subsequently computing the underlying coefficient.

The experimental conditions are kept the same as in Chapter V.2.3.1. The results are presented in Table V.1 and Table V.2 and illustrated in Figures V-3 -V-6 (corresponding to the 4 types of investigated content). The tops (upper parts) of the Figures V-3 -V-6 represent the case of q = 3 while their bottoms (lower parts) represent the case q = 5.

In order to explain the experiments and the meaning of Figures V-3 -V-6 in the sequel the case of high quality stereoscopic content and of q = 5 is considered.

Chapter V.2.3.1 brings to light that, for high quality stereoscopic video content, differences between the Y and Z pdfs are only related to the Good and Excellent labels. Hence, the comparison between and are resumed on uneven partitions of the type , with

. Thus, the binomial tests consider only the values vs. and vs.

. It can be noticed that the differences between and are minimal when 86, 87, 88; actually, this time, the differences are at the limit of the statistical significance. Note that the binomial test is applied at and that it is repeated 48 times, for two type of tests; hence, 2 or even 3 failed tests cannot be considered as a proof of difference between two probabilities. Figure V-4 shows that some psycho-cognitive mechanisms make the observers associate to Excellent an interval , or , instead of an interval that would have been related to a discrete scale with no semantic labels. In other words, the observers are reluctant in assigning the label Excellent and prone to assign the label Good.

A measure of this phenomenon can be the coefficient , defined in Chapter V.2.2 as the ratio of the actual size of the interval on which the labels are assigned to the size of an unlabeled scale interval.

In the experiments, q=5, M=100 and 87,88; hence 0.65 ; 0.6. (blue bottom bar) and P(Y=5) vs.

(top pink bar) for q = 5. (blue bottom bar) and P(Y=3) vs.

4 4 3 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 4
(top pink bar) for q = 5. (blue bottom bar) and P(Y=3) vs.

(top pink bar) for q = 5. (blue bottom bar) and P(Y=5) vs.

(top pink bar) for q = 5. (blue bottom bar) and P(Y=3) vs.

4 4 3 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 4
(top pink bar) for q = 5. (blue bottom bar) and P(Y=3) vs.

(top pink bar) for q = 5.

V.3. Conclusion

The present chapter addresses the second main challenge of the thesis, namely the would-be semantic impact of the ITU labels in the evolution scores: the existence of the semantic impact is first investigated, then the possibility of its assessment is addressed.

The existence of the semantic impact is verified (Chapter V.1) by a comparison (based in the Student's paired test) between the average values (representing the MOS) corresponding to the continuous (unlabeled) and discrete, semantically labeled scales. The experiments bring to light that such an semantic impact exist for all the investigated cases, i.e. for the 4 types of content and for the two numbers of quality levels on the discrete scale (q = 3 and q = 5). Yet, this is only a preliminary result and does not bring any in-depth information about where this impact comes from.

The main contribution of the chapter is the definition a methodological framework for quantifying the semantic impact. The investigation method is composed of three main steps. First, by using the results presented in Chapter IV, the Y r.v. modeling the scores assigned by the observers on an discrete unlabeled scale is computed. Secondly, the Z r.v. modeling the scores assigned by the observers on a discrete labeled scale is estimated. Finally, the assessment of the semantic impact is achieved by defining a coefficient (denoted by ρ) expressing the differences between the Y and Z.

The experimental results have two-folded outcomes. First, reference values for the semantic impact (i.e. general values, independent with respect to the observers) are computed for each type of content (stereoscopic and 2D video, high and low quality) and for the two investigated numbers of quality levels (3 and 5). Secondly, the fact that practically all the labels (and not only Excellent) are involved in the overall semantic impact is demonstrated. Of course as expected, the extent to which each label impact the overall result, depend on the type of content.

Quality evaluation is an ever-fascinating field, covering at least a century of research works emerging from psychology, psychophysics, sociology, marketing, medicine, ... While for visual quality evaluation, the ITU Recommendations pave the way towards well-configured, consensual evaluation conditions granting reproducibility and comparability of the experimental results, an in-depth analysis of the state-of-the-art studies shows at least three open challenges, related to the:

(1) the continuous vs. discrete evaluation scales, (2) the statistical distribution of the scores assigned by the observers and (3) the usage of semantic labels on the grading scales. Thus, the present thesis turns these challenges into three research objectives:

1. bridging at the theoretical level the continuous and the discrete scale evaluation procedures and investigating whether the number of the classes on the discrete scales is a criterion meaningful in the results interpretations or just a parameter; studying the theoretical influence of the statistical model of the evolution results and of the size of the panel (number of observers) in the accuracy of the results are also targeted; 2. quantifying the bias induced in subjective video quality experiments by the semantic labels (e.g.

Excellent, Good, Fair, Poor, and Bad) generally associated to the discrete grading scales; 3. designing and deploying and experimental test-bed able to support their precision and statistical relevance.

With respect to these objectives, the main thesis contributions (detailed in the conclusive sections included in each chapter) are at theoretical, methodological and experimental levels.

First, at the theoretical level, the continuous and the discrete scale evaluation procedures are bridged by a non-linear random variable transformation formula. The instantiations of this formula for two opposite cases of practical relevance, namely the case in which the scores according on a continuous scales are Gaussian distributed and the case in which these scores are not Gaussian (but estimated via a Gaussian mixture) brings to light that the MOS and the related confidence limits solely depend on the average and variances of the continuous scale models and of the q number of quality levels on the discrete scale.

Secondly, at the methodological level, the main contribution consists in the definition of a methodological framework for quantifying the semantic impact induced by the ITU semantic labels. In this respect, an auxiliary discrete random variable is defined: it is characterized by uneven partition but by equal a posteriori probabilities. The semantic impact is quantified by assessing the differences in the partition class length between this auxiliary random variable and the random variable corresponding to the semantically labeled scale. The experimental results not only identify reference values for the semantic impact, according to the type of content (stereoscopic and 2D video, high and low quality) and to the two investigated numbers of quality levels (3 and 5) but also bring to light that practically all the labels (and not only Excellent) are involved in the overall semantic impact.

A test-bed designed and deployed for investigating the evaluation procedure per-se is also a result of the thesis. It considers a total of 640 human observers, about 80 minutes of video content, and scores assigned on three types of scales (continuous, 5 level labeled scale and 3 level labeled scale).

Hence, as a global conclusion, in order to converge to a unique answer to the controversial issue related to the number of quality levels on a stereoscopic video grading scale, the thesis suggests to perform the evaluation on a continuous grading scale, with no semantic labels associated to the scores and to subsequently map these values on the discrete scales. The ITU labels Excellent, Good, Fair, Poor, and Bad should be avoided or, at least, during the result interpretation, it should be kept in mind that semantic impacts as large as 0.55 or 0.6 (while a non-semantic behavior would have been denoted by a value of 1) can bias the results.

The future work will be structured at three levels.

First, a richer interpretation of the scores already gathered thanks to the experiments carried out in the thesis is targeted, as for examples studies on: inter-gender variability, on the correlation among the scores corresponding to the two views (stereoscopic content) vs. a single view, or on the dependency of the semantic impact coefficient with the MOS.

Work will be also devoted to provide the video corpus and the scoring application as open research resources. This way, the research community can capitalize on our results and bring additional results related to various aspects, from language dependability to multimodal investigations.

Finally, the theoretical results presented in Chapter IV and methodological framework in Chapter V.2 will be reverse investigated in order to find out how correction formulae for the semantic impact can be defined. It should be emphasized that the It should be emphasized that while the semantic impact does depend with the type of content (stereoscopic or 2D video) and its quality, it is always important. Such a result will emphasis the thesis pragmatic relevance and will turn it into a support for day-by-day subjective quality evaluation.

  ABSTRACT ............................................................................................................................................................... I. INTRODUCTION .............................................................................................................................................

Figure I- 1 :

 1 Figure I-1: Visual acuity and color vision charts: Snellen (left) and Ishihara (right)..……………………………………….……………………………25 Figure I-2: Continuous rating scale for DSCQS method………………………………………………………………………………………………………………..26 Figure I-3: Five-grade impairment scale for DSIS method…………………………………………………………………………………………………..……….26 Figure I-4: Categorical scale for Comparison method…………………………………………………………………………………………………………..….….27 Figure I-5: Categorical scale for Single Stimulus method……………………………………………………………………………………………………..……...27 Figure II-1: Grading scales used in [SIM11]; (a) DSIS method scale, (b) DSCQS method scales……………………………………………………..38 Figure II-2: Grading scales used in [McK78]…………………………………………………………………………………………………………………………………44 Figure II-3: Grading scales used in [Alb81]…………………………………………………………………………………………………………………………………..45 Figure II-4: Grading scales used in [JON86]………………………………………………………………………………………………………………………………….47 Figure II-5: Graphical user interface for ITU visual quality assessment………………………………………………………………………………………..51 Figure II-6: State-of-the-art synopsis………………………………………….……………………………………………………………………………………………….53 Figure III-1: Left and right views sampled from the 3DLive corpus………………………………………………………………………………………………61 Figure III-2: Scoring synopsis………………………………………………………………………………………………………….……………………………………………64 Figure III-3: Login interface…………………………………………………………………………………………………………………………………………………..…….65 Figure III-4: Video control interface……………………………………………………………………………………………………..........................................65 Figure III-5: Scoring interfaces…………………………………………………………………………………………………………………………………………………….67 Figure IV-1: Continuous to discrete scale mapping for the subjective evaluation scores................................................................74 Figure IV-2: Subjective evaluations for high-quality stereoscopic video content…………………………………………………………………………79 Figure IV-3: Subjective evaluations for low quality stereoscopic video content………………..............................................................81 Figure IV-4: Subjective evaluations of high quality 2D video content…………………………………………..………………………………………………83 Figure IV-5: Subjective evaluations for low quality 2D video content …………………………….…………………..………………………………………85 Figure IV-6: Precision in the evaluation of high quality stereoscopic video content (Image quality)……………………………………..…….88 Figure IV-7: Precision in the evaluation of high quality stereoscopic video (Depth perception)…………………………………………………..88 Figure IV-8: Precision in the evaluation of high quality stereoscopic video (Visual comfort)……………………………………..………………..88 Figure IV-9: Precision in the evaluation of low quality stereoscopic video (Image quality)………………………………………………………….89 Figure IV-10: Precision in the evaluation of low quality stereoscopic video (Depth Perception)…………………………………..…..…………89 Figure IV-11: Precision in the evaluation of low quality stereoscopic video (Visual Comfort)………………………………………..…………….90 Figure IV-12: Precision in the evaluation of high quality 2D video content…………………………………………..........................................90 Figure IV-13: Precision in the evaluation of low quality 2D video content……………………………………………………………………………………90 Figure IV-14: Subjective evaluations for high-quality stereoscopic video content, non-Gaussian case ...........................................96 Figure IV-15: Subjective evaluations for low quality stereoscopic video content, non-Gaussian case ……….………………………..………98

  1: Grading scales on the subjective quality assessment: constraints, challenges, current limitations and contributions. ...................................................................................................................................................................................................... …15 Table I-1 General viewing conditions for subjective assessments .................................................................................................... 22 Table I-2 Preferred Viewing Distance-PVD for video according to ITU-R BT.500-11. ....................................................................... 22 TableII-1: Methodological panorama of subjective visual quality assessment ................................................................................ 39 Table II-2: Background studies related to grading scales. ................................................................................................................ 44 Table II-3: Grading scales in visual quality assessment. ................................................................................................................... 47

Figure I- 1 :

 1 Figure I-1: Visual acuity and color vision charts: Snellen (left) and Ishihara (right).

  Figure I-2.

Figure I- 2 :

 2 Figure I-2: Continuous rating scale for DSCQS method (source: ITU-R BT.500-13, page 15).

FigureFigure I- 3 :

 3 Figure I-3: Five-grade impairment scale for DSIS method (c.f ITU-R BT.500-13 page 21).

FigureFigure I- 4 :

 4 Figure I-4: Categorical scale for Comparison method (c.f ITU-R BT.500-13, page 20).

Figure I- 5 :

 5 Figure I-5: Categorical scale for Single Stimulus method (c.f ITU-R BT.500-13, page 18).

  Figure II-1.b.

Figure II- 4 :

 4 Figure II-4: Grading scales used in [SIM11]; (a) DSIS method scale, (b) DSCQS method scale.

Figure

  Figure II-5 : Grading scales used in [MCK78].

Figure

  Figure II-6: Grading scales used in [ALB81].

Figure II- 4 :

 4 Figure II-4: Grading scales used in [JON86].

  discrete labeled scale 11-point continuous labeled scale 5-point continuous labeled scale 9-point discrete labeled scale.

  Figure II-5.b. According to the MSU-CQE method, two sequences are played simultaneously; if during the playback, the assessor estimates that one sequence is worse than another, he/she strokes the left or the right arrow keys on his keyboard, depending on a position of the sequence he/she dislikes. In such a case, a red mark appears on top of the video the observer is voting against, see Figure II-5.c. An interface for evaluating visual quality content according to an ACR method is provided by AccepTV [WEB05]. Here, the assessor asks to play the video, score it and then he/she can ask to watch it again, see Figure II-5.d.As a current trend, these examples show that the ITU recommendations can be translated into userfriendly, application oriented software implementations. The study developed in the present thesis follows such a principle and develops an interface matched to the peculiarities of our investigation.

Figure

  Figure II-5: Graphical user interface for ITU visual quality assessment: (a) A graphical user interface designed for the DSCQS methodology [MSU13], (b) A graphical user interface designed for the DSISS methodology [MSU13], (c) A graphical user interface designed for the MSU-CQE methodology[MSU13], (d) A graphical user interface designed for the ACR method by AccepTV [WEB05].

  ]) or for impairments (Imperceptible, Perceptible but non annoying, Slightly annoying, Annoying and Very annoying in [NIN06], [NIN09], [SIM11], [BOS11], [HAN15], [SU15]);  some studies utilize discrete scale without labels [HEW08],[PIE14], [HAN15]. Yet, this taxonomy is far for being complete, and various intermediate situations are encounted:  the studies [TEU96], [PRE00], [WIN03] consider both continuous and discrete labeled scale;  in its turn, [WIN09] utilize continuous and discrete unlabeled scales;  [MCK78] utilize discrete labeled and unlabeled scale;  [ALB81], [SVE00] utilize continuous and discrete, labeled and unlabeled scales.

Figure II- 6 :

 6 Figure II-6: State-of-the-art synopsis, according to a taxonomy based on the type of scale (continuous vs. discrete) and the presence of the labels (unlabeled vs. labeled). The thesis has as objective to theoretically and methodologically bridge the evaluations achieved on continuous unlabeled scales (lower-left quarter) and discrete labeled scales (top-right quarter).

  Rec. ITU-R BT.500-11 ARTEMISRatio of luminance of inactive screen to peak luminance Ratio of the luminance of the screen, when displaying only black level in a completely dark room, to that corresponding to peak white:Display brightness and contrast: set up via PLUGE softwareThe viewing distance and the screen sizes are to be selected in order to satisfy the Preferred Viewing Distance PVD, see TableI-2. Maximum observation angle relative to the normal (this number applies to CRT displays, whereas the appropriate numbers for other displays are under study): not applicable Ratio of luminance of background behind picture monitor to peak luminance of picture: Chromaticity of background: D65 Other room illumination: low III.3. The evaluation methodology and the grading scales III.3.1.1.

  Figure III-1: Left and right views sampled from the 3DLive corpus.

III. 6 .

 6 Scoring and post-processing of the scores III.6.1. Scoring application Following the current state-of-the-art trend for translating the ITU recommendations into user-friendly, application oriented software implementations, an interface matched to the peculiarities of our investigation is developed. The scoring application is Android-based and runs on compatible tablets and smartphones. It is connected to the server on which the video content to be evaluated is stored and prepared by the test administrator. Of course, the video itself is displayed on the TV screen, see Figure III-2. The scoring application offers to the observer three successive interfaces, as illustrated in Figure III-3, Figure III-4 and Figure III-5.

Figure

  Figure III-2: Scoring synopsis

Figure III- 3 :

 3 Figure III-3: Login interface.

Figure III- 4 :

 4 Figure III-4: Video control interface.

  Figure III-5.a, Figure III-5.b and Figure III-5.c. The evaluation concerns the perceived Visual quality.

Figure III- 5 :

 5 Figure III-5: Scoring interfaces: (a) Continuous scale scoring interface for 2D video content; (b) 5 levels discrete scale scoring interface for 2D video content; (c) 3 levels discrete scale scoring interface for 2D video content; (d) Continuous scale scoring interface for stereoscopic video content; (e) 5 levels discrete scale scoring interface for stereoscopic video content; (f) 3 levels discrete scale scoring interface for stereoscopic video content.

Figure

  applying basic non-linear random variable filtering properties, pdf of denoted by ) can be expressed as: (IV-3) where denotes the Dirac's Delta function, see Figure IV-1.

  Figure IV-1: Continuous to discrete scale mapping for the subjective evaluation scores.

  right), corresponding to N = 50, N = 30 and N = 15. For each N value, 16 types of sequences are presented, as detailed in Chapter III.4.1 (they correspond to watermarked content obtained by applying 4 different methods -SS, 2-QIM, 5-QIM and IProtect, in 4 different insertion domainsview-based, NTSS, FS-MPEG, and 3DV-NTSS).  Figure IV-3 is similar to Figure IV-2 but refers to low quality stereoscopic video content, as detailed in Chapter III.4.2.  Figure IV-4 is similar to Figure IV-2 but refers to high-quality 2D video, as detailed in Chapter III.4.3; hence, this time, the observers scores the Visual quality.  Figure IV-5 refers to low quality 2D video, as detailed in Chapter III.4.4. Hence, here again the Visual quality is scored but this time 28 types of sequences are scored.

Figure IV- 2 :Figure IV- 4 :Figure

 24 Figure IV-2: Subjective evaluations for high-quality stereoscopic video content, for grading scales of: (a) quality levels; (b) quality levels; (c) quality levels; and for a number of observers N=50, N=30, N=15.

Figure IV- 6 :Figure IV- 7 :FigureFigure IV- 9 :FigureFigure

 679 Figure IV-6: Precision in the evaluation of high quality stereoscopic video content (Image quality); on the abscissa -the number of the quality levels ; on the ordinate -the 95% error value: (a) Relative error; (b) Absolute error.

  right), corresponding to N = 50, N = 30 and N = 15. For each N value, 16 types of sequences are presented, as detailed in Chapter III.4.1 (they correspond to watermarked content obtained by applying 4 different methods -SS, 2-QIM, 5-QIM and IProtect, in 4 different insertion domainsview-based, NTSS, FS-MPEG, and 3DV-NTSS.  Figure IV-15 is similar to Figure IV-14 but refers to low quality stereoscopic video content, as detailed in Chapter III.4.2.  Figure IV-16 is similar to Figure IV-14 but refers to high-quality 2D video, as detailed in Chapter III.4.3; hence, this time, the observers scores the Visual quality.  Figure IV-17 refers to low quality 2D video, as detailed in Chapter III.4.4. Hence, here again the Visual quality is scored but this time 28 types of sequences are scored.

FigureFigure

  Figure IV-14: Subjective evaluations for high-quality stereoscopic video content, non-Gaussian case, for grading scales of: (a) quality levels; (b) quality levels; (c) quality levels; and for a number of observers N=50, N=30, N=15.

FigureFigureFigureFigureFigure

  Figure IV-17: Subjective evaluations for low quality 2D video content, non-Gaussian case grading scales of: (a) quality levels; (b) quality levels; (c) quality levels; and for a number of observers N=50, N=30, N=15.

FigureFigure

  Figure IV-23: Precision in the evaluation of low quality stereoscopic video, non-Gaussian case (Visual comfort) on the abscissa -the number of the quality levels ; on the ordinate -the 95% error value: (a) Relative error; (b) Absolute error.

Figure V- 1 :

 1 Figure V-1: Principle of the semantic impact evaluation

Figure V- 2 :

 2 Figure V-2: The methodological framework.

For

  each type of content, let us consider the reference panel, the same estimated by means of mixture of Gaussian laws, and the same evenly distributed partition of the interval. 's Delta function, see Figure IV-1.

Figure V- 3 :

 3 Figure V-3: The number of failed binomial tests between the values taken by the Y and Z r.v., when scoring high quality stereoscopic video content by the validation panel: (a) for q=3; (b) for q=5.

  5, FigureV-4.b shows that no statistical differences are encountered for i=1, (i.e. for Bad ). However, differences are spotted out for i 2,3,4,5 , ( i.e. for Poor, Fair, Good and Excellent): 9, 12, 2 and 1 tests (out of the total of 48) are not passed, respectively.

Figure V- 4

 4 Figure V-4 strengthen the conclusion from Figure V-3 that statistical differences between and r.v. exist and that the various semantic labels come across with psycho-cognitive side effects in the scores.

Figure V- 4 :

 4 Figure V-4: The number of failed binomial tests between the values taken by the Y and Z r.v. when scoring low quality stereoscopic video content by the validation panel: (a) for q=3; (b) for q=5.

Figure V- 5 :

 5 Figure V-5: The number of failed binomial tests between the values taken by the Y and Z r.v. when scoring high quality 2D video content by the validation panel: (a) for q=3; (b) for q=5.

  Figure V-6, which is organized the same way as Figure V-3. This time, the semantic impact it is mainly associated to the Bad and Fair labels for q=3: 2, and 3 tests (out of the total of 16) are not passed, respectively, see Figure V-6.a. For q = 5, Figure V-5.b shows differences for i 1,2,3,4,5 , (i.e. for Bad, Poor, Fair, Good and Excellent):2, 14, 12, 5 and 2 tests (out of the total of 28) are not passed, respectively.

Figure V- 6

 6 Figure V-6 demonstrates the same overall behavior: the existence of the semantic impact, as a consequence of several semantic labels.

Figure V- 6 :

 6 Figure V-6: The number of failed binomial tests between the values taken by the Y and Z r.v. when scoring high quality 2D video content by the validation panel: (a) for q=3; (b) for q=5.

  The investigated results are presented in Figure V-4. The horizontal axis gives the values. The vertical axis gives the number of binomial tests which did not pass, represented by bars: the low (blue) bar corresponds to (Good) while the top (pink) bar corresponds to (Excellent).

  Figure V-7: The number of failed binomial tests when assessing high quality stereoscopic video content: (a) P(Y=2) vs.(blue bottom bar) and P(Y=3) vs.(top pink bar) for q = 3; (b) P(Y=4) vs.

  Figure V-8: The number of failed binomial tests when assessing low quality stereoscopic video content: (a) P(Y=1) vs.(blue bottom bar) and P(Y=2) vs.(top pink bar) for q = 3; (b) P(Y=2) vs.

  Figure V-9: The number of failed binomial tests when assessing high quality 2D video content: (a) P(Y=2) vs. (blue bottom bar) and P(Y=3) vs. (top pink bar) for q = 3; (b) P(Y=4) vs. (blue bottom bar) and P(Y=5) vs.(top pink bar) for q = 5.

  Figure V-11: The number of failed binomial tests when assessing high quality stereoscopic video content: (a) P(Y=2) vs.(blue bottom bar) and P(Y=3) vs.(top pink bar) for q = 3; (b) f P(Y=4) vs.(blue bottom bar) and P(Y=5) vs.(top pink bar) for q = 5.

  Figure V-12: The number of failed binomial tests when assessing low quality stereoscopic video content: (a) P(Y=1) vs.(blue bottom bar) and P(Y=2) vs.(top pink bar) for q = 3; (b) P(Y=2) vs.(blue bottom bar) and P(Y=3) vs.(top pink bar) for q = 5.

  Figure V-13: The number of failed binomial tests when assessing high quality 2D video content: (a) P(Y=2) vs. (blue bottom bar) and P(Y=3) vs. (top pink bar) for q = 3; (b) P(Y=4) vs.(blue bottom bar) and P(Y=5) vs.(top pink bar) for q = 5.

  1: Subjective evaluations for high-quality stereoscopic video content (Image Quality), for grading scales of: quality levels and for a number of observers N=50, N=30 and N=15. Figure A1-2: Subjective evaluations for high-quality stereoscopic video content (Depth Perception), for grading scales of: (a) , (b) , (c) , (d) , (e) , (f) , (g) , (h) quality levels and for a number of observers N=50, N=30 and N=15..

  Figure A1-3: Subjective evaluations for high-quality stereoscopic video content (Visual Comfort), for grading scales of: (a) , (b) , (c) , (d) , (e) , (f) , (g) , (h) quality levels and for a number of observers N=50, N=30 and N=15..

  4: Subjective evaluations low-quality stereoscopic video content (Image Quality), for grading scales of: (a) , (b) , (c) , (d) , (e) , (f) , (g) , (h) quality levels and for a number of observers N=50, N=30 and N=15.

  5: Subjective evaluations low-quality stereoscopic video content (Depth Perception), for grading scales of: (a) , (b) , (c) , (d) , (e) , (f) , (g) , (h) quality levels and for a number of observers N=50, N=30 and N=15.

  Figure A1-6: Subjective evaluations low-quality stereoscopic video content (Visual Comfort), for grading scales of: (a) , (b) , (c) , (d) , (e) , (f) , (g) , (h) quality levels and for a number of observers N=50, N=30 and N=15.

  Figure A1-7: Subjective evaluations high-quality 2D video content ,for grading scales of: (a) , (b) , (c) , (d) , (e) , (f) , (g) , (h) quality levels and for a number of observers N=50, N=30 and N=15..

  1: Subjective evaluations for high-quality stereoscopic video content (Image Quality), for grading scales of: quality levels and for a number of observers N=50, N=30 and N=15.

Table III -

 III 1 Viewing conditions for test-bed.………………………………………………..……………………………………………………………………….……56 Table III-1: Panel composition and size: a total of 640 observers, split in 4 sub-panels of 160 observers each..........…………………..61 Table III-3: Outliers' detection results……………………………..………………………………………………………………………………………………………….66 Table V-1: Quantifying the semantic impact for q=3…………………………….……………………………….………………………………………………….123 Table V-2: Quantifying the semantic impact for q=5……………………………………………………….…………..……………………………………………124 Table V-3: The generality of the quantization of the semantic impact for q=3……..………………..……………………………..………………130 Table V-4: The generality of the quantization of the semantic impact for q=5…………..………………………………………………………….131 Table V-5: Reference values for the semantic impact for q=3……………………………..…………………………………………………………………132 Table V-6: Reference values for the semantic impact for q=5……………….………………………………………………………………………………..133

Tableau 1 : Les échelles de classement de l'évaluation subjective de la qualité: contraintes, défis, limitations et contributions.

	Contraintes	Défis		Limitations				Contributions de la thèse
	Etude théorique pour les échelles de notation			 Échelles de notation continues et discrètes  Nombre variable de niveaux de qualité sur l'échelle de notation  Comportement statistique des scores	 Utilisation de différentes échelles de notation avec différents niveaux sans aucune étude sur leur impact sur les résultats  L'hypothèse implicite que les scores ont des distributions gaussiennes		• •	Etablir une formule théorique pour mapper les notes attribuées sur une échelle de notation continue à toute échelle de notation discrète (partitionnée de manière uniforme ou non), avec une précision statistique contrôlée: o Formule de filtrage non linéaire de variable aléatoire o Calcul des paramètres (MOS et écart type) Estimation de la fonction de densité de probabilité pour les scores des observateurs via un mélange gaussien o Application de l'algorithme de Expectation Maximization EM) o Nombre optimal des paramètres du mélange gaussien Evaluer l'impact du nombre d'observateurs humains sur la précision de l'évaluation de la qualité
	Cadre méthodologique pour	l'évaluation de l'impact des	étiquettes sémantiques	 Quantification l'influence cognitive	de psycho-	 Au cours de la procédure d'évaluation, on suppose que: o aucune contrainte psycho-cognitive n'est imposée aux observateurs par les étiquettes sémantiques o les scores sont attribués uniquement en fonction du classement de l'étiquette sur les		•	o o o	Définir un cadre méthodologique pour l'identification et l'évaluation de l'impact des étiquettes sémantiques: Comparaison basée sur les tests appariés de Student pour déterminer si un tel impact existe Définition d'une variable aléatoire auxiliaire: partition inégale mais distances de probabilités a posteriori égales Définition et évaluation d'un coefficient d'impact sémantique par des tests binomiaux répétés
						échelles
	Cadre expérimental			 Répétabilité pertinence statistique et pour les résultats  Possibilité d'analyser la dépendance par rapport au contenu 2D et stéréoscopique  Vidéo haute et basse qualité  Logiciel de notation	 Configurations expérimentales vers l'évaluation du contenu orientées plutôt que vers l'évaluation de la procédure  Manque d'approches consensuelles  Pas de confrontation entre vidéo 2D et stéréoscopique  Aucune comparaison entre les échelles de notation à 5 et 3 niveaux		• • •	o o o o o	640 observateurs humains (160 observateurs pour chaque type de contenu), répartis en 3 types de sous-panels 4 x 60 pour le panneau de référence 4 x 50 pour le panneau de validation 4 x 50 pour le contrôle de contrôle croisé corpus vidéo 4 x 20 minutes: contenu 2D et stéréoscopique vidéo haute et basse qualité (35dB <PSNR <40dB et 25dB <PSNR <30dB) Limites de confiance à 95% pour le MOS et niveaux de signification de 0,05 pour chacun des tests considérés (Chi -2, test de Student, binomial)
								•		Discussion sur le type de contenu
									o	vidéo 2D vs vidéo stéréoscopique
									o	vidéo de basse qualité vs haute qualité
								•		Logiciel de notation Android:
									o	Echelle de notation continue
									o	Echelle de notation discrète étiquetée
										sémantiquement (utilisée alternativement
										à 5 et 3 niveaux)

Table A -

 A 1: The grading scales on the subjective quality assessment: constraints, challenges, current limitations and contributions.

	Constraints	Challenges	Current limitations	Thesis contributions
	Theoretical basis for grading scales	 Continuous discrete scales  Variable number of vs. grading quality levels on the grading scale  Observers scores statistical behavior	 Use of different grading scale with different quality levels, without any investigation about their impact in the meaning of results  Observers' scores are implicitly assumed to be Gaussian distributed	 Establishing theoretical formula for mapping the scores assigned on a continuous grading scale to any discrete grading scale (be it evenly partitioned or not), with controlled statistical accuracy: o Non-linear random variable filtering formula o Parameter computation (MOS and standard deviation)  Estimating the probability density function for the observers scores through a Gaussian mixture o Expectation maximization (EM) algorithm application o Optimal number of components in the Gaussian mixture investigation
				 Assessing the impact of the number of human
				observers in the precision of the quality
				evaluation

Methodological framework for assessing the semantic label impact

		 Psycho-cognitive influence quantification	 During procedure, it is assumed the evaluation that: o no psycho-cognitive constraint is imposed to the observers by the semantic labels o the scores are assigned solely based on the label rank on scales	 Specifying a methodological framework for identifying and assessing the impact of semantic labels: o Student's paired test based comparison for identifying whether the semantic impact does exist o auxiliary discrete random variable definition: un-even partition but a posteriori equal measurements distances o auxiliary random variable estimation o definition and evaluation of a semantic
				impact coefficient trough repeated
				binomial tests
				o
	Experimental test-bed	 Repeatability statistical relevance and for the results  Possibility of investigating the dependency on the content  2D and stereoscopic content  High and low-quality video  Soring tool	 Experimental oriented towards content set-ups evaluation rather then procedure evaluation  Lack of consensual approaches  No confrontation between 2D and stereoscopic video  No comparison between 5 and 3 levels grading scales	 640 human observers (160 observers for each type of content), partitioned in 3 types of sub-panels o 4 x 60 for the reference panel o 4 x 50 for validation panel o 4 x 50 for cross-checking panel  4 x 20 minutes video corpora: o 2D and stereoscopic content o high and low-quality video (35dB < PSNR < 40dB and 25dB<PSNR<30dB)  95% confidence limits for MOS and 0.05 significance levels for each and every considered test (Chi-square, Student's paired, Binomial)
				 Discussion on the content type
				o 2D video vs. stereoscopic content
				o low vs. high quality content
				 Android scoring application:
				o continuous grading scale evaluation
				o discrete, semantically labeled, evaluation
				(alternatively used with 5 and 3 levels)

Rec. ITU-R BT.500-11

	Table I-1: General viewing conditions for subjective assessments
	Ratio of luminance of inactive screen to peak luminance
	Ratio of the luminance of the screen, when displaying only black level in a completely dark room, to that
	corresponding to peak white:
	Display brightness and contrast: set up via PLUGE software
	The viewing distance and the screen sizes are to be selected in order to satisfy the Preferred Viewing
	Distance PVD, see Table I-2.
	Maximum observation angle relative to the normal (this number applies to CRT displays, whereas the
	appropriate numbers for other displays are under study):
	Ratio of luminance of background behind picture monitor to peak luminance of picture:
	Chromaticity of background: D65
	Other room illumination: low

Table I -

 I 2: Preferred Viewing Distance-PVD for video according to ITU-R BT.500-11.

	Screen diagonal (inch)
	4/3 ratio	16/9 ratio	Screen height (m)	PVD
	12	15	0.18	9
	15	8	0.23	8
	20	24	0.30	7
	29	36	0.45	6
	60	73	0.91	5
	> 100	> 120	> 1.53	3-4

I.2.2. Panel composition

	In order to set-up a proper panel, ITU recommends at least 15 observers to compose the panel. They
	should be non-expert, in the sense that they are neither directly concerned with image quality as part of
	their normal professional activity, nor experienced assessors.
	Prior to the session, the observers should be screened for normal (or corrected to normal) visual acuity,
	color vision and dynamic stereopsis. The visual acuity can be screen by the Snellen or Landolt charts
	while color vision can be tested by the Ishihara chart, see Figure I.1. The dynamic stereopsis can be
	tested by 8 main vision tests (referred to as VT-01 to VT-08): the tests VT-04 and VT-07 are compulsory
	while the remaining six tests are for more detailed characterization.

  For instance, [WEB03], since 2005, YouTube continuously increases the quality of its video archives. Initially, YouTube video is displayed at a 320×240 resolution, using a version of H.263 codec. Two years latter, an option to watch videos in 3GP format on mobile phones is added. In March 2008, a high-quality mode is introduced, the aspect ration is changed to 16:9 and the MPEG-4 AVC/H.264 encoder is provided by default. The 1080p HD support appears in 2009 while starting by 2010, the 4K format (up to 4096×3072 pixels) becomes available. In June 2015,

support for 8K (7680×4320 pixels) is added and HDR video is made available one year latter. Beyond Internet video, note that at the time of writing (the fourth quarter of 2017) Netflix reached 117.58 million streaming subscribers worldwide [WEB04].

  Table II-1: Methodological panorama of subjective visual quality assessment.

			Context		Testing conditions	Panel		Scale	Evaluated content
	[MUE02]	The variation in subjective quality evaluation introduced by the interaction between the audio and video modalities	ITU-T 1984	20	5 levels discrete quality scale labeled by Excellent, Good, Fair, Poor and Bad	-passive test: the subject is simply listening to/viewing the content -interactive communication: person to person audio/video
								data exchange
		the role of the visual attention in	ITU-R	20	5 levels discrete impairment	-10 undistorted images
	[NIN06]	image quality assessment	BT.500-10 DSIS		grading Imperceptible, Perceptible but scale labeled by non annoying, Slightly annoying,	-10 images with various artifacts
							Annoying and Very annoying
	[CAM07]	the methodology for subjective assessment of stereo images	ITU-R BT.1438 SAMVIQ	17	continuous scale from 0 to 100 with labels	six different distorted contents (JPEG and JPEG2000 compressions, blur)
		objective quality metrics for	ITU-R	17	continuous scale ranging from 0	-6 original images.
	[BEN08]	stereo images quality assessment relying on both the use of 2D metrics and depth information	BT.500-11. SAMVIQ		to 100	-60 stereo images with different degradation levels (JPEG and JPEG2000)
	[HEW08]	the subjective correlation and assessment of color plus depth between objective map 3D video	DSCQS	32	discrete, unlabeled scale ranging from 1 to 5	-13 original uncompressed videos -13 coded video sequences
		full reference video quality	ITU-R	36	5 levels discrete impairment	10 reference video sequences
	[NIN09]	assessment metric based on the temporal evolutions of the spatial distortions	BT.500-10 DSIS		grading Imperceptible, Perceptible but scale labeled by non annoying, Slightly annoying,	distorted with the MPEG-4 AVC (H264) codec (used at 5 different encoding
							Annoying and Very annoying	configurations)
		a procedure for subjective	ITU-R	16	continuous scale from 0 to 100	still pictures compressed with
	[SIM09]	codec for compression of still pictures evaluation of the new JPEG XR	DSCQS BT.500-11		Fair, Poor and Bad with the labels Excellent, Good,	JPEG XR codec
	[GOL10]	objective stereoscopic video quality metrics	ITU-R BT.500-11 (SSCQS)	20	5 levels discrete quality scale labeled by Excellent, Good, Fair, Poor and Bad	6 different stereoscopic videos content; each video is captured with different parameters
		a	subjective	evaluation	ITU-R	494	5 levels discrete impairment	HD content divided into 5
		framework serving the HEVC	BT.500-11		grading	scale	labeled	by	classes with different spatial
	[SIM11]	standard		DSIS DSCQS and		5 labels discrete quality scale Annoying and Very annoying non annoying, Slightly annoying, Imperceptible, Perceptible but	and temporal resolutions
							labeled by Excellent, Good, Fair,
							Poor and Bad
		reliability of objective quality	ACR	43	5 levels discrete quality scale	84 synthesized sequences: 3
	[BOS11]	metrics in 3DTV considering DIBR (Depth Image-Based Rendering) synthesized view evaluation problem			labeled by Excellent, Good, Fair, Poor and Bad	test sequences generating 4 viewpoints, corresponding to 12 synthesized sequences for each tested algorithm

  Table II-1: (continuing) Methodological panorama of subjective visual quality assessment.

	[URV12]		a database of high quality Full-HD stereoscopic sequences, shot with a semi-professional 3D camera	ITU-R BT.500 ACR-HR		29	5 levels discrete quality scale labeled by Excellent, Good, Fair, Poor and Bad	99 sequences of 16s and 11 sequences of 13s of full-HD stereoscopic sequences
	[CHE12-01]		perception of MPEG-4 AVC compressed stereoscopic 3D videos	SSCQS		NA	continuous scale ranging from 0 to 100	video sequences (a 3D pristine video, a 2D pristine video (right view), and 9 distorted videos with MPEG-4 AVC compressed
	[CHE12]-	02	3D QoE by constructing the visual experience as a weight sum of image quality, depth quantity and visual comfort	ITU-R BT.500-11 SAMVIQ	28	5 levels discrete quality scale labeled by Excellent, Good, Fair, Poor and Bad	3D distorted content with variation of depth focus and different levels of JPEG2000 compression.
	[CHA13]		robust techniques for stereoscopic watermarking video protection	ITU-R BT.500-12 DSCQS	25	5 levels discrete quality scale labeled by Excellent, Good, Fair, Poor and Bad	watermarked video content (34 randomly stereoscopic chosen video excerpts of 40 seconds each)
	[BOS13]		subjective quality assessment for free-viewpoint video decompressed data. sequences generated from	ACR-HR		27	5 levels discrete quality scale labeled by Excellent, Good, Fair, Poor and Bad	Free-viewpoint sequences generated from video decompressed data
			blind image quality assessment	ITU-R		24	continuous scale ranging from 0	distorted	images	with
	[GAO13]			BT.500-13 Paired comparison		to 100	(JPEG2000 compression, JPEG compression, Gaussian Blur, WN and FF)
	[PIE14]		the effect of the chromatic perceived visual quality variations in 3D video on the	ACR-HR		25	continuous scale with 11 points	8 video sequences of 3D video content
			method for no-reference	ITU-R	BT	57	discrete scale from 1 to 5	distorted images based on the
	[FAN15]		principle of natural scene statistics (NSS). quality assessment of contrast distorted images based on the	500-12			discrete scale from 0 to 9	principle of natural scene statistics (NSS)
			extensive benchmarking of	ITU-R		24	5 levels discrete impairment	240 compressed HDR images:
	[HAN15]		objective quality metrics for HDR image quality assessment	BT.2022 DSIS			grading Annoying and Very annoying scale labeled Imperceptible, Perceptible but by non annoying, Slightly annoying,	20 HDR contents encoded by 3 compression algorithms at 4 bit rates
			an oriented correlation models	ITU-R		NA	5 levels discrete impairment	distorted stereopair images
	[SU15]		of distorted natural mages with application to natural stereopair quality evaluation	BT.500-11 DSIS		grading Imperceptible, Perceptible but scale labeled by non annoying, Slightly annoying,
							Annoying and Very annoying

  Table II-2: Background studies related to grading scales.

		Context	Testing conditions	Panel	Scale	Applicative field
	[MAT71]	in a discrete scale the optimal number of categories	N.A	360	number of categories ranging from 2 to 19. 18 different labeled discrete scales, with the	psychology
	[MCK78]	on a graphic rating scale the optimal number of categories	NA	30	discrete scales with 5, 7 and 11 categories continuous scale	psychology
	[COX80]	alternatives for a scale the optimal number of response	NA	225	ranging from 2 to 9 discrete scales with the number of categories	marketing
	[ALB81]	semantic differential rating scales, continuous vs. discrete	NA	176	10 different categorical labeled scales (5 categories) continuous scale	academic
	[SVE00]	assessments using continuous and discrete ordinal rating scales comparison of the quality of	NA	174	discrete labeled scale continuous unlabeled scale	academic
		optimal number of response	NA	146	continuous 101-point scale (from 0 to 100).	marketing
	[PRE00]	categories in rating scales			discrete scale with number of categories from 2 to 11

-2

  Table II-3: Grading scales in visual quality assessment.

		Context		Testing conditions	Panel		Scale	Evaluated content
	[JON86]	terms graphic scaling of qualitative	NA	73	continuous labeled scale	picture and sound
		graphic scaling and validity of	EBU method	40	continuous labeled scale	HDTV picture
	[NAG93]	Japanese descriptive terms used in subjective evaluation tests	(European broadcastin g union)
		the validity of CCIR quality	CCIR 500-4	135	5 levels discrete quality scale	TV content
		indicators along a graphic	Recommend		labeled by Excellent, Good, Fair,
		scale		ations		Poor and Bad
	[TEU96]					continuous scale with labeled extremities
						continuous scale with numbers
						and labeled extremities
	[WAS98]	perceived quality of speech and video in multimedia conferencing applications	ITU-R	24	continuous unlabeled scale	speech and video in multimedia conferencing applications
		video quality evaluation for	ITU-R	20	5 levels discrete impairment	Internet	streaming
		internet	streaming	BT.500		grading scale labeled by	content
		applications				Imperceptible, Perceptible but
	[WIN03]			DSIS		non annoying, Slightly annoying, Annoying and Very annoying
				SSCQE		continuous scale with labeled
						extremities "Bad" and "Good"
		use of graphic scales in	ITU-R	13	5 levels discrete impairment	audio
		modern listening tests	BS.1534		grading	scale	labeled	by
	[ZIE07]			MUSHRA		Imperceptible, Perceptible but non annoying, Slightly annoying, Annoying and Very annoying
						A 5 labels discrete quality scale
						labeled by Excellent, Good, Fair,
						Poor and Bad
		examination of the SAMVIQ	SAMVIQ	N.A.	continuous labeled scale	reference sequences at
		methodology for the				CIF resolution (352 x 288
		subjective assessment of				pixels) and derived from
	[HUY07]	multimedia quality					standard-definition content	or	high-original
							video content encode at
							MPEG4,	H.264	and
							WMV9 codec
	[PEC08]

Table III -

 III 1 Viewing conditions for test-bed.

  Excellent, Good, Fair, Poor and Bad; such a scale is in concordance with ITU-R BT.500-11/13 and with the most of the state-of-the-art studies (cf. Section II);

	continuous unlabeled scale, ranging from 0 to 100; such a scale is in concordance with [CAM07],
	[BEN08], [CHE12-01], [GAO13] and ITU-T P.913;
	 discrete labeled scales:
	o with 5 levels:

o with 3 levels: Good, Fair and Bad; such a scale follows the ITU principles and the explicit suggestion in [GOL10-01] ITU-T P.913.

Table III -

 III 2: Panel composition and size: a total of 640 observers, split in 4 sub-panels of 160 observers

each Type of panels Panel composition and size Panel 1: Stereoscopic video content - High Quality Reference: 35

  males / 25 females aged from 22 to 37 with average of 26.

	Panel 2:	Reference: 31 males and 29 females aged from 20 to 31 with average of 25
	Stereoscopic	Cross-checking:
	video content -	o 5 Levels grading scale: 15males /10 females aged from 21 to 25 with average of 22
	Low Quality	o 3 Levels grading scale: 12 males /13 females aged from 22 to 27 with average of 23
		Validation:
		o 5 Levels grading scale: 13 males /12 females aged from 21 to 25 with average of 22
		o 3 Levels grading scale: 15 males /10 females aged from 22 to 28 with average of 24
	Panel 3:	Reference: 37 males and 23 females aged from 19 to 34 with average of 27
	2D video content	Cross-checking:
	-High Quality	o 5 Levels grading scale: 17 males /8 females aged from 22 to 25 with average of 23
		o 3 Levels grading scale: 13 males /12 females aged from 21 to 27 with average of 24
		Validation:
		o 5 Levels grading scale: 11 males /14 females aged from 23 to 26 with average of 24
		o 3 Levels grading scale: 15 males /10 females aged from 21 to 25 with average of 22
	Panel 4:	Reference: 28 males and 32 females aged from 19 to 30 with average of 26
	2D video content	Cross-checking:
	-Low Quality	o 5 Levels grading scale: 16 males /9 females aged from 21 to 25 with average of 22

Cross-checking:

o 5 Levels grading scale: 14 males /11 females aged from 21 to 25 with average of 22. o 3 Levels grading scale: 12 males /13 females aged from 20 to 25 with average of 22.

Validation:

o 5 Levels grading scale: 14 males /11 females aged from 23 to 30 with average of 24. o 3 Levels grading scale: 12 males /13 females aged from 21 to 24 with average of 22 o 3 Levels grading scale: 10 males /15 females aged from 21 to 31 with average of 26 Validation: o 5 Levels grading scale: 14 males /11 females aged from 21 to 25 with average of 23 o 3 Levels grading scale: 11 males /14 females aged from 20 to 25 with average of 22

  Of course, as in any statistical study, one key issue is the generality of the results. Consequently, the experiments reported in Chapter V.2.3.3 are resumed: the reference panels is kept unchanged but the validation panels are replaced by new panels of 25 observers each (the so-called cross-checking panels, Chapter III.5). All the other experimental conditions are kept unchanged with respect to Chapter V.2.3.2.

	1	1	1	V.2.3.3.		The generality of the results
				1
					1	1
							1	1
																						Fair
	13	13	13	12					1	1												Bad
											1
					10	10
												1	1	1
							8	8
	6 The new results are presented in Tables V.3 -V.4 and Figures V-11 -V-14, which are organized the same 6 5 1 1 1 2 3
	way as Tables V.1 -V.2 and Figures V-11 -V-14.			4	4	4	3	2	2			2	2	2
																		1	1	1
																		0	0
	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	31	32	33	34
	2	2																		10	11
			2	2															9			Fair
					2													8
						2											7					Poor
																6
	14 14					2	2							5
			12 12 11	9	7	6	4 2	4 2	4 2	3 2	3 4	3 4	5	6	7	8	9	10 10
	20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
											134

Note that in the present thesis, the watermarking details are irrelevant; yet, more details are available in [HAS17-01], [CHA13-01], [CHA13-02].

The distortion modes are irrelevant for the present study; they are derived from a research study related to the compressedstream watermarking saliency [AMM17-01], [AMM17-02]

Acknowledgment

I would like to thank all members of the jury. I thank Professor Patrick LE CALLET and Professor Azeddine BEGHDADI for accepting being my thesis reviewers and for their insightful comments and encouragement. I also thank Professor Patrick GALLINARI, Professor Faouzi GHORBEl, Professor Maryline LAURENT and Dr. Didier NICHOLSON for accepting being my thesis examiners.

Contexte Abstract

IV.2.3. Investigation on the accuracy of the results

In order to assess the accuracy (precision) of the results, both the absolute and the relative errors in computation will be investigated as functions of N and q. That is, we shall first investigate the influence of q for a fixed N value.; then, for a given q value, we shall investigate the influence of N.

For a given number of quality levels on the grading scale q, the absolute and relative errors in computation with a 95% confidence level, denoted by and can be approximated by where and can be computed according to (IV-7) and (IV-8) from the and (i.e. from the values estimated on the levels grading scale).

The numerical results, corresponding to the experiments described in Chapter IV.2.2 are presented in Figures IV-6, IV-7, IV-8, IV-9, IV-10, IV-11, IV-12 and IV-13. This time, rather then presenting individual results corresponding to each individual content, average values for a type of content are computed:

IV.3.4. Investigation on the accuracy of the results

The experiments are conducted the same way as in the Gaussian case, Chapter IV.2.3.

For a given number of quality levels on the grading scale q, the absolute and relative errors in computation with a 95% confidence level, denoted by and can be approximated by (IV-11) where and can be computed according to (IV-14) and (IV-15) from the and (i.e. from the values estimated on the levels grading scale). ). Yet, the upper limit for the differences between the relative errors corresponding to the various investigated configurations (4 types of content, N=50, 30 and 15 and q=2,3,..,9) is now reduced from 0.04 (see Chapter IV.2.3) to 0.02.



MOS  q 131

The conclusion from Figure V-7-V-10 is presented in Table V-1 and V-2. These two tables present the ρ values, for each of the 4 types of the investigated content and for q=3 and q=5, respectively.

Reference values for the semantic impact coefficient

When comparing the results presented in Chapter V.2.3.3 to the ones presented in Chapter V.2.3.2, it can be noticed that small differences exist; yet, each and every time common values exist in the corresponding cells in Tables V.1 -V.2 and V.3 -V.4. This demonstrates that the study carried out in this thesis allows the computation of some reference value for the semantic impact coefficient, see Tables V.5 and V.6. The word reference means here that such values are validated by all the three types of panels organized in the study.

List of publications

Published papers

List of acronyms