Keywords: New Caledonia, Erythroxylaceae, Flacourtiaceae, Malpighiaceae, Rhizophoraceae, Salicaceae

Taxonomic notes are made on the Malpighiales of New Caledonia. The names Acridocarpus austrocaledonicus Baill. (Malpighiaceae), Crossostylis grandiflora Brongn. & Gris, C. multiflora Brongn. & Gris, C. sebertii Pancher ex Brongn. & Gris (Rhizophoraceae), and Erythroxylum novocaledonicum O.E.Schultz (Erythroxylaceae) are lectotypified. Based on published molecular phylogenetic and morphological evidence, Lasiochlamys Pax & K.Hoffm

Taxonomic notes on New Caledonian Malpighiales: Acridocarpus, Crossostylis, Erythroxylum, and Xylosma YOHAN PILLON LSTM, IRD, INRAE, CIRAD, Institut Agro, Univ. Montpellier, Montpellier, France yohan.pillon@ird.fr; https://orcid.org/0000-0003-1760-329X New Caledonia has a rich and unique flora that is still imperfectly known (e.g. [START_REF] Gâteblé | One new endemic plant species on average per month in New Caledonia, including eight more new species from Île Art (Belep Islands), a major micro-hotspot in need of protection[END_REF][START_REF] Lannuzel | Mining rare Earth elements: Identifying the plant species most threatened by ore extraction in an insular hotspot[END_REF]. A checklist of the native vascular plant species of this archipelago is actively curated [START_REF] Morat | The taxonomic reference base Florical and characteristics of the native vascular flora of New Caledonia[END_REF][START_REF] Munzinger | FLORICAL: checklist of the vascular indigeneous flora of New Caledonia[END_REF]. Many plant groups have received little attention since the monumental work of André Guillaumin between 1910[START_REF] Sleumer | A concise revision of the Flacourtiaceae of New Caledonia and the Loyalty Islands[END_REF][START_REF] Hamel | André Guillaumin, huitième et dernier Professeur de Culture au Muséum national d'Histoire naturelle[END_REF], which included the last complete treatment of the entire New Caledonian flora [START_REF] Guillaumin | Flore analytique et synoptique de la Nouvelle-Calédonie[END_REF], and more specific treatments of the groups dealt with below [START_REF] Guillaumin | Matériaux pour la flore de la Nouvelle-Calédonie I[END_REF][START_REF] Guillaumin | Matériaux pour la flore de la Nouvelle-Calédonie XXX. Révision des Malpighiacées[END_REF][START_REF] Guillaumin | Matériaux pour la Flore de la Nouvelle-Calédonie. LXVI. Révision des Flacourtiacées (sensu lato)[END_REF]. Some novel lectotypifications, synonymies, combinations and other nomenclatural matters concerning Malpighiales, a particularly wellrepresented order on the island [START_REF] Pillon | Ultramafic soils and species sorting in the flora of New Caledonia[END_REF], are presented below. All specimens with (!) have been seen physically at P and elsewhere online.

ACRIDOCARPUS (MALPIGHIACEAE)

Acridocarpus austrocaledonicus Baillon (1874: 248) Acridocarpus alexandrinae Niedenzu (1928: 262) nomen nud.

When he described Acridocarpus austrocaledonicus, Baillon (1874) cited several collections (syntypes): Balansa (1039, 1475, 1688), Vieillard (s.n.) and Pancher (s.n.). One sheet of Balansa 1475 (P00130332), with Baillon's handwriting, is chosen here as the lectotype. [START_REF] Niedenzu | Malpighiaceae Pars II[END_REF] attributed the unpublished name Acridocarpus alexandrinae to Planchon, but this name was probably first coined by Pancher (written by hand on Vieillard 249, P05529955!, P05529958!).

CROSSOSTYLIS (RHIZOPHORACEAE)

CROSSOSTYLIS Forster & Forster (1776: 87). Type:Crossostylis biflora Forster & Forster (1776: 87).

Haplopetalon Gray (1854: 608). Type:Haplopetalon richii Gray (1854: 608).

Tomostylis Montrouzier (1860: 201). Type:Tomostylis multiflora Montrouzier (1860: 202). Brongniart & Gris (1861a: 376). Lectotype (designated here):NEW CALEDONIA, s.loc., 1861, Pancher & Vieillard 572bis (P05574247!).

Crossostylis grandiflora

Tomostylis multiflora Montrouzier (1860: 202). Type:NEW CALEDONIA, Montrouzier 125

(holo-: MPU312436!).
The name Crossostylis grandiflora can be considered to have been validly published by Brongniart & Gris (1861a), who compared this taxon to C. biflora J.R. Tomostylis multiflora Montrouzier (1860: 202) should not be considered as the basionym or as a homotypic synonym of Crossostylis multiflora Brongniart & Gris. At the time they published their work, Brongniart & Gris (1861a) were probably unaware of the paper that Montrouzier published the year before. Indeed, they published the genus Vieillardia Brongniart & Gris (1861b: 375), now considered as a synonym of Ceodes Forster & Forster (1776: 141), while Montrouzier had published the same generic name the year before for a different plant: Vieillardia Montrouzier (1860: 196), now considered a synonym of Castanospermum A.Cunn ex Mudie (1829: 149). Since the combination with the epithet multiflora is pre-empted in Crossostylis, the name Tomostylis multiflora Montrouzier is treated as a synonym of the junior name C. grandiflora Brongniart & Gris. Brongniart & Gris (1861a: 378). Haplopetalon multiflorum (Brongn. & Gris) Baillon (1862: 40)

Crossostylis multiflora

ERYTHROXYLUM (ERYTHROXYLACEAE)

Erythroxylum novocaledonicum Schultz (1907: 140) Erythroxylum couveleense Guillaumin (1964: 56) nomen inval.

When he described Erythroxylum novocaledonicum, Shultz (1907) cited three collections, all from the herbarium at K: Caldwell s.n. (without locality), Vieillard 2572 ("Gatop"), and Deplanche 275 ("Pum, Nené"). It is likely that Schultz misinterpreted the handwriting on the label of Deplanche 275, which is more likely to be from "Neué", corresponding to current day Néhoué, south of Poum. The sheet of Deplanche 275 at K is chosen as the lectotype. A number of specimens labelled Vieillard 2572 (K000700854!, P05486499!, P05486060! P00723580!), with the locality Neué, instead of Gatope (as cited in the protologue), and not annotated by Shultz, are excluded from the types. The name Erythroxylum couveleense Guillaumin (1964: 56) was not validly published according to article 40.1 [START_REF] Turland | International Code of Nomenclature for algae, fungi, and plants (Shenzhen Code) adopted by the Nineteenth International Botanical Congress[END_REF] because [START_REF] Guillaumin | Résultats scientifiques de la mission franco-suisse de botanique en Nouvelle-Calédonie (1950-1952) III[END_REF] cited two collections, Guillaumin & Baumann 13189 (P00522302!), and Baumann 15303 (P00522301!), but he did not designate either explicitly as the type.

XYLOSMA (SALICACEAE)

XYLOSMA Forster (1786: 22). Lectotype:-Xylosma orbiculata (Forster & Forster) Forster (1786: 22).

Lasiochlamys Pax & Hoffmann (1922: 17). Type:-Lasiochlamys reticulata (Schlechter) Pax & Hoffmann (1922: 17), syn nov.

Within his treatment of New Caledonian Euphorbiaceae, Schlechter described Cyclostemon reticulatus Schlechter (1906: 148); at the time, his knowledge of the genus Cyclostemon Blume (1826: 597) suggested that it occurred only as far west as New Guinea. Today, Cyclostemon is treated as a synonym of the pantropical genus Drypetes Vahl (1796: 49) (Putranjivaceae). Pax & Hoffmann considered that C. reticulatus was misplaced in Cyclostemon, and described the novel genus Lasiochlamys Pax & Hoffmann (1922: 17) to accommodate it within the tribe Antidesmateae of Euphorbiaceae-Phyllanthoideae (now Phyllanthaceae). Guillaumin was apparently unaware of the description of the genus Lasiochlamys, which does not appear in the name index of his flora of New Caledonia [START_REF] Guillaumin | Flore analytique et synoptique de la Nouvelle-Calédonie[END_REF]. He recognized Cyclostemon reticulatus, the sole species of the genus in New Caledonia, which he placed in Euphorbiaceae subfamily Phyllanthoideae. At that time, he recognized Xylosma as the sole native genus of Flacourtiaceae, while placing Homalium in Homaliaceae, and Casearia in Samydaceae. He subsequently described several species of Xylosma, including X. fasciculata Guillaumin (1953aGuillaumin ( : 414), X. koghiensis (1953aGuillaumin ( : 415), X. planchonellifolia (1953a: 415): 415), X. hurlimannii Guillaumin (1953b: 505), and X. trichostemona Guillaumin (1959: 124). [START_REF] Sleumer | A concise revision of the Flacourtiaceae of New Caledonia and the Loyalty Islands[END_REF] published concise revision of New Caledonian Flacourtiaceae, in which he included Casearia, Homalium, Xylosma, and Lasiochlamys. He was apparently the first author to place Lasiochlamys in Flacourtiaceae, although he did not go as far as providing a critical comparison with Xylosma, except in his generic key. According to the latter, Lasiochlamys has two whorls of tepals, whereas Xylosma has only one. He nevertheless combined in Lasiochlamys several species of Xylosma described by Guillaumin: X. fasciculata, X. koghiensis, X. planchonellifolia, X. hurlimannii, and X. trichostemona, and described additional species: L. peltata Sleumer (1974: 125), L. coriacea Sleumer (1974: 126), L.

pseudocoriacea Sleumer (1974: 126), L. rivularis Sleumer (1974: 127), L. cordifolia Sleumer (1974: 127), L. grossecrenata Sleumer (1974: 128), and L. mandjeliana Sleumer (1974: 128). [START_REF] Lescot | Flacourtiacées[END_REF] wrote the monographic treatment of Flacourtiaceae for "Flore de la Nouvelle-Calédonie et Dépendances", in which she provided more details about the floral morphology of Xylosma and Lasiochlamys. She considered that the perianth of Lasiochlamys consists of two whorls each of (3) 4 (5) pieces, whereas the perianth of Xylosma is composed of a single whorl with 4-6 (7-8) pieces, the latter numbers implying supernumerary organs. Based on her observations and interpretation of the floral morphology, she proposed the transfer L. grossecrenata and L. peltata to Xylosma.

New Caledonian Flacourtiaceae have received little attention since the work of [START_REF] Lescot | Flacourtiacées[END_REF]. Molecular phylogenetic studies showed that Flacourtiaceae was an unnatural assemblage [START_REF] Chase | When in doubt, put it in Flacourtiaceae: a molecular phylogenetic analysis based on plastid rbcL DNA sequences[END_REF], and a large number of its genera are now included in Salicaceae (Angiosperm Phylogeny Group II 2003), and that is the case for all the native species of New Caledonia [START_REF] Morat | The taxonomic reference base Florical and characteristics of the native vascular flora of New Caledonia[END_REF]. In his molecular phylogenetic study, [START_REF] Alford | Systematics studies in Flacourtiaceae[END_REF] found that Lasiochlamys was nested in Xylosma. The only species of Lasiochlamys that was sampled was the type, L. reticulata, which was sister to Xylosma vincentii Guillaumin (1953a: 415), also from New Caledonia. This result has been confirmed by data from additional accessions (Pillon & Gotty, unpublished). [START_REF] Alford | Systematics studies in Flacourtiaceae[END_REF] also noted that the presence/absence of petals is highly homoplasious in the family.

Since Xylosma and Lasiochlamys are not reciprocally monophyletic, are morphologically similar, have a confused history, and differ in a single character (one vs. two perianth whorls), which is homoplasious and the interpretation of which is subjective, Lasiochlamys is better treated as a synonym of Xylosma. The necessary combinations are provided below.

Xylosma cordifolia (Sleumer) Pillon comb. nov. (LSID: 77312120-1). Basionym: Lasiochlamys cordifolia Sleumer (1974: 127). Lectotype ([second step], designated here):-NEW CALEDONIA, pente est du Mt. Aoupinié 700 m, 7 November 1972, MacKee 25783 (P04849378!, iso-: P04849379!, G003866797!, L0011076!, L0011077!, NOU005996!).

Xylosma mandjeliana (Sleumer) Pillon comb. nov. (LSID: 77312121-1). Basionym:

Lasiochlamys mandjeliana Sleumer (1974: 128). Type:-NEW CALEDONIA, Pouébo: Mont Mandjélia (exploitation forestière Frouin), 600-750 m, 4 November 1971, MacKee 24581 (holo-: P00648808!, iso-: L0011083!, NOU005994!).

Xylosma pseudocoriacea (Sleumer) Pillon comb. nov. (LSID: 77312122-1). Basionym: Lasiochlamys pseudocoriacea Sleumer (1974: 126). Type:-NEW CALEDONIA, Haute Kouaoua: pente nord-est du Mé Ori, 800 m, 3 December 1969, MacKee 21232 (P00714499, P00714500 [physical specimens not located in October 2022]; iso-: L0011087!, NOU005992!, NOU005993!).

Xylosma reticulata (Schltr.) Pillon comb. nov. (LSID: 77312123-1). Basionym: Cyclostemon reticulatus Schlechter (1906: 148). Lasiochlamys reticulata (Schlechter) Pax & Hoffmann (1922: 17). Lectotype (designated by Lescot 1980: 64) Xylosma rivularis (Sleumer) Pillon comb. nov. (LSID: 77312125-1). Basionym: Lasiochlamys rivularis Sleumer (1974: 127). Type:-NEW CALEDONIA, La Rivière Bleue, 3 February 1965, Veillon 452 (holo-: L0011090!, iso-: NOU005991!, P00800997!, P00648810!, P00648811!).

Xylosma zongoi Pillon nom. nov. (LSID: 77312126-1). Replaced synonym: Lasiochlamys coriacea Sleumer (1974: 126). Type:-NEW CALEDONIA, Pouébo: Mont Mandjélia (exploitation forestière Frouin), 600-700 m, 4 June 1971, MacKee 24594 (holo-: P00648821!, iso-: L0011078!, NOU005995!). Non Xylosma coriacea (Poiteau) Eichler (1871: 449). Non Xylosma coriacea Lignier & Bey (1904: 167) nomen nudum. Non Xylosma coriacea Pancher ex Guillaumin (1919: 313) nomen nudum.

The combination in Xylosma in pre-empted by three earlier names. The new epithet chosen commemorates Charly Zongo (1980Zongo ( -2015)), for his contribution to our knowledge on the seed physiology of New Caledonian plants [START_REF] Bombarda | Fatty acids profile of Alphitonia neocaledonica and Grevillea exul var. rubiginosa seed oils, occurrence of an omega 5 series[END_REF]Zongo 2010;[START_REF] Zongo | Seed characterisation for propagation and conservation of Oxera pulchella subsp. grandiflora (Dubard) de Kok, a New Caledonian ornamental endemic[END_REF].

Acknowledgements.

I thank the curators of the herbarium at P (Le [START_REF] Bras | The French Muséum national d'histoire naturelle vascular plant herbarium collection dataset[END_REF] for providing access to their collections, Florian Jabbour (P) for discussions on floral morphology, Véronique Andro (P) for locating missing types, Caroline Loup (MPU) for providing a digital picture of the type material of Tomostylis multiflora, Helen Hopkins (K) for pictures of type material of Erythroxylum novocaledonicum and proofreading the language, and Jérôme Munzinger for comments on an earlier version of the manuscript. The collections at NOU were accessed online [START_REF] Bruy | L'Herbier de Nouvelle-Calédonie[END_REF].

  Lectotype (designated here):NEW CALEDONIA, versant méridional du Mont-d'Or, 17 May 1969, Balansa 1475 (P00130332!, iso-: P00537785!, P00537786!). Remaining syntypes: Koé, 14 December 1968, Balansa 1039 (P00130334!, P00537783!, P00537784!), Bai Lebris, December 1868, Balansa 1688 (P00130335!), s.loc., s.dat., Pancher 151 (P00130333!, P05529966!, P05529968!, P05529969!).

  Forst. & G.Forst. They later (1871) provided an expanded description, which has been used as the date of formal description in IPNI (2022). No type was indicated, Brongniart & Gris (1861a) only referring vaguely to collections made by Vieillard and Pancher. One such collection dated 1861 bears the name C. grandiflora, presumably in the hand of Brongniart, and it is chosen here as the lectotype.

  as 'Haplopetalum multiflorum'. Lectotype (designated here):NEW CALEDONIA, Balade, 1855-1860, Vieillard 43 (P00402853!). The name Crossostylis multiflora was validly published by Brongniart & Gris (1861a) in a footnote added in press with the citation of a specimen, Vieillard 43, and a comparison with C. biflora and C. grandiflora, which can be considered as a diagnosis. It was therefore already a valid name when Sebert & Pancher (1874) published a description of the species. Only one sheet of this number at P appears to bear Brongniart's original handwriting and it is treated here as the lectotype (P00402853!). Another sheet of Vieillard 43 (P05574280!), has a different locality (Wagap) and cannot be considered as an isolectotype. Crossostylis sebertii Pancher ex Brongniart & Gris (1871: 396). Lectotype (designated here):NEW CALEDONIA, Baie du Sud, s.dat., Sebert & Fournier 42 (P00645338!, iso-: P00645339!). When they published the name Crossostylis sebertii Pancher ex Brongniart & Gris, Brongniart & Gris (1871) mentioned a collection made by Sebert. Only a single gathering at P of Crossostylis was collected by Sebert & Fournier; one of the two sheets has the name Crossostylis sebertii handwritten by Pancher and is chosen here as the lectotype.

  . Lectotype (designated here):NEW CALEDONIA, Pum [=Poum], Nené [or Neué], 1861-1867, Deplanche 275 (K000700853!, iso-: P00547149!, P00547150!, P05486501!). Remaining syntypes: Caldwell s.n. (K000700852!), Gatope, 1861-1867, Vieillard 2572 (K000700851!, P00547147!, P00547148!, P05486498!).

  :-NEW CALEDONIA, Auf den Bergen bei Ou-Hinna, 600 m, 4 January 1903, Schlechter 15628 (P00648815!).


Andreas Pastor

Lukáš Krasula

Xiaoqing Zhu

Zhi Li

Patrick Le Callet

Andr Éas Pastor

Luk Á Š Krasula

Comparison of subjective methodologies for local perception of distortion in videos

come

Introduction

Subjective quality assessment methodologies provide essential feedback on the quality of a system and how users perceive it. They are necessary to benchmark objective quality metrics and to create datasets to train machine learning and deep learning models. However, running an in-lab or crowdsourced experiment takes time and effort. Furthermore, due to the subjectivity of the stimuli and the task of annotating them, there is not always an agreement in people's judgment. Accurate estimations are needed to reduce noise and uncertainty in collected data. It is then critical to select the most suited methodology to boost and allocate the annotation resources to the proper set of stimuli.

Multiple methodologies exist to rate stimuli, with absolute or relative quality estimations like those performed by Absolute Category Rating (ACR) or Degradation Category Rating (DCR) protocols defined in ITU standard [START_REF]Methodologies for the Subjective Assessment of the Quality of Television Images[END_REF]. Moreover, methods based on a ranking of stimuli are two-Alternative Forced Choice (2-AFC), pairwise comparison (PC), and protocols involving triplets or quadruplets. Ranking methodologies are more sensitive than ACR or DCR since observers only need to provide their prefer-ence over a set of stimuli, which makes it easier to build a judgment, reduce cognitive load, and thus increase the sensitivity of a subjective experiment. This improves what we call the discriminability between stimuli.

Ranking-based subjective experiments yield matrices of choices indicating how often a stimulus is preferred over another. A Pair Comparison Matrix (PCM) requires a model and mathematical solver to translate to a continuous scale. Examples of models are Thurstone [START_REF] Thurstone | A law of comparative judgement[END_REF], Bradley and Terry [START_REF] Bradley | The rank analysis of incomplete block designs -I. The method of paired comparisons[END_REF], and Tversky [START_REF] Tversky | Elimination by aspects: A theory of choice[END_REF]). Due to the pairwise manner of presenting stimuli, the PCM size and the number of possible comparisons rise quadratically with the number of stimuli. Hence, the efficiency of a subjective protocol becomes essential to reduce the number of comparisons to perform. Multiple previous works focused on active-sampling solutions [START_REF] Glickman | Adaptive paired comparison design[END_REF][START_REF] Pfeiffer | Adaptive polling for information aggregation[END_REF][START_REF] Li | Analysis and improvement of a paired comparison method in the application of 3dtv subjective experiment[END_REF][START_REF] Chen | Pairwise ranking aggregation in a crowdsourced setting[END_REF][START_REF] Ye | Active sampling for subjective image quality assessment[END_REF] and more recently [START_REF] Li | Hybrid-mst: A hybrid active sampling strategy for pairwise preference aggregation[END_REF][START_REF] Xu | Hodgerank with information maximization for crowdsourced pairwise ranking aggregation[END_REF][START_REF] Simpson | Scalable bayesian preference learning for crowds[END_REF][START_REF] Mikhailiuk | Active sampling for pairwise comparisons via approximate message passing and information gain maximization[END_REF] to select only the most informative pairs and minimize experimental effort while holding accurate estimations and robustness to poor annotator behavior (e.g., spammers).

Another ranking-based methodology is the Maximum Likelihood Difference Scaling (MLDS) methodology [START_REF] Maloney | Maximum likelihood difference scaling[END_REF][START_REF] Knoblauch | Mlds: Maximum likelihood difference scaling in r[END_REF]. It estimates how pre-ordered stimuli are perceived from comparisons of triplets or quadruplets to retrieve supra-threshold perceptual differences. Stimuli are usually generated from a reference stimulus with an increasing alteration process (e.g., encoding, colorgrading, rotation). Preferences of observers on the triplets or quadruplets are aggregated in matrices and used to estimate a perceptual scale per evaluated reference stimulus. This method lacks a global scale where all sets of stimuli from multiple reference stimuli can be represented and scaled. [START_REF] Pastor | Improving maximum likelihood difference scaling method to measure inter content scale[END_REF] presents a solution to modify the MLDS solver to address this limitation. Validation of the solution is performed through simulated annotations. In [START_REF] Pastor | Perceptual annotation of local distortions in videos: Tools and datasets[END_REF], more validation is performed on actual data from three datasets of annotations using only ranking-based methodologies (i.e., quadruplets, triplets, and pairwise comparisons). In this work, we compare to an additional fourth dataset. This new dataset of subjective opinions was collected using the DCR methodology [START_REF]Methodologies for the Subjective Assessment of the Quality of Television Images[END_REF]. All datasets are available on GitHub 1 .

Based on these data, we illustrate how subjective test methodologies influence the quality of collected subjective data by focusing on discriminability among estimated stimuli Mean Opinion Scores (MOS) [START_REF] Nehmé | Comparison of subjective methods for quality assessment of 3D graphics in virtual reality[END_REF][START_REF] Fela | Perceptual evaluation on audio-visual dataset of 360 content[END_REF][START_REF] Pastor | Towards guidelines for subjective haptic quality assessment: A case study on quality assessment of compressed haptic signals[END_REF][START_REF] Pastor | Comparison of conditions for omnidirectional video with spatial audio in terms of subjective quality and impacts on objective metrics resolving power[END_REF]. This aspect is crucial for objective metrics development and standardization activities, as evaluating and comparing system performances is essential. Sta-tistical testing that considers subjective data reliability is commonly used for comparing differences between correlations: Root Mean Square Error (RMSE) [START_REF] Rec | Methods, metrics and procedures for statistical evaluation, qualification and comparison of objective quality prediction models[END_REF], Spearman Rank Order correlation (SRCC) with the usage of MOS Confidence Interval (CI) [START_REF] Naderi | Transformation of mean opinion scores to avoid misleading of ranked based statistical techniques[END_REF], and methods estimating Resolving Power of objective metrics [START_REF] Bt | Methodological framework for specifying accuracy and cross-calibration of video quality metrics[END_REF][START_REF]Method for specifying accuracy and cross-calibration of Video Quality Metrics (VQM)[END_REF][START_REF] Krasula | On the accuracy of objective image and video quality models: New methodology for performance evaluation[END_REF]. However, this is an afterthought, and the subjective data quality requirement and how much one shall invest in subjective testing to allow discriminating between different quality estimation models has only been weakly studied. We focus on popular Video Quality Assessment (VQA) models validated to local perception estimation in [START_REF] Pastor | On the accuracy of open video quality metrics for local decision in av1 video codec[END_REF], such as VMAF [START_REF]Vmaf v0.6.1 model[END_REF], to demonstrate how subjective data discriminability affects our ability to compare these metrics. Our study encompasses different test methodologies, offering a unique perspective on the relationship between experimental cost and the conclusions drawn from the data. Our contributions include:

• Comparative analysis of subjective quality assessment methodologies for evaluating small video patches as a proxy for local distortion perception in videos. • Examination of experiment cost and its connection with discriminability. • Investigation of metric resolving power as influenced by subjective discriminability.

Local perception of distortion in videos

Estimating the perception of local distortions by the Human Visual System (HVS) is paramount in video compression. Since distortions introduced by video encoding algorithms can change the perception of content. Encoding is performed through Rate-Distortion estimation and optimization to achieve the highest quality while reducing the amount of bitrates consumed to store the encoded information. These decisions (e.g., selection of modes, partitionings, transforms) operate at the scale of Coding Units (CU) or Coding Tree Units (CTU) levels, which are spatially located, 128 × 128 pixels for the largest CU in AV1 2 .

On the other hand, the human eye performs gazes and saccades to focus on specific regions of interest. Each gaze is spatially localized, with the highest resolution achieved in the central region of the retina: the fovea region covers 1°of visual angle and has a sensitivity of 30 cycles per degree. Gazes are also temporally located with a duration averaging around 200 to 300ms when performing a fixation. Lastly, eye movements can track moving objects and anticipate their future position, commonly called smooth pursuit. Hence, the connection between these two aspects, video encoding CTU and HVS fovea, is not straightforward and needs to be understood. We define Percpetual Units (PU) as this spatio-temporal granularity of the HVS we want to understand and model to predict the perception of distortions and integrate it in video encoders. Figure 1 illustrates an example of PU. A PU is a 64x64 block of pixels and spans over 12 consecutive frames (400ms).

To understand how the HVS perceives distortions introduced by video encoding algorithms, we must collect data from human observers to possess a subjective ground truth. The choice of subjective methodology to collect, evaluate, and train objective metrics is essential. Multiple methodologies exist to collect the perceived quality of visual stimuli. Each subjective methodology requires a different number of observers and a different number of presentations per stimulus. The choice of the methodology will impact the time, cost, and quality of the data collection. An efficient and reliable methodology is needed to collect data on these Perceptual Units. Moreover, the results of methodologies need to be compared to select the most adapted and efficient one.

As we have PU to define the spatio-temporal granularity of the HVS we want to model, we can also define tube-contents. A tube-content TC i is a set of tubes, with a reference tube extracted at a specific spatiotemporal location in the video source (SRC) and then multiple versions of this reference tube but distorted with different encoding parameters. Hence, this set can be written as

TC i = {TC re f , TC 1 dist , ..., TC N dist }.
The size of these tubes follows the definition of a PU.

Subjective Datasets on Perceptual Units

This section presents the subjective datasets we collected using quadruplets, triplets, pairs, and double stimuli for local perception of distortions in videos.

The stimuli evaluated are from the dataset presented in [START_REF] Pastor | On the accuracy of open video quality metrics for local decision in av1 video codec[END_REF]. Tube-contents are extracted from high-quality videos encoded using AV1, reflecting types of contents and encoding recipes used at Netflix. 8 tube-contents were selected with 5 distortion levels in each tube-content TC i = {TC re f , TC 1 dist , ..., TC 5 dist }.

Quadruplet, Triplet and pair based dataset

This section recaps the three subjective experiments conducted in [START_REF] Pastor | Perceptual annotation of local distortions in videos: Tools and datasets[END_REF] with methodologies using ranking of stimuli.

Quadruplet-based intra-content dataset

We collected data on the 8 tube-contents described above for the quadruplet dataset. Each tube-content with 6 stimuli yields 15 possible quadruplets. From a

TC i = {TC re f , TC 1 dist , ..., TC 5 dist }, quadruplets are {(TC re f , TC 1 dist , TC 2 dist , TC 3 dist ), (TC re f , TC 1 dist , TC 2 dist , TC 4 dist ), ..., (TC 2 dist , TC 3 dist , TC 4 dist , TC 5
dist )}. We divided these 120 quadruplets from the 8 tube-contents into three playlists of 40 trials. We collected 1800 annotations in total, with 15 participants on each of the playlists.

Triplet-based intra-content dataset

In this dataset, we collected data on same 8 tube-contents using triplets generated following the procedure presented under "The Method of Triads", section 5 of [START_REF] Knoblauch | Mlds: Maximum likelihood difference scaling in r[END_REF]. Here, the 6 stimuli of a TC (i.e., reference + 5 levels of distortions) yield 20 triplets to annotate. We divided the resulting 160 triplets into four playlists of the reference (R) and distorted tubes (Di) are located. This information is not provided during tests and is just for the readers. From left to right, the first depicts quadruplets containing four unique distortion levels, while the second showcases triplets comprising three unique distortion levels. These quadruplets and triplets were sampled following the strategy outlined in [START_REF] Maloney | Maximum likelihood difference scaling[END_REF][START_REF] Knoblauch | Mlds: Maximum likelihood difference scaling in r[END_REF], adhering to the predetermined ordering of stimuli based on distortion strength. The third type of stimuli is pairs for quality ranking. The last type is for the DCR experiment with reference and distorted stimuli pairs. Observers provide an impairment level assessment on the ITU BT.500 impairment scale [START_REF]Methodologies for the Subjective Assessment of the Quality of Television Images[END_REF], see Table 1.

Quadruplet-based inter-content dataset

We used the active-sampling method proposed in [START_REF] Pastor | Improving maximum likelihood difference scaling method to measure inter content scale[END_REF] to collect inter-content scaling information. An example of intercontent comparison quadruplet is provided in Figure 3. The data used to initialize the active-sampling is a concatenation of the three previously described datasets to avoid any unfair advantage in later analysis. The sampling procedure was stopped after 55 samplings. Each sampling is a batch that contains 40 quadruplets. In between each sampling, the batch is annotated by an observer and fed to the active sampling algorithm.

DCR methodology based dataset

The test procedure follows the Degradation Category Rating (DCR) method specified in ITU-T Rec. P.910 [START_REF]Subjective video quality assessment methods for multimedia applications[END_REF]. Observers Imperceptible), see Table 1. The first stimulus is the reference tube, displayed on the left, and the second is the impaired tube on the right; see the rightmost example in Figure 2. The average viewing duration is 7 min for the 50 trials in the DCR test. Trials containing twice the reference tube are included in the test as attention checks. We expect observers to vote for "Imperceptible" impairment for these trials. If an observer, for at least one attention check, voted more than "Perceptible but not annoying", all of his answers are discarded for later analysis. 98 observers remain after this cleaning. Moreover, a stabilization phase of four pairs of stimuli is included at the beginning of the test: a no-impairment example, two with clearly perceptible impairment, and one with severely annoying impairment. This phase helps observers adjust to the testing methodology, contents, and interface. Stabilization scores are excluded during later analysis. The stimuli used for calibration differ from those assessed later in the test.

Table 2 summarizes all datasets and provides the total number of annotations gathered for each dataset. The term DCRequivalent observers denotes the equivalent number of observers needed to accumulate the same volume of annotations following the number of trials annotated in a DCR session. For example, the dataset based on quadruplet assessments yielded 4000 annotations, equivalent to 80 DCR-equivalent observers, as calculated by dividing 4000 by 50. 50 is the number of trials annotated in a DCR session.

Comparison of methodologies

This section compares the data collected with the four subjective quality assessment methodologies.

Usage of the scale

Figure 4 analyses the MOS obtained in the four test conditions. We fit a power function (in black) and extract the slope and the power exponent. We analyze these coefficients to see how assessors perceive the stimuli differently and use the rating scales. The red line translates the relationship where a PD-MOS from one test is equivalent to the PD-MOS of another test. In Figure 4 (top plots), we compare DCR test results with each ranking experiment. We can see strong agreement regarding the Pearson and Spearman correlation (PLCC, SRCC). SRCC CI from [START_REF] Naderi | Transformation of mean opinion scores to avoid misleading of ranked based statistical techniques[END_REF] is also inline. Nevertheless, the mapping between PD and MOS is not always linear, translating differences in estimating small distortion ranges. In Figure 4 (bottom plots), PLCC, SRCC, and SRCC CI agreement is even higher, up to 0.99 correlation. RMSE is smaller, indicating that the PD-MOS score mapping is more consistent across scales and differences in small distortion range are smaller.

Discriminability analysis

In [START_REF] Nehmé | Comparison of subjective methods for quality assessment of 3D graphics in virtual reality[END_REF][START_REF] Fela | Perceptual evaluation on audio-visual dataset of 360 content[END_REF][START_REF] Pastor | Towards guidelines for subjective haptic quality assessment: A case study on quality assessment of compressed haptic signals[END_REF][START_REF] Pastor | Comparison of conditions for omnidirectional video with spatial audio in terms of subjective quality and impacts on objective metrics resolving power[END_REF], the authors suggested examining the MOS discriminability evolution with increasing numbers of assessors to show how well a subjective methodology can recover accurate MOS scores and compare subjective methodologies efficiency [START_REF] Pastor | Discriminability-Experimental Cost" tradeoff in subjective video quality assessment of codec: DCR with EVP rating scale versus ACR-HR[END_REF]. A two-sample Wilcoxon test is applied to all the possible pairs of MOS in a dataset to test the proportion of significantly different ones. We plot the evolution of this ratio in function of the assessors' number.

Figure 5 presents the results on the four datasets with 95% confidence intervals over 100 simulations. Quadruplet methodology is the most discriminative one. 30 equivalent observers in the quadruplet or pair-based scenario achieve the discriminability of the 98 observers in the DCR experiment.

After 40 DCR-equivalent observers (i.e., 2000 annotations), we can comment that for the three ranking-based datasets, continuing to annotate the contents provides a marginal gain in discriminability and thus in data quality. A plateau appears, and adding more time or resources is not necessarily worth it.

Objective Metrics Resolving power

ITU-T Rec. J.149 [START_REF]Method for specifying accuracy and cross-calibration of Video Quality Metrics (VQM)[END_REF], ITU-R Rec. BT.1676 [START_REF] Bt | Methodological framework for specifying accuracy and cross-calibration of video quality metrics[END_REF] and in [START_REF] Krasula | On the accuracy of objective image and video quality models: New methodology for performance evaluation[END_REF] defines how to measure a metric resolving power. Resolving power represents the lowest quality difference a metric can measure that has a statistical difference from the subjective ratings point of view. It characterizes the meaningfulness of quality differences measured by prediction models. However, resolving power depends on metric accuracy and the dataset quality used for evaluating the metric performance. This section will show how much performance the different models can express depending on subjective data quality. Here, we consider five video quality metrics: VMAF [START_REF]Vmaf v0.6.1 model[END_REF], DLM [START_REF] Li | Image quality assessment by separately evaluating detail losses and additive impairments[END_REF], VIF [START_REF] Sheikh | Image information and visual quality[END_REF], MS-SSIM [START_REF] Wang | Image quality assessment: from error visibility to structural similarity[END_REF], and PSNR. Table 3 presents the performances of these metrics on the quadruplet-based dataset. We chose to analyze this dataset since it achieves the highest discriminability. This analysis could be replicated for the other datasets as well.

Results are presented in Figure 6. Increasing the discriminability, hence the quality of subjective data, improves the resolving power of quality metrics and shows the importance of having reliable PD-MOS. Moreover, linking with results presented in table 3, VMAF performs the best across the different metrics. Its low resolving power also suggests it. Lower resolving power is best. Poorly performing metrics like PSNR have accordingly high resolving power, which does not change much with higher discriminative subjective data.

Conclusion

In conclusion, our study compares multiple subjective methodologies for assessing local perception of distortion in videos and their impact on objective metrics' resolving power. We have shown that ranking-based methodologies, such as quadruplets, triplets, and pairwise comparisons, offer higher discriminability and efficiency than directly rating impairment, like with the DCR method. Quadruplet-based assessments demonstrated the highest discriminability among the tested methodologies, allowing for accurate estimation of perceptual differences with fewer observers than other methodologies.

Furthermore, we illustrated how the quality of subjective data, particularly discriminability among PD-MOS, influences the resolving power of objective quality metrics. Higher discriminability in subjective data leads to improved resolving power, enabling more accurate measurement of quality differences by prediction models. Our analysis highlights the importance of investing in high-quality subjective data collection to leverage objective quality metrics' capabilities fully.

Moving forward, our work lays the groundwork for future research to refine subjective testing methodologies and improve the evaluation of objective quality metrics by better understanding the interplay between subjective data quality, experimental costs, and metric performance. By understanding the relationship between subjective data quality, experimental cost, and resolving power, researchers can make informed decisions to optimize resource allocation and improve the reliability of video quality assessment.

Figure 1 .

 1 Figure 1. Example of a Perceptual Unit (PU) aligned on motion. The PU is a 64x64 block of pixels and spans over 12 consecutive frames (400ms). The PU is the spatio-temporal granularity of the Human Visual System (HVS) we want to understand and model to predict the perception of distortions and integrate it into video encoders.

Figure 2 .

 2 Figure 2. Displayed are examples of stimuli evaluated by observers in the four subjective tests. Under each example, in white is the information about where

Figure 3 .

 3 Figure 3. Example of quadruplet-based inter-content comparison. Each pair in the quadruplet contains the reference tube and a distorted version of this tube. Stimuli are displayed on a neutral gray background. on each of the playlists. From TC i = {TC re f , TC 1 dist , ..., TC 5 dist }, triplets are {(TC re f , TC 1 dist , TC 2 dist ), (TC re f , TC 1 dist , TC 3 dist ), ..., (TC 3 dist , TC 4 dist , TC 5 dist )} Pairwise-based intra-content dataset This dataset uses pairwise comparisons with 6 stimuli; 15 pairs must be annotated. From TC i = {TC re f , TC 1 dist , ..., TC 5 dist }, pairs are {(TC re f ,C 1 dist ), (TC re f ,C 2 dist ), ..., (C 4 dist ,C 5 dist )} We decided to divide the 120 pairs into three playlists. We collected 1800 annotations with 15 participants for each playlist.

Table 1 :

 1 5-grade DSIS scale used in DCR experiment. the five-point impairment scale (1: Very annoying, 2: Annoying, 3: Slightly annoying, 4: Perceptible but not annoying, 5:

Figure 4 .

 4 Figure 4. Comparison of rating scale usage by observers across the four subjective quality assessment methodologies. PD-MOS stands for Perceptual Difference Mean Opinion Score and represents the visibility of distortions perceived by observers. A PD-MOS of 0 corresponds to an imperceptible difference with the reference tube, and increasing PD-MOS represents higher and higher visible distortions.

Figure 5 .

 5 Figure 5. MOS discriminability evolution for the four datasets, as a function of DCR-equivalent observers' number. Higher discriminability is better.

Figure 6 .

 6 Figure 6. Objective video quality metrics Resolving Power [25] on the quadruplet-based dataset.

Table 2 : Summary of datasets collected over the 8 tube- contents with 6 stimuli each and "#" meaning "number of".

 2

	Subj. test name	# Annotations	DCR-equivalent Observers
	DCR patches	4900	98
	Quadruplet patches	4000	80
	Triplet patches	3960	79.2
	Pair patches	4000

Table 3 : Performances of Full-Reference quality metrics on the quadruplet-based dataset.

 3

		PLCC	KRCC	SRCC	RMSE	MAE
	VMAF [28]	0.824	0.729	0.896	0.329	0.222
	DLM [31]	0.795	0.684	0.861	0.337	0.223
	VIF [32]	0.671	0.669	0.846	0.406	0.302
	SSIM [33]	0.742	0.669	0.840	0.384	0.258
	PSNR	0.550	0.641	0.803	0.474	0.347

https://github.com/andreaspastor/MLDS inter content scaling

AV1 encoder v3.1.2, from AOM Alliance Open Media: https://aomedia.googlesource.com/aom/