Yejing Xie

Harold Mouchère

Foteini Simistira Liwicki

Sumit Rakesh

Rajkumar Saini

Masaki Nakagawa

Cuong Tuan Nguyen

Thanh-Nghia Truong

ICDAR 2023 CROHME: Competition on Recognition of Handwritten Mathematical Expressions

Keywords: mathematical expression recognition, handwriting recognition, bimodal, evaluation, dataset

This paper overviews the 7th edition of the Competition on Recognition of Handwritten Mathematical Expressions. ICDAR 2023 CROHME proposes three tasks with three different modalities: on-line, off-line and bimodal. 3905 new handwritten equations have been collected to propose new training, validation and test sets for the two modalities. The complete training set includes previous CROHME training set extented with complementary off-line (from OffRaSHME competition) and on-line samples (generated).

The evaluation is conducted using the same protocol as the previous CROHME, allowing a fair comparison with previous results. This competition allows for the first time the comparison of the on-line and off-line systems on the same test set. Six participating teams have been evaluated. Finally the same team won all 3 tasks with more than 80% of expression recognition rate.

Introduction

Handwritten mathematical recognition is an important and challenging task with numerous real-world applications. The CROHME (Competition on Recognition of On-line Handwritten Mathematical Expressions) competition has been organized since 2011 to foster research and development in this field. The competition provides a platform for researchers to compare and evaluate their methods for recognizing on-line and off-line handwritten mathematical expressions. CROHME has contributed to significant advances in this area by promoting the development of new approaches and datasets, as well as facilitating the exchange of ideas among researchers.

At the very beginning CROHME (2011 [START_REF] Mouchère | Crohme2011: Competition on recognition of online handwritten mathematical expressions[END_REF], 2012 [START_REF] Mouchère | Icfhr 2012competition on recognition of on-line mathematical expressions (crohme 2012)[END_REF], 2013 [START_REF] Mouchère | Icdar 2013 crohme: Third international competition on recognition of online handwritten mathematical expressions[END_REF], 2014 [START_REF] Mouchère | Icfhr 2014 -competition on recognition of on-line mathematical expressions (crohme 2014)[END_REF], 2016 [START_REF] Mouchère | Icfhr2016 competition on recognition of handwritten mathematical expressions (crohme2016)[END_REF]) focused only on on-line content. Recently, CROHME 2019 [START_REF] Mahdavi | ICDAR 2019 CROHME + TFD: Competition on Recognition of Handwritten Mathematical Expressions and Typeset Formula Detection[END_REF] has considered also off-line content using the rendering of the on-line signal to produce a perfect image of the handwritten expressions. This was a first step towards the offline domain. After that, other competitions and dataset as OffSRAHME [START_REF] Wang | Icfhr 2020 competition on offline recognition and spotting of handwritten mathematical expressions-offrashme[END_REF] appeared focusing on the off-line domain with more realistic content. This time the competition includes bimodal expressions that combine both on-line and off-line data.

In this paper, we provide an overview of the ICDAR 2023 CROHME competition, describing the three tasks, the corresponding datasets and the evaluation metrics. We also provide the results of the participating systems with a short description. We conclude by discussing the main trends and challenges in handwritten mathematical recognition.

Tasks

Task 1. On-line Handwritten Formula Recognition. As the main task in the previous CROHME, participants utilise InkML format data, which has series of handwritten strokes collected by a tablet or similar device (as in Figure 1), and should convert it to Symbol Label Graph (SymLG), which is a specific Symbol Layout Tree since CROHME 2019. Note that the participants were advised to use strictly only the on-line information and not the off-line information. Task 2. Off-line Handwritten Formula Recognition. Real scanned images (as in Figure 3) or rendered images from InkML (as in Figure 2) are used and should be converted to SymLG. In the same manner as for Task 1, the participants were advised to use strictly only the off-line information and not the on-line information. Task 3. Bi-modal Handwritten Formula Recognition. Both the on-line strokes and off-line images which come from the same acquisition were used for the bi-modal system. While similar with the 2 tasks mentioned above, the outputs of bi-modal system are also SymLG. Note that the participants in this task could use both the on-line and off-line information.

All three tasks involved in the above system are ranked according to the rate of completely correct recognition, but each task is ranked separately.

Datasets and formula encodings

In this section, we describe the data sources, data collection, data format and data encoding that we used in this competition. As indicated in Table 1, CROHME 2023 collects new bimodal data both from off-line real images and on-line strokes, as well as merges data from the previous CROHME. On this basis, it is supplemented by the off-line dataset OffRaSHME and large scale of artificial on-line data.

Handwritten Formulas Data Format.

In general, CROHME 2023 provides 2 types of input handwritten formula data, which are InkML for on-line and PNG format image for off-line, and the bi-modal is combining these 2 formats of one same handwritten formula. All 3 tasks used Symbol Label Graph (SymLG) as the ground truth for system training and also as the final output of the system for further performance evaluation.

InkML: Strokes are defined by lists of (x, y) coordinates, representing sampled points which are collected by on-line input equipment. Figure 1 gives a example of on-line version of a expression, each point is visible and connected to the next point to see the strokes. The specific strokes compose the symbols, and these symbols compose the handwritten formulas. Each InkML (except test set), in addition to giving information on the position of the strokes, also provides stroke-level annotation information to indicate the relationship between strokes with the assistance of MathML (XML-based representation) in presentation mode. Thus the symbol segmentations and labels are available in this type of data. Similarly, the Label Graph (LG), which is a CSV-based representation for Stroke-level Label Graph used in previous CROHME is also provided Image: Due to the different sources of data, there are two different images types. Indeed, in first competitions several types of acquisition devices have been used: Annoto pens (which produce on-line and off-line signal), tablet-pc with pen based sensitive screen, and numeric white board. So a number of early CROHME data, which are only available as on-line data, lack real off-line images, so these were rendered automatically from the on-line data with size 1000 × 1000 pixels and 5 pixels of edge padding. At the same time, as OffSRaHME [START_REF] Wang | Icfhr 2020 competition on offline recognition and spotting of handwritten mathematical expressions-offrashme[END_REF], we scanned all the formulas that existed in real writing on paper; these real off-line images are not gray-scale and have no fixed size due to the different sizes of handwritten formulas scale.

Symbol Label Graph: With the development of end-to-end handwritten mathematical recognition system, the stroke level segmentation of each symbol is no longer produced. Therefore, CROHME 2023 does not demand stroke level Label Graph (LG) files as the final output of the system, only Symbol Label Graph (SymLG) is required as formula structure representation for evaluation. SymLG and the related metrics are detailled in section 4.

Formula Data Selection and Collection.

Firstly a corpus of expressions is built then these expressions are written by volunteers.

Formula Data Selection. The selection of the mathematical expressions is an important step because it selects the language domain of the competition. To build this list of expressions, we have followed the same process as in the past contests, as described in [START_REF] Mouchère | Advancing the state of the art for handwritten math recognition: the CROHME competitions, 2011-2014[END_REF]. Using a public set of L A T E X sources of scientific papers [START_REF] Gehrke | Overview of the 2003 KDD cup[END_REF] all math expressions have been extracted and filtered using the grammar IV defined in previous context. Then the expressions are selected in this list to keep the term frequency and expressions length frequency comparable to the training set. 4000 expressions have been pre-selected.

Handwritten Formula Collection. The 2023 new data collection was done over 150 of participants in three sites: France, Japan and Sweden with the Wacom Intuio device, which is allowed to record the pen trajectory while writing on a standard paper. Each A4 size paper has 5 or 8 formulas written on it. Most of the formulas need to be written in rectangular boxes in order to easily separate the formulas on the same sheet of paper. We also tried to let the participants write down a series of formulas freely, without any limitation of boxes.

After the collection, each expression have been individually checked. A part of them has been discarded because of issues in the acquisition (miss matching of on-line and off-line), connecting symbols (the same stroke shared by 2 symbols), or scratching. . . leading to a new set of 3900 handwritten expressions.

Generated artificial HME

Two pattern generation sets are provided for Task 1. The first set consists of syntactic patterns generated by parsing the structure of handwritten mathematical expressions (HMEs) to identify the syntactic role of each component within an HME [START_REF] Truong | Syntactic data generation for handwritten mathematical expression recognition[END_REF]. After that, each HME is decomposed to identify all valid sub-HMEs based on their syntactic role. These sub-HMEs are considered as new meaningful HMEs. Moreover, we also interchange sub-HMEs that have the same syntactic role inside each HME to get new HME patterns. The generated syntactic patterns differ from the original patterns, and we have provided approximately 76,224 syntactic patterns in CROHME 2023.

The second set consists of synthetic patterns generated from L A T E X sequences. We create an XML layout from each L A T E X sequence and update the layout with the handwritten symbols extracted from CROHME 2019 training set. We generated approximately 68,884 HME patterns from L A T E X sequences of the CROHME 2019 Wikipedia corpus and collected L A T E X sequences.

The table 1 describes clearly the usage of the different data sets in the competition.

Evaluation metrics

Symbol Layout Graph: SymLG

As CROHME 2023 concentrates on formula recognition for a variety of input formats, the evaluation metric of formula recognition is always the same symbollevel evaluation for different tasks. The unified SymLG representation forgets which strokes belong to which symbol and allows all systems, whether they produce stroke-level or symbol-level results, to have an identical standard that can be compared directly. The Figure 4(a) gives an example of a SymLG for an expression containing 4 symbols. This graph can be generated from the L A T E X string, from the mathML representation (available in InkML files) or from Stroke level Label Graphs.

Metrics

As previous CROHME, initially described in [START_REF] Mouchère | Advancing the state of the art for handwritten math recognition: the CROHME competitions, 2011-2014[END_REF], updated in [START_REF] Mahdavi | ICDAR 2019 CROHME + TFD: Competition on Recognition of Handwritten Mathematical Expressions and Typeset Formula Detection[END_REF], we consider expression and structure recognition rates as the most important metrics.

For each system, we generated a table giving the percentage of expressions with matching MathML trees, and with at most 3 incorrect symbols or relations. And the completely correct recognition rates were used to rank systems for all Tasks. It should be noticed that the evaluation metric applied on SymLG looses a part of its advantages.

Indeed, originaly designed for stroke level comparison, the tool lgeval first aligned the node identifiers. For one expression, stroke identifiers are unique and the same whatever the recognized output. In Symbol level LG, the node identifiers are based on the position of the symbol in the relation tree, starting from the origin. This identification leads to add new errors during the evaluation in the case of relation errors.

For example, in Figure 4(c) the symbols x, + and 7 are correctly recognized, but because of the relation error at the beginning of the expression, all following nodes are renamed and miss-matching with the ground-truth. So, in case of perfect recognition, every nodes are matching and no errors will be produced. In case of only symbol recognition errors (without structure errors), the matching is pertinent and the provided errors are meaning full. In case of a relation recognition error, all the symbols from the concerned sub-expression will be in error.

Participating methods

As clearly shown in Table 4, there are in total 6 teams joined in 3 tasks. Sunia, YP OCR and TUAT participated all the 3 tasks, while DPRL RIT only participated in task 1, PERO and UIT@AIClub Tensor only participated in task 2. We need to mention that, probably because of our technique problem, the system description of team UIT@AIClub Tensor is missed, thus the descriptions of the 5 team are shown as follow.

Sunia PTE.LTD 5 A standard encoder-decoder models are used to translate a bitmap image or a sequence of offsets and pen-up flags to a sequence of tokens in a compact language where each mathematical expression has a unique representation. The encoder consists of stacked CNN/BLSTM layers, and the decoder relies on an attention mechanism that can model convergence and estimate the location of each symbol. For on-line recognition, preprocessing steps such as reordering (for some of the combined models which were trained on spatially sorted strokes and thus stroke-order free), resampling, and normalization are performed to ensure that our system is not too sensitive to stroke order, device, location, and size. Model ensembling and LL(1) grammar parsing are also employed during beam search to boost the accuracy and avoid illegal output. We have augmented the training set by assembling synthetic samples from the subexpressions in the official training set and applying local and global distortions. For data usage, based on on-line official data, more synthetic samples were generated with stroke-level annotations by official labels of these artificial samples. And for off-line and bimodal task, rendered version of on-line samples are also used in additional to the official train set.

YP OCR, CVTE Research: The system utilized an attention-based encoderdecoder structure, where the encoder adopted the DenseNet architecture. In the decoding stage, both a bidirectional tree decoder and a regular decoder are applied, and two different models are obtained. For Task 1, the InkML files were rendered uniformly before feeding them into the models with different decoders. Additionally, a multi-stage curriculum learning training strategy is employed to train the models. For Task 2, the models is trained by only using the official training dataset and alleviated the problem of limited training data by using data augmentation methods. Finally, the trained models are fused in different stages to obtain better results and integrated the results of both decoders through beam search during the inference stage.

PERO System: The system consists of an optical model (OM) and a language model (LM). The OM is based on a CRNN architecture trained using the CTC loss function. It operates on images with a normalized height of 128 pixels and arbitrary length. The LM is an LSTM network trained to predict the next LaTeX token given its predecessors. To train the OM, the provided off-line dataset and also the provided synthetic data are used. As training data for the LM, the ground truth provided in the datasets and also publicly available equations from Wikipedia1 are used. The OM is trained to produce a LaTeX code. The LM decodes logits generated by the OM using prefix search decoding and the hypothesis representing a valid LaTeX code with the highest probability is taken as a result. Finally, LaTeX codes are converted into SymLG using the provided tools.

Team DPRL RIT QD-GGA [START_REF] Mahdavi | Visual parsing with query-driven global graph attention (qd-gga): preliminary results for handwritten math formula recognition[END_REF] system is extended, a visual parser that (1) creates a line-of-sight (LOS) graph over strokes, (2) scores stroke segmentation, symbol class, and relationship hypotheses with a multi-task CNN, (3) segments and classifies symbols, and (4) selects relationships using a maximum spanning tree. We have improved visual features and speed. Input images are now 64x64 for all stroke and formula window images, with formula windows centered on the target/parent stroke. The 2 nearest neighbors [START_REF] Veličković | Graph attention networks[END_REF] of strokes define context windows. A modified graph attention network is used with cropped strokes and their two cropped nearest neighbors as additional context features. Spatial pyramidal pooling (SPP) [START_REF] He | Spatial pyramid pooling in deep convolutional networks for visual recognition[END_REF] was added to avoid spatial information loss. These features have improved symbol segmentation and classification rates, while relationships require more work. Memory optimizations provide inference times of 25.9 ms/formula on a desktop system with two Nvidia GeForce GTX 1080 Ti GPUs (12GB each).

Team TUAT Two models and their combination are provided for on-line, off-line, and bi-modal recognition tasks. For the on-line recognition model, a deep BLSTM network is used for jointly classifying symbols and relations. Then, a 2D-CFG is used to parse the symbols into mathematical structure [START_REF] Nguyen | Global context for improving recognition of online handwritten mathematical expressions[END_REF]. For the off-line recognition model, an end-to-end deep neural network that is trained using weakly supervised learning is used, and a symbol classifier is added to the encoder-decoder model to improve the localization and classification of the CNN features. More details of this method can be found in the paper [START_REF] Truong | Improvement of endto-end offline handwritten mathematical expression recognition by weakly supervised learning[END_REF]. For the combination of on-line and off-line models, firstly several candidates with their probability scores during the Beam Search process are generated for each online and off-line recognition model. Then, their prediction scores are combined using a trained weight. The weight is chosen by the best combination recognition rate on the validation set. For the data, the systems are trained using the given dataset as provided by the competition.

Results

In this section, we summarize all of the results in CROHME 2023. Table 2 and3 present respectively results on the new test set 2023 and on the previous test set 2019. There are in total 6 teams participating in 3 tasks, 3 of them participate in all of these 3 tasks, while the other 3 teams only selected one of on-line or off-line task. Sunia team obtained all the highest recognition rate in on-line, off-line and bi-modal tasks, and is ahead of YP OCR who also took part in all of 3 tasks.

Compared to the Test set 2019, Test set 2023 had lower recognition rates for the same participants. AS this is a global observation, we can conclude that the new test set is harder than the previsous one. However, Sunia, which won CROHME 2023, had a better recognition rate than the systems that participated in CROHME 2019 on the same data. It is a significant indication of the Since the 3 tasks in this competition use exactly the same evaluation metrics, it is interesting to conduct a comparison between different tasks. In general, bi-modal systems have the best performance, which has a more enriched data dimension therefore fits the common sense. In most cases, on-line systems have better capabilities than off-line systems, except on-line and off-line systems from TUAT. This illustrates that on-line data with the combination of spatial and temporal information is more conducive than off-line data which only consider the spatial information. However, we do not exclude that generated artificial online HME greatly increases the sample size of the on-line dataset, and improving the performance of the deep learning system. Therefore, it is also necessary to further explore the usage of artificial data for off-line datasets, which can greatly increase the data volume of off-line datasets and save manual annotation costs.

A summary of confusion histograms that tabulate errors in symbols and symbol pairs as defined in [START_REF] Mouchère | Advancing the state of the art for handwritten math recognition: the CROHME competitions, 2011-2014[END_REF] is presented in Table 4. It is important to note that the most common errors are confusion with absent symbols which are not shown in the table. Indeed, the confusion with absent symbols is always caused by the errors in structure, different structures can not be matched symbol by symbol, that raises errors in the position and classification of the symbols, as explained in section 4.2. The statistics in the table show that symbols 1, 2, and -are the most difficult to recognize, which have commonalities across systems, maybe because these symbols are the most frequent. For symbol pairs, most of systems are hard to identify -1, 1 , and 2 , there is also specificity in the different systems.

Table 4: Most frequent symbol and sub-structure recognition errors (Test set 2023). E columns give the GT symbol or sub-structure which have been misrecognized. # columns give the number of occurrences of this errors.

Symbols

Symbol Pairs

1 st 2 nd 3 rd 1 st 2 nd 3 rd E # E # E # E # E # E # Task1: On-line Sunia 1

Conclusion

CROHME 2023 provides 1,045 new expressions in train set, 555 in validation set and 2,300 in test set of handwritten formulas in bi-modal with manual annotations, as well as large-scale of artificial data. Compared to the CROHME 2019 results, the CROHME 2023 winner had a superior performance, especially in on-line task. But the winner's improvements are not as impressive as those in 2019. CROHME 2023 participants all adopted the deep learning approaches, training models with large amounts of data. However, several system continue to use structural constraints (as LL [START_REF] Gehrke | Overview of the 2003 KDD cup[END_REF] or CFG grammars). Furthermore, statistical language models are now integrated by some participating systems. It is worth noting that, the winner team Sunia, used data augmentation strategy to increase the amount of provided official data. In addition, most on-line systems have better performance than off-line systems, there is still a significant improvement potential for Handwritten Mathematical Expression Recognition systems, especially off-line.

On the basis of this topic, we have more in-depth ongoing discussions scheduled for the future. Nowadays, deep learning models, especially those with encoder-decoder structure, have become the dominant solution for Handwritten Mathematical Expression Recognition. These systems are sensitive to samples quantity and quality of data sets, therefore, it will be interesting to extend the amount of existing data set by exploring more cost-effective annotation methods as well as gaining more data with the help of synthetic algorithms. Exploring nonsupervised training strategy would also be benefit. In addition, more benchmarking experiments are possible to explore the highlights and weaknesses of each system, in order to provide more guidance for them.

Fig. 1 :

 1 Fig. 1: On-line handwritten formula. Strokes are sequence of points (in Inkml file format).

Fig. 2 :

 2 Fig. 2: Off-line handwritten formula rendered from Inkml (PNG file format).

Fig. 3 :

 3 Fig. 3: Off-line handwritten formula scanned from real paper (400 dpi, PNG file format).

7 Fig. 4 :

 74 Fig.4:A visualisation example for formula recognition evaluation with SymLG. Compared with the Ground Truth (a) '2 x + 7', (b)'2 x + 1' and (c)'2x + 7' are 2 possible mistakes of system recognition. (b) has a symbol recognition error, which means that '1' and '7' are mistaken, while (c) has two structure recognition errors, which means that the relationship between '2' and 'x', '2' and '+' are misunderstood, which leads to new errors of mismatching nodes oRR and oRRR.

Table 1 :

 1 CROHME 2023 Data Sets. On-line data are only inkml files, off-line are images, bimodal are inkml and image for each equation.

	Tasks Train	#	Validation	#	Test	#
	Task 1
	On-line Train 2019	9 993	Test 2016	1147
	Validation 2019	986
	artificial	145 108
	new samples	1045	new samples	555	new samples 2 300
	Total= 157 132	Total= 1 702	Total= 2 300
	Task 2
	Off-line rendered	9975	rendered	1147
	real	1604
	OffRaSHME	10 000
	new samples	1045	new samples	555	new samples 2 300
	Total= 20 979	Total= 1 702	Total= 2 300
	Task 3:
	Bimodal InkML + rendered 9 975	InkML + rendered 1147
	InkML + real	1 604
	new samples	1045	new samples	555	new samples 2 300
	Total= 10 979	Total= 1 702	Total= 2 300

Table 2 :

 2 Formula Recognition Results (Test set 2023)

		Structure + Symbol Labels	Structure
		Correct	1 ≤ s.err	2 ≤ s.err	Correct rate
	Task 1: On-line
	Sunia*	82.34	90.26	92.47	92.41
	YP OCR	72.55	83.57	86.22	86.60
	TUAT	41.10	54.52	60.04	56.85
	DPRL RIT	38.19	53.39	58.39	59.98
	Task 2: Off-line
	Sunia*	70.81	81.74	86.13	86.95
	YP OCR	67.86	80.86	85.13	85.99
	PERO	58.37	71.22	75.57	75.55
	TUAT	51.02	63.17	67.48	64.59
	UIT@AIClub Tensor	14.83	22.87	28.57	32.19
	Task 3: Bi-modal
	Sunia*	84.12	91.43	93.70	94.13
	YP OCR	72.55	83.57	86.22	86.60
	TUAT	53.76	68.83	74.22	70.81

Table 3 :

 3 Formula Recognition Results (Testset 2019)

		Structure + Symbol Labels	Structure
		Correct	1 ≤ s.err	2 ≤ s.err	Correct rate
	Task 1: On-line
	Sunia*	88.24	93.08	93.99	94.25
	YP OCR	84.74	90.99	92.33	92.66
	TUAT	56.88	71.89	76.31	70.73
	DPRL RIT	40.70	59.47	67.06	70.23
	2019 winner	80.73	88.99	90.74	91.49
	Task 2: Off-line
	Sunia*	77.73	87.41	90.08	90.58
	YP OCR	73.39	85.74	88.49	89.16
	PERO	67.39	77.23	81.07	78.98
	TUAT	50.13	63.55	66.89	63.30
	UIT@AIClub Tensor	38.28	52.29	58.80	59.13
	2019 winner	77.15	86.82	88.99	89.49
	Task 3: Bi-modal
	Sunia*	86.91	92.16	93.58	93.74
	YP OCR	84.74	90.99	92.33	92.66
	TUAT	57.88	73.81	82.82	72.14
	continuous developments in Handwritten Mathematical Expression Recognition
	over recent years.

https://patentscope.wipo.int/search/zh/detail.jsf?docId=DE375155214

Acknowledgments

The organization team of CROHME 2023 would like to thank all the writers who participated in the input of the thousands of equations, colleagues who helped with the annotation, the participants for their constructive feedback. We particularly Da-Han Wang and the OffRASHME team for sharing their data.


Ali Ak

Anne Flore Perrin

Denise Noyes

Ioannis Katsavounidis

Video Consumption in Context: Influence of Data Plan Consumption on QoE

Keywords: CCS Concepts:, Human-centered computing, Acceptability and Annoyance, Quality of Experience, Eliminated-By-Aspects"

User expectations are one of the main factors on providing satisfactory QoE for streaming service providers. Measuring acceptability and annoyance of video content, therefore, provide a valuable insight when measured under a given context. In this ongoing work, we measure video QoE in terms of acceptability and annoyance for the remaining data in a mobile data plan context.. We show that simple logos can be used during the experiment to prompt the context to subjects and the different context levels may impact the user expectations and consequently their satisfactions. Finally, we show that objective metrics can be used to determine the acceptability and annoyance thresholds for a given context.

INTRODUCTION

Quality of Experience (QoE) in video streaming defines the observer's level of satisfaction and how well their expectations are met while viewing the video content. Several factors may impact the QoE and they are categorized as system, context, and human in the Qualinet white paper [START_REF] Le | Qualinet white paper on definitions of quality of experience. European network on quality of experience in multimedia systems and services[END_REF]. Along with the essential video quality, other vital considerations in measuring QoE include but are not limited to fidelity, cost, ecological impact, and display device specifications.

Recent advancements provide a plethora of metrics and methodologies to assess QoE for video content. However, from the point of view of the streaming service provider [START_REF] Zeithaml | Delivering quality service : balancing customer perceptions and expectations[END_REF], knowing the exact quality of the video content is not always priority and might not be enough to understand whether the delivered content satisfies the user expectations. In this regard, Acceptability and Annoyance (AccAnn) scale has been introduced and frequently used in recent years [START_REF] Angela | Quality in Context-an ecological approach to assessing QoS for mobile TV[END_REF][START_REF] Knoche | The Big Picture on Small Screens Delivering Acceptable Video Quality in Mobile TV[END_REF][START_REF] Li | AccAnn: A New Subjective Assessment Methodology for Measuring Acceptability and Annoyance of Quality of Experience[END_REF][START_REF] Song | Acceptability-Based QoE Models for Mobile Video[END_REF]. AccAnn scale often contains three categories as "not acceptable (1)", "acceptable but annoying (2)", and "not annoying (3)". Mapping of quality measurements from Absolute Category Rating (ACR) or Degradation Category Rating (DCR) scale to AccAnn scale has also been investigated [START_REF] De Pessemier | Quantifying Subjective Quality Evaluations for Mobile Video Watching in a Semi-Living Lab Context[END_REF][START_REF] Li | AccAnn: A New Subjective Assessment Methodology for Measuring Acceptability and Annoyance of Quality of Experience[END_REF][START_REF] Oeldorf-Hirsch | How Bad is Good Enough? Exploring Mobile Video Quality Trade-offs for Bandwidth-Constrained Consumers[END_REF].

AccAnn ratings are classically collected via a multi-step evaluation procedure. After the stimulus is presented to subjects, a first evaluation screen is prompted, asking whether the stimulus is acceptable or not. If the subject answers no, the stimulus is rated with lowest rating as "not acceptable" and if the subject answers yes a second question is Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). prompted asking whether the stimulus is annoying or not. If subject answers the second question as yes, the stimulus is rated as "acceptable but annoying" and if the answer is no the stimulus is rated with highest quality as "not annoying".

Li et al. proposed a a single-step procedure [START_REF] Li | AccAnn: A New Subjective Assessment Methodology for Measuring Acceptability and Annoyance of Quality of Experience[END_REF] where the three ratings are presented at once with color coding to guide the subjects. Among others, context plays an important role in satisfaction of the user expectations [START_REF] Jumisko-Pyykkö | Does Context Matter in Quality Evaluation of Mobile Television?[END_REF]. Depending on the use-case, context can contain various factors such as subscription level of a streaming service (premium plan vs basic plan), signal strength of the display device (4g vs 3g), remaining power of the device (low vs full battery), etc. Previous studies take context into account for acceptability and annoyance by either collecting pre/post-experiment surveys [START_REF] Jumisko-Pyykkö | Does Context Matter in Quality Evaluation of Mobile Television?[END_REF] or setting the context for subjects with set of instructions prior to the experiment [START_REF] Li | AccAnn: A New Subjective Assessment Methodology for Measuring Acceptability and Annoyance of Quality of Experience[END_REF].

Many mobile phone users subscribe to monthly data plans that allow them a certain volume to be used over a month, for example, 10GBytes, while others buy data that can be used over 24 hours, or 1-week. As such, users are aware of the concept of "how much data I have left in my quota" and tent to adjust their usage so as to maximize that, without exceeding it. In this ongoing work, we focus on remaining data plan context and its influence on the user expectations and acceptance and annoyance of the video content.

SUBJECTIVE EXPERIMENTS

Several subjective study were conducted as part of this work. Initially, an ACR experiment was conducted to collect Mean Opinion Scores (MOS) and afterwards three AccAnn experiments with the same context and three context levels were conducted. Details for each experiment are given below.

Content

3 source video (SRC) with 1080p resolution and horizontal orientation were used in the experiment. Each source video is 5 seconds long with 30 fps. To generate the processed video sequences (PVS), each SRC was compressed with VP9 coding algorithm at 6 different levels. Generated PVS with SRC were used in all experiments.

ACR Experiment

In order to collect the mean opinion scores (refered as ACR-MOS for the rest of the paper), an absolute category rating with hidden reference (ACR-HR) experiment was conducted on a 5 category quality scale. The experiment was conducted in Nantes University IPI laboratories with 25 subjects from the in-house participant panel. All subjects were checked for visual acuity and compensated for their participation. The experiment room was set according to ITU recommendations [START_REF] Itu-R | Methodology for the Subjective Assessment of the Quality of Television Pictures[END_REF].

ACR experiment was conducted without any context provided to observers in order to collect traditional MOS values corresponding to each video. This will allow us to analyze the relation between MOS values and the acceptability

annoyance MOS (AccAnnMOS) collected in the AccAnn experiments.

AccAnn Experiments

We followed the experiment design proposed in [START_REF] Li | AccAnn: A New Subjective Assessment Methodology for Measuring Acceptability and Annoyance of Quality of Experience[END_REF] as color coded acceptability annoyance test with a single-step evaluation procedure. The evaluation screen with three scales is shown in Figure 1-a which is prompted to subjects after each video stimulus. For the analysis, we also use the numerical representations of the AccAnn results as 1, 2, 3 for "Not Acceptable", "Annoying but Acceptable" and "Not Annoying" respectively. Similar to the ACR experiment, the experiment room was arranged according to ITU Recommendations [START_REF] Itu-R | Methodology for the Subjective Assessment of the Quality of Television Pictures[END_REF]. The experiments were conducted in Nantes University IPI laboratories with 60 subjects from the in-house participant panel.

In order to understand the impact of context on the acceptability and annoyance of the video stimuli, AccAnn experiment was conducted in three sessions where in each session subjects were assigned by a profile. The context used in the experiment was remaining data and in each session a different context level was used. Prior and during the experiment, the logos shown in Figure 1-b were used to provide the context to subjects. By collecting AccAnn-MOS with different context levels, we seek to reveal the change in expectations of the subjects.

MAPPING OF ACCANN-MOS TO ACR-MOS

Initially, AccAnn-MOS values to ACR-MOS values were mapped to understand the acceptability and annoyance thresholds. For each context level ("Low", "Medium", "High" data quota levels), we plotted AccAnn-MOS values against the ACR-MOS values in Figure 2. Furthermore, a 4 parameters logarithmic function was fitted. We can observe that the AccAnn-MOS values for the "Low" are particularly higher for the same content in low to mid quality range.

The thresholds where contents start to be unacceptable are 1.86, 2.21, 2.25 in ACR-MOS scale for "Low", "Medium" and "High" context levels, respectively. This indicates that the subjects have a lower expectation in terms of quality for an acceptable content when they have "Low" amount of data quota in their data plans. Meanwhile, we observe no statistically significant difference between "Medium" and "High" context levels.

On another front, the thresholds where contents start to be annoying are calculated as 3.38, 3.56, 3.59 in ACR-MOS scale for "Low", "Medium" and "High" context levels, respectively. It can be seen that the differences between context levels in annoyance thresholds are not as pronounced as unacceptability threshold.

EBA ANALYSIS

In this section, we use the EBA model explained in Section 4.1 to further investigate our initial observations (see Section

Model

Tversky has proposed a set of models to study and analyze pairwise comparison data [START_REF] Tversky | Elimination by Aspects: A Theory of Choice[END_REF] as a generalization to the Bradley-Terry-Luce (BTL) model. According to EBA model, a subject prefers certain stimulus over an alternative due to presence of set of attributes one has over the other. In EBA model, stimuli may contain several attributes and all of them can impact the choice of the subject. On the other hand, BTL, is a specific case of EBA where each stimulus is defined by a unique attribute.

In QoE domain, EBA model can be used to study the effect of parameters that ultimately define the measured quality in a subjective test. By conducting a set of subjective studies with same stimuli and varying context levels (e.g. Low, Medium, High Data Quota), we can define the set of attributes that affects each measurement (AccAnn opinion scores) as the visual quality of each stimulus and the context level. In a previous study [START_REF] Li | Quantifying the Influence of Devices on Quality of Experience for Video Streaming[END_REF], Li et al. studied the influence of subscription levels and display device on the QoE for video streaming services.

Formally, we can define the 𝑖 𝑡ℎ video sequence with its visual quality attribute defined as 𝑢 (𝑞 𝑖 ) and its visual quality as the logarithmic of the attribute, i.e., log(𝑢 (𝑞 𝑖 )). Furthermore, we can define the attributes of each context level as 𝑢 (𝑑 𝐿 ), 𝑢 (𝑑 𝑀 ) and 𝑢 (𝑑 𝐻 ) for "Low", "Medium" and "High" data quota, respectively. Consequently, we can represent the measured QoE (i.e., AccAnnMOS) in each AccAnn experiment as log(𝑢 (𝑞 𝑖 ) + 𝑢 (𝑑 𝑖 )) where 𝑑 𝑖 is either 𝑢 (𝑑 𝐿 ), 𝑢 (𝑑 𝑀 ) or 𝑢 (𝑑 𝐻 ) depending on the context level. Therefore, following the EBA model, the probability that a subject prefers video 𝑖 over video 𝑗 based on the influence of context level and the visual quality of the videos can be defined as: We rely on the Matlab implementation of EBA model proposed in [START_REF] Wickelmaier | A Matlab function to estimate choice model parameters from paired-comparison data[END_REF]. In order to so, AccAnn results need to be represented as a pairwise comparison matrix (PCM) 𝑀 𝑖 𝑗 . 𝑖 and 𝑗 are in the range [START_REF] Ak | From Just Noticeable Differences to Image Quality[END_REF]63], since we have 21 stimuli in the dataset and each observed with three context levels ("Low", "Medium", "High"). When converting AccAnn results into a PCM, we assign 𝑀 𝑖 𝑗 with 1 if the video 𝑖 has higher rating than the video 𝑗, and 0 for the opposite case. In the cases where the AccAnn scores are equal, we randomly sample 0 or 1 from a uniform distribution and assign to 𝑀 𝑖 𝑗 .

Finally, we can calculate the maximum likelihood estimates of the EBA model attributes (𝑢 (𝑞𝑖) and 𝑢 (𝑑 𝑖 )) by the following likelihood function:

𝐿 = 𝑖< 𝑗 𝑝 𝑀 𝑖 𝑗 𝑖 𝑗 (1 -𝑝 𝑖 𝑗 ) 𝑀 𝑗𝑖 (2)
where the 𝑝 𝑖 𝑗 is calculated as in Equation 1and 𝑀 𝑖 𝑗 are the corresponding entries of the PCM.

Results

After solving the EBA model described above, we can obtain the parameters 𝑢 (𝑞𝑖) and 𝑢 (𝑑 𝑖 ). As described, we can acquire the visual quality of each of the 21 video sequences by 𝑙𝑜𝑔(𝑢 𝑞 [ 𝑖 ] ) and its measured QoE with the influence of the context as 𝑙𝑜𝑔(𝑢 𝑞 [ 𝑖 ] + 𝑢 (𝑑 𝑖 )). Then, we can obtain the influence of context levels (defined as 𝑄 𝑑 𝑖 ) for each video (𝑖)

with the following equation:

𝑄 𝑑 𝑖 = 𝑙𝑜𝑔(𝑢 𝑞 𝑖 + 𝑢 (𝑑 𝑖 )) -𝑙𝑜𝑔(𝑢 𝑞 𝑖 ) (3)
Figure 3 presents the results of the EBA analysis. We observe a greater influence to the acceptability of the "Low" context level (i.e., Low remaining data). On the other hand, we don't observe a significant difference between "Medium" and "High" context levels (i.e., Medium and High remaining data). Furthermore, we observe that the impact is much greater in low quality range. These results verify our observations and conclusions derived in Section 3.

ACCEPTABILITY ANNOYANCE THRESHOLDS OF OBJECTIVE QUALITY METRICS

In this section, we use the following abbreviations "NAnn", "UAnn", "AA", "UAcc", "NAcc" for "Not Annoying", "Unsure about Annoyance but sure about the Acceptability", "Annoying but Acceptable", "Unsure about Acceptability but sure about the Annoyance", "Not Acceptable", respectively. The 21 PVSs in each experiment were assigned to one of these categories as proposed in Algorithm 2 in [START_REF] Li | AccAnn: A New Subjective Assessment Methodology for Measuring Acceptability and Annoyance of Quality of Experience[END_REF]. Differently from the proposed approach, we rely on Barnard's exact test instead of Fisher's exact test. "UAnn" category can be seen as the threshold for the videos to start to be annoying whereas "UAcc" is the threshold for the videos to start to be unacceptable.

After assigning each video in each experiment to one of these categories, we can analyze the objective quality metric predictions for different categories. Note that the quality metric predictions don't vary between the experiments since the content are the same and the only difference is the provided context level ("Low", "Medium" and "High" remaining Data). Two metrics (VMAF [START_REF] Li | Toward a practical perceptual video quality metric[END_REF], DJNDQ [START_REF] Ak | From Just Noticeable Differences to Image Quality[END_REF]) were selected for this analysis and the results are presented in Figure 4.

VMAF scores range from 0 to 100 with higher numerical values indicating a higher quality, whereas DJNDQ ranges from 0 to 5 (for this dataset, otherwise no theoretical upper bound) with lower numerical values indicating a higher quality. Note that, none of these metrics are designed to measure Acceptability and Annoyance. Although, by exploring good distinction between the three main categories ("NAnn", "AA", "NAcc"), while showing difficulties in identifying content on the thresholds ("UAnn", "UAcc").

Moreover, we can determine a metric threshold by simply averaging the metric score predictions of the content in the thresholds ("UAnn", "UAcc"). Table 1 presents the result of this analysis. Note that the results may not be generalized due to low number of content available in the experiment. For each context, we can estimate a threshold in metric score range where the content start to become annoying (UAcc) and where content start to become unacceptable (UAnn).

In accordance with the EBA analysis results, we don't see a significant difference in metric score thresholds ("UAnn", "UAcc") for "Medium" and "High" remaining Data scenarios. Again, similar to EBA Analysis results, the subjects' expectations in quality for acceptability and annoyance are lower for metric scores in "Low" remaining Data scenario

CONCLUSION AND PERSPECTIVE

In this ongoing work, we provided an analysis on the influence of context over the Acceptability and Annoyance of video content. We showed that the subjects are capable to incorporate the provided context in their evaluation.

Moreover, we conducted a set of detailed analysis to quantify the influence of context on the subject expectations and consequently the acceptability and annoyance of the video content. We showed that when the QoE of the video content is low, the context has a higher impact on the user expectations and consequently their satisfaction. On another front, we provided preliminary results on the metric thresholds for predicting acceptability and annoyance of a video content.

Although due to low number of samples in the experiments the results might not be generalized reliably, we still believe that these results provide crucial insights for the streaming service providers. We believe that, this work can lead to new avenues to explore for streaming service providers in regard the satisfying user expectations. Streaming service providers can utilize the preliminary findings and the analysis scheme to adjust their video encoding recipes to provide a similar satisfaction with lower bandwidth in certain contexts. Moreover, we believe that the preliminary results may inspire and lead the community to explore different context and their impact on the user satisfaction.

Fig. 1 .

 1 Fig. 1. (a) AccAnn experiment color coded scale as shown in the evaluation screen, and (b) presentation of different levels of the remaining data quota.

3 )

 3 by comparing the AccAnn-MOS values to ACR-MOS values with different context levels. By utilizing EBA model, we can quantify the influence of the context level as a function of measured QoE. Results of this analysis are presented in Section 4.2.

Fig. 2 .

 2 Fig. 2. Mapping of AccAnn-MOS to ACR-MOS for the three AccAnn experiment sessions with different context levels. For each context level, a 4 parameters logarithmic function is fitted. Thresholds where the stimuli start to be annoying (below 2.5 AccAnn-MOS) and start to be unacceptable (below 1.5 AccAnn-MOS) are shown with dashed lines for each context level.

Fig. 3 .

 3 Fig. 3. Plot of influence of context level 𝑄 𝑑𝑖 in AccAnn-MOS range and the measured AccAnn-MOS of each corresponding context level. Horizontal axis represents the AccAnn-MOS and the vertical axis represents the 𝑄 𝑑𝑖 calculated as Equation 3.

Fig. 4 .

 4 Fig. 4. Metric (VMAF and DJNDQ) score predictions for each content in the dataset. Content are ordered based on their categories indicated on the vertical axis of each plot(also color-coded). Due to categorical differences of the stimuli between the experiment, each experiment is plotted seperately.

Table 1 .

 1 Metric thresholds for Acceptability of Annoyance as the average of the predicted scores of all content in corresponding categories estimated in each metric's own scale. metric predictions, we can estimate a numerical threshold in metric score range to determine the Acceptability and Annoyance of a video content for a given context. It can be observed from the figure that metrics provide a relatively

	Low Data	Medium Data	High Data
	UAcc UAnn UAcc UAnn UAcc UAnn
	VMAF 35.88 74.85 27.72 90.51 27.72 89.38
	DJNDQ 2.35	1.03	2.97	0.45	2.97	0.42

the

© 2023 Copyright held by the owner/author(s). Manuscript submitted to ACM