Accordingly, the goal of this thesis is to shed new light on some of the challenges emerged in recent years. In particular, we provide an in-depth analysis of some of the most prominent aspects of modern Internet. A particular emphasis is given on the World Wide Web, which among all, is undoubtedly one of the most popular Internet applications, and a speci c regard to its interaction with machine learning.

The rst part of this work studies the Quality of Experience of users' browsing the Web, with measurements led both in the wild and in controlled environments. Our contributions follow with an original analysis of both the subjective user feedback and the objective QoE metrics, showing how hard it is to build accurate supervised data-driven models capable to predict the user satisfaction, along with an in-depth discussion of the multi-modal nature of the subjective user opinions.

In the second part of this work, we analyze and discuss the fairness of state-of-the-art transformer-based language models, which are pre-trained on Web-based corpora and which are typically used to solve a wide variety of Natural Language Processing (NLP)

tasks. Here, we question whether the sheer size and heterogeneity of the Web guarantee diversity in the models. The core of our contributions rests in the measure of the bias embedded in the models, that we discuss under di erent angles.

Finally, the last part of this dissertation addresses the classi cation of objects generated by machines through some of the simplest state-of-the-art supervised machine learning algorithms. Through a minimally intrusive, robust and lightweight framework, we show that the di erent behaviors of a eld of the IP packet, the IP identi cation (IP-ID), could be easily classi ed with few features having high discriminative power. We At time (c) a popup arise, inviting users to login in the website.

Figure

Scenario in which the active probing is performed: only one sender is used to ease the synchronization of packets generation, whilst both the machines are used to receive and collect the stream of IP-IDs generated at the target machine. . . . . . . La première partie de ce travail étudie la qualité de l'expérience de navigation des utilisateurs sur le Web (Web QoE), avec des mesures e ectuées à la fois "in the wild" et dans des environnements contrôlés. Dans le chapitre , nous abordons le problème de l'évaluation de la qualité de l'expérience sur un site web populaire en fonctionnement.

Plus précisément, nous le faisons en recueillant l'acceptance de Wikipédia par les utilisateurs, soit plus de k de réponses au sondage, ce qui représente plus du double des réponses recueillies dans des études similaires à grande échelle sur Wikipédia. Nos contributions continuent avec une analyse originale de l'avis subjectif des utilisateurs et des mesures objectives de la qualité d'expérience, montrant, d'un côté, une dépendance spatiale entre des caractéristiques collectées, d'un autre côté, la di culté de construire des modèles supervisés précis, basés sur les données disponibles, capables de prédire la satisfaction des utilisateurs. Dans le chapitre , nous nous concentrons plutôt sur la mesure de la qualité de l'expérience dans des expériences contrôlées. En particulier, nous avons mesuré le temps de chargement des pages perçu par les utilisateurs, c'est-à-dire le moment où un utilisateur considère qu'une page web est chargée et prête à être parcourue, sur pages web via la plateforme Eyeorg. Dans ce chapitre, on trouve une discussion approfondie de la nature multimodale des avis subjectifs des utilisateurs.

Dans la deuxième partie de ce travail, nous analysons et discutons l'équité des modèles de langage basés sur des transformateurs de pointe, qui sont pré-entraînés sur des corpus basés sur le Web et qui sont généralement utilisés pour résoudre une grande variété de tâches de traitement du langage naturel (NLP). Dans le chapitre nous nous demandons si la taille et l'hétérogénéité du Web garantissent la diversité des modèles.

Le coeur de nos contributions repose sur la mesure du biais intégré dans les modèles, que nous discutons sous di érents angles. Nous observons que l'équité des grands modèles, At its dawn, the Internet was designed to support very few simple services, which were working over rigidly-speci ed protocols, and mostly text-based (e. g., electronic mail, remote login, etc.). Over the last two decades, along with interconnecting more and more users, it sparked the proliferation of many and diverse applications, signicantly increasing the number and the type of activities that users can carry out online.

Nowadays, the Internet is a complex and big network, with multiple layers of protocols that interact with each other. It is an evolving system which is constantly changing in operations, size, technologies, and economic relationships, all of which evolve at di erent time scales.

Among all, the World Wide Web is undoubtedly one of the most successful Internet application. Indeed, nowadays, the Web usage is no longer restricted to sending and reading emails or nding information through search engines but also for watching videos in streaming (e. g., Twitch, Youtube, etc.), for building social relations (e. g., Twit-ter, LinkedIn, etc.) and for buying goods (e. g., Amazon, AliExpress, etc.). It is then no surprise to nd in the Alexa top-websites rank of [ ] the predominance of social networks (baidu.com, facebook.com, etc.) and shopping platforms (tmall.com, taobao.com, etc.) . Due to its pervasive and heterogeneous nature, the Internet raised the interest of both industry and academia researchers, who face a constantly renewed interest in exploring and understanding the dynamics interacting in this continuously evolving system. Over the years, to cope with the always increasing Internet demand and popularity, novel network infrastructures, devices and tools, as, for instance, lighter browsers in the case of the Web, have been introduced.

Moreover, along with technology evolution and development, also the habits and the expectations of the users have changed signi cantly. Indeed, the pervasiveness of the Internet still raises new challenges: tra c growth, security concerns, users' quality of experience, economic interests, transparency and fairness. Of those, Quality of Experience (QoE) embraces many of the others, so that, in this regard, all the Internet players (Domain Name Services, Internet Service Providers, etc.) have to keep up using cuttingedge technologies in order to provide a satisfactory QoE to the users. This is crucial since if only one of them is a ected by an outage, the entire Internet ecosystem might be a ected as well, compromising the nal experience of the users.

.

. Quality of Experience on the Web

Web browsing is one of the most popular applications for both desktop and mobile users.

Slow rendering of the websites was due in the s to dial-up connections, in the s to slow G connections and it persists nowadays for a wide range of reasons, including the growingly more complex structure of websites [ ] and an increased usage of mobile devices [ , ]. This sometimes caused to the World Wide Web, since its inception, the title of World Wide "Wait" [ ]. Indeed, not surprisingly the rst assessment of Web performance can be traced back to the early s. Since then, a lot of e ort has been devoted to speed up the Web, as well as in designing metrics that can accurately tell whether a webpage loaded fast or not and that can capture well the users' Quality of Experience.

The user-perceived latency, in particular, plays an important role in this regard and has tangible consequences: Amazon claims that ms latency penalty results in a 1% years to the QoE o ered to nal users and, more speci cally, this is the reason why a lot of e ort has been devoted to reduce delays. However, it is still unclear if and by how much a latency reduction translates into a better perceived experience from the user point of view. This, coupled with the manifest aspect of Web QoE heavily impacting revenues for Web-based companies, motivated a proliferation of new metrics proposals and validation studies attempting to better capture human perception on browsing experience. Despite the considerable progresses in this direction, the quality of experience of Web users' remains still largely impenetrable. This di culty is the result of the fact that users QoE on the Web is a ected by several factors. Some of them are often measurable, like those tied to the system (e. g., the network and application Quality of Service metrics), some others more frequently unknown and not trackable, more tied to the context in which the user is located, as the user expectations or expertise. Indeed, the concept of QoE combines user perception, experience, and expectations with Quality of Service (QoS) metrics. Figure shows all the elements which contribute to users' QoE, including the relationship between the system in uence factor, i. e., network and application QoS, the context and the human subjective component. These factors are strongly interdependent: network-layer QoS (e. g., the quality of the connection, packet losses, etc.)

a ects and can degrade application-layer QoS metrics, e. g., the time at which the page is loaded (PLT) or when it becomes interactive (TTI) etc., which in turn in uences the way in which the user experiences the browsing.

As a consequence, the analysis of Web QoE needs to couple the collection of two type of measurements: the objective metrics, which include the system in uence factors, typically automatically collected with the browser, and the subjective human feedback, which instead require involving the users.

The objective QoS metrics measured from an endpoint at network layer, e. g., latency, packet loss, bandwidth, etc., from a session viewpoint are gathered together as meaningful metrics at application layer. These metrics rely on measurable data (e. g., network, browser events) capturing Web quality [ , , , , ]. Most of these metrics are available from the browser navigation timing [ ] as they pinpoint precise time instants of the page loading progress, as the time when the page becomes interactive (TTI), or the time when the rst pixel is painted (TTFP), or can be inferred from packet/ ow-level tra c [ , ] and are easy to include as proxy of user experience.

On the other side, the subjective metrics require the user feedback and rely instead on directly collecting responses from users regarding di erent questions related to Web QoE. Di erent approaches have been proposed in the literature: collect the Mean Opinion Score (MOS) by averaging answers from a set of many users which have been asked

to rate on a -scale range their QoE [ ], or ask users to comment on a video of the website rendering process [ , , ], or even measure the "user acceptance" of a ser-

vice [ ].
Finally, to map these two kind of measurements two main approaches have been established in literature: expert models, where domain experts specify a closed form function Yet, machine learning should ideally not be used as a black-box tool, but rather considered as a methodology, with a rational thought process that is entirely dependent on the problem under study. Speci cally, the use of algorithms should ideally require an adequate understanding of their mechanisms, properties and limitations, in order to better contextualize and interpret their results. This is particularly relevant in the case of the algorithms which run the risk of replicating and even amplifying human biases, particularly those a ecting protected groups.

Indeed, models pre-trained on large datasets could encode biases potentially damaging towards marginalized populations and could reinforce existing hegemonic viewpoints [ ]. The deployment and wide commercialization of biased algorithms would expose potential ethical implications that should not be understated. This already manifested in recent years in several ways with varying degrees of consequences for the subject group. Notable cases brought to the fore include the gender bias in Amazon online recruitment tools [ ], where the AI software penalized any resume that contained the word "women's" in the text, hence downgrading the resumes of women who attended women's colleges, and the COMPAS algorithm [ ] used by some U.S. states to predict whether defendants should be detained or released, which was found to be biased against African-American. This is crucial in the case of natural language processing (NLP), whose applications are among the most pervasive (e. g., hiring and recruitment, chatbots, conversational systems, etc.) and, for which, the presence of a bias would be detrimental and have high impact harmful consequences. In this area, the last three years were characterized by the emergence of several innovative NLP algorithms designed to solve di erent tasks and engineered to assist the most diverse applications. Among those, groundbreaking transformer-based language models (LMs) have been proposed and gained lots of scienti c interest due to their sizable improvements on a wide range of NLP tasks [ , ,

, , ]. All the proposed models have been pre-trained trained on Web-based corpora, ranging from user-generated content, as Reddit, to encyclopedia, as Wikipedia, to literary works, as B C , and news articles, as CC N .

Actually, given the sheer size and heterogeneity of the Web, one could expect these models to guarantee diversity and be not prone to bias. However, the assumption of considering large amounts of Web text as representative of all of humanity has already been widely questioned in literature. Several works [ , , , , ] highlighted the risk that this could have in perpetuating dominant viewpoints, resulting in models that embed stereotypical and derogatory associations along gender, race, ethnicity, and disability status. More speci cally, when pre-trained LMs demonstrate a preference depending on the way a group of people speak, i. e., by understanding one group better than the other, we are dealing with a problem of algorithmic bias and, consequently, of fairness.

. This thesis is organized in six chapters and structured as follows. This rst introductory chapter is devoted to provide a global context and present the problems addressed.

During this introductory, we divide the objective of the thesis in three main research questions, enumerated in the following:

-Q : How accurately can we measure and predict the Quality of Experience of users browsing on the Web?

-Q : Does the Web sheer size and heterogeneity ensure the fairness of the models trained on Web-based content?

-Q : Are machine learning models better at classi cation tasks when trained on content generated by machines instead of humans?

. . Thesis Organization

In the following, we brie y sum up our contributions and schematize the outline in Figure :

• In Chapter we study the users' Quality of Experience on the Wild Web, addressing question Q . We do this by performing a large-scale study of one of the most popular websites in operation, namely Wikipedia, and explicitly asking a fraction of its users for feedback on their actual browsing experience.

-We show that the analysis of the collected subjective users' feedback reveals both expected (e. g., the impact of browser and network connectivity) and surprising ndings (e. g., absence of day/night, weekday/weekend seasonality and other temporal dependencies).

-Also, we leverage user survey responses to build supervised data-driven models to predict user satisfaction which, despite including state-of-the art quality of experience objective metrics and a considerable number of features of di erent type, are still far from achieving accurate results.

-The collected dataset is made publicly available, hopefully contributing in enriching and re ning the scienti c community knowledge on Web users' Quality of Experience (QoE). • In Chapter we continue the study of question Q and focus on QoE when this is measured in controlled environments, conducting experiments diametrically opposed to those shown in Chapter . An often implicit assumption made by industrial and academic research communities is that a single metric is su cient to assess whether a webpage loaded fast.

-In this work we collect and make publicly available a unique dataset which contains webpage features (e. g., number and type of embedded objects) along with both objective and subjective Web quality of experience metrics. This dataset was collected by crawling over websites-representative of the top M websites in the Wild Web-while crowdsourcing the subjective user opinions on the user perceived page load time (uPLT).

-We show that the uPLT distribution is often multi-modal and that, in practice, no more than three modes are present.

-Our analysis reveals that, for complex webpages, each of the di erent objective QoE metrics proposed in the literature (such as ATF, TTI, PLT, etc.) is suited to approximate one of the di erent uPLT modes.

• In Chapter we investigate the relationship between the Web and machine learning models, addressing question Q . Speci cally, we test whether the sheer size and heterogeneity of the Web guarantee also that the models trained therein are bias-free and, consequently, are inclusive with respect to di erent social groups.

-We study the fairness of state-of-the-art transformer-based language models recently proposed and widely adopted for a plethora of NLP tasks. They have been trained on Web-based content of di erent size and type.

-We propose and validate an evaluation technique to assess the quality and the bias of the predictions of language models on transcripts of both spoken African American English and Standard American English.

-Our analysis shows the presence of diverse biases encoded by di erent stateof-the-art language models, like BERT and RoBERTa, de facto revealing that the heterogeneity of the Web is a feature that does not imply diversity.

• In Chapter we instead focus on Q and observe the behavior of some of the simplest state-of-the-art supervised machine learning models when they are trained on content generated by machines. We show how predictions are accurate when it comes to the classi cation of objects with very pre-determined patterns, in sharp contrast with human-generated data as shown in Chapter .

-We propose a framework to classify the di erent behaviors of the identi cation eld of the IP packet (IP-ID). This in the past was mostly implemented in the operating systems as a simple packet counter, which allowed to perform a wide range of tasks. However, this behavior has been discouraged over the years for security reasons and other policies, as the use of random values, have been suggested.

-Despite being only minimally intrusive, our technique is signi cantly accurate (99% true positive classi cation), robust against packet losses (up to 20%) and lightweight (few packets su ces to discriminate all the IP-ID behaviors).

-We then apply our technique to an Internet-wide census, where we actively probe one alive target per each routable / subnet, and provide an updated picture of the Internet-wide adoption of the di erent known IP-ID implementations.

• Finally, in Chapter we summarize and report the main contributions to the research community and conclude the thesis with a discussion of the open issues.

. . Publications

The content of this dissertation has been partially published in international conferences and journals. In the following we report the list of papers published or under review: ]. Finally, because these tests are carried on a limited number of pages, it is possible to evaluate computationally costly metrics, such as those that require processing the visual rendering of the website, which would hardly be doable in the World "Wild" Web.

Our aim is instead to take a completely di erent approach and perform a large-scale study of a popular website in operation, by explicitly asking a fraction of users for feedback on their actual browsing experience. Clearly, the approach is challenging but it opens the possibility to gather more relevant user-labels, as they are issued from real users of a real service, as opposite to crowdworkers payed to play a game (e. g., nd which video completes rst as in A/B testing).

We do so by launching a measurement campaign over Wikipedia, that has gathered over k survey responses in nearly months. We complement the collection of user labels with objective metrics concerning the user browsing experience (ranging from simple PLT [ ] to sophisticated SpeedIndex [ ]), and harvest several data sources to further enrich the dataset with several other informations (ranging from technical speci cation of the user device to techno-economic aspects tied to the user country) so that each user survey answer is associated with over features. Summarizing our main contributions:

• rst, we use survey data to deeply characterize user satisfaction along both temporal and spatial dimensions: shortly, we nd that on average 85% of users are satis ed and show that user satisfaction does not exhibit seasonality at daily/weekly timescales (which is unexpected) and document evidence of spatial dependency across many of the collected features (e. g., network access, browsing equipment, country wealth, etc.);

• second, we use labels to build data-driven models of user experience: despite including performance metrics considered to be the state-of-the art in user quality of experience, we nd that the model still falls short from attaining satisfactory performance in operational settings;

• third, in spirit with the current trends toward research reproducibility, we release the collected dataset as open-source (after having carefully ensured that no sensitive information is leaked in the process, see Section . . ), as we hope this can help the scienti c community in re ning its understanding of Web users' experience.

In the remainder of this chapter, after overviewing the related work (Section . ), we explain the feedback collection process and dataset (Section . ), which we dissect under both temporal and spatial angles (Section . ) and that we leverage to build a data-driven model of Wikipedia users' quality of experience experience (Section . ).

We nally discuss current limitations in Web QoE assessment and possible directions to circumvent them (Section . ) and summarize our ndings (Section . ).

. . Wikipedia is, according to Alexa [ ], the th most popular website, with over billion monthly visitors, that spend over minutes over pages on average per day on the site. We engineer a survey that is triggered after the page ends loading and collects user feedback (Section . . ), that we augment with additional information (Section . . ).

Assessment of

We note that, while this work is not the rst in leveraging Wikipedia surveys in general (see e. g., [ ]) this is the rst to gather user feedback on quality of Web browsing experience from operational websites, for which we believe releasing the dataset can be valuable for the community. To make sharing of the dataset possible, we take special care into making user and content deanonymization as hard as possible, without hurting the dataset informative value as much as possible (Section . . ). In this section, we also perform a preliminary assessment of the collection methodology, to con rm the absence of bias in the response process (Section . . ).

.

. Technical aspects of the survey collection

Due to limitations in Wikimedia's caching infrastructure, the survey is injected into the page via client-side code. Wikimedia continuously collects Navigation Timing performance data of a randomly selected sample T of page views (less than every , pageviews). The survey is then displayed to a randomly selected sub-sample Sf this population (less than every , of the pageviews with Navigation Timing information)

and only part of the surveys do receive an answer A. Since A ⊂ T, several features (that we detail in Section . . and analyze in Section . . ) related to page loading performances are also available for pages sampled in the survey responses.

The survey appears on Russian, French and Catalan Wikipedias, as well as English Wikivoyage, and it is displayed in the appropriate language to the viewer. We collect the survey on mobile & desktop version of the site (but not on the mobile app). The goal of The survey is injected in the DOM after the page nished loading (i. e., when the loadEventEnd [ ] res). In order to give the survey visibility, it is consistently inserted in the top-right area of the wiki article, ensuring that it typically appears above the fold. However, as the users can freely browse the page before the survey appears, it might be out of sight when it's injected in the DOM, which is why we also record the time elapsed between the loadEventEnd and the moment the user sees the survey.

Also users that are shown the survey are free not to respond to the survey, or might as well respond very late (e. g., possibly browsing to other tabs in the meanwhile).

Overall, users responded as reported in Tab. to about 3.6% of the over 1.7M surveys that have been displayed in the period, for a total of over 62k anwers: 84.8% of the users respond positively to the survey with an almost equal split of the remaining answers to a neutral (7.7%) or negative (7.5%) grades.

.

. Collected features

We enrich the collected corpus with external sources that can be useful for a better understanding of the survey responses (Section . ) as well as being instrumental to the purpose of feedback prediction (Section . ). A terse summary of the metrics collected (as well as those we plan to release) is reported in Tab. . We discuss rationales of the selection for metrics that we make available in the publicly available dataset in Section . . .

Page:

For each page, we record features that concerns it (e. g., its URL, revision ID, size, etc.) and that thus are critical from a privacy point of view. We additionally record the time lapse at which the survey is shown to users, which is instead innocuous.

Performance: Since S ⊂ T, then all the Navigation Timing performance-related metrics (such as DOM, PLT, TTI, TTFP, connection duration, number of HTTP redirects and their duration, DNS wait time, SSL handshake time, etc.) are also collected. Finally, we compute the page download speed which is a simple, yet non linear, transformation of page size and connection duration, by quantizing it in steps of Kbps. These informations are speci c to page views, and are less critical to be shared.

User:

The collected user-related metrics include the browser, device and OS families. Additionally, we know whether users are logged in Wikipedia, if they are accessing Wikipedia through a tablet device and the number of edits that users have made (coarse bins). These informations are of course highly critical. Country-level precision also allows us to relatively compare performances across users . in the same environment, i. e., we normalize the page download speed with respect to the median per-country speed observed in our dataset (in terms of ratio, absolute and relative error).

Environment

Additionally, ties between country wealth and network tra c volumes have been established in the literature (particularly, deviation from expected volume [ ]): it is thus worth investigating whether there also exist ties between wealth and users' impatience.

We use the Gross Domestic Product (GDP) information made available by the World Bank Open Data project [ ]. The per-country economic features we consider (namely, per-country GDP, country GDP rank, per-country per-capita GDP, etc.) are expressed in terms of Geary-Khamis dollars, which relate to the purchasing power parity, i. e., how much money would be needed to purchase the same goods and services in two countries.

The rationale in so doing is that, albeit Web users perception is tied to psychophysics laws [ ], there may be environmental conditions that tune this law di erently in each country. For instance, a xed amount of delay (the stimulus) may have a smaller perceptual value to users of countries with poor Internet access which GDP-related features might capture: e. g., in other words, one can expect users in a high-GDP country to have better average performance and thus be more impatient than users from a low-GDP one.

In particular, we use the per-country dataset provided by [ ] since arguably the world-level statistics evolve on a relatively long timescale.

Finally, we expect user-home gateways [ ] and particularly end-user devices [ , ] to have a direct impact on the overall performance. As such, we complement the ISP-level view with a device-level information. Particularly, we harvest the Web [ ] to nd techno-economic information about user devices and in particular, collect device CPU, memory and pricing information. Intuitively, this information complements the per-country GDP information as, e. g., there may be further perceptual di erences between users with a costly smartphone in low-GDP vs high-GDP countries. We recognize that device CPU and memory specs are only an upper-bound of the achievable performance, as it is the mixture of applications installed and running on a device that determine the amount of available CPU and RAM resources, from which user perception will be ultimately a ected [ , ]. Missing this information on a per-sample basis, we attempt to at least construct the per-device statistics, by considering Navigation Timing information of a large representative sample of Wikipedia users. Particularly, we consider the month of August during which we observe over million Navigation Timing samples from , di erent devices, including all , devices in our survey.

We then construct deciles of per-device performance (e. g., of page load time and similar

Note that we collect pricing information at the time of our query, and not at the time when the device was actually bought; we also ignore price di erences among countries, and per-ISP o er bundles. Publicly Available timing information): indeed, it can be expected that users of knowingly slow devices be less impatient, which this additional data source could provide.

.

. Ethics

The dataset we collect contains obviously sensitive information allowing to deanonymize

Wikipedia visitors (such as IP addresses, version of their browser and handsets), as well as linking them to the content they visited (e. g., page, revision ID, time of their visit,

etc.).

Despite the dataset release policy explicitly forbids user deanonymization, in the interest of respecting personal privacy we have to obscure information so to render user deanonymization as hard as possible, while still allowing meaningful information to be extracted from the data.

At rst, as discussed in the WWW [ ] paper, we proposed a conservative vetting process, that selectively ltered/obscured/aggregated information to select which features could be ultimately released publicly, out of the total (T) dataset, that in this chapter we denote with (WWW): the risk in this case was that perfect unlinkability could not be claimed, since we do not control all other sources (e. g., a survey responder wishing to deanonymize himself, well-funded opponents, capable researchers, etc.). After a rigorous Wikipedia legal vetting process, we instead choose another option that aggressively lters/obscures/aggregates features in an extremely conservative manner, considering only features of the performance class, that are not linked to sensitive information. In particular, the publicly available (PA) dataset is provided only for the Russian and French Wikipedia.

Method: Speci cally, for the conservative (WWW) case, we opted for an approach where we transformed data in a non bijective way (e. g., IP to ASN and ISP mappings that provide network-related properties, while preventing user deanonymization at the same time), or aggregated at a su ciently coarse grain (e. g., country-level geolocation; obfuscation of browser major/minor version; aggregation of unpopular devices, etc.). For the same reason, we decided to aggregate time-related information at a coarse-grain (hour-level) and drop most content-related features (e. g., page ID). We quantized the page size with a resolution of kB, to also make it hard to reverse-engineer which page was visited. We maintained most of the Navigation Timing related performance features, that have the highest mutual information, which we obfuscated wherever necessary (e. g., given that with precise PLT and download speed one could easily reverse engineer the page size, and thus the content, we quantize the download speed in steps of Kbps). In the publicly available (PA) set, only performance metrics are considered, that are not linkable to any property related to time-of-day, user, content, geography, device, etc.

Results: As a consequence, comparing results in these two scenarios is useful to see if this loss of information potentially has an impact on the global prediction accuracy, which we assess in Section . : at the same time, from results presented in Tab. , we can expect this e ect to be rather limited. Indeed, Tab. reports the number of features that are collected overall (T) vs those that would have been available under a conservative (WWW) vetting process and the publicly available ones (PA). For each class ( rst column), the table reports the number of T/WWW/PA features (second column), and additionally reports boxplots of the mutual information MI(x, y) between features in the class and the survey answer (last column). MI expresses the amount of information (in bits) that can be obtained about the survey answers through the observed variable.

Tab. shows that, while we only consider about half of the collected features (T), the (WWW) features overall have a higher mutual information (particularly, note that the th percentile, median and th percentile are higher in the (WWW) feature set). Thus, we conclude that:

• on the one hand, classi cation results of Section . are only minimally a ected by selecting all (T), some (WWW) or very few (PA) features, so that repeatability of

Wikipedia (T) Answered (A) 0.1 1 10 0.1 1 10 A set A + , A 0 and A - sets Positive (A + ) Neutral (A 0 ) Negative (A -) Figure : Quantile-quantile plot of PLT statistics for di erent sets (T ⊃ S ⊃ A = A + ∪ A ∪ A -).
the QoE study is not a ected by the vetting process: under this angle, it is fortunate that features belonging to the performance class, which are those exhibiting the highest mutual information with the user grade, are also the ones made available, being the least critical to share.

• on the other hand, the type of study we conduct in Section . would be impossible to reproduce with the available features (PA) set: under this angle, we decide to provide in this work a through spatio-temporal characterization of the collected dataset, as it would hardly be doable otherwise.

. . Validity of the collection methodology

Despite our care in engineering the survey questioning process, we cannot exclude apriori the existence of bias in the user survey answer process. For instance, users might refrain to answer when the page loading experience was positive, and be more willing to express their opinion in case of bad experience, which would lead to under-estimate the user satisfaction.

To assess whether our survey collection methodology yields to such (or other) biases, we compare three sets of page view experiences, namely (i) the set T where we record Navigation Timing information from the browser (ii) the set S where users have been shown the survey (iii) the set A where users have actually answered to the survey. Finally, we further slice the set of answered surveys A according to the answer in three additional datasets with (iv) positive A + , (v) neutral A 0 and (vi) negative A -grades.

Among the numerous features we collect, without loss of generality we now limitedly consider the Page Load Time (PLT) distribution. Since S ⊂ T is selected with uniform random sampling, by construction we have that S and T are statistically equivalent as far as individual features, such as PLT, are concerned. However, in case where users decision to answer to the survey (irrespectively of the actual grade that we consider in Section . ) would be biased by the performance of the page, then the PLT statistics should di er among the set of displayed S vs answered A surveys. The left-side of Fig.

reports a quantile-quantile (QQ)-plot of the empirical PLT distribution, using quantiles of S on the x-axis and T, A on the y-axis, from which one can clearly remark the absence of such bias.

Conversely, one would expect that, shall the PLT a ect the actual grading of the browsing experience, then PLT statistics should di er among the

A + ∪ A 0 ∪ A -= A
sets. This is shown in the right-side of Fig.

, comparing the quantiles of the answer set

A on the x-axis to its per-grade slices on the y-axis. Several remarks are in order. First, it can clearly be seen that browsing experience with negative scores fall above the equality line, con rming as expected that the set of negatively rated pages A -contains pages with longer download time compared to the positive A + and neutral A 0 sets. Second, similar considerations hold for neutral (slightly above) and positive (slightly below) answers, although they are less visible -in part, this is due since positive grades represent the bulk of the answers |A + |/|A| = 84.8%, for which the PLT statistics of A + and A are mechanically more similar (we will take care of class imbalance when appropriate later on in Section . ). Third, we notice that the QQ-plots of positive, neutral and negative answers overlap for quantiles corresponding to low and moderate PLT values, indicating as expected that the PLT alone cannot fully capture user perception.

. We start by analyzing the user feedback along aggregate (Section . . ), temporal (Section . . ) and spatial (Section . . ) viewpoints, including for the time being the neutral answers. Second, scores are hardly separable along any of the TTI, RSI or PLT metrics: notice for instance that 75% of positive (57% negative) pages have a TTI up to second, and that similar considerations hold for RSI s (59% positive vs 43% negative) and PLT s (47% vs 32%). This raises the need for additional metrics beyond those related to performance timing, which hopefully can further assist the prediction of user scores. .

. . Aggregate view

As previously illustrated in

. Temporal breakdown

At a glance: We next present the daily amount of user answers over the whole -months period, with annotation of di erent Wikipedia-related events. Such events, some of which are reported in Fig. , are of di erent nature and include, e. g., the injection of banners for fundraising or the call for volunteering contributions to Wikipedia content; network-related events such as data center switchover/switchback; browser-related event such as new versions that introduce known regression in performance metrics (e. g., Chrome release that introduces a firstPaint regression); back-end events and deployment of new features (e. g., RUM metric "MediawikiLoadEnd" improved). As it can be seen from Fig.

, an operational website at scale continuously has events that are generally not available in testbeds (such as those overviewed in Section . ), that thus sample very narrow and speci c conditions that are not representative of real deployments.

Yet, these operational changes appear to have only a moderate e ect on browser tim- time activity that is preserved by our random sampling. In the day-of-week case one can notice a slight increase in the answer frequency on Sundays, which in our dataset is due to a combination of (i) a slightly higher tra c volume on some Sundays over the -months period, (ii) as well as a higher propensity to answer the survey on Sunday, especially during some weeks of September. In turn, from daily variability in user responsiveness, one could have expected a higher tolerance to, e. g., slow websites performance, that however does not appear in our results. One likely reason is that the largest discrepancy between maximal and minimal user click time is on average of about second during the day (see Fig.

(b) in [ ]), which
may not be enough to trigger perceptual changes so important to a ect the acceptability of the page rendering process (whereas they could have appeared had our survey involved a ner-grained -grade ACR scale feedback).

Similarly, from the bottom plot of rates. In case of numerical features, the natural ordering is otherwise preserved (so that breakdown is not monotonously increasing). On each subplot, the top x-axis report the cardinality of samples for each bar, and the bottom label reports the feature name and is further annotated with the mutual information value.

Page-related metrics: Particularly, we aggressively censure features that would allow content-linkability, making only two page-related features available in the (WWW) set:

namely, the survey viewtime after the page is rendered and the HTML page size. The plot in the top left corner of Fig. reports the variation on scores as a function of the HTML page size. It can also be seen that breakdown is very similar irrespectively of the HTML page size, with the exception of smaller pages, that have a slightly higher negative scores (which deserves further attention). Thus, in our dataset the page size only plays a minor role in the user feedback, which can be expected since Wikipedia pages tend to be relatively small. Concerning the smallest bin of pages up to 10kB, notice that it comprises 7.8% of the over 46k pages (i. e., a bag of 3.6k pages) con rming that a 10kB granularity make linkability complex.

User-related metrics: Among user-related metrics, we select the browser, device and OS families ( ner grain information is precluded from sharing), and report whether users are logged in Wikipedia (binary ag), if they are accessing Wikipedia through a tablet device (binary ag) and the number of edits that users have made (coarse bins). These features are reported in the top row (and the rst two features in the second row) of Fig.

.

For the family of browsers, device and OS, we report the top-and aggregate all others into a "other" bin. Interestingly, from the browser family one can notice a remarkable discrepancy of users score breakdown for di erent browsers. Particularly, one can observe "mobile" versions of popular browsers to have poorer scores than their "laptop/desktop" counterpart: in this case, one cannot easily disambiguate whether poor scores are tied to bad implementation of the browser, or to bad performance of the mobile device (a nevertheless very likely cause [ , ]). Considering only laptop/desktop browsers, we have that Safari ( st), Opera ( nd) and Chrome ( rd) are on the podium, with Firefox ( th) a close next.

It is also interesting to observe that, whereas users scores quite clearly di er among browsers, the amount of mutual information is still relatively low (comparable to the HTML page size) -which is due to the fact that browsers are not equally represented in the dataset, with Chrome and Chrome mobile taking up over 50% of the samples in our dataset. Similarly, score breakdown is remarkably di erent across devices, yet the number of devices is so large (over 2.7k) and the categories either too precise (as for the di erent XiaoMi models) or too coarse (iPhone and iPad do not unfortunately report the model version, which mixes old and new devices in a single bin) resulting in a very low mutual information.

Score breakdown per OS con rms that users score are better on laptop/desktop. However class imbalance across OSs makes it so that a simple binary indicator (isTablet) has a higher predictive power with respect to more precise labels (e. g., twice as much as the OS and browsers family).

Next, concerning user experience on Wikipedia, we notice that readers ( edit) are more likely to provide a negative answer than writers (from to over edits). This is somewhat surprising since whereas our survey population is mostly European, logged editors are always directed to US servers, incurring in higher latency. The higher fraction of positive answers can be due, on the one hand to the fact that higher Round Trip Time (RTT) delay may be masked from warm-up caches for the page they are editing, or on being more accustomed with (and thus more adapted to) Wikipedia service. At the same time, given that most (97%) of Wikipedia users are readers, the knowledge of the edit counts is irrelevant for predicting user satisfaction -so that even in this case a simple binary information such as whether the user is logged in has more predictive power (high MI ).

Environment-related metrics: Features in the environment class include network-related and per-country information, reported in the middle and bottom column of Fig.

respectively. Network information is represented by ASN, connection type and speed information (particularly, we report in the picture the ratio of the download speed to the median speed in the country observed in our dataset). We see that all have a clear impact on the user scores, with consistent di erences across ASN, very strong di erences across connection type (although there are only very few G and G connections in our dataset, thus a low MI) and strong di erence on the relative connection speed. Interestingly, concerning the latter one can notice that the ratio of negative scores decreases for increasing speed, and nally exhibits a slight decrease again for users having × the median speed in the country -likely well equipped and possibly more impatient users.

In terms of country-level information, bottom-row plots in Fig. inspect the country name and its GDP rank. Two phenomena appear: on the one hand, we observe that users living in countries with poor GDP (high rank) consistently report poor performance (likely tied to poorer infrastructures); on the other hand, we observe that users of wealthy countries, that have comparably better performance (e. g., higher rates), also possibly report negative scores, but possibly due to di erent reasons (e. g., tied to higher user impatience due to higher expectations).

We nally consider further information concerning the user device (such as RAM and price harvested from the Web), which we report to the median per-country per-capita GDP. We gather that, whereas poor maximum RAM ( GB) is symptomatic of bad performance, scores are strikingly similar across a range of device prices: as performances are likely di erent across devices [ ] (which in part justi es the price di erence), this seems to suggest that owners of cheap devices are prepared to be more tolerable in spite of poorer performance. However, if we do take into account the relative wealth of the country by normalizing the price tag over the per-capita GDP, we see that there is a negative correlation with user scores (possibly, expectations of users owning a pricey device in a lower-GDP country are also higher, and users are more likely to report bad performance as negative experience).

. We continue by disregarding the neutral scores and now build data-driven models that forecast user answers.

Problem formulation: Keeping only negative and positive answers for the user feedback Clearly, from an operational standpoint a conservative estimation of user satisfaction is preferable. Indeed, the service operator wants to avoid that a malfunctioning service that is truly a ecting user experience goes undetected, as when the ratio of dissatis ed users increases above a given level this can prompt alert to repair or ameliorate the service. In our settings, conservative prediction results translate into maximizing the recall of negative scores.

Reference classi cation results: Given the class imbalance, we have to preliminarily downsample the dataset : indeed, given that after discarding the neutral scores 92% of the users are satis ed, a naïve -R classi er that just learns the relative frequency of the scores and systematically answers with the majority class, would achieve . accuracy -but would entirely miss negative scores, having thus a null A -recall. Hence, a more appropriate baseline for recall of unsatis ed users requires performing undersampling, i. e., keep only a portion of the positive scores, equal to the size of the negative ones, to obtain a balanced dataset. We denote the balanced dataset B and the complementary dataset, only containing positive answers ltered out in the downsampling as B.

We prefer to avoid the diametrically opposite approach of synthetically generating users score, which is in stark contrast with the very same nature of our survey work. the true positive rate by letting the probability threshold vary. We remark that for all the other results we keep the probability threshold xed to 0.5. This is obtained again when averaging the results of the -fold validation with the abovementioned di erent models. This plot con rms that classi cation performs just slightly better than random guessing (blue dotted line), regardless of the model adopted, without any surprising result. XGBoost is the model leading to the highest Area Under the Curve (AUC) (0.65), followed by the MLP.

Feature subsampling: We next consider how the classi cation results change with respect to the above reference (where we consider the collected overall (T) features) when we slice the dataset by keeping only subsets of (T). We repeat the experiments with the same parameters used to obtain the above reference results and the same statistical metrics. set (all belonging to the performance class). In this case, results show a slightly higher, but still very limited specially in the random forest case, reduction of the classi cation performance: on the one hand, performance-related features consistently rank high in terms of Gini importance, though on the other hand they lack discriminative power for telling user answers apart. We also report the classi cation results obtained when we remove the outliers from each feature using the three-sigma rule, that we denote with

(PA w.o.
). In this case, we observe that ltering out the outliers do not the performances. The solid black line in top of metrics we collect, enriched with environmental information as described in Section . , are apparently not enough to discriminate among satis ed and unsatis ed users.

Explainability: In step with the current trend towards human interpretable machine learning and model explainability, we leverage SHAP (SHapley Additive exPlanations) [ ] to explain which are the relevant features that can help revealing whether a user is satised or not. We report in . This work is the rst to leverage user feedback from real browsing sessions in operational settings. As any new work, there are a number of limits, which requires communitylevel-e orts which we discuss next.

Collection and validation methodologies: We remark that this work is the rst to collect user feedback from real users in real browsing activity, from an operational deployment.

This is in stark contrast with most lab research, where volunteers or crowdworkers are exposed to a very limited heterogeneity (e. g., single device/browser), are not carrying on a browsing activity (e. g., A/B testing uses videos) and are not asked about their satisfaction but about other metrics as a proxy (e. g., which video nished rst?). We argue that lab/crowdsourcing experiments and collection in the wild should coexist.

On the one hand, we stress that while A/B testing is a necessary step, it is however not su cient. Survey data discussed in this chapter seems to suggest that metrics that are considered as state-of-the art for Web QoE, seems to be ultimately poorly correlated with the experience of real Wikipedia users. In turn, it also follows that lab/crowdsourcing experiments should diversify the type of user feedback: e. g., the fact that a user is able to notice which video nishes rst (which uPLT metrics attempt to model), does not imply that he would grade that Web rendering process as positive (or the rendering corresponding to the other video as negative).

On the other hand, we are aware that part of the challenges in real-world experiments comes from diversity and variance: it follows that surveys such as those we are carrying on should be kept running continuously, as it is commonplace for VoIP applications that regularly poll their users for a QoE opinion. Operating continuously would lower barriers for further experiments [ ], empower website operators with a very relevant performance indicator for their service, informing them in near-real time about impact of new features deployment. Additionally, long-time surveys allow to collect signi cant volumes of data to keep ameliorating models for user prediction in spite of high variance and heterogeneity. Moreover, there exist other QoE in uence factors that we did not include in this study, like the sentiment linked to the topic and the content of the page or more information about the context in which the measurement is carried out, as the earlier user browsing experience. These undoubtedly have an important impact, that is however hard to capture.

RSI: not needed, or not enough?: Concerning Web user QoE metrics, this study seems to suggest a poor discriminative power of the RUM SpeedIndex (RSI) so as to predict users scores, at least for Wikipedia users. In part, this may be due to the structure of Wikipedia pages (where, e. g., text may be more prevalent that in other pages in the Alexa top-typically considered in similar studies, see Section . ). This nevertheless raises the question so as to whether it is possible to (i) design metrics that are better t to the spatial structure of the page, or (ii) metrics capable of better weighting the focus of user attention, and at the same time (iii) raises questions about the accuracy vs generality of QoE metrics.

As for (i), we have currently improved the system to also collect navigation timing statistics for speci c elements that are believed to be important for Wikipedia, such as the "time to the top image". This is a good compromise between collecting the whole waterfall (which is impossible in operational settings) and could yield to metrics that are website-speci c (losing generality), but better correlated with user experience (gaining discriminative power).

As for (ii), we are aware that more complex approaches involving spatial dimension (i. e., eye gaze) also exist [ , ]. However, including the spatial dimension in the user perception is hard to capture in the lab, and challenging in the wild: a good starting point would be to leverage mouse-movements as a proxy of eye gaze activity (which are known to be strongly correlated [ ]), and that can help further re ning QoE metric in the spatial direction (e. g., by adding the knowledge of whether the rendered element is under the user gaze). Additionally, mouse-movements can capture user anxiety which further reduces the user viewport [ ]. Clearly, further research is needed on whether user-touch can be useful for similar purposes in case of mobile handsets.

Finally, (iii) previous work [ ] already has pointed out a tension between accuracy vs generality of QoE metrics and models: on the one hand, it seems rather challenging to capture the rich diversity of over one billion pages with a single QoE model, so that it may be tempting to develop website-speci c models, as it is our focus here; on the other hand, it may be possible to develop models for groups of websites that sharing similarities in their underlying structure (e. g., picture-dominant vs text-dominant sites; interactive vs static pages; etc.), which remains an open question to date.

Per-server vs per-device statistics: In this work, we did not explicitly leverage time-series of server-related operational metrics, as these are gathered live at minute-timescale on

Prometheus [ ] but are not readily available on the Hive platform [ ]. At the same time, the raw load on during the considered period appears too low in practice to have an impact so signi cant to a ect user satisfaction.

Conversely, given that mobile browsers performances are signi cantly dependent on the handsets, as already shown in [ , ] and con rmed in this work, collecting perdevice statistics seems a mandatory step to ameliorate prediction performance, as "com-putation activities are the main bottleneck when loading a page on mobile browsers" [ ].

Unfortunately, average per-device performance we considered in this work are not telling enough, as they merely report the resource upper-bound (i. e., CPU and RAM capacity)

as opposite to the actual state of the device (i. e., free RAM and available CPU cycles) corresponding to the page view that the user answered about -which could hopefully ameliorate prediction performance.

.

In this work we engineer, collect, analyze and predict user survey scores pertaining to the quality of their Web browsing experience. Out of over 1.7 million queries, we gather over 62k answers corresponding to either positive (84.8%), neutral (7.7%) or negative (7.5%) experiences. Associated to each answer, we collect features, part of which we make publicly available taking care of rendering user deanonymization and contentlinkability as hard as possible.

The main takeaways in our analysis are that users are consistently satis ed, and that scores do not exhibit seasonality at circadian or weekly timescales, which is unexpected. Quite surprisingly, scores are also not a ected by network-related events (e. g., data center switchover) happening during the period, nor by Wikipedia-related events (e. g., banner campaigns that alter the page rendering) nor by known browsers events (e. g., Chrome rst paint regression). Additionally, we nd that scores are, as expected, heavily in uenced by user-level expertise and equipment (e. g., device, OS and browser), as well as network and country-level characteristics (including access technologies, ISP and economical factors). Interestingly, scores are not a ected by the Wikipedia page size, nor by the device price (unless economical factors are also weighted in).

Concerning user score prediction, perhaps the most important (and equally disturbing) takeaway is that it is surprisingly hard to predict even a very coarse-grained indication of user satisfaction. This can be tied in part to the lack of more informative indicators in our dataset (such as content and context factors that are known to a ect user QoE), and also raises a number of interesting questions and challenges for the whole community.

T

H E U S E R S A N D T H E C O N T R O L L E D W E B

Contents

. Originally, quality of user experience on the Web was approximated using simple showing that users rarely agree on a single uPLT [ , ]. However, the multi-modality of uPLT was not the main focus of these studies, and as such it was not studied in depth.

UPLT multi-modality is rooted in many factors, such as personal preferences with respect to what is considered important on a webpage, e. g., text rather than images, carousels of elements, popups or ads. Fig. illustrates this issue when asking for feedback from recruited participants regarding the uPLT of www.booking.com. About 40% of users believe uPLT to be around 2 seconds, another 40% indicates uPLT≈ 3.7 seconds and nearly 20% report a uPLT≈ 9.1 seconds. These uPLT values appear in conjunction with distinct webpage loading events; we report the snapshot of these events in Fig.

.

This example illustrates the challenges of measuring uPLT, and raises questions about which among the numerous objective Web QoE metrics (e. g., PLT, TTFP, ATF [ ]) is more suitable as a proxy for these remarkably di erent user opinions.

To address these questions, we collect a comprehensive data set of webpage features (e. g., number and type of embedded objects) along with both objective and subjective Web quality metrics. We nd that around 50% of the webpages in our study present a multi-modal uPLT distribution and that, in practice, three modes are su cient to accurately describe uPLT distribution. Moreover, we show that the number of images and the number of objects in a webpage can help in predicting uPLT modality. To promote cross comparison and enable further studies, we make this dataset publicly available [ ].

This chapter is organized as follows. We describe the methodology used to produce the representative set of webpages for our analysis and how we employed the Eyeorg platform [ ] to crowdsource uPLT on these pages (Section . ). Next, we thoroughly characterize the collected user feedback (Section . ), rigorously quantifying violations of the hypotesis that uPLT is uni-modal and nally contrast uPLT modes with objective QoE metrics. Finally, we discuss our ndings (Section . ) and put our results in perspective with recent related work.

.

To explore the relationship between uPLT and objective Web QoE metrics we need to (i) collect a comprehensive dataset comprising "representative" webpages, and (ii) crowdsource feedback from real users on uPLT. We rst devise a novel methodology to iden- is important to sample popular webpages, since they attract the majority of the tra c, unpopular webpages might have a completely di erent set of characteristics yielding to di erent results. In this chapter we argue that popularity should not prime over diversity of webpages, as otherwise the results may lose generality. We therefore opt for a strati ed selection of both popular and diverse pages by clustering them according to the complexity of their HTML content.

Initial hitlist:

We crawl all URLs from Cisco's Umbrella top -million list, which became popular in the research community after Alexa became paywalled, on August . The Umbrella list is generated by tracking the total number of worldwide DNS requests.

The main advantage of this approach is that this gives us insights not only on popular top level domains (e. g., wikipedia.org), but also on popular actual pages with content (e. g., https://en.wikipedia.org/wiki/Main_Page). However, the Umbrella list also contains URLs that are the target of automated DNS requests (i. e., not associated to an actual user request) notably for ads and analytic services. We additionally nd that some webpages on the list are either no longer valid or implement access control. By discarding URLs that either never responded to our request, or returned non-HTML content (e. g., JSON

or XML) we obtain 317, 000 valid HTML pages.

Clustering: On this set of webpages, we compute six features that, as reported by Butkiewicz et. al [ ], are distinctive of the page characteristics and in particular have high correlation with webpage complexity: page size (in MB), total number of objects, number of images, CSS, javascript, and number of distinct origins. Then, we rely on K-means to nd pages of similar complexity. Given our crowd-measurement budget, we x K = 100. We experiment with three standard feature normalization techniques: min-max, mean-std, and a quantile-based feature normalization, where we transform the original values of each feature to the quantiles they correspond to in the dataset (e. g., for amazon.com, page size: 38%, num imgs: 99%, num domains: 61%).

Fig. reports the number of pages per cluster produced by K-means, ordered by decreasing cluster size. We observe that, due to the wide range of values for page size and number of objects in the dataset (up to 75 and 1, 429Mb, respectively), both min-max and mean-std normalization create several "outlier clusters" near the extreme ranges of each feature with very few pages (less than 10), while creating a single overcrowded cluster for simple and small pages. We note that quantile-based normalization results

in clusters that represent a sizeable number of pages (the smallest clusters contain between 155 and 347 pages) while at the same time helps in better representing the ne-grain diversity of relatively small pages (there is no single giant cluster). Upon a closer analysis, we put aside 14 clusters that contained a large number of "error pages".

These cluster included regular HTML pages reporting , customized pages, pages with valid HTML but no actual content, as well as login pages. We observe that the 5 largest clusters still represent 30% of all pages: therefore, a strati ed selection strategy helps avoiding oversampling these pages.

Strati ed selection: From each of the remaining 86 clusters, we manually pick one webpage for user evaluation in Eyeorg. We do this by choosing a popular webpage according to the ranking, i. e., which is simultaneously (i) the closest to the centroid, (ii) in English language, and that (iii) does not contain o ensive or adult content (e. g., porn, gambling), in order to avoid exposing crowdsource participants to upsetting content.

Given the fair amount of work involved, this list of "representative" webpages is interesting per se, and we make it available [ ]. Finally, we add 22 handpicked webpages that we also studied in previous work [ ] to obtain a total of 108 sampled webpages.

.

. Objective Web Quality Metrics

For each webpage of the set, we collect the objective Web quality metrics discussed in Section . , notably the TTFP, the TTI, the Approximated Above-The-Fold (AATF) [ ] and the PLT. We rely on a Chrome extension [ ] to measure all the metrics, since some metrics require the rendered position of all objects in the page, cannot be measured from the HAR le (as opposed to PLT, DOM, TTI, etc.) and thus are better measured directly from the browser. Further, we use FFmpeg to record videos of a webpage rendering process.

We instrument a stock version of Chrome (v . . . ) with the above extension and attempt to load the 108 selected webpages, consecutively. For each load, we set a maximum duration of seconds and also record webm videos of the rendering process.

This is needed since we then plan to crowdsource users responses with Eyeorg. Since headless Chrome currently does not support extensions, we leverage the X virtualframe bu er Xvfb to allow remote execution without the need for a physical monitor. We measure each webpage times, ensuring warm DNS caches, and a clean browser pro le at the beginning of each run. We then select the experiment (video and set of performance metrics) with the median PLT among the repetitions.

.

. UPLT Crowdsourcing

We measure uPLT via Eyeorg's timeline experiment [ ] where a participant is asked to "scrub" the video of a webpage load until when (s)he considers the page to be ready.

We run a single Eyeorg campaign targeting the above 108 webpages and 1, 000 paid participants from Figure Eight (total cost: $120). Each participant evaluates videosthus generating 6, 000 uPLT values or about 54 valid feedbacks per webpage, on average.

In Figure Eight, we request the highest quality participants. As discussed in the Eyeorg paper, we also lter user responses using a mix of their engagement (i. e., the time spent on task) and the quality of their opinions using some control questions. Eyeorg implements control questions on top of the frame selection helper, a tool that helps the user "rewinding" her uPLT selection if an equal (earlier) frame is identi ed. This is needed because, for some users, it can be hard to scrub a video exactly to the earliest point associated with a selected frame. For one video out of six, the frame selection helper suggests the very rst video frame as a rewind option. Users that blindly accept this suggestion without noticing the obvious di erence between the two frames are considered as potentially distracted, and their responses are discarded. In total, we discard 172 users due to low engagement and due to failing the control questions.

. '

In order to provide an in-depth characterization of user feedback, we start our analysis by checking the existence of multiple modes on the uPLT distribution, considering for https://www. gure-eight.com/ No more than 1% di erent in a pixel-by-pixel comparison. each webpage the valid uPLT feedbacks. Then, we study the number and parameters of the di erent modes exhibited by the uPLT distribution for each webpage. Finally, we investigate how the complexity of modern webpages (e. g., number of objects, domains, etc.) and user browsing behavior may a ect uPLT multi-modality.

.

. UPLT Distribution Analysis

We next analyze the uPLT distributions to inspect the presence of multi-modal behaviors. For this purpose, we rely on a non-parametric statistical test widely used to assess whether a distribution of real-valued random variables, such as the uPLT, is likely to be uni-modal [ ]. This test computes the dip statistic as the maximum di erence between the empirical cumulative distribution function (ecdf), and the uni-modal distribution function that minimizes that maximum di erence. When we perform the dip test, we employ the common threshold p < 0.05 to reject the null hypothesis of uni-modality.

We nd on our set that 56 webpages are likely to exhibit a uni-modal distribution of uPLT and 52 a multi-modal one. By lowering this threshold, the number of likely multi-modal webpages decreases, e. g., when p < 0.01 only 42 pages are estimated as multi-modal.

For the webpages found to be likely multi-modal, we model their uPLT distributions with a Gaussian mixture model (GMM), i. e., a weighted sum of K independent Gaussian distributions. The question that naturally arises is how many Gaussian components K have to be considered per webpage. By letting the parameter K of the GMM range from 2 to 10, we observe that the GMM accurately models the uPLT distribution for K 3.

However, we nd that even for K = 3 some webpages have small modes (34 webpages have at least one of the three components with weight lower than 0.05). We run the goodness-of-t Kolmogorov-Smirnov test, with a con dence level of 0.95.

The null hypothesis is that the empirical uPLT and the mixture distribution (which we sample to obtain the same number of samples as the empirical one) with K = 3 come from the same distribution. The result shows that, for more than 70% of the multi-modal webpages, the null hypothesis is con dently accepted. Hence, for each likely multimodal webpage, we set K = 3 and nd the corresponding GMM parameters from its uPLT distribution: mean, standard deviation, and weight of each component. We observe that the weight ranges from 0.40 to 0.95 for the major component (M major ), which represents on average 69% of users (median 72%), 0.03 to 0.43 for the middle (M middle ) component, and 0.02 to 0.27 for the minor component (M minor ). For some outlier webpages, such as booking.com, users are split into multiple well de ned modes of similar size. In the opposite case, there are webpages such as paperpile.com

where nearly all the users agree on a single uPLT value, with two other smaller modes (M major,mass = 0.86, M middle,mass = 0.10, M minor,mass = 0.04).

The uPLT components can alternatively be sorted by occurring time, in such a way that the user opinion is split among M time = [M first , M second , M third ] on multimodal webpages (so that M first refers to the earliest in time and M third to the latest one). By analyzing the modes de ned in these two distinct ways, we can check when each M mass mode coincides with each M time mode.

Tab. shows the percentage of occurrences for each of the couples of M mass = M time (M major = M first , M major = M second , etc.). This gives us information on which time sorted mode M time is more liable to be the most or least popular one (M mass ). The table highlights that the majority of users are more likely to prefer the earliest modes: the major mode M major is indeed equivalent to the earliest mode M first on 33 pages (63% of the whole multi-modal webpages set), it is equal to the second Page Feature µ σ % % %

Size [MB]

Table : Statistics of uni-modal/multi-modal pages.

one M second on 17 pages (33%), and nally it is equivalent to the latest third mode M third on just 2 pages (4%). Reversely, the minor mode M minor tends to rarely coincide with the earliest one M first (8%): it is actually most of the time equal to the latest mode M third (63%) and sparingly to the second one M second (29%). We can nally conclude that the mapping between mass and time sorted modes is such that the most popular mode is generally also the earliest in time and vice versa.

.

. Page Characteristics and uPLT

Given that half of the webpages exhibit a multi-modal uPLT, we investigate which of their characteristics (e. g., number of objects, images, domains, etc.) may cause a split of users' feedback with respect to when the page is loaded. For example, ads heavy webpages might be (at least) bi-modal since some users consider the page to be loaded before ads are shown, while some others would wait for the whole content to be retrieved and displayed.

Tab. illustrates several statistics (average, standard deviation, / / th percentile)

of the webpage characteristics we considered during our strati ed URL selection (see Section . ). We can observe that the standard deviation of the size of multi-modal webpages is double with respect to that of uni-modal ones (at the 100% percentile we have 10, 338 vs 3, 460). We also observe that the mean number of images and of distinct origin domains for multi-modal webpages is respectively 34 and 11 compared to 20 and 7 for uni-modal webpages. This is inline with the intuition that complex webpages are more likely to be multi-modal. We next inspect how prevalent is advertising across these websites by matching the content received against EasyList, a list of known advertisehttps://easylist.to/easylist/easylist.txt ment domains. We nd that multi-modal websites contain, on average, times more advertisements than uni-modal websites, likely segmenting user opinions on uPLT.

We are now interested in assessing the importance of each of the above features for predicting uPLT multi/uni-modality of webpages. For this task, we train a Random Forest Classi er with estimators, which on a -fold cross validation achieves an average precision of 0.69 and an average recall of 0.68. In line with the current trend towards human interpretable machine learning and model explainability, we leverage SHAP (SHapley Additive exPlanations) [ ] to understand which features can better reveal whether a webpage is uni-modal or not. We report in Fig.

the features, sorted by the sum of the SHAP magnitude values computed for all the webpages. SHAP values capture the e ect of removing a feature for a given prediction under all possible combinations of presence or absence of the other features. Hence, they provide a quantitative insight of the importance of each feature for the model. The positive x-axis values assess the impact on the model output for predicting the uni-modal class, whilst the negative ones refer to the multi-modal class. We can observe that the two most in uential features are the number of images and the number of objects present in the webpage. In particular, the lower the values of these features, the higher their SHAP value (up to 0.2 for the number of images and 0.15 for number of objects). In other words, for simple webpages with few images and objects, users more likely agree on a single uPLT, making the uPLT distribution uni-modal. Such e ect is less evident for the other webpage properties, where low and high feature values overlap, causing a decrease in the impact Seven-fold cross-validation ensures that the validation dataset is at least 15% of the size of the whole dataset. factors on model prediction, probably due to the lack of additional data points to train the model. These ndings provide valuable insights for designing webpages with more predictable user perception. For instance, we might expect that the uPLT measured via mobile browsers presents a unimodal distribution, as they generally load a simpli ed version of the webpage. We acknowledge that future studies are needed to further elaborate relevant design guidelines in this direction.

Finally, we quickly investigate if a di erence in performance metrics can also explain the multi-modality of uPLT. We check whether the time di erence between the early and late events (such as TTI and PLT) of the page loading process provides strong evidence of multi-modality. Fig.

shows the ecdf of |PLT -T T I| for webpages we previously categorized as uni-modal or multi-modal. We can observe that multi-modal websites are, overall, characterized by larger |PLT -T T I| di erences compared to uni-modal ones.

On the other hand, less than 10% of uni-modal pages had a |PLT -T T I| > 2.2s. This nding suggests that the rendering of these webpages naturally segment the users: some believe the page loaded as soon as the major part of the page loaded (usually closer to TTI) whereas others wait for all visible images to nish loading to consider the page fully loaded (usually closer to PLT).

. We rely on the weighted RMSE (wRMSE) to assess the quality of approximation of objective Web quality metrics for the 52 webpages with multi-modal uPLT (see the bottom part of Tab. ). This approach weights the average towards larger components, which is particularly important for better evaluating the error on M middle and M minor . We conduct this analysis from three di erent perspectives: (i) we compare the wRMSE of the mean of the uPLT distribution µ(uPLT ), as we did for uni-modal webpages,

(ii) we examine the three modes sorted by their mass M mass = [M major , M middle ,

M minor ] (M major is the mode with the largest mass of the distribution), and (iii) by

occurring time M time = [M first , M second , M third ] (M first is the earliest).
The results summarized in Tab. show that TTFP and TTI better approximate M major and, not unexpectedly, given the duality shown in Sec. . . , M first . On the other hand, AATF and PLT better approximate M middle , M minor , M second and M third . The former suggests that, to enhance the uPLT analysis, measuring and optimizing the last updates, usually achieved with PLT, is less relevant with respect to the earlier ones, e. g., TTI and TTFP. The latter instead con rms that the users choosing a late uPLT agree on a page to be loaded close to the last two page tracking events. It is also an interesting validation to note that, on uni-modal pages, PLT better matches µ(uPLT )

whereas TTI does that for multi-modal ones.

. Only few among recent works highlighted the existence of possible multi-modal behaviors for the uPLT. However, none of them deepened the study of the uPLT multi-modality or further explored the existence of these underlying di erent user behaviors, by carrying out the analysis of user feedback under this angle.

A summary of closely related work is reported in Tab. , where we distinguish for each study whether its authors identify or mention the multi-modal trait of uPLT ("uPLT multi-modality") or if they analyze that the uPLT is insu ciently captured by singlevalued objective metrics ("Metrics evaluation"). For the sake of comparison with our work, we report when available, the experimental settings and the size of the measurements (amount of users and webpages involved). Speci cally, we note that previous studies [ , , ] observe that uPLT does not match PLT, while Gao et al. [ ] nd that, more generally, "commonly used navigation metrics such as onLoad and TTFB fail to represent majority human perception". We note that although these works remarked either the multi-modality of uPLT or the di culty in mapping uPLT to single Web QoE metrics, their focus was not on characterizing the uPLT multi-modal nature. This con rms that the main hypothesis of our work is in line with the recent empirical observations in Web QoE modeling.

In this chapter, we went beyond related work by (i) evaluating the fraction of unimodal versus multi-modal pages according to a rigorous statistical test, (ii) thoroughly characterizing the di erent uPLT modes, and nally (iii) mapping between the di erent uPLT modes and the Web QoE metrics proposed in the literature. Speci cally, our analysis shows that (i) the uPLT distribution is uni-modal for approximately half of the webpages in our dataset, for which a simple PLT indicator (measured via the browser onLoad event) is a good estimator of user perception. We also show that, among classical indicators of webpage complexity, the number of objects and the number of images are good indicators for uPLT modality. We then show that (ii) multi-modal webpages are, in practice, never characterized by more than three modes. The most prevalent mode represents no less than 40% of users (69% on average, 72% median) in our dataset. We also observe that the earliest and most popular modes tend to match.

Finally, we demonstrate that (iii) we can approximate the earliest and most popular mode by TTFP and TTI, whereas metrics such as ATF and PLT better approximate the other modes. These ndings can be summarized in the following rule of thumb for measuring Web QoE using existing metrics. On the one hand, given that user browsing statistics are likely to exhibit multi-modality, one metric is generally not su cient to faithfully capture user perception. On the other hand, the whole spectrum of user perception seems to be captured by relatively few user modes, so that a small number of metrics are good at capturing uni-modal (e. g., where PLT or AATF will su ce) as well as multi-modal behavior (e. g., where additionally TTI should be measured for increased representativeness).

. In this chapter, we have asked a very simple but yet important and challenging question: to which extent users agree on a single time for when a page is loaded? This question is important because, traditionally, Web quality metrics (e. g., PLT and SpeedIndex) are conceived to produce a unique time indicator, implicitly assuming that user opinions would statistically converge to a single value. This question is also challenging, because of the sheer size of the Web coupled with the complexity to collect and understand user opinions. We show that for around half of the webpages considered, the uPLT distribution is multi-modal and that instead, for simple webpages users more likely agree on a single uPLT. We point out our results are representative (as per the strati ed sampling Actually, given the sheer size and heterogeneity of the Web, one could expect these models to be bias-free. However, already before the explosion of transformer-based LMs, a variety of biases have been identi ed in standard word embeddings [ , ]. Recently, some e ort has been devoted to highlight the presence of possible biases encoded by transformer-based LMs along gender, race, ethnicity, and disability status. Yet, whereas the study of such biases is commonly tackled via sentiment analysis and named entity recognition tasks, in this work we take a di erent approach. Inspired by the frequent scenario occurring in conversational systems, where a word could be unheard or unrecognized by the Automatic Speech Recognition system and would therefore need to be predicted, we measure how token predictions change based on their context. Similar to prior work [ ] considering social biases in BERT, we assess the bias for a target token by directly querying the underlying Masked Language model.

In this work we focus on the study of potential bias towards English dialects spoken by underrepresented and historically discriminated groups, such as African American English (AAE). Particularly, AAE slightly di ers from mainstream English, also known as Standard American English (SAE). In linguistics, these two variants are regarded as two di erent languages because highly structured with their own phonological, syntac- For these reasons, we set out to measure the robustness and the quality of transformerbased LMs in the prediction of missed words when the input is either SAE or AAE. Here we question whether the heterogeneity of Web content, over which the LMs have been trained, guarantees diversity. We resort to two renowned corpora of spoken SAE and AAE and evaluate the LMs in a Masked Language Modeling (MLM) task. This is a llin-the-blank task, where we mask and predict a token simulating its absence in every utterance. We next de ne two metrics to compare the likelihood that the model assigns to the predicted token and to the actual masked one, that we use as a proxy of quality and fairness for the model itself.

tic
Speci cally, we rigorously quantify the model bias and nd that BERT, in both its cased and uncased variants, exposes a non-negligible bias towards SAE (up to 21% more accurate results with respect to AAE). Surprisingly we nd this bias to be reversed for RoBERTa and BART models. We additionally observe distilled variants of these LMs to be fairer with respect to their teachers. Finally, our analysis reveals how most of the bias resides in the AAE structural di erences, and identi es the particles, the pronouns and the adpositions as principal parts of speech sources of bias.

This chapter is organized as follows. After overviewing the related work (Section . ), we present the corpora and the methodology to measure of the bias (Section . ). In Section . we show the results of the LMs when operated to predict the tokens from AAE and SAE, illustrating that the metrics we de ne reveal a bias. Finally, we discuss our ndings and possible directions (Section . ). DistilBERT and DistilRoBERTa, including both their cased and uncased versions.

These works have proven that the large dimension of the training datasets for stateof-the-art LMs is not synonymous of diversity and, as a consequence, of inclusion [ ].

Therefore, in this regard, our analysis is essential to provide a framework to assess, reveal and counteract the existing biases, which we hope will contribute in enriching the scienti c community knowledge on this domain.

. Spoken language tends to have incomplete sentences, spontaneous self-corrections and interruptions and its register is more of an informal one, while written language is typically more structured and pre-planned. The features of spoken language are of particular interest for studying conversational systems.

Hence, to capture and provide an accurate and comprehensive account of societal biases embedded in state-of-the-art LMs, we leverage two corpora of spoken English.

These are widely used by the linguists because considered a fair representation of their spoken language. We note that, while this work is not the rst in studying the presence of societal biases, to the best of our knowledge, this is the rst to provide a thorough characterization of it for AAE, across di erent models tested on a MLM task. In this case, the use of the MLM task would replicate the often occuring scenario where a word could be unrecognized by the Automatic Speech Recognition system which would normally have to infer it given the surrounding context. We summarize LMs performance by means of statistical metrics, which are used to characterize both the bias and the quality of the models.

.

. Corpora for Spoken English

For SAE, we leverage the Santa Barbara Corpus of Spoken American English (SBC-SAE) [ ], which has been already widely adopted for di erent applications, as the assessment of political risk faced by U.S. rms [ ], the measure of grammatical convergence in bilingual individuals [ ] and the exploration of new-topic utterances in

naturally occurring dialogues [ ].
The SBCSAE is the only existing large-scale corpus of naturally occurring spoken interactions from people with di erent regional origins in USA. It includes conversations from a wide variety of people, di ering in gender, occupation and social background, recorded in various real everyday life situations. All the audio recordings are complemented with their transcribed counterparts, which are the ones we use in this work.

The fact that SBCSAE consists of speakers from several regional origins prevents us from crafting the results and unintentionally inducing a bias by comparing AAE with an academic version of SAE, which is instead rather di erent from the commonly spoken English and, hence, far from the purpose of this work. Therefore, we lter out Hispanic and African American speakers ( AAE utterances, a negligible number with respect to the size of the corpus) and obtain a corpus of SAE language.

For AAE, we leverage the Corpus of Regional African American Language (CORAAL) [ ],

which also provides the audio recordings along with their time-aligned orthographic transcription, of particular interest for this work. CORAAL includes sociolinguistic interviews for over a million words. It is periodically updated and is the only publicly available corpus of AAE. As such, it has been used in literature for a plethora of tasks, ranging from dialect speci c speech recognition [ ] to cross-language transfer learn-

ing [ ].
In this work, we only focus on the CORAAL:DCB portion, since it is the one comprising the most recent interviews (carried out between and ) and the largest amount of data (more than k words). It includes conversations from speakers raised in Washington DC, a city with a long-standing African American population.

For each corpus we de ne U = {u 1 , u 2 , ..., u n } as the set of all the available utterances, and T = {t 1 , t 2 , ..., t n } as the set of all terms (unique words). Since we perform an utterance-level analysis, we rst lter out noise. Particularly, we discard both short utterances (composed by just one or two words) and very long ones (greater than words). Therefore, we only keep utterances having a number of words ranging from to . in terms of total number of words L and terms |T|, we can see that, after the ltering, the average utterance length u is very similar (∼ 8 words per utterance).

.

. Bias in Masked Language Modeling

In order to measure the bias in LMs we perform a MLM task. We leverage the transformer- These LMs have all been pre-trained using a MLM objective, which consists in randomly masking % of the tokens using a special [MASK] token. Note that these models are trained on di erent corpora, summarized in Tab. . In practice, they have been instructed to predict a masked token, referred to as a [MASK], given the surrounding context of the sentence.

Therefore, by directly querying the underlying MLM in each LM, we simulate the typical scenario where a conversational system has to infer a missed word in an utterance.

original utterance (u) And I be okay with it . u with w 1 masked [MASK] I be okay with it .

u with w 2 masked And [MASK] be okay with it .

• • •

u with w 7 masked And I be okay with it [MASK] Table : Example showing the masked token experiment.

Speci cally, we encode each utterance of the two corpora with the tokenizer of the LM considered, then, in turn, we mask each word w mask and nally predict it by feeding the model with only a context of tokens surrounding the masked one w mask . Tab.

shows an example, illustrating how the experiment is carried on: (i) we let the LM encode the original utterance u (the one reported in the table has a length lower than tokens so there is no need for the window), (ii) we mask and predict the rst token w 1 , (iii) we iteratively repeat this process until the last token of the utterance is masked.

The LM provides for each run a list of possible terms to ll-in-the-blank. In this vocabulary set (T) we select the predicted term t p having the highest probability P(t p |c) and, as such, ranking rst in the list ρ(t p |c) = 1, where c is the context surrounding t p and ρ is the rank of t|c provided by the model. In this notation, a word w is a term t in a context c (t|c). We next retrieve from the vocabulary of possible terms T the corresponding probability P(t m |c) and the rank ρ(t m |c) for the actual masked token t m . The latter provides a measure of how likely the LM will choose t m as a candidate token to replace the masked one w mask . It is then natural to employ the probabilities di erence ∆P(t|c)

as a proxy of the quality of the prediction for a single token, so de ned:

∆P(t|c) = P(t p |c) -P(t m |c) = ∆P(w). ( )
We further de ne for each token t|c the Complementary Reciprocal Rank (CRR) as:

CRR(t|c) = 1 -ρ(t m |c) -1 = CRR(w). ( )
Note that this is the di erence between the reciprocal rank (RR) of the predicted token, which is always equal to (ρ(t p |c) -1 = 1), and the RR of the masked token.

We then de ne the probability di erence for an utterance by averaging the probability di erence for each token in the utterance:

∆P(u) = 1 u w∈u ∆P(w), ( )
with u being the length of the utterance in terms of tokens. Similarly, we de ne the CRR for an utterance as:

CRR(u) = 1 u w∈u CRR(w).
( )

Note that the metrics based on the ranks ρ(t|c) generated by the LMs are necessary to fully capture the bias embedded in the models, as the ∆P(t|c) alone could be insucient. This because, the ∆P(t|c) strongly depends on how the LM assigns the probability.

Indeed, the probability distribution of P(t|c) could be more uniform, and consequently would lead, on average, to a smaller ∆P(t|c), or more skewed, causing instead larger di erences ∆P(t|c). Instead, this e ect is not present in CRR that remains una ected by such di erences in the output probability distribution of P(t|c).

.

In this section, we rst provide an accurate overview of the measured LMs fairness, and then further analyze the discovered biases from di erent viewpoints. We show how they varies when we take into account the syntactical, grammatical, and lexical patterns typical of AAE language rst, and then, when we slice the corpus based on parts of speech.

.

. Measuring the Bias of LMs

As described in Section . , we test the fairness of transformer-based LMs by running experiments in a MLM setting. As aforementioned, we use ∆P and CRR as metrics for measuring the quality and the fairness of the models towards the two investigated languages. We are interested in observing the expected behavior of the LMs with respect to each utterance, therefore we consider an aggregate measure of the metrics on a perutterance level.

Tab. reports an overview of the results of ∆P(u) and CRR(u). After having assessed that the di erence between the means of AAE and SAE for both ∆P(u) and CRR(u) with a Welch's t-test [ ] is signi cant (p-value < 0.05), we measure their e ect size using the Cohen's d [ ]. This is reported in the last two columns of Tab. . According to

Cohen's classi cation there is a small e ect for both the metrics, and a medium e ect for BART on ∆P(u) (d> 0.5). We summarize the quality of the prediction in the corpora by means of two error measures. We report the Mean Absolute Error (MAE) for each of the two distributions:

MAE(∆P(u)) = 1 |U| u∈U |∆P(u)|, ( ) MAE(CRR(u)) = 1 |U| u∈U |CRR(u)|. ( )
We also report the Mean Squared Error (MSE), de ned as:

MSE(∆P(u)) = 1 |U| u∈U ∆P(u) 2 , ( ) MSE(CRR(u)) = 1 |U| u∈U CRR(u) 2 . ( )
Indeed, these error measures can be used to quantify the quality of the predicted terms. which leads to the highest relative bias. Speci cally, notice that the MAE(∆P(u)) for SAE is more than 20% lower than AAE, % lower for the MAE(CRR(u)), % for the MSE(∆P(u)) and % for the MSE(CRR(u)).

Second, DistilBERT, in both its cased and uncased avors, and DistilRoBERTa, are the models which perform better as regards the average probability di erence ∆P(u). that such LMs sometimes exceed the performance of the original ones. However, as mentioned in Sec. . , it is crucial to also look at the CRR(u), since a better behavior in terms of ∆P(u) could in practice just be tied to the fact that the model generates more uniformly distributed probabilities P(t|c) with respect to the others.

Finally, we observe that BART, despite leading to a decent quality of the prediction for AAE (MAE(∆P(u)) and MSE(∆P(u)) are lower than BERT), shows an opposite trend with respect to BERT and DistilBERT. This reverse unexpected bias towards AAE is also introduced by RoBERTa and DistilRoBERTa. This is somewhat surprising and could probably be ascribable to the type of datasets they have been trained on. In- • You don't need nothing but you.

• You don't need anything but you. • I wasn't no lifeguard cause I couldn't swim.

• I wasn't a lifeguard because I couldn't swim. • Don't never try to chase another person happiness.

• Never try to chase another person's happiness. • I don't know nobody over there no more.

• I don't know anyone over there anymore.

Copula be ( . %)

• And I be okay with it.

• And I am okay with it.

• It depends on where you going to.

• It depends on where you are going to.

• All of my friends was from like DC. • All of my friends were from DC.

• Okay, we having a baby.

• Okay, we are having a baby.

Contractions ( . %)

• I'm'a ask you.

• I'm going to ask you.

• I ain't coming home.

• I'm not coming home.

• something gonna happen.

• something is going to happen. • you gonna be there for a couple of hours.

• you will be there for a couple of hours.

Table : A sample of AAE utterances selected based on their syntactical features and their translations to SAE. In brackets the prevalence of the feature over the utterances in the AAE corpus.

.

. Bias on AAE Features

We next investigate how results change when we acknowledge the lexical, syntactical, morphological and also phonological rules of AAE. Following AAE grammar [ ], we choose to focus on three major syntactical features: (i) the use of double negatives, (ii) the di erent usage of copula be and, nally, (iii) the contractions of words and groups of words.

As for (i), we search for the close presence of multiple forms of grammatical negation (which in Standard English are instead understood to resolve to a positive) in all the utterances of the AAE corpus, and nd, that 0.7% of the utterances contains such a feature. Concerning (ii), we select the AAE utterances exhibiting the use of the aspectual be verb, typically used to denote habitual or iterative meaning (e. g., I be okay with it in Tab. ). Additionally, we also lter on utterances with the verb tense in the -ing form where the copula is either omitted (e. g., It depends on where you going to in Tab. ) or left at the base form (e. g., they be getting mad in Tab. ), for a total of 2.8% of utterances. Finally, for (iii) we include those utterances containing not-standard contractions, e. g., I'm'a, ain't or omitting the auxiliary before gonna, e. g., something gonna happen in Tab. . We do not include contractions which are popular in SAE, as wanna, won't, aren't, etc. We obtain 4.6% of the utterances in this class. After having properly ltered the utterances corresponding to the speci c grammar patterns, we carefully manually validate our selection, by random picking and inspecting % of them. We check that the 1% random sampled utterances are actually satisfying the criteria we were looking for.

. MAE MSE ∆P(u) CRR(u) ∆P(u) CRR(u) Model AAE AAE ∆[%] d AAE AAE ∆[%] d AAE AAE ∆[%] d AAE AAE ∆[%] d
From this manual labeling we double check our selection strategies based on syntactical rules and nd that for both the cases these are % accurate.

Next, we randomly choose utterances from each AAE case and build a ground truth by translating the AAE utterances into a version compliant to SAE, that we de ne as AAE . We keep the translation process as neutral as possible, by preserving the standard o cially recognized contractions and by only adjusting the selected grammar rules.

Tab. reports some examples of the utterances extracted from each AAE grammar case bucket and the corresponding translated ones.

Finally, we repeat the MLM experiments, as described in Section . , on these translated utterances AAE and measure the bias. We report the results in Tab. . Ac- cording to Cohen's classi cation there is a prevalent medium e ect for both the metrics, with the exception of MSE(CRR(u)) for the copula class, where it is large.

At a rst glance, we observe that the errors for the set of the AAE utterances in the copula class are larger than both the other two classes and the whole AAE corpus (reported in Tab. ). More in general, we observe that, on average, both the three classes, and therefore, all the AAE utterances, come with a less accurate average prediction with respect to the overall AAE corpus. We observe instead that the translated utterances AAE are better predicted with respect to AAE surprisingly for all the seven LMs.

Notably, we observe that for the translated utterances in the double negative class, the four metrics are always smaller (and hence sign of better performance) than those measured for the SAE corpus. This is somewhat unexpected since we observed for RoBERTa and BART an opposite bias on SAE. However, we remind that the SAE corpus, SBC-SAE, is made up of conversations collected from people with di erent regional origins. Consequently, despite the e ort we make in trying not to excessively standardize the utterances during the translation process, we could be generating sentences which are free from regional bias and consequently "cleaner" than those found in the SAE corpus.

. . Bias on Part-of-Speech Finally, we investigate to which extent the POS tags are tied to the measured bias towards AAE or SAE. To produce these results, we preliminary tag the tokens independently generated by each language model with the NLTK [ ] POS-tagger. Next, we group by the main tags of the universal tagset [ ] and compute the MAE and the MSE on the term-level measurements ∆P(t) and CRR(t).

Indeed, rather than averaging across the tokens in one utterance, we consider all the terms t belonging to a given POS tag. Tab. reports the results obtained for the top-POS featuring the highest cumulative bias, computed by summing the absolute bias |∆[%]| introduced by each LM and measured with the MAE(CRR(t)): the particles (e. g., to, up, out, etc.), the pronouns (e. g., you, it, my, etc.) and the adpositions (e. g., like, of, with, etc.). The results for the rest of the POS are reported in Tab. . In order to trust the results of the POS-tagger we manually check the correctness of tokens for each class and language. We nd that the accuracy is 100% for the pronouns, 99% for the adpositions and 92% for the particles. Also in this case, we measure the e ect for both the metrics, and nd that, according to the -grade Cohen's classi cation scale, it is very small. Interestingly, for the particles class, one can notice the same pattern reported in Tab. .

Particularly, DistilBERT cased is the LM which performs better in terms of ∆P(t) and, DistilRoBERTa the one that leads to the lowest bias. Conversely, BERT is the model that shows the highest bias towards SAE: it is up to 29% more accurate with respect to AAE for MAE(∆P(t)). BART presents the opposite largest bias in favor of AAE: 23% (18%) more for the MAE of ∆P(t) (CRR(t)) on the particles class. It is also interesting to note that DistilBERT also at a token-level analysis presents better values for ∆P(t) rather than CRR(t).

Quite surprisingly, we discover a bias presented by all the tested LMs towards AAE in the pronouns class. This holds for both the ∆P(t) and the CRR(t) and is revealed with both the error measures, with the exception of BERT and DistilBERT cased for the MSE of ∆P(t). This result deserves further investigation.

.

In this chapter we proposed a methodology for the evaluation of the fairness of transformerbased language models. We assess and analyze the bias for two corpora, one of the spoken SAE and one of the AAE. By directly querying the underlying MLM in seven LMs,

we study the quality and the bias of their predictions under several angles. The focus is then narrowed down to the partitioning of the measured bias, by selecting the AAE corpus according to its language features. Additionally, we assess the impact of the POS tags on the found bias.

In a nutshell, results presented in this chapter suggest that di erent models embed diverse biases. Particularly, the most popular state-of-the-art LMs, namely BERT and DistilBERT, show a non-negligible bias towards SAE (quality of the predictions up to 21% more accurate than AAE). Instead, BART, RoBERTa and DistilRoBERTa exhibit an opposite bias. Our experiments reveal also that the distilled variants of BERT and RoBERTa are the fairest among the seven tested LMs.

Yet, despite this work provides a rst insightful snapshot of linguistic bias embedded in di erent LMs, it opens a number of research questions. First, can fairer prediction outcomes be achieved with an ensemble learner of LMs embedding opposite biases, as, for instance, BERT cased and BART? Second, our results give insights on how the bias could be consistently mitigated with more inclusive corpora, by taking into account AAE features. Finally, a special care could be put in the analysis of the distilled LMs, narrowing the gap on the causes which lead them to fairer predictions with respect to their teacher models, with a particular emphasis on the Web-based corpora used for training.

. Table : MAE and MSE of ∆P(t) and CRR(t) measured on AAE and SAE corpora: results obtained through the ll-in-the-blank task with di erent language models, averaging token predictions for each POS class. † signi es that the AAE and SAE expectations are statistically signi cant according to the Welch's two-tailed t-test (p-value < 0.05). The column d contains their e ect size computed according to the Cohen's d.

MAE MSE ∆P(t) CRR(t) ∆P(t) CRR(t) Model AAE SAE ∆[%] d AAE SAE ∆[%] d AAE SAE ∆[%] d AAE SAE ∆[%] d Verbs [ k (AAE) k (SAE) terms], |∆[%]| =
T Given this context, and in particular the emergence of new IP-ID behaviors, it is important to de ne methods to classify them, as well as using these methods to quantify the prevalence of IP-ID implementation in the current Internet. To summarize our main contributions:

H E M O D E L S A N D T H E M A C H I N E S

Contents

• we design and implement a lightweight framework to classify the full range of IP-ID behaviors, based on a handful of ICMP packets;

• we carefully validate our method against two datasets comprising the replies from about 1, 855 sample hosts, chosen in di erent manners, for which we build a ground-truth by manual inspection and against multiple synthetic datasets, tailormade to test robustness against various forms of shortfalls;

• we apply the methodology to an Internet-wide campaign, where we classify one alive target per each routable / subnet, gathering a full blown picture of the IP-ID adoption in the wild.

Speci cally, whereas the global counter (18% of occurrencies in our measurement) implementation was the most common a decade ago [ ], we nd that other behaviors (constant 34% and local counter 39%) are now prevalent. We also nd that security recommendations expressed in [ ] are rarely followed (random, 2%). Finally, our census quanti es a non marginal number of hosts (7%) showing evidence of a range of behaviors, that can be traced to poor or non-standard implementations (e. g., bogus endianness, non-standard increments) or network-level techniques (e. g., load balancing, or exogenous tra c intermingled to our probes confusing the classi er). To make our ndings useful to a larger extent, we make all our dataset and results available at [ ].

This chapter is structured as follows. After overviewing the related work (Section . ), we describe the methodology and illustrates the work ow and the datasets involved (Section . ). We next show the performance of the supervised classi cation approach chosen in the following order: system validation, robustness assessment and probing overhead analysis (Section . ). Then, we present the results of the classi er when operated in the wild and put in perspective our ndings with those of the previous works (Section . ).

Finally, we summarize the main outcomes (Section . ).

.

. . Normative reference

The IP identi cation (IP-ID) eld identi es the unique fragments of a packet and it is used to handle the re-assembling process. First documented in the early s by RFC [ ] its use has been updated in several RFCs [ , , , this was a not legal implementation as the eld was supposed to be unique. random IP-IDs are extracted according to a pseudo-random number generator. Finally, a special mention is worth for the class of (v, vi) odd IP-ID behaviors, that are not systematically documented in the literature and that arise for several reasons (including bugs, miscon guration, non-standard increments, unforeseen interaction with other network apparatuses, etc.) and for which we report two di erent samples occurring in real experiments.

.

. Active probing

To gather the above described sequences, our measurement technique relies on active probing. We craft a tool able to send and receive ICMP packets, running at two vantage points (VP) with public IP addresses in our campus network. Speci cally, we send a stream of N ICMP echo requests packets in a back-to-back fashion, which forces the target machine to generate consecutive ICMP echo replies: thus, assuming for the time being that no packet were lost, we gather a stream of N IP-IDs samples for that target. suggest that sending packets back-to-back is a good strategy, although we do not feel results to be conclusive for all the devices available in the network (e. g., router, setup, shaper, etc.). However, even in case the reality was not as nice as our experimental lab results (which is likely to be the case), at the same time this a ects at most some of the odd behaviors, which already are a tiny (7%) fraction of the overall cases. Indeed, it is very unlikely that the amount of real tra c is so perfectly varying between probes to erroneously confuse a classi er to believe that a global sequence is a random one just due to exogenous tra c. Very high information entropy of those sequences is not a sideproduct of some variable tra c, but truly coming from a random number generator (it is pretty well known that is hard to generate good pseudo-random sequences, and the arrival rate is surely not a source of perfect entropy).

To overcome reordering, packet loss and duplication events, we additionally control the sequence number in the stream of generated probe packets.

Feature Constant

Local Global Random Odd H(x) log 2 N 2 log 2 N 2 log 2 N 2 - H(s) log 2 N log 2 N log 2 N - H(x ) log 2 N 2 - H(s ) log 2 N - E[x ] (2 16 -1) 2 - σ x (N 2 -4) 48
(N 2 -4) 12
(2 16 -1) √ 12 - σ s (2 16 -1) √ 12 (N 2 -1) 12
(2 16 -1) 2 - σ x (2 16 -1) √ 12 - σ s |x 1 -y 1 -1 2 | (2 16 -1) √ 12
-Table : Tabulated expected values for selected features

. . Features De nition

As anticipated, to build a robust classi er we need to manually de ne a set of tailormade features able to discriminate among the di erent IP-IDs implementations. The experiment and the measurements can be formalised as follows: we send N packets to a given target t, with the source address eld alternating between consecutive requests, whose replies are sent back to our two vantage points x and y: we indicate with s the aggregated sequence comprising the N IP-IDs sent back by t, as we receive it at the edge of our network . By abuse of language, we indicate with x and y the sub-sequences (each of length N/2) of IP-IDs, sent back by t and received by the homonymous host.

From these sequences x, y and s we further construct derivative series x , y and s by computing the discrete di erences between consecutive IP-IDs (i. e., x i = x i -x i-1 ).

We summarize these series with few scalar features by computing the rst E

[X] = 1 N • N i x i and second moments of the IP-ID series, σ = E[X 2 ] -(E[X]
) 2 as well as their information entropy, de ned as the expected value of the information content H(X) = E I(X) = -N i p i log 2 p i where p i is the provability that the discrete random variable X takes the x i value.

Speci cally, we consider the mean E[X] of the derivative series x and y , the entropy H(X) and the standard deviation of s,x and y and of their derivatives s , x and y .

Actually, for each feature we can derive an expected value in the ideal case (so that no Notice that packet losses and reordering may let us receive less than N packets, or receive packets in a slight di erent order than what sent by the target. Sequences from well behaving hosts that have no software bug or malicious behavior, and that are neither a ected by losses nor reordering expected values is reported for the odd class) that we summarize in Tab. . For the sake of brevity, we report in Tab.

only once the expectations of the features of the subsequences x and y, given that they are conceptually equivalent. Intuitively, we expect the mean of the constant sequence to be unknown, but that of its derivative to be null.

Similarly, the derivative of a global counter would have a value of ( ) for the aggregate sequence s (subsequences x and y). The entropy of the sequence is expected to increase from the minimum of a constant sequence equal to H(X) = -1 log 2 (1) = 0 . Accordingly, in the global implementation, the sequence s is made up of two not-overlapping sequences, leading to an expected maximum entropy of H(s global ) = log 2 N. Di erently, in the local implementation this is true only when the two counters do not overlap, otherwise this remains only an upper bound. A similar observation can be done for the entropy expectations for the random sequences, in which the presence of duplicate values would reduce the entropy. For the local implementation, the sequences x , y , derivatives of two independent counters, are constant thus the entropy, as said, is expected to be 0.

to the maximum of H(X) = -N • 1 N log 2 1 N = log 2 N
On the other hand, the derivative s of the aggregate sequence s is made up of two alternating values, corresponding to the two o sets:

s local (n) =      θ 1 = y 1 -x 0 if n even θ 2 = x 2 -y 1 if n odd ( )
Both θ 1 and θ 2 are repeated for N 2 times, so each one occurs with a probability of 1 2 .

The entropy becomes H(s local ) = -2 • 1 2 log 2 1 2 = 1 Conversely, being the expected derivative sequence of a global counter always equal to s global = 1, as a result the entropy becomes H(s global ) = 0.

Similarly, the other expectation values can be easily derived by analogy.

. . Datasets

In this work, we collect four di erent datasets, that we use in the di erent stages of the work alternatively to make the classi er learn the classi cation function, i. e., as training dataset, and to evaluate performances as testing dataset. ]). We send a batch of N = 100 backto-back probe packets to each target, but otherwise probe at a low average rate, so that we complete a / census in about days. Fig.

shows the empirical cumulative distribution function (ECDF) of the received packets at our VPs. We observe that we receive almost all the replies from most of the targets: the 90% (80%) of the targets answer to more than (all) packets per each host, corresponding to a 20% (0%) loss scenario. A large plateau in the CDF also indicates that the distribution is bi-modal, i. e., the remaining hosts generally reply with very few packets (e. g., 10 or less per each VP or over 90% loss rate). This suggests that future campaigns could be safely conducted with a smaller N < N. To provide accurate classi cation results, in light of our robustness analysis done with synthetic dataset and whose results are shown in Sec. . . , we limit our attention to the 2, 588, 148 hosts for which we have received at least N = 80 packets.

Ground Truth G and G :

The second real dataset is G, made of IP-ID sequences for which we manually construct a ground truth. For this purpose, we extract the replies from a subset of targets of L which satisfy some pre-established requirements. We include in this dataset only the , hosts from which we receive % of the replies, and perform the manual inspection of each of the sequences. We repeat the process twice, with two very di erent choices of the ground-truth datasets: G sampled uniformly from the hitlist paying attention to guarantee class balance and G where about 75% samples belong to the same IP/ subnet. Interestingly, when performing the manual labelling, we nd a small but non marginal fraction (about 7%) of sequences that are hard to classify: a deeper investigation reveals these odd behaviors to be due to a variety of reasons -including A summary of all the datasets with their description and properties is reported in Tab. .

. From the values tabulated in Tab , we expect classi ers that use this set of features to be able to fully discriminate the set of IP-ID well-de ned behaviors under ideal conditions. However, as we shall see, unexpected behavior may arise in the Internet, due to a variety of reasons, which are hard to capture in general. We thus opt for a supervised classi cation approach, which allows to learn a predictive model with decision trees (DTs), based on the above features. Additionally, we investigate to what extent the classi er is robust against losses and reordering, and nally assess the minimum number of samples N needed to achieve a reliable classi cation.

.

. Classi cation accuracy and validation

We rst train and validate our classi er using the the real dataset G of IP-ID sequences for which we have manually constructed a ground truth. Note that, for the moment we train the classi er only over the dataset G, but later we will show the independence of the model from this choice.

We assess the classi cation accuracy over G with a -fold cross-

. . Robustness

Before operating the classi er on the data collected in the wild, we study its robustness both to packet losses and to the number of packets employed in the experiments.

It is fundamental to test the robustness of the features to losses in a controlled scenario, in order to emulate the real measurements, in which events such as packet losses or outof-order arrivals are not so rare.

Robustness to Losses:

For the previously shown six features we evaluate their values in the lossy synthetic S lossy sequences and tabulate the results averaged over the dataset, respecting the IP-ID and loss type partitioning, in order to compare with the ones evaluated for the lossless sequences in S ideal . In Tab. we report those values evaluated for the simulated local implementations. We compare the features evaluated for the lossless dataset S ideal with those of the lossy S lossy with uniform random, hole, extremal and equidistant losses.

As a whole, results obtained with the synthetic dataset S lossy do not signi cantly diverge from the ones obtained with S ideal , proving the strength of the features and their robustness to change and alteration of the original sequences. Speci cally, H(s), which turns out to be the most important feature, as shown in Fig. , does not vary in presence of any kind of losses, while H(x ) can vary more depending on the avour of the loss.

Given these results, we next assess the robustness of the classi er against packet losses, which may introduce distortion in the features. Since, as previously described, the expected values in the ideal conditions are signi cantly apart, we expect the classier to be resilient to a high degree of losses. Without loss of generality, we consider an extreme case where only 80 out of 100 samples are correctly received (i. e., a 20% loss rate) by exploiting the lossy synthetic dataset S lossy .

We want to assess the accuracy of the previously validated model, i. e., the one trained on the real lossless dataset G over S lossy . Results of these experiments are reported in . The last step of the analysis consists in using the previously trained classi er over G to classify the IP-ID behaviors present in the dataset L. In this section, we rst show the results of the classi cation and, then, we put them in perspective with those obtained Notice that even in the extreme case with as few as N = 2 packets, random and constant classi cation are correctly labeled, whereas the remaining global vs local cannot be discriminated, yielding to . accuracy in the G set. Finally, we perform a spatial analysis and we deepen the analysis of the odd behaviors.

.

. Longitudinal Comparison (over the years)

We apply our classi er in the wild, speci cally on the previously introduced dataset L (Sec. . . ), made with the data collected through a large scale Internet measurement campaign. We observe that, while our classi er is able to perform a very accurate classication even with few samples, we need to deal with loss rates, which is unknown a priori.

Hence, even though our probing overhead analysis in Sec. . . revealed high accuracy for few number of samples, we prefer for the time being to use a simple and conservative approach and select N = 100 samples, being very accurate also in presence of very high loss rates. We apply the classi cation to batches of , hosts, and for each class c, we compute the relative breakdown of the class in that batch nc = n c / i n i , evaluating the con dence intervals of nc over the di erent batches. Results are reported in Fig.

( .

. Sensitivity Analysis

Training Set Choice: In order to prove the indipendence of the results from the choice of the training dataset we exploit the second manually validated dataset G , which satises the previously described requirements and it is purposely biased, as it contains 75%

of the samples from the same / , which is something not desiderable from a IP coverage point of view.

We This might be due again to the lack of fundamental information to correctly classify those behaviors. Conversely, constant behaviors are easy to be identi ed even with a bunch of few packets. These results show that to correctly detect random and odd IP-ID classes more care is needed: more data might be required to spot the proper behavior of the series. Whilst, for the other classes, few packets are more than enough to correctly classify them.

.

. Spatial analysis

Odd class: As already mentioned, during the manual labelling phase we discovered some targets setting the IP-ID in not-standard unexpected manners, which may be ascribable to di erent causes, and that we named as odd behaviors. In this section we try to investigate a bit more the odd class, trying to gure out whether we can re-map some of those IP addresses in other classes or not. The rst analysis we perform consist in The data collection block relies on an experimental testbed comprising one sender and two receivers, which collect the IP packets and, speci cally, the information related to the IP-ID eld. The sender sends a burst of packets, minimizing the impact of external tra c and purposely exploiting spoo ng to precisely alternate addresses in the sequence.

To classify the di erent IP-ID classes, we trained and validated di erent classi ers on datasets gathered from real measurements and additionally tested in the presence of controlled losses to assess its robustness. Training of the model required manual validation of thousands of sequences: during this phase, we also discovered some odd behaviour, not documented in any of the previous RFCs, and which may be attributed to di erent causes.

In instances where odd behaviour was previously reported, our classi er is the rst to automatically and correctly label such instances, making it easier to perform large-scale analysis over the Internet. Moreover, classi cation only requires a handful of packets, making the methodology extremely lightweight.

Experimental results show that the majority of hosts adopt local IP-IDs ( %) or a constant counter ( %) of which:

• A fraction of global counters ( %) is signi cantly lower than expected;

• A non-marginal number of hosts have an odd behaviour ( %);

• Random IP-IDs are only slightly more than an exception ( %).

. This outcome provides a picture of Internet-wide adoption of the di erent IP-ID implementations. Indeed, we gather that the % breakdown of the global implementation in is three times lower with respect to the % reported in [ ]. While the quantitative reduction is in line with the statistics reported by recent work that leverages global IP-ID behaviour to detect censorship in the Internet [ ], one could have expected the decrease in global implementation to be compensated by an increase of random IP-IDs, which is not the case.

Contributions: Our rst contribution is to devise an accurate, lightweight and robust classi er: accuracy of the classi er follows from a principled de nition of the statistical features used to succinctly describe the IP-ID sequence; robustness is a consequence of this choice, as features remains wide apart even under heavy losses.

Our second contribution is to carry on a manual investigation e ort for a moderate size dataset coming from real Internet measurements: this valuable ground truth allow us to adopt a supervised classi cation techniques to train a model able not only to detect well-de ned behaviors, but also to correctly recognize a wide range of odd behaviors.

Finally, all our datasets, including the testing with manual ground truth, as well as the results of our census, are publicly available at [ ]: we hope that the former can assist scientists to build and test new techniques for IP-ID classi cation, whereas the latter provides practitioners with readily usable lists of the hosts with global IP-ID implementations for their inference. This chapter presents some important results. First, we observe that the concerned Wikipedia users are consistently satis ed, and that not only user answers unexpectedly do not exhibit seasonality at circadian or weekly timescales but also they are not a ected by network-related events, which are typically inducing measurable delay changes.

Second, we nd evidence of spatial dependency across many of the collected features:

particularly, we observe that user scores are in uenced by user-level expertise and equipment as well as network and country-level characteristics.

Finally, we observe that supervised data-driven models of user experience, used to predict the user scores still falls short from attaining satisfactory performance in operational settings. This occurs despite when they include performance metrics considered to be the state-of-the art of Web QoE and even when reducing the variance and heterogeneity of the data.

.

. User Perceived Page Load Time in controlled experiments

In Chapter we focused instead on measuring Quality of Experience in controlled experiments. Particularly, we crowdsource the user perceived page load time, the time when a user considers a webpage to be loaded and ready to browse, on webpages via the Eyeorg platform. Through the Eyeorg's timeline experiment, participants are shown a video of webpage load and asked to scrub it until when (s)he considers the page to be ready. Similarly to Chapter , also in this work we gather both objective and subjective Web quality metrics. This chapter showed two main results.

First, we observe that half of the webpages involved in our study present a multimodal uPLT distribution and that, in practice, three modes are su cient to accurately describe the uPLT distribution.

Second, we nd that the number of images and the number of objects in a webpage can help in predicting the uPLT modality.

In spirit with the current trends toward research reproducibility, also this dataset is made publicly available. * * *

In the second part of the thesis we presented some results related to the interaction of machine learning models with (i) Web and (ii) machine-generated content.

. . The fairness of models trained on the Web

In Chapter we question the assumption that the Web sheer size and heterogeneity ensure the fairness of the models trained on Web-based content. Here, we propose a methodology for the evaluation of the fairness of state-of-the-art transformer-based language models. This chapter showed two main results, related to the fact that di erent models encode diverse biases when used for the prediction of terms of both Standard American English and African American English.

First, we observe that the fairness of large models, trained on huge amount of Webbased content, is unbalanced. Speci cally, the predictions done with BERT and Distil-BERT on Standard American English are up to 21% more accurate with respect to those done on African American English. We show also how instead BART, RoBERTa and DistilRoBERTa exhibit an opposite bias, favouring then African American English.

Second, results highlight that the distilled variants of BERT and RoBERTa, designed to be lighter and trained on a lower amount of data, are the fairest among the seven tested language models.

. . Supervised learning to infer machine-generated content

In Chapter we observe instead how state-of-the-art machine learning algorithms behave when they are trained on content generated by machines, i. e., the IP identi cation (IP-ID) eld of the IPv header. Despite being only minimally intrusive and fairly lightweight, the proposed technique is signi cantly accurate and unveils two main ndings, the rst one more related to the methodology, the second one more tied instead to the classi cation results.

From the methodology perspective, we nd that few scalar features and a simple classi er, as a decision tree, are enough to accurately predict the di erent IP-IDs implementations. This is in sharp contrast with the results shown in Chapter . Here the classi cation of objects with very pre-determined behaviors, depending on the speci c implementation of the OS, leads to very accurate From the point of view of the results, instead, the application of this technique to an Internet-wide census provides an updated view of the adoption of the di erent known IP-ID implementations in the wild. Particularly, the results of the census reveals that the global is no longer the most common IP-ID implementation and that, instead, other be-haviors, as local and constant, are present. This is particularly relevant, being the global implementation of invaluable help to infer a wealth of information concerning the network. Releasing all our datasets and results publicly, including two manually labeled ground truth datasets and a list of the approximate half million hosts with the IP-ID implementations, can make works relying on this IP-ID class still possible.

. There are several ways in which the results discussed in this thesis could be extended.

We next present some perspectives that could be accomplished as future work.

As concerns the Web QoE, di erent directions can be further explored. With regards to the experiment carried out in Chapter , we must point out that there exist other QoE in uence factors that we did not include in the analysis, such as content and context factors that are known to a ect user QoE. For instance, the sentiment linked to the topic and the content of the page or more informative indicators about the context in which the measurements are carried out, as the earlier user browsing experience, heavily impact QoE. However, they are hard to capture. Moreover, adding the knowledge of whether the rendered element is under the user gaze, using mouse-movements as a proxy of eye gaze activity, can help further re ning QoE metric in the spatial direction. Clearly, further research is needed on whether user-touch can be useful for similar purposes in case of mobile handsets.

As for the bias embedded in models trained on Web based corpora, di erent debiasing techniques can be thought and implemented. Besides traditional debiasing approaches, as those based on the loss function modi cation, research directions which rely on ensemble methods could be explored. Ensemble learning might lead to fairer prediction outcomes, by combining language models which embed opposite biases, as, for instance, BERT cased and BART in our analysis. Moreover, distilled language models need further investigations. Particularly, a special emphasis should be given on the study of the causes which lead them to have fairer predictions with respect to their teacher models. This could pose new research questions, such as the comparison of the cost-bene t analysis of large base models with respect to the smaller distilled counterparts. The time immediately before the user agent starts checking any relevant application caches.

dnsLookup 5 Time it took to resolve names (domainLookupEnd -domain-LookupStart).

secureConnectionStart 5 The time immediately before the user agent starts the handshake process to secure the current connection.

connectStart 5

The time immediately before the user agent start establishing the connection to the server to retrieve the document.

connectEnd 5

The time immediately after the user agent nishes establishing the connection to the server to retrieve the current document.

requestStart 5

The time immediately before the user agent starts requesting the current document from the server, or from relevant application caches or from local resources.

responseStart 5

The time immediately after the user agent receives the rst byte of the response from the server, or from relevant application caches or from local resources.

responseEnd 5

The time immediately after the user agent receives the last byte of the current document or immediately before the transport connection is closed, whichever comes rst.

loadEventStart 5

The time immediately before the load event of the current document is red.

loadEventEnd 5

The time when the load event of the current document is completed.

mediawikiLoadEnd (Mediawiki-speci c.) The time at which all ResourceLoader modules for this page have completed loading and executing.

domComplete 5

The time immediately before the user agent sets the current document readiness to "complete".

domInteractive 5 The time immediately before the user agent sets the current document readiness to "interactive".

  il valore dell'inchiostro e come impugnare la penna nel modo giusto. . . . . . . . . . . . . . . . . . . . . . . . . . . . . A B S T R A C T Today, a world without the Internet is unimaginable. By interconnecting billions of people worldwide and by o ering an uncountable number of services, it is now fully embedded in the modern society. Yet, despite technology evolution and development, its pervasiveness and heterogeneity still raise new challenges, such as security concerns, monitoring of the users' Quality of Experience (QoE), care for transparency and fairness.

  FigureRelationship between QoS at di erent layers and user QoE. . . .FigureOutline of the thesis. . . . . . . . . . . . . . . . . . . . . . . . . Figure Appearance of the Survey in the English Wikipedia (answer order is randomized). . . . . . . . . . . . . . . . . . . . . . . . . Figure Quantile-quantile plot of PLT statistics for di erent sets . . . . . Figure Aggregate statistics of navigation timing performance (TTI, RSI and PLT in the gure), conditioned by survey response. . . . . . Figure Annotation of major Wikipedia-related events occurred during the whole -months observation period. . . . . . . . . . . . . . Figure Temporal view: daily mean of PLT, TTI and RSI during the observation period. . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure Temporal view: breakdown of daily survey answers among positive, neutral and negative scores. . . . . . . . . . . . . . . . . .Figure Temporal view: absence of night/day seasonality of survey answers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure Temporal view: absence of weekday/weekend seasonality of survey answers. . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure Illustration of spatial breakdown of user scores across page, user and environment features obtained by conditioning each of them over di erent values and showing on the top x-axis the cardinality of samples for each bar. . . . . . . . . . . . . . .

Figure

  Figure ROC curves, obtained when averaging the results gathered with a -fold cross validation on B with the PA features set. . . . . . Figure Classi cation results, feature subsampling: performance obtained by limiting the (T) total features, (WWW) features, (PA) publicly available features, both with (PA w.o. ) and without (PA) outliers ltering. . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure

  Figure Classi cation results, dataset subsampling: performance obtained by restricting the attention to (a) Chrome-only browser, (b) Russian population (c) Android OS and (d) top-pages (and combinations thereof). . . . . . . . . . . . . . . . . . . . . . . . Figure Ranking of the features according to their SHAP values. . . . .

Figure uPLT distribution for

  Figure uPLT distribution for www.booking.com, highlighting the issue that users do not agree on a single time instant to identify completion of webpage rendering. . . . . . . . . . . . . . . . . . . . Figure Number of webpages per cluster. . . . . . . . . . . . . . . . . . . Figure Component weights for pages with multi-modal uPLT. . . . . . Figure Ranking of the features according to their SHAP values when predicting uni-modal pages. . . . . . . . . . . . . . . . . . . . .

Figure

  Figure |PLT -T T I| ECDF for uni/multi-modal pages. . . . . . . . . . . Figure The ECDF of the utterance length u for both AAE and SAE corpora. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure

  FigureThe di erence between the ECDFs of SAE and AAE for the ∆P(u) measure. When the values are greater than zero the LMs are more biased towards SAE, vice versa otherwise. . . . . . . .

Figure

  FigureThe di erence between the ECDFs of both AAE and SAE for the CRR(u) measure. When the values are greater than zero the LMs are more biased towards SAE, vice versa otherwise. . .

Figure

  Figure Illustration of Constant, Local, Global, Random and Odd sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure

  Figure Sensitivity analysis to external tra c: derivative of the sequence of IP-IDs x in the two di erent scenarios . . . . . . . . . . . . . Figure Internet campaign: ECDF of the number of packet replies . . . . Figure Manual Ground Truth: Normalized classes occurrences for the training datasets G and G . . . . . . . . . . . . . . . . . . . . .

Figure Validation :

 Validation Figure Validation: Confusion Matrix of -fold validation over G done both with Decision Tree and Random Forest Classi ers . . . . .

Figure Validation :

 Validation Figure Validation: Relative importance for the most useful features of the classi er. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure Robustness :

 Robustness Figure Robustness: Misclassi cation breakdown of the (local,odd) (14%) for the di erent loss models. . . . . . . . . . . . . . . . . . . . . Figure Probing Overhead analysis: Accuracy as a function of the sample set size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure ( a )

 a Figure (a) Internet campaign: Normalized classes occurrences for the training G and Internet-scale L dataset; (b) Measured occurrences of Global IP-ID implementations over the years; (c) Breakdown of the classes of L obtained with both G and G . . . . . Figure Breakdown of the classes of L obtained with both a Decision Tree and a Random Forest Classi er. . . . . . . . . . . . . . . . Figure Normalized classes occurrences for L and its lighter version when only N= packets out of are considered. . . . . . . .

Figure IP -

 IP Figure IP-ID census results, shown as a th order Hilbert curve, a fractal space-lling curve that allows the mapping of the onedimensional IPv address space into a bi-dimensional image. . .

Figure

  Figure Standard Deviation of IP-ID classes of IP addresses owned by same AS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  entraînés sur une énorme quantité de contenu Web, est déséquilibrée. Plus précisément, les prédictions faites avec BERT et DistilBERT sur l'anglais américain standard sont jusqu'à 21% plus précises que celles faites sur l'anglais afro-américain. Nous montrons également qu'au contraire, BERT, RoBERTa et DistilRoBERTa présentent un biais opposé, favorisant alors l'anglais afro-américain. Nos résultats soulignent également que les variantes distillées de BERT et RoBERTa, conçues pour être plus légères et entraînées sur une quantité moindre de données, sont les plus justes parmi les sept modèles de langage testés. En n, la dernière partie de cette thèse traite de la classi cation d'objets générés par des machines à l'aide de certains des plus simples algorithmes d'apprentissage automatique supervisés à l'état de l'art. Dans le chapitre , grâce à un framework solide mais peu intrusif, nous montrons que les di érents comportements d'un champ du paquet IP, l'identi cation IP (IP-ID), peuvent être facilement classi és avec peu de caractéristiques ayant un haut pouvoir discriminatoire. Nous appliquons en n notre technique à un census à l'échelle de l'Internet et fournissons une vue actualisée de l'adoption de ses différentes implémentations dans l'Internet. En particulier, les résultats du census révèlent que le global n'est plus l'implémentation d'IP-ID la plus courante et que, au contraire, d'autres comportements, comme le local et le constant, sont présents. Du point de vue de la méthodologie, nous constatons que quelques caractéristiques scalaires et un classicateur simple, su sent pour prédire avec précision les di érentes mises en oeuvre des IP-ID. Du point de vue des résultats, en revanche, l'application de cette technique fournit une vue actualisée de l'adoption des di érentes implémentations d'IP-ID connues. I N T R O D U C T I O N Contents . Context and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . Quality of Experience on the Web . . . . . . . . . . . . . . . . Machine Learning in modern science . . . . . . . . . . . . . Reading map and Contributions . . . . . . . . . . . . . . . . . . . . . . Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . Originally conceived to allow the communication among multiple computers on a single network, the Internet is nowadays a worldwide network used by more than 59% of the global population every day [ ]. It has become an integral part of the society that today, a world without the Internet is unimaginable. By connecting billions of people worldwide and by enabling an uncountable number of services, the Internet is now a core pillar of the modern information society and a critical piece of the human infrastructure.

Figure :

 : Figure : Relationship between QoS at di erent layers and user QoE.

  and use subjective data to t model parameters[ , , ], or machine learning models, where instead the subjective data is used to train the model[ , ].. . Machine Learning in modern scienceMachine learning is a well established subset of Arti cial Intelligence (AI) crawling with strong technical and scienti c background. Its success in several application domains yielded to a growing demand for systems that can be used by both experts and novices in the eld of machine learning. Nowadays, machine learning has become an integrative part of the modern scienti c methodology and an essential building block for data science. It o ers the possibility to adopt automated procedures for the prediction of different phenomena based on past observations, unraveling underlying patterns present in data and providing insights about the problem under investigation. For instance, as aforementioned, machine learning has been used to build models which map the two di erent kind of measurements for Web QoE (objective metrics and subjective user feedback), in order to automatically extract the user-label data from the collected metrics.

Figure :

 : Figure : Appearance of the Survey in the English Wikipedia (answer order is randomized).

  plot of Page Load Time (PLT)

Figure :

 : Figure : Aggregate statistics of navigation timing performance (TTI, RSI and PLT in the gure), conditioned by survey response.

Figure : Figure : Figure :

 ::: Figure :Annotation of major Wikipedia-related events occurred during the whole -months observation period.

  ing metrics: Fig shows that events and banner campaigns do not alter in a signi cant fashion the evolution of PLT/RSI/TTI metrics, that are intrinsically variable at a daily timescale. Particularly, from Fig. , one can notice that the daily fraction of positive, neutral and negative answers remains remarkably steady over the observation period, with a stationary fraction of about 85% satis ed users.On the one hand, this could be somewhat unexpected since, one could argue events such as, e. g., data center switchover or browser regression to directly a ect the objective measurable delay. At the same time, in light of Fig. , it appears that the observed level of variability in the PLT/RSI/TTI metrics happen in a range that is not enough to a ect human perception -or in other words that the measured delay changes do not necessarily harm user QoE.Seasonality: We next study if user scores follow classic night/day and weekday/weekend e ect. The rst circadian timescale is intrinsic to variation in human cognitive capability throughout the day, whereas the second can possibly re ect a change in the environment (work/leisure), which not only a ects the environment (e. g., user mood) but also possibly the devices used to access the service (e. g., company vs personal). Fig.and

  Fig. report the raw answer frequency (top plots) as well as the breakdown of users scores (bottom plots) at hour-of-day and day-of-week aggregation granularities respectively. Plots report the mean (line) and 95% con dence interval (shadowed band) of the metrics of interest. Top plots in Fig. and Fig. do exhibit a seasonal variation in the answers volume. Particularly, in the hour-of-day case in Fig. the volume is merely correlated with the volume of users activity, which as expected follows a seasonal pattern with lower night-

Figure : Figure :

 :: Figure : Temporal view: absence of night/day seasonality of survey answers.

  Fig. and , in this section we further dig into page, user and environment-related features. Speci cally, whereas lab studies have rather poor diversity in terms of handsets, browser software, and geographical diversity, the collected dataset allows to peek at Web users' QoE under each of these angles. Fig. reports, for cherry-picked features in the dataset, the breakdown of positive/negative scores (neglecting neutral answers for the sake of simplicity). For each subplot, we condition over di erent values of the feature and visually report the positive/negative breakdown as stacked bars. For categorical features without a natural ordering, the bars are ordered in increasing satisfaction

Fig

  Fig. presents the Receiver Operating Curve (ROC) plots produced when evaluating

  Figure : ROC curves, obtained when averaging the results gathered with a -fold cross validation on B with the PA features set.

Fig

  Fig. reports the classi cation results, expressed again in terms of accuracy (top), precision (middle) and recall (bottom), when considering the portion of (WWW) features:as expected, since features in the (WWW) set are fewer with respect to (T) but having better mutual information with the survey answers, classi cation results are practically una ected. We reduce this set even further by only considering the features in the (PA)

Figure : Figure :

 :: Figure : Classi cation results, feature subsampling: performance obtained by limiting the (T)total features, (WWW) features, (PA) publicly available features, both with (PA w.o. ) and without (PA) outliers ltering.

  Figure : Ranking of the features according to their SHAP values.

Figure :

 : Figure : Relevant snapshots of the www.booking.com rendering process corresponding to the di erent modes that are visible in the distribution reported in Fig. . Notice that the "above the fold" content is almost all rendered in (a) and fully rendered in (b). At time (c) a popup arise, inviting users to login in the website.

Figure

  Figure : uPLT distribution for www.booking.com, highlighting the issue that users do not agree on a single time instant to identify completion of webpage rendering.

Figure :

 : Figure : Number of webpages per cluster.

Figure :

 : Figure :Component weights for pages with multi-modal uPLT.

Fig. shows the

  Fig. shows the weights' distribution of the three components sorted by their mass, M mass = [M major , M middle , M minor ], across the 52 likely multi-modal webpages.

Figure :

 : Figure : Ranking of the features according to their SHAP values when predicting uni-modal pages.

Figure :

 : Figure :|PLT -T T I| ECDF for uni/multi-modal pages.

  [ ], transformers-based bidirectional encoder representations language models (LMs) gained lots of scienti c interest due to their sizable improvements on a wide range of Natural Language Processing (NLP) tasks. The success of BERT pushed researchers to expand the state-of-the-art by introducing a plethora of model variants with di erences in the architecture [ ], the size [ , , ] and the training [ , ]. This resulted in a growing concern of the research community to discuss the potential risks coming from the pervasive adoption of these models [ ]. Indeed, several studies highlight that this would hinder an equitable and inclusive access to NLP technologies and have real-world negative consequences in di erent areas, as education, work and politics [ ]. In this context, given the consistent emergence of new LMs trained on Web-based corpora, it is crucial to de ne to which extent such models are fair and not instead prone to bias.

.

  The success of transformer-based LMs is down to several factors, among which it is worth mentioning the large architectures and the training done on huge amounts of textual data. Moreover, recently a special e ort has been devoted to reducing the size of large LMs, e. g., BERT [ ] and RoBERTa [ ], by means of various compression techniques as knowledge distillation or quantization. This made possible the emergence of much smaller, but no less accurate, LMs as DistilBERT [ ] and DistilRoBERTa [ ]. Nevertheless, the emergence and massive spread of all these models raised the interest of the research community towards the potential societal risks linked to the employment of these models for either generating text tasks or as components of classi cation systems [ ]. These works have studied the e ects of transferring the stereotypical associations present in the training datasets to LMs, which cause unintended bias towards underrepresented groups. Recent work [ ] studies di erences in performance for BERT, showing that it often favors the majority group with regards to gender, language, ethnicity, and insurance status, whereas [ ] nds racial bias encoded in di erent models. A signi cant research e ort has been done to show race and gender bias embedded in large models [ , , , , ]. Authors in [ ] highlight instead the presence of topical biases in the words predicted by BERT on sentences mentioning disabilities. In addition to bias measurement works, researchers have proposed methods to mitigate societal biases with debiasing techniques [ , , ]. As for the bias towards languages, most studies have focused on o ensive language and hate speech detection [ , , ], while assessing the bias against dialects spoken by underrepresented groups is quite recent [ ]. Whereas the above works mostly focus on the negative sentiment and stereotypical associations towards speci c groups in BERT [ ], in this work we quantify the linguistic bias towards AAE for di erent LMs: BERT, RoBERTa, BART [ ],

Fig. shows theFigure :

 : Fig. shows the empirical cumulative distribution function (ECDF) of the utterance length u for both AAE and SAE corpora. We can see that the two distributions are almost equal, therefore our utterance-level analysis does not introduce any bias.In Tab. we report a terse summary of the corpora statistics, both before and after having applied the ltering based on the utterance length. Even though the sizes of the two datasets are very di erent, not only in terms of number of utterances |U|, but also

  based BERT base LM [ ] and its recent variants, including DistilBERT base [ ], in both their cased and uncased avors, RoBERTa base [ ], DistilRoBERTa base and BART base [ ].

Figure :

 : Figure : The di erence the ECDFs of SAE and AAE for the ∆P(u) measure. When the values are greater than zero the LMs are more biased towards SAE, vice versa otherwise.

Figure :

 : Figure :The di erence between the ECDFs of both AAE and SAE for the CRR(u) measure.When the values are greater than zero the LMs are more biased towards SAE, vice versa otherwise.

.

  IP-ID classi cation via supervised learning . . . . . . . . . . . . . . Background and related work . . . . . . . . . . . . . . . . . . . . . . . Normative reference . . . . . . . . . . . . . . . . . . . . . . . IP-ID Classi cation Breakdown . . . . . . . . . . . . . . . . . IP-ID Based-Inference . . . . . . . . . . . . . . . . . . . . . Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Active probing . . . . . . . . . . . . . . . . . . . . . . . . . . Features De nition . . . . . . . . . . . . . . . . . . . . . . . . Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . IP ID Classi cation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Classi cation accuracy and validation . . . . . . . . . . . . . Robustness . . . . . . . . . . . . . . . . . . . . . . . . . . . . Internet Census . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Longitudinal Comparison (over the years) . . . . . . . . . . . Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . Spatial analysis . . . . . . . . . . . . . . . . . . . . . . . . . Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. The IP identi cation (IP-ID) is a ( ) bits eld in the IPv (v ) header [ ]. Originally, along with the fragment o set, the IP-ID was used to assist packet segmentation and reassembly and it was unique per each combination of source, destination and protocol. Yet, with technology evolution and the adoption of the MTU path discovery [ ], IP fragmentation becomes less common nowadays, so that the last normative reference [ ] allows IP-ID of atomic datagrams to be be non-unique. As a consequence, IP-ID elds values are determined by the speci c implementation of the Operating System [ ]. In particular, the majority of research work focus their attention on the global counter implementation, which used to be the most common implementation about a decade ago [ ]. However, due to recent evolution of the standards [ , ], a wider range of behaviors can be expected nowadays. Over time, di erent behaviors have been observed such as global and per-ow counters, pseudo-random sequences and constant values [ ], as well as odd behaviors such as those due to load balancing [ ] middleboxes, or host implementations using the wrong endianness [ ]. Given that some of the above implementations maintain state at the IP level, the IP-ID eld has been of invaluable help to infer a wealth of information concerning the network. Particularly, by leveraging inference from global IP-ID implementation, researchers have been able to count the numer of hosts behind NATs [ , ], or even assess the tra c they generate [ , ] and nally expose censorship in the Internet [ , , , ].

Figure :

 : Figure : Illustration of Constant, Local, Global, Random and Odd sequences

  ] does not fully specify the IP-ID behavior (i. e., it only states that each packet must have a unique IP-ID for the triplet of source, destination and protocol), di erent behaviors (namely global, local and random, illustrated in Fig. ) are detailed in by that some hosts set the IP-ID to zero: at the time of [ ],

  that the actual IP-ID implementation depends on the speci c Operating System (OS) and versions . In , RFC [ ] updated the speci cations by a rming that the IPv ID uniqueness applies to only non-atomic datagrams: in other words, if the don't fragment (DF) bit is set, fragmentation and reassembly are not necessary and hence devices may set the IP-ID to zero. At the same time, concern has In particular [ ] reports Windows and FreeBSD to use a global counter, Linux and MacOS to use local counters and OpenBSD to use pseudo-random IP-IDs.

a

  Due to the rounding done by the authors [ ], the sum of all the percentages is % been raised about security problems following the predictability of IP-ID sequences [ , , , ]. In particular, in RFC [ ] discouraged the use of a global counter implementation for many security issues, such as stealth port scan to a third (victim) host, and in RFC [ ] addressed concerns concerning IPv -speci c implementations. In light of the recent evolution of the standards, a re-assessment of IP-ID usage in the wild is thus highly relevant. . . IP-ID Classi cation Breakdown In the last decade, to the best of our knowledge, few research works have provided a complete picture of the breakdown of the existing IP-IDs behaviors. That is what makes the comparison of the results of this work with the previous ones with the purpose of analysing the temporal changes on the IP-ID popularity an hard task.

  to accurately classify IP-ID behaviors, that we apply to the Internet at large, gathering a picture of the relative popularity of each IP-ID behavior. In terms of methodologies, authors in [ ] use ICMP timestamp and IP-ID to diagnose paths from the source to arbitrary destinations and nd reordering, loss, and queuing delay. In [ ], the authors identify out-of-sequence packets in TCP connections that can be the result of di erent network events such as packet loss, reordering or duplication. In [ ], they use HTTP requests from two di erent machines toward target websites, to discover the number of load-balancing server. Authors in [ ] use TCP SYN-ACK from multiple vantage points to identify connectivity disruptions by means of IP-ID elds, which then they use as a building block of a censorship detection framework.In this chapter, we leverage ICMP tra c (spoo ng IP addresses to craft sequences of packets that are precisely interleaved when they hit the target under observation) to build an accurate, robust and lightweight IP-ID classi cation technique..To provide an accurate and comprehensive account of IP-ID behavior in the wild, we need (i) a reliable classi er, able to discriminate among the di erent typical and anomalous IP-ID behaviors. At the same time, to enable Internet coverage, (ii) the classi er should rely on features with high discriminative power, extracted from the data gathered through an active probing technique that is as lightweight as possible. In this section we illustrate the practical building blocks and their theoretical foundations, that our classi cation framework builds upon.IP-ID classes: From the host perspective, several IP-ID behaviors are possible as depicted in Fig. . The image shows the sequences of IP-ID samples sent from different host (orange and blue) where the packets are sent alternatively to the target. The di erent behaviors depicted are, from left to right: (i) constant counters are never incremented (and for the most part are equal to 0x0000); (ii) local or per-host counters that are incremented at each new packet arrival for that ow (mostly by unit, 99.7% of the times in our large scale measurements): as a consequence, while the orange or blue per-host sub-sequences are monotonically increasing, the aggregate sequence alternates between the two; (iii) global counters are incremented by unit at each new packet arrival for any ow: thus, the sequence s is monotonically increasing (90.3% of the times by unit, 4.7% by units and 4.6% by units), and the orange or blue perhost sub-sequences are monotonically increasing but at a faster rate (by units); (iv)

Figure :

 : Figure :Scenario in which the active probing is performed: only one sender is used to ease the synchronization of packets generation, whilst both the machines are used to receive and collect the stream of IP-IDs generated at the target machine.

Figure :

 : Fig. shows the scenario in which the experiments are carried out. It provides information about how the hosts are involved and the kind of data collected: there are two receivers but only one real sender, and the information gathered at the two vantage points regards the sequences of IP-IDs generated by the target machine. To validate our assumptions, we carry on additional experiments on a preliminary testbed to test the sensitivity of the algorithm to external tra c hitting the target. In these experiments:• we send UDP CBR tra c with Iperf at T X rate = 10Mbps and vary the packet size over time (in particular decrease), so that we increase the packet rate during the experiment (to control the IP-ID generation);• in one experiment, we send ICMP Echo Request packets with an inter-packet gap of ∆t interpacketgap = 10ms and collect the IP-ID sequence x, for which we derive the derivative series (gray color line);

  occurring when all the N elements of the series are di erent. Consequently, by considering the global and local implementations, we can observe that the entropy for the sequences x and y of length N 2 is expected to be maximum H(x global ) = H(y global ) = log 2 N 2

Figure : Figure :

 :: Figure : Manual Ground Truth: Normalized classes occurrences for the training datasets G and G

Fig

  Fig and Fig . In particular, the confusion matrix reported in the left side of Fig shows the aggregated results over all loss models: we can observe that most of the classes have a true positive classi cation of 99% or 100% even in presence of 20% packet losses, and irrespectively of the actual loss pattern.

.

Figure : Figure :

 :: Figure : Robustness: (left) Confusion Matrix of a classi er trained on the real lossless dataset G and tested on the synthetic lossy dataset S lossy with purposefully injected 20% packet losses on each sequence, (right) Confusion Matrix of a classi er trained on the real lossless dataset G and tested on the dataset where 20% of each sequence is intentionally randomly swapped S reorder .

Figure :

 : Figure : (a) Internet campaign: Normalized classes occurrences for the training G and Internetscale L dataset; (b) Measured occurrences of Global IP-ID implementations over the years; (c) Breakdown of the classes of L obtained with both G and G

  then use these two datasets to classify the IP-ID behaviors in the whole large scale dataset L covering the all the responsive IP addresses of the full hitlist. Results, shown in Fig. (c) con rms indeed the validity of our methodology since, statistically, there are only slight di erences between the occurrences breakdown when the classi er is trained on G or G . Both datasets yield to consistent results ensuring the independence of the model from the training dataset and proving that as long as the behaviors are balanced the IP-pre x level imbalance is irrelevant.Lightweight Census: Additionally, we may want to further investigate how the classi cation results change when we have a fewer number of packets building the IP-ID series that we aim at classify. This is important since we want to avoid injecting useless tra c in the network. Similarly to what previously described in Sec. . . , we take the measurements dataset G and only consider that we have at our disposal only the rst N = 10 < N out of the N = 100 samples gathered in the full experiment. Given that in this case we are only looking at a small portion of the collected series, we may expect that in this case we can have a loss in terms of amount of oddities really present in the dataset, and behaviors like the one depicted in Fig. might not be correctly classi ed, simply due to to lack of information about it. In fact, in this case, it is possible that the jump of the IP-ID counter occurs later in the sequence, so all the features are evaluated on a resembling simple counter. What practically happens in reality con rms the expectations: about half of the oddities are spread between the global and local implementations. What is instead more surprising is the substantial decrease for the population random class.

Figure :

 : Figure : Normalized classes occurrences for L and its lighter version when only N= packets out of are considered.

Figure :

 : Figure : IP-ID census results, shown as a th order Hilbert curve, a fractal space-lling curve that allows the mapping of the one-dimensional IPv address space into a bidimensional image.

Feature

gaps 5

 5 The gaps in the Navigation Timing metrics. Calculated by taking the sum of: domainLookupStart -fetchStart, connectStart -do-mainLookupEnd, requestStart -connectEnd and loadEventStart -domComplete. rstPaint The time when something is rst displayed on the screen. rsi RUMSpeedIndex. Estimate of the SpeedIndex value based on Re-sourceTiming data. Now moved to the RUMSpeedIndex Event-Logging schema, but was collected as part of the NavigationTiming schema at the time of the study. metrics coming from the browsers' implementation of the NavigationTiming API rstPaint comes from the Paint Timing API or vendor-speci c implementations predating the standards. RUMSpeedIndex is a compound metric combining several NavigationTiming and ResourceTiming (level and level ) metrics into a single score. It's a rd-party FLOSS library found here: https://github.com/ WPO-Foundation/RUM-SpeedIndex B I B L I O G R A P H Y [ ] [Online]. Available from: https://www.

Table

  Summary of recent related work gathering user feedback for Web quality of experience assessment. . . . . . . . . . . . . . .Table Collected corpus of Wikipedia users' QoE feedback. . . . . . . . Table Summary of the features (T/WWW/PA) that are associated to each users' survey response. The mutual information between the survey answer and T/WWW/PA features in the class is reported as a boxplot. . . . . . . . . . . . . . . . . . . . . . . . . .

	Table	User feedback prediction: Confusion matrixes, obtained when
		averaging the results obtained with a -fold cross validation
	Table	MAE and MSE of ∆P(u) and CRR(u) measured on AAE and
		SAE corpora: results obtained through the ll-in-the-blank task
		with di erent language models. † signi es that the AAE and SAE expectations are statistically signi cant according to the
		Welch's two-tailed t-test (p-value < 0.05). The column d con-
		tains their e ect size computed according to the Cohen's d. . . .
	Table	A sample of AAE utterances selected based on their syntactical
		features and their translations to SAE. In brackets the preva-

on B with the PA features set. . . . . . . . . . . . . . . . . . . .

Table User feedback prediction: classi cation results expressed through several metrics, obtained when averaging the results obtained with a -fold cross validation on B with the PA features set. . . Table Breakdown for M mass = M time . . . . . . . . . . . . . . . . . Table Statistics of uni-modal/multi-modal pages. . . . . . . . . . . . . Table RMSE of (top) uni-modal and (bottom) multi-modal uPLT with Web quality metrics. . . . . . . . . . . . . . . . . . . . . . . . . Table Summary of recent related work. . . . . . . . . . . . . . . . . . Table Corpora summary: with and without ltering utterances (U) based on their length. With u we indicate the average utterance length; with L, the length of the corpus in number of words, and; with |T|, the number of terms (unique words). . . . Table Training data for the tested LMs. . . . . . . . . . . . . . . . . . . Table Example showing the masked token experiment. . . . . . . . . . lence of the feature over the utterances in the AAE corpus. . . . xv xvi List of Tables

Table Similar to

 Similar Table. but calculated over a sample of utterances of AAE and their translated version (AAE ) for each feature of AAE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Table Similar to Table. but calculated for t rather than u, for three POS classes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Table MAE and MSE of ∆P(t) and CRR(t) measured on AAE and SAE corpora: results obtained through the ll-in-the-blank task with di erent language models, averaging token predictions for each POS class. † signi es that the AAE and SAE expectations are statistically signi cant according to the Welch's twotailed t-test (p-value < 0.05). The column d contains their e ect size computed according to the Cohen's d. . . . . . . . . . . . . Aujourd'hui, un monde sans Internet est inimaginable. En inter-connectant des milliards de personnes dans le monde et en o rant un nombre incalculable de services, il est désormais pleinement intégré à la société moderne. Pourtant, malgré l'évolution et le développement de la technologie, son omniprésence et son hétérogénéité soulèvent encore de nouveaux dé s, tels que les problèmes de sécurité, le contrôle de la qualité d'expérience des utilisateurs (QoE), le souci de transparence et celui d'équité. En conséquence, l'objectif de cette thèse est d'apporter un nouvel éclairage sur certains des

	xviii
	L I S T O F A B B R E V I AT I O N S POS QoE QoS R É S U M É D E L A T H È S E E N F R A N Ç A I S Part Of Speech Quality Of Experience Quality Of Service	1
		RMSE	Root Mean Squared Error
		wRMSE Weighted Root Mean Squared Error
	AAE	ROC	African American English Receiver Operating Curve
	ACR	RUM	Absolute Category Ranking Real User Monitoring
	AS	RR	Autonomous System Rectiprocal Rank
	AI	RSI	Arti cial Intelligence Rum SpeedIndex
	ASN	RTT	Autonomous System Number Round Trip Time
	AATF SAE	Approximated Above The Fold Standard American English
	ATF	Above The Fold SHAP SHapley Additive exPlanations
	AUC SVM	Area Under the Curve Support Vector Machines
	CRR	TTC	Complementary Rectiprocal Rank Time To Click
	DOM TTFB Document Object Model Time To First Byte
	GDP	Gross Domestic Product TTFP Time To First Paint
	GMM TTI	Gaussian Mixture Model Time To Interactive
	K-NN uPLT K Nearest Neighbor User Page Load Time
	IP		Internet Protocol
	IP-ID	Internet Protocol IDenti cation
	ISP		Internet Service Provider
	LM		Language Model
	MAE	Mean Absolute Error
	MI		Mutual Information
	MLM	Masked Language Modeling
	MLP		Multi Layer Perceptron
	MOS		Mean Opinion Score
	MSE		Mean Squared Error
	NLP		Natural Language Processing
	OS		Operating System
	PA		Publicly Available
	PLT		Page Load Time
				xvii

Table Summary of related work . . . . . . . . . . . . . . . . . . . . . . Table Tabulated expected values for selected features . . . . . . . . . . Table Summary of the Datasets. . . . . . . . . . . . . . . . . . . . . . . Table Features values for both lossless and lossy synthetic dataset S lossy with % losses -local implementation case of IP-ID. . . Table List of features, informing whether each comes from raw data or is derived, and if it is present in the T, WWW or PA set. . . . Table Schema of the Public Available Features. . . . . . . . . . . . . . dé s qui ont émergé ces dernières années. En particulier, nous fournissons une analyse approfondie de certains des aspects les plus importants de l'Internet moderne. Un accent particulier est mis sur le World Wide Web, qui, parmi tous, est sans doute l'une des applications Internet les plus populaires, et un regard spéci que sur son interaction avec l'apprentissage automatique. Nous suivons donc deux directions de recherche principales: premièrement, nous nous concentrons sur l'analyse de la qualité de l'expérience des utilisateurs du Web (Chap. et Chap. ); deuxièmement, nous mettons l'accent sur l'utilisation de l'apprentissage automatique appliqué aux mesures d'Internet, en étudiant, d'une part, l'impact de son interaction avec le Web (Chap. ) et, d'autre part, son utilisation pour prédire les objets générés par les machines (Chap. ).

  Background and related work . . . . . . . . . . . . . . . . . . . . . . . Web QoE metrics . . . . . . . . . . . . . . . . . . . . . . . . . State of the art limitations . . . . . . . . . . . . . . . . . . Web users' experience is still largely impenetrable [ , ]. As such, a number of alternative metrics that attempt at better tting the human cognitive process (such as SpeedIndex, user-PLT etc., see Section . ) have been proposed as a proxy of users'

	Quality of Experience (QoE), whose monitoring is important for both Over The Top
	(OTT) operators to keep users engaged as well as for Internet Service Providers (ISP) to
	ComNet Conference. lower user churn. At the same time, studies involving more advanced metrics are typically validated .
	with rather small-scale experiments, either with a small number of volunteers, or by • (UNDER REVIEW) Salutari, F., Linguaglossa L. & Lipani A., "Quantifying the Bias of Transformer-Based Language Models for African American English in Masked relying on crowdsourcing platforms to recruit cheap labor and produce a dataset labeled
	Language Modeling", The with user opinion. Often, videos of websites rendering process are used (as opposite to Conference on Empirical Methods in Natural Lan-
	guage Processing (EMNLP). actual browsing), with possibly very speci c instruction (e. g., such as in A/B testing, by .
	clicking on the fastest of two rendering processes) that are however rather di erent from Moreover, the collaboration with other researchers on di erent topics resulted in the the cognitive process in action during the typical user browsing activities. Additionally, following contributions: such tests are carried on a limited number of xed conditions, with a small heterogeneity
	of devices, Operatying Systems (OSs) and browsers, and are not exempt from cheating
	so that ingenuity is needed to lter out invalid answers from the labeled dataset [ ,
	reasons including unexpected sources of latencies [ ], interactions between network
	protocols [ ], the growingly more complex structure of websites [ ], an increased
	usage of mobile devices [ ,	] and the emergence of new protocols [ ]. Yet, whereas
	the study of Web performance is commonly [ , , ,	,	,	,	,	] tackled

• Salutari, F., Cicalese, D., & Rossi, D., "A closer look at IP-ID behavior in the Wild".

International Conference on Passive and Active Network Measurement, (PAM' ). .

• Salutari, F.,

Da Hora, D., Dubuc, G., & Rossi, D.

, "A large-scale study of Wikipedia users' quality of experience", The Web Conference (WWW' ). . • Salutari, F., Da Hora, D., Dubuc, G., & Rossi, D., "Analyzing Wikipedia Users' Perceived Quality Of Experience: A Large-Scale Study", IEEE Transactions on Network and Service Management. . • Salutari, F., Da Hora, D., Varvello, M., Teixeira, R., Christophides, V., & Rossi, D., "Implications of the Multi-Modality of User Perceived Page Load Time", IEEE Med-• (UNDER REVIEW) Bahri, M., Salutari, F., Putina, A. & Sozio, M., "Automated Machine Learning with a Focus on the Unsupervised Learning: a Survey", The International Journal of Data Science and Analytics (JDSA). . • (UNDER REVIEW) Putina, A., Salutari, F., Bahri, M. & Sozio, M., "AutoAD: an Automated Framework forUnsupervised Anomaly Detection", European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML/PKDD). . T H E U S E R S A N D T H E W O R L D W I D E W E B Contents . The Wikipedia case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Our contribution . . . . . . . . . . . . . . . . . . . . . . . . User feedback collection . . . . . . . . . . . . . . . . . . . . . . . . . . Technical aspects of the survey collection . . . . . . . . . . . Collected features . . . . . . . . . . . . . . . . . . . . . . . . . Ethics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Validity of the collection methodology . . . . . . . . . . . . User feedback characterization . . . . . . . . . . . . . . . . . . . . . . Aggregate view . . . . . . . . . . . . . . . . . . . . . . . . . . Temporal breakdown . . . . . . . . . . . . . . . . . . . . . . . Spatial breakdown . . . . . . . . . . . . . . . . . . . . . . . User feedback prediction . . . . . . . . . . . . . . . . . . . . . . . . . Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Since its inception, the World Wide Web has sometimes been dubbed as World Wide "Wait" [ ]. Slow rendering of the websites happened due to dial-up connections in the s, slow G connections in the s and so on, but it also persists nowadays for several via simple objective metrics [ ], and rather typically the Page Load Time (PLT), the quality of

  At the same time, the above metrics su er from a limited validation with user feedback.

	. .
	. . Web QoE metrics . . State of the art limitations for their feedback on the quality of their browsing experience. We remark that this ap-
	proach is rather common with VoIP services (e. g., Skype, Hangouts often ask for Mean
	Web QoE metrics fall in two main categories: objective and subjective. As we are inter-Opinion Score (MOS) rating at the end of the call), but to the best of our knowledge this
	ested in measuring browsing experience on individual pages, engagement metrics such has not been attempted before on the wide and wild Web. Speci cally, instead of col-
	as those used in [ , Subjective: Subjective metrics rely on directly collecting responses from users regard-] are clearly out of scope. lecting user feedback on a -grade Absolute Category Ranking (ACR) scale, we ask for
	ing di erent questions related to Web QoE. Typical approaches are to crowdsource the a slightly more than binary feedback (see Section . ), which let us carry on a thorough
	Table : Summary of recent related work gathering user feedback for Web quality of experience Objective: As such, objective metrics of interest for Web user QoE rely on measurable validation with A/B testing [ , ], or by performing experiments on real pages in characterization of user satisfaction (see Section . ) and formulate a simple, yet hard, assessment. Year [ref] Scale/heterogeneity Experimental settings data (e. g., network, browser events) capturing Web quality [ , , , , ]. These controlled conditions [ , , ]. Both approaches have their downsides. Controlled binary classi cation problem (see Section . ). Particularly, we provide a wide variety of Main focus metrics can be further categorized in two classes. experiments with real HTTP server/clients and emulated network conditions for a more classi cation results but we especially provide a thorough characterization of user QoE
	Lab+CW 1 On the one hand, there are tracking metrics that either pinpoint precise time instants Pages Network 2 Sw 3 Hw 4 Samples faithful and interactive browsing experience, but are harder to scale, topping to few hun-along the temporal and spatial dimensions (Section . ).
	[ ] and track speci c events of the W C Navigation Timing [ ]: notable examples include + ---. k Prioritize ele-ments (Above Per-user content pri-oritization dreds users and few thousands data points [ ]. A/B tests try to circumvent this limit, The usefulness of the investigation we carry on and of the models we propose is clear
	The Fold and the time at which the Document Object Model (DOM) is loaded or becomes interactive but introduce other limitations. First and foremost, A/B testing is hardly representative when we consider that recent work such as [ ] still employs simple response times as a user ratings) + k n.a. k Side-by-side uPLT metric de ni-(TTI), the time at which the rst element is painted (TTFP), the Time to The First Byte of Web browsing activity, since crowdworkers are instructed to select which among two proxy of user sastisfaction for Web performance, whereas authors [ ] go at a deep level
	[ ] (TTFB), the time at which the page is fully loaded (PLT) or the time when the Above videos (of the tion to investigate performance of Video application, considering a more involved Pseudo-videos, that they are passively screening side-by-side and that correspond to two di er-same site) + k Controlled HTTP vs HTTP/ The Fold (ATF) portion of the page is rendered [ ]. Most of these metrics are available ent Web rendering processes, appears to nish rst -whereas it is known that even Subjective Quality Assessment (PSQA) involving a Random Neural Network (RNN). It is
	[ ] from the browser Navigation Timing [ ] or can be inferred from packet/ ow-level traf-browsing thus clear that, whereas models for video quality abunds [ , ], scienti c community for a simple Web browsing task such as information seeking, already di erent types of experiments + . k Side-by-side HTTP/ push impact c [ , ] and are easy to include (though not necessarily relevant) as proxy of user searches are rather di erent from the user standpoint in terms of cognition, emotion still misses an established and agreed MOS model for Web performance. At the same
	[ ] experience: for instance, [ ] aim at prioritizing delivery of content that is rendered videos or the and interaction [ ]. In other words, these experiments inform us that humans can time, we point out that the community started adopting slightly more accurate objective
	same website in di erent pro-above the fold (futher specializing content relevance for each user). perceive di erences in these rendering processes, however they fail to signify if these models (as in [ , , ] to perform large scale studies), that are inspired by metrics
	tocol settings On the other hand, there are the integration metrics that are founded on the idea that perceptible rendering changes would impact the user satisfaction through the course of such as SpeedIndex that we consider in this work, although the human component -
	[ ] one page can render faster than another despite nishing loading at the same "time" + . k k Side-by-side videos ( dif-PSI metric de nition a normal browsing session. which is among the main contribution of this work-is generally missing. Particularly,
	ferent website (e. g., in terms of PLT). These metrics integrate all events of the waterfall representing the our collection e ort allows us to perform a large scale study across the human dimension, The time at which users consider the process nished is denoted as user-perceived-pairs) + . k Webcam, Eye gaze, uPLT visual progress of the page, such as SpeedIndex [ ] and variants [ , , ], that have PLT (uPLT) [ ] or Time To Click (TTC) [ ] and is often used as a ground truth of to levels that were previously unprecedented.
	[ ] user perception. Yet, when users select a uPLT in [ ], they are proposed with similar eye-tracking glasses + n.a. k Controlled ATF metric de ni-Compared to recent literature, compactly summarized in Tab. , we are the rst to received signi cant attention lately. Denoting with x(t) ∈ [0, 1] the visual completeness ratio of a page, metrics in the SpeedIndex family are de ned as the integral of the residual frames at earlier times, which has the bene cial e ect of clustering answers and make involve a large number of real users (62k from 59k distinct IP addresses) accessing a
	[ ] completion (1 -x(t))dt and di er in the way they express x(t). Initial de nitions in browsing tion diverse set of pages (46k Wikipedia pages, which are more likely similar among them uPLT more consistent at the price of possibly inducing a bias. Similarly, [ ] employs experiments + n.a. User rating QoE-aware network-this family required capturing movies of the rendering process [ ], or to further use SpeedIndex and TTC to forecast which among the left or right video was selected by the than the set of di erent websites used in other studies), gathering over 62k user re-
	[ ] similarity metrics SSim [ ], making them di cult to use outside a lab environment. of video ren-ing user at time TTC: the classi er in [ ] is accurate in predicting which of the two videos sponses overall (more than twice the survey responses collected in similar large-scale
	dering of Web browsing To counter this issue, simple approximations such as the ObjectIndex/ByteIndex [ ] is perceived as fastest by users. Yet, ndings in [ ] are not informative about whether Wikipedia studies [ ]). Particularly, whereas most of the studies involving lab volun-
	+ . k that merely count the fraction of objects/bytes received (over the total amount), or as k User rating QUIC protocol the user would have been dissatis ed from the slower rendering had s/he actually been teers & crowdworkers employ a single browser and hardware (since crowdworkers are
	[ ] the RUM SpeedIndex (RSI) [ ] that use areas of rectangles for objects as they are of video ren-dering of Web browsing. shown videos rendered with a single browser and hardware combination) on a relatively
	browsing painted on screen (over the total screen size) have been proposed. In this chapter, we use small set of synthetic controlled network conditions (1-32), in our dataset we observe
	this study RSI, which is among the most advanced Web QoE metrics considered to be the current 62k users 46k 3.8kISPs 45 2.7k 62k User feed-back from User satisfaction . . Our contribution 45 distinct browsers software used on over 2, 716 hardware devices on 3, 827 ISPs -a
	real browsing signi cant change with respect to arti cial and controlled lab conditions, which make industry standard. Finally, while we are aware that more complex approaches involving activity 1 Crowdworkers, 2 Number of controlled network conditions, 3 Software browser, 4 Hardware device the spatial dimension (i. e., eye gaze) also exist [ , ], but they are not covered by this To get beyond the limitations of controlled and crowdsourced experiments just exposed the dataset that we release at [ ] of particular interest.
	work (cfr. Section . ). in Section . . (e. g., few users involved, lack of real user behavior representativeness
	PLT corresponds to a browser's onload event, which indicates that all of the objects in the document are and low data heterogeneity), in this work we are the rst to query, at scale, Web users
	in the DOM, and all the images, scripts, links and sub-frames have nished loading.

As inferred from the User-Agent header eld, after having ltered bots

Table :

 : Collected corpus of Wikipedia users' QoE feedback.

	Period	May th -Oct th
	No. of survey requests	|S| =
	No. of survey answers	|A| =	|S|/|A| = . %
	No. of positive answers |A + | =	|A + |/|A| = . %
	No. of neutral answers	|A 0 | =	|A 0 |/|A| = . %
	No. of negative answers |A -| =	|A -|/|A| = . %
	the survey is to assert whether there are Quality of Experience issues that a signi cant
	fraction of users consider to be problematic, and that Wikipedia should thus deal with.
	Since it is well known that "results that are only based on user ratings do not re ect
	user acceptance" [ ], instead of asking users a -grade Absolute Category Ranking
	(ACR) score, the survey explicitly asks for user acceptance, i. e., users can respond with
	a positive, neutral or negative experience. For the sake of completeness, a snapshot of
	the survey question as it is rendered for English readers is reported in Fig. . To avoid
	biasing user answers, we randomize the order of survey answers and we avoid priming
	e ect by refraining to explain/formulate speci c survey goal (e. g., collect data to make
	Wikipedia faster) prior of the answer (survey purpose and data collection policies are
	available through the "privacy statement" hyperlink shown in Fig. ). Similarly, neutral
	feedback is meant for, e. g., users that have no honest opinion, as well as users who were
	not paying attention during the rendering, or users that do not understand the question,
	etc., to avoid biasing the results (Section . . ).

Table :

 : Summary of the features (T/WWW/PA) that are associated to each users' survey response. The mutual information between the survey answer and T/WWW/PA features in the class is reported as a boxplot.

	Class	T/WWW/PA	Sample features	MI(x,y)
			Wiki, Page size,
	Page	/ /	Survey viewtime,
			etc.
			PLT, TTI
	Performance	/ /	TTFP, RSI, etc.
			Device, Browser,
	User	/ /	editCountBucket, etc.
			Connection Type,
	Environment	/ /	Time, Geolocation, etc.
			Total
	Overall	/ /	WWW [ ] paper

Table :

 : User feedback prediction: Confusion matrixes, obtained when averaging the results obtained with a -fold cross validation on B with the PA features set.

	Model	True Predicted All
		-		+
	Perceptron	-+	. .	. .
	Random Forest	-+	. .	. .
	XGBoost	-+	. .	. .
	K-NN	-+	. .	. .
	SVM	-+	. .	. .

prediction analysis is a simpli cation which directly stems from the structure of our survey, and allows to turn the problem into a binary classi cation one. This simple formulation enables immediate and intuitive statements of performance objective, that we express in terms of the classic information retrieval metrics.

Table :

 : User feedback prediction: classi cation results expressed through several metrics, obtained when averaging the results obtained with a -fold cross validation on B with the PA features set. Accuracy Precision A -Recall A -Precision A + Recall A + F1 A -F1 A +

	Model
	Perceptron	.	.	.	.	.	.	.
	Random Forest	.	.	.	.	.	.	.
	XGBoost	.	.	.	.	.	.	.
	K-NN	.	.	.	.	.	.	.
	SVM	.	.	.	.	.	.	.
	Tab. reports confusions matrixes, obtained when training a -trees random for-
	est [ ], a XGBoost [ ] classi er, a Multi Layer Perceptron (MLP) (two layers percep-
	tron: a Recti ed Linear Unit (ReLU) and a sigmoid), a K Nearest Neighbor (K-NN) (with
	K = ) and nally a Support-Vector Machines (SVM) classi er. Results are gathered when

considering all the (PA) publicly available features on a -fold cross validation with a : training and testing dataset split, and nally averaging the outcomes of each fold. The entire classi cation results are reported in Tab. and expressed in terms of accuracy, precision, recall and F score for both the positive A + and the negative A - answers sets. We obtain, in terms of model accuracy, similar results with both the models, with a slightly higher accuracy when using XGBoost but lower A -recall with respect to the Random Forest Classi er. Prediction outcome is clearly deceiving and only slightly better than the naïve baseline, despite the relatively large number of features collected: speci cally, with the Random Forest classi er only 61% of the unsatis ed users are correctly captured, with a precision of 0.58. Interestingly, performance on the remaining dataset B, i. e., the set of positive scores ltered out due to class imbalance, remains consistent with an average accuracy of 0.55.

  The Multi-Modality of uPLT . . . . . . . . . . . . . . . . . . . . . . . Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Representative Webpage Selection . . . . . . . . . . . . . . . Objective Web Quality Metrics . . . . . . . . . . . . . . .

. . UPLT Crowdsourcing . . . . . . . . . . . . . . . . . . . . . . Understanding Users' Feedback . . . . . . . . . . . . . . . . . . . . . . UPLT Distribution Analysis . . . . . . . . . . . . . . . . . . . Page Characteristics and uPLT . . . . . . . . . . . . . . . . . . Evaluation of Web Quality Metrics . . . . . . . . . . . . . . Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A good Quality of Experience (QoE) on the Web is essential for both content providers and consumers. QoE directly a ects end-users' willingness to visit a webpage [ ] as well as content providers' business revenues [ ]. Both industry (e. g., QUIC, SPDY, and HTTP/ ) and academia [ , , , ] have made signi cant e ort to design tools and novel protocols to reduce page load times as the main factor that determines Web QoE is how fast a page loads [ ].

  . Evaluation of Web Quality Metrics

	RMSE µ,Metric TTFP	TTI	AATF PLT
	µ(uPLT )	. s	. s	. s	. s
	wRMSE	TTFP	TTI	AATF	PLT
	µ(uPLT )	. s	. s	. s	. s
	M major	. s	. s	. s	. s
	M middle	. s	. s	. s	. s
	M minor	. s	. s	. s	. s
	M first	. s	. s	. s	. s
	M second	. s	. s	. s	. s
	M third	. s	. s	. s	. s
	Finally, we investigate to which extent single-valued objective Web quality metrics (see
	Section . ) can approximate the di erent modes of the uPLT distributions exhibited by
	the webpages of our study. For each of the 56 webpages showing a uni-modal uPLT be-
	havior, Tab. reports on the top the Root Mean Square Error (RMSE) of the mean of the

uPLT distribution µ(uPLT ) with respect to each of the following objective Web perfor-Table : RMSE of (top) uni-modal and (bottom) multi-modal uPLT with Web quality metrics.

mance metrics: TTFP, TTI, PLT and ATF, computed as Approximated Above-The-Fold (AATF) [ ]. Results reveal that PLT is the metric which better approximates (lowest error term) the uPLT for webpages showing a uni-modal behavior of uPLT.

Table :

 : Summary of recent related work.

	Year [ref]	Experiments scale	Measurement design	uPLT multi-	Metrics evaluation	uPLT modes analysis	Main focus
						modality
	[ ]	n.a.	uPLT crowd-	No	Yes	No	WQL de nition
				sourcing				and demonstra-
									tion
	[ ]	n.a.	uPLT crowd-	No	Yes	No	Assessment and
				sourcing				Models for Web
									QoE
	[ ]	n.a.	n.a.		No	Yes	No	Web	QoE
									overview
	[ ]	users, webpages	uPLT crowd-sourcing	Yes	No	No	uPLT de nition metric
	[ ]	users, webpages	uPLT crowd-sourcing	Yes	No	No	uPLT optimiza-tion by tracking
									user's eye gaze
	[ ]	. k users, webpages	A/B by	testing showing	No	Yes	No	Web browsing QoE assessment
				side-by-side
				videos
	work]	[This	k users, webpages	uPLT crowd-sourcing	Yes	Yes	Yes	uPLT modality	multi-
									analysis	and
									characteriza-
									tion
	.

  is interesting per se, that ensures our 100 target pages cover the initial M set) and repeatable (for which we have already open sourced our dataset [ ]).

	Whereas this work is far from entirely closing the Web QoE measurement issue, we
	hope that open sourcing our dataset [ ] can help the community into further nailing T H E W E B A N D T H E M O D E L S
	down the smallest set of relevant Web QoE metrics covering all user modes, as opposite
	to attempting to de ne yet another single Web QoE indicator, that would by de nition
	fail in this task.
	Contents
	.	Is the Web diverse enough?: on the fairness of language models . .
	.	Background and related work . . . . . . . . . . . . . . . . . . . . .
	.	Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
		. .	Corpora for Spoken English . . . . . . . . . . . . . . . . .
		. .	Bias in Masked Language Modeling . . . . . . . . . . . . .

. selection, which . Quantifying the Bias . . . . . . . . . . . . . . . . . . . . . . . . . . . . Measuring the Bias of LMs . . . . . . . . . . . . . . . . . . . . Bias on AAE Features . . . . . . . . . . . . . . . . . . . . . . . Bias on Part-of-Speech . . . . . . . . . . . . . . . . . . . . . Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Table :

 : Corpora summary: with and without ltering utterances (U) based on their length. With u we indicate the average utterance length; with L, the length of the corpus in number of words, and; with |T|, the number of terms (unique words).

	Corpus Language	|U|	u	L	|T|
	Original
	CORAAL	AAE	,	.	,	,
	SBCSAE	SAE	,	.	,	,
	Preprocessed
	CORAAL	AAE	,	.	,	,
	SBCSAE	SAE	,	.	,	,

.

  : MAE and MSE of ∆P(u) and CRR(u) measured on AAE and SAE corpora: results obtained through the ll-in-the-blank task with di erent language models. † signi es that the AAE and SAE expectations are statistically signi cant according to the Welch's two-tailed t-test (p-value < 0.05). The column d contains their e ect size computed according to the Cohen's d.

						MAE								MSE
			∆P(u)			CRR(u)			∆P(u)			CRR(u)
	Model	AAE SAE	∆[%]	d	AAE SAE	∆[%]	d	AAE SAE	∆[%]	d	AAE SAE	∆[%]	d
	BERTcased BERTuncased DistilBERTcased DistilBERTuncased RoBERTa DistilRoBERTa BART	. . . . . . .	. . . . . . .	† † † † - † - † - †	. . . . . .	. . . . . . .	. . . . . . .	† † † † - † - † - †	. . . . . . .	. . . . . . .	. . . . . . .	† † † - † - † - †	. . . . . . .	. . . . . . .	. . . . . . .	† † † † - † - † - †	. . . . . . .

Table

  Similar to Table. but calculated over a sample of utterances of AAE and their translated version (AAE ) for each feature of AAE.

					Double Negative [ utterances]
	BERT cased BERT uncased DistilBERT cased DistilBERT uncased RoBERTa	. . . . .	. . . . .	† . . † . † . .	. . . . .	. . . . .	† . † . † . † . .	. . . . .	. . . . .	† . . † . . .	. . . . .	. . . . .	† . † . † . † . .
	DistilRoBERTa BART	. .	. .	† . † .	. .	. .	† . † .	. .	. .	† . .	. .	. .	† . .
					Copula be [ utterances]
	BERT cased BERT uncased DistilBERT cased DistilBERT uncased RoBERTa DistilRoBERTa BART	. . . . . . .	. . . . . . .	† . † . . † . † . † . .	. . . . . . .	. . . . . . .	† . † . † . † . † . † . † .	. . . . . . .	. . . . . . .	† . † . . . . . .	. . . . . . .	. . . . . . .	† . † . † . † . † . † . † .
					Contractions [ utterances]
	BERT cased BERT uncased DistilBERT cased DistilBERT uncased RoBERTa DistilRoBERTa BART	. . . . . . .	. . . . . . .	† . † . . † . . † . .	. . . . . . .	. . . . . . .	† . † . † . † . † . † . † .	. . . . . . .	. . . . . . .	† . † . . . . † . .	. . . . . . .	. . . . . . .	† . † . † . † . † . † . † .

Table :

Table :

 : Summary of related work

	Work Year	Features Census		Classes Break-	Methodology	Scope of the
					down (%)				work
	[ ]	∆IP-ID	no (	target	70%	global,	Analysis of replies	Packet reorder-
			routers)		remaining	%	to active probing	ing and losses
					between	con-	(ICMP requests)	diagnosis.
					stant (equal to
					) and counters
					with increment
					by .
	[ ]	∆ IP-ID	no (	target	38% global	Analysis of replies	Discover	the
			web-servers)			to active probing	amount of load
							(HTTP requests)	balanced servers,
									measure	the
									tra c generated
									by a server.
	[ ]	-	no		57%	global,	-		O -path	DNS
					14% local, 9%			cache	poison-
					constant, 20%			ing attacks and
					mixed IP-ID, 1%			defense against
					random/other			them	through
					( a )				DNSSEC valida-
									tion.
	[ ]	IP-ID	no		16% global	TCP SYN-ACK from	Reveal Internet
		accelera-					multiple	vantage	censorship.
		tion					points

  14% local, 9% constant, 1% random/other. Additionally, [ ] suggests that 20% of DNS TLD exhibit evidence of "two or more sequential sequences mixed up, probably due to multiple machines behind load balancer".The remaining works concentrate instead on assessing the popularity of just the global implementation being it only the focus of their studies, proving once again the relevance of a Internet-wide list comprising IP addresses generating IP-ID with the afore-ID behavior, in this work we not only consider all expected IP-ID behavior, but additionally quantify non-standard behaviors: in particular, we provide a methodology

	Speci cally, the sole quantitative assessment of IP-ID behavior over multiple classes dates back to . This is limited to Top Level Domains TLDs probed by [ ] (whose main aim is to propose practical poisoning and name-server blocking attacks on stan-dard DNS resolvers, by o -path, spoo ng adversaries). In particular, the study nds , [ ] reported that 70% (over probed targets) were using an IP-ID counter (global or local implementation); in , [ ] reported that 38% (over hosts) used a global IP-ID; in , [ ] a rms the global implementa-tion to be the most common assignment policy (among behaviors). . . IP-ID Based-Inference Additionally, the IP-ID has been exploited for numerous purposes in the literature. No-tably, IP-ID side-channel information helped to discover load balancing server [ ], count hosts behind NAT [ , ], measure the tra c [ , ] and detect router alias [ , , ]. More recently, [ ] leverages IP-ID to detect router aliases, or infer router up time[ ] and to reveal Internet censorship [ ], refueling interest in the study of IP-ID behavior. Whereas the above work [ , , , , ] mostly focus only on the 57% global, mentioned behavior. Namely, in global IP-

Table :

 : Summary of the Datasets.

	Name L	Type Real		Description Large scale measurements	Properties 78.4% of targets Presence of presence of receive all the probes	Size [Targets] , M	URL [ ]
		Measurements	dataset comprising the IP-ID	odd behaviors of the IP-
				sequences received from the portion of hitlist [ ] providing response rate 80%	ID, possibility of losses 89% of targets receive or out-of-order packets more than 80 probes
	G	Real		Manually labeled dataset con-	Targets chosen to pro-	k	[ ]
		Measurements	taining the IP-IDs contained	vide IP-pre x level and
				in the replies of a set of IP	class balance, presence
				addresses sampled uniformly	of odd behaviors, used
				from the hitlist to guarantee	for training and classi -
				class balance	cation of L
	G	Real	Manually labeled dataset con-Figure : Internet campaign: ECDF of the number of packet replies Targets chosen to k	[ ]
		Measurements	taining the IP-IDs from the	provide IP-pre x level
	replies of a set of IP addresses where % of it belong to the c, we decide to be conservative: we preliminary probe the 6 millions potential targets imbalance, presence of odd behaviors,
	same IP/ subnet sending two ICMP echo requests, and include in our nal target list the approximately used for validation of performances
	S ideal 3, 2 million responsive hosts (in line with [ , Synthetic Dataset manually designed	Lossless, absence of	k	[ ]
				to intentionally contain the	odd behaviors, used
				four possible IP-ID imple-	for	validation	of
				mentations in the ideal case	performances
				evenly distributed emulating
				the replies collected through
				real measurements
	S lossy	Lossy		Dataset manually designed to	Lossy, absence of odd	k	[ ]
		Synthetic	intentionally contain the four	behaviors, used for test-
				possible IP-ID implementa-	ing resilience to losses
				tions spoiled with four di er-
				ent avour of losses (S lossy = ∪(S unif. , S hole , S extr. , S equi. )) evenly distributed
	S reorder Synthetic	G dataset spoiled when of %	Used for testing re-	k	[ ]
				of each IP-ID sequence is inten-	silience to sequence
				tionally randomly swapped	alteration	due	to
					out-of-order packets
	Large scale census L: The rst dataset is made up of real measurements coming from
	a large scale measurement campaign and includes the replies coming from a subset of a
	hitlist of alive IP addresses [ ]. We avoid putting stress on the infrastructure carrying a
	full Internet census: as we aim at providing an accurate picture of the relative popularity
	of IP-ID implementations on the Internet, it su ces to collect measurements for portion
	of targets, namely 1 alive IP/ host per each / pre x. For this reason, for the targets
	selection, we rely on the public available hitlist regularly published by [ ], comprising
	16 millions of targets IP/ . The hitlist contains targets for all / , including those who
	have never been replying to the probing: excluding them from our target list, leaves us
	with approximately 6 millions of potential targets. To reduce the amount of probe traf-

  validation, whose results are reported in Fig. as a confusion matrix: we can observe that the classi er is extremely accurate, with 100% true positive in the constant and local classes, 99%Figure : Validation: Relative importance for the most useful features of the classi er.for the random and 98% for the global class. The worst case is represented by 95% true positive for the odd class (that represent only 7% of the samples): these very few misclassi cations are erroneously attributed to local, global or random classes, and addi-

	Relative Importance	0.1 0.2 0.3						H(.) σ(.) E[.]
		0.0	H(s)	σ x	E(y )	σ s	H(x )	σ s
					Features

tional series de nition (e. g., to compensate for wrong endianness) could help reducing if needed. For completeness, we compare the results obtained with the Decision Tree [ ] with the ones achieved with Random Forest. Results, shown in Fig. , again as a confusion matrix, show that the small misclassi cation gaps introduced by the Decision Tree are fully lled when using a Random Forest Classi er, which leads to 100% classi cation accuracy for all the classes. Additionally, Fig.

reports the importance for the most useful features of the classi er. Four main takeaways can be gathered from the picture: rst, just four features are necessary for a full discrimination, which is reasonable as the cardinality of the classes to discriminate is small; second, as expected features that measure the dispersion (entropy and standard deviation) are prevalent; third, both original and derivative sequences are useful in the detection; fourth, subsequence metrics are highly redundant (i. e., H(x) = H(y), σ x = σ y , etc.).

Table :

 : Features values for both lossless and lossy synthetic dataset S lossy with % losseslocal implementation case of IP-ID.

		S ideal		S lossy
	Feature Lossless Uniform	Hole	Extremal Equidistant
	H(s)	.		.	.	.
	H(x )		.	.		.
	E[y ]		.	.		.
	σ(x)	.	.	.	.	.
	σ(s)	. e 3	. e 3	e 3	. e 3	. e 3
	σ(s )	. e 3	. e 3	. e 3	. e 3	. e 3

  a), where we additionally report the breakdown in our G training set comprising just 1, 855 population samples: it can be seen that while G has no statistical relevance for the census, it is not a ected by class imbalance and thus proves to be a good training set.

		0.4			Decision Tree Classifier
	Normalized Occurrences	0.1 0.2 0.3			Random Forest Classifier
		0.0	Constant	Local	Global	Random Oddities
	Figure : Breakdown of the classes of L obtained with both a Decision Tree and a Random
	Forest Classi er.
	This decreasing trend, summarized in Fig.	(b), is possibly a ected by the compara-
	bly smaller population size of early studies. However, we believe this to be rooted into
	OS-level changes in IP-ID policy implementations: e. g., Linux and Solaris, which pre-
	viously adopted a global counter, for security reasons later moved to a local counter
	implementation [ ].
	Results are particularly interesting to put in perspective with current literature knowl-
	edge. Speci cally, past work [ , , ,	] consistently reported the global counter
	to be more widespread: in		, [ ] 70% ; in	, [ ] 38%; in	, [ ] a rms
	the global implementation to be the most common assignment policy; in	, [ ]
	57%. On the contrary, we nd that only 18% (over 2, 5 million targets) are still using
	global counter implementation: this in line with	results that reports slightly more

than 16% global IP-IDs [ ] (whose main aim is to detect censorship in the Internet). By comparing our results with the only one providing the occurrences of both the normatives-compliant IP-ID behaviors and some odd practices [ ], the study (our census) nds 57% (18%) global, 14% (39%) local and 9% (34%) constant IP-IDs, which testify of a signi cant evolution. Additionally, recalling that [ ] suggests that 20% of DNS TLD generate mixed IP-IDs, we nd out that this is much larger than the 7% fraction of the larger "odd" class (including but not limited to load balance) that we nd in this work. Finally, despite recommendations [ ], the percentage of random IP-ID sequence was (and remains) limited 1% (2%). For completeness and in light of what showed in Sec. . . , we compare the results obtained with the Decision Tree those of the Random Forest. From the outcomes reported in Fig. we can observe that no statistical di erence is present in the two cases.

  Speci cally, the available readily usable list of the approximate half million hosts with global IP-ID implementations global implementations [ ] can make work such as [ , , , ] still possible. Moreover, by updating and consolidating the scattered knowledge [ , , , , ] of IP-ID prevalence, this work contributes in re ning the current global Internet map. or negative ( . %) experiences, each one associated with over features. The collected dataset is interesting per se, as it is particularly heterogeneous, comprising k distinct IP addresses, browsers, . k ISPs and . k hardware devices. A portion of it, including features which ensures that no sensitive information allowing to deanonymize Wikipedia visitors is present, is made available to the research community as we hope this can help in re ning the understanding of Web users' experience.

Table :

 : List of features, informing whether each comes from raw data or is derived, and if it is present in the T, WWW or PA set.

Table :

 : Schema of the Public Available Features.provided in a separate dataset together with the user survey answer, where the order of the entries has been randomly shu ed in order to prevent user deanonymization wiki Which wiki the request was on (ruwiki, cawiki, eswiki, frwiki or

		enwikivoyage)
	unload	The time spent on unload (unloadEventEnd -unloadEventStart).
	redirecting 5	Time spent following redirects.
	fetchStart 5

  alexa.com/topsites. [ ] [Online]. Available from: https : / / sites . google. com / a / webpagetest . org / docs / using-webpagetest/metrics/speed-index. [ ] [Online]. Available from: https://webqoe.telecom-paristech.fr/data/. [ ] [Online]. Available from: https://xkcd.com/ /. [ ] [Online]. Available from: https://bit.ly/ VW Gnd. . [ ] E. Aguiar et al. "Video quality estimator for wireless mesh networks. " In: IEEE th International Workshop on Quality of Service. . [ ] Tim Altho et al. "Harnessing the web for population-scale physiological sensing: A case study of sleep and performance. " In: Proc. The World Wide Web Conference (WWW). . Alemnew Asrese et al. "Measuring Web Quality of Experience in Cellular Networks. " In: Proc. Passive and Active Measurement Conference (PAM). . [ ] Eytan Bakshy, Dean Eckles, and Michael S. Bernstein. "Designing and Deploying Online Field Experiments. " In: Proc. The World Wide Web Conference (WWW). Athula Balachandran et al. "Modeling Web Quality-of-experience on Cellular Networks. " In: Proc. ACM MOBICOM. ACM, . [ ] Christine Basta, Marta R. Costa-jussà, and Noe Casas. "Evaluating the Underlying Gender Bias in Contextualized Word Embeddings. " In: Proceedings of the First Workshop on Gender Bias in Natural Language Processing. Association for Computational Linguistics, .

	Management . (	), pp.	-.
	[ ] .
	[ ]

[ ] Apache Hive. [Online]. Available from: https://hive.apache.org/. [ ] Approximate ATF chrome extension. [Online]. Available from: https://chrome. google.com/webstore/detail/approximate-atf/eedmonedcfjniaagehchbkdolbobmfhb. [ ] A. S. Asrese et al. "Measuring Web Latency and Rendering Performance: Method, Tools, and Longitudinal Dataset. " In: IEEE Transactions on Network and Service [ ] Steven M Bellovin. "A technique for counting NATted hosts. " In: Proc. Internet Measurement Workshop (IMW).

.

This manuscript o ered a walk through di erent studies aimed at providing an overview of some of the most prominent aspects of modern Internet. Here we followed two main research directions: rst, we focused on the analysis of Web users' Quality of Experience (Ch. and Ch. ); second, we gave emphasis on the use of machine learning applied to Internet measurements, studying, on one side, the impact of its interaction with the Web (Ch. ); and, on the other, its use to predict objects generated by machines (Ch. ).

We next summarize the achievements of this thesis work and discuss possible future research directions.

.

In the rst part of this thesis we focused on the measurement and on the analysis of the Quality of Experience of users browsing the Web.

. . Users' acceptance in the Wild Web

In Chapter we tackled the problem of assessing the quality of experience on a popular website in operation. Speci cally, we do this by gathering the user acceptance of Wikipedia, over k user answers, more than twice the survey responses collected in similar large-scale Wikipedia studies. We collect either positive ( . %), neutral ( . %) En cons équence, l'objectif de cette th èse est d'apporter un nouvel éclairage sur certains des d éfis qui ont émerg é ces derni ères ann ées. En particulier, nous fournissons une analyse approfondie de certains des aspects les plus importants de l'Internet moderne. Un accent particulier est mis sur le World Wide Web, qui, parmi tous, est sans doute l'une des applications Internet les plus populaires, et un regard sp écifique sur son interaction avec l'apprentissage automatique. La premi ère partie de ce travail étudie la qualit é de l'exp érience de navigation des utilisateurs sur le Web, avec des mesures effectu ées à la fois "in the wild" et dans des environnements contr ôl és. Nos contributions continuent avec une analyse originale de l'avis subjectif des utilisateurs et des mesures objectives de la qualit é d'exp érience, montrant la difficult é de construire des mod èles supervis és pr écis, bas és sur des donn ées, capables de pr édire la satisfaction des utilisateurs, ainsi qu'une discussion approfondie de la nature multimodale des avis subjectifs des utilisateurs. Dans la deuxi ème partie de ce travail, nous analysons et discutons l' équit é des mod èles de langage bas és sur des transformateurs de pointe, qui sont pr é-entraîn és sur des corpus bas és sur le Web et qui sont g én éralement utilis és pour r ésoudre une grande vari ét é de t âches de traitement du langage naturel (NLP). Nous nous demandons ici si la taille et l'h ét érog én éit é du Web garantissent la diversit é des mod èles. Le coeur de nos contributions repose sur la mesure du biais int égr é dans les mod èles, que nous discutons sous diff érents angles. Enfin, la derni ère partie de cette th èse traite de la classification d'objets g én ér és par des machines à l'aide de certains des plus simples algorithmes d'apprentissage automatique supervis és à l' état de l'art. Gr âce à un framework solide mais peu intrusif, nous montrons que les diff érents comportements d'un champ du paquet IP, l'identification IP (IP-ID), peuvent être facilement classifi és avec peu de caract éristiques ayant un haut pouvoir discriminatoire. Nous appliquons enfin notre technique à un census à l' échelle de l'Internet et fournissons une vue actualis ée de l'adoption de ses diff érentes impl émentations dans l'Internet.

Title : Longitudinal, large-scale and unbiased Internet measurements

Keywords : Internet Measurements, Quality of Experience, Machine learning Abstract : Today, a world without the Internet is unimaginable. By interconnecting billions of people worldwide and by offering an uncountable number of services, it is now fully embedded in the modern society. Yet, despite technology evolution and development, its pervasiveness and heterogeneity still raise new challenges, such as security concerns, monitoring of the users' Quality of Experience (QoE), care for transparency and fairness. Accordingly, the goal of this thesis is to shed new light on some of the challenges emerged in recent years. In particular, we provide an in-depth analysis of some of the most prominent aspects of modern Internet. A particular emphasis is given on the World Wide Web, which among all, is undoubtedly one of the most popular Internet applications, and a specific regard to its interaction with machine learning. The first part of this work studies the Quality of Experience of users' browsing the Web, with measurements led both in the wild and in controlled environments. Our contributions follow with an original analysis of both the subjective user feedback and the objective QoE metrics, showing how hard it is to build accurate supervised data-driven models capable to predict the user satisfaction, along with an in-depth discussion of the multi-modal nature of the subjective user opinions.

In the second part of this work, we analyze and discuss the fairness of state-of-the-art transformer-based language models, which are pre-trained on Web-based corpora and which are typically used to solve a wide variety of Natural Language Processing (NLP) tasks. Here, we question whether the sheer size and heterogeneity of the Web guarantee diversity in the models. The core of our contributions rests in the measure of the bias embedded in the models, that we discuss under different angles. Finally, the last part of this dissertation addresses the classification of objects generated by machines through some of the simplest state-of-the-art supervised machine learning algorithms. Through a minimally intrusive, robust and lightweight framework, we show that the different behaviors of a field of the IP packet, the IP identification (IP-ID), could be easily classified with few features having high discriminative power. We finally apply our technique to an Internet-wide census and provide an updated view of the adoption of the different implementations in the Internet.

Institut Polytechnique de Paris 91120 Palaiseau, France