Diana Joumblatt

Renata Teixeira

Jaideep Chandrashekar

Nina Taft

HostView: Annotating end-host performance measurements with user feedback

INTRODUCTION

Network disruptions can adversely impact a users' web browsing, cause video/audio interruptions, or render web sites and services unreachable. Such problems are frustrating to Internet users, who are oblivious to the underlying problems, but completely exposed to the service degradations. Ideally users' end systems would have diagnostic tools that can automatically detect, diagnose and possibly repair, performance degradations. Hopefully, this can be done without user intervention. Clearly, the first step for any such (end-host) diagnostic tool is a methodology to automatically detect performance degradations in the network that can affect a user's perception of application performance.

While empirical network performance studies have been carried out in the past, the bulk of these have focused on well established performance metrics-delay, loss rate, throughput. Very little attention has been devoted to how these metrics affect end-users' interactions with applications that rely on the network. Understanding the more subjective quality of experience has been limited to a few niche applications, for instance VoIP [START_REF] Jiang | Modeling of packet loss and delay and their effect on real-time multimedia service quality[END_REF], online-gaming [START_REF] Chen | How sensitive are online gamers to network quality?[END_REF], and video playback [START_REF] Tao | Real-time monitoring of video quality in IP networks[END_REF][START_REF] Chen | Oneclick: A framework for measuring network quality of experience[END_REF]. Building such application specific models are useful in improving the specific application, but cannot be easily generalized to capture a particular user's "online experience", which is likely to span a number of different applications. Moreover, building application-specific models requires a great deal of domain knowledge.

We argue for building a general purpose tool that can detect network performance degradations that affect users' quality of experience. Moreover, we wish to carry out this detection with the user's mix of applications and working environment. This tool is fundamentally hard to build. First, perception of performance varies dramatically across users and even for the same user according to her mood and expectations. A user is more likely to expect better performance from her network at work, than from a free wireless at a coffee shop. Second, the quality of experience is very tightly coupled to the application. For instance, a sudden increase in round-trip times (RTTs) may have no discernable impact on a video connection (playback buffers mask jitter), but it can make a remote login session unusable. These factors clearly imply that we cannot really study network performance (or its degradation) without taking into account the application, the end-user, and the environment. Understanding how to construct a diagnostic tool requires analysis of network performance data annotated with users' perception of network quality. However, no such dataset exists today! A few end-host measurement datasets exist [START_REF] Guha | How Healthy are Todayís Enterprise Networks?[END_REF][START_REF] Cooke | Reclaiming Network-wide Visibility Using Ubiquitous Endsystem Monitors[END_REF][START_REF] Simpson | NETI@home: A distributed approach to collecting end-to-end network performance measurements[END_REF], but these lack the all important annotations of users' perceived quality. Other studies incorporate user feedback with the performance measurements, for specific applications [START_REF] Chen | Oneclick: A framework for measuring network quality of experience[END_REF], but the data is not collected at the end-host. Clearly if one is to correlate a user's perception of performance with that of the network, applications and mobile environment her device operates in, the data collection utility must reside on the end-host itself.

In this paper, we describe the design of HostView, an endhost measuring tool to collect network performance data annotated with users' perceived quality of the network. The questions of what data to collect, and how to collect it are fundamental design questions that are also driven by the user. Certainly, the more data a tool collects the better it will be able to do diagnosis. But privacy concerns, while subjective, will govern the types of data collected. The mechanism for collecting data is also critical because heavy handed methods that affect the basic performance of the machine will deter a user from installing such a tool. Thus a study of the overhead of any measurement method is imperative in the design of end-host tracing tools. We describe how we came to our design decisions regarding what type of data to collect, and using our first prototype we present the overhead of candidate collection techniques ( §2).

Another key design question focuses on how to extract feedback from users across different network conditions with a minimum level of annoyance ( §3). We build a second prototype of HostView1 that includes an algorithm to sample user experience three times a day. Our two-week experiments with a small set of users show that users' unpredictable behavior is an important challenge for any sampling algorithm. Our long-term goal is to collect traces via a largescale campaign that will enable a broad research agenda.

END-HOST DATA

We concentrate on collecting three kinds of data: (i) traffic and network performance statistics, (ii) application level context (for traffic flows), and (iii) system performance and environmental data (network type, etc). This section explores what specific data should be collected within these broad categories and also evaluates the accuracy versus overhead tradeoff of candidate collection methods. The performance study was carried out with a first implementation of HostView that runs on MacOS X and Linux, via a small pilot with 7 students from LIP6 who ran the tool on their laptops for 2 weeks.

Network performance

Active network probing, as a mechanism to measure network performance, is appealing because it sidesteps issues related to recording potentially private information (as compared to passively recording traffic). However, the performance inferred by active probing might not reflect that of any particular application: the destinations are different; probe packets may be handled differently from those of applications; the probing may happen outside of performance episodes, and so on! Thus, to overcome these difficulties, HostView uses passive measurements and incorporates a number of mechanisms to allay privacy concerns with recording traffic. With the packet traces, we can infer RTTs, losses, and reachability issues of active application connections [START_REF] Zhang | PlanetSeer: Internet Path Failure Monitoring and Characterization in Wide-area Services[END_REF]. We log the traffic, rather than simply log summary statistics as in [START_REF] Simpson | NETI@home: A distributed approach to collecting end-to-end network performance measurements[END_REF], because the summaries are not fine-grained enough to allow detailed analysis on specific connections. Logging all the traffic packets is easily accomplished with the libpcap library; which has been extensively tested and optimized. Previous results show that the overhead with libpcap is under 20% in the worst case [START_REF] Schneider | Packet capture in 10-gigabit ethernet environments using contemporary commodity hardware[END_REF] (full packet capture on 100Mbps Ethernet). In our own testing, we never saw the overhead exceed 5%; thus, we do not expect the overhead of libpcap to be significant.

Recording packets raises issues with some users about privacy; few people are readily willing to run such a measurement tool. To understand this issue better, we conducted a user survey of 400 computer scientists to assess user discomfort with specific kinds of data being logged. Our results, summarized in [START_REF] Joumblatt | Perspectives on Tracing End-Hosts: A Survey Summary[END_REF], lead us to believe that a tool that is restricted to recording packet headers (and not the entire packet) and which appropriately anonymizes the traces, would be acceptable to a large fraction of users.

To this end, HostView incorporates a number of privacy protection features. First, it removes any information that identifies the host: users are identified by a randomly generated id, which is used to prefix trace file names. Source IP addresses are anonymized with a SHA-256 hash. In addition, the traces are periodically uploaded using s-ftp and the server where the traces reside has very restricted access controls. Third, HostView allows users to pause the logging (all logging is disabled in this period) in one hour increments, when the user carries out any activity they do not want recorded.

Application context

To understand how individual applications are affected by network events, we also need to associate network flows (reconstructed from the packet traces collected by libpcap) with the application (executable) that is terminating the flow and also correctly identify the application protocol embedded in the stream of packets. The traffic stream seen at the end-host is a multiplexing of the traffic from several individual applications (Firefox, Yahoo Messenger, Skype, etc.), each of which uses a multiplicity of application protocols (HTTP, FTP, SMTP, RPC, etc.). Using destination ports alone (without other information) of a network flow is reputed to have limited value in understanding which application generated the flow. This is because the destination port numbers are often overloaded, or used in unorthodox ways (applications often tunnel their protocols over HTTP to traverse firewalls, as an example). In the following, we describe mechanisms to associate these two labels with net-work flows seen at the end-host.

Identifying Application name: There are several APIs available that allow mapping network flows to application names (in short, app name) or process identifiers-ETW [START_REF]Event Tracing: Improve Debugging and Performance Tuning with ETW[END_REF] and lsof are two in particular. However, they are often platform specific (ETW is only supported in Windows), or come with very high overhead (lsof), which makes them hard to use for a broad deployment. Among the options we considered, the gt toolkit [START_REF] Gringoli | Gt: picking up the truth from the ground for internet traffic[END_REF] was found to have the best balance of platform support (it runs on our target platforms), and low overhead, and this has been integrated into HostView. This particular method periodically samples the kernel socket tables, which contain mappings between applications and sockets, and associates outgoing flows with the appropriate socket entry. This method has an intrinsic tradeoff between sampling interval and coverage: shorter intervals are likely to label more flows, but at a higher cost. Longer sampling intervals reduce the overhead, but tend to label fewer flows (missing those that start and terminate between sampling periods). This tool has been profiled extensively in previous work [START_REF] Gringoli | Gt: picking up the truth from the ground for internet traffic[END_REF]. The results showed CPU loads of 5% with 1 s polling intervals, and 20% when the interval is reduced to 125 ms. In our own testing, we also ran it with a more extreme sampling interval of 0.01 s and with this sampling rate, we observed average CPU loads of 59%. While this did give us a very small increase in classification coverage (a few percentage points), the substantial increase in the CPU loads makes this operating point undesirable; the default setting of 1 s seems like a reasonable choice, and this is the sampling interval that is used in HostView.

Identifying Application protocol: The naive method to associate a network flow with the application protocol is to simply consult the IANA list of well known port numbers (/etc/services in most *nix distributions). This method tends to be unreliable when applications use non-standard or dynamic port numbers. More robust application protocol classifiers include those that scan for known signatures in payloads [START_REF]Appln. layer classifier for linux[END_REF], and statistical methods that apply learned traffic models to traffic distributions. While such methods are well matched for traffic collected at network gateways [START_REF] Kim | Internet Traffic Classification Demystified: Myths, Caveats, and Best Practices[END_REF], they have costly CPU requirements, and a careful evaluation of their feasibility on end-hosts has not yet been carried out.

To understand whether the more complex classifiers can practically be deployed on end-hosts, a first implementation of HostView incorporated the l7 classifier [START_REF]Appln. layer classifier for linux[END_REF]. This particular instance has built into it a comprehensive database of protocol signatures. We choose this particular payload classifier since it is considered the authoritative payloadbased classifier and has been extensively used to determine "ground-truth" in other studies of classification accuracy. We do not include port-based classification or any of the statistical-based methods in this version of the tool, because these can be run offline using the packet header traces we already export to the server (l7 inspects the full packet payload, and hence cannot be run on packet headers alone). In the data collected from our pilot deployment over 7 users, we observed that CPU load of l7 is directly proportional to traffic load; we also observed frequent episodes where the CPU load spiked very high (this was found to correlate with bursts in traffic volume). Since we expect traffic to be bursty, it would seem that running l7 on a user's production end-host would not be recommended and we will have to resort to alternate methods of performing the classification.

Using the traces from our initial pilot, Table 1 presents the fraction of flows and bytes, that were correctly classified either by the l7 method, and the port inspection method. The row labeled l7=port shows the flows and bytes for which both methods gave the same classification. The last row presents the fraction of flows and bytes with which gt could associate an application name.

Comparing the results of port-and payload-based classification in Table 1, we see that the two techniques return the same application protocol for 86.43% of bytes. This fraction is even higher if we only consider TCP connections (over 94% of flows and bytes), which confirms a recent study with residential DSL customers [START_REF] Maier | On dominant characteristics of residential broadband internet traffic[END_REF]. We find surprisingly that the port-based classifier does reasonably well. Similarly, gt is able to assign an application name to almost all the TCP connections. Therefore, the combination of port inspection to obtain application protocol and gt for application name suffices for labeling TCP connections. Although a different application mix would lead to different classification accuracy for port inspection, the application name should allow us to identify these cases. For instance, one of our traces contains BitTorrent traffic. Although the connections use port 80, the application name contains the string "BitTorrent".

The fraction of flows and bytes labeled by l7 and by gt is much lower for UDP. We examine UDP destination ports of all unlabeled flows both for gt and l7 and find that the vast majority of these flows are broadcast and multicast traffic on the local network (for instance, NetBIOS, MDNS, and UPnP). If we assume that these ports are always used consistently, and hard code these into the classifier, the results are remarkably better (over 90% accurate classification). Port inspection is the only technique to infer the application context for UDP flows.

HTTP traffic accounts for a very large fraction of the bytes seen in our traces. Given our overall goal, we feel it advantageous to also record the content-type fields (video, audio, images, or text) to gain an understanding of the type of application being transported. However, this does require some form of online payload inspection, just to extract the leading portion of the packet (we find that the first 500 bytes are sufficient to identify the content-type in most cases).

To sum, HostView logs application names associated with open sockets every second using the gt toolkit. It does not incorporate any online methods to identify the application protocol; this identification is relegated to the back end server where the traces are uploaded. Finally, HostView tries to identify the content-type from looking at the first few packets of potential HTTP responses.

System performance and environment

We also log system and environment data to broaden our understanding of the end-host's performance. 1) HostView logs CPU load every second and this data is uploaded to a back-end server. 2) The active network interface is recorded whenever it changes; if the active network is wireless, a popup asks the user to assign a label to the network (work, airport, coffee-shop, etc) when the wireless network SSID is encountered for the first time. As a privacy mechanism, HostView only logs a hash of the SSID (and not the SSID text string) along with the user supplied descriptive label for the network. HostView periodically logs a few wireless network performance indicators 3) received signal strength, 4) noise level and 5) last transmission rate. 6) HostView also attempts to find the home autonomous system (AS) that the end-host is in each time its IP address changes. This is carried out with a 3-hop limited traceroute to google.com after which the IP addresses of each hop are mapped to ASes using an IP-to-AS mapping tool; only the AS numbers are actually exported from the end-host. We collect the AS information to have the ability to understand how a user's perception of performance can vary when connected to different ASes.

USER FEEDBACK

While network performance can be objectively measured, the actual impact felt by the user is much harder to capture and quantify. At the same time, capturing the user's perception of network performance is crucial to understanding how, and to what extent, network disruptions affect different users and applications. Motivated by the extensive user studies research carried out in the HCI community, we incorporated user feedback mechanisms directly into our tool.

Questionnaire design

We capture user feedback by asking people to fill out a set of questions about their network connection quality. Interesting tradeoffs arise when designing such a questionnaire; on the one hand we want the questionnaire to be short so that users will take the time to fill it out, but on the other hand the more information we obtain the better for our research. We limited our questionnaire to 5 questions (a common amount in HCI surveys [START_REF] Consolvo | Using the Experience Sampling Method to Evaluate Ubicomp Applications[END_REF]) because more people are willing to participate when the questionnaire takes less than a minute or two to complete. User feedback can be both quantitative (e.g. rating performance on a scale, or things for which statistical summaries can be produced) and qualitative (e.g., free-form text). Multiple choices are easier to interpret than open format questions, but they can limit the feedback a user can provide. To balance these issues, our questionnaire uses a mix of both quantitative and qualitative. For example, we ask the user to rate the quality of their perceived network connection on a standard Likert scale [START_REF]Likert scale[END_REF] from 1 to 5 (widely used scale in survey research). We also ask the user to list which applications (if any) experienced problems accessing the Internet in the last 5 minutes prior to the questionnaire. For qualitative input, users can enter (via a free form text box) any additional comments they would like describing their computer's performance (at the application/network/machine level); this includes even potential hypotheses the user might have as to why poor performance occurred.

Triggering the questionnaire

There are two approaches to trigger the questionnaire in order to elicit user feedback: system-or user-driven. The first approach, Experience sampling (ESM) [START_REF] Csikszentmihalyi | Validity and Reliability of the Experience-Sampling Method[END_REF], is widely adopted in the HCI community [START_REF] Consolvo | Using the Experience Sampling Method to Evaluate Ubicomp Applications[END_REF]. By triggering the questionnaire using system measurements such as throughput or CPU load, the user can be queried over a range of performance levels that should theoretically translate into a range of annoyance/satisfaction levels.

The second approach is user-driven and is implemented with an "I am annoyed" button; the user is encouraged to hit a button at any moment when they are unsatisfied with the performance. This mechanism was first proposed with the OneClick framework [START_REF] Chen | Oneclick: A framework for measuring network quality of experience[END_REF] where it was coupled with a single specific application. Our scenario differs in two ways: (i) OneClick runs at a web server with pre-recorded content and ours runs on the end-host itself; (ii) we do not know a priori which application the user is annoyed with, whereas OneClick runs in the context of a single specific application.

These two approaches have different advantages. ESM will sample more evenly throughout the performance range which is useful because we want to understand the range of user satisfaction across levels of performance. However, we also seek extra samples at moments of performance problems since our end goal is to be able to do diagnosis. We thus need extra questionnaires from these rarer events. The I'm annoyed button has the benefit of obtaining more samples at poor performance epochs. We thus elected to incorporate both mechanisms to obtain the benefits of each.

Experience sampling algorithm: The purpose of the experience sampling algorithm is to decide under what conditions and when to pop up the questionnaire to obtain user feedback. One could simply trigger the questionnaire at random epochs. However, as one might guess, and as the endhost data we obtained so far indicates, most of the time a user's machine is lightly loaded, and thus a purely random scheme results in large numbers of user samples at low load. Although there are many different phenomena that can underlie a performance degradation, we simplify things by using network load as a proxy for performance assuming that higher user annoyance usually co-occurs with a heavily loaded network. It is thus intuitive that a weighted random sampling scheme which weighs more heavily the higher load epochs will yield a more even distribution of samples. (Each sample here refers to a single completed questionnaire.)

The basic form of our sampling algorithm incorporated into HostView is presented in Algorithm 1. The algorithm has six static parameters (α l , αm, α h , γ l , γm, γ h ,) and three adaptive parameters (T l , Tm and active-time) that are adjusted per user and updated on a daily basis. The thresholds T l and Tm denote load ranges in the traffic. The throughput, denoted throu, is considered low if 0 ≤ throu ≤ T l , medium if T l < throu ≤ Tm, and high if throu > Tm. These parameters are computed per user as the 85 and 95 percentile values of the user's empirical distribution. These parameters are adaptive because they are updated each day. On a given day, the value is taken from the same day of the previous week. If no such measurements exist, we use data from the previous day. The sampling is bootstrapped by setting T l = 117Kbps and Tm = 727Kbps; these values were extracted from the data collected in the first pilot study.

The α parameters capture the fraction of time a user spends in the different load ranges. Since we do not know this a-priori, we make a reasonable guess and set α l = 0.85, αm = 0.10 and α h = 0.05 implying that a user spends 85% (10%, 5%) of the time in a low load state (medium, high, respectively). The γ parameters, γ l , γm, γ h , are the probabilities of taking a sample when the load is low, medium or high respectively. These are set to 0.02, 0.4 and 0.9. Finally, the parameter N is a polling interval. Our algorithm works as follows. Every N seconds, the ESM algorithm computes the load and determines whether it is low, medium or high based on the values of T l and Tm. Depending on the current state, the algorithm computes a random value and compares it to the corresponding γ values. The decision of whether to pop-up the ESM questionnaire is based on the outcome of this comparison and also on whether the user was actually using his machine in the previous minute. Our goal is to obtain a maximum of three completed questionnaires per day; this number was chosen based on the results of our survey [START_REF] Joumblatt | Perspectives on Tracing End-Hosts: A Survey Summary[END_REF]. We require that the samples be complete; a deferred sample (users are allowed to postpone the questionnaire), or a timeout (the questionnaire goes away after 6 minutes of the pop up) does not add to this number. Thus the selection of the polling interval N is important.

The choice of N presents a fundamental tradeoff: if N is small and we sample too much, user annoyance increases because either there are too many questionnaires or the questionnaires occur too close in time. On the other hand, if N is too large, we won't get enough samples to perform diagnostic analysis. We limit the maximum number of questionnaires per day to 4, and design the algorithm to hit an average between 2 and 3 ESM questionnaires per day. There are various heuristic schemes that could be selected to choose N , including static and adaptive approaches. In our pilot study, we elected to use an Adaptive Polling Interval scheme that computes N = (0.102 /4) * active time where 0.102 is the average probability of taking a sample at any interval, 4 is the maximum number of questionnaires per day, and active time is defined as the number of seconds per day the user is actually using the keyboard or the mouse of their computer. Each new day, before the ESM algorithm starts, HostView predicts the active time value for the day based on the same day of last week, or if that is not available, then it uses the active time of the previous day. We set the minimum value of active time to be three hours.

When the questionnaire pops up, the user has the choice to answer it immediately, to postpone it, or to ignore it. If it is ignored, the questionnaire will "timeout" and disappear after 6 minutes. If the user answers the questionnaire, then we return to the basic algorithm to select the next time to pop up a the questionnaire. However, if the user postpones or ignores it, then they were be asked again in 30 minutes, on average.

Takeaways from pilot study

We conducted a second pilot study of two weeks with four LIP6 students to evaluate the user feedback mechanism. In this experiment, we collect packet headers, application context, system performance logs and user feedback. Table 2 presents the number of ESM samples per user, the number of days each user ran HostView, and the 85 th and 95 th percentile of the load distribution of each user. A sample is labeled low, medium or high depending on the value of the instantaneous load with respect to T l and Tm.

We make a few observations from this pilot study. First, we see that a third or more of the samples collected for each user occurred during heavy network load. This reflects our choice of giving a higher weight to sampling heavy traffic episodes than sampling light episodes and confirms correct implementation of the approach. Second, the last two columns of Table 2 justify HostView's design decision to customize the load thresholds to each user. The distribution of load for User 1 and User 4 are dramatically different; in fact the 85 th and 95 th percentiles of User 1 are not even in the same order of magnitude as User 4. Third, users behave differently when it comes to dealing with the ESM questionnaire. For instance, User 1 only used the "Ask me later" option to defer the questionnaire twice, but his/her questionnaire timed out 26 times. User 2, however, deferred answering the questionnaire 38 times. This justifies the design decision to include multiple options for answering, postponing, and timing-out the questionnaire because all of these end up being used. Fourth, we observe that the total number of times the questionnaire popped up varies across users. Users 2 and 4 ran HostView for a roughly similar amount of time (12 and 10 days). However, user 2 received 43 pop-ups while user 4 was only queried for feedback 18 times. In this case, the reason for this variation is the behavior of user 2, who only filled in 15% of the pop-ups, which forced the ESM algorithm to oversample. In other cases, the variation in the total number of pop-ups is also due to the sampling probabilities (see Alg. 1), the adaptive thresholds and whether the user was actively using her machine one minute before the pop-up.

In our pilot study, some users complained that pop-ups were too close in time. After studying their data, we realized that our predictors for user active time could be quite erroneous and that user active time is not easily predictable from their behavior on the previous day or the same day the previous week. We evaluated the correlation between completed responses and the frequency of pop-ups as an indication of user annoyance. Figure 1 plots the fraction of unanswered pop-ups as a function of the distance to previous pop-up. We observe that when two pop-ups happen within less than one hour, users are less likely to answer them (over 50 % for all users). Therefore, we decided to modify our algorithm to ensure a one hour minimum gap between popping up two consecutive questionnaires.

Tradeoff: Polling frequency vs. number of daily samples

Because our pilot study showed that it is very difficult to predict a user's active time, and since we suspect that this complexity may not be necessary in practice, we considered other heuristics for defining the polling interval N . The simplest solution is set it to a fixed time, such as 1, 5, 10 or 20 minutes. Using the data collected in the second pilot, we simulated each of these settings to assess how many questionnaires each of our users would get each day. The simulation for each user utilizes their actual active time, load levels, etc.

Figure 2 plots the daily number of pop-ups for the various fixed-N schemes. We observe that a polling interval of 1 minute yields the highest number of days where we achieve the goal of getting three samples per day. However, a 1 minute interval presents the risk of taking the three samples in a short time duration (a little more than three hours) which means that for people who are very active on their machines, the samples will not be spread throughout the day. We conclude that a polling interval of 5 minutes is a reasonable choice and we incorporate this design decision in the final version of HostView along with the one hour minimum gap between consecutive pop-ups.

DISCUSSION

In this paper, we outlined the tradeoffs involved in collecting end-host measurements annotated with user feedback. We spent a considerable amount of time studying and analyzing the overhead of different candidate techniques for collecting end-host data that is complete and sufficient enough for us to study the root causes of performance degradations. The techniques for data collection that we elect to implement are those that have low or moderate overheads and are associated with high utility. We also discussed a candidate experience sampling algorithm and highlighted the difficulties of sampling users because of the unpredictability of their behavior. We described how we used the feedback of our second pilot to modify some of the parameters of our experience sampling algorithm. Our next step is to run HostView for a longer period of time, with a larger set of users, so that we can research characterizations and solutions to end-host diagnostic problems that combine performance and perception.

Figure 1 :

 1 Figure 1: Distribution of unanswered pop-ups.

Figure 2 :

 2 Figure 2: Average number of pop-ups per day (100 simulations with a maximum standard error of 0.15)

Table 1 :

 1 Application classification coverage

	Output	Method	TCP	UDP	Total traffic
			Flows Bytes Flows Bytes Flows Bytes
		l7	99.20% 99.90% 47.72% 13.66% 60.08% 90.04%
	Protocol	port	99.60% 99.85% 87.82% 69.90% 90.65% 96.39%
		l7 =port 94.77% 95.84% 46.76% 13.51% 58.29% 86.43%
				93.69% 98.41% 94.89% 95.82%
	App Name	gt	99.56% 99.89% 10.30% 1.36% 31.73% 88.63% 79.13% 96.13% 83.87% 99.64%

Table 2 :

 2 User sample types

	Algorithm 1 Experience sampling algorithm
	1: Every N seconds :
	2: if (machine not idle) and (Num samples <= 3) : 3: if (throu < T l ) and (Rand < γ l ) take sample 4: 5: else if (T l <= throu < Tm) and (Rand < γm) take sample 6: 7: else if (Tm <= throu) and (Rand < γ h ) 8: take sample

http://cmon.lip6.fr/EMD/

ACKNOWLEDGEMENTS

The authors would like to thank all participants of the pilot studies. This work was supported by the European Community's Seventh Framework Programme (FP7/2007-2013) no. 223850 and the ANR project C'Mon.


Diana Joumblatt

Renata Teixeira
email: renata.teixeira@lip6.frjaideep.chandrashekar

Jaideep Chandrashekar

Nina Taft
email: nina.taft@technicolor.com

Performance of Networked Applications: The Challenges in Capturing the User's Perception

Keywords: C.2.3 [Network Operations]: Network monitoring Human Factors, Measurement, Performance End-host measurement, Network performance diagnosis, User experience

There is much interest recently in doing automated performance diagnosis on user laptops or desktops. One interesting aspect of performance diagnosis that has received little attention is the user perspective on performance. To conduct research on both end-host performance diagnosis and user perception of network and application performance, we designed an end-host data collection tool, called HostView. HostView not only collects network, application and machine level data, but also gathers feedback directly from users. User feedback is obtained via two mechanisms, a systemtriggered questionnaire and a user-triggered feedback form, that for example asks users to rate the performance of their network and applications. In this paper, we describe our experience with the first deployment of HostView. Using data from 40 users, we illustrate the diversity of our users, articulate the challenges in this line of research, and report on initial findings in correlating user data to system-level data.

INTRODUCTION

Slow web downloads, choppy Skype calls and YouTube videos, or broken access to your email server are just few examples of how Internet disruptions can frustrate the user experience. A number of research projects aim to help users with tools that automatically diagnose performance problems. Some of this research has focused on enterprise networks [START_REF] Bahl | Towards Highly Reliable Enterprise Network Services via Inference of Multi-level Dependencies[END_REF][START_REF] Kandula | Detailed diagnosis in enterprise networks[END_REF] and ISP networks supporting popular applications such as VoIP [START_REF] Singh | DYSWIS: An architecture for automated diagnosis of networks[END_REF] or IPTV [START_REF] Mahimkar | Towards Automated Performance Diagnosis in a Large IPTV Network[END_REF]. Other efforts diagnose performance and configuration problems from an end-host upon the user request [START_REF] Kreibich | Netalyzr: illuminating the edge network[END_REF].

The ultimate goal of such tools is to improve the user experience, yet such efforts rarely engage the user in the process in order to understand their perception of how the network is performing. Studies of user quality of experience have been limited to a few niche applications, for instance VoIP [START_REF] Jiang | Modeling of packet loss and delay and their effect on real-time multimedia service quality[END_REF], online-gaming [START_REF] Chen | How sensitive are online gamers to network quality?[END_REF], and video playback [START_REF] Chen | Oneclick: A framework for measuring network quality of experience[END_REF][START_REF] Tao | Real-time monitoring of video quality in IP networks[END_REF]. We believe it is important to understand the user's perspective in assessing performance degradation. Clearly, a user's perspective of what constitutes a problem is a subjective matter. Such a perspective could potentially be used to decide whether or not to launch a diagnosis process. For example, if a user's network experiences a problem, but the application masks it and the result doesn't affect the user, it may not necessary to launch a diagnosis process.

The critical first step towards automated performance diagnosis and repair of end-hosts is to gather data directly from laptops or desktops and annotate that data with the user's perspective. Although some end-host datasets exist [START_REF] Cooke | Reclaiming Network-wide Visibility Using Ubiquitous Endsystem Monitors[END_REF][START_REF] Guha | How Healthy are TodayŠs Enterprise Networks?[END_REF][START_REF] Simpson | NETI@home: A distributed approach to collecting end-to-end network performance measurements[END_REF], none of them is annotated with the user perceived quality of the performance of the network and applications. The only exception is HomeMaestro [START_REF] Karagiannis | Homemaestro: Distributed monitoring and diagnosis of performance anomalies in home networks[END_REF], which instrumented the gateway in a small set of homes to capture all packets and asked home users to keep logs of how they perceive the network performance. Because it collects traces for a home and not per end-host and the logs are not automated, they cannot have fine-grained correlation of each user's perception of network performance.

Towards this goal we designed and built HostView1 [START_REF] Joumblatt | HostView: Annotating End-Host Performance Measurements with User Feedback[END_REF], an end-host tracing tool that runs on Mac OS and Linux PCs. HostView collects network traffic, system performance information, and the application associated with network traffic. Importantly, it also prompts the user for feedback on network performance. HostView incorporates two mechanisms for getting user feedback: a system-triggered questionnaire based upon the Experience Sampling Methodology [START_REF] Consolvo | Using the Experience Sampling Method to Evaluate Ubicomp Applications[END_REF], and a user-triggered mechanism called an "I'm annoyed!" button. The ESM mechanism prompts the user no more than three times per day to respond to a questionnaire about their network experience in the five minutes preceding the questionnaire. The "I'm annoyed!" button is always displayed at the edge of the screen and we ask users to click on this button when they are not happy with their network performance. Clicking the button brings up the same feedback questionnaire as in the ESM prompts. The particular questions in the questionnaire can be seen in [START_REF] Joumblatt | HostView User Manual[END_REF]. For more details on the design of HostView, please refer to [START_REF] Joumblatt | HostView: Annotating End-Host Performance Measurements with User Feedback[END_REF].

In this paper, we report on preliminary findings from our first deployment with HostView. First, we find that there is a great deal of diversity in the installed user base -in network speeds, application usage, and the level of network use ( § 2). While expected, the extent of this diversity took us by surprise and necessitated several modifications to Hostview after it was released (to support particular users). This experience leads us to believe that monitoring tools must be designed so as to adapt to particular individuals. Second, collecting feedback from users, particularly about poor network performance, is fundamentally challenging because it is hard to collect sufficient statistics about rare events. By allowing two different mechanisms to register their feedback, our tool is able to collect a reasonable number of reports ( § 3). We were able to collect 576 total reports from users in a 2 week period, 95 of which were deemed instances of poor network performance. Finally, we find that a small set (seven, to be exact) of relatively simple performance metrics are able to uncover problems in a vast majority of cases where user's report performance problems. In 82 out of the 95 poor performance instance, at least one of the metrics also exhibits "anomalous" behavior ( § 4).

USER CHARACTERISTICS

Our first deployment of HostView was between November 2010 and February 2011. The data in this paper comes from 40 users (26 on Mac OS and 14 Linux users). Although many users ran HostView for over a month, this paper analyzes data from the first two weeks of each of these users for the purposes of consistency. We gathered these users both via publicity and via incentives. We publicized HostView by distributing leaflets during the ACM Internet Measurement Conference 2010 and by sending emails to a number of CS mailing lists in November 2010. We offered two types of incentives for users to install HostView. First, HostView reports "network usage" summaries, where users can see their throughput as well the fractional bandwidth of active applications. Second, we offered 50 USD Amazon gift cards to a set of 40 users randomly picked from the first 100 who run HostView for an entire month.

Given that we mainly recruited users from the networking community and that HostView runs only for Mac OS and Linux, we expect that most of our users are computer experts. Despite this bias and the small number of users, our users exhibit a great deal of diversity in many ways. First, our users come from 14 different countries in Europe, North and South America, Asia-Pacific, and Middle-East. Second, the total number of different applications employed by these users over two weeks was 385. Table 1 presents the top-ten applications in our traces in terms of the number of users and the number of bytes transferred. Application names are identified in HostView from the system process names using the gt toolkit [START_REF] Gringoli | Gt: picking up the truth from the ground for internet traffic[END_REF], so traffic to web-based applications such as Facebook or YouTube will appear under the browser name. We see that a number of our popular applications, such as web browsers, e-mail, and Skype, are similar to popularity rankings by the broader community captured in the the wakoopa ranking2 . It is interesting to contrast the most popular applications with the applications that transfer the most bytes. The web browsers and Skype appear in both lists. Given that Bittorent transfers files, videos and other (often large) content, it isn't surprising that it tops the list in terms of number of bytes. The fact that SSH appears in the top-ten reflects the working habits of computer scientists. Third, we observe behavioral diversity, even with a bias towards computer scientists. For example, network usage varies considerably across the set of studied users. We compute data rates per user as the number of bits downloaded plus the bits uploaded every second. A user's data rate reflects both the capacity of her access network and the amount of data exchanged by applications. If a user is connected to a 100 Mbits/s Ethernet but only reading text email, the data rate will be low. Inversely, if a user is downloading movies over a 512 Kbits/s DSL line, the download can only go as fast as the DSL link. Fig. 1 presents the cumulative distribution of the median and 95 th percentile of data rates per user (computed over the two week period) in log scale. This figure shows that both statistics vary by orders of magnitude across users. There is also a significant difference between the median and the 95 th percentile data rates, which indicates that individual users generate very different data rates over time.

This difference in network usage also reflects on the volume of trace data collected for each user. Fig. 2 shows the cumulative distribution of the HostView trace size (in bytes) collected during the first two weeks for each of the 40 users (x-axis is in log scale). This figure shows that trace sizes vary from tens of megabytes to over ten gigabytes. The trace size depends not only on the data rates, but also on other factors such as the amount of time a user spends online, the applications used, and a person's use of HostView's pause feature. The offered pause function allows users to temporarily halt data collection. Our analysis of HostView logs shows that 26 users never paused the data collection, other users paused it only few times (at most 4 times during the two week period), except for one user who paused it 13 times. The users with the smallest traces never used the pause button; they simply spent little time online. In summary, we observe user diversity over orders of magnitude in terms of data rates and volume of network data generated; and we observe some outlier user behavior in terms of their use of the pause feature.

CHALLENGE: USER DIVERSITY

Although the diversity in user behavior is attractive for gathering a broad set of data, and for subsequent analysis, this same diversity leads to challenges in this line of research, as discussed in this section.

Monitoring tool design

There are often many ways to carry out the software design of a particular component in the monitoring tool itself. The "best" design may in fact depend upon the configuration and usage patterns of a user's computer. We illustrate this point with one example and discuss the implications for monitoring tool design.

A core component of an end-host monitoring tool is the ability to upload the collected data from an individual machine to a back-end server for later processing. This should be done in a way that does not impact the PC performance nor annoy the user. One approach is to upload the most recent trace all at once, per each data transfer attempt (we try every 4 hours). This way the task completes quickly and removes potentially large trace files off a user's machine in one step. We tried this approach initially and found that although it worked for the majority of users, there were a few who reported problems because uploading traces was consuming too much bandwidth and impacting machine performance. We learned that these users in fact suffered (with some regularity) from very low bandwidth rates, illustrating that one cannot make assumptions even about minimal upload quality.

We redesigned this mechanism following a second approach that also attempts to upload every four hours, but that introduces two features to avoid consuming too much bandwidth. First, it caps the upload bandwidth at 100 Kbits/s. Second, it sends traces split in 2 Mbytes chunks. After sending each chunk, if the chunk is larger than 150 Kbytes and the achieved throughput is less than 10 Kbits/s, then HostView aborts uploading the trace and tries again after four hours. This check ensures that if a user is at a place with bad network connectivity, HostView will not exacerbate the problem. Four of our users are running the new upload algorithm and no uploading problems have since been reported.

We conclude that the diversity in trace sizes and network connectivity of an individual's machine implies that a "one-size-fits-all" approach isn't necessarily the best design paradigm for end-host tracing tools. Moreover it seems a shame to have to design a tool for the worst case scenario, i.e. the few users with problems. We have come to believe that developing monitoring software whose configurations are adaptive is the right design paradigm for the future development of end-host tracing tools. For example, a tool could be adaptive in that it first observes the user's behavior or the machine performance and then selects among a few options for uploading the trace data.

Sampling users for feedback

The study of network disruptions as perceived by end-users presents fundamental challenges in getting sufficient data for the task at hand. The challenge is in figuring out at what time and under what conditions to sample a user. By "sample a user" we mean the act of popping up a questionnaire and getting the user to complete it. We identify five issues in gathering both machine and user data during the very moment of a performance problem. First, poor performance events are by definition supposed to not be the norm, and thus depending upon the user's machine and network, such events could be far and few between. Second, in order to avoid user annoyance, the questionnaire should pop up a limited number of times per day. We selected to use 3 based upon the results of our 400-person survey [START_REF] Joumblatt | Perspectives on Tracing End-Hosts: A Survey Summary[END_REF]. Third, the questionnaires need to be spaced somewhat far apart -again to minimize user annoyance. This is complicated by the fact that predicting user uptime is hard [START_REF] Joumblatt | HostView: Annotating End-Host Performance Measurements with User Feedback[END_REF] which makes the task of deciding when to next pop up a questionnaire difficult. On days when the user doesn't use her PC much, fewer than 3 samples could be collected. Fourth, promising the user that they will be asked to fill out the questionnaire less than a small finite number (e.g. 3 times a day), means that once they are all filled out, another questionnaire cannot be issued even if a poor performance epoch occurs. Fifth, there is a large diversity of user willingness [START_REF] Joumblatt | Perspectives on Tracing End-Hosts: A Survey Summary[END_REF] and we suspect that there will also be a wide range in the consistency within users fill-in the questionnaires.

All of this means that popping up questionnaires randomly, when limited to 3 a day and multiple hours in between questionnaires, isn't going to generate many surveys at poor performance epochs. To improve the chances we designed HostView (described in [START_REF] Joumblatt | HostView: Annotating End-Host Performance Measurements with User Feedback[END_REF]) to pop up the questionnaires using a scheme that weighs bad performance moments more heavily when deciding when to pop-up the questionnaire. Remember, that our "I'm annoyed!" button also increases the chances of getting feedback since the user can elect to engage it at any time.

How did our users respond to the two mechanisms for gathering data? Table 2 presents a summary of the responses we gathered. The numbers in the first 3 columns, under the ESM heading, refer to system-triggered questionnaires. We received 541 completed surveys that came from 35 users, indicating that most of our users (88%) used the ESM mechanism regularly. This also implies that 5 of our users never responded to the system-triggered questionnaire. Unfortunately we found that 448 questionnaires were never answered; this count refers to questionnaires that popped up and either were closed by the user or timed out after 6 minutes because the user ignored it. The 311 Deferred questionnaires refers to the number of times a user hit the "Ask Me Later" button when the survey popped up. Of the deferred questionnaires, we found that roughly 23% were completed the following time the questionnaire popped up. We found that 80% of the deferred questionnaires were answered within a 12 hour period. This illustrates that the deferral option is a good feature to have in ESM system. Overall we found that the majority of our users did use the ESM mechanism and complete the questionnaires, however this same set of users also left plenty of questionnaires empty and made regular use of the deferral option. We obtained 28 reports from 8 users that were generated from the "I'm annoyed!" button. Only 11 of our users used both the ESM and "I'm annoyed!" button mechanisms. This indicates that many users prefer one mechanism over the other and thus having both definitely improves the chances of obtaining the target user feedback data.

In the following, we focus on the poor performance reports out of the 569 completed questionnaires (541 from ESM and 28 from the "I'm annoyed!" button) . We say a completed questionnaire constitutes a poor performance report if the user rated their connection quality as either a 1 or a 2, out of a 5-point scale (with 5 being the fastest) or if the user explicitly named an application they believe to be performing poorly or provided a description of the problem. Among the 541 ESM questionnaires, 67 of them were poor performance reports, and these were generated by 23 users. So in total, over our 2 week period, we received 95 poor performance reports from 24 users.

All of this points to the fact that in order to get sufficient data, a large number of users are needed, or the measurement experiment needs to be run for a very long time. Simply scaling up the number of users is at odds with well known practices in the HCI community. There, most surveys are done with a small set of users (say 20) [START_REF] Consolvo | The Wi-Fi Privacy Ticker: Improving Awareness and Control of Personal Information Exposure on Wi-Fi[END_REF] and the users agree to respond to the questionnaires. This well accepted methodology has the advantage that the users participating will use the tool properly, thus enabling a qualitative study. However the disadvantage of this approach is that the number of users that participate is very limited (i.e. rarely in the hundreds). In the Internet measurement community, we are used to data gathering activities that assembles thousands or millions of data points from an ongoing stream of events. Further exploration is needed in approaches that target a large user base, even if they are only partially trained or aren't guaranteed to be diligent in responding to feedback questionnaires. (Although we only present data from 40 users here, our next deployment will target a much larger user base.)

USER REPORTS ON PERFORMANCE

We now compare user reports of poor performance with system and network performance metrics that are easily measured automatically. We start with a simple analysis in which we consider seven basic metrics, and seek to understand if the poor performance epochs (as flagged by the user) co-occur with any of our automated metrics experiencing atypical, or "anomalous" behavior.

The specific performance indicators, seven in all, that we extracted from the traces are as follows: Round-trip times (RTTs), computed as the elapsed time between sending a TCP packet and receiving the corresponding acknowledgment; Data rate, the aggregate bandwidth (in both directions) measured in bits/second; TCP retransmissions, which counts the number of segments retransmitted (per second); TCP resets, measured as the number of packets containing a RST flag (TCP reset); CPU load, the CPU utilization measured at 30 second intervals. Finally, when the end-host had an active and operational wireless interface, we also logged wireless signal strength and wireless noise, as reported by the airport or iwconfig utilities (on Mac OS and Linux, respectively).

For each of these seven performance indicators, we define an anomaly as follows: we build a distribution of the values using two weeks of data and identify the 5 th and 95 th percentile values of the distribution. We consider an instance of a metric that falls inside this range (5 th to 95 th percentile values) to be normal. We consider the following cases to be anomalous: i) if the RTT, CPU load, TCP retransmissions or wireless noise exceed the 95 th percentile value; ii) if the data rate or wireless signal strength fall below the 5 th percentile value; or iii) any instance of a any TCP reset.

We now look at some detailed examples that illustrate the sequence of events in time that occur just before and after a user completes a poor performance report. Figures 3(a), 3(b) and 4 all follow the same format; the time the user submitted the report is marked 0 on the x-axis (indicated by the vertical line) and we assume the poor performance epoch occurred close to this time. The brief time series on the x-axis indicates what occurred for 2 minutes before and after the report was filed. Because the performance metrics all have different ranges, we rescaled them so as to be able to view them (y-axis) on the same plot. The normal range is mapped to values between [0, 1] with the 5 th (95 th ) percentile value mapped to 0 [START_REF] Bahl | Towards Highly Reliable Enterprise Network Services via Inference of Multi-level Dependencies[END_REF] TCP RSTs are treated differently, and we simply mark the time that the RST was observed.

In Figure 3(a) the user reported a bad performance epoch by using the "I'm annoyed!" mechanism and provided the additional context: "Speed rate: Slow"; "Application: iTerm"; "Problems: Can't connect to some sites or services". Here we see that the RTTs appear to rise and fall, and occasionally cross the y = 1 line (i.e., exceed the 95 th percentile of the distribution), and we posit that the performance problem experienced by the user was due to the very large round trip times observed. This variation (pattern) in RTTs is typical behavior when the user is behind a large buffer. The stream of packets exchanged with the end-host gradually fills up the buffer introducing additional latency. When the buffer fills up, it starts dropping packets causing TCP connections to reduce their rate; this buffer bloat issue is becoming a major problem for ISPs (see http://www.bufferbloat.net/). To keep the plot readable, we only indicate the performance metrics relevant to this instance being anomalous.

Figure 3(b) is another instance where the user complained about bad performance. Here, the metrics plotted are data rate, wireless signal strength, and wireless noise. We see that the wireless signal strength is consistently below the 5 th percentile value (below the y = 0 line) around the time the user complained. In this case, the user annoyance with their performance may have been due to poor wireless quality that adversely affected the available bandwidth.

In the above two cases we found anomalous events co-occurring with the user's poor performance report and thus could postulate that when network performance was at the fringe of its normal behavior, this particular user unhappy with it. Two other scenarios are possible in terms of the consistency between system metrics and user reporting. If the performance metrics are anomalous, but the user files a report indicating there is no problem, then this is an indication that the user can tolerate such performance levels. On the other hand, if a user files a report indicating poor performance and none of our metrics exhibit anomalous behavior, then it means that more metrics are needed to understand what annoyed the user. (Clearly, our seven metrics considered here are merely a starting point, and we plan to examine others, especially application level metrics, shortly.) Figure 4 shows an example of a completed ESM performance report. The user said there was no problem when completing the report, yet it is very clear from the figure that the RTT values consistently exceed the 95 th percentile values, and this is also reflected in the low data rates. We looked into this case in more detail and found that at the time the report was completed, the user had 3 processes running: a large file transfer to a foreign country, mail and HostView. HostView was engaged in a data upload to a central server but this activity was capped at 100 Kbps and the RTTs to the server were between 0.7 and 0.8 in Figure 4; the mail activity generated minimal traffic, however the RTTs for the file transfer traffic were above 95 th percentile. In this mix of applications, it is possible that the user was happily reading emails and not bothered if the file transfer occurred slowly.

We performed this type of analysis over all of the 95 poor performance reports we received. Overall, we found that in 82 out of these 95 instances, at least one of our seven performance metrics exhibited anomalous behavior. The anomalies that we could associate with each of these 82 poor performance reports are summarized in Table 3. The first column indicates the type of problem the user selected (among a list of choices that included 'Other' and 'None'). The number next to the problem indicates the number of reports received of that problem type; for example, we received 27 reports about slow browsers. The count in the data rate column indicates that in 14 of those 27 reports (row 1), the data rate metric crossed its threshold line for our simple definition of anomalous. Note that the values in a single row can exceed the total number of reports because sometimes in a single report, multiple metrics can be anomalous. In order to explain the 13 reports for which we found no anomalous event across our performance indicators, we need to examine additional performance metrics.

We see in the table that all of our metrics, except wireless noise, were implicated in some problem. We are not claiming that the poor performance epoch a user experienced was caused by the anomaly we observe; in fact our metrics often themselves indicate some other underlying performance problem. We are merely claiming that there can often be simple indicators that track user annoyance, and the fact that our seven metrics are helpful in explaining 82 of the 95 poor performance reports we received is promising; one may not need to look at hundreds of features to understand the majority of incidents. Pinpointing the exact root causes for poor network performance problems is a challenging task and much more work needs to be done. Our work here is an initial foray toward the eventual goal of detecting performance degradations on the fly and performing proactive mitigation where possible.

CONCLUSION

We learned some interesting lessons in our first deployment of HostView. First, we believe that designing for diversity is an important paradigm in the design of end-host tracing tools. Designing for diversity implies that both the monitoring tool and the user-feedback mechanism should be adaptive. Second, due to the multiple challenges in getting user feedback data at poor performance epochs, we believe that the sampling mechanism should be weighted to prompt users when anomalous events occur. Third, we found that despite all of the challenges in collecting feedback for performance diagnosis, the two sampling methods we used together generated a fair amount of feedback. Fourth, we found that most users reporting poor performance episodes did so consistently and that reports often correlate with some performance metrics exhibiting outlier behaviors. This is promising for research in the area of automated diagnosis at end-hosts.

Figure 1 :

 1 Figure 1: Median and 95 th percentile of data rates.

Figure 2 :

 2 Figure 2: Trace size for the first two weeks of collection.

Figure 3 :

 3 'I am annoyed : Speed rate: Slow; Application: iTerm; Problems: "Can't connect to some sites or services" I am annoyed: Speed rate: Super slow; Problems: "Slow download or upload" Correlation of poor performance reports with system and network performance indicators

Figure 4 :

 4 Figure 4: ESM: Everything Good

Table 1 :

 1 Top-ten applications.

	Ranking by users	Ranking by bytes
	1. Firefox	1. uTorrent
	2. Skype	2. Safari
	3. Safari	3. SSH
	4. Mail	4. Firefox
	5. Google Chrome 5. Google Chrome
	6. Dropbox	9. smbd
	7. iTunes	6. Skype
	8. Thunderbird	7. Mail
	9. Adium	8. VLC media player
	10. VirtualBox	10. Chromium

Table 2 :

 2 User feedback (1335 questionnaires)

		ESM		I'm annoyed!
	Completed Deferred	Not Answered
	541	311	448	35

  , respectively. Values in the range [min, 5 th percentile] (and that in [95 th percentile, max])

	Problem (number of reports)	Data rate	Wireless signal strength	Wireless noise	CPU load	RTT	TCP reset	TCP retransmissions
	Slow browser (27)	14	2	0	10	18	23	14
	Slow download or upload (24)	12	2	1	7	17	22	14
	Poor voice or audio quality (14)	7	0	0	2	8	9	2
	Poor video quality (11)	5	2	0	2	7	9	6
	Can't connect to some sites or services (10)	3	1	1	3	8	8	6
	Total (95 reports)	38	8	2	24	56	74	38

Table 3 :

 3 Breakdown of anomalies across bad performance problems picked by users are mapped to [-0.1, 0] (and [1, 1.1] respectively). Thus anomalies are those falling below 0 or above 1 (depending upon the metric).

http://cmon.lip6.fr/EMD

Wakoopa is a social networking site that tracks the applications used by its members: http://social.wakoopa.com/software

Acknowledgements

This project is supported by the European Community's Seventh Framework Programme (FP7/2007-2013) no. 258378 (FIGARO) and the ANR project C'MON.