Indira Paudel

Dr Anderson Morais

Dr Badii Jouaber

Dr Felipe Lalanne

Dr Wissam Mallouli

Jacob Søgaard

Kun Wang

Samira Tavakoli

Natalia Kushik

Uddhav Pokhrel

Sabitra Pokhrel

Sita Dahal

Sunita Dhakal

Jeevan Pokhrel

Keywords: Quality of Experience, Quality of Service, Video, Over the top, wireless network, video on demand, web services v

Today user experience is becoming a reliable indicator for service providers and telecommunication operators to convey overall end to end system functioning (client, terminal, network, services infrastructure, media encoding, etc.). Moreover, to compete for a prominent market share, different network operators and service providers should retain and increase the customers' subscription. To fulfil these requirements they require an efficient Quality of Experience (QoE) monitoring and estimation. QoE is a subjective metric which deals with user perception and can vary due to the user expectation and context. Moreover, subjective QoE evaluation is expensive and time consuming since it requires human participation. Therefore, there is a need for a tool that can objectively measure the QoE with reasonable accuracy in realtime.

In the context of service and network providers, to fulfill user requirements, they need to provide sufficient QoS to relevant services/applications. However, QoS parameters do not necessarily reflect the user's satisfaction and feelings towards a particular service/application. Therefore it is necessary to compute the mapping or correlation between QoS and QoE. This mapping helps them understand the behavior of the overall network on user experience (QoE) and efficiently manage the network resources.

In this thesis work, we utilize this correlation between QoS and QoE to objectively estimate the QoE. For this, we use subjective methodology to create a dataset which represents the correlation between objective QoS parameters and subjective QoE. Following this, different intelligent machine learning algorithms are trained with this subjective dataset to objectively predict the QoE. The machine learning algorithms we used in our work are fuzzy expert system, fuzzy rough hybrid expert system and random neural network.

As a first contribution, we analyzed the impact of network conditions on Video on Demand (VoD) services. We also proposed an objective QoE estimation tool that uses fuzzy expert system to estimate QoE from network layer QoS parameters. As a second contribution, we analyzed the impact of MAC layer QoS parameters on VoD services over IEEE 802.11n wireless networks. We also proposed an objective QoE estimation tool that uses random neural network to estimate QoE from the MAC layer perspective. As our third contribution, we analyzed the iv effect of different adaption scenarios on QoE of adaptive bit rate streaming. We also developed a web based subjective test platform that can be easily integrated in a crowdsourcing platform for performing subjective tests. As our fourth contribution, we analyzed the impact of different web QoS parameters on web service QoE. We also proposed a novel machine learning algorithm (i.e., fuzzy rough hybrid expert system) for objectively estimating web service QoE.

List of figures

List of tables

According to the ITU-T Focus Group on IPTV [START_REF] Itu-T | Quality of experience requirements for IPTV services[END_REF], Quality of Experience (QoE) refers to "the overall acceptability of an application or service, as perceived subjectively by the end-user".

Similarly, Qualinet [2] defines QoE as "the degree of delight or annoyance of the user of an application or service. It results from the fulfilment of his or her expectations with respect to the utility and/or enjoyment of the application or service in the light of the user's personality and current state". Therefore, QoE is a subjective measure and can vary according to the user expectation and context. Moreover, it is an overall end to end system effect (client, terminal, network, services infrastructure, media encoding, etc.) and depends on a number of factors that cannot simply be measured. It requires tests with actual users in a controlled environment to properly estimate the QoE; which is costly and time consuming.

In the Internet, multimedia services and applications have become an important source of income for the network operators and service providers. To compete for a prominent market share, different network operators and service providers should retain and increase the customers' subscription. For this, they should fulfil user's multimedia QoE requirements. To fulfil these requirements they require an efficient QoE monitoring and estimation tool.

However, QoE is a subjective metric and can vary due to the user expectation and context.

Moreover, subjective QoE evaluation is expensive and time consuming since it requires human participation. Therefore, there is a need of a tool that can objectively measure the QoE with reasonable accuracy.

Traditional approaches for measuring the user satisfaction rely on objective QoS parameters collected from the network. In this case, the QoS parameters are monitored and controlled in order to provide a satisfactory level of service quality. Different QoS parameters like bandwidth, delay, packet loss, etc. are essential metrics for determining the service quality from a technical point of view. However, QoS parameters do not necessarily reflect the users' satisfaction and feelings towards a particular service.

In this thesis work, to overcome these shortcomings, we proposed different hybrid approaches that combine both subjective and objective metrics to objectively predict QoE. These approaches utilize subjective methodology to create a dataset which represents the correlation between objective QoS and subjective QoE. Following this, different intelligent machine learning algorithms are trained with this subjective dataset to objectively predict the QoE. We applied our approaches to estimate the QoE of multimedia and web services. Moreover, we also created a novel web based video subjective test platform that can be easily integrated into a crowdsourcing platform.

Motivation

The motivation for this thesis work revolves around searching answers for two basic questions.

Why multimedia services?

The popularity of multimedia services over the Internet is immense. It is supported by the exponential growth in consumption and adoption of Internet multimedia services. Figure 1, Figure 2 and Figure 3, show the forecast of video consumption and adoption from 2013 to 2018 [3] [4]. This growth has created a threat on network resources scarcity as well as opportunities for network operators and service providers for revenue generation. As a consequence, in the recent years, multimedia traffic has diverted the attention of the research as well as industrial communities.

Mobile video: On-demand video content downloaded or streamed to the mobile handset.

Why multimedia QoE monitoring and estimation?

Multimedia QoE monitoring and estimation is a multi-disciplinary approach which involves user psychology, engineering science, economics, etc. The QoE depends on different elements (i.e., content, network, application, business model, etc.) that directly or indirectly affect the user's perception towards the multimedia service. Moreover, the diversity in these elements makes the QoE estimation rather complex and unpredictable. This motivates us to address some prominent challenges related to QoE monitoring and estimation as described below.

QoE is a subjective metric: QoE is a subjective metric as it reflects the perception of the users to a particular service. User attitude and expectation towards multimedia services play a vital role in determining the QoE. Moreover, the QoE can depend on different user profiles like age, sex, interest, skills, frame of mind, experience, etc. Also, different environmental conditions can impact how users perceive the multimedia service content. Multimedia QoE can vary according to when, where, and with whom the service is used.

Subjective vs. objective evaluation:

There exist two methodologies for evaluating multimedia QoE i.e., subjective and objective. The subjective evaluation requires human participation and controlled environment to perform the test. However, this methodology cannot be applied in real-time and requires more time and cost (due to human participation). On the other hand, the objective evaluation of multimedia QoE is based on objectively measured network/media parameters. However, it may include complex mathematics and algorithms and may require objective metrics that can be hard to extract and also may not highly correlate with subjective QoE (no human participation).

Network QoS impact on multimedia QoE: The ability to identify the perceived degree of multimedia impairment due to network perturbation is a key point in the multimedia QoE estimation. Moreover, the effect of network perturbations on multimedia traffic can range from distortion-less to intolerable distortion. Also the selection of network related key performance indicators (KPIs) or QoS parameters that can impact multimedia QoE is difficult.

Need for objective tool:

There is a pressing need for objective QoE estimation tools capable of processing high speed network links in real time to extract QoS parameters and correlate them with user perceived QoE. However, the relationship between QoS and QoE is fuzzy and nonlinear. To address this issue there is a large number of intelligent algorithms proposed in the literature however, there is still room for innovative mechanisms to efficiently correlate QoE from QoS in real time. two novel intelligent algorithms that are fuzzy rough hybrid system [5] and logic networks [6].

Contribution of the thesis

 Objective QoE estimation of VOD services based on network QoS metrics

A methodology and a system based on fuzzy expert system to estimate the impact of network conditions (QoS) on the QoE of video traffic (VoD/IPTV) [7] is presented here. At first, we conducted subjective tests to correlate network QoS metrics with participants' perceived QoE of video traffic. Second, we proposed a No Reference method based on fuzzy expert system to estimate the network impact on the video QoE. The membership functions of the proposed fuzzy system are derived from normalized probability distributions correlating the QoS metrics with QoE. We proposed a simple methodology to build the fuzzy inference rules. We evaluated our system in three different sets of experiments. The estimated video quality showed high correlation with the subjective QoE obtained from the participants in a controlled test. We integrated our system as part of a monitoring tool in an industrial IPTV test bed and compared its output with standard Video Quality Monitoring (VQM).

Additionally, we performed benchmarking analysis where we compared the result obtained from our system with Indra's probe [162] for QoE estimation in the real IPTV test bed. The evaluation results show that the proposed video quality estimation method based on fuzzy expert system can effectively measure the network impact on the QoE.

 Objective QoE estimation of VOD services based on MAC-level QoS metrics

The impact of different

 Subjective QoE estimation of OTT video services based on crowdsourcing

We present insights into a study that investigates perceptual preferences of various adaptive video streaming scenarios through crowdsourcing based subjective quality assessment [START_REF] Shahid | Crowdsourcing based subjective quality assessment of adaptive video streaming[END_REF] here. For this, we created a novel web based video subjective test platform that can be easily integrated into the crowdsourcing platform. With the help of this platform we performed a subjective test of OTT video services to analyze whether users prefer bit rate switching or buffering events in adverse network conditions. Our study suggests that in a network environment with fluctuations in the bandwidth, a medium or low video bitrate which can be kept constant is the best approach. Moreover, if there are only a few drops in bandwidth, one can choose a medium or high bitrate with a single or few buffering events.

 Objective QoE estimation for web service selection

Apart from multimedia services, we also worked on estimating QoE for web service selection.

Here, we present a novel method based on a fuzzy-rough hybrid expert system for estimating QoE of web services for web service selection [5]. We also analyzed how different QoS parameters impact the QoE of web services. For this, we conducted subjective tests in a controlled environment with real users to correlate QoS parameters to subjective QoE. Based on this subjective test, we derive membership functions and inference rules for the fuzzy system. Membership functions are derived using a probabilistic approach [11] [12] and inference rules are generated using Rough Set Theory (RST) [13]. We evaluated our system in a simulated environment in MATLAB. The simulation results show that the estimated web quality from our system has a high correlation with the subjective QoE obtained from the participants in controlled tests.

Overview of the work

The remainder of the thesis is divided into different chapters. In chapter After that intelligent machine learning algorithms learns the relationship between QoS parameters and QoE, this learned intelligent system can be used as an objectively tool for QoE estimation.

Components of the QoE evaluation framework

In the following section, we describe each components of the QoE evaluation framework and highlight our contributions.

Applications/services

Applications/services help users to perform certain tasks. We can find abundant of Internet applications that helps us to perform different activities in our daily life such as multimedia, web, email, games, etc. The popularity of these applications is growing rapidly and, moreover, we can find a number of applications having similar features and performing similar tasks. This offers a number of options to the customers and introduces a higher demand for quality. Applications and services that fulfill the customers experience requirements are the ones which are favorite to the customers. Therefore, estimating the QoE is a must for application/service providers. In this thesis work, we are focused on estimating QoE of multimedia and web services.

Multimedia applications/services: Applications that communicate any combination of audio, image, video, or data traffic are known as multimedia applications. In this modern era, multimedia applications are the medium of information, communication and entertainment for the customers. Some of the most used multimedia applications are audio/video conferencing, television, video-on-demand, telephony, etc. Moreover, multimedia traffic has become a principal source of traffic in today's Internet. However, these multimedia applications are delay sensitive, bandwidth intense, and loss tolerant [15], which makes them very sensitive to varying network conditions. Hence, it is of utmost importance for network and service providers to monitor and estimate user experience to retain the higher service quality.

Web applications/services: Web Services (WSs) are self-contained software systems that can be published, advertised, located, and invoked through the web, usually relying in standardized XML technologies (REST, SOAP, WSDL, and UDDI [15]) for description and publication, and on Internet Protocols for invocation [START_REF] Lalanne | Quality of experience as aselection criterion for web services[END_REF]. Web applications/services are very common in the Internet. Some of the popular web services are weather forecast, currency conversion, search engine, maps, etc. With the proliferation of web services on the Internet, it has become important for service providers to select the best web services for their clients in accordance to their functional and non-functional requirements. Users gets intolerant if the content is not served in expected time and easily switch to other options if their needs are not fulfilled [START_REF] Lohr | For impatient web users, an eye blink is just too long to wait[END_REF]. About 90% of the people do not want to complain for the low service quality.

They just leave the service and move to another ones [START_REF] Nokia | Quality of experience of mobile services Can it be measured and improved?[END_REF]. So service providers and operators should not wait for user feedback for improving the service quality, instead they should continuously monitor QoE and improve it as required. They should provide users with services that can offer high QoE values.

Quality of Service (QoS) parameters

The term QoS refers to the probability of the telecommunication network meeting a given traffic contract [START_REF] Mehta | Performance Analysis of QoS Parameters for Wimax Networks[END_REF]. QoS is actually the ability of a network element (e.g., an application, host, or router) to have some level of assurance that its traffic and service requirements would be satisfied [START_REF] Talwalkar | Analysis of Quality of Service (QoS) in WiMAX networks[END_REF]. QualityCrowd [START_REF] Keimel | Qualitycrowd-a framework for crowd-based quality evaluation[END_REF], PC-Video-Test-Interface [START_REF] Shahid | Crowdsourcing based subjective quality assessment of adaptive video streaming[END_REF], etc.

Intelligent machine learning algorithms

Intelligent machine learning algorithms automatically learn from the past observations to make accurate predictions in the future. They are mostly used in classification problems.

Some of the most popular machines learning algorithms are decision tree, neural networks, fuzzy expert system, etc. In our work, the QoE estimation system uses machine learning algorithms to learn the correlation between QoS parameters and subjective QoE. The learning process involves training the algorithm with subjective data set. Once the system has learnt, it can predict the QoE based on any combination of input QoS parameters.

Advantages of using machine learning algorithms include [START_REF] Schapire | Machine Learning algorithms for classification[END_REF]:

 They are much more accurate than the human crafted rules since they are data driven.

 They do not require expert or programmer for making decision.

 They can be performed automatically and in real time.

 They are flexible and cheap.

Disadvantages of using machine learning algorithms include:

 They required lot of labelled data.

 Impossible to get perfect accuracy.

Quality of Experience (QoE)

In this thesis work, the output QoE scores are represented in terms of MOS scores [START_REF] Itu-T | Mean opinion Score (MOS) terminology[END_REF] as presents in Section 2.2.3. Therefore, the output of the proposed QoE estimation framework is a MOS score that reflects the mean opinion score of different customers. This MOS score is as close as possible to those voted by the human users.

Conclusion

In

Chapter 3

State of art

Introduction

The multimedia era started in the middle of the last century when the television appeared and people got used to watch TV shows and movies at home. Moreover, with the VCR, DVD and Blue-ray disks and players, viewers could use the TV set to watch recorded material. In parallel, in the late 90s internet video service started. This was a time when video Internet technologies were merely innovation and novelty. Due to the technological limitations with dial-up connections and slow modems, it was hard to achieve good transfer rate and superior quality. In addition, the limited graphics processing power at that time prevented wide adoption of high quality videos. However, with the proliferation of broadband internet and the tremendous increase in processing power, multimedia technologies boomed [START_REF] Motorola | The Evolution of OTT video[END_REF]. The combination of high speed internet and sophisticated powerful devices introduced a completely new way for multimedia consumption through Internet. It was the boom of "Over

The Top" and "Video on Demand" services.

Moreover, the advancement in different concurrent digital multimedia technologies and the proliferation of smart mobile terminals with their application ecosystem have exponentially increased the popularity of Internet multimedia services, which are a key player in current ICT business. It is expected that video traffic will reach 66% of the global mobile traffic by the year 2015 with one million minutes of video content crossing the Internet every second [START_REF]Cisco. Cisco Visual Networking Index: Forecast and Methodology 2010-2015[END_REF]. At the same time, video consumers are getting more demanding about the quality of multimedia content. Therefore, in order to satisfy users' demands with acceptable viewing quality, it is utmost necessary to monitor their satisfaction.

The traditional approach of measuring the user satisfaction relies on QoS parameters collected from the network. In this case, the QoS parameters are monitored and controlled in order to provide a satisfactory level of service quality. Different QoS parameters like bandwidth, delay, packet loss, etc. are essential metrics for determining the service quality from a technical point of view. However, QoS parameters do not necessarily reflect the user's satisfaction and feelings towards a particular service. In order to accurately address the human perception of the service quality, a new concept of measuring the Quality of experience (QoE) is involved.

The QoE refers to "the degree of delight or annoyance of the user of an application or service.

It results from the fulfilment of his or her expectations with respect to the utility and/or enjoyment of the application or service in the light of the user's personality and current state" [2]. Accordingly, the QoE is a subjective metric and can vary due to the user expectation and context. Moreover, the QoE is a reliable indicator for service providers and telecommunication operators to convey overall end to end system functioning (client, terminal, network, services infrastructure, media encoding, etc.). Furthermore, it is a multidisciplinary approach when determining QoE, which involves user psychology, engineering science, economics, etc. The QoE depends on different elements (i.e., content, network, application, etc.) that directly or indirectly affect the user's perception towards the multimedia service. These elements should perform to their best to provide high user experience.

However, the diversity in these elements makes the QoE estimation rather complex and unpredictable.

It has been shown that multimedia customers are willing to pay for better quality of experience [START_REF] Accenture | Hearts, Minds and Wallets, Accenture Video-Over-Internet[END_REF]. The success of paid VoD services (not to mention that of Netflix) is just a simple proof. However, customers get intolerant if the multimedia service is not satisfactory and they easily shift to other options if their needs are not fulfilled. Therefore, the user satisfaction is utmost important for retaining customers and has become the main differentiator for the success of network operators and service providers. Correspondingly, network operators and service providers should accurately estimate and monitor the multimedia QoE in order to track the performance and quality of a particular service.

In the following sections, we will look at the different components of the multimedia QoE ecosystem and their roles in the multimedia QoE. We will present different multimedia transmission components and highlight some key performance indicators relevant to the multimedia QoE. We will also explain different multimedia QoE estimation methods and techniques along with the QoS/QoE learning algorithms. Finally, we will discuss some of the future challenges and issues related to multimedia QoE.

Multimedia QoE ecosystem

Quality of Experience (QoE) is an important indicator for network operators and service providers to help them assessing the user acceptability towards a particular service or a can affect the overall multimedia QoE. The multimedia service provider should monitor, evaluate and adapt these components according to different service requirements in order to offer a better experience. The success of a multimedia service depends on how well all these components are behaving in the ecosystem. In the following section, the role of different components of the multimedia QoE ecosystem is described.

Content

If the original multimedia quality is unsatisfactory, then the QoE at the destination is also unsatisfactory. In order to have a better multimedia QoE, the multimedia content should be of high quality. Some of the characteristics of multimedia contents and their description are listed in Table 1.

Table 1: Multimedia content characteristics and their description

Content Characteristics

Description

Bit rate

Bit rate in terms of video transmission refers to the minimum rate at which video bits are transferred from a source to a destination. The higher is the multimedia bit rate, the better is the multimedia quality.

Frame rate

Multimedia frame rate refers to a number of multimedia frames presented per second. The higher is the frame rate, the smoother the video appears and hence, the better is the video QoE.

Resolution

Video resolution refers to the number of pixels in both directions (width and height) of a video frame. A higher frame resolution yields to a better video quality.

2D/3D

Video types i.e., 2D/3D, refers to the visual dimensions of a video content. These content types have different service and network requirements.

Therefore, different content characteristics represent the diversity of multimedia content and their properties influence the multimedia QoE. However, content characteristics like higher bit rate, frame rate and resolution increase the complexity of the video encoding and require higher network bandwidth for getting better QoE. Moreover, different multimedia devices and applications should support different content characteristics for a smooth playback.

Network

Network represents the segments from the content servers, the core and distribution segments and the access network with a combination of copper, fiber and wireless links. In order to experience better multimedia QoE, transmission conditions at the network end should be reliable. The network condition is represented by the quality of service (QoS) parameters.

Each multimedia service has its own QoS requirements. Some of the network QoS parameters and their descriptions are presented in Table 2.

Table 2: Network QoS parameters and their description

QoS parameters

Description

Packet loss rate

Packet loss rate is the ratio of the total number of packets lost in transmission compared to the total number of packets sent. The higher is the packet loss rate, the lower is the multimedia QoE.

Burst loss

If a group of consecutive packets are lost then it is defined as a burst packet loss. A higher burst loss results in a lower multimedia QoE.

Jitter

Jitter is the variation in the packet inter-arrival delay. Higher jitter results in a lower multimedia QoE.

Bandwidth

Bandwidth is the amount of information that can flow in a network during a specific period of time. To some extent, the higher is the network bandwidth, the higher is the multimedia QoE.

Packet error rate

Packet error rate is the ratio of the total packets received with errors to the total number of transmitted packets. The higher is the packet error rate, the lower is the multimedia QoE.

The variation in time of the network conditions will directly impact the network QoS parameters and therefore, the multimedia QoE.

Users

Each individual user has his/her own perception towards a multimedia quality. User attitude and expectation towards multimedia services play a vital role in determining the quality of the service. Multimedia QoE can depend on different user profiles like age, sex, interest, skills, frame of mind, experience, etc. For example, if a person is young and sportive, he/she might be inclined to sport content and his/her perception towards the QoE of sport content can be different than for a movie content. Thus, it is necessary to categorize users and to analyze their needs.

Environment

Different environmental conditions can impact how users perceive the multimedia service content. Multimedia QoE can vary according to when, where, and with whom the service is used. For example, multimedia QoE will be different when the multimedia service is used while commuting, in a cafe, at home, or in a bar. These places offer different levels of external noise that can directly or indirectly impact how a user perceives the service.

Moreover, time of day (morning, afternoon, evening, etc.) also has an impact on the user multimedia experience. It was observed that for the same multimedia content, a higher level of user experience was obtained in the afternoon and in the mid-day [START_REF] Msakni | Impact of non-QoS related parameters on video QoE[END_REF]. Also, the results obtained from lab tests (having controlled environment) and crowdsourcing experiment for multimedia QoE are different [START_REF] Shahid | Crowdsourcing based subjective quality assessment of adaptive video streaming[END_REF].

Device

Multimedia content can be consumed through different devices like televisions, personal computers, laptops, tablets, mobile phones, etc. These devices offer different level of screen size resolution as well as different sound quality to the users. Therefore, the user experience can vary when different devices are utilized. For example, the user experience of watching a video content on a personal computer (PC) at a fixed distance is very different from that when watching it on a handheld mobile device where you have a possibility to move [START_REF] Khan | Impact of end devices on subjective video quality assessment for qcif video sequences[END_REF].

Subjective tests for multimedia QoE conducted by [START_REF] Khan | Impact of end devices on subjective video quality assessment for qcif video sequences[END_REF] on PC and mobile handset revealed that, for the same content, almost all users preferred the PC over the mobile handset.

Application

Multimedia content can be consumed as a video-on-demand (VOD) service, over the top (OTT) service, broadcast service, etc. These services provide different multimedia applications for usability and interactivity. Multimedia applications should be self-intuitive and easy to use for getting a higher QoE. In the study performed by Bernhaupt et al. [START_REF] Bernhaupt | Usability and usage of iTV services: lessons learned in an Austrian field trial[END_REF], it was found that the usability rating is higher when the service is easier to use. Moreover, every application has its own video buffering scheme, encoding, and decoding, which can affect the playback of multimedia content and the overall multimedia QoE.

Each component of the ecosystem plays an important role in determining how users perceive the multimedia quality. If any element of the ecosystem is imbalance, the overall multimedia ecosystem is disturbed with lower QoE. This implies that concerned parties involved in the multimedia content distribution should be very careful in monitoring, evaluating and adapting these corresponding components for getting a higher QoE.

Multimedia transmission and Key performance indicators

Nowadays, multimedia traffic is the major source of traffic over the Internet. It is used for different purposes like entertainment, education, advertisement, etc. Internet technology boosted the growth of multimedia by providing a transmission platform from where each individual can stream the content to their devices. However, Internet is a best effort service and is shared by different other data greedy applications which means that multimedia traffic are vulnerable to various network perturbations. To overcome this time varying nature of Internet's network quality, different operators provide dedicated service (in the form of SLA agreement) to the managed multimedia services like IPTV and VOD. However, most of the multimedia services are over the top (OTT) and need to share the same bandwidth with competing applications. In the following section, we study different aspects of multimedia transmission in Internet.

Multimedia transmission in Internet

Figure 6 illustrates different components of multimedia transmission system in the Internet.

The raw analog multimedia contents are huge in size and can consume a large amount of network resources. To compensate this, analog multimedia contents are digitalized and compressed by video and audio compression algorithms in the source encoder, and stored in the multimedia server. Upon the client's request, the multimedia content is retrieved from the server and the channel encoder adapts the multimedia stream to the network QoS requirements by adding redundant information for error recognition and correction. After that, the multimedia stream is transmitted (streamed) to the client terminal over Internet.

Multimedia stream can suffer from different types of perturbation like packet loss and delay due to congestion. At the client terminal, a channel decoder performs the error detection and correction and transmits the digital data to the source decoder. The digital data received from the channel decoder is transformed into continuous waveform in the source decoder, which can be viewed or listened to using different players at the application layer. To compensate the network jitter different multimedia players implement a playout buffer. The playout buffer stores some multimedia content in order to have a smooth playback. and RTP over UDP. In the following, we explain the characteristics of these protocols.

RTP/UDP based streaming

The Real Time Transport Protocol (RTP) [START_REF] Schulzrinne | RFC 3550. RTP: a transport protocol for real-time applications[END_REF] is one of the popular and most used streaming protocols for multimedia transmission. Most of the IPTV and VOD services use RTP/UDP based multimedia steaming. RTP is transmitted over UDP and is basically used for real-time transfers (TCP is unsuitable for real-time transfers due to high delay [START_REF] Avermedia | HTTP/TCP vs. RTP/UDP[END_REF]). RTP works in conjunction with the Real Time Control Protocol (RTCP), which operates at the session layer.

The main function of RTCP is to provide a feedback on the quality of the data distribution [START_REF] Mcgath | Basics of streaming protocol[END_REF].

However, when using UDP, data packets often have difficulties getting through firewalls and network address translators [START_REF] Biernacki | On the influence of network impairments on YouTube video streaming[END_REF]. UDP is an unreliable protocol and the multimedia streams can suffer packet loss which might cause distortion of the multimedia content.

HTTP/TCP based streaming

Nowadays, most of the OTT video streaming applications use HTTP/TCP as a transport mechanism for multimedia streaming. This is because HTTP/TCP can easily pass through firewalls and routers. It also does not require special proxies or caches. In addition, HTTP media streaming is easier and cheaper to deploy because web streaming can use generic HTTP solutions and does not require specialized servers at each network node [START_REF] Biernacki | On the influence of network impairments on YouTube video streaming[END_REF].

However, the use of TCP for streaming has some shortcomings. Due to the TCP congestion avoidance mechanism, it produces transmission rate which is saw-tooth shaped and has high variation. Also, due to TCP reliability, retransmission of lost packets occurs, which introduces additional transmission delays. To cope up with this short term bandwidth variation of TCP, streaming services deploy playout buffer [START_REF] Seyedebrahimi | Model and Performance of a No-Reference Quality Assessment Metric for Video Streaming[END_REF]. The use of TCP guarantees the reception of all packets to ensure no video impairment, however, due to severe network conditions multimedia transmission can freeze for a long time. Also to cope up with different network conditions different adaptive HTTP based video deliver schemes have been developed [START_REF] Networks | Comparing Adaptive HTTP Streaming Technologies[END_REF].

In adaptive HTTP based video delivery, video contents are encoded in different data rates and stored as small fragments (a few seconds each) on the server. It has an ability to switch between different data rates based upon network conditions and other variables [START_REF] Networks | Comparing Adaptive HTTP Streaming Technologies[END_REF]. This will eliminate the unwanted freezing events at the client terminal however, reducing the multimedia quality, i.e., the resolution. Different implementations of adaptive HTTP based video delivery are Apple's HTTP Live Streaming (HLS), Microsoft's Silverlight Smooth Streaming (MSS), Adobe's HTTP Dynamic Streaming (HDS) and Dynamic Adaptive

Streaming over HTTP (MPEG-DASH) [START_REF] Networks | Comparing Adaptive HTTP Streaming Technologies[END_REF].

Impacts of different transmission components on multimedia QoE

Different components are involved when a multimedia content is being transmitted from the server to a client. Each of these components has its own effect on the multimedia QoE. Out of these different components source encoding, network and playout buffer have a significant effect on multimedia QoE. In the following section some of these components and their effect on multimedia QoE are explained.

Source coding

A source encoder converts the uncompressed raw media signals into bit streams. However, the size and the bitrate of this bit stream are huge. Therefore, different compression techniques are used in source coding to make the source data light weighted with the lowest possible bit rate in order not to exceed the available network bandwidth. The compression is performed in such a way that the same data can be easily reproduced at the decoder. However, data compression adds visible distortion and artifacts to the multimedia quality since a lot of redundant information is lost. Moreover, the highly compressed multimedia streams are easily susceptible to network impairments. Examples of video compression standards are MPEG-2, H.264, H.265, VC-2, etc.

Network

Today's Internet services rely on unreliable and best effort network. The network bandwidth available between the source and destination is unknown in advance and can change over time. Therefore, there is no guarantee for the bandwidth, the packet loss, the burst loss, the jitter for having a good multimedia quality. Moreover, multimedia streams are most sensitive to the network perturbations and the effect of network perturbations can range from distortion-less to intolerable distortion. The impact of network perturbation on multimedia content streaming can result in a frame loss, freezing, pixelization, etc.

Playout buffer

A multimedia service requires a multimedia player at the client terminal. Most of the players employ a playout buffer to compensate the jitter in the network. If the playout buffer gets empty the playback of the multimedia stream gets interrupted. The size or length of the playout buffer can determine the start-up time or freeze events of the multimedia stream. If the length of the playout buffer is high, then the start-up time will be high; that means users need to wait longer time before playback. On the other hand, if the length of the playout buffer is small, the buffer gets empty fast and there will be risk of freezing events. Therefore, it is required to design the playout buffer smartly. Till now there is no systematic guideline for buffering strategy and dimensioning of player buffer size, however; in the following section we outline video buffering strategies as discussed in [START_REF] Sonnati | New buffering strategies in Flash Player 9 and Flash Media Server 3[END_REF].

• Standard buffering

In the standard buffering strategy, the player maintains one buffer threshold level (buffer size). To start playback, the player continuously downloads the content up to this threshold and when the threshold is achieved the playback starts immediately. In parallel, the player downloads the video content in order to maintain this threshold. However, if the download bandwidth drops below the video data rate, the data available in the buffer decreases continuously. This will empty the buffer and playback stalls until the buffer fills up again to the particular threshold. The main limitation of the standard buffering scheme is that the player needs to have a larger buffer threshold level to compensate longer bandwidth drops, and this leads to a longer prebuffering time which can affect the user QoE.

• Dual-threshold buffering strategy

To avoid the limitations of the standard buffering scheme, the dual-threshold buffering strategy maintains two buffer threshold levels. The lower threshold level is chosen to start the playback fast and the higher threshold level is chosen to compensate longer bandwidth drops.

In this scheme, when the buffer level reaches the lower threshold it starts playback and increases its buffer threshold to the higher threshold level. Therefore, in good network conditions, the buffer grows continuously to achieve the higher buffer threshold level, which can be used in the future to compensate sudden bandwidth drops. However, if the buffer is empty due to adverse network conditions, then a lower value of buffer threshold is triggered for a fast start.

Indicators (KPIs) for multimedia Quality

Key performance indicators (KPIs) are parameters that indicate the overall success or quality of a particular service. In case of multimedia services, KPIs indicate the overall quality of the multimedia service. The measurement of KPI parameters at different transmission points can help to identify and to locate the problem. In general, multimedia KPIs are classified into two groups namely: network level and application level KPIs.

Network level KPIs

Network level KPIs are the QoS parameters associated with the network layer that indicate the quality of a multimedia service. Different packet level information (i.e., IP level information) is used to measure network level KPIs. Some of the important network level KPIs are described below.

Packet loss ratio (PLR)

Packet Loss Ratio is the ratio of the total number of packets lost in a transmission compared to the total number of packets sent. The higher packet loss ratio helps to conclude that the network has problems. Multimedia traffic is highly susceptible to the packet loss ratio.

PLR is defined as follows

PLR=N loss /N send ,
where N loss is the total number of packets lost and N send is the total number of packet sent.

Average Delay

Average delay refers to an average time needed for a packet to reach from source to

Burst loss

If a group of consecutive packets is lost then it is defined as burst packet loss. The larger is the burst loss, the greater is the multimedia quality degradation. A single burst loss can impact a single video frame or can propagate to a number of video frames.

Jitter

Jitter is the change in the packet inter-arrival delay. Jitter shows how much the latency changes from packet to packet. A low jitter indicates a good and uninterruptable connection.

A high jitter is a sign that there is congestion in the network. If we consider the RTP protocol specified in RFC3550 [START_REF] Schulzrinne | RFC 3550.RTP: a transport protocol for real-time applications[END_REF], the jitter is calculated as:

If Si is the RTP timestamp from packet i, and Ri is the time of arrival in RTP timestamp units for packet i, then for two successive packets i and i-1, inter-arrival delay D may be expressed as:

D i = (R i -R i-1 ) -(S i -S i-1 ) = (R i -S i ) -(R i-1 -S i-1 )
The inter-arrival jitter is calculated continuously according to the formula:

J i = ( 15 * J i-1 + |D i | )/16

Throughput

Throughput refers to the number of bits received during a time unit. It is measured in bits per second (bps). The higher is the multimedia content streaming throughput; the better is the multimedia service quality. If the throughput is less than the required multimedia bit rate, the multimedia quality degrades. For example, if throughput is less than desired multimedia bit rate, then the congestion occurs and packets are lost.

Number of duplicate packets

If the same packet is received more than once it is considered as duplicate packet. When duplicated packets appear this means that there are some configuration error in the network or some devices are defective. If the number of duplicate packets increases, the multimedia quality decreases.

Number of reordered packets

A packet is considered as reordered if the sequence number is smaller than the sequence number of the packet previously received. If the number of reordered packets increases, the multimedia quality decreases.

Application level KPIs

Application level KPIs are the performance parameters that are directly associated to the application layer and the presentation of a multimedia content. Some of the important application level KPIs are described below.

Resolution

Resolution is taken into account when video content is considered. Video resolution refers to the number of pixels in both directions (width and height) of a video frame. It is represented in width  height format. Different video formats such as HD (1280 720), SD (720 480), QCIF (176 144), etc. have their own video resolutions. The higher is the video resolution;

the better is the video quality. It can be noticed that, the low resolution videos are preferred for handheld devices and the higher resolution videos are preferred for bigger display screens.

Frame rate/sample rate

Multimedia frame rate refers to the number of video frames presented per second. The higher is the frame rate, the smoother the video appears i.e., the better is the video QoE. Video frame rate is measured in frame per second (fps). In case of audio, a sample rate is used, which is measured as the number of audio samples carried per second (Hz).

Encoding rate

Multimedia encoding rate refers to the data rate at which the multimedia files are encoded.

Multimedia files are generally encoded in Constant Bit rate (CBR) or in Variable Bit rate (VBR). The higher is the bit rate, the higher is the image quality, however, a higher bit rate adds more complexity to the system and requires a larger network bandwidth. This parameter is measured in bits per second.

Freezing time

The duration of time when the multimedia playback stops is called freezing time. This phenomenon occurs when the application's playout buffer is empty. For smooth playback, application's playout buffer should be always full so that video frames/ audio samples can be always available for the playback. The higher is the freezing time, the lower is the multimedia QoE. This parameter is measured in seconds.

Freezing frequency

The number of times the multimedia playback stops or freezes per unit time is referred as freezing frequency. If the freezing frequency increases during the multimedia playback, then the multimedia QoE decreases.

Blurriness and blockiness

Blurriness and Blockiness are applicable to video content. If there are insufficient bits available to represent some details of the image, blurriness and blockiness occur. This phenomenon is basically caused due to the compression techniques.

Estimation of Multimedia Quality of Experience

Limitations of subjective methods:

 Testing environment requires strict attention.

 Real-time implementation is difficult.

 Process is hard for automation.

 Subjective estimation is costly and time consuming.

Limitations of objective methods:

 These methods are hard to correlate with human perception.

 Objective methods may require high calculation power /time.

 When using objective methods, it is hard to integrate all quality affecting parameters of the model.

As listed above, both methodologies have their own shortcoming; to overcome them, various hybrid approaches that combine both subjective and objective methodologies are proposed.

For example, in different approaches such as those presented in [START_REF] Rubino | Evaluating users' satisfaction in packet networks using random neural networks[END_REF] In the following section, we describe some QoE estimation techniques that can be used when estimating the QoE for a video content.

Video QoE estimation

Video QoE estimation refers to the quality estimation of a video signal originated from different sources (movies, sports, cartoons, etc.). Similar to the speech QoE estimation, the video QoE estimation can be classified into two categories which are subjective and objective estimation. Different methodologies for subjective QoE estimation of video signals are described below.

Subjective QoE estimation

Subjective video quality estimation methods require an appropriate test environment. General environment conditions for a subjective laboratory test and a test at home are defined in [START_REF] Itu-R | Methodology for the subjective assessment of the quality of television pictures[END_REF].

Moreover, each user should be screened for (corrected-to-) normal visual acuity on the Snellen or Landolt chart, and for normal color vision using specially selected charts (Ishihara, for instance) [START_REF] Itu-R | Methodology for the subjective assessment of the quality of television pictures[END_REF]. The participation of at least 15 participants is considered as statistically reasonable for this kind of subjective tests [START_REF] Itu-R | Methodology for the subjective assessment of the quality of television pictures[END_REF].

Double Stimulus Continuous Quality Scale (DSCQS)

In the DSCQS method, participants are required to give scores to multiple video sequence pairs. These pairs consist of original and test sequences of duration of around 10 seconds each. The pairs are shown in alternating fashion. These sequences are shown twice and in a random order. Participants are unaware which sequence is an original and which one is a test sequence. They rate the quality on a scale of bad to excellent. This scale has an equivalent scale from 0 to 100. The difference between these two scales is used to remove the uncertainties caused by the material content and/or viewer's experience [START_REF] Itu-R | Methodology for the subjective assessment of the quality of television pictures[END_REF].

Double Stimulus Impairment Scale (DSIS)

The DSIS method [START_REF] Itu-R | Methodology for the subjective assessment of the quality of television pictures[END_REF] differs from DSCQS by showing multiple video sequence pairs only once and the original sequence is always shown before the test sequence. If longer test sequences (for example, over 10 seconds) are shown when using the DSIS or DSCQS method, the time between the original sequence and the test sequence can be increased.

Furthermore, it can be hard to rate the sequences accurately and a psychological "recency" effect can be noticed [START_REF] Alpert | Subjective quality evaluation: the SSCQE and DSCQE methodologies[END_REF].

Single Stimulus Continuous Quality Scale (SSCQE)

In the SSCQE method, a video sequence of around 5 minutes is shown to the participants.

Each participant needs to evaluate the video quality instantaneously by continuously adjusting the slider in each 1-2 seconds. The DSCQS scale (from Excellent to bad) is used in this case [START_REF] Itu-R | Methodology for the subjective assessment of the quality of television pictures[END_REF]. The reference video sequence is not provided to participants. One may notice that in this case it is difficult to compare scores for different test sequences as well as to provide the overall quality rating for a particular test sequence. Moreover, the scores provided by endusers can be also affected by a "recency or memory" effect [START_REF] Alpert | Subjective quality evaluation: the SSCQE and DSCQE methodologies[END_REF].

Simultaneous Double Stimulus Continuous Evaluation (SDSCE)

In the SDSCE method, two video sequences, the original and the test, are simultaneously shown to participants. Participants will check the difference between these two sequences and rate the quality by moving the slider of a handset-voting device using a 0-100 scale. If the difference between the two sequences is null then the slider should be at the top i.e. 100, while when the quality difference is maximum the slider should be at the bottom i.e. 0.

Participants are unaware if which sequence is the original and which one is the test sequence [START_REF] Itu-R | Methodology for the subjective assessment of the quality of television pictures[END_REF].

Subjective Assessment Methodology for Video Quality (SAMVIQ)

Most of the methodologies discussed above were developed to perform the subjective test in the TV or similar environment. However, the SAMVIQ method can be used for PC and for mobile environments. Participants are shown different versions of the same video sequence and when all the sequences are rated, the following sequence content can be then accessed [START_REF] Itu-R | Methodology for the subjective assessment of video quality in multimedia applications[END_REF]. In this case, the participants can access any version of the sequence and they can replay, start or stop, change or keep the current score if they prefer. Participants are allowed to view one version at a time and they use the DSCQS method for rating the sequence. Figure 7 illustrates the test organization in the SAMVIQ. Pairwise Comparison (PC)

The pairwise comparison methodology is preferred when participants are not always capable to express their view by means of exact rating score. The PC method tries to evaluate all possible combinations of two samples. In the pairwise comparison methodology, participants are shown video sequence pairs one after another. The duration of each video sequence is around 10 seconds. Participants observe the quality of these two video sequences one after another and based on their perception they select the video sequence with higher quality i.e., provide the preference [START_REF] Itu-T | Subjective video quality assessment methods for multimedia applications[END_REF]. Preference values are converted to regular quality values (MOS scale) using various algorithms [START_REF] Li | Subjective assessment methodology for preference of experience in 3DTV[END_REF].

Results can be stored in a so called comparison matrix (C)

C1 C2 C3 C1 0 3 1 C2 3 0 2 C3 5 4 0
Objective methods can be classified in three groups [START_REF] Itu-T | User requirements for objective perceptual video quality measurements in digital cable television[END_REF] according to the availability of the original video sequence for estimation; those are full reference method, reduce reference method, and no reference method.

Full reference (FR)

In the full reference method, the distorted sample is compared with the original sample including per-pixel processing and temporal/spatial alignment. The comparison between the original and distorted samples is then used by measuring device (algorithms) for QoE estimation. Figure 8 illustrates the full reference method.

Reduce reference (RF)

The reduce reference method is somehow between the full reference and no reference methods. It is developed to predict the multimedia quality based only on partial information about the original sample. In this case, partial information indicates the features extracted from the original sample. Figure 9 illustrates the reduce reference model. The no reference method uses a degraded signal for the quality estimation and does not rely on any information about the original reference sequence. Due to the lack of an original sample for the comparison, the accuracy of NR methodology is less than that of the FR or RR methods. Figure 10 shows the no reference model. the objective input metrics. Different models available for the objective quality estimation assessment are listed below [START_REF]Video quality assessment methods[END_REF].

Media layer model

The media layer model utilizes media signals as an input for the multimedia QoE estimation.

This model uses content-dependent features (noise level, interruptions, etc.) for the QoE evaluation. Intrusive speech quality estimation methodology along with FR and RR video quality methodologies falls into the media layer model.

Parametric Packet layer model

The packet layer model relies on the information gathered from IP and RTP headers (IP/RTP packet loss, IP/RTP jitter, etc.) for the multimedia QoE estimation. This model is efficient for estimating the "in-service quality". This is because the computational complexity for this technique is not hard as it does not require media signal to be decoded for the estimation purpose.

Bit stream layer model

The bit layer model exploits the information gathered from the payload bit stream before decoding and the packet header information for estimating the multimedia QoE. This model often needs to decrypt the encrypted multimedia payload therefore, it can have high computation complexity.

Hybrid model

The hybrid model utilizes several or all of the above methodologies for estimating the multimedia QoE. It exploits as much information as possible to estimate the multimedia quality; therefore, it is considered as one of the most effective models for the multimedia QoE estimation.

Parametric Planning model

The parametric model relies on network and terminal quality design and management parameters (for example, coding bit rate, packet loss rate, etc.). These models typically use a mathematical formula, representing the quality estimation as a function of different parameters [START_REF] Joskowicz | A general parametric model for perceptual video quality estimation[END_REF].

Figure 11 illustrates the scope of various objective quality estimation models while Table 4 presents the comparison of different objective quality estimation models with existing standards.

Subjective test environment

Subjective tests involve human participants and require appropriate environment to perform the test. Test environment should be unbiased, should not be affected by external noise, and should be close to real world scenario. Most of the research works use lab environment for subjective tests, however, crowdsourcing environment is getting popular as well [START_REF] Shahid | Crowdsourcing based subjective quality assessment of adaptive video streaming[END_REF]. In the following sections, we discuss laboratory and crowdsourcing environments in detail.

Laboratory environment

The laboratory experiment provides a controlled environment for performing subjective tests for evaluating the multimedia quality. Different parameters associated with the test like noiselevel, distance between screen and users, screen size, etc. can be easily controlled according to the requirements. However, lab based experiments have limitations such as 1) high cost in terms of time and labor 2) limited participants diversity [START_REF] Yen | Lab experiment vs. crowdsourcing: a comparative user study on Skype call quality[END_REF]. A laboratory experiment takes weeks for preparing tests, recruiting users, scheduling time slots for supervising the experiments, etc. Also users need to be physically present in the laboratory to perform the test [START_REF] Yen | Lab experiment vs. crowdsourcing: a comparative user study on Skype call quality[END_REF]. Generally lab tests are performed in university or research laboratory so the participants for the test are the either students or researchers. This limits the diversity of participants: most of them are boys which are young and well-educated, etc.

Crowdsourcing environment

In crowdsourcing environment, subjective experiments can be performed from distance and there is little control over the participant's environment. It is computer software assisted method generally performed on a web platform. This methodology mainly involves collecting subjective assessment of quality through ubiquitous streaming via the Internet. Different screen test and cheat detection mechanisms are employed in a crowdsourcing experiment [START_REF] Hoßfeld | Best practices for QoE crowdtesting: QoE assessment with crowdsourcing[END_REF] [24] to make the test reliable and resilient. Crowdsourcing based subjective experiments have gained attention to replace the needs of lab-based tests and these experiments offer promising correlation with the former [START_REF] Li | Subjective assessment methodology for preference of experience in 3DTV[END_REF]. This allows an investigator to get opinions from a vast variety of subjects; in a time-flexible, test-data size scalable, and swift manner [START_REF] Shahid | Crowdsourcing based subjective quality assessment of adaptive video streaming[END_REF]. Different crowdsourcing software available for multimedia subjective test are QualityCrowd [START_REF] Keimel | Qualitycrowd-a framework for crowd-based quality evaluation[END_REF], PC-Video-Test-Interface [START_REF] Shahid | Crowdsourcing based subjective quality assessment of adaptive video streaming[END_REF], etc.

QoS/QoE learning algorithms

The QoS parameters reflect objective network and service level performance and they do not directly address the user satisfaction of the delivered service or application. On the other hand, QoE become an important indicator, useful for network operators and service providers to help them understand the user acceptability towards a particular service or application. The paradigm is shifting towards user-centric evaluation of a service or application performance.

To attract or bind users to a service, real time estimation of QoE is a must for network operators and service providers. Therefore, it is necessary to derive a correlation between the QoS parameters and the QoE, which can be used to identify the impact of different QoS parameters on the QoE of the users and moreover, objectively estimate the QoE. However, the relationship between QoS and QoE is hard to estimate, since this relationship is not linear.

Moreover, higher QoS level does not always yield the higher QoE value. In Table 5 we present different QoS/QoE correlation methodologies available in the literature. A number of models have been developed for automatic evaluation of user satisfaction however, the accuracy of these models are still questionable. Moreover, nowadays, people study human cognitive processes very carefully and try to elaborate models that behave similar to brain neurons. Since a human mind is known to be non-deterministic, it is challenging to get a formal algorithm of the human behavior. That is the reason why researchers turn their attention to self-adaptive models and learning algorithms.

Table 6 provides the comparison of estimation techniques in terms of their modeling capabilities [START_REF] Gray | A Comparison of Techniques for Developing PredictiveModels of Software Metrics[END_REF]. Various QoS/QoE learning algorithms can be found, for example, in [START_REF] Staelens | No-Reference Bitstream-Based Visual Quality Impairment Detection for High Definition H.264/AVC Encoded Video Sequences[END_REF], [START_REF] Agilent | Whitepaper: IPTV QoE: Understanding and interpreting MDI values[END_REF], [START_REF] Kang | An artificial-neural-network-based QoE estimation model for Video streaming over wireless networks[END_REF], [START_REF] Menkovski | Predicting Quality of Experience in Multimedia Streaming[END_REF], [START_REF] Machado | A New Proposal to Provide Estimation of QoS and QoE over WiMAX Networks[END_REF], [START_REF] Du | Research on Relationships between QoE and QoS based on BP Neural Network[END_REF], [START_REF] Khan | Video Quality Prediction Models Based on Video Content Dynamics for H.264 Video over UMTS Networks[END_REF], [START_REF] Malinovski | Quality-of-Experience Perception for VideoStreaming Services: Preliminary Subjective andObjective Results[END_REF], [START_REF] Frank | A neural network based test bed for evaluating the quality of video streams in IP networks[END_REF], [START_REF] Calyam | Multi-ResolutionMultimedia QoEModels for IPTV Applications[END_REF], [START_REF] Mushtaq | Empirical study based on machine learning approach to assess the QoS/QoE correlation[END_REF]. These models use different learning algorithms in different use cases and claim to provide higher accuracy. In Table 7, we present some of the models that use learning algorithms for QoS/QoE estimation. In this section, we briefly discuss the three QoS/ QoE learning algorithms that are used in our works.

Neural networks for QoE prediction

Neural networks are widely used for solving various problems in the artificial intelligent area.

Neural networks are composed of interconnected processing elements (neurons/nodes) working together to solve specific problems. The network can be divided into different levels (layers) which contain a number of neurons and each level can be treated as a collection of states. Usually, for each neuron there exists a formula (firing rule) that calculates its output according to weighted inputs that is used when coming to the next level via weighted edges.

The firing rule decides whether a neuron should be fired for any given input patterns.

The neuron operates in two modes i.e., training mode and use mode. In the training mode, the neurons are trained according to the input patterns, so that the actual output of the network matches the desired output with minimum error. In the use mode, if the network detects known pattern (taught input), its current outputs is an associated output. Otherwise, the firing rule is used to determine whether to fire or not.

In the following section we describe random neural network that is often used for QoS/QoE correlation and QoE estimation.

Random neural network [42]

Generally 3-layer (1 input layer, 1 hidden layer, and 1 output layer) feed forward RNN is considered for the QoS/QoE correlation purpose. The input layer corresponds to the QoS parameters and the output layer represents the QoE score. The output of the network can be represented by a function f (QoS), which is calculated as:

f (QoS) = ∑ ∑ where, Q h = ∑ ∑
Here, C represents input QoS parameters C = (C 1 , •••, C I ), h represents hidden nodes h=1….H, and O represents the output node. Q h is the activity rate of the hidden neuron h. The strictly positive numbers r o , r h for h =1…..H and r i for i =1…..I correspond to the firing rates of the neurons in the network (respectively, for the output one, the hidden nodes, and the I input ones). The weights are the variables tuned during the learning process. We denote by w+ uv (resp. by w - uv ) the weight corresponding to an exiting (resp. inhibiting) signal going from neuron u to neuron v (observe that both numbers w+ uv and wuv are ≥ 0). Learning will thus consist of finding the appropriate values of the weights capturing the mapping from (c (k) to the real number q (k) where q (k) is the quality given by a panel of human observers to some audio or video sequence (depending on the application) when the source parameters had the values present in c (k) for k =1..K [START_REF] Rubino | Evaluating users' satisfaction in packet networks using random neural networks[END_REF].

Fuzzy logic for the QoE prediction

Fuzzy logic was introduced by Lotfi A. Zadeh [START_REF] Zadeh | Fuzzy sets[END_REF] in 1956 and it extends the classical set theory to the so called fuzzy set theory. In the general set theory, a set A can be specified by its characteristic function , and the set A contains an element a, i.e., a  A if (a) = 1, otherwise (a) = 0 and a  A. If A is a fuzzy set then the  function is not a classical predicate but its value belongs to an interval between 0 and 1, i.e.,  a (a)  [0, 1]. In this case, the  function shows the degree how likely that the statement 'a belongs to A' holds. In other words, differently from classical Boolean algebra, a partial truth is introduced in the fuzzy logic where the truth degree may range between completely true and completely false [START_REF] Novák | Mathematical principles of fuzzy logic[END_REF].

Fuzzy logic expert systems are one of the well-known estimation/prediction techniques that is used for making decisions based on imprecise/ambiguous information in various fields [START_REF] Baghel | Survey on Fuzzy Expert System[END_REF].

For instance, Adeli and Neshat [START_REF] Adeli | A fuzzy expert system for heart disease diagnosis[END_REF] proposed a fuzzy expert system approach for diagnosis of heart diseases, while in [START_REF] Ngai | Design and development of a fuzzy expert system for hotel selection[END_REF] a fuzzy expert system is used for a hotel selection. Usually, the aim of a fuzzy logic expert system is to draw a concise result based on ambiguous information. Fuzzy logic expert system has three main components, namely fuzzifier, fuzzy interference engine and defuzzifier as shown in Figure 12 [START_REF] Attaullah | Intelligent vertical handover decision model to improve QoS[END_REF]. The fuzzifier contains the membership functions (fuzzy sets). In the fuzzifier, input parameters (service parameter values) are mapped into membership functions to determine the membership of these parameters to appropriate fuzzy sets. The fuzzy inference engine contains a collection of IF-THEN rules, which are obtained from experts or learnt using other intelligent techniques. The inputs taken from the fuzzifier (i.e., membership values) are applied to the antecedents of the fuzzy rules. The obtained value is then applied to the consequent membership function (i.e., the output QoE membership function). In other words, the consequent membership function is clipped or scaled to the level of the truth value of the rule antecedent. If more than one rule is triggered from one set of input parameters, the outputs of all the rules are aggregated into the aggregated output fuzzy set. The defuzzifier is used to perform a defuzzification, namely a single output value is obtained from the defuzzifier with the use of the aggregated output fuzzy set. There exist various defuzzification p 2 , …, p k that are used for the QoE evaluation, the output QoE of the multimedia service is represented by the variable QoE(W).

Each service parameter p i , i  {1, …, k} is classified into A j classes, j  {1, …, l}, and a membership function  pi Aj (x) is derived for each class of service parameters. In this thesis work, we use the MOS score, thus l = 5 and the score belongs to the set {excellent, good, fair, poor, and bad}. Similar classes are considered for the output QoE. As mentioned above, the membership functions and inference rules are constructed based on the subjective data set, i.e., statistical data provided by end-users and/or by experts.

In order to estimate the QoE of a multimedia service, different patterns (p 1 _value, p 2 _value, …, p k _value) are submitted to the fuzzy logic expert system. The output of the system represents the QoE value for a multimedia service. Below, we provide the algorithm for evaluating the QoE value based on the fuzzy logic expert system.

Algorithm for evaluating/predicting the

2.

Apply the justified inputs (membership values) taken from the fuzzifier to the fuzzy inference rules in fuzzy inference engine.

Rule: IF p i is A j AND p i is A j then QoE is A j .

The output of the rule evaluation is the consequent membership function (output fuzzy set) clipped or scaled to the level of the truth value of the rule evaluation. If more than one rule is triggered the outputs of all the rules are aggregated into the aggregated output fuzzy set.

3.

Defuzzify the aggregated output fuzzy set and get a single QoE value (QoE) in the defuzzifier.

Return QoE value.

There exist various defuzzification techniques (Step 3) [START_REF] Ross | Fuzzy Logic with Engineering Applications[END_REF], for example, the centroid method, the weighted average method, the maximum method, etc., and, in this work we use the centroid method. The mathematical basis of this method relies on the center of gravity (COG) that can be expressed by the following formula.

Center of Gravity (COG)

    dx x xdx x y i i ) ( ) (   ,
where y is the defuzzified output, µ i (x) is the aggregated membership function and x is the output variable.

Rough set theory

Rough set theory (RST) is one of the tools for machine learning. It provides algorithms, explanations and some theoretical basis for the research on learning [START_REF] Hua | Study on the application of rough sets theory in machine learning[END_REF]. It was first proposed by Prof Z. Pawlak in 1980. It can be used for extracting knowledge or decision from uncertain and incomplete information.

RST is used for discovering patterns, rules and knowledge in data. RST has many advantages, such as it does not have information loss, is flexible and extendable as compared with other data mining technologies [START_REF] Laghari | Quantitative and qualitative assessment of qoe for multimedia services in wireless environment[END_REF]. It has also found application in data mining, policy-making analysis, process control, and pattern recognition.

Basic definitions and concepts of RST

In rough set theory we define data in a form of information system (S). Let S = (U, A, V, f) where U and A represent non-empty universal set. Let U has finite n objects {x 1 , x2,…n} and A is non-empty finite attribute set (a1,a2…n).One attribute set corresponds to one equivalence relation, i.e., A= C  D, and C  D =Ø, where C (in our case QoS parameters) is the conditional attribute set and D decision attribute set ( in our case MOS scores) . Here V a represents the domain value for the attribute set a and similarly, f represents the information function.

Evaluation metrics for QoE prediction techniques

In this section, we shortly discuss how QoE prediction techniques can be evaluated with respect to two main criteria: prediction accuracy and self-adaptability. The prediction accuracy refers to the ability of the model's estimated score to match that of the subjective QoE. Therefore, for a model to have high prediction accuracy, the score difference should be minimal. On the other hand, self-adaptability refers to the ability of the model to adapt to new dataset. An efficient model should automatically adapt (re-organize) to the new dataset without much complexity and time. However, both accuracy and self-adaptability highly depend on the size and coverage of the learning dataset and the number of considered input parameters.

Furthermore, in spite of the fact that machine learning algorithms are widely used for evaluating user satisfaction, almost all these models rely on statistics provided by high quality experts and/or collected from ordinary users. Collecting these statistics is the most critical task since it requires time and cost. Moreover, when training a model there are a number of additional points specific to the learning algorithms that need to be considered like defining the number of branches of a decision tree or designing membership functions and inference rules of a fuzzy expert system.

Multimedia QoE standardization bodies

There are different multimedia QoE standardization bodies, industry forums, and others that work on different aspects of multimedia QoE. These include definitions and terms of reference, requirements, recommended practices, test plans, and many more [START_REF] Winkler | Video Quality Measurement Standards -Current Status and Trends[END_REF]. Some of the most active ones are discussed below.

Video Quality Experts Group (VQEG)

The Video Quality Experts Group (VQEG) works in the field of multimedia QoE assessment (particularly video) [START_REF] Vqeg | Video Quality Expert Group -Motivation, Objectives and Rules[END_REF]. It was founded by ITU-T and ITU-R group members in 1997. The group works in the field of video quality assessment and investigates different new and advanced subjective and objectives methods and measurement techniques (VQEG). Moreover, they plan, test, and validate objective quality estimation. VQEG is an open group and does not require fees, membership applications, or invitation to join. Some of the past projects of VQEG groups are FRTV Phase I, FRTV Phase II, multimedia phase I, RRNR-TV etc. Some of the active projects of VQEG groups are 3DTV, Audiovisual HD (AVHD), HDR (High Dynamic Range Video), Hybrid Perceptual/Bitstream JEG-Hybrid, MOAVI (Monitoring of Audio Visual Quality by Key Indicators), Quality Recognition Tasks (QART), RICE (Real-Time Interactive Communications Evaluation), and Ultra HD (VQEG).

ITU-T

The ITU-T (Telecommunication Standardization Sector of the International Telecommunications Union) is the primary international body working in the field of standardization of telecommunications equipment and systems [START_REF][END_REF]. ITU-T is a part of ITU (International Telegraph Unit). This organization is based on public-private partnership and requires membership to be involved in. There are different study groups inside ITU-T that work for multimedia quality standardization work. ITU-T Study Group 9 is focused on studies of cable television and quality assessment methods for video and multimedia services. ITU-T, Study Group 12 (SG12) is mainly studying QoE requirements and assessment methods for multimedia services including IPTV [START_REF] Takahashi | Standardization activities in the ITU for a QoE assessment of IPTV[END_REF]. A joint group has been established i.e., Joint Rapporteur's Group on Multimedia Quality Assessment (JRG-MMQA) to harmonize the work of these two study groups. IUT-T works on all areas of multimedia QoE assessment i.e., speech, audio, video, and multimedia (combined) and has developed different standards for a quality assessment of multimedia for example ITU-T J.148, ITU-T P. NBAMS, ITU-T P.862, etc.

ATIS IIF

The ASTIF-IIF [START_REF]The IPTV Interoperability Forum[END_REF]

Future challenges and issues related to multimedia QoE assessment

Multimedia QoE assessment and monitoring is essential to deliver an optimized end to end high QoE service. This requires a deep understanding and efficient identification of different objective and subjective parameters that impact the user experience. Multimedia content delivery is a large and continuously evolving field that involves various actors from content service providers to Internet service providers, and to content consumers (users) themselves.

Therefore, a comprehensive QoE assessment requires the understanding, the role, and impact of these actors on multimedia content from delivery till consumption [START_REF] Baraković | Survey and challenges of QoE management issues in wireless networks[END_REF]. A multidisciplinary approach involving different measures at the server, network, application, or user levels for a wide range of objective (QoS) and subjective (user perception) metrics is necessary for building QoE assessment models. A typical process for building such model includes:

 Conducting subjective lab or crowdsourcing tests to evaluate user perception in different scenarios. As the number of impacting parameters is relatively high, the objective of subjective tests is to measure user acceptability with respect to a limited number of parameters like screen size change, player buffering strategy, network conditions change, etc.

 Building correlation model to map between parameters (like QoS parameters)

measured during the subjective tests with the QoE scores given by subjects. This phase is considered as the learning phase.

 Evaluating the model against user scores to measure its accuracy.

The QoE assessment model requires the extraction of QoS parameters from different points of the network. For this perspective, the measurement of potential QoS parameters plays a key role in providing the required input data for the quality estimation model. Such measurements can be achieved by installing network monitoring probes on key points in the network infrastructure. These tools deploy Deep Packet Inspection (DPI) techniques to extract relevant parameters from the network traffic. When the traffic is encrypted, DPI needs to decrypt the content in order to extract those relevant parameters. In most of the cases, this operation is simply impossible. Therefore, new models using statistical properties of the network traffic need to be designed. Furthermore, with the push for personalized and user centric services, there is a pressing need for QoE estimation probes capable of processing high speed network links in real time to extract QoS parameters and correlate them with user perceived QoE. However, the relationship between QoS and QoE is fuzzy and non-linear. To address this issue there are large numbers of intelligent algorithms proposed in the literature however, there is still room for innovative mechanisms to efficiently correlate QoE from QoS, in real time. IPTV supports both multicast and unicast delivery of content as shown in Figure 14 and Figure 15 respectively. Multicast IPTV enables TV content provider to deliver TV content in a single stream (using broadcast technology) to many customers at the same time. Since only one stream is transmitted over the network, it saves a considerable amount of bandwidth.

Using IGMP (Internet Group Management Protocol) protocol [START_REF] Fenner | RFC 2236: Internet Group Management Protocol, version 2[END_REF], clients can receive multicast packets and enable the routing of the broadcast stream to their network device through the network. Unicast IPTV is a point to point delivery. There is a separate content stream on the network for each unicast session for each user. VoD is an example of Unicast IPTV where customers request for particular multimedia content and receive it on their TV sets. Since the server needs to send the content to each user in separate streams, unicast IPTV is usually not bandwidth efficient.

Since IPTV multimedia flows share the same bandwidth with other internet traffic, there is a high possibility of network congestion. Moreover, multimedia contents require larger bandwidth and are very sensitive to timing schedule. Therefore, the lack of network resources can lead to packet losses, jitter, delay, etc. with the corresponding result in multimedia impairments.

RTP is generally used as a multimedia transport protocol since it addresses the time critical requirement of multimedia bit streams. RTP provides a timestamp and sequence number to IP packets containing media data. This can be used by the receiver to synchronize play back and manage buffers minimizing network jitter [START_REF]What is IPTV: Unicast vs multicast[END_REF].

In this chapter, we focus on a specific scenario with VoD services that use RTP/UDP as transport protocol and analyze the impact of network perturbation on video QoE. Moreover, we will present a real-time video QoE estimation system based on network QoS parameters.

The accurate estimation of video quality requires access to both the original and the received video streams, however this is unsuitable for real-time monitoring. We are aiming at a real time QoE estimation tool that does not require complete decoding of video stream and access to original reference video, it is based on network QoS parameters that accurately estimates video QoE. This permits monitoring to be performed at intermediate measuring points along the networks path.

Related works

The ability to identify the perceived degree of video impairment due to network perturbation is a key point in the quality estimation of video traffic. Moreover, the impact of network perturbations on video can range from distortion-less to intolerable distortion. Moreover, not all network impairments result necessarily in a visible degradation, which has also been accounted for in ITU-T G.1080 [START_REF]Quality of experience requirements for IPTV services[END_REF] and TR-126 [START_REF]Triple-play Services Quality of Experience (QoE) Requirements[END_REF]. Therefore, measuring the impact of network perturbation on the quality of the video traffic is a challenging task as shown in several works [START_REF] Winkler | Video quality evaluation for Internet streaming applications[END_REF] [75] [START_REF] Issa | Estimation of time varying QoE for high definition IPTV distribution[END_REF].

There have been a number of research works done with the objective of video traffic QoE estimation based on media parameters (PSNR [START_REF] Itu-T | Objective perceptual video quality measurement techniques for digital cable[END_REF], VQM [START_REF] Pinson | A new standardized method for objectively measuring video quality[END_REF], SSIM [START_REF] Wang | Spatio-temporal ssim index for video quality assessment[END_REF], and PEVQ [START_REF] Itu-T | Objective perceptual multimedia video quality measurement in the presence of a full reference[END_REF]) however, very limited work can be found on QoE estimation of video transmission from the network perspective without any media parameters.

Media Delivery Index (MDI) [START_REF] Agilent | Whitepaper: IPTV QoE: Understanding and interpreting MDI values[END_REF] is a scalable metric for assessing the effect of delivery network on the video and can be measured from any point between the video end points. It has two components, delay factor (DF) and media loss rate (MLR), both using packet loss and jitter as predictors of video quality. However, it does not consider users perception in quality estimation and only gives an objective metric for quality evaluation. Furthermore, the work presented in [START_REF] Kim | A study on a QoS/QoE correlation model for QoE evaluation on IPTV service[END_REF] proposed a numerical formula to evaluate QoE using different QoS parameters such as packet loss, burst loss, jitter, delay, GoP (Group of Picture) length.

However, it also lacks user perception and experimental and validation results. Therefore, to incorporate the user perception in quality estimation, a combinational (hybrid) approach which involves both objective and subjective approaches is proposed. In this combinational approach, a subjective test is performed to create a dataset which represents the relationship between objective metrics and subjective QoE. Following this, different intelligent systems are trained with this subjective dataset to objectively predict the QoE. These methodologies rely on the correlation between objective metrics and subjective QoE. However, the relationship between objective QoS metrics and QoE is fuzzy and non-linear, and hard to calculate. Therefore, to address this issue there are numbers of intelligent algorithms proposed in the literature however, there is still room for innovative mechanisms to efficiently correlate QoE from QoS in real time. Most of the intelligent algorithms used for this purpose are based on neural networks [START_REF] Kang | An artificial-neural-network-based QoE estimation model for Video streaming over wireless networks[END_REF] [82] [START_REF] Machado | A new proposal to provide estimation of QoS and QoE over WiMAX networks: An approach based on computational intelligence and discrete-event simulation[END_REF] however, neural networks are computationally complex, requires large training data and more training time, and moreover, the reasoning process is not transparent. Decision tree [START_REF] Staelens | No-Reference Bitstream-Based Visual Quality Impairment Detection for High Definition H.264/AVC Encoded Video Sequences[END_REF] based learning approach was proposed for multimedia quality estimation based on bit stream information (QoS parameters), however, decision trees only partially suit for small datasets, small variations in the dataset need a regeneration of the tree and also the reasoning process is not completely transparent.

In this thesis work, we propose a no-reference QoE estimation system based on fuzzy logic [START_REF] Negnevitsky | Artificial Intelligence, A Guide to Intelligent System[END_REF] to estimate the impact of the network conditions on the video quality, i.e., the QoE. An advantage of using fuzzy expert systems is that they are simple, computationally less intensive and reasoning process is transparent. It can be seen in Table 6 in section 37, that fuzzy system outperforms other estimation techniques in terms of modelling capabilities. Moreover, they are good at making decisions with imprecise information. We consider three QoS metrics (packet loss rate, packet loss burstiness, and jitter) as the network condition indicators. However, it must be noted that our methodology can also incorporate additional parameters as well. The variation of these QoS metrics impacts the quality of the delivered video and, consequently, the user satisfaction level. Our objective is to design and implement a method to estimate the variation of the user satisfaction level in function of the network QoS conditions. At first, we performed a set of subjective tests with real participants to measure the correlation between QoS metrics and QoE of video traffic. Second, we proposed a fuzzy expert system which is based on No Reference method that can estimate the video quality based on the network conditions. Figure 16 illustrates the multimedia QoE estimation framework based on network QoS parameters. The correlation between the QoS metrics and the participants' QoE is transformed into fuzzy membership functions using probability distribution functions and curve fitting methods. We also proposed a simple methodology for fuzzy inference rules generation by assigning weights to the video impairment scores. Three different sets of experiments were performed to evaluate our system. In the first one, we simulated our system in MATLAB and compared the estimated QoE output (also called estimated MOS or eMOS) with the subjective QoE obtained from the participants in a controlled test. This experiment validated our methodology showing high correlation between our estimated and the subjective QoE. In the second experiment, we integrated our system as part of a monitoring tool in an industrial IPTV test bed and compared its output with standard Video Quality Monitoring (VQM). The outputs of both video quality estimation methods were also highly correlated. In the third experiment, we performed benchmarking analysis between our monitoring tool and Indra's monitoring tool [162], the experimental results showed that the scores predicted by both the tools were highly correlated. The experimental results show that the proposed video quality estimation method based on fuzzy expert system can effectively measure the network impact on the QoE. In the following sections, we describe our methodology and the experimental results.

Services

Methodology

In order to develop our video quality estimation technique, we followed a methodology that consists on conducting subjective tests with end user participants in order to build a learning set for correlating objective network QoS metrics with the subjective QoE provided by the participants. This correlation was then used to build the membership functions and inference rules of our fuzzy expert system for video QoE estimation. Figure 17 illustrates the methodology for the proposed multimedia QoE estimation system. a pause of 5 minutes was taken. Figure 18 illustrates the experimental environment for the subjective tests.

Correlation between QoS Metrics and QoE

From the subjective test, we built a learning set that consisted of the mapping between the participants' scores and the QoS metrics for each of the considered video clips. We used a probabilistic approach to correlate QoS metrics to the participants' scores. Therefore, for every QoS metric, we built five different probability distribution functions (pdf) (one function per QoE score) that provide the variation of the participants' ratio (%) with the QoS metric for a specific QoE score. This probabilistic information was changed into a fuzzy set by dividing the pdf by its peak value (normalized pdf) [11]. The fuzzy set, which has the same form as that of the original pdf, is converted into an equivalent triangular or trapezoidal fuzzy set by using a curve fitting method [12]. The triangular or trapezoidal fuzzy set represents the membership functions for the different QoS metrics.

Figure 19 illustrates the QoE scores membership functions associated with the packet loss rate QoS metric. For example, the packet loss rate of 0.2% has membership values of 0, 0.6, 0.8, 1 and 0.35 corresponding respectively to the QoE scores 5, 4, 3, 2, and, 1. We note that a membership value of 1 represents a high degree of membership to the corresponding class and decreasing membership value represents deviation from the class. Figure 20 and Figure 21 illustrate the membership functions for packet loss burstiness and jitter metrics respectively.

Similarly, in Figure 22, the membership functions for the estimated QoE are defined according to the standard MOS [START_REF] Itu-T | Mean opinion Score (MOS) terminology[END_REF] definition.

Membership_value

MOS

Very annoying Annoying slightly annoying

Perceptible but not annoying Imperceptible

Video Quality Estimation Fuzzy Inference Rules

The correlation between the QoS metrics and the video quality, described in the subsection 4.3.2, allowed to build five fuzzy membership functions for the three considered QoS metrics (packet loss rate, packet loss burstiness, and jitter). Based on the combinations of QoS metrics and their rating, we have to estimate whether the network impact on the video quality (QoE) is imperceptible (excellent conditions), perceptible but not annoying (good conditions), slightly annoying (fair conditions), annoying (poor conditions), or very annoying (bad conditions).

That is, we need to associate an estimated QoE score for the different combination of QoS metrics scores. For example:

IF (Packet loss is very annoying) & (Burst Loss is very annoying) & (Jitter is very annoying) THEN (the estimated QoE is very annoying).

Considering the combination of the QoS metrics scores, we have a set of 5 3 possible rules. We follow the following methodology to define the rules while at the same time reducing their number. We associate a "weight" to each rating as follows: 0 for "imperceptible", 1 for "perceptible but not annoying", 3 for "slightly annoying" 5 for "annoying", and, 7 for "very annoying". For every combination, we calculate the rule weight as the sum of the weights of the QoS metric scores. The rule output corresponds to the estimated QoE score as defined in Table 9. For instance, if all the QoS metrics scores are imperceptible then rule score is 0 (0+0+0), which corresponds to "imperceptible" QoE score. Likewise, if two QoS metrics scores are "perceptible but not annoying" and one is "slightly annoying", then the rule weight is 5 (1+1+3), which corresponds to an "annoying" QoE score.

QoE Estimation System

Our proposed video QoE estimation system is based on fuzzy logic that is powered with a learned membership function (QoS/QoE correlation) and a set of fuzzy inference rules. Fuzzy logic is a well-known technique that can handle problems with imprecise and incomplete data [START_REF] Negnevitsky | Artificial Intelligence, A Guide to Intelligent System[END_REF]. Figure 23 illustrates the building blocks of our proposed system that can be placed at any point in the network between the video source and the terminal. The system performs a perflow analysis. A Deep Packet Inspection (DPI) engine at the entry point of the system inspects the network traffic to identify video flows and extract relevant per-flow QoS metrics (packet loss rate, packet loss burstiness, and jitter). These metrics are constantly fed to the fuzzy expert system that uses the defined membership function and inference rules to estimate the QoE on a per flow basis.

Validation of the Proposed Methodology

To validate the proposed methodology, we compared the results obtained from the subjective tests in section 4.3.1 with those obtained from our proposed system. For this end, we used the Fuzzy logic toolbox of MATLAB [START_REF]MATLAB tools for designing, and simulating systems based on fuzzy logic[END_REF] and developed a simulation scenario with our membership functions and rules for validation. Video clips with different QoS metric values were used for validation. For each video clip, we obtained subjective QoE from subjective tests as described in section 4.3.1; and, estimated QoE from our system simulated in MATLAB.

Figure 24 represents the comparison between the subjective and estimated MOS. We can see that subjective MOS and estimated MOS have a linear relationship and are highly correlated with correlation coefficient of 0.95. This indicates that the proposed system succeeds in reflecting user's perception. This is also illustrated in Figure 25 that considers the probability distribution of the difference between the participants' subjective scores (MOS) and the estimated scores. We can see that in around 60% of the tests, the score difference were less than 0.5. It reached 83% for score difference less than 1. This means that in 83% of the cases, the difference in the subjective and estimated QoE were at most one score level (e.g., if the participants reported "Imperceptible" video quality, our method would have reported either "imperceptible" or "perceptible but not annoying"). Only 4% of the tests showed a score difference of more than 1.5. This estimation accuracy emphasizes the ability of the proposed system to measure the impact of the network conditions on the user satisfaction.

Real Test Bed Experimentations

To evaluate our system in close to real network conditions, we implemented the proposed fuzzy expert system for video quality estimation as a module in a MMT [START_REF]Montimage monitoring tool (MMT)[END_REF] probe. Then we integrated MMT probe and a VQM probe in an industrial IPTV test bed [START_REF] Cuadra-Sanchez | D2.2. Definition of requirements of the management systems to keep up with QoE expectations based on QoS and traffic monitoring[END_REF]. The experiments consisted in comparing the results of both probes in the presence of different emulated IP level and copper line perturbations.

In the following sections, we first describe the industrial evaluation test bed; then we present an analysis of the experimental results. We implemented our QoE estimation system as a module in the MMT to facilitate its deployment in IP based networks. Now, the MMT method refers to our QoE estimation method. This monitoring tool uses DPI to detect video flows and extract the QoS metrics of interest to be used for the QoE estimation.

Test Bed for QoE Measurement

At the CPE side, we installed the MMT probe along with a VQM probe in order to compare the results of the two methods. The QoE measurements consisted in analyzing the network traffic of the broadcasted video clips. Two different types of deterioration factors were simulated: xDSL noise and copper faults on the copper cables and, packet loss and message delay/jitter on the IP level on the DSLAM modem.

We compare the results of the MMT method with those obtained using the VQM method.

Experimental Results

A total number of around 600 video streams were analyzed. For every stream, we collected the applied perturbation type, the QoS parameters, and the estimated MOS given by the MMT method and by the VQM method. Figure 27 illustrates the variation with the packet loss rate of the estimated Mean Opinion Score (eMOS) given by MMT and by VQM. For both methods, we can see that the eMOS values decrease with the increase of packet loss. The decrease, however, is sharper using MMT method. This is due to the fact that MMT estimation model is mainly designed for high definition video quality. In fact, the subjective test described in section 4.3.1 used exclusively HD video clips. In the test bed experiments, average quality videos were used. This fact made the transmitted video clips less sensitive to packet loss than HD videos. As the packet loss rate increases, the eMOS of both methods decreases to reach a point where the estimated quality becomes very bad. We should note here that MMT eMOS has a maximum value of 4.51 and a minimum value of 0.494. This is due to the centroid method used for defuzzification in the fuzzy expert system [START_REF] Negnevitsky | Artificial Intelligence, A Guide to Intelligent System[END_REF]. Furthermore, the VQM method is calculated based on media signal parameters therefore, it is expected to yield higher accuracy. On the other hand, the MMT method is based on network parameters and not all network impairments result necessarily in a visible degradation [START_REF]Quality of experience requirements for IPTV services[END_REF] [78] [START_REF] Staelens | No-Reference Bitstream-Based Visual Quality Impairment Detection for High Definition H.264/AVC Encoded Video Sequences[END_REF]. However, the results of the test bed evaluation show that MMT proposed video quality estimation succeeds in reflecting the impact of network perturbations on the video quality with reasonable accuracy. Packet loss (%)

Benchmarking

For benchmarking analysis, we compared the result obtained from our MMT probe with the Indra's probe [162] for QoE estimation in the real IPTV test bed. Video traces from different network monitoring probe with different level of network impairments were used for this purpose. After that, for different level of network impairment we compared the estimated MOS obtained from MMT probe and Indra's probe. In Figure 29 we can see that both probes behave in similar way and their MOS scores are highly correlated with correlation index of 0.86.

Conclusion

In this chapter, we have analyzed the effect of different network QoS parameters on video QoE. From the subjective test, we have found that video QoE is very sensitive to network perturbations and different values of QoS parameters (due to network perturbation) correspond to different levels of video impairments with different QoE values (MOS scores). Therefore it is very hard to accurately estimate video QoE based on observations or simple mathematics as the relationship between QoS and QoE is fuzzy. To address this problem we have proposed a video quality estimation method based on a fuzzy expert system to measure the impact of network condition (QoS parameters) on the user perceived satisfaction level (QoE) of video services. We have performed a set of subjective tests with real participants in order to correlate network QoS parameters levels with the user perceived video quality. The defined fuzzy membership functions were derived from the QoS/QoE correlation using probability distribution functions. We have also proposed a simple method to build estimation inference

Estimated_MOS

Network impairment level

MMT probe

Indra probe 70 rules by assigning weights to the different video impairment scores. The proposed methodology has been validated versus the results of subjective tests. The proposed system has been integrated and tested in an industrial IPTV evaluation test bed. Moreover, we performed a benchmarking analysis between our QoE estimation system and Indra's QoE estimation system. The validation and experimental results have shown that our QoE estimation method is correlated to both the participants' subjective QoE scores as well as to the estimated MOS given by the VQM method and Indra's method. However, to increase the accuracy in estimation/prediction it is necessary to incorporate other parameters like codec parameters, MAC level parameters, users profile, etc. as the network parameters alone cannot precisely evaluate the QoE at the user end.

Chapter 5

QoE estimation of VoD services: MAC Layer technology is also fuelled because of its easy deployment, availability and cost effectiveness.

Moreover, the advancement of this technology has attracted the operators to use it as a supplement to enhance their service and business coverage.

Although there is a huge advancement in wireless technologies, the issues related to QoS provisioning still remain challenging. The legacy IEEE 802.11 a/b/g are capable of supporting bandwidth intensive applications such as audio/video streaming and interactive gaming, but they cannot guarantee QoS when traffic load increases.

Recently, in order to meet multimedia service requirements, the IEEE 802.11 working groups have specified the IEEE 802.11n standard with many new promising enhancements such as

MIMO (Multiple Input Multiple Output), OFDM (Orthogonal Frequency Division

Multiplexing), frame aggregation and Block Acknowledgement (BA). The high data rate that 802.11n allows has enabled the adoption of demanding applications like high definition TV, VOD (Video on Demand), video conferencing, etc. It is interesting to investigate the performance of these multimedia services over such wireless networks. Furthermore, when it comes to wireless networks, MAC (Medium Access Control) protocols play a crucial role in the performance of the whole system as it governs how resources (channel) get allocated to different stations.

For the quality estimation of video traffic in wireless networks, the ability to identify the perceived degree of video impairment with respect to MAC-level parameters is important.

Moreover, the impact of MAC-level parameters on video can be significant and may completely degrade the video quality. Therefore, careful parameterization of these parameters can increase overall performance of video transmission. Many research were conducted regarding quality estimation of video traffic based on network and upper level parameters, however, very few research has been done from the MAC layer perspective.

Generally, QoS parameters are used to determine the service quality, which do not necessarily reflect the user's satisfaction towards a particular service. QoS parameters reflect network and service level performance; however, they do not address user's reaction to the service or application. Therefore, it is necessary to estimate QoE to indicate service quality.

In this chapter, we focus on mapping MAC-level parameters to user perceived QoE in order to provide tools to measure the impact of the MAC-level parameters on the video QoE. In order to achieve this objective, intensive video subjective test with a number of participants were performed. In the video subjective tests, participants were asked to provide MOS [START_REF] Itu-T | Mean opinion Score (MOS) terminology[END_REF] (Mean Opinion Score) values to video clips with impairments (varied MAC-level parameters). These MOS scores reflect the user satisfaction towards video quality. As a first contribution, we evaluate the performance of video transmission over 802.11n from MAC level perspective. Furthermore as a second contribution, we propose a QoE estimation system based on Random Neural Networks (RNN) [START_REF] Rubino | Evaluating users' satisfaction in packet networks using random neural networks[END_REF] to estimate the impact of some MAC-level parameters on video quality.

Related works

The With frame aggregation multiple data units or sub frames are concatenated with one leading physical header. The objective is to reduce the inter-frame space and the preamble all together into to a single radio preamble. This reduces the overhead and resources wastage as compared Moreover, our work analyzes the combinational impact of various MAC-level parameters on video QoE in IEEE 802.11n wireless network.

Performance Evaluation (Effect of individual MAC layer parameters on video QoE)

In this section, we analyzed the performance of video transmission which is affected by different MAC level parameters. In order to achieve this objective, intensive video subjective tests with a number of participants were performed. In these video subjective tests, the participants were asked to provide MOS [START_REF] Itu-T | Mean opinion Score (MOS) terminology[END_REF] (Mean Opinion Score) values as defined in section 2.2.3 to video clips with impairments (varied MAC-level parameters). These MOS scores reflect the user satisfaction towards video quality. Furthermore, we also analyzed how the perceived QoE varies for different video content types.

Potential MAC-layer parameters

In this thesis work, we have taken into account different MAC-level parameters (including some of the enhanced 802.11n MAC parameters) to evaluate their effects on video QoE.

These parameters are either directly associated to MAC layer (e.g., queue size, aggregation, retransmission limit) or indirectly associated (e.g., number of competing station, BER). They all play a crucial role and can impact the quality of the video traffic. Each of these MAC parameters is described below:

Bit Error Rate

Wireless networks suffer due to the error prone wireless channel characteristics. This results in the variation of the Bit Error Rate (BER) that can cause the MAC frame to be received with errors and trigger retransmissions that can impact the overall performances of the system [START_REF] Wang | IEEE 802.11n MAC enhancement and performance evaluation[END_REF].

Aggregation Size

Frame aggregation, here A-MPDU (MAC Protocol Data Unit) aggregation, allows combining many MAC frames into one larger aggregated frame. Streaming applications can take benefit from frame aggregation and the mean aggregate size can increase with both resolution and the wireless channel conditions. Limiting the maximum frame aggregation length can severely impact both the average delay and the quality of a video stream. However in high BER, aggregation size should be limited to reduce larger frame drops [START_REF] Paudel | QoS-HAN: QoS provisioning in home automated networks over[END_REF].

Number of competing stations

The performance of the wireless network degrades with increasing number of users. Indeed, as the number of competing stations increases in the system, stations have to struggle most of the time to find a free channel (due to high contention).This may lead to higher collision rates, larger delays, and packets dropping .

Queue length

Many applications are sensitive to queue characteristics like queue length, waiting time, service time, etc. A large queue can hold more packets to avoid higher packet drop. However, this needs a larger memory which is not feasible in already deployed physical implementations and may also result in outdated information. On the other hand small queue length can increase packet drop, which is more likely to happen in saturated conditions.

Retransmission Limit (Maximum number of retransmissions)

Retransmission is a good policy to deal with corrupted packets at the MAC layer. However, there are delay bounds for each traffic type and when packets arrive after a certain deadline, they become useless. When channel conditions are not highly error prone, using different values for maximum retransmission limits in different channel conditions can enhance the system performance. Figure 30 shows the effect of these MAC-level parameters on video frame delivery ration. In this figure, we presented the case for "movie" content type, however similar affect was observed for other content types as well. We can see that at different values of these MAClevel parameters, the value of frame delivery ration changes. Since, video quality is extremely sensitive to packet drops/ frame drops, a decrease in frame delivery ratio consequently degrades the video QoE. This justifies the importance of the chosen MAC-level parameters

Simulation Platform

For simulations, we use NS2 simulator embedded with IEEE 802.11n module. We considered a network topology consisting of static stations where each station transmits traffic to its paired station. Two types of traffic flow i.e., video and data are used in the simulation. The video flow consisted of H.264/AVC video traces that were fed as input using the Evalvid tool [START_REF] Klaue | Evalvid-A framework for video transmission and quality evaluation[END_REF] and the data flow consisted of CBR (Constant Bit Rate) traffic. The simulation was conducted for three different types of video content, as shown in Table 10. During our simulation, we have designed different scenarios to examine the performance of video under different MAC-level parameters. Some of the simulation parameters that are common to all scenarios are presented in Table 11 along with their respective values.

In each scenario, each video clip was streamed from a source node to a destination node and correspondingly different MAC-level parameters were varied. This was repeated for all three video content types. This combination resulted in a total of 200 video clips with different perturbations.

Thereafter, for the subjective test the constructed video clips along with the original video clips were shown to different participants in a closed room, in a random order. Participants rated each video clip according to their perceived impairment giving one of the following MOS scores. On average each participant rated 100 video clips. After watching 20 video clips a pause of 5 minutes was allowed. Figure 31 illustrates the experimental environment for the subjective tests. A total of 40 users registered for the test, which is considered reasonable for this kind of subjective tests [START_REF] Nezveda | Test persons for subjective video quality testing:Experts or non-experts?[END_REF].

Experimental Results

As a first set of experiments, we analyzed the effect of individual MAC level parameters (aggregation, BER, number of competing stations, and load) on the MAC QoS parameters.

trend these scores follow. It was observed that the impact is more on sport and animation contents than on movie content. This is because video contents chosen for sport and animation had more objects movements as compared to that in movie. Therefore, video service vendors/operator should be more cautious while providing QoS and QoE to different type of video contents.

Impact of Bit Error Rate on video QoE

Figure 35 shows the impact of BER on the video QoE. As BER increases, MOS value decreases exponentially. This is because an increase in BER will increase the number of corrupted video frames due to errors in transmission, which finally leads to the dropping of the video frame. From the subjective test results, BER value less than 1x10 -4 has almost no effect on video transmission with MOS score of 5. However, if the BER value increases above 2x10 -4 video transmission error increases causing degradation in video quality.

Impact of number of competing stations on video QoE

As the number of competing stations in the system increases, stations need to compete more to find the channel free. This situation can be worse when the total traffic in the system gets saturated. So it is necessary to analyze the impact of the number of competing stations over video QoE in both saturated and unsaturated conditions.

Impact of number of competing stations in unsaturated condition

In this scenario, overall channel capacity is unsaturated. To analyze the impact of the number of competing stations on video QoE, the number of stations with CBR traffic flows in the system is increased. When the total traffic rate in the system is low, packet loss will be less due to less congestion. However, as the number of competing stations increases, the collision probability increases due to limitations in the access mechanism and as a result, frame drop probability increases. Figure 36 shows that as the number of competing stations increases, the user perceived video QoE decreases. The result of subjective tests shows that video transmission is not affected much if the competing station in the system is less than 10.

However, as the competing stations in the system increase above 10, video quality degrades which is shown by the exponentially decreasing curves of the MOS values.

Impact of competing stations in saturation condition

In this scenario, the overall channel capacity is saturated. To analyze the impact of the number of competing stations on video QoE, the number of stations with CBR traffic flows is increased. As the total traffic rate in the system is very high, the increase in competing stations increases both congestion and collision probability. Figure 37 shows that the increase in the number of competing stations decreases the user perceived video QoE. From the subjective tests results in saturated conditions, it can be seen that when the competing stations in the system is less than 3 there is no effect on video transmission. However, as the competing stations increases above 3, the video quality starts degrading which is shown by the exponentially decreasing curves of MOS values.

From the above results we can conclude that the effect of increasing the competing stations is more when the total traffic is saturated in the system. For the saturated case, the video QoE drops even when the number of competing station in the system is 3 as compared to 10 when the traffic is unsaturated. full when packet arrival from upper layer is higher than the transmitted rate. It gets worse when the number of competing stations increases because the station has to struggle more to find a free channel. This results in queue overflow and hence leads to frame drops. Therefore, we analyzed the effect of different queue lengths on video QoE with different number of competing stations in saturated traffic conditions. We can observe that, as the queue length decreases, the video QoE decreases correspondingly. The effect of queue length is higher when the number of competing stations is high i.e., 5 CBR flows. For example, for the sport clip with queue length 200 and 2 competing stations, the MOS value is almost 5; however, for the same queue length with 5 competing stations MOS value is 2.5.

Therefore, to achieve higher value of QoE, sufficient queue length should be assigned to the stations. However, we cannot assign higher value of queue since it requires larger memory which is not feasible in physical implementations and may also result in outdated information if packets are stored in the queue for a longer time. introduced in the channel. We can see that as the retransmission limit increases, the video QoE increases for both high and low BER case. However, the MOS value does not improve after certain level of BER and only increasing the retransmission limit value cannot improve the video QoE in high BER. This is caused by numerous frame corruptions due to high error prone channel conditions. For instance, we can see that at high BER, the MOS values are always between 1 and 2.

Impact of Retransmission Limit on video

From the experimental analysis, we can say that video service vendors/operators should be more cautious while providing QoS/QoE to different type of video contents. It was observed that the impact is more on sport and animation contents than on movie content. Furthermore, to improve the video quality in highly loaded conditions, video traffic should be offloaded over less loaded channels. Moreover, to maintain better video quality, video traffic should be transmitted over channel with low BER. Additionally, queue length and retransmission limit value should be wisely selected in order to achieve high video QoE.

Objective QoE estimation based on MAC level parameters

In this section, we propose a QoE estimation system based on Random Neural Networks (RNN) [START_REF] Rubino | Evaluating users' satisfaction in packet networks using random neural networks[END_REF] to estimate the impact of some MAC-level parameters on video quality. At first, we perform a set of subjective tests with real participants to measure the correlation between MAC-level parameters and QoE of video. Secondly, we propose an algorithm based on RNN that can estimate the video quality based on the MAC-level parameters. We simulated our system using "qoe-rnn" [START_REF]qoe-rnn[END_REF] and compared the estimated QoE output (estimated MOS or eMOS) with the subjective QoE obtained from the participants in a controlled test. The experiments validate our methodology showing acceptable correlation between our estimated and subjective QoE. This shows that the proposed video quality estimation method can effectively measure the impact of MAC-level parameters on the video QoE.

Methodology

In order to develop our video quality estimation technique, we followed a methodology that consists of conducting subjective tests with real end user participants in order to build a learning set and correlate objective MAC-layer metrics with the subjective QoE provided by the participants. This data set was then used to train the RNN for video QoE estimation.

Simulation Environment

For the simulations, we use the NS2 simulation environment embedded with the IEEE 802.11n module. A specific network topology was considered which consisted of static stations, where each station sends out traffic to its paired station. We used two types of traffic flows i.e., video and data. For video flow, we used the H.264/AVC video traces that were supplied as input using the Evalvid tool [START_REF] Klaue | Evalvid-A framework for video transmission and quality evaluation[END_REF] and for the data flow, CBR (Constant Bit Rate) traffic was used.

The characteristics of the video traffic are shown in Table 12. Different simulation scenarios were adopted to examine the performance of video under the combined effect of different MAC-level parameters. In our experiments, we have considered BER, number of competing stations, load, and aggregation as the most influencing MAC parameters. The value of the retransmission limit and the queue size are considered to be constant as in practice they are not changed frequently. Aggregation is also considered as one of the important parameters as its effects can be seen in different BER conditions and since it is considered as one of the fundamental parameters in the IEEE 802.11n standard [START_REF] Wang | IEEE 802.11n MAC enhancement and performance evaluation[END_REF].Simulation parameters that are common for all the scenarios are presented in Table 13.

In order to construct different video clips for the subjective test, we streamed the original video clip from the source node to the destination node and simultaneously varied the MAC-level parameters. This resulted in a variety of video clips at the destination, ranging from distorted to distortion less. In total 390 of such video clips were constructed. Users are subscribed to telecommunication operators and service providers and they expect to have best overall performance. With the increase of OTT-based services, high bandwidth is needed to serve them. This bandwidth scarcity will have negative impact on user experience.

Even though service providers may not own OTT services, they are still expected to provide superior user experience and performance [START_REF] Jacobs | Five Major Challenges for Telecommunications Providers[END_REF].

Nowadays, most of the OTT video streaming applications use adaptive HTTP/TCP as a transport mechanism for multimedia streaming. In adaptive HTTP based video delivery, video contents are encoded in different data rates and stored as small fragments (a few seconds each) on the server. This will allow applications to switch between different data rates based upon different network conditions and other variables [START_REF] Networks | Comparing Adaptive HTTP Streaming Technologies[END_REF]. This will eliminate the unwanted freezing events at the client terminal however, reducing the multimedia quality, i.e., the resolution (bit rates). Also, the adaptive bitrate switching (adaptive HTTP/TCP) which is

Screen test

Screen tests are used to find the end user watching conditions for use in the video and image assessment [152]. If the watching conditions at the workers end are not favorable then the test scores available are not reliable. In our web based application we applied two screen test mechanisms as described in [START_REF] Gardlo | Crowdsourcing 2.0: Enhancing execution speed and reliability of web-based QoE testing[END_REF].

In screen test 1 as shown Figure 49, participants were asked to select highest and lowest numbers in faint white color from white background. The numbers are moving in the screen and not every number has equal visibility. Visibility of these numbers also depends on screen orientation, screen resolution, screen brightness, screen color combination, and participant's eyesight. The numbers between 1 and 7 are visible to the participants. The higher the number is, the lower the visibility.

In screen test 2 as shown in Figure 50, participants were asked to click visible stars in faint black color from black background. These stars are stable in the screen and not every star has equal visibility. Visibility of these stars also depends on screen orientation, screen resolution, screen brightness, screen color combination and participant's eyesight. Star numbers between 1 to 7 are visible to the participants. The higher the number is, the lower the visibility.

From the result obtained from screen test, we selected 215 workers out of 266 workers. The quality of the videos can also be compared versus the average video bitrate. This has been illustrated in Figure 53, where the mean of the subjective scores has been calculated over the video contents. Generally, users prefer videos at higher bitrates, i.e., 3 or 5 Mbps and the difference between them is probably more due to the difference in content than the difference in compression levels. Users dislike buffering events and it seems that the frequency is more important than the total duration of these events (both videos at 3 Mbps with buffering has a total buffering time of 2s), which is in line with earlier studies such as the one presented in [START_REF] Van Kester | Estimating the impact of single and multiple freezes on video quality[END_REF]. But if the bitrate is high enough and the frequency of the buffering events is low enough, e.g., the 1F3M video, this seems to be a viable alternative to decreasing the bitrate of the video or having a constant low bitrate, e.g., 600 kbps or 1 Mbps. found that users prefer video at higher bit rates, however, users are very sensitive to buffering events and their experience decreases with increase in frequency of buffering events.

Moreover, if the bitrate is high enough and the frequency of the buffering events is low enough, e.g., the1F3M video, this seems to be a viable alternative to decreasing the bitrate of the video or having a constant low bitrate, e.g., 600 kbps or 1 Mbps.

Chapter 7

QoE estimation for web services

Introduction

Web Services (WSs) are self-contained software systems that can be published, advertised, located, and invoked through the web, usually relying on standardized XML technologies (REST, SOAP,WSDL, and UDDI [START_REF] Curbera | Unraveling the web services web: an introduction to soap, wsdl, and uddi[END_REF]) for description and publication, and on Internet Protocols for invocation [START_REF] Lalanne | Quality of experience as a selection criterion for web services[END_REF]. Popularity of web services is growing rapidly which leads to a large number of web services or application with similar features. This offers users a number of options and introduces a higher demand on price, response time, availability, reliability, service performance, and other non-functional attributes for selecting a web service.

The availability of a large number of web services providing similar functionalities and features has increased the need for sophisticated discovery and selection processes that can better meet the user's needs. The discovery process is a process of identifying or locating a web service that fulfills certain functional properties. On the other hand, the selection process refers to evaluating and ranking the discovered web services for selecting the one that fulfill a set of non-functional properties [START_REF] Badr | Enhancing web service selection by user preferences of non-functional features[END_REF]. As indicated in [START_REF] Badr | Enhancing web service selection by user preferences of non-functional features[END_REF], the "functional properties describe what the service can do and the non-functional properties depict how the service can do it". Non-functional properties involve qualitative or quantitative features such as, throughput, latency, response time, integrity, availability, security, etc. [START_REF] Wang | A fuzzy model for selection of qos-aware web services[END_REF], [START_REF] Tiwari | A non functional properties based web service recommender system[END_REF]. However, a selection process which relies only on a partial set of non-functional properties can be misleading as this will not necessarily reflect the user's satisfaction. Thus, as we propose here, we need a methodology that considers several parameters to estimate the expected user experience, with each having a greater or lesser impact on the resulting estimation.

Users' demand and expectation for web technology is accelerating with time. Users' gets intolerant if the content is not served in expected time and easily switch to other options if their needs are not fulfilled [START_REF] Lohr | For impatient web users, an eye blink is just too long to wait[END_REF]. About 90% of the people do not want to complain for the low service quality. Generally, QoS parameters are used for selecting web services, which do not necessarily reflect the user's satisfaction towards a particular web service. QoS parameters reflect network and service level performance; however, they do not address the user's reaction to the service or application. Therefore, it is necessary to derive a correlation between the QoS parameters and the subjective QoE, so that it is possible to identify the impact of different QoS parameters on the QoE experienced by the users. This motivates research communities for further studies in quality estimation of web services. In recent years, a high amount of research work has been done on QoE assessments for voice and video services. However, little has been done on QoE assessment of web technology.

In this chapter, we propose a methodology to estimate the quality of web services based on a fuzzy-rough hybrid algorithm. Figure 54 shows the web QoE estimation framework based on web QoS parameters. Fuzzy expert systems [START_REF] Negnevitsky | Artificial Intelligence, A Guide to Intelligent System[END_REF] are good at making decision with imprecise information; however, they cannot automatically formulate the rules that they require for making the decisions. Therefore, a fuzzy-rough hybrid expert system is proposed where rough set theory is used to define the rules necessary for the fuzzy expert system. We consider three web QoS parameters: execution time, availability, and reliability as important indicators for QoE estimation. These parameters have been selected because their variation affects the efficiency of web services and the overall user experience; but, it must be noted that our method can also easily integrate more or other parameters. At first, we conducted subjective tests in a controlled environment with real users to correlate QoS parameters to subjective QoE, i.e., Mean Opinion Score (MOS). Based on the results from these subjective tests, we derived membership functions and rules for the fuzzy system. The probabilistic approach is used for deriving membership functions and the Rough Set Theory [13] is used to derive rules from the subjective tests. We simulated our system in MATLAB and compared the estimated

Related Work

The World Wide Web (WWW) has dominated the Internet since it was commercialized in 1995. It has allowed interconnecting the world by sharing information related to daily life activities, such as education, business, commerce, science, social networking, and entertainment. This has fuelled the increase, in the past years, of an enormous amount of web services and applications based on web technologies.

QoE was first defined in the context of multimedia services. A high amount of research attention has been placed in estimating QoE and correlating network QoS with QoE of multimedia services as shown in [7], [START_REF] Pinson | A new standardized method for objectively measuring video quality[END_REF], and [START_REF] Cherif | A_PSQA: Efficient real-time video streaming QoE tool in a future media internet context[END_REF]. In the case of web services, user satisfaction is often measured in terms of response time. If users need to wait a long time during web service session, it will be perceived negatively. Due to the limitation of resources in mobile networks, the situation with this respect can become even more critical. The effect of response time on user behavior in the web is presented in [START_REF] Rossi | User patience and the web: a hands-on investigation[END_REF], [START_REF] Vicari | Measuring internet user traffic behavior dependent on access speed[END_REF], and [START_REF] Egger | Waiting times in quality of experience for web based services[END_REF]. Response time is one of the important parameters; however, it is not sufficient to evaluate web services quality [START_REF] Bouch | Quality is in the eye of the beholder: meeting users' requirements for internet quality of service[END_REF]. Two practical approaches to measure QoE are presented in [113], where a service level approach using statistical samples and a network management system approach using QoS parameters is used. Here the authors identify different key performance indicators for mobile services based on reliability (i.e., service availability, service accessibility, service access time, and continuity of service) and comfort (i.e., quality of session, ease of use, and level of support). However, it does not provide any methodology that could map QoS parameters to QoE. The work presented in [START_REF] Padovitz | Towards efficients election of web services[END_REF] proposed models that allow selecting web

Subjective test

The subjective test platform used in [START_REF] Lalanne | Quality of experience as a selection criterion for web services[END_REF] has been established to perform the subjective tests.

In the experiment, an interactive web application has been developed to simulate typical web service architecture. Each user is asked to use this web application. From the users' responses, a MOS score is obtained. Three QoS parameters: reliability, execution time (in seconds), and availability (in seconds) are measured during the performance of the tests.

 Execution time is measured as a delay at the server level.

 Availability is measured as a period of server downtime, in which the service responds with "Service unavailable, please retry again in a few moments" after each request until a certain time has elapsed.

 Reliability is measured as a number of consecutive erroneous responses, in which the service responds with "An error has occurred, please try again", until the subject has retried the number of times defined by the test variable.

A total of 88 users were registered for the test which is considered reasonable for this kind of subjective tests [START_REF] Nezveda | Test persons for subjective video quality testing:Experts or non-experts?[END_REF]. The details of the experiments can be found in [START_REF] Lalanne | Quality of experience as a selection criterion for web services[END_REF].

Membership functions

The membership functions required for the fuzzy expert system are designed using the subjective data set. The curve values of the membership functions represent the degree to which a particular QoS parameter value belongs to different QoE scores. We used a Probabilistic Distribution Function (PDF) to derive a membership function as described in [7] and [12]. For every QoS parameter, we built different probability distribution functions (PDF)

(one function per QoE score) that provide the variation of the participants' ratio (%) with the QoS metric for a specific QoE score. This probabilistic information was changed into a fuzzy set by dividing the PDF by its peak value (normalized PDF) [11]. The triangular or trapezoidal fuzzy set represents the membership functions for the different QoS metrics. In our case, we reduce the five scale MOS classes to three scale MOS classes (low, medium, and high) for QoS parameters because it was very difficult to find the boundary region between fair and good, and bad and poor, which has thus been replaced respectively by medium and high classes. intelligent system for web QoE estimation. The web QoS parameters are constantly fed to the fuzzy expert system that uses the pre-defined membership function and inference rules to estimate the web QoE.

Experimental Results

To validate the proposed methodology, we compared the results obtained from the subjective tests with those obtained from our proposed system. For this, we used the Fuzzy logic toolbox of MATLAB [START_REF]MATLAB tools for designing, and simulating systems based on fuzzy logic[END_REF] and developed a simulation scenario with our membership function and rules for validation. Test cases with different input values were used for validation. For each test case, we obtained subjective QoE from subjective tests; and, estimated the QoE using our system simulated in MATLAB. Figure 61 represents the comparison between the subjective MOS and estimation MOS. We can see that subjective MOS and estimated MOS have a linear relationship and are highly correlated with correlation coefficient of 0.84. This indicates that the proposed system succeeds in reflecting the user's perception. We should note here that the estimated QoE obtained by our system has a maximum value of 4.51 and a minimum value of 0.523. This is due to the centroid method used for defuzzification in the fuzzy expert system [START_REF] Negnevitsky | Artificial Intelligence, A Guide to Intelligent System[END_REF]. The results indicate that the proposed system succeeds in reflecting the user's perception. This is also illustrated in Figure 62 that considers the probability distribution of the difference between the subjective scores (QoE) and the estimated QoE scores. We can see that in around 83% of the test cases the score differences were less than 0.5. This estimation accuracy emphasizes the ability of the proposed system to measure the QoE of web services.

Conclusion

In this chapter, we proposed a novel method based on fuzzy-rough hybrid expert system for estimating QoE of web services. The estimated QoE can be used as a selection criterion for web services. We have performed a set of subjective tests with real participants in order to correlate web QoS parameters levels with the user perceived quality. The defined fuzzy membership functions were derived from the QoS/QoE correlation using probability distribution functions. We have used a Rough Set Theory to generate estimation inference rules. The proposed methodology has been validated against the results of subjective tests.

The validation results have shown that our QoE estimation method is highly correlated to the participants' subjective QoE scores. In this thesis, our work was mostly dedicated to analyzing the impact of QoS parameters on different internet services and applications. We considered two services as use cases:

multimedia VoD and OTT services and web applications. We identified different QoS parameters associated to different layers of the OSI model, which could directly or indirectly impact the service quality. After that, intensive subjective tests were performed to correlate QoS parameters to QoE. We also proposed a QoE estimation system based on different machine learning techniques that use subjective database to train and predict or estimate the QoE. In the following, we summarize the contributions of this thesis work listed in Table 18.

QoE estimation of Web services

We proposed an objective QoE estimation system for web service selection using web QoS parameters. The selected web QoS parameters were execution time, reliability and availability. A subjective test was performed in laboratory using a web application to correlate web QoS parameters with web QoE. The subjective dataset obtained from the subjective test was used to train a novel machine learning algorithm i.e. fuzzy rough hybrid expert system to estimate the web QoE objectively. To validate our system we compared the results of the subjective test with our proposed objective web QoE system. The results showed high correlation between the subjective and objective scores with correlation coefficient 0.84, which validates our proposed system.

Future Work

This thesis has several contributions and has provided a framework for monitoring and estimating the QoE of different application and services. At the same time, it unlocks some research issues and future directions. In the following, we remark several open issues that can be explored in the future as a continuation of our work.

We will like to extend our QoE estimation framework to other services such as gaming, OTT video etc. We will also investigate the usage of the feedback of our system to implement corrective actions on network and application levels to recover the QoE to satisfactory levels. Furthermore, we will incorporate feature selection methodologies that could automatically select the most impacting QoS parameters. We also plan to make technical analysis of different methodologies proposed in our system in terms of accuracy, processing speed, computational overhead, etc. and compare with other solutions available in the market.

We will extend our VoD QoE estimation system to enrich the model with relevant media and codec parameters and measure the QoE estimation improvement we get. Special care will be taken to measure the computational overhead as we target real-time estimation of video QoE over high speed data links (10+ Gbps).

We will like to exploit the data set obtained from crowdsourcing based subjective test to objectively estimate the QoE for adaptive bitrate streaming. It can be used to train machine learning algorithms for the QoE estimation, as done in the previous contributions.

Additionally, we will also investigate on how to adapt our estimation tool in different streaming implementations such as Dynamic Adaptive Streaming over HTTP (MPEG-DASH), Apple HTTP live streaming (HLS), Microsoft Smooth Streaming, etc.

xi 7 . 1

 71 Introduction .................................................................................................................. 7.2 Related Work................................................................................................................7.3 Methodology ................................................................................................................ 7.3.1 Subjective test ....................................................................................................... 7.3.2 Membership functions .......................................................................................... 7.3.3 Inference Rules ..................................................................................................... 7.4 Web QoE Estimation system........................................................................................ 7.5 Experimental Results.................................................................................................... 7.6 Conclusion .................................................................................................................... Chapter 8 ..................................................................................................................................... Conclusion and Future work ....................................................................................... 8.1 Conclusion .................................................................................................................... 8.1.1 QoE estimation of VoD services........................................................................... 8.1.2 QoE estimation of OTT services........................................................................... 8.1.3 QoE estimation of Web services ........................................................................... 8.2 Future Work ................................................................................................................. References ................................................................................................................................... xii

Figure 1 :

 1 Figure 1: Consumer Internet video consumption [3] ......................................................................

Figure 2 :

 2 Figure 2: Residence video service adoption [4] ..............................................................................

Figure 3 :

 3 Figure 3: Consumer mobile video service adoption [4]..................................................................

Figure 4 :

 4 Figure 4: QoE estimation framework .............................................................................................

Figure 5 :

 5 Figure 5: Multimedia Quality of Experience ecosystem ..............................................................

Figure 6 :

 6 Figure 6: Multimedia transmission ...............................................................................................

Figure 7 :

 7 Figure 7: Test organization in SAMVIQ [48]...............................................................................

Figure 8 :

 8 Figure 8: Full reference model ......................................................................................................

Figure 9 :

 9 Figure 9: Reduce reference model ................................................................................................

Figure 10 :

 10 Figure 10: No reference model .....................................................................................................

Figure 11 :

 11 Figure 11: Scope of different objective quality estimation model ................................................

Figure 12 :

 12 Figure 12: A fuzzy expert system .................................................................................................

Figure 13 :

 13 Figure 13: A fuzzy logic expert system for QoE estimation ........................................................

Figure 14 :

 14 Figure 14: Multicast IPTV (reference) .........................................................................................

Figure 15 :

 15 Figure 15: Unicast IPTV (VoD) ...................................................................................................

Figure 16 :

 16 Figure 16: Multimedia QoE estimation Framework (Based on network QoS parameters ...........

Figure 17 :

 17 Figure 17: Methodology of proposed video QoE estimation system ...........................................

Figure 18 :

 18 Figure 18: Experimental setup for subjective tests. ......................................................................

Figure 19 :

 19 Figure 19: Membership functions for packet loss rate metric. .....................................................

Figure 20 :

 20 Figure 20: Membership functions for packet loss burstiness metric. ...........................................

Figure 21 :

 21 Figure 21: Membership functions for jitter metric. ......................................................................

Figure 22 :

 22 Figure 22: Membership function for the estimated MOS (eMOS) ...............................................

Figure 23 :

 23 Figure 23: Building blocks of the video QoE estimation system .................................................

Figure 24 :

 24 Figure 24: Comparison between the subjective and estimated MOS ...........................................

Figure 25 :

 25 Figure 25: Probability distribution of the subjective and estimated MOS difference ..................

Figure 26 :

 26 Figure 26: Vierling experimental test bed for QoE measurements ...............................................

Figure 27 :

 27 Figure 27: Variation of eMOS with packet loss rate ....................................................................

Figure 28 :

 28 Figure 28: Estimated MOS in different copper line perturbations ...............................................

Figure 29 :

 29 Figure 29: Benchmarking MMT probe with Indra's probe ..........................................................

Figure 30 :Figure 31 :

 3031 Figure 30: Frame delivery ratio vs different MAC-level parameters ...........................................

Figure 32 :

 32 Figure 32: QoS and QoE versus number of competing stations ...................................................

Figure 33 :

 33 Figure 33: Aggregation without Block Ack..................................................................................

Figure 34 :

 34 Figure 34: Aggregation with Block Ack .......................................................................................

Figure 35 :

 35 Figure 35: Impact of BER on video QoE......................................................................................

Figure 36 :

 36 Figure 36: Impact of number of competing stations on video QoE in unsaturated conditions. ...

Figure 37 :

 37 Figure 37: Impact of number of competing stations on video QoE in saturation condition .........

Figure 38 :

 38 Figure 38: Impact of queue length on video QoE .........................................................................

Figure 39 :

 39 Figure 39: Impact of retransmission limit on video QoE .............................................................

Figure 40 :

 40 Figure 40: Video QoE estimation Framework (Based on MAC level parameters) ......................

Figure 41 :

 41 Figure 41: Experimental environment ..........................................................................................

Figure 42 :

 42 Figure 42: QoE estimation Module...............................................................................................

Figure 43 :

 43 Figure 43: Comparison between subjective and estimated QoE ..................................................

Figure 44 :

 44 Figure 44: Probability distribution of the subjective and estimated MOS difference ..................

Figure 45 :

 45 Figure 45: Variation of QoE with respect to BER and the number of competing stations ...........

Figure 46 :

 46 Figure 46: Variation of QoE in different BER and aggregation lengths ......................................

Figure 47 :

 47 Figure 47: Framework for QoE estimation of OTT video services ..............................................

Figure 48 :

 48 Figure 48: Adaptation Scenarios ...................................................................................................

Figure 49 :

 49 Figure 49: Screen test 1.................................................................................................................

Figure 50 :

 50 Figure 50: Screen test 2.................................................................................................................

Figure 51 :

 51 Figure 51: Comparison between laboratory and crowdsourcing subjective test ..........................

Figure 52 :

 52 Figure 52: Correlation between laboratory and crowdsourcing subjective test ............................

Figure 53 :

 53 Figure 53: MOS score vs target bitrate .........................................................................................

Figure 54 :

 54 Figure 54: Web QoE estimation system (QoS parameters) ........................................................

Figure 55 :

 55 Figure 55: Methodology for Web QoE estimation .....................................................................

Figure 56 :

 56 Figure 56: Membership function for execution time ..................................................................

Figure 57 :

 57 Figure 57: Membership function for availability ........................................................................

Figure 58 :

 58 Figure 58: Membership function for reliability ..........................................................................

Figure 59 :

 59 Figure 59: Membership function for QoE score .........................................................................

Figure 60 :

 60 Figure 60: Web QoE estimation system .....................................................................................

Figure 61 :Figure 62 :

 6162 Figure 61: Comparison between subjective and objective QoE .................................................

Figure 4 :

 4 Figure 4: QoE estimation framework

Figure 6 :

 6 Figure 6: Multimedia transmission

  destination. It can be measured in ms (milliseconds) or μs (microseconds). Larger delay results in increase of start-up time in multimedia playback. End to End Delay = Packet receive time -Packet send time Average delay = Σ (Packet receive time -Packet send time)/ total packets received

Figure 7 :

 7 Figure 7: Test organization in SAMVIQ [48]

Figure 8 :

 8 Figure 8: Full reference model

Figure 9 :

 9 Figure 9: Reduce reference model

Figure 10 :reference model 3 . 5 . 2

 10352 Figure 10: No reference model

Figure 11 : 4 :

 114 Figure 11: Scope of different objective quality estimation model

Figure 12 :

 12 Figure 12: A fuzzy expert system

1 .

 1 QoE value for multimedia service Inputs: A multimedia service W, service parameters p 1 , p 2 , …, p k and their values p 1 _value, p 2 _value, …, p k _value; Inference rules given in the form: IF <antecedent> THEN <consequent>; Membership functions  pi Aj (x), i  {1, ..., k}, j  {1, ..., l}, where x is the service parameter value and A j represents different classes of corresponding parameter values. Output: The QoE value (QoE) Procedure: Map the service parameter values p 1 _value, p 2 _value, …, p k _value into the membership functions  pi Aj (x) and get the membership value in the fuzzifier.

  is working with Industry segments to define necessary standards and specifications for IPTV network architecture, QoS and QoE, Security, and Interoperability (ATIS-IIF). It requires membership for involvement. Different partners working together in ATIS IIF include service providers and manufacturers, content and entertainment providers, manufacturers, and the entire IPTV industry ecosystem. The Quality of Service Metrics Committee (QOSM) inside ATIS IIF works in different issues related to QoS and QoE in IPTV services. It has issued different documents related to QoS and QoE in IPTV services like ATIS-0800008, ATIS-0800004, etc.

Finally, Chapter 4

 4 QoE estimation of VOD services: Network Layer perspective 4.1 Introduction IPTV (Internet Protocol Television) is a means by which television services are transmitted using IP suite over a packet switched network. According to the International Telecommunication Union focus group on IPTV (ITU-T FG IPTV), "IPTV is defined as multimedia services such as television/video/audio/text/graphics/data delivered over IP based networks managed to provide the required level of quality of service and experience, security, interactivity and reliability". IPTV can deliver both Live TV as well as Video-on-Demand (VoD) services.

Figure 14 :Figure 15 :

 1415 Figure 14: Multicast IPTV (reference)

Figure 16 :

 16 Figure 16: Multimedia QoE estimation Framework (Based on network QoS parameters

Figure 17 :

 17 Figure 17: Methodology of proposed video QoE estimation system

Figure 19 :Figure 20 :

 1920 Figure 19: Membership functions for packet loss rate metric.

Figure 21 :

 21 Figure 21: Membership functions for jitter metric.

Figure 22 :

 22 Figure 22: Membership function for the estimated MOS (eMOS)

Figure 23 : 4

 234 Figure 23: Building blocks of the video QoE estimation system

Figure 24 :Figure 25 :

 2425 Figure 24: Comparison between the subjective and estimated MOS

Figure 26

 26 Figure 26 presents the building blocks of the Vierling experimental test bed, which is an industrial evaluation test bed for QoE assessment in Digital Subscriber Line technology (xDSL). The test bed is constituted of an IPTV server that uses VLC to stream video clips over RTP/UDP transport. The streaming server is connected to a Digital Subscriber Line Access Multiplexer (DSLAM) modem emulator at the central office (CO). At this point of the network, an IP traffic perturbation tool is used to emulate network conditions by introducing a variety of network impairments such as packet loss, delay, and jitter. These perturbations are used for quantifying the network impact on the video QoE. The DSLAM modem is connected via kilometers length copper lines to the Customer Premises Equipment (CPE) xDSL modem. Furthermore, the test bed allows performing measurements on multiple copper line configurations at the network access level and allows adding xDSL disturbers with the help of remote controlled HF signal and noise generator. The test bed can reproduce a typical DSL network configuration: (i) the IPTV server represents the service provider network, and (ii) the DSLAM modem emulator and CPE (Customer Premises Equipment) modem correspond to the access network on a physical copper line with thousands of copper configurations. In addition, external network probes can be attached to the modem at the CPE side to sniff the video traffic and analyze the collected data in order to correlate QoS parameters and estimate the corresponding QoE level.

Figure 26 :

 26 Figure 26: Vierling experimental test bed for QoE measurements

Figure 27 :Figure 28 :

 2728 Figure 27: Variation of eMOS with packet loss rate

Figure 29 :

 29 Figure 29: Benchmarking MMT probe with Indra's probe

Perspective 5 . 1

 51 IntroductionThe use of IEEE 802.11 based WLANs (Wireless Local Area Networks) is immense. The reason for this paradigm shift is the tremendous increase of Wi-Fi enabled gadgets, smart phones and tablets in our daily lifestyle. There is an equal amount of increase in wireless multimedia services like Voice over IP (VOIP), IPTV, telemedicine and internet gaming for both home and professional use. The reason of becoming one of the preferred access

Figure 30 :

 30 Figure 30: Frame delivery ratio vs different MAC-level parameters

Figure 31 :

 31 Figure 31: Subjective test Environment

Figure 35 :

 35 Figure 35: Impact of BER on video QoE

Figure 36 :

 36 Figure 36: Impact of number of competing stations on video QoE in unsaturated conditions.

Figure 37 :Figure 38 :

 3738 Figure 37: Impact of number of competing stations on video QoE in saturation condition

Figure 39 :

 39 Figure 39: Impact of retransmission limit on video QoE

Figure 40 :

 40 Figure 40: Video QoE estimation Framework (Based on MAC level parameters)

  deployed to avoid freezing can cause irritation to users because of frequent change in multimedia quality as bit rate changes. Different implementations of adaptive HTTP based video delivery are Apple's HTTP Live Streaming (HLS), Microsoft's Silverlight Smooth Streaming (MSS), Adobe's HTTP Dynamic Streaming (HDS), and Dynamic Adaptive Streaming over HTTP (MPEG-DASH) [39].

Figure 49 :test 1 Figure 51 :

 49151 Figure 49: Screen test 1

Figure 52 :

 52 Figure 52: Correlation between laboratory and crowdsourcing subjective test

Figure 53 :

 53 Figure 53: MOS score vs target bitrate



  QoE output with the subjective QoE obtained from the participants in the controlled tests. The results from the experiments validated our methodology showing high correlation between our estimated QoE and the subjective QoE. Fuzzy rough hybrid expert system

Figure 54 :

 54 Figure 54: Web QoE estimation system (QoS parameters)

Figure 56 107 0. 5 ,

 561075 Figure 56 illustrates the QoE scores membership functions associated with the execution time QoS parameter. For example, the execution time of 1.5 seconds has membership values of

Figure 56 :Figure 57 :Figure 58 :Figure 59 :

 56575859 Figure 56: Membership function for execution time

Figure 60 :

 60 Figure 60: Web QoE estimation system

Figure 61 :Figure 62 :

 6162 Figure 61: Comparison between subjective and objective QoE

Table 1 :

 1 Multimedia content characteristics and their description ...............................................

Table 2 :

 2 Network QoS parameters and their description ..............................................................

Table 3 :

 3 Comparison between different video subjective test methodologies [46][49] ...............

Table 4 :

 4 Comparison of objective quality estimation models with existing standards[START_REF] Takahashi | Standardization activities in the ITU for a QoE assessment of IPTV[END_REF]. .......

Table 5 :

 5 A summary of the selected evaluation approaches for each model[START_REF] Alreshoodi | Survey on QoE\ QoS Correlation Models For Multimedia Services[END_REF] ......................

Table 6 :

 6 Comparison of estimation techniques in terms of modeling capabilities [130] .............

Table 7 :

 7 QoS/ QoE models using learning algorithms .................................................................

Table 8 :

 8 Video characteristics .......................................................................................................

Table 9 :

 9 In this Internet revolution era, technology became an essential part of our daily life. Most of us possess technological gadgets that help accessing different Internet services like voice, video, emails. Providing Internet access to these different services involves three different players: network operators, service providers, and content providers. This multi-party ecosystem survives as these different players fulfill customers' demands and Quality of Experience (QoE) requirements. In return, the customers pay for the service they get.

	xvi

Rule weight to QoE mapping .........................................................................................

Table 10: Video characteristics ..................................................................................................... Table 11: Simulation parameters .................................................................................................. Table 12: Video characteristics .

.................................................................................................... Table 13: Simulation parameters .................................................................................................. Table 14: Soccer comparison matrix ............................................................................................ Table 15: Subjective Test Results ............................................................................................... Table 16: Discretization Table .................................................................................................... Table 17: QoE Index Decision Making Table ............................................................................ Table 18: Thesis contributions ....................................................................................................

  This thesis work starts with investigating different QoE estimation techniques from the state of the art. At first, we focused on identifying the key performance indicators (KPIs) that can impact the users perceived QoE. These KPIs are basically QoS parameters relevant to different OSI layers i.e., network layer, Medium access control (MAC) layer and application layer. Then, we utilize the QoS -QoE relationship of different services to objectively estimate the QoE in real-time. For this, a subjective test is performed to correlate selected QoS parameters to users perceived QoE for a particular service. This correlation is then used

by different intelligent machine learning algorithms to objectively predict QoE in real-time. In this work, we proposed novel subjective test platforms as well as different intelligent algorithms. The most relevant contributions of this thesis are listed below:  Objective QoE evaluation framework-based on QoS and QoE relationship An objective QoE evaluation framework that correlates the relationship between QoS parameters and subjective QoE to objectively estimate the QoE in real-time is presented here.

For this purpose, we designed a subjective test platform for conducting subjective tests with end user participants in order to build a learning set for correlating objective QoS metrics with the subjective QoE provided by the participants. This correlation was then used by different intelligent algorithms to objectively estimate the QoE. This framework includes two novel subjective test platforms i.e., laboratory test platform and crowdsourcing platform. Moreover to correlate QoS with QoE, we used different intelligent algorithms out of which we present

  or having a constant low bitrate. In Chapter 7, we proposed a novel algorithm based on fuzzy rough hybrid expert system to estimate the web QoE from objective web QoS parameters. The estimated web QoE can be used to select the most performing service among different web services. Finally Chapter 8 concludes the thesis work by highlighting the main contributions and results, and discussing future perspective and research directions.In this chapter, we describe an overall QoE evaluation framework that is used for QoE estimation of different application/services in this thesis work. The proposed QoE evaluation framework is shown in Figure4. This framework exploits the relationship between QoS parameters and QoE to objectively estimate the QoE. In order to achieve this relationship we perform a subjective test in which participants provide QoE scores to different service quality.

	Chapter 2
	QoE evaluation framework
	2.1 Introduction

2, we present a QoE estimation framework. The proposed framework can be used by different Internet services to objectively estimate the QoE. This chapter also proposes different novel subjective platforms as well as intelligent machine learning algorithms for correlating QoS-QoE. In Chapter 3, we present a deeper insight on multimedia services, components of multimedia QoE ecosystem, different methodologies and techniques for estimating multimedia QoE etc. In Chapter 4, we analyze the impact of different network QoS parameters on users perceived video QoE for

VoD services. We proposed a methodology based on a fuzzy expert system to objectively estimate the video QoE. To validate our methodology, we integrated our system as part of a monitoring tool in an industrial IPTV test bed and compared its output with standard Video Quality Monitoring (VQM). In Chapter 5, we analyze the impact of different MAC-level QoS parameters on the video QoE for VOD services. We utilize the correlation between QoS parameters and QoE to train a random neural network which is then used for objectively estimating QoE. In Chapter 6, we performed a crowdsourcing experiment to study the perceptual preferences of various adaptive video streaming scenarios. We performed a crowdsourcing based subjective test of OTT video services to analyze whether users prefer bit rate switching or buffering events in adverse network conditions. We found that users are very sensitive to buffering events however, if the bitrate is high enough and the frequency of the buffering events is low enough, this seems to be a viable alternative to decreasing the bitrate of the video These different service qualities are achieved by varying different QoS parameters. The QoS parameters can be associated to different OSI (Open Systems Interconnection model) layers.

  Multimedia services and applications have grown exponentially in the recent years. Different multimedia services like IPTV, VOD, OTT video, video conferencing, etc. are very common in a massive Internet market. To gain a prominent market share, different vendors are competing with each other; such competition includes cable television, Internet service providers, traditional and emerging telephony carriers, etc. On the other hand, multimedia customers are expecting to access a high quality service on any device they are using (PC,

smartphone, tablet...). A user gets intolerant if the multimedia service is not satisfactory and he/she easily shifts to other options if his/her needs are not fulfilled. This requires the service provider to deliver the best multimedia quality to the customers under heterogeneous network conditions as well as for different terminals' capabilities. In these challenging scenarios, it is critical to guarantee an appropriate QoE for the end users. This requires efficient QoE estimation and monitoring techniques to track the performance of the multimedia service in terms of user's perception.

The QoE estimation methods can be implicitly categorised into subjective and objective methods. Subjective methods consist of many participants viewing sample multimedia and rating its quality according to a predefined quality scale depending on their personal perception. On the other hand, objective methods are used to measure the QoE based on objectively measured network/media parameters. Different limitations of subjective and objective methods are listed below.

Table 5 : A summary of the selected evaluation approaches for each model [151]

 5

	Aspects/ Models	AQoS	Parameters	NQoS	Parameters	Video coding	Parameters	Subjective	Metrics	Objective	Metrics	Learning	Technique	Simulation /	Test bed	Technology	Reference	Measurements
	Fiedler et al [134]													Simulation /test-bed	Wired	RR
	Siller et al [135]													test-bed	Wired	FR
	Wang et al [55]													Simulation /test-bed	Wired	NR
	Agboma et al [136]													test-bed	Wireless	NR
	Menkovski et al [137]													Simulation /test-bed	Wireless	NR

AQoS -Application level QoS, NQoS -Network level QoS, Subjective metric -MOS, Objective metric -PSNR, SSIM, VQM, etc.

Table 6 : Comparison of estimation techniques in terms of modeling capabilities [130]

 6

	Technique Model	Can	Explains	Suits	Can be	Reasoning	Suit	Include
		Free	resist	output	Small	adjusted	process is	complex	known
			outliners		data	for new	visible	model	facts
					sets	data
	Least	N	N	P	N	N	Y	N	P
	Square
	Regression
	Robust	N	Y	P	P	N	Y	N	P
	Regression
	Neural	Y	N	N	N	P	N	Y	P
	Networks
	Fuzzy	Y	P	Y	Y	P	Y	Y	Y
	Systems
	Hybrid-	Y	P	Y	P	P	P	Y	Y
	Neuro-
	Fuzzy
	system
	Rule Based	N	N/A	Y	N/A	N/A	Y	Y	Y
	Systems
	Case-	Y	P	Y	P	Y	P	Y	N
	Based
	Reasoning
	Regression	Y	Y	Y	P	Y	P	Y	P
	Trees
	Decision	Y	Y	Y	P	Y	P	Y	P
	Trees
							(Yes = "Y", No= "N", Partially = "P")

Table 7 : QoS/ QoE models using learning algorithms

 7

	Models	Learning	algorithm	Subjective	test	Objective	metric	QoS	parameters	Correlation	Conclusion
	Menkovski	Support	Yes			SI, TI, video	-SVM accuracy	-Classification
	et al [137]	Vector					bitrate, video	Mobile-	based	on
		Machine					framerate	88.59±2.85%,	acceptable	or
		(SVM),							PDA-	unacceptable QoE
		Decision							89.38±2.77%,	(2 class)
		Tree (DT)							Laptop-
										91.45±2.66%;
										-DT, accuracy
										Mobile-
										93.55±1.76%,
										PDA-
										90.29±2.61%
										Laptop-
										95.46±2.09%
	Machado	Artificial	-		MOS from	jitter, delay,	-	-It was observed
	et al [138]	neural			Evalvid	throughput,		that the neural
		network			tool	packet loss		network for the
											scenario had a
											very	good
											prediction.
											-The influence of
											the	video
											dynamics and the
											amount of nodes
											is perceptible,
	Du et al	BP (Back-	-		PSNR,	Delay,	-	-No	subjective
	[139]	Propagation)			DMOS	Jitter,		test
		Neural			Video	Loss,		-visualizes	the
		Network			Quality	Burst,		relationship
						Evaluation	Bandwidth,		between	the
						System	Congestion		network technical
								Period,		parameters
								Disorder		and DMOS
								Packets
	Khan et al	Adaptive	-		PSNR to	Content	ANFIS-0.87	-It is possible to
	[141]	Neural			MOS	Type, video	Regression-0.86	predict the video
		Fuzzy			conversion	Sender		quality if the
		Inference					Bitrate, and		appropriate
		System					frame rate,		parameters	are
		(ANFIS) &					Block Error		chosen.
		non-linear					Rate, Mean		-The content type
		regression					Burst Length		has a	bigger

Table 9 : Rule weight to QoE mapping Rule weight Estimated QoE score

 9

	0	Imperceptible
	1-2	Perceptible but not annoying
	3-4	Slightly annoying
	5-6	Annoying
	7+	Very annoying

  IEEE 802.11n standard is a relatively new WLAN technology that can operate in both 2.4 GHz and 5 GHz band. It brings multiple technological enhancements to improve speed, reliability, and range. The increase in data rate is about 11 times as compared to the former IEEE 802.11g technology (i.e., 600 Mbps versus 54 Mbps). MIMO (Multiple Input Multiple Output)/SDM (Spatial Division Multiplexing) is a key feature introduced at the PHY layer.

	Additional technologies like OFDM, 40 MHz channel width operation, and short guard
	intervals have also contributed to the increase in physical data rate [88]. At the MAC layer,
	enhancements are made to the earlier IEEE 802.11e MAC, like frame aggregation and Block
	Acknowledgement (BA) to reduce the overhead, increase the useful throughput, and improve
	the overall efficiency.

Table 10 : Video characteristics

 10

	Type	Bit	Resolution Frame	Time
		rates		per	(Sec)
		(kbps)		second
	Movie	2549	640 x 360	24	15
	Animation	2508	640 x 360	24	15
	Sports	2686	640 x 360	24	15

Table 11 : Simulation parameters Parameters Values

 11

		Simulation Area	250 m X 250 m
		Simulation time	30 s
		Node mobility	Static
		Bandwidth	96 Mbps
		MAC layer	IEEE 802.11n
		Video packet size	1052 bytes
		CBR packet size	1000 bytes
		Time slot	20 s
		SIFS	10 s
		TXOP limit	3.264 ms
		Antenna Type	Omni Directional
		Network Interface Type	Wireless/Physical/MIMO
		Interface Queue Type	Aggregation queue
		Number of antennas	4
		Transmission protocol	UDP
			NS2 Environment with
			Evalvid
		Simulated wireless
		network with MAC level
		peturbations
		Source	Destination
		Node	Node
	Original video clip	Video traffic trace file	Received video traffic trace file	Received video clip

  Since the nature of the relationship between MAC-level parameters and video QoE is non-linear, RNN learning method is used in this thesis work. Moreover RNN is considered appropriate for the identification of QoE perceived parameters in multimedia applications[START_REF]qoe-rnn[END_REF]. Here, we considered four MAC-level parameters: Bit Error Rate (BER), frame aggregation, number of competing stations, and traffic load. However, it must be noted that our methodology can also incorporate additional parameters as well. Since the variation of these attributes impacts the quality of the delivered video and consequently the level of user satisfaction, our objective is to design and implement a method to estimate the variation of user satisfaction level as a function of these different MAC-level parameters, both individually and combined. Figure40, illustrates the video QoE estimation framework based on MAC level QoS parameters.

			Laboratory
			subjective test
			(QoS->QoE)
	Services	QoS parameters	Intelligent machine learning algorithms	QoE
	 Multimedia	 MAC layer
			 Random neural network

Table 12 : Video characteristics Type Bit rate (kbps) Resolution Frame per sec Duration (Sec)

 12

	Video clip 2508	640 x 360	24	15

Table 13 : Simulation parameters Parameters Values

 13 IntroductionOTT services refer to "Over The Top" services, that provide services over the Internet and are not regulated by traditional distributors (ISP or system operator). From a service provider perspective it is a normal Internet traffic and they are only responsible for transporting IP packets and has no right to control, view, copyrights, and/or make other redistribution of the content[98]. These are the third party content that operates on top of the Internet and are lower in cost than traditional services like cable television, IPTV, VOD, etc. Most the OTT applications that we find in Internet today are related to media and communication like

	Simulation Area	250 m X 250 m
	Simulation time	30 s
	Node mobility	Static
	Bandwidth	96 Mbps
	MAC layer	IEEE 802.11n
	Video packet size	1052 bytes
	CBR packet size	1000 bytes
	Time slot	20 s
	SIFS	10 s
	Antenna Type	Omni Directional
	Network Interface Type	Wireless/Physical/MIMO
	Interface Queue Type	Aggregation queue
	Number of antennas	4
	MAC Retransmission limit 7
	Queue Length	200 packets
	Transmission protocol	UDP
	Thereafter, for the subjective test, both the constructed and the original video clips were shown
	to the participants in a random order. The subjective test was performed in a closed room to
	reduce the effect of external noise and light. Participants rated each video clip according to the
	perceived impairment giving one of the following MOS scores as described in section 2.2.3.
	During the subjective test, on average, each participant rated 40 video clips. After rating 20
	video clips, they were allowed to take a pause of 5 minutes. A total of 25 users were registered
	for the test, which is considered reasonable for this kind of subjective tests [96]. The dataset
	collected from the subjective test was divided into training set and test set for training and
	validating the RNN respectively. Figure 41 illustrates the experimental environment for the
	subjective tests.

NowTV, Netflix, WhereverTV, Hulu, whatsapp, MyTV, etc. Moreover, OTT Video streaming shares the biggest part of Internet video demand

[156]

.

  They just leave the service and move to another ones [113]. So service providers and operators should not wait for user feedback for improving the service quality; instead, they should continuously monitor QoE and improve it as required. They should provide users with services that can offer high QoE values.

Table 18 : Thesis contributions Application QoS parameters Subjective test Machine Learning Algorithms

 18

	Validation

Acknowledgments

We will like to use web QoE as a metric for web service composition. We also plan to consider the contextual parameters for QoE estimation, as well as implement the solution in different web service frameworks.

where, "C ij = n" means that condition C j was preferred over C i n times.

Table 3 illustrates the comparison between various video subjective test methodologies.

Objective QoE estimation

Objective video quality estimation methods are based on media, service, and transmission parameters. These methods mostly rely on mathematical models that can be used when estimating the multimedia QoE based on some objective parameters. The lack of human presence increases error margins for the corresponding estimation. However, the estimation process can be automatic and thus can be performed fast enough. Therefore, service providers and network operators are mostly interested in a tool that can objectively reflect the subjective mean opinion score (MOS) of users with reasonable accuracy. Some of the important properties of RST which are used to classify and reduce data to important data and achieve CORE influencing factors are listed below.

Properties

Indiscernibility of objects:

If two objects are characterized by same information then they are indiscernible (similar). In RST it is defined as

Rough set Approximation:

With any rough set a pair of precise sets, called the lower and the upper approximation of the rough set, is associated. Let P  A is a set of conditional attributes and X  U is set of users, then

The lower approximation P * X consists of all objects which surely belong to the set and the upper approximation P * X contains all objects which possibly belong to the set. The difference between the upper and the lower approximation P* X-P * X constitutes the boundary region of the rough set [5].

Reduct and CORE:

A reduct is a minimal set of attributes discerning one object from all objects with a different decision. A reduct refers to a decision rule. Finding all reduct is an NP-complete problem.

Different algorithms like greedy algorithm and genetic algorithm are used to search for reduct. CORE is the intersection of all the reducts and is the set of indispensably important attributes.

Decision rules:

IF-THEN decision rules are generated by reading values for each attribute in the reduct. With every decision rule two conditional probabilities, called the accuracy (i.e., certainty) and the coverage coefficient, are associated. The accuracy coefficient expresses the conditional probability that an object belongs to the decision class specified by the decision rule, given it satisfies conditions of the rule. The coverage coefficient gives the conditional probability of reasons for a given decision [13]. We calculate support, accuracy and coverage of condition attributes from [13] corresponding to decision rules.

Support of a rule:

⁄ Accuracy of the rule:

Where,

( )

Coverage factor of decision rule:

Where,

( )

efficient QoE management systems aim at reacting before the user even notices the quality degradation [START_REF] Hobfeld | Challenges of QoE management for cloud applications[END_REF]. This requires an efficient feedback loop that can detect, locate and react in real time to degraded network conditions by controlling or reconfiguring different components of the QoE estimation system. Big advances have been made in this direction, however, open questions like when, where, and how to react still need to be addressed.

to the traditional single frame transmission. There are different aggregation types, mainly A-MPDU (Message Protocol Data Unit aggregation), A-MSDU (MAC Service Data Units aggregation), and two-level aggregation. In A-MSDU multiple MSDUs are aggregated and transmitted at once as a single MPDU. In A-MPDU multiple MPDU frames are aggregated into larger A-MPDU frames with only one physical header. The difference in A-MPDU frame aggregation is that aggregation is done after the MAC layer sends the MPDU frame [START_REF] Perahia | Next generation wireless LANs[END_REF]. In this work, we consider A-MPDU as our aggregation scheme as it performs better for videos [START_REF] Zheng | Video transmission over IEEE 802.11 n WLAN with adaptive aggregation scheme[END_REF]. The BA mechanism was introduced in IEEE 802.11e and has been modified in IEEE 802.11n to support the A-MPDU aggregation. With a single BA frame, multiple received frames can be acknowledged instead of a single ACK frame for individual data frames. When an A-MPDU with errors is detected, the receiver transmits a BA to selectively acknowledge the uncorrupted MPDUs. The sender is then able to re-transmit only the non-acknowledged MPDUs. This mechanism can only be used with the A-MPDU aggregation mode and is very efficient in error prone environments [START_REF] Paudel | QoS-HAN: QoS provisioning in home automated networks over[END_REF].

Most of the work in the area of video QoE estimation is from the network layer perspective.

They measure the impact of network parameters on quality of the video traffic, i.e., QoE [7] [74] [75] [START_REF] Issa | Estimation of time varying QoE for high definition IPTV distribution[END_REF]. However, few efforts have been done to analyze the effect of MAC-level parameters on video QoE. Moreover most of the existing works analyses the video QoE based on individual MAC-level parameters [91, 92, 93, 94, and 8].

In [START_REF] Wang | IEEE 802.11n MAC enhancement and performance evaluation[END_REF], authors have evaluated the performance of VOIP services in IEEE 802.11n WLAN environments. They considered IEEE 802.11n specific parameters and Bit Error rate (BER) for analyzing audio quality. However, we are using video traffic and a different set of MAC parameters. Authors in [START_REF] Barreira | Impact of the IEEE 802.11 n frame aggregation mechanisms on video streaming quality[END_REF] presented the impact of IEEE 802.11n frame aggregation mechanisms on video streaming. They used PSNR (Peak Signal-to-Noise Ratio) to represent the quality of video. Authors in [START_REF] Počta | Performance evaluation of video transmission over IEEE 802.11 b WLANs from the video quality perspective[END_REF] analyzed the impact of background traffic on video quality in IEEE 802.11b WLAN environment. In their work, video quality assessment was performed objectively with PSNR, Video Quality Metric (VQM), and Structural SIMilarity (SSIM). However we differ from [START_REF] Barreira | Impact of the IEEE 802.11 n frame aggregation mechanisms on video streaming quality[END_REF] and [START_REF] Počta | Performance evaluation of video transmission over IEEE 802.11 b WLANs from the video quality perspective[END_REF] since we used subjective evaluation as compared to their objective approach. The work presented in [START_REF] Suzuki | Qoe estimation from mac-level QoS in audiovideo transmission with ieee 802.11e edca[END_REF] estimated QoE of audiovideo services from MAC-level QoS based on IEEE 802.11e and used its associated parameters such as distance between the nodes and TXOP limit. Our work share some similarities with [START_REF] Suzuki | Qoe estimation from mac-level QoS in audiovideo transmission with ieee 802.11e edca[END_REF] but differs from them since we performed our experiments over IEEE 802.11n wireless network and considered different MAC-level parameters for evaluation.

increases with the value of aggregation and this is shown by lower MOS values. This is due to the loss of entire A-MPDU frame which is not the case with BA where only the corrupted sub-frame is lost. For the rest of the simulation we consider BA as enabled. Moreover, from Figure 34 we can see that for the case with BA, different levels of aggregation value follow similar trend of performance degradation with an increase in BER. This is because video data rate is not high enough to aggregate sufficient frames for higher aggregation values. However, to get the optimal result we used the highest values of aggregation (64000 bytes) for the rest of our performance evaluation.

Experimental results

For the experiments, we analyzed the combined effect of different MAC-level parameters on video QoE. Following the simulation process explained in Section 5.4.

Related Work

Based on the subjective test, there has been some work done on estimating the quality of adaptive video streaming. The work presented in [157], studied the optimum number of coding quality levels that could be used in adaptive video system in mobile video content.

Authors in [START_REF] Cranley | Incorporating user perception in adaptive video streaming systems[END_REF] studied the effects of frame rate and resolution on the user's perceived QoE.

In [159] authors studied the impact of quality variation on the users' QoE. Different scenarios such as rapid and gradual bit rate drop as well as oscillating the quality were considered as a use case. Furthermore in [START_REF] Yitong | A study on Quality of Experience for adaptive streaming service[END_REF], authors presented the impact of fluency, bitrate distribution, startup bitrate level, bitrate switching frequency, etc. on the users perceived QoE. Similarly, in our work we studied the impact of different adaptation scenarios (bit rate switching, freezing) on adaptive bitrate streaming but using a crowdsourcing based subjective test. We analyzed whether users prefers bit rate switching or freezing in different adaptive scenarios. A novel web based application for performing subjective test is developed for this purpose.

Methodology

The methodology for QoE assessment of OTT video services involves performing a crowdsourcing-based video subjective test. A novel web based application was developed for performing subjective tests which was embedded inside a crowdsourcing platform. In the subjective test, we used PC (Pair wise comparison) methodology where users provide score viewing two video sources and expressing their preference. The PC scores obtained were converted to mean opinion scores using Bradley-Terry model [START_REF] Bradley | Rank analysis of incomplete block designs: I. The method of paired comparisons[END_REF]. The framework for QoE estimation of OTT services is shown in Figure 47.

Services

Test Background

The videos for our subjective test are originally from the subjective lab experiment detailed in [START_REF] Tavakoli | Subjective quality assessment of an adaptive video streaming model[END_REF]. The original videos were all in 1280x720 resolution with a frame rate of 24 fps and encoded using the high profile for H.264/AVC at 4 different bitrates: 600 kbps, 1 Mbps, 3

Mbps, and 5 Mbps. Seven different sources were used, three sources were taken from entertainment movies and the rest of the content were from a soccer match, a sports documentary, a newscast, and a concert.

Several adaptation scenarios for the videos were produced in the original experiment, such as going from a high to a low bitrate in a stepwise manner as shown in Figure 48. In our subjective experiment we used the following scenarios from the original experiment: Gradual Freezing event for 2 seconds in the constant 1 Mbps video (1F1M). In total 9 different scenarios were used, resulting in a total of 63 stimuli. The preference score obtained from the users are processed for preference matrix. The preference matrix contains the probabilities i.e., each element of the rows in the preference matrices corresponds to the amount of times that a video i is preferred over a video j. The actual number of objects that are compared is nine however; they are not compared to all possible combinations but according to the optimized square design. Table 14 shows the preference matrix for soccer video source. For example we can see that video (1F1M) is preferred 5 times to video (1F3M). Where, represents the quality score for stimuli i and can take a value from 0 to 1. This expression can be reformulated using preference values, i.e., Here is the frequency of stimuli i being preferred over j. Now, the can be computed by maximizing the log-likelihood function, which is represented as (for our 63 stimuli):

∑ ∑ ( )

To solve this equation we used the optimization routine in a software package i.e. Matlab function OptiPt.m as mentioned in [START_REF] Wickelmaier | A matlab function to estimate choice model parameters from paired-comparison data[END_REF]. Lastly, the obtained estimations of the probability values have been normalized to the regular scale of 1-5 MOS.

Experimental Results

In order to validate the results obtained from the crowdsourcing experiment, we compared the opinion scores obtained from the lab experiment to the crowdsourcing experiment as shown in Figure 51. The results in Figure 52 shows that the opinion scores obtained from both the experiments are strongly correlated, though not to the degree one of would expect. This can be due to the differences in the test setup, such as evaluation method, viewing environment, and the introduction of new distortions. Our experiments verify the results from earlier studies, e.g., [START_REF] Mok | Measuring the quality of experience of http video streaming[END_REF], that buffering events has a high impact on the QoE. Due to this, users generally prefer viewing videos at lower bitrates than having buffering events in videos at higher bitrates.

services based on client constraints and QoS information gathered by the service providers at runtime. A new scheme for QoS-aware web services selection which exploits fuzzy logic to locate and select the right service based on the customer's preference or satisfaction degree is presented in [START_REF] Wang | A fuzzy model for selection of qos-aware web services[END_REF]. However, the works presented in [START_REF] Padovitz | Towards efficients election of web services[END_REF] and [START_REF] Wang | A fuzzy model for selection of qos-aware web services[END_REF] lack any experiments and validation of results. Authors in [START_REF] Keskes | Web services selection based on context ontology and quality of services[END_REF], present a web service selection methodology based on a context-based ontology and quality of service measurements. In [START_REF] Liu | Qos computation and policing in dynamic web service selection[END_REF], the authors The methodology that we propose relies on subjective tests. Subjective data are strongly influenced by the customer's feeling and experience. So, the correlation between QoS parameters and the participants' QoE remains imprecise, uncertain or ambiguous due to various human mental states and profiles, making the preferences over the criteria hard to quantify. The fuzzy approach [START_REF] Negnevitsky | Artificial Intelligence, A Guide to Intelligent System[END_REF] can deal with the consumers' imprecision by creating preference relations through the use of fuzzy sets and inference rules. An advantage of using fuzzy expert systems is that they are simple and computationally less intensive. Fuzzy expert systems are good at making decisions with imprecise information; however, they cannot automatically acquire the rules they require for making the decisions. Therefore, a fuzzyrough hybrid expert system is proposed where rough set theory is used to acquire the rules for the fuzzy expert system. Rough Set Theory is used for discovering patterns, rules and knowledge from the datasets as in [START_REF] Hua | Study on the application of rough sets theory in machine learning[END_REF] and [13]. Rough Set Theory has many advantages as, for instance, that it does not have information loss and it is flexible and extendable as compared with other data mining technologies [START_REF] Laghari | Quantitative and qualitative assessment of qoe for multimedia services in wireless environment[END_REF].

Methodology

In order to develop our web quality estimation technique, we followed a methodology that consists in conducting subjective tests with end user participants in order to build a learning set that correlates web QoS parameters with the subjective QoE. This correlation was then used to build the membership functions and inference rules of our fuzzy expert system for web QoE estimation. The methodology for designing web QoE estimation is shown in the Figure 55. 15 and Table 16.

Attribute values for QoS parameters have been processed through discretization arithmetic as shown in Table 16. Criteria for different QoS parameters in Table 16 where selected from the subjective dataset, where each QoS parameter were evaluated independently of other QoS parameters. The criteria's value represents the average value of all those particular subjective test cases.

QoE estimation of VoD services

We analyzed the impact of network conditions on the video QoE and proposed a QoE estimation system for VoD services. For this, we performed a lab based subjective test and used fuzzy expert system as a machine learning algorithm for mapping QoS to QoE. The QoE estimation system extracts the relevant QoS parameters i.e. packet loss ratio, packet loss burstiness and jitter from the network and estimates the video QoE. To validate our QoE estimation system we performed three sets of experiments.

-In the first set, we compared the results of subjective tests to the estimated scores from our proposed QoE estimation system. The comparison results were highly correlated with correlation coefficient of 0.95 and our estimated scores followed the subjective MOS scores.

-In the second set, we implemented our QoE estimation system in an industrial test bed and compared its results with VQM system. VQM considers media parameters for the QoE estimation while our estimation system only considers network parameters. Despite this difference, the results obtained from both systems where highly correlated proving that QoE can be effectively estimated by considering network level parameters.

-In the third set, we performed benchmarking analysis where we compared the results obtained from our MMT probe with the Indra's probe for QoE estimation in the real IPTV test bed. The results show that both probes behave in similar way and their MOS score are highly correlated with correlation index of 0.86.

These experiments validate our system and demonstrate the effectiveness of our QoE estimation system.

After considering network level parameters, we were interested in measuring the MAC level disruptions on the video delivery in 802.11n wireless network. An intensive lab based subjective tests were performed to correlated MAC layer parameters with video QoE. We considered different MAC layer parameters like aggregation, Bit error rate, Queue length, retransmission limit and number of competing stations and analyzed their effect on video QoE. The experiment results suggest that careful parameterization (tuning) of these MAC layer parameters can increase the overall QoE of video services. Moreover, we proposed a QoE estimation system based on random neural networks that learns the relationship between MAC layer parameters and video QoE from the subjective test results. We validated our QoE estimated system by comparing the subjective test results with the results from our proposed QoE estimation system, the results showed high correlation with correlation coefficient of 0.89.

In the above contributions, video QoE was estimated based on the network and MAC layer parameters. The main objective of this work was to provide a QoE estimation tool that can objectively predict the subjective mean opinion of users with reasonable accuracy from the network perspective. The proposed methodology provided higher accuracy, however; the coding and media parameters are equally important and needs to be considered. For example, if a media packet is lost, it is necessary to identify whether I, P or B frame is lost. Since, the impact of I frame loss is more important than P or B frame loss in visual quality.

QoE estimation of OTT services

We analyze the effect of different adaptive scenarios (freezing and bitrate switching) on adaptive bitrate video streaming for OTT video services. A crowdsourcing platform was used for performing the subjective tests. We developed a novel web based subjective test application that can be easily integrated with crowdsourcing platform to perform the test. We compared the results obtained from crowdsourcing test with those obtained from laboratory tests. The comparison results were correlated satisfactorily with correlation coefficient of 0.75, which demonstrates the efficiency of oou platform. Moreover, the subjective test revealed that users dislike buffering events and it seems that frequency is more important than the total duration of these events. However, if the bitrate is high enough and the frequency of the buffering events is low enough, then it seems to a viable alternative to decreasing the bitrate of the video or having a constant low bitrate.