Shafqat Ur Rehman

Thierry Turletti

Walid Dabbous

A Roadmap for Benchmarking in Wireless Networks

Keywords: wireless networks, wireless experiments, benchmarking methodology, wireless tools, repeatability, IEEE 802.11 wireless networks, wireless experiments, benchmarking methodology, wireless tools, repeatability, IEEE 802.11

Experimentation is evolving as a viable and realistic performance analysis approach in wireless networking research. Realism is provisioned by deploying real software (network stack, drivers, OS), and hardware (wireless cards, network equipment, etc.) in the actual physical environment. However, the experimenter is more likely to be dogged by tricky issues because of calibration problems and bugs in the software/hardware tools. This, coupled with diculty of dealing with multitude of hardware/software parameters and unpredictable characteristics of the wireless channel in the wild, poses signicant challenges in the way of experiment repeatability and reproducibility. Furthermore, experimentation has been impeded by the lack of standard denitions, measurement methodologies and full disclosure reports that are particularly important to understand the suitability of protocols and services to emerging wireless application scenarios. Lack of tools to manage large number experiment runs, deal with huge amount of measurement data and facilitate peer-veriable analysis further complicates the process. In this paper, we present a holistic view of benchmarking in wireless networks and formulate a procedure complemented by step-by-step case study to help drive future eorts on benchmarking in wireless network applications and protocols.

Introduction

Performance evaluation of wireless networking systems and protocols has recently witnessed tremendous research activity. Evaluation techniques employed range from mathematical modeling to in-eld experimental evaluation each with its own pros and cons. A survey of the wireless networking literature reveals that majority of articles embrace simulation as a convenient approach for the performance evaluation of such systems [START_REF] Kotz | Experimental evaluation of wireless simulation assumptions[END_REF]. However, lack of realism because of simplistic radio models in stochastic simulations can lead to misleading analysis [START_REF] Kotz | Experimental evaluation of wireless simulation assumptions[END_REF] [START_REF] Hamida | Impact of the physical layer modelling on the accuracy and scalability of Wireless Network Simulaltion[END_REF].

The best way to achieve realism is to perform in-eld experiments using 'real' hardware and software. Unfortunately, wireless experimentation is not a smooth process and congurability and management of even a small number of nodes is cumbersome. Moreover, behavior of the network is tightly coupled with the conditions on the physical layer and lack of control over these conditions poses a big challenge to comparative performance analysis. This is because wireless channels are unpredictable (random), error-prone and could vary over very short time scale (order of microseconds). It is also dicult to avoid collocated wireless networks. However, experimenter does have control over some network parameters which can be congured to xed values. Henceforth, all the congurable network parameters are considered to be controllable and the rest uncontrollable.

Uncontrollable parameters include station workload ( memory/cpu usage,etc.), network trac load, multipath fading, path loss (attenuation), channel interference, etc. Controllable parameters entail scenario congurations (topology, trac, wireless card congurations). Controllable parameters can also include meta-data such as system hardware/software specications.

It may seem trivial to keep track of all these parameters, but ensuring correctness and soundness of the measurement data can prove to be tricky. Some parameters and metrics concerning channel characteristics are inuenced by, among other things, calibration settings. For example, if the impact of power/rate adaptation, noise oor calibrations and interference is ignored, it can lead to misleading results [START_REF] Tala | Guidelines for the accuratedesign of empiricalstudiesin wirelessnetworks[END_REF] and hence compromise the trustability of the results. Experimental data sets, code, and software are crucial elements in scientic research; yet, these elements are noticeably absent when the research is recorded and preserved in the form of a scholarly article. Furthermore, most researchers do not deposit data related to their research article [START_REF]Executable Paper challenge[END_REF]. In [START_REF] Jourjon | A Portal to Support Rigorous Experimental Methodology in Networking Research[END_REF], authors conducted an informal study of 33 of last year's accepted articles from the prestigious ACM SIGCOMM 2010 conference. The study indicated problems with 70 % of the accepted papers related to proper description of methodology, experiments, and analysis. This makes it dicult for peers and reviewers to conrm the results and hence reduces trust in the ndings. This boils down to reproducibility of experiments. Reproducibility is the ability of an experiment to be replicated by independent reviewers on their own. It is not necessary for others to get exactly the same results. Usually, variation in the measurements is unavoidable. If the this variation is smaller than some agreed limit, the experiment would be deemed reproducible. Reproducibility is often confused with repeatability which is being able to do the same experiment over and over at the same site by same experimenters and get the same results ( i.e., with variation smaller than the agreed limit) each time. When it comes to simulation, repeatability is easily achievable. Reproducibility, however, may still require more care and eort. In any case, it may be sucient to maintain provenance (i.e., chronological record of measurements and analysis steps) of results ( gures, tables, graphs, etc.) with all the parameters, data, source code and scripts. However, in real-world wireless experiments, both repeatability and reproducibility are non-trivial and elusive.

Reproducibility has been at the core of research in most elds of science. An experimental result would be worthwhile only if it can be reproduced by peers. In natural sciences, an experiment is considered reproducible if the protocol/procedure used is described in sucient detail along with reagents, specications of the equipment, times, temperatures, etc. [START_REF] Manolescu | The repeatability experiment of SIGMOD 2008[END_REF]. However, networking experiments can't be reproduced by such measures because the distributed software is much more complex. Indeed, wireless experiments include additional complexities such as volatile radio spectrum, software/hardware imperfections/calibrations, congurability, management of resources and data, etc. [START_REF] Kiess | On Real-world Experiments With Wireless Multihop Networks[END_REF]. It is impossible to ensure same channel conditions until and unless the experiment setup is insulated from outside interference using shielded enclosure. That may be the reason why rigorous peer verication of experimental results is not becoming a culture in networking eld as yet [START_REF] Ott | Orbit Testbed Software Architecture: Supporting Experiments as a service[END_REF]. In this paper, we restrict ourselves to the repeatability of wireless experiments. However, the roadmap presented is valid for reproducibility as well.

Because of uncontrollable experiment conditions, it would be impractical to repeat a real-world (non-shielded) wireless experiment the way experiments are repeatable in other elds of science. We, therefore, focus on getting around this obstacle via conducting large number of runs, and clustering them according to the similarity of experiment conditions. Essentially, the entire procedure entails following steps: dene the scenario precisely, conduct large number of runs with xed and recorded controllable parameters, measure all the uncontrollable conditions (i.e., parameters/metrics), cluster the runs as per the conditions, and perform an apples to apples comparison. This would provide a level playing ground for the fair comparison of networking protocols. This will also lead researchers to archive and share data and code and hence enable future researchers to compare their experimental results with the previous ones [START_REF] Manolescu | The repeatability experiment of SIGMOD 2008[END_REF].

The above procedure requires a highly systematic and scientically rigorous experimentation approach which is often challenging due to the cost, complexity, and scale of the involved experimental resources, and some potential limitations in the training of the research investigators [START_REF] Jourjon | A Portal to Support Rigorous Experimental Methodology in Networking Research[END_REF]. Generally, an experimenter is required to deal with a host of issues such as testbed setup, installation of hardware/software tools, calibration and instrumentation of tools, sanity checks, imperfections and limitations of tools, scenario description, scheduling and management of experiment runs, meta-data collection, data cleaning, synchronization and merging of traces, data normalizations/transformations, analysis, reports, data management (measurement data, meta-data, code, assumptions, archiving, sharing, etc.).

In this paper, we provide a roadmap for scientically rigorous wireless experimentation in the wild. We promote the notion of wireless benchmarking which encompasses the aforementioned requirements and can provision a level playing ground for the realistic comparative evaluation of networking protocols.

More precisely, benchmarking signies fair apples to apples comparison of wireless systems relative to a reference evaluation. The seemingly simple task of benchmarking is surprisingly complex in wireless networks because it requires RR n°7707

an in-eld evaluation of the system to ensure real world operational conditions.

The complexity of such an evaluation is compounded by the lack of control on experimental conditions and lack of tools to deal with the issues listed above. The remainder of this paper is organized as follows. In section 2, we elaborate state of the art and provide critical analysis of the existing work. In section 3, we provide a brief overview of benchmarking and list benets and challenges for wireless networks. Section 4 provides detailed step-wise account of your proposed methodology for wireless experimentation and benchmarking. In section 5, we provide an experimental case study which implements all aspects of the benchmarking methodology. Section 6 concludes the paper.

Related Work

In this section, we focus primarily on experimentation methodologies that have been designed to make experimental results trustable by facilitating repeatability and peer-verication. Also, we look into other contributions such as guidelines, tools and data repositories aimed at benchmarking in wireless networks.

In [START_REF] Jourjon | Executable Paper Challenge: LabWiki: an Executable Paper Platform for Experiment-based Research[END_REF], after an extensive analysis of published work, authors conclude that lack of information about scenario, methodologies, and meta-data hampers reproducibility and peer verication. They have proposed a web portal called LabWiki where experimenters can describe the experiments and store all the information about an experiment. Each of the artifacts of an experiment and the experiments themselves are identied by public URLs which can be linked easily from any LabWiki pages. Authors propose to use the R language to analyze measurements collected on the portal. After the experiment development process is nished, a Portal user may choose to open up his or her LabWiki permissions to specic reviewers or the general public. However, the data repository is not implemented as yet and the portal is still in development process [66].

MyEmulab [START_REF]Emulab: The Utah Network Emulation Testbed[END_REF] is a web-portal to the Emulab network emulator testbed. It provides services to buid experimental network topologies, upload an experiment description, automatically congure and execute the experiment. Furthermore, it provides Wiki and versioning tools to allow collaboration between members of a given project. However, MyEmulab does not provide services to archive, access, analyze the measurements. Also it does not oer any services to report and share the results. NEPI (Network experiment programming interface) [START_REF]Network Experiment Programming Interface[END_REF] proposes a framework based on a unied object model to describe a networking experiment which could subsequently be executed on dierent environments (e.g. simulations, emulations, testbeds). However, it is an ongoing work and diers in terms of focus. Currently, it does not handle real-world wireless experiments and lacks the support of multiple runs, collection of meta-data, data management, analysis, etc.

In [4], a comprehensive practice has been recommended for 802.11 wireless experiments. It contains essential information for setting up test scenarios in dierent wireless environments and metrics that should be considered in each scenario. In [START_REF]Reporting IP Performance Metrics to Users[END_REF] [START_REF]Framework for Performance Metrics Development[END_REF], authors provide an insight into the selection of performance metrics for benchmarking. Selection of good metrics is necessary for producing useful benchmarks. However, these contributions are not complemented by a methodology.

In order to extend the value of measurements beyond a certain case study and make them accessible to broader research community, online network measurement data repositories are being deployed [START_REF]Internet Measurement Data Catalog[END_REF] [START_REF]A Community Resource for Archiving Wireless Data[END_REF]. The purpose of these repositories is to facilitate archiving, publishing and sharing of network measurement data. Data repositories have tremendous benet for benchmarking. They foster collaborative research. They make it easier for peers to revisit the analysis or calculate additional metrics without repeating the entire experimentation eort when existing traces can serve the purpose. Eort is also being made to develop supporting analysis tools [START_REF] Andersen | Challenges and opportunities in Internet data mining[END_REF] [52] [START_REF] Wipal | 11 traces manipulation software[END_REF]. However, these eorts are still in their infancy and are disconnected from experimentation testbeds and methodologies. The only exception is the expected data repository proposed in [START_REF] Jourjon | Executable Paper Challenge: LabWiki: an Executable Paper Platform for Experiment-based Research[END_REF] which is yet to materialize.

During the last decade, wireless experimentation has received signicant attention. Various shared as well non-shared testbeds have been setup, toolkits/frameworks have been developed and data repositories have been deployed to eciently deal with the complexities of wireless experimentation and facilitate analysis. Among notable testbeds and toolkits are Emulab [START_REF]Emulab: The Utah Network Emulation Testbed[END_REF], ORBIT (Open-Access Research Testbed for Next-Generation Wireless Networks) [START_REF] Ott | Orbit Testbed Software Architecture: Supporting Experiments as a service[END_REF], MiNT (a miniaturized mobile multi-hop wireless network testbed) [START_REF] De | Mint: a miniaturized network testbed for mobile wireless research[END_REF], Ad-hoc Protocol Evaluation (APE) testbed [START_REF] Lundgren | A large-scale testbed for reproducible Ad Hoc protocol evaluations[END_REF], EXC toolkit [START_REF] Kiess | On Real-world Experiments With Wireless Multihop Networks[END_REF], etc. Each of these platforms is tailored to meet the requirements of a specic area of focus. However, a survey of the contemporary experimentation testbeds and toolkits is beyond the scope of this paper. We, rather, focus more on enabling benchmarking in wireless networks by designing a methodology which could be carried out using existing testbed and tools.

We have formulated a rigorous benchmarking methodology [Section 4] for protocol evaluation in wireless networks and developed a supporting experimentation toolbox referred to as WEX (Wireless Experimentation) toolbox [START_REF]WEX Toolbox[END_REF] [Section 5]. Our methodology and the corresponding architecture facilitate workow management of large experimentation campaigns spanning over weeks or even months. An indexing mechanism makes easier parallel processing of all the traces and it is possible to replay an experiment. In addition, we provide data repository and a web portal aimed at making experimentation results shown in networking papers veriable by the reviewers. In the following sections, we provide an insight into wireless network benchmarking and details about the methodology.

Benchmarking Analysis

Benchmarking is a well known concept in many domains of computing and natural sciences. It is often applied to measure quantitative advantage of one system over another similar system. More precisely, benchmarking is evaluation of the performance of a system relative to a reference performance measure. It can be applied to virtually any system (business processes, progress reviews, software/hardware tools, protocols, etc.) which exhibits quantiable performance indicators. It is proven means of improving performance, eciency, cost-savings and competitiveness of a system [START_REF] Mann | Everything You need to know about benchmarking[END_REF]. It facilitates "learning from the experiences of others" and thus enables the identication, adaptation and deployment of protocols that produce the best results. However, the potential of benchmarking hasn't yet been utilized in networking and particulary in wireless networks.

The following subsections shed light on potential benets and challenges in the way of benchmarking in wireless experiments.

Importance

Identifying, improving and deploying superior communication standards and protocols adds to the business value of a network and benchmarking provides the foundations. Some of the reasons that benchmarking is becoming more and more promising are outlined as follows:

Benchmarks can be published and used by interested research groups who then can contribute with relevant metrics, models and test scenarios. They enable increased reproducibility of results and provide a common base for fair and consistent comparisons. Standardized workloads, run rules and benchmark tools can speed up the performance estimation and hence organization's ability to make improvements in a more ecient way. The use of dierent metrics, test scenarios and measurement methodologies complicates comparison. Benchmarks help overcome these problems and promote healthy competition.

Benchmarking helps identify areas of cost reduction, enables a more detailed examination of eciency and facilitates value-add. Benchmarks are also used to prepare proposals for product selection and system development. They can be employed to investigate how well an initial installation is performing. It is helpful in debugging a given conguration, determining where additional equipment needs to be installed and it can go a long way in providing most cost eective and functional installation in a given environment.

Basically, benchmarking is greatly useful in planning, testing and evaluating network performance. It is of great interest to engineering, marketing and executive level personnel. Sta can better prioritize which network problem needs to be addressed and "how good is good enough?"

Wireless Benchmarking Challenges

In wireless networks, radio propagation is highly dependent upon the physical environment including geometric relationship between the communicating nodes, movement of nodes, movement of objects and the type and orientation of the antennas used. Unlicensed ISM (Industrial, Scientic and Medical) bands of the spectrum available are being shared by an increasing number of dierent RR n°7707 devices making wireless medium more interference prone. Also, wireless networks are becoming more and more complicated. Modern APs can dynamically alter power levels and channel assignments in response to changing conditions. The rapid evolution in wireless technologies (e.g., introduction of smart antennas, directional antennas, recongurable radios, frequency agile radios, MIMO systems, multi-radio/multi-channel systems) makes benchmarking more complicated. Reproducibility is at the core of benchmarking but factors mentioned above coupled with volatile weather conditions, ever-shifting landscape of obstructions, network equipment aging, software/rmware bugs, etc. make network retesting, reconguration and hence reproducibility a big challenge [START_REF] Ganu | Addressing Repeatability in Wireless Experiments using ORBIT Testbed[END_REF].

Vagaries of the wireless medium have a sound impact on the performance of upper layers. Deciding what metrics to calculate and what parameters have direct or indirect impact on the low-level or high-level metrics is challenging. There is a need for tools that collect information at dierent layers and combine this information to allow a detailed and comprehensive analysis [START_REF] Kiess | On Real-world Experiments With Wireless Multihop Networks[END_REF]. Data collected can be fairly large. Depending on the number of ows, data rates, duration of the experiment, and number of probes; collected measurement data can run into hundreds of Giga bytes. Synchronizing, merging and managing wireless traces is time consuming. In order to do the analysis, one needs to combine them into one coherent time-ordered sequence. It is costly in terms of time, computational and storage resources.

There are up to fourteen channels on 802.11b/g worldwide out of which only 3 channels are non-overlapping. In most cases, density of wireless nodes and a small number of non-overlapping channels make it impossible to ensure innocuous co-existence of dierent WLANs. Increased channel interference leads to degradation in network performance. In order to investigate channel interference on network performance, spectrum analysis is indispensable. During the course of our experimentation, we employed Wi-Spy [START_REF]Wi-Spy 2.4x[END_REF] in conjunction with kismet spectools [START_REF]Kismet Spectrum Tools[END_REF] for spectrum analysis using standard laptop computers. Wi-Spy enables to capture the entire 2.4 GHz band but, with a sweep time of around 600 ms, it is signicantly slow.

The challenges identied herein are in no way exhaustive. Each wireless technology has its own specic open issues that need to be investigated. No panacea exists that can solve all of these problems. However, it is desirable to identify the strengths and weaknesses of existing solutions through well-formulated benchmarks that enable "apples to apples" comparison and enact a rm basis for future advancements. This will go a long way in alleviating the critical issues such as data rate enhancements, cost minimization, and user security in future wireless networks. Figure 1 outlines the set of activities envisioned for benchmarking in wireless networks. The rst step is to prepare Terms of Reference (TOR) or

Plan. Other steps involve research on state of the art, design and development of benchmarking tools, setting up network management and experiment control tools. Then there is a cycle of activities as shown by circulating arrows in Figure 1. The cycle should be repeated for initial smoke runs in order to establish the precision and logic of results prior to the running of actual benchmarks. The cycle may also be repeated in order to achieve a certain level of condence in the results. The activities include conguration of target test environment, performing the experiment, and undertaking measurements and data collection , analyzing and producing results, managing data-centric activities (such as storage, security, sharing, etc.) and preparing and publishing benchmark reports.

Plan:Terms of Reference

This step forms the basis of benchmarking. It sets goals and provides motivation for the undertaking. Type of the network (e.g., WiFi, WiMAX, Bluetooth, GPRS, IrDA etc), area of focus (e.g., wireless application such as peer-to-peer video streaming, content sharing), scope of measurements (i.e., set of metrics), and target deployment environment (indoor, outdoor, etc.) are key considera-tions. Priority should be given to the area that has greater value to the users or area that consumes larger resources. It has to be decided what should be the key indicators or metrics based on their importance and major aspects of cost. Terminology in the context of area of focus has to be dened so as to avoid confusing connotations. Planning for key benchmarking tasks such as network setup, cluster setup, benchmarking tools (e.g., trac generators, sniers, etc.), trace and meta-data collection, preprocessing, statistical analysis, reporting etc., has to be done. A set of deliverables which conform to the requirements, scope and constraints set out in planning has to be listed and elaborated. A documentation or data management system has to be developed. Terms of reference are subject to change during the process of benchmarking as a consequence of change in high level requirements or system artifacts that may become clear later on.

Investigate: Existing best practices

Research on the current state of benchmarking and evaluation paradigms across domains relevant to benchmarkee [START_REF] Camp | Benchmarking. The Search for Industry Best Practices That Lead to Superior Performance[END_REF] is constructive so that benchmarkers can bring them up to speed with the advances, avoid re-inventing the wheel, be able to make use of the existing knowledge-base of best practices and software tools, and start o from where peer benchmarkers have left. One needs to develop a comfortable understanding of the underlying wireless standards. It is imperative to investigate selection of metrics, run rules, baseline and peak congurations (if any), limitations, risks etc.

For instance if one were interested in improving hand os in wireless networks, she would try to identify other domains that also have hand o challenges. These could include air trac control, cell phone switching between towers etc. Typical aspects/metrics to consider would include (but not limited to) hando costs, eciency, delay, errors, and QoS. Benchmarking of handos is critical because depending on the application scenario, failed or wrong handos can result in enormous cost or have disastrous consequences.

Engineer: Benchmark tools

An ensemble of software tools (benchmarking toolbox) is at the core of any benchmarking endeavor. Before delving into the development of such tools, it is very useful to explore existing tools that serve a similar purpose. The golden rule here is to avoid re-inventing the wheel and re-use the existing code where possible in order to cut down the development cost. Benchmarking tools are desired to evolve as a result of bug-xes, functional enhancements, re-factoring, re-engineering, etc. An agile development approach would be suitable wherein some of the benchmark tools would be implemented or adopted based on their priority. Sometimes adjustments are required to account for new understanding gained through mistakes during the course of benchmarking.

For example, consider wireless mesh networks. Wireless meshes normally facilitate broadband applications with various QoS requirements. Suppose in order to test the mesh network's ability to carry large amounts of trac, say in video surveillance of a metropolis, capacity planning might take precedence over security planning. In this case, we can put benchmarking of security or other qualitative aspects on hold and concentrate on what is more important: throughput.

It is more productive to embed the functional testing or unit testing (in vitro in most cases) within the development process. Indeed, this allows rapid enhancements and re-factoring while ensuring that the core functionality remains intact. A system documentation is necessary to describe the functionality, limitations, direction for future enhancements, dependencies and installation guidelines for the deliverable tools.

Deploy: Resource and Experiment control

The pre-requisite for benchmark tool suite deployment is setting up a computer network in the target test environment such that it meets all the mandatory software and hardware requirements laid out in the test specication. Typical test environments include calibrated over the air test (COAT) environment, conducted test environment, over the air (OTA) outdoor Line of sight (LOS) environment, OTA indoor LOS environment, OTA indoor non-line of sight (NLOS) environment and OTA shielded enclosure environment [START_REF] Rfc | Benchmarking Methodology for Network Interconnect devices[END_REF].

Deployment involves setting in place the network equipment and installing the required software. Setting up of a computing cluster is also desirable in order to manage the execution of experimental tasks on the set of nodes participating in the experiment. It also empowers the benchmarker to perform multiple runs faster and eciently. It is very imperative to have the network equipment calibrated and all the benchmark software tested. It is good practice to use latest versions of rmware and drivers for all the wireless products. Products are normally shipped with default optimal settings. The decision whether to use baseline congurations or peak congurations or any other custom settings must be carefully considered but security settings might have to be adjusted anyway. Whatever settings are used must be carefully documented along with hardware models.

All of the required protocols will be congured and enabled during the setup. Parameters and settings associated with the devices and applications running thereon that aect the performance will have to be listed. Then, within this list, those parameters and settings that will vary during the experimentation have to be identied so that they can be included in the sampling process of network measurement. For example CPU usage, memory usage, swap usage, interference, etc.

We need to document all relevant congurations regarding devices (OS kernel, CPU, memory, etc.), tools (sniers, spectrum analyzers, etc.), network(security usage, (TX, RX) signal levels, RTSCTS usage, etc.), number of senders/recievers, etc. Key to successful benchmarking is holding as many parameters as possible constant in order to isolate the contribution of specic elements being compared.

Congure: Wireless experimentation scenario

Congurations elaborated in section 4.4 are general and are concerned with the network resources and experiment cluster setup. All of the benchmark tests would have to be run without changing the general conguration/setup of the devices in anyway other than that required for the specic test scenario. In this RR n°7707 step, all the tools necessary to carry out the tasks specied in the experimentation scenario have to be calibrated. Nodes participating in the experiment such as source(s), receiver(s), probes and spectrum analyzers have to be congured. This is usually repeated for each run of the experiment to ensure a clean start.

Experiment: Undertake experiment execution and data collection

Multiple independent applications, such as data and streaming media, should be run and behavior of the wireless network should be measured according to the test specications using a suitable sampling rate [START_REF]Framework for Performance Metrics Development[END_REF]. Applications should be representative of the real world situation and capable of associating with the wired/wireless interfaces of the devices and generating trac at the desired rate. Benchmarkee should be tested under dierent frame sizes especially max and min legitimate frame sizes and enough sizes in between to get a full characterization of its performance [START_REF] Rfc | Benchmarking Methodology for Network Interconnect devices[END_REF]. Workload tools of the benchmark toolbox are expected to produce normal workload as well as fault-load. Fault-load represents stressful conditions to emulate real faults that are experienced in the real systems. Appropriate level of detail about the workload is important in order to make meaningful analysis. Network load characteristics along with extraneous and internal RF interference should be measured. Network variations such as link failures and congestions have to be reported. Meta data about the result elements (such as trac samples, RF interference samples) and conguration elements (such as network settings and parameters) would aid in keeping track of the context in which experiment was performed. It is also important to structure the chain of steps between launch and termination of the experiment and maintain version control of the participating scripts. Employing visual tools to explore in-consistencies and changes in scenario denitions of the subsequent runs can result in big payos.

Performing network measurements is a complex process. Precision and accuracy of measurement devices and tools has to be documented. It must be clear to the benchmarkers as to whether they are measuring what they actually wish to measure. A general strategy would be to gather more than one type of data set -either from a dierent location in the network or from a dierent time [START_REF] Paxon | Strategies for sound internet measurement[END_REF]. Measurement data needs to be collected using open industry-standard formats. Collection of meta-data, even if its immediate benet is not obvious, may become useful in future benchmarking practice. It can be extremely helpful to seek out early peer review of proposed measurement eort.

Preprocess: Data cleansing, archiving and transformation

The data collected in the experiment execution stage needs to be cleansed. This could be achieved by employing self-consistency checks and investigating outliers and spikes. The rst question to ask would be if the experiment was performed all right. Validity and integrity of measured data has to be assessed. Traces collected using a snier may contain signicant amount of exogenous trac.

In order to reduce transformation and processing time, it may be desirable to lter out irrelevant data before transformations and analysis. We need tools to verify that measurements indeed provide a true picture of wireless network. One approach would be to create 802.11 nite state machines, look for inconsistencies and come up with an executive summary on the quality of measurements. If the measurements lack the desired level of integrity and validity, it would be required to repeat the experiment with better experience and improvements in the tools gained in previous measurement cycles.

Conducting benchmarking is costly in terms of time and resources. This, therefore, necessitates persistent structured storage of measured data using standard data formats such as XML [START_REF]Extensible Markup Language (XML[END_REF], XBRL [START_REF]Extensible business reporting language (XBRL)[END_REF] and database management systems (DBMS). One such example is CRAWDAD [START_REF]A Community Resource for Archiving Wireless Data at Dartmouth[END_REF]. Meta data (interference, variable resources, etc.) should also be associated with the traces.

Analyze: System peformance

Finally, data would be processed to produce the results which represent the values of metrics as specied in the test specications. For some metrics, data has to be transformed (normalized or converted to dierent scale) to t the analysis needs. Eort should be made to minimize the generation of intermediate data between raw measurements and nal results. Instead caches can be used for transient intermediate results. This would aid in reproducing the same analysis [START_REF] Paxon | Strategies for sound internet measurement[END_REF]. Calculation of mean (arithmetic or geometric) behavior over same measurements performed in dierent ways can provide a good insight of the network performance. Condence intervals and distributions can also be used to depict the network behavior.

The whole chain of analysis must be documented. Versioning and storage of analysis scripts along with the measured data that underpins the results should be stored. We need to archive both measurement traces and benchmark results. Benchmark results are either obtained through internal benchmarking eort or from partner research groups or organizations. Versioning mechanism has to be employed to facilitate reproducible analysis. 4.9 Report: Benchmarking score Reports are the windows through which benchmarkers [START_REF]Reporting IP Performance Metrics to Users[END_REF] can gain a visual access to the results. They provide detailed insight into the strengths and weaknesses of benchmarkee. All the benchmark-related information which is complimentary to the results must be made available. Meta-data (e.g., precision of tools, accuracy of measurements, etc.) which could be useful for trouble-shooting, decision-making, and management action, should also be reported. Reports should include an executive summary consisting of comprehensive graphs, congured parameters and drill-down details (if any). In fact, reports have to be designed and presented in accordance with the full disclosure report (FDR). Full disclosure report, for each benchmark, is prepared in accordance with reporting rules. Producing and interpreting benchmark results crosses into the realm between art and science. Web services are a great way to provide access to the database of benchmark results. Web services, then, can be used by interested organizations and groups to gain access to the results. Web services will also enable distributed access and sharing in the form of web reports. 4.10 Benchmarking methodology in a nutshell Figure 2 illustrates the ow of events in a typical benchmarking process. Each step of the process is annotated with tasks that should be accomplished before proceeding to the next step.

CASE STUDY

In this case study, we investigate all the steps presented in the above recommended practice for wireless benchmarking. We will restrict ourselves to the case of wireless channel characterization. Channel characterization enables researchers to investigate the inuence of environment on wireless network performance and allows them to understand how well a protocol or an application performs in dierent wireless environments. Benchmarking provides a whole new perspective to the analysis by facilitating the identication of performance or benchmarking gaps. It makes easier the diagnosis of performance issues and provisions a better understanding on how to close the benchmarking gap. It took us around one year to investigate the wireless experimentation issues and establish the proposed benchmarking methodology. The following material is a step by step account of the process. The material required to reproduce the results in the case study is available at [START_REF]Channel characterization using WEX Toolbox[END_REF].

Plan: Terms of Reference

In this step, we lay out the foundations for the undertaking by giving a clear direction, setting a stage and provisioning the means to carry out the required benchmarking activities with eciency. We select the wireless technology to be benchmarked, a small set of representative metrics (benchmark score), target deployment environment, resource requirements, set of tasks, output (nal deliverable) and risks involved as detailed in Table 1.

Investigate: Existing best practices

We conducted an extensive analysis of the state of the art in wireless/wired networks as well as other computing and non-computing elds. The purpose was, in part, to understand the notion and utility of benchmarking in various elds. We also contemplated benchmarking jargon, concepts, practices, application scenarios and obtained interesting insight into this important paradigm which has been often undervalued in networking research. Computational benchmarks such as NPB (NAS Parallel benchmarks) [START_REF]NAS Parallel Benchmarks[END_REF], Standard Performance Evaluation Corporation (SPEC) [START_REF]Standard Performance Evaluation Corporation (SPEC)[END_REF] benchmarks, Transaction processing performance council (TPC) [START_REF]Transaction Processing Performance council (TPC)[END_REF] benchmarks were looked into. Non-computational benchmarks especially those employed in the evaluation of business processes were also investigated. In fact, benchmarking forms a key component of business excellence models in today's businesses as a means to measure competitiveness. Examples are Global Benchmarking Network (GBN) [START_REF]Global Benchmarking Network (GBN[END_REF], European Foundation for Quality Management (EFQM) [57], etc. This investigation helped us polish our benchmarking terminology vis-a-vis ne-tune benchmarking methodology for wireless networks. We also investigated practices for wireless performance evaluation [4], development of metrics [START_REF]Framework for Performance Metrics Development[END_REF] [START_REF]Reporting IP Performance Metrics to Users[END_REF], experimentation platforms and RR n°7707 2 which could facilitate wireless experimentation and hence aid in benchmarking. In the end, we selected some of the existing tools to incorporate them in our experimentation platform which is demonstrated in [START_REF]WEX Toolbox[END_REF]. [START_REF]Framework for Performance Metrics Development[END_REF], Strategies for sound internet measurement [START_REF] Paxon | Strategies for sound internet measurement[END_REF], etc. Tools CrunchXML [START_REF] Bilel Ben Romdhanne | Ecient and Scalable Merging Algorithms for Wireless Traces[END_REF] , Wireshark/Tshark [START_REF]Wireshark[END_REF], WiPal [START_REF] Wipal | 11 traces manipulation software[END_REF], Netdude [START_REF]Network dump data displayer and Editor[END_REF], TCPDump [START_REF]TCPDump[END_REF], Iperf, Netperf [START_REF]Netperf: A network performance benchmark[END_REF], ttcp/nttcp/nuttcp [2527], Nettest [START_REF]Secure Network Testing and monitoring[END_REF], DDoS benchmarks [START_REF] Arikan | Attack Proling for DDoS Benchmarks[END_REF], MGEN, Kismet Spectrum tools [START_REF]Kismet Spectrum Tools[END_REF], GNU Radio toolkit [START_REF]GNU Radio[END_REF], etc. Platforms OMF [START_REF]The OMF Testbed Control, Measurement and Management Framework[END_REF], Emulab

Engineer: Benchmark tools

After stepping into the wireless experimentation arena, we have explored and put to test a number of tools in order to gauge their functional suitability, precision and correctness. It became clear to us that we need to develop some new tools, instrument/enhance existing ones and harness them all together to achieve sound wireless experimentation and the larger objective of benchmarking in the end. Some of the existing tools such as TCPDump [START_REF]TCPDump[END_REF], tshark/wireshark [START_REF]Wireshark[END_REF], sun grid engine (SGE) [START_REF]Sun Grid Engine[END_REF], etc., served the purpose well apart from calibrations and ne tunings. TShark/WireShark suer from performance degradation, in case of large trace les, which we hope to overcome by employing more ecient tools such as WiPal [START_REF] Wipal | 11 traces manipulation software[END_REF]. In addition, we needed functionality to perform basic sanity checks, large scale scheduling, trace management, manipulation of wireless headers, (meta-)data management and post-processing, reporting, etc. Therefore, we indulged in the development of our own platform. Also, we developed a packet injector for trac generation called Direct Packet Sender (DPS), based on BSD raw sockets [START_REF]Packet Injection and Sning using Raw Sockets[END_REF], in order to inject packets directly at the link layer. Given in Table 3 is a list of tools that we brought together to build our wireless experimentation platform, henceforth, known as Wireless Experimentation (WEX) Toolbox [START_REF]WEX Toolbox[END_REF]. Amongst the tools listed in Table 3, we modied MGEN, Madwi and kismet spectrum tools as follows:

MGEN [START_REF]Multi-Generator[END_REF] was modied to customize the packet format. We stripped o unwanted elds from the payload other than the sequence number and the timestamp. Madwi was instrumented to disable transmission of duplicate packets in case of packet injection. We customized the format of output from kismet spectrum tools, associated timestamps with the frequency samples and inserted a code snippet to archive spectrum information.

The congurations of tools in our deployment of the platform are discussed below.

Deploy: Resource and experiment control

The experimental LAN and cluster was setup in indoor environment. The deployment details are demonstrated in [START_REF]WEX Toolbox[END_REF].

Resource Control Setup

We setup a wired local area network (LAN) in order to manage experimentation cluster, experiment workow and data collection. All the stations are connected to each other through gigabit switches. MyPLC [43] is used for setting up and managing the LAN computers. Fedora 10 images are prepared using vserver [START_REF]vserver capable kernel[END_REF]. All the tools required on each are bundled into this image. The image is customized for each node to allow for network congurations. The specications of the network equipment and tools are shown in Tables 4 and5. Spectrum Analyzer Kismet Spectrum Tools [START_REF]Kismet Spectrum Tools[END_REF] Wireless Tools Wireless Tools for Linus version 29 [START_REF]Wireless Tools for Linux[END_REF], Compat Wireless 2.6.32 [START_REF]Stable compat-wireless releases[END_REF] RR n°7707

Experiment Control Setup

WEX Toolbox [START_REF]WEX Toolbox[END_REF] employs SGE (Sun Grid Engine) [START_REF]Sun Grid Engine[END_REF] in order to manage scheduling and execution of tasks on the experimental cluster. The cluster consists of master and execution nodes. The functionality of the cluster is divided into two parts, namely Control Network and Experimental Network. The entire cluster, in this scenario, consists of 7 Dell Latitude E6500 laptops, but it can be extended to a large number of computers (a few hundred) quite easily because we employ MyPLC [43] to manage the network resources. MyPLC employs virtual server [START_REF]vserver capable kernel[END_REF] based Fedora OS images which are installed with all the required software. Because of the centralized management, it saves the experimenter from the hassle of catering to the setup issues on individual machines. Tools employed by the cluster are described in [START_REF]WEX Toolbox[END_REF]. They are grouped under control network or experimental network. In our current deployment, control network consists of a master node, a database server and an analysis server, whereas experimental network consists of one access point, one source node, one receiver, two probes and one Wi-Spy based spectrum analyzer. The cluster can easily support groups of senders, receivers, probes, spectrum analyzers, access points, etc.

Control/Management Network

It provisions command and control interface for experimental network and enables remote congurations. Also it provisions a reliable mechanism to schedule tasks and collect data (traces and meta-data) according to the run rules in distributed computing environment. Master or server node is the brain of control network and is used to congure and manage rest of the nodes.

Experimental Network

All the nodes in experimental network are designated as execution nodes mainly because they run experimental tasks and applications as instructed by the scheduler daemon running on master node.

Conguration: The Wireless experiment scenario

Tasks in this activity may vary greatly from scenario to scenario. Therefore, the congurations laid out hereunder are specic to the scenario chosen for this case study. The focus is on capturing the characteristics of wireless medium in order to enable in depth analysis of wireless network performance under varying channel conditions. To that end, we use the packet injection technique to generate trac at the source node, capture trac over the selected channel using probes and monitor RF activity in the 2.4GHz band using Wi-Spy spectrum analyzer [START_REF]Wi-Spy 2.4x[END_REF]. Often, multiple runs of a wireless experiment for the same scenario are necessary. At the end of each run, data is transferred to the content/collection server. At the end of an experimentation session, data is preprocessed, analyzed and full disclosure reports are generated. Scenario congurations consist of relative placement of nodes, software/hardware congurations, wireless interface congurations, experimentation workow congurations, etc., as described below. All the nodes are placed on top of wooden tables with metal structures underneath. All of the stations are at 0.75 m height from the oor. The room is located at the top oor of a 3-storey building and is not RF isolated from the outside world. Actually, many APs are present at the dierent oors of the building, which makes possible to run experiments in a real working environment. As interferences are not controlled, it is crucial to be able to monitor the RF spectrum during the various experimentations.

Placement of nodes

Software Parameters

Sun Grid Engine version 6.2u2_5 [START_REF]Sun Grid Engine[END_REF] is used for scheduling experiment tasks. The scheduler is congured to periodically check the execution queues for pend-RR n°7707 Figure 3: Indoor experimentation setup and placement of nodes ing experimental tasks. Wireless tools for Linux [START_REF]Wireless Tools for Linux[END_REF] version 29 is used for interface congurations. Packet injection [START_REF]Packet Injection and Sning using Raw Sockets[END_REF] is used for trac generation. In order to harness MetaGeek's Wi-Spy 2.4x portable USB spectrum analyzer [START_REF]Wi-Spy 2.4x[END_REF].

Hardware Parameters

All the nodes have x86 based architecture with 2 dual core CPUs. Each node has a total physical memory of 3.5GB and total swap size of 1024.0M B. Wi-Spy 2.4x is congured to scan radio activity in the entire 2.4 GHz band. We use Atheros wireless card (GWL G650) with Madwi (Multimode Atheros driver for Wi-Fi on Linux) version 0.9.4 revision 4128 from the trunk.

Wireless Parameters

MAC and PHY revisions used by the driver are 2414 and 2413 respectively. Channel type is 11g (operates in 2.4 GHz frequency range). Channel 11 is selected (this tells nodes to lock to the channel frequency 2.452). Fragmentation, RTS and retries are turned o. Transmission (TX) power is xed at 18 dBm which is the maximum value for our Atheros wireless cards.

Reference Time duration

The total run time for an experiment is 345 seconds. Reference time duration for which results are calculated is 300 seconds.

Run Rules and Workow Congurations

An experiment is formulated as a set of tasks which are congured to be executed according to a nite state machine (FSM). The workow is as follows:

Wireless interfaces on the Source, Probe 44, and Probe 16 are congured 10 seconds after the launch of an experiment run. After waiting for another 15 seconds, TCPDump is launched on Probe 44 and Probe 16, and spectrum analyzer is launched on the Wi-Spy machine. Tcpdump and spectrum tools are scheduled for execution for total duration of 320 seconds each. After waiting for another 10 seconds, DPS is put into action for exactly 300 seconds. DPS is terminated 10 seconds before the termination of TCPDump and spectools. The timeline of the ow of events during an experiment is demonstrated in Figure 4. Traces obtained for the rst 10 and the last 10 seconds are discarded. The delays at the start and the end serve as grace periods. Long delays at the beginning are intended to allow the driver to reach steady state in terms of noise oor calibration and internal state. Also, there is an inter-run gap (i.e., pause between successive runs) when the experimentation session consists of multiple runs. The gap is set to 75 seconds.

Metrics

For this case study, we will measure RF interference, RSSI, Ricean K factor, Bit error rate, packet error rate and packet loss ratio at the probes.

5.6 Experiment: Undertake experiment execution and data collection

Launch experiment

A bootstrap python program, called primary scheduler, generates an initial schedule for all runs of the experiment. Input parameters of the primary schedule are desired number of runs and session start time. Initial schedule is a set of startup tasks, one for each run. A startup task encapsulates information such as start time of a run and link to the scenario denition les. Startup tasks are submitted to the grid engine right away. When an startup task gets executed by the grid engine, it generates a secondary schedule based on the scenario denition. Secondary schedule formulates the state machine of the run and governs the ow of tasks. These tasks specify each and every action to be performed on the target cluster nodes. Typical actions include scenario congurations, BSS setup, workload generation, trac sning, capturing spectrum information, etc. Each task is converted as a job and submitted to the grid engine. We employ a naming convention based on timestamp and node ID to identify each run.

Workload generation

Our packet injector DPS enables us to generate custom payload with dierent sizes, formats and trac distributions. This is important for the soundness of performance measurements. In this scenario, DPS is used to generate data packets with a payload of 100 bytes each. Packets are transmitted at the maximum rate possible. We set the link bandwidth to 1 Mbps by setting the physical bit rate of the wireless interface to 1 Mbps. This results in DPS transmitting at an eective rate of less than 1 Mbps. In order to be able calculate bit errors, we set bits in the payload to all 1's. First 8 bytes of the payload are reserved. Rest of the bytes is used for calculating bit errors per packet.

Trace capture

We use TCPDump to capture packet trace and spectrum analyzer's to capture RF trace.

Preprocessing: Data cleansing, archiving and transformation

We identify each trace by assigning it an identication tag based on the timestamp and node ID. At the end of each run, traces are collected at the server. However, preprocessing is deferred until the end of entire experimentation session. This makes it easier to manage the traces. We lter out unwanted extraneous packets to save space, speed up packet transfer to database and later on decrease analysis time. Extraneous packets are the ones originating from other wireless networks deployed in the vicinity of wireless experimental setup.

Traces are exported to an intermediate XML format which is then used to lter out relevant packet elds to MySQL database on a database server using CrunchXML [START_REF] Bilel Ben Romdhanne | Ecient and Scalable Merging Algorithms for Wireless Traces[END_REF].

Analyze: System performance

We implemented various scripts and programs to analyze packet traces. Analysis code is an ensemble of C++ programs, python and SQL scripts. An eort was made to avoid maintaining intermediate states and data. This means that analysis is performed on the actual data store each and every time. This practice facilitates reproducible analysis. In this section, we explain selected metrics and the mechanism to calculate each of them.

5.8.1 Channel interference and RF activity in 2.4 GHz band:

Because the radio spectrum used by wireless LAN is freely available for public and research use, it is usually highly congested. Interference can be caused by not only wireless networks but also by devices such as cordless phones, Bluetooth, microwave etc., using the same channel or channels adjacent to the selected communication channel. The purpose is to capture frequency uctuations in the entire wireless spectrum of either 2.4 GHz band (or at least adjacent channels) and study the impact of the level of interference on performance metrics such as BER/PER, packet loss, etc. Spectools [START_REF]Kismet Spectrum Tools[END_REF] is congured to log frequency uctuations for 2.4 GHz band. It collects information consisting of frequency range 2.400 to 2.483 at 419 points with a step size of 119 kHz. The rate at which it can capture samples depends on the processing time, called sweep time, for each sample. We have observed that it takes more than 500 ms to process one RF sample. The trace le is a sequence of tuples of the form time, frequency, amplitude. Using this trace le, one can plot a variety of graphs, e.g., frequency vs. amplitude, amplitude vs. frequency vs. time, frequency vs. running, average and peak amplitudes, etc.

Received Signal Strength Indicator (RSSI)

RSSI is a measure of power present in the RF signal. RSSI implementation varies from vendor to vendor. In madwi, RSSI is equivalent to signal-to-noise ratio (SNR) and essentially is a measure of signal power above the noise oor. It is calculated for each packet by subtracting noise power from the received signal power.

Ricean K Factor:

Ricean K Factor is one of several measures of wireless channel characterization. K factor completely denes Ricean distribution. The higher K is, the less signal fading is. Rayleigh distribution is a special case of Ricean distribution. When the direct LOS or dominant component between the transmitter and the receiver disappears, K approaches 0 and Ricean distribution degenerates to Rayleigh distribution. We estimate K factor from empirical data. We employ a moment based method to estimate K [START_REF] Abdi | On the estimation of the K parameter for the Rice fading distribution[END_REF]. K is obtained using the following equation

K = √ 1 -γ 1 - √ 1 -γ (1)
where

γ = V [R 2 ] /(E[R 2 ]) 2 , with V [.]
denoting the variance, E[.] denoting the expectation and R denoting the received signal envelope.

We developed both Matlab and SQL based scripts for estimating the K factor. Received power measurements are extracted from the received packets. Wireless interface measures the power in dBm which is a logarithmic scale. We convert the power measurements into Watts, normalize and then apply the formula 1.

Bit Error Rate (BER):

BER is a very important metric which shows the eciency and resilience of the entire communication path including transceivers and the communication channel. BER is calculated as the number of bits ipped in each packet having bad CRC ag. We consider only the payload bits and any bits ipped in the packet headers are not accounted for. All the metrics mentioned in Table 8 except channel interference are averaged over 5 runs ( corresponding to one session). In order to show the extent of variations around the average metric score, corresponding condence intervals are also computed for a condence level of 96% and are plotted for each score.

Bit Error Rate (BER):

As shown in Figure 7, bit error rate is more stable on Probe 16 and Probe 44 compared to probes 21 and 49 which are NLOS. Even between Probes 21 and 49, Probe 49 experiences higher bit errors the reason being Probe 49 is farther plus there is greater human activity in the room. 5.9.5 Packet Error Rate (PER): Figure 8 demonstrates average packet error rate as experienced by the probes. PER is more reliable metric than BER because it gives an accurate measurement of how many packets were corrupted during each experiment run. Same explanation as provided for Figure 8 in section 5.9.4 applies to PER in Figure 8.

Packet loss ratio:

Packet loss incurred at each probe demonstrated in Figure 9. Probes 49 and 21 experienced greater packet loss and more variations in packet loss compared to Probes 16 and 44. It is interesting to note that packet loss ratio is roughly 10 times greater than PER. This means that, in the real-world wireless, corrupted packets make only a small part of the lost packets.

Mon Tue Wed Thu Fri Mon Tue Wed Thu Fri Mon Tue Wed Thu Fri 10

Conclusion

Benchmarking is a very powerful tool for in-depth objective performance evaluation of wireless network, troubleshooting and management. Benchmarking However, the potential of benchmarking hasn't yet been realized to its fullest, the reason being inherent complexities in the wireless network and the test environments, lack of best practices and software tools. Over the last few years experimentation has evolved as the de-facto evaluation methodology for wireless networks. We have presented in-depth benchmarking methodology accompanied by practical demonstration of an entire benchmarking cycle. We hope that this will encourage the research community to improve/develop benchmark tools and foster a more collaborative approach by sharing and publishing benchmarks, full disclosure reports and practices. This will make it easier for benchmarkers to conduct large scale experiments with greater control, foster greater collaboration, provide an impetus for objective performance evaluation and hence, proliferate research activities and enhance the ability to innovate.

Figure 1 :

 1 Figure 1: Wireless Network Benchmarking

Figure 2 :

 2 Figure 2: An instance of wireless benchmarking process

Around 20

 20 nodes are positioned in 8 × 5 m room in a regular fashion as shown in Figure 3. The nodes used in the case study are Source (labeled in red), Server, Probe 16, Probe 21, Probe 44, Probe 49 and Wi-Spy. The relative distances between the nodes can be estimated from the room dimensions. Source, Probe 16, Probe 21, Probe 44, Probe 49 and Wi-Spy participate in the experiment.

Figure 4 :

 4 Figure 4: Timeline of events for each run

Figure 5 :

 5 Figure 5: Spectrum analysis: adjacent and co-channel interference

5. 9 . 2

 92 Channel interference and RF activity in 2.4 GHz band:The RF landscape in 2.4 GHz wireless band during the course of one experiment run is shown in the Figure5. The bandwidth of the 2.4 GHz band is 83 MHz i.e., [2400 MHz, 2483 MHz]. IEEE 802.11 divides the band into 14 channels, analogously to how radio and TV channels are sub-divided. All the channels are 22 MHz wide but spaced only 5 MHz apart. Spectrum information captured by Wi-Spy spectrum analyzer is in the form of frequency vs. amplitude. For graphical demonstration, the entire band [2400 -2483 MHz] was mapped to the corresponding 14 WiFi channels. 5.9.3 Ricean K Factor:

Figure 6 shows

 6 Figure 6 shows K factor values as measured on two LOS (line of Sight) probes named Probe 44 and Probe 16 and two NLOS (Non LOS) probes named Probe 21 and Probe 49. Large value of K signies less scattering and reections caused by surrounding objects/walls and hence smaller level of multipath fading which is the case for Probes 21, 44 and 49. Small value of K means greater depth of fading which is the case for Probe 16. Contrary to RSSI which has strong dependence on distance, multipath fading depends on location, orientation with respect to the obstructions in the environment.

Figure 6 :

 6 Figure 6: Average K-factor on each probe during each session. There are 4 sessions per day and each x-axis tick mark represents the rst session in the sequence.

Figure 7 :

 7 Figure 7: Average BER on each probe for each session. There are 4 sessions per day and each x-axis tick mark represents the rst session in the sequence.

Figure 8 :

 8 Figure 8: Average PER on each probe for each session. There are 4 sessions per day and each x-axis tick mark represents the rst session in the sequence.

  RR n°7707Mon Tue Wed Thu Fri Mon Tue Wed Thu Fri Mon Tue Wed Thu Fri 10

Figure 9 :

 9 Figure 9: Average Packet loss ratio for each session on each probe. There are 4 experiments (representing 4 sessions) each day. Each x-axis tick mark represents the rst experiment in the sequence.

Table 1 :

 1 Planning for wireless benchmarking

	Activity	Specications
	Type of network IEEE 802.11
	Area of focus	Channel characterization
	Benchmarkee	WiFi channel
	Metrics	Co-channel and adjacent channel interference, K-Factor (ricean
		fading model), received power, RSSI, packet error rate (PER) ,
		bit error rate (BER), packet loss
	Deployment en-	Over-the-air(OTA) line-of-sight(LOS) or OTA non line-of-
	vironment	sight(NLOS) non-shielded indoor environment
	Tasks	Network setup, cluster setup, experiment description (ED),
		scheduling, data collection, analysis, reporting, etc.
	Resources	Hardware (Computers with PC slot, Atheros Wireless cards, Spec-
		trum analyzers, High speed Ethernet switches, Database Server),
		Software(Madwi driver, Trac generators, snier, MySql DBMS,
		SUN Grid Engine (clustering software), Human ( Benchmark
		facilitators, Network Administrator, Software developer, Bench-
		marker)
	Cost	Cost of Resources
	Deliverables	Metrics, Benchmark score, full disclosure reports
	Risks	Bugs in drivers/sniers, limitation of spectrum analyzers, acqui-
		sition of resources
	tools as listed in Table

Table 2 :

 2 State of the art: Literature and Tool(suite)s

	Research	References
	Understanding the	Everything You need to know about benchmarking [1],
	benchmarkee	Draft Recommended Practice for the Evaluation of 802.11
		Wireless Performance [4], Framework for Performance Met-
		rics Development

Table 3

 3

			: WEX Toolbox
	Function		Tool
	Workload/Trac genera-	Direct Packet Sender (DPS) (New), Multi-Generator
	tors		or MGEN (Instrumented), Iperf
	WLAN device driver	Madwi (Instrumented)
	Spectrum analyzer	Kismet Spectrum Tools (Instrumented)
	Snier		TCPDump
	Packet Analyzer	Tshark, Wireshark
	Sanity checks	Unit test suite (New)
	Scheduler		SGE Scheduler, Scheduler support tools (New)
	Content / data Manage-	Data Cataloguer (New)
	ment (CM)
	Database schemas	DB schema manager (New)
	Database	Management	MySQL
	System (DBMS)
	Merge / Synchronization	CrunchXML(New)
	Extract, Transform and	ETL Scripts (new)
	Load(ETL)
	WEX Cluster	SGE 6.2u2_5

Table 4 :

 4 LAN setup (Hardware requirements)

	Hardware		Specications
	Computers		Dell Latitude E6500 laptops
	Switches		Linksys SRW2016 16-Port 10/100/1000 Gigabit Switch
	Ethernet Card		Intel® 82567LM Gigabit LAN card
	Wireless	Card	Intel® WiFi Link 5300 AGN
	(built-in)
	Wireless Card (Ex-	Atheros 5212 PCI card (For experimental wireless network)
	ternal)
	Spectrum Analyzer Wi-Spy 2.4x
	Processor		Two x86-based Intel core duo processors (@ 2.4 GHz)
	Physical memory	4 GB

Table 5 :

 5 LAN setup (software requirements)

	Software	Specications
	OS	Fedora 10 (Kernel 2.6.27.14)
	Wireless driver	MadWi 0.94 revision 4928
	Sniers	Tshark, tcpdump, wireshark
	Network le shar-	NFS 1.1.4-1
	ing
	Time synchroniza-	NTP 4.2.4p7 (ntp-sop.inria.fr)
	tion
	Network Manage-	MyPLC [43]
	ment

Table 6 :

 6 WEX cluster (server side)

	Tool		Description
	Scheduler		Congured to run every 8 seconds to schedule the execution
			of pending tasks
	NTP		Network Time Protocol to ensure time synchronization
	NFS server		Directories containing SGE binaries and experimentation
			scripts are shared on the cluster server
	MySQL	databse	Time sequenced unied repository for traces
	server
	Crunch XML	Export traces from intermediate XML format to database
			relations.
	Logs		Errors, warnings, information during the course of opera-
			tion of cluster.
	Jobs		Experimental tasks are translated to jobs which are sched-
			uled for execution on EN.
	Java		Java version 1.6.0_17, Java TM SE Runtime Environment
			(build 1.6.0_17 -b04)

Table 7 :

 7 WEX cluster (client side)

	Tool		Description
	SGE	Execution	Responsible for managing the execution of jobs on client
	daemon		nodes
	NTP		Network Time Protocol to ensure time synchronization
	NFS client	Shared directories are mounted

RR n°7707

http://www.inria.fr ISSN 0249-6399

Acknowledgements

This work was supported by European Community's Seventh Framework Program (FP7/2007-2013) under grant agreement no.224263.

Packet Error Rate (PER):

PER is the ratio of number of packets received with CRC errors to the total number of packets sent.

Packet Loss Ratio:

Packet loss is the number of packets that fail to reach the destination. In order to make it more discernible, we report Packet loss ratio which is the ratio of the number of packets lost to the total number of packets transmitted. Number of lost packets is determined from the sequence numbers of correct received packets.

Report: Benchmarking score

This section elaborates benchmarking score or the result set along with the meta data necessary to fully describe and possibly reproduce an experiment. Subsection 5.9.1 highlights full disclosure report (FDR). Subsequent subsection demonstrates the plots for metrics explained in section 5.8.

Reporting Rules:

Table 8 shows what could be included in the full disclosure report. The specic details about individual parameters are provided in aforementioned congurations. To give the reader an insight of how an eventual FDR might look like, we have prepared an FDR for one of the metrics, namely K factor, which is available at [START_REF]Channel characterization using WEX Toolbox[END_REF]. Similar reports would be desirable for all the metrics chosen in any wireless benchmarking undertaking. Ideally, they should be dynamically generated from a central trace repository such as CRAWDAD [START_REF]A Community Resource for Archiving Wireless Data[END_REF] in response