Malisa Vucinic
email: malisav@ac.me

Milica Pejanovic-Djurisic
email: milica@ac.me

Thomas Watteyne
email: thomas.watteyne@inria.fr

Mališa Vučinić

Milica Pejanović-Djurišić

SODA: 6TiSCH Open Data Action

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

I. INTRODUCTION

Industrial Internet of Things (IIoT) solutions require reliable, low-power and secure wireless networking. Today, this is achieved using industrial standards such as WirelessHART and ISA100.11a, both of which use the Time-Slotted Channel Hopping (TSCH) mode of IEEE802.15.4. In TSCH networks, nodes form a mesh, they are tightly synchronized, time is sliced into timeslots, and a communication schedule tells each node what to do in a timeslot: transmit, listen or sleep. Transmission frequency is constantly changing, resulting in frequency diversity, a technique called "channel hopping".

Tens of thousands of TSCH-based networks are deployed today 1 ; they yield wire-like reliability and over a decade of battery lifetime [START_REF] Watteyne | Industrial IEEE802. 15.4 e networks: Performance and trade-offs[END_REF]. The 6TiSCH standardization group within the Internet Engineering Task Force (IETF) 2 couples this industrial-grade performance with the ease of use of IPv6.

The 6TiSCH working group has been standardizing a number of open Internet standards (RFCs) since its creation in 2013. These standards define the missing pieces towards interoperable and autonomic operation of TSCH in IPv6 networks. Apart from IIoT, 6TiSCH also targets smart building, smart home, smart city and smart agriculture applications. 6TiSCH is fundamentally different from WirelessHART and ISA100.11a: communication schedule in 6TiSCH is built in a distributed manner. Each node in a 6TiSCH network, through the scheduling function it uses, decides locally what communication resources it needs. Because the networking paradigms they use are so different, the performance of 6TiSCH is not necessarily the same as the performance of WirelessHART and ISA100.11a.

Yet, companies want to know whether 6TiSCH meets their requirements. They ask questions such as Can 6TiSCH be used to build a long linear network along a road or a 200-meter long train? Does 6TiSCH offer low enough latency for home lightning or industrial control loops? How long does it take to build a 100-node network in a factory? Today, we do not have systematic answers to these questions.

At the same time, the academic community around lowpower wireless networking has launched an initiative to standardize how benchmarking is done, to minimize commonly biased results in academic papers [START_REF] Duquennoy | A benchmark for low-power wireless networking[END_REF]. Different research projects tackle the development and deployment of generic research tools that automate the experimentation 3 and testing [START_REF] Ziegler | F-Interop-online conformance, interoperability and performance tests for the IoT[END_REF]. That is why the 6TiSCH Open Data Action (SODA) project aims at serving as a bridge between the academic initiative and existing tools and testbeds.

The goal of the SODA project is to lower the barrier for wide adoption of 6TiSCH. SODA will provide a reference benchmark methodology and the datasets of standardscompliant 6TiSCH solutions, in industry-relevant test scenarios and conditions. SODA will automate the execution of these scenarios so benchmarking can remain up-to-date with changes to the standards, thereby facilitating comparison between older and newer proposals, resulting in "continuous delivery benchmarking". This paper gives an overview of the SODA project, and discusses the need for community involvement.

The remainder of this paper is organized as follows. Section II gives an overview of 6TiSCH, the state-of-the-practice and state-of-the-art. Section III introduces the SODA project and its expected outputs. Section IV discusses the community involvement that SODA is fostering. Finally, Section V concludes this paper.

II. 6TISCH: A PRIMER

The 6TiSCH protocol stack is rooted in IEEE802.15.4 hardware that enables low-power wireless communication. The state-of-the-art IEEE802.15.4 radio transceivers consume less than 5 mA 4,5 when transmitting or receiving. While low, this consumption allows a device to run off a pair of AA batteries for about a month, if the radio transceiver is always on. To extend this lifetime to a decade, the TSCH Medium Access Control (MAC) layer heavily duty-cycles the radio. The IEEE802.15.4 standard defines the operation of a node for the different timeslot types (transmit, listen, sleep) but it does not define how the schedule is built.

The 6TiSCH standardization group within the IETF fills this gap. Contrary to existing industrial solutions that use a central entity to calculate the network schedule, 6TiSCH relies on distributed scheduling. The 6TiSCH protocol stack is composed of 3 major components: 1) the 6top Protocol [START_REF] Wang | 6top Protocol (6P), Internet Engineering Task Force (IETF)[END_REF], allowing neighbor nodes to make changes to their schedule; 2) a scheduling function [START_REF] Chang | 6TiSCH Minimal Scheduling Function (MSF), Internet Engineering Task Force (IETF)[END_REF]- [START_REF] Anamalamudi | Scheduling Function One (SF1): hopby-hop Scheduling with RSVP-TE in 6tisch Networks[END_REF], an algorithm that drives the 6top Protocol; and 3) the secure join solution [START_REF] Vučinić | Minimal Security Framework for 6TiSCH, Internet Engineering Task Force (IETF)[END_REF], allowing new nodes to securely join the network and obtain keys used for encryption.

6TiSCH is supported, or support is actively being worked on, by all major open-source Internet of Things projects [START_REF] Watteyne | Industrial wireless IP-based cyber-physical systems[END_REF]: OpenWSN, Contiki, Contiki-NG, RIOT, TinyOS. Together with ETSI, the 6TiSCH standardization group has so far organized 4 interoperability test events, in which different entities meet and test their 6TiSCH implementations for interoperability.

The research community around 6TiSCH keeps up with the standardization work. Novel proposals have recently been published that relate to scheduling functions [START_REF] Duy | Distributed cell selection for scheduling function in 6tisch networks[END_REF]- [START_REF] Muraoka | Simple distributed scheduling with collision detection in TSCH networks[END_REF], join procedures [START_REF] Nahas | Network Bootstrapping and Leader Election in Low-power Wireless Networks[END_REF], [START_REF] Vučinić | Broadcasting strategies in 6TiSCH networks[END_REF], routing [START_REF] Righetti | Performance Evaluation the 6top Protocol and Analysis of its Interplay with Routing[END_REF], and co-existence [START_REF] Yaala | Cooperative resynchronization to improve the reliability of colocated IEEE 802.15. 4-TSCH networks in dense deployments[END_REF], to illustrate some. Different research projects also tackle aspects of 6TiSCH. For example, the H2020 F-Interop project6 develops tools for remote interoperability and conformance testing. F-Interop develops a "6TiSCH testing tool", which covers all aspects of the 6TiSCH stack. The H2020 ARMOUR project 7tackles security aspects of 6TiSCH, particularly of its secure join solution.

III. SODA OVERVIEW

The SODA project has three main objectives: 1) Capture industry use cases and requirements, and map them to test scenarios and Key Performance Indicators (KPIs). 2) Collect and analyze performance data of a standardscompliant 6TiSCH stack in these scenarios, under different conditions. 3) Automate data collection and processing to enable "continuous delivery benchmarking".

A. Benchmark

A SODA benchmark consists of 1) a test scenario, 2) a set of KPIs, 3) a test environment, and 4) a 6TiSCH implementation. In an ideal case, the latter would have little influence on the benchmark results, as one may expect the standards to determine the behavior of implementation(s). The practical factors such as implementation choices taken by different developers and stochastic propagation conditions necessarily add a level of indeterminism.

As part of the SODA project, we will focus on the Open-WSN stack [START_REF] Watteyne | OpenWSN: a standards-based low-power wireless development environment[END_REF], ETSI's reference implementation of 6TiSCH. We will put in place all the mechanisms for the community to complement our efforts by providing benchmarks using other available 6TiSCH implementations. We will facilitate this by fully automating the benchmarking process through different tools and a user interface (see III-E), exposing an API and using a standardized data format that a 6TiSCH implementation must implement in order to be integrated with our "continuous delivery benchmarking" tools. This will allow easy comparisons of different 6TiSCH implementations, and will highlight the effects of implementation choices and stochastic propagation conditions on performance.

B. Test Scenarios

The starting point for SODA is the definition of generic test scenarios. Under a "scenario", we want to emulate an industrial use case and typical operating conditions. We illustrate a preliminary list of parameters that each scenario encompasses in Table I. We aim at defining scenarios that correspond to typical "smart factory", "smart agriculture" and "smart city" deployments. For example, RFC5673 discusses the requirements of the industrial low-power wireless networks that 6TiSCH targets. It states that 90% of the market is covered by packet rates between 1/second and 1/hour, with the average under 1/minute. Similarly, we can derive typical coverage requirements (e.g. in maximum number of hops in the network) and interference levels.

Each of these scenarios is then "mapped" to a specific test environment (e.g. a testbed or the simulator). This corresponds to the precise definition of testbed nodes in use, the transmit radio power needed to meet the hop (i.e. coverage) requirement, and any other testbed-specific parameter needed to run the experiment.

C. KPIs and Data Collection

Industry-relevant KPIs for the evaluation of the wireless technology are typically end-to-end reliability, latency and radio duty cycle. As discussed by Duquennoy et al. [START_REF] Duquennoy | A benchmark for low-power wireless networking[END_REF], these are also commonly used by researchers for evaluating new proposals and comparisons with state-of-the-art. Another important KPI for a deployment is network formation time. Typically, this depends on the density of the deployment and the load of the bootstrap protocols.

With SODA, we will approach the data collection from a slightly different perspective than typically used by researchers. As we want to execute our scenarios on multiple testbeds, we aim at bypassing testbed-specific logging capabilities. Furthermore, we want our results to be easily comparable with real deployments where logging is not an option. For that reason, we plan on collecting all the data necessary to calculate high-level KPIs with over-the-air statistic reports, by having nodes in the network locally store and communicate in-band the necessary information. The nodes will periodically report to a central server, encoding the latest measurements. Obviously, with this approach, there is a risk that the reporting of a measurement (i.e. sending packets containing these reports) influences the system we are measuring (i.e. the network). We intend to study this further and find optimal values for the actual sampling period.

For example, to calculate latency, each application packet carries a timestamp of the instant when it was generated, which allows the server to calculate the delta. For reliability, the server needs to know the total number of application packets generated, which can be included in statistics reports, and compared to the actual number of received packets. Duty cycle in 6TiSCH networks is highly related to the communication schedule, so it is sufficient for a node to report how many cells it has installed with its neighbors. Network formation time can be computed by having the nodes report the timestamp of when they joined.

We will use the Concise Binary Object Representation (CBOR) format to encode these statistics in a flexible manner. Datasets which contain the statistics and application traffic will be published in an open-data approach.

D. Test Environments

As discussed in Section III, we will map the generic test scenarios to different test environments. We will map each scenario to 1) two testbeds with IEEE802.15.4-compliant devices, and 2) the 6TiSCH simulator. In the following, we give a brief overview of each of these test environments.

1) IoT-lab Testbed: Judging by the number of published works, IoT-lab 8 in France has so far proven to be the defacto testbed for 6TiSCH experimentation. Many researchers already have custom automation scripts and tools developed to run their experiments on IoT-lab and process results. For this reason, we believe that for SODA to be a success, it must provide reference results on IoT-lab. We have identified that the Saclay deployment site of IoT-lab provides the most realistic propagation conditions and is the one where OpenWSN implementation has been most thoroughly tested on. SODA will use the IoT-lab Saclay site and its A8 Open Nodes that integrate an M3 Open Node with STM32F microcontroller and Atmel's AT86RF231 radio. We are also aware, however, of a drawback of IoT-lab. M3 Open Nodes provided by IoTlab have a technical limitation that prevents 6TiSCH with default timings to be executed with link-layer security enabled. Namely, hardware acceleration module for encryption on M3 Open Nodes is placed within the radio chip and accessed through SPI, introducing large delays. Encryption of a 127byte frame with hardware acceleration takes approximately 5 ms, making it infeasible to run the default 10 ms TSCH slots with security enabled.

2) w-iLab.t Testbed: We will also use the w-iLab.t9 facility in Ghent, Belgium (see Fig. 1) to 1) study the effect of different propagation conditions on performance, and 2) overcome the discussed technical limitations of IoT-lab. More precisely, we will use the IoT-officelab deployment of w-iLab.t providing 120 nodes installed over three floors in a building. w-iLab.t provides 802.15.4-compliant Zolertia Re-Mote nodes, based on CC2538 System on Chip (SoC). This SoC supports 6TiSCH with default timings and link-layer security enabled. To illustrate, hardware-accelerated encryption of a 127-byte frame on CC2538 at 32 MHz takes less than 60 µs.

3) The 6TiSCH Simulator: While testbeds provide results close to real deployments, they may not be the best suited tool for quickly testing an idea, due to the inherent overhead of firmware development and remote debugging. The 6TiSCH simulator 10 was developed for precisely this purpose: to allow a quick performance estimation for a given configuration and a set of conditions [START_REF] Municio | Simulating 6TiSCH Networks[END_REF]. We are currently extending this simulator to support reproduction of connectivity traces obtained from real deployments [START_REF] Brun-Laguna | Moving Beyond Testbeds? Lessons (We) Learned about Connectivity[END_REF]. This will allow us to run a given scenario on the exact same conditions as present in a real-world deployment or a testbed, and look for unexpected performance anomalies.

E. Experimentation Tools

Apart from the scientific component on studying network performance in different conditions and looking for anomalies, the SODA project will also provide different tools that will enable "continuous delivery benchmarking" on defined scenarios and test environments.

As discussed by Duquennoy et al. [START_REF] Duquennoy | A benchmark for low-power wireless networking[END_REF], a user should ideally select a scenario, upload/select a firmware image, and click "play". SODA will achieve this level of automation through the jFed 11 tool provided by the Fed4Fire+ project that allows a federated access and experiment automation. We will also develop a web-based solution enabling a scenario to be executed on each of the considered testbeds, and the 6TiSCH simulator. We will automate the data collection and processing, and automate the graph generation through tools such as Grafana.

On the firmware side, SODA will develop an open-source statistics collection module for OpenWSN, the reference 6TiSCH implementation, that will enable data collection on a generic testbed, as well as in real deployments.

F. Users

SODA takes a challenging approach by targeting the industry, the standardization bodies and the academic community.

On the one hand, industry stakeholders are typically interested in the performance of standards-compliant solutions; so are the standardization bodies that produce the actual specifications. For this reason, SODA will provide a benchmark of the OpenWSN implementation in different settings covering the base 6TiSCH specs without any additional optimizations. This output will serve the industry to asses whether 6TiSCH fits their requirements, as well as the standardization bodies on whether the produced specifications meet the expectations.

On the other hand, the research community is typically interested in the optimizations that result in performance improvements. The benchmarks provided by SODA will therefore serve to the community to assess the impact of these optimizations.

Through the tools SODA will develop, we will enable the "continuous delivery benchmarking", allowing the research community and industry stakeholders to quickly assess the impact of specific optimizations. Our tools will allow the user to immediately share the results or keep them proprietary. All of this comes with an added benefit of using a standardized benchmarking methodology, something that has been missing in our research community [START_REF] Duquennoy | A benchmark for low-power wireless networking[END_REF].

IV. COMMUNITY INVOLVEMENT

Our goal is for the outputs of SODA to be used by the research community; we are looking for feedback on several topics.

First, the main requirement for scenario definitions is that they are relevant to the industry. Are we missing some important parameter to describe a scenario? Capturing multiple potential use cases and typical deployment requirements within a single parameter set is a tempting task. As discussed in Section III-B, the preliminary idea is to survey the existing smart factory, smart agriculture and smart city deployments, and to generalize each with a single parameter set. Can the same goal be achieved with a different approach?

Second, we are looking for KPIs that are interesting for future research. What information other than that discussed in Section III-C should we collect within the firmware statistics module in order to calculate research-relevant KPIs?

Third, we will automate the experimentation on two testbeds (IoT-lab and w-iLab.t) and on the 6TiSCH simulator, also providing an easy-to-use web interface for launching experiments and processing results. The community could develop support for additional IEEE802.15.4 testbeds and/or 6TiSCH implementations, and integrate it with the interface.

Lastly, we are looking for input on the debugging process and the evolution of the tools with future technological developments. How to best capture the firmware development and corresponding network debugging within a generic graphical interface? How to ensure that the benchmark and the tools are up-to-date with the state of 6TiSCH standards and trending research topics?

V. CONCLUSION 6TiSCH Open Data Action (SODA) will provide a reference benchmark and open datasets of a standards-compliant 6TiSCH solution, in different conditions and test environments. SODA will enable "continuous delivery benchmarking" through experiment and data processing automation. We are looking for community involvement both on making the results and tools relevant for the industry and the academic community, and on extending the support of our tools to additional test environments and 6TiSCH implementations.

Fig. 1 .

 1 Fig. 1. An illustration of w-iLab.t facility (reproduced from the w-iLab.t website).

One vendor alone, Emerson, reports over 36,300 networks deployed and over 10 billion operating hours: http://www.emerson.com/en-us/ expertise/automation/industrial-internet-things/pervasive-sensing-solutions/ wireless-technology.

https://datatracker.ietf.org/wg/6tisch/charter/

www.fed4fire.eu

www.linear.com/docs/41870

www.st.com/en/microcontrollers/stm32wb-series.html

www.f-interop.eu

www.armour-project.euThe challenge of Objective 1 is to define scenarios that are relevant for the industry, and at the same time enable future experimentation. We need KPIs that provide confidence to industry stakeholders for adopting 6TiSCH for their IoT products, yet facilitate future developments and comparisons. We discuss these aspects in detail in Sections III-B and III-C.We define generic test scenarios to be able to provide a common ground to technology developers for future experimentation. We will provide a mapping of these scenarios to specific testbeds, as well as to the 6TiSCH simulator, a Python-based tool that answers what-if questions through quick performance estimation[START_REF] Municio | Simulating 6TiSCH Networks[END_REF]. We will collect real-world data by executing these scenarios on different testbeds. Similarly, we will collect simulation data using different propagation models, including real-world traces[START_REF] Brun-Laguna | Moving Beyond Testbeds? Lessons (We) Learned about Connectivity[END_REF] reproduced by the 6TiSCH simulator. This approach allows immediate comparison between realworld results and simulations, the isolation of the effects of propagation conditions and models on different KPIs, and experimentation in industry-relevant conditions. We discuss this in more details in Section III-D.To meet Objective 3, we will complement the existing development projects and provide tools that will automate the benchmarking process on the testbeds and in the 6TiSCH simulator. We discuss these tools in Section III-E.

https://www.iot-lab.info

http://doc.ilabt.iminds.be/ilabt-documentation/wilabfacility.html

https://bitbucket.org/6tisch/simulator

https://www.fed4fire.eu/tools/jfed/


Carlo Alberto Boano

Simon Duquennoy

Anna Förster

Omprakash Gnawali

Romain Jacob

Hyung-Sin Kim

Olaf Landsiedel

Ramona Marfievici

Luca Mottola

Gian Pietro Picco

Xavier Vilajosana

Thomas Watteyne

www.iotbench.ethz Marco Zimmerling

Ch

H.-S Kim

IoTBench: Towards a Benchmark for Low-power Wireless Networking

Unlike other fields of computing and communications, low-power wireless networking is plagued by one major issue: the absence of a well-defined, agreed-upon yardstick to compare the performance of systems, namely, a benchmark. We argue that this situation may eventually represent a hampering factor for a technology expected to be key in the Internet of Things (IoT) and Cyber-physical Systems (CPS). This paper describes a recent initiative to remedy this situation, seeking to enlarge the participation from the community.

I. INTRODUCTION

Low-power wireless communication is a key enabling technology for the Internet of Things (IoT) and Cyber-physical Systems (CPS). Network stacks supporting multi-hop communications have received particular attention, as they bring flexibility in the physical deployment of devices. The last decade has seen tremendous improvement in the performance of these network stacks: near-perfect reliability is now a reality, and can be achieved with extremely low latency and energy consumption. Problem: Lack of confidence in results.. Despite the performance claims in several academic papers and commercial system, even in the many cases where these claims are substantiated by rigorous and realistic evaluations, the extent to which the results for one system hold in the setup of another is still unclear at best.

A simple example about periodic data collection, arguably the most-studied traffic pattern, illustrates the problem. Fig. 1 shows manually-gathered statistics from 32 papers published in flagship conferences in recent years (ACM SenSys, IEEE/ACM IPSN, INFOCOM, MASS, DCOSS and EWSN 2010 to 2015). The chart plots two simple parameters known to have significant impact on system performance: the number of nodes in the tested network and the inter-packet interval (IPI). It is easy to see that different setups cover different areas of the chart. Do the results reported for a given combination hold for another? In practice, the divergence across experimental setups is even higher, as the application and environmental parameters affecting performance are many, and in many cases include the configuration of the protocols under consideration. This situation is not just methodologically flawed from a scientific standpoint; it has also distorting effects on industrial practice and its connection with research. For instance, one of the reasons companies are reportedly reluctant to embrace multi-hop low-power wireless due to the perception that it is too unreliable, or otherwise not performant. Many research papers and even commercial systems claim otherwise; however, due to the aforementioned fragmentation of experimental setups, it is hard to overcome this perception. Solution. To overcome this status quo, we need a common, agreed-upon yardstick against which the performance of systems can be measured-in other words, a benchmark.

Benchmarks have been designed and routinely used in other communities, including database systems [START_REF]Database Benchmarks[END_REF], robotics [START_REF]YCB Benchmarks[END_REF], and image processing [START_REF]Computer Vision Datasets[END_REF], to compare research and commercial products. A benchmark typically defines input parameters representative of several application cases, output metrics quantifying the performance of the benchmarked systems, and possibly the required experimental setup. For instance, a set of representative datasets (e.g., data or image) may be provided as input, and the speed or accuracy in performing some function (e.g., a specific database query or image recognition) may be measured as output.

Therefore, there is a lot of previous expertise one can draw upon to define a benchmark for low-power wireless networking. Nevertheless, what distinguishes this field from, say, image processing, is the fact that system performance is affected by real-world dynamics, such as the vagaries of low-power wireless links. The number and density of nodes defining the network topology matters, but also the characteristics of the environment; factors like indoor vs. outdoor scenarios, the presence of interference, or variation in physical parameters like temperature and humidity, may greatly affect connectivity and system performance. Fields like robotics and control face similar challenges. As a consequence, the experimental environment plays a central role in the definition of a benchmark for low-power wireless networking. Goals. The overarching goal of our benchmarking effort is to bring a standard and consistent way to test and compare low-power wireless networking protocols. We consider main stakeholders for our benchmark design: i) researchers who want to evaluate new protocols against older ones or baselines, ii) companies to showcase the advantage of their protocols, and iii) customers to help decide objectively strengths of protocols. A set of common goals and trade-offs exist that reasonably satisfies the needs of three stakeholders:

• The benchmark should be reproducible across experimental settings. To achieve this goal, a benchmark should contain sufficient context information to recreate the experiments.

• The benchmark should be simple and minimalist. This goal ensures that a benchmark is applicable as broadly as possible. • The benchmark should suggest sufficient and appropriate experiments and metrics to improve the state of experimentation. Thus, benchmarking is also an opportunity to educate the community on best practices. • The benchmark should allow for consistent comparisons of low power wireless protocols across key metrics.

Contribution and road-map. In this paper, we outline the challenges that must be addressed to realize the vision above. We begin by discussing the approach ( §II) we follow in designing this benchmark. This starts from an implementationagnostic definition of application profiles and performance metrics ( §III) and aims to ultimately achieve the implementation ( §IV) of an experimental setup enabling the automated and reproducible execution of the benchmark. Finally, we discuss the current status of our initial efforts ( §V) along with some key elements we expect to determine the practical adoption of the benchmark; these are the elements based on which we conclude the paper with an outlook ( §VI) towards future activities.

II. DESIGN SPACE AND APPROACH

In designing our benchmark, we face conflicting requirements. On one hand, as illustrated in Fig. 2, we would like the benchmark to be as general and representative as possible, i.e., encompassing a wide array of application use cases, technical approaches, and experimental setups. On the other hand, the desire to enable accurate, fair, and repeatable comparison pushes towards a very detailed and prescriptive experimental setup, possibly yielding to the (undesirable) extreme of forcing Fig. 2. The design space of protocol comparison. Our approach aims for increasing comparison accuracy while preserving generality and usability of the benchmark.

the comparison to take place in exactly the same experimental conditions, e.g., the very same testbed. Question is how to reconcile these conflicting goals.

The first step is the definition of parameters, metrics, and profiles. As further detailed in §III, together these effectively provide a specification of the input provided to an experiment, the outputs measured, and the conditions observed during its execution, analogously to the benchmark definitions in other fields. At this level of abstraction, metrics and profiles provide the common conceptual framework that constitutes the minimal foundation for a benchmark. Indeed, without a common definition of these aspects, the whole notion of comparability is at stake. Consequently, the agreed-upon definition of metrics and profiles can also be immediately useful as a way to i) guide system evaluation by clearly identifying what portion of the configuration space is being covered and, as a consequence ii) qualify the results obtained, by separating the conditions in which they hold from those for which nothing can be said.

In their more abstract incarnation, parameters, metrics and profiles are technology-agnostic; the specific radio technology can be regarded as one of the attributes of the profile describing an experiment. Further, they are also implementationagnostic, which means that they are applicable to different evaluation methods, e.g., simulation vs. testbed experiments. The level of detail included in the definition of profiles defines the trade-offs between generality and reproducibility, e.g., by progressively including in the profiles the information necessary to reproduce the environmental conditions.

Specifying experimental parameters, however, may be too weak without a way to enforce them, and at the same time pose too much burden on the researcher performing the experiments. For this reason, our aim is also to define, design, and implement a common experimental infrastructure supporting the execution of experiments based on our benchmark suite. In this vision, an entire toolchain is available to researchers and practitioners, which simplifies and automates the selection, deployment, and execution of benchmarks, along with their sharing and dissemination to the community. Several challenges are associated with this goal. The technical ones are discussed in detail in §IV, both for simulation and testbed experiments. However, a more subtle challenge is to prevent a readily available infrastructure from effectively "locking" a research community to a common hardware/software platform, effectively stifling innovation, as we further discuss in §VI. In this respect, however, the approach illustrated in Fig. 2 also helps, as in principle it enables the definition of different degrees of compliance and reproducibility to a benchmark suite, namely, ranging from the use of the sole specification to the use of the same experimental infrastructure.

III. IOTBENCH PARAMETERS, METRICS AND PROFILES

We seek to provide a well-defined structure to IoTBench, a benchmark for IoT, while not limiting the scope of its applicability. Parameters and metrics. We define key classes of parameters and metrics that we consider in the definition of IoTBench. A metric is a measurable quantity representative of some specific facet of a benchmark, but not necessarily originating from a deterministic process.

1) Input parameters: are quantities that users feed as input to a certain run of a benchmark. They are completely under the user's control and should, in principle, be independent among themselves and of the protocol involved in a run of a benchmark. Examples are the number of nodes involved in a run, or the traffic load. 2) Output metrics: are quantities that users obtain as the result of running a benchmark. They are completely determined by the behavior of a given protocol given a fixed set of input metrics, and thus out of the user's control. Examples are the packet delivery rate or energy consumption of a protocol. 3) Observed metrics: are quantities that users also obtain as the result of a run of a benchmark, typically to characterize the environment when a benchmark was run. They are indeed necessary to enable fair comparison between different protocols. Examples are the level of external interference or people's presence during a run.

It is therefore useful to map metrics commonly found in experimental studies of protocols to the three classes above. These, in turn, may be grouped as:

• System parameters: such as network topology, link qualities, and mobility patterns. Depending on the nature of the metric, these may be either input to a run of a benchmark, or observed. For example, network topologies are usually input to a run of a benchmark, at least to the extent possible in a given environment. Link qualities, however, do not enjoy direct control and thus may only be observed during a run, to check whether different runs are comparable in terms of wireless dynamics. • Traffic parameters: such as traffic load, traffic pattern, and number of sources. These metrics should always be input to a run. Most importantly, a necessary condition for a fair comparison between different protocols is that their behavior is determined by the exact same set of input parameters. Differences in these sets may already question the soundness of the comparison. • External metrics, such as external interference and people's presence are eminently observed. Similar to the observation above for traffic parameters, a necessary condition for a fair comparison between different protocols is that these observed metrics are comparable across runs of different protocols.

• Performance metrics, such as packet delivery or latency, and energy consumption are eminently output as they typically represent the results of a run of a benchmark that users are interested in. They represent the quantitative basis to compare different protocols, and must solely be determined by a protocol's behavior given certain assignments to all input parameters. • Experiment parameters such as the number of repetitions and the acceptable standard deviation of key performance metrics are also typically input. Rather than directly impacting the output metrics, they most often determine the scope of applicability of the output metrics; for example, by identifying the conditions to exclude certain results from the analysis. The metric classification we describe is instrumental in creating combinations of metrics that uniquely characterize a certain run of a benchmark. Profiles. A profile is a possibly partial assignment of concrete values to input parameters, and a precise definition of observed and output metrics to be measured. In other words, a profile is a set of points in the n-dimensional space determined by n input parameters.

A profile should ideally map to a concrete application or domain. For example, we may define a low-rate data collection profile representing applications such as environmental monitoring, with the following structure:

• Input parameters: #nodes: 100, #sources: 99, #destinations: 1, traffic load: from .1 msg/min to 1 msg/min. • Observed metrics: link qualities and external interference.

• Output metrics: packet delivery rate, end-to-end packet latency, and average energy consumption. It goes without saying that the definition of each and every metric is a challenge per se. For example, even seemingly simple or commonly employed metrics may be defined differently. Energy consumption, for instance, may refer to the average energy consumption of all nodes in the network, or only to a subset of nodes deemed significant. Once the structure of metrics and profiles is in place, a benchmark definition should then provide specific semantics of all metrics involved, similar to existing literature in other fields [START_REF]Computer Vision Datasets[END_REF], [START_REF]Database Benchmarks[END_REF], [START_REF]YCB Benchmarks[END_REF].

IV. IMPLEMENTATION

IoTBench will be accompanied by several tools to make it practical and easy to use allowing researchers to perform automated and reproducible execution of the benchmark, both on real testbeds ( §IV-A) and in simulation ( §IV-B).

Regardless of whether results are obtained experimentally or in simulation, profiles and metrics should be permanently stored in a database linked to a centralized public repository. The latter should contain all protocols and implementations that have been benchmarked, to allow anyone to re-run a given experiment or simulation, and should easily allow a user to download the source code that was used in a specific evaluation, e.g., for checking used parameters. Furthermore, a Web interface should summarize, for each profile, which solutions have performed best. An example is the EWSN competition series [START_REF] Boano | EWSN Dependability Competition: Experiences and Lessons Learned[END_REF], which uses a leaderboard filtering results depending on the selected profile, environmental characteristics, and the like.

A. Implementation on Testbeds

The low-power wireless community commonly makes use of testbed facilities to realistically evaluate and compare the performance of newly-proposed protocols to the state of the art. Large-scale public testbeds such as MoteLab, Kansei, TWIST, and-more recently-Indriya [START_REF] Doddavenkatappa | Indriya: A Low-Cost, 3D Wireless Sensor Network Testbed[END_REF], FlockLab [START_REF] Lim | FlockLab: A Testbed for Distributed, Synchronized Tracing and Profiling of Wireless Embedded Systems[END_REF], and FIT IoT-Lab [START_REF] Adjih | FIT IoT-LAB: A Large Scale Open Experimental IoT Testbed[END_REF] have often been used to define an adhoc evaluation scenario, extract the metrics of interest from the collected serial logs, and draw conclusions about the performance of a given solution. Some of these testbeds offer valuable features such as power profiling, fine-grained GPIO tracing, and controlled mobility of nodes [START_REF] Lim | TraceLab: A Testbed for Fine-Grained Tracing of Time Sensitive Behavior in Wireless Sensor Networks[END_REF], but do not embed a number of key properties that are needed to support the benchmark proposed in ( §III). We discuss next a list of these key properties and briefly sketch how they could be implemented in existing testbeds. Extracting performance metrics. To evaluate and compare different solutions, testbeds should offer the ability to extract a minimum set of key performance metrics, such as packet delivery, end-to-end packet latency, and energy consumption. The simplest approach is to have the testbed extracting such metrics automatically by interacting with the benchmarked firmware. This, however, relies on the correctness of the firmware and has the disadvantage of not being indisputable (e.g., if the software to be benchmarked has to compute itself the energy consumption using software-based energy estimation methods [START_REF] Dunkels | Software-based Online Energy Estimation for Sensor Nodes[END_REF]). The quantification of performance metrics should hence ideally be actively measured by the testbed infrastructure in an unobtrusive way. A few testbeds are capable of performing a non-intrusive fine-grained measurement of power consumption [START_REF] Adjih | FIT IoT-LAB: A Large Scale Open Experimental IoT Testbed[END_REF], [START_REF] Bouckaert | The w-iLab.t Testbed[END_REF] and timing information [START_REF] Lim | TraceLab: A Testbed for Fine-Grained Tracing of Time Sensitive Behavior in Wireless Sensor Networks[END_REF], [START_REF] Schuß | A Competition to Push the Dependability of Low-Power Wireless Protocols to the Edge[END_REF]. Testbeds used as benchmarking reference can for example embed these capabilities and make use of GPIO tracing to detect the reception of packets [START_REF] Schuß | A Competition to Push the Dependability of Low-Power Wireless Protocols to the Edge[END_REF] in an nearly unobtrusive way. Feeding input parameters. To create an instance of a given profile, the testbed needs the ability to configure a number of system parameters such as the number of nodes and their topology. One could either specify offline a set of hard-coded scenarios that belong to a given benchmark profile or let the testbed automatically compute this information from an accurately derived connectivity map. Experiment parameters such as the traffic load can be "passed" to the benchmarked firmware through external scripts. For example, one can specify a list of commands passed via serial to the firmware such as "S@13@27" instructing a node to send a message to node 13 with a payload of 27 bytes. Furthermore, the testbed should be able to handle a number of experiment parameters such as the necessary number of repetitions and automatically schedule experiments accordingly. Observing external metrics. A testbed should also have the ability to observe external metrics for each experiment. An example is the observation of the amount of external interference in the surrounding of the wireless nodes. A testbed could make use of a few dedicated devices to scan the relevant frequencies [START_REF] Noda | Quantifying the Channel Quality for Interference-Aware Wireless Sensor Networks[END_REF] or use existing supporting tools [START_REF] Istomin | Trident: In-field connectivity assessment for wireless sensor networks[END_REF]. An open research challenge is, however, how to discern the unwanted traffic from the one caused by the running experiment. Based on these, a testbed can then either autonomously refute and re-run the experiment, for example, in case the measured interference is higher than a given threshold, or automatically associate an experiment to the correct profile. Another example is the observation of people's presence in the surroundings of the wireless nodes. Testbeds that embed cameras for monitoring the movement of robots [START_REF] Johnson | Mobile Emulab: A Robotic Wireless and Sensor Network Testbed[END_REF] could be used for this purpose by using suitable human motion recognition algorithms. Controlling the environment. Testbeds used to benchmark low-power wireless systems should ideally also offer the ability to control the environmental conditions to study their impact on protocol performance. For example, facilities based on off-the-shelf IEEE 802.15.4 radios can make use of Jam-Lab [START_REF] Boano | JamLab: Augmenting Sensornet Testbeds with Realistic and Controlled Interference Generation[END_REF] to generate specific interference patterns over time. Towards this goal, the testbed may offer a pre-defined set of jamming profiles that can be chosen when performing an experiment, and also a way to limit the number of additional sources of interference present in the background. Similarly, specialized testbeds allowing to control the on-board temperature of nodes such as TempLab [START_REF] Boano | TempLab: A Testbed Infrastructure to Study the Impact of Temperature on Wireless Sensor Networks[END_REF]. could offer a pre-defined set of temperature profiles for a given set of nodes.

B. Implementation in Simulation

Many different simulators have been developed for evaluating low-power wireless protocols. A simulator brings several advantages when implementing benchmarks: the evaluations are perfectly repeatable, easy to compare against each other and easy to setup. Different benchmarks can be easily described in setup descriptions, which can be re-used by other developers. Furthermore, the evaluation is typically fast, costefficient and can be completely automated. However, simulations have been also criticized in the community for not being realistic enough. While higher-layer protocols and their effects can be easily implemented, other properties such as energy consumption, wireless propagation and interference, hardware faults or delays, are still hard to capture or require a complex implementation. Recently, progress has been made to address these challenges in simulation and modeling. In this section, we give an overview of the state of the art and identify some areas which still need to be addressed. In general, simulations are complementary to testbed evaluation and can be used to enable cross-validation. Existing simulation environments. TOSSIM [START_REF] Levis | TOSSIM: accurate and scalable simulation of entire TinyOS applications[END_REF] for TinyOS pioneered many ideas in simulation of WSN applications. Cooja [START_REF] Österlind | Cross-level sensor network simulation with COOJA[END_REF], a network simulator that supports both node emulation (only for MSP430 and AVR platforms), and native execution of Contiki applications [START_REF] Kim | Challenging the ipv6 routing protocol for low-power and lossy networks (rpl): A survey[END_REF], is widely used by the research community. With emulated nodes, Cooja directly runs a compiled firmware, which makes the transfer from simulation to reality easy. It also offers a number of good simulation models, e.g. for interference. Other options include OMNeT++ 1 , which is a general-purpose network simulator. It 1 www.omnetp.org hosts models for various applications and the INET Framework offers a complete networking stack, including energy consumption and wireless propagation. However, it does not offer automatic transfer from simulation to real systems and code needs to be re-implemented. NS2, and its successor NS3, include sophisticated models but is not in low-power wireless networking community.

Required simulation models for benchmarking. The set of required models for simulating low power wireless systems can be separated coarsely into three groups: environmental, hardware and software models. While the software models can be easily simulated (in fact, their implementations are often identical to real systems), the environmental and hardware models pose a great challenge, both in terms of designing and implementing the models, and in using them correctly. The three groups of models are also interconnected. a) Environment models: In these models, we strive to mimic the properties of our environment, such as wireless propagation, events (e.g., fire, movement), temperature, and humidity. As a first step towards benchmarking low-power wireless systems, we consider wireless propagation and interference the most important model. It describes how the waves propagate in the environment and captures properties of this propagation such as spreading, fading, attenuation, Doppler effect, and shadowing [START_REF] Rappaport | Wireless Communications[END_REF]. The main challenge may not be the model itself, but describing the propagation environment in the needed level of detail, including people moving around, furniture, and open/closed windows. There exist sophisticated models for wireless propagation, such as ray tracing [START_REF] Yun | Ray Tracing for Radio Propagation Modeling: Principles and Applications[END_REF] or radio irregularity model [START_REF] Khalilov | Radio Irregularity Model in OMNeT++[END_REF]. Another option is tracebased radio propagation models [START_REF] Garg | A TinyOS Based Tool for Gathering Real-World Wireless Traces[END_REF] as done in TOSSIM, for example. These models could be used in conjunction with packet reception models for synchronous transmissions [START_REF] Wilhelm | On the reception of concurrent transmissions in wireless sensor networks[END_REF] to accurately simulate a class of state-of-the-art protocols. b) Hardware models: These include simulating hardware behaviour, such as delays, interrupts and errors. They also include models for power consumption and batteries. It is a rather underrepresented topic, which needs much more attention. In OMNeT++, there exists a framework for power modelling, which is still under active development [START_REF] Feeney | Demo: Making Batteries a First Class Element in the Design and Evaluation of Embedded Wireless Systems[END_REF], [START_REF] Feeney | Energy Framework: an extensible framework for simulating battery consumption in wireless networks[END_REF].

A lifetime estimator able to model non-linear properties of batteries was also proposed for Cooja [START_REF] Dron | An Emulation-based Method for Lifetime Estimation of Wireless Sensor Networks[END_REF]. c) Software models: These include implementations of protocols, applications and other software elements. Cooja is able to run any Contiki application (native nodes), or arbitrary firmwares (emulated nodes). In OMNeT++/INET, some implementations of IEEE standards are available, e.g. 802.15.4 or 802.11. Traffic and application models are abundant. The situation is similar for ns-2/ns-3.

In summary, the simulation environment suffers from exactly the opposite problems compared to testbeds. Extracting performance metrics, feeding input parameters, observing external metrics and controlling the environment are perfectly available in simulation, but new or consolidated models are required for accuracy and realism.

V. EARLY INITIATIVES AND ADOPTION

As a first concrete work item, we intend to evaluate a selection of multi-hop mesh routing protocols against initial, simple benchmark profiles. This will present a number of benefits: i) gain initial experience on building benchmark profiles, ii) provide data relevant to reproducibility analysis, and iii) raise awareness about the benchmark.

1) Simple profiles: initially, we may consider two scenarios: i) periodic data collection and ii) aperiodic any-to-any, two-way communication. Both will have well-defined input parameters and output metrics, such as packet delivery rate, end-to-end packet latency, and energy consumption. To let any interested party run their protocol, we will provide access to one reference testbed (and specify how to reproduce its infrastructure elsewhere), and define interfaces for metric computation. For instance, when it comes to measuring end-to-end packet latency, one only needs to define how packet transmission and reception events are signaled. 2) Proof-of-concept testbed train: as a second step, we aim to fully automate benchmarking and run the complete set of experiments (combination of protocol and profile) periodically, e.g., weekly. We will, in addition, consider capturing external observed metrics, such as the ambient noise level during experiments. This will enable us to study the reproducibility of experiments in nondeterministic environments, and as testbeds evolve. 3) Community engagement: finally, through publishing all results online, and encouraging the community to submit their own protocol, we will raise awareness about the initiative and receive early feedback. We believe this to be crucial towards the realization of a practical and useful benchmark. Similar initiatives will also be instrumental in gaining adoption of the benchmark. The profiles provide ready-touse scenarios. They will be established through a community process and will ensure that traffic patterns and application scenarios are relevant and representative. We believe that this will ease protocol evaluation as researchers do not have to identify, define, and justify their evaluation scenarios to the same degree as today. At a later stage, we will also integrate the profiles as ready to use tools in selected testbeds.

The testbed train will provide benchmark results for established protocols under above profiles. For example, it will ensure that reference implementations are configured correctly for each application scenario. Thus, a researcher does not need to evaluate and configure the protocols that a researcher wants to compare their work against, which is often a complex and time consuming process. In our experience, this testbed train will reduce the time needed for protocol evaluation. Moreover, we expect it to increase the credibility of the evaluation.

As a second concrete work item, we have started to extend the D-Cube testbed infrastructure used during the EWSN dependability competition series in an attempt to make it a fullfledged benchmarking infrastructure for low-power wireless systems [START_REF] Schuß | Moving Beyond Competitions: Extending D-Cube to Seamlessly Benchmark Low-Power Wireless Systems[END_REF]. Currently, D-Cube already embeds the ability of (i) extracting performance metrics; (ii) feeding input pa-rameters such as network density, traffic load, traffic pattern, duration of a run, and number of repetitions; as well as (iii) controlling the harshness of the RF environment. Similar initiatives will shed light on the requirements of a full-fledged benchmarking infrastructure for low-power wireless systems and are also instrumental to gain adoption of the benchmark.

VI. CONCLUSIONS AND OUTLOOK

The roadmap of the proposed benchmark initiative must culminate in a set of materialized concepts with direct impact to the relevant stakeholders, including industry and academia. The most direct application is the adoption of the benchmark in the certification process chain, either by already existing certification companies or through a new entity devoted to that. In any of the cases, certification entities may exploit the benchmark procedures and results to complement or extend certification processes in order to rank the evaluated technology under certain configurations. To be successful, this initiative should seek to co-exist and complement existing industry-driven IoT benchmarks [START_REF] Eembc Iotmark | [END_REF].

This approach may benefit the adopting entities as the performance of the technology will be fully comparable to others. At the same time, vendors would use such a reference to center their improvements or develop new products filling the technical limitations identified through the benchmark evaluation as is happening today in other verticals, e.g., CPU/processor benchmarks. Yet, the benchmark can be used by particular markets to establish entry barriers for competitors or substitute technologies. As a public reference, the benchmark can limit the required lower limits of performance for a given application domain.

Academia, in turn, will benefit from the benchmark metrics and scenarios since the benchmark results will draw the direction of what is more relevant and what are the real issues for a particular technology. This will favor the progress of research in relevant directions with direct application to adopting markets.

We believe in keeping the initiative open, enabling the benchmark initiative to progress without the influence or bias of any particular stakeholder, maintaining results and benchmark specifications away from commercial interests and ensuring the reproducibility and correctness of those publicly reported results. The benchmark community must ensure the support to open tools when possible, eliminating entry barriers to those stakeholders with promising technologies but limited resources and opportunities.

Fig. 1 .

 1 Fig. 1. Comparing the experimental setup of systems supporting periodic data collection (2010-2015).


Jonathan Muñoz
email: jonathan.munoz@inria.fr

Emmanuel Riou
email: emmanuel.riou@gridbeecom.com

Xavier Vilajosana
email: xvilajosana@uoc.edu

Paul Muhlethaler
email: paul.muhlethaler@inria.fr

Thomas Watteyne
email: thomas.watteyne@inria.fr

Overview of IEEE802.15.4g OFDM and its Applicability to Smart Building Applications

This paper compares the performance of two IEEE802.15.4 physical layers in the Smart Building context: 2.4 GHz O-QPSK and sub-GHz OFDM. The former has been in the IEEE802.15.4 standard since 2003, the latter was rolled into its 2015 revision. OFDM promises exceptional performance, in particular in environments with high external interference and multi-path fading. This paper starts with a comprehensive overview of IEEE802.15.4 and IEEE802.15.4g, with a particular focus on OFDM, its design drivers and modes of operation. The second half of this paper presents results from an exhaustive benchmarking campaign of both technologies in a building environment, and discusses lessons learnt. We show how OFDM has a higher range, even at 400 kbps and 800 kbps data rates. We then quantify the importance of frequency repetition in OFDM, and of using a wide communication channel, and we show how the use of OFDM can result in a 2-4× decrease in power consumption compared to 2.4 GHz O-QPSK. We conclude by recommending the use of OFDM option 1, with MCS2 for short (<128 B) frames, and MCS3 otherwise.

I. INTRODUCTION

Low-power wireless mesh networks encompass industrial [START_REF] Pister | Industrial Routing Requirements in Low-Power and Lossy Networks[END_REF], home [START_REF] Brandt | Home Automation Routing Requirements in Low-Power and Lossy Networks[END_REF], urban [START_REF] Dohler | Routing Requirements for Urban Low-Power and Lossy Networks[END_REF] and smart building [START_REF] Martocci | Building Automation Routing Requirements in Low-Power and Lossy Networks[END_REF] applications. In Smart Building applications [START_REF] Martocci | Building Automation Routing Requirements in Low-Power and Lossy Networks[END_REF], low-power wireless mesh networks are used to monitor and automate intrusion detection, fire detection, elevator monitoring, HVAC, and lighting, among others.

The IEEE802. [START_REF] Dias | Evaluation of an RPL/6LoWPAN/IEEE 802.15.4g Solution for Smart Metering in an Industrial Environment[END_REF].4 standard has ruled the Smart Building application space for low-power wireless mesh solutions. Standards have been developed over the last decade to define an entire protocol stack, industrial alliances have built around them, and countless companies, big and small, sell IEEE802.15.4-based Smart Building solutions.

These solutions mostly use the 2.4 GHz version of IEEE802.15.4, with O-QPSK modulation, DSSS, and 127 B frames. This physical layer provides a trade-off between energy consumption, communication range, and reliability which is absolutely suited to Smart Building applications. Tens of thousands of networks based on that physical layer are operating today, and achieve over 99.999% end-to-end reliability and over a decade of battery lifetime. It is not the goal of this paper to argue against IEEE802. 15.4. Yet, there are a number of recent developments in the physical layers of the IEEE802.15.4 standard. Its 2015 revision Fig. 1: In OFDM, data is transmitted on multiple sub-carriers. Multiple sub-carriers are grouped into channels; the OFDM option determines how this grouping is done. includes the IEEE802.15.4g amendment, which defines new physical layers. These have been designed for smart utility applications, and focus on range. Among these is OFDM (Orthogonal Frequency Division Multiplexing), a physical layer which has been extensively used in high-end wireless systems, and which is now entering the low-power wireless space. On paper, OFDM offers longer range, higher bandwidth and better handling of external interference and multi-path fading. This paper explores these claims, experimentally, in the Smart Building application space.

The remainder of this paper is organized as follows. Section II gives an overview of IEEE802.15.4 and IEEE802.15.4g, and presents related work. Section III lists the contributions of this paper. Section IV details the experimental setup. Section V discusses the experimental results and what lessons can be learnt from them. Section VI concludes this paper.

II. OVERVIEW AND RELATED WORK

This section provides a complete overview of both IEEE802. [START_REF] Dias | Evaluation of an RPL/6LoWPAN/IEEE 802.15.4g Solution for Smart Metering in an Industrial Environment[END_REF].4 (Section II-A) and IEEE802. [START_REF] Dias | Evaluation of an RPL/6LoWPAN/IEEE 802.15.4g Solution for Smart Metering in an Industrial Environment[END_REF].4g (Section II-B). Both sections give a comprehensive overview of the technology and go over previously-published related work.

A. IEEE802.15. [START_REF] Martocci | Building Automation Routing Requirements in Low-Power and Lossy Networks[END_REF]

GHz

The IEEE802.15.4 standard [START_REF]IEEE Standard for Information Technology -Telecommunications and Information Exchange Between Systems -Local and Metropolitan Area Networks Specific Requirements Part 15.4[END_REF] has been almost synonymous to smart building applications. This standard defines the physical layer (modulation, data rate, transmit power). All major chip vendors have cheap1 IEEE802.15.4-compliant radio chips in their catalog, making it a very accessible technology. The communication range with those chips (assuming a 0 dBm transmit power and a sensitivity around -100 dBm) is in the order of 100-200 m outdoors and 20-50 m indoors. Their power consumption is 5-25 mA at 3.6 V, making it possible to achieve a decade of battery lifetime when duty cycling. Because it is unlicensed and world-wide, the 2.4 GHz frequency band is the most used for smart building applications.

At the physical layer, IEEE802. The first version of IEEE802.15.4 was ratified in 2003. In the decade that followed, several fully standards-compliant protocol stacks were built on top of the IEEE802.15.4 physical layer. The protocol stack mainly defines how these radios are duty cycled to conserve energy (at the "Medium Access Control" -MAC -layer), and how to organize a network deployed throughout a building as a multi-hop mesh (at the routing layer). Countless proposals of MAC and routing combinations have been proposed by academia; a couplesuch as the RPL routing protocol [START_REF] Winter | RPL: IPv6 Routing Protocol for Low-Power and Lossy Networks[END_REF] -have made it through the standardization process.

Industrial alliances have formed, which typically put together several standards to form a complete protocol stack, and serve as labeling/certification bodies. The most prevalent examples in the Smart Building space are ZigBee2 and Thread 3 .

Tens of thousands of low-power wireless mesh networks are operating today, using variants of these protocols and standards. A large number of companies are working in the Smart Buildings space. One example is Assa Abloy4 , which commercializes wireless door opening solutions. Another example is Yanzi Networks5 , which commercializes indoor temperature, humidity and presence sensors, as well as controllable power sockets.

There are two main challenges for making a network composed of IEEE802.15.4 devices, operating at 2.4 GHz, reliable.

The first challenge is external interference. At 2.4 GHz, external interference mainly comes from IEEE802.11 (WiFi) and IEEE802.15.1 (Bluetooth).

Khaleel et al. [START_REF] Khaleel | Impact of Wi-Fi Traffic on the IEEE 802.15.4 Channels Occupation in Indoor Environments[END_REF] investigate the cross-interference between IEEE802.15.4 and IEEE802.11b (WiFi). They use a Cross-bow Telos device equipped with an IEEE802.15.4-compliant CC2420 radio to sense the frequency spectrum by using RSSI, under different WiFi data rate conditions. They show that when there is a WiFi connection of 3 Mbit/s, the probability of failure to access the medium for a IEEE802.15.4 device reaches 90%, when operating on the same frequency as the WiFi connection.

Watteyne et al. [START_REF] Watteyne | Lessons Learned from Large-scale Dense IEEE802.15.4 Connectivity Traces[END_REF] conduct an experiment to record the connectivity between 350 nodes in a typical office environment, using the IoT-lab large-scale testbed [START_REF] Adjih | FIT IoT-LAB: A Large Scale Open Experimental IoT Testbed[END_REF]. These nodes communicate on each of the 16 available frequencies at 2.4 GHz. They show the impact of WiFi interference on the reliability of the IEEE802.15.4 wireless links: even when the WiFi network sits idle, IEEE802.11 beaconing causes a significant number of links to drop from 90% to 70-80% Packet Delivery Ratio (PDR).

The second challenge is multi-path fading. In any indoor environment, objects in the surroundings of a wireless link cause a reflection of the radio signal. These different "echoes" of the same signal reach the receiver's antenna at slightly different times. All these reflections can interfere constructively, increasing the signal strength. Yet, they can also interfere destructively, making the communication between transmitter and receiver impossible.

Watteyne et al. [START_REF] Watteyne | Mitigating Multipath Fading through Channel Hopping in Wireless Sensor Networks[END_REF] visualize the effect of multi-path fading. They install a transmitting node on a robotic arm which moves inside a 20 cm by 35 cm plane, with a 1 cm step, yielding 735 positions. At each position, the transmitting node sends 1000 29 byte frames to the receiver node 1 m away. The experiment is repeated over each of the 16 available frequencies. Results show that the PDR of the wireless link between transmitter and receiver can swing from 100% to 0% by moving the transmitter by just 3 cm. This is entirely due to multi-path fading.

New techniques have appeared to cope with external interference and multi-path fading in IEEE802.15.4 networks operating at 2.4 GHz. The most disruptive is arguably Time Slotted Channel Hopping (TSCH), a MAC approach by which tightly synchronized nodes, heavily duty cycle their radio to conserve energy, and use frequency hopping to combat interference and multi-path fading. TSCH has been the default MAC approach in IEEE802.15.4, since its 2015 revision. Further standardization at the IETF 6TiSCH working group defines how to combine TSCH with IPv6. The authors in [START_REF] Watteyne | Industrial IEEE802.15.4e Networks: Performance and Trade-offs[END_REF] present the performance of SmartMesh IP, a commercial TSCH solution, which yields over 99.999% end-to-end reliability, and over a decade of battery lifetime. TSCH (and 6TiSCH) networks are widely regarded as the future for low-power wireless networks, and are the base for all major open source implementations [START_REF] Watteyne | Industrial Wireless IP-Based Cyber -Physical Systems[END_REF] as well as several commercial ongoing implementations.

Clearly, using 6TiSCH can yield wire-like reliability and a decade of battery lifetime from an IEEE802.15.4 network operating at 2.4 GHz. This paper looks at whether using a different physical layer -possibly combined with 6TiSCHhas the potential to yield even better performance. A whole new set of sub-GHz physical layers has been developed within the IEEE802.15.4g task group. The goal of this paper is to explore their potential.

B. IEEE802.15.4g sub-GHz

The IEEE802.15.4g amendment [START_REF]IEEE Standard for Local and metropolitan area networks-Part 15.4: Low-Rate Wireless Personal Area Networks (LR-WPANs) Amendment 3: Physical Layer (PHY) Specifications for Low-Data-Rate, Wireless, Smart Metering Utility Networks[END_REF] was created for Smart Utility Network (SUN) applications. One strong requirement, to be able to build neighborhood-wide (mesh) networks, is a multi-km range. The IEEE802.15.4g amendment, first published in 2012, was rolled into the main IEEE802.15.4 specification in its 2015 revision [14] 6 .

IEEE802.15.4g introduces three alternative physical layers (PHYs): FSK (Frequency Shift Keying), O-QPSK (Offset-Quadrature Phase Shift Keying) and OFDM (Orthogonal Frequency Division Multiplexing). Each physical layer was designed for a specific market segment, and is marketed as having distinct advantages. FSK increases the transmit power efficiency by the constant envelope of the signal. O-QPSK shares the characteristics of IEEE802.15.4 DSSS O-QPSK. OFDM provides high data rates, and is designed to operate in environments with frequency selective fading, such as indoors [14]. In all cases, compliant radio chips can exchange frames of up to 2047 B.

FSK and O-QPSK are conventional well-known modulation techniques. While OFDM is commonly used in advanced systems, its introduction to low power wireless devices is new. OFDM was created to combat multi-path fading. Like TSCH, it exploits frequency diversity. But while TSCH does so at the MAC layer, OFDM does so directly at the physical layer. In OFDM, a frequency band (called "channel") is divided into numerous frequencies ("sub-carriers"). The sub-carriers are far enough apart in frequency to be orthogonal: they do not interfere with one another. An OFDM symbol is the combination of the sub-carriers, each carrying a portion of the information to be transmitted. Each sub-carrier is modulated with a low-order modulation (BPSK, O-QPSK or 16-QAM). Combining the sub-carriers is equivalent to having a high order signal modulation (e.g., [START_REF] Mochizuki | Development and Field Experiment of Wide Area Wi-SUN System Based on[END_REF],777,216-PSK for 24 bits per symbol), making it possible to achieve high data rates. Fig. 1 shows the OFDM channels in IEEE802.15.4g 7 , where communication uses multiple sub-carriers, separated by a constant equal frequency distance ∆f . In the center frequency f c , no information is sent (null tone).

In OFDM, the effective duration of a symbol (t s ) is 96 µs. To ensure orthogonality between sub-carriers, they are separated by 1/t s , or 10,416.667 Hz. The standard describes 4 ways of grouping sub-carriers to form an OFDM symbol; they are called "options", and are numbered from 1 to 4. Table I shows the number of sub-carriers for each option, as well as the bandwidth it occupies. The sub-carrier modulation is specified by the Modulation and Coding Scheme (MCS). There are 7, numbered from 0 to 6. The data rate of the signal is given by the combination of the OFDM option and MCS. Table II shows the details of the MCS and the data rates for each OFDM option/MCS combination.

Frequency repetition is an OFDM technique in which more than one sub-carrier transports the same information. While it reduces the effective data rate of the transmission, it makes the signal much more robust against multi-path fading. That is, even if one sub-carrier is not successfully demodulated, there is another sub-carrier (at a different frequency) that carries the same data.

Any IEEE802.15.4g-compliant chip must implement a physical layer with 2-FSK modulation and 50 kbps data rate. All other physical layers are optional. Each physical layer allows further parametrization (data rate, bandwidth). The result is that IEEE802.15.4g comprises an astounding 31 different physical layer options, with data rates ranging from 6.25 kbps to 800 kbps.

With such a variety of settings, it is hard for an implementor to understand which setting to use. This paper aims to contribute to providing an answer. We start by looking at already-published related work, with a particular focus on hands-on performance evaluation in specific scenarios, ideally comparing the different physical layers. We then conduct our own experiments to compare sub-GHz OFDM with the traditional 2.4 GHz O-QPSK to see whether, as an end user, there is some advantage. We are particularly interested in the higher data rates modes of OFDM, and in the impact of frequency repetition on reliability.

We have conducted the experiments introduced in Section IV on all IEEE802.15.4g modes. For reasons of space 8 , we only present the modes which give the reader the most insights: option 1 MCS2 (400 kbps, 1094 kHz bandwidth, 2× frequency repetition), option 1 MCS3 (800 kbps, 1094 kHz bandwidth, no frequency repetition), option 2 MCS5 (800 kbps, 552 kHz bandwidth, no frequency repetition).

This paper is obviously not the first to look at OFDM in a low-power wireless context. Several teams have worked on building complete IEEE802.15.4g-based solutions. That is, given a physical layer, assemble a protocol stack with existing standards, and evaluating the resulting network in some pilot deployment.

Dias et al. [START_REF] Dias | Evaluation of an RPL/6LoWPAN/IEEE 802.15.4g Solution for Smart Metering in an Industrial Environment[END_REF] evaluate a protocol stack they designed for smart metering in an industrial environment. The solution is based on IEEE802.15.4g, combined with an upper stack developed by the IETF (6LoWPAN, RPL OF1, TLS-DTLS and DLMS/COSEM). They use AT86RF215 chips configured to communicate at 915 MHz using O-QPSK modulation and 250 kbps data rate. They test a 10-node network deployed for 20 days in a 120 m × 40 m warehouse. The nodes form a multi-hop network around a border router located in the corner of the building. Using simple reachability tests, they show that packet loss ratio increases rapidly with the number of hops, with almost 90% loss at 4-5 hops. The round-trip time increases by approximately 15 ms per hop.

Mochizuki et al. [START_REF] Mochizuki | Development and Field Experiment of Wide Area Wi-SUN System Based on[END_REF] implement a Wireless Smart Utility Network (Wi-SUN) system. They propose up-links transmissions ("UP", toward the Border Router) at 20 mW (+13 dBm), and down-link transmissions ("DL", toward the nodes) at 250 mW (+24 dBm). The deployment is performed in the city of Kyoto, Japan, consisting of a Border Router and a mobile Measurement Station. The Border Router is located on the roof of the Kyoto City Hall, 17 m high. Data rate is set to 100 kbps, modulation to 2-GFSK, frame length to 250 B, and frames are separated by 50 ms. The Measurement Station is placed at several distances from the Border Router, in both Light-of-Sight (LOS) and Non-Light-of-Sight (NLOS) conditions. 1000 frames are transmitted from the Border Router to the Measurement Station, for each of its locations. The authors record the PDR for different positions of the Measurement Station. They show that, in LOS conditions, a higher transmission power at the Border Router provides more stable communication. By changing the transmit power of the base station from 20 mW to 250 mW, its communication range increases from 150 m to 650 m.

The related work reviewed so far focuses on building entire systems. None of the publications pays particular attention to the IEEE802.15.4g setting (modulation, data rate) used. We believe is that OFDM is very applicable for the Smart Building space, and we use this paper to argue why. We are convinced that OFDM has not received sufficient attention for two reasons. First, IEEE802.15.4-OFDM is a recent technology. The Atmel AT86RF215 9 , arguably the most used radio capable of OFDM, was only available since Q3 2015 10 . Second, OFDM is perceived as being very energy hungry and complex [START_REF] Hanes | IoT Fundamentals[END_REF], and therefore not applicable for battery-powered low-power wireless mesh solutions.

What is missing is work that evaluates IEEE802.15.4g OFDM experimentally, and assesses its suitability for lowpower wireless mesh-based Smart Building solutions. Lee et al. [START_REF] Lee | Implementation of IEEE 802.15.4g Wireless Communication Platform for Smart Utility Service[END_REF] are, to the best of our knowledge, the only authors who have done a similar study, on a home-made implementation of IEEE802.15.4g OFDM (combining a 32-bit micro-controller, an FPGA and a radio frequency ASIC). Their article is, however, focused on the feasibility of a smart home utility service using IEEE802.15.4g FSK and OFDM. They build a data concentrator and remote monitor connected to the Internet and then check the real status of the water and electricity consumption. No details are given about the location of the nodes and the distance covered by the radio links.

Our paper contributes to this missing body of work. Section III details our goal and lists our contributions.

III. CONTRIBUTIONS

This study compares sub-GHz IEEE802.15.4g-OFDM options 1 and 2 in their higher data rates with IEEE802.15.4 O-QPSK PHY at 2.4 GHz, from the user's point of view, in smart building applications.

The contribution of this study is five-fold: 1) We show that the Packet Delivery Ratio (PDR) of IEEE802.15.4g-OFDM is higher than for IEEE802.15.4 O-QPSK, for all locations. 2) We show that this holds even when sending 2047 B frames over IEEE802.15.4g-OFDM and 127 B over IEEE802.15.4 O-QPSK. 3) We show how frequency repetition can double the PDR of marginal links. 4) We show the importance of using wide channels. 5) We further show that the charge consumed for sending a 127 B packet over IEEE802.15.4g-OFDM is at least 2 times lower than IEEE802.15.4 O-QPSK. We understand the impact changing frequencies (2.4 GHz and sub-GHz) has on many different metrics, including range. Some might even qualify comparing 2.4 GHz O-QPSK with sub-GHz OFDM as "unfair". Yet, we approach this comparison purely from an end-user point of view: given the choice between technologies, and given that all operate in unlicensed bands, does it make sense to use Sub-GHz OFDM for a Smart Building application?

IV. EXPERIMENTAL SETUP

We conduct this study through a number of experiments. To run an experiment, we use 4 nodes: one is a transmitter (TX), the other 3 are receivers (RX). During an experiment, the nodes loop through all combinations of modulation, frequency and frame length, and, for each, measure the PDR of the link.

This section provides enough details for the reader to be able to replicate the tests and results. We describe the hardware (Section IV-B), software (Section IV-C), radio characteristics (Section IV-D), and deployments (Section IV-E).

A. Foreword: Alias and Color Codes

We want to compare the performance of 2.4 GHz IEEE802.15.4 O-QPSK and sub-GHz IEEE802.15.4g OFDM. The former is almost synonymous with smart building applications, and therefore represents a baseline. The latter is newer, and promises exceptional performance in environment in which multi-path fading is very present. Our test considers all 31 radio settings, covering all IEEE802.15.4 and IEEE802.15.4g modulations. For reasons of space, and to provide the reader with the most insightful information, we only present results for the following 4 PHYs: 2.4 GHz O-QPSK at 250 kbps, sub-GHz OFDM option 1 at 400 kbps, sub-GHz OFDM option 1 at 800 kbps and sub-GHz OFDM option 2 at 800 kbps. Table III provides an "alias" and assigns a color for each PHY. These will be used throughout the remainder of this paper.

B. Hardware

Each node is equipped with an Atmel AT86RF215 radio chip, which implements both IEEE802.15.4 (2.4 GHz) and IEEE802.15.4g (sub-GHz). We use the AT86RF215 Xplained Pro reference board -manufactured by Atmel -to ensure the setup follows the chip vendor's recommendations. This board has two SMA connectors on which we connect two 2 dBi omni-directional antennas, one for 2.4 GHz, one for sub-GHz.

A node (depicted in Fig. 2) consists of a Raspberry Pi 3 (rPi) model B connected to the radio board over an SPI bus. The electronics and connectors are housed in a plastic box, with the two antennas sticking out. The box is self-contained and powered by a 22,000 mAh battery bank, more than enough charge to power the node during an experiment.

C. Software

The rPi of each node runs a Linux Debian distribution, and is connected to the Internet over WiFi. We connect over SSH to each node to remotely launch the test scripts. The test scripts are written in Python, and drive the radio throughout an experiment 11 . The scripts are responsible for having the TX node loop through all combinations of modulation, frequency 11 As an online addition to this paper, all the software is published under an open-source license at https://github.com/openwsn-berkeley/range_test. and frame length, and, for each, send 100 frames. On the RX side, the scripts are responsible for (re-)configuring the radio so it is listening on the same frequency using the same modulation as the TX node at the same time. TX and RX nodes are synchronized over NTP. Appropriate guard times are introduced to ensure that the RX node is listening when the TX node transmits a frame.

The frame lengths considered depend on the PHY. For OFDM, the TX node sends frames of lengths 127 B and 2047 B. For IEEE802.15.4 O-QPSK, the TX node sends frames of length 127 B. Similarly, the frequencies considered depend on the physical layer. For OFDM, there are 5 and 8 available frequencies for option 1 and 2, respectively. For the 2.4 GHz frequency band, there are 16 frequencies. In all cases, the inter-frame spacing is 20 ms and the TX power is +8 dBm.

An experiment -looping through all modulations, frequencies and frame lengths -takes roughly 30 min. During an experiment, an RX node logs, for each frame received, the modulation and frequency it listens on, the counter contained in the frame, whether its FCS is correct and the RSSI value. Because 100 frames are sent for each modulation/frequency/length, the PDR for that setting can be computed.

D. Radio Characteristics

Table III gives the current draw of the AT86RF215 radio chip, at 3.3 V, for each radio setting. Because the chip's datasheet does not provide the current draw for each setting, we measured it. For each radio setting, we configure the TX node to transmit in continuous transmission mode, and measure the current draw using an ammeter.

Table III also details the sensitivity of the AT86RF215 radio chip, for each radio setting, as read from the datasheet 12 .

E. Deployments

Fig. 3 shows a floorplan of the deployment area, the Inria office building in Paris, France. The ceiling is metallic, the floor is covered with carpet, external concrete walls have glass

V. EXPERIMENTAL RESULTS

In total, we collected 57,200 atomic measurements, one for each frame, RX location, frequency, frame length, and index. This dataset contains a wealth of information. The goal of this section is to explore this dataset and extract the lessons they contain.

A. The Longer Range of OFDM

Table IV gives the average PDR value over all frequencies available per PHY and per RX node location. It shows three tiers of RX positions. In the first tier (positions 1A through 1E), the receiver and transmitter nodes are close. PDR is over 80% in all cases, for both OFDM and O-QPSK. In the second tier (1F-1H), the PDR of O-QPSK starts decreasing (down to 62%), whereas the PDR of OFDM stays above 75%. In the third tier (2A-2C), OFDM still offers some connectivity while O-QPSK is not able to get any frame across.

Fig. 4 depicts the PDR/RSSI relationship for nodes 1F, 1G and 1H. Each dot corresponds to the PDR/average RSSI relationship for 100 frames of 127 B sent on one frequency 13 . All the OFDM samples exhibit a higher RSSI than O-QPSK. OFDM is well inside the sensitivity of its radio, O-QPSK presents samples closer to the sensitivity. This is expected given the difference in frequency [START_REF] Itu-R | Propagation Data and Prediction Methods for the Planning of Indoor Radiocommunication Systems and Radio Local Area Networks in the Frequency Range 300 MHz to 100 GHz[END_REF].

Yet, there is more involved than simply the difference in frequency. Because it operates at 2.4 GHz, O-QPSK should suffer from external interference from WiFi. By transmitting on multiple frequencies at the same time, OFDM should also be more robust against multi-path fading. We witness both phenomena in Sections V-B and V-C, respectively.

B. The (Limited) Impact of WiFi Interference over O-QPSK Fig. 5 shows the average PDR for each IEEE802.15.4 frequency at 2.4 GHz, for positions 1F, 1G and 1H. At the same time as the experiment was conducted, WiFi was operating in the building on IEEE802.11 channels 1 (2.412 GHz), 6 (2.437 GHz) and 11 (2.462 GHz). WiFi channels are 22 MHz wide, each covering roughly 4 IEEE802.15.4 frequencies. Fig. 5 clearly shows a degradation in the PDR for IEEE802.15.4 frequencies in the WiFi channels, from 80-90% down to 50-60%. Yet, this impact is small, and will just require an IEEE802.15.4 radio to retransmit more often when operating in a WiFi channel. That effect alone does not explain the better PDR of OFDM presented in Table IV.

C. The Power of Frequency Repetition

Table IV shows that, at 127 B, OFDM1@400 and OFDM1@800 perform the same. The difference between OFDM1@400 and OFDM1@800 is that only the former uses a 2× frequency repetition, meaning that each portion of data is repeated on two different frequencies. This means that if multipath fading prevents the receiver from correctly decoding the frame on one frequency, it should be able to on the second copy. Of course, enabling 2× frequency repetition reduces the data rate by half.

When increasing the frame length, however, things change. With 2047 B frames, it takes the radio longer to transmit the frame. At a constant bit error rate, it is normal to have a higher frame error rate. The resulting recommendation is hence to use frequency repetition when the PDR of the link is marginal.

D. The Importance of Using a Wide OFDM Band

In order to achieve a high data rate, the radio can be configured to use a lower OFDM option (more sub-carriers in the channel) and/or a higher MCS value (higher data rate per sub-carrier). We want to explore which approach is better, from a PDR point of view. In particular, we compare OFDM1@800 and OFDM2@800: they both result in the same data rate (800 kbps), OFDM1@800 by using more sub-carriers, OFDM2@800 by increasing the data rate of each subcarrier (16-QAM has a constellation size of 16, QPSK a constellation of 4).

Location 2B in Table IV satisfies our trade-off. From a PDR perspective, using a wide band yields good connectivity (74% PDR with 127 B), while using a higher MCS number as in OFDM2@800 causes the communication to be almost impossible (4% PDR with 127 B).

The resulting recommendation is hence to use the lowest possible OFDM option (wider channels), even if this means fewer channels.

E. Resulting Battery Lifetime Comparison

Table III indicates the current draw of the radio in each mode 14 . We assume a state-of-the-art MAC protocol, such as TSCH, which ensures that a node's radio is only on when needed (no idle listening).

We want to compute the charge the TX node needs to successfully send a 127 B to an RX node, for several locations. The term "successfully" implies retransmissions: if the PDR of the link is 50%, the TX node will have to transmit on average twice. Eq. (1) expresses that charge. C is the charge in Coulomb, d is the duration a radio needs to send the 127 B OFDM1@800 OFDM2@800 Fig. 6: Charge needed by the TX node to successfully transmit a 127-byte frame to a particular RX node, including retransmissions. For positions 2A and 2B, the computation cannot be done for O-QPSK and OFDM2@800, as the PDR is zero. frame, I is the current the radio draws when transmitting, and PDR is the Packet Delivery Ratio (a number between 0.0 and 1.0) of the link between the transmitter and receiver nodes.

C = d × I P DR (1)
Fig. 6 plots (1) for nodes 1F-2B, for all 4 radio settings. OFDM is more efficient than O-QPSK, in all cases. This is because the PDR of O-QPSK is lower than OFDM, and because OFDM has a higher data rate. While a more complete benchmarking/analysis (possibly using different radios, and taking radio wake-up times and acknowledgment overhead into account), the superiority of OFDM Fig. 6 shows is so clear that the trend will still hold.

VI. CONCLUSIONS

The overall lesson learnt from this paper is that OFDM should no longer be overlooked for low-power wireless networks, in particular in Smart Building applications. Not only are OFDM-capable radios readily available on the market, their performance meets the expectations. Their range is better than traditional 2.4 GHz O-QPSK, with techniques such as frequency repetition very efficiently handling multi-path fading and external interference directly at the physical layer. Current OFDM radios still consume in the order of 6× more than the best-in-class 2.4 GHz O-QPSK counterparts, but that is bound to change as inter-vendor competition kicks in.

For Smart Building applications, this paper makes the recommendation of using OFDM option 1, with MCS2 with short (<128 B) frames, MCS3 otherwise.

It is clear that a MAC-layer scheme will need to be introduced (for example through the 6TiSCH standardization action) which exploits the agility of these radios: for each frame, agree with your neighbor on the most appropriate radio setting.

Fig. 2 :

 2 Fig. 2: Picture of a test node. Four such test nodes are used in the experiments, each equipped with an AT86RF215 radio communicating on both 2.4 GHz IEEE802.15.4 O-QPSK and sub-GHz IEEE802.15.4g OFDM.

Fig. 3 :

 3 Fig. 3: Floorplan of the deployment area.

Fig. 4 :

 4 Fig. 4: PDR vs RSSI (127 B frames).

Fig. 5 :

 5 Fig. 5: Average PDR per channel on locations 1F, 1G and 1H.

  15.4 uses O-QPSK modulation with DSSS. It cuts the 2.400-2.485 GHz frequency bands into 16 orthogonal frequencies, separated by 5 MHz, each 2 MHz wide. The maximum frame size of IEEE802.15.4 is 127 B; the data rate is 250 kbps. This means that it takes an IEEE802.15.4 radio roughly 4 ms to transmit a 127 B frame.

TABLE I :

 I The OFDM option specifies how many sub-carriers are used in one channel.

		option	option	option	option
		1	2	3	4
	MCS0
	BPSK rate 1/2	100 kbps	50 kbps	-	-
	4× freq. rep.
	MCS1
	BPSK rate 1/2	200 kbps	100 kbps	50 kbps	-
	2× freq. rep.
	MCS2
	QPSK rate 1/2	400 kbps	200 kbps	100 kbps	50 kbps
	2× freq. rep.
	MCS3
	QPSK rate 1/2	800 kbps	400 kbps	200 kbps	100 kbps
	no freq. rep.
	MCS4
	QPSK rate 3/4	-	600 kbps	300 kbps	150 kbps
	no freq. rep.
	MCS5
	16-QAM rate 1/2	-	800 kbps	400 kbps	200 kbps
	no freq. rep.
	MCS6
	16-QAM rate 3/4	-	-	600 kbps	300 kbps
	no freq. rep.

TABLE II :

 II The MCS setting specifies the modulation used on each sub-carrier, its symbol rate, and whether frequency repetition is used. Combined with an OFDM option, this yields a data rate. A dash ('-') indicates a combination that does not exist. Colored cells are settings explored in Section IV.

TABLE III :

 III The PHY characteristics evaluated in this paper.

  Table IV clearly shows the benefit of frequency repetition: from RX location for example enabling frequency repetition for a 2047 B frame raises the PDR from 44% to 80%.

		O-QPSK	OFDM1@400	OFDM1@800	OFDM2@800
	RX node	(250 kbps)	(400 kbps)	(800 kbps)	(800 kbps)
		127 bytes	127 bytes	2047 bytes	127 bytes	2047 bytes	127 bytes	2047 bytes
	1A	96 %	100 %	100 %	100 %	100 %	100 %	92 %
	1B	78 %	100 %	100 %	100 %	100 %	97 %	92 %
	1C	91 %	100 %	100 %	100 %	100 %	100 %	100 %
	1D	88 %	100 %	100 %	100 %	100 %	100 %	100 %
	1E	81 %	100 %	100 %	100 %	100 %	100 %	100 %
	1F	62 %	94 %	92 %	89 %	89 %	85 %	75 %
	1G	81 %	100 %	100 %	100 %	98 %	99 %	96 %
	1H	71 %	100 %	100 %	99 %	99 %	99 %	94 %
	2A	0 %	100 %	100 %	100 %	100 %	100 %	98 %
	2B	0 %	88 %	80 %	74 %	44 %	4 %	0 %
	2C	0 %	0 %	0 %	0 %	0 %	0 %	0 %

TABLE IV :

 IV The Packet Delivery Ratio (PDR) of the wireless link for multiple positions of the RX node, for each radio setting and frame length.

Sub-5 USD IEEE802.15.4 radio chips are commonplace.

http://www.zigbee.org/

https://threadgroup.org/

https://www.assaabloy.com/

https://www.yanzi.se/

Strictly speaking, IEEE802.15.4g was rolled into IEEE802.15.4 in 2015. It is however common to (still) refer to this part of the IEEE802.15.4 specification as "IEEE802.15.4g", as we do throughout this paper.

Throughout this paper, we use the European

863-870 MHz frequency band. Our results hold for other frequency bands, including the 902-928 MHz US band, the only difference being the number of available channels.

The complete results can be found in a technical report, companion to this paper.

http://ww1.microchip.com/downloads/en/DeviceDoc/ Atmel-42415-WIRELESS-AT86RF215_Datasheet.pdf

http://www.atmel.com/about/news/release.aspx?reference=tcm:26-64181

The sensitivity for IEEE802.15.4g is defined as the RSSI which yields 10% PER with 250 B frames. The sensitivity for IEEE802.15.4 is defined as the RSSI which yields 1% PER with 20 B frames.

Some experiments were run multiple times, which is why there are more dots than there are frequencies available.

There are radios on the market which draw significantly less current for O-QPSK (9.7/4.5 mA TX/RX current for Analog Devices' LTC5800). While exact numbers presented in this section will be different with different hardware, the conclusions hold.

ACKNOWLEDGMENTS

This work is partially supported by the European Commission through the H2020 F-Interop and H2020 ARMOUR projects, and by the Spanish Ministry of Economy and the FEDER regional development fund through the SINERGIA project (TEC2015-71303-R).