domain: info spi

text:
Zacharie Brodard
email: brodazac@cisco.com

Hao Jiang
email: hjiang2@cisco.com

Tengfei Chang
email: tengfei.chang@inria.fr

Thomas Watteyne
email: thomas.watteyne@inria.fr

Xavier Vilajosana
email: xvilajosana@uoc.edu

Pascal Thubert
email: pthubert@cisco.com

Rover: Poor (but Elegant) Man's Testbed

Keywords: •Networks → Network experimentation, Network protocol design, Network measurement, Internet of Things, Testbed, OpenWSN, OpenMote

This paper presents the OpenVisualizer Rover testbed, a simple, easy-to-deploy and cheap testbed for the Internet of Things (IoT). The OpenWSN project provides a free and open-source implementation of a standards-compliant protocol stack for the IoT, as well as all the necessary network management and debugging tools. The network management software, OpenVisualizer, is portable across popular operating systems, and connects the OpenWSN low-power wireless mesh network to the Internet. In the current setup, motes are connected to the USB ports of the computer the OpenVisualizer runs on. The OpenVisualizer monitors the internal state of those motes, which it presents through a web interface. Rover extends the OpenVisualizer by allowing motes plugged into different computers to remotely connect to it. Once connected, a user monitors and manages the motes exactly as if they were connected locally. This offers endless experimentation possibilities, as the resulting testbed can be quickly (re)deployed in realistic environments. An example Rover testbed has been deployed at the Cisco Paris Innovation and Research Lab. This paper discusses the Rover architecture, the deployment, and the experimental research done with it.

INTRODUCTION

With the development of new technologies in ICT, we are witnessing during the last few years the emergence and fast growth of the Internet of Things (IoT). To exploit, test and Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. verify new IoT technologies and standards, several experimental testbeds have been developed and deployed to enable experimental research and benchmarking. FIT IoT-lab [START_REF] Adjih | FIT IoT-LAB: A large scale open experimental IoT testbed[END_REF] is a testbed deployed in 6 sites across France, composed of 2728 low-power wireless nodes and 117 mobile robots, available for experimenting with large-scale wireless IoT technologies. Indriya [START_REF] Doddavenkatappa | Indriya: A Low-cost, 3D Wireless Sensor Network Testbed[END_REF] is another large-scale wireless sensor network testbed deployed at the National University of Singapore. It uses TelosB devices and is built on an active-USB infrastructure. Tutornet1 is a testbed running in Ronald Tutor Hall at University of Southern California. It consists of 13 clusters, each cluster composed of a stargate and several motes attached via USB cables. A central PC can program any mote in the testbed through the communication with these stargates over WiFi.

PE-WASUN

Such shared testbeds have a couple of shortcomings. First, they are shared by nature: one has to wait for the motes to become available before being able to start an experiment. Second, they only feature a handful of types of motes (hardware) and its hard -if at all possible -to swap the hardware used. Third, they are deployed in a single location, which is not representative of the variety of environments low-power networks are deployed in. Forth, these testbeds are complex and expensive to install, and often require a team of full-time engineers or grad students to keep them up and running. And finally, these testbeds are academic/open by nature, and not appropriate for in-house developments or proprietary technology. The goal of the Rover architecture presented in this paper is to address these shortcomings.

Rover combines the strengths of the OpenMote hardware and the OpenWSN software. OpenMote 2 [START_REF] Vilajosana | OpenMote: Open-Source Prototyping Platform for the Industrial IoT[END_REF] is an openhardware prototyping ecosystem designed to accelerate the development of the Industrial Internet of Things (IIoT). The OpenMote-CC2538 is the core of this ecosystem. It is a 4 cm × 3 cm board which provides computation and communication capabilities. Its main component is the Texas Instrument CC2538, a System-on-Chip (SoC) with a 32-bit ARM Cortex-M3 micro-controller and a 2.4GHz IEEE802.15.4 radio transceiver. OpenWSN 3 [START_REF] Watteyne | OpenWSN: a Standards-based Low-Power Wireless Development Environment[END_REF] provides an open-source implementation of a fully standards-based protocol stack for the IoT, running on a variety of hardware and software platforms. With a suite of free and open-source debugging and integration tools, it aims to help academia and industry verify the applicability of these standards to the Internet of Things. It is the reference open-source implementation of the IEEE802.15.4-2015 Time Synchronized Channel Hopping (TSCH) standard and IPv6 over TSCH (6TiSCH). Fig. 2 depicts the protocol stack implemented in OpenWSN. OpenMote and OpenWSN have been designed together.

The Raspberry Pi 4 is a credit-card sized single-board computer, which aims at providing low-cost, high-performance computers to stimulate the education of basic computer science at schools. It features a Broadcom BCM2835 SoC, and uses an SD card for booting and long-term storage. The different Debian and Arch Linux flavors that run on it comes with a Python interpreter. It has become very popular gateway computer for IoT networks. Rover uses it as the default platform to connect OpenMote boards to.

The goal of Rover is to offer all the pieces necessary for building a testbed which is easily (re-)deployable, cheap and running cutting-edge IOT protocols and standards. The Rover testbed integrates the Raspberry Pi, OpenMote and 2 www.openmote.com 3 www.openwsn.org 4 www.raspberrypi.org OpenWSN projects.

The contribution of this paper is threefold:

• It presents Rover, an easy-to-deploy, cheap, standardsready testbed for the IoT.

• It shows an example deployment of a Rover testbed at Cisco-Paris.

• It discusses cases for which this type of testbed in particularly suited.

The remainder of this paper is organized as follows. Section 2 introduces the OpenVisualizer, the software running on the PC controller the testbed. Section 3 presents the Rover architecture, highlights the components developed as an contribution to the OpenVisualizer, and lists the key features of the Rover testbed. Section 4 describes how a Rover tesbed is currently deployed at Cisco-Paris. Section 5 discusses example use cases in which Rover-based experimentation is particularly suitable. Finally, Section 6 concludes this paper and discusses future work.

OPENVISUALIZER

The OpenVisualizer is a Python-based debugging and visualization tool that runs on a Windows/Linux/OSX computer and interacts with the motes connected to it over serial ports. It includes 6LoWPAN Low-power Border Router (LBR) functionality, i.e. it translates 6LoWPAN packets into IPv6 packets, thereby connecting the low-power wireless mesh to the Internet. Its web-based user interface provides relevant information about the OpenWSN network. This includes the internal states of each mote (neighbor table, communication schedule, routing table, a.o.), the multi-hop routing structure, and the debug/error messages generated by the motes. The OpenVisualizer also sends commands and data to the motes, in particular packets coming from the Internet.

The OpenVisualizer is built following the "Event Bus" software design pattern. Components connected to the Event Bus use a publish-subscribe approach to send messages to one another. This modular software architecture makes the code maintainable, flexible and extensible. The Event Bus has proven to be a powerful tool to write applications and support the ongoing development of communication standards and protocols.

Fig. 3 depicts the general architecture of the OpenVisualizer. The Event Bus provides the messaging framework for the full chain of services between a mote and the external network. Components are conceptually grouped into the functional areas shown on the bus. Each component implements an event handler and provides a specific service. The openTun component creates and uses an IPv6 virtual TUN interface, allowing motes to communicate with external applications running on the Internet. LBR is the component that provides low-power border router translation between IPv6 packets on the external network, and 6LoWPAN packets in the low-power wireless mesh. The RPL component manages the multi-hop routes by implementing the RPL standard. It receives the RPL DAO messages sent by the motes, and maintains a central view of the RPL topology; it uses that view to build source routes for packet sent into the mesh. Each moteConnector is the primary interface for an individual mote; it pushes events out from the mote onto the bus, for other components to display/interpret. Each moteConnector communicates with a mote over a serial port using a moteProbe component. For each message received from the mote, the moteProbe encapsulates it as an event which it publishes on the Event Bus. Similarly, the moteProbe subscribes to fromMoteConnector@{serial} notifications at the Event Bus, to forward them to the mote over its serial port. The serial frames between the mote and the computer are framed using HDLC, implementing RFC1662 [START_REF] Simpson | PPP in HDLC-like Framing[END_REF]. The payload of the HDLC frame has a leading 1-byte "type" field which indicates the format of the followed payload.

With each mote connected to the OpenVisualizer through the serial, and almost all the necessary information about the low-power wireless mesh fully visualized, it becomes easy to conduct IoT research and development activities. Yet, it requires for all the nodes to be physically connected to a same computer, severely limiting the scalability of the approach.

THE ROVER TESTBED

In any realistic deployment, the different motes of the mesh are not close enough to be connected directly to the same computer running the OpenVisualizer. What is needed is a mechanism to allow some motes to connect remotely. This paper presents the improvement to the OpenVisualizer which allows just that. We implemented the "Rover" module, a version of the moteProbe which is connected to the serial port of the mote, and connects to the main Event Bus of the OpenVisualizer remotely.

Architecture

The new architecture can be divided into two parts, with most components running in the central OpenVisualizer, and the moteProbe components running remotely on each Rover node. The architecture defines a new remoteConnec-torServer component for the OpenVisualizer, and a re-moteConnectorRover component for the Rover node. These bridge the signaling data between the motes and the central computer, without removing any of the OpenVisualizer functionality. Fig. 4 shows the resulting OpenVisualizer architecture.

The original moteProbe component still remains available; the new architecture supports both local and remote motes, concurrently. RemoteConnnectorServer is the new Open- Visualizer component which allows the mote to connect remotely. It maintains a TCP session between each Rover node and the main OpenVisualizer. Through that session, it transfers all the events destined to a remote Rover node. Simlarly, it receives incoming events from the Rover nodes, and dispatches them in the local Event Bus. No other changes to the core components of the OpenVisualizer were necessary.

The remoteConnectorServer communicates with each re-moteConnectorRover using ZeroMQ5 . ZeroMQ is an asynchronous messaging system which offers one-to-many communication between the OpenVisualizer and the Rover nodes. Components exchange messages using a publish-subscribe pattern: the remoteConnectorServer publishes events the Rover nodes subscribe to, and subscribes to all the Rover nodes to receive their events. The result is that the Event Bus of OpenVisualizer is extended remotely to each Rover node; all events are transparently transferred between components, whether local or remote.

Additional Features

The Rover functionality is built into the OpenPi platform 6 , the OpenWSN-ready distribution for the Raspberry Pi. In the OpenPi, an OpenVisualizer Rover program runs as a service called "openrover", and serves socket connection requests from the central OpenVisualizer.

Once the remote connection is established, the OpenVisualizer is able to monitor and manage all the motes, regardless of whether they are local or remote. It also allows the user to remotely reset or reflash the motes with any arbitrary firmware.

The Rover code was contributed to the OpenWSN project, and is available under an open-source BSD license7 . the software running on the Raspberry Pi The moteProbe component is equivalent to that running in the OpenVisualizer. It is the remoteConnectorRover component which maintains the (remote) connection to the OpenVisualizer The remoteConnectorRover component subscribes to the fromMoteProbe@{serial} events from the local Event Bus, and forwards them to the OpenVisualizer. Similarly, the remoteConnectorRover component republishes fromMote-Connector@{serial}@{roverIP} events received from the OpenVisualizer into local Event Bus. The remoteConnectorRover communicates with the re-moteConnectorServer on the OpenVisualizer using ZeroMQ. Each time a Rover node connects, the OpenVisualizer creates dedicated moteState and moteConnector components, identified by the "@{roverIP}" and "@{serial}" suffixes. From that moment on, Rover node and the OpenVisualizer exchange all relevant information.

EXAMPLE DEPLOYMENT

OpenMote

Table 1 gives an approximate cost of a Rover node. A Rover-based testbed is currently deployed on the sixth floor of the Cisco Innovation and Research Lab in Paris, France. Fig. 6 shows the 8-node topology, each label are the last 2 bytes of the OpenMote's MAC address, in hexadecimal notation. Mote 9E-D9 is the DAGroot, and is connected to the Raspberry Pi which serves as the central computer running the OpenVisualizer. The remaining nodes remote, each connected to a Raspberry Pi running the OpenRover software. For this experiment, The topology is hardcoded in a general shape of a ladder, and each mote check at layer 2 if each received packet comes from one of their neighbors, if not the packet is simply dropped.

EXAMPLE USE CASE

This section illustrates how the Rover testbed is used to conduct research on 6TiSCH [START_REF] Thubert | An Architecture for IPv6 over the TSCH mode of IEEE[END_REF] schedule management with Traffic Engineering for Bit Indexed Explicit Repliation (BIER-TE).

BIER (Bit Indexed Explicit Replication) is a new multicast forwarding protocol [START_REF] Wijnands | Multicast using Bit Index Explicit Replication[END_REF]. When a multicast data packet enters a BIER domain, the ingress router determines a set of egress routers to which the packet needs to be sent, and then encapsulates the packet in a BIER header. The BIER header contains a bitstring in which each bit represents exactly one egress router in the domain. To send a packet to a particular set of egress nodes, the ingress node sets the bits for each of those egress nodes, and clears other bits in the bitstring. Each packet can then be forwarded along the unicast shortest path tree from the ingress node to the egress nodes based on some routing protocol. BIER-TE [START_REF] Eckert | Traffic Engineering for Bit Index Explicit Replication BIER-TE[END_REF] is a Traffic Engineering technique inspired from BIER and being standardized. It forwards and replicates packets based on BIER-like bitstrings, but does not require an routing protocol. The key differences over BIER are: (1) BIER-TE replaces in-network, autonomous, and IGP-based path calculation by explicit paths calculated off-path by a controller; (2) in BIER-TE, every bitposition of the bitstring of a BIER-TE packet indicates one or more adjacencies -instead of a set of destination nodes as in BIER;

(3) in BIER-TE, each node has no routing table but only a BIER-TE Forwarding Table (BIFT). The BIFT specifies how the node should replicate packets to its adjacencies according to the bitstring contained in the BIER header of the packet. BIER-TE can thus be used to enable path diversity by controlling replication and elimination by tuning the bitstring in the BIER header. The output bitstring from the last node in path can be used to identify transmission failures, and this information can be passed to the central controller, which in turn can modify the bitString for the next packets.

To explore the performance of BIER-TE on a 6TiSCH lowpower wireless mesh, we implemented BIER-TE on Open-WSN and evaluated its performance on the Cisco-Paris Rover testbed. Fig. 7 plex path, reserves timeslots, and installs a BIFT into each node over their serial port. An associated bitmap for this flow is then assigned by the OpenVisualizer in the ingress node, and inserted into the BIER header of each packet. Each node along the path receiving the packet examines the bitmap against its BIFT and performs packet forwarding, replication or elimination operations. When the destination node receives the packet, it forwards the payload data to the upper layer and feeds the bitmap and delay back to Open-Visualizer. With the Rover testbed, various experiments are conducted to benchmark the performance of BIER-TE over 6TiSCH. For each experiment, the code is instrumented to measure jitter, latency, loss ratio and reliability. The Rover testbed thus proved itself essential to conduct this real-world experimental study. It integrates both the advanced OpenWSN software and the state-of-the-art Open-Mote hardware. It includes an open-source and up-to-date implementation of a complete protocol stack based on IoT standards, which makes the testbed complete and powerful. The real-time network visualization makes it simple and easy to use. The OpenVisualizer offers network monitoring and management capabilities, no matter whether the mote is local or remote, real or simulated. The fact that each Rover node can be positioned anywhere while maintaining the connection with the central OpenVisualizer is a critical feature. Because of their Ethernet connectivity, and the low cost of the nodes, the testbed can be redeployed in minutes. We strongly believe the Rover architecture is key tool for conducting benchmarking and feasibility studies, in particular for the standards for the Industrial IoT.

CONCLUSION

This paper introduces Rover, an IoT testbed which combines OpenWSN software and OpenMote hardware in an easy-to-use, flexible and cheap solution. It offers an ideal real-world experimentation environment for both academic and industry R&D activities, and will help shape the future of the Internet of Things. It improves to the OpenVisualizer software by allowing node to connect remotely. Through the connection, the OpenVisualizer and Rover node exchange signaling and data.

We are currently working on a mechanism for the Open-Visualizer to automatically discover the Rover nodes which are in the local network, either through mDNS, or by using a registration mechanism to a resource directory. We are also working towards a Rover node which uses an Intel Edison rather than a Raspberry Pi. This will allow Rover nodes to connect to WiFi (at 5 GHz) rather than Ethernet, offering even more flexibility.

  '16 13-17 November 2016, Valletta, Malta c 2016 ACM. ISBN XXX-XXXX-XX-XX/XX/XX. . . $15.00 DOI: XX.XXX/XXX X

Figure 1 :

 1 Figure 1: A Rover node.

Figure 2 :

 2 Figure 2: The OpenWSN protocol stack.

Figure 3 :

 3 Figure 3: OpenVisualizer without Rover.

Figure 4 :

 4 Figure 4: OpenVisualizer with Rover.

Fig. 1 Figure 5 :

 15 Fig.1shows a Rover node, consisting of an OpenMote and a Raspberry Pi. Fig.5show the block diagram of

Figure 6 :

 6 Figure 6: Bird view of the Rover testbed running at Cisco-Paris.

Figure 7 :

 7 Figure 7: Using the Rover testbed to benchmark BIER-TE on OpenWSN.

Table 1 :

 1 Approximate cost of a Rover node.

	-CC2538	75 e
	Raspberry Pi 3	35 e
	enclosure	5 e
	SD card	15 e
	RJ45 cable	5 e
	Total	135 e

http://testbed.usc.edu/

http://zeromq.org/

openpi.openwsn.org

As an online addition to this paper, all source code is available at https://github.com/openwsn-berkeley/openpi under a BSD open-source license.

Acknowledgment

This work was partly supported by the European Commission's Horizon 2020 Framework Programme, through the H2020 F-Interop and H2020 ARMOUR projects.

ADDITIONAL AUTHORS

Additional author: Geraldine Texier (Telecom Bretagne, email: geraldine.texier@telecom-bretagne.eu).


Sébastien Ziegler
email: sziegler@mandint.org

Serge Fdida
email: serge.fdida@lip6.fr

Thomas Watteyne
email: thomas.watteyne@inria.fr

César Viho
email: cesar.viho@irisa.fr

F-Interop -Online Conformance, Interoperability and Performance Tests for the IoT

Keywords: Internet of Things, tests, testbed as a service, interoperability, conformance, performance, scalability, privacy

F-Interop is a platform for online and remote conformance, interoperability and performance tests for the Internet of Things (IoT). This paper outlines the challenges faced by the IoT and its online testing, and how F-Interop addresses them. F-Interop provides an extensive experimental platform for online testing. This paper presents an overview of its overall architecture, and an initial set of results.

Introduction and Project Presentation

The Internet of Things (IoT) is widely considered the next technological revolution impacting all application domains. It is touted to be massive and pervasive, with 50 to 100 billion smart "things" connected by 2020 [START_REF] Ericson | More than 50 Billion Connected Devices[END_REF]. It will impact diverse application domains, including smart cities, agriculture, industries and eHealth [START_REF] Ziegler | IoT6 -Moving to an IPv6-Based Future IoT[END_REF].

Since 1995, interoperability is recognized by the International Telecommunication Union (ITU) as the main obstacle for IoT development and market adoption. The success of this new technological revolution will depend on its ability to overcome its current fragmentation. Overcoming fragmentation requires standardization and interoperability [START_REF] Ziegler | Internet of Things -From Research and Innovation to Market Deployment[END_REF].

F-Interop1 is a European research project addressing this challenge. It develops an platform for online and remote conformance, interoperability and performance testing for the IoT.

This paper gives an overview of the F-Interop platform; it is organized as follows. Section 2 outlines the problem F-Interop addresses. Section 3 presents the technical approach taken. Section 4 shows the architecture of the F-Interop platform. Section 5 discusses the methodology, the current results, and the remaining tasks. Section 6 outlines the expected impact of F-Interop project. Finally, Section 7 concludes this paper.

Problem Statement

In order to be widely adopted, new technologies, products and solutions typically go through the following steps:

-Standardization: stakeholders discuss and align their views, and converge towards common standards and specifications. -Conformance & Interoperability: stakeholders test and validate that their implementations conform to the standard(s) it implements, and that they can communicate with one another. -Optimization: in terms of the Quality of Service, scalability or energy consumption. -Market Launch: the solution is ready for roll-out.

Developing a product requires extensive testing. This is all the more true for interoperability testing, which verifies that two independent implementations work together. Traditionally, interoperability testing is done through face-to-face "interop events". Several vendors meet and go through a list of interoperability tests, each one verifying a particular aspect of the implemented standard.

This approach has the following drawbacks:

-It requires engineering teams to travel across the globe, often only to find out they need to make a minor fix. This has a high cost, both in terms of time and money. That cost is often too high for SMEs. -Time-to-market is stretched, as interop events typically happen every 4-6 months. The result is that vendors who want to adopt emerging standards have a disadvantage compared to vendors who go to market with entirely proprietary solutions.

The goal of F-Interop is to allow compliance and interoperability testing to be both online and remote. "Online" means that the testing is orchestrated by a server somewhere on the web, to which users connect from their offices. "Remote" means that interoperability tests between different vendors can happen through the server, i.e. with requiring the teams to be in same room.

F-Interop targets two audience:

-SMEs and companies. F-Interop allows faster development of standards-based interoperable products, with a time-to-market cut by 6-12 months, and significantly lower overhead. -Standardization Development Organizations (SDOs). The communities contribute test cases to F-Interop, ensuring better compliance of products to their standards, faster adoption, and eventually a larger portion of standards-based products.

Besides allowing direct interoperability testing between vendors, F-Interop interfaces to the European FIRE research infrastructure2 , which includes a set testbeds. This allows F-Interop to run tests at scale against a large number of IoT devices at once.

Technical Approach

The goal of F-Interop is to extend FIRE+ with online interoperability and performance testing tools supporting emerging IoT-related technologies from research to standardization and to market launch. This benefits researchers, product development by SME, and standardization processes. F-Interop combines the complementary approaches listed below.

Online Testing Tools. F-Interop develops online testing tools for the IoT, enabling to test interoperability, conformance, scalability, Quality of Service (QoS), the Quality of Experience (QoE), and energy efficiency of IoT devices and services.

Testbeds federations with a shared "Testbed as a Service". F-Interop brings together 3 testbed federations and facilities, encompassing over 32 testbeds and 4755 IoT nodes: -Fed4FIRE3 , which federates 24 FIRE+ testbeds, bringing together cloud, IoT, wireless, wireless mobile, LTE, cognitive radio, 5G, openflow, SDN, NFV and network emulation technologies. -OneLab4 , which federates testbeds for the future Internet, including IoT, cognitive radio, wireless and overlay network technologies. -IoT Lab5 , which federates IoT and crowdsourcing/crowd-sensing testbeds, including smart campus, smart building and smart office testbeds.

In order to support this integration, F-Interop extends the testbed federation architecture model with a new layer, enabling shared services among several testbed federations. This approach enables to interface "Testbed as a Service" (TBaaS) with existing federations through a clearly specified API, enabling remote access and interaction with various experimental platforms.

Supporting IoT Standardization and Industry. F-Interop works in close collaboration with several standardization bodies, and directly contributes to three IoT standardization processes: oneM2M, 6TiSCH (IETF) and the Web of Things (W3C). It also explores the possibility to support and enable new online certification and labeling mechanisms, including the IPv6 Ready logo 6 . More generally, F-Interop intends to enable an easier participation of researchers and industry in standardization processes. It runs an open call for SMEs and developers, inviting them to use and enrich the platform with additional modules and extensions.

Flexible Testing Schemes. F-Interop is researching and exploring various testing schemes and configurations, by interconnecting devices under tests with the server testing tools, resources provided by the F-Interop connected federations of testbeds, and resources provided by other users. This is illustrated in Fig. 1, where a hexagon represents an Implementation Under Test (IUT).

Initial Architecture

The initial architecture of F-Interop is presented in Fig. 2. A companion paper [START_REF] Leone | Technical Overview of F-Interop[END_REF] provides a detailed technical overview of this architecture.

Methodology, Current Results and Remaining Tasks

The first step has been to define the F-Interop platform. It is composed of the following three testbeds of FIRE+: IoT-Lab, Fed4FIRE and OneLab. The F-Interop platform will be extended with additive testing tools, devices and testbeds.

The F-Interop platform identifies different types of actors. An F-Interop "Contributor" is an entity that provides testing tools into the platform, as well as testbeds and devices that are added to extend the existing testbeds. An F-Interop "User" is an entity that has an IoT device, system or application to be tested (an IUT) and wants to use the F-Interop platform to test it. We consider two types of testing tools: online interoperability testing and online performance testing. We target emerging IoT technologies that cover several layers/aspects of the IoT protocol stack, and focus first on IETF 6TiSCH [START_REF] Dujovne | 6TiSCH: Deterministic IP-enabled Industrial Internet (of Things)[END_REF] and IETF CoAP [START_REF] Shelby | The constrained application protocol (coap)[END_REF]. The goal is to use the F-Interop (online) platform for conducting the same tests as conducted during face-to-face interoperability events. Three 6TiSCH plugtests and four CoAP plugtests were organized by ETSI, each with a test description (a list of interoperability tests to execute). The F-Interop platform is built to be able to execute those tests in a remote and online manner. Fig. 4 outlines the resulting overall architecture of the F-Interop platform.

Fig. 5 illustrates how the F-Interop platform can be used for remote online interoperability testing. It distinguishes the control plane -which manages the interactions between components -from the data plane -which executes the tests. A first proof-of-concept has been developed for CoAP online remote interoperability testing [START_REF] Zack | F-Interop: Online Interoperability, Performance and Scalability Tests for the Internet of Things [demo[END_REF]. Similar proof-of-concept developments for 6TiSCH and for oneM2M are under development.

Several challenges remain to be addressed: -Standardization communities, enabling them to save time and resources, and be more inclusive with partners who cannot afford traveling, and to accelerate standardization processes. -SMEs and companies, enabling them to develop standards-based interoperable products with an estimated time-to-market cut by 6-12 months, and significantly lowered engineering and financial overhead. -Academic communities, by providing a new set of online tools to support their own research and development, as well as by reducing the financial barrier to their participation in standardization processes.

Moreover, F-Interop will provide standardization support to several emerging IoT standards at the IETF (with an initial focus on 6TiSCH), ETSI (with a focus on oneM2M) and W3C (with a focus on Web of Things). The F-Interop architecture is designed to be easily extensible to new standards. More generally, faster availability of standards-based products is expected to result in wider adopt standards and a larger portion of interoperable products.

Conclusion -towards online interop and performance tests

In order to support IoT research, development and adoption, F-Interop develops a platform for online testing tools encompassing interoperability, compliance and performance tests. It is progressing towards a new model of interoperability test, enabling a larger participation with remote online testing.

The F-Interop platform is still being developed. However, it already announces an open call with funding to develop new partnerships with third parties research projects interested to develop complementary testing tools, address additional standards and/or organize F-Interop based interoperability tests in standardization processes.

Fig. 1 .

 1 Fig. 1. Multiple testing schemes.

Fig. 2 .

 2 Fig. 2. Initial Architecture.

Fig. 3 .

 3 Fig. 3. F-Interop main components.

Fig. 4 .

 4 Fig. 4. F-Interop general architecture.

Fig. 5 .

 5 Fig. 5. F-Interop online interoperability testing architecture.

http://www.f-interop.eu/

http://www.ict-fire.eu/

http://www.fed4fire.eu/

https://onelab.eu/

http://www.iotlab.eu/

https://www.ipv6ready.org/

Acknowledgments

The work presented in this article is supported by the F-Interop European research project, under the Horizon 2020 Framework Program supported by the

European Commission. It is implemented by a consortium coordinated by UPMC and Mandat International. F-Interop capitalizes on other projects results, including IoT Lab, Fed4FIRE, and OneLab.


Keoma Brun-Laguna

Ana Laura Diedrichs

Javier Emilio Chaar

Diego Dujovne

Juan Carlos Taffernaberry

Gustavo Mercado

Thomas Watteyne

A Demo of the PEACH IoT-based Frost Event Prediction System for Precision Agriculture

Keywords: Smart Agriculture, Precision Agriculture, IoT, SmartMesh IP

come

I. RELATIONSHIP TO PREVIOUS WORK

This demo presents the PEACH project and the associated deployment running in Argentina. The PEACH project has been previously published [START_REF] Watteyne | PEACH: Predicting Frost Events in Peach Orchards Using IoT Technology[END_REF]. Some text and figures in this extended demo abstract are borrowed from that publication.

II. OVERVIEW

In 2013, 85% of the peach production in the Mendoza region (Argentina) was lost because of frost. Because less fruit was produced in the region, 600,000 less work days were needed to process the harvest between November 2013 and March 2014, a reduction in work force of 10,600 people. Across the Mendoza region, frost caused a loss of revenue of 950 million Argentine pesos, roughly 100 million USD (at that time) in the peach business alone. A frost event happens when the temperature is so low that crops cannot recover their tissue or internal structure from the effects of water freezing inside or outside the plant.

Agronomists need precise and representative measurements to understand the impact of frost events on the plants. Today, virtually all industrial peach orchards are equipped with a meteorological station that monitors temperature and humidity. As the meteorological station only measures the environment in one location, this solution lacks spacial representation. Frost countermeasures exist, such as placing heaters across There is a need for a dense monitoring system to gather data that really represents the orchard environment. Because farmers use heavy machinery, using wires is not an option. We choose to use a low-power wireless mesh network to interconnect sensors densely deployed in the orchard. The security and reliability of those wireless networks must be equivalent to that of its wired counterpart. Our target battery lifetime for the devices is "at least a year", the natural grow cycle of peaches. Our goal is to retrieve representative measurements that agronomists can analyze to understand and predict frost events in peach orchards. What we build is a system with a large number of sensing points (air temperature, air relative humidity, soil moisture, soil temperature), at different elevations/depths throughout the orchard.

III. THE PEACH DEPLOYMENT

The PEACH network consists of 21 low-power wireless motes deployed in a peach orchard in Junín, a city close to Mendoza, in the West of Argentina. We use SmartMesh IP motes produced by Linear Technology as it offers best-in-class >99.999% end-to-end reliability and can run over a decade on a pair of AA batteries [START_REF] Watteyne | Industrial IEEE802.15.4e Networks: Performance and Trade-offs[END_REF]. A temperature sensor is built into the motes, publishing a measurement every 30 s. The motes themselves produce network statistics (called "Health Reports") such as the number of neighbors or the amount of charge consumed by the mote, every 15 min. All the information produced by the network (sensor measurements and network statistics) is sent to the network gateway, a SmartMesh IP Manager.

We call "back-end" the system that connects to the SmartMesh IP Manager, retrieves all the information produced by the mesh network, and sends those to a remote database. The back-end is composed of two software components, the solmanager and the solserver. A Raspberry Pi singleboard computer is connected to the SmartMesh IP Manager over USB, and runs the solmanager application that serves as an interface between the mesh network and the Internet. The solmanager transmits all the information the SmartMesh IP Manager receives to a remote server located in the Inria-Paris research center. This server runs the solserver application. The solserver application stores the data into a database and allows users to visualize the sensors measurements seconds after they were acquired in the orchard in Argentina. We use a Grafana dashboard (Fig. 3) to show the measurements, and a map to show the network (Fig. 1), in real-time 1 .

The solserver supports multiple solmanager instances. This means that we can easily deploy a new mesh network and, by running a new solmanager application, forward all the mesh data to the existing solserver. Each mesh network is called a "site"; we show in this demo how to add a new site.

All the deployment details can be found on the project website 2 .

The PEACH deployment uses the "Sensor Object Library" (SOL), a library we developed to use a generic representation for the data produced by the mesh network. The format of the data generated by the SmartMesh IP network depends on the application that runs on the motes. We created the SOL library to wrap the information generated by the SmartMesh IP as SOL objects, composed of the following fields: the identifier of the device generating the information, the timestamp of when the information was generated, the type of information and the value of the information. A SOL object can be seen as a generalization of the Type-Length-Value (TLV) format. The SOL registry contains the list of SOL types currently used, and their associated format. The SOL registry is publicly maintained and can be easily augmented with new types of sensor data 3 .

Both the solmanager and solserver applications use the SOL library. As the data transits from the mesh network to its final destination (i.e. the database), it is subject to multiple format changes. The solmanager turns the SmartMesh IP data into easy-to-manipulate SOL objects in JSON format, then turns those SOL objects in a lightweight binary format to be sent to the solserver. The solserver turns the objects back into JSON format and parse them before storing them into the database.

V. PERFORMANCE RESULTS

We know how the network behaves from the Health Reports retrieved at the SmartMesh IP Manager. After 3 weeks of deployment the mesh network has a 100% end-to-end reliability, 97% stability, 800 ms latency.

The network reliability indicates the portion of the packets generated by the motes that reach the SmartMesh IP manager. The motes have generated 483,114 packets, all successfully received by the SmartMesh IP manager (possibly after multiple hops), yielding 100% reliability.

The network stability represents the average Packet Delivery Ratio (PDR) over all wireless links. The motes have sent 3,009,962 data-link frames, 93,985 unsuccessfully (i.e. no link-layer acknowledgment was received), yielding a network stability of 97%. This number is very high, indicating that the nodes are deployed close enough in an environment with little external interference and multi-path fading.

The network latency represents how long a packet is traveling in the multi-hop wireless mesh network. In the PEACH deployment, it takes on average 800 ms for a sensor measurement to travel from the mote that generated it to the SmartMesh IP manager, over the multi-hop low-power wireless mesh network. The latency depends on the number of hops. Currently, the furthest mote is 5 hops away from the manager; it has a latency of 1540 ms. This latency is small compared to the variation speed of the signal we are observing (temperature).

VI. RELATED DEPLOYMENTS

An equivalent network is deployed for monitoring the plants in the UC Berkeley Botanical Garden 4 . This botanical garden has over 13,000 different kinds of plants from around the world cultivated over an area of 34 acres. The different needs of each plant makes the irrigation a complex task; the mesh network is deployed to help the local botanist team get in-situ measurements.

The ARHO project is adopting the back-end system presented in this extended abstract. In the ARHO project, 945 sensors are deployed in the California Sierra Nevada to monitor the snow-pack [START_REF] Zhang | American River Hydrological Observatory: Monitoring the Sierra Nevada Snowpack[END_REF]. The sensors are organized in 15 low-power wireless 15 networks, each one covering 1-2 km 2 .

VII. THE PEACH DEMO

During the demo, we replicate the PEACH network presented in Section III. We deploy SmartMesh IP motes across the conference venue. They form a low-power wireless mesh network around the SmartMesh IP Manager, which is connected to a computer. Sensor measurements and network statistics are forward by this computer to the back-end system running at Inria-Paris. Visitors can see the data live at https://sol.paris.inria.fr/. During the demo, we have the system reset; visitors witness how the reliable wireless mesh network is formed. A picture of the different elements shown during the demo is shown in Fig. 3.

This demo presents the technology as a turn-key solution. We are interested in partnering with end users and integrators to replicate the PEACH network to other deployments and applications.

4 "Student-installed Sensors Help Monitor Botanical Garden", http://news. berkeley.edu/, 20 October 2015.

VIII. LESSONS LEARNT, ONGOING WORK AND ROADMAP

A. Lessons Learnt

Although frost events may appear harmless, they yield enormous losses in fruit production, for example peaches. As seen throughout this project, IoT technology is ready for this type of precision agriculture applications. The main outcome of this project is a perfectly working end-to-end lowpower wireless distributed sensor system, built exclusively from commercial off-the-shelf components. Deploying such a network took a couple of hours, putting the whole hardware and software together a couple of days.

In a decade, wireless sensor applications went from boards programmed from scratch with a combination of the most promising research papers, to using off-the-shelf products implementing standards-based fully working and mature technology. Wireless Sensor Networks/Internet of Things technology successfully transitioned from the academic to the commercial world.

B. Ongoing Work and Roadmap

A lot still needs to be done. The first step is to add additional sensors to the motes, including relative air humidity, soil moisture, soil temperature and soil conductivity. We then plan to apply machine learning techniques, in particular ensemble regression-tree, on the large-scale network statistics and measurements dataset. On the application side, we want to be able to identify frost events patterns and thus, be able to predict them. On the networking side, we are particularly interested in studying the maximal capacities of the IEEE802.15.4e "Time Synchronized Channel Hopping" (TSCH) standard that is used in those deployments.

The combination of technologies is perfectly suited for micro-climatic modeling in precision agriculture, and we believe it is also perfectly suited for other fields. We are looking to apply this proven technology in other precision agriculture applications and to extend this work to other domains, including infrastructure monitoring and building automation.

Fig. 1 .

 1 Fig. 1. The nodes and wireless links in the PEACH deployment.

Fig. 2 .

 2 Fig. 2. Architecture of the PEACH network.

Fig. 3 .

 3 Fig. 3. The setup of the PEACH demonstration, including the motes, the manager and the web interface of the back-end system.

https://sol.paris.inria.fr/

https://www.savethepeaches.com/

https://github.com/realms-team/sol/blob/master/registry.md

ACKNOWLEDGMENTS

The authors thank the Dust Networks product group at Linear Technology and Metronome Systems for their technical support. This publication was made possible by the support of the STIC-AmSud program, under grant 16STIC08-PEACH and by Inria, through the REALMS associate team.


Thomas Durieux

Benoit Cornu

Lionel Seinturier

Martin Monperrus

Dynamic Patch Generation for Null Pointer Exceptions Using Metaprogramming

Null pointer exceptions (NPE) are the number one cause of uncaught crashing exceptions in production. In this paper, we aim at exploring the search space of possible patches for null pointer exceptions with metaprogramming. Our idea is to transform the program under repair with automated code transformation, so as to obtain a metaprogram. This metaprogram contains automatically injected hooks, that can be activated to emulate a null pointer exception patch. This enables us to perform a fine-grain analysis of the runtime context of null pointer exceptions. We set up an experiment with 16 real null pointer exceptions that have happened in the field. We compare the effectiveness of our metaprogramming approach against simple templates for repairing null pointer exceptions.

I. INTRODUCTION

Null pointer exceptions are the number one cause of uncaught crashing exceptions in production [START_REF] Li | Have Things Changed Now?: An Empirical Study of Bug Characteristics in Modern Open Source Software[END_REF]. Li et al. found that 37.2% of all memory errors in Mozilla and Apache are due to null values [START_REF] Li | Have Things Changed Now?: An Empirical Study of Bug Characteristics in Modern Open Source Software[END_REF]. Other studies 1 found that up to 70% of errors in Java production applications are due to null pointer exceptions. It is an inherent fragility of software in programming languages where null pointers are allowed, such as C and Java. A single null pointer can make a request or a task to fail, and in the worst case, can crash an entire application. Program repair [START_REF] Monperrus | Automatic Software Repair: a Bibliography[END_REF] is a research field concerned with the generation of patches. There are a number of program repair techniques (eg. [START_REF] Le Goues | Genprog: A generic method for automatic software repair[END_REF]) but not one of them is dedicated to null pointer exceptions.

One way of fixing null pointer exceptions is to use a template [START_REF] Kim | Automatic patch generation learned from human-written patches[END_REF]. For instance, one can reuse an existing local variable as follows:

+ if (r == null) { + r = anotherVar; + } r.foo(p);
This example illustrates a template that depends on the context (called parametrized template in this paper): the reused variable is the template parameter. When repairing a statically typed language, such as Java that we consider in this paper, the static type of the template parameter has to be compatible with the variable responsible for the null pointer exception. This is done with static analysis of the code.

However, there may be other variables in the context of the null pointer exception, typed by a too generic 1 http://bit.ly/2et3t79, last accessed Oct 19 2016 class, which may be compatible. The static analysis of the context at the location of the null pointer exception may not give a complete picture of the objects and variables that can be used for fixing a null pointer exception. There is a need to make a dynamic analysis of the repair context. This is the problem we address in this paper.

To overcome this problem, we propose to perform a dynamic analysis of the repair context. This completely changes the way we perform patch generation: instead of applying templates one after the other, we propose to create a metaprogram that is able to dynamically analyze the runtime context, and to dynamically alter the execution of the program under repair to identify a working patch. The resulting technique is called NPEfix. It takes as input a test case that triggers an uncaught null pointer exception and outputs a patch that fixes it if one exists (or several alternative patches if several valid patches exist). NPEfix is composed of three main phases: first, it generates a metaprogram out of the program under repair, second it compiles the metaprogram, third, it runs the failing test case a large number of times, each time altering the behavior to find a way to avoid the null pointer exception.

NPEfix is based on repair strategies for null pointer exceptions. Reusing an existing variable, as seen above, is one such strategy. NPEfix uses a total of 9 different strategies that are categorized in two groups. The first group is about providing an alternative value when a null pointer exception is about to happen. This value can come from elsewhere in the memory (i.e. a valid value that is stored in another variable), or it can be a new object. The second group of strategies is about skipping the execution of the exception-raising expression. It can be either skipping a single statement or skipping a complete method.

In NPEfix, we use a metaprogramming approach -a code transformation -for each of those strategies. All transformations are compatible one with another and can be used in conjunction. Then, we explore the search space of tentative patches for null pointer exceptions by activating the hooks of the metaprogram which have been injected by transformation. Compared to a template based approach, the novelty of our approach is to explore the search space of tentative patches purely at run-time, which gives us a way to finely analyze the runtime context of the null pointer exception.

To evaluate our approach, we build a benchmark of 16 null pointer exceptions that happened in the field for open-source software and were all reported in bug trackers. For each of those bugs, we create the NPEfix metaprogram, and explore the search space to identify patches that are able to avoid the null pointer exception to happen.

To sum up, the contributions of this paper are:

• A taxonomy of 9 alternative execution strategies to repair null pointer exceptions (Section II). • A set of code transformations for implementing those null-pointer repair strategies in Java (Section III).

• The systematic evaluation of our approach on [START_REF] Monperrus | Automatic Software Repair: a Bibliography[END_REF] real null pointer exceptions that we were able to reproduce (Section IV). We compare NPEfix against naive template-based repair. • A publicly available tool and benchmark for facilitating further research on this topic: https://github. com/Spirals-Team/npefix. This paper is a reworked and extended version of a working paper published on Arxiv [START_REF] Cornu | NPEFix: Automatic Runtime Repair of Null Pointer Exceptions in Java[END_REF]. It is structured as follows. Section II presents a taxonomy of repair strategies for null pointer exceptions. Section III describes our metaprogramming approach to apply those repair strategies. Section IV discusses the evaluation of our work. Section V, VI respectively explores the limitations of our approach and the related work.

II. A TAXONOMY OF REPAIR STRATEGIES FOR NULL POINTER EXCEPTIONS

In this section, we present a taxonomy of run-time repair strategies for null pointer exceptions. It unifies previous work on that topic [START_REF] Dobolyi | Changing Java's Semantics for Handling Null Pointer Exceptions[END_REF], [START_REF] Kent | Dynamic Error Remediation: a Case Study with Null Pointer Exceptions[END_REF], [START_REF] Long | Automatic Runtime Error Repair and Containment via Recovery Shepherding[END_REF] and proposes new strategies.

When a harmful null pointer exception is going to happen, there are two main ways to avoid it. First, one can replace the null value by a valid object. Second, one can skip the problematic statement. In both cases, no null pointer exception happens. In this paper, we refine those two techniques in 9 different strategies to repair null pointer exceptions. There are grouped along those two families: replacement of the null by an alternative object, and skipping the execution of the statement associated with the null pointer exception.

A. Strategies based on Null Replacement

One way to avoid a null pointer exception to happen is to change the reference into a valid instance. We can provide an existing value (if one can be found) or a new value (if one can be constructed). To facilitate the presentation, r is an expression (usually a variable reference). We basically want r to reference a valid (non-null) value in order to prevent a null pointer exception when executing r.foo(p). Symbol p is a method parameter. Reuse (S1b) A first strategy is the case of repairing the null pointer exception with an existing object as follows:

+ if (r == null) { + r = anotherVar; + } r.foo(p);
Strategy S1b is parameterized by the variable anotherV ar. The variable is taken from the set S of accessible objects composed of the local variables, the method parameters, the class fields of the current class and all the other static variables. S is further filtered to only select the set of all well typed and non-null values V according to the type of r.

Local Reuse (S1a) A variant of the reuse strategy consists in replacing one null reference by a valid object, without modifying the null variable itself.

B. Strategies based on Execution Skipping

We also propose to skip the statement where a null pointer exception would happen. There are different possible ways of skipping.

Line Skipping (S3) First, the straight-forward strategy S3 consists in skipping only the problematic statement and allows to avoid the null pointer exception at this location.

+ if (r != null) { r.foo(p); + }
We also propose a family of strategies which consists in skipping the rest of the method. For skipping the rest of the method, several possibilities can be considered. Let us consider that the method returns an object of type "Bar".

Return Null S4a If the method expects a return value, one can return null: this is a reasonable option because it is possible that the caller has a non-null check on the returned object. end for 15: end for vs skipping, local vs global, reusing objects vs creating new ones. For each strategy, the corresponding code that needs to be injected is shown in the last column.

C. Novelty

Among those 9 strategies, some of them have already been explored. Dobolyi et al. [START_REF] Dobolyi | Changing Java's Semantics for Handling Null Pointer Exceptions[END_REF] have proposed two of them: S2b and S3. Kent [START_REF] Kent | Dynamic Error Remediation: a Case Study with Null Pointer Exceptions[END_REF] have defined S2b, S3, S4a and S4d. Long et al. [START_REF] Long | Automatic Runtime Error Repair and Containment via Recovery Shepherding[END_REF] have explored S3, S4b and S4d. This means that 4/9 strategies presented in this paper are new: S1a, S1b, S2a, S4c. The novelty lies on the idea of reusing existing objects (S1a, S1b, S4c) and performing local injection (S1a, S2a).

D. A Naive Implementation for NPE Repair

To find patches according to the strategies presented in Section II, a naive implementation consists of exploring all possible strategies one by one. We call this naive implementation TemplateNPE, whose main algorithm is shown in Algorithm 1. The idea of this algorithm is to explore one strategy after each strategy, using source code transformation techniques, and to test whether they repair the null pointer exception under consideration. If the application of a template compiles, we have a "tentative patch". Hence, this algorithm explores the search space of all tentative patches. One sees that this naive implementation requires recompiling one file for each pair (parameter, strategy).

All patch templates are parametrized. The first template parameter contains the Java expression that may be null, it is located in the condition of the template as follows: if (<parameter> == null) ... . The second template parameter refers to the expression that is used to replace the null expression. This expression can be a variable (S1a, S2b), a new instance or a predefined constant (null, 0, 1, "", ' ') for S2a, S2b. This template parameter is not present in S3, S4a and S3). The replacement expressions are statically created based on the static analysis of the context of the line that produces the null dereference. We use code analysis to perform which variables are accessible at the expression causing the null pointer exception, and also to list constructor calls for building new instances.

III. METAPROGRAMMING FOR NULL POINTER REPAIR

We now present a set of code transformations to embed the 9 strategies presented in Section II in a metaprogram.

In this paper, a metaprogram is a program enriched with behavior modification hooks. By default, all behavior modification hooks are deactivated, which means that by default a metaprogram is semantically equivalent to the original program.

Let us consider the program

x = y + z.
A metaprogram is for instance x = y -z if HOOK_IS_ACTIVATED else y + z

(functional style) or

x = HOOK_IS_ACTIVATED ? y -z : y + z (ternary expression, e.g. in Java). Variable HOOK_IS_ACTIVATED is a Boolean global variable which controls the activation of the behavior modification. This metaprogram enables one to transform at run-time an addition into a subtraction. In our context, the metaprogram is automatically created using source-to-source transformation.

The metaprogramming code transformations are realized in a tool for Java called NPEfix. NPEfix is composed of three main phases: first, it generates a metaprogram out of the program under repair, second it compiles the metaprogram, third, it runs the failing test case a large number of times, each time altering the behavior to find a way to avoid the null pointer exception and thus emulate a patch.

A. Core Intercession Hook

To modify the behavior when a null pointer exception happens, we encapsulate each method call and field access as shown in Listing 1.

The call of doSomething that is originally present is now made on the result of method checkForNull. Method checkForNull does the following things. It first assesses whether the object is null, i.e. whether a null pointer exception will occur; if it's not null, the program proceeds with its normal execution. If the object is null, it means a null pointer exception is about to be thrown, and then a strategy is applied. For sake of simplification, this is shown as a switch as in Listing 1, this switch case is the core intercession hook of the metaprogram.

B. Value Replacement Strategies

There are four strategies based on value replacement (the first half of Table I): S1a, S1b, S2a and S2b.

1) Reuse Variable: For replacing a null value with a variable, the challenge is to maintain a set of variables as a pool for replacement at run-time. Listing 2 shows how we tackle this problem; we use a stack to store all the variables of each method. Each variable initialization and assignment inside the method is registered thanks to the NPEfix' method initVar. In addition, at the beginning of each method, we collect all the accessible fields and parameters.

2) Create New Object: Now, let us consider that the strategies that create a new variable (strategies S2a and S2b).

As shown in Listing 1, a call is made to createObject that takes as parameter the static type of the dereferenced variable. createObject uses reflection to access to all the constructors of the given type. In addition, this method is recursive so as to create complex objects if needed.

C. Skipping Strategies

Now we present how we implement the strategies based on skipping the execution (the second half of Table I).

1) Line skipping: The strategy S3 necessitates to know if a null pointer exception will happen in a line, before the execution of the line. For this, the transformation presented in Listing 1 is not sufficient, because the call to method checkForNull implies that the execution of the line has already started. To overcome this issue, we employ an additional transformation presented in Listing 3.

Similarly to checkForNull, method skipLine assesses, before the line execution, whether the dereferenced value is null or not, and whether it is harmful. Method skipline takes an arbitrary number of objects, the ones that are dereferenced in the statement. This list is extracted statically.

2) Method skipping: The remaining strategies are based on skipping the execution of the rest of the method when a harmful dereference is about to happen: these are strategies S4d, S4a, S4c and S4b (the last part of Table I). We implement those strategies with a code transformation as follows.

A try-catch block is added in all methods, wrapping the complete method body. This try-catch blocks handle a particular type of exception defined in our framework (ForceReturnError). This exception is thrown by the skipLine method when one of the method-skipping strategies is activated, as show in Listing 4. This listing also shows a minimalist example of the code resulting from this transformation.

D. Exploration of the Patch Search Space at Runtime

Now that we have a metaprogram that embeds all strategies, we also have a way to explore the search space of null pointer repair purely at run-time. The From Hooks to Patches Given a combination of behavioral modification hooks, one can create the corresponding source code patch by reinterpreting the hooks according to the templates presented in Section II.

E. Implementation

All those transformations have been implemented in a tool called NPEfix, which has been made publicly available for sake of reproducible research and open science 2 . The transformations use the Spoon library [START_REF] Pawlak | Spoon: A Library for Implementing Analyses and Transformations of Java Source Code[END_REF].

IV. EVALUATION

We now evaluate NPEfix. We design a protocol to answer the following questions.

• RQ1. What is the impact of NPEfix' runtime analysis of the repair context and TemplateNPE's static analysis of the repair context on the number of explored tentative patches? • RQ2. Does NPEfix (metaprogramming) produce more valid patches? • RQ3. What are the reasons explaining the presence of different valid patches? • RQ4. Is the performance of NPEfix acceptable?

A. Protocol

In order to evaluate our approach of patch generation based on metaprogramming, we build a benchmark of real and reproducible null pointer exceptions in Java programs (see Section IV-C). Then we compare the ability of TemplateNPE and NPEfix to find different patches and repair each bug of the benchmark. TemplateNPE is the template-based implementation of our nine repair strategies that is described in Section II-D. NPEfix uses our metaprogramming approach to fix null pointer exceptions as described in Section III.

In this paper, we study "test-suite adequate" patches [START_REF] Martinez | Automatic Repair of Real Bugs in Java: A Large-Scale Experiment on the Defects4J Dataset[END_REF] We consider a patch as test-suite adequate when the failing test case that reproduces the null dereference passes as follows: 1) no exception (a null pointer exception or another one) is uncaught and crashes the test case and 2) whether the assertions that come at the end of the test case reproducing the null pointer exception pass. For sake of readability, we refer to testsuite adequate patches as "valid" patches (hence we use test-suite adequate and "valid" interchangeably in this paper)

Note that test-suite adequate patches mare be conceptually incorrect, even if they pass the test suite. Since we use as only oracle the test cases, which can miss correctness assertions, a valid patch can be considered incorrect beyond the test-suite satisfaction correctness criterion [START_REF] Long | An Analysis of the Search Spaces for Generate and Validate Patch Generation Systems[END_REF], [START_REF] Xuan | Nopol: Automatic repair of conditional statement bugs in java programs[END_REF].

B. Evaluation Metrics

After each bug and each repair technique, we collect three metrics: the number of tentative patches (whether valid or not); the number of valid patches (that pass the test case); the execution time required to explore the search space of patches. We interpret those metrics as follows. A larger number of tentative patches means that the search space is richer. As shown in previous work [START_REF] Martinez | Automatic Repair of Real Bugs in Java: A Large-Scale Experiment on the Defects4J Dataset[END_REF], [START_REF] Long | An Analysis of the Search Spaces for Generate and Validate Patch Generation Systems[END_REF], there are often multiple different patches that are able to make a failing test case passing. A larger number of generated valid patches is better, it means that the developer is given more choices to get a really good choice.

The results of this experimentation, incl. all tentative patches are publicly available at https://github.com/ Spirals-Team/npefix-experiments

C. Benchmark

To build a benchmark of real null pointer exceptions in Java, we consider two inclusion criteria. First, the bug must be a real bug reported on a publicly available forum (e.g. a bug tracker). Second, the bug must be reproducible. This point is very challenging since it is really difficult to reproduce field failures, due to the absence of the exact crashing input, or the exact configuration information (versions of dependencies, execution environment, etc.). As a rule of thumb, it takes one day to find and reproduce a single null pointer exception bug. We consider bugs in the Apache Commons set of libraries (e.g. Apache Commons Lang) because they are well-known, vastly used and their bug trackers are public, easy to access and to be searched. Also, thanks to the strong software engineering discipline of the Apache foundation, a failing test case is often provided in the bug report. We have not rejected a single reproducible field null pointer exception.

As a result, we have a benchmark that contains 16 null pointer exceptions (1 from collections, 3 from lang and 7 from math, 3 from PDFBox, 1 from Felix, 1 from Sling). It is publicly available for future research (https://github. com/Spirals-Team/npe-dataset). The main strength of this benchmark is that it only contains real null pointer exception bugs and no artificial or toy bugs. Table II shows the size of applications for which we have real field failures. Table III presents the results of our experiment. The first column contains the bug identifier. The second column contains the number of tentative patches for each bug (ie. the size of the search space). This column is composed of two sub-columns: the number of tentative patches using the template-based approach (TemplateNPE) and the number of patches generated by the metaprogramming approach (NPEfix). For example, TemplateNPE identifies 7 tentative patches and NPEfix identifies 10 tentative patches. The remaining top-level columns will discussed below in Section IV-E and Section IV-G.

In 12/16 (in bold) of the case NPEfix explores more tentative patches than TemplateNPE. This validates that the static analysis and dynamic analysis of the repair context differs, and that the latter is potentially richer. The difference in the number of tentative patches between TemplateNPE and NPEfix is explained as follows.

• During execution, more objects are detected as compatible with the type of the null expression. With the template approach, we do not know the actual runtime type of all variables. • The number of different new objects created varies because NPEfix detects at runtime more compatible constructors.

• Some strategies cannot be applied at certain loca-tions with the template approach. For example, Tem-plateNPE cannot apply the skip-line strategy (S3) on a local variable. This case is naturally handled in the metaprogramming approach.

• In 3/16 cases, the template-based approaches identifies more tentative patches. The reason is that NPEfix filters out equivalent patches by on the runtime variable value. This is further discussed in Section IV-F2 and Section IV-F3.

RQ1. What is the impact of NPEfix' runtime analysis of the repair context and TemplateNPE's static analysis of the repair context on the number of explored tentative patches?

NPEfix explores 273 tentative patches, which is 68 more than the template-based approach. In other words, the search space of the metaprogramming technique is larger. This validates our intuition that the runtime analysis of the repair context is valuable in certain cases.

E. RQ2. Does NPEfix (metaprogramming) produce more valid patches?

Table III also gives the number of tentative patches that are valid, ie. that make the failing test case passing. This is is shown in the 4th and 5th columns under the top-level header "# Valid Patches" presents the results of this experiment. For example, for bug Collection-360, neither TemplateNPE nor NPEfix identifies a valid patch.

For lang304, TemplateNPE identifies 43 valid patches, while NPEfix finds 65 valid patches. As we can see there is a correlation between the size of the explored search space and number of valid patches identified. This means that it is worth exploring more tentative patches to identify more valid patches.

RQ2. Does NPEfix (metaprogramming) produce more valid patches? NPEfix finds 175 patches that avoid the null pointer exception and make the test case passing. Among those valid patches, 48 patches of them are uniquely found by the metaprogramming approach thanks to the runtime analysis of the repair context. Those patches are only test-suite adequate, and if the test suite is weak, they may be incorrect.

F. RQ3. What are the reasons explaining the presence of different valid patches?

We answer to this research question with 3 case studies.

1) Math-305: We now discuss the patches found for bug Math-306, where the null pointer exception is thrown during the computation of a clustering algorithm called Kmeans. The null pointer exception is triggered when a point is added to the nearest cluster. When the library first computes the nearest cluster, it fails because the current point is at a distance largest than Integer.MAXVALUE. Then, the nearest cluster is set to null and a null pointer exception is thrown when the library tries to add the current point to it.

NPEfix succeeds to identify 4 different patches for this bug. They are all presented in Figure 1. The top three first patches (in green) of the figure are test-suite adequate: they all avoid the null pointer exception to be thrown. As we see, they have the same behaviour: they skip the line that produces the null pointer exception. The first and the third patches create a new cluster that is never used in the application: this is useless but it works. The second patch skips the line that produces the null pointer exception, resulting in the point not being added to the cluster. The last patch is invalid because it produces a division per zero later in the execution: the test case still fails (not with the original null pointer exception but with a division-by-zero exception).

This case study illustrates the fact many different valid patches indeed exist. However, while they are all equivalent according to the test case oracle, they are not equivalent for the developers: a developer would likely discard a patch that creates a temporary object only to avoid a null pointer exception.

2) Felix-4960: Felix is an implementation of the OSGI component model. Real bug Felix-4960 is about a null pointer exception that is thrown in method "getRe-sourcesLocal(name)" which is dedicated to searching a ---org/apache/commons/math/stat/clustering/ KMeansPlusPlusClusterer.java +++ org/apache/commons/math/stat/clustering/ KMeansPlusPlusClusterer.java @@ -90,3 +90,7 @@ Cluster<T> cluster = getNearestCluster( clusters, p); resource in a path. The null pointer exception appears when Felix does not succeed to get a list of resources in the path. In this case Felix tries to iterate on a list that is null, which triggers the null pointer exception. For Felix-4960, TemplateNPE is surprisingly able to generate more tentative patches: NPEfix generates 7 tentative patches and TemplateNPE generates 9 tentative patches that compile. The reason is that template generates tentative patches based on reusing variables, however those patches are meaning less. TemplateNPE fills the template parameters with m_activationIncludes and m_declaredNativeLibs. However, those variables are null because not initialized at this location in the code. In other words, TemplateNPE generates a patch ---pdfbox/pdmodel/interactive/form/PDAcroForm.java +++ pdfbox/pdmodel/interactive/form/PDAcroForm.java @@ -250,2 +250,5 @@ + if (fields == null) { + return retval; // reval is null + } for (int i = 0; i < fields.size() && retval == null; i ++)

Listing 5. The additional patch of TemplateNPE for the bug PDFBbox-2965 that replaces a null by a null, which obviously results in the same null pointer exception as before.

On the contrary, NPEfix works at runtime, and hence knows that the actual value of m_activationIncludes and m_declaredNativeLibs are null. Hence, it does not even tries them, because it knows in advance that such a tentative patch would be invalid. In this case, the runtime analysis of the context is interesting to discard incorrect patches early in the process.

3) PDFBbox-2965: PDFBbox is a PDF rendering library. This library allows one to read the properties of a PDF file. PDFBbox-2965 happens when PDFBox searches for a specific field on a PDF that contains no PDF form. Internally PDFBox iterates on all form fields of the PDF and compare the name of the field against the searched field. But when the PDF contains no form, the list of fields is null, and a null pointer exception is thus triggered.

NPEfix generates 3 tentative patches and Tem-plateNPE generates tentative 4 patches for this bug. They are all valid according to the test case. As we see, TemplateNPE generates one additional valid patches. This patch is shown in Listing 5. It consists of returning variable retval if the fields variable is null. This patch is of low-quality because retval is always null in this case. In other words, a simple return null; is a more explicit patch, and would better help the developer. This explains why NPEfix does not generate this patch, as for Felix-4960, thanks to runtime analysis, NPEfix knows that retval is null, and that return retval; is equivalent to return null; (strategy S4a).

G. RQ4. Is the performance of NPEfix acceptable?

The last column of Table III presents the execution time required to explore the whole search space of tentative patches for fixing the null pointer exception. For example, for the bug Collections-360, TemplateNPE explores the search space in 20 seconds while NPEfix requires 2 minutes 32 seconds.

In most cases, the template based approach is faster. The reason is that creating the metaprogram takes a lot of time due to the complexity of code transformations. Also, the additional code injected in the metaprogram slows down each repair attempt, ie slows down each execution of the failing test case. For NPEfix, the complete exploration of the search space takes at most 3 minutes 22 seconds. We consider this acceptable since a developer can wait for 3 minutes before offered a set of automatically tentative patches.

RQ4. Is the performance of NPEfix acceptable?

The complete exploration of the search space of tentative patches is faster with TemplateNPE. This highlights a trade-off between the number of patches explored found and the time to wait.

V. DISCUSSION

A. Patch Readability and Templates

One concern of automatic patch generation is the readability of the generated patches and its impact on the maintainability of applications [START_REF] Fry | A human study of patch maintainability[END_REF]. Let us discuss the readability of NPEfix patches. NPEfix patches repair one specific type of bug: null dereference. We observe that most developers fix these bugs by adding a null check (if(... != null) something()) before the null expression. Most NPEfix strategies resemble those humanwritten patches and thus and can be as easily understood and maintained as human patches. Our experience with NPEfix suggests that template-inspired patches enables one to encode -if not enforce -readable patches.

B. Genetic Improvement and Metaprogramming

Genetic improvement [START_REF] Langdon | Genetic improvement of gpu software[END_REF] refers to techniques that change the behavior of a program in order to improve a specific metric, for example the execution time. There is an interesting relation between genetic improvement and NPEfix. Both are based on dynamically changing the behavior of an application. In this perspective, making failing tests pass can be considered as a functional metric to be optimized. Indeed, we think that metaprogramming techniques similar to that proposed in this paper could be used for genetic improvement by creating a space of different program behaviours to explore.

C. Threat to Validity

A bug in the implementation of TemplateNPE or NPEfix is the threat to the validity of the findings based on quantitative comparison presented in Table III.

Our benchmark is only composed of real bugs due to null pointer exceptions which is a strength. However, since they are all in Java and from 6 projects, there is a threat to the external validity of our findings. Other bugs and an implementation of TemplateNPE and NPEfix in another language may uncover other behavioral differences.

D. Limitations

We now summarize the main limitations of NPEfix that we identified during our experiment. First, NPEfix is complex. As we have seen, the template-based approach is quite straightforward, while the metaprogramming approach is complex to debug and maintain. Due to the richness and complexity of Java, there may be cases where the metaprogramming approach slightly changes the behaviour of the application. Second, as we have seen in Section IV-G, NPEfix is slower than the template based approach. Third, as for all test-suite based patch generation technique, there is a risk of generating testsuite adequate, yet incorrect patches as suggested by the Math-305 example.

E. Reflections on TemplateNPE

We would like to emphasize the fact TemplateNPE works surprisingly well. Initially, TemplateNPE was built as baseline comparison for NPEfix. However, over time, we realized that it is much simpler to implement and maintain, while keeping comparatively good results, and while being faster. This is an important lesson learned for us and for the metaprogramming research community: advanced code transformations and behavior modifications at runtime is not necessarily the best option. Simpler may be better.

VI. RELATED WORK

A. Patch Generation

The literature on patch generation is growing very fast. We only present a brief overview of notable contributions and refer to [START_REF] Monperrus | Automatic Software Repair: a Bibliography[END_REF] for a comprehensive overview. GenProg by [START_REF] Le Goues | Genprog: A generic method for automatic software repair[END_REF] applies genetic programming to the AST of a buggy program and generates patches by adding, deleting, or replacing AST nodes. Debroy and Wong [START_REF] Debroy | Using mutation to automatically suggest fixes for faulty programs[END_REF] propose a mutation-based repair method inspired from mutation testing. This work combines fault localization with program mutation to exhaustively explore a space of tentative patches. SemFix by [START_REF] Nguyen | Semfix: program repair via semantic analysis[END_REF] is a constraint based repair approach. It provides patches for assignments and conditions by combining symbolic execution and code synthesis. Nopol by [START_REF] Demarco | Automatic repair of buggy if conditions and missing preconditions with smt[END_REF], [START_REF] Xuan | Nopol: Automatic repair of conditional statement bugs in java programs[END_REF] is also a constraint based method, which focuses on repairing bugs in if-conditions and missing preconditions. SPR [START_REF] Long | Staged program repair with condition synthesis[END_REF] defines a set of staged repair operators so as to early discard many candidate repairs that cannot pass the supplied test suite and eventually to exhaustively explore a small and valuable search space.

We have discussed in this paper a template-based patch generation approach. PAR by [START_REF] Kim | Automatic patch generation learned from human-written patches[END_REF] uses 10 patch templates for common programming errors, Relifix [START_REF] Tan | Relifix: Automated repair of software regressions[END_REF] defines templates specifically for regression bugs. All of those approaches require a regression test suite to validate the patch, none of them leverage production traffic to assess the absence of regressions.

B. Metaprogramming for Repair

Rinard et al. [START_REF] Rinard | Enhancing server availability and security through failure-oblivious computing[END_REF] presented a technique called "failure oblivious computing" to avoid illegal memory accesses. Their idea is to create a metaprogram that adds additional code around each memory operation during the compilation process. For example, the additional code verifies at run-time whether an array is accessed out of his bounds. If the access is outside the allocated memory, the access is ignored (akin line skipping presented in this paper) instead of crashing with a segmentation fault. Dobolyi and Weimer [START_REF] Dobolyi | Changing Java's Semantics for Handling Null Pointer Exceptions[END_REF] present a technique to tolerate null pointer exceptions using metaprogramming. Using code transformation, they introduce hooks to a recovery framework. This framework is responsible for forward recovery of the form of creating a default object of an appropriate type of skipping instructions. Their strategies are a small subset of ours, and they do not explore the search space as we do in this paper.

The closest related work is by Kern and Esparza [START_REF] Kern | Automatic Error Correction of Java Programs[END_REF] use a metaprogram that integrates all possible mutations according to a mutation operator. The mutations that are actually executed are driven by meta-variables. A repair is a set of values for those meta-variables. The metavariables are valued using symbolic execution. Both the metaprogram and the kind of faults are completely different from what we have presented in this paper.

VII. CONCLUSION

In this paper, we have presented a novel and original technique to explore the repair search space of null pointer exception bugs. Our technique, called NPEfix, is based on 9 strategies that are specific to null pointer exceptions, and uses metaprogramming to explore the search space of all possible strategies. We have evaluated our technique on 16 real null pointer exceptions: NPEfix is able to successfully explore the search space of null pointer repair, and finds 175 valid patches over all considered bugs.

Future work is required to see whether other patch templates can be implemented using metaprogramming. For instance, we are confident that even a generic technique as Genprog [START_REF] Le Goues | Genprog: A generic method for automatic software repair[END_REF] could be implemented in a metaprogramming way, which may be key for performance. Future work is also required to optimize the transformations and to speed-up the execution of metaprograms.

Table I

 I NPEFIX' REPAIR FOR NULL POINTER EXCEPTIONS.

		Strategy	Id	Description
			local	S1a local reuse of an
		reuse			existing compatible object
	Replace	creation	global local	S1b global reuse of an existing compatible object S2a local creation of a new object
			global	S2b global creation of a
					new object
		line	S3	skip statement
			null	S4a return a null to caller
			creation S4b return a new object
	skipping	method	reuse	to caller S4c return an existing compatible object to caller
				S4d return to caller (void
					method)

Return New Object S4b One can return a new in- stance of the expected type. As for strategy S2b, this is a strategy parameterized by all possible constructor calls.

		Algorithm 1 TemplateNPE: Exploration of all tentative
		patches based on parametrized templates
		Input: p: a program
		Input: t: a test case reproducing a NPE
		Input: e: expression that triggers the NPE
		Input: S: a set of repair strategies
		Output: P: set of tentative patches
		Output: Q: set of valid patches
		1: compile p
		2: for s in S do
		3:	A ← possible parameter value for s in e
		4:	for a : A do
		5:	x ← apply s on p parametrized by a
		6:	recompile x
		7:	if x compiles then
		8:	add x to P
		9:	end if
		10:	run t against x
		11:	if t succeeds then
		12:	add x to Q
		13:	end if
		14:
	+ if (r == null) {
	+	return null;
	+ }
	r.foo(p);
	+ if (r == null) {
	+	return new Bar();
	+ }
	r.foo(p);
	Return Variable S4c One can also search in the set of
	accessible values one which corresponds to the expected
	return type and return it. As for strategy S1b, this is
	a strategy parametrized by all possible type-compatible
	variables.
	+ if (r == null) {
	+	return anotherVar;
	+ }
	r.foo(p);
	Vanille Return S4d When the method does not return
	anything (void return type in Java), inserting a simple
	"if (r==null) { return; }" is a valid option.
	All strategies are listed in Table I. The table represents
	the different dimensions of the analysis: replacement

  It tries to create a new instance of the class from each available constructor. Given a constructor, it attempts to create a new instance for each of the

	// before transformation
	value.dereference();
	// after NPEfix transformation
	if (skipLine(value)){
	value.dereference();
	}
	boolean skipLine(Object... objs){ // NPEfix framework
	for (Object o : objs) {
	if (o == null && cannotCatchNPE() && doSkip())
	return false
	}
	return true;
	}
	Listing 3. Implementation of Line-based Skipping
	parameter recursively. The stopping condition is when
	a constructor does not need parameters. Note that the
	primitive types, which don't have constructors, are also
	handled with default literal values.

Table II DESCRIPTIVE

 II STATISTICS OF THE SUBJECTS SUFFERING FROM NULL POINTER EXCEPTIONS. SIZE MEASURED BY CLOC VERSION 1.60.

	Subject Domain	Size
	COLL	Collection library	21,594 LOC
	LANG Utility functions	18,970 LOC
	MATH Math library	90,771 LOC
	PDF	PDFBox library	64,375 LOC
	Felix	Felix library	33,057 LOC
	SLING Sling library	583 LOC
	Total	16 bugs	229,350 LOC

Table III COMPARISON

 III OF THE THE TEMPLATE APPROACH (TEMPLATENPE) AND THE META-PROGRAMMING APPROACH (NPEFIX) ON THREE KEY METRICS.

		# Tentative Patches # Valid Patches	Execution Time
	Bug ID	Template NPEfix Template NPEfix Template NPEfix
	collections360	7	10	0	0	0:00:20 0:02:32
	felix-4960	9	7	4	4	0:00:43 0:03:22
	lang304	44	77	43	65	0:00:11 0:00:26
	lang587	21	28	12	28	0:00:23 0:00:23
	lang703	16	15	0	7	0:00:09 0:00:20
	math1115	8	11	6	5	0:00:47 0:02:08
	math1117	8	11	0	0	0:00:41 0:01:52
	math290	9	10	3	4	0:00:18 0:00:42
	math305	2	4	1	3	0:00:08 0:00:40
	math369	15	16	14	14	0:00:26 0:00:43
	math988a	17	17	11	11	0:01:02 0:01:38
	math988b	17	25	17	17	0:01:39 0:01:48
	pdfbox-2812	6	14	2	2	0:00:28 0:01:46
	pdfbox-2965	4	3	4	3	0:00:16 0:01:34
	pdfbox-2995	4	5	3	1	0:00:13 0:01:35
	sling-4982	18	20	7	11	0:00:05 0:00:06
	Total	205	273	127	175	0:07:57 0:21:44
	Average	12.81	17.06	7.94	10.94	0:00:29 0:01:21
	Median	9.00	12.50	4.00	4.50	0:00:22 0:01:35

D. RQ1. What is the impact of NPEfix' runtime analysis of the repair context and TemplateNPE's static analysis of the repair context on the number of explored tentative patches?

https://github.com/Spirals-Team/npefix