Zacharie Brodard

Laurent Toutain

Géraldine Texier

Keywords: DetNet, Deterministic Networking, 6TiSCH, 6LoWPAN, IEEE Std. 802.15.4, WSN, IOT, Industrial Internet

Je tiens à remercier:

Thomas Watteyne et Norm Finn pour l'enseignement qu'ils m'ont apporté au travers des années, et la collaboration sans faille dans nos travaux communs qui ont mené à la création de 6TiSCH et DetNet; en reconnaissance de leur savoir apparemment infini et de leur patience totale -ou presque ; ) Zacharie Brodard,

Résumé en Français

En s'appuyant sur une connaissance précise du temps, sur la réservation de ressources et l'application distribuée de règles d'admission strictes, un réseau déterministe permet de transporter des flux unicast et multicast pré-spécifiés avec un taux de perte extrêmement bas et une latence majorée, ce qui ouvre la voie au support d'applications critiques et/ou temps-réel sur une infrastructure de réseau d'entreprise convergée.

La technologie de réseau déterministe offre de nouvelles garanties de Qualité de Service (QoS) comme par exemple celle de majorer la latence d'un flux en toutes circonstances. Plus précisément, la latence comme le taux de perte peuvent être maintenus en-dessous des bornes garanties alors même que de nouveaux services sont déployés, augmentant la charge globale du réseau sans impact mesurable sur les flux existants.

Il existe de nombreux cas d'usage, soit déjà déployés, soit émergeants, pour les réseaux déterministes. Audio/Video Bridging, (AVB) pour l'industrie audio-visuelle, Professional Media Networking (PMN) pour les studios de production professionnels, et de nombreuses variations de l'Internet des objets (IoT) dans le cadre de l'Internet Industriel, sont trois catégories significatives. L'Internet Industriel, en particulier, promet des économies en milliards de dollars dans de multiples industries par l'optimisation des procédures industrielles et des coûts d'installation et d'opération.

Introduction

It all started with point-to-point copper wires, transporting analog signals for short messages, then telephone and television, industrial measurements and commands, anything though initially not data. Digital data networks, and then packet networks, came last; but with the advent of determinism, the late comers now show the potential to federate all original forms of wired and wireless communication and lead to the final convergence of all communication networks.

A generic and cheap replacement to serial cables to provide connectivity to all sorts of devices, coupled with resource-sharing meshed networks, are now required to simplify the cabling and drive the costs down in many industries. Dumb as it may seem to emulate the legacy forms of serial communications, reproducing the various aspects of a point-to-point electric cable over a multi-hop packet network is actually the hardest thing to do. Yet, the need is becoming more and more pressing, as 1) managing all the existing sorts of cables and buses has become an increasingly costly hassle in many aspects of our lives and 2) point-to-point wires will not scale to serve the exploding needs of the Internet of Things.

At the Emerson Exchange 2008 conference in Washington, we presented a white paper on Integrating an Industrial Wireless Sensor Network with your Switched Ethernet and IP Network 1 . The paper discussed how Wireless Sensor Networks (WSNs), which are in essence cheaper and faster to deploy than traditional wired field-buses, could leverage the entire network to connect the sensors to a centralized controlling application located afar on the carpeted floor, for Industrial supervisory control or logging; at the same time, the paper stressed issues that are raised when integrating a classical, often proprietary industrial automation network, with tight response time and availability constraints, into a wider IP network based on packet-switched and Internet technologies. With this and a collection 2345 of other papers, the realization is now coming that with techniques such as flow isolation, high availability and a new generation of Quality of Service (QoS), the times of the convergence of these networks are finally approaching.

Converging Networks for the Industrial Internet

"Operational technology (OT) is hardware and software that detects or causes a change through the direct monitoring and/or control of physical devices, processes and events in the enterprise" a . a http://www.gartner.com/it-glossary/operational-technology-ot

On Operational technology

Operational Technology (OT) refers to industrial networks, which focus on highly reliable, secure and Deterministic Networking. In OT environments, Deterministic Networks are characterized as providing a guaranteed bandwidth with extremely low packet loss rates, bounded latency, and low jitter 6 . OT networks are typically used for monitoring systems and supporting control loops, as well as movement detection systems for use in process control (i.e., continuous manufacturing) and factory automation (i.e., discrete manufacturing), and protection systems in the SmartGrid 7 .

Due to its different goals, OT has evolved in parallel but in a manner that is radically different from Information Technology/Information and Communications Technology (IT/ICT), which until now relied on selective queuing and discarding of IP packets to achieve end-to-end flow control over the Internet, and provided limited guarantees in terms of delivery and latency. For that lack of determinism and an associated lack of trust, IT and OT network have been maintained segregated.

Enabling the IT/OT convergence: The motivation behind General Electric's original vision of an Industrial Internet 8 , as depicted in figure 1, was that a single percentile point of operational optimization 9 would save billions of dollars across many vertical industries. But reaching additional levels of optimization is not an easy task; it would require collecting and processing huge amounts of live measurements by widely distributed OT sensing and IT analytics capabilities 10111213 . These currently missing measurements comprise all sorts of diagnostics and sensor data that in many cases today are captured but not reported, either because the cost of wiring the devices would be prohibitive, or, if a device has connectivity to a control network, because the control network is kept isolated from the IT network and the wider Internet for security reasons. In order to avoid skyrocketing operational costs in maintaining highly trained teams for multiple solutions, the missing measurements must share the same infrastructure (network and management) as the deterministic OT flows. It results that the Industrial Internet vision can only be achieved by the convergence of IT and OT, whereby the network becomes capable of emulating the properties of deterministic OT circuits in the same fabric that serves traditional best effort IP applications.

New efforts at the IETF and the IEEE 14 will enable traffic that requires bounded latency in a worst case scenario and is generally sensitive to packet loss and/or jitter, for application in a large variety of use cases [START_REF] Grossman | Deterministic Networking Use Cases[END_REF] 15 with a high degree of operational criticality. The upcoming protocols will support a mix of deterministic and classical besteffort traffic to be transported across Ethernet bridges and over IP networks, respectively. This Deterministic Networking capability is the key enabler for the convergence of IT and OT networks.

The case of Low-power Lossy Networks:

The quality of transmission over IEEE Std 802.15.4 radios is affected by multiple elements, such as the relative location of objects in the environment and interferers of all kinds; these elements may be extremely difficult to control (e.g., radars) and may change brutally (e.g., a door opens); it results that a given channel cannot be expected to remain stable over a long period of time, and that some Channel Agility is required to guarantee a degree of service continuity over a long period of time.

When a radio transmission fails, adding diversity to the transmission characteristics improves the chances to avoid the cause of the failure and thus, those of a successful retransmission. A retransmission over the same channel adds only time diversity, and unless the cause of the loss is really transient, it is bound to fail for the same reason the original transmission did. But, as discussed by Srinivasan [START_REF] Srinivasan | The β-factor: measuring wireless link burstiness[END_REF] 16 in the case for 802.15.4 and 802.11b networks, other forms of diversity can help alleviate the issue.

Considering that those radios are highly sensitive to multipath fading [START_REF] Watteyne | Mitigating Multipath Fading Through Channel Hopping in Wireless Sensor Networks[END_REF] 17 , and that in turn multipath fading is highly sensitive to both location and frequency, it makes sense to add channel diversity to the retransmission. Trying this, it was found that switching channels dynamically based on a variation of the link quality also yields transient periods of instability (more on this in section 3.3 on page 20).

So in the years 2003-2007, providing deterministic services over IEEE Wireless Personal Area Networks (WPANs) 18 such as 802. 15.4 Low-Rate WPANs (LWPAN) appeared to be extremely challenging. Around that time, Pister developed a novel approach to combine frequency diversity and channel hopping based on time synchronization, and they introduced the quantum leap that brought determinism over low-power wireless, with the Time Synchronized Mesh Protocol (TSMP) [START_REF] Pister | TSMP: Time synchronized mesh protocol[END_REF] 19 . TSMP introduced a scheduled combination of frequency and time diversity that defeats most interferences and can reach wire-equivalent reliability on battery-operated devices. TSMP opened the way to Time-Slotted Channel Hopping (TSCH) [START_REF] Watteyne | Industrial ieee802.15.4e networks: Performance and trade-offs[END_REF] 20 , which was published as an amendment 21 in 2012, and is now updated and retrofitted in the mainline IEEE Std 802.15.4 standard.

TSCH was immediately adopted by the Process Control industry, and two competing industrial WSN standards were developed in the following years, both leveraging this technique; first came WirelessHART™, which shipped with HART7 22 , and then ISA100.11a [START_REF] Hasegawa | Industrial wireless standardization-Scope and implementation of ISA SP100 standard[END_REF] 23 , which brought a limited support of IPv6.

The above standards are optimized for application in industrial Process Control; they are not designed to access the global Internet. In order to realize the Industrial Internet vision for wireless, there was a need to develop a new series of open standards combining best-effort and time-sensitive IPv6 traffic over TSCH. This work has started in 2013 at the IETF 6TiSCH WG and is now delivering its first round of standards.

But wire-equivalent reliability is still not determinism. For safety and critical control applications, there is a need to guarantee the timely delivery of packets, even in the face of accidental situations such as the failure of a device or an obstacle moving in the way of the transmission. For this, additional forms of diversity, such as spatial diversity, and sometimes implementation and even technology diversities, are required.

Some Context About This Thesis

This manuscript discusses Deterministic Networks in the Industrial Internet with a focus on wireless industrial use cases, and proposes novel techniques to compute deterministic pathways called Tracks over an IEEE Std 802.15.4 TSCH-based mesh network. This work started around 2007 when I contributed as an editor of the specification to the ISA100.11a standardization work. The ISA100.11a products 24 shipped in 2011; my main contribution was to enable at least a partial support of IPv6 in the standard.

In order to steer the adoption of IPv6 by the ISA100 committee, there was a need to update 6LoWPAN [START_REF] Montenegro | Transmission of IPv6 Packets over IEEE[END_REF] 25 and provide a more efficient, stateful, compression; I jumped in and co-edited the Compression Format for IPv6 Datagrams over IEEE 802.15.4-Based Networks [START_REF] Thubert | Compression Format for IPv6 Datagrams over IEEE[END_REF] 26 , also known as 6LoWPAN Header Compression (HC), and helped reinjecting this work into the ISA100.11a standard.

In parallel, I worked on the standardization of RPL: IPv6 Routing Protocol for Low-Power and Lossy Networks [START_REF] Winter | RPL: IPv6 Routing Protocol for Low-Power and Lossy Networks[END_REF] 27 , which I co-edited and largely influenced. RPL shipped in 2012 and has become the de-facto standard for routing in LLN.

In 2013, Thomas Watteyne and I lead the formation of the 6TiSCH Working Group (WG) at the IETF, which we have been co-chairing ever since. With its current charter 28 , 6TiSCH provides a statistically multiplexed network layer based on IPv6 over the TSCH Medium Access Control (MAC) [5] 29 layer, leveraging RPL and 6LoWPAN to build energy-efficient mesh networks.

The work on deterministic IP is yet to be achieved, but it was clear in 2014 that the problem was more generic than just IPv6 over TSCH. So I went on and helped create the DetNet Working Group, which covers applications that are found in professional audio/video, industrial process control, safety, electric utilities, and many more [START_REF] Grossman | Deterministic Networking Use Cases[END_REF] 30 .

At DetNet, I contributed to the Deterministic Networking Problem Statement [START_REF] Finn | Deterministic Networking Problem Statement[END_REF] 31 , which documents the issues that are in scope for the group, the Deterministic Networking Architecture [START_REF] Thubert | Deterministic Networking Architecture[END_REF] 32 , which supports the overall activity and presents in particular the concept of Packet Replication and Elimination, and the study on technical alternatives for the data plane [START_REF] Korhonen | DetNet Data Plane Protocol and Solution Alternatives[END_REF] 33 . This manuscript brings it all together; it is organized as follows:

In part I, we study the characteristics of Deterministic Networking, how the concept can be achieved on wired infrastructures, and the direction taken by the standards bodies to enable deterministic properties on packet networks.

In part II, we focus on low-power wireless technology, and consider the specific techniques that are applied today to achieve some degree of determinism on this medium, in particular for use in industrial Process Control networks.

In part III, we walk through our contributions, namely:

• with BIER-TE-based OAM, Replication and Elimination [START_REF] Thubert | Replication and Elimination, Internet-Draft draft-thubert-bier-replication-elimination-00[END_REF] 34 , we introduce a new set to Operations, Administration and Maintenance (OAM) tools [START_REF] Mizrahi | An Overview of Operations, Administration, and Maintenance (OAM) Tools[END_REF] 35 that is based on Bit Indexed Explicit Replication (BIER) [START_REF] Wijnands | Multicast using Bit Index Explicit Replication[END_REF] 36 Traffic Engineering (BIER-TE) [16] 37 ; this novel approach enables new degrees of control and trace-ability for the Replication and Elimination function in a generic Deterministic Network; an interesting aspect of that innovative technique is the capability to control, on a perpacket basis, which adjacencies (e.g., links) participate to the redundancy scheme, and, in return, to observe which transmission failed for each individual packet; and from that unique capability we derive a protection control loop that limits the redundant transmissions in stable conditions and routes around breakages when they are detected;

• we apply that redundancy control mechanism to 6TiSCH networks; we propose different arrangements to improve the highly predictable properties of the TSCH MAC, leveraging Replication, Retries and Elimination along novel interleaved multipath structures that we call Tracks, to enhance the art of industrial networks and Introduction achieve a higher level of reliability and manageability as required for mission-critical operations and safety-related applications;

• in addition, we propose a novel collaborative frame overhearing technique that is intended to balance for the extra transmissions that are associated with Replication and Elimination; an intuition of the proposed technology is that though a transmission from one node to another may have less chances to succeed over a radio than over a wired Ethernet, this might be compensated by the fact that, over the radio, other nodes than the intended recipient may have captured the transmission, and may be in a position to progress it towards the final destination. The collaboration can be applied to improve end to end transmissions as long as a packet that is transmitted can be uniquely identified, and as long as the extra listeners are well-chosen to provide spatial diversity in the rough direction to the destination.

Introduction i

Converging Networks for the Industrial Internet .

Deterministic Networking: the Vision

Chapter 1

What is deterministic?

Common definitions

In Mathematics and Physics:

Wikipedia: "A deterministic system is a system in which no randomness is involved in the development of future states of the system. A deterministic model will thus always produce the same output from a given starting condition or initial state. "

In Philosophy:

Wikipedia: "A deterministic system is a conceptual model of the philosophical doctrine of determinism applied to a system for understanding everything that has and will occur in the system, based on the physical outcomes of causality. In a deterministic system, every action, or cause, produces a reaction, or effect, and every reaction, in turn, becomes the cause of subsequent reactions. The totality of these cascading events can theoretically show exactly how the system will exist at any moment in time."

A sense of what Deterministic Networking is has emerged as the capability to effectively emulate point-to-point wires on switched networks that were initially designed to serve IT and then IoT applications, such as Ethernet and IEEE Std 802.15.4, and the IETF is now extending the Pseudo-Wire emulation of Ethernet over IP to provide Deterministic Services over larger networks.

Various flavors of this concept can be found around, for instance with the concept of Network Slicing at the 3rd Generation Partnership Project (3GPP) 1 . With these methods, an overlay, which is effectively a logical structure of meshed tunnels, would inherit physical properties such as a portion of the available buffers and bandwidth.

In this part, we study the characteristics of Deterministic Networking, how the concept can be achieved on wired infrastructures, and the direction taken by the standards bodies to enable deterministic properties on packet networks.

A game of convergence

At first, people realized that an on/off electrical signal could be perceived at very long distances and the telegraph was born. They realized that the voltage could be modulated and the analog audio signals, which still available today from devices equipped with those legacy jack and RCA connectors, joined the party.

In the last twenty years, voice, data and video networks have converged to digital over IP. Mail delivery has become quasi-immediate and volumes have multiplied; long distance voice is now mostly free and the videophone is finally a reality; TV is available on-demand and games became interactive and massively multi-player.

The convergence of highly heterogeneous networks to Information Technology (IT) resulted in significant drops in price for the end user while adding distinct new value to the related services.

Yet, and even though similar benefits can be envisioned when converging new applications over the Internet, there are still many disjoint branches in the networking family tree, many use cases where mission-specific applications continue to utilize dedicated point-to-point analog and digital technologies for their operations.

And even as a number of industrial protocols are now migrating to open standards such as Ethernet and IP, the typical OT network is kept isolated from the IT network and operated by a different crew of OT specialists, which yields double operational expenses compared to a converged network infrastructure and management.

The way to the IT/OT Convergence

Forty years ago, OT people found that information encoded as an analog modulation of current could be carried virtually instantly and with no loss over the distance; the basis of industrial control was laid out. Even today, most control signals rely on modulated current, typically between 4mA and 20mA, to report the variation of an observed phenomenon.

Then came digitization, which enabled to multiplex data with the control signal and manage the device, but at the same time introduced latency to industrial processes, the necessary delay to encode a series of bits on a link and transport them along, which in turn may limit the amount of transported information [8] The need to save cable and simplify wiring lead to Time Division Multiplexing (TDM) of signals from multiple devices over shared digital buses, each signal being granted access to the medium at a fixed period for a fixed duration; with TDM, came more latency, waiting for the next reserved access time.

A new context for the convergence

Statistical multiplexing, with Ethernet and IP, was then introduced to achieve higher speeds at lower cost, and with it finally came jitter and congestion loss. Some OT applications evolved to compensate for the transport degradation with recovery mechanisms and jitter absorption buffers, at the expense of yet some additional latency. But this did not seem acceptable for all, and in order to avoid those unwanted statistical effects, competing and not interoperable solutions appeared, driven by multiple standard defining organizations (SDOs), consortia and individual vendors.

Notable examples are, in the wired space 34 [44] 5 , PROFINET Isochronous Real-Time (IRT), POWERLINK, SERCOS III, CC-Link IE, and to some degree Modbus TCP and EtherNet/IP which are now converged in the CIP specification, and in the wireless space, WirelessHART™, ISA100.11a, WIA-PA, WIA-FA, iPCF 6 , and WISA 7 .

In the real world today, operational signals are still massively carried as a simple analog modulation of the electrical current, over costly point-to-point wires. In order to replace those wires with a cheaper Ethernet-based switched fabric federating multiple access links, a limited subset of deployed control networks made all the steps towards digital statistical multiplexing; but even those cannot interoperate with IT technology, and the convergence is stalled. The main technical reason for not converging those networks derives from the limits of the protocols that sustain the Internet as we know it today. With multi-hop operations and statistical multiplexing, IP technologies lack "determinism".

A new context for the convergence

In response to needs from different vertical OT industries, a new trend is emerging to bring determinism in IEEE and IETF technologies that were until then associated to IT: Likewise, a deterministic flow may experience short buffering in intermediate hops to guarantee that there are never two deterministic packets scheduled for transmission at overlapping times, in other words, that the link is always available at the scheduled time of transmission; because of the extra fixed latency that is added at each hop, most deterministic flows will incur a higher latency than with best effort hot potato forwarding.

The bus analogy (to deterministic circuit switching)

The goal of the reserved bus lanes in a city is to avoid that the bus is delayed by traffic jams and guarantee a good and repeatable experience for the passengers. Say a bus takes 30 minutes to travel from a given bus stop down the street from a user's home, all the way to his office. If there is a bus every 10 minutes, then the transit time for a particular user will always be somewhere between 30 and 40 minutes, depending on how lucky he is on a particular day, and this, regardless of the traffic in the car lanes. This user will certainly not use every bus that passes by, but he knows that there will be one soon enough, and that, 40 minutes later at worst, he will be at his desk.

Because there cannot be as many lines as end-to-end travels, our user may have to hop from one bus to another one or more times before reaching his final destination; switching between lines represents some additional transport time for him, and yields a huge reduction of number of lines and thus of operational complexity, resulting in increased profitability for the bus company.

A bus line is analog to the reserved circuits that can be scheduled in a packet-switched Deterministic Network; regardless of the load, they guarantee a periodic transmission opportunity with a bounded latency for asynchronous commands and alerts. In that analogy, the ultimate realization of a Deterministic Network appears to be a perfect emulation of the good old serial cable over a packet-switched network, transporting the exact same application, at a much lower cost than a full mesh of wires.

Determinism in a network

The vacation place analogy (to time-sharing)

For those of us who do not have 52 weeks of vacation per year, clever marketing has invented this concept of time sharing acquisition whereby an individual owns a flat in a nice vacation resort for just one week a year. That week, the individual will use the flat as if it was always his, will not find unexpected people in his bed at night, and, provided that all owners are decent enough, he will have no clue whatsoever of how many other owners also live in that flat at other times along the year.

We saw in section 1.5.1 that determinism requires that packets from a particular flow may need to be held in an intermediate node until the scheduled time when the free access to the transmission medium is guaranteed. This means that enough buffers must be available during that period to hold the packet; by analogy to the vacation place, there can be no unwanted occupant in the master bed.

The object of Deterministic Networking, which is to remove chance from the picture, requires tying physical resources to the protected flows. With scheduled operation, a same resource may be affected to different flows at different times, and the duration of this affectation, thus the number of deterministic flows that can make use of a same resource, is directly affected by the precision of the shared sense of time in the network.

It results that the reservation system that locks physical resource to well-dimensioned and identified flows must be aware of the device capabilities such as clock precision and forwarding latency, as well as the amount of buffers, timers and queues that are available to control in that particular device.

With appropriate shaping, a deterministic flow is fully isolated from any influence from other traffic, with no leak, no loss, and no latency whatsoever that could be imputed to other flows, whatever the load on the network. This isolation goes beyond that provided by a Virtual LAN (VLAN) 17 and its Layer 3 equivalent, the Virtual Routing and Forwarding (VRF) 18 ; those only prevent leak and eavesdropping, but cannot protect against congestion loss and latency induced by some other traffic that happens to share the same physical resources.

In other words, Deterministic Networking brings a new level of isolation and guarantees that are critical to converge OT control flows onto a shared IT infrastructure spanning the campus or the factory. To realize this to its full extension, a strict policing and shaping must be performed that filters out misbehaving devices, whether it is an external attack or a failing network node that may, for instance, repeat the last frame forever at line speed. In turn, the isolation brings a new form of security whereby attempts to influence a flow by injecting another in the same physical infrastructure becomes totally inefficient.

Determinism in a network

This is why until now, the extreme OT uses cases such as industrial motion control, which operates control loops at several hundred times per second, had to resort to specialized standards or proprietary designs, as detailed in section 1.3 on page 4.

In contrast, scheduled operations can utilize the medium for critical flows up to high loads, limited only by the precision of the clocks and the compute capability of the central controller. Voice calls and industrial control loops are like trains, the traffic is predetermined and with adequate dimensioning and scheduling, the network can be programmed to ensure a seamless and repeatable experience. This requires new shapers, a fine sense of time, and a better control of individual physical resources such as buffers in memory.

A deterministic flow must traverse the network in the same predictable fashion every time, regardless of the load of the network. There can be no observable influence whatsoever from any other flows. A Deterministic Network may be primarily loaded with deterministic flows and still maintain its predictable properties unmodified for each of them. This goes beyond the capabilities of the current QoS-based Internet technologies, where an increased load in one flow generates delays and losses, which are rapidly observable, on adjacent flows.

All in all

Transporting OT traffic requires a network that is reliable and jitter-free, and can be trusted to transport periodic and asynchronous commands with a bounded latency. An ever increasing bandwidth is not always a valid response in the face of high loads of time sensitive flows found in OT applications. This strategy was not accepted by the industrial community, which developed its own adaptions of Ethernet and IP to meet its specific needs. As opposed to QoS-based networks, a Deterministic Network must always retain its properties, even under high loads of critical flows.

To achieve this, it is necessary to schedule the timely operation of the network with the granularity of numerous tiny physical resources. While scheduling enables to provide the required guarantees, it may also yield an additional latency for the transported flows. A precise synchronization of the network must be obtained, so the required resources can be reserved for the individual packets within that particular flow at their precise time of arrival. Over the distance, tiny flows must be aggregated into larger ones that can be processed as a single entity.

With Deterministic Networks, the worst-case data loss and latency can be guaranteed in a consistent fashion when multiple services are deployed on a common converged network infrastructure. A deterministic flow is completely isolated in its own time-shared set of physical resources and cannot be influenced by any other traffic in any observable fashion; from the perspective of the application, a deterministic end-to-end connection appears as a dedicated point-to-point wire.

Partie I, Chapter 1 -What is deterministic?

As Deterministic Networking capabilities are deployed, wired and wireless links that are today limited to provide Internet connectivity can be used to replace any of the cables that we see in day-to-day applications, RCA connectors on the stereo gear, Serial Digital Interface (SDI) in professional video and AES3 in audio, HDMI and S/PDIF, their matches in the consumer domain, RS232 serial cables, fancy bus connectors in cars, 2 and 4-wires cables in industrial control networks, all of them. Wouldn't life be drastically simpler with just one connector?

A shared sense of time enables to transport -in fact to recreate-a clock signal that is used to synchronize both ends of an emulated serial cable.

And the perfect isolation between flows that is obtained to guarantee the required latency is also an improvement on the security side, and a factor of trust in the convergence for OT people.

Awareness of Deterministic Networking is now spreading 23 , and the new technology is generally perceived as an evolution of quality of service to bring a new level of guarantees for network-based services. True as it is, what Deterministic also and mostly brings to Networking is more revolutionary; it is the capability, for the first time, to carry any signal that was ever transported across a point-to-point wire, for any form of application that men ever devised, over multi-hop packet networks.

Chapter 2

The precursors

On Fast Reroute

Use-cases where two non-congruent paths are set up to ensure either a full redundancy, or at least a rapid fail-over, by selecting at the ingress between two pre-computed paths, abound in the art of networking. An example of such, the Linear Protection [62]1 of the Multi-Protocol Label Switching (MPLS) Transport Profile (MPLS-TP) [6]2 [23]3 uses specific OAM frames [START_REF] Allan | Administration, and Maintenance Framework for MPLS-Based Transport Networks[END_REF] 4 to monitor the liveliness of the routes and make rapid fail-over decisions.

To avoid the complexity of setting up non-congruent paths, and in order to react even quicker from where the problem actually happens, the IETF has devised the concept of fast-reroute [START_REF] Shand | IP Fast Reroute Framework[END_REF] 5 , which means rerouting around a failure from anywhere inside the network.

The IETF has proposed two approaches for IP fast-reroute, the original IP Loop-Free Alternates (LFA) [START_REF] Zinin | Basic Specification for IP Fast Reroute: Loop-Free Alternates[END_REF] 6 [10]7 , which attempts to find a path around a failing node or a failing link, with a variable coverage, and a new technique for IP Label Distribution Protocol (LDP) using Maximally Redundant Trees (MRT) [START_REF] Envedi | An Architecture for IPLDP Fast Reroute Using Maximally Redundant Trees (MRT-FRR)[END_REF] 8 , which draws a pair of trees in any bi-connected topology, each tree connecting every node and both rooted at the destination, in such a fashion that a breakage only blocks one of the trees, so that the other tree offers a path to destination.

One key benefit that is found in both techniques is to be compatible with, and actually leverage, an existing OSPF-based infrastructure; and one limit to both approaches is the lack of control on the generated alternate paths, in particular with respect to their relative costs. The detour may be long and expensive, in particular with MRT which computes global trees.

Partie I, Chapter 2 -The precursors

On SDN and Traffic Engineering

Software Defined Networking (SDN) 9 [59] 10 promotes a model that simplifies the network operation by automating the deployment of network resources from a centralized controller; at the extreme, the networking gear becomes a basic subservient to the master controller in practice, though, a distributed routing plane may still be associated with the SDN operation so as to reach the controller and handle the general purpose bulk of the traffic. Though still highly overhyped 11 , the momentum started with Openflow [START_REF] Mckeown | OpenFlow: enabling innovation in campus networks[END_REF] 12 is getting traction in campuses and cloud data centers through various vendor incarnations such as the Network Element Drivers 13 of Cisco's Network Services Orchestrator and Arista's Directflow 14 .

The art of Traffic Engineering (TE) at the IETF leverages routing protocol extensions to link-state Interior Gateway Protocols (IGP) such as OSPF [START_REF] Yeung | Traffic Engineering (TE) Extensions to OSPF Version 2[END_REF] 15 [4] 16 and IS-IS [7] 17 , as well as extensions to BGP [START_REF] Fedyk | BGP Traffic Engineering Attribute[END_REF] 18 , to report the topological information to a central routing component, which is implemented by a Path Computation Element (PCE) [START_REF] Farrel | A Path Computation Element (PCE)-Based Architecture[END_REF] 19 . Relying on that topological information learned, the PCE computes diverse paths and assign flows to those paths. But this technique yields a lot of human intervention and does not yet support deterministic properties. [START_REF] Phinney | Industrial Routing Requirements in Low-Power and Lossy Networks[END_REF] 20 discusses the need for the LLN routing protocol to compute multiple forwarding solutions. The centralized approach in TE is echoed by the best practice found in the art of industrial and vehicular networks, operating Time-Triggered Ethernet [START_REF] Kopetz | The time-triggered ethernet (TTE) design[END_REF] 21 , WirelessHART™and ISA100.11a. The limit of the model is probably the scalability of the controller, considering that the optimization of a Time-Triggered schedule is an NP-complete [13] 22 [61] 23 problem with a complexity that grows rapidly with the number of engineered flows. 9 www.hpc.mil/dren/sdn 10 22 Craciunas and Oliver: "SMT-based task-and network-level static schedule generation for timetriggered networked systems" 23 Oliver, Craciunas, and Stöger: "Analysis of deterministic ethernet scheduling for the industrial internet of things"

More closely related to OT applications of IoT and low-power radios, Industrial Routing Requirements in Low-Power and Lossy Networks

On BIER and BIER-TE

Bit Indexed Explicit Replication (BIER) [START_REF] Wijnands | Multicast using Bit Index Explicit Replication[END_REF] 24 is a network plane replication technique that was initially intended as a new method for multicast distribution. In a nutshell, a BIER header is added to IPv6 or MPLS packets; this new header includes a bitmap called the BitString that explicitly signals the destinations that are intended for a particular packet.

This means that:

1. the source is aware of the individual destinations and 2. the BIER control plane is a simple extension of the unicast routing as opposed to a dedicated multicast data plane, which represents a considerable reduction in OPEX.

The simplicity of the BIER technology makes it very versatile as a network plane signaling protocol. Already, a new Traffic Engineering (BIER-TE) [16] 25 variation is emerging that uses bits to signal segments along a TE path.

While the more classical BIER is mainly a multicast technology that typically leverages a unicast distributed control plane through IGP extensions, BIER-TE is mainly a unicast forwarding plane technology that leverages a central computation to setup a path, compute segments and install the mapping in the intermediate nodes.

From the BIER-TE architecture, the key differences between BIER-TE and BIER are as follows:

• BIER-TE replaces in-network autonomous path calculation by explicit paths calculated offpath by the BIER-TE controller host.

• In BIER-TE every BitPosition of the BitString of a BIER-TE packet indicates one or more adjacencies -instead of a BFER as in BIER.

• BIER-TE in each Bit-Forwarding Router (BFR) has no routing table but only a BIER-TE Forwarding Table (BIFT) indexed by SI:BitPosition and populated with only those adjacencies to which the BFR should replicate packets to.

The generic abstraction of an adjacency can be implemented as a link, a tunnel or a route along a path segment. With Segment Routing (SR) [START_REF] Filsfils | Segment Routing Architecture[END_REF] 26 a segment can be signaled as an MPLS label, or an IPv6 Routing Header. A segment may be loosely of strictly source routed, depending on the need for full non-congruence and the confidence that loose routing may still achieve that need.

Chapter 3

Enabling determinism in communication networks

Expected benefits in wired networks

A perfectly Deterministic Network would ensure that every packet would reach its destination, and would always do so in due time. In an imperfect world, Deterministic Networking must nearly eliminate packet loss, with the associated goal to guarantee a worst case latency for a packet, and all this whatever the overall network conditions are, effectively emulating the case of a point to point serial cable.

The main cause of data loss in a wired switch fabric is a statistical effect called congestion loss, whereby, at a particular moment, multiple flows entering a switch converge to a same outgoing port, in volumes that exceed the capacity of that port to output the traffic. It results that some of the traffic must be dropped, and one of the desired effects of Quality of Service (QoS) is to intelligently select the frames to be discarded.

Making networks more deterministic eliminates this statistical effect by maintaining at all time the amount of critical packets within the physical capabilities of the hardware. This can be achieved by the use of time-shared resources (bandwidth and buffers) per circuit, and/or by shaping and/or scheduling the packets at every hop.

Equipment failure, such as a switch rebooting, a broken interface adapter, or an unplugged physical wire, is a secondary source of data loss. When a breakage occurs, multiple packets are lost in a row before the flows are rerouted or the system may recover. This is not acceptable for critical applications such as related to safety. A typical process control loop will tolerate an occasional packet loss, but a loss of several packets in a row will cause an emergency stop (typically after 4 packets lost, within a period of 1 second).

Making networks more deterministic improves the resiliency against breakages and statistical transmission loss such as due to cosmic particles, typically by adding redundancy in the network path.

Finally, since the operation of a Deterministic Network rely on precisely applying a tight schedule, and the worst case time of delivery must be guaranteed, there must be a shared sense of time throughout the network that can be exposed to and leveraged by other applications.

Partie I, Chapter 3 -Enabling determinism in communication networks

Can we make Ethernet deterministic?

On precise time: A synchronization of clocks in the order of the microsecond can be achieved by a software implementation using the Precision Time Protocol (PTP) [START_REF] Meier | IEEE 1588 applied in the environment of high availability LANs[END_REF] 1 IEEE Std 1588, or one of its derived profiles, such as IEEE Std 802.1AS. With hardware assistance, this can be brought down to tens of nanoseconds for use in Smartgrid [START_REF] Ingram | Use of precision time protocol to synchronize sampled-value process buses[END_REF] 2 and even down to nanoseconds for military [START_REF] Mackay | Applications and Opportunities for the IEEE 1588[END_REF] 3 and 4/5G so-called fronthaul and crosshaul [START_REF] De La | An overview of the CPRI specification and its application to C-RAN-based LTE scenarios[END_REF] 4 applications.

With a precise shared sense of time, the switches can guarantee the exact forwarding time at each hop for TSN applications, and in-time delivery at the egress for AVB applications. Precise time is leveraged by Time-Triggered Ethernet (TTE) [START_REF] Kopetz | The time-triggered ethernet (TTE) design[END_REF] 5 to control the exact time of transmission; with TTE, the sender and the listener can agree that a certain packet belongs to a certain deterministic flow just because it is transmitted at a certain precise time, without the need to tag the packet. Keeping the Ethernet frames unmodified enables inter-working between TTE and classical Ethernet switches.

On timely transmission:

Another key aspect to provide deterministic latency guarantees is to ensure that the medium is free and can be accessed with no wait when the time comes for sending a deterministic packet.

In the case of full-duplex switched Ethernet, a sender owns the transmit medium, so there is no need to defend against an interfering transmission from other parties; a collision may be avoided either by preventing transmission ahead of the scheduled time by a guard time that is more than the transmission time of the largest frame, or by suspending a frame being transmitted to free the medium for the deterministic packet, which is the solution that the IEEE as selected with 802.1Qbu Frame Preemption.

By construction of the schedule, only one frame may be programmed during any particular window of time sized for one frame of maximum size, and there can never be a contention between deterministic frames.

If a non-deterministic frame is being transmitted at the precise time scheduled for a deterministic frame, its transmission is interrupted and a CRC is attached to validate the partial transmission. The deterministic packet can then be transmitted in time and in full, and then the non-deterministic transmission can resume starting at the offset where the transmission was interrupted, yielding no access latency for the deterministic frame.

On redundancy: Fast recovery techniques, in particular leveraging rings, already exist in the IT world, in particular with the ITU-T G.8032 Ethernet Ring Protection Switching (ERPS) 6 and Cisco's Resilient Ethernet Protocol (REP) 7 .

But a real Deterministic Network must bring resiliency against physical and logical failures and guarantee the continuity of operations in all conditions; this requires the physical redundancy of each involved piece of networking equipment8 , and the capability to compute non-congruent paths between source and destination and leverage Frame Replication and Elimination techniques.

Kirmann first introduced the Parallel Redundancy Protocol [START_REF] Kirrmann | IEC 62439 PRP: Bumpless recovery for highly available, hard real-time industrial networks[END_REF] 9 (PRP) and then the High-availability Seamless Redundancy [START_REF] Kirrmann | HSR: Zero recovery time and low-cost redundancy for Industrial Ethernet (High availability seamless redundancy, IEC 62439-3)[END_REF] 10 (HSR) for that purpose. The International Electrotechnical Commission (IEC) further standardized those methods [START_REF]IEC, Industrial communication networks -High availability automation networks -Part 3: Parallel Redundancy Protocol (PRP) and High-availability Seamless Redundancy (HSR)[END_REF] 11 for both the industrial [3] 12 and Smartgrid [START_REF] Tan | IEC 61850 based substation automation system architecture design[END_REF] 13 applications, and is now evolving the IEC 61850 [START_REF]IEC, IEC 61850 Communication networks and systems for power utility automation -ALL PARTS[END_REF] 14 standard for substation communication networks to include them [START_REF] Goraj | Migration paths for IEC 61850 substation communication networks towards superb redundancy based on hybrid PRP and HSR topologies[END_REF] 15 .

On reliability: Finally, critical applications such as Pro Audio16 demand a packet delivery ratio (PDR) that orders of magnitude better than the capabilities of a simple chain of switches along a path. Additional diversity, such as replication and elimination of a packet over non-congruent paths, enables to reach the required figures.

The maths is simple: If the probability of loss along a path is P , then, if total diversity and independence can be achieved between two parallel non-congruent paths, then the probability of loss with Replication and Elimination over those two paths becomes P 2 . With a loss ratio of an Ethernet fabric that is typically in the order of 10 -5 , sending redundant copies over two fully diverse paths yields a 10 -10 loss ratio, which enough to burn a blu-ray disk without a scratch in an AVB studio.

Applying this logic to a highly lossy multi-hop media such as a wireless link with, where a worse than 10% packet error rate (PER) without retries is not uncommon [START_REF] Shu | Packet loss analysis of the IEEE 802.15. 4 MAC without acknowledgements[END_REF] 17 , a redundant path still yields at best a PER of 1%, far from the 10 -5 wire equivalence that industrial networks are after.

Partie I, Chapter 3 -Enabling determinism in communication networks

Can we make wireless deterministic?

On transmission reliability: The radio medium, in particular in the industrial, scientific, and medical (ISM) bands that are shared not only between data networks but also with all sorts of interferers such as microwave ovens and radars, is orders of magnitude less reliable than classical wired networks such as Ethernet over copper or fiber; providing deterministic services over wireless appears extremely challenging.

Co-channel interferences are not the only possible causes for a frame loss; for instance, physical obstacles may happen to move in the way of the transmission and block the communication. Multi-Path Fading, which is due to multiple reflections that may reinforce or cancel one another out a few centimeters away, is a major cause of transmission errors; it does not affect all channels in a same fashion, and which channels are impacted is highly sensitive to the relative position of the sender and the receiver, and to their environment.

In short, the quality of a given channel is affected by multiple parameters that may vary brutally, and IEEE Std 802.15.4 transmissions over a fixed channel cannot be expected to remain stable over a long period of time. It results that basic 802.15.4 implementations which operate on a fixed channel will often suffer from intermittent delivery issues, and can only apply to low-end applications for which a consistent reliability is not a concern.

Channel Agility was added to improve the availability of the radio links by permanently sensing the channel, looking for an increase in Bit Error Rate (BER), and switching channel when loss becomes too high. This technique represents a clear improvement from the art of fixed channel, but it only fixes an error condition reactively, and a period of lossy transmissions is experienced before a new channel is selected. This is well suited for applications such as AMI/AMR metering, which do not require a wire equivalent reliability, but, at the same time, this does not provide the deterministic guarantees that industrial applications require.

A predictive technique that would enable to switch to a better channel before the problem even occurs would be ideal but early attempts [START_REF] Hwang | Energy efficient channel agility utilizing dynamic multi-channel CCA for ZigBee RF4CE[END_REF] 18 did not make it to mainstream. Effectively, one can leverage transmission statistics, observe activity on other channels and remember channels with a bad transmission record, so as to black list them. This may be efficient to protect against a stable Wi-Fi co-channel interferer, but there is no way to know if this is effectively the case, and if the problem will last and for how long. The physical phenomenons that are the most common cause of channel degradation, moving an object or starting a process, cannot be fully predicted by the radio device in most practical situations, and making Channel Agility proactive has appeared so far to be a red herring, yielding more complexity than actual benefits. Time-Slotted Channel Hopping (TSCH) [START_REF] Watteyne | Industrial ieee802.15.4e networks: Performance and trade-offs[END_REF] 19 is the best of breed with its simple

Partie I, Chapter 3 -Enabling determinism in communication networks

The only way to approach determinism in wireless transmissions -most people will prefer to use terms such as "highly predictable" instead of deterministic-is to schedule all the transmissions, and that is what TSCH and industrial WSNs do. A schedule such as illustrated with colored codes in figure 3.1 on the preceding page controls at which time and on which channel a frame is forwarded between which pair of nodes.

Scheduling implies that all nodes control the precise time of emission, which in turn requires a shared and precise sense of time. It is, for instance, possible to achieve in the order of tens of microseconds clock synchronization over an 802.15.4 link.

With fully scheduled operations, it is now possible to guarantee the time of delivery for those packets that make it to destination, in a deterministic fashion.

Additional Benefits from scheduling in wireless: In addition to the benefits listed in section 3.1 on page 17, scheduling provides specific value to the wireless medium. On the one hand, scheduling reduces transmission losses and with TSCH and its industrial derivatives, a wire-equivalent loss ratio of 10 -5 can be obtained: to achieve this, routes are computed so as to enable at least two forwarding solutions for every node, which ensures spatial diversity, whereas retries over the TSCH MAC provide both time and frequency diversity and effectively combats co-channel interference as well as multipath fading.

On the other hand, scheduling optimizes the bandwidth usage: compared to CSMA/CA operations, there is no blank related to IFS and exponential back-off in scheduled operations, though some minimal Clear Channel Assessment may be needed to comply with the local regulations such as ETSI 300-328. And because TSCH time slots provide a full time sharing operation, there is no limit to the ratio of guaranteed critical traffic.

Additionally, the capability to schedule the operation at the scale of a Service Provider (SP) network enables an operator to guarantee Service Level Agreements (TSCH) [START_REF] Gaillard | Managing IoT networks using service level agreements (SLA)[END_REF] 21 across customers who use the shared infrastructure.

Finally, scheduling plays a critical role to save energy; in IoT, energy is the foremost concern, and synchronizing sender and listener enables to maintain them in deep sleep at all times when there is no scheduled transmission; this optimizes sleeping periods by avoiding idle listening and long preambles; TSCH enables battery operated nodes that actually forward packets in a mesh topology for multiple years [START_REF] Watteyne | Industrial ieee802.15.4e networks: Performance and trade-offs[END_REF] 22

In a nutshell: While scheduling transmissions can guarantee the time of delivery, it is impossible, in the ISM band, to keep all possible interferers at bay. Co-channel interference, as well as the self-inflicted Multi-Path fading, which is due to echoes of the transmission, are unavoidable. So there is no way to guarantee the delivery of all frames.

It takes different mitigation techniques to avoid the different issues that affect wireless 3.3. Can we make wireless deterministic? transmissions. To combat them all, all possible forms of diversity should be leveraged, in the spatial domain by routing over multipath, in the temporal domain by retrying transmissions or sending copies over parallel paths at distinct times, and in the frequency domain with frequency hopping (within frames) or channel hopping (between frames).

Several times along the way to the destination, the work presented in this manuscript replicates and then eliminates the copies of a packet that are forwarded along parallel disjoint paths, hopping between frequencies with each transmission so that a copy along one path does not interfere with a copy along the other.

Chapter 4

The IETF DetNet architecture "Deterministic Networking (DetNet) provides a capability to carry specified unicast or multicast data flows for real-time applications with extremely low data loss rates and bounded latency. Techniques used include: 1) reserving data plane resources for individual (or aggregated) DetNet flows in some or all of the intermediate nodes (e.g. bridges or routers) along the path of the flow; 2) providing explicit routes for DetNet flows that do not rapidly change with the network topology; and 3) distributing data from DetNet flow packets over time and/or space to ensure delivery of each packet's data' in spite of the loss of a path. The capabilities can be managed by configuration, or by manual or automatic network management."

On Deterministic Networking

In parallel to industrial applications, the need for determinism in professional and home audio / video markets drove the formation of the Audio/Video Bridging (AVB) standards efforts in IEEE at 802.1. With the demand for connectivity and multimedia in transportation, AVB is being evaluated for application in vehicle head units, rear seat entertainment modules, amplifiers, camera modules, and engine control systems. Automotive AVB networks share the OT requirements for Deterministic Networks characteristics. This wider application scope for Deterministic Networks has led to the 802.1 AVB Task Group becoming the Time-Sensitive Networking (TSN) Task Group (TG), covering industrial and vehicular applications, and then to the formation of the DetNet1 WG at the IETF. With DetNet, the networks in consideration can extend beyond the LAN boundaries and secure deterministic forwarding over a composite Layer 2 / Layer 3 network will eventually be enabled.

The architecture in a nutshell

The architecture in a nutshell

The Deterministic Networking Architecture [START_REF] Thubert | Deterministic Networking Architecture[END_REF] 2 applies a centralized approach for a limited amount of deterministic flows, which share the network with a more classical distributed path computation and statistical multiplexing operation for traffic of lesser criticality and requirements.

In that approach, diverse applications can push their requirements over a Northbound Interface to a centralized controller, which translates these requirements in term of complex routes with replication and elimination capabilities, and pushes the result onto the network over a Southbound Interface, including precise operating schedules and timebased resource reservations. 1. the definition of data models to report the topology and the devices capabilities to the controller which is aware of the application requirements and can perform 2. the computation of a path that matches those needs; 3. the protocol elements to request a path set up for a given flow and configure 4. the Network Interface Card (NIC) in the end nodes, and the time-shared reservation of physical resources in the network nodes along the end-to-end path, and 5. the forwarding behavior for each flow.

Limits and perspectives

Deterministic Networking is not a matter of standard bodies and networking layers, (L2 vs. L3); nor is it about virtualization. What it is really about is a tight control of physical operations and scheduling of real buffers and queues, receiving, shaping and resending real packets at precise times with dedicated Hardware.

Such operation requires a network that is precisely synchronized; DetNet will inherit the precise clock synchronization from work done at other bodies, such as but not limited to, GPS, IEEE802.1AS, the Precision Time Protocol (PTP) IEEE Std 1588, or the Network Time Protocol (NTP) [START_REF] Neagoe | NTP versus PTP in com puter networks clock synchronization[END_REF] 6 . The precision of the synchronization limits the capability to tightly control the resources and thus the amount of deterministic traffic that can be applied onto the network.

Other limits of the centralized approach are on the one hand the complexity of the computation that hinders the scalability of the solution, and on the other hand the control plane overhead and the delays that are required to install a new path, or modify an existing path, from the controller across the network to all the intermediate nodes.

More of a constraint than a limit, the DetNet operation at Layer 3 will require services from lower layers to achieve the required properties end-to-end; the DetNet group will collaborate with 802.1 TSN, which is responsible for Layer 2 operations, to define a common design to support deterministic applications. A number of abstractions such as the end-host operation should be defined in a fashion that is agnostic to the choice of network used for the connectivity. This common design should ensure that the definition of the southbound and northbound interfaces is kept homogeneous between Layer 2 and Layer 3 so as to enable various inter-working models, such as DetNet transporting TSN on a deterministic pseudo-wire, or a mapping interface between a TSN network and a DetNet network at the edge of a deterministic L2 fabric.

The weakest link along a path will limit the whole chain, and an imperfect mapping at the interconnection between two networks with slightly different operations may induce a reduction of end-to-end results. The interconnection between a wired and a wireless network, in particular, can be expected to yield complex mapping issues.

The following chapters explore how deterministic can be applied on wired and wireless media, and which particular benefits are expected in either case. We'll see that determinism can be practically achieved, but through different methods and with different capabilities. The interconnection and the enablement of end-to-end deterministic capabilities are still a complete green field, open for further research.

Part II

Diving Deeper in Wireless

Chapter 5

The art of low-power wireless sensor network

So how do we make wireless highly predictable?

The industrial WSNs listed in section 1.3 on page 4 apply the principles discussed in section 3.3 on page 20 and schedule all possible transmissions to minimize collision losses.

Scheduling is achieved through TDM, by slicing time and affecting time slots to particular transmissions. This scheduled mode of operation is particularly adapted to well known, periodic flows for which a schedule can be computed in advance.

TSCH combines TDM with channel agility in order to defeat interferences, in particular Multi-Path Fading, which generally affects 2 to 4 channels out of the 16 available with IEEE Std 802.15.4 in the 2.4GHz band, and co-channel interference, in particular when it is located in a limited number of adjacent channels.

TSCH recognizes that smarts in channel selection do not pay off in practice, and all channels that are not black-listed are equally used; they are tried in a pseudo-random order, hopping between non-adjacent channels at each transmission; and though typically, at any given point of time, several channels present a high BER for a particular pair of devices, a series of retransmissions over a sequence of alternate channels eventually bypasses the issue after a few attempts, and an industrial-class reliability can effectively be achieved.

The complexity in TSCH is elsewhere; it comes from the need to ensure that the receiver is tuned to the same channel as the sender at the precise time of the transmission; this requires additional protocol elements to synchronize the network and schedule the transmissions.

A TSCH schedule can be viewed as the program of a mechanical piano that would play the same tune over and over, whereby the channels used would be the music. But the analogy stops here, since at the next iteration of a schedule, all transmissions are rotated by a pseudo random number, with a sequence such as below: Partie II, Chapter 5 -The art of low-power wireless sensor network the initial settings of Layer 2 keys and a secure connectivity for the configuration of the device. Since the Minimal Security assumes the one-touch manual setting of Pre-Shared Keys (PSK) on each individual device, there is an additional desire to enable fully autonomic operations and avoids the manual intervention.

Based on certificates installed by the vendor on the device, and a backend collaboration between the vendor and the target domain, the 6TiSCH Secure Join Protocol [START_REF]Richardson, 6tisch Secure Join protocol[END_REF] 15 will cover the phase that precedes the Minimal Security and enable the initial trust and key exchange between the device and the domain, eliminating the need for a PSK.

The group is now addressing the challenge to make the schedule dynamic, which involves a Scheduling Function [14] 16 that allocates and releases time slots dynamically for parent/child unicast communication along the RPL graph, based on the observed needs of the current flows, and a peer-to-peer protocol [START_REF] Wang | 6top Protocol (6P)[END_REF] 17 between adjacent nodes to negotiate the time slots.

When this work is complete, 6TiSCH may recharter to work on adapting DetNet to wireless and ultimately enable the IT/OT convergence for WSNs.

Chapter 6

The vision of 6TiSCH centralized scheduling

In order to fully realize its architecture, the 6TiSCH WG has to make work together: The wireless convergence

PCE vs. 6TiSCH

With 6TiSCH, a PCE controls the network via a Command and Configuration interface that implements specific device drivers, e.g. CLI, NETCONF or CoMi. This PCE inherits from the overall DetNet design but requires some specific awareness for such notions as channels, which must be added to the common DetNet design.

A similar concept arises when implementing DetNet on fiber optics, which requires the awareness of the light wavelength -the lambda-that is used for multiplexing. In that context, (MPLS) was already generalized as G-MPLS [START_REF] Mannie | Generalized multi-protocol label switching (GMPLS) architecture[END_REF] 1 by adding the implicit context of the lambda to make a switching decision.

6TiSCH base elements (time slots, schedule, chunks and bundles)

This can be ensured in a distributed fashion by the 6top Protocol [START_REF] Wang | 6top Protocol (6P)[END_REF] 4 , which leverages a new IEEE Std 802.15.4 Information Element (IE) in 802.15.4 frames to update the cells that are allocated between adjacent 6TiSCH nodes; that new IE was delegated by the IEEE Assigned Numbers Authority5 (ANA) to the IETF and allocations will be handled by the Internet Corporation for Assigned Names and Numbers (ICANN) 6 Internet Assigned Numbers Authority (IANA) 7 .

The schedule is enforced by the 6top sublayer; 6top resides at the upper Layer 2, above the IEE802.15.4 Medium Access Control (MAC) but below the 6LoWPAN sublayers for compression and fragmentation, and the IP(v6) Layer above it.

As illustrated in Figure 6.10, a classical IoT device will operate all layers, and may either switch or route a packet. Operating in 6top is akin to MPLS switching, which is sometimes seen as an intermediate Layer, like a Layer 2.5. This contrasts with the classical 6LoWPAN fragment operation in Layer 3 -aka routeover-mesh networks, where fragments must be reassembled at each hop, in order to enable the IP routing operation based on the IP header that is only present in the first fragment. Work on LLN Fragment Forwarding and Recovery [START_REF] Thubert | LLN Fragment Forwarding and Recovery[END_REF] 8 is ongoing to improve that situation and forward individual fragments all the way.

Partie II, Chapter 6 -The vision of 6TiSCH centralized scheduling

Applying DetNet to 6TiSCH

This section shows how a variation of the Available Routing Construct (ARC) technology can be leveraged to form the ladder, and then how timeslots can be scheduled and ACKs optionally suppressed to save additional energy.

In a nutshell, the major steps for 6TiSCH centralized operations as controlled by a PCE are as follows:

• Discovering and exposing the 6TiSCH topology to the PCE/controller • Programming a full schedule per node vs. TE-like path setup • Retransmission vs. elimination and replication • Detecting and rerouting around interferences

The WG has documented which kind of operations may be required from a central computer to establish a deterministic path over the 6TiSCH network. Some of this text was integrated in the Deterministic Networking Use Cases [START_REF] Grossman | Deterministic Networking Use Cases[END_REF] 9 Work Group document, in section 5 Wireless for Industrial 10 .

In order to avoid paraphrasing, that section is integrated here by reference.

A more complete description of the 6TiSCH requirements for DetNet and the concept of a Track can be found in the 6TiSCH requirements for DetNet, which is also integrated by reference to this thesis.

In that document, we detail the dependencies on DetNet and PCE controller to express topologies and capabilities, as well as abstract state that the controller must be able to program into the network devices to enable deterministic forwarding operations.

In particular, we elaborate on the design of a 6TiSCH Track. In order to cope with the high loss ratio on the wireless medium, there is a need to support the concept of Replication and Elimination, as combined with that of Retransmission, which is more specific to radio links. It results that a Track may be a lot more complex than a serial sequence of hops, and involve correlating multiple transmissions and reception as relating to a same packet.

In other words, a node may need to:

• Expect more than one copies of the same frame at some subsequent slotOffset/channelOffset and eliminate the duplicate; • Forward replicated copies of that frame at multiple slotOffset/channelOffset; • Perform retries when the copies are not received, which is a concept that is not present in wired Deterministic Networks.

Forwarding along 6TiSCH Tracks

In this thesis, we suggest to extend the concept of G-MPLS so that the physical properties of a transmission indicate not simply a next hop but a full context for Replication, Retries and Elimination, all done without the need to add information inside the frame itself, and leverage that concept to form complex Tracks.

The concept of a 6TiSCH

Track is yet to be fully elaborated, but this work, in particular with the 6TiSCH requirements for DetNet [START_REF] Thubert | 6TiSCH requirements for DetNet[END_REF] 11 , has largely contributed to shape it in its current form.

A deterministic flow is qualified with information such as:

• bandwidth requirements, which translates in a number of cells in a bundle; • precise time of packet transmission, which translates in the slotOffset in the schedule for those cells, and a • maximum latency, which translates in the alignment of schedules along a path, which must be computed by a PCE so as to meet the end-to-end deadline.

In order to closely indicate the deterministic scheduling of every individual packet, this work suggests a number of operational rules for 6TiSCH Tracks:

• The IEEE Std 802.15.4 destination MAC address in the frames is always set to multicast (0xFFFF), meaning that all nodes that have a Track programmed to listen to that particular timeslot are expected to accept the frame and handle it;

• one (or more than one) bundle are assigned uniquely to each frame in the flow; in other words, all the cells in the bundle carry a copy of a same frame;

• Multiple cells in a bundle indicate that a packet is to be retried in case of a loss; an acknowledgement is the indication that the other cells in the bundle will not be used. This denotes an OR operation (more on this in section 10.1 on page 67) between the cells in the bundle. The experimental phase of this thesis relies on multi-cell bundles to experiment with Retries associated with retransmissions. It can be noted that, from the standpoint of the transmitting node, it would not matter whether the receiving node is the same for all tries; a single bundle can be used to try to send to node A, and if that fails to node B, etc...

• Multiple transmit bundles denote a Replication. Regardless of the success or failure of the transmissions over bundle A, bundle B will be tried. This denotes an AND operation between the cells in the bundle. The experimental phase of this thesis relies on multiple one-cell bundles to experiment with Replication and Elimination while keeping the loss ratio high enough by avoiding retransmissions.

• Multiple receive bundles indicate an Elimination. In some cases, there is no point in listening to the next bundles once the frame is received. With the BIER-TE-based technique that is presented in chapter 11 on page 71, this thesis relies on receiving all the possible copies and performing an AND operation between the bitmaps of each copy so as to determine which transmissions worked or failed along the Track.

Partie II, Chapter 7 -Enabling the convergence

Converging deterministic and stochastic flows on a same network requires breaking that natural coupling. Either the network is deterministic and the challenge is to transport best effort traffic, or the network is deterministic and the challenge is the reverse.

As it goes, both have been tried. Applying QoS (IEEE Std 802.11e or derivatives) and with a widely underutilized physical medium, it is possible to make a stochastic packetswitched network transport Deterministic Flows. But under load from best effort traffic, this solution fails rapidly, faster on Wi-Fi than on Ethernet since 802.11e is a statistical QoS operation [START_REF] Mangold | Analysis of IEEE 802.11 e for QoS support in wireless LANs[END_REF] 1 .

The 6TiSCH architecture defines the other way around, with the challenge to enable stochastic IP traffic over a deterministic TSCH MAC. The idea is that a PCE reserves hard cells from the time/frequency matrix CDU matrix for deterministic flows; those cells cannot be reused of displaced; they are allocated when a flow is established, along complex paths called Tracks. The reservation ensures that a flow that is placed on a Track cannot be influenced whatsoever by stochastic IP flows, which can only use the unreserved cells.

What's free in the CDU matrix is partitioned in chunks of soft cells that are made available for best effort traffic. This Time and Frequency Division Multiplexing technique is how 6TiSCH ensure the co-existence of deterministic and best effort traffic on a same medium.

Introducing the ARC technology

In both cases, the packets exit on the other end of the ARC and from there take the shortest path to reach the destination.

The yellow packet continues to the destination Omega, but the red packet faces a second breakage in another ARC further down the road, and is turned again within that other ARC. Since each ARC is its own recovery domain, the second breakage is corrected independently and the red packet also makes it to the destination.

Technically-speaking, an ARC can be implemented as a sequence of hops with two ends. It results that one ARC is, configuration-wise, very similar to an MPLS-TP path; rerouting along an ARC can also be seen as a particular application of existing fast-reroute.

An ARCset is typically computed by a separate Network Management System or an SDN controller. ARCs extends the concept of SDN centralized routing computation to any traffic, and proposes a new balance of responsibilities between, on the one hand, the slow-response centralized routing computation, and, on the other hand, the quick-response forwarding-plane operations, which are still distributed and can be automated.

An ARC deployment in an IPv6 or an MPLS network may leverage Segment Routing [START_REF] Filsfils | Segment Routing Architecture[END_REF] 8 to direct packets towards the appropriate edge.

ARCs offers a new model and a number of issues that were difficult to solve with greedy technologies appear a lot simpler from that new perspective. We explored some of those capabilities in past work (published as US patents):

(i) Shared Risk Link Groups (SRLG) refers to groups of links that may fail together because they share physical properties. An example of SRLG may be a bundle of optical fibers that may be cut at the same time by a digging engine. [

Centralized diverse paths computation with Replication and Elimination

As discussed in chapter 8, there are a number of techniques in the art that can be leveraged to build a pair of non-congruent paths between a source and a destination. But in order to enable replication and elimination inside the network, there is an additional need to provide a number of bridges, or rendez-vous points, between the diverse paths, and new technology must be introduced.

Since we must avoid single points of failure, we want to build a topology with the general shape of a ladder based on two non-congruent paths, as opposed to that of chain of a sausages for instance. To achieve this, a number of interconnection paths must be setup to transport the frames between the two non-congruent paths.

Running an example of ARC Chain building

The application of ARCs to bi-casting for replication and elimination was presented in Applying Available Routing Constructs to bicasting [START_REF] Thubert | Applying Available Routing Constructs to bicasting[END_REF] 1 and an associated patent [START_REF] Thubert | Bicasting using non-congruent paths in a loop-free routing topology having routing arcs[END_REF] 2 . This section presents a variation of the OLAF [START_REF] Thubert | Available Routing Constructs[END_REF] 3 technique that is designed to form complex Tracks, which, in their simplest form, adopt the general shape of a ladder; the ladder has two parallel disjoint paths between a source and a destination, and then steps that connect the two paths and enable replication and elimination several times along the way.

This new mechanism is complemented in section 10.2 by one that schedules timeslots in an appropriate order, and whereby ACKs are optionally suppressed to save energy.

In figure 9.1 and 9.2, the adapted OLAF technique is applied to build an ARC Chain to protect a Track between A and Z with multiple replication and elimination points. 1 Thubert and Wijnands: Applying Available Routing Constructs to bicasting 2 Thubert, Bellagamba, and Wijnands: Bicasting using non-congruent paths in a loop-free routing topology having routing arcs 3 Thubert and Bellagamba: Available Routing Constructs

Partie III, Chapter 9 -Centralized diverse paths computation with Replication and Elimination

Chapter 10

Computing and scheduling time slots

Forwarding rules

The expectation is that a central computer, such as a Network Manager or a System Manager used in existing industrial deterministic WSNs, will compute the ARC chain and then set up forwarding steps in each node in the form of a schedule. The schedule in the node may be complex, and will be represented in the form of logical 'AND' and 'OR' operation between time slots, 'OR' meaning that a successful completion for one of the OR'ed steps removes the need to perform the other OR'ed steps.

Due to operations such as retries and elimination, a same packet may be received at any of several time slots in each iteration of the schedule, resulting in an 'OR' operation. That packet is then forwarded at some other time slots, using 'OR' operations between equivalent time slots for retries, and 'AND' operations between timeslots that reach different next hops and are used for replication. This way 6TiSCH uses that concept is discussed in more details in section 4.1.2. Replication, Retries and Elimination of 6TiSCH requirements for DetNet [START_REF] Thubert | 6TiSCH requirements for DetNet[END_REF] 1 and section 6.4 on page 53 details how bundles of cell can be related to these logical operations.

If the cause of a loss is linked to the transmission channel, for instance in case of a cross-channel or a multi-path interference, a retry in the same conditions (same peers, channels, etc. . . ) has an increased probability to fail as well, for the exact same reasons that caused the first loss. TSCH restores some of the chances by adding channel diversity, and the proposal below uses an additional spatial diversity to increase the chances again, using replication and elimination inside ARCs along an ARC chain. On the top right of the picture, a summary of the arc and incoming links from upper ARCs are represented; arrows are represented in the same direction on that picture and in the schedule below that is progressing stepwise below, each step a timeslot.

Detailed steps

Partie III, Chapter 10 -Computing and scheduling time slots

Computation rules

The exact times or exact channels that are being used at each hop can actually be chosen by the central computer -a PCE. Depending on the number of flows that have to be accommodated, the PCE has more or less degrees of liberty to find a solution that fits all the requirements for all the paths. The overall problem is similar to that of Time Triggered Ethernet in the wired space, knwon to be NP-Complete.

The sequence has to be computed so that the packets progress towards the destination, and to ensure that all possible chances of reception have happened before attempting transmission; and so is the overall maximal latency for a certain frame, which is implemented as the delay between the first injection of the frame inside the ARC chain by A, and the last chance that Z had to receive the frame, in this scenario from L.

With this patented technique [START_REF] Thubert | Installation of time slots for sending a packet through an ARC chain topology network[END_REF] 2 , the central computer respects the rules summarized below to compute the schedule in each node (and the 'AND' and 'OR' operations):

• A scheduled frame or packet is associated to a number of "receive" and "transmit" time slots at any given node;

• From the perspective of a node, the first slot is a "receive" timeslot and the last is a "transmit" timeslot;

• There may be intermixed "receive" and "transmit" time slots, though having all the "receive" time slots first is preferable;

• A node in an ARC copies a frame received from a parent ARC as soon as possible inside the ARC, left and right, not down;

• But it does not copy in one direction if it already got the frame from that direction;

• And it may not listen on a "receive" timeslot if the frame was already received from any direction along the ARC.

If there is no retry scheduled, the Layer 2 ACK may be omitted, so both sender and receiver can go to sleep immediately after the transmission, which saves additional energy.

Chapter 11

Controlling Replication and Elimination using BIER headers

This seminal idea for this chapter was presented as OAM and Time Slot Control in a Deterministic ARC Chain Topology Network [START_REF] Thubert | OAM and Time Slot Control in a Deterministic ARC Chain Topology Network[END_REF] 1 . It was then adapted to in-band signaling based on Traffic Engineering for Bit Index Explicit Replication (BIER), which is introduced in section 2.3 on page 15, and in that shape, was proposed to the IETF Design Team for the DetNet data plane. This section was finally exported and published at the IETF as BIER-TE-based OAM, Replication and Elimination [89]2 .

Technical solution description

We leverage BIER-TE to activate the Replication and Elimination functions in a manner that is abstract to the data plane forwarding information. An adjacency, which is represented by a bit in the BIER header, can correspond in the data plane to an Ethernet hop, a Label Switched Path, a network segment [START_REF] Filsfils | Segment Routing Architecture[END_REF] 3 , or it can correspond to an IPv6 loose or strict source routed path.

In a nutshell, BIER-TE is used as follows:

1. A controller computes a complex path, which 6TiSCH calls a Track. The complex path takes the general form of a ladder to enable Replication and Elimination between disjoints path with no single point of failure, as illustrated in figure 11.1 on the following page.

The analog of the steps of a ladder are the adjacencies that enable to copy packets from one path onto the other and recover intermediate losses, whereas the side rails are the main disjoint paths.

2. The steps, and the segments of side rails in-between steps, are the adjacencies. With the methods proposed in this chapter, the adjacencies can be activated on-demand and on a per-packet basis by setting or resetting bits in the BIER header.

Additional details

The BIER header is of variable size, and a DetNet network of a limited size can use a model with 64 bits if 64 adjacencies are enough, whereas a larger deployment may be able to signal up to 256 adjacencies for use in very complex paths.

For the DetNet data plane, a replication point is an ingress point for more than one adjacency, and an elimination point is an egress point for more than one adjacency.

A pre-populated state in a replication node indicates which bits are served by this node and to which adjacency each of these bits corresponds. With DetNet, the state is typically installed by a controller entity such as a PCE. The way the adjacency is signaled in the packet is fully abstracted in the bit representation and must be provisioned to the replication nodes and maintained as a local state, together with the timing or shaping information for the associated flow.

The DetNet data plane uses BIER-TE to control which adjacencies are used for a given packet. This is signaled from the path ingress, which sets the appropriate bits in the BIER BitString to indicate which replications must happen. The replication point clears the bit associated to the adjacency where the replica is placed, and the elimination points perform a logical AND of the BitStrings of the copies that it gets before forwarding.

As is apparent in the examples above, clearing the bits enables to trace a packet to the replication points that made any particular copy. BIER-TE also enables to detect the failing adjacencies or sequences of adjacencies along a path and to activate additional replications to counter balance the failures.

Finally, using the same BIER-TE bit for both directions of the steps of the ladder enables to avoid replication in both directions along the crossing adjacencies. At the time of sending along the step of the ladder, the bit may have been already reset by performing the AND operation with the copy from the other side, in which case the transmission is not needed and does not occur (since the control bit is now off).

Chapter 12

Compensating loss with overhearing: the Leapfrog Collaboration

So far we studied ways to build a redundant path with Replication and Elimination, program it in a TSCH network and then signal dynamically which adjacencies are to be walked through on a per packet bases. This provides a deployable solution to realize a Deterministic Network over a wireless TSCH medium, but is energy greedy and some optimization should be found in order to address a larger scope of application.

This chapter proposes a novel variation of promiscuous listening -overhearing-techniques that allows two parallel paths to protect one another within one Track. In other words, the chances of loss in radio transmission along one path are counterbalanced by another radio property which produces redundancy on the receiver side.

Related work

Experimentation has proven that simple transmission over IEEE Std 802.15.4 without retries, though it is permitted by the standards, yields a high packet error rate [76]1 .

As shown in previous studies [START_REF] Mahmood | Reliability in wireless sensor networks: A survey and challenges ahead[END_REF] 2 , the mechanisms used to improve reliability in WSNs are typically either based on retransmission or on redundancy. The most widespread paradigm is retransmission and acknowledgement, aka ARQ, for Automatic Repeat re-Quest, also known as Automatic Repeat Query. ARQ at Layer 2 is compatible with the use of an IP stack above, but care must be taken on how its implementation at various layers of the network interact with one another [START_REF] Fairhurst | Advice to link designers on link Automatic Repeat reQuest (ARQ)[END_REF] 3 . The principle remains at all layers to detect the transmission failure (often by a gap between transmissions or from an acknowledgement timer expiration) and try that transmission again, until a positive acknowledgement is received, or a maximum amount of retransmission attempts have failed, in which case the connection is dropped.

Although ARQ is a simple and quite efficient mechanism, the technique falls short for deterministic WSNs. On the one hand, retransmission leads to additional delays and jitter that may affect the transport protocol or the application. On the other hand, re-Partie III, Chapter 12 -Compensating loss with overhearing: the Leapfrog Collaboration transmitting a packet consumes energy for both the sender and the receiver and utilizes the spectrum without ensuring that this sending will be more successful than the previous one.

In order to alleviate the energy consumption of retransmission mechanisms, several pieces of work used overhearing to implicitly acknowledge the packet reception. For example, Lee and Huh [START_REF] Lee | Reliable data transfer using overhearing for implicit ack[END_REF] 4 use implicit acknowledgement by overhearing when the receiver in turn forwards the packet. Maalel, Roux, Kellil, and Bouabdallah [START_REF] Maalel | Adaptive reliable routing protocol for wireless sensor networks[END_REF] 5 enhanced the mechanism with spatial diversity by maintaining an ordered list of neighboring node susceptible of retransmitting the packet.

In his PhD thesis, Lampin [START_REF] Lampin | Urban wireless sensors networks : Applications and protocols[END_REF] 6 suggests to leverage overhearing along a wireless graph with a particular addressing to skip one hop if the next hop already heard a packet. A similar technique was envisioned for application in Wi-Fi in Path shortening in a wireless mesh network [START_REF] Thubert | Path shortening in a wireless mesh network[END_REF] 7 . The two concepts that were missing at the time were the 6TiSCH Track to define the overhearing nodes and the schedule to optimize the operation by having only the right nodes listen at the right time as the packet progresses along the Track.

Overhearing is also widely used to address security issues as shown by Alghamdi, Wu, Zheng, and Kanhere [1]8 .

Introducing the Leapfrog Collaboration

In our model, a controller computes (at least) two parallel or interleaved paths for one Track, such that the nodes on one path may promiscuously overhear transmissions along the other path. For a scheduled transmission from a node X to a node Y, a listen-only time slots is scheduled in at least one alternate node Z that is also in the complex Track and preferably closer than X to the destination. This entry is added to Z's normal forwarding schedule to give it an additional chance to receive a frame by overhearing transmissions on the other path before its turn to transmit.

It results that globally, the schedules in the devices are organized with more receive slots than transmit slots, and that the extra receive slots are configured to refrain from generating an acknowledgement, since the extra receiver is overhearing and is not the intended recipient, and as optional, so the node listens only if it did not already receive the packet in a prior receive slot.

Running an example of Leapfrog Collaboration

Running an example of Leapfrog Collaboration

In the example illustrated in figure 12.1, we assume that due to radio transmission range and instabilities, the chances to receive decrease rapidly with the distance, e.g., B receives roughly 80% of A's transmissions, but C only gets 35% of them, D receives roughly 75% of C's transmissions, but E only gets 40% of them, etc... A, C, E, G, I andvia B, D, F andH, respectively. As illustrated in the top left of figure 12.2 on the following page, we compute the ladder as a double helix, making it so that the transmission across the steps is actually better than that along the rails. Whereas the art of frame Replication and Elimination would blindly forward a copy of the frame on each of these paths, the proposed Leapfrog Collaboration takes advantage to the shared properties of the medium to compensate for the additional loss that is incurred with radios by leveraging overhearing by other nodes than the expected destination of the transmission.

A schedule is set up with promiscuous listening between the paths, one path being operated on even time slots and the other on odd time slots. In order to save additional energy, a policy is set up to send on an even timeslot only upon detection of a failure on the preceding odd timeslot, as observed from a missing overheard L2 Ack.

Part IV

Experimentations

With the art of IEEE802.14.5 TSCH, we can build radio meshes that exhibit a packet delivery ratio similar to that of an an Ethernet switched path, typically with a loss in the order of 10 -5 . In order to experiment anything, we need a certain amount of loss in the transmission. This amount must be realistic, and yet, in order to run our test in reasonable times, we had to find ways to keep them orders of magnitude above the normal levels that would be experienced in a real industrial deployment. So we voluntarily placed our devices in crowded areas, and applied different types of errors and anomalies, which appear clearly in the resulting graphs. In turn, these anomalies affect some of the results, and in particular give the wrong image of the actual delivery ratio that can be obtained with the technologies that are being demonstrated here. This should be considered when reading this part.

A warning on loss ratio

Partie IV,

Rover: the test tool

OpenWSN boasts an active support community and is maintained up to date with the latest 6lo, ROLL and 6TiSCH specifications; it was ported on a variety of software and hardware platforms including OpenMote and FreeRTOS. The OpenMote/OpenWSN combination was used at ETSI interop tests 5 which took place in the recent years. OpenWSN comes with a Python test tool called OpenVizualiser (OV). With Rover, we split the OV software so that a central computer running the full-fledged master code can control a number of slaves running on the remote Raspberry Pis over IP, providing a single system image for the user and the test tools, as shown in figure 13.3.

We described the test bed in full details in "Rover: Poor (but Elegant) Man's Testbed" [9]6 . Zacharie Brodard modified the OpenWSN software that processes the 6LoWPAN Routing Header (6LoRH) [START_REF] Thubert | 6LoWPAN Routing Header[END_REF] 7 to support the BIER 6LoRH encoding that we described as a new draft proposal [START_REF] Thubert | Replication and Elimination, Internet-Draft draft-thubert-bier-replication-elimination-00[END_REF] The semantics of the BIER bits was directly encoded as a schedule in the individual devices and programmed through Rover.

Jiang Hao then added support to read the RPL topology and generate the track automatically. He also added the automation that is experimented with in chapter 15 on page 97.

In summary, Rover is an IoT testbed which combines OpenWSN software and Open-Mote hardware in an easy-to-use, flexible and cheap solution. It offers an ideal real-world experimentation environment for both academic and industry R&D activities, and will help shape the future of the Internet of Things.

Rover improves to the OpenVisualizer software by allowing node to connect remotely. Through the Rover bases, we could reprogram or reflash the OpenMotes, install new schedules and export all the statistics and results if the tests we ran.

The test layout

Using Rover, we ran the series of tests described in this part with a kit of eight Rover bases, interconnected over the Lab Ethernet wires over IP, as described in figure 13.4. The exact positioning of the devices in the PIRL lab is visible in figure 13.1 on page 89.

Experimental results

Experimental results

The top plots on Figure 14.2 on the facing page represent, over the course of one experiment, with time as the x-axis, the number of time slots, represented as the y-axis, that it took to reach destination. Each packet is represented by a blue cross. The packets that did not make it to the destination are represented with a latency of -1, which shows as a negative number of time slots.

The third plot, bottom left, represents the same thing for the deterministic Track, and the fourth on the bottom right has the aggregated results, by number of time slots used on the x-axis, and percentage of successful deliveries on the y-axis, the green column representing the deterministic Track.

Analysis

The top figures show the expected statistical effect of ARQ, whereby depending on the number of retries the packets suffer a variable delay with a minimum observed latency of 4 slots (all transmissions successful) and a maximum of 19 slots (very rare, observed at 18:00 on the red path).

This statistical behavior explains the roughly Poissonian shape of the envelope of the blue and red columns on the aggregated view, bottom right. The most significant result in this experiment appears with the comparison, on a same graph, of this envelope with the green peak that represents the latency along the complex Track. We confirm observations that were made with TTE and the theory that we developed in the train analogy in section 1.5.1 on page 7, that deterministic traffic usually -but not always-experiments a higher latency than hot potato traffic, but also that this latency is a constant. Indeed, the bottom left figure shows that the green packets experience a constant latency of exactly 10 time slots observed along the complex Track, which is explained by the fully scheduled operation and the total absence of retries along that Track.

We saw in section 3.2 on page 18 that if the probability of loss along one hop on the red or the blue path is roughly P then the probability of a loss on both of the parallel hops is P 2 . This explains why, though rare, we still see packet losses on the green Track.

Comparing the blue and the green Tracks, we see that for most packets, a linear path with TSCH takes the packet faster and with a better energy budget than the complex Track. This is expected since the extra transmissions are performed reactively, only when a first transmission failed, whereas in the case of the complex Tracks, duplicate transmissions happen even if there is no loss. The channel rotation in TSCH preserves equal chances of success for a retransmission. Now, comparing the red and the green Tracks, we see that for the cases that TSCH does not compensate, e.g., the device that reboots at 21:00, the complex Track brings a superior behavior, maintaining a rate of delivery at all times.

Partie IV, Chapter 14 -Eliminating jitter with Replication and Elimination Chapter 15

Saving energy with BIER-TE-based replication automation

In this test, we evaluate the benefits of controlling Replication and Elimination using BIER headers as discussed in chapter 11 on page 71 in order to save energy. With our BIER-TE approach, each bit in the transported BIER bitmap encodes an adjacency; when a given bit is set, the corresponding adjacency is traversed at its scheduled time, otherwise it is not used for this packet. Note that the schedule is the same regarless of the bit settings in individual packets, so the latency is constant regardless of the portion of the Track that is being used.

Test description

We saw in section 14.3 on page 95 that with TSCH operation along a serial path and the classic hot potato operation, most though not all packets make it to destination faster and at a lower energy budget than with full replication and elimination, which consumes extra time and energy even when there is no loss. This shows that TSCH coupled with ARQ is indeed an excellent technique, to the point that apart from extreme use cases such as safety, the energy budget that is consumed in replication and elimination appears difficult to justify.

For our test, we reuse the logical topology that was described in section 14.1 on page 93 and make it so that the lower path on the top left experiences a higher loss rate than the top path on the right. We establish a control loop whereby a serial path is used till there is a loss; upon a loss, the source is instructed to set all the bits in the BIER bitmap, so as to establish where the problem comes from. If a link is found to be faulty, a new serial path is computed that routes around the problem, and this path is used for a while. Periodically, the normal path is retested and if it is operative again, then the initial bitmap with the minimum number of bits set is restored.

As illustrated in figure 14.1 on page 93, the shortest serial path has four hops, A → C, C → E, E → G, and G → H, so it can be encoded with only four of the eleven bits that are required to represent all the adjacencies in that graph. When the full replication and elimination is used, a transmission requires almost three time as much energy. Both the yellows and the green bars correspond to Replication and Elimination, but the yellow one leverages the BIER bitmap to discover faulty radio hops, and dynamically adapt the forwarding plane to route around them without affecting the routing or the scheduling that are maintained the same during the whole experiment.

As before, the red and blue bars represent the serial paths with standard reactive retransmission upon loss (ARQ).

For all transmissions, we leverage TSCH over 16 channels so the frequencies that are being used are rotated for each packet. Finally, figures 15.4 on page 100 and 15.5 on page 100 represent the number of losses in a row, the latter being zoomed to at least two losses in a row so that the number of occasions of four losses is still visible.

Analysis

Looking at the loss that is represented for a latency of -1 in figure 15.1 on page 98, we see that the delivery on the fully replicated Track on the bottom right inherits from the good properties of the top path, whereas the automated energy-saving Track on the bottom left, which is laid out along the lower path, experiences an even higher loss since it has no scheduled retransmission. This is also illustrated in the left of figure 15.1 on page 98 with the yellow bar associated to latency -1, which is much higher than the blue or the red.

Comparing the latency spectrum of the full replication in green, with that of the automated energy-saving Track in yellow, we see that the success rate drops from 95% to 83% in our experiment. This is the trade off that we are making in this model to save energy.

Indeed, figure 15.3 on page 99, that illustrates the energy consumption per packet, shows that almost all the packets are sent at the minimal energy cost. Figure 15.3 on page 99 is a zoomed window of time to make each transmission visible. Packets that are sent with the full bitmap require eleven transmissions, so they appear as a green cross at y = 11. In contrasts, packets that make it along the shortest path appear as a green cross at y = 4. We see small bursts of rerouted packets with one additional hop at 22:31:00, and a slightly longer one at 23:31:00.

In the former case, we blocked two links and the workable path involved two additional hops, so the rerouted packets appear as green crossed at y = 6, whereas in the latter case, we rebooted one device and only one additional hop was required for the time necessary for the node to rejoin the network.

Figure 15.5 on the facing page shows that the rapid reaction to the loss effectively permits to avoid the critical case of four losses in a row, even with the high degree of loss that is seen in this case. But it must be noted that this requires an efficient back channel for communication. If the back channel is implemented with 6TiSCH minimal support [START_REF] Vilajosana | Minimal 6TiSCH Configuration[END_REF] 1 , it may be that the latency involved delays the reaction to the problem, adding complexity to the set up.

In summary, the control loop that we put in place excels at saving energy and avoids the condition of four losses in a row, which effectively meets the requirements that we considered for this experiment. On the other hand, this technique yields a high loss ratio, and the return channel to the controller may be problematic to operate in a low power lossy network without a reliable back channel.

With that information, the root can build a source-routed path to the target device by starting from the target, looking up one of its parents, add it to the tail of the source route path, and recurse till it reaches self.

In our example, Node A is the root, and node H is the target of the messages. With RPL, H advertises to the root that it has 2 parents, F, and G; G advertises D and E, etc... C and B have a single parent, that is A.

Since RPL cannot advertise siblings, we end up with a slightly different vision of the topology, and instead of building the ladder-shape logical structure like the one represented in figure 14.1 on page 93, we end up with braided paths instead, as illustrated in figure 16.1 on the preceding page.

In addition to the red and blue paths that use TSCH with retries, we introduce a purple Track that has the full replication and elimination, a green Track that is only composed of the upper and lower paths operating in parallel but with neither intermediate replication nor retry, and a yellow Track that similar to the green one but which enables overhearing of the transmissions on the other path.

The green Track exhibits a low latency by avoiding to schedule the crossing transmissions between the paths. But since there is no retry, it is highly lossy. On the other hand, the purple Track is reliable but slow. We apply the schedule that is represented in figure 16.2 on that topology and run it in a noisy environment. The objective of this test is to demonstrate how close we can be to the best of both worlds by overhearing the activity of one path from devices along the other. We need an extremely high error rate for our experiment. So we place it in an environment with a high level of co-channel interference from Wi-Fi users, and place obstacles on the line of sight path.

Analysis

Though this test involves a much higher loss rate than the previous ones, the envelope of the red and blue plots, as compared to the purple peak, reproduces the results that we have already seen and analyzed in section 14.3 on page 95; the purple peak representing replication and elimination is much shifted to the right compared to the serial paths with retries in red and blue, indicating more transmissions thus more latency and energy spending.

The green Track, which corresponds to full replication and elimination only at the edges and forwarding on the blue and red paths in between, is certainly an improvement in terms of latency and energy spending, but it yields a loss rate that is in the same order as that experienced by either blue or red.

The yellow bar in figures 16.4 on the facing page and 16.5 on the preceding page, which corresponds to the Leapfrog Collaboration, effectively seems to approach the best of both worlds; the latency is reduced in the exact same fashion for the yellow Track as it is for the green Track, but the delivery ratio is increased by almost thirty percent.

The energy consumption [START_REF] Vilajosana | A realistic energy consumption model for TSCH networks[END_REF] 2 for the Leapfrog Collaboration Track is somewhere in between the green and the purple cases, since, compared to the green case, we spend additional energy for listening, but not for sending. Since the energy spent on listening is roughly equivalent to that spent in transmitting, we can estimate the energy budget of the green Track as roughly 16 such units, that of the purple Track as 30 and that of the yellow Track as 21, so the Leapfrog Collaboration in this case consumes roughly 30% more energy than the green Track, whereas the purple Track is 50%more.

Considering the huge amounts of errors that we forced, it is not surprising that four losses in a row happened on all types of Tracks, and this manuscript should not be used to dismiss the proposed techniques for industrial application, all the contrary; what we wanted to illustrate is visible in figure 16.5 on the facing page, which shows that compared to serial paths with retries or in parallel, the full replication and the Leapfrog Collaboration both provide a three to four times improvement.

At the same time, we see that the Leapfrog Collaboration is only a few percentile points below the full replication, for the considered criteria of reliable delivery, as visible on the left of figure 16.4 on the preceding page, and avoidance of four losses in a row, as visible on the right of figure 16.5 on the facing page, within a latency that is reduced to 8 time slots.

Chapter 17

Conclusion on our experimentation

We saw that hopping channels with TSCH is mandatory to run our tests reliably and used TSCH at all times, for all types of Tracks. Then we applied different techniques to optimize the two critical properties by which we can rate the applicability of a wireless solution to industrial applications, reliable delivery within a bounded latency and maximum number of losses in a row.

Here is what we found:

• we saw that a serial nailed down path with retries is actually challenged if packets must be delivered reliably within a bounded latency; retransmissions introduce a delay such that some packets arrive too late and are considered as good as lost; using two parallel paths aggregated into a complex Track in place of ARQ enabled to deliver the packets timely, but the loss ratio appeared too high;

• for the criterion that the loss of one packet is acceptable, but four losses in a row are not, we saw that replication and elimination along a complex Track enables the expected user experience at all times, but also that it introduces a latency that is sometimes not acceptable; moreover, the extra energy budget may reduce the applicability of the approach.

the Leapfrog Collaboration appears to be nearing the best of both worlds, passing the reliability and latency criteria and yet doing so within reasonable time and energy budgets.

Compared to hot potato with ARQ along a serial path, Leapfrog provides a guarantee against accidental link or device failures, and a bounded latency for all packets. A serial path with retries has a better energy budget and average latency, and if this the foremost criterion, then it can be selected; but when the trail of late packets is considered as good as lost, or if the delivery of packets must be ensured at all times including rare failures, then the Leapfrog Collaboration should be preferred.

Compared to full replication and elimination, the Leapfrog Collaboration provides slightly lower yet similar benefits for both reliable and timely delivery of packets, as well as avoidance of four losses in a row. The energy consumption and the latency are greatly improved, nearing the budget obtained with two parallel paths and no retry, yet still significantly higher than that of a serial path.

Wrapping up

Conclusion

A trend towards determinism in IT networks: Over the last forty years, most of the communication technologies that we use on a daily basis, mail, books, music, voice and video, have converged onto digital networks, and, by and large, the Internet, not only bringing costs down but also adding unprecedented new value such as immediate service and improved interactivity.

In a similar fashion, converging IT and OT technology on a shared network yields vastly unrealized benefits in multiple vertical industries, including but not limited to manufacturing, commercial sector, building automation, vehicles, and the power grid; despite the huge potential, the convergence is not happening.

In some cases, the operational resistance comes from a lack of trust in the technology and between different professional groups; only time, probably decades, will change that. In other cases, the limitation is technical and the lack of determinism in IT networks now appears as the gating factor.

Both wired and wireless networks are evolving towards more determinism, in particular with work done at the IEEE 802.1 for bridged Ethernet networks, and at 802.15 for Lowpower Wireless PANs, but the techniques used in wired and wireless environments are largely different; the DetNet group at the IETF is now considering the establishment of end-to-end paths with Deterministic properties from the perspective of Layer 3, and this thesis proposes improvements to the work done at 6TiSCH for the particular case of LWPANs.

Applicability of deterministic IT networks to solve OT problems:

The applicability of the various techniques in the art, and of those proposed in this manuscript, really depends on the use case:

• If an industrial application has no degree of liberty in terms of acceptable jitter and latency whatsoever, then the traditional technique of modulating current over a point to point wire is probably still the best, if not the only option. But the cost and operational complexity of deploying new cables in an existing production facility hinders the addition of new devices for upgrades and enhancements.

• If a high rate and an ultra-precise determinism is required, but some limited latency is acceptable, then a solution based on high speed deterministic Ethernet could be considered; but even if that does not require end-to-end wiring but only from device to the switched fabric, deploying Ethernet comes at the incremental cost for

Partie IV, Chapter 17 -Conclusion on our experimentation deploying wires the devices, which may or may not be doable in a particular environment, or may be hardly affordable, depending on the use case. When applicable, for instance for low speed Process Control applications, huge savings in incremental deployment time and cost of goods may be achieved by relying of wireless solutions as opposed to wires.

• If energy is critical and some rare interruptions of service are acceptable, for instance for non-critical monitoring applications, then a hot potato forwarding along a serial TSCH path will be optimal, providing both the best average delivery time and the lowest energy consumption. If the additional latency is acceptable, then a controlled BIER-TE based scheduling associated with a limited number of retries will reduce the jitter at a minimal energy expenditure, and increase the reliability and the availability of the system.

• If more determinism in delivery time is required, or if the application requires availability at all times, then the proposed Leapfrog Collaboration scheme optimizes the PDR/energy ratio.

• But if the prominent goal is to get every packet to the destination and energy is only secondary to it, then a full Replication and Elimination offers the best chances of success. If the latency budget permits, one might then balance an even more complex Track and more spatial diversity with a mix replication and retransmission.

In any case, the proposed BIER-TE bitmap technique provides an efficient encoding to control the forwarding operation along complex Tracks on a per packet basis, which can adapted to meet the required Quality of Service for each individual packet.

Based on the TimeSlotted Channel Hopping MAC technology, the combination of this novel technique and the capabilities offered by the 6TiSCH architecture enables all these possible arrangements and can provide a deterministic delivery for IPv6 packets even in the face of the stochastic behavior of the radio medium.

Perspectives and next research on deterministic wireless networking: Our research opens new grounds for providing determinism in wireless networks, leveraging physical properties of the medium as opposed to simply emulating the operations that are already being studied and standardized for the wired space, in particular for Ethernet. The property that we have been focusing on is the capability by multiple downstream nodes to receive a packet that is destined to only one of them, and we found novel ways to leverage that to compensate the lack of reliability of unicast radio transmissions.

Other properties of the radio medium may be considered. For instance, the degradation of the unicast transmission that is due to multipath fading may be compensated by sectorized or beam-forming antennas and more complex receive chains, enabling Replication and Elimination of a same physical transmission between the same pair of nodes, as received over different sectors of the antenna, having travelled across different (reflected) physical paths and arriving with microseconds of offset.

As time passes and technologies progress, the energy budget of memory and computation will go down while the physics of radio propagation will not change. It results that more and more, we'll attempt to rely on more complex computation to avoid costly retransmissions and save as much on Replication and Elimination as we can. An alternate form of diversity that we have not evaluated yet, the use of network coding at intermediate points along a Track, may help drive the transmission budget farther down.

Some less ambitious research may also be conducted to propose improvement to the LeapFrog Collaboration, in particular:

• using the Leapfrog on a parallel transmission only upon the detection of a lack of acknowledgement on the main path

• coupling LeapFrog with a routing protocol such as RPL, for instance having a snooping child retransmit on behalf of a first child if the acknowledgement of a parent appears to be missing, thereby providing additional spatial diversity

Enabling an end-to-end DetNet:

The DetNet WG at the IETF aims at solving the deterministic networking problem at Layer 3, across technologies. Initially, people consider this as an encapsulation problem, e.g. adding new control fields to pseudo-wires to provide information such as timing and flow identification. But at some point, the Layer 3 abstraction has to be instantiated on the physical network, and if people can figure how IEEE Std 802.1 TSN and DetNet can interact, applying DetNet on other physical layers will bring its own variety of problems.

In particular, enabling determinism on wired and wireless networks separately is certainly not the end of the journey for DetNet. And it can be foreseen that maintaining deterministic properties at the interconnection of wired and wireless networks will be problematic; the protocols used to synchronize, the typical throughput and the methods used to schedule and access the medium, everything is widely different, yielding nonhomogeneous capabilities that will be difficult to abstract and account for at an upper layer.

It results that DetNet across wired and wireless links is a complete green field for future research; but this is certainly not barring from deploying the emerging technology in homogeneous wired or wireless control networks. The vision of an IT/OT convergence in an Industrial Internet is now becoming a reality, with new standards such as TSN and 6TiSCH becoming available for production.

Partie IV, Chapter 17 -Conclusion on our experimentation

A number of co-authored papers were published in the course of this work and are included in this thesis.

Contribution to DetNet and BIER

A mere two years after 6TiSCH, the DetNet WG was formed in 2015 to define Deterministic Networking in a general Internet context. I contributed to the creation of the WG, and the foundation documents listed below. I was also a member of the forwarding plane alternatives Design Team.

A roughly the same time, the BIER8 WG (for Bit Index Explicit Replication) formed to study a new model for directed multicast flows, whereby bits in a special header indicate which individual destinations are targeted for a particular packet.

The BIER technique was later extended for the purpose of Traffic Engineering (TE); in the BIER-TE [16] 9 approach, a position in the BIER bitmap effectively indicates an adjacency (say, a segment in a complex path) as opposed to a destination.

I am the principal inventor of US Patent No. 9,456,444 [START_REF] Thubert | OAM and Time Slot Control in a Deterministic ARC Chain Topology Network[END_REF] 10 which is a seminal work for BIER-TE, and is included in this thesis. I then extended this work with the BIERbased Replication and Elimination Internet-Draft listed below. The discussion and the experimental work in this thesis are focusing on 6TiSCH networks where transmission errors are more readily observable, but the work has a more global applicability to Deterministic Networks in general.

For that reason, the method is being proposed at DetNet and BIER for standardization with a generic applicability to all types of networks. The proposal was initially submitted as a section of a draft version of the DetNet dataplane alternatives, and was then moved to the BIER draft listed above.

Patents issued or still pending

It must be noted that I work in the industry for a private company; as many such companies do, Cisco encourages the production of Intellectual Property Rights (IPR) as part of the innovation process within the company. It results that patents, as opposed to academic papers, are the most common form for the seminal work presented in this thesis. Papers and Standards usually follow, though not all are encumbered by IPR.

On IPR

We developed and patented a number of novel techniques to provide improvements over the art of industrial quasi-deterministic wireless networks. Some of these methods are detailed in this thesis, with experimental and simulation-based evaluations.

Seminal Work:

The following patents are seminal work for this thesis and included by reference: En apportant le déterminisme dans les réseaux des Technologies de l'Information (IT), une nouvelle génération de réseaux commutés de l'IT va permettre l'émulation de ces liens série et la convergence de réseaux autrefois dédiés sur une infrastructure commune à base d'IP. En retour, la convergence de l'IT et de l'OT permettra de nouvelles optimisations industrielles, en introduisant des technologies héritées de l'IT, comme le BigData et la virtualisation des fonctions du réseau (NFV), en support des opérations de l'OT, améliorant les rendements tout en apportant une réduction supplémentaire des coûts.

Les solutions de réseaux déterministes réclament des possibilités nouvelles de la part des équipements, possibilités qui vont bien au-delà de celles demandées pour les besoins classiques de la QoS.

Les attributs-clé sont :

• la synchronisation précise de tous les noeuds, en incluant souvent la source et la destination des flux • Le calcul centralisé de chemins de bout en bout à l'échelle du réseau • De nouveaux filtres de mise en forme du trafic à l'intérieur comme à l'entrée du réseau afin de le protéger en tous points • Des moyens matériels permettant l'accès au medium à des échéances précises.

Au travers de multiples papiers, de contributions à des standards, et de publication de propriété industrielle, le travail présenté ici repousse les limites des réseaux industriels sans fils en offrant :

Abstract

Based on time, resource reservation, and policy enforcement by distributed shapers, Deterministic Networking provides the capability to carry specified unicast or multicast data streams for real-time applications with extremely low data loss rates and bounded latency, so as to support time-sensitive and mission-critical applications on a converged enterprise infrastructure.

As of today, deterministic Operational Technology (OT) networks are purpose-built, mostly proprietary, typically using serial point-to-point wires, and operated as physically separate networks, which multiplies the complexity of the physical layout and the operational (OPEX) and capital (CAPEX) expenditures, while preventing the agile reuse of the compute and network resources.

Bringing determinism in Information Technology (IT) networks will enable the emulation of those legacy serial wires over IT fabrics and the convergence of mission-specific OT networks onto IP. The IT/OT convergence onto Deterministic Networks will in turn enable new process optimization by introducing IT capabilities, such as the Big Data and the network functions virtualization (NFV), improving OT processes while further reducing the associated OPEX.

Deterministic Networking Solutions and application use-cases require capabilities of the converged network that is beyond existing QOS mechanisms.

Key attributes of Deterministic Networking are:

• Time synchronization on all the nodes, often including source and destination • The centralized computation of network-wide deterministic paths • New traffic shapers within and at the edge to protect the network • Hardware for scheduled access to the media.

Through multiple papers, standard contribution and Intellectual Property publication, the presented work pushes the limits of wireless industrial standards by providing:

Figure 1 -

 1 Figure 1 -The Power of One (Source GE estimates)

2

 2

  .

Figure 1 . 3 -

 13 Figure 1.3 -The bus analogy

Figure 4 . 2 -

 42 Figure 4.2 -The DetNet Architecture

Figure 4 .

 4 Figure 4.2 illustrates the steps that take place in order to setup a deterministic path (in blue), and the interfaces (in purple and green) and data models that DetNet should standardize. Following the numbers on that figure, the main steps are:

4. 5 .

 5 Limits and perspectives

Figure 6 .

 6 Figure 6.10 -G-MPLS switching along Tracks It is worth noting that, because the timeslot information is sufficient for the 6top sublayer to make a switching decision, and this, regardless of the payload of the frame, 6LoWPAN fragments can be forwarded along the G-MPLS-switched path without the need of reassembly in intermediate nodes. This was true for any frames in the art of 802.15.4 TSCH and remains true for 6TiSCH Tracks.

Figures 10 .

 10 Figures 10.1 and 10.2 depict the operation for the first ARC computed in section 9.1 on page 63 (the last in the chain) using rules that are further detailed in section 10.3.

Partie III, Chapter 11 -

 11 Controlling Replication and Elimination using BIER headersIf the frame is lost, then it is probably due to a failure in either I → A or C → E, and the controller should enable segments I → B and D → E to isolate the problem with the precision of an adjacency.A BitString of 00010000 indicates unequivocally a transmission error on the A → C adjacency, whereas a BitString of 01001100 indicates a loss in either A → B, B → D or D → C. Enabling D → E on the next packets may provide more information for the determination of the problem.

Figure 12 . 1 -

 121 Figure 12.1 -Example PDR for a LeapFrog Track

Figure 13 . 3 -

 133 Figure 13.3 -Test bed: test control flows

8 .

 8 Partie IV, Chapter 13 -Testing environment

Figure 13 . 4 -

 134 Figure 13.4 -Test bed: layout

Partie IV, Chapter 15 -

 15 Figure 15.1 represents the spectrum of latency that is experienced by the packets along the different Tracks.

Figure 15 . 1 -

 151 Figure 15.1 -Latency spectrum

Figure 15 .

 15 Figure 15.3 on the next page represents the energy spent per packet over a small window of time where events of interest take place.

Figure 15 . 4 -Figure 15 . 5 -

 154155 Figure 15.4 -Number of losses in a row (zoomed)

Figure 16 . 2 -

 162 Figure 16.2 -Overhearing Schedule

Partie IV, Chapter 16 -Figure 16 . 4 -Figure 16 . 5 -

 16164165 Figure 16.4 -Percentage of occurence per type of Track

1 .

 1 Le calcul centralisé de chemin complexes basé sur une technologie innovante appelée ARC 2. La signalisation de ces chemins complexes et la traçabilité des paquets par une extension de la technologie BIER-TE 3. Réplication, Renvoi et Elimination des doublons le long de ces chemins complexes 4. Un temps-réel basé sur un échéancier qui assure un haut taux de délivrance et garantit une latence bornée 5. La capacité de transporter à la fois des flux déterministes et du trafic IPv6 à multiplexage statistique sur un maillage 6TiSCH partagé Ce manuscrit rapporte des améliorations apportées aux techniques existantes des réseaux sans fils à basse puissance (LoWPAN) comme Zigbee, WirelessHART™et ISA100.11a, afin d'amener ces nouveaux bénéfices jusqu'aux réseaux opérationnels sans fil. Elle a été implémentée en programme et sur du matériel open-source, et évaluée face à du IEEE Std. 802.15.4 classique ainsi que du 802.15.4 TSCH, utilisés en topologie maillée. L'expérience menée montre que notre nouvelle proposition permet d'éviter les à-coups et de garantir des taux élevés de délivrance, même face à des évènements exceptionnels comme la perte d'un relais ou la dégradation temporaire d'un lien radio. Mots-clés: DetNet, Deterministic Networking, 6TiSCH, 6LoWPAN, IEEE Std. 802.15.4, WSN, IOT, Industrial Internet.

Deeper in Wireless 5 The art of low-power wireless sensor network 5

  Networking in DetNet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Controlling a Deterministic Network . . . . . . . . . . . . . . . . . . . . . 4.4.1 Reporting the topology to the controller . . . . . . . . . . . . . . . 4.4.2 Implementing the needs of the application . . . . . . . . . . . . . . 4.4.3 Automating the network operation . . . . . . . . . . . . . . . . . . 4.5 Limits and perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 So how do we make wireless highly predictable? . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . ii About This Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v I Deterministic Networking: the Vision 1 What is deterministic? 3 1.1 Common definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2 A game of convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.3 The way to the IT/OT Convergence . . . . . . . . . . . . . . . . . . . . . 4 1.4 A new context for the convergence . . . . . . . . . . . . . . . . . . . . . . 5 1.5 Determinism in a network . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.5.1 The train analogy (to control loop traffic) . . . . . . . . . . . . . . 7 1.5.2 The bus analogy (to deterministic circuit switching) . . . . . . . . . 8 1.5.3 The vacation place analogy (to time-sharing) . . . . . . . . . . . . . 9 1.5.

4 The casino analogy (to statistical effects) . . . . . . . . . . . . . . . 10 1.5.5 All in all . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 The precursors 13 2.1 On Fast Reroute . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.2 On SDN and Traffic Engineering . . . . . . . . . . . . . . . . . . . . . . . . 14 2.3 On BIER and BIER-TE . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Enabling determinism in communication networks 17 3.1 Expected benefits in wired networks . . . . . . . . . . . . . . . . . . . . . . 17 3.2 Can we make Ethernet deterministic? . . . . . . . . . . . . . . . . . . . . . 18 3.3 Can we make wireless deterministic? . . . . . . . . . . . . . . . . . . . . . 20 The IETF DetNet architecture 25 4.1 Positioning of work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.2 The architecture in a nutshell . . . . . . . . . . . . . . . . . . . . . . . . . 27 ix CONTENTS 4.3 II Diving 5.2 WSNs in Industrial Process Control . . . . . . . . . . . . . . . . . . . . . . 5.3 6TiSCH and best effort IPv6 . . . . . . . . . . . . . . . . . . . . . . . . . . 6 The vision of 6TiSCH centralized scheduling 6.1 PCE vs. 6TiSCH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 6TiSCH base elements (time slots, schedule, chunks and bundles) . . . . .

6.3 Applying DetNet to 6TiSCH . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Forwarding along 6TiSCH Tracks . . . . . . . . . . . . . . . . . . . . . . . 7

Enabling the convergence III Novel Approaches for Higher Predictability 8 Computing a complex Track: the ARC technology 8

.1 Introducing the ARC technology . . . . . . . . . . . . . . . . . . . . . . . . 9 Centralized diverse paths computation with Replication and Elimination 9.1 Running an example of ARC Chain building . . . . . . . . . . . . . . . . . 10 Computing and scheduling time slots 10.1 Forwarding rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2 Detailed steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3 Computation rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Controlling Replication and Elimination using BIER headers 11.1 Technical solution description . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Additional details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x CONTENTS Compensating

loss with overhearing: the Leapfrog Collaboration 77 12

  .1 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 12.2 Introducing the Leapfrog Collaboration . . . . . . . . . . . . . . . . . . . . 78 12.3 Running an example of Leapfrog Collaboration . . . . . . . . . . . . . . . . 79 12.4 Quick retrospective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 12.5 Additional perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 The PIRL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 13.2 Rover: the test tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 13.3 The test layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

	IV Experimentations	85
	Testing environment	89
	13.1

Eliminating jitter with Replication and Elimination 93 14

  .1 Test description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 14.2 Experimental results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 14.3 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

Saving energy with BIER-TE-based replication automation 97 15

  .1 Test description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 15.2 Experimental results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 15.3 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

101 Saving time and energy with the Leapfrog Collaboration 103 16

  .1 Test description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 16.2 Experimental results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 16.3 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

107 Conclusion on our experimentation 109 Conclusion and Perspectives 113 Contributions of this thesis 117 xi

Part I

  Nunes et al.: "A survey of software-defined networking: Past, present, and future of programmable networks"

11

www.itential.com/blog/overhyped-sdn/

12

McKeown et al.: "OpenFlow: enabling innovation in campus networks" 13 www.cisco.com/c/en/us/products/collateral/cloud-systems-management/network-services-orchestrator/ datasheet-c78-734669.html 14 www.arista.com/en/um-eos-4172f/eos-section-45-1-introduction#ww1148008

15

Yeung, Katz, and Kompella: Traffic Engineering (TE) Extensions to OSPF Version 2

16

Atlas et al.: OSPF Traffic Engineering (TE) Metric Extensions 17 Berger, Bartlett, and Harrison: IPv6 Traffic Engineering in IS-IS 18 Ould-Brahim and Fedyk: BGP Traffic Engineering Attribute 19 Farrel and Ash: A Path Computation Element (PCE)-Based Architecture 20 Phinney, Thubert, and Dwars: Industrial Routing Requirements in Low-Power and Lossy Networks 21 Kopetz et al.: "The time-triggered ethernet (TTE) design"

  also the editor of a number of IETF standards that participate to the 6TiSCH overall design and to the 6TiSCH ecosystem, in particular at the 6lo WG which continues and extends the work done at 6LoWPAN, and the ROLL 6 WG where RPL[START_REF] Winter | RPL: IPv6 Routing Protocol for Low-Power and Lossy Networks[END_REF] 7 was initially developed.

	• Book chapters
	-Palattella, Thubert, Vilajosana, Watteyne, Wang, and Engel: "6tisch wireless The following contributions are closely related to this thesis though not included
	industrial networks: Determinism meets IPv6" [64] (award winning) therein:
	• Papers published in journals
	• Informational Internet-Drafts
	-Dujovne, Watteyne, Vilajosana, and Thubert: "6TiSCH: deterministic IP-enabled
	industrial internet (of things)" [15] -Thubert: An Architecture for IPv6 over the TSCH mode of IEEE 802.15.4 [82]
	-Thubert: "Low power wireless protocols boost industrial networking" [84]
	-Thubert: 6TiSCH requirements for DetNet [81]
	• Papers published in conferences
	-Thubert, Watteyne, Palattella, Vilajosana, and Wang: "IETF 6tsch: Combin-• Standard Track Internet-Drafts
	ing IPv6 connectivity with industrial performance" [112]
	-Thubert, Palattella, and Engel: "6TiSCH centralized scheduling: When SDN -Chakrabarti, Thubert, and Nordmark: An Update to 6LoWPAN ND [11]
	meet IoT" [95] -Thubert: IPv6 Backbone Router [83]
	-Watteyne and Thubert: "Efficient 6LoWPAN Neighbor Discovery applied to -Sarikaya, Thubert, and Sethi: Address Protected Neighbor Discovery for Low-
	Multilink IoT subnets" [127] power and Lossy Networks [73]
	-Brodard, Jiang, Chang, Watteyne, Vilajosana, Thubert, and Texier: "Rover: Poor (but Elegant) Man's Testbed" [9] -Thubert and Pylakutty: Root initiated routing state in RPL [96]
	-Robles, Richardson, and Thubert: When to use RFC 6553, 6554 and IPv6-in-• Papers accepted for future conferences IPv6 [72]
	-Matsui, Papadopoulos, Thubert, Watteyne, and Montavont: "Poster: 4th In-
	dustrial Revolution: Toward Deterministic Wireless Industrial Networks" [50] • RFCs
	accepted at ACM EWSM 2017
	-Thubert and Cragie: IPv6 over Low-Power Wireless Personal Area Network -Jiang, Brodard, Ahmed Bouabdallah, Montavont, Texier, Thubert, Watteyne, and Papadopoulos: "Competition: Controlled Replication for Higher Reliability (6LoWPAN) Paging Dispatch [90]
	and Predictability in Industrial IoT Networks" [35] accepted at ACM EWSM -Thubert, Bormann, Toutain, and Cragie: 6LoWPAN Routing Header [109]
	2017
	-Papadopoulos, Matsui, Thubert, Texier, Watteyne, and Montavont: "Leapfrog • Prior related RFCs
	Collaboration: Toward Deterministic and Predictable in Industrial-IoT appli--Thubert and Hui: Compression Format for IPv6 Datagrams over IEEE 802.15.4-cations" [66] accepted at IEEE ICC 2017 Conference Based Networks [93]
	• Papers submitted or in progress

-Brodard, Thubert, Jiang, Watteyne, Chang, Texier, Papadopoulos and Montavont: "Multipath Redundancy for Wireless Time-Slotted Networks" aimed at the IEEE Transactions on Industrial Informatics I am -Winter, Thubert, Brandt, Vasseur, Hui, Pister, Levis, Struik, Kelsey, and Clausen: RPL: IPv6 Routing Protocol for Low-Power and Lossy Networks [133] Partie IV, Chapter 17 -Conclusion on our experimentation

-

  Also considered as part of the work done for this thesis are the following patents, either still pending or already issued: Thubert, Vasseur, Wetterwald, and Mundenmaney: Dynamic installation of local storing mode paths in a non-storing low-power and lossy network[START_REF] Thubert | Dynamic installation of local storing mode paths in a non-storing low-power and lossy network[END_REF] -Thubert, Vasseur, and Wetterwald: Timeslot distribution in a distributed routing protocol for deterministic wireless networks[START_REF] Thubert | Timeslot distribution in a distributed routing protocol for deterministic wireless networks[END_REF] -Thubert, Bellagamba, Anteunis, and Levy-Abegnoli: Generating a loop-free routing topology using routing arcs[START_REF] Thubert | Generating a loop-free routing topology using routing arcs[END_REF] 6LoWPAN IETF IPv6 over LWPANs WG. v, 42, 45, 51, 91, 118 6TiSCH IETF IPv6 over the TSCH mode of IEEE Std 802.15.4 WG. v-vii, 41-54, 56, 67, 71, 78, 91, 93, 101, 113-115, 117-120 6lo IETF IPv6 over Networks of Resource-constrained Nodes WG (continues 6LoWPAN). 91, 118 AMI Advanced Metering Infrastructure. 20 AMR Automatic Meter Reading. 20 ARC Available Routing Construct. 52, 59-65, 67-70 ARQ Automatic Repeat reQuest (or Query). 77, 80, 82, 93, 95, 97, 98, 109 AS Autonomous System. 61 AVB Audio/Video Bridging. 5, 18, 19, 25, 28 BBR Backbone Router. 43 BER Bit Error Rate. 20, 37 BIER Bit Indexed Explicit Replication. vi, 15, 54, 71-76, 91, 92, 94, 97, 98, 100, 102, 114, 119, 120 Carrier Sense Multiple Access with Collision Avoidance. 21, 22 DAG Directed Acyclic Graph. 60 DetNet IETF Deterministic Networking WG. vi, 5, 25-33, 42, 44, 45, 52, 71, 76, 113, 115, 119, 120 DODAG Destination Oriented Directed Acyclic Graph. 103 ERPS Ethernet Ring Protection Switching. 19 G-MPLS Generalized MPLS. 45, 46, 51, 53 HSR High-availability Seamless Redundancy. 19 ICT Information and Communication Technologies. ii IEC International Electrotechnical Commission. 19, 39 IEEE Institute of Electrical and Electronics Engineers. iii-v, 3, 5, 18, 20, 21, 25, 30, 33, 37, 39, 42, 45, 46, 51, 53, 56, 77, 87, 113, 115, 118 IETF Internet Engineering Task Force. iii, v, 3, 5, 13, 14, 25, 26, 28, 30, 32, 41, 42, 45, 51, 59, 71, 113, 115, 117, 118, 122 IoT Internet of Things. 3, 14, 22, 40, 51, 92, 117 IP Internet Protocol. i-iii, v, 3-6, 10, 11, 13, 40, 49, 51, 55, 56, 77, 91, 92 IPv6 Internet Protocol Version 6. v, 15, 40-43, 45, 48, 49, 61, 71, 114, 117 ISDN Integrated Services Digital Network. 10 ISM Industrial, Scientific, and Medical. 20, 22 IT Information Technology. ii, iii, 3-5, 9, 18, 44, 55, 113, 115 LAN Local Area Network. 9, 25, 29, 61 LDP Label Distribution Protocol. 13 LFA Loop-Free Alternates. 13 LLN Low-Power and Lossy Network. v, 14, 42, 43 LWPAN Low-power Wireless Personal Area Network. iv, 113 MAC Medium Access Control. v, vi, 5, 21, 22, 39, 41, 43, 48, 51, 53, 55, 56, 114 OAM Operations, Administration and Maintenance. vi, vii, 13, 26 OLAF Open Lowest ARC First. 60, 63 OPEX Operational Expenditures. 15, 40, 55 OT Operational Technology. ii, iii, 4, 5, 9, 11, 12, 14, 25, 40, 41, 43, 44, 55, 113, 115 PAN Personal Area Network. 113 PCE Path Computation Element. 14, 28, 32, 45, 52, 53, 56, 70, 76 PCI Peripheral Component Interconnect. 6 PER Packet Error Rate. 19 PIRL Cisco's Paris Innovation Labs. 89, 92 PMN Professional Media Networking. 6, 31 PRP Parallel Redundancy Protocol. 19 PSK Pre-Shared Keys. 44 PTP Precision Time Protocol. 18, 33 QoS Quality of Service. i, 10, 11, 17, 56 REP Resilient Ethernet Protocol. 19 RFC Request For Comment (as an IETF standard). 117, 119, 122 ROLL IETF Routing Over Low power and Lossy networks WG. 91, 118 RPL IPv6 Routing Protocol for Low-Power and Lossy Networks. v, 42-45, 47, 92, 103, 104, 117, 118 TDM Time Division Multiplexing. 4, 10, 37, 55 TE Traffic Engineering. vi, 14, 15, 30, 39, 52, 54, 71, 73, 76, 94, 97, 98, 100, 102, 114, 119 TSCH Time-Slotted Channel Hopping. iv-vi, 5, 20-22, 30, 37-39, 41-43, 45, 46, 48, 49, 51, 56, 67, 77, 87, 93-95, 97, 98, 103, 104, 109, 114, 117 TSMP Time Synchronized Mesh Protocol. iv TSN Time-Sensitive Networking. 5, 18, 25, 33, 115 TTE Time-Triggered Ethernet. 18, 95 VLAN Virtual Local Area Network. 9 WG Working Group. v, 5, 25, 26, 41-43, 45, 52, 115, 117-119 WPAN Wireless Personal Area Network. iv WSN Wireless Sensor Network. i, iv, 22, 37, 39-41, 44, 67, 77 Résumé En s'appuyant sur une connaissance précise du temps, sur la réservation de ressources et l'application distribuée de règles d'admission strictes, un réseau déterministe permet de transporter des flux pré-spécifiés avec un taux de perte extrêmement bas et une latence maximale majorée, ouvrant la voie au support d'applications critiques et/ou temps-réel sur une infrastructure de réseau convergée. De nos jours, la Technologie Opérationnelle (OT) s'appuie sur des réseaux déterministes mais conçus à façon, en général propriétaires, utilisant typiquement des liens série spécifiques, et opérés en isolation les uns des autres, ce qui multiplie la complexité physique et les coûts d'achat et de déploiement (CAPEX), ainsi que d'opération et maintenance (OPEX), et empêche l'utilisation agile des ressources.

	Glossary
	• Issued Patents
	-Thubert, Wetterwald, Ribiere, Moon, and Kruys: Path shortening in a wireless mesh network [114] 3GPP 3rd Generation Partnership Project. 3
	• Issued Patents
	-Thubert, Vasseur, and hui: OAM and Time Slot Control in a Deterministic
	ARC Chain Topology Network [97]
	-Thubert, Vasseur, and Wetterwald: Installation of time slots for sending a • Patents Pending
	packet through an ARC chain topology network [101] -Thubert, Wijnands, and Shepherd: Reliable multicast in low-power and lossy CAC Call Admission Control. 10
	• Patents Pending -Thubert, Wijnands, and Shepherd: USING BIT INDEX EXPLICIT REPLI-CATION (BIER) IN LOW-POWER AND LOSSY NETWORKS [108] networks [107] -Vasseur, Wetterwald, and Thubert: DISTRIBUTED ARBITRATION OF TIME CONTENTION IN TSCH NETWORKS [120] -Levy-Abegnoli, Thubert, and Wetterwald: Establishing deterministic multicast CAN Controller Area Network. 6 CDU Channel Distribution/Usage. 46-50, 56 SDN Software-Defined Networking. 6, 14, 28, 32, 39, 61, 62, 117 CSMA/CA SDH Synchronous Digital Hierarchy. 10 SONET Synchronous Optical Networking. 10
	paths in a network [42]

-Thubert, Wetterwald, and Vasseur: LEAPFROG COMMUNICATIONS IN A

TSCH NETWORK

[105]

Partie IV, Chapter 17 -Conclusion on our experimentation

Related contributions:

-Thubert, Vasseur, and Wetterwald: Reassignment of unused portions of a transmission unit in a network [102] -Vasseur, Thubert, and Wetterwald: Efficient network probing for deterministic wireless networks [118] -Wetterwald, Vasseur, and Thubert: Configuring new paths in a wireless deterministic network [131] -Thubert, Vasseur, and Wetterwald: Synchronized routing updates for TSCH networks [103] -Thubert, Vasseur, and Wetterwald: Distributed rescheduling of bounded flows in a time sensitive network [99] -Thubert, Vasseur, and Wetterwald: Exclusive and overlapping transmission unit allocation and use in a network [100] -Thubert, Vasseur, and Wetterwald: DISTRIBUTED APPROPRIATION AND DEFENSE OF TIME SLOT OWNERSHIP IN A CHANNEL HOPPING SCHED-ULE [98] MPLS MultiProtocol Label Switching. 13, 15, 45, 51, 61 MPLS-TP MPLS Transport Profile. 13, 61 NFV Network Functions Virtualization. 117 SP Service Provider. 22 SPF Shortest Path First. 64 SR Segment Routing. 15 N° d'ordre : 2017IMTA0011

  1. Complex Track computation based on a novel ARC technology 2. Complex Track signaling and traceability, extending the IETF BIER-TE technology 3. Replication, Retry and Duplicate Elimination along the Track 4. Scheduled runtime enabling highly reliable delivery within bounded time 5. Mix of IPv6 best effort traffic and deterministic flows within a shared 6TiSCH mesh structure This manuscript presents enhancements to existing low power wireless networks (LoWPAN) such as Zigbee, WirelessHART™and ISA100.11a to provide those new benefits to wireless OT networks. It was implemented on open-source software and hardware, and evaluated against classical IEEE Std. 802.15.4 and 802.15.4 TSCH radio meshes. This manuscript presents and discusses the experimental results; the experiments show that the proposed technology can guarantee continuous high levels of timely delivery in the face of adverse events such as device loss and transient radio link down.

De nouvelles capacités sont nécessaires pour permettre l'interconnexion de milliards d'objets, et rendre accessibles les vastes quantités de données générées par les objets connectés, et ce, en accord avec les besoins de performance spécifiques aux diverses applications opérationnelles, qui vont typiquement bien au-delà de ceux de l'informatique de bureau et des technologies classiques de l'Internet.De nos jours, la Technologie Opérationnelle (OT) s'appuie sur des réseaux déterministes mais conçus à façon, en général propriétaires, utilisant typiquement des liens série spécifiques, et opérés en isolation les uns des autres, ce qui multiplie la complexité physique et les coûts d'achat et de déploiement (CAPEX), ainsi que d'opération et maintenance

www.odva.org/Portals/0/Library/Conference/2015_ODVA_Conference_Ditzel-Didier_TSN.pdf

Grossman et al.: Deterministic Networking Use Cases

Srinivasan et al.: "The β-factor: measuring wireless link burstiness"

Watteyne et al.: "Mitigating Multipath Fading Through Channel Hopping in Wireless Sensor Networks"

[START_REF] Fairhurst | Advice to link designers on link Automatic Repeat reQuest (ARQ)[END_REF] standards.ieee.org/about/get/802/802.15.

html[START_REF] Fang | Call admission control schemes and performance analysis in wireless mobile networks[END_REF] Pister and Doherty: "TSMP: Time synchronized mesh

protocol"[START_REF] Farrel | A Path Computation Element (PCE)-Based Architecture[END_REF] Watteyne et al.: "Industrial ieee802.15.4e networks: Performance and

trade-offs" 21 standards.ieee.org/findstds/standard/802.

15.4e-2012.html 22 en.hartcomm.org/hcp/tech/aboutprotocol/aboutprotocol_specs.html

www.3gpp.org/

Borella et al.: Methods for determining sendable information content based on a determined network latency

www.cisco.com/c/en/us/td/docs/net_mgmt/ciscoworks_lan_management_solution/4-0/user/guide/ configuration_management/cmug/configvlan.pdf

www.cisco.com/c/en/us/td/docs/net_mgmt/ciscoworks_lan_management_solution/4-0/user/guide/ configuration_management/cmug/vrf.pdf

www.embedded.com/electronics-blogs/cole-bin/4406659/Deterministic-networking--from-niches-to-the-mainstream-

Osborne et al.: MPLS Transport Profile (MPLS-TP) Linear Protection

Beller and Sperber: "MPLS-TP-The New Technology for Packet Transport Networks."

Frost, Bryant, and Bocci: MPLS Transport Profile Data Plane Architecture

Allan and Busi: Operations, Administration, and Maintenance Framework for MPLS-Based Transport Networks

Shand and Bryant: IP Fast Reroute Framework

Zinin: Basic Specification for IP Fast Reroute: Loop-Free Alternates

Bryant et al.: Remote Loop-Free Alternate (LFA) Fast Reroute (FRR)

Envedi, Atlas, and Bowers: An Architecture for IPLDP Fast Reroute Using Maximally Redundant Trees (MRT-FRR)

Wijnands et al.: Multicast using Bit Index Explicit Replication

Eckert et al.: Traffic Engineering for Bit Index Explicit Replication BIER-TE

Filsfils et al.: Segment Routing Architecture

Partie I, Chapter 2 -The precursors

Meier and Weibel: "IEEE 1588 applied in the environment of high availability LANs"

Ingram, Schaub, and Campbell: "Use of precision time protocol to synchronize sampled-value process buses"

MacKay: Applications and Opportunities for the IEEE 1588 Standard in Military Applications

Oliva et al.: "An overview of the CPRI specification and its application to C-RAN-based LTE scenarios"

 5 Kopetz et al.: "The time-triggered ethernet (TTE) design"

www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-G.8032-201508-I!!PDF-E

www.cisco.com/c/en/us/support/docs/lan-switching/ethernet/116384-technote-rep-00.html

www.processonline.com.au/content/industrial-networks-buses/article/redundancy-in-industrial-networks-part-2-712482

Kirrmann, Hansson, and Muri: "IEC 62439 PRP: Bumpless recovery for highly available, hard realtime industrial networks"

 10 Kirrmann et al.: "HSR: Zero recovery time and low-cost redundancy for Industrial Ethernet (High availability seamless redundancy, IEC 62439-

3)" 11 IEC: Industrial communication networks -High availability automation networks -Part 3: Parallel Redundancy Protocol (PRP) and High-availability Seamless Redundancy

(HSR)12 Araujo et al.: "High availability automation networks: PRP and HSR ring

implementations"13 Tan and Luan: "IEC 61850 based substation automation system

architecture design" 14 IEC: IEC 61850 Communication networks and systems for power

utility automation -ALL PARTS15 Goraj and Harada: "Migration paths for IEC 61850 substation communication networks towards superb redundancy based

on hybrid PRP and

HSR topologies" 16 avnu.org/proav/ 17 Shu et al.: "Packet loss analysis of the IEEE 802.15. 4 MAC without acknowledgements"

Hwang: "Energy efficient channel agility utilizing dynamic multi-channel CCA for ZigBee RF4CE"

Watteyne et al.: "Industrial ieee802.15.4e networks: Performance and trade-offs"

Gaillard: "Managing IoT networks using service level agreements (SLA)"

Watteyne et al.: "Industrial ieee802.15.4e networks: Performance and trade-offs"

Partie I, Chapter 3 -Enabling determinism in communication networks

datatracker.ietf.org/wg/\gls{detnet}/charter/

Thubert and Finn: Deterministic Networking Architecture

Neagoe, Cristea, and Banica: "NTP versus PTP in com puter networks clock synchronization"

Richardson: 6tisch Secure Join protocol

Dujovne et al.: 6TiSCH 6top Scheduling Function Zero (SF0)

Wang and Vilajosana: 6top Protocol (6P)

Mannie: "Generalized multi-protocol label switching (GMPLS) architecture"

Wang and Vilajosana: 6top Protocol (6P)

www.ieee802.org/15/ANA.html

https://www.icann.org/resources/pages/welcome-2012-02-25-en

www.ietf.org/iana.html

Thubert and Hui: LLN Fragment Forwarding and Recovery

Grossman et al.: Deterministic Networking Use Cases

https://tools.ietf.org/html/draft-ietf-detnet-use-cases-11#section-5

Thubert: 6TiSCH requirements for DetNet

Mangold et al.: "Analysis of IEEE 802.11 e for QoS support in wireless LANs"

Thubert: 6TiSCH requirements for DetNet

Thubert, Vasseur, and Wetterwald: Installation of time slots for sending a packet through an ARC chain topology network

Thubert, Vasseur, and hui: OAM and Time Slot Control in a Deterministic ARC Chain Topology Network

Thubert, Brodard, and Jiang: BIER-TE-based OAM, Replication and Elimination

Filsfils et al.: Segment Routing Architecture

Shu et al.: "Packet loss analysis of the IEEE 802.15. 4 MAC without acknowledgements"

Mahmood, Seah, and Welch: "Reliability in wireless sensor networks: A survey and challenges ahead"

Fairhurst and Wood: Advice to link designers on link Automatic Repeat reQuest (ARQ)

Lee and Huh: "Reliable data transfer using overhearing for implicit ack"

Maalel et al.: "Adaptive reliable routing protocol for wireless sensor networks"

Lampin: "Urban wireless sensors networks : Applications and protocols"

Thubert et al.: Path shortening in a wireless mesh network

Alghamdi et al.: "Constructing a Shortest Path Overhearing Tree with Maximum Lifetime in WSNs"

www.etsi.org/technologies-clusters/technologies/testing/10-news-events/events/942-6tisch-plugtests

Brodard et al.: "Rover: Poor (but Elegant) Man's Testbed"

Thubert et al.: 6LoWPAN Routing Header

Thubert, Brodard, and Jiang: BIER-TE-based OAM, Replication and Elimination

Partie IV, Chapter 15 -Saving energy with BIER-TE-based replication automation

Vilajosana, Pister, and Watteyne: Minimal 6TiSCH Configuration

Vilajosana et al.: "A realistic energy consumption model for TSCH networks"

Partie IV, Chapter 16 -Saving time and energy with the Leapfrog Collaboration

Partie IV, Chapter 17 -Conclusion on our experimentation

datatracker.ietf.org/wg/roll/charter/

Winter et al.: RPL: IPv6 Routing Protocol for Low-Power and Lossy Networks

datatracker.ietf.org/wg/bier/charter/

Eckert et al.: Traffic Engineering for Bit Index Explicit Replication BIER-TE

Thubert, Vasseur, and hui: OAM and Time Slot Control in a Deterministic ARC Chain Topology Network

Acknowledgments

This work took place at the Cisco Paris Innovation and Research Lab (PIRL) and funded in part by the Cisco-Polytechnique Chair.

It was also partly supported by the European Commission's Horizon 2020 Framework Programme, through the H2020 F-Interop and H2020 ARMOUR projects.

Simulations of this work were ran at IMT-Atlantique by Tadanori Matsui.

Part III

Novel Approaches for Higher Predictability

Contributions of this thesis

Contribution to the ETSI IPv6 Industry Specification

Group

The ETSI IPv6 Industry Specification Group (IP6 ISG) 1 is composed of industry experts in IPv6 and produces highly regarded and widely publicized best practice and recommendation documents to ease and accelerate the deployment of IPv6 in the enterprise, in service provider networks, academia and education networks, datacenter, IoT, SDN and NFV, 5G mobile wireless access and core, and more.

I adapted this manuscript for publication as a contribution to IP6 ISG as ETSI Group Report (GR) IP6 0009 IPv6-Based Industrial Internet leveraging 6TiSCH Technology, the produced document largely echoing this manuscript.

Being published as one of those documents is a clear recognition for this work by eminent peers, and a fulfillment for the author.

Contribution to 6TiSCH

The IETF 6TiSCH WG was initially chartered to develop an architecture [START_REF] Thubert | An Architecture for IPv6 over the TSCH mode of IEEE[END_REF] 2 and the minimal support [START_REF] Vilajosana | Minimal 6TiSCH Configuration[END_REF] 3 to provide an IPv6 stack over TSCH, in order to enable stochastic best effort) IPv6 traffic over a basic schedule.

The second round of 6TiSCH still considers stochastic traffic, but adds capabilities to adapt the schedule dynamically to adapt to the variations of the traffic. The work also includes a secure join process, and in preparation for a potential next phase, requirements [START_REF] Thubert | 6TiSCH requirements for DetNet[END_REF] 4 for the co-existence with deterministic flows.

The most probable next steps, should the work continue, will involve deterministic IPv6 flows, with Tracks that are either controlled through RPL [START_REF] Thubert | Root initiated routing state in RPL[END_REF] 5 , or through Yang models yet to be standardized. This thesis comes in preparation for that work; outcomes include multiple papers describing 6TiSCH activities in relation with determinism, patents, and Internet-Drafts at the IETF, some of which resulting in Standard Track RFCs. -Thubert, Wetterwald, Vasseur, and Levy-Abegnoli: Reverse directed acyclic graph for multiple path reachability from origin to identified destination via multiple target devices [START_REF] Thubert | Reverse directed acyclic graph for multiple path reachability from origin to identified destination via multiple target devices[END_REF] -Thubert, Vasseur, Levy-Abegnoli, and Wetterwald: Parent device allocation of retransmit slot to child network device on behalf of peer child device in a deterministic network [START_REF] Thubert | Parent device allocation of retransmit slot to child network device on behalf of peer child device in a deterministic network[END_REF] In short