Mehdi Kherbache
email: mehdi.kherbache@univ-lorraine.fr

Moufida Maimour
email: moufida.maimour@univ-lorraine.fr

Eric Rondeau
email: eric.rondeau@univ-lorraine.fr

Network Digital Twin for the Industrial Internet of Things

Keywords: Digital Twin, Internet of Things, Industy 4.0

Digital Twins are starting to revolutionize many industries in the last decade providing a plethora of benefits to optimize the performance of industrial systems. They aim to create a continuously synchronized model of the physical system which enables rapid adaptation to dynamics, mainly unpredicted and undesirable changes. A wide range of industrial fields have already benefited from digital twins technology such as aerospace, manufacturing, healthcare, city management and maritime and shipping. Furthermore, recent research works are starting to study the integration of digital twins for computer networks to allow more innovation and intelligent management. One of the basic building blocks of digital twins technology is the Internet of Things where wireless sensors and actuators are deployed to ensure the interaction between the physical and digital worlds. This type of network is complex to manage due to its severe constraints especially when it controls critical industrial applications, resulting in the Industrial Internet of Things (IIoT). We believe that the optimization of the IIoT will lead to efficient integration of Digital Twins in Industry 4.0. In this paper, we design a Network Digital Twin for the IIoT where sensors, actuators and communication infrastructure are replicated in the digital twin to enable intelligent realtime management of such networks. This way, new networking services such as predictive maintenance, network diagnosis, resource allocation, energy optimization with other intelligent services can be efficiently integrated and exploited in the network life-cycle. We validate the proposed architecture by providing a promising prototype implementation that should unleash the full potential of the network digital twin.

I. INTRODUCTION

The Internet of Things (IoT) market is growing rapidly as a consequence of developing smart buildings, smart industries, intelligent transportation, among others. It is expected to grow by 26% during the period of 2017-2022 [START_REF]Internet of things (IOT) market[END_REF]. The IoT is seen as a key technology in enabling the fourth industrial revolution with the advent of Industrial Internet of Things (IIoT) enabling a broad range of industrial applications to profit from full automation and high productivity.

IIoT are known to be complex and hard-to-manage systems since they control constrained industrial applications where various requirements need to be satisfied. Reliability, strict real-time constraints and energy requirements are the most crucial ones among others. To meet these diverse growing requirements, novel technologies are needed to assist these wireless systems. Digital Twins (DT) is one promising candidate technology to this purpose. In fact, a DT is particularly promising for building a continuously updated model of a physical system to allow rapid adaptation to changing dynamics [START_REF] Grieves | Digital Twin: Mitigating Unpredictable, Undesirable Emergent Behavior in Complex Systems[END_REF]. Moreover, it provides a powerful simulation framework and can enable cognitive capabilities for the replicated system [START_REF] Barricelli | A Survey on Digital Twin: Definitions, Characteristics, Applications, and Design Implications[END_REF].

DT technology is classified as one of the top ten most promising technological trends for the next decade by Gartner [START_REF] Cearley | Top 10 strategic technology trends for 2020[END_REF]. Moreover, it is expected to grow at a Compound Annual Growth Rate (CAGR) of 58% during the period 2020-2026 [14]. The digital replication of the physical system enables intelligent monitoring, innovation and various augmented functionalities. Various fields are already taking advantage of this fast growing technology such as manufacturing [START_REF] Mandolla | Building a digital twin for additive manufacturing through the exploitation of blockchain: A case analysis of the aircraft industry[END_REF], healthcare [START_REF] Bruynseels | Digital twins in health care: ethical implications of an emerging engineering paradigm[END_REF], aerospace [START_REF] Medina | A Maturity Model for Digital Twin Implementations in the Commercial Aerospace OEM Industry[END_REF], smart city [START_REF]Smart city digital twins[END_REF], among others.

Recent research work have started to investigate the opportunities that DT could bring to computer networks. The IETF draft [START_REF] Zhou | Digital Twin Network: Concepts and Reference Architecture[END_REF] proposes a reference architecture for the Digital Twin Network (DTN) along with issues and open research perspectives related to its implementation. More detailed architectures are proposed in [START_REF] Almasan | Digital Twin Network: Opportunities and Challenges[END_REF][START_REF] Khan | Digital Twin of Wireless Systems: Overview, Taxonomy, Challenges, and Opportunities[END_REF] and their implementation is discussed across various aspects such as DTN model construction using machine learning algorithms, procedures to enable new networking functionalities (what-if analysis, troubleshooting, network planning, etc.) along with enabling technologies for efficiently implementing the DTN. Moreover, data collection for the DTN is further addressed in [START_REF] Zhu | A knowledge graph based construction method for Digital Twin Network[END_REF] and knowledge graphs are leveraged to concretely construct the DTN. With an application-level viewpoint, the DTN is leveraged to predict the global QoS (Quality of Service) by using Graph Neural Networks in [START_REF] Ferriol-Galmés | Building a Digital Twin for Network Optimization Using Graph Neural Networks[END_REF] while an applicationdriven DTN middleware is proposed in [START_REF] Bellavista | Application-Driven Network-Aware Digital Twin Management in Industrial Edge Environments[END_REF].

While DTN-related research are gaining a progressive interest recently, there is still a gap in providing a detailed architecture of a network digital twin for the IIoT given its constrained nature and complex characteristics that should be taken into consideration. We recently proposed in [START_REF] Kherbache | When Digital Twin Meets Network Softwarization in the Industrial IoT: Real-Time Requirements Case Study[END_REF], a holistic DTbased architecture that should enable closed-loop IIoT network management and that enables augmented functionalities for the design and during the service of the IIoT. The current paper is a completion and a more in-depth architecture of the Network Digital Twin for the IIoT taking advantage of the SDN (Software Defined Networking) paradigm and enabling intelligent networking services such as predictive maintenance, network diagnosis, energy optimization, resource allocation, sustainability, among others. It is designed with a terre-à-terre vision that should allow its efficient implementation.

The remainder of the paper is organized as follows. Section II covers related work on the DTN. In Section III, the architecture of the NDT for IIoT is presented while Section IV an implementation of a prototype providing basic functionalities of the NDT is introduced. Section V concludes the paper.

II. RELATED WORK

Several research work are studying the DTN concept, proposing different architectures, modeling aspects and methods along with the proposition of various networking services to enable the real potential of digital twins in the networking context. The IETF group is currently engaged in standardizing the DTN concept. Their latest draft [START_REF] Zhou | Digital Twin Network: Concepts and Reference Architecture[END_REF] outlines the basic concepts and a reference architecture as well as the main challenges and issues for building a DTN.

An ML-based DTN is introduced in [START_REF] Almasan | Digital Twin Network: Opportunities and Challenges[END_REF] as a key enabler of efficient control and management of modern real-world networks. The adoption of DTN allows network operators to efficiently design network optimization solutions, perform troubleshooting, what-if analysis and network planning. The authors argue that Deep Learning techniques can be leveraged to model the DTN of a network taking as input some parameters such as traffic, topology, routing, scheduling policies, etc. Then producing as output some network-related performance metrics (utilization, delay, packet drops, etc). By adopting a feedback-based iterative approach between the DT and a network optimizer, this latter can find the best network configuration that satisfies the requirements set by a network operator.

Another work in [START_REF] Khan | Digital Twin of Wireless Systems: Overview, Taxonomy, Challenges, and Opportunities[END_REF] presents a high-level architecture for digital twin of wireless systems in a three layers fashion: Physical interaction layer, Twin layer and Services layer. Two aspects are considered to discuss different details concerning digital twins of wireless systems. Twins for wireless deals with the role of the digital twin in enabling wireless systems. On the other hand, wireless for twins deals with the efficient utilization of wireless resources for enabling effective twin signaling over a wireless link.

With a particular focus on DTN modeling and addressing the problem of data collection for the DTN, [START_REF] Zhu | A knowledge graph based construction method for Digital Twin Network[END_REF] proposes a four-layer architecture for the DTN consisting of a Physical network layer, Data lake layer, DT layer and Network application layer. The data lake layer collects, stores and preprocesses the collected data to provide knowledge and relation extraction to the DT layer for DTN model construction. The DT layer consists of : i) physical entity modeling that completes the modeling of the different networking components, ii) requirements modeling that is used to develop a multitude of scenario models such as network resource prediction, anomaly detection, automatic operation through AI algorithms, iii) Twin management & control center which orchestrates the DT layer to satisfy the intended requirements. A sketch based data collection algorithm is proposed for efficiently collecting data in the data lake layer and knowledge graphs are used to construct the DTN. The relationship between different network characteristics (topology, routing, queue scheduling, and input traffic) is modeled in [START_REF] Ferriol-Galmés | Building a Digital Twin for Network Optimization Using Graph Neural Networks[END_REF] using a digital twin that exploit Graph Neural Networks (GNNs) to predict the global QoS (predict the mean per-flow delay). It includes an optimizer that finds the best routing and/or scheduling policies fulfilling complex Service Level Agreements (SLAs). The particularity of the proposed model is its generalization capability as it performs well in network scenarios unseen in the training set.

An Application-driven Digital Twin Networking (ADTN) middleware is proposed in [START_REF] Bellavista | Application-Driven Network-Aware Digital Twin Management in Industrial Edge Environments[END_REF] to facilitate the interaction with heterogeneous distributed industrial devices and the dynamic management of network resources by adopting an applicationlevel approach. This architecture is based on an SDN crosslayer approach to dynamically manage the industrial environment as well as considering the quality of service requirements and the network configuration adaptation abilities.

In the context of mobile networks, [START_REF] Deng | A Digital Twin Approach for Selfoptimization of Mobile Networks[END_REF] presents an algorithmic framework combining digital twin technology, reinforcement learning and expert knowledge to enable selfoptimization in mobile networks. First, a digital twin network is constructed to predict the future state of the network based on its current state. After that, reinforcement learning and expert knowledge extract the predicted state to make optimization decisions and send them back to the digital twin which evaluates these decisions and applies the best one in the physical network. This latter sends the new network state back to the digital twin after applying these decisions after an evaluation time duration and thus allowing a closed-loop selfoptimization of the physical network. A 5G Digital Twin is provided in [START_REF] Nguyen | Digital twin for 5G and beyond[END_REF] to assist the development and deployment of complex 5G networks and permit cost-effective access to 5G, given that the deployment of 5G networks is too expensive.

In the context of vehicular networks, The Software Defined Vehicular Network (SDVN) based on an intelligent digital twin in [START_REF] Zhao | Intelligent digital twin-based softwaredefined vehicular networks[END_REF] was developed to enable intelligent and adaptive routing in vehicular networks. Another work in [START_REF] Zhang | Adaptive Digital Twin and Multiagent Deep Reinforcement Learning for Vehicular Edge Computing and Networks[END_REF] designed a digital twin network to model an edge service aggregation scheme of vehicular networks. Potential edge service matching among massive vehicle pairs are revealed by the DTN. This latter divides the complex vehicular network into multiple parts according to potential service associations which reduces considerably the edge service scheduling complexity.

In order to enable flow emulation in the DTN, [START_REF] Yang | A Systematic Network Traffic Emulation Framework for Digital Twin Network[END_REF] proposes a complete flow emulation framework for digital twin network to keep physical network traffic in consistency with that in digital twin network. The authors demonstrate through a case study how the proposed framework can enable accurate delay measurement capabilities without affecting the physical network.

III. NETWORK DIGITAL TWIN FOR IIOT

The IIoT is enabling the fourth industrial revolution through the integration of sensors and actuators in industrial systems. These devices are of a constrained nature in terms of energy since they only use batteries as a source of power supply. In addition, in terms of reliability since the communication between nodes is ensured with a wireless shared medium. Several research efforts are made to enable energy saving while ensuring a sufficient reliability in IIoT. Also, in time-sensitive industrial systems the communication between nodes should respect strict deadlines. The required information should reach its destination in time while on the other hand energy efficiency should be maintained to increase the network's lifetime.

These varying requirements can be satisfied using a Network Digital Twin (NDT) that models the IIoT network continuously and keeps an updated view of the network through its entire life-cycle. In other words, from the design to the service phase including production and maintenance as shown in Figure 1. The physical network sends reports to the NDT which in turn analyzes them for potential enhancements of the networking performance. From another angle, the NDT iterates continuously through the network development process (design, production and maintenance phases) while getting feedback in return from the physical network. This iterative process enables a cycle of continuous improvement and innovation through the coupling of modeling and analysis with execution [START_REF] Kherbache | When Digital Twin Meets Network Softwarization in the Industrial IoT: Real-Time Requirements Case Study[END_REF].

The NDT enables multiple intelligent features thanks to the data collected continuously from the physical network. This data can be translated to knowledge by intelligent algorithms and simulation capabilities integrated in the NDT. In other words, the NDT allows predictive maintenance, network diagnosis, efficient energy optimization, security management, optimized resource allocation and real-time network monitoring. Furthermore, interoperability between different networking devices can be ensured since their corresponding components in the digital world are platform independent and they can be managed following common mechanisms without worrying about the technicalities of the different devices. To enable such intelligent services, the NDT should be carefully crafted to take into consideration how data is collected and stored, how it is processed to model the network in the digital world and how services are implemented and requested to satisfy the applications requirements. That is why a detailed architecture of the NDT is proposed in Figure 2.

The physical network interacts with the SDN-Controller to share relevant data such as nodes general information (e.g., address, radio interference range, the list of direct neighbors, etc.) or data representing the node's state (e.g., energy level, link quality, number of sent packets, number of received packets, etc.) among others.

The SDN-Controller interpretes the received node's general information data to construct a global topology and stores the node's state information as well as the constructed topology in the data lake. Also, the SDN-Controller acts as an interface between the NDT and the physical counterpart in a way that it can interpret service requests from the real network and translate them so that the NDT can understand them and provide the requested service. In addition, the SDN-Controller is responsible for applying the network decisions made by the NDT in the real network. The SDN paradigm is adopted because it allows Network Function Virtualization (NFV) and network slicing. NFV makes it possible to run multiple applications on the same network infrastructure while network slicing can divide the network into virtual slices independent from each other and each tailored to satisfy the requirements of a specific application. The capabilities brought by these technologies play an essential role in unlocking the full potential of the NDT.

The data lake is part of the NDT and it is considered as a database where different data structures are stored and retrieved. Logical models of the network can be built based on the data provided by the data lake. They can either be basic models representing the network components or they can be more advanced models including more features for functional purposes such as anomaly detection, communication scheduling, security management, among others.

Digital Twin Entity Orchestrator (DTEO) is the management entity of the whole NDT system. Requesting models to assess the network performance by executing different what-if scenarios and learning continuously the best network parameters. Setting simulation parameters into a simulation/emulation asset and getting back performance metrics that should help in improving the network operation. This orchestrator leverages intelligent networking services to satisfy the applications requirements.

Real-time network monitoring is implemented in the NDT by interacting with the DTEO and also requesting the intelligent networking services when needed. It can be seen as the human-machine interface of the NDT system since it includes a visualization tool where the network engineer can intervene when needed to better assist the network.

The NDT exposes the networking capabilities to the applications which in return send their QoS/QoE (Quality of Service/Quality of Experience) requirements. An interface between the NDT and the applications is provided to translate the networking capabilities to the applications and also the QoS/QoE requirements to the NDT. Based on the sent applications requirements, the NDT finds the best networking configuration that should satisfy these needs.

Intelligent networking services are implemented in a modular fashion and can be requested anytime by the DTEO or the network monitoring tool : a) Predictive maintenance: the objective of this service is to predict network failures in advance, AI algorithms could be leveraged for this purpose based on data related to the networking operation. In addition to the historical data collected from the physical network, possible what-if scenarios could be simulated to get more insights and further enrich the dataset used for predictive maintenance.

b) Network diagnosis: through this service, the network behavior is observed continuously and possible breakdowns such as node failures, link failures, network misconfigurations, security attacks, etc. can be detected. Comparing the NDT behavior with the real network's behavior continuously and whenever an unexpected behavior in the physical network occurs, an anomaly is arised. This service outputs diagnosis reports that can be visualized in the network monitoring tool giving the network engineer insights on the networking operation. Based on these reports, accommodation of the current network policies is performed or in more severe cases the network is reconfigured to overcome the detected anomalies. c) Security management: this service enables testing security attacks safely in the NDT in order to detect possible security breaches and reinforce the network's security.

d) Resource Allocation: this service allows efficient resource allocation for scheduling the communication between the wireless network nodes. Reinforcement Learning (RL) is a potential algorithm for this purpose since it can provide a general policy that can give the best communication scheduling valid for various scenarios. Taking advantage of the NDT, the learning phase will be executed in a replicated version of the real network and continuous enhancements of communications scheduling can be ensured. During training, the actions chosen in each episode can be simulated and evaluated in the NDT. This latter returns the reward to the RL agent based on the simulation performance and the training agent iterates in this way until converging to an optimal policy. e) Interoperability: this service is responsible for ensuring the interoperability between the different application digital twins and breaking the software silos between their various closed platforms or interfaces. It includes formatting capabilities allowing the translation of multiple closed platforms/interfaces specificities to a common format understood by the NDT.

f) Mapping and Mobility: this service aims at tracking the network nodes and can provide their location coordinates or even estimate them in case of the unavailability of such information. It plays an essential role in the sustainability service, e.g., giving the location of a node that should be removed or replaced for environmental purposes.

g) Energy Optimization: this service has the objective of increasing the network's lifetime by providing mechanisms that should reduce the energy consumption of the network. It can be requested for example when a node's energy level has reached a certain threshold in order to reduce its usage by adapting the deployed network policies according to that.

h) Sustainability: dedicated sustainability features provide a global view of the environmental footprint of the IIoT infrastructure. They take into account issues related to energy, battery management to change and recycle batteries, management of wireless nodes that will have to be replaced and removed to avoid contaminating nature in the case of environmental monitoring applications, monitoring the impact of radio waves, etc. Node obsolescence models integrated in the digital twin coupled with real system information allow to anticipate maintenance and recycling phases. This module leverages the Mapping and Mobility service providing the location of the wireless nodes when an intervention should be carried out on them (e.g., replacement, removal). This sustainability aspect is essential, given that in 2025, the number of connected objects is estimated at more than 75 billion.

IV. PROTOTYPE IMPLEMENTATION

In this section, a functional prototype of the NDT is presented as depicted in Figure 3. It is based on an existing SDN solution for WSNs (Wireless Sensor Networks) called SDN-WISE [START_REF] Galluccio | SDN-WISE: Design, prototyping and experimentation of a stateful SDN solution for WIreless SEnsor networks[END_REF]. Data collection from the IIoT network is ensured via an SDN controller that receives packets containing nodes general information and decrypts them to construct a graph describing the network topology. These packets are called reports and they are sent periodically to make sure that the network topology is always updated in the SDN controller. This latter shares the constructed topology with the NDT.

The NDT has two implemented services in our case which can be requested by the DTEO : a) Communication scheduling: allows an efficient resource allocation by scheduling the communication between the nodes. In our case, the WSN uses TSCH (Time Slotted Channel Hopping) as a MAC (Medium Access Control) protocol and TASA (Traffic Aware Scheduling Algorithm) algorithm is executed to perform timeslots/channels allocation. This service produces TSS (TSCH Scheduling Specification) packets containing scheduling information for each node and shares them with the DTEO that ensures their forwarding to the physical network via the SDN controller.

b) Routing management: ensures the routing function for the WSN. It executes the SPF (Shortest Path First) Dijkstra algorithm on the topology graph to get the shortest path from each node to the sink in a tree form. OpenPath packets are constructed from the resulting SPF tree and shared with the DTEO as a response to a routing request. The DTEO forwards such packets to the physical network. The DTEO interacts with Cooja simulator which takes as input an XML (Extensible Markup Language) file describing the simulation scenario to be executed. Cooja outputs simulation log files that are processed by the DTEO to get insights on the networking performance.

We believe that this first prototype can be enhanced with more functionalities and more services. For example, we can integrate more scheduling algorithms in the communication scheduling service and more routing algorithms in the routing management service. These various algorithms can be simulated and evaluated thanks to the DTEO and Cooja simulator interaction. After that, the algorithms giving the best performance are chosen to be deployed in the physical network.

The SDN-WISE data plane also includes a request packet, that is sent to the controller when a node doesn't know a route for a packet. This is a service request for routing and a response is sent from the controller giving the node instructions on how to route the packet. The request packet can be generalized for other service requests that can be demanded from the WSN to the NDT, e.g., requesting energy optimization when the node's energy level is low, requesting resource allocation accommodation when a node is overcharged with high traffic flows, etc.

Furthermore, the current implementation allows sending a Config packet from the controller to the WSN to read or write some parameters. This packet can be enhanced to enable more configuration capabilities for the NDT to update the network configuration for better performance.

V. CONCLUSION

In this article we presented an NDT for the IIoT by taking into consideration the complexity of such constrained networks. The core of the NDT is investigated and developed to enable a set of intelligent network services such as predictive maintenance, network diagnosis, security management, interoperability, sustainability, etc. Each service is carefully defined to meet the requirements of the IIoT. Finally, a prototype implementation of the proposed architecture based on SDN-WISE is presented.

In the future, we plan to further develop the proposed prototype with a particular focus on a resource allocation case study using reinforcement learning for scheduling the communication between the network nodes.

Fig. 1 .

 1 Fig.1. NDT throughout the network lifecycle[START_REF] Kherbache | When Digital Twin Meets Network Softwarization in the Industrial IoT: Real-Time Requirements Case Study[END_REF]

Fig. 2 .

 2 Fig. 2. Network Digital Twin architecture for the IIoT.

Fig. 3 .

 3 Fig. 3. Implementation of the NDT prototype.


Mehdi Kherbache
email: mehdi.kherbache@univ-lorraine.fr

Moufida Maimour
email: moufida.maimour@univ-lorraine.fr

Eric Rondeau
email: eric.rondeau@univ-lorraine.fr

Digital Twin Network for the IIoT using Eclipse Ditto and Hono

Keywords: Digital Twin Network, Internet of Things, Industy 4.0

Digital twins are beginning to revolutionize many industries in the last decade, offering a plethora of benefits to optimize the performance of industrial systems. They aim to create a continuously synchronized model of the physical system that allows for rapid adaptation to dynamics, primarily to unpredicted and undesired changes. A vast range of industrial domains have already benefited from digital twin technology, such as aerospace, manufacturing, healthcare, city management and shipping. In addition, recent research is beginning to explore the integration of digital twins into computer networks to enable more innovation and intelligent management. One of the building blocks of digital twin technology is the Internet of Things, where wireless sensors and actuators are deployed to provide interaction between the physical and digital worlds. This type of network is complex to manage due to its strong constraints, especially when controlling critical industrial applications, which gave rise to the Industrial Internet of Things (IIoT). We believe that optimizing IIoT will lead to effective integration of digital twins in Industry 4.0. In this paper, we design a Digital Twin Network (DTN) for IIoT where sensors, actuators, and communication infrastructure are replicated in the digital twin to enable real-time intelligent management of these networks. By taking advantage of Eclipse Hono which allows an efficient connectivity for the network devices and Eclipse Ditto for representing the devices states in a digital form and also providing easy access to these states for the DTN. In this way, cognitive network services such as predictive maintenance, sustainability features, network diagnostics, security management, resource allocation, energy optimization can be efficiently integrated and operated in the network lifecycle. We validate the proposed architecture by providing a resource allocation case study where we explain how the Time Slotted Channel Hopping mechanism is exploited in our architecture.

INTRODUCTION

The Internet of Things (IoT) market is growing rapidly in response to the significant development in the areas of smart buildings, smart industries, intelligent transportation, among others. It is expected to grow by 26% during the period 2017-2022 according to Markets and Markets (2017). The IoT is seen as a key technology to enable the fourth industrial revolution with the emergence of the Industrial Internet of Things (IIoT) enabling a broad set of industrial applications to attain full automation and high productivity.

IIoT systems control constrained industrial applications where various requirements need to be satisfied. Reliability, strict real-time constraints and energy requirements are the most crucial ones among others. The highly constrained environment of IIoT networks qualifies them as complex and hard to manage systems. In order to meet the previously mentioned various growing requirements, novel technologies are needed to assist these wireless systems. The Digital Twin (DT) is a strong promising candidate technology for this purpose. In fact, a digital twin is particularly promising for building a continuously updated model of a physical system to enable rapid adaptation to changing dynamics, as highlighted in the paper [START_REF] Grieves | Digital Twin: Mitigating Unpredictable[END_REF]. In addition, it offers a powerful simulation framework and can enable cognitive capabilities for the replicated system, [START_REF] Barricelli | A survey on digital twin: Definitions, characteristics, applications, and design implications[END_REF].

DT technology is classified as one of the top ten most promising technological trends for the next decade by Gartner in [START_REF] Cearley | Top 10 strategic technology trends for 2020[END_REF]. As stated in Markets and Markets (2020), it is expected to grow at a Compound Annual Growth Rate (CAGR) of 58% during the period 2020-2026. The digital replication of the physical system enables intelligent monitoring, innovation and various augmented functionalities. Various fields are already taking advantage of this fast growing technology such as manufacturing [START_REF] Mandolla | Building a digital twin for additive manufacturing through the exploitation of blockchain: A case analysis of the aircraft industry[END_REF], healthcare [START_REF] Bruynseels | Digital twins in health care: ethical implications of an emerging engineering paradigm[END_REF], aerospace [START_REF] Medina | A maturity model for digital twin implementations in the commercial aerospace oem industry[END_REF], smart city in IEEE (2017), among others.

Recent research work have started to investigate the opportunities that DT could bring to networks. The IETF (Internet Engineering Task Force) draft in [START_REF] Zhou | Digital Twin Network: Concepts and Reference Architecture[END_REF] proposes a reference architecture for the Digital Twin Network (DTN) along with issues and open research perspectives related to its implementation. More detailed architectures are proposed in [START_REF] Almasan | Digital twin network: Opportunities and challenges[END_REF]; [START_REF] Khan | Digital twin of wireless systems: Overview, taxonomy, challenges, and opportunities[END_REF] and their implementation is discussed across various aspects such as DTN model construction using machine learning algorithms, procedures to enable new networking functionalities (what-if analysis, troubleshooting, network planning, etc.) along with enabling technologies for efficiently implementing the DTN. Moreover, data collection for the DTN is further addressed in [START_REF] Zhu | A knowledge graph based construction method for digital twin network[END_REF] where knowledge graphs are leveraged to concretely construct the DTN. With an application-level viewpoint, the DTN is leveraged to predict the global QoS (Quality of Service) by using Graph Neural Networks in Ferriol-Galmés et al. ( 2021) while an application-driven DTN middleware is proposed in [START_REF] Bellavista | Application-driven network-aware digital twin management in industrial edge environments[END_REF].

While DTN-related research are gaining a progressive interest recently, there is still a gap in providing a detailed architecture of a digital twin network for the IIoT given its constrained nature and complex characteristics that should be taken into consideration. We recently proposed in [START_REF] Kherbache | When digital twin meets network softwarization in the industrial iot: Real-time requirements case study[END_REF], a holistic DT-based architecture that should enable closed-loop IIoT network management and that enables augmented functionalities for the design and during the service of the IIoT. The current paper is a completion and a more in-depth architecture of the DTN for the IIoT taking advantage of Eclipse Hono (Eclipse (2017a)) and Eclipse Ditto [START_REF] Eclipse | Eclipse Ditto™ • open source framework for digital twins in the IoT[END_REF]). Hono acts as a connectivity layer supporting various protocols for IoT devices while Ditto is an open source framework for creating digital IoT twins. The provided architecture enables intelligent networking services such as predictive maintenance, network diagnosis, energy optimization, resource allocation, sustainability, among others. It is designed with a terre-à-terre vision that should allow its efficient implementation.

The remainder of the paper is organized as follows. Section 2 covers related work on the DTN. In Section 3, the architecture of the DTN for IIoT is presented while in Section 4 a resource allocation case study exploiting the IEEE 802.15.4e-TSCH (Time Slotted Channel Hopping) mode in the proposed architecture is discussed. Section 5 concludes the paper.

RELATED WORK

A number of research works are investigating the DTN concept, proposing different architectures, aspects and modeling procedures as well as proposing a variety of networking services to unleash the true potential of digital twins in the context of networking. The DTN concept is now being developed by the IETF group in order to standardize it. Their most recent draft [START_REF] Zhou | Digital Twin Network: Concepts and Reference Architecture[END_REF] outlines the basic notions and a reference architecture along with the main challenges and issues for building a DTN.

An ML-based DTN is presented in [START_REF] Almasan | Digital twin network: Opportunities and challenges[END_REF] as a key tool for efficient control and management of modern real-world networks. The efficient design of network optimization solutions, performing troubleshooting, simulation analysis and network planning are enabled to network operators by the adoption of DTN. The authors claim that deep learning techniques can be used to model the DTN of a network by considering certain parameters such as traffic, topology, routing, scheduling policies, etc. Then delivering as output some performance metrics related to the network (utilization, delay, packet loss, etc.). By pursuing an iterative feedback-based process between the DT and a network optimizer, the latter can find the best network configuration that meets the requirements set by a network operator.

A high-level architecture for the digital twin of wireless systems is introduced in [START_REF] Khan | Digital twin of wireless systems: Overview, taxonomy, challenges, and opportunities[END_REF] in three layers: Physical Interaction Layer, Twin Layer and Services Layer. Two aspects are considered to discuss different details regarding digital twins for wireless systems. Wireless for twins considers the efficient use of wireless resources to enable efficient signaling of twins over a wireless link. On the other hand, twins for wireless discusses the role of digital twins in the implementation of wireless systems. [START_REF] Zhu | A knowledge graph based construction method for digital twin network[END_REF] proposes a four-layer architecture for DTN including a physical network layer, a data lake layer, a DT layer, and a network application layer. The data lake layer collects, stores and preprocesses the collected data to provide knowledge and relationship extraction to the DT layer for building the DTN model. The DT layer includes: i) physical entity modeling that complements the modeling of individual network components, ii) requirements modeling that is used to develop a variety of scenario models such as network resource prediction, anomaly detection, automatic operation through AI algorithms, iii) the twin management & control center that manages the DT layer to satisfy the needed requirements. A sketch-based data collection algorithm is proposed to efficiently collect data in the data lake layer and the DTN is built using knowledge graphs.

TwinNet, presented in [START_REF] Ferriol-Galmés | Building a digital twin for network optimization using graph neural networks[END_REF], models the relationship between different network characteristics (topology, routing, queue scheduling and ingress traffic) using a digital twin that exploits graph neural networks (GNNs) to predict the overall QoS (prediction of average delay per flow). It includes an optimizer that finds the best routing and/or scheduling policies meeting complex SLAs (Service Level Agreements). The particularity of the proposed model is its generalization capability, since it performs well in network scenarios that have not been seen in the training set.

An Application-driven Digital Twin Networking (ADTN) middleware is proposed in [START_REF] Bellavista | Application-driven network-aware digital twin management in industrial edge environments[END_REF] to smooth the interaction with heterogeneous distributed industrial devices and dynamic management of network resources by following an application-level perspective. This architecture is based on a cross-layer SDN approach to actively manage the industrial environment while taking into consideration QoS requirements and network configuration adaptation capabilities.

To enable self-optimization of mobile networks, [START_REF] Deng | A digital twin approach for self-optimization of mobile networks[END_REF] presents an algorithmic framework combining digital twin technology, reinforcement learning and expert knowledge. A DTN is built to predict the future state of the network based on its current state. Then, reinforcement learning and expert knowledge extract the predicted state to make optimization decisions and send them back to the digital twin which evaluates these decisions and applies the best one in the physical network. The latter sends the new network state back to the digital twin after applying these decisions after an evaluation time, thus enabling a closed-loop self-optimization of the physical network. In [START_REF] Nguyen | Digital twin for 5g and beyond[END_REF], a 5G digital twin is designed to help with the development and deployment of complex 5G networks and enable cost-effective access to 5G, as the deployment of 5G networks is too expensive.

In [START_REF] Zhao | Intelligent digital twinbased software-defined vehicular networks[END_REF], the Software Defined Vehicular Network (SDVN) based on an intelligent digital twin was developed to enable intelligent and adaptive routing in vehicular networks. Another work in [START_REF] Zhang | Adaptive digital twin and multiagent deep reinforcement learning for vehicular edge computing and networks[END_REF] designed a digital twin network to model an aggregation scheme for edge services of vehicular networks. The DTN reveals potential correspondences between edge services and massive vehicle pairs. The latter divides the complex vehicular network into multiple parts based on potential service associations, which significantly reduces the complexity of scheduling edge services.

In order to enable flow emulation in the DTN, [START_REF] Yang | A systematic network traffic emulation framework for digital twin network[END_REF] proposes a comprehensive flow emulation framework for the digital twin network to keep the physical network traffic consistent with the digital twin network traffic. The authors demonstrate, through a case study, how the proposed framework can accurately measure delays without affecting the physical network.

DIGITAL TWIN NETWORK FOR IIOT

In this section we describe the DTN architecture for the IIoT as an effort to enable intelligent management and more innovation for such networks.

As depicted in Figure 1, the physical IoT network interacts with the connectivity layer where Eclipse Hono (Eclipse (2017a)) is used. This latter is designed for simplified IoT device connectivity by eliminating the protocol silos of the different devices. In fact, Hono contains for each supported protocol (e.g., HTTP, MQTT or CoaP) a microservice called Protocol Adapter which maps the connection protocol of the device to Hono's APIs (Application Programming Interface). A device registration API is included to make Hono aware of devices that can or will connect to the service. In addition, authentication API is used to verify the identity of the devices willing to connect to Hono. Telemetry and events APIs are used by protocol adapters to send telemetry/events downstream (from devices to business applications). Command & Control API is used by business applications to send commands to connected devices.

Eclipse Hono communicates with back-end business applications using Advanced Message Queuing Protocol (AMQP) 1.0. In our case it is connected to Eclipse Ditto [START_REF] Eclipse | Eclipse Ditto™ • open source framework for digital twins in the IoT[END_REF] which is an open-source frame-work for creating digital IoT twins. It focuses on back-end scenarios by providing web APIs in order to simplify working with already connected devices and things from customer apps or other back-end software. Ditto ensures that access to twins can only be done by authorized parties via the policies service. Eclipse Ditto structures the data sent by devices via Hono into Digital IoT Twins. A digital twin of an IoT device in Ditto is represented as a Thing consisting of attributes and features. The Things service is responsible for persisting things and features in Ditto. The follow-up of the modifications made to the Things, Features and Policies is ensured with the Things-search service. Also, this latter updates an optimized search index and executes queries on it.

Two channels are available for use in Ditto, a twin channel that connects to the digital representation of a Thing and a live channel that routes a command/message towards an actual connected device. Ditto uses MongoDB as a database and only stores the latest state of Things. Furthermore, Ditto provides an HTTP API and a WebSocket to applications to facilitate retrieving Things-related information.

The Digital Twin Network interacts with Ditto via this HTTP API or a WebSocket. The DTN stores things states in the database and updates them wherever a modification to the Things is made in Ditto. This is done because Ditto only stores the latest states of the Things in its data store (mongoDB). This interaction allows storing historical data of the Digital IoT Twins during the entire network lifecycle. As shown in Figure 2, The collected data is exploited to construct different types of models, they can be either basic ones representing the network topology or more advanced ones built for functional purposes (anomaly detection, security management, resource allocation, among others). These models are stored in the database. Moreover, simulation capabilities are provided in the DTN in order to enable running what-if scenarios and getting insights on the network performance in different situations. Network monitoring is also included to better assist the network knowing that it is considered as the human-machine interface between the DTN and the physical IIoT.

The DTN is composed of different components and managed by a central entity called Twin Management & Control. This latter triggers the construction of the logical models when needed, pushes simulation settings in the simulator and receives performance metrics, requests the cognitive services and gets responses from them.

The cognitive services are implemented in the DTN for the IoT network knowing that it is aware of its state. These services run intelligent algorithms continuously with the goal of satisfying QoS/QoE (Quality of Service/ Quality of Experience) requirements giving the available resources. For instance, predictive maintenance as a service has the goal of predicting network anomalies in advance by exploiting the recorded historical data in the DTN. In addition, efficient communication scheduling is enabled by the resource allocation service which runs AI-based algorithms to find the optimal schedule for the IIoT to satisfy the applications requirements. Reinforcement learning is a promising candidate algorithm for communication scheduling problems according to recent research studies [START_REF] Chilukuri | Recce: Deep reinforcement learning for joint routing and scheduling in time-constrained wireless networks[END_REF]).

Further, sustainability features are included in a sustainability service. Taking into consideration energy, batteries management, monitoring the impact of radio waves and management of wireless nodes that risk to endanger the environment. This service makes use of the mapping and mobility service that is responsible for tracking the network nodes and provide their location coordinates. Besides, the energy optimization service's role is to increase the network lifetime by giving insights on factors that are causing increased energy consumption.

Moreover, network diagnosis is implemented to generate reports giving insights on possible network misconfigurations, nodes failure, security attacks, among others. These reports can be visualized in the GUI (Graphical User Interface) to aid the network engineer in assisting the networking operation. Testing security attacks in the DTN is enabled by the security management service in order to enforce network security. Also, the interoperability service includes formatting capabilities to break the software silos between the different application digital twins. This way, the DTN collaborates efficiently with application digital twins regardless of the technologies used in their implementation.

Finally, the DTN interacts continuously with applications by exposing the network's capability to them. On the other hand, the applications send their QoS/QoE requirements to the DTN which is in charge of finding the best networking configuration to satisfy these needs. An interface between the applications and the DTN is integrated to make a two-way translation of the networking capabilities to the applications and the QoS/QoE requirements to the DTN.

RESOURCE ALLOCATION CASE STUDY

In this section we present a resource allocation case study using TSCH (Time Slotted Channel Hopping) mode as a MAC protocol. It was introduced among several channel access modes in the IEEE 802.15.4e amendment, surveyed in [START_REF] Kurunathan | Ieee[END_REF]. TSCH has been designed to meet the requirements of IIoT applications as it provides time-critical guarantees and very high reliability. In order to exploit the benefits of TSCH, a schedule specification in terms of slots/channels assignment for each network node is required. Various scheduling algorithms for TSCH exist in the literature, they are either centralized, decentralized, static or hybrid according to [START_REF] Urke | A survey of 802.15.4 tsch schedulers for a standardized industrial internet of things[END_REF]. This categorization allows distinguishing the way a schedule is generated.

In the proposed DTN architecture, TSCH scheduling can be revolutionized by defining the schedule in the digital layer and leveraging the simulation capabilities provided. In fact, TSCH scheduling in such architecture can be obtained by executing the current physical network scenario in the simulation asset then sending the resulting schedule to the physical layer. Moreover, the resource allocation service of the DTN works continuously to improve the deployed schedule in an effort to satisfy the applications requirements depending on the current state of the physical network. This service also runs what-if scenarios to detect possible bottlenecks due to high traffic loads and predicts possible drops in performance then accommodates the schedule consequently. In addition, the resource allocation service can coordinate with the other intelligent network services to update the schedule, e.g., accommodate the schedule upon a request from the energy optimization service when the remaining energy of some sensor nodes is low.

The first step in developing the DTN using Eclipse Ditto is to define a JSON (JavaScript Object Notation) Thing structure in terms of attributes and features. In fact, the attributes are defined as static values that does not change frequently while features are dynamic state values that change more often. To show our vision in the development process of the DTN for the IIoT, Figure 3 represents a first example prototype of a Thing that represents the IoT devices in a digital form. The policyID is the ID of the access policy for this Thing which describes the authorized parties to read/write its values. One attribute representing the mote type is defined. We also defined some features that contribute in defining a TSCH schedule along with the schedule itself that is updated after executing a scheduling algorithm in the DTN : Advanced intelligent methods can be exploited for TSCH scheduling in the DTN architecture because the energy constraint is excluded in the digital layer. For instance, using reinforcement learning algorithms as they seem to adapt well and give promising results in this context, as in [START_REF] Chilukuri | Recce: Deep reinforcement learning for joint routing and scheduling in time-constrained wireless networks[END_REF] where a deep reinforcement learning scheduling algorithm for TDMA-based time constrained wireless networks is proposed. In fact, such algorithms require a learning phase in order to converge to an optimal policy, but this can considerably increase the energy expenditure if this phase is executed on the physical network. In our architecture, the optimal policy is learned in the DTN by means of simulation.

Furthermore, the proposed architecture can take into account application digital twins requirements. For instance, deadlines requirements of a time-sensitive application DT are sent to the DTN which is in charge of finding the optimal policies satisfying the aforementioned needs. The Twin Management & Control (TM&C) entity interpretes the received requirements and understands that the application packets should reach their destinations within a fixed deadline. After that, the resource allocation service is requested to adapt the communication schedule according to the received requirements. It simulates various scheduling algorithms by interacting with the TM&C which in turn defines the simulation settings and sends them to the simulation asset. This latter returns the performance metrics for each simulation, the TM&C analyzes these results and decides which policy responds best to the deadlines requirements.

CONCLUSION

In this paper we proposed a DTN architecture for the IIoT using two open-source tools. First, Eclipse Hono which acts as a connectivity layer eliminating protocol silos of the different IoT devices. Second, Eclipse Ditto which is a framework for building digital IoT twins and provides APIs for business applications to easily interact with these DTs. In our case it is a DTN application that is implemented on top of Ditto and interacts with it to record the networking operation all along its lifecycle. The proposed architecture enables implementing cognitives services such as predictive maintenance, network diagnosis, resource allocation, security management, sustainability, among others. Besides, network monitoring is enabled in the DTN along with simulation capabilities. All of the cognitive services exploit the simulation capabilities provided in the DTN to run what-if scenarios giving performance insights on the networking operation.

Finally, we presented a resource allocation case study by focusing on the service responsible for that and supposing the use of IEEE802.15.4-TSCH mode. An example structure of a Thing in Ditto is also presented with the attributes and features that should help in communication scheduling between the network nodes.

In the future, it is envisaged to develop the DTN application on top of Eclipse Ditto and develop the resource allocation and energy optimization services and assess their impact on the networking performance.

Fig. 1 .

 1 Fig. 1. Digital Twin Network using Eclipse Ditto + Hono.

•

  nodeAddress : the address of the IoT device in the physical network ; • batteryLevel : shows the current battery state of the IoT device, useful for the schedule definition ; • dataRate : the frequency of sending data packets in the IoT device , useful for the schedule definition ; • bufferState : gives the number of remaining packets that should be transmitted by the IoT device, useful for the schedule definition ; • Schedule : defines the schedule for the IoT device in terms of transmission cells (Tx cells) and reception cells (Rx cells), both represented by two lists : timeslots and channel offsets. The values of this feature are updated after executing a TSCH scheduling algorithm in the resource allocation service included in the DTN.

Fig. 2 .

 2 Fig. 2. Digital Twin Network internal architecture.

Fig. 3 .

 3 Fig. 3. Thing structure in Eclipse Ditto for the TSCH scheduling case.It is envisaged to further enrich the Thing structure in the future by considering more features that should allow giving a more precise state of the IoT device.


Mehdi Kherbache
email: mehdi.kherbache@univ-lorraine.fr

Otabek Sobirov
email: otabek.sobirov8@etu.univ-lorraine.fr

Moufida Maimour
email: moufida.maimour@univ-lorraine.fr

Eric Rondeau
email: eric.rondeau@univ-lorraine.fr

Abderrezak Benyahia
email: a.benyahia@univ-batna2.dz

Reinforcement Learning TDMA-Based MAC Scheduling in the Industrial Internet of Things: A Survey

Keywords: Reinforcement Learning, Communication Scheduling, Wireless Sensor Networks, TDMA, IIoT

As technology progresses, almost all Industrial Internet of Things applications are becoming more data-driven. Hence, more advanced ways of communication are required to both save energy and become efficient to collect huge amounts of data. Researchers have now agreed that the root improvement of IIoT communication can start from a lower layer which is Data Link. Time Division Medium Access-based protocols on this layer have drawn academia's attention to decrease the energy consumption of IIoT devices and achieve higher levels of throughput. Yet, a problem arises, that is, developing a scheduler has always been troublesome while designing a Medium Access Control protocol. Scheduling communication in TDMA-based MAC protocols is considered an NP-hard problem. It is one of the topics highly debated among researchers to make the scheduling adaptive to the huge variation of IIoT application requirements. Conventional scheduling algorithms may suffer from the rapid changes in sensor network topology, network throughput, and applications' delay targets. Therefore, Reinforcement Learning algorithms are being integrated into scheduling techniques because of their adaptability and efficiency. This paper surveys RL-based TDMA MAC protocols and compares them in terms of several unified features. Each protocol is explained and discussed with others regarding its advantages and drawbacks.

INTRODUCTION

Industrial Internet of Things (IIoT) is mainly comprised of sensor devices sharing a common wireless medium whose optimal use is critical in terms of energy, throughput, delay, etc. Time Division Multiple Access (TDMA)-based medium sharing protocols have been considerably studied to meet the IIoT application requirements at the MAC layer as stated by Teles Hermeto et al. (2017). TDMA is one of the most prominent medium-sharing strategies that provide delay-deterministic communication. In TDMA, time is divided into small units called frames which will be further divided into slots. A slot is a base entity for communication between two devices. Scheduling timeslots among nodes should be efficient and optimal so that nodes can save energy and achieve the targeted throughput. TDMAbased communication requires nodes to have some form of duty-cycle management mechanism that periodically determines the sleeping and active timeslots to save nodes' energy, [START_REF] Ergen | TDMA scheduling algorithms for wireless sensor networks[END_REF].

Wireless Sensor Networks (WSN) tend to be multi-hop (ad-hoc) networks where a node can send its data to another node to reach the sink node, which makes energysaving the most decisive factor while choosing a scheduling algorithm. However, IIoT application requirements necessitate high levels of throughput and low latency, and as invoked by [START_REF] Zibakalam | A New TDMA Scheduling Algorithm for Data Collection over Tree-Based Routing in Wireless Sensor Networks[END_REF], this is where TDMA-based scheduling algorithms play an important role due to their inherent characteristics of providing intended delay and throughput. The preceding TDMA-based scheduling algorithms generally utilize a single channel. However, with the recent advancements, most protocols integrated the channel hopping technique, which resulted in the compatible usage of TDMA with FDMA, such as Time Slotted Channel Hopping (TSCH) standard. TSCH, standardized in 2015, is defined in Watteyne et al. (2015) as a mediumsharing MAC protocol that incorporates time slotting and frequency hopping as the foundations. However, TSCH standard did not specify any scheduling technique or a way to build it, which allows researchers to create custom application-specific schedulers.

Scheduling communication in TDMA-based MAC protocols is considered an NP-hard problem, according to [START_REF] Saifullah | Real-time scheduling for wirelesshart networks[END_REF]. It is one of the topics highly debated among researchers to make the scheduling adaptive to the huge variation of IIoT application requirements. The rise in the number of IIoT devices has shown that conventional scheduling algorithms may suffer from the rapid changes in sensor network topology, network throughput, and applications' delay targets. Therefore, researchers started to integrate Reinforcement Learning (RL) algorithms into scheduling techniques because of their adaptability and efficiency. RL algorithms applied in schedulers usually gather information from nodes' medium usage and application performance, and then by processing it, they find the optimal time slot and/or channel offset schedules for communication.

Several surveys tried to collect and analyze TDMA-based schedulers. For example, [START_REF] Hammoudi | Scheduling in ieee 802[END_REF] covered some RL-based schedulers for TSCH without being exclusively reserved to that. A more recent survey in [START_REF] Urke | A survey of 802[END_REF] covers more TSCH scheduling techniques and classifies them into five categories : centralized, collaborative, autonomous, hybrid, and static. However, it does not include machine learning-based schedulers. This paper surveys RL-based scheduling techniques for TDMA-based MAC protocols in the IIoT. To the best of our knowledge, this is the first survey dedicated to RL-based TDMA schedulers in the IIoT.

We collected TDMA-based RL schedulers to date from the most popular search engines, including Google Scholar, ResearchGate, and IEEEXplore library with the following keywords : TDMA-based, MAC protocol, RL, IIoT, WSN, Scheduling. The initial number of papers was relatively big (for instance, google scholar showed 42), but it was narrowed down to 9 due to the general requirements of this survey. This survey focuses on general TDMA-based MAC layer (single or multi-channel) that make use of RL that can be deployed in the IIoT for different types of applications.

This survey paper classifies RL-based schedulers into two groups: single-channel and multi-channel TDMA-based scheduling. And, it has the following structure. First, a general background on RL is provided in Section 2. Section 3 describes each paper individually and summarizes the advantages and disadvantages of a protocol in question. After that, a table is given to compare the chosen protocols based on several features. Finally, Section 4 concludes the survey.

REINFORCEMENT LEARNING

Reinforcement learning is defined in [START_REF] Sutton | Reinforcement learning: An introduction[END_REF] as a machine learning technique based on learning from interactions with the environment. This learning process is goal-directed where an agent must discover which actions yield the most reward by trying them. The interaction between a learning agent and its environment is defined in terms of states, actions and rewards using the formal framework of Markov Decision Processes. An agent in state s takes an action a that takes it to another state s and then receives reward r from its environment. This learning process is repeated enough times for the algorithm to converge and results in an optimal policy π * . This latter maps the states to the optimal actions to ensure that the design goal of the algorithm is achieved.

RL methods can be classified in three categories according to [START_REF] Heidrich-Meisner | Reinforcement learning in a nutshell[END_REF] :

• Critic-only methods or learning based on value functions : their idea is to first find the optimal value function and then derive an optimal policy. • Actor-only methods : here the optimal policy is directly searched in the policy space. • Actor-critic methods : which are a combination of the above approaches, the policy (actor) and the value functions (critic) are represented and improved separately. The critic measures the performance of the actor and decides when it should be improved.

The most known critic-only method is Q-learning which is a model-free reinforcement learning algorithm, detailed in [START_REF] Watkins | Technical note: Q-learning[END_REF]. A QL agent takes an action to maximize the value function, Q(s, a), of the state-action pair. Q(s, a) represents the expected total discounted returns from the action taken in state s :

Q(s, a) ← α Q(s, a) + (1 -α)(r + γ max a Q(s , a))
where α is the learning rate, γ is the discounting factor that sets the preference either towards immediate rewards or to long-term rewards, r is the received reward from executing action a in state s that leads to state s . Further, max a Q(s , a) represents the largest Q-value that can be obtained from the actions that can be taken in the next state.

RL can be combined with deep learning to deal with the problem of high-dimensional state and action spaces, resulting in Deep Reinforcement Learning as stated in [START_REF] Arulkumaran | A brief survey of deep reinforcement learning[END_REF]. It is generally based on training Deep Neural Networks (DNN) to approximate the optimal policy and/or the optimal value functions. Furthermore, DNNs could be used to model the environment in which RL agents will be trained.

RL-BASED MAC SCHEDULERS

In this section we will detail the RL-based TDMA MAC schedulers by providing a description in terms of objectives, how RL is used in the proposed method and the performance results for each. Table 1 summarizes the discussed techniques in a comparative way, provides details of the experimentations done for each and also their advantages and drawbacks.

Single-channel

Liu and Elhanany ( 2006) introduced RL-MAC as a reinforcement learning based MAC protocol for WSNs. It was designed to optimize the nodes' radio on-off function in order to minimize energy consumed by the sensor nodes.

The main particularity of the proposed algorithm is that it adapts to the traffic generation pattern of the node and also of its neighboring nodes. The proposed learning scheme uses Q-learning with a reward function that tries to find a trade-off between minimizing energy consumption and ensuring an acceptable throughput. Precisely, the state s is represented by the number of packets queued for transmition at the beginning of the slotframe. The action is the reserved active time for the node. The reward function is a combination of two components, the first one reflects on the internal state of a node at timeslot t which aims to maximise the energy efficiency. The second component reflects on the state of other nodes as perceived at timeslot t + 1 which tries to minimize the number of missed packets. An -greedy method is followed in the learning algorithm to ensure a balance between exploitation and exploration. Performance evaluation of RL-MAC shows that it outperforms in terms of energy efficiency and data throughput S-MAC proposed in [START_REF] Ye | Medium access control with coordinated adaptive sleeping for wireless sensor networks[END_REF], and T-MAC proposed in van [START_REF] Van Dam | An adaptive energyefficient mac protocol for wireless sensor networks[END_REF], two MAC protocols designed to reduce energy waste caused by idle listening by managing the nodes' duty cycle. Also, latency is reduced considerably in RL-MAC compared to S-MAC especially when traffic load is heavy. Finally, we can say that despite the fact that the proposed protocol is one of the first works including RL for scheduling nodes' radio on-off, it is outdated (2006) and does not fit the strict requirements of nowadays IIoT applications.

Always in the context of energy-saving directed protocols, Mihaylov et al. (2011b) proposed a self-organizing reinforcement learning approach for scheduling the wakeup cycles of nodes in a wireless sensor network. It allows to adapt the use of sensor resources to the applications requirements in terms of latency, data rate and lifetime. Concretely, each node will learn to stay awake during the periods where it needs to communicate with its parents/children nodes (nodes that belong to the same coalition), this behaviour is called synchronization. At the same time, the node learns to stay asleep when neighboring nodes on the same hop are communicating (nodes in another coalition). In other words, the node desynchronizes with the neighboring nodes that are not in its coalition to avoid radio interferences and packet loss. The algorithm uses a value iteration approach similar to Q-learning with an implicit exploration strategy. The action space contains the timeslot numbers within the slotframe, an agent selects a slot when its radio will be switched on for the duration of the duty cycle which is fixed by the user. Each agent stores a "quality value" for each timeslot which is updated every time an event (overheard, sent or received packets, idle listening) occurs during that slot. The node will stay awake for those consecutive timeslots (of a length equal to the duty cycle) that have the highest sum of Q-values. Evaluating this protocol in different topologies has showed that it provides much lower end-to-end latency compared to S-MAC. However, various shortcomings can be addressed in the proposed protocol. For example, the duty cycle is fixed and equal for all nodes in the network which means that it is not traffic-adaptive. In addition, communications between active nodes on the same routing branch can collide since they are synchronized. These drawbacks has been solved in a subsequent extension of the algorithm called DESYDE, proposed in Mihaylov et al. (2011a). [START_REF] Savaglio | Lightweight reinforcement learning for energy efficient communications in wireless sensor networks[END_REF] proposed a Q-learning MAC protocol (QL-MAC) which self-adjusts the node's duty-cycle to minimize energy consumption without impacting the other network parameters. It optimizes the sleeping and active periods of the nodes based on traffic predictions and transmission state of neighboring nodes. This is ensured by applying a Q-learning scheme where each slot is assigned a Q-value that is updated based on the actions of a node or the state of neighbor nodes. A node decides whether it should stay active or in sleep mode during each time slot, so the action space depends on the number of timeslot in a frame. The reward function is crafted carefully to take into consideration the state of neighboring nodes in addition to node state. Performance results show that QL-MAC reduces considerably energy expenditure with a minimal negative impact on the PDR compared to CSMA/CA. However, the authors did not provide insights on the impact of the learning algorithm on latency which allows to say that there is no guaranties of low latency.

The previous works are protocols developed from scratch and do not rely on a standardized stable protocol, such as TSCH. Based on TSCH MAC protocol, Nguyen-Duy et al. ( 2019) presented RL-TSCH, a reinforcement learning solution for scheduling TSCH nodes. The algorithm schedules the process of turn on/off node's radio at each beginning of timeslot based on the current state and the previous state of the node. It takes into consideration the number of packets in the transmission buffer along with the remaining energy in every node. The learning agent determines the number of active and non-active timeslots at the beginning of each slotframe. Thus, the node behaves as in MSA (Minimal Scheduling Algorithm) during the active timeslots and turns off its radio during the nonactive timeslots. The algorithm applies Q-learning with an action space consisting in choosing the number of active timeslots. The reward function is designed to minimize energy consumption and ensure a high throughput and reliability. Evaluating the proposed algorithm in different small-scale topologies showed that it reduces the energy consumed to about one third compared to MSA, achieves a similar PDR but the latency is much higher to that obtained with MSA. Based on the obtained results, it is logical to say that the proposed algorithm does not fit for low-latency applications but may be well suited for applications with strict energy requirements.

Another work based on TSCH protocol is the one presented by [START_REF] Park | Multiagent reinforcement-learning-based time-slotted channel hopping medium access control scheduling scheme[END_REF]. The authors introduced a multi-agent reinforcement learning based scheduler for TSCH, called QL-TSCH. The proposed algorithm reduces collisions while allowing contention, so multiple links can be scheduled in the same timeslot. This has the effect of increasing network throughput while allowing low energy consumption, thus the algorithm fits for high-density and high-traffic applications. In fact, each node acts as a Qlearning agent that learns the transmission slot with the lowest transmission failure rate and transmits only in that slot. During the learning phase, an agent performs a transmission with a probability P exploration in a slot chosen by an action peeking mechanism that selects the least active timeslot. Otherwise, the action (the timeslot) with the highest Q-value is selected. The chosen timeslot is scheduled as a transmission slot and the remaining slots are scheduled as listening slots. Rewards are assigned based on the success/failure of the performed action, a positive value when a transmission succeeds and a negative value is assigned if it fails. This algorithm addresses the nonstationarity problem of the multi-agent system that may interrupt the convergence by including an action peeking mechanism. This latter consists in a node observing the activity of other neighboring nodes during its listening slots, the node concludes that a timeslot is already reserved whenever a nearby communication is detected during that slot. Performance evaluation of QL-TSCH has been done in a large-scale network of 99 nodes by considering the PDR and the end-to-end packet delay in three industrial scenarios, compared with Orchestra scheduler proposed in [START_REF] Duquennoy | Orchestra: Robust mesh networks through autonomously scheduled tsch[END_REF] and FTA scheduler proposed in [START_REF] Park | Frametype-aware static time slotted channel hopping scheduling scheme for large-scale smart metering networks[END_REF]. Results show that QL-TSCH outperforms both Orchestra and FTA in terms of PDR, a better end-to-end packet delay compared to Orchestra but FTA has the best performance regarding this latter metric. In spite of all the good results that QL-TSCH achieved, its energy expenditure has not been evaluated. This leaves an open future research perspective.

Multi-channel

Phung et al. ( 2013) proposed a multichannel protocol for data gathering WSNs with a reinforcement learning based scheduling algorithm. The aim of the algorithm is to minimize energy consumption caused by collision, idle listening and deafness problem in WSNs. It addresses the joint problem of route selection and transmission scheduling and solves it in a fully distributed manner without a need for a coordination between the nodes. In other words, it makes nodes learn not only to which parent but also on which channel they should forward their data. Concretely, in each slot a node executes an action from the set of available actions (listen on its own channel for reception or transmit to one of its parents default channels) and keeps track of the probability of successfully performing each action in that slot. Such a probability is updated for the selected action in a given slot and will be the basis for choosing the best actions in the scheduling phase. The trade-off between exploitation and exploration is ensured by applying a "win-stay lose-shift" policy. In fact, a successful action will be repeated in the same slot in the next frame while a failed action leads to choose randomly another action from the action space in the next frame. The algorithm is traffic-adaptive because a node only contends for channel access when it has packets in its queue. Evaluation results of the proposed protocol show that it outperforms a frequency-hopping protocol called McMAC, proposed in Hoi-Sheung et al. (2007), in terms of PDR, end-to-end latency and it provides 9 times better energy efficiency.

Phung et al. ( 2018) introduced a scheduler for TSCH networks supporting multiple QoS (Quality of Service) objectives. It is based on a trial-and-error process where each node acts as an autonomous agent learning from feedback from its environment and the interaction with the other nodes. In fact, two RPL (Routing Protocol for Low-Power and Lossy Networks) instances are considered in the design of the learning scheme, one is reserved for delaysensitive data and the other one for regular data. A node either listens for data coming from the children nodes, transmits data to the parent of delay-sensitive instance or transmits to the parent of regular instance. The reward function is a combination of rewards for the objective of reliability and energy and rewards for the objective of low latency. Based on the received reward signal, the action to execute in the next slotframe would be the same as in the current one if the reward equals 1 or another action is selected if the reward equals 0. The learning algorithm keeps track of the probabilities of successfully performing each action in each slot, the same way as the precedently described algorithm. The probabilities are updated for each executed action and the allocation process exploits the final obtained probabilities to choose the best actions for each node during the slotframe. The scheduler has been evaluated with two RPL instances where one is for delay-sensitive data (bounded latency setup at 2 timeslots) and the other is for regular data. Results show that the proposed scheduler provides much lower data delivery latency than Orchestra (one order of magnitude lower) while keeping a similar energy consumption. This proves that the scheduler fits for delay-sensitive applications requiring low latency.

In order to consider the more general problem of satisfying packets' deadlines in delay-sensitive environments, Chilukuri et al. (2021) proposed RLSchedule as a reinforcement learning based TDMA slot scheduler for networks with strict time constraints. It has the objective of finding a schedule where the least possible number of packets miss the deadlines by the least amount of time. The paper identifies a set of node features that will be the basis for network state representation. This enables the RL scheme to take scheduling decisions based on an up-to-date dynamic network status and not on the same static criterion every time. The proposed framework follows a centralized approach where a controller gathers information about the most frequently seen network scenarios and sends this knowledge to a server. Deep Reinforcement Learning with PPO (Proximal Policy Optimization) is applied to learn the optimal policy which will be sent to the centralized controller. This latter exploits the received policy to build schedules for any scenario it sees. The action space consists in choosing the first M (the set of available channels) non-conflicting transmissions based on a proven baseline heuristic. The used heuristics are the following five : a) Deadline Monotonic (DM), b) Earliest Deadline First (EDF), c) Proportional Deadline (PD), d) Earliest Proportional Deadline (EPD) and e) Least Laxity First (LLF). A sixth possible action consists in choosing the top M nonconflicting transmissions with the minimum set of features. The reward function is designed in such a way to minimize the number of packets missing their deadlines and even if they exist the time by which they miss their deadlines is minimized. Performance evaluation of RLSchedule shows that it provides much lower packet delay and it has the least percentage of missed packets compared to the other scheduling heuristics (DM, EDF, PD, EPD, LLF). The obtained results confirm that RLSchedule is well suited for time-constrained networks. Chilukuri and Pesch (2021) presented RECCE, a scheduler that follows the same principles of RLSchedule but considers a more general problem by including routing in the learning scheme. In fact, RLSchedule considers the best (shortest) routes following Dijsktra algorithm while RECCE explores and learns multiple routes and schedules to deliver more packets within the deadline. Experimentation results showed that by exploring different routing paths and choosing the one (not necessarily the shortest) with the minimum delay allows RECCE to meet the scheduling goal better than RLSchedule.

CONCLUSION

In this paper we surveyed the most prominent RL-based TDMA MAC schedulers for the IIoT. We provided a summary table that can be the basis for comparing the different surveyed schedulers, enumerating their advantages and drawbacks. From this table, we can notice that 44% of the papers focus mainly on energy objectives while the rest are more QoS directed. But there are no papers that consider both of the issues at the same time. In addition, Quality of Experience (QoE ) is not really exploited in these works and applications' needs should be better considered. Moreover, 44% of papers use network simulation tools for performance evaluation and 44% of papers use either matlab simulations or custom simulators. Further, only 33% of the papers evaluated their works on a real test-bed with real sensors. This means that the results truthfulness can be discussed regarding two angles : a) non network simulation tools do not integrate all the complexity of the WSN protocols and their environment, b) real platforms should be required for a complete validation.

As future work, it is intended to extend the survey to cover machine-learning based schedulers in general and not only RL-based ones as in this paper.

Table 1 .

 1 General Comparison of RL-based Schedulers

	Limitations		-synchronization problems -	no real test-bed experiment	-old proposal and compared	with outdated protocols.	-not well suited for irregular	data traffic. -fixed duty cy-	cle for all nodes -coordination	problem.	-no insurance of low latency -	tested against a basic protocol	(CSMA/CA)		-no extensive experimenta-	tions -not suited for low la-	tency applications. -bench-	marked against the most ba-	sic TSCH scheduling function	(MSA)	-no garanties of recuced energy	consumption.				-synchronization problems. -	experimentations done in a	matlab simulation.	-support of only two types of	traffic (delay-sensitive and reg-	ular). -no garanties of reduced	energy consumption. -experi-	mentations done in a matlab	simulation.	-no real test-bed experimenta-	tions -home-made custom sim-	ulator -need for a centralized	controller	similar to Chilukuri et al.	(2021)
	Advantages		increased throughput and en-	ergy saving.			-low end-to-end latency. -re-	duced energy consumption			-adaptable to topology	changes . -reduced energy	consumption		-reduced energy consumption	-traffic-adaptive					-fit for high-traffic, high-	density large scale networks. -	low latency and collision rate.			-traffic-adaptive -low energy	consumption		-traffic prioritization included	-low data delivery latency -fit	for delay-sensitive applications				-almost garanteed satisfac-	tion of deadline requirements.	-suitable for time-constrained	applications -scheduling deci-	sions based on different pre-	selected nodes features -adapt-	ability to topology changes.	-routing included in the learn-	ing scheme → increased perfor-	mance relatively to Chilukuri	et al. (2021)
	Compared with		S-MAC and T-MAC (star	with 5 nodes, linear with 10	nodes, mesh topologies with	100 nodes)	S-MAC (Topology: Line with 4	nodes, Single-hop mesh with 6	nodes, Grid with 4x4 nodes)		CSMA/CA (100% and 60%	duty cycles), Topology: mesh	with 7 nodes, grid with 4x4,	7x7, 10x10 nodes	TSCH Minimal Scheduling	Function ( In simulation : 4-8	nodes in star, linear and mixed	topologies. In real test-bed : 3	nodes)		Orchestra and FTA scheduler	(Mesh topology with 99 nodes)				Mc-MAC (grid and random	topologies of 25 nodes)		Orchestra (grid topology of 16	nodes)					Scheduling heuristics (DM,	EDF, PD, EPO, LLF,CFLLF,	Random)	Scheduling heuristics (DM,	EDF, PD, EPO, LLF,CFLLF,	Random)
	RL method Simulation Testbed		Q-Learning NS-2 simu-No	lation			Q-Learning OMNet++ No	simulation			Q-Learning Cooja sim-Implemented	ulation on TelosB	motes with	TinyOS	Q-Learning Cooja Nodes run-	simulation ning Con-	(Zoletia tiki OS	Z1 sensor	nodes)		Q-Learning No ARM	Cortex	M4 MCU-	based IoT	platform	General RL Matlab No	simulation		Trial-and-Matlab No	error process simulation					Deep RL Home-No	-Proximal made	Policy Op-simulator	timization	(PPO)	Deep RL Home-No	-Proximal made	Policy Op-simulator	timization	(PPO)
	Class		Single-	channel			Single-	channel			Single-	channel			Single-	channel					Single-	channel				Multi-	channel		Multi-	channel					Multi-	channel	Multi-	channel
	Main	Objective	Energy	saving			Low latency				Energy	saving			Energy	saving					High	throughput				Energy	saving		Low latency						Satisfying	packets	deadlines	requirements	Satisfying	packets	deadlines	requirements
	Scheduler		Liu and	Elhanany	(2006)		Mihaylov	et al.	(2011b)		Savaglio	et al.	(2019)		Nguyen-	Duy et al.	(2019)				Park et al.	(2020)				Phung	et al.	(2013)	Phung	et al.	(2018)				Chilukuri	et al.	(2021)	Chilukuri	and Pesch	(2021)