Masahiro Yamguchi

Trong Phuc Truong

Shohei Mori

Vincent Nozick

Hideo Saito

Shoji Yachida

Hideaki Sato

Superimposing Thermal-Infrared Data on 3D Structure Reconstructed by RGB Visual Odometry

In this paper, we propose a method to generate a threedimensional (3D) thermal map and RGB + thermal (RGB-T) images of a scene from thermal-infrared and RGB images. The scene images are acquired by moving both a RGB camera and an thermal-infrared camera mounted on a stereo rig. Before capturing the scene with those cameras, we estimate their respective intrinsic parameters and their relative pose. Then, we reconstruct the 3D structures of the scene by using Direct Sparse Odometry (DSO) using the RGB images. In order to superimpose thermal information onto each point generated from DSO, we propose a method for estimating the scale of the point cloud corresponding to the extrinsic parameters between both cameras by matching depth images recovered from the RGB camera and the thermal-infrared camera based on mutual information. We also generate RGB-T images using the 3D structure of the scene and Delaunay triangulation. We do not rely on depth cameras and, therefore, our technique is not limited to scenes within the measurement range of the depth cameras. To demonstrate this technique, we generate 3D thermal maps and RGB-T images for both indoor and outdoor scenes.

Introduction

Thermal-infrared cameras measure infrared rays emitted from any objects and can use this ray information to estimate the object temperature. This temperature, which can not be retrieved from RGB cameras, is a valuable information that leads to many applications in industrial domains as well as in academic research. Indeed, thermal-infrared cameras can be used to detect gas leaks, fires, abnormalities in electronic apparatus, and so on. Most of these issues are hardly detectable with visible light, thus thermal-infrared cameras constitute a precious tool in these cases. However, thermal-infrared camera also presents some drawbacks. In practice, the material used to produce the lenses of thermal-infrared cameras usually makes their field of view usually narrow compared to the lenses of standard RGB cameras. Moreover, textures perceptible in visible spectral domain (i.e. captured by RGB cameras) are likely lost in invisible light. Since the human eye deals with visible light, it is often difficult for humans to understand thermal-infrared images due to this texture alteration. These issues would be easily solved if it was possible to perfectly superpose a thermal-infrared image with its corresponding RGB image. This is feasible only if both RGB and thermal-infrared cameras share the same projection center (i.e., the same position). Some very specific cameras, such as the one used in [START_REF] Yasuda | Thermo-key: Human region segmentation from video[END_REF], can simultaneously capture RGB and thermal-infrared images. However, in standard situations, RGB and thermal-infrared images are captured using different cameras. This paper addresses the problem to effectively associate thermal data to its corresponding RGB counterpart for visualization purposes. The correspondence between RGB and thermal-infrared images is obtained using the 3D structure generated from DSO. The top row shows a reference view and a 3D reconstructed result, the middle row shows a 3D thermal model generated by the proposed method, and the bottom shows an RGB-T image and reference frames of an RGB and a thermal-infrared cameras.

Related Works

In this paper, we consider the situation where the acquisition device is composed of at least an RGB and a thermal-infrared camera. In a standard situation, the two (or more) cameras can not share the same position. Thus, the matching of these data requires some geometric computation.

A feasible way to proceed consists of adding a 3D sensor to the thermal-infrared and RGB cameras to handle these geometric constraints. Borrmann et al. [START_REF] Borrmann | Thermal 3D mapping of building façades[END_REF] use a laser scanner and an RGB camera. Due to its heavy weight, the laser scanner is attached to the RGB camera and mounted on a wheeled robot. The robot moves around the scene and performs laser scans repeatedly to generate a 3D RGB map. For each scan of the scene, the robot also acquires the corresponding 3D thermal map. While this method performs well, both the laser scanner and the robot are very expensive. Moreover, the need for a wheeled robot prevents the acquisition process to be conducted in scenes composed of stairs or other low-quality paths.

A cheaper alternative consists of using an RGB-D sensor to generate the 3D model of the scene. Vidas et al. [START_REF] Vidas | 3D thermal mapping of building interiors using an RGB-D and thermal camera[END_REF], as well as Matsumoto et al. [START_REF] Matsumoto | AR visualization of thermal 3D model by hand-held cameras[END_REF] mount a Kinect with a thermal-infrared camera on a hand-held stereo rig. They obtain a 3D structure of the scene using KinectFusion [START_REF] Newcombe | Kinectfusion: Real-time dense surface mapping and tracking[END_REF]. Although, this method performs well in indoor situations, low cost depth sensors, such as Kinect, usually fail in outdoor environments where the sunlight affects the depth value acquisition [START_REF] Horaud | An overview of depth cameras and range scanners based on time-of-flight technologies[END_REF][START_REF] Alenyà | Using ToF and RGBD cameras for 3D robot perception and manipulation in human environments[END_REF]. Moreover, the size of the acquired scene is usually limited to small areas with the use of KinectFusion.

Prakash et al. [START_REF] Prakash | 3D mapping of surface temperature using thermal stereo[END_REF] combines two thermal-infrared cameras and use stereo vision to compute a 3D model of the scene. In this method, they use epipolar geometry from the thermal-infrared images to constrain the computation of correspondence. However, the thermal stereo method is relevant only when there are significant temperature gradients on the object surface. Up to now, finding stereo correspondences between 2 thermalinfrared images is still very challenging, as stated in [START_REF] Treible | CATS: A Color and Thermal Stereo Benchmark[END_REF]. Indeed, since the thermal-infrared images are low textured, standard matching method often find very few correspondences.

Finally, Ham et al. [START_REF] Ham | An automated vision-based method for rapid 3D energy performance modeling of existing buildings using thermal and digital imagery[END_REF] compute the geometry of the scene using the Structure from Motion (SfM) technique [START_REF] Agarwal | Building rome in a day[END_REF] and superimpose the thermal information on the resulting 3D structure. This system uses only an RGB camera and a thermal-infrared camera, and thus handles both indoor scenes and outdoor scenes. However, the calculation cost of the 3D reconstruction from the SfM is significant. Indeed, SfM combines all the images of the considered sequence in order to match them. Thus, the generated 3D point cloud is accurate, however this process leads to a significant computational cost. On the other hand, SLAM uses selective frames (e.g., keyframes) or the last consecutive frames to improve its real-time performance during the video capture. Thus, this approach results in a low computational cost for a 3D reconstruction not as accurate as for SfM [START_REF] Davison | MonoSLAM: Real-time single camera SLAM[END_REF]. In this work, we propose a method combining an RGB camera with a thermal-infrared camera. As discussed in [START_REF] Ham | An automated vision-based method for rapid 3D energy performance modeling of existing buildings using thermal and digital imagery[END_REF], this setup does not include any depth sensor and thus also supports outdoor scenes. We use a simultaneous localization and mapping (SLAM) method instead of a SfM approach, so the calculation cost is lower than in [START_REF] Ham | An automated vision-based method for rapid 3D energy performance modeling of existing buildings using thermal and digital imagery[END_REF]. Moreover, we propose a robust calibration process for thermal-infrared cameras as well as an accurate camera relative pose for a more efficient thermal to RGB mapping.

Overview of the Proposed Method

In this paper, we present a method to generate RGB-T images (RGB plus Thermal) by superimposing thermal information obtained from the thermal-infrared camera onto the RGB images. More precisely, our system consists of four stages: first, we attach an RGB camera and a thermal-infrared camera on a stereo rig, then calibrate them using a common calibration board that can be simultaneously detected by both cameras, as described in Section 4. Second, we acquire a set of images of the scene with the RGB camera and generate at runtime a 3D structure of the environment, as explained in Section 5. In the third step, we estimate the projective scale between the 3D structure and the calibration board using depth images generated from both RGB and thermal-infrared camera sequences, as described in Section 6. Finally, the thermal map and the RGB-T images are generated using the 3D structure computed from the RGB images and the estimated scale. An overview of the whole process is depicted in Fig. 2.

Our main contribution is this paper consists in both the accurate thermal camera calibration process (Section 4) and the overall process to superimpose Thermal-infrared data on 3D reconstruction from the RGB images, including the specific problem of finding the unknown projective scale between RGB and Thermal-infrared cameras (Section 6).

Camera Calibration

Calibration Board Issues

Camera calibration is a well known process for pinhole cameras. This process, that usually requires point correspondences between images, has been wildly studied for RGB images but appears to be more challenging for thermal-infrared cameras, due to the lack of textures. Assuming that the calibration process is performed using a calibration board, the thermal-infrared camera hardly detects the calibration pattern. Indeed, a thermal-infrared camera measures the temperature from the infrared rays emitted by objects. The locally emitted infrared light intensity depends more on the object tem-perature than on the color of the object. Even though it is true that black objects absorb and emit a larger amount of infrared energy than brighter objects, in practice, a thermal-infrared camera cannot detect a checker pattern if the temperature of the calibration board is uniform across the black and white parts.

A general way to make the board detectable by the thermal-infrared camera is to heat the calibration board in order to make the re-emitted light significantly hotter than the intrinsic object temperature. Matsumoto et al. [START_REF] Matsumoto | AR visualization of thermal 3D model by hand-held cameras[END_REF] uses a hair dryer to heat the calibration board while Ham et al. and Weinmann et al. [START_REF] Weinmann | Thermal 3D mapping for object detection in dynamic scenes[END_REF][START_REF] Ham | An automated vision-based method for rapid 3D energy performance modeling of existing buildings using thermal and digital imagery[END_REF] prefer to use a set of lamps. Some other methods proposed by Prakash et al. and Saponaro et al. [START_REF] Prakash | Robust thermal camera calibration and 3D mapping of object surface temperatures[END_REF][START_REF] Saponaro | Improving calibration of thermal stereo cameras using heated calibration board[END_REF] use a flood lamp to heat the calibration board. However, Vidas et al. [START_REF] Vidas | A mask-based approach for the geometric calibration of thermalinfrared cameras[END_REF] suggest that the use of a flood lamp is comparatively inaccurate.

In our method, we designed our calibration board from a printed chessboard pattern and placed black plastic tape on the black parts. Then, we used a freezer to cool the calibration board. Heating the board with a lamp is also a good way to proceed, however using a freezer is more convenient in our case.

The resulting image obtained after heating or cooling still presents some very blurred textures, as shown in Fig. 3. This blur is caused by the fact that the board temperature quickly tends to diffuse from one cell to the next. Thus, after the heating or cooling process, the measured temperature gradually diffuse from a black part to a white one. This blur is a serious issue for the calibration process, in term of accuracy but also for any automatic chessboard detection. In our method, we first estimate the camera lens distortion parameter using the plumb line method (more details in Section 4.3). Next, we perform an initial estimation of the checker pattern corners. These estimated points are significantly refined using a non-linear process involving constrains such as cross-ratio and vanishing point consistency.

Camera Model

In this paper, we use the pinhole camera model [START_REF] Hartley | Multiple view geometry in computer vision[END_REF] for both the RGB and thermal-infrared cameras. Let x = (u, v, 1) and X = (X, Y, Z, 1) be the image coordinates and the world coordinates, respectively. A camera projection matrix is defined as follows:

P = K R t , with K =       f x 0 x 0 0 f y y 0 0 0 1       R ∈ SO(3), t ∈ R 3 . ( 1
)
where K is the camera's intrinsic matrix, defined by the camera principal point (x 0 , y 0 ) and the camera focal length f x , f y expressed in pixel units. R is a 3 × 3 rotation matrix defining the camera orientation and t is a transformation vector representing the camera position. A 3D world coordinate point X = (X, Y, Z, 1) projects to a 2D image point x = (u, v, 1) by sx = PX, with s ∈ R * .

Distortion Correction

The camera lens distortion correction is performed only once per camera unless the lenses are changed or manipulated (zoom or strong refocus). We use the plumb line method described by Devernay and Faugeras [START_REF] Devernay | Straight lines have to be straight[END_REF] in order to correct the lens distortion by making straight lines of the scene also straight in the image. This method requires a set of points assumed to be aligned in the real world. In our case, we manually extract these points from buildings images that contain good straight lines with strong thermal contrasts with the sky. The method defined in [START_REF] Devernay | Straight lines have to be straight[END_REF] non-linearly computes the fine radial distortion parameters such that the selected points become aligned. The resulting images present a very good correction, as depicted in Fig. 4. This radial distortion correction procedure is strongly required for the next steps of the camera calibration process.

Corner Refinement

This section presents our chessboard corner points refinement. This process highly improve the accuracy of the camera calibration.

Initial Detection

A very common way to detect a calibration chessboard is to use the automatic tools provided by OpenCV1. This tool automatically detects the corner of the chessboard and is wildly used to calibrate RGB cameras. However, in the thermal-infrared camera case, the inherent blur (shown in Fig. 3) makes the point detection accuracy drastically fall. The point detection is clearly inaccurate and messy for thermal images. Since a good accuracy in camera calibration is required for the thermal to RGB mapping, we propose a chessboard detection refinement process.

Corner Refinement based on Line Intersection

This process starts from an undistorted image of the calibration chessboard obtained from Section 4.3. Thus, 1Open Source Computer Vision Library (OpenCV) http://opencv.org/ every corner of the checkerboard pattern should be detected such that each point lying on the same row (respectively column) should be aligned. We refine the corner points using line intersection optimization inspired by the method proposed by De la Escalera and Armingol [START_REF] De La Escalera | Automatic chessboard detection for intrinsic and extrinsic camera parameter calibration[END_REF]. In our method, we perform a least square fitting of the lines from the "inaccurate" detected corners instead of using the Hough transform on the image.

Naturally, these first estimated lines are not accurate due to the blur on the thermal-infrared images, see Fig. 5 (left). However, they can be considered as a good starting estimation for the first step of our refinement process. These lines are transformed to fit the neighboring maximum gradient of the image intensity, computed only in the orthogonal direction of the lines. In other words, the lines are transformed to lie on the middle of the blur between the black and the white cells of the chessboard. Since the maximum gradient point set may include outliers, the line fitting is performed with RANSAC line fitting. Finally, the corner points are estimated based on the intersection between each rows and columns.

Corner Refinement based on Geometric Constraints

Firstly, let P denote the set of the corners initially detected on the calibration board and refined in Section 4.4.2. The next step of our refinement consists in a non-linear process. Indeed, we also optimize the corners' position according to both the cross-ratio extracted from each chessboard cell and the vanishing points of the chessboard rows and columns respectively.

Cross ratio Corner points are placed at constant intervals on the chessboard and thus should satisfy some constraints of cross-ratio. The cross-ratio is a perspective invariant defined in the invariance theory of crossratio [START_REF] Zhang | Calibrating camera radial distortion with cross-ratio invariability[END_REF][START_REF] Hartley | Multiple view geometry in computer vision[END_REF]. If the points A, B, C and D are collinear in space, the four corresponding projected image points A , B , C and D are also collinear. The cross-ratio of two sets of any such consecutive four points from any row (or column) of the chessboard are identical and satisfies the following equation:

ρ = AC CB / AD DB = A C C B / A D D B (2)
In practice, it is possible to measure how the detected points satisfy this cross-ratio constraint by a cost function C cross (P) inspired by a method proposed by Ricolfe-Viala et al. [START_REF] Ricolfe-Viala | Robust metric calibration of non-linear camera lens distortion[END_REF]. This cross-ratio cost is equally di- vided into two costs dedicated respectively to rows and columns constrains:

C cross (P) = C rows cross (P) + C cols cross (P)
where C rows cross (P) and C cols cross (P) are computed in the same way. For clarity purpose, we will detail only the computation of C rows cross (P). According to the cross-ratio constraint, each four successive points of a row should lead to the theoretical cross-ratio ρ defined by any consecutive points on the chessboard. Let Q rows be the set of all combination of four consecutive points q = A, B, C, D of P, extracted only from the rows of the calibration board. The cross-ratio ρ(q i ) of each q i ∈ Q rows is defined as:

ρ(q i ) = A i C i C i B i / A i D i D i B i
Then, the cross-ratio cost function dedicated to the rows points can be defined as:

C rows cross (P) = 1 card(Q rows ) q i ∈Q rows 1 - ρ(q i ) ρ (3)
The same procedure holds for Q cols with:

C cols cross (P) = 1 card(Q cols ) q i ∈Q rows 1 - ρ(q i ) ρ ( 4
)
Vanishing points As for the cross-ratio cost function, the vanishing point cost function is divided into two costs since a calibration board contains two main vanishing points:

C vanish (P) = C rows vanish (P) + C cols vanish (P)

Again, let's just consider the chessboard row vanishing point. The cost function procedure starts by an estimation of the vanishing point (x v , y v ) computed as the least square intersection of all the line extending each rows. Then, the vanishing point cost C rows vanish (P) is defined as the average distance from this vanishing point to these lines. Since each of those lines should pass throw the vanishing point, the average distance from the lines to the vanishing point should tend to zero if the corner points are accurately positioned. More formally, let r i = (a i , b i , c i ) be the Hessian form of the i st row, and V the set of rows of P, then the cost to minimize can be expressed as:

C vanish (P) = 1 card(V) r i ∈V |a i x v + b i y v + c i | a 2 i + b 2 i ( 5
)
The denominator transforms the line equations in their normalized Hessian form. Moreover, note that we don't use this formulation when the calibration board is orthogonal to the principal ray of the camera since in this situation, the vanishing points would lie at infinity which is not compatible with our vanishing point scoring method. Fig. 6 depicts both the cross-ratio and the vanishing points constraints.

Data fidelity term

Using only the two first terms would lead to a set of points perfectly aligned and with the perfect distances from one to the next, but not necessary consistent with the calibration board since the refined points could freely move anywhere. Thus, to maintain a certain consistency between the points and the calibration board, we add a data fidelity term C dist (P).

C dist (P) = 1 card(P) x∈ P d(x i , xi ) (6)
global cost function The final cost function to optimize the chessboard corner position estimation can be described as the weighted sum of the three first cost functions:

C(P) = arg min P C vanish (P) + αC cross (P) + βC dist (P)

where α, β are given parameters. Once the corners are refined by minimizing the weighted sum of Eq.( 7), these refined points are used to calibrate the camera parameters using Zhang's method [START_REF] Zhang | A flexible new technique for camera calibration[END_REF].

Deciding the Weights α and β

The weights α and β of Eq. ( 7) are decided experimentally. We generate a virtual chessboard consisting in a set of points regularly arranged and transform them using a random but realistic perspective transformation (homography). Then, we added a Gaussian noise on these points. For a large set of values for α and β, we generate many noised virtual chessboards and ran our algorithm. We selected the couple (α, β) that reconstruct the best the original virtual chessboard in average over all the virtual noised chessboards.

Reconstructing the 3D Structures

Assuming the cameras to be calibrated, the next step consists in the computation of a 3D structure of the scene and in the camera pose estimation of each frame in the 3D structure coordinate system. This 3D structure will be the support to projected thermal data for the final visualization. Note that we only need the camera pose of the RGB camera since we already know the relative pose between the RGB camera and the thermal-infrared camera, from the camera stereo rig calibration.

Many different techniques exist to compute a 3D reconstruction. Table 1 shows a comparison between state-of-the-art methods. Density refers to how dense the 3D structure is, cost corresponds to the calculation cost, and scale specifies whether the method can handle large scenes. Among these methods, techniques using depth sensors based on time of flight or structured light in the infrared domain (e.g., Microsoft Kinect sensor) are immediately discarded since such low-cost depth sensors do not perform well in outdoor scenes due to interference with sunlight. Moreover, low-cost depth sensors cannot handle large scenes due to their intrinsic limited range of action. These constrains exclude the use of KinectFusion. Better sensors are expensive and thus do not match our purpose of keeping the whole process affordable. Moreover, our method should be easy to set up and to implement, so we also discard Lidar 3D scanners. The remaining solutions suggest to use an RGB camera to perform the 3D reconstruction.

There are two main approaches to generate 3D structures from RGB images: direct methods, like Engel et al. [START_REF] Engel | Semi-dense visual odometry for a monocular camera[END_REF], referring directly to image intensity, and feature-based methods, such as Klein et al. [START_REF] Klein | Parallel tracking and mapping for small AR workspaces[END_REF], that generates and matches feature points to construct the 3D map. Since our goal is to generate a large thermal map with real-time performance, possibly on outdoor scenes, the best candidates are direct sparse odometry (DSO) [START_REF] Engel | Direct sparse odometry[END_REF], large-scale direct monocular simultaneous localization and mapping (LSD-SLAM) [START_REF] Engel | LSD-SLAM: Large-scale direct monocular SLAM[END_REF] and ORB-SLAM [START_REF] Mur-Artal | ORB-SLAM: a versatile and accurate monocular slam system[END_REF]. In these three 3D reconstruction systems, DSO and ORB-SLAM are more accurate than LSD-SLAM [START_REF] Engel | Direct sparse odometry[END_REF][START_REF] Mur-Artal | ORB-SLAM: a versatile and accurate monocular slam system[END_REF]. Moreover, the 3D structures generated from DSO have a better density than those generated from ORB-SLAM. Thus, we selected DSO for our 3D reconstruction. [START_REF] Engel | Direct sparse odometry[END_REF] direct sparse small large LSD-SLAM [START_REF] Engel | LSD-SLAM: Large-scale direct monocular SLAM[END_REF] direct semi-dense small large PTAM [START_REF] Klein | Parallel tracking and mapping for small AR workspaces[END_REF] feature sparse small small ORB-SLAM [START_REF] Mur-Artal | ORB-SLAM: a versatile and accurate monocular slam system[END_REF] feature sparse small large SfM [START_REF] Agarwal | Building rome in a day[END_REF] feature sparse large large KinectFusion [START_REF] Newcombe | Kinectfusion: Real-time dense surface mapping and tracking[END_REF] RGB-D dense small small Lidar 3D scanner dense -large

Relative Scale Between the Point Clouds

Any 3D models generated by a monocular RGB camera is build and defined up to an unknown scale. Since we want to back-project the thermal-infrared images on the 3D model obtained by DSO, this unknown scale should be estimated. In practice, this scale is also the scale that relates a depth map generated from the RGB image to the corresponding depth map generated from the thermal-infrared images. This Section defines how to estimate the scale of the DSO point cloud in the stereo rig coordinate system, and how to back-project the thermal-infrared images on this point cloud. DSO divides frames into two types: key frames for which a depth map is computed, and the other frames just used to refine the depth map generated by the key frames. We use these depth maps to superimpose thermal information on the 3D structure. For a given key frame, the depth value of each pixel x refers to a 3D point that can be projected on the infrared thermal image on x . This projected point x corresponds to a thermal value that can be associated to the pixel x in the RGB image.

More precisely, let I rgb express an RGB image and I ir a thermal-infrared image. The depth maps of DSO are sparse so some points have depth value and some others do not. Image coordinates with depth value d are denoted by x = (x, y, 1) . A general back-projection function [START_REF] Hartley | Multiple view geometry in computer vision[END_REF] to converts the RGB image coordinate to a 3D coordinate according to the depth d in the DSO depth map is given by:

X(d) = KR -1 (dx -Kt) 1 ( 8
)
If we consider that P rgb defines the coordinate system of the stereo rig, then P rgb = K rgb [Id|0] and

P ir = K ir [R ir |t ir ],
where [R ir |t ir ] is the relative pose between the two cameras, usually with the arbitrary constraint t ir 2 = 1. Since we are back projecting from P rgb = K rgb [Id|0], the back-projection function simplifies to :

P rgb (x, d) = dK -1 rgb x 1 (9)
where K rgb is the intrinsic matrix of RGB camera. Then, the following equation is used to project the DSO depth map onto the thermal-infrared image to read its thermal intensity value:

x = M rgb_ir P rgb (x, d) (10)
such that the pixels of the RGB image can be associated to the corresponding pixel in the thermal-infrared image:

I rgb (x) ↔ I ir (x ) (11)
However, since the relative pose of P ir in P rgb coordinate system is defined up to scale, the transformation matrix M rgb_ir from P rgb in P ir is also defined up to scale s:

M rgb_ir = K ir R ir st ir (12)
Here, t ir is a translation vector corresponding to the position of the thermal infrared camera in the stereo rig coordinate system. Since t ir is defined up to an arbitrary scale (here t ir 2 = 1), it can be also defined up to any other scale factor s. Moreover, the 3D structure and thus the depth values provided by any monocular RGB SLAM are also defined up to scale, thus we cannot project the DSO point cloud to thermal-infrared camera images unless we can determine a common scale factor between the 3D structure system coordinate and the stereo rig system coordinate. For this process, we can fix the 3D structure scale factor to its default value and search for the fine the stereo rig scale factor s of that fits the best to the 3D structure, as depicted in Fig. 7. Fig. 8 shows projection result from DSO point cloud to thermal-infrared images using difference scale s in Eq. ( 12), varying the parameter s. In our method, we estimate the scale of the point cloud generated by a monocular RGB camera and project it to thermal-infrared images accurately.

Thermal values and RGB values have different modalities, and thus cannot be compared directly. We use instead the RGB and Thermal depth value to compare both images, since depth is a common modality. In our method, we generate a depth map from both RGB and thermal-infrared images using a multi view stereo (MVS) algorithm [START_REF] Okutomi | A multiple-baseline stereo[END_REF][START_REF] Pradeep | MonoFusion: Realtime 3D reconstruction of small scenes with a single web camera[END_REF]. A depth map is generated using several images near focus frames and the trajectory of the camera obtained from DSO. We generate a patch for each pixel, then get the score by comparing the patch and other frame patches using zero-mean normalized cross correlation (ZNCC).

d i = arg min d ∈D C(i, d) (13)
Eq. ( 14) describes the score of the ZNCC where C expresses the ZNCC score of the patch centered in pixel i with a certain depth value d.

C(i, d) = -

This score is calculated comparing a square window I p in depth image I and a square window I p in depth image I . I p (j) and I p (j) describe the intensity value on pixel j. Finally, I p and σ(I p ) refer to the means and the standard deviation of the window I p . Fig. 9 shows some results of depth maps generated from both RGB and thermal-infrared images.

The depth map generated from the RGB image sequence is successively translated to thermal-infrared camera coordinates by Eq. ( 12) by iterating on t. The translated RGB depth map and the thermal-infrared depth map are compared using the mutual information defined in [START_REF] Studholme | An overlap invariant entropy measure of 3D medical image alignment[END_REF][START_REF] Viola | Alignment by maximization of mutual information[END_REF]. This mutual information is often used to compare images of different modalities. In our case, this modality variety can be expressed as the difference of accuracy between the RGB and thermal camera depth map computation. Indeed, thermal-infrared depth maps are always significantly worse than the depth maps build from RGB images. The translation t leading to the In Eq. ( 15), I rgb is the patch converted by Eq. ( 11) using given scale s, and η describes the ratio of appear the RGB depth map on the thermal-infrared camera coordinate after converting. H(I) describes the appearance ratio of depth i on the depth image I and H(I, I ) describes the two dimensional appearance ratio of depth i on the depth image I and depth j on the depth image I .

H(I) = - i=0 p I (i) log p I (i) (16)
H(I, I ) = - i=0 j=0
p II (i, j) log p II (i, j)

Fig. 10 shows the variation of the mutual information scores with different scales s. The green curve is the average score for each scale. The optimal scale can be estimated uniquely as a global maximum from this the graph. This scale is a valuable information to correctly superimpose the temperature data on the point cloud.

Generating RGB-T Images from the Point cloud

At that stage, we have a point cloud computed by the RGB camera and the correct scale between this point cloud and the stereo rig referential. Thus, it is basically possible to project associate to each point a RGB value as well as a thermal value. In practice, the point cloud generated from DSO is sparse, so we sometimes have to interpolate the thermal information where the point cloud is not dense enough. Thus, the 3D point cloud is first back-projected on the targeted RGB image using the camera pose provided by DSO. Second, we generate triangle meshes from the projected point cloud using Delaunay triangle division2 [START_REF] Lee | Two algorithms for constructing a Delaunay triangulation[END_REF]. Then each triangle vertex is back projected on the thermal image using Eq. ( 10), in order to compute the right texture coordinates of each triangle in the thermal image. The final rendering consists in the superimposition of the dense thermal textured mesh back projected onto the corresponding RGB image. Fig. 11 shows an example of an RGB-T image build with this process.

2Fade2D Delaunay Triangulation http://www.geom.at/products/fade2d/

Experiments

Experimental Setups

In our experiments, we fixed a Flea3 in Point Grey monocular RGB camera and a PI640 Optris thermalinfrared camera into a hand-held stereo rig, as depicted in Fig. 12. This device can be easily manipulated to capture two videos of the scene.

Calibration Refinement

The calibration refinement process described in Section. 4 is evaluated as follows. We first calibrate the cameras with the non-optimized chessboard corners detected by OpenCV, then compute the reprojection error on each camera, i.e. the average distance from each detected corner in the image and the projected corned with the projection matrix. Second, we repeated this process with the refined chessboard corners. These reprojection errors are shown in Table 2. Even through the reprojection error of the thermal camera is still not as accurate as for the RGB camera, we can note a noticeable improvement, roughly by a factor of 2, in the reprojection error of the thermal camera calibration.

Thermal Maps and RGB-T Images

As detailed in Section 7, the final RGB-T images are computed by superimposing thermal information on the RGB-images. The thermal data is rendered with a colormap with red, green, blue gradation, where red indicates high temperature and blue indicates lower temperature. For our experiments, we generated thermal maps from both an outdoor sequence and an indoor sequence. Fig. 13 shows the resulting generated thermal maps.

In the outdoor sequence, the building equipments and the asphalt exposed to the sun light get a high temperature, and thus appear in red. The cars parked in the shadows of buildings have a lower temperature and are rendered in blue. In the indoor sequence, we can notice the high temperature of the displays in red, and low temperature of the beverage rendered in blue.

We also generated RGB-T images from various scenes. Fig. 14 shows the resulting RGB-T images and reference RGB images.

Evaluation the Estimated Scale

In this section, we describe the evaluation of our scale estimation method and its accuracy. Fig. 15 shows the environment we used to evaluate the scale. In this experiment, we reconstruct a calibration board of known length. Using the scale estimated by our method, we compared the estimated size of the calibration board with the actual size.

First, we reconstruct a thermal map as usual, as shown on the left of Fig. 15, then we reconstruct the calibration board in the right part of the figure. Second, the point cloud is reprojected to the image capturing the calibration board and we pick up the points on the corners of each cell of the calibration board. The points are very noisy, so we remove the outliers that are not on the plane of the calibration board by RANSAC and compute a estimated length L of each cell side:

L = t s (18)
We then estimate the average length of the cells, which is the length of the DSO point cloud scale. Using Eq. ( 18), we converted the scale to the actual size. We calculated the mean relative error against the actual size L using Eq. ( 19):

= L -L L (19)
Considering the randomness of RANSAC, we repeated the process 100 times. Table 3 describes the mean and the standard deviation of the relative error. We evaluated the indoor and outdoor scenes in two sequences.

Conclusion

In this paper, we visualize temperature more effectively by generating a thermal map and RGB-T images using 3D structures obtained from DSO. We further demonstrate the results of our method using indoor and outdoor scenes. In the proposed method, first, we calibrate an RGB and a thermal-infrared camera using a calibration board that can be detected by the thermal-infrared camera. Then, temperature information is superimposed onto the generated 3D structure using the extrinsic parameter between both cameras. At that time, we have to obtain the scale of the 3D point cloud. Thus, we estimate the scale using depth maps generated from MVS. Moreover, we generate RGB-T images that can superimpose temperature on RGB images where sufficient 3D points are not obtained from DSO using Delaunay triangulation in order to create triangle mesh.

We can generate thermal maps and RGB-T images by using the proposed method. We plan to expand our system to generate automatic alerts for abnormalities in electricity in plants based on variations in the temperature.

Figure 1 :

 1 Figure 1: Example of a generated thermal map and RGB-T image. The correspondence between RGB and thermal-infrared images is obtained using the 3D structure generated from DSO. The top row shows a reference view and a 3D reconstructed result, the middle row shows a 3D thermal model generated by the proposed method, and the bottom shows an RGB-T image and reference frames of an RGB and a thermal-infrared cameras.

Figure 2 :

 2 Figure 2: Overview of the proposed method.

Figure 3 :

 3 Figure 3: Blur on the thermal-infrared camera image. The top row shows images of a calibration board captured by both a thermal-infrared camera and an RGB camera. At the bottom row, a zoomed corner on a thermal-infrared image and an RGB image are compared. It is more difficult to detect a thermal-infrared image than an RGB image because of the blur.

Figure 4 :

 4 Figure 4: Estimation of distortion parameter. (Left) A set of points supposed to be aligned in the real scene and their least-square fitted line. (Right) Corresponding undistorted image.

Figure 5 :

 5 Figure 5: Initial detection and refining the points. Refined corner points should lie along straight lines in the left image. The center image shows the refinement of the points. Blue points are points detected by OpenCV and Red Points are the refined points, and the right image shows lines drawn connecting the corners.

Figure 6 :

 6 Figure 6: Constraints of cost function. The left figure describes the cross-ratio constraint, which is satisfied by Eq. (2). The right figure shows vanishing point of the calibration board.

Figure 7 :

 7 Figure 7: Variation of back-projection by different scale. The red circle represent a 3D object of the scene, with a certain arbitrary scale. The translation t between the two cameras is initially subject to t ir 2 = 1 can be scaled with the fine scale factor s to fit the 3D reconstruction from the first camera.

Figure 8 :

 8 Figure 8: Result of back-projection to thermalinfrared images using different scales. The top left is a reference RGB camera image. The other images are generated by back-projection of the DSO point cloud using different scales. The bottom left image shows the accurate scale.

  j∈I p I p (j) -I p I p (j) -I p σ(I p ) σ(I p )

Figure 9 :

 9 Figure 9: Generating a depth map using a multi-view stereo (MVS) algorithm. The top left shows a reference RGB camera image and the bottom left shows a reference thermal-infrared camera image. Right images show the result of MVS using each image.

Figure 10 :

 10 Figure 10: Variation of the MI score with scale. The green curve is the average score for each scale. We use some samples to estimate the scale and get the average of the score. The red vertical line is the place where the score is maximum, and the blue vertical line is the ground truth obtained by the experiment in Section 8.4.

Figure 11 :

 11 Figure 11: RGB-T image. The left shows the result of overlaying a 2D mesh on the image, and the right shows the resulting generated an RGB-T image.

Figure 12 :

 12 Figure 12: Camera rig. Left: thermal-infrared camera. Right: RGB camera.

Figure 13 :

 13 Figure 13: Thermal maps. Left: outdoor sequence, made out of 1330 pairs of RGB and thermal-infrared images. Right: indoor sequence with 1264 pairs of RGB and thermal-infrared images.

Figure 14 :

 14 Figure 14: RGB-T images from various scenes. This images are generated from 3 different sequences.

Figure 15 :

 15 Figure 15: Evaluation of the estimated scale. The reconstructed environment includes a calibration board. First, estimate the scale of the point cloud. Second, compare the estimated scale and the actual scale by using the side of the calibration board.

Table 3 :

 3 Mean and standard deviation of the relative error between the estimated scale and the ground truth.

Table 1 :

 1 Comparison of 3D reconstruction methods.

	method	density	cost	scale
	DSO

Table 2 :

 2 Calibration RMS in pixel.

		Initial Refined
	Thermal 1.4980 0.6311
	Stereo	1.2345 0.7594
	RGB	0.1723	-


Maurizio Gabbrielli
email: maurizio.gabbrielli@unibo.it

Saverio Giallorenzo
email: saverio.giallorenzo@gmail.com

Ivan Lanese
email: ivan.lanese@gmail.com

Stefano Pio Zingaro
email: stefanopio.zingaro@unibo.it

A Language-based Approach for Interoperability of IoT Platforms

The Internet of Things (IoT) promotes the communication among heterogeneous entities, from small sensors to Cloud systems. However, this is realized using a wide range of communication media and data protocols, usually incompatible with each other. Thus, IoT systems tend to grow as homogeneous isolated platforms, which hardly interact. To achieve a higher degree of interoperability among disparate IoT platforms, we propose a language-based approach for communication technology integration. We build on the Jolie programming language, which allows programmers to easily make the same logic work over disparate communication stacks in a declarative, dynamic way. Jolie currently supports the main technologies from Service-Oriented Computing, such as TCP/IP, Bluetooth, and RMI at transport level, and HTTP and SOAP at application level. As technical result, we integrate in Jolie the two most adopted protocols for IoT communication, i.e., CoAP and MQTT. In this paper, we report our experience and we present high-level concepts valuable both for the general implementation of interoperable systems and for the development of other language-based solutions.

INTRODUCTION

IoT systems are being developed for a wide range of applications and target areas [START_REF] Gubbi | Internet of Things (IoT): A vision, architectural elements, and future directions[END_REF], [START_REF] Atzori | The internet of things: A survey[END_REF], using a number of different technology stacks [START_REF] Al-Fuqaha | Internet of things: A survey on enabling technologies, protocols, and applications[END_REF]. Nevertheless, as reported, e.g., in [START_REF] Soursos | Towards the cross-domain interoperability of IoT platforms[END_REF], [START_REF] Gojmerac | Bridging IoT islands: the symbIoTe project[END_REF], IoT platforms frequently take the shape of vertical solutions (usually dubbed "IoT islands") that focus on a specific application domain and rely on a single communication technology stack. Such platforms provide little support for collaboration and integration. How to overcome this limitation is currently a hot topic, tackled also by ongoing EU projects, e.g., symbIoTe [START_REF] Gojmerac | Bridging IoT islands: the symbIoTe project[END_REF] and bIoTope [START_REF]The bIoTope project[END_REF].

The problem of integration involves many layers, spanning from link-layer communication technologies, such as ZigBee and WiFi, to application-layer protocols like HTTP, CoAP [START_REF] Bormann | CoAP website[END_REF], [START_REF] Shelby | The constrained application protocol (coap)[END_REF], and MQTT [START_REF]MQTT community[END_REF], [START_REF] Banks | MQTT Version 3.1.1[END_REF], reaching the top-most layers of data-format integration [START_REF] Milenkovic | A case for interoperable IoT sensor data and meta-data formats: The internet of things (ubiquity symposium)[END_REF].

In this work, we tackle the main issues of intercommunication among IoT islands, by focusing on integration at both the transport (TCP or UDP) and application level. The approach we propose is language based, that is, we aim at devising a programming language where different communication protocols can seamlessly coexist and interoperate. Thanks to proper abstractions provided by the language we propose, programmers can easily change the transport and application protocols used for a given communication, even at runtime. Notably, when the application protocol supports different representation formats (such as JSON, XML, etc.) of the message payload, as in the case of HTTP and CoAP, the language we propose can automatically marshal and un-marshal data as required.

Following our approach, most of the complexity of guaranteeing interoperability among protocols is managed by the programming language, and hidden from the programmer. This hidden complexity is particularly high when many technologies are involved. The problem is exacerbated when one has to replace the technology used for some specific interaction. The replacement may be either static, e.g., because of the deployment of new, heterogeneous devices in a pre-existing system, or dynamic, e.g., to support a changing topology of disparate mobile devices. These are scenarios where our language-based approach can make the difference with respect to other approaches.

As an illustrative example, let us consider a scenario where we want to integrate two islands of IoT devices, both collecting temperature data, but relying on different communication stacks, namely HTTP over TCP and CoAP over UDP. We want to program a collector that receives temperature measurements from both islands and uses them for further elaboration.

In the language we propose, the description of such a collector is divided into two parts: a behavior specifying the logic of the elaboration and a deployment describing in a declarative way how communication is performed.

The behavior could be of the shape: where Line 3 is a reception statement, expecting a temperature measurement on operation receiveTemperature (an operation is an abstraction for technology-specific concepts such as channels, resources, URLs, . . . ) and storing it in variable data.

Once we defined the logic of the collector, we need to specify on which technologies the communication happens; in the example above, how the collector accepts communications from devices. In our language this information is defined within ports. For instance, the port for receiving (denoted with keyword inputPort) HTTP measurements can be defined as follows:

interface TemperatureInterface { OneWay: receiveTemperature( string ) } inputPort CollectorPort1 { Location: "socket://collector.net:8000" Protocol: http Interfaces: TemperatureInterface } Port CollectorPort1 specifies that the collector expects inbound communications via Protocol http using a TCP/IP socket receiving at URL "collector.net" and on TCP port 8000. The port and the operation are linked by the definition of interface TemperatureInterface. The interface declares the operation receiveTemperature, including the type of expected data (string), as a OneWay operation, namely an asynchronous communication that does not require any reply from the collector.

Thanks to port CollectorPort1, the collector can receive data from the HTTP island. To integrate the second island, we just need to define an additional port, similar to CollectorPort1, except for using UDP/IP datagrams at the transport layer and CoAP at the application layer. Hence, the whole code of the collector becomes: The example above highlights how, using the proposed language abstractions, the programmer can write a unique behavior and exploit it to receive data sent over disparate technology stacks. Clearly, one can define different operations for different technologies, if the required elaborations differ. Our language supports both inbound and outbound communications, the latter declared with outputPorts, whose structure follows that of inputPorts. In addition, we let programs change the Location and Protocol of its outputPorts at runtime, enabling the selection of the appropriate technologies for each context.

Our contribution

To enable the programming of IoT integration in the above style, we do not start from scratch, but we leverage the work done in the area of Service-Oriented Architectures (SOAs) [START_REF] Erl | Soa: principles of service design[END_REF] and, in particular, we build on the Jolie programming language [START_REF] Montesi | JOLIE: a Java Orchestration Language Interpreter Engine[END_REF], [START_REF] Montesi | Composing services with JOLIE[END_REF], [START_REF]Jolie website[END_REF]. Indeed, the example above uses the Jolie syntax and abstraction mechanisms.

As mentioned, Jolie enforces a strict separation of concerns between behavior, describing the logic of the application, and deployment, describing the communication capabilities. The behavior is defined using the typical constructs of structured sequential programming, communication primitives, and operators to deal with concurrency (parallel composition and input choice).

Jolie communication primitives comprise two modalities of interaction. Outbound OneWay communications send a message asynchronously, while RequestResponse communications send a message and wait for a reply (they capture the well-known pattern of request-response interactions [START_REF] W3c | Transport message exchange pattern: Single-requestresponse[END_REF]). Dually, inbound OneWay communications wait for a message, while RequestResponse communications wait for a message and send back a reply.

A main feature of the Jolie language, and the reason why we base our approach on it, is that it allows one to switch among many communication media (via keyword Location) and data protocols (via keyword Protocol) in a simple, uniform way. Being born in the field of SOAs, Jolie supports the main communication media (TCP/IP sockets, Bluetooth L2CAP, Java RMI, and Unix local sockets) and data protocols (HTTP, JSON-RPC, XML-RPC, SOAP and their respective SSL versions) from this area.

We think that the ability to use different communication modalities in a uniform way and to easily switch between them is very useful in the area of IoT. However, to make this approach practical, we also need to support the main communication stacks used in the IoT setting. Indeed, the main technical contribution of the present paper is the introduction in Jolie of the support for two application protocols relevant in the IoT scenario, namely CoAP [START_REF] Shelby | The constrained application protocol (coap)[END_REF], [START_REF] Bormann | CoAP website[END_REF] and MQTT [START_REF] Banks | MQTT Version 3.1.1[END_REF], [START_REF]MQTT community[END_REF].

Even if Jolie provides support for easy integration of new protocols, the task is non trivial. Indeed, all the protocols currently integrated in Jolie support the same internal interface, based on two assumptions: i) the usage of underlying technologies that ensure reliable communications and ii) a point-to-point communication pattern. However, both these assumptions fall when considering the two IoT technologies we integrate:

• CoAP communications can be unreliable since they are based on UDP connectionless datagrams. CoAP provides options for reliable communications, however these are usually disabled in an IoT setting, since battery and bandwidth preservation is important. • MQTT communications are based on the publishsubscribe paradigm, which contrasts with Jolie pointto-point communication primitives. Hence, we need to define a mapping of the general abstractions of the Jolie language into the publish-subscribe paradigm, balancing two factors: i) preserving the simplicity of the point-to-point communication style and ii) capturing the typical flow of communications as programmed in a publish-subscribe style. An evident example of the challenges of our mapping is the implementation of request-response communications on top of publish-subscribe interactions. Remarkably, the mapping that we present in this work is general and could be used also in other contexts.

The flexibility provided by Jolie can be used to support and interconnect multiple IoT islands, as discussed above. Jolie supports also more advanced scenarios where the selection of the protocol to use changes according to internal or environmental conditions, such as available energy or quality of communication, but of course this requires some capability of switching the protocol also from the side of the Things, which may not be the case in many practical situations.

Indeed, in the rest of the paper, we omit to model IoT devices -like Arduino and other microcontrollers -that are at the edge of the network, since they are normally programmed by using low-level languages. In principle, these devices could be programmed by using Jolie-like languages, possibly extending them to provide those low-level abstractions needed by programmers to access the in-board sensors and actuators. However, the constraints on the hardware and the usually limited amount of energy available to edge devices require a low-footprint, lightweight execution environment. Although these requirements could be achievable also for a language like Jolie, this would require a strong engineering effort, which is not considered in this paper. We argue that this direction of work is not urgent, since currently developers tend to program very simple behaviors for edge devices [START_REF] Al-Fuqaha | Internet of things: A survey on enabling technologies, protocols, and applications[END_REF], which usually capture some data (e.g., through one of their sensors) and then send them to other devices (gateways, aggregators, servers). These other devices have more powerful hardware and less constraints on energy consumption, and can then implement the logic for the data processing. Hence, here we neglect the programming of edge devices and we focus on those devices that can both host the Jolie runtime and whose topological context can benefit from the flexibility offered by the language.

JOLIE FOR IOT

As mentioned above, Jolie currently supports some of the main technologies used in SOAs. However, only a limited amount of IoT devices uses the media and protocols already supported by Jolie. Indeed, protocols such as CoAP/REST [START_REF] Shelby | The constrained application protocol (coap)[END_REF], [START_REF] Bormann | CoAP website[END_REF] and MQTT [START_REF] Banks | MQTT Version 3.1.1[END_REF], [START_REF]MQTT community[END_REF], which are widely used in IoT scenarios, are not yet implemented in Jolie. Implementing these protocols is essential in order to allow Jolie programs to directly interact with the majority of IoT devices. However there are some challenges linked to the implementation of these technologies within Jolie:

• lossless vs. lossy protocols -In SOAs, machine-tomachine communication relies on lossless protocols, as there are no strict constraints on energy consumption or bandwidth, hence the number of message exchanges at the transport level needed for ensuring a message delivery is not critical. On the contrary, in IoT networks these constraints exist and are important, and the choice of the protocol needs to take them into account. Many protocols, and the CoAP application protocol in particular, rely on the UDP transport protocol -a connectionless protocol that gives no guarantee on the delivery of messages, but allows one to limit message exchanges and energy and bandwidth consumption. Since Jolie assumes lossless communications, the inclusion of connectionless protocols in the language requires careful handling to prevent misbehaviors; • point-to-point vs. publish-subscribe -In order to provide language constructs that do not depend on the chosen protocol, we need to find a uniform setting covering both point-to-point communications, such as the ones of HTTP and CoAP, and publish-subscribe communications typical of MQTT. Jolie already provides language constructs usable with many communication protocols, hence the less disruptive approach is to use the same constructs, which are designed for a point-to-point setting, also for MQTT. This requires to find for each point-to-point construct a corresponding effect in the publish-subscribe paradigm, such that typical programming patterns produce similar effects in both settings. In this way, one can program a unique behavior valid for both point-to-point and publish-subscribe scenarios.

We detail how we integrate CoAP/UDP and MQTT in the Jolie language respectively in sections 3 and 4. The Jolie language interpreter, including our extensions at version 1.0, is available at [START_REF] Gabbrielli | Jolie for IoT website[END_REF]. The integration of our extension into an official Jolie release is ongoing work.

SUPPORTING COAP IN JOLIE

The Constrained Application Protocol (CoAP) [START_REF] Bormann | CoAP website[END_REF], [START_REF] Shelby | The constrained application protocol (coap)[END_REF] is a specialized web transfer protocol for constrained scenarios where nodes have low power and networks are lossy. The goal of CoAP is to import the widely adopted model of REST architectures [START_REF] Fielding | Architectural styles and the design of networkbased software architectures[END_REF] into an IoT setting, that is, optimizing it for Machine-to-Machine applications. In particular, CoAP makes use of GET, PUT, POST, and DELETE methods like HTTP. Following the RFC [START_REF] Shelby | The constrained application protocol (coap)[END_REF], CoAP is implemented on top of the UDP transport protocol [START_REF] Postel | User datagram protocol[END_REF], with optional reliability. Indeed, CoAP provides two communication modalities: a reliable one, obtained by marking the message type as confirmable (CON), and an unreliable one, obtained by marking the message type as non confirmable (NON).

As an example, we consider a scenario with a controller, programmed in Jolie, that communicates with one of many thermostats in a home automation scenario. Thermostats are accessible at the generic address "coap://thermostat/##" where "##" is a two-digit number representing the identifier of a specific device. Thermostats accept two interactions: a GET request on URI "coap://thermostat/##/getTemperature", that returns the current temperature, and a POST request on URI "coap://thermostat/##/setTemperature", that sets the temperature of the HVAC system. We report and comment below the code of a possible Jolie controller. .messageCode = "GET" .alias = "/%!{id}/getTemperature", }; .osc.setTmp << { .contentFormat = "text/plain", .alias = "/%!{id}/setTemperature", .messageCode = "POST" } } Interfaces: ThermostatInterface } main { getTmp@Thermostat( { .id = "42" } )( temp ); if ( temp > 27 ){ setTmp@Device( 24 { .id = "42" } ) } else if ( temp < 15 ){ setTmp@Device( 22 { .id = "42" } ) } } Listing 2. Jolie controller communicating over CoAP/UDP.

Our scenario uses two CoAP resources:

"/getTemperature" and "/setTemperature". We model them in Jolie at lines 3-6 of Listing 2, by defining the interface ThermostatInterface, which includes a RequestResponse operation getTmp, representing resource "/getTemperature", and a OneWay operation setTmp, representing resource "/setTemperature". By default, we map operation names to resource names, hence in our example we would need resources named "/getTmp" and "/setTmp" respectively. However, as described below, one could override the default mapping, defining the coupling of protocol-specific concepts (here CoAP resources) and operations inside ports. In this way, programmers can define interactions at a high level with interfaces, while the grounding to the specific case is done in the deployment.

At lines 8-24 we define an outputPort to interact with the Thermostat. At line 9 we specify the Location of the thermostat. Recalling that the scheme of the resources of the thermostats is "coap://thermostat/##/...", we define the Location of the port using the UDP "datagram://" protocol, followed by the first part of the resource schema "thermostat" and the UDP port on which it accepts requests. Here we assume thermostats to use CoAP standard UDP port, which is "5683". Note that, in the Location, we do not define the address of a specific thermostat, e.g., "datagram://thermostat:5683/42". On the contrary, we just specify the generic address to access thermostats in the system, while the specific binding will be done at runtime, thanks to the .alias parameter of the coap protocol, described later on.

At line 10 we define coap to be the protocol used by the outputPort. At lines 11-21 we specify some parameters of the coap protocol -this matches the standard way in which Jolie defines parameters for Protocols in ports.

Here, we follow the methodology presented in [START_REF] Montesi | Process-aware web programming with Jolie[END_REF] for the implementation of the HTTP protocol in Jolieindeed CoAP adopts HTTP naming schema and resource interaction methods. In particular, we draw from [START_REF] Montesi | Process-aware web programming with Jolie[END_REF] the parameter prefix .osc, whose name is the acronym of "operation-specific configuration" and which is used for configuration parameters related to a specific operation.

In the example, we define .osc parameters for both operations getTmp and setTmp. At line 15 we specify that the CoAP method used is GET. At line 14 we define, using the .contentFormat parameter, that the encoding of the payload of the message in a binary format ("text/plain"). Other accepted values for the .contentFormat parameter are "json" and "xml". Marshalling and un-marshalling is automatic and transparent to the programmer. This feature is enabled by the structure of Jolie variables, which are always tree-shaped, hence they can easily be translated into representations based on that shape. At line 15, following the practice introduced in [START_REF] Montesi | Process-aware web programming with Jolie[END_REF], we specify that getTemp aliases a resource whose path concatenates a static part, given by the Location, and the instantiation of the template "/%!{id}/getTemperature" provided by protocol parameter .alias. The template is instantiated using values from the parameter of the operation invo-cation in the behavior, e.g., value 42 at line 27 1 . Hence, the interpretation of the declaration at line 15 is that, when invoking operation getTmp at runtime, the element id of the invocation will be removed from the payload and used to form the address of the requested resource. The aliasing for operation setTmp (line 20) is similar to that of getTmp, while the operation use method POST. Since here the .contentFormat parameter is omitted, the default "text/plain" is used.

To conclude, we briefly comment the runtime execution of the example, described in the behavior at lines 26-33. At line 27 the controller invokes operation getTmp. Being an outgoing RequestResponse, the invocation defines on which port to perform the request (Thermostat) and presents two pairs of round brackets: the first contains the data for the request, the second points to the variable that will store the received response. Recalling the aliasing defined at line 15, at line 27 we define the value of element id = 42, thus the URI of the resource invoked at runtime is "coap://thermostat/42/getTemperature". Notably, in the example we hard-coded the id of the device, however in a realistic setting the value of id would be retrieved from a variable. Once received, the response from thermostat 42 is assigned to variable temp. The example concludes with a conditional in which, if the temperature is above 27 (line 28), the thermostat is set to lower room temperature to 24 degrees, while, if the temperature lies below 15 degrees, the thermostat is set to raise the temperature to 22 degrees.

Dually to outputPorts, inputPorts allow the programmer to specify inbound communications. The parameters described above are valid also for inputPorts, with the only difference that messageType works only for RequestResponses, and specifies whether the communication of the reply is reliable or not. Note that, concerning the .alias parameter, the template is instantiated using the address of the incoming communication and the values are inserted among the elements of the payload.

Implementation of CoAP/UDP in Jolie

We end this section reporting the most relevant issues met during the implementation in Jolie of the CoAP/UDP stack. In Jolie the implementations of the supported application and transport protocols are independent. This enables the composition of any transport protocol with any application protocol. In particular, the implementation of UDP that we provide can also be used to support other protocols relying on UDP like MQTT-SN [START_REF] Hunkeler | MQTT-S-a publish/subscribe protocol for wireless sensor networks[END_REF]. For this reason, we separately present the integration in Jolie of UDP and of CoAP.

Concretely, the Jolie language is written in Java and provides proper abstract classes that represent application and transport protocols. Each protocol is obtained as an implementation of the corresponding abstract classes. Each implementation is a separated module which is loaded only if the protocol is used. This expedites the integration of new protocols in the language.

The implementation of UDP consists in a listener and a sender class, both based on the Netty framework [START_REF] Maurer | Netty in Action[END_REF]. Since the structure expected by Jolie and the one provided by Netty are similar, the integration of UDP is smooth.

The implementation of CoAP consists in a unique class, taking care of both encoding and decoding messages, and is based on nCoAP [START_REF] Kleine | nCoAP[END_REF].

We notice that CoAP supports request-response communications and, in particular, CoAP messages include fields i) to specify at which address the reply is expected and ii) to match a reply with a previous request. Hence, the implementation of RequestResponse communications in CoAP is sound also with a transport protocol which is not connection-oriented, such as UDP. This would be a problem for protocols that do not provide such a facility, such as HTTP, which is indeed not commonly used over UDP.

Notably, Jolie comes with a formal semantics (in terms of a process calculus) [START_REF] Guidi | SOCK: a calculus for service oriented computing[END_REF], which enables to rigorously reason on the behavior of Jolie programs. The semantics in [START_REF] Guidi | SOCK: a calculus for service oriented computing[END_REF] only considers reliable communications and needs to be extended to also cover the unreliable case. We do not report here on this topic, since it is not central for the purpose of this paper.

SUPPORTING MQTT IN JOLIE

Message Queue Telemetry Transport (MQTT) [START_REF]MQTT community[END_REF], [START_REF] Banks | MQTT Version 3.1.1[END_REF] is a publish/subscribe messaging application protocol built on top of the TCP transport protocol.

A typical publish/subscribe interaction pattern can be diagrammatically represented as in Fig. 1 where:

1) a Subscriber subscribes to topic (a) at some Broker; 2) a Publisher publishes a message to topic (a) at the same Broker; 3) the Broker forwards the message to topic (a) to the Subscriber.

Subscriber

Broker Publisher 1) Subscribe to (a)

2) Publish at (a)

3) Forward msg in (a)

Fig. 1. Typical publish/subscribe interaction pattern.

On top of the basic mechanism of publish/subscribe, MQTT defines three levels of quality of service (QoS) for the delivery of each message published by a publisher. QoS levels determine whether messages can be lost and/or duplicated. Concretely, QoS levels are as follows:

• At most once -the message can be lost, no duplication can occur. • At least once -delivery of the message is guaranteed, but duplication may occur. • Exactly once -delivery of the message is guaranteed and duplication cannot occur.

To present how we model the MQTT protocol in Jolie, we first detail the simpler case of OneWay communications in section 4.1. Then, we address the more complex case of RequestResponses in section 4.2. Both mappings are general, i.e., in principle they can be applied to implement one-way and request-response communication patterns on top of any publish/subscribe protocol. Remarks on the implementation are provided in section 4.3.

One-Ways in MQTT

We first consider the case of inbound communications and then the case of outbound communications.

To exemplify OneWay inbound communications, we use the example in Listing 3, which is a revision of the example in Listing 1 by omitting the HTTP and CoAP ports and by adding an MQTT inputPort. As expected, the program behavior and the structure of the inputPort are unchanged. Main novelties are:

• the used Location (line 6) has the prefix "socket://" (as seen in the HTTP port) since MQTT relies on TCP transport protocol; • the used Protocol (line 7) is mqtt;

• the .broker protocol parameter (line 8), which is compulsory, specifies the address of the Broker.

While the syntax and the effect of the communication primitive from the point of view of the programmer are the same as the ones in Listing 1, the actual message exchanges performed to obtain such an effect are different.

Beyond defining such message exchanges, we also need to decide how to identify the topic on which the message exchange is performed.

Regarding the message exchanges, from the point of view of the programmer, an inbound OneWay communication receives a datum from the communication partner. To obtain the same effect using the publish/subscribe paradigm, one has first to subscribe at the Broker to the chosen topic and then wait to receive a message on that topic, forwarded by the Broker. How topics are selected will be detailed later on. The execution of a reception on a OneWay operation comprises two actual communications: a subscription from the program to the Broker and a message delivery in the opposite direction. However, subscription to topics and the execution of a message reception are logically separated and can be done at different moments. Indeed, the subscription is performed when the Jolie program is launched for all operations present in MQTT inputPorts. This choice is more in line with the expected behavior of Jolie programs -and of Service-Oriented programs in general -where messages to operations, whose reception statements are not yet enabled, are stored until the actual execution of the reception. In Jolie, the compulsory parameter .broker is needed precisely to know the address at which the subscription needs to be performed. The address for the delivery of the actual message is the usual Location of the inputPort.

Regarding the selection of topics, similarly to what done for CoAP resources, in MQTT we default to mapping Jolie operations to topics, otherwise we use the .osc parameter .alias to enable loose coupling between operations and topics. We remark that .alias in inputPorts have a different behavior in MQTT with respect to HTTP and CoAP. In CoAP the name of the resource extracted from the received message is used to derive the correct instantiation of the .alias template. The values resulting from the match are then inserted among the elements of the payload before storing it in the target variable data. Instead, in MQTT, the .alias parameter is used to identify the topic for subscription. For example, in Listing 3, one could add the Protocol parameter .osc.receiveTemperature.alias = "temperature" to specify that the selected topic for operation receiveTemperature is "temperature". Note that, since there is no outgoing data, templates in MQTT inputPorts, such as "temperature" in the example, are constants (we require all such constants defined within the same inputPort to be distinct). Having only constant aliases is not a relevant limitation in the context of IoT, where topics are mostly statically fixed. Addressing this limitation without disrupting the uniformity of the Jolie programming model is not trivial and is left as future work.

To conclude the mapping of OneWay operations in MQTT, we consider here the case of outbound operations, exemplified in Listing 4. Outgoing OneWay operations simply cause the publication of the value passed as the parameter of the invocation (line 17) at the Broker. The address of the Broker is defined by the Location (line 6) of the outputPort Broker. The topic is derived from the name of the operation and the parameter of the invocation, using protocol parameter .alias as usual. Being an MQTT publication, we specify the .QoS protocol parameter (line 10), which selects the QoS level "Exactly once" for the operation setTmp.

Request-Responses in MQTT

To discuss RequestResponse communications, let us consider the example in Listing 2, revised in Listing 5 by replacing the CoAP protocol with MQTT. We omit OneWay communications and concentrate on the outbound RequestResponse. Afterwards, we will also discuss the dual inbound RequestResponse.

interface ThermostatInterface { RequestResponse: getTmp( TmpType )( int ) } outputPort Broker { Location: "socket://iot.eclipse.org:1883" Protocol: mqtt { .osc.getTmp << { .format = "raw", .QoS = 2, // exactly once QoS .alias = "%!{id}/getTemperature", .aliasResponse = "%!{id}/getTempReply", } } Interfaces: ThermostatInterface } main { getTmp@Broker( { .id = "42" } )( temp ) } Listing 5. Jolie controller communicating over MQTT.

Syntactically, the main novelty with respect to the outputPort in Listing 4 is the addition of Protocol parameter .aliasResponse. This parameter specifies the name of the topic where the receiver will publish its response.

From the point of view of the programmer, an outbound RequestResponse is composed of an outgoing communication followed by an inbound reply. The outgoing communication is implemented using the approach already seen for OneWay communications, i.e., using the .alias Protocol parameter to identify the topic. Then, one has the issue of relating the outgoing request with its reply. Many standard point-to-point communication technologies, such as HTTP/TCP and the already discussed CoAP/UDP, support request-response communications by defining means to link a given outgoing request to its reply. MQTT does not provide dedicated means to do such a linking, hence we exploit the content and the topic of messages to this end. We identify the topic for the reply with the .aliasResponse Protocol parameter. Like for .alias parameters, the template of the .aliasResponse parameter is instantiated using the content of the message sent in the behavior. For example, in Listing 5, we use .id in line 19 to obtain "42/getTemperature" and "42/getTempReply", respectively the publication and reply topics.

We can now describe the pattern of interactions that we use to implement the outgoing RequestResponse communication at line 19 in Listing 5. As a reference, the pattern of interactions is depicted in the left part of Fig. 2. We will describe the right part later on, after having introduced inbound request-response communications.

Controller

Broker Thermostat 1) Subscribe to "42/getTemperature" 2) Subscribe to "42/getTempReply" 3) Publish to "42/getTemperature" 4) Forward msg in "42/getTemperature" 5) Publish to "42/getTempReply" 6) Forward msg in "42/getTempReply" Fig. 2. Interaction in the home automation example in MQTT.

First, the controller subscribes to the reply topic "42/getTempReply" at the Broker. Then, the controller sends to the Broker the request message on topic "42/getTemperature"; the payload of the message contains the reply topic "42/getTempReply". The execution of the RequestResponse terminates when the Broker for-wards the reply received on topic "42/getTempReply" to the controller.

Differently from inbound OneWay communications, here we do not subscribe to the reply topic when the program is launched. Indeed, it would be useless since no relevant message can arrive on this topic before the controller sends its message to the Broker and, by anticipating the subscription, it would complicate the usage of runtime information in templates.

To exemplify inbound RequestResponse communications, we assume that the thermostat in our example is programmed in Jolie. We report its code in Listing 6.

interface ThermostatInterface { RequestResponse: getTmp( TmpType )( TmpType ) } inputPort Thermostat { Location: "socket://localhost:9000" Protocol: mqtt { .broker = "socket://iot.eclipse.org:1883"; .osc.getTmp << { .format = "raw", .alias = "42/getTemperature" } } Interfaces: ThermostatInterface } main { getTmp( temp )( temp ){ // retrieves temperature and stores // it within the root of variable temp } } Listing 6. Jolie thermostat communicating over MQTT.

At line 11 in Listing 6, the .alias parameter "42/getTemperature" is static, as usual for inputPorts.

When the thermostat program is launched, it subscribes to topic "42/getTemperature". When a message on this topic arrives, the reply topic, e.g., "42/getTempReply", is extracted and the rest of the payload (empty in this case) is passed to the behavior. The body of the RequestResponse (lines [START_REF] Postel | User datagram protocol[END_REF][START_REF] Montesi | Process-aware web programming with Jolie[END_REF] is executed to compute the return value. Finally, the return value is published on the reply topic "42/getTempReply".

We now summarize the exchange between the controller and the thermostat (left part of Fig. 2):

1) when the thermostat is started, it subscribes to topic "42/getTemperature" at the Broker; 2) when the outgoing RequestResponse is executed, the controller subscribes to topic "42/getTempReply" at the Broker; 3) the controller publishes the request message to topic "42/getTemperature"; 4) the Broker forwards the message in topic "42/getTemperature" to the thermostat; 5) the thermostat computes the response and publishes it at topic "42/getTempReply"; 6) the Broker forwards the message in topic "42/getTempReply" to the controller.

We remark that RequestResponse operations are meant to be one-to-one communications. To ensure this in a publish/subscribe setting while using the approach above, one has to ensure that no other participant subscribes to the selected topics, which essentially act as namespaces.

Implementation of MQTT in Jolie.

The implementation of MQTT in Jolie required the creation of two main classes: the actual MQTT protocol and a generic publish/subscribe meta-channel that bridges between the end-to-end style of Jolie communications and publish/subscribe interactions. Furthermore, we had to update the Jolie class for channel creation so that it could choose between the standard end-to-end media and the new publish/subscribe meta-channel.

The MQTT protocol class both encodes and decodes messages and implements the QoS policies of the MQTT standard. Concretely, as for CoAP, we based the implementation of MQTT on Netty [START_REF] Maurer | Netty in Action[END_REF]. The main difficulty in the implementation of the protocol is the definition of the message patterns needed to implement OneWay and RequestResponse communications, which have been described above. Beyond being invoked when operations are executed, the MQTT class is also invoked when the program is started, to perform port initialization. In particular, this is when subscriptions to topics identified in inputPorts are performed (along with the related connections to the Brokers).

In addition to the MQTT protocol, we also implemented a generic publish/subscribe meta-channel. Indeed, since Jolie is based on an end-to-end communication pattern, it assumes that the caller requires the creation of a connection to the server, which waits for inbound requests. For this reason, given a certain medium, inputPorts and outputPorts use a medium-specific implementation of, respectively, a listener class and a sender class.

This pattern, separating listeners from senders, does not apply to publish/subscribe protocols, where both the subscriber and the publisher need to establish a connection with the broker. For this reason we use the publish/subscribe meta-channel to bridge between the two styles. On the one hand, it implements the interfaces of listeners and senders as required by Jolie. On the other hand, it relies on the pre-existing Jolie sender classes (TCP socket in the case of MQTT) to create the connection to the broker.

RELATED WORK

In the literature there are many proposals for platforms, middlewares, smart gateways, and general systems, all aimed at solving the interoperability problem arising from the current "babel" of IoT technologies (protocols, formats, and languages). Without any claim of being complete, here we mention a few notable examples which are somehow related to our current research.

Recently the W3C started the Web of Things (WoT) Working Group [START_REF]Web of things[END_REF]. The aim of WoT is to define a standard stack of layered technologies, as well as software architectural styles and programming patterns, to uniform and simplify the creation of IoT applications. In this context, the W3C is working on a WoT Architecture [START_REF]Web of things architecture[END_REF]. The main concept of the architecture is the notion of "servient", a virtual entity that represents a physical IoT device. Servients provide technology-independent, standard APIs that developers can use to transparently operate in heterogeneous environments. Remarkably, both the WoT proposal and ours concern high-level abstractions for lowlevel access to devices provided via, e.g., HTTP, CoAP, and MQTT. However, while we propose a dedicated language, they provide API specifications. More in general, there are many proposals for the integration of WoT and IoT. For example [START_REF] Dominique | A web of things application architectureintegrating the real-world into the web[END_REF] and [START_REF] Corredor | A lightweight web of things open platform to facilitate context data management and personalized healthcare services creation[END_REF] define general platforms covering different layers of IoT, including an accessibility layer which integrates concepts like smart gateways and proxies to facilitate the connection of (smart) Things into the Internet infrastructure, using architectural principles based on REST. Smart gateways and proxies are used in several industrial proposals to facilitate the development of applications. Common denominator of some of these proposals, e.g., [START_REF]Smartthings[END_REF], [START_REF]Meshlium[END_REF], [START_REF]Thinking things[END_REF], is the abstraction of low-level functionalities provided by embedded devices (e.g., connectivity and communication over low-level protocols like ZigBee, Z-Wave, Wi/IP/UPnP, etc.). Smart gateways are used also to translate (or integrate) CoAP into HTTP [START_REF] Sulaeman | Performance evaluation of HTTP-CoAP proxy for wireless sensor and actuator networks[END_REF], [START_REF] Ludovici | A proxy design to leverage the interconnection of CoAP wireless sensor networks with web applications[END_REF], [START_REF] Mingozzi | CoAP proxy virtualization for the Web of Things[END_REF] and to integrate both CoAP and MQTT by means of specific middlewares [START_REF] Thangavel | Performance evaluation of MQTT and CoAP via a common middleware[END_REF]. Eclipse IoT [START_REF]The Eclipse for IoT Project[END_REF] is an IoT integration framework proposed by the Eclipse IoT Working Group. Aim of Eclipse IoT is to build an open IoT stack for Java, including the support for deviceto-device and device-to-server protocols, as well as the provision of protocols, frameworks, and services for device management. There exist several European projects, notably INTER-IoT [START_REF] Ganzha | Semantic technologies for the IoT -an inter-IoT perspective[END_REF] and symbIoTe [START_REF] Gojmerac | Bridging IoT islands: the symbIoTe project[END_REF], that address the issue of interoperability in IoT and have produced several concrete proposals. Finally, a work close to ours is [START_REF] Zhiliang | A SOA based IoT communication middleware[END_REF], where a middleware converts IoT heterogeneous networks into a single homogeneous network.

Although related to our aim in this paper, the cited proposals tackle the problem of IoT integration from a framework perspective: they provide chains of tools, each addressing a specific level of the integration stack. Differently, we extend a language specifically tailored for system integration and advanced flow manipulation, Jolie, to support integration of IoT devices. This offers a single linguistic domain to seamlessly integrate disparate low-level IoT devices and intermediate nodes (collectors, aggregators, gateways). Moreover, Jolie is already successfully used for building Cloud-based, microservice solutions [START_REF] Gabbrielli | Self-reconfiguring microservices[END_REF], [START_REF] Callegati | Insider threats in emerging mobility-as-a-service scenarios[END_REF]. This makes the language useful also for assembling advanced architectures for IoT, e.g., to handle real-time streaming and processing of data from many devices. The benefit, here, is that, while solutions based on frameworks require dedicated proficiencies on each of the included tools, Jolie programmers can directly work at any level of the IoT stack, without the need to acquire specific knowledge on the tools in a given framework.

To conclude our revision of related work, we narrow our focus on language-based integration solutions for IoT. The work most related to ours is SensorML [START_REF]The sensorML project[END_REF]. SensorML, abbreviation of Sensor Model Language, is a modeling language for the description of sensors and, more in general, of measurement processes. Some features modeled by the language are: discovery and geolocalization of sensors, processing of sensor observations, and functionalities to program sensors and to subscribe to sensor events. While some traits of SensorML are common to our proposal, the scopes of the two languages sensibly differ. Indeed, while Jolie is a high-level language for programming generic architectures (spanning from cloudbased microservices to low-level IoT integrators), Sen-sorML just models IoT devices, their discovery, and the processing of sensor observations.

DISCUSSION AND CONCLUSION

In this paper, we proposed a language-based approach for the integration of disparate IoT platforms. We built our treatment on the Jolie programming language. This first result is an initial step towards a more comprehensive solution for IoT ecosystem integration and management. Concretely, we included in Jolie the support for two of the most widely used IoT protocols. The inclusion enables Jolie programmers to interact with the majority of present IoT devices. Summarizing our results: i) we included in Jolie the CoAP application protocol, also extending the Jolie language to support the UDP transport protocol, ii) we added the support for the MQTT protocol and, in doing so, iii) we tackled the challenging problem of mapping the renowned pattern of request-responses (typical of HTTP and other widely used protocols) into the publish/subscribe message pattern of MQTT. The mapping abstracts from peculiarities of MQTT and is applicable to any publish/subscribe protocol.

Regarding future work, we are currently investigating the integration in Jolie of more IoT protocols [START_REF] Al-Fuqaha | Internet of things: A survey on enabling technologies, protocols, and applications[END_REF], in order to extend the usability of the language in the IoT setting.

Another interesting direction comes from the inclusion of publish/subscribe protocols in Jolie. Indeed, the publish/subscribe pattern is renowned for enabling high scalability of networks, as well as supporting flexible and highly dynamic network topologies [START_REF] Eugster | The many faces of publish/subscribe[END_REF]. Our intuition is that, besides efficiency and scalability, publish/subscribe architectures can achieve a higher degree of reliability if programmed using the Jolie language.

Another interesting direction for future developments is studying how Jolie can support the testing of IoT technologies, e.g., to test how different protocol stacks perform over a given IoT topology. Thanks to the simplicity of changing the combination of the used protocols (application and transport), experimenters can quickly test many configurations, also enjoying a more reliable platform to compare them. Indeed, usually even changing one of the protocols in the configured stack would require an almost complete rewrite of the logic of network components. Contrarily, in Jolie, this change just requires an update of the deployment part of programs, leaving the logic unaffected. Moreover, such an update could even be done programmatically, making the practice of repeated experimenting on IoT networks easier and more standardized.

Finally, as future work, we also consider the possibility of developing a light-weight version of the language, to be used on low-power IoT devices. Indeed, in this paper, we assumed that these devices are programmed with low-level languages, since they can support only a very constrained execution environment. Clearly, letting programmers develop all the components of an IoT network in the same language would not only ease its implementation but also testability, deployment, and maintenance. However, achieving such a result would require a very challenging engineering endeavor.

type

  TmpType: void { .id: string } | int { .id: string } interface ThermostatInterface { RequestResponse: getTmp( TmpType )( int ) OneWay: setTmp( TmpType ) } outputPort Thermostat { Location: "datagram://thermostat:5683" Protocol: coap { .osc.getTmp << {

interface

  ThermostatInterface { OneWay: setTmp( TmpType ) } outputPort Broker { Location: "socket://iot.eclipse.org:1883" Protocol: mqtt { .osc.setTmp << { .format = "raw", .QoS = 2, // exactly once QoS .alias = "%!{id}/setTemperature" } } Interfaces: ThermostatInterface } main { setTmp@Broker( 24 { .id = "42" } ) } Listing 4. Example of outgoing MQTT OneWay communication.

  Listing 3. Code of the Collector Example, revised for MQTT.

	interface TemperatureInterface {
	OneWay: receiveTemperature( string )
	}
	inputPort CollectorPort3 {
	Location: "socket://localhost:8050"
	Protocol: mqtt {
	.broker = "socket://iot.eclipse.org:1883"
	}
	Interfaces: TemperatureInterface
	}
	main {
	receiveTemperature( data )
	}

In Jolie the dot . defines path traversals inside trees. Hence, the notation {.id = 42} indicates a tree with an empty root and a subnode called id, whose value is 42.

Acknowledgments

We thank Marco Di Felice, Luca Bedogni, and Federico Montori for useful suggestions and comments.