Wenbin Zou
email: wzou@szu.edu.cn

Di Wu

Shishun Tian
email: stian@szu.edu.cn

Canqun Xiang
email: xiangcanqun2018@email.szu.edu.cn

Xia Li
email: lixia@szu.edu.cn

Lu Zhang

End-to-end 6DoF Pose Estimation from Monocular RGB Images

.

I. INTRODUCTION

The increase in consumer demand for safer vehicles greatly promote the development of the advanced driver-assistance systems (ADASs) and autonomous driving. Recently, visionbased systems [START_REF] Bećirbašić | Videoprocessing platform for semi-autonomous driving over 5g networks[END_REF] have been drawn extensive attention in autonomous driving and ADASs due to their great potential in roadway-environment understanding [START_REF] Shubham | An ai-based real-time roadway-environment perception for autonomous driving[END_REF], such as traffic light detection [START_REF] Vitas | Traffic light detection in autonomous driving systems[END_REF], road semantic segmentation [START_REF] Kim | Road semantic segmentation oriented dataset for autonomous driving[END_REF] etc., of which one crucial component is to detect, estimate and reconstruct the 3D shape of vehicles directly from the captured RGB video cf. Fig. 1.

The 2D object detection has gained significant improvement in recent years thanks to the development of deep learning, while the estimation of object shape and pose in 3D remains a challenging problem. The current state-of-the-art RGB-based 6DoF pose estimation methods [START_REF] Sundermeyer | Implicit 3d orientation learning for 6d object detection from rgb images[END_REF], [START_REF] Do | Real-time monocular object instance 6d pose estimation[END_REF], [START_REF] Kehl | Ssd-6d: Making rgb-based 3d detection and 6d pose estimation great again[END_REF] are two-staged: the first stage is to detect the object with 3D rotation via a trained network, the second stage is to estimate the full 3D translation Fig. 1: 6D-VNet is trained end-to-end to estimate vehicles' six degree of freedom poses from a single monocular image. The output will precisely estimates the vehicle's voxels in 3D occupancy.

via projective distance estimation. The aforementioned twostaged systems primarily focus on the industry-relevant binpicking tasks. Typically, a robot needs to grasp a single arbitrary instance of the required object, e.g. a component such as a bolt or nut, and operate with it. In such scenario, the surface alignment in the Z dimension, i.e. the optical axis of the camera, is less important than the alignment in the X and Y dimensions. Such industrial setting requires accurate estimation of rotation, whereas the translation tolerance can be relaxed. However, in autonomous driving, translation distance of traffic participants along longitudinal axis varies significantly. The translation estimation is thus more challenging and the estimation of vehicle's translation is more critical than that of orientation.

Traditional methods usually leave the translational estimation as a separate procedure after the object class prediction and rotation estimation by using a geometric projection method. However, the geometric projection method assumes that: (i) the object centre in 3D will be projected to the object bounding box centre in the 2D image; (ii) the predicted object class and rotation vector is correctly estimated. Therefore, by using geometric projection as a post-processing step, the error from object class estimation and rotation regression will be aggregated in the following projective distance estimation. Fig. 2: The core component of the proposed network: following the detection head in 2D space, the 6D-VNet regresses the 6DoF rotation and translation vector simultaneously in 3D space.

To accommodate the requirement for accurate translation estimation in autonomous driving, we propose a framework, termed 6D-VNet, aiming at regressing the vehicle's rotation and translation simultaneously cf. Fig. 2. 6D-VNet streamlines the vehicle's 6DoF via the intermediate outputs from the Region Proposal Network (RPN) [START_REF] Girshick | Fast r-cnn[END_REF]. The detection part of the network is the canonical 2D object detection network (Mask R-CNN). The 6DoF estimation part of the network takes the intermediate output from the detection head. The challenging aspect of learning 6DoF vehicle pose is to design a loss function which is able to learn both rotation and translation. The model learns a complementary representation when supervised by both translation and orientation signals. Moreover, traffic participants exert mutual influence among their neighbours. Therefore, we introduce a weighted nonlocal block, which is a modified version of [START_REF] Wang | Non-local neural networks[END_REF], to capture the collective information between traffic participants with interpretable self-attention map.

Specifically, the network is trained end-to-end by joint losses designed with solid geometric ground. Experimental results show that the proposed method outperforms the stateof-the-art two-staged systems. We list our contributions as follows:

• To our best knowledge, this is the first work which successfully regresses the rotation and translation simultaneously for deep learning-based object 6DoF pose estimation. And we show the effectiveness of the translation head inclusion into end-to-end training scheme (Sec. III-A). A Preliminary version of this manuscript was published previously [START_REF] Wu | 6d-vnet: End-toend 6-dof vehicle pose estimation from monocular rgb images[END_REF]. Since then, we have conducted experiments on the Pascal3D+ dataset and extensive ablation studies.

II. RELATED WORK

Monocular-based 3D object detection were helped by the early work on face detection to popularise bounding box object detection. Later, PASCAL VOC [START_REF] Everingham | The pascal visual object classes challenge: A retrospective[END_REF], MS-COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF], [START_REF] Aslan | Semantic food segmentation for automatic dietary monitoring[END_REF] datasets pushed the detection towards a more diverse, challenging task. A tracking and estimation integrated model is proposed in [START_REF] Zhao | Learning a tracking and estimation integrated graphical model for human pose tracking[END_REF] to determine spatial configuration of body parts in each frame. 3D convolutional neural network is presented in [START_REF] Ge | Real-time 3d hand pose estimation with 3d convolutional neural networks[END_REF] for real-time 3D hand pose estimation, and more relevant works are given in [START_REF] Patacchiola | Head pose estimation in the wild using convolutional neural networks and adaptive gradient methods[END_REF] and [START_REF] Bao | Tiny hand gesture recognition without localization via a deep convolutional network[END_REF], [START_REF] Al-Hammadi | Hand gesture recognition using 3d-cnn model[END_REF]. 3D head pose estimation [START_REF] Chen | Cumulative attribute space regression for head pose estimation and color constancy[END_REF], [START_REF] Li | A survey on 3d hand pose estimation: Cameras, methods, and datasets[END_REF] is also a challenge. 3D head pose and facial actions in monocular video sequences that can be provided by low quality cameras are initialized and tracked by a 3-D pose estimator and 2-D face detector [START_REF] Dornaika | Three-dimensional face pose detection and tracking using monocular videos: Tool and application[END_REF]. Recognizing the visual focus of attention of meeting participants based on their head pose is addressed in [START_REF] Ba | Recognizing visual focus of attention from head pose in natural meetings[END_REF] using a Gaussian mixture model. Studies of human pose can be referred from [START_REF] Duc | Human detection from images and videos: A survey[END_REF]. KITTI dataset [START_REF] Geiger | Are we ready for autonomous driving? the kitti vision benchmark suite[END_REF], [START_REF] Fan | Environment detection of 3d lidar by using neural networks[END_REF] propelled the research for traffic participants under autonomous driving scenario. However, the 3D object detection task in the KITTI dataset primarily focuses on using point clouds data from Velodyne laser scanner, which is an expensive apparatus. In addition, the KITTI 3D object detection task only has half degree of freedom for rotation overlooking vehicle's heading direction. Camera pose estimation is the problem of determining the position and orientation of a calibrated camera, which is to infer where you are and is key to the applications of mobile robotics, navigation and augmented reality, where localization is crucial for performance. [START_REF] Kendall | Posenet: A convolutional network for real-time 6-dof camera relocalization[END_REF] propose a robust and real-time monocular 6DoF relocalization framework, called PoseNet. It trains a convolutional neural network to regress the 6DoF camera pose from a single RGB image in an end-to-end manner with no requirement of additional engineering or graph optimisation. It is robust to difficult lighting, motion blur and unknown camera intrinsics, where point based SIFT registration fails. However, it was trained using a naive loss function, with hyper-parameters which require expensive tuning. To address this issue, a more fundamental theoretical treatment is given in [START_REF] Kendall | Geometric loss functions for camera pose regression with deep learning[END_REF] by exploring a number of loss functions based on geometry and scene re-projection error. [START_REF] Wang | Dels-3d: Deep localization and segmentation with a 3d semantic map[END_REF] proposes a unified framework to tackle self-localisation and camera pose estimation simultaneously. Instead of using a single RGB image, it integrates the signals from multiple sensors to achieve high efficiency and robustness. The problem of camera pose estimation is egocentric, in other words, a single vector of 6 dimensions will suffice to relocalise the camera pose. 6DoF object detection [START_REF] Sundermeyer | Implicit 3d orientation learning for 6d object detection from rgb images[END_REF] is also essential for robotic manipulation [START_REF] Lee | A hardware architecture of face detection for human-robot interaction and its implementation[END_REF] and augmented reality applications [START_REF] Cuevas | Moving object detection for real-time augmented reality applications in a gpgpu[END_REF]. The BOP benchmark [START_REF] Hodan | Bop: Benchmark for 6d object pose estimation[END_REF] consists of eight datasets in a unified format that cover different practical scenarios and shows that the methods based on point-pair features currently outperform the methods based on template matching, learningbased and 3D local features. Encouraging results have been shown in recent researches [START_REF] Sundermeyer | Implicit 3d orientation learning for 6d object detection from rgb images[END_REF], [START_REF] Do | Real-time monocular object instance 6d pose estimation[END_REF], [START_REF] Kehl | Ssd-6d: Making rgb-based 3d detection and 6d pose estimation great again[END_REF] which either RGB or RGB-D images are used to detect 3D model instances and estimate their 6DoF poses. Especially, [START_REF] Sundermeyer | Implicit 3d orientation learning for 6d object detection from rgb images[END_REF] proposes a two-staged 6DoF object detection pipeline: firstly, a Single Shot Multibox Detector (SSD) [START_REF] Liu | Ssd: Single shot multibox detector[END_REF] is applied to provide object bounding boxes and identifiers. Then, an Augmented Autoencoder (AAE) is applied to estimate the object rotation using a Domain Randomisation [START_REF] Tobin | Domain randomization for transferring deep neural networks from simulation to the real world[END_REF] Strategy. However, the aforementioned methods focuses on the industry-relevant objects with neither significant texture nor discriminative colour or reflectance properties. Moreover, objects of interests are lying on a uniform ground plane (e.g., in T-LESS dataset [START_REF] Hodan | T-less: An rgb-d dataset for 6d pose estimation of textureless objects[END_REF], the range of object distance is from 650mm to 940mm). Hence, the tolerance of rotation needs to stay low, whereas translation can be relaxed.

III. MODEL

6D-VNet is conceptually intuitive and structurally hereditary: Faster R-CNN has two outputs for each candidate object, a class label and a bounding-box offset, Mask R-CNN adds a third branch that outputs the object mask. Likewise, 6D-VNet streamlines the objects 6DoF prediction via the intermediate outputs from the Region Proposal Network (RPN). However, in order not to break already learnt functionality of the pretrained network, careful design choices must be customised for the end-to-end training. Next, we present the overall architecture in Sec. III-A. Particularly, we introduce the end-toend training paradigm with translation estimation integration which greatly outperforms other two staged frameworks in terms of translation estimation accuracy. The design choices for the joint losses are presented in Sec. III-B. Lastly we show the spatial relationship between traffic participants can be incorporated via a modified weighted non-local block in Sec. III-C.

A. Network Architecture

6D-VNet is built upon the canonical object detection network as shown in Fig. 3. The system is a two-staged network which is trained end-to-end to estimate the 6DoF pose information for object of interest. The first stage of the network is a typical 2D object detection network (Mask R-CNN). The second stage of the network is the customised heads to estimate the object 6DoF pose information.

The 6DoF pose estimation branch is the main novelty of the model and is split into two parts: the first part only takes RoIAlign [START_REF] He | Mask r-cnn[END_REF] from each candidate box if the candidate is of the vehicle class and performs sub-class categorisation and rotation estimation. Since in-plane rotation is unique for a given vehicle class, all vehicles share similar rotational features for the same yaw, pitch, and roll angles. Therefore, the fixed-size visual cue from RoIAlign layer is sufficient for estimating the candidate sub-category and rotation.

The second part takes both RoIAlign feature and bounding box information (in world unit as described in Sec. III-B) into consideration via a concatenation operation to estimate the 3 dimensional translational vector. To our knowledge, this novel formulation is the first of its kind to regress the translational vector directly. The joint feature combination scheme implicitly encodes the object class and rotation information via the concatenation operation (⊕ in Fig. 3). The translation regression head functions in synergy when it is combined with the joint loss from sub-category classification and quaternion regression. We show in the experiment that our novel formulation for translation regression produces much more accurate position estimation comparing to the methods that treat the translation estimation as a post-processing step. This accurate estimation of translational vector is particularly crucial for the applications where the distance of the objects are of primary importance (e.g., in the autonomous driving scenario).

B. Joint Losses

We minimise the following loss L to train our network in an end-to-end fashion: L = L det + L inst , where L det denotes the multi-task loss as in a canonical detection network:

L det = L cls + L box + L mask .
The classification loss L cls , 2D bounding box loss L box and 2D mask loss L mask are identical as those defined in [START_REF] He | Mask r-cnn[END_REF]. In order to accelerate the network training and keep the functionality of the pre-trained multi-task module (e.g., mask head for instance segmentation), we can freeze these heads and their corresponding child nodes, i.e., the convolutional backbone and set the L det to zero during back propagation phase. L inst denotes the individual instance loss for 6DoF estimation with subclass categorisation. Specifically, it is defined as a triple-loss:

L inst = λ sub cls L sub cls + λ rot L rot + λ trans L trans ,
where λ sub cls , λ rot , λ trans are hyper-parameters used to balance their corresponding loss. Next we explain the design choices for the above triple losses. Sub-category classification loss L sub cls . Sub-category denotes the finer class of the vehicle corpus: e.g., Audi-A6, BMW-530, Benze-ML500, etc. In order to balance the rare cases for infrequently appearing cars in the training images, weighted cross entropy is used for sub-category classification loss. Rotation loss L rot . There are generally three representations for providing orientation information: Euler angles, SO(3) rotation matrices and Quaternions. Euler angles are easily understandable and interpretable parametrisation of 3D rotation. However, there are two issues when directly regressing the Euler angles: (1) non-injectivity: the same angle could be represented by multiple values due to the wrapping around 2π radians, which make the regression a non uni-modal task;

(2) Gimbal lock: possible loss of one degree of freedom does not make Euler angles invalid but makes them unsuited for practical applications. Given 3D models of the objects, one way to work around the problem is to rotate each view at fixed intervals to cover the whole SO(3) and then find the nearest neighbour [START_REF] Sundermeyer | Implicit 3d orientation learning for 6d object detection from rgb images[END_REF] or closest viewpoint [START_REF] Kehl | Ssd-6d: Making rgb-based 3d detection and 6d pose estimation great again[END_REF], which treat the rotation estimation problem as a classification problem. But this requires a complete CAD model of the object and a discretisation step of orientation angles. To estimate rotation matrix directly, [START_REF] Do | Real-time monocular object instance 6d pose estimation[END_REF] propose a LieNet to regress a Lie algebra based on rotation representation. However, a 3 × 3 orthogonal matrix is over-parameterised and enforcing the orthogonality is non-trivial.

Quaternions are favourable due to the universality mapping from 4 dimensional values to legitimate rotations. This is Fig. 3: System pipeline: 6D-VNet takes a monocular image as input and performs the vehicles' 6DoF estimation. The grey box represents a canonical instance segmentation network and the dark blue branch is for estimating object 6DoF pose and its sub-category. a simpler process than the orthonormalisation of rotation matrices. Quaternions are continuous and smooth, lying in a unit manifold, which can be easily enforced through backpropagation. Therefore, the rotation head in our network focuses on the regression of quaternion representation. However, the main problem with quaternions is that they are not injective: the quaternion q and -q represent the same rotation because two unique values (from each hemisphere) map to a single rotation. To address the issue, we constrain all quaternions to one hemisphere such that there is a unique value for each rotation as follows:

Enforcement of Quaternions to one hemisphere: Let Q represents the set of quaternions, in which each quaternion, q ∈ Q, is represented as q = a + bi + cj + dk, and a, b, c, d ∈ R. A quaternion can be considered as a four-dimensional vector. The symbols i, j, and k are used to denote three "imaginary" components of the quaternion. The following relationships are defined:

i 2 = j 2 = k 2 = ijk = -1, from which it follows that ij = k, jk = i, and ki = j.
The quaternion q and -q represent the same rotation because a rotation of θ in the direction v is equivalent to a rotation of 2π -θ in the direction -v. One way to force uniqueness of rotations is to require staying in the "upper half" of S 3 . For example, require that a ≥ 0, as long as the boundary case of a = 0 is handled properly because of antipodal points at the equator of S 3 . If a = 0, then require that b ≥ 0. However, if a = b = 0, then require that c ≥ 0 because points such as (0, 0, -1, 0) and (0, 0, 1, 0) are the same rotation. Finally, if a = b = c = 0, then only d = 1 is allowed.

Hence, for the rotation head, given the ground truth unique quaternion q and the predicted q, the rotation loss is defined as:

L rot (q, q) = qq q γ (1)

An important choice for regressing in Euclidean space is the regression norm γ . Typically, deep learning models use

L 1 = 1 or L 2 = 2 .
With the datasets used in this paper, we found the L 1 norm performs better: the error does not increase quadratically with magnitude nor over-attenuate large residuals. Translation loss L trans . Regressing translation vector in world unit instead of pixel unit stabilises the loss. The transformation of the detected object takes 2D bounding box centre, height and width u p , v p , h p , w p in pixel space and then outputs their corresponding u w , v w , h w , w w in world unit as:

u w = (u p -c x )z s f x , v w = (v p -c x )z s f y , h w = h p f x , w w = w p f y
where the matrix [f x , 0, c x ; 0, f y , c y ; 0, 0, 1] is the camera intrinsic calibration matrix. Huber loss is adopted to describe the penalty in translation estimation: give ground truth 3 dimensional translation vector t and the prediction t, the translation loss is:

L trans (t, t) = 1 2 (t -t) 2 /δ if |t -t| < δ, |t -t| -1 2 δ otherwise.
(

) 2
where the hyperparamter δ controls the boundary of outliers.

If δ is set to 1, then it becomes the smooth-L1 loss used in [START_REF] Girshick | Fast r-cnn[END_REF].

In this paper, δ is set as 2.8 which is the cut off threshold for translational evaluation as described in Sec. IV-B.

C. Weighted Non-local neighbour embedding

In order to capture spatial dependencies among detected objects of interest, we introduce a non-local block with a weighted operation. We reason that the dependencies among neighbouring objects will assist the network to regularise the 6DoF pose estimation collectively better than treating them individually. For example, neighbouring cars on the same lane will follow almost the same orientation and maintain certain distance. There are several advantages of using a weighted non-local operations comparing with other social embedding schemes (i) non-local operations capture long-range dependencies directly by computing interactions between any two positions, regardless of their positional distance; (ii) non-local operations maintain the variable input sizes and can be easily combined with other operations; (iii) our proposed weighted operation renders it possible to associate the output maps with self-attention mechanisms for better interpretability.

The non-local means (NL-means) is first introduced in [START_REF] Buades | A non-local algorithm for image denoising[END_REF], based on a non-local averaging of all pixels in the image. Later [START_REF] Wang | Non-local neural networks[END_REF] introduced the non-local operations as an efficient and generic component for capturing long-range dependencies with deep neural networks. The non-local operations maintain the variable input sizes. Intuitively, a non-local operation computed the response at a position as a weighted sum of the features at all positions in the input feature maps. The generic non-local operation in deep neural network is defined as:

y i = 1 C(x) ∀j f (x i , x j )g(x j ) (3)
where i is the index of an output position (here in space) whose response is to be computed and j is the index that enumerates all possible positions. x is the input signal and y is the output signal of the same size as x. A pairwise function f computes a scalar (representing relationship such as affinity) between i and j. The unary function g computes a representation of the input signal at the position j. The response is normalised by a factor C(x). The non-local models are not sensitive to the design choices of f and g. For simplicity and fast computation, we consider g in the form of a linear embedding: g(x j ) = W g x j , where W g is a weight matrix to be learnt. The pairwise function f is implemented in the form of embedding Gaussian as: f (x i , x j ) = e θ(xi) T φ(xj ) , where θ(x i ) = W θ x i and φ(x j ) = W φ x j are two embeddings. And C(x) = ∀j f (x i , x j ). The recently proposed selfattention module [START_REF] Vaswani | Attention is all you need[END_REF] is a special case of non-local operations in the embedding Gaussian version: when, for a given i, 1

C(x)
f (x i , x j ) becomes softmax computation along the dimension j. So we have y = sof tmax(x T W T θ W φ x)g(x). However, we found out that when the input dimension d (for a feature map of H × W × C where C is the channel number, d = H ×W ) gets large, the dot products grow large in magnitude, pushing the softmax function into regions where it has extreme gradients. As a result, y will have extreme value as well. 1 To counteract the effect, we propose to use a weighted non-local operation for calculating the self-attention map A 1 To illustrate why the dot products get large, assume that the components of x are random variables with mean 0 and variance 1. Then its self dot product,

x T • x = d i x 2
i , has mean 0 and variance d (d is the input dimension).

as:

A = sof tmax( x T W T θ W φ x √ d ) (4)
So that y = A • g(x). The weighted non-local operation scales the dot-product attention variance to unit 1, which consequently does not push the softmax operation to extreme, saturated values. Intuitively, the weighted operation has the same form of expression. However, the suitable temperature in softmax is finicky to determine. Alternatively, we scale the dotproduct input variance to unit 1. Consequently, the output map after softmax operation will provide a justifiable interpretation in the form of a self-attention formulation.

IV. EXPERIMENTS

We benchmark our method on the challenging PAS-CAL3D+ [START_REF] Xiang | Beyond pascal: A benchmark for 3d object detection in the wild[END_REF] dataset and perform comprehensive studies on the challenging Apolloscape dataset [START_REF] Huang | The apolloscape dataset for autonomous driving[END_REF]. For both Pascal3D+ and Apolloscape dataset, we focus our experiments on the "Car" category in urban scene. Implementation Details: ResNet-101 is adopted as the convolutional body with Feature Pyramid Networks (FPN) as the detection backbone. The instance segmentation head is pre-trained using the Apolloscape scene dataset2 with "car, motorcycle, bicycle, pedestrian, truck, bus, and tricycle" as 8 instance-level annotations.

The hyperparameters λ sub cls , λ rot , λ trans in Eqn. III-B are set to 1.0, 1.0, 0.1 to scale the loss accordingly. To decrease the translational outlier penalty and stabilise the network training, the hyperparameter δ in Eqn.III-B is set to 2.8 metres as the loose end of the translational metric in Sec. IV-B. The base learning rate starts from 0.01 with warm start-up scheme and the models are trained up to 5 × 10 4 iterations with learning rate divided by 10 at 1.5 × 10 4 th and 3 × 10 4 th iterations. We use a weight decay of 0.0001 and a momentum of 0.9. The RoIAlign takes a feature map of 7 × 7 from each RoI. The weighted non-local block is plugged into the last layer (5th) of the convolutional body with a receptive field of 32×32. During training, the images are resized with the largest side randomly in [2000,2300]. Due to memory limitation, batch size is set as one image per single GPU. The incorporation of non-local block will increase the memory requirement, hence the training images are resized with the largest side randomly in [1500,2000] when non-local block is plugged in. Top 1000 regions are chosen as per FPN level with 100 batch size per image. During testing phase, 0.1 is chosen as the detection threshold in the Faster R-CNN head with multi-scale augmentation.

A. Analysis on Pascal3D+ dataset

We first evaluate our method on the task of joint detection and viewpoint estimation. The object orientation is represented in terms of viewpoint: azimuth, elevation and tilt angles. The results are reported using Average Viewpoint Precision(AVP) under different quantisation of angles, as proposed by [START_REF] Xiang | Beyond pascal: A benchmark for 3d object detection in the wild[END_REF] [START_REF] Tulsiani | Viewpoints and keypoints[END_REF] 0.89 8.8 • Viewpoints & KeyPoints [START_REF] Tulsiani | Viewpoints and keypoints[END_REF] 0.90 9.1 • RenderForCNN [START_REF] Su | Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views[END_REF] 0.88 6.0 • Deep3DBox [START_REF] Mousavian | 3d bounding box estimation using deep learning and geometry[END_REF] 0.90 5.8 • 3D-RCNN(VGG16) [START_REF] Kundu | 3d-rcnn: Instance-level 3d object reconstruction via render-and-compare[END_REF] 0.94 3.4 • 3D-RCNN(ResNet50) [START_REF] Kundu | 3d-rcnn: Instance-level 3d object reconstruction via render-and-compare[END_REF] 0.96

3.0 • 6D-VNet 0.95 3.3 •
TABLE II: Evaluation of viewpoint estimation with groundtruth detections on Pascal3D+ for "Car" category.

3D-RCNN [START_REF] Kundu | 3d-rcnn: Instance-level 3d object reconstruction via render-and-compare[END_REF] where a differentiable Render-and-Compare loss is introduced. This loss is complementary to our framework and we believe the inclusion of the loss that allows 3D shape and pose to be learned with 2D supervision can further improve the accuracy of the pose estimation.

We follow [47] [46] to evaluate viewpoint on ground-truth boxes in Tab. II which provides an upper-bound of viewpoint accuracy independent of the object detector adopted. The viewpoint estimation error is measured as geodesic distance over the rotation group SO(3). The Acc π/6 that estimates accuracy at π 6 and the median angular error MedErr which was used in [START_REF] Xiang | Subcategory-aware convolutional neural networks for object proposals and detection[END_REF]. Our method performs on par with the stateof-the-art on both the π 6 and median angular error metrics.

B. Analysis on Apolloscape 3D Car Instance dataset

The Apolloscape 3D Car Instance challenge contains a diverse set of stereo video sequences recorded in the street scenes from different cities. There are 3941/208/1041 high quality annotated images in the training/validation/test set. 3The monocular RGB images are of pixel size 2710 × 3384. It is worth noticing the high resolution of the images: the total number of pixels of a single image is 100 times than those of other canonical image datasets (e.g., MS-COCO, Mapillary Vistas, ImageNet). The camera intrinsic parameters are provided in the form of camera focal lengths (f x , f y ) and optical centres expressed in pixels coordinates (c x , c y ). Car models are provided in the form of triangle meshes. The mesh models have around 4000 vertices and 5000 triangle faces. One example mesh model is shown as in Fig. 2. There are total 79 car models (Fig. 4) in three categories (sedan1, sedan2, SUV) with only 34 car models appearing in the training set. In addition, ignored marks are provided as unlabelled regions and we only use the ignored masks to filter out detected regions during test. Evaluation Metrics: The evaluation metrics follow similar instance mean AP as the MS-COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF]. However, due to 3D nature, the 3D car instance evaluation has its own idiosyncrasies: instead of using 2D mask IoU to judge a true positive, the 3D metric used in this dataset contains the perspective of shape (s), 3D translation (t) and 3D rotation (r). The shape similarity score is provided by an N car * N car matrix where N car denotes the number of car models. For 3D translation and 3D rotation, the Euclidean distance and arccos distance are used for measuring the position and orientation difference respectively. Specifically, given an estimated 3D car model in an image C i = {s i , t i , r i } and ground truth model C * i = {s * i , t * i , r * i }, the evaluation for these three estimates are as follows: for 3D shape, reprojection similarity is considered by putting the model at a fix location and rendering 10 views (v) by rotating the object. Mean IoU is computed between the two poses (P ) rendered from each view. Formally, the metric is defined as:

c shape = 1 |V | v∈V IoU (P (s i ), P (s * i )) v
, where V is a set of camera views. For 3D translation and rotation, the evaluation metric follows that of the canonical self-localisation:

c trans = t i -t * i 2 and c rot = arccos(|q(r i ) q(r * i )|).
Then, a set of 10 thresholds from loose criterion to strict criterion (c 0 , c 1 , . . . , c 9 ) is defined as: shapeT hrs -[.5 : .05 : .95] rotT hrs -[50 : 5 : 5] transT hrs -[2.8 : .3 : 0.1] where the most loose metric c 0 : 0.5, 50, 2.8 means shape similarity > 0.5, rotation distance < 50 • and translation distance < 2.8 metres, and stricter metrics can be interpreted correspondingly: all three criterion must be satisfied simultaneously so as to be counted as a true positive.

It is worth noting the strict translation distance threshold of 2.8 metres: it requires that the detected vehicle's distance from the camera centre needs to be correctly estimated within a 2.8 metres threshold even if the vehicle is hundreds metres away from the camera, otherwise the detection will be counted as a false positive. The precise translational estimation requirement is the major factor for the network to produce incorrect false positive, which is a challenging task from a human perspective as well.

1) Main Results & Ablation Studies:

We present the results of the proposed 6D-VNet which reaches the 1st place in Apolloscape challenge 3D Car Instance task. Furthermore, we perform comprehensive evaluations to analyse the "bells and whistles" in 6D-VNet, which further improve state-of-the-art cf. Tab. VII. Our ablation studies gradually incorporated all components and are detailed as follows. Effect of End-to-End Training. We first provide the 2D bounding box mAP from Faster R-CNN head to serve as the upper bound by 2D object detection as in Tab. III. We can find that small objects are more challenging to detect. Small object also indicates the object longitudinal axis distance is typically far away from the camera. The accurate estimation of large translational distance value is thus more challenging.

In Tab. IV we show that the translational head is crucial for improving the mAP in our end-to-end training scheme. The projective distance estimation was the defacto approach in previous state-of-the-art methods [START_REF] Sundermeyer | Implicit 3d orientation learning for 6d object detection from rgb images[END_REF], [START_REF] Kehl | Ssd-6d: Making rgb-based 3d detection and 6d pose estimation great again[END_REF], [START_REF] Do | Real-time monocular object instance 6d pose estimation[END_REF] as a second stage to measure the translational distance as described as follows: Projective Distance Estimation: Fig. 5 illustrates the projective distance estimation via geometric method which were adopted in previous state-of-the-art methods [START_REF] Sundermeyer | Implicit 3d orientation learning for 6d object detection from rgb images[END_REF], [START_REF] Kehl | Ssd-6d: Making rgb-based 3d detection and 6d pose estimation great again[END_REF], [START_REF] Do | Real-time monocular object instance 6d pose estimation[END_REF]. For each object we precomputed the 2D bounding box and centroid.

To this end, the object is rendered at a canonical centroid distance z r (z r should be set larger than the object length in longitudinal axis so that the entire object can be projected onto the image plane). Subsequently, the object distance z s can be inferred from the projective ratio according to z s = lr ls z r , where l r denotes diagonal length of the precomputed bounding box and l s denotes the diagonal length of the predicted bounding box on the image plane. Given its depth component z s , the complete translational vector can be recovered geometrically as:

x s = (u -c x )z s f x , y s = (v -c x )z s f y
where [u, v] is the bounding box centre, and the matrix [f x , 0, c x ; 0, f y , c y ; 0, 0, 1] is the camera intrinsic calibration matrix. The formulation assumes that: (i) the object centre in 3D will be projected to the object bounding box in the 2D image; (ii) the predicted object class and rotation vector is correctly estimated. The depth component estimation is treated as an entirely independent process given the rotation pose estimation. Hence, the geometric post-processing method achieves around only 3.8%mAP due to the crude translation estimation. We show that using the same bounding box information (in world unit) to train a translation regression head improves the mAP to 8.8%. The projective distance is the most commonly adopted approach in previous state-of-the-art methods [START_REF] Sundermeyer | Implicit 3d orientation learning for 6d object detection from rgb images[END_REF], [START_REF] Kehl | Ssd-6d: Making rgb-based 3d detection and 6d pose estimation great again[END_REF], [START_REF] Do | Real-time monocular object instance 6d pose estimation[END_REF] as a second stage to measure the translational distance. w/o ⊕ denotes the network without the concatenation operation in Fig. 3. ⊕ represents the branch trained with joint losses. ⊕ (fine-tune) denotes, when training the network, the learnt parameters in convolutional body and Faster R-CNN head are unfreezed. We visualize the self-attention map A of Eqn. 4 in the weighted non-local block in Fig. 7. The heat maps on the right exhibit meaningful attention positions (without the weighted operation, the heat map will exhibit extreme high temperature with hard to interpret positions on the attention map), which demonstrates that the weighted non-local block can learn to find meaningful relational clues regardless of the distance in space. In Fig. 8, we visualise the predicted vehicles in 3D space. When incorporated with weighted non-local block, the model is able to capture the spatial dependencies so that it adjusts the predictions according to the distance and orientation of neighbouring vehicles.

V. CONCLUSIONS

We proposed an end-to-end network 6D-VNet for 6DoF pose estimation of vehicles from monocular RGB images. It can not only detect the traffic participants, but also generate their translations and rotations. To the best of our knowledge, the incorporation of translation regression into the network is the first of its kind and greatly improve the pose estimation accuracy compared to those methods which treat the translation estimation as a post-processing step. In addition, we design the joint losses according to solid grounding in geometry, which are crucial to achieve accurate pose estimation. The experiments show that the proposed method reaches the first place in Apolloscape challenge 3D Car Instance task and achieves the performance of current state-of-the-art framework on the PASCAL3D+ dataset. Particularly, a large improvement for position estimation is observed when the translation head is trained from both visual clues and bounding box information. Furthermore, we demonstrate that the spatial dependencies among neighbouring vehicles can be incorporated via a weighted non-local block and an interpretable self-attention map. It can help regularise the 6DoF object pose estimation collectively which is better than treating them individually. In this paper, the 6DoF pose is directly generated from the network, post-refinement is not considered. In the future work we will try to further improve the 6DoF pose estimation by using post-processing techniques such as iterative closet point based algorithms or iterative refinement network. VII: Performance on test set in terms of mAP. c 0 is the most loose criterion for evaluating AP and c 5 is in the middle of criterion. Superscript S, M, L of average precision (AP s) and average recall (ARs) represent the object sizes. Superscript number 1, 10, 100 represent the total number of detections for calculating recalls. Projective distance (PD) [START_REF] Do | Real-time monocular object instance 6d pose estimation[END_REF] estimation is adopted in the state-of-the-art methods [START_REF] Do | Real-time monocular object instance 6d pose estimation[END_REF], [START_REF] Kehl | Ssd-6d: Making rgb-based 3d detection and 6d pose estimation great again[END_REF], [START_REF] Sundermeyer | Implicit 3d orientation learning for 6d object detection from rgb images[END_REF]. Triple head (TH) is the baseline 6D-VNet. Translation head is then concatenated with visual branch (TV), represented by ⊕ operation in Fig. 3. Fine-tuning (FT) the convolutional body and detection head gives the task specific network a 3% boost in mAP. Multi-scale testing (MS) further increases the accuracy.

The incorporation of weighted non-local block (NL) improves both precision and recall. Using ignore mask (IM) to filter 2D detection bounding box with 0.5 IoU as threshold improves the precision, however, slightly degrades the recall. Finally, enforcing the quaternions to one hemisphere (QS) achieves the current state-of-the-art.

Fig. 4 :

 4 Fig. 4: 79 car model meshes for the Apolloscape 3D Car Instance dataset.

Fig. 5 :

 5 Fig. 5: Projective Distance Estimation.

Fig. 6 :

 6 Fig. 6: Losses w/o & w/ weighted non-local block (NL).

Fig. 7 :

 7 Fig. 7: Examples of the behaviours of a weighted non-local block. These two examples are from held-out validation images. On the left images, the starting point (pink square) represents one x i . Since we insert the weighted non-local block in res5 layer, one pink square denotes a receptive field of pixel size 32 × 32. We position the starting points on one of the vehicles. The end points of arrows represent x j . The 10 highest weighted arrows for each x i are visualised. The arrows clearly indicate that the weighted non-local block attends to the neighbouring traffic participants. Note that in the second illustration, end points are able to locate neighbouring vehicles' wheels and rear-view mirrors which are critical clues to position and orient the vehicle. The right images are the visualisations of self-attention map A of x i from Eqn. 4. These visualisations show how the weighted non-local block finds interpretable, relevant indications of neighbouring vehicles to adjust the pose estimation.

Fig. 8 :

 8 Fig. 8: 3D renderings of predicted vehicles with camera coordinate axis. The bottom renderings framed by silver plate represent the results from the model with weighted non-local block. The vehicles in colour silver, green, red represent ground truth, true positive and false positive respectively. Due to the strictly fixed 2.8-metre translation criterion, vehicles that are farther way from the cameras are more difficult to measure. The pink arrows highlight the predictions that have been adjusted according to their neighbouring vehicles when weighted non-local block is plugged in.

  . The results are shown in Tab. I. Our proposed framework improves almost all previous methods apart from the recent

	Method			Car
		AV P4	AV P8	AV P16	AV P24
	Pepik et al. [41]	36.9	36.6	29.6	24.6
	Viewpoints & KeyPoints [42]	55.2	51.5	42.8	40.0
	RenderForCNN [43]	41.8	36.6	29.7	25.5
	Poirson et al. [44]	51.4	45.2	35.4	35.7
	Massa et al. [45]	58.3	55.7	46.3	44.2
	Xiang et al. [46]	48.7	37.2	31.4	24.6
	3D-RCNN [47]	71.8	65.5	55.6	52.1
	6D-VNet	72.2	63.0	54.8	48.7

TABLE I :

 I Joint detection and viewpoint evaluation on Pas-cal3D+ dataset for "Car" category.

	Method	Acc π 6 ↑ M edErr ↓
	Viewpoints & KeyPoints(TNET)

TABLE III :

 III

2D bounding box detection mAP from the Faster R-CNN head. It is the upperbound for 3D detection result. Subscripts of AP represent squared object one side size in pixels as: S:28-56, M:56-112, L: 112-512, XL:512+.

TABLE IV :

 IV Effect of end-to-end training and joint losses.

TABLE V :

 V Effect of Weighted Non-local Block and runtime analysis.

TABLE VI :

 VI Number of parameters of each component of the proposed method.

	the training procedure. Tab. V shows that the incorporation
	of weighted non-local block consistently improves mAP on
	both validation and test set with marginally increased inference
	runtime. According to Tab. VI, the backbone Mask RCNN and
	the triplet heads contain most parameters (242Mb+54Mb), the
	additional parameters are caused by the proposed weighted
	non-local block only achieve 518kb which is extremely low
	compared to the backbone. Thus, similar conclusion can be
	drawn that the proposed method can improve the performance
	with marginally increased number of parameters.

  PD TH TV FT MS NL IM QS mAP c0 c5 AP S AP M AP L AR 1 AR 10 AR 100 AR S AR M AR L

	"	0.037 0.067 0.088 0.121 0.141 0.144 0.147 " " " " " " " 0.148 0.353 0.209 0.115 0.138 0.371 0.042 0.191 0.244 0.115 0.259 0.490 0.125 0.042 0.072 0.042 0.076 0.016 0.091 0.128 0.072 0.133 0.245 " 0.190 0.089 0.069 0.057 0.194 0.026 0.116 0.139 0.069 0.138 0.305 " " 0.237 0.122 0.100 0.077 0.246 0.030 0.141 0.174 0.100 0.169 0.360 " " " 0.331 0.164 0.126 0.115 0.297 0.035 0.162 0.231 0.126 0.246 0.424 " " " " 0.351 0.198 0.128 0.131 0.357 0.040 0.179 0.243 0.128 0.253 0.477 " " " " " 0.352 0.199 0.131 0.133 0.360 0.040 0.187 0.249 0.131 0.261 0.481 " " " " " " 0.357 0.206 0.117 0.137 0.366 0.041 0.190 0.246 0.117 0.262 0.489

TABLE

http://apolloscape.auto/scene.html

After manual examination, we have deleted visually distinguishable wrongly labelled images, leaving us with 3888/206 images for training/validation.


Maotong Xu

Min Tian

Eytan Modiano
email: modiano@mit.edu

Suresh Subramaniam
email: suresh@gwu.edu

RHODA Topology Configuration Using Bayesian Optimization

Keywords: Bayesian Optimization, data center networks, inter-cluster network, RHODA

, a simple variant of the Hungarian algorithm that maximizes the one-hop or direct traffic among clusters was used. In this paper, we leverage the Bayesian Optimization (BO) framework and propose a fast algorithm to minimize the average number of hops in the inter-cluster network of RHODA. To the best of our knowledge, this is the first paper that employs BO to optimize optical DCN performance. We present our design decisions and modifications to BO based on the network constraints. Results show that BO can achieve optimal or near-optimal results, and outperforms a well-known regular topology (Gemnet) and the Hungarian-based method by up to 13% and 58%, respectively.

Introduction

Data center (DC) traffic has experienced dramatic growth, increasing at an annual rate of 31%, and will reach 3.3 ZB per year [START_REF] Wang | Scheduling with machine-learning-based flow detection for packetswitched optical data center networks[END_REF]. To store and process Supported by NSF award CNS-1618487 and CNS-1617091 huge amounts of data, DCs will consist of millions of servers. For example, Microsoft owns over one million servers and a single DC alone contains over 250,000 servers [START_REF]Who Has the Most Web Servers: Available online: whos-got-the-most-web-servers[END_REF]. On the other hand, interactive applications, e.g., web searches and social media, require low network latency. For example, the acceptable latency range for stock exchange transactions is 5-100 ms [START_REF]Some interesting bits about latency[END_REF]. Traffic within a DC is usually not uniformly distributed. For instance, measurements on a 1500-server Microsoft production DC network (DCN) reveal that only a few ToRs (Top-ofthe-Rack, used as an alternative for rack) are hot and most of a ToR's traffic goes to a few other ToRs [START_REF] Kandula | Flyways to de-congest data center networks[END_REF]. Finally, power consumption of data centers will reach 140 billion kilowatt-hours annually by 2020, and it will cost $13 billion annually [16].

Conventional electrical DCNs (e.g., FatTree [START_REF] Al-Fares | A scalable, commodity data center network architecture[END_REF], Bcube [START_REF] Guo | BCube: a high performance, server-centric network architecture for modular data centers[END_REF], VL2 [START_REF] Greenberg | VL2: a scalable and flexible data center network[END_REF] and Flattened Butterfly [START_REF] Csernai | Towards 48-fold cabling complexity reduction in large flattened butterfly networks[END_REF]) are built using a multi-layer approach, with a large number of switches at the bottom level to connect with servers/racks and a few highend switches located at the upper layers to aggregate and distribute the traffic. Those networks rely heavily on high-cost and power-hungry electrical switches, and operators are facing limited scalability, high latency and low energy efficiency problems.

Optical Wavelength Division Multiplexing (WDM) is a promising technology for meeting the traffic demand of data centers. It can support hundreds of wavelengths per fiber and 100 Gbps transmission rate per wavelength. Moreover, large-scale optical switches consume less power per bit/s, making the network architecture scalable and energy-efficient. RHODA [START_REF] Xu | A hierarchical wdmbased scalable data center network architecture[END_REF] is a WDM-based reconfigurable hierarchical optical DCN architecture. The architecture can scale up to 10+ million servers and support various traffic patterns. RHODA achieves high throughput through reconfigurable clustering of racks of servers. Racks with large amounts of mutual traffic can be grouped into clusters, and a high-bandwidth intra-cluster network connects racks within a cluster. The clusters are connected through an inter-cluster network which is also reconfigurable based on the traffic demands among clusters. The inter-cluster network uses wavelength selective switches (WSSs) and optical space switches to achieve topology reconfigurability. The resulting inter-cluster network topology is degree-constrained because of port count limitation of WSSs. RHODA saves large amount of power and cost by extensively using passive devices (couplers, Arrayed Waveguide Grating Routers (AWGRs), and mux/demuxes), and minimally using power-hungry and expensive devices (e.g., WSSs).

Once clusters are defined and the intra-and inter-cluster topologies have been established, flows are routed within and between clusters over those topologies, possibly using multiple hops, where each hop corresponds to one optical transmission and reception. As packets have to be converted from optical to electrical to optical form for each hop, and queued up for transmission at intermediate nodes (ToRs), the network latency is largely determined by the number of hops. The focus of this paper is on the design of the inter-cluster network topology to minimize the average number of hops (weighted by traffic demands). In [START_REF] Xu | A hierarchical wdmbased scalable data center network architecture[END_REF], the inter-cluster topology is constructed using a Hungarian-based method [START_REF] Kuhn | The hungarian method for the assignment problem[END_REF]. The method first builds a circle/ring among clusters to first make the inter-cluster topology connected, and then the Hungarian algorithm is iteratively used to find a perfect bipartite matching to maximize the total traffic over the directly connected clusters (i.e., single-hop traffic). General regular topologies, e.g., Hypercube [START_REF] Dally | Principles and practices of interconnection networks[END_REF] and Gemnet [START_REF] Iness | Gemnet: A generalized, shuffle-exchangebased, regular, scalable, modular, multihop, wdm lightwave network[END_REF], are attractive for uniform traffic, but they do not minimize the average hop distance for skewed traffic. In this paper, we propose an algorithm for inter-cluster topology design based on Bayesian Optimization (BO), and demonstrate that it produces optimal or near-optimal results very quickly.

Bayesian optimization (BO) [START_REF] Mockus | Bayesian heuristic approach to global optimization and examples[END_REF] is a powerful tool to find optimal or nearoptimal solutions for black-box problems, i.e., it is suitable in situations where a closed-form expression for the objective function is unknown. BO first uses a statistical model, e.g., Gaussian Process, to fit the objective function. Then, a pre-defined acquisition function is used to locate the next point to sample. The acquisition function can trade off between exploration and exploitation. This means that BO does not get stuck in local optima and can often find the global optimal solution.

In this paper, we use BO to find an inter-cluster topology for RHODA to minimize the average traffic-weighted hop distance. We describe our design decisions -the choice of prior model, acquisition function and optimizing algorithm. Further, we describe several modifications to BO based on topology constraints. Finally, we compare BO with the optimal solution obtained through solving an integer-linear program (ILP), and with the hop distances in the Gemnet topology and using Hungarian-based method. Our results show that BO can achieve optimal or near-optimal results, and outperforms Gemnet and Hungarian-based method by up to 13% and 58%, respectively.

The rest of this paper is organized as follows: Section 2 briefly describes the RHODA architecture. Section 3 presents the ILP model and our design decisions and modifications of BO based on network requirements. Section 4 presents performance evaluation results, and Section 5 concludes the paper.

RHODA Architecture

In this section, we briefly describe the RHODA architecture. The reader is referred to [START_REF] Xu | A hierarchical wdmbased scalable data center network architecture[END_REF] for more design details of RHODA.

As Fig. 1 illustrates, RHODA is a DCN architecture consisting of N servers grouped into M racks, so that there are m = N/M servers per rack. Each rack has a ToR switch used for both electronic switching of the packets within a rack and for communication with other racks. Each ToR connects to all of the m servers within the rack. For communication to and from other ToRs, each ToR has T intra + T inter tunable transceivers (TRXs). RHODA consists of five parts, namely, cluster membership network (CMN), intra-and inter-signals demultiplexing part, intra-cluster network, inter-cluster network, and intra-and inter-signals multiplexing part. RHODA lets cluster membership be reconfigurable. This is achieved by the CMN using a set of k/α V ×V cluster configuration switches (CCS), where V = α•C and α is a positive integer parameter that is chosen as a trade-off between the cost and complexity of the CMN and the reconfiguration flexibility.

Since most of the heavy communication in a data center is carried over small subsets of ToRs [START_REF] Roy | Inside the social network's (datacenter) network[END_REF] (and these ToRs would ideally be configured to the same cluster), RHODA equips each cluster with a k×k AWGR to support large amounts of intra-cluster traffic. In the inter-cluster network, each cluster can be considered as the smallest communication element. Flows from racks are merged (using optical-to-electrical-to-optical conversion) by grooming switches (GSs) to reduce the number of wavelengths needed. The communication graph between clusters (i.e., the cluster topology) is then determined by C 1×d WSSs and d C×C optical space switches (OSWs). In particular, each cluster can send signals to up to d other clusters. Demultiplexers (DMUXs) split signals carried by different wavelengths. A signal carried on wavelength w is forwarded to the w k th port of the DMUX.

Topology Configuration Algorithm

Our objective is to minimize the traffic-weighted average number of hops in the inter-cluster network of RHODA. We first present an ILP formulation, and then present the BO framework adapted to solve our topology design problem.

ILP Formulation

The network has a set of nodes (which are the clusters) C, and a traffic matrix T . The number of nodes/clusters, C = |C|. Each node has both an in-degree and out-degree constraint of d. The following ILP constructs the degree-constrained and directed topology, and minimizes the traffic-weighted average hop distance over all node-pairs, assuming that multi-hop traffic is routed over the shortest path in the topology.

min i,j∈C T ij • H ij i,j∈C T ij (1) s.t. L ij ∈ {0, 1}, ∀i, j ∈ C (2) j∈C,j =i L ij ≤ d, ∀i ∈ C (3) j∈C,j =i L ji ≤ d, ∀i ∈ C (4)
L ii = 0, ∀i ∈ C (5) H ii = 0, ∀i ∈ C (6) 0 < H ij < C, ∀i, j ∈ C, i = j (7) H ij = 1, for L ij = 1, ∀i, j ∈ C, i = j (8) H ij = min(H ik + ((L kj -1) * (-a)) + 1)), for L ij = 1, ∀i, j, k ∈ C, i = j, j = k, i = k (9)
In the above formulation, the decision variables are the L ij 's, where L ij = 1 implies the establishment of a link between node i and j, and no link is established if L ij = 0. H ij represents the number of hops on the shortest path from i to j. In this ILP formulation, (1) is our objective, to optimize the traffic-weighted average number of hops. Inequalities (3) and ( 4) ensure that the in-degree and out-degree of each node is not more than D. The values of H ij 's are determined by ( 6), [START_REF] Greenberg | VL2: a scalable and flexible data center network[END_REF], and [START_REF] Guo | BCube: a high performance, server-centric network architecture for modular data centers[END_REF]. If a link from node i to j exists, H ij is 1; on the other hand, if there is no link from i to j, H ij is defined as the minimum value of H ik + ((L kj -1) * (-a) + 1), where a is defined as a large positive integer, so that if L kj does not exist, H ik + ((L kj -1) • (-a) + 1) will be an integer far larger than C -1.

Bayesian Optimization

The ILP is too time-consuming for large instances (e.g., when there are more than 10 clusters and the degree is 4 or higher) and cannot be used in an online setting where the traffic dynamically changes and the topology needs to be configured quickly so that packets are not blocked for extended periods while the configuration takes place. We therefore seek a fast heuristic algorithm to minimize average hop distance. In [START_REF] Xu | A hierarchical wdmbased scalable data center network architecture[END_REF], the topology is constructed by adapting the Fig. 2: An example of BO's working process [START_REF] Alipourfard | Cherrypick: Adaptively unearthing the best cloud configurations for big data analytics[END_REF].

well-known Hungarian assignment method [START_REF] Kuhn | The hungarian method for the assignment problem[END_REF] for finding the maximum-weight matching in a bipartite graph. Once a circle is formed among the clusters to ensure that the topology is connected, our approach in [START_REF] Kuhn | The hungarian method for the assignment problem[END_REF] iteratively applied the Hungarian algorithm until the degree constraint is violated. The resulting topology, however, may turn out to have a large average hop-distance. Grid Search and Random Search [START_REF] Bergstra | Random search for hyper-parameter optimization[END_REF] can be used for our problem, but time consumption would be too large.

In this paper, we turn to a widely used framework to solve optimization problems, namely, Bayesian optimization (BO). For instance, it is useful for solving the following problem (shown in one dimension here):

min x∈A f (x),
where structure/concavity of objective function f (x) is unknown but can be observed through evaluations. Further, BO aims to minimize the number of evaluations to save optimization cost/time. Figure 2 shows an example of running BO on a 1D problem. The blue area is an area in which the objective function is expected to lie in with, say, 95% probability. The optimization starts with two points, i.e., x 1 and x 2 . At each iteration, BO decides that the next point is sampled at the argmax of a predefined acquisition function. As shown in Fig. 2, x 3 and x 4 are sampled in the next two iterations.

BO consists of two essential components, i.e., a statistical model for modeling the objective function, and an acquisition function for deciding the next point to sample. First, BO evaluates the objective function based on multiple randomly chosen initial points (e.g., 5). Then, BO iteratively uses all available data to update the posterior probability on the objective function, use the current posterior probability to compute the acquisition function, and argmax the acquisition function to find the next point at which to evaluate the objective function. The pseudocode is shown in Alg. 1.

Algorithm 1: Bayesian Optimization Algorithm

1: Use a statistical model to model the objective function f 2: while Stopping condition is not satisfied do 3:

Update the posterior probability distribution 4:

Calculate the acquisition function using posterior distribution 5:

Argmax the acquisition function to locate next point 6:

Evaluate f with sample point 7:

Update dataset 8: end while To leverage BO to find a good topology in our problem, we need to make several design decisions and modifications based on our requirements.

Choice of prior model

We choose Gaussian Process as the statistical model for modeling the objective function. The Gaussian Process has desirable features, e.g., it is non-parametric and the model is approximately Gaussian (central limit theorem).

Random point

We provide several random "points" for BO's first step. In our case, we provide several adjacency matrices (representing random degreeconstrained topologies) as input to evaluate f (x). We generate such a random matrix as follows. First build a circle among nodes to guarantee that all nodes are connected. We then randomly assign d -1 1s (edges) on each row of the adjacency matrix. Then, we iterate over all columns and rows of the adjacency matrix in random order and adjust the entries to guarantee that each node has no more than d ingress/egress edges. The pseudocode is shown in Alg. 2.

Acquisition function There are three main acquisition functions used in BO, i.e., Probability of Improvement (PI) [START_REF] Kushner | A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise[END_REF], Expected Improvement (EI) [START_REF] Brochu | A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning[END_REF], and GP Upper Confidence Bound (GP-UCB) [START_REF] Srinivas | Gaussian process optimization in the bandit setting: No regret and experimental design[END_REF]. PI can get stuck in local optima and under-explored globally [START_REF] Snoek | Practical bayesian optimization of machine learning algorithms[END_REF], and GP-UCB needs extra effort to tune its own parameter. So, in this paper, we use EI as the acquisition function.

Optimization algorithm As shown in Line 5 of Alg. 1, BO finds the argmax of the acquisition function to locate the next sample point. In our case, BO inputs a random point (i.e., adjacency matrix) for an optimizing algorithm (Opt-Alg), e.g., Limited-memory Broyden-Fletcher-Goldfarb-Shanno algorithm (L-BFGS) [START_REF] Mokhtari | Global convergence of online limited memory bfgs[END_REF]. Then, L-BFGS uses the random point as an initial point and obtains derivatives of the acquisition function to identify the direction of steepest descent. However, the result (a matrix) from L-BFGS might not satisfy our topology degree constraints. for r in Lrow do 12:

if AM [r, c] == 1 then 13:

if n edge ≤ d -1 then 14:

n edge + = 1 15:

else 16: AM [r, c] = 0 17: end if 18: end if 19:
end for 20: end for We use two steps to make the result from L-BFGS be a valid (i.e., connected and degree-constrained) topology. First, we build a circle among clusters. Then, we sort the elements of the result matrix from L-BFGS in descending order and build edges between nodes one by one without violating degree constraints. The pseudocode is shown in Alg. 3.

Algorithm 3: Conversion Algorithm

Input: Result/Matrix from L-BFGS (RM ) Output: Adjacency matrix (AM ) Build a circle among nodes in AM Sort elements of RM in descending order (Lsort) for e in Lsort do if Egress degree of ci and ingress degree of cj are both less than d then Build an edge from ci to cj end if end for Stopping condition BO needs a stopping condition, e.g., based on time and/or iterations, and the best solution achieved in that period is taken as the final result. The more iterations, the better the final result, but as we seek a solution quickly, we use a stopping condition of sec, e.g., 1 sec. After sec, BO outputs the best achievable topology.

Evaluation

In this section, we conduct simulation results to compare the BO-based method with ILP, Hungarian-based method (HG), and Gemnet [START_REF] Iness | Gemnet: A generalized, shuffle-exchangebased, regular, scalable, modular, multihop, wdm lightwave network[END_REF], in terms of the average number of hops. The ILP gives the optimal value, but it can only be obtained for small topologies in reasonable time. Gemnet provides a general method to build a topology with several desirable properties such as small average hop distance, diameter, etc. As opposed to ShuffleNet [START_REF] Hluchyj | Shuffle net: An application of generalized perfect shuffles to multihop lightwave networks[END_REF] and de Bruijn graph [START_REF] Sivarajan | Multihop lightwave networks based on de bruijn graphs[END_REF], Gemnet has no restriction on the number of nodes in its topology, and typically achieves smaller hop distances. Gemnet(M, K) arranges clusters in a cylinder of K columns and M clusters per column.

Cluster Topology

We first compare BO with ILP, Gemnet, and HG in terms of the average number of hops of the cluster topologies they produce. Flows are sent from each cluster to other clusters (uniform traffic), and the flow rate equals 1 Gbps. Figure 3 shows in terms of average number of hops, BO can achieve optimal results in some cases, and ILP outperforms BO by no more than 6% in the considered cases. Also, BO outperforms Gemnet and HG by up to 21% and 15%, respectively.

Then, we compare BO with Gemnet and HG with different traffic patterns in large networks and show comparison results in Figure 4. Figure 4(a) shows comparison results with different number of clusters under uniform traffic. Results show BO outperforms Gemnet and HG by up to 21% and 58%, respectively. Clearly, HG is not a good choice to configure topology under uniform traffic. The reason is that HG aims to find a matching to maximize single-hop traffic, and under uniform traffic, HG might perform the same as a random topology. chosen between 1 and 100 Gbps. Results show BO outperforms Gemnet and HG by up to 13% and 58%, respectively. Also, results show that HG outperforms Gemnet in a network with low traffic density. Under low traffic density, building direct links between clusters with large traffic benefits more in terms of the average number of hops.

RHODA

We now evaluate the various algorithms based on how the generated inter-cluster topologies perform in the entire RHODA architecture. Recall that the overall hop distance in RHODA is based on both the intra-and inter-cluster hop distances, and we have only tried to optimize the inter-cluster topology in this paper. We assume the following numbers for the DCN: the number of clusters is 64, and each cluster has 64 racks (k = 64). The number of wavelengths on a fiber is 128, the bandwidth of a wavelength is 100 Gbps, and both T intra and T inter are equal to 2 (i.e., a total of 4 transceivers per rack). The degree of WSS is 4 (i.e., d = 4). We consider three traffic patterns, i.e., uniform, low density traffic (LT), and high density traffic (HT). We set the rack flow rate to be 1/k 2 Gbps (the total flow rate from a cluster to another cluster equals 1 Gbps). Under uniform traffic pattern, a rack sends a flow to each of the other racks. Under LT and HT, the probability that a cluster sends flows to another cluster is 0.2 and 0.8, respectively. Given cluster c i sends flows to cluster c j , each rack in c i sends a flow to each rack in c j .

In Figure 5, we show comparison results of BO, Gemnet, and HG in RHODA with different metrics, i.e., the average number of hops, the average switch load, and the maximum switch load. Results show that RHODA outperforms Gemnet by up to 11%, 11%, and 77% in terms of average hop distance, average switch load, and maximum switch load, respectively. Also, RHODA outperforms HG by up to 58%, 58%, and 77% in terms of average hop distance, average switch load, and maximum switch load, respectively. Thus, the topology configured by BO not only achieves small hop distances, but also balances the traffic well among switches.

Low Traffic (0.

Conclusion

A reconfigurable hierarchical optical DCN architecture (RHODA) was introduced in an earlier paper. RHODA groups racks into clusters and enables both clusters and the inter-cluster topology to be configurable. In this paper, we focus on optimizing the inter-cluster topology in terms of the weighted hop distance, and present an approach based on Bayesian Optimization (BO). We compare BO with ILP, Gemnet, and the Hungarian-based method. Results show that BO can achieve optimal or near-optimal results within a small amount of time, and it outperforms Gemnet and Hungarian-based method by up to 13% and 58%, respectively.

Fig. 1 :

 1 Fig. 1: The RHODA architecture. M : number of racks; k: number of racks per cluster; d: egress degree of a cluster; C: number of clusters.

Algorithm 2 :

 2 Random points Algorithm 1: Input: C, d 2: Output: Adjacency Matrix (AM ) 3: Build a circle among clusters 4: for r in range(C) do 5: Randomly assign d -1 elements to 1 in r th row 6: end for 7: Find a random permutation of list range(1, C) called (L col ) 8: for c in L col do 9: n edge = 0 10: Find a random permutation of list range(1, C) called (Lrow) 11:

Fig. 3 :

 3 Fig. 3: Performance comparisons of Bayesian-based, Gemnet and Hungarian algorithms in cluster network.

Figure 4 (Fig. 4 :

 44 Fig. 4: Comparisons of Bayesian-based, Gemnet and Hungarian algorithms in cluster networks: (a) Comparison with different number of clusters and cluster degrees. (b) Comparison with different traffic densities.

Fig. 5 :

 5 Fig. 5: Comparisons of Bayesian-based, Gemnet and Hungarian algorithms in RHODA: (a) Comparison in terms of number of hops. (b) Comparison in terms of average switch load. (c) Comparison in terms of maximum switch load.


Ahmed Mohammed Mikaeil
email: ahmed_mikaeil@yahoo.co.uk

Weisheng Hu
email: wshu@sjtu.edu.cn

Q-Learning Based Joint Allocation of Fronthaul and Radio Resources in Multiwavelength-Enabled C-RAN

Keywords: 5G, C-RAN, mobile fronthaul, reinforcement learning, resource allocation, WDM-PON, TWDM-PON

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Cloud radio access network (C-RAN) is a leading technology for next generation mobile network 5G. In 5G C-RAN the traditional base station functions are split between three entities known as the central unit (CU) which contains a number of virtualized baseband units (vBBUs) pooled in a central location to facilitate signal processing, transmission scheduling and resource sharing, the remote radio units (RRUs) which are remotely deployed at the cell sites, and the distributed units (DUs) which can be independently deployed together with CUs or DUs [START_REF] Wey | Passive Optical Networks for 5G Transport: Technology and Standards[END_REF]. The interface connecting between CU and DU is known as midhaul interface (also known as Fronthaul-II or F1 interface), and the interface connecting between RRU and DU is known as mobile fronthaul interface (also known as Fronthaul-I or Fx interface) [START_REF] Mikaeil | Traffic-Estimation-Based Low-Latency XGS-PON Mobile Front-Haul for Small-Cell C-RAN Based on an Adaptive Learning Neural Network[END_REF].

Passive optical networks (PONs) are promising technologies for supporting fronthaul and mid-haul interfaces in next generation mobile network (5G). For example, current commercial PONs such as XGS-PON and 10GEPON are capable of supporting mid-haul interface without any modification as the capacity and latency requirement for such an interface is similar to traditional backhaul network [START_REF] Mikaeil | Traffic-Estimation-Based Low-Latency XGS-PON Mobile Front-Haul for Small-Cell C-RAN Based on an Adaptive Learning Neural Network[END_REF]. However, for fronthaul interface some modifications regarding the latency and bandwidth efficiency are required because such an interface requires a high capacity and low latency transport network solution [START_REF] Mikaeil | Traffic-Estimation-Based Low-Latency XGS-PON Mobile Front-Haul for Small-Cell C-RAN Based on an Adaptive Learning Neural Network[END_REF].

There are many proposals in literature that studied the latency and bandwidth efficiency issues of PON based mobile front-haul. The existing popular proposals are: 1-Traffic estimation low-latency PON based mobile front-haul [START_REF] Nomura | First demonstration of optical-mobile cooperation interface for mobile fronthaul with TDM-PON[END_REF], which relies on predictive method to estimate the scheduling grants for the optical network units (ONUs) to minimize mobile front-haul scheduling latency. 2-Mobile-DBA front-haul [START_REF] Siyu | Low-latency high-efficiency mobile fronthaul with TDM-PON (mobile-PON)[END_REF], which utilizes the mobile uplink scheduling information to compute the scheduling grants for the ONUs in order to eliminate the scheduling delay and the waiting time of ONUs. 3-Mobile-PON proposal [START_REF] Tiwari | LONG TERM EVOLUTION (LTE) PROTOCOL Verification of MAC Scheduling algorithms in NetSim[END_REF] which relies on PHY-2 split option to increase front-haul efficiency and unifies PON and LTE schedulers by dynamically or statically mapping of LTE radio resource blocks (RBs) into the PON front-haul transport blocks (TBs) to eliminate front-haul latency.

The major limitation of these proposals is that all of them consider single wavelength PONs mobile front-haul; whereas, due to the huge data-rate requirement for front-haul interface in 5G mobile network C-RAN architecture, single wavelength PONs are insufficient for supporting 5G C-RAN. Another limitation is that in Ref [START_REF] Tiwari | LONG TERM EVOLUTION (LTE) PROTOCOL Verification of MAC Scheduling algorithms in NetSim[END_REF] the authors assume a fixed front-haul TB size to be allocated to every RB independent of actual RB capacity. However, in practical LTE network the actual capacity of the RB depends on many factors such as user equipment UE request size, channel quality status and modulation and coding (MCS) schemes used during uplink transmission [START_REF] Stefan | Combined use of reinforcement learning and simulated annealing: algorithms and applications[END_REF]. A fixed TB allocation can decrease front-haul efficiency and increase front-haul uplink latency. Our major contribution in this paper is that we extend the low-latency PONs based mobile front-haul proposal to the multi-wavelength domain (e.g. WDM and TWDM-PON) and try to overcome the latency and the bandwidth efficiency problems we mentioned earlier. To do that, we propose to jointly allocate C-RAN air interface resources and fronthaul uplink resources to the users at the granularity of LTE media access control (MAC) layer sub-frame cycle which known as transmission time interval (TTI) (i.e. one TTI equals 1ms). We formulate the joint radio and fronthaul resource allocation framework as an optimization problem with the objective of finding an optimum or sub-optimum (RBs/TBs) to UE allocation pattern that minimizes total uplink scheduling latency (as well as fronthaul delay) and improves the total system throughput. Due to the complexity of such an optimization problem, because of the contiguity constraint on single-carrier frequency-division multiple access (SC-FDMA) uplink transmission, we introduce a reinforcement learning algorithm to solve the problem and evaluate its performance against some other heuristic approaches The rest of this paper is organized as follows. In section II, we present the system model for multi-wavelengths enabled C-RAN and formulate the uplink resource allocation optimization problem. In Section III, we introduce a solution to our resource allocation optimization problem based on Q-leaning algorithm. In section IV we evaluate the performance of our solution, and in section VI we give the conclusion for our paper.

Introduction

Multi-wavelength enabled C-RAN architecture

The system model considers a C-RAN network consists of M RRUs; each RRU is attached to an optical network unit (ONU) (Fig. 1). The ONUs are aggregated over an optical splitter to a TWDM or WDM optical line terminal (OLT) which is connected directly to a DU unit. The DU and CU are co-located together at the central office and connected to each other via a mid-haul network (e.g.,TDM-PON or Ethernet). The CU system is virtualized into M vBBUs. Each vBBU is assigned a fixed wavelength channel to connect to its associated RRU. Each vBBU has a bandwidth equal to N RBs, and total C-RAN system is designed to serve K active mobile users. We assume that a learning based software agent that coordinates between CU and DU/OLT (assuming a 5G system with dual split as in Fig. 1) is in charge of the scheduling process of uplink air interface and front-haul resources. During the uplink scheduling process, every UEs in the network sends scheduling requests to ONU/ RRU. These requests contain UEs buffer status report (BSR) and channel quality indicator (CQI). The ONU/RRU transmits on single wavelength the UE requests to OLT which passes these requests to the CU unit at the C-RAN center. The scheduling agent at CU utilizes BSR and CQI information to compute the scheduling decision for the radio interface and fronthaul resources (i.e.RB /TBs allocation to UEs) every TTI period.

UE K

The final scheduling decision in form of grant allocations is broadcasted over all wavelength channels of the fronthaul aggregation network to ONUs. Each ONU in the network receives these grant allocations; however, its MAC layer protocol permits only the processing of the allocation associated with the RRU that it is connected to. Finally, the RRU sends the scheduling allocation grants to UEs over the air interface.

Multi-wavelength enabled C-RAN architecture

In C-RAN system described above, we assume that the allocation of air interface resource block (RB) and fronthaul upstream transport block size (TB) to users is done in a slotted scheduling base, with a slot duration equal to one TTI. At each scheduling slot, the RB/TB can be allocated to a one user at most. In order to efficiently utilize RB/TB resources during uplink scheduling while achieving a minimum UE uplink delay in multi-wavelength mobile fronthaul network, we formulate an optimization problem with the objective to minimize the total sum of idle time over the all wavelengths and vBBUs in the network. Fig 2 illustrates the calculation process of sum of wavelength during a TTI duration cycle. In this figure,A &' denotes the j )* incoming scheduling requests processing time on the i )* wavelength channel of fronthaul network; where, j ∈ {∈ {1, 2,3. . . , J} is the index of the request with J as the total number of requests. i ∈ {1, 2,3. . . , M} denotes the index of the wavelength channel with Mas the total number of the wavelengths.B &' denotes the off-scheduling time on the i )* wavelength channel ,and λ & denotes thei )* wavelength channel i ∈ {1, 2,3. . . , M}.Assuming the above notation and referring to Ref [START_REF] Lee | Proportional fair frequency-domain packet scheduling for 3GPP LTE uplink[END_REF] flow-shop scheduling problem, the total sum of ide time as can be written as illustrated in Fig. 2.

𝑦 D,E,F T ERS ≤ 1 , ∀𝑛 ∈ 1,2 … 𝑁 , ∀𝑖 ∈ 1,2 … 𝑀 (1) 𝑦 D,E,F Q FRS T ERS U DRS ≤ 𝑀 * 𝑁 (2) 𝑦 D,E,F Q FRF Z U DRS * 𝐵 D,E,F ≤ 𝐵 D,E,\]^ , ∀𝑘 ∈ 𝐾 * ( 3
)
where 𝐵 D,E,F is the rate (in bytes) that user 𝑘 obtains if RB 𝑛 _ is assigned to it, and 𝐵 D,E,\]^ is the maximum number of bytes requested by user 𝑘. 𝐾 * is a set contains the UE who has the highest rate over the RB 𝑛 _ .

𝑛 _ = 𝑎𝑟𝑔𝑚𝑎𝑥 F∈Q * 𝑊 E,F Q FRS T TRS 𝑦 E,F (4)
where

𝑊 E,F = g h,i,
The constraint in Eq. ( 1) is used to limit the allocation of each RB/TB to one user during a single TTI period to avoid the interference (Note: LTE does not allow the allocation of less than one RB to UE). The constraint in Eq. ( 2) is used to limit the total number of scheduled RBs over all wavelength not to exceed the total capacity of the system (i.e. system stability constraint). The constraint in Eq. ( 3) is used to avoid overallocating of RBs/ TBs to the UEs (i.e. ensure that the agent will not assign transport blocks more what the users have requested). The constraint in Eq. ( 4) is used to ensure that each RB is allocated to the UE that maximizes the total C-RAN PF metric (i.e. ensure each RB is allocated to UE that achieves highest CQI index or SNR value over that specific RB). The constraint in Eq. ( 5) is SC-FDMA contiguity constraint which is used to ensure all of the allocated RBs to a single UE are adjacent to each other in frequency domain.

The optimization problem we describe above belongs to the class of NP-hard problems due to the constraint given in Eq. ( 5) (the proof of the NP-hard can be found in [START_REF] Da Mata | Resource allocation for the LTE uplink based on Genetic Algorithms in mixed traffic environments[END_REF]). Therefore, classical optimization methods such as branch and bound methods can only be used to solve the small-scale scheduling problems, for large-scale and complex scheduling problem heuristic approaches or reinforcement learning can be used. Some heuristic approaches such as genetic algorithms [START_REF] Khdhir | Tabu Approach for Adaptive Resource Allocation and Selection Carrier Aggregation in LTE-Advanced Network[END_REF] and Tabu search [START_REF] Watkins | Learning from delayed rewards. Diss. King's College[END_REF] have been already evaluated for uplink scheduling problem for disturbed RAN (D-C-RAN case). In this paper a reinforcement learning based solution is presented and its performance is compared with the above-mentioned heuristic methods under C-RAN architecture The resource allocation optimization problem in C-RAN is a complex scheduling problem that fits RL context. reinforcement learning, mainly Q-Learning algorithm [START_REF] Gao | Q-learning-based power control for LTE enterprise femtocell networks[END_REF], has shown positive results in solving some resource allocation problems similar to our problem (e.g. [START_REF] Ye | Deep reinforcement learning for resource allocation in V2V communications[END_REF], and [START_REF]ns-3.25[END_REF]). QL is an iterative model that can be defined by sets of states, actions and a reward function that produce a reward for each state-action interaction. As shown in Fig. 3, at each iteration the learning agent (TB/RB assignment agent) observes the environment state 𝑠 | ∈ 𝑆, then; applies an action 𝑎 } ∈ 𝐴 to the environment according to the strategy π. The environment transits into a new state 𝑠 |yS ∈ 𝑆 producing a reward signal 𝑟𝑤 | ∈ 𝑅 to the agent. The agent updates its strategy based on the new state and the received reward. The basic goal of the agent is to choose the best action for each state that maximizes the cumulative reward as

𝑄 𝑠 | , 𝑎 | : = 𝐸 𝛾 | * 𝑟𝑤 | 𝑠 | , 𝜋 … |𝑠 ‡ = 𝑠 | R𝟎 ( 6
)
where γ is a discount factor that reflects the significance of the upcoming reward relative to the current reward. When the selected action a is the optimal one 𝜋 * (𝑠) , 𝑄 𝑠 | , 𝑎 | is the maximum of the state. The update formula is given as

𝑄 |yS 𝑠 | , 𝑎 | ≔ 1 -𝛼 𝑄 | 𝑠 | , 𝑎 | + 𝛼 𝑟𝑤 | 𝑠 | , 𝑎 | + 𝛾 𝑚𝑎𝑥 ] }•m Ž• 𝑄 | 𝑠 |yS , 𝑎 |yS (7)
where 𝛼 ∈ [0,1] is the learning rate that balances new information against previous knowledge. The Q-learning algorithm does not determine how the actions can be chosen in each state. To determine that, this paper considers 𝜖 -greedy policy, in this policy 𝜖 is the exploration rate which is used to choose a random action 𝑎 } ∈ 𝐴 with a probability falling between 0 and 1 (i.e. 𝜖 : 0 < 𝜖 < 1) this known as exploration, in contrast of choosing an action based on previous experience (i.e. selecting action with 1-ϵ probability), which known as exploitation. The exploration rate decays over the course of the learning until it reaches the minimum value.

The uplink resource allocation scheduling problem in reinforcement learning context

To write the uplink resource scheduling problem we described earlier in reinforcement learning context we can define the states, actions and reward function as follow:

1. State: S: {s S , s -, s ˜, … … … , s ™ }: as a combination of the total sum of idle time over the all wavelength channels w ™ and the total C-RAN system PF gain G ™ calculated the state transition (i.e. s ™ =(G ™ , w ™ )). G ™ and w ™ can be written as :

𝐺 | = 𝑊 E,F,| Q FRS T TRS 𝑦 E,F,| (8)
, and The optimization objective is to find the optimal/suboptimal RB to UE allocation pattern that maximizes the system PF gain and RB/TB to wavelength scheduling strategy that gives a minimum sum of idle time over the wavelength channels of the fronthaul network. Later on, this allocation pattern and scheduling stagey will be used to update the allocation of RBs to UEs and TBs to ONU/RRUs every TTI scheduling cycle. The complete algorithm for the scheduling is summarized by Algorithm 1.

𝑤 | = 𝑃 D,| (9)

Performance Evaluation Results

We evaluate the performance of our uplink scheduling algorithm in NS-3 simulator [14]. Since NS-3 does not support C-RAN and BBU virtualization, we use eNodeBs to play the role of vBBUs in our simulations. In these simulations, we consider a C-RAN network with 4 RRU connected to over 4 WDM wavelength channels to 4 vBBUs resides in the cloud center. We assume different distances between each RRU/ONU and CU unit at the cloud center as follows: 5,10,15 and 20km. We consider urban propagation environment, where UEs are uniformly distributed in the network, and experience

Algorithm 1

Input: The initial UE to RB/TB allocation strategy (i.e. 𝐺 ‡ , 𝑤 ‡ ) Output: The optimal allocation strategy. different MCS indexes ranging between 2~28. We assume adaptive modulation schemes for the uplink transmission, in which the C-RAN system senses the UEs channel quality condition and accordingly chooses the modulation scheme and the quantization resolution to be used. In this paper, we adopt three modulation schemes namely; QPSK, 16-QAM and 64-QAM, each with different quantization resolution bits as follow, 8 bit with 64QAM, 6 bit with 16-QAM and 4 bit with QPSK. We consider a random walk mobility model with an average UE movement speed equal 3km/h. For the traffic model, we assume a full buffer model with UE traffic load equal to 640kbps. The overall system parameters used during the simulation are summarized in table 1. For the Q-learning scheduling algorithms, we set the following parameters: α = 0.5 and γ = 0.5. We use ϵ -greedy as action selection policy with ϵ = 0.90 at the beginning and decays until became 0.010 when enough number of the episodes have been explored. The complete parameters and settings used for the scheduling algorithms are given in table 2. We choose the total system throughput, total scheduling time, and the speed of convergence as performance evaluation metrics. To evaluate these metrics, we run multiple. Fig 4 shows the overall performance comparisons.

Initialize 𝑄 (… § ,] § ) ← 𝑄 (… § R(© § ,ª § ),] § R ‡) , 𝜏 =0,
Fig. 4(a) shows the achieved system throughput by each scheduling algorithm plotted versus the number of the active users during the simulation. From this figure, we can notice that the highest system throughput is achieved by RL algorithm followed by GA whereas TS algorithm achieves the lowest system throughput. We explain RL's superior performance by its ability to produce allocation patterns very close to the optimal as it does not require a long time to simulate the optimization solver as opposed to TS and GA algorithms (see Fig 4(c)).

Fig. 4(b) shows a comparison of the scheduling time consumed by each algorithm. As we can see, the RL algorithm also attains the lowest scheduling time compared to TS and GA algorithms. However, this time TS outperforms GA and achieves lower scheduling time. All of the three algorithms show a total scheduling time of less than 1ms (TTI period) when the number of active users in the system was less than 150 UEs. However, the scheduling time of GA exceeded 1ms when the number of users was 200 active UEs.

Fig 4(c). shows a performance comparison of the three algorithms in term of the speed of convergence considering the objective function given in equation 1. As we can see RL algorithm achieves the fastest speed of convergence on the objective function compared to GA and TS algorithms. In other words, RL algorithm converges to the minimum sum of idle time in the first 50 iterations while GA algorithm converges in about 80 iterations and TS algorithm converges in about 60 iterations; however, the convergence of TS is slightly unstable compared to RL and GA. Fig 4(d). compares the performance of the total uplink delay for static RB to TB mapping, dynamic RB to TB mapping [START_REF] Tiwari | LONG TERM EVOLUTION (LTE) PROTOCOL Verification of MAC Scheduling algorithms in NetSim[END_REF] and our new adaptive-TB allocation method. From this figure we can see that our new adaptive-TB allocation method achieves the lowest total uplink scheduling delay in comparison with static and dynamic RB to TB mapping proposals. The reason behind the improved delay performance achieved by adaptive-TB allocation method is the efficient utilization of fronthaul uplink resources (see Fig 5(a) and(b)). This is due to the fact that adaptive-TB method allocates an adaptive fronthaul TB size equal to the actual RB size calculated by the scheduling algorithm (RL) based on UEs traffic load and channel condition. This method can greatly reduce the capacity required on fronthaul as opposed to static and dynamic RB to TB mapping methods which assume fixed fronthaul TB size for every RB.

Conclusion

In this paper a reinforcement learning based scheduling algorithm is proposed to address the resource allocation optimization problem for multi-wavelength Enabled-C-RAN architecture. The performance of the algorithm is validated with simulation and compared with two other heuristic approaches. The simulation results have shown that RL based scheduling is the most promising approach, as it outperforms the two other heuristic methods in all performance evaluation metrics, offerings the highest system throughput, lowest scheduling time and total uplink scheduling latency. The results have also shown that adaptive allocation of fronthaul transport resources with RL based scheduling which rely on UE traffic load and actual radio condition can greatly enhance the C-RAN system performance in terms of uplink scheduling delay and fronthaul efficiency.

Fig. 1 .

 1 Fig. 1. Multi-Wavelength-Enabled C-RAN Architecture.

Fig. 2 .

 2 Fig. 2. The of sum of idle time for the wavelengths.In order to describe our problem define the following notations: 𝐾 (𝑘 = 1, 3. . . , 𝐾) as the number of active user, 𝑀 (i= 1, 3 … 𝑀) as the number of wavelengths/vBBUs in the C-RAN system, 𝑁 (𝑛 = 1,3, … … 𝑁) as the number of RBs in C-RAN network , P & as the sum of idle time on wavelength i when assigning TB/RB n to UE requests k and 𝑦 D,E,F, 𝑦 D,E,F ∈ (0,1) as a selection variable that indicates whether the RB/TB n on wavelength i is allocated to UE k or not ( y &,I,J == 1 if TB J is allocated to UE k and 0 otherwise). Given the above notations our optimization's objective function can be written as:

Fig. 3 .

 3 Fig. 3. Reinforcement learning elements

U DRS 2 . 3 .

 23 Action: 𝐴: {𝑎 S , 𝑎 -, 𝑎 ˜, … … … , 𝑎 | }: as the permutation of RBs allocation strategy to UEs ,and the permutation of sequencing order of the allocated RBs over the wavelength channel TBs as well as the permutation of the wavelength channels order. Reward function: 𝑅: {𝑟𝑤 … m, ] m, , , 𝑟𝑤 … Ÿ, ] Ÿ, , , 𝑟𝑤 ˜, … , 𝑟𝑤 … }, ] } } as a function that rewards the unity value if the action has taken by the agent increases the total system PF gain and decreases the total sum of idle time over the past episode, otherwise it rewards the value (-0.1), this function is written as follow 𝑟𝑤 … }, ] } = 1 𝑖𝑓 𝐺 |yS > 𝐺 | 𝑎𝑛𝑑 𝑤 |yS < 𝑤 | -0.1 𝑂𝑡ℎ𝑒𝑤𝑖𝑠𝑒 (10)

Fig . 4 .Fig . 5

 45 Fig .4. Performance comparison: (a) The achieved throughput ;(b) total scheduling time consumed by each algorithm (The convergence speed) ;(d) comparison of UE total uplink scheduling delay with static RB to TB mapping, dynamic RB to TB mapping and adaptive TB allocation methods.

Table 1 .

 1 Simulation Parameters

	Parameter	Value	Parameter	Value
	Simulation	1000TTI	Channel band-	10 MHZ
	length		width
	Link adaptation QPSK,16QAM,	Number of	400 RB
		64QAM	RB(per TTI)
	Propagation	COST 231	Maximum
	model		Number of us-
			ers

Table 2 .

 2 Algorithms Settings

	Parameter	Value
	Number of Iteration	200
	RL Discount Factor	0.5
	RL Learning Rate	0.5
	RL Exploration Rate	0.90
	RL Minimum Exploration Rate	0.010
	RL Exploration Rate Decay	0.99
	GA Parameters	Same as [9]

ACKNOWLEDGMENT

This work was supported by the National Natural Science Foundation of China (NSFC) (61431009, 61371082, and 61521062) and the National Science and Technology Project -China (2015ZX03001021).