Zhihao Chen
email: zhihao.chen@groupe-esigelec.org

Benoit Decoux
email: benoit.decoux@esigelec.fr

Real Time Object Detection, Tracking, and Distance and Motion Estimation based on Deep Learning: Application to Smart Mobility

Keywords: Pedestrian Detection, Pattern Recognition, Object Detection, Tracking, YOLO V3, SSD, Deep Learning

In this paper, we will introduce our object detection, localization and tracking system for smart mobility applications like traffic road and railway environment. Firstly, an object detection and tracking approach was firstly carried out within two deep learning approaches: You Only Look Once (YOLO) V3 and Single Shot Detector (SSD). A comparison between the two methods allows us to identify their applicability in the traffic environment. Both the performances in road and in railway environments were evaluated. Secondly, object distance estimation based on Monodepth algorithm was developed. This model is trained on stereo images dataset but its inference uses monocular images. As the output data, we have a disparity map that we combine with the output of object detection. To validate our approach, we have tested two models with different backbones including VGG and ResNet used with two datasets : Cityscape and KITTI. As the last step of our approach, we have developed a new method-based SSD to analyse the behavior of pedestrian and vehicle by tracking their movements even in case of no detection on some images of a sequence. We have developed an algorithm based on the coordinates of the output bounding boxes of the SSD algorithm. The objective is to determine if the trajectory of a pedestrian or vehicle can lead to a dangerous situations. The whole of development is tested in real vehicle traffic conditions in Rouen city center, and with videos taken by embedded cameras along the Rouen tramway.

I. INTRODUCTION

The work presented in this paper is a part of ADAPT 1 project (Assistive Devices for empowering disAbled Peo- 1 This work is carried out as part of the INTERREG VA FMA ADAPT project "Assistive Devices for empowering disAbled People through robotic Technologies" http://adapt-project.com/index.php. The Interreg FCE Programme is a European Territorial Cooperation programme that aims to fund high quality cooperation projects in the Channel border region between France and England. The Programme is funded by the European Regional Development Fund (ERDF) ple through robotic Technologies) which focuses on smart and connected wheelchair to compensate for user disabilities through driving assistance technologies. The work presented in this paper is based on pedestrian detection is part of ADAPT project. In general, ADAS is used to improve safety and comfort in vehicles. ADAS is based on the combination of sensors (RADAR, LIDAR, cameras, etc.) and algorithms that ensure vehicle, driver, passenger and pedestrian safety based on different parameters such as traffic, weather etc. Here in this project, ADAS aims to detect object like pedestrian, vehicles, etc. Our contribution aims on the development of perception system based object detection-based deep learning with different approaches such as YOLO V3, and SSD.

The principal objective is to apply robust approaches to detect object and avoid traffic accidents on real time system. The detection of object and estimation of its distance are the most important tasks to determinate the object's position. To carry out these tasks, we had, in old research work, evaluated traditional image processing approaches like HOG and DPM approaches. To get a more robust result, we have applied deep learning based object detection and distance estimation approaches, and then combined them together to get the final result. Results show that the object are detected with high accuracy and is considered as satisfied result. Furthermore, the object distance estimation is carried out with some errors which need to be improved. Our training platform is a comuting cluster with Nvidia P100 and K80 GPUs (Graphical Processing Units), and inference platform is portable computer with Nvidia GTX960m, which has about 1.5 TFLOPS of calculation capacity.

This paper is organized as follows: Section 1 introduces the motivation of the paper. Section 2 presents the state of the art about object/pedestrian detection based deep learning. Section 3 presents the object detection and tracking based on YOLO V3 and SSD deep learning algorithms. Section 4 illustrates the object distance estimation approach based on deep learning. The pedestrian behavior analysis will be presented in Section 5. Results are presented through the different section 3, 4, and 5. Finally, in Section 6, we will conclude this paper.

II. RELATED WORK

Object detection is a key problem in computer vision. It has two big challenges: detection of objects in images and estimation of their position, and estimation of their class. Over the past few years, several methods based on Convolutional Neural Network (CNN) have been proposed to tackle this problem, with great success. Those methods can be divided into two main categories : one-stage methods, which provide estimation of position and estimation of classes in one step, and two-stage methods, which first detect regions of the images where object could be present, and then apply these regions to a classifier.

Two of the most popular methods in the one-stage category are Single-Shot Detector (SSD) [START_REF] Liu | SSD: Single Shot MultiBox Detector[END_REF] and You Only Look Once (YOLO) [START_REF] Redmon | YOLOv3: An Incremental Improvement[END_REF]. They both provide as outputs the probability of each possible classes, not for the whole image but for a set of regularly spaced positions and for different scales and aspect ratio of rectangles called boxes. One of the two-stage methods giving the best results is RCNN (Region-proposal Convolutional Neural Network) [START_REF] Girshick | Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation[END_REF] and its improved versions [START_REF] Girshick | Fast R-CNN[END_REF] [5], base on two independent neural networks: a region-proposal network and a classification network. Those three methods have similar performance from the point of view of the mean Average Precision (mAP), a criterion which quantifies quality of detections (proportion of correct detections) as a function of the recall, which is the proportion of objects that are detected. However, two independent networks make the prediction slow on the most embedded computation platforms. In onestage detection architectures, the classification is made on fixed size and fixed number of bounding boxes at predefined layers, then tune the localization of detected objects by regression. Those architectures are generally faster than two-stage ones, with similar performance [START_REF] Liu | SSD: Single Shot MultiBox Detector[END_REF].

Considering that our application is based on sequential media (video for example), loss of too small objects and imprecision on position are less critical than the real-time performance. Some trade-off like increasing number of bounding box and resolution of inference also could be adopted to reduce it. Based on the above analysis, we choose two widely applied one-stage object detection approaches: SSD [START_REF] Liu | SSD: Single Shot MultiBox Detector[END_REF] and YOLO V3 [START_REF] Redmon | YOLOv3: An Incremental Improvement[END_REF] to evaluate.

For estimation of distance from objects to the vehicle, many sensors are available : laser, ultrasonic, infrared ray, etc. Those kinds of sensors are widely used in civil field. But the use of multiple types of sensors make the system more complex, and more expensive. Since we plan to use a low cost monocular camera for object detection, it seems valuable to estimate the distance with the same sensor as for object detection, that is a monocular camera. If the datasets used for object detection had the distance from object as ground-truth information, information of distance could be directly learned by adding a regression output to the CNN, but it is not the case. One solution is to use an unsupervised approach to estimate depth for monocular images, like the one called Monodepth [START_REF] Godard | Unsupervised Monocular Depth Estimation with Left-Right Consistency[END_REF]. This model is trained on stereo images but infers disparity maps from monocular images, so it is intersting for our needs.

III. OBJECT DETECTION AND TRACKING BASED ON DEEP LEARNING A. Choice of the Approach to be Developed

In order to analyze the performance of object-detection models, speed and accuracy are important parameters to consider. Comparison of the performance of different approaches must be done carefully because the experiments are not always done under the same conditions. Indeed, there are several parameters that can vary from one experience to another. This is for example the learning database which is used, the resolution of the input image or the threshold of the Intersection-over-Union criterion (IoU, which is used to evaluate the quality of the detections) etc. We have analyzed the properties of three well known approaches for object detection : SSD [1] YOLO V3 [START_REF] Redmon | YOLOv3: An Incremental Improvement[END_REF], and Faster-RCNN [START_REF] He | Deep Residual Learning for Image Recognition[END_REF]. Faster RCNN seems to be more accurate than SSD and YOLO V3. On the other hand, it is slower. Which means that if the task requires high quality precision, RCNN Faster is the right solution. On the other hand, if it is the speed that is essential, the RCNN Faster is not the best candidate. YOLO V3 is faster than SSD and Faster RCNN. So, if speed is the main criterion of choice, YOLO V3 would to be the best candidate. If we want at the same time a good accuracy and a good speed, SSD could be a better solution, as fastest approach after YOLO V3, and its accuracy is almost as good as Faster-RCNN. So it represents a good compromise between accuracy and speed.

B. SSD vs YOLO V3

During the course, YOLO V3, the newer version of YOLO is released. According to the literature, it would have better performance than the SSD in terms of speed and accuracy. So we decided to make the inference of SSD and YOLO V3 on the same environment to compare them. Table III shows the evaluation environment used. The model of the SSD that we evaluated was trained with the PascalVOC learning base. While the YOLO V3 model we evaluated was trained with the Coco learning base. We conducted the evaluation of all the classes of each learning base. This is the reason why instead of presenting the information of the AP(Average Precision) which is relative to the average accuracy of a class, we presented the information of the mAP(mean Average Precision) which represents the average of the APs of all classes in the training base. Table ?? presents the results of the evaluation and the inference of YOLO V3 and SSD. According to the results of our tests, the YOLO V3 gets better performance. The YOLO V3 at the resolution 416*416 has the similar inference FPS as the SSD at the input resolution 300*300 but the precision is markedly higher. On the other word, with the parameter of the same precision, the YOLO V3 is quicker than SSD.

C. SSD Algorithm

We made inferences using the Keras and Tensorflow libraries. Below are the characteristics of the inferences made with the SSD approach.

• VGG (SSD, 300x300, 120000 iterations): model trained with an SSD architecture having a VGG base. Model taking in pictures of size 300x300. This is the model on which the inference was made. • SSD: name of the approach used to perform the inference We obtain a processing time of 6.2fps and a confidence rate approaching 100 % in the majority of case.

The purpose of this method is to determine the performance of the model trained. This is the VGG (VOC0712, SSD 300x300, 120000 iterations), model presented above. There are two main parameters that come into play in this context. This is the accuracy and the rate of callback. The figure below [START_REF] Geiger | Vision meets robotics:The KITTI dataset[END_REF] presents the operations to be applied in order to obtain these two parameters.

This evaluation allows us to know how much the model we use is accurate with respect to each class in the learning base on which it was trained. The model we used was trained on the PascalVOC learning base. But we did not evaluate the 20 classes of this learning base. We evaluated just six classes that we considered likely to end up in the tramway environment. These include the following classes: person, car, motorcycle, bike, bus and dog. The graphs in the figure above show the accuracy according to the recall rate. This precision is calculated relative to a parameter named confidence rate which is set at 0.5 % in our case. If we were at the top right corner of the graph, that would mean that we have an average accuracy of 100%. Unfortunately, this is not the case but we are very close to it. In fact, of the six classes evaluated, the lowest rate is that of the person class and it corresponds to an AP of 84%. Apart from that, all other results are around 90% AP. The 84% AP rate for the person class is a good result, although it is slightly lower than the rest of the classes. This can be explained by the fact that the person class is more complicated to detect than a class like the car class for example. Indeed, because of its movement during the detection of a person, one can not detect in some cases the totality of the person. That is to say, sometimes there may be missing in the box delimiting the perimeter of the detected object an arm or a leg. This will result in the person being detected but their confidence will be weakened. A car during its detection does not lose a tire or a door because it is moving. When it is detected, it is detected completely. The table summarizes the AP of each of the evaluated classes. In view of this table and the previous analysis, we can conclude that the model VGG (SSD 300x300, iter: 120000) allows to have very good results in terms of specify. As for speed, with a GTX 660 we are at 6.2 fps. We could improve this performance by using much better GPUs.

D. YOLO V3 Algorithm

YOLO V3 is an improvement version of the former YOLO and YOLO 9000, and adopts some state-of-the-art architectures like Residual Network. The implementation which we use is its original, on C language. This implementation doesn't depend on other high level deep learning libraries like Tensorflow or Caffe and it could slightly improve its efficiency.

IV. OBJECT DISTANCE ESTIMATION

Monodepth [START_REF] Cordts | The Cityscapes Dataset for Semantic Urban Scene Understanding[END_REF] is an unsupervised CNN based approach for distance estimation. The model is trained on stereo images and makes inference on monocular images. The output of this approach is a disparity map. The authors provide the source code with many models, trained on different datasets (including Cityscape [START_REF] Cordts | The Cityscapes Dataset for Semantic Urban Scene Understanding[END_REF] and Kitty [START_REF] Geiger | Vision meets robotics:The KITTI dataset[END_REF]) and with different backbones (including VGG and ResNet). We have tested two models on some images, with different backbones (VGG and ResNet) and with different datasets (Cityscape and Kitty). The disparity values represent relative distances because the training images and inference images are taken by cameras of different focal distance. To get the true distancewe should use calibration images to calculate the coefficient of the baseline and focal distance. Figure 4 shows the resulting disparity maps for one example of image. These output disparity maps show the influence of the dataset and the backbone. A suitable training dataset can improve the precision of the disparity maps: the model trained by Cityscape gets better result for an input image of city scene. 5 shows that the backbone network has also influence on the result: for this example image, VGG has a better performance than ResNet, even though ResNet generally gets better precision for classification in many works [START_REF] He | Deep Residual Learning for Image Recognition[END_REF].

The disparity maps are not sufficient for our objective, as they are not related to the objects which are present in the images. So we need to combine an object detection approach, which provides estimations of the bounding boxes of objects, and Monodepth which can give distance information inside these bounding boxes. Firstly we apply the input image to the object detection algorithm to get objects bounding boxes, and then compute histogram in the corresponding regions of the disparity map to get the distributions of disparity values of the detected objects. The major distribution interval gives the estimation of distance of the object. Figure 6 shows an example of application of this method on the bounding box of a detected object and the region of the corresponding disparity map.

We have applied this method on several images including real and synthetic ones. The results are difficult to quantify, as there is no dataset available with ground truth for the distance to the objects. Distance estimation gives good results when object are not too numerous on the images. When there are many objects with overlapping bounding boxes, the estimation is distorted, as all pixels in the bounding box are taken into account. Figure 7 shows an example of results with bad distance estimation. This problem could be reduced by applying a semantic segmentation to the images in parallel to the object detection process, to use only the pixels belonging to the objects inside the bounding boxes. Figure 8 shows some examples of correct distance estimation.

V. PEDESTRIAN BEHAVIOR ANALYSIS

For our application, it is very important to be able to give an estimation of the direction of movement and speed of the object in motion. So we need to make a tracking of the detected objects in the images. However, in a video sequence, the confidence score of a detected object can vary to a large extent from image to image, and then can fall below the detection threshold on some images. So it is important to be able to estimate their positions in this case. Below are the steps that allowed us to track down detected objects:

• At time t, make the list of all boxes of the detected objects. • At time t + 1, compare the abscissa of the boxes of each newly detected object with those of the boxes of the detected objects at time t ; associate this box with the nearest one, provided that the difference of abscissa is below a giver threshold. • The boxes of time t + 1 which have a corresponding box at t are marked as tracked. When tracking an object in a video sequence, we can have a loss of detection on some images. In this case, we need to estimate the new position of the object anyway. The objects which are not detected by the object-detection algorithm This means that on the current image, they could not be detected by the SSD but we could estimate their positions using the information of their positions in the previous images. For this, we apply a processing with the following sequence of steps:

• For each tracked object, calculate its speed (in pixels/s) at time t, which is the difference of abscissae of the bounding box centers between positions of this object in images at times t and t-1 (the sign of this difference gives the direction of motion) • In order to attenuate the effect of uncertainty of the estimation of position given by the object-detection algorithm, the final speed is taken as the moving average of the values calculated over a few preceding images • In case of no detection, consider that the object has the same speed as previous calculation, and consider that the dimensions and ordinate of its bounding box are the same as the previously calculated one • Go back to the first step ; in case of no detection after a few iterations of this sequence of steps, exit from this sequence and mark this object as no-tracked Figure 9 shows examples of application of this principle on two images of a sequence. In this paper, we have presented a contribution based three deep learning approaches for object detection, tracking, and distance estimation for smart mobility applications (traffic road and railway). The object detection approach has been well developed by taking into account not only high accuracy for object detection, but real-time applications constraints too. We have developed object detection by both SSD and YOLO V3 algorithm in order to find which algorithm is more adapted for our application. The comparison carried out illustrate that the YOLO V3 is more than SSD algorithm.

The object distance estimation based on monodepth algorithm was developed. The model was trained under stereo images dataset and makes inference on monocular images dataset. As output data, the monodepth algorithm gives a disparity map. We can merge the object detection approach and estimation distance to share the feature extraction layers, which could improve its efficiency. We have validate our approach under different datasets like Cityscape and Kitty, but also in real time within ESIGELEC vehicle in traffic road of Rouen city center. We have also validate the development under railway dataset of the Tramway of Rouen.

As a last contribution in this paper, we have presented a new method based SSD in order to analyze the behavior of object like pedestrian or vehicle. After detecting object with the SSD modified algorithm, we estimate its future position in order to have its direction of movement: pedestrian who want to cross the road, who do not cross the road, who goes through, etc.

We have presented a comparative study between SSD and YOLO V3 algorithm for object detection and tracking. To optimize their performance on low-consumption platform, a rich and suitable dataset could be very important. Change the number of detection class cant get significant improvement.

Finally, the whole development presented in this paper were validated on both real traffic circulation conditions in the city center of Rouen, and through the railway videos acquired from an embedded camera in the tramway of Rouen.

Figure 8

 8 Figure 8 present the results of the inference performed on the SSD and YOLO3.

Fig. 1 .

 1 Fig. 1. SSD inference (left) vs YOLO V3 inference (right).

  • PascalVOC: database dedicated to pedestrian traffic containing 20 classes • nVIDIA GTX 660: graphics card used • 2G: size of the ram used for the graphical calculation

Fig. 2 .

 2 Fig. 2. SSD Inference Result.

Fig. 3 .

 3 Fig. 3. SSD Inference Evaluation.

  Firstly we evaluate its pretrained model which is trained by the author. This model is trained on Pascal VOC + COCO. To get quantifiable evaluation results, we evaluate this model on the dataset Pascal VOC 07 Evaluation, for the class Person. Then, we train our model to figure out if we can improve its performance by training on different dataset. We trained by ourselves 2 models, one is trained on Pascal VOC 07 with one class Person and another is trained on COCO. The training is executed on the cluster of computation MYRIA in the Normandy calculate center CRIANN. The model trained on Pascal VOC 07 takes about 9000 batches and the model trained on COCO takes more than 16000 batches with the batch size of 32. 2 Nvidia P100 can train about 1500 batches per hour and 2 Nvidia K80 can train around 800 batches per hour. To quantify our models performance, we use the same evaluation dataset as the pretrained model, Pascal VOC 07 Evaluation. Beside this quantifiable result, we also make the inference on a video of driver's perspective which is taken by us in the centre and urban of the city Rouen. Those results show two important consequences. 1. a rich dataset is critical for the performance of precision (mAP). The model trained on COCO dataset with 64115 images of persons shows 5.6mAP improvement over the Pascal VOC dataset with 2000 images of persons. 2. The number of classes doesnt significantly influence the speed of inference. The one class model is only about 0.4FPS faster than the 80 classes model. The inference time mainly depends on the quantity of parameters in the neural network, but the suppression of classes has only impact on the number of parameters of the fully connected layer's, which is a small part of the whole model.

Fig. 4 .

 4 Fig. 4. Test of different datasets : disparity maps resulting from application of Monodepth on a scene of sidewalk, with backbone VGG and training on two different datasets : Cityscape (middle) and Kitty (right).

Fig. 5 .

 5 Fig. 5. Test of different backbones : disparity maps resulting from application of Monodepth on a scene of sidewalk, with training on Cityscape dataset and two backbones: VGG (middle) and Resnet (right).

Fig. 6 .

 6 Fig. 6. Combination of the results of object detection and depth estimation : for each detected object, the histogram of estimated disparities is computed. The figure shows an example of bounding box : the corresponding histogram represents the number of disparity values (ordinate), as a function of disparities (abscissa). Those numbers are accumulated on small intervals, and the interval with maximum sum is chosen as the one corresponding to the object in the bounding box.

Fig. 7 .

 7 Fig. 7. Example of results with bad distance estimation.

Fig. 8 .

 8 Fig. 8. Example of correct results of distance estimation. Estimated disparity is superimposed on each detected object.

Fig. 9 .

 9 Fig. 9. Examples of tracked pedestrians and vehicle on two parts of images, with estimated direction of motion and speed (in pixels/s). When tracked, objects are marked with estimated speed (at the bottom left corner of the bounding box) and direction of motion (arrow at the bottom right corner) on the abscissa axis of the image. Objects which are tracked are bounded by two rectangles: one with the color of the estimated class and a black one to denote tracking in action

TABLE II SSD

 II AND YOLO V3 INFERENCE.

		mAP	fps
	SSD	79.5	8.6
	YOLO V3	85.0	11.2

TABLE III AP

 III OF THE EVALUATED CLASSES.

	Classes	AP
	Bike	97
	Bus	91
	Car	89
	Dog	99
	Motorbike	99
	Person	84

ACKNOWLEDGMENT

This research is supported by ADAPT Project (co-financed by the European Regional Development Fund within the framework of the INTERREG VA France (Channel) England program). Many thanks to Segula company for its contribution in the project and to the engineers of Autonomous Navigation Laboratory of IRSEEM for their support in testing phase.


R Khemmar

M Gouveia

B Decoux

J Y Ertaud

Real Time Pedestrian and Object Detection and Tracking-based Deep Learning: Application to Drone Visual Tracking

Keywords: Object detection, object recognition, visual tracking, tracking, pedestrian detection, deep learning, visual servoing, HOG, DPM

This work aims to show the new approaches in embedded vision dedicated to object detection and tracking for drone visual control. Object/Pedestrian detection has been carried out through two methods: 1. Classical image processing approach through improved Histogram Oriented Gradient (HOG) and Deformable Part Model (DPM) based detection and pattern recognition methods. In this step, we present our improved HOG/DPM approach allowing the detection of a target object in real time. The developed approach allows us not only to detect the object (pedestrian) but also to estimates the distance between the target and the drone. 2. Object/Pedestrian detection-based Deep Learning approach. The target position estimation has been carried out within image analysis. After this, the system sends instruction to the drone engine in order to correct its position and to track target. For this visual servoing, we have applied 1 This work is carried out as part of the INTERREG VA FMA ADAPT project "Assistive Devices for empowering disAbled People through robotic Technologies" http://adaptproject.com/index.php. The Interreg FCE Programme is a European Territorial Cooperation programme that aims to fund high quality our improved HOG approach and implemented two kinds of PID controllers. The platform has been validated under different scenarios by comparing measured data to ground truth data given by the drone GPS. Several tests which were ca 1 rried out at ESIGELEC car park and Rouen city center validate the developed platform.

INTRODUCTION

The works presented in this paper are a part of ADAPT 1 project (Assistive Devices for empowering disAbled People through robotic Technologies) which cooperation projects in the Channel border region between France and England. The Programme is funded by the European Regional Development Fund (ERDF).

focuses on smart and connected wheelchair to compensate for user disabilities through driving assistance technologies. One of the objectives of the project is to develop an Advanced Driver-Assistance System (ADAS) platform for object detection, recognition, and tracking for wheelchair applications (object detection, obstacle avoidance, etc.). The work presented in this paper is related to object/pedestrian detection. In general, ADAS is used to improve safety and comfort in vehicles. ADAS is based on the combination of sensors (RADAR, LIDAR, cameras, etc.) and algorithms that ensure safety of vehicle, driver, passenger and pedestrian based on different parameters such as traffic, weather, etc. [START_REF] Yan | Robust Multi-Resolution Pedestrian Detection in Traffic Scenes[END_REF]. Here in this project, ADAS aims to detect pedestrian. Our contribution aims to develop a perception system based on object detection with different approaches such as HOG, DPM, and Deep Learning. This paper is organized as follows: Section 1 introduces the motivation of the paper. Section 2 presents the state of the art about object detection/tracking and visual control. Section 3 presents a comparison between the adopted object detection algorithms and some results obtained. Section 4 illustrates our improved HOG/DPM approach applied to object/pedestrian detection and tracking. In the same section, we present an innovative solution to estimate the distance separating objects to the vehicle or to the drone. The visual control system-based multi approach controller will be presented in Section 5. Finally, in Section 6, we will conclude this paper.

STATE OF THE ART AND RELATED WORK State of the Art

Object detection is a key problem in computer vision with several applications like automotive, manufacturing industry, mobile robotics, assisted living, etc. Pedestrian detection is a particular case of object detection that can improve road security and is considered as an important ADAS component in the autonomous vehicle. In [START_REF] Yan | The fastest deformable part model for object detection[END_REF], a very in-depth state of the art for pedestrian detection is presented. Three main contributions are developed by the authors: 1. Study of the statistics of the size, position, and occlusion patterns of pedestrians in urban scenes in a large pedestrian detection image dataset, 2. A refined per frame evaluation methodology that allows to carry out informative comparisons, including measuring performance in relation to scale and occlusion, and 3. Evaluation of 16 pre-trained state of the art detectors [START_REF] Yan | The fastest deformable part model for object detection[END_REF]. As a main conclusion of the study, detection is disappointing at low resolutions and for partially occluded pedestrians. In [START_REF] Cho | Real-time Pedestrian Detection with Deformable Part Models[END_REF], a real-time pedestrian detection with DPM algorithm applied to automotive application is presented. The system is based on a multiresolution pedestrian model and shows superior detection performance than classical DPM approaches in the detection of small pixel-sized pedestrians. The system was evaluated with the Caltech Pedestrian benchmark [START_REF]Caltech Pedestrian Detection Benchmark Homepage[END_REF], which is the largest public pedestrian database. The practicality of the system is demonstrated by a series of use case experiments that uses Caltech video database. The discriminatively trained, multiresolution DPM is presented as an algorithm between generative and discriminative model [3][7]. The algorithm has different steps: building a pyramid of images at different scales, using several filters and part filters to get responses. The algorithm combines these different responses in a starlike model then uses a cost function, and trains classifiers by Support Vector Machine (SVM) classifier. This algorithm is still a widely used algorithm particularly in combination with DPM [START_REF] Dollar | Pedestrian Detection: An Evaluation of the State of the Art[END_REF]. As another method of object detection, the Integral Channel Features (ICF) [START_REF] Cho | Real-time Pedestrian Detection with Deformable Part Models[END_REF], can find a combination of multiple registered image channels, which are computed by linear and nonlinear transformations [START_REF] Lowe | Distinct Image Features from Scale-Invariant Keypoints[END_REF]. Integrating some features like HOG and do a training by AdaBoost in a cascade way can lead to pedestrian detection with good accuracy [START_REF] Lowe | Distinct Image Features from Scale-Invariant Keypoints[END_REF]. The sliding window methods (also called pyramid methods) are used in object detection with a high cost of detection time. In a recent work, proposing high-recall important regions is widely used [START_REF]KLT Feature Tracker[END_REF]. In another way, the approaches based on Deep Learning, also called Convolutional Neural Networks (CNN), become very successful for feature extraction in the image classification task [START_REF] Girshick | Fast R-CNN[END_REF] [START_REF] Ren | Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks[END_REF]. Rich Feature Hierarchies for Convolutional Neural Networks (RCNN) model [START_REF]Object Detection Homepage[END_REF], that combines CNN and selective search can be taken as an example. This algorithm has made a huge progress on object detection task like PASCAL VOC. It will be presented in the next section.

Related Work

In the literature, for similar tasks, several approaches have been used for object detection and pattern recognition, such as HOG/DPM, KLT/RMR (Kanade-Lucas-Tomasi/Robust Multiresolution Estimation of Parametric Motion Models) and Deep Learning [START_REF] Cho | Real-time Pedestrian Detection with Deformable Part Models[END_REF]. For example, in Google Robot's Project [START_REF] Gershgon | Google's Robot Are Learning How to Pick Things Up[END_REF], a deep learning model is applied to articulated robot arm for the picking up of objects. In Kitti Vision Benchmark Suite Project (KIT and Toyota Technological Institute) [START_REF] Geiger | Object Detection Evaluation. The Kitti Vision Benchmark Suite[END_REF], an object detection and orientation estimation benchmark is carried out. The system allows not only localization of objects in 2D, but also estimation of their orientation in 3D [START_REF] Geiger | Object Detection Evaluation. The Kitti Vision Benchmark Suite[END_REF]. In [START_REF] Forson | Understanding SSD MultiBox-Real Time Object Detection in Deep Learning[END_REF], an example of end-to-end object detection from Microsoft is described. For the task of detecting objects in images, recent methods based on convolutional neural networks (CNN, Deep-Learning) like SSD [START_REF] Geiger | Object Detection Evaluation. The Kitti Vision Benchmark Suite[END_REF][20] allow detection of multiple objects in images with high accuracy and in real-time. Furthermore, the SSD model is monolithic and relatively simple compared to other models, making it easier to use for various applications. For the task of grasping objects, people from the AI-Research of Google have recently used Deep Learning to learn hand-eye coordination for robotic grasping [START_REF] Levine | Learning Hand-Eye Coordination for Robotic Grasping with DeepLearning and Large-Scale Data Collection[END_REF]. The experimental evaluation of the method demonstrates that it achieves effective real-time control, and it can successfully grasp new objects, and correct mistakes by continuous servoing (control). In many applications of detection of objects like pedestrian, cyclists and cars, it is important to estimate their 3D orientation. In outdoor environments, solutions based on Deep Learning have been recently shown to outperform other monocular state-of-the-art approaches for detecting cars and estimating their 3D orientation [START_REF] Chabot | Deep MANTA: A Coarse-to-fine Many-Task Network for joint 2D and 3D vehicle analysis from monocular image[END_REF] [START_REF] Mousavian | 3D Bounding Box Estimation Using Deep Learning and Geometry[END_REF]. In indoor environments, it has been shown that using synthetic 3D models of objects to be detected in the learning process of a CNN can simplify it [START_REF] Mousavian | 3D Bounding Box Estimation Using Deep Learning and Geometry[END_REF]. In [START_REF] Georgakis | Synthesizing Training Data for Object Detection in Indoor Scenes[END_REF] and [START_REF] Dollar | Pedestrian Detection: A Benchmark[END_REF], we can find an evaluation of the state of the art object (pedestrian) detection approaches based on HOG/DPM. In [START_REF] Louvat | Analyse d'image pour l'asservissement d'une camera embarquée dans un drone[END_REF], B. Louvat et al. have presented a double (cascade) controller for drone-embedded camera. The system is based on two aspects: target position estimation-based KLT/RMR approaches and control law for the target tracking. The developed platform was validated on real scenarios like house tracking. In [START_REF] Hérissé | Asservissement et navigation autonome d'un drone en environnement incertain par flot optique[END_REF], B. Hérissé has developed in his PhD thesis an autonomous navigation system for a drone in an unknown environment based on optical flow algorithms. Optical flow provides information on velocity of the vehicle and proximity of obstacles. Two contributions were presented: automatic landing on a static or mobile platforms and field following with obstacle avoidance. All algorithms have been tested on a quadrotor UAV built at CEA LIST laboratory.

OBJECT DETECTION APPROACH

In order to identify the most suitable approach to our object and/or pedestrian detection application for the autonomous vehicle (a drone in this study), we need to establish the feasibility of several scientific concepts in pattern recognition approaches such as the KLT/RMR, SIFT/SURF, HOG/DPM but especially recent approaches of the artificial intelligence field, such as Deep-Learning. In this paper, we will focus on object detection and tracking-based through improved HOG/DPM approaches.

Classical Approaches

We have started out our experimentations by implementing point of interest approaches like Scale Invariant Feature Transform (SIFT) [START_REF] Lowe | Distinct Image Features from Scale-Invariant Keypoints[END_REF] and Speeded-Up Robust Features (SURF) [START_REF] Bay | SURF: Speeded Up Robust Features[END_REF], based on descriptors that are very powerful to find matching between images and to detect objects. These methods allow the extraction of visual features which are invariant to scale, rotation and illumination. Despite their robustness to changing perspectives and lighting conditions, we found that the approach is not robust for object/pedestrian detection. SURF is better as it is up to twice as fast as SIFT. However, SIFT is better when a scale change or an increase in lighting is applied to the image. For pedestrians detection, SIFT and SURF therefore remain insufficient for our application. We have also experimented KLT approach [START_REF] Koen | Segmentation As Se-lective Search for Object Recognition[END_REF] for extraction of points of interest and tracking them between an image taken at t-1 and another image taken at t. The approach has high accuracy and is fast, but on the other hand, it is not robust to perturbations, for example when the target is displaced too much, and in case of highly textured images. This is why we have decided to experiment the RMR approach [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], which has very low precision but is considered to be very robust. We have obtained results which are same as KLT approach. A hybrid KLT/RMR approach would be a better solution where we can have benefits of both the approaches.

Object Detection-based HOG/DPM

HOG and DPM algorithms first calculate image features. They apply classifiers on databases of positive images (with pedestrian) and negative images (without pedestrian). They have the advantage of being accurate and give relatively good results; however, their calculating time is high. We have therefore experimented three approaches: HAAR, HOG and DPM.

Pseudo-HAAR Features

HAAR classifiers use features called pseudo-HAAR [START_REF] Gerónimo | Haar wavelets and edge orientation histograms for on-board pedestrian detection[END_REF][27]. Instead of using pixel intensity values, the pseudo-HAAR features use the contrast variation between rectangular and adjacent pixel clusters. These variations are used to determine light and dark area. To construct a pseudo-HAAR feature, two or three adjacent clusters with relative contrast variation are required. It is possible to change the size of features by increasing or decreasing the pixel clusters. Thus, it makes it possible to use these features on objects with different sizes.

HOG Algorithm

HOG is a descriptor containing key features of an image. These features are represented by the distribution of image gradient directions. HOG is an algorithm frequently used in the field of pattern recognition. It consists of five steps:

1. Sizing of the calculation window ; by default the size of the image to be processed is 64x128 pixels 2. Calculation of image gradients using simple masks

DPM Algorithm

The hardest part for object detection is that there are many variances. These variances arise from illumination, change in viewpoint, non-rigid deformation, occlusion, and intra-class variability [START_REF] Su | State of the Art object Detection Algorithms[END_REF]. The DPM method is aimed at capturing those variances. It assumes that an object is constructed by its parts. Thus, the detector will first find a match by coarser (at half the maximum resolution) root filter, and then using its part models to fine-tune the result. DPM uses HOG features on pyramid levels before filtering, and linear SVM as a classifier with training to find the different part locations of the object in the image. Recently, new algorithms have been developed in order to make DPM faster and more efficient [START_REF] Su | State of the Art object Detection Algorithms[END_REF] [START_REF] Dollar | Pedestrian Detection: An Evaluation of the State of the Art[END_REF].

Pedestrian Detection-based HAAR/HOG/DPM

We have carried out all the experiments with 6 different databases dedicated to the pedestrian detection: ETH, INRIA, TUD Brussels, Caltech, Daimler and our own ESIGELEC dataset dedicated to the pedestrian detection. Fig. 1 and Fig. 2 shows respectively results obtained under ETH and ESIGELEC datasets.

Object Detection-based Deep Learning

Many detection systems repurpose classifiers or localizers to perform detection. They apply the classification to an image at multiple locations and scales. High scoring regions of the image are considered as positives detections [START_REF] Su | State of the Art object Detection Algorithms[END_REF] [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]. CNN classifier, like RCNN, can be used for this application. This approach gives good results but require many number of iterations to process a single image [START_REF] Su | State of the Art object Detection Algorithms[END_REF] [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]. Many detection algorithms using selective search [START_REF] Su | State of the Art object Detection Algorithms[END_REF][5] with region proposals have been proposed to avoid exhaustive sliding window. With deep learning, the detection problem can be tackled in new ways, with algorithms like YOLO and SSD. We have implemented those two algorithms for object detection in real time [START_REF]Yolo Homepage[END_REF], and obtained very good results (qualitative evaluation). Fig. 3 shows an example of our deep learning model applied in pedestrian detection within ESIGELEC dataset at the city center of Rouen.

OBJECT DETECTION AND TRACKING-BASED FASTER DPM ALGORITHM Improved HOG Algorithm

We have identified four methods to improve HOG/DPM algorithm, which correspond to the five step of the HOG algorithm. In order to be able to make comparisons with the original version, a classifier was trained on the INRIA and TUD-Brussels image datasets. The number of available images in these datasets is relatively small compared to some other datasets, which has an impact on the pedestrian detection quality. The objective is to improve the quality of HOG/DPM pedestrian detection and to significantly reduce the computation time for the detection in road scenes. The tests were carried out on the ESIGELEC dataset; 1. Gamma Correction: in order to improve the pedestrian image quality, we performed a gamma pre-processing on the images to be trained and tested. Gamma correction can illuminate more the dark areas in images if gamma coefficient is greater than 1 or, in the contrary, darken them if the coefficient is less than 1. We found fewer parasites in the processed images. 2. Image Resizing: HOG algorithm performs calculations on images of size 64x128 pixels. We performed calculation with windows size of 128x256. By doubling the size of the calculation window, we have improved the accuracy but also doubled the computation time (for example from 58ms in the classic HOG to 115ms for the improved HOG). 3. Negative Gradients: when calculating gradients, improved HOG uses negative gradients (180° to 0) and positive gradient (0 to 180°) like classical HOG. This allows the calculation of histogram with 18 values (9 values in classic HOG). The calculation time does not vary, however, by taking the negative gradients (signed gradient) into account, a small improvement was carried out but considered as not significant. However, we found presence of noise, which does not improve the classic HOG. 4. Normalization Vector: as a last step, HOG performs a standardization of 2x2 cells, ie. 32x32 pixels with a pitch of 16x16. The object detection is degraded and the computation time doubles, which cannot considered as an improvement of classic HOG.

Faster-DPM Algorithm

The DPM algorithm is applied on the entire image and this is done at each iteration of the pedestrian video sequence. In order to reduce the computing time, the idea is to apply HOG algorithm only in a Region of Interest (RoI) in which the target (here the pedestrian) is located. This will drastically reduce the computing time and will also better isolate the object. Firstly, the DPM is applied all over the image once to locate the object. Secondly, and after obtaining a RoI surrounding the object to be detected as a bounding box (yellow box in figure 4), we built a New RoI (NRoI) by providing a tolerance zone (a new rectangle like the green one in figure 4 which is larger than the first one). Starting form the second iteration, DPM algorithm is applied only in this new image, which is represented by NRoI. If the target to be detected is lost, the DPM algorithm is re-applied over the entire image. In Fig. 4, we can see that Faster-DPM improves target detection by reducing time from 4 to 8 times less than classic DPM.

Pedestrian Distance Estimation

To have a better information on the detected pedestrians, it is necessary to estimate the distance separating the pedestrians from the vehicle. As our system is based on a monocular camera, the measurement of this distance has to be estimated. The law called "Inverse Squares", used in astronomy to calculate the distance between stars, inspired us: "physical quantity (energy, force, etc.) is inversely proportional to the square of the distance of stars". By analogy, the physical quantity represents the area of our RoI (bounding box surrounding the target). We have used a parametric approach. We have taken measurements at intervals of 50 cm, and the corresponding object RoI surfaces have been recorded. Fig. 5 shows the result obtained.

Figure 5. Detected RoI surface (blue curve) vs distance from the object (red curve): y axis represents the surface (pixels 2 ), and x axis represents distance (m).

The blue curve looks like a square root. This is validated by the trend curve, whose as equation ( 1) is:

𝑦 = 262673 * 𝑥 -1.905 (1)
where x is the abscissa and y the ordinate. The equation of blue curve is (2):

𝑆 = 𝐴 * 𝑑 -2 (2)
where 𝑆 is the surface in pixel 2 , 𝑑 is the distance to be estimated, and A = 262673.

The ideal detection zone is between 3 and 7 meters with an error of 5%. The Tab. 1 illustrates the calibration process carried out for the measurement environment. Table 1. Measurement environment calibration.

VISUAL CONTROL-BASED MULTI APPROACH CONTROLLER Drone Double Controller

The speed servo control of the drone allows it to track the target continuously and in real time. The image processing as a closed loop gives the coordinates of the target in (x,y,z) plan. Using this information, the system send instruction to the Drone engines corresponding to the real time position of the target in order to correct the position of the drone. We have developed two kinds of controllers: Proportional-Integral-Derivative (PID) and Adaptive Super Twisting (AST) controller [START_REF] Viola | Rapid Object Detection using a Boosted Cascade of Simple Features[END_REF]. The controller system is based on three corrections: 1. Drone altitude correction, 2. Drone rotation speed, and 3. Drone navigation. Under Robot Operating System (ROS) system, the instructions are considered as speeds sent to drone engines.

Firstly, we have applied a Proportional (P) controller:

 Altitude : K p = 0.004  Forward translation : K p = 0.08 Secondly, we have applied a classic PID controller (3):

V rotation = K p * erreur + K i * ∫ erreur * dt t 0 + K d * d dt (erreur) (3)
With Rotation: K p = 0.0041, K i = 0.0003, K d = 0.0003

Lastly, in (4), we have applied an AST (nonlinear PID) controller:

V rotation = K p * ( erreur |erreur| + A ) * √|erreur| + K i * ∫ erreur |erreur| + A t 0 dt (4)
with A = 0.001, K p = 0.013, and K i =0.02.

The P controller is amply enough to enslave the altitude of the drone and its translation. However it is the rotational slaving that predominates; we need to keep the target in the center of the image and the target moves strongly from right to left and vice versa.

Overall, the visual servoing that we have adopted uses the data extracted during the HOG/DPM or deep learning image processing (visual controller) and sends commands to the drone actuators. This is a double controller: speed control (internal loop) to control the motors and servo positioning (external loop) to position the drone according to the target position to track. The correction can be done with classical correctors (P, PI, PID) or more advanced commands like for example AST controller. Fig. 6 illustrates the architecture of our cascade control used.

Test & Validation

Indoor and outdoor environment tests were carried out throughout the project. For the final test phases, scenarios have been established. In order to validate the final platform, several scenarios has been defined. The platform gives very good results. Better results are obtained when the target (person) is alone in the environment. Performance is reduced when the target is in a group, despite the use of an adaptive region of interest. In addition, there is always the risk of losing the target, object detection algorithms do not have a detection rate of 100% and sometimes the person to track is no longer detected. If the target is no longer detected, the drone is ordered to switch to stationary mode as a filled situation. We are correcting this problem through two different methods: 1. An estimation of the future position of the target based on improved Kalman filter that gives better results; from now on, we are able to predict the position of the target to be followed and thus to minimize a possible confusion of this target with another target of the same nature (as for example the case of several people who walk together). A second approach is also under development which concerns deep learning not only for object/people detection and tracking, but also for object distance and orientation estimation. To compare the performance of the two implemented control laws (PID controller and AST controller), the GPS coordinates of each trajectory were recorded and compared. In order to illustrate the results obtained, the target (here a person) has made a reference trajectory in the form of a square. Fig. 7 shows the performances obtained on a square shape trajectory. We can see that the PID controller is more accurate than the AST controller.

EXPERIMENTAL RESULTS

The developments were carried out on an Ubuntu Linux platform with ROS, which is a set of computer tools for the robotics environment that includes a collection of tools, libraries and conventions that aim to simplify the process of operation, thus allowing more complex and robust robot behavior

CONCLUSION

In this paper, we have presented a new approach for object detection and tracking applied for drone visual control. A comparison study between different approaches dedicated to pattern recognition and object/pedestrian detection has been carried out. We have present our contribution to improve the quality of both HOG and DPM algorithms. We have also implemented deep learning based Yolo and SSD algorithm for pedestrian detection. An innovative method for distance calculation of pedestrian/object is presented in this paper. The approach were validated within experiments and the accuracy is 5% of errors. The system detects not only object/target to follow but also their distance from the drone. The system is generic so that it is "applicable" on any type of platform and/or environment (pedestrian detection for autonomous vehicle and smart mobility, object detection for smart wheelchair, object detection for autonomous train, etc.). This work aims to show the feasibility of scientific and technological concepts that affect object detection and tracking-based classic approaches like HOG/DPM, but also deep learning approaches-based Yolo or SSD. We have validated the whole development under several scenarios by using both parrot drone platform and real vehicle in real traffic conditions (city center of Rouen in France).

3 .

 3 Image division of 64x128 into 8x8 cells. For each cell, HOG algorithm calculates the histogram. 4. Normalization of histograms by 16x16 blocks (ie. 4 cells) 5. Calculation of the size of the final descriptor. The obtained descriptor is then given to a SVM classifier. The classifier needs many positive and negative images. The mixture HOG/SVM gives good results with limited computing resources.

Figure 1 .Figure 2 .

 12 Figure 1. Comparison between HOG and DPM applied under ETH dataset (640x480-image

Figure 3 .

 3 Figure 3. Object/Pedestrian detection-based deep learning (Yolo and SSD). The results have been obtained under GPU Nvidia Quadro K4000M machine.

Figure 4 .

 4 Figure 4. Comparison between classic DPM and Faster-DPM. Left top and bottom image: object detection-based DPM, Right top and bottom image: object detection-based Faster-DPM with adaptive RoI (green box).

Figure 6 .

 6 Figure 6. Double controller with external (black) and internal loop (in red).

Figure 7 .

 7 Figure 7. PID and AST comparison in Object/Person detection and tracking-based on our improved HOG/DPM. Top image: target square trajectory (blue line) carried out but the mobile target (person). Bottom image: comparison for the different trajectory carried out by person (ground truth data), PID controller and AST controller. x and y axis represents the geographic coordinate system coordinates.

  [START_REF] Mohamed | Adaptive Super Twisting control design for manufactured diesel engine air path[END_REF]. The ROS architecture developed comprises 4 different nodes: Node 1: image acquisition from the drone's embedded camera, Node 2: image processing and calculation of the position of the target to follow, Node 3: visual servoing-based on PID or AST controller, and Node 4: Sending commands to the drone (manual and/or autonomous steering). The drone used is a Parrot Bebop 2 with 22 minutes of autonomy, weight 420g, range of 300m, camera resolution of 14 Mpx, and maximum speed of 13m/s. We have carried out several tests with six datasets dedicated to pedestrian detection. In this section, we present the tests carried out on the ESIGELEC dataset including tests scenarios under real traffic conditions in Rouen shopping center car park and Rouen city center. The results obtained are shown in Fig. 8.

Figure 8

 8 Figure 8. Pedestrian Detection with improved HOG/DPM approach under ESIGELEC Indoor/Outdoor dataset. Left top image: Pedestrian detection with calculated distance under DPM, Right top image: Pedestrian detection-based SSD deep learning model, Left bottom image: Pedestrian detection-based DPM, Right bottom image: Pedestrian detection-based Faster-DPM.

ACKNOWLEDGMENTS

This research is supported by ADAPT project (This work is carried out as part of the INTERREG VA FMA ADAPT project "Assistive Devices for empowering disAbled People through robotic Technologies" http://adapt-project.com/index.php. The Interreg FCE Programme is a European Territorial Cooperation programme that aims to fund high quality cooperation projects in the Channel border region between France and England. The Programme is funded by the European Regional Development Fund (ERDF). Many thanks to the engineers of Autonomous Navigation Laboratory of IRSEEM for the support during platform tests.


G Vienne
email: g.vienne@unity-sc.com

L Milord

J-Ph Piel

W A Iff

Ph Coste

A Sippel

T Combier

A Tarnowka

J.-F Boulanger

G.-O Reymond

D Alliata

A Bosseboeuf

A Comparison of TSV Etch Metrology Techniques

We use three metrology techniques, vertical scanning interferometry (VSI), confocal chromatic microscopy (CCM), and time domain optical coherence tomography (TD-OCT), for depth measurement of through-silicon vias (TSVs) of various cross sections and depths. The merits of these techniques are discussed and compared.

Introduction

While sales of semiconductor equipment broke a new record this year, many metrology needs should be addressed to support the development and production of electronic chips based on "More than Moore" scaling. Among these scaling approaches, 3D integration based on TSVs offers superior integration density and reduces interconnect length/latency. Measurements are needed to evaluate the depth uniformity of etched TSVs. Indeed, upon metal filling, geometrical variations of TSVs can affect Cu nails coplanarity and can warp the wafer, resulting in a low stacking yield. Measuring the depth of TSVs is an increasingly challenging task as the diameter of many TSVs has now shrunk to only a few microns.

Method

A lithography mask comprising circular and square patterns of diameter/side 3, 5, 7, and 9 µm was used to etch two silicon wafers at different depths, resulting in TSVs further referred to as shallow and deep TSVs. The TSVs' aspect ratio (ratio of depth to diameter, or ratio of depth to side, for circular or square TSVs, respectively) varied approximately in the range 1 to 20. VSI measurements were performed on a UnitySC tool from the NST series with a x50 Mirau objective lens scanned over up to 100 µm in steps of 40 nm. The same tool incorporated a TD-OCT device from Fogale Nanotech operating at a wavelength of 1300 nm with a FWHM spectral width of 110 nm [START_REF] Alliata | International Microelectronics Assembly and Packaging Society (IMAPS), 14th International Conference and Exhibition on Device Packaging WekoPa Resort and[END_REF] and focused with a x50 microscope objective. VSI provided measurements in excellent agreement with the SEM for shallow TSVs but failed to return sufficient signal from deep TSVs (except for the square TSV with 9 µm side). CCM produced results in good agreement with the SEM in record speed but only for shallow TSVs. It was unable to measure the narrowest TSVs (3 µm in diameter or side). TD-OCT was the only method able to resolve the depth of both shallow and deep TSVs. Its accuracy is still under evaluation and improvement. The depth of the shallow TSVs is close to the coherence length of the TD-OCT. It results in interferences between top and bottom surfaces which affect the measurement accuracy. We are developing a model-based approach which will correct for this effect.

Conclusions

VSI offers high accuracy, CCM offers fast scanning, but TD-OCT is superior to these techniques for deep TSVs. We are further investigating and improving its capabilities.

Fig. 1 .

 1 Fig. 1. Metrology techniques applied to TSV etch metrology: (a) Vertical scanning interferometry (VSI) , (b) Confocal chromatic microscopy (CCM), (c) Time domain optical coherence tomography (TD-OCT)

Fig. 2 .Fig. 3

 23 Fig. 2. Examples of metrology results in shallow TSVs: (a) VSI on a circular TSV 7 µm in diameter, (b) CCM height map on square TSVs with side ranging from 9 to 3 µm (3 µm not detected), (c) TD-OCT interferograms from the upper surface and the bottom of a circular TSV 7 µm in diameter. (a) (b) (c)

Acknowledgements

This work was performed in the frame of the OLOVIA Project ANR 15-CE24-0028 supported by the French National Agency of Research.