Hamdi Ben

Abdallah Ab

Jean-José Orteu

Benoit Dolives

Igor Jovančević
email: igorjovancevicijo@diotasoft.com

3D Point Cloud Analysis for Automatic Inspection of Aeronautical Mechanical Assemblies

Keywords: robotic inspection system, CAD model, viewpoint selection, cable segmentation, 3D point cloud analysis, non-rigid/flexible/deformable part

In this paper, we address the problem of automatic robotic inspection in two parts: first, automatic selection of informative viewpoints before the inspection process is started, and, second, automatic treatment of the acquired 3D point cloud from said viewpoints. We apply our system to detecting defects on aircraft Electrical Wiring Interconnection System (EWIS) in order to comply with the growing amount of safety regulations such as interference and allowable bend radius of cables in mechanical assemblies.

INTRODUCTION

Industrial quality control is often realized as an automated end-of-line inspection system that checks certain features of the products. In the aeronautical field, many industrial companies need to automate numerous and complicated operations of quality control of aircraft's Electrical Wiring Interconnection System (EWIS) to address the various security requirements.

The system that we have developed is based on standard industrial robots that use a vision-based sensor system mounted on their end-effectors, such as the one presented by Worn et al. [START_REF] Worn | Arikt: Adaptive robot based visual inspection[END_REF] for visual weld seam inspection for car manufacturing, or the robot arm equipped with an optical 3D scanner for inspection of mechanical components by Raffaeli et al. [START_REF] Raffaeli | Context dependent automatic view planning: The inspection of mechanical components[END_REF] In this paper, we present a robust approach for detecting defects on aircraft EWIS in order to comply with the growing amount of safety regulations, such as interference and allowable bend radius of cables in mechanical assemblies. It mainly consists of two processes: First, automatic selection of informative viewpoints before the inspection process is started. In computer vision community the viewpoint selection has been investigated in the past. The most notable examples are inspection path planning for complex 3D structures, [START_REF] Bircher | Structural inspection path planning via iterative viewpoint resampling with application to aerial robotics[END_REF] robot based inspection, [START_REF] Liu | Flying robot based viewpoint selection for the electricity transmission equipment inspection[END_REF] human actions [START_REF] Rudoy | Viewpoint selection for human actions[END_REF] or global 3D reconstruction. [START_REF] Jonnalagadda | Viewpoint selection for object reconstruction using only local geometric features[END_REF] The method proposed in this paper to find the best viewpoint, uses the 3D CAD model of the object to be inspected. This approach consists of two steps: generating candidate viewpoints and constructing scoring function for finding the best viewpoints with the highest score.

Second, after the offline process of viewpoint selection is completed, the robot inspection system is ready for operation. In this paper, we present an original comprehensive method for cable segmentation and reconstruction based on cylinder fitting. The method is fast, almost immediately producing a model that is compact compared to the measurement data. Using the model, we can carry out quantitative analysis of the interference and bend radius of cables.

The paper is organized as follows. The overview of the proposed inspection system is given in section 2. Our viewpoint selection scheme, considerations about generating a set of candidate local viewpoints and the scoring function of our approach are detailed in 3. The proposed inspection method is explained in section 4. Finally, in section 5. some future directions are presented and the paper is concluded.

OVERVIEW

Overview of the proposed system

We address the problem of automatic inspection in two parts: first, automatic selection of informative viewpoints before the inspection process is started (offline preparation of the inspection), and second, automatic treatment of the acquired 3D point cloud from said viewpoints (online inspection process).

Robot-based inspection

The used robot-based inspection platform consists of a mobile robot equipped with three sensors (two cameras and a 3D scanner) mounted on the robot arm effector (Fig. 1a). This setup allows to perform a certain range of actions: (1) localization: a wide field tracking camera allows to precisely localize the effector with respect to the part it controls. (2) inspection: a high-resolution camera with a reduced field of view allows to capture the details and to observe the elements very finely with 7 degrees of freedom. (3) 3D scan: a structured light stereo sensor complements the sensor's capabilities by digitizing the areas observed. (4) controlling: the whole system is controlled by an industrial PC.

Example of inspection

The automatic inspection method based on 2D information obtained by matching the 3D CAD model and real 2D image is presented in our recent paper. [START_REF] Ben Abdallah | Automatic inspection of an aeronautical mechanical assembly by matching the 3D CAD model and real 2D images[END_REF] We have found that some types of inspection cannot be performed by only 2D image analysis. A typical example of such types is the interference between the elements. It requires to check if two flexible elements (e.g. cables, harnesses) or a flexible and a rigid element (e.g. pipe, support) are at a safe distance from each other. For example, Fig. 2 illustrates that 2D image analysis does not solve the problem. For this type of situations, we use a 3D sensor that provides 3D point clouds. The dataset is captured from a single position (one-shot 3D snapshot) using a pre-calibrated ENSENSO N35 3D scanner (Fig. 1b) mounted on an autonomous navigation mobile robot. The data from the robotic platform is meant for offline analysis after the data-collection routine is completed. An example of our dataset is shown on Fig. 3.

OFFLINE PROCESS: VIEWPOINT SELECTION

The initial setup of an inspection task for a 3D free-form surface cannot be done manually, because a human operator cannot define with sufficient accuracy the camera position that will allow him to get the best viewpoint of the element to be inspected. Therefore, we need a (semi-)automatic offline configuration process that is used to compute the best viewpoints which can help to improve the quality and the efficiency of inspection. [START_REF] Eitzinger | Integrated Imaging and Vision Techniques for Industrial Inspection[END_REF] Therefore, the camera needs to be positioned in order to obtain an optimal view of each component to be inspected.

Strategy to find the best viewpoint

The strategy proposed to find the best viewpoint (compute the 6D location (position and orientation) of the camera with respect to the scene to be observed) for the aeronautical mechanical assembly inspection, according to which the candidate viewpoints can be generated, is based on the 3D model of the assembly. First, the candidate viewpoints are sampled over a sphere (ρ, ϕ, θ ), hence each viewpoint is at the same distance from the center of mass (c x , c y , c z ) of the element to be inspected. This is to comply with industrial security standards concerning distance between inspected elements and the robot [START_REF] Liu | Flying robot based viewpoint selection for the electricity transmission equipment inspection[END_REF] and also tradeoff between the field of view and image resolution. Next, we compute the value of visibility score for each candidate viewpoint. The visibility score is calculated by taking into account the occlusion and the visible area of the rendered element. Finally, the viewpoint with the highest score is selected.

Generate a set of candidate viewpoints

A viewpoint V point is defined by a 3D point V (x, y, z) (sensor's position) and sensor's orientation, both expressed in CAD model frame. The candidate viewpoints V C point (x, y, z) are described by the equations:

     x = c x + ρ × sin(ϕ) × cos(θ ) y = c y + ρ × sin(ϕ) × sin(θ ) z = c z + ρ × cos(ϕ) θ min ≤ θ ≤ θ max ϕ min ≤ ϕ ≤ ϕ max θ ϕ x ŷ ẑ = |0 -ẑ = |1 |V point ρ Figure 4: Candidate viewpoint generation
The viewpoint management algorithm provides a visibility sphere (Fig. 4), which defines a set of viewpoints. The candidate viewpoints are evenly distributed on the sphere surface according to the predefined sampling step and the range of longitude θ and colatitude ϕ.

The scoring function

The scoring function f score quantifies the visibility of the element from each candidate viewpoint. We evaluate each generated viewpoint of the inspected element by taking into account the occlusion. The viewpoint that yields a render with the greatest number of pixels belonging to the object of interest is chosen as the best point of view for performing the inspection task according to this criteria. In Fig. 5 we show some of the considered viewpoints for the cable in red rectangle shown in Fig. 5a.

ONLINE PROCESS: INSPECTION METHOD

The online inspection process consists of three main steps, which are detailed in this section. First, 3D point cloud is acquired using a 3D scanner mounted on the robot arm (section 2.1.1). Further, we perform noise filtering and data re-sampling as pre-processing steps. Next, we propose a two-step global-to-local registration procedure that allows for alignment of the 3D data with the 3D CAD model. Finally, the decision is made by segmenting cables from point cloud and calculating measures such as distance between two cables and bending radius.

Usually, the inspection of manufactured parts in industry is done by performing a comparison between a reference model, such as a CAD model, and 3D measurements taken on the surface. [START_REF] Jaramillo | On-line 3-D inspection of deformable parts using FEM trained radial basis functions[END_REF] However, the matching with 3D CAD model becomes more problematic when the target objects are non-rigid (flexible), or otherwise prone to change in shape or appearance (e.g. cables and harnesses). To solve this problem, we present an original method for cable segmentation and reconstruction based on cylinder fitting. Moreover, an interference detection algorithm based on measured data determines if the part fulfills the required tolerances specified by the designer.

Automatic registration process

The automatic registration for 3D CAD models and scanned data illustrated on Fig. 6 includes two main processes: (1) preprocessing and (2) registration. Besides removing noise, to perform registration, the 3D CAD model and point cloud must be converted into the suitable representation. The first step in our method is to filter some noise from the input point cloud. In order to remove separate single points or few-point clusters, we iterate over all points in the input point cloud. We consider each point to be the center of a 3D sphere that has a radius r. Next, the number of neighbours inside a sphere is computed. If a point has less than k neighbours, it is removed from the point cloud. The radius of the sphere and the number of neighbours inside a sphere depends on the resolution of the point cloud data.

The computation of our approach is based purely on local neighborhood. The first step is to determine the spatial resolution of the point cloud R(PC) by calculating the average of the distance between each cloud point and its nearest neighbor. Given a raw 3D point cloud P = p 1 , p 2 , ..., p N , a k-d tree structure is constructed to represent this point cloud and the neighborhood for each point p i is investigated by using the k-nearest neighbor (KNN) method. In our case, we have approximately 40% of the points being considered as noise (see Fig. 7). In order to perform the registration, the 3D CAD model and 3D point cloud must be converted into the suitable representation. We use weighted random sampling: it generates a given number of points uniformly distributed according to triangular areas. However, it is difficult to obtain accurate registration results when two data sets have different resolutions. [START_REF] Li | Automatic registration for 3d shapes using hybrid dimensionality-reduction shape descriptions[END_REF] To solve this problem, the model points and point cloud were subjected to a data re-sampling process.

Registration

Once the appropriate data is obtained, the resulted point clouds (CAD one and real one) are automatically registered using a two-step global-to-local registration. An in-house developed 2D-3D alignment algorithm is used to get an initial estimate for wide field of view camera pose in the model frame. Since our 2D-3D system is previously calibrated, it is straightforward to obtain the pose of our 3D scanner in model frame to align the 3D data with the 3D CAD model by applying the obtained transformation parameters R c and T c , which yields the initial estimate of the pose. The results of the global registration are refined by a local registration process that uses an iterative non-linear Levenberg-Marquardt (ICP) algorithm. [START_REF] Changmin | Fully automated registration of 3D data to a 3D CAD model for project progress monitoring[END_REF]

Cable segmentation

Cables are designed to be flexible. The registration or matching with 3D CAD model becomes problematic when the target objects are non-rigid, or prone to change in shape or appearance. Nevertheless, cable connection points, where the cable meets connectors or clamp, are rigid. These connection points are the most important elements for matching with CAD model. Indeed, these points make it possible to guarantee a rigidity of a portion of cable.

In this section we present a comprehensive method for cable segmentation and reconstruction based on cylinder fitting. First of all, using a two-step global-to-local registration described in section 4.1, we generate a set of candidate points belonging to the cable for the initialization step. Further, we propose a search structure that allows to find the cables in the point cloud received from 3D scanner and to reconstruct them as collections of cylinders of varying radius, length, and orientation. Each cylinder, called sub-cable is fitted using least-squares method [START_REF] Raumonen | Fast automatic precision tree models from terrestrial laser scanner data[END_REF] to the point cloud data corresponding to a cable.

Initialization of the search structure

After the registration (Fig. 8a and8b) explained in section 4.1, the next step is to initially place our search structure (a starting cylinder). Therefore, we iterate through all the points in the registered synthetic cable's point cloud. For each point p i from synthetic cable, we construct a 3D sphere that has p i as a center and has a radius r that is equal to the synthetic cable's radius. For each sphere we compute the number of 3D points inside the sphere that belong to the input point cloud. Then, we sort the candidate points in decreasing order according to the number of points inside their corresponding spheres (Fig. 8c).

We start with the candidate point that has the highest number of points inside its corresponding sphere. This sphere is considered to be the starting one for the fitting phase. We go through all the points inside this sphere and we consider them as centers of new constructed spheres with the same radius r (step 1 Fig. 8d). For each of these new spheres, we compute the number of points inside of it. Finally, the center of the sphere with the greatest number of points is taken as an fitted initialization point (step 2 Fig. 8d). If the initialization is successful, we continue the process of segmentation. Otherwise, we take the next candidate point (the one that has the next largest number of 3D points inside of it). Having found an initialization point (step 2 Fig. 8d), our algorithm further relies on a search sphere virtually placed around this point, to find an initial cylinder that will be used to start reconstruction of the cable. The used sphere is defined as a set of 3D points evenly distributed on the sphere's surface according to the range of longitude θ and colatitude δ that are located at a distance r from a given center. Two points on the sphere are antipodal if they are opposite through the center. Fig. 9a. shows two points A and B which are diametrically opposite and, therefore, situated in such a way that a line candidate drawn from the one to the other passes through the center of the sphere. The purpose was to reduce the number of cylinder parameters from seven to three using spherical coordinates: one parameter for the radius of the cylinder, and two parameters for the start-point P s and end-point P e positions in a local coordinate system (u, v, w).

Next, we virtually generate many cylinders, where each cylinder is defined by 2 antipodal points on the sphere. For each cylinder, we count the number of points within. An empty space around cables is assumed, therefore the cylinder with the greatest number of points is taken as a model of the sub-cable that we are searching for.

A B w v u θ δ P s P e (a) (b) (c) (d
) Figure 9: Search structure: (a) example of two antipodal points, (b) virtual rotating cylinder, (c) the set of 3D points distributed on the sphere surface for the initialization step and (d) simplified structure after the initialization step.

Fitting the sub-cable parameters

The start-point P s and end-point P e are defined by the two antipodal points which form a cylinder with the greatest number of cloud points inside it. The main axis of sub-cable is defined by these 2 antipodal points on the sphere. Next, the height is computed by projecting the 3D points inside the cylinder onto its main axis (see Fig. 10). Finally, the mean distance of all 3D points inside cylinder to its main axis gives the cable radius. Another possible way to fit sub-cables could be to fix the axis direction and to project the points into a plane orthogonal to the axis, and then to fit a circle to the projected points in the least square sense. [START_REF] Raumonen | Fast automatic precision tree models from terrestrial laser scanner data[END_REF] But, this method proved to be less robust than the one we have implemented.

Propagation of search structure

The search structure described in section 4.2.2 is moved in both directions along the cable (see Fig. 11c). It starts from where it stopped last time until it completes the cylinder model of cable. In other words, a detected cylinder end point will coincide with another cylinder start point (Fig. 11).

Inspection process

The reason for doing the segmentation of a flexible element (cable, pipe or harness) is to measure the bend radius and its minimum distance to the other elements in the assembly.

Minimum bend radius

The minimum bend radius is the smallest allowed radius the cable is allowed to be bent around. The segmentation method presented in section 4.2 produces almost immediately a model segmented from the measurement data. Using this model, we can carry out quantitative analysis of a bend radius of a cable. By projecting the set of start-points P s and end-points P e for each sub-cable (Fig. 12a.) onto a plane perpendicular to the camera optical axis, a circle is fitted to the projected point by RANSAC algorithm. Finally, the bend radius can be estimated from the radius of fitted circle (Fig. 12b).

Interference detection

Now, we have two main point clouds. The segmented point cloud, which we will call S and all the other points, excluding the segmented point cloud, which we will call P. We will iterate through all the points included in S. In each point, we will compute the distance between this point and its k nearest neighbours in P with a distance less or equal than our tolerance (interference distance). Next, a data clustering approach in an euclidean sense is implemented by using a k-d tree data structure. Then, we go through each of these clusters and we keep only the ones that are close enough and have a significant size because they are most likely representing another element in the mechanical assembly and the inspected element is considered to be in danger (interference detected). An example of our results of interference detection process is shown on Fig. 12c.

Experiments and discussion

The proposed method has been tested on 140 point clouds, both with and without defects. We acquired point clouds using a pre-calibrated ENSENSO N35 3D scanner mounted on an autonomous navigation mobile robot in different lighting conditions. The distance between camera and inspection element was set at 600 mm with a tolerance of ±6mm.

Despite the fact that we used a one-shot scanning approach and that the point cloud had 40% of noise, we managed to set up a robust system for verifying the radius of curvature and measuring the interference. Our method has been validated in an industrial context.

CONCLUSION

In this paper, we propose an original framework for detecting defects on aircraft Electrical Wiring Interconnection System (EWIS) in order to comply with the growing amount of safety regulations, such as interference and allowed bend radius of cables in mechanical assemblies. We address the problem of automatic robotic inspection in two parts. First, automatic selection of informative viewpoints is performed. The proposed method is based on the 3D model of the assembly and it consists of two steps: generating candidate viewpoints and constructing scoring function for finding the best viewpoints with the highest score. Second, after the offline process of viewpoint selection is completed, the robot inspection system is ready for operation.

In this work an approach for interference detection of aircraft's components assembly is presented and carried out based on measured data. Furthermore, we have summarized the main steps and demonstrated the potential of an original comprehensive method for cable segmentation and reconstruction based on cylinder fitting. The method is fast, almost immediately produces a cable point cloud segmented out of the 3D measurement data. Using the model, we can carry out quantitative analysis of the interference and bend radius of cables.

Repeatability of the proposed approach was evaluated by introducing Gaussian noise on data. Furthermore, the experimental results show that our proposed method is highly accurate in detecting defects on aircraft's EWIS.

Figure 1 :

 1 Figure 1: Acquisition system: (a) Robot with its vision-based sensory system, (b) Ensenso N35 3D scanner

Figure 2 :

 2 Figure 2: Example of inspection: (a) airplane engine (b) the interference between some cables cannot be determined by 2D image analysis

Figure 3 :

 3 Figure 3: Example of our dataset. (a) CAD model, (b) 2D image, (c) 3D point cloud and (d) 3D point cloud with texture

Figure 5 :

 5 Figure 5: (a) CAD model, (b), (c), (d) and (e) different candidate viewpoints for inspecting the element in the red rectangle, with the corresponding f score

Figure 6 :

 6 Figure 6: Overview of the automatic registration process

Figure 7 :

 7 Figure 7: Mean K-nearest neighbours (KNN) distance (input point cloud in red, filtered point cloud in green)

Figure 8 :

 8 Initialization step: (a) initial 2D-3D global registration, (b) ICP-based local registration, (c) some candidate start points and (d) illustration of the fitting method to estimate the initialization point.

4. 2 . 2

 22 Search structure (sphere, cylinder, antipodal points)

Figure 11 :Figure 10 :

 1110 (a) segmented 3D point cloud, (b) the final cylinder model and, in blue, a detected hole in the cable due to two overlapping cables (c) different colors show the different sub-cables Projection Height calculation: (a) sub-cable inside cylinder fitted in the least-squares sense, (b) projection of the point cloud inside cylinder onto main axis, (c) bar graph showing the amount of points per bin, (d) the fitted sub-cable.

Figure 12 :

 12 Example of inspection results: (a) the set of start-points P s and end-points P e for each sub-cable, (b) the corresponding bend radius, (c) results of interference detection process (segmented point cloud in green and representative cluster which causes interference problem in blue)

We are currently working on a more general framework for cylindrical shape segmentation for objects other than cables and pipes (e.g. arteries in the medical field, tree branches in forestry applications, etc.).

We are planning to publish a library CSSL (Cylindrical Shape Segmentation Library) in a scientific review.