Van Thao Le

Yann Quinsat
email: yann.quinsat@ens-paris-saclay.fr

In-situ calibration of galvanometric scanning head for laser powder bed fusion machines based on a vision system

Keywords: Additive manufacturing, Laser powder bed fusion, Galvanometric scanning system, Insitu calibration, Vision system, Image processing

In situ calibration of galvanometric scanning head for laser powder bed fusion machines based on a vision system van Thao Le, Yann Quinsat

To cite this version:

van Thao Le, Yann Quinsat. In situ calibration of galvanometric scanning head for laser powder bed fusion machines based on a vision system. International Journal of Advanced Manufacturing Technology, 2020, 111 [START_REF] Chen | Compensating technique of field-distorting error for the CO2 laser galvanometric scanning drilling machines[END_REF][START_REF] Ehrmann | Optics for vector scanning[END_REF], pp.1767-1783. [START_REF] Lüdtke | Calibration of Galvanometric Laser Scanners Using Statistical Learning Methods[END_REF].1007/s00170-020-06189-7. hal-02997911

1. Introduction

Context

Laser galvanometric scanning systems are widely used in many fields such as medical imaging, optical metrology, and material processing due to its capability to achieve very high processing velocities, high-positioning repeatability, and small laser spot size [START_REF] Chen | A scribing laser marking system using DSP controller[END_REF][START_REF] Hsiao | Development of portable laser machining system for laser writing applications[END_REF][START_REF] Ai | Focused laser lithographic system for efficient and cross-scale fabrication of large-area and 3D micro-patterns[END_REF]. In Laser Powder Bed Fusion (LPBF) additive manufacturing machines such as Selective Laser Melting (SLM) and Selective Laser Sintering (SLS), a laser galvanometric scanning head (Fig. 1) is also used to control the positions of laser spot in the work plane. This scanning head consists of a positioning device (i.e. two rotatable mirrors driven by two galvanometers) and a focusing device.

The focusing device can be either an f-theta lens located after the positioning device, or a dynamic focus module (DFM) located before the positioning device [START_REF] Cui | Calibration of a laser galvanometric scanning system by adapting a camera model[END_REF][START_REF] Chen | Compensating technique of field-distorting error for the CO2 laser galvanometric scanning drilling machines[END_REF][START_REF] Ehrmann | Optics for vector scanning[END_REF]. The focusing device controls the laser spot size in the work plane, whereas the positioning device controls the positions of laser spot in the work plane. Errors between desired positions and actual positions of laser spot in the work plane directly affect the geometry accuracy of built parts. Therefore, the laser galvanometric scanning head of LPBF machines must be calibrated in an accurate manner to control lasing paths.

Fig. 1. Laser galvanometric scanning head of LPBF machines with dynamic focus module (DFM), adapted from [START_REF] Godineau | Calibration of galvanometric scan heads for additive manufacturing with machine assembly defects consideration[END_REF].

Literature review

In the literature, much research has reported the calibration of galvanometric scanning systems. For instance, Mao et al. [START_REF] Mao | Plane-constraint-based calibration method for a galvanometric laser scanner[END_REF] and Yang et al. [START_REF] Yang | Modeling and Calibration of the Galvanometric Laser Scanning Three-Dimensional Measurement System[END_REF] reported the calibration of one-mirror galvanometric scanners, which were used in 3D measurement systems. Lüdtke et al. [START_REF] Lüdtke | Calibration of Galvanometric Laser Scanners Using Statistical Learning Methods[END_REF] and Tu et al. [START_REF] Tu | Effective data-driven calibration for a galvanometric laser scanning system using binocular stereo vision[END_REF] calibrated two-mirror galvanometric laser scanners by using employed statistical learning methods (e.g., linear regression and artificial neural networks). For two-mirror galvanometric scanning systems, they were generally calibrated by calculating correction tables for all joint componentsi.e., two rotatable mirrors. The correction tables allow us to determine the relationship between positions of laser spot in the work plane (also called the task space) and corresponding positions of all joint components in the join space.

Several mathematical models have been developed to describe the relationship between the task space and the joint space. For example, the models presented in [START_REF] Xie | Correction of the image distortion for laser galvanometric scanning system[END_REF][START_REF] Manakov | A Mathematical Model and Calibration Procedure for Galvanometric Laser Scanning Systems[END_REF] have taken into account the geometry of all components of the laser galvanometric scanning system. However, these works have used many assumptions related to the geometry, for example, rotatable axes of mirrors confused with their reflection surfaces, and so on. Thereby, the real laser galvanometric scanning system was not modeled completely. To fill this gap, certain authors have developed more representative models by taking into account different sources of defects such as defects related to laser source orientation, positions and orientations of mirrors [START_REF] Brosens | Scanning Accuracy of the oving-Iron Galvanometer Scanner[END_REF][START_REF] Blais | Control Of Low Inertia Galvanometers For High Precision Laser Scanning Systems[END_REF]. Recently, Godineau et al. [START_REF] Godineau | Calibration of galvanometric scan heads for additive manufacturing with machine assembly defects consideration[END_REF] developed a complete analytical model for the laser galvanometric scanning head of LBPF machines. In their model, thirty defects related to all components of the laser scanning head were taken into consideration. Thus, the position of a laser spot in the work plane was determined as a function of all joint commands and thirty assembly defects.

In most of the published works [START_REF] Godineau | Calibration of galvanometric scan heads for additive manufacturing with machine assembly defects consideration[END_REF][START_REF] Brosens | Scanning Accuracy of the oving-Iron Galvanometer Scanner[END_REF][START_REF] Blais | Control Of Low Inertia Galvanometers For High Precision Laser Scanning Systems[END_REF][START_REF] Bessmeltsev | Method of Calibration of a Laser-Based Galvanometric Scanning System with Submicron Resolution[END_REF][17], the calibration of laser galvanometric scanning systems by calculating correction tables was performed as follows: The laser beam was controlled to desired positions, which were defined by a grid of points. The laser beam engraved a grid of marks on a calibration plate to establish real positions of laser spot. Subsequently, the real positions of laser spot were measured by a measurement system in another location. Finally, based on mathematical models (with assembly defects [START_REF] Godineau | Calibration of galvanometric scan heads for additive manufacturing with machine assembly defects consideration[END_REF] or without defects [START_REF] Bessmeltsev | Method of Calibration of a Laser-Based Galvanometric Scanning System with Submicron Resolution[END_REF]17]) and experimental data, the correction tables/matrices were calculated. These methods allow the laser galvanometric scanning heads to be calibrated effectively and precisely. However, they were "rigid" and time consuming, because the steps of measuring the actual laser positions, which were engraved on the calibration plate was performed by a measuring system in another location. In addition, it is difficult to integrate such a measurement system in a closed-loop calibration procedure. Lastly, the measured data accuracy was significantly affected by the repositioning of the calibration plate and the deviations of the measuring systems.

Nowadays, vision inspection systems -also called machine vision systems are widely used for automated inspection, robot guidance, quality control, and manufacturing applications because of their accuracy, flexibility, repeatability and efficiency [START_REF] Zhang | In-situ monitoring of laser-based PBF via off-axis vision and image processing approaches[END_REF][START_REF] Hsu | Vision-based inspection system for cladding height measurement in Direct Energy Deposition (DED)[END_REF][START_REF] Dubreuil | Calibration based on part set-up measurement for onmachine inspection using vision[END_REF][START_REF] Tapia | A Review on Process Monitoring and Control in Metal-Based Additive Manufacturing[END_REF]. The vision inspection systems equipped with a camera or multiple cameras, and even video and lighting are capable of measuring parts, verifying parts in the correct position, and recognizing the shape of parts at high speeds. In the literature, many researchers have used such vision systems for the in-situ inspection/monitoring of AM processes [START_REF] Tapia | A Review on Process Monitoring and Control in Metal-Based Additive Manufacturing[END_REF][START_REF] Chua | Process monitoring and inspection systems in metal additive manufacturing: Status and applications[END_REF][START_REF] Grasso | Process defects and in situ monitoring methods in metal powder bed fusion: a review[END_REF][START_REF] Everton | Review of in-situ process monitoring and in-situ metrology for metal additive manufacturing[END_REF]. In the vision inspection systems using a camera, the camera can be installed in a coaxial configuration or an off-axial configuration [START_REF] Grasso | Process defects and in situ monitoring methods in metal powder bed fusion: a review[END_REF]. In the coaxial configuration, the camera follows the same optical path with the laser beam of LPBF systems to locally acquire melting pool signals. Therefore, the melting pool can be observed from the top view. This configuration was widely used for measuring the size and temperature of melting pool. On the other hand, in the off-axial configuration, the optical axis of the camera is not perpendicular to the work plane of LPBF machines. Thus, images captured by the camera always present perspective distortion. However, the images provide more comprehensive information, for example, melting pool shape in side views, temperature distribution, geometry and topography of deposited layers [START_REF] Scime | Anomaly detection and classification in a laser powder bed additive manufacturing process using a trained computer vision algorithm[END_REF][START_REF] Scime | A multi-scale convolutional neural network for autonomous anomaly detection and classification in a laser powder bed fusion additive manufacturing process[END_REF][START_REF] Zhang | In situ surface metrology of laser powder bed fusion processes using fringe projection[END_REF][START_REF] Zhang | Situ Monitoring of Laser Powder Bed Fusion Additive Manufacturing Using Digital Fringe Projection Technique[END_REF][START_REF] Zhang | In situ surface topography of laser powder bed fusion using fringe projection[END_REF][START_REF] Zhang | Extraction and evaluation of melt pool, plume and spatter information for powder-bed fusion AM process monitoring[END_REF].

In the context of using a vision system for the calibration of a laser galvanometric scanner, Delgado and Lasagni [START_REF] Ortega Delgado | Reducing field distortion for galvanometer scanning system using a vision system[END_REF] proposed a method using a self-designed calibration plate and a coaxial coupled vision device (i.e., a camera and an illumination source) that shared the same optical path with the laser beam. The authors demonstrated that their calibration method was capable of reducing the field distortion of the laser galvanometric scanner efficiently, and avoiding the necessity of repetitive marking and measuring phases of laser marks. However, this calibration method was only relied on experimental measurements. The geometry of the laser galvanometric scanning system was not taken into account in the calibration process.

Research aim

In this study, taking into account the complete analytical model of the laser galvanometric scanning head developed in [START_REF] Godineau | Calibration of galvanometric scan heads for additive manufacturing with machine assembly defects consideration[END_REF] and the advantages of a vision system simultaneously, an in-situ calibration method for the laser galvanometric scanning head of LBPF machines was proposed based on a vision system with a high-resolution monochrome camera. The aim is to acquire experimental data rapidly during the calibration process, and avoid the laser marking and the measuring phases of laser marks. The research work presented in this paper was carried out in two phases: Firstly, the feasibility of using such a vision system to measure 2D objects in a plane with four available reference points was demonstrated. Secondly, the proposed measurement method was used to acquire the positions of laser spot intended for the calibration of the optical chain of LPBF machines according to the algorithm developed in [START_REF] Godineau | Calibration of galvanometric scan heads for additive manufacturing with machine assembly defects consideration[END_REF].

This paper is organized as follows: an overview of the proposed method is described in Section 2. Section 3 details the materials and experimental setup. In Section 4, the intrinsic parameters of the camera are estimated via the camera calibration process. These parameters are used for the image processing and the measurement. The homography computation for correcting perspective distortion of images is also presented. In Section 5, the method for acquiring laser spot positions in the work plane of the LPBF machine is presented. The measured positions of laser spots are subsequently used for the calculation of correction tables to calibrate the optical chain of the LPBF machines (Section 6). Section 7 is intended for some conclusions and future works.

Method overview

An in-situ calibration method for the optical chain of LPBF machines was proposed in this study basing on a vision system -i.e., a high-resolution monochrome camera. In this method, the measurement of laser spot positions was directly performed on a reference plate positioned in the work plane of the LPBF machine (Fig. 2).  In the first step -"Camera calibration & homography computation", the intrinsic parameters of the camera were estimated via the camera calibration process. These parameters were used for the image processing and the measurement method. To correct the perspective distortion of images, a homography transformation was also estimated based on four points of the reference plate (Section 4).

 In the second step -"Measurement of laser spot positions" (Section 5), before detecting the laser spot in an image, lens distortion (radial and tangential lens distortions) and perspective distortion of the image were corrected. The perspective distortion was corrected by using the homography transformation matrix estimated in the first step. The homography transformation allows an image captured in an off-axial configuration to be transformed into an orthogonal image, and the computation of world coordinates of a point from its pixel coordinates to be easily performed in an automatic manner for all positions of laser spot. The pixel coordinates of laser spot center in the corrected image were identified by using some image processing algorithms. Finally, the world coordinates (in millimeters) of laser positions in the world coordinate system were calculated by using the ratio of millimeters per pixels.

 The calibration of the optical chain of the machine was performed in two steps (Section 6): The pre-correction tables calculated from the model of the machine without defects and a set of theoretical points were used to generate the real laser spot position in the work plane. Once the positions of laser spot in the work plane were measured, they were subsequently used to identify all defects related to the geometric model of the machine and calculate the correction tables for all joint components of the optical chain. Finally, the accuracy of the proposed calibration method was analyzed.

Experimental setup

Camera

In this work, a monochrome camera VIEWORKS VC-71MC and a lens ZEISS DISTAGON T2.0/35 ZF were used for the measurement. The monochrome camera possesses a high resolution CMOS imaging of 10000 (H) x 7096 (V) pixels. The size of each pixel is 3.1 µm x 3.1 µm. The camera was installed on the machine according to an off-axial configuration beside the laser scanning head of the machine (Fig. 2b). This is due to the fact that the build compartment of LPBF machines requires an isolable environment from the outside of the machines, and it is filled with inert gases such as argon or nitrogen to ensure the quality of built parts and avoid the phenomena of oxidation and contamination [START_REF] Kruth | Additive Manufacturing of Metals via Selective Laser Melting: Process Aspects and Material Developments[END_REF]. The fixture system of the camera was designed so that the position and orientation of the camera can be adjusted to capture expected images of the objects on the work plane of the machine (Fig. 2a and2b). Once the camera was installed on the LBPF machine prototype, the pose of the camera was fixed during the measurement.

Reference plate

To acquire the laser spot, a plate with four reference points was positioned in the work plane of the machine (Fig. 2c). The world coordinates of four reference points were measured a Microvue -Excel 651 HM-HC optical system. A reference point is the intersection of a vertical line and horizontal line engraved on the reference plate. The images of laser spot on the plate at different positions were captured by the camera. In order to obtain a proper image for the image processing, the exposure time of the camera was adjusted, and a light projection source was used.

Laser source

In the current study, a visibly red laser with a very low power was used instead of a highpower laser source as in real LBPF machines. The aim of using this laser source is to ensure the safety of operators and all experimental devices. Thereby, the reference plate can be used many times in the experiments. Moreover, with a visible laser source, the images of laser spots in the work plane captured by the camera present in common shapes such as circles or ellipses, avoiding the large deviation in the detection of laser spot centers in images. The use of a visible laser instead of a high-power laser source would also not significantly change the calibration accuracy of the machine's optical chain. This will be demonstrated at the end of section 6.3.

Test platform

The test platform used is composed of a 3-axis CTI (Cambridge Technology) laser scanning head and an SMC control board. It is identical to the technology used in an Addup Formup 350.

The geometry of the test platform has been chosen to match the geometry of the same machine.

These choices make it possible to carry out tests that can be directly transposed to an industrial machine.

Camera calibration, homography computation, and measurement of 2D objects in a plane

An image captured by the camera always presents lens and perspective distortions, which have significant effects on the accuracy of the identification of entities in the image, and subsequently on the accuracy of the measurement method. Hence, the camera should firstly was calibrated (Subsection 4.1). The calibration process aims at estimating the intrinsic and extrinsic parameters of the camera. The intrinsic parameters include the focal length, principal point, pixel skew coefficient, radial and tangential distortion coefficients, whereas the extrinsic parameters describe the pose of the camera (i.e., the camera's orientation and position). In order to detect and measure the laser spot position in images automatically, a homography transformation matrix based on four reference points was calculated (Subsection 4.2). Finally, the measurement method for 2D objects in the work plane was validated to ensure the quality of acquired data for the calibration process (Subsections 4.3 and 4.4).

Camera model and camera calibration

In computer vision, the pinhole camera model is widely used [START_REF] Dubreuil | Calibration based on part set-up measurement for onmachine inspection using vision[END_REF][START_REF] Quinsat | A novel approach for in-situ detection of machining defects[END_REF]. This model presents the mapping relation between an arbitrary 3D point P = (X w , Y w , Z w ) in the world coordinate system (R w ) and its corresponding pixel point p = (u, v) in the pixel coordinate system (R uv ) of the image, as shown in Eq. ( 1) and Fig. 4:

(1)
where is a non-zero scale factor; f x and f y are the effective focal length on the u-axis and vaxis of the pixel coordinate system (R uv ); c x and c y are the coordinates of the principal point that is the intersection of the optical axis and the image plane in the image coordinate system (R im ); s is the skew coefficient, which is non-zero if the image axes are not perpendicular. The parameters f x , f y , c x , c y and s constitute the intrinsic parameter matrix A of the camera. The matrix R and the vector T are the rotation matrix and the translation vector between the camera coordinate system (R c ) and the world coordinate system (R w ). They describe the extrinsic parameters of the camera. To present a real camera, the radial and tangential distortions of the camera lens are also taken into account. The equation describing these distortions is given by Eq. ( 2) [START_REF] Heikkila | A four-step camera calibration procedure with implicit image correction[END_REF]:

+ (2)
where (x, y) are the coordinates of the point p in the image coordinate system (R im ) without distortions: x = uc x and y = v -c y ; (x distorted , y distorted ) denote the coordinates of the distorted point of the point p; r 2 = x 2 +y 2 ; k 1 , k 2 and k 3 are radial distortion coefficients; p 1 and p 2 are tangential distortion coefficients. These distortion coefficients are also considered as intrinsic parameters of the camera.

The camera calibration aims at estimating the intrinsic and extrinsic parameters of the camera. These parameters were estimated by minimizing the cost function ( 3):

(

) 3
where is the projection of the world point in the image according to Eq. ( 1); m ij is the j-th image point corresponding to the world point in the image i; n is the number of images used for the calibration process; and m is the number of world points, equal to the number of image points in each image.

In this study, the camera parameters were estimated by using the "Camera Calibrator" toolbox available in Matlab and a calibration plate certificated by GOM Company with dimensions of 500 mm x 400 mm and 69 x 53 circular black markers on the white background (Fig. 5). In the literature, most existing camera calibration approaches [START_REF] Zhang | A flexible new technique for camera calibration[END_REF][START_REF] Usamentiaga | Highly accurate geometric calibration for infrared cameras using inexpensive calibration targets[END_REF][START_REF] Wang | Out-of-focus color camera calibration with one normalsized color-coded pattern[END_REF][START_REF] Sun | A new method of camera calibration based on the segmentation model[END_REF][START_REF] Yu | A calibration method based on virtual large planar target for cameras with large FOV[END_REF] and the Camera calibrator toolbox developed in Matlab consider that the world points must be coplanar and their z-coordinate is equal to 0. In fact, the z-coordinates of marker centers of this plate present a deviation between 0 and 0.377 mm. To take into account this flatness defect and improve the parameter determination, a least square fitting plane (LSFP) was firstly estimated from the given coordinates of all marker centers. Finally, the projections of all marker centers on the LSFP and their coordinates in the coordinate system associated with the LSFP were used as the world points for the camera calibration.

To calibrate the camera, twenty-nine images of the calibration plate captured in different positions and orientations were used. The estimated intrinsic parameters of the camera were given in Table 1. It can be seen that the values of the tangential distortion coefficients (p 1 and p 2 ) are very small, which shows the quality of the optics. Thus, the influence of tangential distortion can be negligible. The standard deviations of f x , f y , c x , c y and s are fewer than 0.3 pixels, meaning a good uncertainty in the estimation of camera parameters. On the other hand, the radial distortion coefficients (k 1 to k 3 ) are noticeable and have significant effects on the identification precision of expected entities in the images. Therefore, this distortion should be corrected before the detection of laser spot centers.

Homography computation

The perspective distortion always exists in an image taken in a non-orthogonal configuration. This distortion can be corrected by estimating a 2D homography matrix H defined as Eq. ( 4) [START_REF] Hartley | Multiple View Geometry in Computer Vision[END_REF]: [START_REF] Cui | Calibration of a laser galvanometric scanning system by adapting a camera model[END_REF] where (u, v) are the coordinates in the initial image and (u', v') are the coordinates in the orthogonal image. Herein, the matrix H was estimated by using four correspondences (A, B, C, D)  (A', B', C', D'), as shown in Fig. 6a.

The points (A, B, C, D) in the initial imageeach point is the intersection of a vertical line and a horizontal line engraved on the plate, were detected by using the Algorithm 1 (described in subsection 4.3), whereas the points (A', B', C', D') in the orthogonal configuration were defined from the corresponding world points, the focal length and principal point of the camera.

By using the perspective warping with the homography transformation matrix H, the perspective distortion of the initial image was effectively corrected, and the corrected image appears as an image captured in the orthogonal configuration of the camera (Fig. 6b).

Measurement method of 2D objects in a plane

Detection of a grid point in the image

In this work, the interested 2D objects were the grid points engraved in the reference plate.

The pixel coordinates of a gird point in the image were determined through two steps: (i) correct lens distortion of the image, and (ii) identify the intersection points of the gird on the image. The lens distortion correction was easily performed by using the "camera calibrator" toolbox of Matlab software.

In the image, each line of the grid is characterized by a bright line with a width of about ten pixels. Each point of the grid is the intersection of a vertical line and a horizontal line. Hence, a grid point in the image was identified by the Algorithm 1 developed in Matlab, as follows:

Algorithm 1: Identification of a grid point in the image 1. Crop a square region of interest (RoI) around the interested point to isolate it from the total image and reduce the image processing time (Fig. 7).

2. In the cropped image, detect the points on boundaries of the vertical and horizontal lines by using the CANNY algorithm [START_REF] Hartley | Multiple View Geometry in Computer Vision[END_REF]. Sometimes it also detects boundary points of bright regions in the RoI (Fig. 7b).

3. From the detected boundary points, apply the RANSAC algorithm [START_REF] Hartley | Multiple View Geometry in Computer Vision[END_REF] to classify and construct two vertical lines (green) and two horizontal lines (yellow), which are the boundary lines corresponding to the vertical and horizontal lines.

4. Calculate four intersections (x, red) of four constructed lines and their gravity centroid (*, violet). This gravity centroid was considered as the grid point needs to identify (Fig. 7b).

5. Transform the pixel coordinates of the identified grid point in the cropped image (Fig. 7b) into the pixel coordinate system of the corrected image (Fig. 7a) The algorithm 1 was used to detect four reference points (Fig. 2c), which are intended for the homography computation, and detect all grid points on the plate (Fig. 7a) for the validation of the measurement method.

Detection of all grid points in the image for the measurement method

In order to detect all grid points engraved on the plate automatically, the correction of lens distortions and perspective distortion was firstly performed. Thereafter, a grid of theoretical points was created from four points (A', B', C', D') (Figs. 6b and7a)each theoretical point is very close to the grid point to be detected. Finally, all grid points in the corrected image were automatically detected by the Algorithm 1.

Computation of the world coordinates of grid points

To calculate the coordinates of the grid points in the world coordinate system (R w ), the ratio (r) between a distance in the world coordinate system (R w ) per the corresponding distance in the orthogonal image was used. This ratio was defined as Eq. ( 5):

(mm/pixel) [START_REF] Chen | Compensating technique of field-distorting error for the CO2 laser galvanometric scanning drilling machines[END_REF] where p j , p k are the image points selected from four reference points (A', B', C', D') in the orthogonal image (Fig. 7b) and p j  p k ; P j and P k are respectively the corresponding world point of p j and p k .

Finally, the coordinates of the grid points measured by the proposed method were calculated as Eq. ( 6):

P mesure = r*(p -C) (6)
where C = (c x , c y ) is the principal point of the camera, P mesure is the grid point measured in the world coordinate system in millimeters, and p is the corresponding image point in the pixel coordinate system of the orthogonal image.

Validation of measurement method

Fig. 8 shows the position errors between the grid points measured by the proposed method and the reference points given by the manufacturer of the plate. For all measured points, the position errors are less than 0.058 mm, and the mean error is approximately 0.028 mm. In the error histogram (Fig. 8 on the right), it is found that about 82% of measured points have errors less than 0.04 mm. Only 8% of measured points present errors between 0.045 mm and 0.058 mm. It is also observed that the distribution of errors is not homogeneous. This phenomenon can be explained by the inhomogeneity of the grid lines engraved on the plate, and the image-taking condition (for example, the light intensity and reflection of light on the plate). This produces the lines in the image with different levels of clarity. In fact, the accuracy of the proposed measurement method greatly depends on the accuracy of the homography calculation and the grid point detection.

Therefore, the detection of grid points in the images plays an important role. The quality of the detection of grid points in the images strongly depends on the brightness of the lines in the image.

This problem can be eliminated by improving and adjusting the image-taking condition and selecting the proper threshold values for detecting boundary points of lines in the CANNY algorithm (Fig. 8b).

In summary, the results shown in this section confirm that the proposed method can detect image points in the image correctly, and the points in the work plane of the machine can be efficiently measured with an average position error lower than 0.03 mm. Therefore, the method can be used for the in-situ calibration of the optical chain of LPBF machines.

Measurement of laser spot positions in the work plane

Shape of laser spot

In the experiment, the reference plate was used to acquire the laser spot shape. This plate was positioned on the work plane of the machine so that the axis X p and Y p of the plate coordinate system R p are respectively parallel to the axis X w and Y w of the world coordinate system R w of the work plane of the machine (Fig. 9). The images of laser spot in the reference plate were captured by the camera. In images, a laser spot is ideally represented by a small region of bright and white pixels and the boundary of the laser spot is a circle or an ellipse. However, depending on many factors such as the firing angle of the laser, the surface state, the surface material, the light of the working environment, the resolution of the camera, the exposure time of the camera, and so on, the actual laser spot form is often elliptical (Fig 10a) or irregular (Fig 10b). In addition, the laser spot images present some types of noises, for example, background noise (Fig. 10c), photon noise, and readout noise (Fig. 10d) [START_REF] Vyas | Centroid Detection by Gaussian Pattern Matching in Adaptive Optics[END_REF]. To limit reflections and reduce noise on the image, the surface of the plate was covered with a mat material except four reference points (Fig. 2c) that were used to calculate the homography in the measurement method. After that the background noise was eliminated (Fig. 10a).

Detection of laser spot in the images

The steps to detect the laser spot center were shown in Fig. 11. Firstly, the image of the laser spot was corrected in terms of lens and perspective distortions. In addition, a square region of interest (RoI) including the laser spot was cropped to isolate the laser spot from the total image and reduce the image processing time.

To reduce the photon and readout noises while leaving the edges of the laser spot unchanged, some image filtering approaches can be used. Several algorithms are frequently used to filter images, such as adaptive median filtering, Gaussian filtering, medium filtering, and Wiener filtering [START_REF] Zhang | Research on the measurement accuracy of different laser spot center location[END_REF]. In this work, to reduce the photon and readout noises while leaving the edges of the laser spot unchanged, the 2D Gaussian filtering [START_REF] Wang | Out-of-focus color camera calibration with one normalsized color-coded pattern[END_REF][START_REF] Tang | Defocus map estimation from a single image via spectrum contrast[END_REF] was employed. Thereafter, the filtered image was converted into a binary image using a threshold value defined as Ts = I max /1.1 [START_REF] Zhou | Laser Spot Center Detection Based on Improved Circled Fitting Algorithm[END_REF],

where I max is the maximum brightness in the image.

There are two groups of methods in the literature for detecting the center of the laser spot:

(i) In the first group, the contour of the laser spot was firstly detected, and the center of the laser spot was then determined by estimating a circle or an ellipse from the detected contour by the least square method [START_REF] Zhou | Laser Spot Center Detection Based on Improved Circled Fitting Algorithm[END_REF][START_REF] Thomas | A simple approach for the estimation of circular arc center and its radius[END_REF][START_REF] Qian | Precision improved spot centroid-locating algorithm based on profile regularization[END_REF]. These methods are effective for detecting laser spots with a relatively large size and a regular shape (e.g. circle or ellipse). (ii) The second group of methods was based on analyzing the characteristics of the laser spot area as a function of pixel intensity of the image, for example, Gaussian quadric estimation [START_REF] Dong | Non-iterative spot center location algorithm based on Gaussian for fisheye imaging laser warning system[END_REF] and center of gravity [START_REF] Vyas | Centroid Detection by Gaussian Pattern Matching in Adaptive Optics[END_REF][START_REF] Hou | Charge-coupled devices combined with centroid algorithm for laser beam deviation measurements compared to a position-sensitive device[END_REF][START_REF] Liu | A high precision laser spot center location method based on multi-scale image analysis[END_REF][START_REF] Vázquez-Otero | Laser Spot Detection Based on Reaction Diffusion[END_REF][START_REF] Singh | Analysis of accuracy of laser spot centroid estimation[END_REF][START_REF] Thomas | Optimized centroid computing in a Shack-Hartmann sensor[END_REF]. The center of the light spot is generally located at the top of the intensity surface. Hence, these methods are more suitable for determining the center of the laser spot, in particular for well-structured laser spots.

In this work, three methods were considered to detect the position of the laser spot, namely ellipse fitting (1), center of gravity (2), and Gaussian quadric estimation (3), as depicted in Fig. 11.

(m1.1) is the method (1) using the input data detected by the CANNY algorithm [START_REF] Hartley | Multiple View Geometry in Computer Vision[END_REF], while (m1.2)

is the method (1) using the input data detected by the binarization. In the same way (m3.1) is the method (3) using the input image without Gaussian filtering, and (m3.2) is the method (3) using the input image filtered by the Gaussian filtering operation.

Influence of the noise

In order to evaluate the accuracy and performance of these methods, three cases of laser spots were simulated by using a super-Gaussian distribution function of order p [START_REF] Stratan | Measuring effective area of spots from pulsed laser beams[END_REF][START_REF] Dickey | Laser Beam Shaping Applications[END_REF][START_REF] Du Preez | High power infrared super-Gaussian beams: generation, propagation, and application[END_REF], as shown in Table 2. These images were representative for certain actual shapes of laser spot on the reference plate taken by the camera. The size of each image is 160 x 160 pixels and the center of laser spot for all ideal images without noises is the image center (x 0 , y 0 ) = (80, 80) (pixels). Fig. 12 presents the detection error between the laser spot center detected by the considered methods and that of the ideal image center (x 0 , y 0 ). It is revealed that the methods (m1.2), (m2), and (m3.2) are more stable and the detection error is less than 0.3 pixels in the three study cases.

Influence of the set up environment

Due to the vibration of the machine and the change of light intensity of the experimental room, the position of an actual laser spot is also unstable. Fig. 13 shows the positions of an actual laser spot detected by the methods (m1.2), (m2), and (m3.2) in 20 images taken successively by the camera. For the detection methods (m1.2) and (m2), the maximum laser spot variation distance in the images is 0.26 and 0.27 (pixels), respectively, whereas the method (m3.2) shows the maximum laser spot variation distance of 0.90 (pixels). This observation demonstrates that the methods (m1.2) and (m2) allow us to obtain more stable detection results of the real laser spot center. In terms of the image processing time, the method (m2) is more rapid. Thus, the method (m2) -Center of gravity was finally taken into account for the study.

Determination of laser spot positions in the world coordinate system

Once the laser spot center in the cropped image was detected, its coordinates were transformed into the pixel coordinate system of the corrected initial image, and the coordinates of laser spot position in the world coordinate system of the plate were determined as presented in Subsection 4.3.3.

6.

In-situ calibration of galvanometric scanning head for a LPBF machine prototype

Model of LBPF machines

In this study, the models developed by Godineau et al. [START_REF] Godineau | Calibration of galvanometric scan heads for additive manufacturing with machine assembly defects consideration[END_REF] for the optical chain of industrial SLM machines -i.e. forward kinematic model (FKM) and inverse kinematic models (IKM), and the proposed measurement method (Section 2) were used for calibrating the optical chain of an LPBF machine prototype installed in our laboratory ( from X [START_REF] Godineau | Calibration of galvanometric scan heads for additive manufacturing with machine assembly defects consideration[END_REF].

The LPBF machine prototype considered in this work possesses the same optical chain (i.e.

the laser galvanometric scanning head) to that of the industrial LPBF machine. However, the CTI block of mirrors of the prototype machine has only a protective window, whereas the CTI block of the industrial machine possesses two protective windows (Fig. 1). Thus, the model of the optical chain in this study was adapted from the model in [START_REF] Godineau | Calibration of galvanometric scan heads for additive manufacturing with machine assembly defects consideration[END_REF] by removing the protective window 2 of the CTI block (Fig. 1) and two related defects. The model in this study has 28 assembly defects instead of 30 defects as in [START_REF] Godineau | Calibration of galvanometric scan heads for additive manufacturing with machine assembly defects consideration[END_REF]. Moreover, in this study, we used the reference plate (Fig. 2c) positioning on the work plane of the machine to measure the laser spot position in the step of experimental measurement phase. Thus, the work distance from the CTI head and the work plane must be modified by subtracting the thickness of the plate. This means the work plane considered in this study is the top surface of the reference plate.

Calibration procedure

The calibration process of the laser galvanometric scanning head consists of two steps, which are detailed in the next two sections.

Calculation of pre-correction tables

In the first step, the pre-correction tables/matrix for all joint commands were calculated from the IKM without defects and a gird of theoretical laser spot positions in the work plane -X th (x th , y th ).

The theoretical positions of laser spot consist of 65 x 65 points in a calibration region of 560 mm x 560 mm. The pre-correction tables/matrix ([ ] 65x65 and [ ] 65x65 ) in bits were calculated according to the schema presented in Fig. 14.

Fig. 14. Calculation of pre-correction tables for joint commands.

Calculation of correction tables

In the second step, the correction tables were calculated from the experimental data and the IKM with identified assembly defects, as shown in Fig. 15. Firstly, the pre-correction tables were integrated in the ScanMaster Controller (SMC) of the controlling device. A subset of theoretical points extracted from the theoretical gird of points X th was used to generate real positions of laser spot in the top surface of the reference plate by using the ScanMaster Designer software. At each time the laser projected in the reference plate surface, its spot was captured by the camera.

Subsequently, all captured images were used as the input for the measurement method of laser spot positions, as described in Section 5. The obtained measuring data X exp was then used for the identification of 28 assembly defects  by using the algorithm developed in [START_REF] Godineau | Calibration of galvanometric scan heads for additive manufacturing with machine assembly defects consideration[END_REF]. Finally, the correction tables were calculated in the same manner for calculating the pre-correction tables, but using the IKM with already identified defects.

Data acquisition

Due to the dimensions of the reference plate are limited in (350 mm x 350 mm), the measurement of laser spot positions was performed at 19 x 11 positions. These positions of laser spots were generated by shooting the laser according to the order of theoretical grid points (209 points), as shown in Fig. 16. In the experiment, the positions of laser spot were measured in the world coordinate system of the reference plate (R p ). Hence, it is necessary to measure the position of laser spot corresponding to the origin of the world coordinate system of the machine (R w ) in (R p ) (Fig. 9).

Denote

is the origin of the world coordinate system of the machine (R w ) in (R p ) and is the position of the laser spot measured in the world coordinate system of the reference plate (R p ), the position of the laser spot in the world coordinate system of the machine X exp(Rw) were determined by Eq. ( 7):

(7) To evaluate the accuracy of the calibration, the same steps in the data acquisition presented in Fig. 15 was repeated. However, in this case, the correction tables were used instead of the precorrection tables. The measured positions of actual laser spot were then compared with those generated by the identified model of the machine. Fig. 18 presents the position error distribution between the measurement and the identified model with defects of the machine. It reveals that the position errors between the measured positions and those generated by the identified model are very small with an average error of 0.03 mm. The maximum position error is less than 0.075 mm. There are approximately 80% of positions with an error less than 0.04 mm (Fig. 19). The distribution of position errors was not homogeneous. In fact, the laser spot shape varies according to the laser shooting positions and the conditions of taking the images, resulting in the deviation of laser spot positions in the images. In comparison with the model without defects of the machine (i.e., the model using the precorrection tables), after the calibration process the average position error was significantly reduced from 0.38 mm (Fig. 17) to 0.03 mm (Fig. 18). The maximum error was also reduced from 1.00 mm to 0.075 mm.

Calibration results and discussion

In comparison with the calibration results obtained in [START_REF] Godineau | Calibration of galvanometric scan heads for additive manufacturing with machine assembly defects consideration[END_REF] for an industrial LPBF machine, which uses the same laser scanning head as the machine prototype in this study, the residual errors between the measurement and the identified model of the machine were higher. In [START_REF] Godineau | Calibration of galvanometric scan heads for additive manufacturing with machine assembly defects consideration[END_REF], after the calibration process, 95% of positions in the calibration zone presents an error less than 0.02 mm, and the maximum error is below 0.025 mm. The calibration results in this study were still not good as those obtained in [START_REF] Godineau | Calibration of galvanometric scan heads for additive manufacturing with machine assembly defects consideration[END_REF]. This can be explained by some following reasons:

 The difference between the geometry of the machine prototype used in this study and the model developed in [START_REF] Godineau | Calibration of galvanometric scan heads for additive manufacturing with machine assembly defects consideration[END_REF] for a real industrial machine. The calibration and measurement results are also affected by the vibration of the machine structure, the variation of light during measurements.

 The shape of the laser spot greatly varies over all the studied points. Thus, detection quality criterion should be defined in order to eliminate outliers.

 The area used for calibration in our case is limited by the size of the plate. Theoretically, the calculation of the correction tables/matrix with dimensions of 65x65 requires 65x65 measured points. However, the experimental data was only measured at 19x11 points in a small region (Fig. 16). For the elements of the correction matrix not having the corresponding experimental data, an interpolation strategy was applied. These elements can have an important influence on the identification of the machine model with defects. However, the final result of the calibration process reveals a deviation (Fig. 18 and19) similar to that observed on the measurement results carried out on the reference plate (Fig. 8). This is a very interesting finding and allows us to confirm the performance of the proposed in-situ measurement method.

A final point concerns the used laser source, which is different from that used in industrial machines. In the current study, a red laser with a wavelength of 633 nm in the visible range was used. On the other hand, in a commercial LPBF machine, the used laser has a wavelength of 1070 nm and located in the infrared range; therefore it is not visible with the used camera. One of possible variation sources in the optical path is due to a variation in the refractive index used in [START_REF] Godineau | Calibration of galvanometric scan heads for additive manufacturing with machine assembly defects consideration[END_REF].

For the silica glass used in the machine prototype, it is possible from the Sellmeier's formula to compute the refractive index [START_REF] Malitson | Interspecimen comparison of the refractive index of fused silica[END_REF][START_REF] Tan | Determination of refractive index of silica glass for infrared wavelengths by IR spectroscopy[END_REF] for the two laser source: n = 1.4551 for the 633 nm wavelength, and n' = 1.4496 for the 1070 nm wavelength. The difference of the laser spot location according to these two indexes with the model without defects is represented in Fig. 20. The results highlight the very small influence of the wavelength variation on the final result, which confirms our approach using a visible laser.

Conclusions and future works

In this work, we proposed an in-situ calibration approach for the laser galvanometric scanning head of LBPF machines by using a vision system. The calibration algorithm was based on the models developed in [START_REF] Godineau | Calibration of galvanometric scan heads for additive manufacturing with machine assembly defects consideration[END_REF]. The main contribution of this work is the development of an in-situ method to rapidly measure the positions of the laser spot used for the calibration process, and avoiding the marking and measuring phases of laser marks as in previous works. The obtained results on the calibration show that the position errors between the positions generated by the identified machine with defects and those measured by our method are very small with an average error of 0.03 mm and the maximum position error less than 0.075 mm. There are approximately 80% of positions with an error less than 0.04 mm.

The calibration results to date are not yet satisfactory with the constraints required by the manufacturer due to some reasons related to the difference related to the geometry between the used machine prototype and real industrial machines. However, by comparing between the identified model with defects and the measurement, we can confirm that the developed method is acceptable for the in-situ measurement with a good accuracy level.

In future work, the calibration area in the work plane and the number of laser spot position should be extended and augmented as in [START_REF] Godineau | Calibration of galvanometric scan heads for additive manufacturing with machine assembly defects consideration[END_REF] to obtain a sufficiently measured data and enhance the accuracy of the calibration process. Moreover, the proposed method will be implemented with an industrial LBPF machine, which has a geometric model suited with the algorithm developed in [START_REF] Godineau | Calibration of galvanometric scan heads for additive manufacturing with machine assembly defects consideration[END_REF] to confirm the effectiveness of the proposed method in terms of time and accuracy. Moreover, compared to other calibration methods, the method we propose also allows process monitoring

Fig. 2 .

 2 Fig. 2. (a) 3D modeling of the LBPF machine prototype used in the study, (b) high resolution monochrome camera and its fixture system installed on the machine prototype, and (c) the plate with four reference points placed on the work plane of the machine.

Fig. 3 .

 3 Fig. 3. The flowchart of the proposed in-situ calibration method.

Fig. 3

 3 Fig.3presents the flowchart of the proposed calibration method:

Fig. 4 .

 4 Fig. 4. Schematic diagram of the camera model.

Fig. 5 .

 5 Fig. 5. Plate certificated by GOM used for the camera calibration.

Fig. 6 .

 6 Fig. 6. Perspective distortion correction by applying a homography transformation: (a) initial image with perspective distortion and (b) image after the perspective warping with the homography H.

Fig. 7 .

 7 Fig. 7. Identification of grid points in the corrected image: (a) creating a theoretical grid of points and (b) determining a point of the grid (intersection of two lines) in the image.

Fig. 8 .

 8 Fig. 8. Distribution of position errors between the measured points (red) and the reference points (white) and the cumulative histogram of position errors.

Fig. 9 .

 9 Fig. 9. Illustrating the positioning of the reference plate on the work plane of the machine.

Fig. 10 .

 10 Fig. 10. Various shapes of the laser spot appearing in the image: (a) ellipse, (b) irregular shape, (c) the laser spot with the background noise, and (d) the laser spot with readout noise.

Fig. 11 .

 11 Fig. 11. Different methods of laser spot center detection.

Figure 12 .

 12 Figure 12. Detection errors caused by different methods.

Figure 13 .

 13 Figure 13. Influence of system vibration on measurement accuracy.

  Fig 2.a). The forward kinematic models (FKM) expresses the coordinates of the laser spot position in the work plane X = (x, y) as a function of the joint commands Q = ( x ,  y ) (Fig. 1 on the left) and assembly defects : X = f(Q, ), while the inverse kinematic models (IKM) allows determining the joint commands Q and assembly defects 

Fig. 15 .

 15 Fig. 15. Calculation of correction tables for joint commands.

Fig. 16 .

 16 Fig. 16. Order for laser shooting and laser spot positions to be measured.

Fig. 17

 17 Fig. 17 shows the deviation between the measured positions (red) and those of the model without defects (white). It is found that position error values are still important in compared to the requirement of the manufacturer. The average value of position errors is about 0.39 mm and the maximum value is about 1 mm.

Fig. 17 .

 17 Fig. 17. Distribution of position errors between the measured positions (red) and those of the model without defects (white).

Fig. 18 .

 18 Fig. 18. Distribution of position errors between the measured laser spot positions (red) and those of the identified model (white) after the calibration.

Fig. 19 .

 19 Fig. 19. Cumulative histogram of position errors from Fig. 18.

Fig. 20 .

 20 Fig. 20. Difference of the laser spot location when using two refractive indexes with the model without defects.

Table 1 .

 1 Estimated value of intrinsic parameters and standard deviation of the camera.

	Intrinsic parameters	f x (pixels)	f y (pixels)	c x (pixels)	c y (pixels)	s (pixels)	k 1	k 2	k 3	p 1	p 2
	Value and	11911.16	11911.83	5030.64	3593.25	0.2560	-0.0986	0.1171	-0.0410	-0.0002	0.0004
	the standard	+/-	+/-	+/-	+/-	+/-	+/-	+/-	+/-	+/-	+/-
	deviation	0.2783	0.2756	0.1240	0.1139	0.0127	0.0001	0.0011	0.0039	0.0000	0.0000

Table 2 .

 2 Images of laser spot simulated by the super-Gaussian distribution function without noises and with white Gaussian noises.

	p	Ideal image without noises	Image with noises
	1
	4
	8

based on the geometric link between the camera mark and the production drawing mark. This aspect will also be developed in future work.


Weize Quan

Kai Wang
email: kai.wang@gipsa-lab.grenoble-inp.fr

Dong-Ming Yan
email: yandongming@gmail.com

Denis Pellerin

Xiaopeng 2019 Zhang

Impact of Data Preparation and CNN's First Layer on Performance of Image Forensics: A Case Study of Detecting Colorized Images

Keywords: image forensics, colorized image, JPEG compression, convolutional neural network, generalization

In the field of image forensics, many convolutional neural network (CNN)-based forensic methods have been proposed and generally achieved the state-of-the-art performance. However, some questions are worth studying and answering regarding the trustworthiness of such methods, including for example the appropriateness of the discriminative information automatically extracted by CNN and the generalization performance on "unseen" data during the testing phase. In this paper, we study these questions in the case of a specific forensic problem of distinguishing between natural images (NIs) and colorized images (CIs). Through a series of experiments, we analyze the impact of data preparation and setting of the first layer of a recent state-of-the-art CNN-based method on the detector's forensic performance, in particular the generalization capability. We obtain some interesting observations which can serve as useful hints for carrying out image forensics experiments. Moreover, we propose a very simple method to improve the generalization performance of colorized image detection by combining decision results from CNN models with different settings at the network's first layer.

INTRODUCTION

With today's rapid growth of digital data communication, many of us communicate in and get information from cyberspace. We usually pay much attention to convenience, but have few considerations for the security of numerical data circulating on the internet, such as digital images. In consequence, people can be easily misled by fake contents in cyberspace, and the detection of such contents has received more and more attention 1 and become an important research direction in the field of cybersecurity and information forensics. In the meanwhile, convolutional neural network (CNN) has recently obtained notable success in computer vision and pattern recognition. An important reason is that CNN attempts to automatically learn hierarchical representation from available data in an "end-to-end" manner. Inspired by this success, many researchers have proposed CNN-based approaches for image forensics. For example, CNNs have been used to identify camera model [START_REF] Bondi | First steps toward camera model identification with convolutional neural networks[END_REF], to expose image forgery [START_REF] Zhou | Learning rich features for image manipulation detection[END_REF], and to detect synthetic images [START_REF] Quan | Distinguishing between natural and computer-generated images using convolutional neural networks[END_REF][START_REF] Yan | Recolored image detection via a deep discriminative model[END_REF]. These CNN-based forensic methods in general work better than traditional handcrafted-feature-based approaches. Despite this, some questions hidden behind the high performance are worth studying and answering, including the following ones: What is the CNN model using as the discriminative information, i.e., is it the "essential" difference between different kinds of images? Is the CNN just overfitting on training data in some aspects that are not the primary factors for the considered forensic problem? How can CNN generalize well on "unknown" data during the testing stage? These questions are closely related to the trustworthiness and the practical applicability of CNN-based forensics.

In this paper, we focus on the case of CNN-based colorized image detection and try to study the above questions. State-of-the-art colorization algorithms, more or less leveraging the powerful capacity of deep neural networks, can automatically colorize a grayscale image to obtain a high-quality color image. Fig. 1 shows a group of [START_REF] Deng | ImageNet: A large-scale hierarchical image database[END_REF] and three colorized images generated by the colorization algorithm proposed in [START_REF] Larsson | Learning representations for automatic colorization[END_REF], [START_REF] Zhang | Colorful image colorization[END_REF], and [START_REF] Iizuka | Let there be color!: joint end-toend learning of global and local image priors for automatic image colorization with simultaneous classification[END_REF], respectively.

images, the left-most column is the original color images, and the remaining three columns are colorized images produced by three recent and advanced colorization algorithms (respectively with the name Ma [START_REF] Larsson | Learning representations for automatic colorization[END_REF], Mb [START_REF] Zhang | Colorful image colorization[END_REF] and Mc [START_REF] Iizuka | Let there be color!: joint end-toend learning of global and local image priors for automatic image colorization with simultaneous classification[END_REF], from left to right), which take the grayscale version of the left-most column as input. It is indeed difficult to distinguish, with naked human eyes, which images are artificially colorized. Accordingly, distinguishing between natural images (NIs) and colorized images (CIs) has drawn increasing interest among the image forensics research community [START_REF] Guo | Fake colorized image detection[END_REF][START_REF] Zhuo | Fake colorized image detection with channel-wise convolution based deep-learning framework[END_REF].

Recently, Guo et al. [START_REF] Guo | Fake colorized image detection[END_REF] first considered and studied this new and important forensic problem. According to the statistical difference between NIs and CIs in the hue, saturation, dark, and bright channels, two different features were designed to catch this forensic difference. Then, they trained the support vector machine (SVM) classifiers to identify fake colorized images. Zhuo et al. [START_REF] Zhuo | Fake colorized image detection with channel-wise convolution based deep-learning framework[END_REF] achieved the state-of-the-art detection performance on the experimental database shared by [START_REF] Guo | Fake colorized image detection[END_REF], by making use of an advanced CNN-based color image steganalyzer called WISERNet (WIder SEparate-then-Reunion Network) [START_REF] Zeng | WISERNet: Wider separatethen-reunion network for steganalysis of color images[END_REF]. WISERNet is particularly good at detecting weak signals in images and its first layer has thirty 5 × 5 residual filters from the well-known steganalytic Spatial Rich Model (SRM) [START_REF] Fridrich | Rich models for steganalysis of digital images[END_REF]. As shown later in this paper, we find that data preparation and setting of CNN's first layer can have big impact on the forensic performance of WISERNet, especially the generalization capability. Here the generalization (or blind detection) performance indicates the detection performance on testing data in which CIs are generated by an "unknown" colorization method with regard to the training procedure, e.g., when WISERNet is trained with CIs from Ma [START_REF] Larsson | Learning representations for automatic colorization[END_REF] and later tested on CIs from Mb [START_REF] Zhang | Colorful image colorization[END_REF] and Mc [START_REF] Iizuka | Let there be color!: joint end-toend learning of global and local image priors for automatic image colorization with simultaneous classification[END_REF].

The remainder of this paper is organized as follows. Section 2 presents the studied problem and technical details. Section 3 reports the experimental results and our analysis. Section 4 draws the conclusions and proposes some future working directions.

DATA PREPARATION AND NETWORK 2.1 Motivation

For this specific forensic problem of colorized image detection, we have some observations about data and network: CIs shared by the authors of [START_REF] Guo | Fake colorized image detection[END_REF] and used in [START_REF] Guo | Fake colorized image detection[END_REF][START_REF] Zhuo | Fake colorized image detection with channel-wise convolution based deep-learning framework[END_REF] are in a lossless format without compression on the artificially generated color information, and NIs from ImageNet are in the lossy JPEG format; in the meanwhile, the weights of first layer of WISERNet are initialized with SRM residual filters [START_REF] Fridrich | Rich models for steganalysis of digital images[END_REF] and untrainable2 . Therefore, it is natural to raise the following question: Does WISERNet, as used in [START_REF] Zhuo | Fake colorized image detection with channel-wise convolution based deep-learning framework[END_REF], rather capture the difference of processing history between NIs and CIs (i.e., JPEG compressed or not), or the desired "essential" color difference?

In order to answer this question, in this work, we experimentally study the impact of two important but until now ignored and underestimated factors on CNN's forensic performance as follows:

(1) we study the impact of data by constructing two datasets in which CIs are with/without JPEG compression, and (2) we study the impact of network by adopting two different strategies for the setting of the first layer of WISERNet.

Data Preparation

We construct two sets of data and the only difference is whether CIs are JPEG compressed or not. Following [START_REF] Guo | Fake colorized image detection[END_REF] and [START_REF] Zhuo | Fake colorized image detection with channel-wise convolution based deep-learning framework[END_REF], three stateof-the-art colorization algorithms (Ma [START_REF] Larsson | Learning representations for automatic colorization[END_REF], Mb [START_REF] Zhang | Colorful image colorization[END_REF], and Mc [START_REF] Iizuka | Let there be color!: joint end-toend learning of global and local image priors for automatic image colorization with simultaneous classification[END_REF]) are adopted for producing CIs. NIs come from ImageNet dataset [START_REF] Deng | ImageNet: A large-scale hierarchical image database[END_REF]. We use 10,000 natural images from ImageNet validation dataset to construct training and validation dataset (with the ratio 4:1). The exact indexes of these images can be found from [START_REF] Larsson | Learning representations for automatic colorization[END_REF]. Then, we remove 899 grayscale images and 1 CMYK image from the remaining 40,000 images of ImageNet validation dataset (the total number of images in this dataset is 50,000), and obtain 39,100 NIs to construct testing dataset. Note that, the magnitude of testing dataset is far larger than the settings reported in [START_REF] Guo | Fake colorized image detection[END_REF] and [START_REF] Zhuo | Fake colorized image detection with channel-wise convolution based deep-learning framework[END_REF]. We employ the three colorization methods mentioned above to produce the corresponding colorized images. In addition, we construct another dataset where we only replace the CIs (the original output of colorization algorithms) with a JPEG compressed version. In details, the compressed CI is generated in the following way: given an original CI, we first obtain the quantization table of the corresponding NI (i.e., the NI which shares the same grayscale version) using the Matlab JPEG Toolbox [START_REF] Sallee | Matlab JPEG Toolbox[END_REF]. Then, we estimate the quality factor from the above quantization table of NI using the method proposed in [START_REF] Cogranne | Determining JPEG image standard quality factor from the quantization tables[END_REF] and compress the CI with estimated quality factor. Hereafter, for ease of presentation, dataset with/without JPEG compression means that CIs in this dataset are with/without JPEG compression.

It is necessary to justify the JPEG compression of CIs mentioned above and prove that the artificial color information in CIs before compression is indeed considered as uncompressed. To this end, we quantitatively analyze the JPEG blocking artifacts of the two datasets with the forensic measure of K F [START_REF] Fan | Identification of bitmap compression history: JPEG detection and quantizer estimation[END_REF]. Larger K F means stronger JPEG blocking artifacts. We analyze the blocking artifacts of NIs, as well as CIs with or without JPEG compression, in the color space of YCbCr, which partitions images into luminance and chrominance and which is the space adopted by JPEG standard. We calculate K F of validation dataset (2,000 images for each class) and report the average values in Table 1. Compared with the column of "NI", K F of "Y" of the columns of "Ma", "Mb" and "Mc" are very similar, while that of "Cb", "Cr" have a large gap. This is an experimental proof that the color information in CIs without JPEG compression is in fact considered as forensically uncompressed. Furthermore, this gap is significantly decreased after compressing the original CIs with the same quality factor as that of the corresponding NIs (compare K F of "Cb", "Cr" of the column of "NI" with that of the columns of "Ma-C", "Mb-C" and "Mc-C"). This implies that the difference between NIs and JPEG compressed CIs becomes very small in terms of JPEG compression trace, and that this trace seems quite obvious before compression which may impact the colorized image detection, e.g., as in the experimental setting of [START_REF] Guo | Fake colorized image detection[END_REF][START_REF] Zhuo | Fake colorized image detection with channel-wise convolution based deep-learning framework[END_REF] where CIs are not compressed.

Network Settings

The first layer of WISERNet used in [START_REF] Zhuo | Fake colorized image detection with channel-wise convolution based deep-learning framework[END_REF] is a channel-wise convolutional layer where the convolutional kernels are fixed as the thirty 5 × 5 SRM residual filters borrowed from [START_REF] Fridrich | Rich models for steganalysis of digital images[END_REF]. In order to study the sensitivity and impact of the first layer of WISERNet on the different datasets, i.e., CIs with/without JPEG compression, we adopt another setting in which WISERNet's first layer is initialized in a conventional way with Gaussian random distribution and is trainable (denoted by WISERNet-Gauss). We expect that under these two settings, the network focuses on different information in NIs and CIs to carry out the classification. In the next section, we present the experimental results related to the two datasets and the two settings of the CNN's first layer, as well as our proposed simple combination method to improve generalization.

EXPERIMENTAL RESULTS

Implementation Details

All the experiments are implemented with PyTorch 0.3.1. The GPU version is GeForce ® GTX 1080Ti of NVIDIA ® corporation. We train the network according to the experimental setting described in [START_REF] Zhuo | Fake colorized image detection with channel-wise convolution based deep-learning framework[END_REF].

Table 2:

The performance (HTER, in %, lower is better) of WISERNet [START_REF] Zhuo | Fake colorized image detection with channel-wise convolution based deep-learning framework[END_REF] and WISERNet-Gauss trained on dataset without JPEG compression. For each row, "X" (e.g., "WISER-Net") means testing on dataset without JPEG compression and "X-cro" (e.g., "WISERNet-cro") means cross-testing on dataset with JPEG compression. The generalization performance results are presented in italics (same in Table 3). Following [START_REF] Guo | Fake colorized image detection[END_REF] and [START_REF] Zhuo | Fake colorized image detection with channel-wise convolution based deep-learning framework[END_REF], the half total error rate (HTER) is employed to evaluate the detection performance. The HTER is defined as the average of misclassification rates (in %) of NIs and CIs. In this work, all reported results are the average of 5 runs.

Method

Results and Analysis

In this subsection, we first study the impact of CIs with/without JPEG compression and setting of WISERNet's first layer on the classification accuracy (i.e., trained and tested on CIs of same colorization algorithm) and generalization (i.e., trained and tested on CIs of different colorization algorithms). Then, we propose a simple yet effective method to improve the generalization of WISERNet. Table 2 reports the performance of WISERNet and WISERNet-Gauss trained on dataset without JPEG compression and tested on dataset without/with compression. On the contrary, Table 3 reports the performance of these two networks trained on dataset with JPEG compression and tested on dataset with/without compression. We also tested a variant of WISERNet with trainable first layer initialized with SRM filters and found that the performance is similar to original WISERNet, so here we do not show results of this variant.

Compared with the row of "WISERNet" in Table 2, we can find that the classification and generalization error rate in Table 3 both increases. In other words, when only replacing CIs with corresponding compressed version, the forensic performance (especially generalization) obviously drops. Meanwhile, the detection performance significantly decreases when we train WISERNet on dataset without JPEG compression and test it on dataset with compression (compare the rows of "WISERNet" and "WISERNet-cro" in Table 2), whereas this phenomenon does not exist in the case of training on dataset with JPEG compression and testing on dataset without compression (compare the rows of "WISERNet" and "WISERNetcro" in Table 3). These results indicate that WISERNet takes the trace of JPEG compression as the important discriminative feature and thus has good generalization when trained and tested both on dataset without JPEG compression of CIs (the row of "WISERNet" in Table 2). In addition, in Table 2, when the first layer of WISER-Net is initialized with Gaussian random distribution and trainable (the so-called WISERNet-Gauss), the generalization is not as good as the original WISERNet (compare the rows of "WISERNet" and "WISERNet-Gauss"). On the contrary, the rows of "WISERNet" and "WISERNet-Gauss" in Table 3 are relatively close, where the networks are trained and tested on datasets with JPEG compression of CIs. This further implies that the SRM filters can strongly capture the trace of JPEG compression. To summarize, when the CIs are without JPEG compression, the WISERNet can achieve very good detection performance, especially generalization, because this model uses SRM filters in the beginning of network which coincidentally and mistakenly detects the trace of JPEG compression. This is however not desirable and leads to the dramatic performance drop in the row of "WISERNet-cro" in Table 2. In the meanwhile, when the training database is carefully prepared as in Table 3, the original WISERNet is indeed a good choice for this forensic task, providing better overall performance than WISERNet-Gauss. Furthermore, we notice that the detection performance differs for WISERNet and WISERNet-Gauss, e.g., the rows of "WISERNet" and "WISERNet-Gauss"in Table 3. The only difference between these two networks is in the first layer. Intuitively, this difference may lead two networks to extract different discriminative features to some extent. We qualitatively analyze this difference, and visualize the FFT (fast Fourier transform) of the first-layer kernels of these two networks after training, and the corresponding results are shown in Fig. 2. Many kernels in the first layer of WISERNet [Fig. 2(a)] have an apparent high-pass response, whereas the kernels of WISERNet-Gauss [Fig. 2(b), (c) and(d)] mainly capture the band-pass frequency information. Based on these observations, we introduce a simple yet effective method to further improve the generalization somehow borrowing idea from ensemble learning. Specifically, we combine the predictions of these two networks to obtain the final prediction according to a simple criterion: the final prediction is CI when the prediction of either of two networks is CI, otherwise the image is NI. The rationale behind this criterion is that we trust the (different) discriminative features of both networks which are used to determine whether an image is CI. The corresponding results are reported in rows of "WISERNet-Ensemble" and "WISERNet-Ensemble-cro" in Table 2 and3. Obviously, the generalization can be improved by this method while decreasing very slightly the classification accuracy (e.g., compare the rows of "WISERNet", "WISERNet-Gauss", and "WISERNet-Ensemble" in Table 3). Despite of its simplicity, to the best of our knowledge, this ensemble strategy with different initialization methods (SRM and Gaussian) is first used in the literature for improving the generalization of CNN-based image forensic detector. The success is probably due to the high diversity of the two initializations, more diverse than the conventional way of using only one initialization method (e.g., Gaussian) with different random sampling.

Discussion and Summary

Unlike traditional handcrafted-feature-based forensic methods, recent CNN-based approaches are relatively difficult to understand concerning what is the discriminative information used by CNN, and sometimes this information can be surprising and misleading. Taken the above CNN-based colorized image detection as an example, when within the dataset there is an apparent difference in JPEG compression, the CNN with SRM filters captures to some extent this trace and takes it as part of the discriminative information. Consequently, the high performance of CNN model benefits from and covers up the potential pitfall existing in the dataset. As far as we know, there is no existing work that considers and studies this kind of phenomenon for CNN-based image forensics. From this case study, we get some useful hints: 1) reducing as much as possible the impact of image generation and processing history (this information is not relevant to the task at hand), so we need to carefully prepare the data; 2) carefully using some existing filters (e.g., SRM filters) in the beginning of CNN because these filters have strong capacity of capturing image processing history and thus are risky to be used if the dataset has not been properly prepared.

CONCLUDING REMARKS

This paper considered the CNN-based colorized image detection as an example and studied the impact of image generation pipeline and CNN's first layer on the classification accuracy and generalization. From this case study, we learned some lessons related to the trustworthiness of CNN-based forensics, which until now have been ignored among the community but would be helpful for researchers in the field to avoid biased data preparation and network design. We think that our first study in this direction can be useful for other forensic tasks, e.g., detection of computer graphics images and GAN-generated fake images where similar problems and pitfalls may exist. We would like to continue our work on improving the generalization of deep-learning-based detectors, either based on an ensemble of classifiers or other approaches.

Figure 1 :

 1 Figure1: From left to right: each row includes a natural image taken from ImageNet[START_REF] Deng | ImageNet: A large-scale hierarchical image database[END_REF] and three colorized images generated by the colorization algorithm proposed in[START_REF] Larsson | Learning representations for automatic colorization[END_REF],[START_REF] Zhang | Colorful image colorization[END_REF], and[START_REF] Iizuka | Let there be color!: joint end-toend learning of global and local image priors for automatic image colorization with simultaneous classification[END_REF], respectively.

Figure 2 :

 2 Figure 2: Visualization of FFT of the first-layer filters of WISERNet [(a)] and WISERNet-Gauss [(b), (c), and (d)]. From left to right: SRM, filters in R, G, and B channel, respectively. Note that, filters for R, G, and B in WISERNet are all SRM.

Table 1 :

 1 K F of validation dataset in color space of YCbCr. "X-C" means the JPEG compressed version of colorized images produced by X colorization method.

	Channel	NI	Ma	Mb	Mc	Ma-C Mb-C Mc-C
	Y	0.3491 0.3227 0.3121 0.3144 0.3650 0.3667 0.3650
	Cb	0.6757 0.0661 0.0596 0.0434 0.6552 0.6318 0.6741
	Cr	0.7023 0.0653 0.0625 0.0489 0.6185 0.6548 0.6811

Table 3 :

 3 HTER (in %, lower is better) of different networks trained on dataset with JPEG compression. For each row, "X" (e.g., "WISERNet") means testing on dataset with JPEG compression and "X-cro" (e.g., "WISERNet-cro") means crosstesting on dataset without JPEG compression. Ensemble-cro 0.80 11.72 15.81 6.72 0.85 1.10 2.30 1.40 0.95

		Ma		Mb	Mc
		Ma Mb Mc Ma Mb Mc Ma Mb Mc
	WISERNet	0.34 4.49 3.67 3.27 0.23 0.30 3.15 0.98 0.21
	WISERNet-cro	20.56 40.06 40.65 36.99 22.97 32.47 34.82 28.37 26.31
	WISERNet-Gauss	0.58 20.90 25.93 24.98 0.43 2.03 23.39 1.41 0.46
	WISERNet-Gauss-cro	0.89 27.96 31.70 28.48 10.08 27.95 22.18 14.37 10.53
	WISERNet-Ensemble	0.60 3.86 3.61 2.82 0.38 0.43 2.68 0.82 0.38
	WISERNet-Ensemble-cro 0.96 26.44 29.40 25.79 8.99 23.92 19.68 13.58 9.66
	Method	Ma Ma Mb	Mc	Mb Ma Mb Mc Ma Mb Mc Mc
	WISERNet	0.78 18.90 24.28 9.35 0.93 2.98 4.78 2.79 0.89
	WISERNet-cro	0.69 13.78 17.40 10.31 0.67 1.08 4.40 1.29 0.66
	WISERNet-Gauss	0.76 25.52 27.97 9.03 0.89 5.26 4.05 3.53 0.96
	WISERNet-Gauss-cro	0.72 22.97 29.27 12.53 0.74 3.86 5.49 2.44 0.84
	WISERNet-Ensemble	0.82 16.00 20.43 6.08 0.93 2.10 2.44 2.14 1.00
	WISERNet-

See https://www.cbc.ca/news/technology/fighting-fake-images-military-1.4905775

Confirmed by email with authors of[START_REF] Zhuo | Fake colorized image detection with channel-wise convolution based deep-learning framework[END_REF].

ACKNOWLEDGMENT

This work is funded in part by Beijing Natural Science Foundation (L182059), National Natural Science Foundation of China (61620106003 and 61772523), and the French National Research Agency (DEFALS ANR-16-DEFA-0003, ANR-15-IDEX-02). W. Quan acknowledges the support from the UCAS Joint PhD Training Program. We thank L. Zhuo for helpful discussions about the details of [15] and Dr. W. Fan for sharing implementation of the JPEG blocking measure [4].