Baptiste Magnier

Edge Detection: A Review of Dissimilarity Evaluations and a Proposed Normalized Measure

Keywords: Edge detection, supervised evaluation, distance measures

In digital images, edges characterize object boundaries, so edge detection remains a crucial stage in numerous applications. To achieve this task, many edge detectors have been designed, producing different results, with various qualities of segmentation. Indeed, optimizing the response obtained by these detectors has become a crucial issue, and effective contour assessment assists performance evaluation. In this paper, several referenced-based boundary detection evaluations are detailed, pointing out their advantages and disadvantages, theoretically and through concrete examples of image edges. Then, a new normalized supervised edge map quality measure is proposed, comparing a ground truth contour image, the candidate contour image and their associated spatial nearness. The effectiveness of the proposed distance measure is demonstrated theoretically and through several experiments, comparing the results with the methods detailed in the state-of-the-art. In summary, compared to other boundary detection assessments, this new method proved to be a more reliable edge map quality measure.

I. INTRODUCTION

In computer science, all systems, especially automated information processing structures, must be evaluated before being developed, principally for industrial applications or medical data. Image processing is no exception to this rule. In image analysis, image segmentation is one of the most critical tasks and represents an essential step in low-level vision. All the methods developed therefore have to be tested and assessed, whether regarding edge detection, point matching, region segmentation or image restoration/enhancement.

Edge detection represents one of the pioneer theoretical works in image processing tasks [START_REF] Marr | Theory of Edge Detection[END_REF] and remains a key point in many applications. It is extensively used because boundaries include the most important structures in the image [START_REF] Torre | On edge detection[END_REF] [START_REF] Papari | Edge and line oriented contour detection: State of the art[END_REF]. Furthermore, edge detection itself could be used to qualify a region segmentation technique [START_REF] Lopez-Molina | On the impact of anisotropic diffusion on edge detection[END_REF] [START_REF] Magnier | Texture Removal by Pixel Classification using a Rotating Filter[END_REF]. In addition, contour extraction remains a very useful preprocessing step in image segmentation, registration, reconstruction, interpretation and tracking [START_REF] Magnier | Demonceaux Highly Specific Pose Estimation with a Catadioptric Omnidirectional Camera[END_REF]. An efficient boundary detection method should create a contour image containing edges at their correct locations with a minimum of misclassified pixels. Edge detection assessment is therefore an essential field of study, but research has still not necessarily gone deep enough regarding contour detection for digital images. Contrary to region segmentation evaluations, which may consider color attributes, contour detection assessments generally use binary images. Thus, an Ecole des Mines d'Alès, Site de Croupillac, LGI2P, 7. rue Jules Renard, 30100 ALES, France. E-mail: baptiste.magnier@mines-ales.fr Special thanks to Adam Clark Gimming. edge detection evaluation leads to a direct assessment of the considered binary image. Moreover, measures of boundary detection evaluation are based on the fact that a reliable edge map should characterize all the relevant structures of the image. It should also create a minimum of spurious pixels or holes (oversights). Therefore, an evaluation can be used to assess and improve an algorithm, or to optimize edge detector parameters [START_REF] Heath | A robust visual method for assessing the relative performance of edge-detection algorithms[END_REF], for example thresholds in the last stage of the detection.

In edge detection assessment, the measurement process can be classified as using either unsupervised or supervised evaluation criteria. The first class of methods exploits only the input data image and gives a score of coherence that qualifies the result given by the algorithm [START_REF] Heath | A robust visual method for assessing the relative performance of edge-detection algorithms[END_REF][17] [START_REF] Panetta | Non-reference medical image edge map measure[END_REF]. The second class computes a similarity/dissimilarity measure between a segmentation result and a ground truth obtained from synthetic data or an expert judgment [START_REF] Chabrier | Comparative study of contour detection evaluation criteria based on dissimilarity measures[END_REF][22] [START_REF] Lopez-Molina | Quantitative error measures for edge detection[END_REF]. As edge extraction is performed for image processing tasks and computer vision, similarity measures are important within a broad flied of study, for example in image interpretation, consisting in automatically extracting or recognizing objects in an image, compared to a model. Moreover, the assessment can compute a similarity or a dissimilarity of the shapes attributes between two of binary objects [START_REF] Dubuisson | A modified Hausdorff distance for object matching[END_REF][22] [START_REF] Magnier | A quantitative error measure for the evaluation of roof edge detectors[END_REF]. Thus, the problem of supervised edge detection assessment amounts to a question of pattern recognition. Nevertheless, the edge position of an object could be interpreted in different ways, as represented in Fig. 1. Indeed, for a vertical step edge, an edge can be located either on the left, or on the right (which corresponds to an inner or an outer contour). Considering a vertical blurred edge (i.e. ramp contour [START_REF] Bourennane | Generalization of Canny-Deriche filter for detection of noisy exponential edge[END_REF]), the true edge is placed in the middle of the blur, but the question of the position remains the same when the size of the blur is even. In the crest lines case [START_REF] Magnier | A quantitative error measure for the evaluation of roof edge detectors[END_REF], true edges are chosen in the middle of the ridge or of the valley when the width of the ridge/valley is equal to a odd number (i.e. at the maxima -top of ridges-or minima -bottom of valleys- [START_REF] Magnier | Multi-scale crest line extraction based on half gaussian kernels[END_REF] ). Finally, for a single real or synthetic image, as several contour chains constituting the ground truth depend on several positions of the true pixels, thus the combinations of the boundary pixel placements are numerous, so many chains could be created/chosen. For example, we can create a ground truth boundary map of the image (a) in Fig. 1 with as many combinations as there are pixels in Contour 1, 2 and 3. This observation shows that the evaluation must not be binary, but rather illustrates the importance of taking into account the misplaced edge distance regarding the desired contour of the ground true image. For all these reasons, as described in [START_REF] Papari | Edge and line oriented contour detection: State of the art[END_REF], quantitative assessments are generally lacking in proposed edge detection methods.

In image segmentation evaluation, the Structural Similarity Index (SSIM ) estimates the visual impact of shifts in an image [START_REF] Wang | Image Quality Assessment: from Error Visibility to Structural Similarity[END_REF]. This measure is based on the grayscale information concerning the attributes that represent the structure of objects in the scene. SSIM consists of three local comparison functions, namely luminance comparison, contrast comparison, and structure comparison between two signals excluding other remaining errors. Unlike the P SN R (Peak Signal to Noise Ratio) or RM SE (Root-Mean-Square Error), which are measured at the global or the image level, the SSIM is computed locally by moving a 8×8 window for each pixel. The final score of an entire image corresponds to the mean of all the local scores. Applications of this image quality evaluation include automatically judging the performance of compression or restoration algorithms, concerning grayscale or color images. Even though SSIM can be applied in the case of an edge detection evaluation, in the presence of too many areas without contours, the obtained score is not efficient or useful (in order to judge the quality of edge detection with the SSIM , it is necessary to compare with an image having detected edges situated throughout the image areas).

This work focusses on comparisons of supervised edge detection evaluations with respect to binary representation of the boundaries. As introduced above, a supervised evaluation process estimates scores between a ground truth and a candidate edge map (both binary images). These scores could be evaluated counting the number of erroneous pixels, but also through spatial distances of misplaced or undetected contours; this paper outlines the various algorithms presented in the literature. In addition, a new supervised edge map quality measure based on the distances of misplaced pixels is presented and compared to the others; the score obtained by this measure is normalized and can be interpreted for each type of contour. In order to present the effectiveness and efficiency of the proposed method, this new formula is compared with others presented and detailed in the state-of-theart. Thus, several comparisons of edge detection assessments are carried out for different perturbations such as: addition of false positive points, creation of false negative points, oversegmentation near the edge (until total dilation of the edge), contour displacement, modification of the image size... These experiments indicate that some measures are not appropriate for the evaluation of edge detection for each type of boundary, and an ideal measure must be one that reacts as coherently possible to reality. Therefore, this paper shows the relevance of a boundary detection assessment that takes into consideration the distance of both the false positive and the false negative points created or missed by the boundary detection process.

The remainder of this paper is organized as follows. Section II is devoted to an overview of error measures based on statistics. Next, Section III reviews most existing referencebased edge measures involving distances of erroneous points, then points out the advantages and drawbacks of different measures, followed by the presentation of a new measure. Section IV presents experimental results for different synthetic data, and then quality measure results for real images. Finally, Section V gives perspectives for future work and draws the conclusions of the study.

II. CONFUSION MATRIX-BASED ERROR ASSESSMENTS

To assess an algorithm, the confusion matrix remains a cornerstone in boundary detection evaluation methods. Let G t be the reference contour map corresponding to ground truth and D c the detected contour map of an image I. Comparing pixel per pixel G t and D c , the first criterion to be assessed is the common presence of edge/non-edge points. A basic evaluation is compounded from statistics resulting from a confusion matrix. To that effect, G t and D c are combined. Afterwards, denoting | • | as the cardinality of a set, all points are divided into four sets:

• True Positive points (TPs), common points of G t and D c : On the one hand, let us consider boundary detection of natural images, FPs appear in the presence of noise, texture or other contours influencing the filter used by the edge detection operator. On the other hand, FNs represent holes in a contour of D c (generally caused by blurred edges in the original image I). For example, an incorrect threshold of the segmentation could generate both FPs and FNs. In the experiment, TPs, FPs and FNs are respectively represented in green, red and blue: see illustrations in Fig. 2 for more details. Computing only FPs and FNs enables a segmentation assessment to be performed [START_REF] Magnier | Fast anisotropic edge detection using gamma correction in color images[END_REF] [START_REF] Magnier | Multi-scale crest line extraction based on half gaussian kernels[END_REF]. Yet, combining at least these two quantities enables

T P = |D c ∩ G t |, • False
P * m (Gt, Dc) = 1 - T P T P + F P + F N None Segmentation Success Ratio [55] SSR (Gt, Dc) = 1 - T P 2 |Gt| • |Dc| None Complemented Φ measure [56] Φ * (Gt, Dc) = 1 - T P R • T N T N + F P None Complemented χ 2 measure [60] χ 2 * (Gt, Dc) = 1 - T P R -T P -F P 1 -T P -F P • T P + F P + F P R T P + F P None Complemented Fα measure [39] F * α (Gt, Dc) = 1 - P REC • T P R α • T P R + (1 -α) • P REC , with P REC = T P T P + F P α ∈]0; 1]
a segmented image to be assessed more precisely. Thus, we present a list of statistical measures -others are detailed in [START_REF] Basseville | Distance measures for signal processing and pattern recognition[END_REF]-, and a good-quaity edge detection method should obtain the smallest response for the three following indicators [START_REF] Baddeley | An error metric for binary images[END_REF] [9]:

               Over-detection error : Over(G t , D c ) = F P |I| -|G t | , Under-detection error : U nder(G t , D c ) = F N |G t | ,
Localization-error [START_REF] Lee | A comparative performance study of several global thresholding techniques for segmentation[END_REF]:

Loc(G t , D c ) = F P + F N |I| .
One of the pioneer works in quantitative edge evaluation, reported by Deutsch and Fram [START_REF] Deutsch | A quantitative study of the orientation bias of some edge detector schemes[END_REF] computed two parameters P 1 and P 2 . The first evaluates the distribution of false positive points in function of the number of columns containing edge points, whereas P 2 quantifies the edge detection by counting the missing points for each line of D c . Thus, P 1 and P 2 parameters are given by:

P 1 (G t , D c ) = 1 - n sig n sig + (n noise + F P ) • w stan • |G t | w 1 • |I| . (1)
and

P 2 (G t , D c ) = 1 - n r w 2 -1 -1 - n noise |G t | w1 1 - n noise |G t | w1 , (2)
with:

       n noise = F P • |G t | T N + F P n sig = T P -n noise 1 -n noise ,
where n r and w 1 represent respectively the number of rows and columns in D c which contains a least one edge point (F P or T P point). Also, w stan corresponds to the number of columns of G t (which is the same as D c ), w 2 is the number of rows of the envelop rectangle containing D c . These two parameters are close to 0 when the segmentation is efficient and tend towards one for poor detection. However, P 1 and P 2 would be more suited for the evaluation of vertical edges (see Fig. 3). On the contrary, the score of P 1 can be over 1 when w 1 w stan , as in Fig. 5. Moreover, concerning P 2 , when n r = w 2 , so nr w2 = 1 and

P 2 (G t , D c ) = 1 -(1-nnoise/|Gt|) w 1
(1-nnoise/|Gt|) w 1 = 0, translating, wrongly, a perfect segmentation, as in Fig. 3 (b), (c), (e), (f) and Fig. 5. Several edge detection evaluations involving confusion matrices are presented in Table I. Only TPs are an indicator, as for SSR, this measure is normalized in function of |G t |•|D c | and decreases with improved quality of detection, with SSR = 0 qualifying a perfect segmentation.

{ { (a) T (b) C (c) L (d) C1 P1(L, C) = 1.0244 P1(C, L) = 1.200 P1(C, C1) = 0 P2(L, C) = 0 P2(C, L) = 0 P2(C, C1) = 0.1429 (e) C2 (
The Binary Signal-to-Noise Ratio [START_REF] Wilson | A new metric for greyscale image comparison[END_REF] B SN R (see table I) is inspired by the Signal-to-Noise Ratio (SN R). It corresponds to a measure comparing the level of a signal in a reference signal to the level of the noise in a desired signal. Thus, B SN R computes a global error in function of the FPs and FNs; the fewer the numbers of FPs and FNs, the more the score increases and tends to the infinity when G t = D c .

Finally, the complemented Performance measure P * m presented in table I considers directly and simultaneously the three entities T P , F P and F N to assess a binary image [START_REF] Grigorescu | Contour detection based on nonclassical receptive field inhibition[END_REF]. The measure is normalized and decreases with improved quality of detection, with P * m = 0 qualifying perfect segmentation. By combining F P , F N , T P and T N , another way to display evaluations is to create Receiver Operating Characteristic (ROC) [START_REF] Bowyer | Edge detector evaluation using empirical ROC curves[END_REF] curves or Precision-Recall (PR) [START_REF] Martin | Learning to detect natural image boundaries using local brightness, color, and texture cues[END_REF], involving True Positive Rates (T P R) and False Positive Rates (F P R):

     TPR = T P T P + F N FPR = F P F P + T N .
Derived from T P R and F P R, the three measures Φ, χ 2 and F α (detailed in Table I) are frequently used in edge detection assessment. Using the complement of these measures results in a score close to 1 indicating good segmentation, and a score close to 0 indicating poor segmentation. Among these three measures, F α remains the most stable because it does not consider the TNs, which are dominant in edge maps. Indeed, taking into consideration T N in Φ and χ 2 influences solely the measurement (as is the case in huge images). These measures evaluate the comparison of two edge images, pixel per pixel, tending to severely penalize an (even slightly) misplaced contour. Furthermore, they depend heavily of the reference image G t which could be interpreted in different ways, as the different contours presented in Fig. 1. So statistical measures do not indicate enough significant variations of the desired contour shapes in the course of an evaluation (as illustrated in Fig. 5). As this penalization tends to be too severe, some evaluations resulting from the confusion matrix recommend incorporating spatial tolerance, particularly for the assimilation of TPs [48] [7] [39]. This inclusion could be carried by a distance threshold or a dilation of D c and/or G t , as in [START_REF] Demigny | On optimal linear filtering for edge detection[END_REF] (see Eq. ( 58)). Such a strategy of assimilation leads to counting several near contours as stripes parallel to the desired boundary (issued from the edge detector itself or a blur/texture in the original image). For example, awarding a spatial tolerance for the FPs will ensure the same score for an over-segmentation near the edges (experiments illustrated in Fig. 13), if the spatial tolerance stays greater than the FP distances from the true pixel. Tolerating a distance from the true contour and integrating several TPs for one detected contour are opposite to the principle of unicity in edge detection expressed by the 3rd Canny criterion: an optimal edge detector must produce a single response for one contour [START_REF] Canny | A computational approach to edge detection[END_REF]. Thus, from the discussion below, only one FP or one TP should be considered in the boundary detection evaluation process. Finally, to perform an edge evaluation, the assessment should penalize a misplaced edge point proportionally to the distance from its true location.

III. ASSESSMENT INVOLVING DISTANCES OF MISPLACED

PIXELS

A reference-based edge map quality measure requires that a displaced edge should be penalized in function not only of FPs and/or FNs but also of the distance from the position where it should be located. Table II reviews the most relevant measures in the literature with a new one called Ψ. Some distance measures are specified in the evaluation of over-segmentation (i.e. presence of FPs) and others in the assessment of under segmentation (i.e. missing ground truth points). A complete edge detection evaluation measure takes into account both under-and over-segmentation assessment, the studied measures are detailed in the following subsection.

A. Existing quality measures involving distances

The common feature between these evaluators corresponds to the error distance d Gt (p) or/and d Dc (p). Indeed, for a pixel belonging to the desired contour p ∈ D c , d Gt (p) represents the minimal distance between p and G t . On the contrary, if a pixel p belongs to the ground truth G t , d Dc (p) is the minimal distance between p and D c . Mathematically, denoting (x p , y p ) and (x t , y t ) the pixel coordinates of two points p and t respectively, thus d Gt (p) and d Dc (p) are described by: distance of FPs

F oM (Gt, Dc) = 1 - 1 max (|Gt| , |Dc|) • p∈Dc 1 1 + κ • d 2 G t (p) κ ∈ ]0; 1]
F oM of oversegmentation [START_REF] Strasters | Three-dimensional image segmentation using a split, merge and group approach[END_REF] F oMe (Gt, Dc) = 1 -1 max (e -F P , F P )

• p∈Dc∩¬G t 1 1 + κ • d 2 G t (p) κ ∈ ]0; 1] F oM revisited [47] F (Gt, Dc) = 1 - 1 |G t |+β•F P • p∈G t 1 1 + κ • d 2 Dc (p) κ ∈ ]0; 1] and β ∈ R +
Combination of F oM and statistics [START_REF] Boaventura | Method to evaluate the performance of edge detector[END_REF]

d 4 (Gt, Dc) = 1 2 • (T P -max (|Gt| , |Dc|)) 2 + F N 2 + F P 2 (max (|Gt| , |Dc|)) 2 + F oM (Gt, Dc) κ ∈ ]0; 1] and β ∈ R +
Edge map quality measure [START_REF] Panetta | A New Reference-Based Edge Map Quality Measure[END_REF] Dp (Gt, Dc) =

1/2 |I|-|G t | • p∈Dc 1 - 1 1 + κ•d 2 G t (p) + 1/2 |G t | • p∈G t 1 - 1 1 + κ•d 2 G t ∩Dc (p) κ ∈ ]0; 1] Yasnoff measure [59] Υ (Gt, Dc) = 100 |I| • p∈Dc d 2 G t (p) None Hausdorff distance [24] H (Gt, Dc) = max max p∈Dc d G t (p), max p∈G t d Dc (p) None Distance to Gt [46] [24][13][31] D k (Gt, Dc) = 1 |Dc| • k p∈Dc d k G t (p), k = 1 for [46] k ∈ R + Maximum distance [13] f 2 d 6 (Gt, Dc) = max   1 |Dc| • p∈Dc d G t (p), 1 |Gt| • p∈G t d Dc (p)   None Oversegmentation [20][40] Θ (Gt, Dc) = 1 F P • p∈Dc d G t (p) δ T H k , k = δ T H = 1 for [20]
for [START_REF] Odet | Scalable discrepancy measures for segmentation evaluation[END_REF]:

k ∈ R + and δ T H ∈ R * + Undersegmentation [20][40] Ω (Gt, Dc) = 1 F N • p∈G t d Dc (p) δ T H k , k = δ T H = 1 for [20]
for [START_REF] Odet | Scalable discrepancy measures for segmentation evaluation[END_REF]:

k ∈ R + and δ T H ∈ R * + Baddeley's Delta Metric [2] ∆ k (Gt, Dc) = k 1 |I| • p∈I |w(d G t (p)) -w(d Dc (p))| k k ∈ R + and a con- vex function w : R → R Symmetric distance [13][31] S k (Gt, Dc) = k p∈Dc d k G t (p)) + p∈G t d k Dc (p) |Dc ∪ Gt| , k = 1 for [13] k ∈ R + Magnier et al. measure [37] Γ(Gt, Dc) = F P +F N |G t | 2 • p∈Dc d 2 G t (p) None Symmetric distance mea- sure Ψ(Gt, Dc) = F P +F N |G t | 2 • p∈G t d 2 Dc (p) + p∈Dc d 2 G t (p) None              for p ∈ D c : d Gt (p) = Inf (x p -x t ) 2 + (y p -y t ) 2 , t ∈ G t , for p ∈ G t : d Dc (p) = Inf (x p -x t ) 2 + (y p -y t ) 2 , t ∈ D c .
These distances refer to the Euclidean distance, even though some authors include other types of distance, see [START_REF] Lopez-Molina | Quantitative error measures for edge detection[END_REF]. The measures presented in Table II are tuned in function of the various parameters referenced in Table IV. After marking a theoretical comparison of all these edge detection assessments and carrying out in depth comparison of their parameters, the experimental results are presented.

1) Figure of Merit and its derivatives: First, to achieve a quantitative index of edge detector performance, one of the most popular descriptors is the Figure of Merit (F oM ). This distance measure ranges from 0 to 1, where 0 corresponds to a perfect segmentation [START_REF] Abdou | Quantitative design and evaluation of enhancement/thresholding edge detectors[END_REF]. The constant value κ is fixed at 1/9, 1/4 or 1/10 (the last value is the one used in the experiments of this paper). The more κ is close to 1, the more F oM tackles FPs, as illustrated in Fig 4 . Widely utilized for comparing several different segmentation methods, in particular thanks to its normalization criterion, this assessment approach nonetheless suffers from a main drawback. Whenever FNs are created, for example a contour chain (even long) which is not totally extracted, the distance of FNs (d Dc (p)) are not recorded. Indeed, F oM can be rewritten as:

F oM (Gt, Dc) = 1- p∈Dc∩G t 1 1 + κ • d 2 G t (p) + p∈Dc∩¬G t 1 1 + κ • d 2 G t (p) max (|Gt| , |Dc|) ,
Over(G t , D {1,2} ) = 0.12 U nder(G t , D {1,2} ) = 1 Loc(G t , D {1,2} ) = 0.24 B SN R (G t , D {1,2} ) = 0.73 P * m (G t , D {1,2} ) = 1 Φ * (G t , D {1,2} ) = 1 χ 2 * (G t , D {1,2} ) = 0.98 F * α=0.5 (G t , D {1,2} ) = 1 P 1 (G t , D 1 ) = 16.25 P 1 (G t , D 2 ) = 1.63 P 2 (G t , D 1 ) = 0 P 2 (G t , D 2 ) = 0 F oM (G t , D 1 ) = 0.22 F oM (G t , D 2 ) = 0.60 F oM e (G t , D 1 ) = 0.22 F oM e (G t , D 2 ) = 0.60 F (G t , D 1 ) = 0.63 F (G t , D 2 ) = 0.75 d 4 (G t , D 1 ) = 0.84 d 4 (G t , D 2 ) = 0.89 D p (G t , D 1 ) = 0.51 D p (G t , D 2 ) = 0.54 SF oM (G t , D 1 ) = 0.25 SF oM (G t , D 2 ) = 0.40 M F oM (G t , D 1 ) = 0.29 M F oM (G t , D 2 ) = 0.40 Υ(G t , D 1 ) = 12.99 Υ(G t , D 2 ) = 37.34 H(G t , D 1 ) = 1.41 H(G t , D 2 ) = 7.67 H 5% (G t , D 1 ) = 1.41 H 5% (G t , D 2 ) = 6.71 D k k=1 (G t , D 1 ) = 1.06 D k k=1 ((G t , D 2 ) = 3.05 D k k=2 (G t , D 1 ) = 0.16 D k k=2 ((G t , D 2 ) = 0.48 f 2 d 6 (G t , D 1 ) = 1.06 f 2 d 6 (G t , D 2 ) = 3.05 Θ δT H =1 (G t , D 1 ) = 1.19 Θ δT H =1 (G t , D 2 ) = 12.26 Θ δT H =5 (G t , D 1 ) = 0.05 Θ δT H =5 (G t , D 2 ) = 0.49 Ω δT H =1 (G t , D 1 ) = 1.04 Ω δT H =1 (G t , D 2 ) = 5.12 Ω δT H =5 (G t , D 1 ) = 0.04 Ω δT H =5 (G t , D 2 ) = 0.20 ∆ k w (G t , D 1 ) = 0.96 ∆ k w (G t , D 2 ) = 2.31 SD k k=1 (G t , D 1 ) = 1.04 SD k k=1 ((G t , D 2 ) = 2.57 SD k k=1 (G t , D 1 ) = 1.05 SD k k=2 ((G t , D 2 ) = 2.98 Γ(G t , D 1 ) = 0.34 Γ(G t , D 2 ) = 0.57 Ψ(G t , D 1 ) = 0.46 Ψ(G t , D 2 ) = 0.72 KP I Γ (G t , D 1 ) = 0.16 KP I Γ (G t , D 2 ) = 0.27 KP I Ψ (G t , D 1 ) = 0.22 KP I Ψ (G t , D 2 ) = 0.37 Fig. 5.
Results of evaluation measures. For the two candidate images, the number of FPs and number of FNs are the same: FPs: |D 1 ∩¬Gt|=|D 2 ∩¬Gt|=54 and FNs: |¬D 1 ∩Gt|=|¬D 2 ∩Gt| = |Gt|=48. Also, D 1 ∩Gt=D 2 ∩Gt=∅, so T P = 0 and SSR(Gt, D 1 ) = SSR(Gt, D 2 ) = 1. The assessments involving distances of FPs and/or FNs heavily penalize Dc, including misplaced points with greater distances. hence:

F oM (Gt, Dc) = 1 - T P + p∈Dc∩¬G t 1 1 + κ • d 2 G t (p) max (|Gt| , |Dc|) , (3)
because, for p ∈ D c ∩ G t , d 2 Gt (p) = 0 and

1 1+κ•d 2 G t (p) = 1.
Knowing that T P = |G t | -F N , for the extreme cases, the F oM measures takes the following values:

                 if F P = 0 : F oM (G t , D c ) = 1 - T P |G t | , if F N = 0 : F oM (G t , D c ) = 1 - 1 max(|Gt|,|Dc|) • p∈Dc∩¬Gt 1 1 + κ • d 2 Gt (p)
. ). Moreover, for F P > 0, as

Consequently
1 1+κ•d 2 G t (p) < 1,
it can be easily demonstrate that the F oM measure penalizes the over-detection very low compared to the under-detection. The curve in Fig. 6 shows that when F N > F P , the penalization of missing points (FNs) becomes higher wheareas it is weak F N < F P . Incidentally, F oM e represents an extension of F oM by counting only FPs [START_REF] Strasters | Three-dimensional image segmentation using a split, merge and group approach[END_REF], strictly evaluating the oversegmentation. In fact, it computes a mean of

1 1+κ•d 2 G t
(p) for all the FPs. Consequently, in the presence or absence of FNs, a contour image is considered by the F oM e criterion as correctly segmented. The experiments in Fig. 6, Fig. 8 and Section IV illustrate this undesirable behavior. Some improvements have been developed, such as F and d 4 . In order to overcome the gaps in F oM , F computes the distances of FNs from G t and the measure is tuned by the number of FPs and the cardinality of G t (choosing β = 1 [START_REF] Pinho | Edge detection filters based on artificial neural networks[END_REF]). As Segmentation level F oM

|T P -max(|G t |,|D c |)| M F N M F P M Good ≈ 0 ≈ 0 ≈ 0 ≈ 0 F P , F N = 0 > 0 > 0 0 > 0 F P = 0, F N > 0 > 0 > 0 0 F P , F N > 0 > 0 > 0 > 0
pointed out through Eq. 3 for F oM and illustrated in Fig. 6, F behaves inversely to F oM concerning FPs/FNs points: F is more sensitive to FPs than FNs. Also, d 4 represents another enhancement, this edge measure depends particularly on T P , F P , F N and F oM ; d 4 is normalized with the 1 2 coefficient [START_REF] Boaventura | Method to evaluate the performance of edge detector[END_REF]. Nonetheless, even though d 4 behaves correctly for the experiment in Fig. 6, this measure focuses on FPs and penalizes FNs like the F oM measure, as detailed in Table III. This measure suffers from two main drawbacks. On the one hand, the sum of all its terms yields a high sensibility to edge displacements. On the other hand, in the case of a pure undersegmentation, when F N → 0, d 4 belongs to [0.7, 0.85] but does not attains the score of 1.

As described here, an effective measure for the evaluation of the edge detection is not only directed by d Gt or d Dc . Thus, inspired by f 2 d 6 (see bottom and [START_REF] Dubuisson | A modified Hausdorff distance for object matching[END_REF]), another way to avoid the computation of only the distance of FPs in F oM (or only the distance of FNs in F ) is to consider the combination of both F oM (G t , D c ) and F oM (D c , G t ), as in the following two formulas:

• Symmetric Figure of Merit: SF oM (G t , D c ) = 1 2 • F oM (G t , D c ) + 1 2 • F oM (D c , G t ) (4)
• Maximum Figure of Merit: M F oM (G t , D c ) = max (F oM (G t , D c ) , F oM (D c , G t )) .
(5) As F oM is a normalized measure, SF oM and M F oM are also normalized. Finally, SF oM and M F oM take into account both distances of FNS (i.e. d Dc ) and FPs (i.e. d Gt ), so they can compute a global evaluation of a contour image. Nevertheless, Fig. 6 shows that SF oM behaves like F oM when F N > F P and that M F oM is not monotonous for this experiment (whereas only FNs are added).

2) Hausdorff distance its enhancements: A second measure widely computed in matching techniques is represented by the Hausdorff distance H. In object recognition [START_REF] Hemery | Comparative study of localization metrics for the evaluation of image interpretation systems[END_REF], the algorithm aims to minimize H, which measure the mismatch of two sets of points [24][45]. This max-min distance could be strongly deviated by only one pixel which can be positioned sufficiently far from the pattern (illustrated in Fig. 8); so the measured distance becomes that between the pattern and the (erroneous) point, in that case disturbing the score of H. To improve the measure such that H becomes less sensitive to outliers, one idea is to compute H with a proportion of the maximum distances (for example 5%, 10% or 15% of the values [START_REF] Huttenlocher | A multi-resolution technique for comparing images using the Hausdorff distance[END_REF]); let us note H n% this measure for n% of values (n ∈ R * + ). Even though a mean percentage of the distance does not guarantee an optimized comparison, as illustrated in Fig. 7, this enhancement helps with regard to robustness, and some other modifications are proposed in [START_REF] Zhao | A new Hausdorff distance for image matching[END_REF] and in [START_REF] Baudrier | Binary-image comparison with local-dissimilarity quantification[END_REF].

P 1 (A, A n ) = 0.01 P 1 (A, L) = 0.80 P 2 (A, A n ) = 0 P 2 (A, L) = 0 P * m (A, A n ) = 0.06 P * m (A, L) = 0.95 SSR(A, A n ) = 0.06 SSR(A, L) = 0.99 Φ * (A, A n ) = 0.01 Φ * (A, L) = 0.92 χ 2 * (A, A n ) = 0.07 χ 2 * (A, L) = 0.99 F * α=0.5 (A, A n ) = 0.03 F * α=0.5 (A, L) = 0.90 F oM (A, A n ) = 0.05 F oM (A, L) = 0.60 F oM e (A, A n ) = 0.50 F oM e (A, L) = 0.83 F (A, A n ) = 0.06 F (A, L) = 0.66 d 4 (A, A n ) = 0.05 d 4 (A, L) = 0.78 SF oM (A, A n ) = 0.06 SF oM (A, L) = 0.53 M F oM (A, A n ) = 0.06 SF oM (A, L) = 0.60 D p (A, A n ) = 0.003 D p (A, L) = 0.29 Υ(A, A n ) = 2.93 Υ(A, L) = 1.82 H(A, A n ) = 7 H(A, L) = 5.83 H 5% (A, A n ) = 5.5 H 5% (A, L) = 5.37 D k,k=1 (A, A n ) = 0.30 D k,k=1 ((A, L) = 2.05 D k,k=2 (A, A n ) = 0.18 D k,k=2 ((A, L) = 0.44 f 2 d 6 (A, A n ) = 0.30 f 2 d 6 (A, L) = 2.11 Θ δT H =1 (A, A n ) = 4.87 Θ δT H =1 (A, L) = 2.34 Θ δT H =3 (A, A n ) = 2.89 Θ δT H =3 (A, L) = 0.79 Ω δT H =1 (A, A n ) = 0 Ω δT H =1 (A, L) = 2.31 Ω δT H =3 (A, A n ) = 0 Ω δT H =3 (A, L) = 0.77 ∆ k (A, A n ) = 2.11 ∆ k (A, L) = 2.56 SD k,k=1 (A, A n ) = 2.12 SD k,k=1 (A, L) = 2.09 SD k,k=2 (A, A n ) = 2.71 SD k,k=2 (A, L) = 2.51 Γ(A, A n ) = 0.01 Γ(A, L) = 0.48 Ψ(A, A n ) = 0.01 Ψ(A, L) = 0.75 KP I Γ (A, A n ) = 0.01 KP I Γ (A, L) = 0.24 KP I Ψ (A, A n ) = 0.01 KP I Ψ (A, L) = 0.
Inspired by the Hausdorff distance with a view to developing a new method that is robust with regard to a small number of outliers, some researchers have proposed other measures and studied their behaviors in the presence of misplaced edge points [2] [13]. As H n% remains close to the Hausdorff distance, the rank n acts as a threshold for erroneous pixels and H n% behaves as H. As pointed out in [START_REF] Dubuisson | A modified Hausdorff distance for object matching[END_REF], an average distance from the edge pixels in the candidate image to those in the ground truth is more appropriate for matching purposes than H and H n% . A first proposition of this distance is D k which represents an error distance only in function of d Gt . Also, the Yasnoff measure, called Υ, seems to D k , with k = 2, using a different coefficient of 100 |I| [START_REF] Yasnoff | Error measures for objective assessment of scene segmentation algorithms[END_REF]. The distance measures D k and Υ estimate the divergence of FPs; in other words, they correspond to a measure of over-segmentation. On the contrary, the sole use of a distance d Dc instead of d Gt enables an estimation of the FN divergences, representing an undersegmentation (as in Ω). Precisely, θ and Ω represent two overand under-segmentation assessment measures, where δ T H is the maximum distance allowed to search for a contour point (see also distortion rates in [START_REF] Goumeidane | New discrepancy measures for segmentation evaluation[END_REF]). These distance measures penalize misplaced points further than δ T H from G t [START_REF] Odet | Scalable discrepancy measures for segmentation evaluation[END_REF]. Choosing, δ T H > 1 is equivalent to taking into account the relative position for the over-and under-segmentation and goes back to the problem of the spatial tolerance detailed in Section II). Finally, as concluded in [START_REF] Chabrier | Comparative study of contour detection evaluation criteria based on dissimilarity measures[END_REF], a complete and optimum edge detection evaluation measure should combine assessments of both over-and under-segmentation, as f 2 d 6 and S k . Thus, the score of the f 2 d 6 corresponds to the maximum between the over-and the under-segmentation whereas the values obtained by S k reprensents their mean. Moreover, S k takes small values in the presence of low level of outliers whereas the score becomes large as the level of mistaken points increases [START_REF] Dubuisson | A modified Hausdorff distance for object matching[END_REF] [START_REF] Lopez-Molina | Quantitative error measures for edge detection[END_REF] but is sensitive to remote misplaced points as represented in Fig. 8.

Combining both d Dc and d Gt , Baddeley's Delta Metric (∆ k ) [START_REF] Baddeley | An error metric for binary images[END_REF] is a measure derived from the Hausdorff distance which is intended to estimate the dissimilarity between each element of two binary images. Finally, as this distance measure is based on the mean difference between the two compared images and is useful in region segmentation [START_REF] Falah | A Region-Region and Region-Edge Cooperative Approach of Image Segmentation[END_REF]. For this measure, w represents a weighting concave function, in general w(x) = x or w(x) = min( √ n 2 + m 2 , x) for an image of size m×n, with m and n∈N * [START_REF] Lopez-Molina | A gravitational approach to edge detection based on triangular norms[END_REF] (in our experiments, we use w(x)=x). Compared to automatic threshold algorithms such as [START_REF] Otsu | A threshold selection method from gray-level histograms[END_REF], the threshold at which the edge detector obtains the best edge map is more appropriate when it is computed by the minimum value of ∆ k [START_REF] Fernández-Garcıa | Characterization of empirical discrepancy evaluation measures[END_REF]. The main drawback of ∆ k is its hypersensitivity to false positive points, i.e. this measure tends to over-penalize images with false detections. Indeed, when a false positive pixel is far from the true edge, the |w(d Gt (p)) -w(d Dc (p))| value creates a high impact on the evaluation, thus penalizing the measure (as in the example in Fig. 8).

Another way to compute a global measure is presented in [START_REF] Panetta | A New Reference-Based Edge Map Quality Measure[END_REF] with the normalized edge map quality measure D p . In fact, this distance measure is similar to SF oM , with different coefficients. The over-segmentation measure (left term) evaluates d Dc , the distances between the FPs and G t .

The under-segmentation measure (right term) computes the distances of the FNs between the closest correctly detected edge pixel, i.e. G t ∩ D c . That means that FNs and their distances are not counted without the presence of TP(s), and D p is sensitive to displacements of edges. Moreover, both the left and the right terms are composed of a 1 2 coefficient, so in the presence of only under-or over-segmentation, the score of D p does not go above 1 2 .

3) A new edge detection measure evaluation: In [START_REF] Magnier | A quantitative error measure for the evaluation of roof edge detectors[END_REF], a normalized measure is developed after computation of the edge assessment Γ. The Γ function represents an over-segmentation measure which depends also of F N and F P . As this measure is not sufficiently efficient concerning FNs, Ψ is an alternative function which also considers d Dc for false negative points. Inspired by S k , Ψ uses different coefficients which change the behavior of the measure, as discussed in the next subsection.

B. On the importance of the coefficients

As shown in Table IV, distance measures compute the evaluation using a coefficient mostly including:

|G t |, |D c |, |G t ∪D c |, F N or F P . Concerning the Υ distance measure, the coefficient 100
|I| compress the measurement, especially when the image is large (as demonstrated in Fig. 12). For D p , the coefficient |I| ∼ 0 + . Thus, the assessment of FNs is given more weight in the edge evaluation, strongly penalizing edge displacements. The experiments presented in the next section illustrate this drawback (Fig. 12). However, ∆ k uses |I| as denominator term because the mean distance is computed for all the pixels of the image.

The authors of Γ have studied the influence of the coefficient in different concrete cases [START_REF] Magnier | A quantitative error measure for the evaluation of roof edge detectors[END_REF]. Actually, using only |G t | or |D c | penalizes severely the measurement when one misclassified pixel is placed at a significant distance of its true position. For a counterexample, let us consider the following formula:

Measure |G t | |D c | d G t d D c Other F oM F oM e F P F F P d 4 T P , F P , F N M F oM SF oM D p |I|, p ∈ G t : d G t ∩D c (p) Υ |I| H, H n% D k Θ F P Ω F N ∆ k |I| f 2 d 6 S k |G t ∪D c | Γ F P + F N Ψ F P + F N
A(G t , D c ) = p∈Gt d 2 Dc (p) + p∈Dc d 2 Gt (p) |G t | 2 . ( 6
)
A represents an evaluation measure of contour detection involving both d Dc and d Gt plus a coefficient term 1 |Gt| 2 . Now, considering two images (images (b) and (d) in Fig. 8) with identical ground truth points G t (Fig. 8(a)). However, in image (d), a single FP point at a distance sufficiently high behaves for A as several FP points on another image; these images have the same interpretation in terms of A measure, which is not the case with ratios 1/10 and 9/10. The table in Fig. 8 illustrates, with a simple example, this drawback for A, ∆ k and S k , and a similar example can be found in in Fig. 7.

Close to A, the following measure B depends strongly on the desired contour number |D c |:

B(G t , D c ) = p∈Gt d 2 Dc (p) + p∈Dc d 2 Gt (p) |D c | 2 . ( 7
)
B suffers from the same problem as A. Furthermore, in general, a measure depending mainly on a coefficient 1 |Dc| (as a coefficient 1 |I| ) evaluates a boundary image with a value close to 0 whereas D c stays totally misplaced, especially when |D c | is huge (when |D c | is huge, it corresponds to a total saturation of the desired contour with an inconsistent number of FPs).

Finally, the authors concluded in [START_REF] Magnier | A quantitative error measure for the evaluation of roof edge detectors[END_REF] that such a formulation must take into consideration all observable and theoretically observable cases, as illustrated in Fig. 8. That means that an effective measure has to take into account all the following input parameters |G t |, |D c |, F N and F P , whereas the image dimensions should not be considered. Thus, the coefficient parameter F P +F N |Gt| 2 seems a good compromise for an evaluation measure of edge maps and has been introduced into the new formula of assessment Ψ.

C. Normalization of the edge detection evaluation

In order to compare each boundary detection assessment, all the measures must be normalized, and must also indicate the same information: an error measure close to 1 means poor segmentation whereas a value close to 0 indicates good segmentation. The values of F oM , F oM e , F , d 4 , M F oM , SF oM and D p comply with this [0, 1] condition. However, concerning the other distance measures in Table II, normalization is required. Introduced in [START_REF] Magnier | A quantitative error measure for the evaluation of roof edge detectors[END_REF], a formula called Key Performance Indicator (KP I), with KP I ∈ [0, 1] gives a value close to 1 for a poor segmentation. Alternatively, a KP I value close to 0 indicates a good segmentation. The Key Performance Indicator is defined using the following equation:

KP I u : [0; ∞[ → [0; 1[ u → 1 - 1 1 + u h . ( 8
)
where the parameter u is replaced by a distance error and h a constant such that h ∈ R + * . A key parameter of the KP I formula is the power of the denominator term called h. It may be called a power of observation. Inasmuch as KP I depends on its value, it evolves more or less quickly around 0.5 and embodies a range of observable cases. Average values have been determined for the error distance term

d 2 Dc + d 2 G t concerning Ψ and d 2 G t for Γ.
The choice of values between 1 and 2 can be easily checked.

Otherwise, the more abrupt the KP I evolution, the less the transition between 0.5 and 1 is marked (i.e. the slope of the KP I curve, for example h = 1 in Fig. 9). Moreover, fixing h = 1, KP I stagnates far from 1 when to increase slowly, then the slope becomes sharp around 0.5 and converges quickly towards 1. Finally, to fix the power at the golden ratio φ 1.6180339887 in order to ensure an evolution of KP I that would not be too abrupt from 0 to 1 and also not penalize when the error distances are not high (contrary to KP I with h = 1, see Fig. 9). Furthermore, Fig. 10 illustrates KP I values in function of the distance of the mistake points and their numbers. For a small skeleton G t = 200, it shows that KP I draws near 1 quickly when the number of FPs, number of FNs and their distances are growing. Alternatively, for a bigger skeleton, KP I rises more slowly. These three evolution surfaces show the coherence and robustness of KP I regarding the starting conditions: a displaced edge is penalized in function of the number of false pixels and also of the distance from the position where it should be located.

d 2 Dc + d 2 G t or
Such a value enables a displaced edge to be penalized in function of the false pixel number and also of the distance from the position it should be located at. As a compromise, using the KP I formula with h = φ, the measurement neither becomes too strong in the presence a small number of positive or negative misclassified pixels nor penalizes D c too severely if d Gt/Dc is small. Compared to Γ, Ψ improves the measurement by combining both d Gt and d Dc . The next section is dedicated to experimental results where candidate edge images are progressively degraded in order to study the behaviors of the presented boundary detection assessments. Note that the other measures are also normalized with another formula in the next section in order to compare all of them.

IV. COMPARISON OF EXPERIMENTAL RESULTS

The two sections above describe the main error measures concerning edge detection assessment, explaining the advantages and drawbacks of each one. The tests carried out in the experiments are intended to be as complete as possible, and thus as close as possible to reality. To that end, considering an edge model (i.e. ground truth) the edge detection evaluation measures are subjected to the following studies:

• addition of false negative points (under-segmentation),

• addition of false positive points (over-segmentation),

• addition of both false negative and false positive points, • addition of false positive points close to the true contour,

• translation of the boundary,

N (f ) =          0 if M = m = 0 1 if M = m = 0 f -m M -m if M > 1 and m = 0 f otherwise. (9)
Note that the parameters for each evaluation measure are indicated directly in the captions of the curves and that the matlab code of the distance measures as F oM , D k , S k and ∆ k are available at http://kermitimagetoolkit.net/library/code/.

A. Behavior comparison

The simulation of the degradation of the ground truth G t is studied using synthetic data. Thus, concerning the first 6 experiments, a vertical line represents G t in a size image of 100×100, as illustrated in Fig 11(a). FNs, FPs or displacements corrupt the image, and, as G t corresponds to a line, the interpretation of the results remains simpler. For example, adding a false pixel to G t or a translation of G t can easily be represented mathematically whereas, for a more complicated shape, the translation of G t can cross other pixels and an appropriate measure will not obtain a monotonic score. Hence, the degradations applied to the vertical lines are chosen in order to obtain a monotonic score for a complete measure. Then, the following two tests concern a ground truth which is a square followed by the computation of the minimal value of the edge detection evaluation measure in function of the threshold.

1) Addition of false positive points:

The first test consisted in randomly adding undesirable pixels to G t until 100 pixels, as represented in Fig. 11(b). This perturbation is equivalent to impulsive noise. The curves in Fig. 11 indicate the response of each measure in function of the number of created Fs; they enable comparison of the results delivered by the different measures.

Firstly, concerning measures automatically given values between 0 and 1, χ 2 * , P * m , F * α , SSIM , F oM , F , d 4 , SF oM and M F oM overlap and penalize corrupted images by FPs as soon as they appear, but their values are under 0.5 whereas the last corrupted image is composed of more FPs that TPs(see Fig. 11(b)). F oM e is not monotonic and computes a mean in function of the apparition of FPs. Φ * and D p overlap and stay close to 0 (same remark for the under-segmentation Ω). Contrary to the previous measures, KP I Γ and KP I Ψ ensure an evaluation which does not become too high in the presence of a small number of FPs, but penalizes D c more severely for 50 or more undesirable points. Secondly, concerning the non-normalized measures, Θ is not monotonic or sensitive to FP distances, as H and H 5% which stay blocked at the higher distances values, whatever the number of FPs. Furthermore, Υ, D k , ∆ k , f 2 d 6 and S k have a coherent behavior, even though Υ, D k , ∆ k and S k (with k = 2) are sensitive to FPs at the beginning of the assessment.

This first experiment is produced by randomly adding FPs to the same G t image. The second test is presented in the image of Fig. 12 (a) which is composed of a vertical line and 100 FPs randomly placed around the line. The first image in this test image is as large as the image in Fig. 11(b). In a second step, the TPs and FPs are maintained in the same position and the margins are increased, as illustrated in Fig. 12 (

ϒ H H 5% D k , k = 2 Θ, δ TH = k = 1 Ω, δ TH = k = 1 Θ, δ TH = 5, k = 2 Ω, δ TH = 5, k = 2 ∆ k , k = 2 f 2 d 6 S k , k = 1 S k , k = 2

% of FNs

Normalized measure scores are changed by the elongations of the margins (note that the margins lengthen both downward and to the right at the same time). Even though the values of χ 2 * do not change significantly, SSIM , Υ and ∆ k evolve strongly. The distance measure D p , which remains insensitive to FPs becomes increasingly close to zero as the margin increases.

ϒ H H 5% D k , k = 2 Θ, δ TH = k = 1 Ω, δ TH = k = 1 Θ, δ TH = 5, k = 2 Ω, δ TH = 5, k = 2 ∆ k , k = 2 f 2 d 6 S k , k = 1 S k , k = 2
2) Over-segmentation close to the contour: The idea of this experiment is to create an over-segmentation at a maximum distance of 5 pixels, as illustrated in Fig. 13(a). Therefore, 100% of over-segmentation represents a dilation of the vertical line with a structural element of size 1×6, corresponding of a total saturation of the contour, see Fig. 13(b). The curves presented in Fig. 13 show that F oM e , Υ, S k (k = 2), ∆ k and Θ (δ T H = 1) are very sensitive to FPs whereas, Φ * , D k , F oM e , SSIM and D p do not penalize D c enough. Also, Ω stagnates at 0 because it corresponds to an under-segmentation measure. F oM e , D k and Θ are not monotonic whereas the saturation of the contour is progressive. Note that H and H 5% keep the same result throughout the test after 5% of degradation and that ∆ k is nearly the same. Finally, χ 2 * , F * α , F oM , F , d 4 , SF oM , M F oM , KP I Γ , KP I Ψ , f 2 d 6 and S k (k = 1) ensure a measure evolution which is not too abrupt, even though F oM stagnates around 0.35, then d 4 and SF oM around 0.6. the experiment. As this test concerns only the addition of FNs, the D p distance measure obtains a result of 0.5 at the finish, but no higher because it corresponds to a measure which separates under-and over-segmentation. As feared in Section III, the d 4 measure does not sufficiently penalizes the under-segmentation because the maximum score it attains is around 0.8. Moreover, compared to the experiment in Fig. 11, it is clear that F oM , F , d 4 , SF oM , M F oM and D p are more sensitive to FN addition than FP points. The non-normalized measure scores in the lower graph of Fig. 14 evolve progressively, even though S k (with k = 1) is less sensitive to FNs Note that, for ∆ k (which overlaps with f 2 d 6 ), FN distances are considered to be less detrimental than FP distances (experiments reported in Fig. 11). Finally, KP I Ψ ensures an evaluation which is not too high in the presence of a small number of FNs, followed by a greater penalization when the number of FNs increases.

ϒ H H 5% D k , k = 2 Θ, δ TH = k = 1 Ω, δ TH = k = 1 Θ, δ TH = 5, k = 2 Ω, δ TH = 5, k = 2 ∆ k , k = 2 f 2 d 6 S k , k = 1 S k , k = 2 FP / |G t | FP = FN = 50
ϒ H H 5% D k , k = 2 Θ, δ TH = k = 1 Ω, δ TH = k = 1 Θ, δ TH = 5, k = 2 Ω, δ TH = 5, k = 2 ∆ k , k = 2 f 2 d 6 S k , k = 1 S k , k = 2
ϒ H H 5% D k , k = 2 Θ, δ TH = k = 1 Ω, δ TH = k = 1 Θ, δ TH = 5, k = 2 Ω, δ TH = 5, k = 2 ∆ k , k = 2 f 2 d 6 S k , k = 1 S k , k = 2
4) Addition of both false positive and false negative points: Some measure are specialized and have an adaptive behavior for the evaluation of only under-or over-segmentation. Nevertheless, it is interesting to study their sensitivities by combining these two degradations because edge detectors often create both FPs and FNs in real/noisy images. Thus, adding randomly undesirable pixels to G t (until 100 pixels) and creating FNs at the same times gives a discontinuous line in the middle of randomly placed FPs, as illustrated in In this precise case, Φ * , F * α , F oM , d 4 , M F oM and SF oM measures give a result close to 0.5 whereas G t is disturbed by more than 50%, as represented by the measurement of χ 2 * , P * m , KP I Γ and KP I Ψ (note that Υ, D k and S k -with k = 2behave the same way even though they have a sensitivity to misplaced points at the beginning of the experiment). F seems to evolve correctly but is not monotonic. Note that the SSIM remains unsuitable for this type of degradation because the best score it attains is little more than 0.5. Finally, the evolution of D p is monotonic until 0.5 but not thereafter, due to the coefficient 1 |I|-|Gt| which compresses the result, whereas the final image represents a cloud of FPs without TPs (the next experiment with edge translation shows more clearly the influence of this coefficient).

5) Edge translation:

As pointed out in Section I, a blur in the original image can shift the detected contour. When this displacement remains not too high, the shape of the desirable object stays detectable and the evaluation should not overly penalize the edge detector. Hence, edge detection evaluation measures must be assessed in function of the translation of a true contour. Indeed, the vertical line is shifted by a maximum distance of 20 pixels (Fig. 16 bottom left) and the score of the measures is plotted in Fig. 16 in function of the displacement of the desired contour. Thus, regarding the statistical measures Φ * , χ 2 * , P * m and F * α , the curves perfectly illustrate the need to consider the distances of the misplaced pixels for the edge detection evaluation. F oM , SF oM and M F oM overlap completely, but their evaluations are correct, like KP I Γ and KP I Ψ . F and d 4 measures are sensitive to the first 2 translations. As pointed out above with the previous test, D p is not suited to the evaluation of translations, like SSIM (the SSIM computes locally a score in a window 8×8, so the score does not change after a switch of 8 pixels). The evolution of the other measures (Fig. 16, bottom) perform monotonically but no information can be infered because the measures are not normalized between 0 and 1 to evaluate the segmentation.

6) Comparison of the false positive distance and false negative distance points: In the experiment proposed in Fig. 17, two desired contours are compared with a ground truth, illustrating the importance of considering both the distance of the false negatives points (d Dc ) and the distance of the false positive points (d Gt ). Indeed, a square (G t in Fig. 17 (a)) is compared with another square (Fig. 17 (b)) and also with a shape of two edges forming an angle (Fig. 17 (f)). The minimum square size for G t is 3×3 large and the image size stays fixed at 50×50. The shapes C 1 and C 2 keep the same position as in (c) and (g) and increase in proportion to G t . All the shapes grow at the same time, as represented in Figs. 17 (d) and (h) and the scores for each measure are plotted in Fig. 17 (i), (j), (k) and (l). Thus, the more G t grows, the more C 1 is visually closer to G t whereas FNs deviate strongly in the case of C 2 , but keep the same percentage of FPs and FNs for each desired contour. That is the reason why statistical measures obtain the same evolutions for each shape: Φ * , χ 2 * , P * m and F * α . Despite these two different edge evolutions, some distance measures obtain almost the same measurements for C 1 and C 2 : F oM , F oM e , D p , D k , Θ (with δ T H = 1) and KP I Γ . On the contrary, concerning the normalized error distance measures, M F oM and KP I Ψ increase to about 0.5 for C 2 , due to around 50% of TPs whereas they converge towards 0 for C 1 , since C 1 becomes visually closer to G t . Nevertheless, SF oM does not sufficiently penalize FN distances in C 2 , and F is too severe with C 1 . Concerning non-normalized measures, H 5% , f 2 d 6 and S k behave correctly.

B. Threshold corresponding to the minimum measure

The aim of the experiments presented here is to obtain the best edge map in a supervised way. The edges are extracted using the Canny filter [START_REF] Canny | A computational approach to edge detection[END_REF] (i.e. isotropic Gaussian filter with a standard deviation of σ). Then, thin edges are created after the non-maximum suppression of the absolute gradient in the gradient direction η [START_REF] Rosenfeld | Edge and curve detection for visual scene analysis[END_REF] (see table V) and are normalized for easier comparison. In order to study the performance of the contour detection evaluation measures, one approach is to compare each measure by varying the threshold of the thin edges computed by an edge detector 1 . Indeed, compared to a ground truth contour map, the ideal edge map for a measure corresponds to the desired contour at which the evaluation obtains the minimum score for the considered measure among the thresholded gradient images. Theoretically, this score corresponds to the threshold at which the edge detection represents the best edge map, compared to the ground truth contour map [START_REF] Fernández-Garcıa | Characterization of empirical discrepancy evaluation measures[END_REF] [START_REF] Chabrier | Comparative study of contour detection evaluation criteria based on dissimilarity measures[END_REF]. Since a small threshold leads to heavy over-segmentation and a strong threshold may create numerous false negative pixels, the minimum score of an edge detection evaluation should be a compromise between under-and over-segmentation.

The two images used in these experiments are a synthetic and a real image. For the first one, in Fig. 18 (a), several white shapes are present immersed in a black background, creating step edges. To avoid the problem of edge pixel placements, as stated in Section I, Fig. 1, a blur is created by adding a 1 pixel width of gray around each shape. Thus, the ground truth corresponds to this gray. The second image in Fig. 19(a) is a real image with a corresponding hand-made ground truth. Even though the problem of hand-made ground truths on real images is discussed by some researchers, only the comparison of D c with a G t is studied here.

1) Edges of synthetic data:

To evaluate the measures performances, the original image in Fig. 18(a) is disturbed with random Gaussian noise and edges are extracted from the noisy image (Fig. 18(d) and (e)). Scores are plotted in function of the threshold and the image under each curve corresponds to the ideal edge map for the considerate measure. Statistic measures and D p correctly extract the edges at the price of numerous FPs. The Hausdorff distance H, H 5% and ∆ k threshold the edges too severely, losing the majority of TPs. The over-segmentation distance measures D k , θ and Γ lose almost all the contours because the minimum corresponds to the threshold where no false pixel appears (even though true pixels disappear). Finally, SSIM , f 2 d 6 , F , S k and Ψ do not create significant holes in the contours and the FNs extracted are positioned close to the true contours.

2) Edges of a real image: Real images contain other disturbances than the previous noisy synthetic image, such as texture or blurred edges. The ground truth and the original image in Fig. 19 arise from the database available at the following website: http://www.cs.rug.nl/ ∼ imaging/databases/ contour database/contour database.html.

In edge detection assessment, the ground truth is considered as a perfect segmentation. However, the boundary benchmarks are built by human observers and errors may be created by human labels (oversights or supplements). Indeed, an inaccurate ground truth contour map in terms of localization (sometimes several pixels, see [START_REF] Köthe | Reliable low-level image analysis[END_REF], part 2.2.2.3) penalizes precise edge detectors and/or advantages the rough algorithms. In [START_REF] Hou | Boundary detection benchmarking: Beyond f-measures[END_REF] the question is raised concerning the reliability of the datasets regarded as ground truths for novel edge detection methods. Thus, an incomplete ground truth penalizes a method detecting true boundaries and efficient edge detection algorithms obtain between 30% and 40% of errors. Furthermore, contrary to the previous experiment, where only noise is added to the synthetic image, the true contours are only reported by human perception and, locally, the image intensities (for example the texture) create some disturbances which are nether spatially positioned as a result of the filtering technique, nor humanly perceptible. Unfortunately, this type of contour can be amplified by a strong gradient created by the edge detector [START_REF] Canny | A computational approach to edge detection[END_REF]. As these undesirable pixels could be far from G t , however, an edge detection method must converge to a threshold which preserves acceptable contours, close to the ground truth, and removes undesirable contour pixels. In other words, the final shapes created by the contours of the binarized image should be similar, especially nearby the edges pixels of G t . Thus, in this case, the segmented images are totally different from G t than those in the synthetic case. For example, the ideal edges indicated by the SSIM miss most of the main edges. Also, P * m , F * α , χ 2 * and d 4 indicate an ideal edge image composed of numerous false positive pixels (d 4 obtains this result because most statistics include this measure). On the contrary, the statistical measure Φ, and the distance measures H, D p and ∆ k lose most of the desired boundaries; furthermore, the ideal threshold for F oM e and Υ is 1. Ideal contour images concerning S k k=2 , H 5% , f 2 d 6 , F oM , SF oM and M F oM are less polluted by FPs. Finally, the ideal contour map computed for Ψ is less noisy and better-quality, leading to a compromise between optimum under-and over-segmentation.

3) Comparison of several edge detectors by filtering: In this section, we compare the score of different evaluation measures involving real images (Figs. 20(b) and 26(a)) and 9 filtering edge detectors: Sobel [START_REF] Sobel | Camera Models and Machine Perception[END_REF], Shen [START_REF] Shen | An optimal linear operator for edge detection[END_REF], Bourennane [START_REF] Bourennane | Generalization of Canny-Deriche filter for detection of noisy exponential edge[END_REF], Deriche [START_REF] Deriche | Using Canny's criteria to derive a recursively implemented optimal edge detector[END_REF], Canny [START_REF] Canny | A computational approach to edge detection[END_REF], Steerable filter of order 1 (SF 1 ) [START_REF] Freeman | The design and use of steerable filters[END_REF], Steerable filter of order 5 (SF 5 ) [START_REF] Jacob | Design of steerable filters for feature detection using canny-like criteria[END_REF], Anisotropic Gaussian Kernels (AGK) [START_REF] Geusebroek | Fast anisotropic gauss filtering[END_REF], Half Gaussian Kernels (H-K) [START_REF] Magnier | Fast anisotropic edge detection using gamma correction in color images[END_REF]. Table V shows how gradients are computed using these methods. The parameters of the filters are chosen to keep the same spatial support for the derivative information (see Fig. 27(o)). As above, the comparison is the same: the segmented image corresponds to the one having the minimum score for the considered measure.

Box [START_REF] Sobel | Camera Models and Machine Perception[END_REF] and exponential [START_REF] Shen | An optimal linear operator for edge detection[END_REF] [6] filters do not delocalize contour points [START_REF] Laligant | Regularization preserving localization of close edges[END_REF] whereas they are sensitive to noise (i.e., addition of FPs). The Deriche [START_REF] Deriche | Using Canny's criteria to derive a recursively implemented optimal edge detector[END_REF] and Gaussian filters [START_REF] Canny | A computational approach to edge detection[END_REF] are less sensitive to noise but suffer from rounding corners and junctions (see [START_REF] Laligant | Regularization preserving localization of close edges[END_REF]) as the oriented filters SF 1 [START_REF] Freeman | The design and use of steerable filters[END_REF], SF 5 [START_REF] Jacob | Design of steerable filters for feature detection using canny-like criteria[END_REF] and AGK [START_REF] Geusebroek | Fast anisotropic gauss filtering[END_REF], but the more the 2D filter is elongated, the more the segmentation remains robust against noise. Finally, as a compromise, H-K correctly detects contours points having corners and is robust agains noise [START_REF] Magnier | Fast anisotropic edge detection using gamma correction in color images[END_REF]. Consequently, the scores of the evaluation measures for the first 3 filters must be higher than the three last ones. Furthermore, the ideal segmented image should be visually closer to the ground truth edge image concerning SF 5 , AGK and H-K. Thus, Fig. 21 illustrates that F α gives coherent segmentation and scores even though it is high noisy for images (a)-(f) but the scores are not consistent for the noisy image in Fig. 26(a), see Fig. 27(a). H and ∆ k measures are very sensitive to FPs, therefore, important contours are missing in the segmented images (Figs. 22 and24). Worse still, concerning the image in Fig. 26(a), the scores of AGK are qualified as the second worst segmentation. The segmented images with respect to F oM remain noisy (Fig. 23) and the scores in Fig. 27(h) penalize both AGK and H-K filters almost like the Sobel filter.

Finally, segmented images using f 2 d 6 , S k and Ψ are close to the ground truth edge image and, the more efficient the edge detection filter is, the more the main contours are visible with reasonable FPs and the scores decrease in function of the effectiveness of the filter used. On the contrary, other evaluation measures give either segmented images with high level of FPs, or incoherent scores (bars in Figs. 20 and27). Type of operator Gradient magnitude Gradient direction Fixed operator [START_REF] Sobel | Camera Models and Machine Perception[END_REF], [START_REF] Shen | An optimal linear operator for edge detection[END_REF], [START_REF] Bourennane | Generalization of Canny-Deriche filter for detection of noisy exponential edge[END_REF], [START_REF] Deriche | Using Canny's criteria to derive a recursively implemented optimal edge detector[END_REF], [START_REF] Canny | A computational approach to edge detection[END_REF]

|∇I| = I 2 0 + I 2 π/2 η = arctan I π/2

I0

Oriented Filters [START_REF] Freeman | The design and use of steerable filters[END_REF], [START_REF] Jacob | Design of steerable filters for feature detection using canny-like criteria[END_REF], [START_REF] Geusebroek | Fast anisotropic gauss filtering[END_REF]

|∇I| = max θ∈[0,π[ |I θ | η = arg max θ∈[0,π[ |I θ | + π 2
Half Gaussian Kernels [START_REF] Magnier | Fast anisotropic edge detection using gamma correction in color images[END_REF] |∇I| = max

V. CONCLUSION

This study presents a review of supervised edge detection assessment methods. A supervised evaluation process estimates scores between two binary images: a ground truth and a candidate edge map. Eight statistical evaluations based on the number of false positive, false negative, true positive or true negative points are detailed. Firstly, examples of the evaluation of contour images prove the need to evaluate an edge detector using assessments involving distance measures. Fifteen distance measures are therefore also evaluated. The The rest of the paper is dedicated to the experimental results for synthetic and real data. Most of the edge detection evaluation methods are subjected to the following studies: addition of false negative points and/or false positive points, translation of the boundary, remoteness of false positive and false negative contour chains. Finally, as the minimum value of an edge detection evaluation corresponds to the threshold at which the edge detection is the best, the last experiment concerns the minimum value of the measures on image edges compared to the ground truth. This enables the performance of edge detection algorithms to be studied based on filtering through the optimal threshold obtain by the supervised assessment. Therefore, as the proposed function of dissimilarity seems sufficiently robust to evaluate binary edge maps, we plan in a future study to compare more deeply several edge detection algorithms using the optimum threshold computed by the minimum of the evaluation.

Fig. 1 .

 1 Fig. 1. Several edge chains are available for the same object. The image in (a) is of size 30×30. Contour points are represented in blue for image (b), (c) and (d). Contour 1 represents the inner edge whereas contour 2 corresponds to the outer boundary. Contour 3 is the result of a step edge detector.

  Positive points (FPs), spurious detected edges, i.e. erroneous pixels of D c and not in G t defined as boundary: F P = |D c ∩ ¬G t |, • False Negative points (FNs), missing boundary points of D c , i.e. holes in the true contour: F N = |¬D c ∩ G t |, • True Negative points (TNs), common non-edge points of G t and D c : T N = |¬D c ∩ ¬G t |.

Fig. 2 .

 2 Fig. 2. Illustration of TP (green), FP (red) and FN (blue) points. In (b), Dc is contaminated with 6 FPs ans 4 FNs, illustrated with colors in (c).

Fig. 3 .

 3 Fig. 3. Computation of parameters P 1 and P 2 using contour images 7×7. The C image in (b) is considerate as the Gt image. P 1 penalizes Dc having FPs whereas P 2 penalizes vertical contours with hole(s). In (d), P 1 (C, C 1 ) = 0 because P 1 records only FPs. Note in (a) that P 1 (L, C) = P 1 (C, L).

Fig. 4 .

 4 Fig. 4. Evolution of F oM in function of the the distance of the false positive/negative points and κ parameter. A vertical line of false positive points (a) or false negative points (c) is shifted by a maximum distance of 16 pixels and the measured scores are plotted in function of the displacement of this distance.

( a )

 a Gt, |Gt| = 48 (b) D1, F P = 54 (c) D2, F P = 54 |Dc| = Error distance: Gt v.s. D1 (e) Error distance: Gt v.s. D2 Negative values (in blue) represent d Dc while positive values correspond to d G t .

  , F oM counts only TPs to penalize FN points (Fig 4 (d)) whereas only distances of FPs are recorded (Fig 4 (b)

1 (

 1 (a) Image Gt, (b) Image L 1 , (c) Image L 2 , |Gt| = 21 |Dc| = 31, T P =, 21 |Dc| = 10, T P = 0, F P = 10, F N = 0 F P = 10, F N = 21 d) Scores of F oM and its derived measures.

Fig. 6 .

 6 Fig.[START_REF] Bourennane | Generalization of Canny-Deriche filter for detection of noisy exponential edge[END_REF]. Results of F oM and its derived measures in function of the % of FNs, comparing Gt and L 1 . When the number of FNs attains 100%, the candidate edge image corresponds to L 2 .

  (a) Image A, (b) Image An, (c) Image L, |Gt| = 45 |Dc| = 48, T P =, 45 |Dc| = 32, T P = 4, F P = 3, F N = 0 F P = 28, F N = 41 FPs distances FNs distances ↔ FPs distances (d) Distance map: A vs. An (e) Distance map: A vs. L

39

 39

Fig. 7 .

 7 Fig.7. Results of evaluation measures. Even edge evaluations involving distances can be disturbed by few misplaced pixels and do not respect the shape of the true pattern.

1 |I|- 1 |I|-|Gt| ∼ 1

 111 |Gt| affects the measure concerning the false positive term, thereby creating an insensitivity to FPs, because, generally, |G t | |I|and

2

 2 Ground truth, (b) T1 with 19 FPs, (c) Distance map (d) T2 with 1 FP, (e) Distance map |Gt| = 20 distance of 2 pixels. from (a) vs. (b) distance of 15 pixel. from (a) vs. (d) D c |G t | p∈Gt d Dc (p) + p∈Dc d

Fig. 8 .

 8 Fig. 8. Errors quantified by F oMe, H, H 5% , ∆ k , S k (k = 2), A, B and Γ for two different candidate edge images compared to the same ground truth.

d 2 Gφ d 2 d 2 Fig. 9 .

 229 Fig. 9. Evolution of the KP I ψ in function of the mistake points distance, for different powers h, with F N + F P = 4000 and |Gt| = 2200.

Fig. 10 .

 10 Fig. 10. Evolution of the KP I in function of the erroneous points distance, the number of FPs and FNs, for different |Gt| and fixing h = φ.

•

  remoteness of a false positive and false negative chains, • computation of the minimum value of the measures on edge images (synthetic and real) compared to the ground truth. Therefore, 26 measures are tested and compared with each other: statistical measures in Table I (except B SN R ), distance measures in Table II, plus SF oM , M F oM and SSIM . Firstly, the normalized measures Φ * , χ 2 * , P * m , F * α , SSIM , F oM , F oM e , F , d 4 , SF oM , M F oM , D p , KP I Γ and KP I Ψ are plotted together. The values of SSR are not recorded because this measure behaves like P * m . Secondly, the scores of the other measures are also normalized to be plotted together, and normalized using the following equation for easy visual comparison. Denoting by f ∈ [0; +∞[ the scores vector of a distance measure such that m = min(f ) and M = max(f ), then the normalization N of a measure is computed by:

Fig. 11 .Fig. 12 .

 1112 Fig. 11. Evolution of the dissimilarity measures in function of the number of added false positive points.

1 %

 1 b), influencing the evaluation in certain cases. The curves presented in Fig. 12 (b) show which dissimilarity measure (a) 30 % of FPs (b) 100 % of FPs of oversegmentation at a distance of 5 pixels Normalized measure

Fig. 13 .

 13 Fig.[START_REF] Dubuisson | A modified Hausdorff distance for object matching[END_REF]. Evolution of the dissimilarity measures in function of the oversegmentation in the contour area.

Fig. 14 .

 14 Fig. 14. Evolution of the dissimilarity measures in function of the number of false negative points addition. F oM and M F oM overlap.

3 )

 3 Addition of false negative points: In this experiment, pixels of the vertical line are missing, creating false negative points. Fig. 14 presents the evolution of the criteria obtained by the studied segmentation measures. These curves indicate that almost all the measures show the same behavior. The over-segmentation measures F oM e , Γ, Υ, D k and Θ stagnate at 0. The SSIM stays close to 0, whereas there are almost no more remaining pixels in D c at the end of , i.e. FN / |G t |, with |G t

Fig. 15 .

 15 Fig. 15. Evolution of the dissimilarity measures in function of both the number of false positive and false negative points addition.

2 Ω 2 ∆Fig. 16 .

 2216 Fig. 16. Evolution of the dissimilarity measures in function of the translation of Dc. Note that F oM , F oMe, SF oM and M F oM overlap.

( k )Fig. 17 .

 k17 Fig. 17. Evolution of dissimilarity measures in function of the size of the original square.

  Fig 15, bottom right. Fig. 15 shows a hyper-sensitivity of FPs/FNs distances for F oM e , H, H 5% , Θ, ∆ k , S k and ∆ k . Moreover, Θ and Ω are not monotonic. Also, Φ * , F * α , F oM , d 4 , M F oM , f 2 d 6 and S k (with k = 1) increase (almost) constantly from 0 to 1 whereas both FNs and FPs are added. For example, the image in Fig 15, bottom right, illustrates the line where both 50% of the correct pixels are missing and 50 FPs are added.

Fig. 18 .

 18 Fig. 18. Comparison of the minimal score for several edge detection evaluation measures concerning synthetic data. The edges are extracted using Canny's theory (σ = 1) on the noisy image in (d) and the binary images correspond to the thin edge image in (e) which is thresholded at the threshold corresponding to the minimum value of the bottom curve. The results for F * α are the same as the P * m , χ 2 * measures and do not need to be reported. The scores of the over-segmentation measure D k are the same as F oMe, Υ, Θ and Γ. Moreover, F oM , d 4 , SF oM and M F oM obtain almost the same result.

  (a) Image 512×512 (b) Gt, |Gt| = 5951 (c) Thin edges , σ = 3 (d) P * m , Fα, χ 2 * , d4 th = 0.27 (e) Φ, Dp, th = 0.04 (h) SSIM , D k k=2 , th = 0.57 (i) H, th = 0.1 (j) H 5% , th = 0.16 (k) f2d6, S k k=1 , F , th = 0.18 (l) S k k=2 , F oM , th = 0.17

Fig. 19 .

 19 Fig. 19. Comparison of the minimal score for several edge detection evaluation measures concerning a real image and a hand-made ground truth. The edges are extracted using Canny's theory (σ = 3) on the real image in (a) and the binary images correspond to the thin edge image in (c) which is thresholded at the value (th) corresponding to the minimum score for each measure. The best images for the measures S k k=1 , f 2 d 6 , SF oM and M F oM are the same. The optimal threshold for the over-segmentation measures Θ δ T H =1 or 5 , Γ is th = 1, and is th = 0 for the under-segmentation measure Ω δ T H =1 or 5 .

Fig. 20 .

 20 Fig. 20. Comparison of the minimal score for several edge detection evaluation measures concerning a real image and several edge detection methods.

Fig. 21 .

 21 Fig. 21. Comparison of the minimal score for the Fα measure concerning a real image and several edge detection methods.

Fig. 22 .

 22 Fig. 22. Comparison of the minimal score for the H measure concerning a real image and several edge detection methods.

Fig. 23 .

 23 Fig. 23. Comparison of the minimal score for the F oM measure concerning a real image and several edge detection methods.

  Minimum scores of ∆ k

Fig. 24 .

 24 Fig. 24. Comparison of the minimal score for the ∆ k measure concerning a real image and several edge detection methods.

Fig. 25 .

 25 Fig. 25. Comparison of the minimal score for the Ψ measure concerning a real image and several edge detection methods.

Fig. 26 .First 5 (Fig. 27 .

 26527 Fig. 26. Comparison of the minimal score for the Ψ measure concerning a real image and several edge detection methods.

TABLE I LIST

 I OF ERROR MEASURES INVOLVING ONLY STATISTICS.

	Error measure name	Formulation	Parameters
	Binary noise-to-signal ratio [58]	B SN R (Gt, Dc) =	|Dc| F P + F N	None
	Complemented P erf ormance measure
	[51] [19]

TABLE II LIST

 II OF ERROR MEASURES COMPARED IN THIS WORK. IN THE LITERATURE, THE MOST COMMON VALUES ARE k = 1 OR k = 2.

	Error measure name	Formulation	Parameters
	Figure of Merit (F oM )
	[1]

TABLE III BEHAVIOR

 III

OF d 4 MEASURE IN FUNCTION OF DIFFERENT SEGMENTATION LEVELS, NOTING M = (max(|Gt|, |Dc|)) 2 .

TABLE IV PARAMETERS

 IV AFFECTING THE ERROR DISTANCE MEASURES.

TABLE V GRADIENT

 V MAGNITUDE AND ORIENTATION COMPUTATION FOR A SCALAR IMAGE I WHERE I θ REPRESENTS THE IMAGE DERIVATIVE USING A FIRST-ORDER FILTER AT THE θ ORIENTATION (IN RADIANS).

The matlab code and several measures are available on MathWorks:https://fr.mathworks.com/matlabcentral/fileexchange/ 63326-objective-supervised-edge-detection-evaluation-by-varying-thresholds-of-the-thin-edg