Didier Anis Hoayek

Didier Rullière

Assessing clustering methods using Shannon's entropy

come

Assessing clustering methods using Shannon's entropy

Introduction

In unsupervised learning, clustering methods are very popular methods that associate each observation to a cluster index. Such methods are known as hard clustering approaches. In situations where some observations are difficult to associate with certainty to specific clusters, fuzzy clustering may be used: fuzzy clustering methods, known also as soft clustering methods, associate to each observation the probability to belong to each possible cluster index, or more generally some membership degrees (see e.g [START_REF] Ruspini | Fuzzy clustering: A historical perspective[END_REF][START_REF] Yang | A survey of fuzzy clustering[END_REF]De Oliveira and Pedrycz, 2007, among

many other references).

There is a wide diversity of clustering methods: a first subdivision is to distribute clustering methods into two families: 1) partitional clustering algorithm (e.g. k-means, density based clustering, genetic algorithm and many other methods), (see e.g. [START_REF] Macqueen | Classification and analysis of multivariate observations[END_REF]; [START_REF] Kriegel | Density-based clustering[END_REF]; [START_REF] Forrest | Genetic algorithms[END_REF]; among many other references) in which data is organized into a sequence of groups without any hierarchical structure [START_REF] Ezugwu | A comprehensive survey of clustering algorithms: State-of-the-art machine learning applications, taxonomy, challenges, and future research prospects[END_REF]); 2) hierarchical clustering algorithm (e.g. Linkage algorithm, divisive clustering), (see e.g. [START_REF] Dawyndt | The complete linkage clustering algorithm revisited[END_REF]; [START_REF] Roux | A comparative study of divisive hierarchical clustering algorithms[END_REF]) .

A problem for fuzzy clustering is to compare different available methods. Are they overconfident, as in the case where proposed probabilities are all close to 0 or 1? are they underconfident, as in the case where each observation can belong to any cluster, with equal probabilities? how to measure this confidence of the clustering method? how to compare clustering methods? how to propose corrections to proposed probabilities in case of over or underconfident clustering method?

The confidence of a clustering method is indeed of practical interest, beyond the quality of a classification. If the clustering method is too uncertain, too fuzzy (underconfident), it may induce avoidable checks (medical investigations for example, with costs or adverse effects). If the clustering method is too categorical (overconfident), it may disable useful alerts (need of medical checks for example). Quantifying this under/overconfidence, or equivalently too fuzzy/not fuzzy enough characteristic of a clustering method requires the definition of some kind of fuzziness level.

A goal of the present paper is to carefully define the fuzziness of a clustering method. As demonstrated in various sections of this paper, defining such a fuzziness level allows for the comparison of 1 fuzzy clustering methods, determining whether a clustering method is compliant with a given probabilistic model, improving classification or clustering accuracy on test samples for both supervised and unsupervised learning, and assisting in the selection of a hidden number of clusters.

In the literature, to our knowledge there is very few works on the fuzziness assessment of a clustering method. Some existing indices, as Dave's, Bezdek, and Xi-Beni validity measurement indices (see for example [START_REF] Bataineh | A comparison study between various fuzzy clustering algorithms[END_REF], do not directly assess the confidence of the clustering method.

However, some papers in the literature deals with overconfidence issue, see [START_REF] Park | Improving unsupervised image clustering with robust learning[END_REF], in a specific context of computer vision. [START_REF] Aghababyan | Enhancing the clustering of student performance using the variation in confidence[END_REF] proposed an entropy based metric to evaluate the confidence of clustering algorithm without proposing any correction of the underlying probabilities in case of over/underconfident methods. [START_REF] Yao | Entropy-based fuzzy clustering and fuzzy modeling[END_REF] introduced a new fuzzy clustering algorithm based on entropy without investigating about the confidence level of the proposed algorithm with respect to other state of the art methods. In addition, to the best of author's knowledge no work has been done on proposing a statistical hypothesis test to decide about accepting or rejecting a clustering method based on its confidence level.

In this paper an entropy based clustering confidence metric is introduced. Based on this metric one will be able to compare the performance of any two clustering algorithms. A statistical test is also introduced to decide about accepting or rejecting a clustering method. Furthermore, in the context of over/underconfident clustering method, a parametric correction of the underlying probabilities is proposed in order to improve the accuracy of a clustering. An application for adjusting the number of clusters is also presented.

The outline of the paper is as follows. We first introduce in Section 2, a metric to measure clustering fuzziness using entropy, for a theoretical mixture model as well for a practical clustering algorithm. A Section 3 is devoted to the comparison of clustering fuzziness level of different clustering algorithms, with some applications based on numeric simulations. A statistical test helping users to decide about accepting or rejecting a given clustering algorithm is also described. Then, in Section 4, we propose a parametric correction method of the underlying probabilities of under/overconfident clustering algorithms. Applications of the whole methodology on real data are shown in Section 5. A conclusion closes the paper.

Measuring clustering fuzziness

Let I be a finite set of cluster indexes. In this work, (Ω, F, P) is a probability space and for any i ∈ I, X i is a R d valued random variable defined on Ω with a cumulative distribution function F i (•), and a probability density function f i (•) with respect to the Lebesgue measure in R d .

These random variables X i , i ∈ I have distinct distributions, as they correspond to individuals of different clusters. Assume that one chooses a cluster index with a random variable I taking values in I. One labelled observation of the corresponding individual is given by the couple (X I , I). The purpose of a clustering procedure is to retrieve the association between observations of the mixture distribution X I and labels I when the labels are lost.

In this paper, we aim at assessing the fuzziness of different clustering methods, and at comparing it with expected values when they are known.

Notice that the precise value of the labels has no impact, so that a perfect clustering procedure can associate I or any permutation σ(I) to the observations X I : the fuzziness measure should be insensitive to permutations of I. It should also allow to compare clustering that use different numbers of clusters. We present hereafter a measure and some tests based on Shannon's entropy.

An originality of this section is that the fuzziness level associated to a clustering will be represented by the distribution of a real-valued random variable. This will provide a detailed information on the clustering fuzziness, while allowing comparison between these fuzziness levels.

Theoretical entropy

A labeled clustering is given by the joint distribution of (X I , I). When the labels are lost, we aim at inferring the values of the random variable I given the observations X I .

Recall that I = {1, 2, . . . , |I|} is the finite set of all possible cluster indexes. Consider the finite mixture distribution given by:

G := X I = i∈I X i 1 {I=i} , with I ∈ I and each X i ∼ F i , i ∈ I.
F i is the cumulative distribution function (cdf ) of the random variable (rv ) X i . Then, the cdf F G and the probability density functions (pdf ) f G of G are as follows:

F G (x) = i∈I α i F i (x) , f G (x) = i∈I α i f i (x) ,
where, f i is the pdf of X i , with α i = P [I = i] and i∈I α i = 1.

Therefore, the probability that a given observation x is sampled from the underlying rv X i is:

p i (x) := P [I = i | G = x] , = α i f i (x) j∈I α j f j (x)
. Now, considering the joint random variables (G, I), we compute Shannon's entropy of the rv I given G = x, measuring the information on the fact that an observation x is sampled from the distribution F i , i ∈ I:

H I (x) = - i∈I p i (x) log 2 p i (x) ,
under the convention that 0 log 2 0 = 0.

Recall that I is the hidden cluster index associated to X I . This index I will be unknown in practice. The former entropy H I (x) measures the uncertainty of the clustering at a given point x. When it is equal to 0, e.g. in the case when p i (x) = 1 for a given i and 0 otherwise, then it is certain that x was sampled from a specific known index i. This entropy is maximal when all p i (x) are equal, so that one has totally lost the information about the index I that has generated the observation x. This entropy is insensitive to cluster index permutations, which is a desirable property, as stated in the introduction.

Consider the function H

I : x → H I (x) ∈ R + . Applied to a random argument G, this function defines the random variable H I (G) ∈ R + . H I (G)
measures the uncertainty of the clustering at a random point having the same distribution as G.

We give below an original definition of the theoretical fuzziness level of a known mixture model.

Definition 1 (Theoretical fuzziness level). For a given mixture distribution G and associated labels I, i.e. given the joint distribution of (G, I), the theoretical fuzziness level is defined as the non-negative real-valued random variable H I (G).

The distribution of this random variable shows the discriminating power of the considered clustering approach: e.g., the higher its mean, the more ambiguous the situation is (high fuzziness). However, a mean close to zero reflects a clustering where one expects to easily associate a unique label to each point x (low fuzziness). On the other hand, a low dispersion shows that the difficulty of cluster labeling is the same for all points x, whereas a high dispersion indicates that some points are easier to label than others.

It shall be noted that, for a given distribution G, several distributions of I and X i , i ∈ I can lead to the same mixture G = X I .

As an example, picking a uniform r.v. U (0, 1) with probability 1/2, and a uniform r.v. U (0, 0.5) with probability 1/2 leads to the same mixture as an uniform r.v. U (0, 0.5) with probability 3/4 and U (0.5, 1) with probability 1/4, but the conditional distribution of I given G are changed.

Another example is the (overlapping) mixture of Gaussian distributions, that can also be modelled as a non-overlapping mixture of truncated Gaussian distributions; this leads to what is sometimes called a bias or an inconsistency for k-means unsupervised clustering, as the mean of any resulting truncated Gaussian differs from the one of the initial Gaussian (see [START_REF] Hiemstra | On the bias and inconsistency of k-means clustering[END_REF]).

Thus in some cases, identifiability problems may occur when the joint distribution (G, I) is unknown. This problem however depend on the families of possible distributions F i , i ∈ I: for example, it does not occur if all F i are assumed to be Gaussian distributed, but it may occur if all F i can be overlapping truncated Gaussian distributed. Such problems will be discussed in a further Section 5, when F i are unknown. A short review on identifiability problem for finite mixture models is given in [START_REF] Mclachlan | Finite mixture models[END_REF]. We assume in this section that the joint distribution (G, I) is known, so that all F i are known, i ∈ I, and no identifiability problem raises here.

To illustrate the function H I (.) we consider a few basic examples:

Case A Let G be a Gaussian mixture distribution in one dimension with a pdf :

f G (x) = 0.3f N (0,1) (x) + 0.5f N (10,1) (x) + 0.2f N (3,0.1) (x).
where f N (.,.) denotes the pdf of a Gaussian r.v. with indicated parameters. The left side of Figure 1 represents the pdf of G, while the right side illustrates the function H I . One can remark that the labeling is perfectly accurate when the values x of G are far from the central area of the distribution (i.e. left and right queues of the mixture distribution). However, the difficulty of cluster labeling is higher for central values where the region is fuzzy in terms of distribution selection.

Case B Let G be a Dirac mixture distribution in one dimension with a probability distribution:

P [G = x] = 0.31 {x=0} + 0.71 {x=3}
where, 1 {x=a} , a ∈ R is the indicator function defined by:

1 {x=a} = 0 if x ̸ = a 1 if x = a .
The left side of Figure 2 represents the probability masses of G, while the right side illustrates the function H I . One can remark that the labeling is perfectly accurate for all values of x. This is well suited to the Dirac case where the discrimination between different clusters/labels is obvious.

Case C Let G be a two dimensional Gaussian mixture distribution with a pdf :

f G (x) = 0.4f N (µ1,Σ1) + 0.6f N (µ2,Σ2) ,
where, µ 1 = (0, 0) T and µ 2 = (4, 4) T . In addition Σ 1 = 1 0 0 1 and Σ 3 = 2 0.4 0.4 2 .

The left side of Figure 3 represents the contour lines of the pdf of G, while the right side illustrates the contour lines of H I . Once again, one can say that the accuracy of the labeling is lower on zones that are with a high fuzziness level (e.g. between the two modes), and thus difficult to label.

In Figure 4 we are showing the impact of increasing the variance of the underlying distribution of a mixture model on the distribution of the entropy by showing the corresponding entropy's boxplot. Hence, the mixture distribution of Case C is considered and the matrices of the underlying distributions are multiplied by a parameter α. The considered values of α are 1, 4 and 8 respectively. One may remark that when the values of α are increasing the fuzziness level is going to be higher with a lower labeling accuracy, which confirms that the entropy is a suitable tool to show the impact of the variance increase on the global fuzziness level.

Empirical entropy

In practice, even when the true mixture G is unknown, one can derive an associated empirical fuzziness level from both a dataset and a clustering algorithm. Indeed, the dataset provides an empirical distribution of the underlying mixture, and the clustering algorithm provides associated weights, summing to one, that can be used to define an underlying random index. The aim of this section is to define the empirical fuzziness level deriving from both a dataset and a clustering algorithm.

Notice that the weights (or membership grades) in traditional fuzzy clustering methods represent the degree to which data points belong to each cluster. These weights have not originally been meant to be probabilities, or to estimate underlying probabilities. But in some situations, they need to be used as probabilities, or are implicitly used as probabilities. We investigate here the resulting fuzziness level of these weights, in order to compare them to other fuzziness levels. The purpose is not to depreciate one method or another, but to understand the implicit fuzziness level of each method (and associated parameters), and to propose possible clustering weights corrections.

The considered data is an iid sequence of random variables {G 1 , . . . , G n }, sampled from a mixture distribution G. Let G n be a rv distributed as this data (e.g. a uniformly randomly chosen element of the sequence).

Even when G is unknown, a clustering algorithm associate to each data point x the probability that it belongs to different clusters of a set J . Denote by p j (x) the probability that a data point x belongs to a cluster j ∈ J , according to the clustering algorithm.

For any data point x, given G n = x, the probabilities {p j (x) : j ∈ J , } define the marginal distribution of a rv J n . Hence the joint distribution of a couple of rv (G n , J n ) can be defined from the data and the clustering algorithm. Based on these probabilities, given G n = x, the Shannon's entropy of the random variable J n is:

H Jn (x) = - j∈J p j (x) log 2 p j (x) .

Now, consider the function H

Jn : x → H Jn (x) ∈ R + . Applied to a random argument G n , this function H Jn (.) defines a rv H Jn (G n ).
For a given clustering algorithm, we give below the resulting original definition of the empirical fuzziness level.

Definition 2 (Clustering empirical fuzziness level). Let G n be a rv distributed as the empirical data, and J n be the associated random labels according to a given clustering algorithm. Given the joint distribution of (G n , J n ), the clustering empirical fuzziness level is defined as the non-negative realvalued random variable H Jn (G n ).

The higher the average values of the random variable H Jn (G n ), the higher the fuzziness level of the underlying clustering, the higher the labeling difficulty on a random point of the data. In the next section, we develop the analysis and compare this fuzziness level to a theoretical one, given an underlying mixture model.

Comparing clustering fuzziness

We aim here at comparing two clustering fuzziness levels, or equivalently at comparing two clustering confidence levels. Using the fuzziness levels defined in the former section 2, this ends up in comparing the distribution of two entropies.

Comparing the fuzziness level of theoretical or empirical distributions can be beneficial in many ways: it can aid in the selection of one fuzzy clustering algorithm over another, it can aid in the tuning of some fuzziness parameter, it allows for the comparison of the performance of several algorithms exhibiting the same fuzziness level, and it also allows for the correction of cluster weights in order to comply with some probabilistic model. As detailed in numerical illustration, it will also help improving the accuracy of clustering algorithm on some test samples.

As a first illustration, we present here the case where the theoretical distribution (G, I) is known, i.e. when one knows the underlying reality of the data. The case of real data with unknown distribution will be treated in a further Section 5. We compare here a theoretical fuzziness level with an empirical fuzziness level. A direct comparison between two empirical fuzziness level, even when the theoretical mixture is unknown, would also be possible, although not treated here.

The idea we develop below is that a fuzzy clustering algorithm, applied to an iid sample of G, should end up with an entropy distributed as the random variable H I (G). If it concludes with a lower mean entropy, then the clustering algorithm exhibits too low fuzziness, is too overconfident in its associations/labeling. If it concludes with a higher mean entropy, then the clustering algorithm has a too high fuzziness level and is too hesitating. Indeed, the distribution of H I (G) reflects the difficulty of the labeling problem. In fact, depending on the underlying structure of G, some clustering algorithms may perform better then others regarding this fuzziness level. We first assume that the joint random distribution (G, I) is given, so that no identifiability issue raises here. Notice that we will be able to compare algorithms that associate a cluster index probability for each point, and algorithms that associate a unique label to each point.

We aim at comparing the distributions of the two random variables H I (G) and H Jn (G n ). Therefore, in order to compare the entropy distributions of these random variables, a distance (or a dissimilarity) is calculated. Furthermore, as the distribution of H I (G n ) is easier to obtain than the one of H I (G), which requires a numerical integration, we will consider in practice the following dissimilarity:

D n := D (H I (G n ) || H Jn (G n )) ,
where D(.) is the chosen dissimilarity. Note that, the distribution of H I (G n ) represents the theoretical labeling difficulty for the mixture distribution (at data points), whereas H Jn (G n ) represents the perceived labeling difficulty of the clustering algorithm on the data.

The fuzzy clustering weights will be more in line with the underlying theoretical model as the distance decreases. A zero distance means that the clustering algorithm delivers the true underlying cluster probability at each data point. The choice of using here H I (G n ) instead of H I (G) also relies on this zero distance property.

In the next subsections, we compare clustering empirical fuzziness with some underlying known theoretical one, using either a Kolmogorov-Smirnov distance or a Jensen-Shannon Divergence.

Comparison using Kolmogorov-Smirnov distance

In this subsection, we compare some theoretical and empirical fuzziness level, using Kolmogorov-Smirnov distance. To do this, we compute the following statistic:

KS n := D (H I (G n ) || H Jn (G n )) .
(1)

We reconsider the first two examples of Section 2.1 and we test the proposed approach on two clustering methods, using Kolmogorov-Smirnov distance (KS):

1. In the first application, the k-means clustering approach is applied on a sample of n = 10 000 observations generated from the mixture distribution G. In the one dimensional Gaussian mixture model (Case A), the cardinal of J was considered to be equal to three. Kolmogorov-Smirnov statistics has a value of KS n = 0.43 with a p -value of ≈ 0%. Then, one can say that the two distributions are significantly different and that the proposed clustering method fails to identify the labeling difficulty. This is expected as a hard clustering here is clearly overconfident in its labeling. However, the k-means applied on Dirac mixture underlying distribution (Case B), with J = 2, gives KS n ≈ 0 and a p -value of ≈ 100%. Then, in the Dirac case, k-means succeed perfectly in re-finding the original labels of the observations. Note that these results fit well with the nature of the k-means algorithm, which proposes a labeling with no degree of fuzziness, known as hard clustering approach. Hence, one can confirm that such kind of clustering algorithm is to be selected when the underlying structure of G is more categorical, as in the case of a Dirac distribution.

2. In the second application, a soft clustering approach is applied. To do this, we have chosen the fuzzy clustering algorithm called "fanny" which was introduced in Chapter four of [START_REF] Kaufman | An introduction to cluster analysis[END_REF]. Here, for each observation, instead of considering only one clustering label, probabilities of different clustering labels is computed. Now, under the same hypothesis of the first application, Kolmogorov-Smirnov statistics has a value of KS n ≃ 1 with a p-value of 0% for both Gaussian and Dirac cases. Then, for the Gaussian underlying distribution, the considered soft approach has no additional value compared to the k-means and the algorithm fails again in identifying the labeling difficulty. However, for the Dirac case the soft approach, unlike the k-means, is not able to identify well original labels, which is logical due to the fact that in Dirac case the mass function is concentrated at only one point. Finally, the comparisons of the cdf of both H I (G n ) and H Jn (G n ) in Figure 5 shows that, as a result of its high fuzziness, the chosen clustering is under-confident. 1-D Gaussian case 2-D Gaussian case

H I (G n ) H Jn (G n ) H I (G n ) H Jn (G n ) Mean 0.

Comparison using Jensen-Shannon Divergence

In this subsection, we aim at comparing an empirical entropy deriving from a fuzzy clustering algorithm (and associated parameters), with the known underlying entropy of the known mixture G. To this aim, the distance between two entropy distributions P and Q is calculated based on Jensen-Shannon divergence (JSD) metric defined by:

JSD (P || Q) = 1 2 [KL (P || R) + KL (Q || R)] ,
where R = 1 2 (P + Q) and KL (P || R) denotes the Kullback-Leibler divergence between probability distributions P and R. Note that the JSD has the following properties: 1) Non negative measure; 2) Symmetric measure and 3) JSD ∈ [0, 1] , with JSD = 0 if and only if P = Q. Hence, this metric is suited to the case of fuzzy underlying distributions when soft clustering algorithm are applied.

We compare here some theoretical and empirical fuzziness level, using the Jensen-Shannon Divergence. To do this, we compute the following statistic:

JSD n := JSD (H I (G n ) || H Jn (G n )) .
(2) By applying the JSD on our previous examples introduced in Section 2.1, we get the following results:

1. For the Gaussian mixture distribution and "fanny" clustering algorithm context with a sample size n = 10 000: JSD n = 0.985 for one dimensional Gaussian (Case A) and JSD n = 0.897 for the two dimensional (Case C). Then, for this setting and for this criterion, the performance of "fanny" algorithm is better for the two dimensional Gaussian mixture distribution (in terms of fuzziness level). But it is still an under-confident approach (see Figure 5 and Table 1). Note that, in the numerical illustrations of this paper, in order to compute different JSD, random variables H I (G n ) and H Jn (G n ) were discretized with a constant step size of 0.001.

2. Now in order to compare the efficiency of two different soft clustering algorithm "fanny" and "Fuzzy k-means (FKM)" introduced by [START_REF] Bezdek | Pattern recognition with fuzzy objective function algorithms: Advanced applications in pattern recognition[END_REF] we consider a sample n = 10 000 of the two dimensional Gaussian mixture distribution (Case C). Under these assumptions we get JSD fanny n = 0.897 > JSD FKM n = 0.658. Then with this setting, the FKM algorithm is more suited to find similar fuzziness as the true underlying model. The result is illustrated in Figure 6.

To study the impact of the sample size n on the quality and the performance of different clustering algorithms, Table 2a shows JSD n for different values of n in the case of one and two dimensional Gaussian underlying distribution and in the context of "fanny" clustering algorithm. On the other hand Table 2b shows a comparison between JSD fanny n and JSD FKM n for different values of n in the case of two dimensional Gaussian mixture distribution. In both cases, especially for 2-D Gaussian case, JSD is decreasing for high values of n. Then, one can say that the ability of a clustering algorithm to reconstruct original hidden labels is partly related to the uncertainty on the empirical distribution.

As not to be limited to the comparative aspect of our approach, its important to propose a method that helps setting a kind of threshold or boundary between accepted and rejected clustering method, in terms of labels reconstruction performance, for a given set of data.

Statistical test

Using the results of Section 2, we can make some progress on the problem of selecting a good clustering method that fits well the original mixture distribution assumption, when it is known. This way, one can understand if a clustering method tends to be over or underconfident, compared to a given underlying model. We will investigate the case of unknown mixture distribution in Section 5.

Recall that G n is a rv having the same empirical distribution as an iid sample of G, and that H Jn (G n ) is the empirical entropy of the clustering method under investigation, as defined in Section 2.2, for the corresponding sample.

We make the assumption that the clustering method under consideration has a specific consistency property: more precisely, we assume that there exist joint random variables (G, J) such that

H Jn (G n ) d → H J (G) in distribution, as n → ∞.
In other words, the algorithm's empirical fuzziness distribution should converge to some limit distribution, which occurs especially if the clustering weights are not too altered as the sample grows. As an illustration of this assumption, Figure 7 shows that for both fanny and FKM algorithms, and after a reasonable number of observations (e.g. n = 500) the cdf s of entropies H Jn (G n ) seem to empirically converge to some limit distribution.

We consider the context of a statistical test with H 0 : "the entropy of clustering method complies with the original mixture distribution assumption" vs H 1 : "the entropy of the clustering method does not comply with the mixture distribution assumption". In other words one can write:

H 0 : H J (G) d = H I (G) .
The considered test compare here a clustering method with an underlying given theoretical mixture distribution, but it would be quite straightforward to adapt it to the comparison of two different clustering algorithms.

Assume observations are an iid sequence of n random variables distributed as G, say G 1 , . . . , G n . For this data and for a given clustering algorithm, the deterministic function H Jn (.) is returned at each of these data points. Under the above mentioned consistency assumption, we assume for n large enough that the sequence

H Jn (G 1 ) , . . . , H Jn (G n ) is an iid sample of H J (G).
Now assume that the underlying joint distribution of (G, I) is known. Given G = x, the value of H I (x) is known. For an iid sequence of m random variables G ′ 1 , . . . , G ′ m distributed as G, independent from the previous sequence, the sequence H I (G ′ 1 ) , . . . , H I (G ′ m ) is an iid sample of H I (G). Finally, we get two independent iid samples of the respective real-valued random variables H I (G) and H J (G), with respective sample sizes m and n. Under H 0 the two samples derive from the same distribution. Hence any two-sample test of equality of distribution can be applied, like the two-sample Kolmogorov-Smirnov test. This can also be applied if G is discrete or derived from some particular data. More specifically, the test statistic can be written

T m,n := D (H I (G ′ m ) || H Jn (G n )) (3)
where D is the chosen dissimilarity between two distributions of real-valued random variables, for example, the Kolmogorov-Smirnov distance, or the JSD divergence, and where G ′ m and G n are two independent rv distributed as the respective samples {G ′ 1 , . . . , G ′ m } and {G 1 , . . . , G n }.

When the distribution of the test statistic is unknown, the problem of obtaining an approximate critical region of the considered test, under H 0 , can be solved by applying a numerical simulation method.

Under the assumption H 0 , the distribution of T m,n can be estimated by simulation: two independent iid samples of m and n observations respectively are generated according to the joint original distribution (G, I). The distance T H0 m,n between corresponding empirical distributions of H I (G ′ m ) and H I (G n ) is calculated. By repeating this procedure a large number of times, one obtains under H 0 the approximate quantiles of the distribution of T H0 m,n , from which one can get a critical region of approximated level α.

Finally, on a sample of n observations, we choose here m = n and a clustering method is accepted if T n,n < (1 -α) th quantile of the obtained distance distribution T H0 n,n under H 0 .

As an application, using JSD metric, we consider testing H 0 vs H 1 where the underlying distribution is the two dimensional Gaussian mixture defined in Section 2.1. Based on the numerical simulation method described above, with m = n = 10 000, and by repeating the procedure 1000 times, we obtain the approximate quantiles of the corresponding JSD distribution and the critical region of approximated level α = 5%, which is q JSD 0.95 = 0.04. Then for a given clustering algorithm if T n,n > q JSD 0.95 we reject H 0 . Hence the clustering method fails to understand the labeling difficulty of the considered observation. In our case, both fanny and FKM are rejected as both T fanny n,n and T FKM n,n are greater than 0.04.

It is important to note that the proposed test only evaluates the compatibility of a clustering method with an underlying probabilistic model in terms of fuzziness. A test rejection does not imply that the clustering method is ineffective; rather, it indicates that it must be modified if it is to be compatible with a given probabilistic model. This is the goal of the following section.

Correction of clustering probabilities

In this section, we assume that the true model behind observations is known, i.e. that the joint law of (G, I) is known. We will discuss the case of unknown model in the further Section 5. Assume that the statistical test of Section 3.3 is rejecting the clustering method. Before proposing another clustering algorithm, a possible solution is to adjust the underlying cluster probabilities using a parametric transformation.

Recall that p j (x ℓ ) is the probability of assigning observation x ℓ to cluster j. In order to adjust these probabilities to fit the original mixture distribution we propose the following transformation:

p j (x ℓ ) * = p j (x ℓ ) θ i∈J p C i (x ℓ ) θ , ( 4
)
where J is the proposed set of cluster indices and θ ∈ R + a parameter, to be estimated, reflecting the reason behind the bad clustering quality.

Then, based on these new probabilities a new clustering entropy is calculated. We denote it by H Jn (G, θ). Hence, one may assess the effect of this transformation by recomputing JSD (H

I (G n ) || H Jn (G n , θ)).
Now, the optimal value of the parameter θ is the one minimizing the previous JSD, which will be given by: θ = arg min

θ JSD (H I (G n ) || H Jn (G n , θ)) .
So, in case θ > 1, one can say that the proposed clustering algorithm is not categorical enough and has a higher fuzziness level then the one of the original mixture distribution. Otherwise, if θ < 1, the current clustering algorithm is too categorical and we need to regularize it by injecting some source of fuzziness. Due to this interpretation, the parameter θ can be considered as an indicator of fuzziness.

To study the efficiency of this approach, we apply it on a previous example considering two different soft clustering algorithm "fanny" and "Fuzzy k-means (FKM)". We generate a sample n = 10 000 of the two dimensional Gaussian mixture distribution with the same parameters as the one in Section 2.1. Based on this training sample we calculate the estimator θ of θ. Then, we use the value of this estimator to correct the probabilities proposed by each of the clustering algorithm on a testing sample of 3000 observations and we compare the impact of this correction on the quality of the clustering on both training and testing sets by calculating the corresponding JSD. Under these assumptions we get results summarized in Table 3.

Then, by applying the parametric transformation proposed in Equation ( 4) we are able to reduce enormously the JSD and correcting the behavior of both algorithms, so that they become more suited to find the hidden labels of the original data. Even on the testing set, the correction has a considerable favorable influence on both algorithms' clustering quality. In addition, comparison of the cdf of different entropies H I (G n ) vs H Jn G n , θ are illustrated in Figure 8 for training set and Figure 9 for testing.

Note that a more complex transformation than the one proposed in (4) can be done using more than one parameter and focusing on local parts of the distribution. By adapting the same approach described in this section we can select the best transformation according to, say, the least JSD criterion. Also, as the application is done on data with gaussian underlying distributions, it is obvious to expect that the best performance, in terms of clustering, will be for the Gaussian Mixture Model (GMM) based clustering method, introduced by [START_REF] Mclachlan | Mixture models: Inference and applications to clustering[END_REF], where gaussian distributions are considered as priors. As expected, even before correction, JSD of the GMM method is about 0.042 for training and 0.089 for testing with a good fit of the cdf s of the two entropies (see Figure 10). Furthermore, after applying the proposed probability transformation, we used the theoretical labels of the 2-D Gaussian mixture distribution generated in Subsection 2.1 to examine the ability of fuzzy clustering algorithms in reconstructing the original clusters of an underlying data. Then, the idea is to make a confusion matrix comparing theoretical labels with the labels proposed by the clustering method. The accuracy of each clustering method is calculated as the proportion of well classified observation and formulated by:

Accuracy = 1 n n ℓ=1 j∈J
P [ assign observation ℓ to cluster j /Theoretically observation ℓ is in cluster j] .

(5)

Tables 4 and5 show the clustering accuracy as defined in Equation ( 5) for several soft clustering method applied on 2-D Gaussian mixture distribution generated data, before and after applying the correction of clustering probabilities on training and testing sets respectively. Note that here also θ is estimated on the training set and used to correct clustering probabilities on the testing set. In general, the accuracy of the clustering is improving in a significant way after applying the correction method, except for the GMM clustering method where the accuracy is almost the same as this method is designed to deal with gaussian mixture underlying distributions which is fitting perfectly the generated data. In addition, the estimated value of θ is giving information about the quality of the clustering method.

For Fanny and FKM algorithms the fuzziness index θ is far greater than one, so these two methods are not categorical enough and have higher fuzziness level than the original mixture. However, θ for GMM is close to one, which is reasonable because GMM is designed in a way to generate original clusters of a gaussian mixture distribution. Also, an added value of the proposed correction method, is that after performing the correction one can revisit the result of the statistical test that was introduced in Section 3.3 to reassess if after modification, a certain clustering method that was rejected previously is 6 and7 for training and testing sets respectively. It is clear that after applying the parametric correction of the probabilities, the values of the statistic of the test of Section 3.3 decrease significantly to become much closer to the critical region empirical quantile q JSD 0.95 = 0.04 . In fact some of the clustering algorithms that was rejected before correction are accepted after it. Then, correcting clustering probabilities makes: 1) different clustering method more accurate in reconstructing original clusters and 2) the comparison between different clustering method more reasonable in addition to the selection of the best performing one.

Application on real data

Up to this point, we have assumed that the hidden mixture model and the (G, I) distribution were known. Of course, this is not always the case in the real world. Indeed, once the model is understood, no clustering technique is needed. The goal of this section is to demonstrate that the presented methods are still applicable even when the true mixture model is hidden, that they can aid in comparing different algorithms, that they can improve prediction accuracy on test samples for both labeled and unlabeled data, and that they can aid in selecting the appropriate number of clusters. For each of these purposes, specific examples are provided.

Supervised learning, improving classification accuracy

We consider here a supervised classification task. We show that the proposed fuzziness correction of Section 4, together with a standard estimation of some bandwidth parameter, improve the clustering accuracy of most fuzzy clustering algorithms, on an unlabeled test sample. To do so, we look into possible estimation procedures for comparing clustering fuzziness to an estimated probabilistic model.

As an application of our approach, we use an open source data from the UCI Machine Learning Repository, [START_REF] Dua | UCI machine learning repository[END_REF], available at https://archive.ics.uci.edu/ml/datasets/Breast+ Cancer+Wisconsin+%28Diagnostic%29.

This data classifies n = 569 individuals in either malignant (M) or benign (B) breast cancer using 30 numeric features characterizing each person. I.e., a space of dimension R 569×30 , labeled into I = {1 = "M", 2 = "B"} cluster indexes. Therefore, assuming that we are in the context of a mixture distribution G, the probability that a given observation x is sampled from the underlying rv X i can be estimated for any i ∈ I, by for example:

p i (x) = α i f i (x) j∈I α j f j (x) , (6)
with, f i (•) the kernel density estimation (KDE) of the observations generated by the underlying rv X i (i.e., observations in the cluster of index i). In other words it is an estimation of the pdf of the rv X i . Note that, practically one can use kernel density estimation (KDE) functions, already implemented in statistical software, to compute these values. The quantity α i is the proportion of observations in the same cluster, i.e.,

α i = Number of observations in cluster i n .
In this application, we used a standard KDE of the R software, provided by the kde function, corresponding here to a plug-in bandwidth as defined in [START_REF] Chacón | Multivariate plug-in bandwidth selection with unconstrained pilot bandwidth matrices[END_REF] accuracy on a test sample.

For the three different soft clustering methods introduced previously, we compare the impact of the parametric correction proposed in Equation ( 4) on the quality of the clustering by first estimating the optimal value of θ, on a training set representing 80% of the considered data set, and then calculating the corresponding JSD on both training and testing sets. Results are in Tables 8 and9.

Considering clustering fuzziness, Fanny is the worst clustering method for the considered data, even after the correction it is affecting the same probability for both clusters (1 and 2) without any discrimination power. On the other hand, the performance of FKM and GMM is improving enormously after the correction with a preference to the GMM method. The significant impact of the parametric correction on FKM and GMM can be seen also when comparing the cdf of the entropies before and after correction in Figures 11 and12. Note that, the impact of the correction for the GMM is limited because even before correction GMM is performing well on reconstructing original labels. In addition, one can remark that the estimated value of θ for the FKM is far higher than one which is equivalent to say that FKM has a higher fuzziness level than the original mixture of the considered cancer data set. However, the fuzziness index θ for GMM is close to one, which is reasonable because GMM is already performing well on discrimination level even before the correction.

Finally, the illustrations show that the correction of clustering weights, together with the use of a standard bandwidth parameter, significantly improve the clustering accuracy on a test sample. The definition of the fuzziness level, at the origin of the proposed correction, appears here to be of practical interest, even when the hidden mixture model is unknown. Now, by focusing only on FKM and GMM we can also compute the accuracy of each clustering method before and after the parametric correction based on the approach described in Equation ( 5).

Here also θ is estimated on the training set and used to compute the clustering accuracy for both training and testing sets. Results are in Tables 10 and11. On both training and testing datasets one may remark that the clustering accuracy of the FKM method is improving significantly after the correction of the corresponding probabilities. However, the correction has no impact on the accuracy of the GMM. This is also reflected by the value of θ which is close to one, indicating that the fuzziness level of the GMM is acceptable and is capturing well the original labels of the clusters. This may be explained by the possibility that the underlying distribution of the considered dataset is following a gaussian mixture distribution. This simple illustrative example could be extended in several ways, that are let as future work.

Firstly, we considered here a supervised learning problem. For unsupervised learning, one option is to perform hard clustering of the data, obtain labels for each point, and then estimate the entropy as in a supervised setting. Repeating the procedure for several hard clusterings would account for both the uncertainty of the initial hard clustering and the uncertainty caused by the intrinsic overlapping of cluster densities for a given labelling and bandwidth.

Secondly, this illustration was heavily depending on the choice of bandwidth parameter. Intuitively, a larger bandwidth results in more fuzziness, hence impacting the correction parameter. Here, a standard bandwidth estimator helped improving the accuracy of the clustering algorithm, but the choice of other bandwidths would be possible.

Unsupervised learning, finding clusters number

In the context of selecting the optimal number of clusters in unsupervised analysis, several state-of-theart methods exist. These include the Elbow Method, which identifies the point of maximum curvature in the within-cluster sum of squares plot [START_REF] Thorndike | Who belongs in the family?[END_REF]; the Silhouette Coefficient, which measures the compactness and separation of clusters [START_REF] Rousseeuw | Silhouettes: a graphical aid to the interpretation and validation of cluster analysis[END_REF]; the Gap Statistic [START_REF] Tibshirani | Estimating the number of clusters in a data set via the gap statistic[END_REF], which compares within-cluster dispersion to that of reference data; methods using the Bayesian Information Criterion (BIC), which balances model fit and parameter complexity [START_REF] Zhao | Knee point detection in bic for detecting the number of clusters[END_REF]; and more other approaches like the Davies-Bouldin Index for cluster validation [START_REF] Davies | A cluster separation measure[END_REF]. Notable recent references on these methods include studies by [START_REF] Milligan | An examination of procedures for determining the number of clusters in a data set[END_REF] on cluster validity indices, [START_REF] Rousseeuw | Silhouettes: a graphical aid to the interpretation and validation of cluster analysis[END_REF] on silhouettes, [START_REF] Tibshirani | Estimating the number of clusters in a data set via the gap statistic[END_REF] on data-driven approaches, [START_REF] Sugar | Finding the number of clusters in a dataset: An informationtheoretic approach[END_REF] for an information theoretic approach, and Halkidi and Koutsopoulos (2019) for advancements in clustering techniques and validation methods. Software packages are also available, see e.g. [START_REF] Charrad | Nbclust: an r package for determining the relevant number of clusters in a data set[END_REF].

Our work proposes a novel approach that considers the fuzziness level generated by each cluster method, incorporating the degree of uncertainty into the evaluation of the optimal number of clusters for a more comprehensive analysis. In this application, we show that measuring the fuzziness level of a clustering algorithm may help choosing the right number of clusters.

Intuitively, the fuzziness level is related to the right number of clusters: any segmentation of a true cluster in two parts tends to create some fuzziness areas, so that the proposed number of clusters cannot be too high. Proposing a smaller number of clusters may be reasonable when it leads to fuse true clusters, but it is clearly inappropriate when it leads to cut true cluster in parts, creating fuzziness. Based on this idea, we will use the distribution of the empirical entropy to choose the right number of clusters.

For the sake of clarity, let us consider first a toy example. Let G be a Gaussian mixture distribution in one dimension with a pdf : f G (x) = 0.25f N (0,1) (x) + 0.25f N (3,0.1) (x) + 0.2f N (6,1) (x) + 0.2f N (8,0.5) (x) + 0.1f N (10,1) (x).

where f N (.,.) denotes the pdf of a Gaussian rv with indicated parameters. Associated labels are lost, so that the setting is similar to an unsupervised clustering algorithm. The left side of Figure 13 represents the pdf of G, while the right side illustrates the function H I . Remark that this is a difficult example: even when the pdf is known, the number of clusters is here difficult to find, the truth is five clusters, but the values of entropy indicate the presence of a gray zone, where the level of fuzziness reaches its peak around the center of the given mixture distribution.

The right side of Figure 13 represents the entropy H I (G). Once again, one can remark that the labeling is perfectly accurate when the values x of G are far from the central area of the distribution (i.e. left and right queues of the mixture distribution). However, the difficulty of cluster labeling is higher for central values where the region is fuzzy in terms of distribution selection.

As the maximal value of the entropy depends on the cluster number, we use here a rescaled versions of the empirical entropy, so that all rescaled entropies are belonging to [0, 1]:

H Jn (x) = 1 log 2 (card(J )) H Jn (x)
Now, for FKM algorithm, we draw the distribution of H Jn (G n ), with n = 25000, for different numbers of clusters, varying from 2 to 8. Table 12 shows that by considering the number of clusters that minimizes the mean and the 75th percentile the preference goes to five clusters which fits with the simulated underlying mixture distribution. In other words, one selects the number of clusters with the lowest level of uncertainty for more than 75% of the available data. On the other hand, based on Figure 14 the entropies H Jn (G n ) show that with 5 clusters, the entropy tends to be smaller, which is another hint on the optimal number of clusters. Now, beyond considering the entropy mean as a primary indicator, it is valuable to delve into additional insights provided by the 95th percentile which is an extreme percentile. This extreme percentile can shed light on situations where a higher number of clusters may be justified. In conclusion, while the mean of the entropy typically serves as a good indicator, examining nonextreme percentiles, e.g. 75th, can provide useful information. However, taking into account extreme percentiles of the entropy can offer also valuable insights, particularly in situations where the complexity of the data warrants a more nuanced approach to cluster analysis. This multifaceted analysis enhances the decision-making process, allowing for a more informed determination of the optimal number of clusters in various cluster analysis applications.

The example here is chosen to be very clear to understand and visualize, but as the dimension d increases, the visual assessment of the right number of clusters is very difficult, whereas the considered Now, the same methodology is applied on the real dataset considered in Subsection 5.1. Based on FKM algorithm, we draw the distribution of H Jn (G n ), for different numbers of clusters, varying from 2 to 5. Table 13 shows that by considering the number of clusters that minimizes the empirical mean and the 75th percentile of H Jn (G n ) we get n = 2, which is matching the data separated into two classes M and B. On the other hand, based on Figure 15 the box-plots of entropies H Jn (G n ) show that with 2 clusters, the entropy tends to be smaller, which is another hint on the optimal number of clusters. It is important to note that we are observing a similar pattern as in the previous example with simulated data, where the extreme percentile (95th percentile) also highlights a preference for a higher number of clusters.

As a third application in the same unsupervised context, we propose to apply our method for selecting the optimal number of clusters to the MNIST dataset, aiming to demonstrate, once again, its effectiveness in extracting meaningful information within the data. In fact, the MNIST dataset is a widely recognized benchmark in the field of machine learning and computer vision. It stands for Modified National Institute of Standards and Technology database and is a collection of handwritten digits, from 0 to 9, commonly used for training and evaluating various classification algorithms. The dataset consists of a total of 70,000 grayscale images, each measuring 14 shows that by considering the number of clusters that minimizes the mean and the 75th percentile the preference goes to ten clusters which is matching the data classified into ten digits. On the other hand, based on Figure 16 the entropies H Jn (G n ) show that with 10 clusters, the entropy distribution tends to be skewed more to the left.

Conclusion

This paper introduced an innovative metric, based on Shannon's entropy for assessing the quality of clustering algorithms in terms of fuzziness level. The proposed metric can be used to compare the performance of two clustering algorithms in a way to conclude which one is more over/under confident.

In addition, a statistical test has been constructed, based on the introduced fuzziness level metric. It helps to make a decision about accepting or rejecting a clustering algorithm. Moreover, a parametric adjustment of the underlying probabilities of a clustering algorithm has been introduced in order to improve the fuzziness level of the corresponding clustering method. According to many numerical simulations and real world data applications, it was noticed that the proposed methodology helps users significantly in getting better discrimination power from a given clustering method. Applications to find the optimal number of clusters were also proposed. So far, a first perspective of this work is to try to develop a theoretical proof of the probability distribution of the proposed metric. A second perspective is to adapt the proposed methodology on data of higher level of complexity (i.e., data of mixed typology).

Figure 1 :

 1 Figure 1: Gaussian mixture distribution of Case A. Left: pdf of the mixture distribution G, right: entropy H I representing the variation of the labeling difficulty.

Figure 2 :

 2 Figure 2: Dirac mixture distribution of Case B. Left: probability masses of the mixture distribution G, right: zero-valued entropy H I representing the trivial labeling of any observation of G.

  lines of the pdf (b) Contour lines of the entropy HI

Figure 3 :

 3 Figure 3: Two dimensional Gaussian mixture distribution of Case C Left: pdf of the mixture distribution G, right: entropy H I representing the variation of the labeling difficulty.

Figure 4 :

 4 Figure 4: Entropies of two dimensional Gaussian mixture distribution of Case C for different scales

  (a) 1D, Fanny algorithm (b) 2D, Fanny algorithm

Figure 5 :

 5 Figure 5: cdf of different entropies (under fanny algorithm) in the case of a Gaussian mixture distribution (n = 10 000 observations). On the left: one dimensional case of Case A, on the right: two dimensional case of Case C. High abscissas values correspond to points with high labeling difficulty, or equivalently high uncertainty. The cdf of H I (G n ) are blue upper curves, cdf of H Jn (G n ) are red lower curves.

  (a) 2D, Fanny algorithm (b) 2D, FKM algorithm

Figure 6 :

 6 Figure 6: cdf of different entropies in the case of a 2-D Gaussian mixture model, Case C, (n = 10 000 observations). On the left fanny algorithm, on the right FKM algorithm.. The cdf of H I (G n ) are blue upper curves, cdf of H Jn (G n ) are red lower curves.

Figure 7 :

 7 Figure 7: cdf of entropies H Jn (G n ) for different values of n in the case of a 2-D Gaussian mixture model, Case C. On the left fanny algorithm, on the right FKM algorithm.

  (a) 2D, Fanny algorithm (b) 2D, FKM algorithm

Figure 8 :

 8 Figure 8: cdf of different entropies in the case of a 2-D Gaussian mixture model (n = 10 000 observations, Case C). On the left fanny algorithm ( θ = 5.958), on the right FKM algorithm ( θ = 3.348), with modified probabilities applied on training data

Figure 9 :

 9 Figure 9: cdf of different entropies in the case of a 2-D Gaussian mixture model (n = 3 000 observations, Case C). On the left fanny algorithm ( θ = 5.958), on the right FKM algorithm ( θ = 3.348), with modified probabilities applied on testing data

Figure 10 :

 10 Figure 10: cdf of different entropies in the case of a 2-D Gaussian mixture model (n = 10 000 observations, Case C) for GMM algorithm ( θ = 0.970) applied on training data without modified probabilities

  (a) real data, FKM without correction (b) real data, FKM with correction

Figure 11 :

 11 Figure 11: cdf of different entropies for FKM algorithms. On the left without modified probabilities, on the right with modified probabilities, applied on cancer data

Figure 12 :FKM

 12 Figure 12: cdf of different entropies for GMM algorithms. On the left without modified probabilities, on the right with modified probabilities, applied on cancer data

  Figure 13: Gaussian mixture distribution. Left: pdf of the mixture distribution G, right: entropy H I (G) representing the variation of the labeling difficulty.

Figure 15 :

 15 Figure 15: Boxplots of entropies H Jn (G n ) for different number of clusters based on FKM algorithm and using real data

Figure 16 :

 16 Figure 16: Mean, 75th percentile and 95th percentile of entropies H Jn (G n ) for different number of clusters based on FKM algorithm and using MNIST data

Table 1 :

 1

		008	0.585	0.056	0.681
	Variance	0.0014	0.1004	0.027	0.024
	Minimum	0	0.144	≃ 0	0.415
	Maximum	0.993	1.534	≃ 1	1

Moments of entropy distributions for soft clustering fanny-algorithm (n = 10 000 observations)

Table 2 :

 2 JSD for different values of n, the lower the better.

Table 3 :

 3 Comparison of JSD: original vs corrected on testing and training data, the lower the better.

		Training set		Testing set
		JSD(θ = 1)	θ	JSD θ	JSD(θ = 1) JSD θ
	Fanny	0.895	5.958	0.043	0.918	0.088
	FKM	0.630	3.348	0.039	0.678	0.084

Table 4 :

 4 Accuracy of different clustering methods calculated on training data before and after correction, the higher the better. θ is estimated based on the training set and used to correct clustering probabilities on both training and testing sets.

	Clustering method Accuracy before correction Accuracy after correction	θ
	GMM	0.968	0.980	0.970
	Fanny	0.802	0.968	5.958
	FKM	0.904	0.969	3.348
	Clustering method Accuracy before correction Accuracy after correction	θ
	GMM	0.978	0.97	0.970
	Fanny	0.798	0.967	5.958
	FKM	0.901	0.967	3.348

Table 5 :

 5 Accuracy of different clustering methods calculated on testing data before and after correction, the higher the better. θ is estimated based on the training set and used to correct clustering probabilities on both training and testing sets. now accepted. Conclusions of the statistical test applied on several soft clustering methods before and after probability correction are presented in Tables

Table 6 :

 6 Statistical test conclusions on training data. JSD is calculated on training data before and after correction, the lower the better. θ is estimated based on the training set and used to correct clustering probabilities on both training and testing sets.

		Value of the JSD statistic	Decision about H 0
	Clustering method Before correction After correction Before correction After correction
	GMM	0.042	0.035	Accepted	Accepted
	Fanny	0.895	0.043	Rejected	Accepted
	FKM	0.630	0.039	Rejected	Accepted

Table 7 :

 7 Statistical test conclusions on testing data. JSD is calculated on testing data before and after correction, the lower the better. θ is estimated based on the training set and used to correct clustering probabilities on both training and testing sets.

Table 8 :

 8 . Results would naturally differ for other bandwidths, but we show here that this classical KDE helps improving the clustering Comparison of JSD: original vs corrected on training data. θ is estimated on the training set and used to compute JSD (the lower the better) on both training and testing sets.

		Training set
		JSD(θ = 1)	θ	JSD θ
	Fanny	1	0.001	1
	FKM	0.989	9.883	0.249
	GMM	0.0834	1.652 0.0556
		Testing set
		JSD(θ = 1)	θ	JSD θ
	Fanny	1	0.001	1
	FKM	1	9.883	0.352
	GMM	0.0709	1.652 0.0827

Table 9 :

 9 Comparison of JSD: original vs corrected on testing data. θ is estimated on the training set and used to compute JSD (the lower the better) on both training and testing sets.

Table 10 :

 10 Accuracy of different clustering methods based on training data, the higher the better. θ is estimated on the training set and used to compute JSD on both training and testing sets.

	20

Table 11 :

 11 Accuracy of different clustering methods based on testing data, the higher the better. θ is estimated on the training set and used to compute JSD on both training and testing sets.

Table 12 :

 12 Number of clusters Mean of HJn (G n ) 75th percentile of H Jn (G n ) 95th percentile of H Jn (G n ) Empirical mean,75th percentile and 95th percentile of H Jn (G n ) for different number of clusters based on fanny algorithm.Figure 14: Mean, 75th and 95th percentile of entropies H Jn (G n ) for different number of clusters based on FKM algorithm and using simulated data Number of clusters Mean of H Jn (G n ) 75th percentile of H Jn (G n ) 95th percentile of H Jn (G n )

	2	0.307	0.398	0.886
	3	0.236	0.412	0.796
	4	0.248	0.469	0.658
	5	0.201	0.365	0.612
	6	0.214	0.399	0.619
	7	0.209	0.386	0.587
	8	0.250	0.400	0.599

Table 13 :

 13 Empirical mean, 75th percentile and 95th percentile of H Jn (G n ) for different number of clusters based on FKM algorithm and applied on real data.

Table 14 :

 14 Number of clusters Mean of HJn (G n ) 75th percentile of H Jn (G n ) 95th percentile of H Jn (G n ) Empiricalmean, 75th percentile and 95th percentile of H Jn (G n ) for different number of clusters based on FKM algorithm and using MNIST data.

	5	0.956	0.992	0.999
	6	0.955	0.992	0.999
	7	0.952	0.990	0.999
	8	0.941	0.984	0.999
	9	0.940	0.983	0.998
	10	0.939	0.982	0.998
	11	0.941	0.985	0.998
	12	0.942	0.985	0.998
	13	0.943	0.986	0.998
	14	0.945	0.987	0.998
	15	0.946	0.988	0.998