Christophe Biernacki

Matthieu Marbac

Vincent Vandewalle

Ensai V Vandewalle

Gaussian Based Visualization of Gaussian and Non-Gaussian Based Clustering

Keywords: Dimension reduction, Gaussian mixture, factorial analysis, linear discriminant analysis, model-based clustering, visualization

come   L'archive ouverte pluridisciplinaire

Introduction

The exploratory field of multivariate statistics essentially encompasses the clustering and the visualization tasks. Both are often jointly involved: either visualization is performed 1 in the hope of revealing the "graphical evidence" of a cluster structure in the data set; or clustering is performed first and the visualization task follows in the hope of providing a better understanding of the estimated cluster structure. We are primarily interested in the second scenario.

Clustering [START_REF] Jajuga | Classification, clustering and data analysis: recent advances and applications[END_REF] serves to summarize (typically large) data sets by assessing a partition among observations, the latter being thus summarized by (typically few) characteristic classes. Model-based clustering [START_REF] Mclachlan | Finite mixture models[END_REF][START_REF] Mcnicholas | Mixture model-based classification[END_REF][START_REF] Biernacki | Mixture models[END_REF]) achieves the clustering purpose in a probabilistic framework, usually consisting of modeling the whole data distribution using a finite mixture model. Classical challenges can thereby be solved by using tools that rely on theoretical statistics, e.g., estimating the partition using an EM algorithm [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF], selecting the number of groups using information criteria such as BIC or ICL [START_REF] Schwarz | Estimating the dimension of a model[END_REF][START_REF] Biernacki | Assessing a mixture model for clustering with the integrated completed likelihood[END_REF], dealing with missing values among observations [START_REF] Larose | Model-Based Clustering of Incomplete Data[END_REF]. Moreover, this framework allows for the analysis of different types of data by "simply" adapting the related cluster distribution: continuous data [START_REF] Banfield | Model-based Gaussian and non-Gaussian clustering[END_REF][START_REF] Celeux | Gaussian parsimonious clustering models[END_REF][START_REF] Mcnicholas | Parsimonious Gaussian mixture models[END_REF], categorical data [START_REF] Goodman | Exploratory latent structure analysis using both identifiable and unidentifiable models[END_REF][START_REF] Celeux | Clustering criteria for discrete data and latent class models[END_REF][START_REF] Gollini | Mixture of latent trait analyzers for model-based clustering of categorical data[END_REF][START_REF] Marbac | Latent class model with conditional dependency per modes to cluster categorical data[END_REF], mixed data [START_REF] Kosmidis | Model-based clustering using copulas with applications[END_REF][START_REF] Mcparland | Model based clustering for mixed data: clustmd[END_REF][START_REF] Punzo | Clustering bivariate mixed-type data via the clusterweighted model[END_REF][START_REF] Marbac | Model-based clustering of gaussian copulas for mixed data[END_REF][START_REF] Mazo | A semiparametric and location-shift copula-based mixture model[END_REF], functional data [START_REF] Samé | Model-based clustering and segmentation of time series with changes in regime[END_REF][START_REF] Bouveyron | Model-based clustering of time series in group-specific functional subspaces[END_REF][START_REF] Jacques | Model-based clustering for multivariate functional data[END_REF], networks data [START_REF] Daudin | A mixture model for random graphs[END_REF][START_REF] Zanghi | Fast online graph clustering via erdős-rényi mixture[END_REF][START_REF] Ambroise | New consistent and asymptotically normal parameter estimates for random-graph mixture models[END_REF].

Once the clustering process has been performed, the next step is to provide a good understanding of it to practitioners. However, a rendering based on a raw delivery of the model parameters and/or the resulting partition (or the related conditional membership probabilities) can be quite inefficient: understanding of the parameters requires specific knowledge of the model at hand, and the partition can be also hard to read since it is just a numerical list of length equal to the sample size, which must be large enough to have initially motivated the clustering process.

Visualization is designed to express, in a user-friendly manner, the estimated clustering structure. Its general principle is to design a mapping of the data, or of other related statistical results such as the cluster shape, within a "friendly" space (generally R 2 ) while maintaining some properties that the data, or the related statistical results, have in their native space. The vast majority of proposed mapping relies on different variants of factorial analysis or other distance-based methods (like multidimensional scaling). For a thorough list of visualization methods, see Section 2.2, and references therein. However, all standard mappings waste most clustering information that is conveyed by the probabilistic approach, except [START_REF] Scrucca | Dimension reduction for model-based clustering[END_REF] which uses the full model-based approach for the mapping. However, this approach is limited to continuous data. This paper defends the key idea that only a so-called model-based visualization output can exploit the model-based clustering input, since both involved objects are of the same nature (probabilistic objects). More precisely, the mixture model used for the visualization output will inherit from the overlap of the initial mixture model. In fact, this is similar to defining a particular mapping but without any explicit distance design. This process has the clear advantage of being straightforwardly suitable for any type of data without any specific definition of the mixture output since only the conditional memberships need to be estimated. In fact, the specificity of initial data has been taken into account by the initial clustering modeling process. The mixture output involves spherical Gaussian components, with the same number of components as the clustering mixture. This particular Gaussian choice is informed by both some technical arguments and some user-friendly arguments.

The resulting drawing displays meaningful spherical cluster shapes in the bivariate continuous space. Finally, accuracy of this drawing is assessed by comparing the apparent overlap mixture on the graph and the overlap of the initial mixture. To have a good understanding of our proposal, in particular its link with model-based clustering techniques, its general outline can be summarized as follows:

1. select a model-based clustering technique for data at hand; 2. extract the whole distribution of the classification probabilities from the fitted model; 2 Clustering: from modeling to visualizing

Model-based clustering of multi-type data

Clustering aims to estimate a partition z = (z 1 , . . . , z n ), composed of K clusters, of a data set x = (x 1 , . . . , x n ), composed of n observations. The component membership of each observation x i is given by z i = (z i1 , . . . , z iK ), with z ik = 1 if x i arises from component k and z ik = 0 otherwise. Z denotes the space of any z i . In a very general situation, each observation x i is defined on a space X described by d X variables which can be continuous, categorical or functional.

Model-based clustering aims to solve the clustering task in a full probabilistic framework by modeling the distribution of the full data set (x, z), z being considered as a latent part of the data set. This framework has the decisive advantage of consolidating the exploratory clustering result through the background of mathematical statistics (estimation, model selection; see for instance [START_REF] Mclachlan | Finite mixture models[END_REF], [START_REF] Biernacki | Mixture models[END_REF]). More precisely, all couples (x i , z i ) are assumed to independently arise from the distribution defined by the probability density function (pdf)

f (x i , z i ) = K k=1 [π k f k (x i )] z ik (1)
where π k is the proportion of the kth component (π k > 0 and k π k = 1) and f k is the pdf of this component.

From such a modeling, two interesting by-product distributions are available. Firstly, the (marginal) distribution of each x i corresponds to the so-called K component mixture defined by the pdf f

(x i ) = k π k f k (x i ).
Secondly, under distribution f , the probability that x i arises from component k, denoted by t ik (f ), is expressed by

t ik (f ) = p(z ik = 1|x i ; f ) = π k f k (x i ) K =1 π f (x i )
.

(2) Thus, all information about the classification probabilities for observation x i can be stored in a K -1 continuous vector t i (f ) (because K k=1 t ik (f ) = 1) where

t i (f ) = (t i1 (f ), . . . , t i(K-1) (f )). ( 3
)
Information about the classification probabilities of sample x is given by t

(f ) = (t 1 (f ), . . . , t n (f )).
Traditionally, components f k are parametrized by finite dimensional vectors and an EM algorithm, or one of its variants [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF], is used to provide an estimate f of f (the π k 's and the parameters associated with the f k 's). Alternatively semi-or non-parametric mixtures can be considered [START_REF] Benaglia | An em-like algorithm for semiand nonparametric estimation in multivariate mixtures[END_REF]). Finally, an estimated partition ẑ can be straightforwardly deduced from t( f ) by using the rule of maximum a posteriori (MAP) defined by ẑik = 1 iif k = arg max t i ( f ).

Thus, the key point to achieve this model-based clustering procedure is to define the distributional space F where f stands for (f ∈ F). In fact, only the space of components f k has to be defined. Clearly, choosing component pdf f k depends on X . Many proposals already exist such as multivariate Gaussian or multivariate t-distributions for continuous data [START_REF] Mclachlan | Finite mixture models[END_REF][START_REF] Mcnicholas | Mixture model-based classification[END_REF], a product of multinomial distributions for categorical data [START_REF] Goodman | Exploratory latent structure analysis using both identifiable and unidentifiable models[END_REF]; see also the running example later), a product of Gaussian and multinomial distributions when mixing continuous and categorical data [START_REF] Moustaki | Latent class models for mixed variables with applications in archaeometry[END_REF], see also numerical experiments in Section 6.1), specific models for functional data or for network data (see respectively numerical experiments in Section 6.2 and 6.3, with references therein).

However, because of their potential complexity, such previous mathematical features may fail to provide a user-friendly clustering understanding. Indeed, it may be difficult to have a useful overview of individuals in clusters through ẑ (or through t( f )) if n or K is too large. Similarly, it may be difficult to get a useful overview of the whole clusters (proportions, shapes, positioning, etc.) through f if the space X involves many features (d large) or involves features of complex types (like a mix of categorical and functional features), a situation where the pdf of the components can be particularly hard to embrace as a whole. As a matter of fact, the need for a user-friendly understanding of the mathematical clustering results (at both individual and pdf levels) is the very reason for using some specific visualization procedures.

Overview of clustering visualization

Mapping vs. drawing Visualization is probably one of the most appealing data analysis tasks for practitioners since its fundamental purpose is to display some potentially complex and technically demanding statistical objects (typically a data set or a pdf) on simple and seamlessly accessible graphs (typically a scatter plot or an isodensity curve). The whole process can be viewed as the achievement of two different successive steps. The mapping step transforms the initial statistical object into a simpler statistical one typically through a dimensionality reduction of a data set or of a pdf (marginal pdf). It produces no graphical output at all. The drawing step provides the final graphical display from the output of the previous mapping step and usually entails the use of conventional graphical toolboxes. It fine-tunes all the possible graphical parameters.

Individual mapping

The clustering visualization task is probably thought as firstly as visualizing simultaneously the data set x and its estimated partition ẑ. Typically, the corresponding mapping, designated below by M ind , transforms the data set x, defined on X , into a new data set y = (y 1 , . . . , y n ), defined on a new space Y, as follows:

M ind ∈ M ind : x ∈ X n → y = M ind (x) ∈ Y n . (4)
Here M ind denotes a particular mapping family. This family varies according to the type of data involved in X and also depending on whether they use only data x or additional clustering information ẑ or t( f ).

Methods x and also for outputs y, the resulting coordinate matrix ŷ being obtained by minimizing a loss function. However, dissimilarities have to be defined specifically in respect of the type of data under consideration. For just illustrating this point, the Euclidean distance is frequent for continuous data whereas the Hamming distance is more suitably for binary data.

In an machine learning framework, methods such as self-organized map (SOM; [START_REF] Kohonen | Self-organized formation of topologically correct feature maps[END_REF]) or generative topographic mapping (GTM; [START_REF] Bishop | Gtm: The generative topographic mapping[END_REF]) have been developed to summarize the data in terms of a set of reference points having a regular spatial organization corresponding generally to a two-dimensional regular network. But, even if nodes of the network are usually interpreted as clusters, these ones essentially serve as a preprocessing step for limiting the number of prototypes to be considered at a second step in a hierarchical clustering [START_REF] Vesanto | Clustering of the self-organizing map[END_REF].

Methods taking into account additional clustering information ẑ or t( f ) are less common and are mostly restricted to continuous data. We can cite linear discriminant analysis (LDA; [START_REF] Fisher | The use of multiple measurements in taxonomic problems[END_REF], [START_REF] Xanthopoulos | Linear Discriminant Analysis[END_REF]) which takes into account cluster separation by defining the mapping through to a particular factorial analysis of the cluster means. Also, in the specific case of continuous data, [START_REF] Hennig | Asymmetric linear dimension reduction for classification[END_REF], [START_REF] Scrucca | Dimension reduction for model-based clustering[END_REF] and [START_REF] Morris | Dimension reduction for model-based clustering via mixtures of multivariate t-distributions[END_REF] defined a specific linear mapping between X and Y. In that case, the distribution of y is itself a (less-dimensional) Gaussian mixture or a multivariate t-mixture, with the same number of components and the same proportions, which can be expressed as g = k π k g k . Finally, their method aims to preserve the related conditional membership probabilities t( f ) and t(g), namely the classification probabilities of x with f and the classification probabilities of y with g, respectively. In other words, the aim is to find a linear mapping that preserves as far as possible, through the mapping mixture g, the cluster separation occurring in the original mixture f . Somewhat the method we proposed in this paper is related to this idea but it is not restricted to continuous distributions in the mixture and it does not relay on a linear mapping.

Pdf mapping Many visualizations are in practice overlaid by additional information relating to the corresponding mapping distribution. This mapping transforms the ini-

tial mixture f = k π k f k , defined on the distributional space F, into a new mixture g = k π k g k ,
defined on the distributional space G. It can be expressed as the following mapping, designated here by M pdf :

M pdf ∈ M pdf : f ∈ F → g = M pdf (f ) ∈ G, (5)
where M pdf denotes a particular mapping family. It is important to note that the pdf mapping M pdf is rarely defined "from scratch" since it can be obtained as a "simple" byproduct from the previous individual mapping M ind . However, in practice, the resulting mixture g can be particularly tedious to calculate (possibly no closed-form solution available outside linear mappings), which can be partially overcome by displaying the empirical mapping of a very large sample. But the resulting pdf can also have non-conventional isodensity shape per cluster (for instance clusters with disconnected parts), undermining somewhat all the user-friendliness that is expected when using pdf visualization.

Running example

As a running example for this paper, we consider the data set of [START_REF] Schlimmer | Concept acquisition through representational adjustment[END_REF]. It is composed of votes for each of the n = 435 U.S. House of Representatives Congressmen on d X = 16 key votes. For each vote, three levels are considered: yea, nay or unknown disposition. Data are clustered by a mixture of products of multinomial distributions [START_REF] Goodman | Exploratory latent structure analysis using both identifiable and unidentifiable models[END_REF]. Parameter estimation is performed by maximum likelihood and model selection is done by the BIC [START_REF] Schwarz | Estimating the dimension of a model[END_REF], which selects K = 4 components. The r package Rmixmod [START_REF] Lebret | Rmixmod: the r package of the model-based unsupervised, supervised and semisupervised classification mixmod library[END_REF] is used for inference.

As an output of this estimation step, the user is provided with a partition and a parameter. It may be not really convenient to have a detailed look at the partition of 435 individuals. In regard to the parameters, the mixing proportions can be suitable for a quick, but partial, understanding of the clustering result. However, going further into the clustering understanding by analyzing the multinomial parameters can be very laborious since it entails 192 = 16 × 3 × 4 values to be observed and compared.

It is also possible to analyze the clustering results graphically in a conventional way.

Figure 1 presents the scatter plot of the Congressmen and their partition on the first map of the MCA, obtained by the r package FactoMineR [START_REF] Lê | Factominer: an r package for multivariate analysis[END_REF]). It appears that the scatter plot provided by MCA is quite hard to read. Firstly, it is well-known that total inertia is hard to interpret, and consequently the information about a possible relative positioning of clusters can be questionable. Secondly, even if faithful, overlap between components is not fully visible and thus does not allow for a straightforward interpretation of f .

Mapping clusters as spherical Gaussians

In this section, we focus our attention on the so-called pdf visualization. We argue that, asymptotically on the sample size n, similar objects result from both pdf and individual visualization processes. However, we will hold a specific discussion on individual visualization below.

Changing the mapping objects to be controlled

Traditional way: controlling the mapping family As described in Section 2.2, the cornerstone of all traditional pdf visualization procedures is based on defining the mapping family M pdf (or more exactly M ind from which M pdf is almost always deduced). As just an example, the reader can have in mind the classical linear mapping for the continuous case. Then, the pdf family G of g is a simple by-product of M pdf , and thus can be denoted by G(M pdf ). Using the general mapping expression (5), G(M pdf ) is naturally expressed as follows:

G(M pdf ) = {g : g = M pdf (f ), f ∈ F, M pdf ∈ M pdf } . (6)
As an immediate consequence, the nature of G can depend to a great extent on the choice of M pdf , leading potentially to very different cluster shapes. Arguments that lead to traditional M pdf (or M ind ) rely essentially on a combination of user-friendly and easy-to-compute properties. For instance, in the continuous case, linear mappings are often retained (like for PCA). In the categorical case, a continuous space Y is often targeted (like for MCA).

It is a similar situation for functional data with FPCA or also for mixed data with MFA or MDS, even if MDS is a somewhat more complex procedure since it is not always defined in closed-form. However, such choices may vary significantly from one statistician to another one. For instance, MDS relies on defining dissimilarities both inside spaces X and Y and changing them could significantly affect the resulting mapping.

New proposed method: controlling the distribution family Alternatively, the general mapping expression (5) can be seen as indexed by the distribution family G, the mapping M pdf being now obtained as a by-product, and thus now denoted by M pdf (G).

This new point of view is straightforwardly expressed as:

M pdf (G) = {M pdf : g = M pdf (f ), f ∈ F, g ∈ G} . ( 7
)
It corresponds to the reversed situation of ( 6) where G has to be defined instead of M pdf .

This new freedom indeed provides an opportunity to directly force G to be a user-friendly mixture family.

Constrained spherical Gaussians as matching candidates

Spherical Gaussians One of the most simple candidate belonging to the "user-friendly mixture family" is probably the spherical Gaussian mixture defined on

Y = R d Y . Its pdf is defined for any y ∈ R d Y by g(y; µ) = K k=1 π k φ d Y (y; µ k , I), (8)
where µ = (µ 1 , . . . , µ K ) and φ d Y (.; µ k , I) is the pdf of the Gaussian distribution with mean

µ k = (µ k1 , . . . , µ kd Y ) ∈ R d Y and covariance matrix equal to identity I.
Because clustering visualization is the central task of this work, it is expected to require that both mixtures f and g(•; µ) have the most similar clustering information. This information is measured by the probabilities of classification (see ( 3)). We denote now p f as the probability distribution function of the probabilities of classification under mixture f and p g (•; µ) as the probability distribution function of the probabilities of classification under mixture g(•; µ). In this manner, a standard way for measuring the difference between both f and g(•; µ) clustering property could be the following Kullback-Leibler divergence (p f being the reference measure):

δ KL (f, g(•; µ)) = T p f (t) ln p f (t) p g (t; µ) dt (9)
where T = {t : t = (t 1 , . . . , t K-1 ), t k > 0, k t k < 1}. Then, the set G is defined as

G = {g : g = g(•; µ), g ∈ arg min δ KL (f, g), f ∈ F} . ( 10
)
This is mimicking the idea of [START_REF] Scrucca | Dimension reduction for model-based clustering[END_REF] and [START_REF] Morris | Dimension reduction for model-based clustering via mixtures of multivariate t-distributions[END_REF] which imposes (as far as possible) the retention of the overlap of mixture distributions before and after the mapping.

More constraints on Gaussians Another requirement should be that p g (•; µ) and g are linked by a one-to-one mapping, meaning that for one distribution f , there is a unique distribution g(•; µ) which minimizes ( 9). This target is reached firstly by setting

d Y = K -1
and secondly by setting µ K = 0, µ kh = 0 if h > k, and µ kk ≥ 0. This last restriction prevents any rotation and/or translation of y from providing the same distribution p g (•; µ) but a different distribution g(•; µ).

For technical convenience, we consider now the following one-to-one mapping Λ between t and a classical transformation of t, which we express hereafter as t

Λ : t = (t 1 , . . . , t K-1 ) ∈ T → t = ( t1 , . . . , tK-1 ) ∈ [0, ∞) K-1 with tk = t k 1 -K-1 =1 t . (11)
It is essential to note that, by considering mixture g(•; µ), there is also a one-to-one mapping Ψ between y and t

Ψ(•; µ) : y → t with Ψ(y; µ) = π 1 φ d Y (y; µ 1 , I) π K φ d Y (y; µ K , I) , . . . , π K-1 φ d Y (y; µ K-1 , I) π K φ d Y (y; µ K , I) . (12)
Moreover, we have

Ψ -1 ( t; µ) = M -1      ln t1 π K π 1 + 1 2 ||µ 1 || 2 . . . ln tK-1 π K π K-1 + 1 2 ||µ K-1 || 2      with M =      µ 1 . . . µ K-1      , (13)
where matrices M and M -1 are lower triangular.

Estimating the Gaussian centers

Invoking a log-likelihood From (10), we consider the distribution g(•; µ * ) where the centers µ * are defined by

µ * = arg min δ KL (f, g(•; µ)). Noting that |Jac Λ (t)| -1 = K-1 k=1 t k , µ * = arg min [0,∞) K-1 pf ( t) ln pf ( t) pg ( t; µ) K-1 k=1 t k d t, (14)
where pf (•) and pg (•; µ) denote the pdf of t by considering distribution f and g(•; µ), respectively. It is possible to explicitly and easily express the previous pg (•; µ) distribution by using the change of variables theorem combined with the linear transformation (13), which leads to the following term, obtained by noting that |Jac

Ψ -1 (•;µ) ( t)| -1 = K-1 k=1 (µ kk tk ) -1 , pg ( t; µ) = g(Ψ -1 ( t; µ); µ) K-1 k=1 (µ kk tk ) -1 . ( 15
)
Unfortunately, the Kullback-Leibler divergence defined in ( 14) has generally no closed-form.

However, it is easy to independently draw a sample of S ratios of conditional probabilities t = ( t(1) , . . . , t(S) ) from pf . This sample can be used to estimate the previous integral such that maximizing the following normalized (observed-data) log-likelihood function

L(µ; t) = 1 S S s=1 ln pg ( t(s) ; µ), (16)
is equivalent to solving (14) asymptotically on S.

Maximizing the log-likelihood The log-likelihood ( 16), combined with (15), entails the pdf of a mixture model. Thus, it can be classically broken down into a normalized complete-data log-likelihood L comp and a normalized (empirical) entropy term E as follows [START_REF] Hathaway | Another interpretation of the em algorithm for mixture distributions[END_REF]):

L(µ; t) = L comp (µ; t) + E( t), (17)
where, noting Λ -1 ( t) = (Λ -1 1 ( t), . . . , Λ -1 K-1 ( t)) the inverse function of Λ,

L comp (µ; t) = c - K-1 k=1 ln µ kk - 1 2S K-1 k=1 S s=1 Λ -1 k ( t(s) )||Ψ -1 ( t(s) ; µ) -µ k || 2 , (18)
E( t) = - 1 S K-1 k=1 S s=1 Λ -1 k ( t(s) ) ln Λ -1 k ( t(s) ), ( 19
) with a constant term c = 1 S K-1 k=1 S s=1 Λ -1 k ( t(s) ) ln π k -1 S K-1 k=1 S s=1 ln Λ -1 k ( t(s) ).
Since the normalized entropy does not depend on µ, an estimate μ of µ * is obtained only via the maximization of the normalized complete-data likelihood. Note that this maximization is straightforward if (and only if) K = 2. In such a case, we have for component 1 (remind that for component 2 we have μ2 = 0 for identifiability reasons), with μ1 ∈ R K-1 = R:

μ1 = -1 + 1 + 1 S 2 ( S s=1 Λ -1 k ( t(s) ))( S s =1 Λ -1 k ( t(s ) )[ln( t(s ) π 2 π 1 )] 2 ) 1 2S S s=1 Λ -1 k ( t(s) ) . ( 20
)
If the number of components is more than two, a standard Quasi-Newton algorithm should be run with different random initializations, in order to avoid possible local optima. In practice, we use S = 5000 which allows for a fast estimation of the centers and stability of the results.

Remark It can be noticed that generative topographic mapping [START_REF] Bishop | Gtm: The generative topographic mapping[END_REF] (GTM) could have some similarities with our approach since it is also based on a spherical

Gaussian mixture model of the data, estimated through an EM algorithm. However, this fitted distribution is a mixture where the locations of the centers of the clusters on the latent space (typically two-dimensional) are defined by advance on a regular grid avoiding any clustering interpretation. Thus GTM is essentially a non-linear dimensionality reduction where no particular clustering focus is taken into account.

4 Final visualization as bivariate spherical Gaussians 4.1 From a multivariate to a bivariate Gaussian mixture

Because g is defined on R K-1 , it is inconvenient to draw this distribution if K ≥ 4. Therefore, we apply an LDA to g to represent this distribution on its most discriminative map (i.e., eigen value decomposition of the covariance matrix computed on the centers μ by considering the mixture proportions π), leading to the following bivariate spherical Gaussian mixture g:

g(ỹ; μ) = K k=1 π k φ 2 (ỹ; μk , I), (21)
where ỹ ∈ R 2 , μ = ( μ1 , . . . , μK ) and μk ∈ R 2 . The (standard) percentage of inertia of LDA serves to measure the quality of the mapping from g to g. In addition, the accuracy of the mapping from the initial mixture f to the final "ready-to-be-drawn" mixture g can be easily compared through the following difference between the normalized (theoretical) entropy of the partition related to f and the normalized (theoretical) entropy of the partition related to g, namely

δ E (f, g) = - 1 ln K K k=1 X t k (x; f ) ln t k (x; f )dx - R 2 t k (ỹ; g) ln t k (ỹ; g)dỹ . ( 22
)
The two normalized entropy terms involved in δ E are well-referenced in the literature as detailed in the book Bezdek et al. (1999). Such theoretical quantities can be easily estimated using empirical values, similarly to the normalized empirical entropy E given by Equation ( 19)1 . Its meaning is particularly relevant: if δ E (f, g) is close to zero then the component overlap conveyed by g (over f ) is accurate; if it is close to one, then g strongly underestimates the component overlap of f ; if it is close to negative one, then g strongly overestimates the component overlap of f . Thus, δ E (f, g) serves to evaluate the bias of the visualization.

Remark When the initial data set x is in the continuous space X = R d and also when the initial clustering relies on a Gaussian mixture f whose covariance matrices are identical, then the proposed mapping is strictly equivalent to applying a LDA to the centers of f .

Proposal for drawing the bivariate Gaussian mixture

The aim is now to draw the pdf g on the most discriminative map in a manner that highlights as much as possible the overlap between components. Indeed, it is primarily such information that acted as a guideline to transform f into g. The proposed graph will display the following elements:

• Cluster centers: the locations of μ1 , . . . , μK are materialized by vectors.

• Cluster spread: the 95% probability level is displayed by a black border which separates the area outside the probability level in white from the area inside the probability level in gray levels i.e., the set Ω α is in gray where

Ω α = {ỹ : g(ỹ; μ) > u α } (23)
with u α such that Ωα g(ỹ; μ)dỹ = 1 -α (by default plots are made with α = 0.05).

The white area around the black border is designated by "outside the proba. level" in the legend.

• Cluster overlap: curves of iso-probability of the MAP classification are also displayed for different levels (associated with a specific grayscale), a curve being composed of the set of ỹ such that max k=1,...,K π k φ 2 (ỹ; μk , I) g(ỹ; μ) = .

(24)

Default plots are made with ∈ {0.8, 0.95}. Areas with gradual grayscale correspond to different ranges of levels . For instance, the gray area displaying the graphical region between levels 0.8 and 0.95 is designated by "0.8 < Pr. Classif. < 0.95" in the legend.

• Mapping accuracy: the accuracy of this representation is given by the difference between entropies δ E (f, g) and also by the percentage of inertia by axis. Those two criteria provide complementary information. δ E is the most important criterion. Indeed, it indicates if the observed overlap within the visualization Gaussian mixture (thus in the visualization space R 2 ) is faithful or not to the overlap within the initial mixture (thus in the initial data space X ). The percentage of inertia associated to a given axis indicates the classical LDA accuracy value between the multivariate spherical Gaussian mixture in R K-1 and the final visualization spherical Gaussian mixture in R 2 . Its main interest is to rank the two axes of the final visualization, the first one being the most discriminative axis.

Remark When some components are highly overlapped, it is difficult to visualize their respective cluster spreads since they are highly mixed, as expected. However, for wellseparated components, their respective cluster spreads tend to be disjointed, meaning that

Ω α ≈ ∪ K k=1 Ω k,α with Ω k,α = {ỹ : π k φ 2 (ỹ; μk , I) > u α }.
In that limit and simplified case, each gray area associated to cluster k, with k ∈ {1, . . . , K} such that Ω k,α = ∅, is a disc with area A k,α ≈ 2π(ln π k -ln(2π) -ln u α ). This formula leads to the following two comments.

First, the area of the disc related to each well-separated component is directly related to the logarithm of its mixing proportion. Consequently, it allows to visually rank populous clusters by their visual size. Second, the threshold α has to be chosen small enough for visualizing particularly clusters associated with a small mixing proportion π k . Indeed, it can happen in such cases that, for the proposed value α = 0.05, Ω k,α = ∅, meaning that the related cluster shape is not displayed on the graph2 .

It has also to be noticed that alternative drawings for having simultaneously spread, overlap and mixing proportions can be freely proposed by practitioners, eventually by using existing graphical tools for bivariate Gaussian isodentities (see examples later in Section 4.3). As expressed in the title of Section 4.2, this paper just offers a proposal. Indeed, the key innovation of the paper is to offer a new mapping from any mixtures to Gaussians, the drawing part being more incidental, even if useful.

Tutorial on the bivariate spherical Gaussians visualization

We offer here a tutorial for avoiding any misinterpretation of the proposed bivariate spherical Gaussians visualization. It illustrates also its potential interest for having a fast, easy, unifying and faithful overview of the potentially complex underlying clustering structure.

The selected illustrative mixture corresponds to four bivariate Gaussians with non-spherical covariance matrices and different mixing proportions cases. In these simple and well-known situations, many standard bivariate illustrations of Gaussians and/or a related data set already exist, of which users are familiar with them. By this way, users would easily understand how to properly analyze the new drawing we propose, the general understanding principle being unchanged for more complex scenarios as the higher dimensional and/or non-Gaussian cases that are considered in the running example or also in Section 6 later.

We decompose this tutorial into four characteristic scenarios. Scenario 1: "reference" mixture The 1st bivariate (d X = 2) Gaussian layout is composed of four components (K = 4) with mixing proportions π 1 = π 2 = 0.4 and • Axes meaning: the 1st axis is the most discriminative one provided by the LDA mapping (66.09% of the discriminant power). The first two axes sum to 66.09 + 23.41 = 89.50% of the discriminant power, thus most of the discriminant information is present on this two dimensional mapping.

π 3 = π 4 = 0.1, with means ν 1 = (-1, 3), ν 2 = (3, 2), ν 3 = (5, -3), ν 4 = (2, -6)
• Mixing proportions: the probability areas in gray color are directly related to the mixing proportions as discussed at the end of Section 4.2. As a consequence, it appears immediately that components 1 and 2 are more populous than the two others.

• Cluster overlap: it clearly appears also that components 1 and 2 overlap much more than components 3 and 4 do. This fact does not appear clearly at all on Figure 2a since mixing proportions are not involved in the isodensity representation. More generally, separation of all couples of components appears to be faithful. For instance, components 2 and 4 (and also components 1 and 3) are the most separated ones.

Scenario 2: effect of changing the centers Figure 3 considers a similar case to Scenario 1 but where components 1 and 2 are closer with regards to their means, so their overlap has increased. Indeed, here ν 1 = (1, 3), Figure 3b has now a lower displaying accuracy compared to Figure 2b since difference between the entropies is 0.15. However its absolute value is sufficiently close to zero and far from one (its maximum theoretical value)

to allow faithful interpretation of the overall components displaying. Figure 4b shows that this means adaptation was successfully enough since the difference between the entropies is very close to zero. And it can be seen on the same figure that components 3 and 4 overlap very significantly, as expected. Finally, notice that the graphical overlap display of clusters 1 and 2 appears to be changed between Scenario 1 and 3, whereas the real probabilistic overlap between both is unchanged. It is the consequence of the graphical cluster spread as defined in Equation ( 23) which depends on the relative positionning of all clusters simultaneously.

Scenario 4: effect of changing the mixing proportions Finally, Scenario 4 considers Scenario 3 where the component proportions are now equal. Figure 5b illustrates that the size of the gray areas around the centers has been adapted in comparison to Figure 4b.

Indeed, we see now that the gray areas are substantially equal (even if components 3 and 4 still quite overlap). As a comparison, Figure 5a is strictly the same as Figure 5a, illustrating that the simplest isodensity curves representation from mclust we used as a reference candidate here is unable to represent the component proportions. Remark: other competing representations Throughout previous scenarios, we have just considered as a "reference" visualization the basic isodensity curves representation provided by the classical R package mclust [START_REF] Scrucca | mclust 5: clustering, classification and density estimation using Gaussian finite mixture models[END_REF]. Indeed, this one, by its simplicity, appears to be convenient for helping to understand the main behaviour of the new visualisation proposal. There obviously exists many other possible graphical competitors that users can try by themselves for each previous scenario, some of them being good candidates for this (simple) tutorial. However each of them is expected to have some limitations for the complex situations we will consider in the next sections.

For instance, even if mclust offers more advanced graphical outputs, all are restricted to Gaussian mixtures. The R package fpc (Hennig 2010), through the function "weightplots", visualizes classification probabilities independently of the type of mixture, Gaussian or not.

However, it requires multiple figures (each figure displays one component vs. the others)

instead of a unique one, which can be less comfortable for a synthetic mixture overview in case of multiple components. Another generic and classical graphical tool (enable to deal with any type of data) is multidimensional scaling (MDS; [START_REF] Young | Multidimensional scaling: History, theory, and applications[END_REF], [START_REF] Cox | Multidimensional Scaling[END_REF]) but, as already discussed in Section 4, specific dissimilarities have to be defined for each type of data under consideration, thus complexifying the user task.

Remark: theoretical study with two (possibly non-Gaussian) components It is also possible to theoretically describe the behaviour of our Gaussian-like visualization in the case of (possibly non-Gaussian) components, but in the restricted two-component situation. From Equation ( 20), it can be observed that, if the overlap between the two components increases (i.e. t(s) → 1, Λ -1 k ( t(s) ) → 0.5 and π 2 /π 1 → 1), then μ1 → 0. Since μ2 = 0, it implies that the two Gaussians also strongly overlap in our Gaussian-like visualization. On the contrary, if the overlap between the two components decreases (i.e., for some s ∈ {1, . . . , S} values, t(s) → 0 while Λ -1 k ( t(s) ) → 0 and, for other s ∈ {1, . . . , S} values, t(s) → +∞ while Λ -1 k ( t(s) ) → 1), then μ1 → +∞. In that case, the two Gaussians are more and more separated.

Continuation of the running example

We now illustrate the previous visualization proposition on the running example. Figure 6 is the component interpretation graph obtained for the congressional voting records. It presents the Gaussian-like component overlap on the most discriminative map. In this way, it provides in a more visual way than a traditional confusion table the overlap information of the initial mixture f . We also graphically observe the ranking between the different cluster spreads, indicating some variety in mixing proportions (numerically speaking, we have π1 = 0.21, π2 = 0.05, π3 = 0.35 and π4 = 0.39). Note that the mapping of f on this graph is accurate because the difference between entropies is almost zero (i.e., δ E (f, g) = -0.08).

For instance, this figure also shows that the components with most observations (i.e., components three and four) are composed of strongly different Congressmen. Indeed, the overlap between these components is almost zero. Moreover, component one contains Congressmen which are more moderate that Congressmen of components three and four.

5 Proposal for drawing a pseudo bivariate scatter plot

From pdf visualization to individual visualization

We have limited our attention to the mapping of the initial cluster pdf f , described by (5), intentionally discarding the mapping of the initial individual data set x, described by (4).

We have already discussed that the pdf mapping (5) can be a by-product of the individual mapping (4). However, the reverse is mathematically impossible, the distributional information being weaker than the random variable information.

Nevertheless, a pseudo scatter plot y of x can be mapped onto R K-1 by transforming the ratios of probabilities Λ(t i (f )), associated with x i by f , into values y i through the reverse application of Ψ -1 (•; μ) associated with g(•; μ), namely y i = Ψ -1 (Λ(t i (f )); μ) (i = 1, . . . , n). Then, each observation y i is projected on the LDA map, leading to a pseudo scatter plot ỹ = (ỹ 1 , . . . , ỹn ), with each ỹi ∈ R 2 .

We use the term "pseudo" for ỹ (or for y) because some caution has to taken in order to avoid misunderstanding. Indeed, the distribution of ỹ is expected to be different from g(•; μ), the essential property of ỹ being to respect as far as possible the conditional probabilities t(f ) associated to x, but not to respect as far as possible the distribution f itself.

In fact, only when f corresponds to a spherical Gaussian mixture do distributions of ỹ and of g(•; μ) match.

Such remarks strongly affect the related drawing we propose for the scatter plot ỹ:

• Data drawing: display ỹ on the LDA best discriminative map as dots of different colors representing the partition membership z.

• Conditional probabilities: information about the uncertainty of classification is given by the curves of iso-probability of classification.

• Mapping accuracy: again, the accuracy of this representation is given by the difference between entropies δ E (f, g) and also by the percentage of inertia by axis.

• No pdf overlay: do not display ỹ simultaneously with g(.; μ) to avoid misunderstanding; therefore use another graph. Here the scatter plot is not necessarily Gaussian (spherical or other), phenomenon that appears clearly. Indeed, remember that the only property of the initial mixture which is preserved through the procedure we propose is the conditional membership distribution (or in short its "overlapping") under the constraint that this conditional distribution is a byproduct of a spherical Gaussian mixture. Thus, each data sample drawn on Figure 5b has to be seen as a faithful representation of its conditional membership representation under the spherical constraint, but absolutely not a faithful representation of its mixture distribution.

Tutorial on the pseudo bivariate scatter plot visualization

The interest is to quicky access to the membership uncertainty of each individual, what becomes also clearer by the borderlines displayed on the figure. Notice obviously that this membership interpretation is accurate as soon as the difference between the entropies is not far from zero (in absolute value), what is the case for this particular scenario.

Continuation of the running example

Figure 8 displays the scatter plot of the observation memberships obtained on the congressional voting records. It overlays on the most discriminative map the curve of isoprobabilities of classification and the cloud of observations. Three levels of probabilities of classification are considered (0.95, 0.80 and 0.50) and observations are represented with the label of the component maximizing the posterior probability of classification. This plot serves to focus on specific observations and, for instance, to detect observations classified with a high uncertainty of classification. Note that some points which are classified in component two (the blue points) are located in the area containing the observations of component three. This is a standard phenomenon in LDA. Indeed, in such a case, if the maximum a posteriori rule is applied on the native space (i.e., R K-1 ) than its results can be different to the results of maximum a posteriori rule when applied on the low-dimension space (i.e., R 2 ).

Numerical illustrations for complex data

We present applications of the visualization method on three real data sets composed of complex features (mixed, functional and network data). They illustrate the ability of the method to deal with extremely different kinds of data and of mixtures, without any new specific development. Results are obtained by the r package ClusVis which implements the visualization method.

Mixed data: Contraceptive method choice

Data This dataset x is a subset of the 1987 National Indonesia Contraceptive Prevalence Survey [START_REF] Lim | A comparison of prediction accuracy, complexity, and training time of thirty-three old and new classification algorithms[END_REF]. It describes 1473 Indian women with two numerical variables (age and number of children) and eight categorical variables (education level, education level of the husband, religion, occupation, occupation of the husband, standard-of-living index and media exposure). which detects six components.

Model drawing Figure 9 presents the component interpretation graph obtained for the contraceptive method choice data. It shows overlaps between component one, two and three. Moreover, components four and five are significantly different from component six. Such a visualization is in accordance with a fine study of Table 1, which presents the parameters of the continuous variables. Indeed, we can see that components one, two and three are all composed of middle-age women who have many children. On the contrary, components four and five are composed of young women who have few children. Finally, component six is composed of the oldest women. Therefore, the first axis can be interpreted as the age of the women (left side is composed of older women than on the right side).

Finally, the second axis distinguishes components two, four and six from the others. As shown in Table 2, these three components have the same mode for the eight categorical variables. Table 3: Confusion matrix between the component memberships and the political party memberships.

from R 5 to R 2 . Indeed, the entropy of g (in R 5 ) is closed to those of f with a value of 0.004. The loss of information due to the data projection can also be detected by the inertia, because only 56.76% of the inertia is represented by this most discriminative map.

Therefore, the components overlaps should be interpreted with more caution than in the previous examples, where the differences between entropies were close to zero.

The graph shows that components three and six overlap significantly. This result is expected because component three mainly comprises UMP members ("French Republican") and component six is composed of supporters of economic-liberalism. Finally, component one, which comprises politicians from different political parties, is the most isolated. The proposed procedure has been developed by considering that the model used to visualize is a constrained Gaussian mixture. Obviously, other continuous distributions could be considered. However, these distributions must define a one-to-one relation between the space of the probability of classification and the continuous space. If several distributions compete, then the best distribution could be the distribution that leads to minimization of the Kullback-Leibler divergence δ KL (f, g). Alternatively, because there is a step of LDA-like projection, the best distribution could be the distribution that minimizes the difference between the normalized entropies obtained by f and by the projected distribution g, namely δ E (f, g). Finally, if non-Gaussian mixtures are considered, it is crucial that the resulting

Scatter plot drawing

  3. fit a multivariate spherical Gaussian mixture respecting as far as possible the distribution of the previous classification probabilities; 4. (a) draw the spherical Gaussian mixture pdf on the most discriminative bivariate map; (b) draw a "pseudo" bivariate scatter plot representing the individual classification probabilities on the most discriminative bivariate map. This paper is organized as follows. Section 2 focuses on the context of model-based clustering for mixed data and reviews the main existing visualization techniques of a clustering result. Section 3 presents the central contribution of this work consisting of matching any clustering mixture and a spherical multivariate Gaussian visualization mixture according to their component overlap. We describe, in Section 4, how to draw this Gaussian mixture in the most discriminative map. Like any visualization method, our proposition can introduce a bias. However, because we propose a full model-based visualization approach, an index measuring this bias (and thus the quality of the representation) is presented. Section 5 then proposes a means of displaying a kind of individual plotting on the same discriminative map to access each individual data cluster membership positioning. Section 6 illustrates in depth the Gaussian model-based proposition on three real data sets with different types of features (mixed data, functional data and network data). Throughout this paper, the proposition is also illustrated via a categorical running example. Section 7 concludes this work.

Figure 1 :

 1 Figure 1: Scatter plot of the Congressmen and their partition on the first MCA map.

Figure

  Figure 2adisplays isodensity curves of the related mixture provided by the classical R package mclust[START_REF] Scrucca | mclust 5: clustering, classification and density estimation using Gaussian finite mixture models[END_REF]. Just the component number has been manually overlayed on the means. Many other packages are expected to offer similar visualization choices. Figure2bdisplays the proposed bivariate spherical Gaussian visualization associated to this 1st mixture scenario. Note that this Gaussian representation is really spherical, even if it can appear distorted due to the axes scaling. First of all the difference between the entropies has to be checked. Its low absolute value (0.03) indicates that the cluster

  Figure 3b clearly indicates that components 1 and 2 overlap significantly more, what is really the fact in the underlying experimental design.

Figure 2 :

 2 Figure 2: Representation of the clusters for Scenario 1.

Figure 3 :

 3 Figure 3: Representation of the clusters for Scenario 2.

  Figure 4: Representation of the clusters for Scenario 3.

  Figure 5: Representation of the clusters for Scenario 4.

Figure 6 :

 6 Figure 6: Component interpretation graph of the congressional voting records.

Figure

  Figure 7a displays, in a classical way, a sample of size n=1000 from Scenario 3 described in Section 4.3. Figure 7b displays the related pseudo scatter plot we propose. The LDA map is exactly the same between this figure and Figure 4b. However, some comments are required for avoiding misinterpretation of this new plot.

  (a) classical bivariate scatter plot (b) proposed bivariate pseudo scatter plot

Figure 7 :

 7 Figure 7: Scatter plot related to Scenario 3.

Figure 9 :

 9 Figure 9: Component interpretation graph of the Contraceptive method choice.

  Figure15presents the scatter plot of the observation memberships. It confirms the proximity between components three and six.We presented a generic method for visualizing the results of a model-based clustering in a "Gaussian way". This method allows for visualization of any model-based clustering made on any type of data, because it is only based on the distribution of classification probabilities. It permits to interpret the results of a model-based clustering but not to select the best clustering method (choosing a clustering method has to be performed before through a classical model selection process). In this way, it is not an exploratory visualization method, as such methods are often dedicated to. This method produces two graphs. The first graph allows for the component interpretation through all component overlaps. The second graph represents a scatter plot of the observations and many curves of iso-probabilities of classification. It serves to focus on the classification of specific observations and to quantify the risk of misclassification related to the conditional probability of membership. Finally, the accuracy of the procedure can be measured by taking the difference between the normalized entropies obtained by the model used to cluster and by the model defined on the visualization map.

Figure 11 :

 11 Figure 11: Partition among the 1 189 bike stations in Paris (each row corresponds to a single component and gives the mean curve in bold and observations belonging to this component in thin).

Table 1 :

 1 Parameters of the continuous variables for the Contraceptive method choice.

		Age		Number of children
		Mean Variance Mean	Variance
	Component 1	35	30	4	4
	Component 2	35	22	3	2
	Component 3	40	42	5	9
	Component 4	25	10	1	1
	Component 5	24	13	2	1
	Component 6	45	7	5	8
			33

Note that the normalized empirical entropy E could also eventually be additionally normalized by ln K. We have not do it in order to follow the seminal definition of Hathaway. And, obviously, it has no impact at all on the related parameter estimation in Section 3.3.

We never observed this event through experiments of this paper. But, in case it happens, we propose to draw at least the center of the unobserved cluster on the graph.

(a) mclust isodensity curves representation (b) proposed spherical-like representation

Model used to cluster These mixed data are clustered by a mixture f assuming that variables are independent within components [START_REF] Moustaki | Latent class models for mixed variables with applications in archaeometry[END_REF]. Within a component, the continuous variables follow Gaussian distributions and categorical variables follow multinomial distributions. Maximum likelihood inference is performed by the r package Rmixmod [START_REF] Lebret | Rmixmod: the r package of the model-based unsupervised, supervised and semisupervised classification mixmod library[END_REF]. Model selection is done by the BIC criterion due to the projection of the observations y i from R 5 to R 2 when projecting a discriminative rule.

Functional data: Bike sharing system

Data We consider now the study of the Bike sharing system data presented by [START_REF] Bouveyron | The discriminative functional mixture model for a comparative analysis of bike sharing systems[END_REF]. We analyze station occupancy data collected over the course of one month on the bike sharing system in Paris. The data were collected over 5 weeks, between February, 2008) subsum them into a single one. Note that intra-domain links can be considered if hostnames are not identical. Finally, in this experiment we consider that edges are not directed, which is not realistic but which does not affect the interpretation of the groups.

This network presents an interesting community-based organization due to the existence of several political parties and commentators. We assume that authors of these blogs tend to link, blogs with similar political positions as a result of their political affinities.

Model used to cluster We use the graph clustering via Erdös-Rényi mixture proposed by [START_REF] Zanghi | Fast online graph clustering via erdős-rényi mixture[END_REF] and implemented on the r package mixer. As proposed by these authors, we consider K = 6 components. The confusion matrix between the component memberships and the political party memberships is given in Table 3.

Model drawing Figure 14 presents the component interpretation graph obtained for the French political blogosphere data. The graph moderately over-represents the component overlaps (i.e., δ E (f, g) = -0.17). Indeed, the (normalized) entropy of f is equal to 0.016 while the entropy of the projection of g into the most discriminative space is equal to 0.221. Note that this difference between entropies is due to the projection of the data graph presenting the component overlaps is still meaningful and does not entail excessively time consuming calculus (visualization should be a fast step). In particular, it could be meaningful to explore non-unimodal component candidates, like mixture of mixtures, sometimes called also multilayer mixture (see for instance [START_REF] Hennig | Methods for merging gaussian mixture components[END_REF]).