Serguei Barannikov

Ilya Trofimov

Grigorii Sotnikov

Ekaterina Trimbach

Alexander Korotin

Alexander Filippov

Huawei Noah'

Evgeny Burnaev

Manifold Topology Divergence: a Framework for Comparing Data Manifolds

We develop a framework for comparing data manifolds, aimed, in particular, towards the evaluation of deep generative models. We describe a novel tool, Cross-Barcode(P,Q), that, given a pair of distributions in a high-dimensional space, tracks multiscale topology spacial discrepancies between manifolds on which the distributions are concentrated. Based on the Cross-Barcode, we introduce the Manifold Topology Divergence score (MTop-Divergence) and apply it to assess the performance of deep generative models in various domains: images, 3D-shapes, time-series, and on different datasets: MNIST, Fashion MNIST, SVHN, CIFAR10, FFHQ, chest X-ray images, market stock data, ShapeNet. We demonstrate that the MTop-Divergence accurately detects various degrees of mode-dropping, intramode collapse, mode invention, and image disturbance. Our algorithm scales well (essentially linearly) with the increase of the dimension of the ambient highdimensional space. It is one of the first TDA-based practical methodologies that can be applied universally to datasets of different sizes and dimensions, including the ones on which the most recent GANs in the visual domain are trained. The proposed method is domain agnostic and does not rely on pre-trained networks.

Introduction

Geometric perspective in working with data distributions has been pervasive in machine learning [START_REF] Belkin | Laplacian eigenmaps and spectral techniques for embedding and clustering[END_REF][START_REF] Michael | Geometric deep learning: Going beyond euclidean data[END_REF][START_REF] Goodfellow | Deep learning[END_REF][START_REF] Chazal | An introduction to topological data analysis: fundamental and practical aspects for data scientists[END_REF][START_REF] Moor | Topological autoencoders[END_REF][START_REF] Kim | PLLay: Efficient topological layer based on persistence landscapes[END_REF]. Reconstruction of the data from observing only a subset of its points has made a significant step forward since the invention of Generative Adversarial Networks (GANs) [START_REF] Goodfellow | Generative adversarial networks[END_REF]. Despite the exceptional success that deep generative models achieved, there still exists a longstanding challenge of good assessment of the generated samples quality and diversity [START_REF] Borji | Pros and cons of gan evaluation measures[END_REF].

35th Conference on Neural Information Processing Systems (NeurIPS 2021).

For images, the Fréchet Inception Distance (FID) [START_REF] Heusel | Gans trained by a two time-scale update rule converge to a local nash equilibrium[END_REF] is the most popular GAN evaluation measure. However, FID is limited only to 2D images since it relies on pre-trained on ImageNet "Inception" network. FID unrealistically approximates point clouds by Gaussians in embedding space; also, FID is biased [START_REF] Bińkowski | Demystifying mmd gans[END_REF]. Surprisingly, FID can't be applied to compare adversarial and non-adversarial generative models since it is overly pessimistic to the latter ones [START_REF] Ravuri | Classification accuracy score for conditional generative models[END_REF].

The evaluation of generative models is about comparing two point clouds: the true data cloud P data and the model (generated) cloud Q model . In view of the commonly assumed Manifold Hypothesis [START_REF] Belkin | Laplacian eigenmaps and spectral techniques for embedding and clustering[END_REF][START_REF] Goodfellow | Deep learning[END_REF], we develop a topology-based measure for comparing two manifolds: the true data manifold M data and the model manifold M model , by analysing samples P data ⊂ M data and Q model ⊂ M model .

Contribution. In this work, we make the following contributions:

1. We introduce a new tool: Cross-Barcode(P, Q). For a pair of point clouds P and Q, the Cross-Barcode(P, Q) records the differences in multiscale topology between two manifolds approximated by the point clouds; 2. We propose a new measure for comparing two data manifolds approximated by point clouds:

Manifold Topology Divergence (MTop-Div); 3. We apply the MTop-Div to evaluate performance of GANs in various domains: 2D images, 3D shapes, time-series. We show that the MTop-Div correlates well with domain-specific measures and can be used for model selection. Also it provides insights about evolution of generated data manifold during training; 4. We have compared the MTop-Div against 7 established evaluation methods: FID, discriminative score, MMD, JSD, 1-coverage, IMD and Geometry score and found that MTop-Div is able to capture subtle differences in data geometry; 5. We have essentially overcame the known TDA scalability issues and in particular have carried out the MTop-Div calculations on most recent datasets such as FFHQ, with dimensions D up to 10 7 .

The source code is available at https://github.com/IlyaTrofimov/MTopDiv.

Related work. GANs try to recover the true data distribution via a min-max game where two players, typically represented by deep neural networks, called Discriminator and Generator, compete by optimizing the common objective. Training curves are not informative since generator and discriminator counter each other, thus, the loss values are often meaningless. Several other measures were introduced to estimate the quality of GANs. However, there is no consensus which of them best captures strengths and limitations of the models and can be used for the fair model selection. The likelihood in Parzen window works poorly in high-dimensional spaces and does not correlate with the visual quality of generated samples [START_REF] Theis | A note on the evaluation of generative models[END_REF]. The Inception Score (IS) [START_REF] Salimans | Improved techniques for training gans[END_REF] measures both the discriminability and diversity of generated samples. It relies on the pre-trained Inception network and is limited to 2D images domain. While IS correlates with visual image quality, it ignores the true data distribution, doesn't detect mode dropping and is sensitive to image resolution. The Fréchet Inception Distance (FID) [START_REF] Heusel | Gans trained by a two time-scale update rule converge to a local nash equilibrium[END_REF] is a distance between two multivariate Gaussians. These Gaussians approximate the features of generated and true data extracted from the last hidden layer of the pretrained Inception network. Similar variant is KID [START_REF] Bińkowski | Demystifying mmd gans[END_REF] which computes MMD distance between two distributions of features. The work [START_REF] Grnarova | A domain agnostic measure for monitoring and evaluating gans[END_REF] proposed to use the Duality Gap, a notion from the game theory, as a domain agnostic measure for GAN evaluation. Another variant of a measure inspired by game theory was proposed in [START_REF] Olsson | Skill rating for generative models[END_REF]. Conventional precision and recall measures can be also used [START_REF] Lucic | Are gans created equal? A large-scale study[END_REF][START_REF] Kynkäänniemi | Improved precision and recall metric for assessing generative models[END_REF]. Accurate calculation of precision and recall is limited to simple datasets from low-dimensional manifolds. The Geometry Score (GScore) [START_REF] Khrulkov | Geometry score: A method for comparing generative adversarial networks[END_REF] is the L2-distance between mean Relative Living Times (RLT) of topological features calculated for the model distribution and the true data distribution. The GScore is domain agnostic, does not involve auxiliary pretrained networks and is not limited to 2D images. However, GScore is not sensitive even to some simple transformations -like constant shift, dilation, or reflection (see our Fig. 2,4). The barcodes in GScore are calculated approximately, based only on the approximate witness complexes on 64 landmark points sampled from each distribution. That's why the procedure is stochastic and should be repeated several thousand times for averaging. Thus, the calculation of GScore can be prohibitively long for large datasets. We also refer reader to the comprehensive survey [START_REF] Borji | Pros and cons of gan evaluation measures[END_REF].

2 Cross-Barcode and Manifold Topology Divergence

Multiscale simplicial approximation of manifolds

According to the well-known Manifold Hypothesis [START_REF] Goodfellow | Deep learning[END_REF] the support of the data distribution P data is often concentrated on a low-dimensional manifold M data . We construct a framework for comparing numerically such distribution P data with a similar distribution Q model concentrated on a manifold M model . Such distribution Q model is produced, for example, by a generative deep neural network in one of applications' scenarios. The immediate difficulty here is that the manifold M data is unknown and is described only through discrete sets of samples from the distribution P data . One standard approach to resolve this difficulty is to approximate the manifold M data by simplices with vertices given by the sampled points. The simplices approximating the manifold are picked based on proximity information given by the pairwise distances between sampled points [START_REF] Belkin | Laplacian eigenmaps and spectral techniques for embedding and clustering[END_REF][START_REF] Niyogi | Finding the homology of submanifolds with high confidence from random samples[END_REF]. The standard approach is to fix a threshold r > 0 and to take the simplices whose edges do not exceed the threshold r. The choice of threshold is essential here since if it is too small, then only the initial points, i.e., separated from each other 0-dimensional simplices, are allowed. And if the threshold is too large, then all possible simplices with sampled points as vertices are included and their union is simply the big blob representing the convex hull of the sampled points. Instead of trying to guess the right value of the threshold, the standard recent approach is to study all thresholds at once. This can be achieved thanks to the mathematical tool, called barcode [START_REF] Barannikov | Framed Morse complexes and its invariants[END_REF][START_REF] Chazal | An introduction to topological data analysis: fundamental and practical aspects for data scientists[END_REF], that quantifies the evolution of topological features over multiple scales. For each value of r the barcode describes the topology, namely the numbers of holes or voids of different dimensions, of the union of all simplices included up to the threshold r.

Measuring the differences in simplicial approximation of two manifolds

However, to estimate numerically the degree of similarity between the manifolds M model , M data ⊂ R D , it is important not just to know the numbers of topological features across different scales for simplicial complexes approximating M model , M data , but to be able to verify that the similar topological features are located at similar places and appear at similar scales. with Q-points(blue), and also P -points between them, are added for three thresholds: α = 0.2, 0.4, 0.6

Our method measures the differences in the simplicial approximation of the two manifolds, represented by samples P and Q, by constructing sets of simplices, describing discrepancies between the two manifolds. To construct these sets of simplices we take the edges connecting P -points with Q-points, and also P -points between them, ordered by their length, and start adding these edges one by one, beginning from the smallest edge and gradually increasing the threshold, see Figure 1. We add also the triangles and k-simplices at the threshold when all their edges have been added. It is assumed that all edges between Q-points were already in the initial set. We track in this process the births and the deaths of topological features, where the topological features are allowed here to have boundaries on any simplices formed by Q-points. The longer the lifespan of the topological feature across the change of threshold the bigger the described by this feature discrepancy between the two manifolds.

Homology is a tool that permits to single out topological features that are similar, and to decompose any topological feature into a sum of basic topological features. More specifically, in our case, a k-cycle is a collection of k-simplices formed by Pand Qpoints, such that their boundaries cancel each other, perhaps except for the boundary simplices formed only by Q-points. For example, a cycle of dimension k = 1 corresponds to a path connecting a pair of Q-points and consisting of edges passing through a set of P -points. A cycle which is a boundary of a set of (k + 1)-simplices is considered trivial. Two cycles are topologically equivalent if they differ by a boundary, and by collection of simplices formed only by Q-points. A union of cycles is again a cycle. Each cycle can be represented by a vector in the vector space where each simplex corresponds to a generator. In practice, the vector space over {0, 1} is used most often. The union of cycles corresponds to the sum of vectors. The homology vector space H k is defined as the factor of the vector space of all k-cycles modulo the vector space of boundaries and cycles consisting of simplices formed only by Q-points. A set of vectors forming a basis in this factor-space corresponds to a set of basic topological features, so that any other topological feature is equivalent to a sum of some basic features.

The homology are also defined for manifolds and for arbitrary topological spaces. This definition is technical and we have to omit it due to limited space, and to refer to e.g. [START_REF] Hatcher | Algebraic topology[END_REF][START_REF] Rojas Moraleda | Computational Topology for Biomedical Image and Data Analysis: Theory and Applications[END_REF] for details. The relevant properties for us are the following. For each topological space X the vector spaces H k (X), k = 0, 1 . . . , are defined. The dimension of the vector space H k equals to the number of independent k-dimensional topological features (holes,voids etc). An inclusion Y ⊂ X induces a natural map

H k (Y ) → H k (X)
In terms of homology, we would like to verify that not just the dimensions of homology groups H * (M model ) and H * (M data ) are the same but that more importantly the natural maps:

ϕ r : H * (M model ∩ M data ) → H * (M model ) (1) ϕ p : H * (M model ∩ M data ) → H * (M data ) (2)
induced by the embeddings are as close as possible to isomorphisms. The homology of a pair is the tool that measures how far such maps are from isomorphisms. Given a pair of topological spaces Y ⊂ X, the homology of a pair H * (X, Y ) counts the number of independent topological features in X that cannot be deformed to a topological feauture in Y plus independent topological features in Y that, after the embedding to X, become deformable to a point. An equivalent description, the homology of a pair H * (X, Y ) counts the number of independent topological features in the factor-space X/Y , where all points of Y are contracted to a single point. The important fact for us is that the map, induced by the embedding, H * (Y ) → H * (X) is an isomorphism if and only if the homology of the pair H * (X, Y ) are trivial. Moreover the embedding of simple simplicial complexes Y ⊂ X is an equivalence in homotopy category, if and only if the homology of the pair H * (X, Y ) are trivial [START_REF] Whitehead | Elements of homotopy theory[END_REF].

To define the counterpart of this construction for manifolds represented by point clouds, we employ the following strategy. Firstly, we replace the pair (M model ∩M data ) ⊂ M model by the equivalent pair M model ⊂ (M data ∪ M model ) with the same factor-space. Then, we represent (M data ∪ M model ) by the union of point clouds P ∪Q, where the point clouds P , Q are sampled from the distributions P data , Q model . Our principal claim here is that taking topologically the quotient of (M data ∪ M model ) by M model is equivalent in the framework of multiscale analysis of topological features to the following operation on the matrix m P ∪Q of pairwise distances of the cloud P ∪ Q: we set to zero all pairwise distances within the subcloud Q ⊂ (P ∪ Q).

Cross-Barcode (P,Q)

Let P = {p i }, Q = {q j }, p i , q j ∈ R D are two point clouds sampled from two distributions P, Q. To define Cross-Barcode(P, Q) we construct first the following filtered simplicial complex. Let (Γ P ∪Q , m (P ∪Q)/Q ) be the weighted graph with the distance-like weights on edges defined as the complete graph on the union of point clouds P ∪ Q with the distance matrix given by the pairwise distance in R D for the pairs of points (p i , p j ) or (p i , q j ) and with all pairwise distances within the cloud Q that we set to zero. Our filtered simplicial complex is the Vietoris-Rips complex of (Γ P ∪Q , m (P ∪Q)/Q ).

Recall that given such a graph Γ with matrix m of pairwise distances between vertices and a parameter α > 0, the Vietoris-Rips complex R α (Γ, m) is the abstract simplicial complex with simplices that correspond to the non-empty subsets of vertices of Γ whose pairwise distances are less than α as measured by m. Increasing parameter α adds more simplices and this gives a nested family of collections of simplices know as filtered simplicial complex. Recall that a simplicial complex is described by a set of vertices V = {v 1 , . . . , v N }, and a collection of k-simplices S, i.e.

(k + 1)-elements subsets of the set of vertices V , k ≥ 0. The set of simplices S should satisfy the condition that for each simplex s ∈ S all the (k -1)-simplices obtained by the deletion of a vertex from the subset of vertices of s belong also to S. The filtered simplicial complexes is the family of simplicial complexes S α with nested collections of simplices: for α 1 < α 2 all simplices of S α1 are also in S α2 .

At the initial moment, α = 0, the simplicial complex R α (Γ P ∪Q , m (P ∪Q)/Q ) has trivial homology H k for all k > 0 since it contains all simplices formed by Q-points. The dimension of the 0-th homology equals at α = 0 to the number of P -points, since no edge between them or between a P -point and a Q-point is added at the beginning. As we increase α, some cycles, holes or voids appear in our complex R α . Then, some combinations of these cycles disappear. The persistent homology principal theorem [START_REF] Barannikov | Framed Morse complexes and its invariants[END_REF][START_REF] Afra | Computing and comprehending topology: Persistence and hierarchical Morse complexes[END_REF] implies that it is possible to choose the set of generators in the homology of filtered complexes H k (R α ) across all the scales α such that each generator appears at its specific "birth" time and disappears at its specific "death" time. These sets of "birth" and "deaths" of topological features in R α are registered in Barcode of the filtered complex.

Definition. The Cross-Barcode i (P, Q) is the set of intervals recording the "births" and "deaths" times of i-dimensional topological features in the filtered simplicial complex R α (Γ P ∪Q , m (P ∪Q)/Q ).

Examples of Cross-Barcode i (P, Q) are shown on Fig. 2, 11, 14, 17, 18. Topological features with longer "lifespan" are considered essential. The topological features with "birth"="death" are trivial by definition and do not appear in Cross-Barcode * (P, Q).

Basic properties of Cross-Barcode

* (P,Q) Proposition 1.
Here is a list of basic properties of Cross-Barcode * (P, Q):

• if the two clouds coincide then Cross-Barcode * (P, P ) = ∅;

• for Q = ∅, Cross-Barcode * (P, ∅) = Barcode * (P ), the barcode of the single point cloud P itself;

• the norm of Cross-Barcode i (P, Q), i ≥ 0, is bounded from above by the Hausdorff distance

Cross-Barcode i (P, Q) B ≤ d H (P, Q). ( 3
)
The proof is given in appendix.

The Manifold Topology Divergence (MTop-Div)

The bound from eq.( 3) and the equality Cross-Barcode * (P, P ) = ∅ imply that the closeness of Cross-Barcode * (P, Q) to the empty set is a natural measure of discrepancy between P and Q . Each Cross-Barcode i (P, Q) is a list of intervals describing the persistent homology H i . To measure the closeness to the empty set, one can use segments' statistics: sum of lengths, sum of squared lengths, number of segments, the maximal length (H i max) or specific quantile. We assume that various characteristics of different H i could be useful in various cases, but the cross-barcodes for H 0 and H 1 can be calculated relatively fast.

Our MTop-Divergence(P, Q) is based on the sum of lengths of segments in Cross-Barcode 1 (P, Q), see section 2.6 for details. The sum of lengths of segments in Cross-Barcode 1 (P, Q) has an interesting interpretation via the Earth Mover's Distance. Namely, it is easy to prove (see Appendix B.2) that EM-Distance between the Relative Living Time histogram for Cross-Barcode 1 (P, Q) and the histogram of the empty barcode, multiplied by the parameter α max from the definition of RLT, see e.g. [START_REF] Khrulkov | Geometry score: A method for comparing generative adversarial networks[END_REF], coincides with the sum of lengths of segments in H 1 . This ensures the standard good stability properties of this quantity.

Our metrics can be applied in two settings: to a pair of distributions P data , Q model , in which case we denote our score MTop-Div(D,M), and to a pair of distributions Q model , P data , in which case our score is denoted MTop-Div(M,D). These two variants of the Cross-Barcode, and of the MTop-Divergence are related to the concepts of precision and recall. These two variants can be analyzed separately or combined together, e.g. averaged.

Algorithm

To calculate the score that evaluates the similitude between two distributions, we employ the following algorithm. First, we compute Cross-Barcode 1 (P, Q) on point clouds P, Q of sizes b P , b Q sampled from the two distributions P, Q. For this we calculate the matrices m P , m P,Q of pairwise distances within the cloud P and between clouds P and Q. Then the algorithm constructs the Vietoris-Rips filtered simplicial complex from the matrix m (P ∪Q)/Q which is the matrix of pairwise distances in P ∪ Q with the pairs of points from cloud Q block replaced by zeroes and with other blocks given by m P , m P,Q . Next step is to calculate the barcode of the constructed filtered simplicial complex. This step and the previous step constructing the filtered complex from the matrix m (P ∪Q)/Q can Algorithm 1 Cross-Barcode i (P, Q)

Input: m[P, P ], m[P, Q] : matrices of pairwise distances within point cloud P , and between point clouds P and Q Require: VR(M ): function computing filtered complex from pairwise distances matrix M Require: B(C, i): function computing persistence intervals of filtered complex

C in dimen- sion i b Q ← number of columns in matrix m[P, Q] m[Q, Q] ← zeroes(b Q , b Q ) M ← m[P, P ] m[P, Q] m[P, Q] m[Q, Q] Cross-Barcode i ← B(VR(M ), i) Return: list of intervals Cross-Barcode i (P, Q)
representing "births" and "deaths" of topological discrepancies

Algorithm 2 MTop-Divergence(P, Q), see section 2.6 for details, default suggested values:

b P = 1000, b Q = 10000, n = 100 Input: X P , X Q : N P × D, N Q × D arrays rep- resenting datasets for j = 1 to n do P j ← random choice(X P ,b P ) Q j ← random choice(X Q ,b Q ) B j
← list of intervals Cross-Barcode 1 (P j , Q j ) calculated by Algorithm1 mtd j ← sum of lengths of all intervals in B j end for MTop-Divergence(P, Q) ← mean(mtd) Return: number MTop-Divergence(P, Q) representing discrepancy between the distributions P, Q be done using one of the fast scripts1 , some of them are optimized for GPU acceleration, see e.g. [START_REF] Zhang | Gpu-accelerated computation of Vietoris-Rips persistence barcodes[END_REF][START_REF] Bauer | Ripser: efficient computation of Vietoris-Rips persistence barcodes[END_REF]. The calculation of barcode from the filtered complex is based on the persistence algorithm bringing the filtered complex to its "canonical form" ( [START_REF] Barannikov | Framed Morse complexes and its invariants[END_REF]). Next, sum of lengths or one of other numerical characteristcs of Cross-Barcode 1 (P, Q) is computed. Then this computation is run a sufficient number of times to obtain the mean value of the picked characteristic. In our experiments we have found that for common datasets the number of times from 10 to 100 is generally sufficient. Our method is summarized in the Algorithms 1 and 2.

Complexity. The Algorithm 1 requires computation of the two matrices of pairwise distances m[P, P ], m[P, Q] for a pair of samples

P ∈ R b P ×D , Q ∈ R b Q ×D involving O(b 2 P D) and O(b P b Q D) operations.
After that, the complexity of the computation of barcode does not depend on the dimension D of the data. Generally the persistence algorithm is at worst cubic in the number of simplices involved. In practice, the boundary matrix is sparse in our case and thanks also to the GPU optimization, the computation of cross-barcode takes similar time as in the previous step on datasets of big dimensionality. Since only the discrepancies in manifold topology are calculated, the results are quite robust and a relatively low number of iterations is needed to obtain accurate results. Since the algorithm scales linearly with D it can be applied to the most recent datasets with D up to 10 7 . For example, for D = 3.15 × 10 6 , and batch sizes b P = 10 3 , b Q = 10 4 , on NVIDIA TITAN RTX the time for GPU accelerated calculation of pairwise distances was 15 seconds, and GPU-accelerated calculation of Cross-Barcode 1 (P, Q) took 30 seconds.

Experiments

We examine the ability of MTop-Div to measure quality of generative models trained on various datasets. Firstly, we illustrate the behaviour of MTop-Div on simple synthetic datasets (rings (Fig. 2), disks (Fig. 13)). Secondly, we show that MTop-Div is superior to the GScore, another topology-based GAN quality measure. We carry out experiments with a mixture of Gaussians, MNIST, CIFAR10, X-ray images, FFHQ. The performance of MTop-Div is on par with FID. For images, MTop-Div is always calculated in pixel space without utilizing pre-trained networks. Thirdly, we apply MTop-Div to GANs from non-image domains: 3D shapes and time-series, where FID is not applicable. We show that MTop-Div agrees with domain-specific measures, such as JSD, MMD, Coverage, discriminative score, but MTop-Div better captures evolution of generated data manifold during training2 .

Simple synthetic datasets in 2D

As illustrated on Fig. 2 the GScore does not respond to shifts of the distributions' relative position. One of common tasks to assess GAN's performance is measuring it's ability to uncover the variety of modes in the data distribution. A common benchmark for this problem is a mixture of Gaussians, see Fig. 3. We trained two generators with very different performance: original GAN, which managed to capture all 5 modes of the distribution and WGAN-GP, which have only covered poorly two. However, the Geometry score is not sensitive to such a difference since two point clouds have the same RLT histogram. While the MTop-Div is sensitive to such a difference. Figure 4 shows an experiment with MNIST dataset. We compare two point clouds: "5"s vs. vertically flipped "5"s (resembling rather "2"s). These two clouds are indistinguishable for Geometry Score, while the MTop-Div is sensitive to such flip since it depends on the relative position of the two clouds. We evaluate the proposed MTop-Div(D,M) using a benchmark with the controllable disturbance level.

Synthetic modifications of CIFAR10

We take CIFAR10 and apply the following modifications. Firstly, we emulate common issues in GAN: mode drop, mode invention and mode collapse by doing class drop, class addition and intra-class collapse (removal of objects within a class). Secondly, we apply two disturbance types: erasure of a random rectangle and a Gaussian noise. As 'real' images we use the test set from CIFAR10, as 'generated' images -a subsample of the train set with applied modifications. The size of the later always equals the test set size. Figure 5 shows the results. Ideally, the quality measure should monotonically increase with the disturbance level. We conclude that Geometry Score is monotone only for 'mode invention' and 'intra-mode collapse' while MTop-Div(D,M) is almost monotone for all the cases. The average Kendall-tau rank correlation between MTop-Div(D,M) and disturbance level is 0.89, while for Geometry Score the rank correlation is only 0.36. FID performs well on this benchmark, not shown on Figure 5 for ease of perception. Additionally, we calculated MTop-Div for higher order Cross-Barcodes, see Appendix D. We trained WGAN and WGAN-GP on various datasets: CIFAR10, SVHN, MNIST, Fashion-MNIST and evaluated their quality, see Table 1. Experimental data show that the ranking between WGAN and WGAN-GP is consistent for FID and MTop-Div.

GAN model selection

Experiments with StyleGAN

We evaluated the performance of StyleGAN [START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF] and StyleGAN2 [START_REF] Karras | Analyzing and improving the image quality of StyleGAN[END_REF] generators trained on the FFHQ dataset 3 . We generated 20 × 10 3 samples with two truncation levels: ψ = 0.7, 1.0 and compared them with 20 × 10 3 samples from FFHQ. The truncation trick is known to improve average image quality but decrease image variance. Figure 7 show the results (see also Table 2 in Appendix for more data). Thus, the ranking via MTop-Div(M, D) is consistent with FID. We also tried to calculate Geometry Score but found that it takes a prohibitively long time.

Chest X-rays generation for COVID-19 detection

Waheen et al. [START_REF] Waheed | Covidgan: data augmentation using auxiliary classifier gan for improved covid-19 detection[END_REF] described how to apply GANs to improve COVID-19 detection from chest X-ray images. Following [START_REF] Waheed | Covidgan: data augmentation using auxiliary classifier gan for improved covid-19 detection[END_REF], we trained an ACGAN [START_REF] Odena | Conditional image synthesis with auxiliary classifier gans[END_REF] on a dataset consisting of chest X-rays of COVID-19 positive and healthy patients 4 . Next, we studied the training process of ACGAN. Every 10'th epoch we evaluated the performance of ACGAN by comparing real and generated COVID-19 positive chest X-ray images. That is, we calculated FID, MTop-Div(D,M) and a baseline measure - discriminative score 5 of a CNN trained to distinguish real vs. generated data. The MTop-Div agrees with FID and the discriminative score. PCA projections show that generated data approximates real data well. Figure 20 in Appendix presents real and generated images.

Additionally, we compared real COVID-positive and COVID-negative chest X-ray images, see horizontal dashed lines at Fig. 6. Counterintuitively, for FID real COVID-positive images are closer to real COVID-negative ones than to generated COVID-positive images; probably because FID is overly sensitive to textures. At the same time, evaluation by MTop-Div is consistent.

3D shapes generation

We use the proposed MTop-Div score to analyze the training process of the GAN applied to 3D shapes, represented by 3D point clouds [START_REF] Achlioptas | Learning representations and generative models for 3d point clouds[END_REF]. For training, we used 6778 objects of the "chair" class from ShapeNet [START_REF] Angel X Chang | Shapenet: An information-rich 3d model repository[END_REF]. We trained GAN for 1000 epochs and tracked the following standard quality measures: Minimum Matching Distance (MMD), Coverage, and Jensen-Shannon Divergence (JSD) between marginal distributions. To understand the training process in more details, we computed PCA decomposition of real and generated objects (Fig. 8, bottom). For computing PCA, each object (3D point cloud) was represented by a vector of point frequencies attached to the 3-dimensional 28 3 grid. Figure 8, top, shows that conventional metrics (MMD, JSD, Coverage) doesn't represent the training process adequately. While these measures steadily improve, the set of generated objects dramatically changes. At epoch 50, the set of generated objects (green) "explodes" and becomes much more diverse, covering a much larger space than real objects (red). Conventional quality measures (MMD, JSD, Coverage) ignore this shift while MTop-Div has a peak at this point. Next, we evaluated the final quality of GAN by training a classifier to distinguish real and generated object. A simple MLP with 3 hidden layers showed accuracy 98%, indicating that the GAN poorly approximates the manifold of real objects. This result is consistent with MTop-Div: at epoch 1000 it is even larger than at epoch 1.

Time series generation

Next, we analyze training dynamics of TimeGAN [START_REF] Yoon | Time-series generative adversarial networks[END_REF] tailored to multivariate time-series generation. We followed the experimental protocol from [START_REF] Yoon | Time-series generative adversarial networks[END_REF] and used the daily historical market stocks data from 2004 to 2019, including as features the volume, high, low, opening, closing, and adjusted closing prices. The baseline evaluation measure is calculated via a classifer (RNN) trained to distinguish real and generated time-series. Particularly, the discriminative score equals to the accuracy of such a classifer minus 0.5. Fig. 9, top, shows the results. We conclude that the behaviour of MTop-Div is consistent with the discriminative score: both of them decrease during training. To illustrate training in more details we did PCA projections of real and generated time-series by flattening the time dimension (Fig. 9, bottom). At 2000-th epoch, the point clouds of real (red) and generated (green) time-series became close, which is captured by a drop of MTop-Div score. At the same time, discriminative score is not sensitive enough to this phenomena.

Conclusions

We have proposed a tool, Cross-Barcode * (P, Q), which records multiscale topology discrepancies between two data manifolds approximated by point clouds. Based on the Cross-Barcode * (P, Q), we have introduced the Manifold Topology Divergence and have applied it to evaluate the performance of GANs in various domains: 2D images, 3D shapes, time-series. We have shown that the MTop-Div correlates well with domain-specific measures and can be used for model selection. Also, it provides insights about the evolution of generated data manifold during training and can be used for early stopping. The MTop-Div score is domain agnostic and does not rely on pre-trained networks. We have compared the MTop-Div against 7 established evaluation methods: FID, discriminative score, MMD, JSD, 1-coverage, IMD, and Geometry score and found that MTop-Div outperforms many of them and captures well subtle differences in data manifolds. Our methodology permits to overcome the known TDA scalability issues and to carry out the MTop-Div calculations on the most recent datasets such as FFHQ, with the size of up to 10 5 and the dimension of up to 10 7 .

A Simplicial Complexes, Cycles, Barcodes A.1 Background

The simplicial complex is a combinatorial data that can be thought of as a higher-dimensional generalization of a graph. Simplicial complex S is a collection of k-simplices, which are finite (k + 1)-elements subsets in a given set V , for k ≥ 0. The collection of simplices S must satisfy the condition that for each σ ∈ S, σ ⊂ σ implies σ ∈ S. A simplicial complex consisting only of 0and 1-simplices is a graph.

Let C k (S) denotes the vector space over a field F whose basis elements are k-simplices from S with a choice of ordering of vertices up to an even permutation. In calculations it is most convenient to put F = Z 2 . The boundary linear operator Filtration on simplicial complex is defined as a family of simplicial complexes S α with nested collections of simplices: for α 1 < α 2 all simplices of S α1 are also in S α2 . In practical examples the indexes α run through a discrete set α 1 < . . . < α max .

∂ k : C k (S) → C k-1 (S) is defined on σ = {x 0 , . . . , x k } as ∂ k σ = k j=0 (-1) j {x 0 , . . . , x j-1 , x j+1 , . . . , x k }.
The inclusions S α ⊆ S β induce naturally the maps on the homology groups

H k (S α ) → H k (S β ).
The evolution of the cycles through the nested family of simplicial complexes S α is described by the barcodes. The persistent homology principal theorem [START_REF] Barannikov | Framed Morse complexes and its invariants[END_REF][START_REF] Zomorodian | Computing persistent homology[END_REF][START_REF] Afra | Computing and comprehending topology: Persistence and hierarchical Morse complexes[END_REF] states that for each dimension there exists a choice of a set of basic topological features across all S α so that each feature appears in H k (S α ) at specific time α = b j and disappears at specific time α = d j . The H i barcode of the filtered simplicial complex is the record of these times represented as the collection of segments [b j , d j ]. The barcodes are defined and calculated through bringing the set of matrices of the boundary operators ∂ k to the "Canonical Form" by a change of the basis in C k preserving the nested family S α [START_REF] Barannikov | Framed Morse complexes and its invariants[END_REF][START_REF] Barannikov | Canonical Forms = Persistence Diagrams[END_REF].

Let (Γ, m) be a weighted graph with distance-like weights, where m is the symmetric matrix of the weights attached to the edges of the graph Γ. The Vietoris-Rips filtered simplicial complex of the weighted graph R α (Γ, m), is defined as the nested collection of simplices:

R α (Γ, m) = {{x 0 , . . . , x k }, x i ∈ Vert(Γ) m(x j , x l ) ≤ α}
where Vert(Γ) is the set of vertices of the graph Γ. Even though such weighted graphs do not always come from a set of points in metric space, barcodes of weighted graphs have been successfully applied in many situations (networks, fmri, medical data, graph's classification etc).

A.2 Simplices, describing discrepancies between the two manifolds

Here we gather more details on the construction of sets of simplicies that describe discrepancies between two point clouds P and Q sampled from the two distributions P, Q. As we have described in section 2, our basic methodology is to add consecutively the edges between P -points and Q-points and between pairs of P -points. All edges between Q-points are added simultaneously at the beginning at the threshold α = 0 . The P P and P Q edges are sorted by their length, and are added at the threshold α ≥ 0 corresponding to the length of the edge. This process is visualized in more details on Figures 10 and11. The triangles are added at the threshold at which the last of its three edges are added. The 3and higher k-simplices are added similarly at the threshold corresponding to the adding of the last of their edges. The added triangles and higher dimensional simplices are not shown explicitely on Figure 1 for ease of perception, as they can be restored from their edges. As all simplices within the Q-cloud are added at the very beginning at α = 0, the corresponding cycles formed by the Q-cloud simplices are immediately killed at α = 0 and do not contribute to the Cross-Barcode.

Figure 10: We are adding edges between P -points(orange) and Q-points(blue) and between pairs of P -points consecutively. The edges are sorted by their length, and are added at the threshold α ≥ 0 corresponding to the length of the edge. Here at the thresholds α = 0.2, 0.4, 0.6, 0.8 edges with length less than α were added. For ease of perception the simultaneously added triangles and higher simplices, as well as the added at α = 0 all simplices between Q-points, are not shown explicitly here. Notice how the 1-cycle, shown with green, with endpoints in Q-cloud is born at α = 0.4. It survives at α = 0.6 and it is killed at α = 0.8.

The constructed set of simplices is naturally a simplicial complex, since for any added k-simplex, we have added also all its (k -1)-faces obtained by deletion of one of vertices. The threshold α defines the filtration on the obtained simplicial complex, since the simplices added at smaller threshold α 1 are contained in the set of simplices added at any bigger threshold α 2 > α 1 .

With adding more edges, the cycles start to appear. In our case, a cycle is essentially a collection of simplices whose boundary is allowed to be nonzero if the boundary consists of simplices with from Q. For example, a 1cycle in our case is a path consisting of added edges, that can start and end in Q-cloud and that passes through P -points. This is because any such collection can be completed to a collection with zero boundary since any cycle from Q-cloud is a boundary of a sum of added at α = 0 simplices from Q.

A 1-cycle disappears at the threshold when a set of triangles is added whose boundary coincides with the 1-cycle plus perhaps some edges between Q-points.

Notice how the 1-cycle with endpoints in Q-cloud is born at α = 0.4 on Figure 10, shown with green. It survives at α = 0.6 and it is killed at α = 0.8. The process of adding longer edges can be visually assimilated to the building of a "spider's web" that tries to bring the cloud of red points closer to the cloud of blue points. The obstructions to this are quantified by "lifespans" of cycles, they correspond to the lengths of segments in the barcode. See e.g., Figure 11 where a 1-cycle is born between α = 0.5 and 0.9, it then corresponds to the green segment in the Cross-Barcode.

Figure 11: The process of adding the simplices between the P -cloud(red) and Q-cloud(blue) and within the P -cloud. Here we show the consecutive adding of edges together with simultaneous adding of triangles. All the edges and simplices within Q-cloud are assumed added at α = 0 and are not shown here for perception's ease. Notice the 1-cycle born between α = 0.5 and 0.9, it corresponds to the green segment in the shown Cross-Barcode Remark A.1. To characterise the situation of two data point clouds one of which is a subcloud of the other S ⊂ C, it can be tempting to start seeking a "relative homology" analog of the standard (single) point cloud persistent homology. The reader should be warned that the common in the literature "relative persistent homology" concept and its variants, i.e. the persistent homology of the decreasing sequence of factor-complexes of a fixed complex: K → . . . K/K i → K/K i+1 → . . . K/K, is irrelevant in the present context. In contrast, our methodology, in particular, does not involve factorcomplexes construction, which is generally computationally prohibitive. The point is that the basic concept of filtered complex contains naturally its own relative analogue via the appropriate use of various filtrations.

A.3 Sub-manifolds and bars in Cross-Barcode * (P, Q)

It is natural to start analyzing the closeness of the data point cloud P to the data point cloud Q by looking at the matrix of the P Q pairwise distances. If there are many points from P such that their distance to their closest point from Q is relatively big then the clouds P and Q are not close. However, in applications, it is important to distinguish the different situations here. The first case is when all these remote from Q points are close to each other. Then this remote from Q cluster of P points represents a single topological feature distinguishing cloud P from Q. Another case is when the remote from Q points form several clusters so that each such remote from Q cluster represents a separate topological feature. The long bars in the zero-dimensional Cross-Barcode record the lifespans on the distances' scale of these remote from Q clusters of P -points.

In practice it also happens more often that it is not possible to distinguish a separate cluster of P points which are all remote from Q. Rather, there are some P -points inside the same P -cluster that are close to Q and other P -points from the same P -cluster which are further away from Q, as on Fig. 1. This situation is captured and quantified by the higher dimensional topological features distinguishing cloud P from Q. Intuitively such an i-dimensional topological feature represents an i-dimensional P -cloud's sub-manifold whose boundary is close to the Q-cloud, but whose interior P -points are remote from the Q-cloud, like the green polygonal chain on Fig. 10 at α = 0.4. Such features are constructed in the algorithm using the distance matrix combinatorics from (i + 1)-tuples of P -points or P & Qpoints. The distances within each of these tuples are less or equal to the feature's appearance, or birth, threshold. The disappearance, or death, of such a feature calculated by the algorithm corresponds approximately to the scale at which the feature becomes indistinguishable from the Q-cloud. The i-dimensional Cross-Barcode i (P, Q), i ≥ 1, is the set of segments (bars) recording the birth and the death thresholds of such topological features.

A.4 Cross-Barcode * (P, Q) as obstructions to assigning P points to distribution Q Geometrically, the lowest dimensional Cross-Barcode 0 (P, Q) is the record of relative hierarchical clustering of the following form. For a given threshold r, let us consider all points of the point cloud Q plus the points of the cloud P lying at a distance less than r from a point of Q as belonging to the single Q-cluster. It is natural to form simultaneously other clusters based on the threshold r, with the rule that if the distance between two points of P is less than threshold r then they belong to the same cluster. When the threshold r is increased, two or more clusters can collide. And the threshold, at which this happens, corresponds precisely to the "death" time of one or more of the colliding clusters. At the end, for very large r only the unique Q-cluster survives. Then Cross-Barcode 0 (P, Q) records the survival times for this relative clustering. Notice that in situations, like, for example, in Figure 12, it is difficult to attribute confidently certain points of P to the same distribution as the point cloud Q even when they belong to the "big" Q-cluster at a small threshold r, because of the nontrivial topology. Such "membranes" of P -points in void space, are obstacles for assigning points from P to distribution Q. These obstacles are quantified by the segments from the higher barcodes Cross-Barcode ≥1 (P, Q). The bigger the length of the associated segment in the barcode, the further the membrane passes away from Q.

A.5 More simple synthetic datasets in 2D Figure 13: The first picture shows two clouds of 1000 points sampled from the uniform distributions on two different disks of radius 1 with a distance between the centers of the disks of 0.5. The second and third pictures show the dependence of the GScore metric (the GScore is equal to zero independently of the distance between the disks), maximum of segments in H0 and the sum of lengths of segments H1 as a function of the distance between the centers of the disks averaged over 10 runs. The length of the maximum segment barcode in H0 grows linearly and equals to the distance between the pair of closest points in the two distributions.

As illustrated on Figures 2,13 the GScore is unresponsive to changes of the distributions' positions. The Cross-Barcode captures well the precision vs. recall aspects of the point cloud's approximations, contrary to FID, which is known to mix the two aspects. For example, in the case of mode-dropping, bad recall but good precision, the Cross-Barcode 0 (P data , Q model ) contains the long intervals, one for each dropped mode, which measure the distance from the data's dropped mode to the closest generated mode. The mode-dropping case (bad recall, good precision) is illustrated on Figure 14.

A.6 Cross-Barcode and precision-recall

Analogously, in the case of mode-invention, with good recall but bad precision, the Cross-Barcode 0 (Q model , P data ) contains long intervals, one for each invented mode, which measure the distance from the model's invented mode to the closest data's mode.

The mode-invention (good recall, bad precision) case is illustrated on Figure 15. The Bottleneck distance [START_REF] Efrat | Geometry helps in bottleneck matching and related problems[END_REF], also known as Wasserstein-∞ distance W ∞ , defines the natural norm on the Cross-Barcodes:

Cross-Barcode i (P, Q) B = max [bj ,dj ]∈Cross-Barcodei (d j -b j ).
The Hausdorf distance measures how far are two subsets P, Q of a metric space from each other. The Hausdorf distance is the greatest of all the distances from a point in one set to the closest point in the other set:

Proof. Let c ∈ R α0 (Γ P ∪Q , m (P ∪Q)/Q ) be an i-dimensional cycle appearing in the filtered complex at α = α 0 . Let us construct a simplicial chain that kills c. Let σ = {x 1 , . . . , x i+1 } be one of the simplices from c. Let q j denote the closest point in Q to the vertex x j . The prism {x 1 , q 1 , . . . , x i+1 , q i+1 } can be decomposed into (i + 1) simplices p k (σ) = {x 1 , x 2 , . . . , x k-1 , q k , . . . , q i+1 }, 1 ≤ k ≤ i + 1.

The boundary of the prism consists of the two simplices σ, q(σ) = {q 1 , . . . , q i+1 }, and of the (i + 1) similar prisms corresponding to the the boundary simplices of σ.

If c = n a n σ n then c = ∂( n a n k p k (σ n )) + n a n q(σ n ) For any k, j, d(x j , x k ) ≤ α 0 since c is born at α 0 . Therefore d(x j , q k ) ≤ d(x j , x k ) + d(x k , q k ) ≤ α 0 + sup x∈P d(x, Q).
Therefore all simplices p k (σ n )) appear no later than at (α 0 +sup x∈P d(x, Q)) in the filtered complex. All vertices of the simplices q(σ n ) are from Q. It follows that the lifespan of the cycle c is no bigger than sup x∈P d(x, Q))

To illustrate the proposition 1 we have verified empirically the diminishing of Cross-Barcode * (Q 1 , Q 2 ) when number of points in Q 1 , Q 2 goes to +∞ and Q 1 , Q 2 are sampled from the same uniform distribution on the 2D disk of radius 1. The maximal length of segments in H1 as function of number of points in the clouds of the same size is shown in Figure 16. Let's use all the distinct b i , d i to split [0, α max ] to disjoint segments s j :

[0, α max ] = j s j . Each s j is included in K(j) segments [b i , d i ] from the Cross-Barcode i (P, Q). Thus, RLT (k) = 1 α max j:K(j)=k |s j |.
At the same time:

MTop-Div(P, Q) = i (d i -b i ) = j K(j)|s j | = +∞ k=1 j:K(j)=k K(j)|s j | = +∞ k=1 α max × k × RLT (k) = α max EMD(RLT (k), O 0 ).
C Hyperparameters, Software used, and Experiments' Details

We have made experiments in various settings and on the following datasets:

• on a set of gaussians in 2D in comparison with distributions generated by GAN and WGAN.

• MNIST We have observed that GScore is not sensitive to the flip of the cloud of "fives", while our score MTop-Divergence is sensitive to such flip since it depends on the positions of clouds with respect to each other • CIFAR10 We have evaluated our MTop-Div(D,M) using a benchmark with the controllable disturbance level. We have observed that Geometry Score is monotone only for 'mode invention' and 'intra-mode collapse' while MTop-Div(D,M) is almost monotone for all the cases. • FFHQ We have evaluated the quality of distributions generated by StyleGAN and StyleGAN-2, without truncation and with ψ = 0.7 truncation. We observed that the ranking via MTop-Div is consistent with FID • ShapeNet 6 We have studied the training dynamics of the GAN trained on 3D shapes. We observed that MTop-Div is consistent with domain specific measures (JSD, MMD, Coverage) and that MTop-Div better describes the evolution of the point cloud of generated objects during epochs; • Stock data We have studied the training dynamics of TimeGAN 7 applied to market stock data. We observed that MTop-Div is consistent with the discriminative score but better captures the evolution of point cloud of generated objects during epochs; • Chest X-ray images We have studied the training dynamics of ACGAN applied to chest X-ray images. We observed that MTop-Div is more consistent with the discriminative score than FID;

For computation of FID we used Pytorch-FID 8 . For computation of Geometry Score we used the original code 9 patched to supported multi-threading, otherwise it was extremely slow. The RLTs computation was averaged over 2500 trials. We calculated persistent homology via ripser++ 10 .

We used the following hyperparameters to compute MTop-Div: MTop-Div scores were were averaged over 20 runs.

We compared Geometry Score and MTop-Div in the experiment with mixtures of Gaussians. Table 4 shows the results. We conclude that the MTop-Div is consistent with the visual quality of GAN's output while Geometry Score fails.

Figure 17 shows Cross-Barcodes for the experiment with StyleGAN's trained on FFHQ. Figure 19 shows one of Cross-Barcodes in H 0 from the experiment with CIFAR10 dataset to illustrate that the 0-dimensional Cross-Barcode can also be applied. Figure 18 shows the Cross-Barcodes in H 1 from the experiments with GAN 11 and WGAN-GP 12 trained on mixture of Gaussians.

Table 3 shows extended experimental results on GAN model selection including standard error of sample means of MTop-Div. As proposed by a reviewer, we did additional experiments with IMD [START_REF] Tsitsulin | The shape of data: Intrinsic distance for data distributions[END_REF] applied to point clouds from our experiments. IMD is not sensitive to the rings shift (Section 3.1) and the digits flipping on MNIST (Section 3.3). For the experiment "Mode dropping on Gaussians" (Section 3.2), IMD incorrectly ranks poorly performing WGAN-GP (see Fig. 3) higher than the original GAN (Table 4).

For the experiments "GAN model selection" (Section 3.5), IMD ranks a better performing model lower in one case, while the ranking via MTop-Div is consistent with true GAN performance. For the "Synthetic variations of CIFAR10" experiment (Section 3.4), the average Kendall-tau correlation between IMD score and the disturbance level is 0.55, which is lower than the same measure of MTop-Div (0.89).

Figure 1 :

 1 Figure 1: Edges(red) connecting P -points(red) with Q-points(blue), and also P -points between them, are added for three thresholds: α = 0.2, 0.4, 0.6

Figure 2 :

 2 Figure 2: MTop-Div and H0 max compared with GScore, for two ring clouds of 1000 points, as function of d =distance between ring centers, the Cross-Barcode 1 (P, Q) is shown at d = 0.5

Figure 3 :

 3 Figure 3: Difficulty of the Geometry Score to detect the mode dropping.

3. 3 Figure 4 :

 34 Figure 4: Two point clouds:"5"s from MNIST vs. vertically flipped "5"s from MNIST (resembling rather "2"s). The two clouds are indistinguishable for Geometry Score, while the MTop-Div is sensitive to such flip as it depends on the positions of clouds with respect to each other.

Figure 5 :

 5 Figure 5: Experiment with modifications of CIFAR10. The disturbance level rises from zero to a maximum. Ideally, the quality score should monotonically increase with the disturbance level.

Figure 6 :

 6 Figure 6: Training process of GAN applied to chest X-ray data. Top: normalized quality measures FID, MTop-Div, Disc. score vs. epoch. Each measure is divided by it's maximum value: max. FID = 304, max MTop-Div = 21, max. Disc.score = 0.5. Lower is better. Dashed horizontal lines depict comparison of real COVID-positive and COVID-neg. chest X-rays. Bottom: PCA projections of real objects (red) and generated objects (green).

Figure 7 :

 7 Figure 7: Comparison of the quality measures, FID vs MTop-Div, on StyleGAN, StyleGAN2 trained on FFHQ with different truncation levels. MTop-Div(M,D) is monotonically increasing in good correlation with FID.

Figure 8 :

 8 Figure 8: Training process of GAN applied to 3D shapes. Top, middle: quality measures MMD, JSD, 1-Coverage, MTop-Div vs. epoch. Each quality measure is normalized, that is, divided by its value at the first epoch. Lower is better. Bottom: PCA projection of real objects (red) and generated objects (green). Vertical red line (epoch 50) depicts the moment, when the manifold of generated objects "explodes" and becomes much more diverse.

Figure 9 :

 9 Figure 9: Training dynamics of TimeGAN applied to market stock data. Top: discriminative score vs. epoch, MTop-Div vs. epoch. Lower is better. Bottom: PCA projection of real timeseries (red) and generated time-series (green). Vertical red line (epoch 2000) depicts the moment when manifolds of real and generated objects become close.

  The k-th homology group H k (S) is defined as the vector space ker ∂ k / im ∂ k+1 . The elements c ∈ ker ∂ k are called cycles. The elements c ∈ im ∂ k+1 are called boundaries. The general elements c ∈ C k (S) are called chains. The elements of H k (S) represent various k-dimensional topological features in S. A basis in H k (S) corresponds to a set of basic topological features.

Figure 12 :

 12 Figure 12: Paths/membranes (red) in the void that are formed by small intersecting disks around P points (orange), and are ending on Q (blue), are obstacles for identification of the distribution P with Q. These obstacles are quantified by Cross-Barcode 1 (P, Q). Separate clusters are the obstacles quantified by Cross-Barcode 0 (P, Q).

Figure 14 :

 14 Figure 14: Mode-dropping, bad recall & good precision, illustrated with clouds P data (red) and Q model (blue). The Cross-Barcode 0 (P data , Q model ) contains long intervals, one for each dropped mode, which measure the distance from the data's dropped mode to the closest generated mode.

Figure 15 :

 15 Figure 15: Mode-invention, good recall & bad precision, illustrated with clouds P data (red) and Q model (blue). The Cross-Barcode 0 (Q model , P data ) contains long intervals, one for each invented mode, which measure the distance from the model's invented mode to the closest data's mode.

d 4 ) 1 .

 41 H (P, Q) = max sup x∈P d(x, Q), sup y∈Q d(y, P ) . (Proposition The norm of Cross-Barcode i (P, Q), i ≥ 0, is bounded from above by the Hausdorff distance Cross-Barcode i (P, Q) B ≤ d H (P, Q).

Figure 16 :

 16 Figure 16: Diminishing of the length of maximal segment in H 1 with increase of the number of sampled points for point clouds of the equal size n sampled from the same uniform distribution on the 2D disk of radius 1.

•

  MNIST: b P = 10 2 , b Q = 10 3 ; • Gaussians: b P = 10 2 , b Q = 10 3 ; • CIFAR10: b P = 10 3 , b Q = 10 4 ; • FFHQ: b P = 10 3 , b Q = 10 4 . • ShapeNet: b P = 10 2 , b Q = 10 3 ; • Market stock data: b P = 10 2 , b Q = 10 3 ; • Chest X-ray data: b P = 10 2 , b Q = 10 3 .

Figure 20 presents

 20 Figure 20 presents real and generated chest X-ray images. The generated images are of high visual quality and resembles real images.

Figure 21 shows

 21 Figure21shows real and generated 3D shapes. Generated 3D shapes (bottom row) are relatively blurry.

Figure 17 :Figure 18 :

 1718 Figure 17: Cross-Barcodes for GAN's trained on FFHQ. Cross-Barcodes are shown by decrease of generator performance. For clarity, only H 1 barcodes are shown. The number and the total length of segments give the same ranking as the FID score

Figure 19 :

 19 Figure19: Cross-Barcodes for CIFAR10 vs. disturbed CIFAR10. Gaussian noise with 2 levels of variance was applied. For clarity, only H 0 barcodes are shown. For ease of perception of differences in Cross-Barcodes 0 they are shown on the same plot. The dataset with higher level of noise is distinguished here by the longer segments in the Cross-Barcode 0

Figure 20 :

 20 Figure 20: Top: real chest X-ray images. Bottom: generated chest X-ray images.

Figure 21 :

 21 Figure 21: Top: real 3D shapes. Bottom: generated 3D shapes. Generated 3D shapes are relatively blurry.

Table 1 :

 1 MTop-Div is consistent with FID for model selection of GAN's trained on various datasets.

	Dataset	FID		MTop-Div(D,M)
		WGAN WGAN-GP WGAN WGAN-GP
	CIFAR10	154.6	399.2	353.1	1637.4
	SVHN	101.6	154.7	332.0	963.2
	MNIST	31.8	22.0	2042.8	1526.1
	FashionMNIST	52.9	35.1	919.6	660.4

Table 2 :

 2 Performance measures of StyleGANs trained on FFHQ.

	GAN	ψ	FID	MTOP-DIV(M,D)
	STYLEGAN2 1.0	4.75	162.08
	STYLEGAN	1.0	8.25	234.33
	STYLEGAN	0.7 15.86	712.57
	STYLEGAN2 0.7 19.75	1011.53

Table 3 :

 3 MTop-Div is consistent with FID for model selection of GAN's trained on various datasets.

	Dataset	FID		MTop-Div(M,D)
		WGAN WGAN-GP	WGAN	WGAN-GP
	CIFAR10	154.6	399.2	370.5±17.3 2408.5±27.0
	SVHN	101.6	154.7	332.0±12.4 963.2±22.62
	MNIST	31.8	22.0	2365.6±40.1 1474.2±29.7
	FashionMNIST	52.9	35.1	1052.6±24.8 872.9±21.8

E Comparison with the "Intrinsic Multi-scale Distance(IMD)"

Table 4 :

 4 MTop-Div and G. Score for GAN's trained of mixtures of Gaussians.

	WGAN-GP	1.083	0.562	0.206	2.65
	ORIG. GAN	1.087	0.081	0.149	13.87

GAN

G. SCORE MTOP-DIV(M,D) MTOP-DIV(D,M) IMD

Persistent Homology Computation (wiki)

we additionally calculated IMD[START_REF] Tsitsulin | The shape of data: Intrinsic distance for data distributions[END_REF] for the pairs of point clouds from our experiments, see Appendix E.

https://github.com/NVlabs/ffhq-dataset (CC-BY 2.0 License)

We used the ACGAN implementation https://github.com/clvrai/ACGAN-PyTorch, (MIT License) and chest X-ray data was from https://www.kaggle.com/tawsifurrahman/ covid19-radiography-database (Kaggle Data License)

Discriminative score equals accuracy minus 0.5. MTop-Div better correlates with the discriminative score than FID: 0.75 vs. 0.66.

Acknowledgements

The problem statement was developed in the framework of Skoltech-MIT NGP program. The work of Serguei Barannikov and Evgeny Burnaev was supported by Ministry of Science and Higher Education grant No. 075-10-2021-068. Authors are thankful to Alexei Artemov for help with the 3D shapes experiment.


P Godignon

J Montserrat

J Rebollo

D Planson

Edge terminations for 4H-SiC power devices: a critical issue

Keywords: edge termination, JTE, breakdown voltage, OBIC

Edge termination is a critical part of a power devices. Numerous edge termination types have been developed for silicon devices. Implementation of these termination architectures are not straightforward in SiC due to physical and processing specificities: lower junction depths, higher electric field, trench depth and shaping limitations, etc. Two main families of terminations are currently used in commercial devices, pure Field Guard Rings, and JTE + Rings combination. The increasing number of trench commercial devices requires new approaches based on etched rings filled with dielectrics or polysilicon. For epitaxied bipolar devices, MESA with bevel angle termination combined with JTE based architecture are also suitable. In any case, and especially regarding avalanche capability requirements, not only the termination architecture is relevant, but also the passivation type, the channel stopper design, the 3D design. As modelling using conventional tools is not fully reliable, specific characterization methods are needed. For instance, micro-OBIC can be very effective to determine the electric field distribution in the periphery of the power devices.

Introduction

Most of planar and trench power devices are based of one or several n-type/p-type junctions. When the active part of the junction ends on the lateral side of the device, a very high electric field appear under reverse voltage biasing, due to junction curvature or trench corners. Then, a periphery protection, called edge termination is necessary. Reliable and robust high voltage devices need effective edge termination structures to protect the device periphery, so that blocking values close to the ideal 1D avalanche voltage value can be achieved. Many types of terminations have been developed in Silicon technology, the main ones being MESA termination for rectifiers and thyristors, field plates for low voltage range, field guard rings (FGR) and junction termination extension (JTE) for planar rectifiers and transistors. Silicon Carbide is not as friendly as Silicon in terms of processing technology for power device fabrication. Several physical limitations make SiC edge terminations more complex to design in a same way to Silicon. The main limitation is the low diffusion coefficients of dopant atoms inside the semiconductor. Deep junction (> 1μm) are difficult to implement and lateral dopants diffusion is very limited. On the other hand, one of the theoretical advantages of SiC versus Si, its high critical electric field, is also a drawback when looking for compatible dielectric layers capable of operating under such high fields in a reliable way. It clearly limits the use of field plates for instance. Finally, the higher interface traps level observed in the SiC/SiO2 interface compared to Si is also affecting the efficiency of the standard Si terminations architectures in the SiC power devices. Charges accumulate at the interface, generating breakdown voltage instabilities, especially with temperature. Finally, the termination design is also relevant to reach high avalanche ruggedness in order to comply with applications prone to unclamped inductive switching. In this case, the edge termination must reach breakdown at higher voltage than the main active junction [1].

Edge termination structures

Termination optimization is relevant in power device design since it may affect the reliability of the component, but also because it may consume a large area of the die, especially in the case of very high voltage devices (Fig. 1). This is particularly relevant for SiC semiconductor, considering the cost of the starting material and the defects density, which usually limit the die area of the components. In addition, in some cases, edge termination integration increases the fabrication process complexity (increasing the number of photolithography mask levels), like in trench JFETs, pure Schottky diodes, BJTs or thyristors. These aspects have a direct impact on cost and reliability. Usual terminations architectures. The more common periphery protection used in SiC commercial devices is the field guard ring (FGR) termination (Fig. 2a). It is formed with P + -type implantation wells, having a width of few microns. The key design parameters are the rings distances, which usually increase with the ring position, and the first ring distance with the device main junction. This termination usually consumes a large area and require very precise photolithography and mask etching resolution. The optimal first ring distance depends on the epilayer doping and must be lower than 1μm for devices voltage ranges below 1.7kV. Rings termination also depends on the surface charges at the SiC/dielectric interface. In this sense, buried rings configuration have been proposed in order to shift the high electric field peaks from the surface to the SiC bulk [START_REF] Mihaila | th Int. Symp. on Power Semiconductor Devices and ICs (ISPSD06)[END_REF], limiting interface parasitic effects. This is efficient but more complex to implement, since epilayer regrowth or very high energy implantation are required.

The second common type of edge protection is the Junction Termination Extension (JTE), which enable smaller area, but which typically requires an extra Al ion implantation step, with an accurate control of the implanted dose. JTE is also sensible to surface charges [START_REF] Matsushima | th Int. Symp. on Power Semiconductor Devices and ICs (ISPSD17)[END_REF]. Then, single implantation JTE is not used anymore, as multiple JTE combination, having decreasing implantation dose, have shown to be more efficient in terms of process control. SiC etching process can be also use to mitigate the JTE dose (see Fig. 2b) and create a double or triple JTE structure without the need for 3 Al implantation steps [START_REF] Dai | 33 th Int. Symp. on Power Semiconductor Devices and ICs (ISPSD21)[END_REF]. Even if the number of photolithographic masks is not reduced, the global technological process results simpler and cheaper. However, a combination of JTE + FGR, or JTE + FGR + JTE outer rings (see Fig. 3) is the most stable, reproducible and efficient combination for most of the SiC devices structures, especially for blocking voltages higher than 1.7kV. For instance, this architecture has been used in [START_REF] Kaji | [END_REF] for 27 kV diodes demonstration. Other combination of JTE and rings have been proposed in order to reduce termination length [START_REF] Mihaila | th Int. Symp. on Power Semiconductor Devices and ICs (ISPSD16)[END_REF]. In a previous paper [START_REF] Soler | [END_REF], we analyzed different field guard rings, JTE and JTE + rings combinations for 1.7kV and 4.5kV planar MOSFETs fabrication. Main results are summarized in Table 1. We intended to use the MOSFET P-well doping to form deep FGR termination (Fig. 1). The efficiency (calculated using simulated ideal 1D breakdown) of these FGR terminations (designs D3 to D8) is lower than the JTE + Rings one (designs D9 to D11). We believe this difference is due to the surface charges type present in our devices. We also tried to use the P-well doping profile as the 1 st JTE in a double JTEs configuration (design D2), without clear success. The lateral extension of the P-well in the periphery has basically no impact on the termination efficiency. Other 2D and 3D parameters must be defined when designing a JTE (+ rings) termination. JTE length or corner radius must be optimized to minimize the consumed area. In table 2 are reported breakdown voltages of 1.7kV JBS diodes with the termination architecture shown in Fig. 3a. When comparing design D5 to the reference, we can see that a reduction of the corner radius decreases the breakdown capability. The corner radius will also have a strong impact on the device avalanche ruggedness. On the other hand, the recent availability of SiC trench devices also required adaptation of edge termination. In this sense, 3D edge termination like trench guard rings terminations have been recently proposed for SiC devices [8]. This termination configuration has shown to be efficient in Silicon technologies. It allows to reduce the consumed area, and is compatible with the trench fabrication process. Additionally, the bottom of the trench can be implanted with self-aligned buried P + ring (see Fig 4a), allowing a reduction of the electric field at the SiC surface. In Fig. 4b, we can see another alternative using a wide trench filled with Benzocyclobutene (BCB) in an as JTE configuration [9][START_REF] Liu | 31 st Int. Symp. on Power Semiconductor Devices and ICs (ISPSD19)[END_REF]. The critical part of this termination is the trench side wall charges displacement and the possible parasitic channel that can be created with the electric field peaks. The field plate geometry on top of the termination structure is also a key parameter. For high voltage components, especially for bipolar SiC device, positive and negative beveled MESA terminations can also be used. These terminations, formerly used in thyristor and early power diodes components presents the advantage of lower dimensions than planar terminations [START_REF] Sung | [END_REF]. The main issue, especially for the negative bevel, is the SiC etching process necessary to reach low aspect ratio angles. However, solution playing with the etching mask topology (photoresist) or using multiple etching steps are possible, even if not fully compatible with mass production requirements. Other termination elements. One element of the termination never mentioned in the literature is the channel stopper (see Fig. 5a). Nevertheless, it is part of the termination design, as the distance between the end of the JTE or the rings and this channel stopper may affect the breakdown capability, the dynamic behavior and directly contribute to the total area required for the periphery protection of the power devices. To determine the optimal position of the channel stopper by modelling is tricky as it depends on a wide range of parameters such as the interface traps and fixed charge of the different passivation layers, the type of packaging technique used, the position of the top metal electrode, etc. We studied the optimal position of the channel stopper in 1.7 kV JBS diodes passivated with 5µm of polyamide. Diodes with a distance of 100µm between the end of the JTE and the channel stopper were able to withstand more than 2 kV while the diodes with a JTE-CS distance of 60µm exhibit a premature breakdown around 1.6 kV after arcing between the anode and channel stopper metals. A picture of a failing diode at 1.6 kV is shown in Fig. 5b, where the damaged area is clearly seen. Last but not least, the passivation of the surface in the termination area is a critical aspect for the wide band gap semiconductor high voltage devices. When no passivation is used, premature breakdown due to arcing in air can occur. However, the addition of a passivation layer (for instance polyamide) may change the surface charges state, and impact the breakdown voltage. This is particularly visible in JTE terminations, as the surface charges directly impact the optimal value of JTE implantation dose. Again, a combination of JTE and rings is the most suitable solution to mitigate these charges issues. For high voltage devices (> 2.5 kV), additional effects due to ambient factors like humidity are enhanced and may seriously affect the reliability of power modules if not properly passivated [START_REF] Hoffman | th Int. Symp. on Power Semiconductor Devices and ICs (ISPSD18)[END_REF]. In this case, solutions pass through a proper packaging technology, for instance using organic passivant like GlobTop. Stability and reliability evaluation of the termination can be done using High Temperature Reverse Bias test (HTRB) [START_REF] Hoffman | th Int. Symp. on Power Semiconductor Devices and ICs (ISPSD18)[END_REF], including Humidity (H3TRB) [START_REF] Jormanainen | [END_REF][14], if a proper evaluation of the passivation is to be done. It is also relevant to test the termination under Unclamped Inductive Switching (UIS) to check the avalanche capability of the device [1].

Reference

Termination optimization through advanced characterization:

A good understanding of termination behavior also requires advanced characterization tools such as lock-in thermography, photoluminescence, Electron Beam Induced Current (EBIC). Optical Beam Induced Current technique (OBIC) applied to WBG semiconductor devices have shown to be useful to study the efficiency of different edge protection techniques [15]. In order to illustrate some OBIC experiments on different terminations, PiN diodes with several JTE protections (1.7 kV-class) has been measured with increasing reverse voltage up to 400V. Since OBIC setup is under air, as described in [START_REF] Planson | this conference (We-2B-05)[END_REF], it limits the maximum voltage bias for unpassivated devices. Scans were performed from the center of the diode towards the outer periphery. For each results of Fig 6, one can find the drawing of the edge termination at the bottom. Diodes (#1-#3) exhibit increasing length of JTE. Diode #4 has a JTE length of 35 µm + 3 JTE rings. As it can be seen, the OBIC signal appears at least on the JTE length even with 0V. As the reverse voltage increases, the OBIC signal increases and extends laterally towards the outside. For diode #4, none of the JTE rings are depleted at 0V. However, these rings are quickly depleted with a reverse voltage as low as 50V.

Conclusion

Current edge termination in SiC can reach 100% efficiency, which is relevant for the avalanche ruggedness capability. Termination design is still improving to adapt to the new Trench and Super-Junction technologies or to reduce the consumed area, especially in high and very high voltage devices. However, physical limits as the minimum distance between main top electrode and channel stopper will fix the maximum size reduction achievable. Passivation and packaging are also two of the main concerns for the next generation of high voltage devices.

Fig. 1 :

 1 Fig. 1: 6.5kV commercial device picture and evaluation of its respective active and termination area

Fig. 2 :

 2 Fig. 2: Schematic of typical termination types including (a) guard rings termination using VDMOS pwell (b) triple JTE created by SiC dry etching

Fig. 3 :

 3 Fig. 3: Schematic of JBS diode termination including (a) JTE with JTE outer rings and P-well FGR (b) comparison of simulated breakdown voltage of a 6.5kV wafer for different JTE + rings designs

Fig. 4 :

 4 Fig.4: Schematic of a) a trench rings termination[8] (b) a trench termination filled with Benzocyclobutene BCB[START_REF] Liu | 31 st Int. Symp. on Power Semiconductor Devices and ICs (ISPSD19)[END_REF]

Fig. 5 :

 5 Fig. 5: (a) schematic of the termination with channel stopper (b) A picture of a failed 1.7 kV diode after premature breakdown at 1.6 kV due to short distance between anode and channel stopper.

6 :

 6 PiN Diode #1 protected with a 15 µm JTE PiN Diode #2 protected with a 40 µm JTE PiN Diode #3 protected with a 50 µm JTE PiN Diode #4 with a 35 µm JTE and 3 JTE rings Fig. OBIC measured current versus distance in experiments on 1.7 kV Class PiN diodes with several JTE protections geometries. Basic drawing of each termination is shown as inset.

Table 1 :

 1 Measured breakdown voltage and termination efficiency of 1.7kV & 4.5kV MOSFET[START_REF] Soler | [END_REF]

			Wafer#1		Wafer#2
	Design	Schematic cross section	Average VBR	Efficiency (%)	Averag e VBR	Efficiency (%)
			(V)		(V)
	D1 (130 μm)		1930	85	4500	77
	D2 (100 + 30μm)		1870	82	4000	68
	D3 (12 rings)		2000	88	4870	83
	D4		1360	60	4550	78
	D5		960	42	2880	49
	D6 (20 rings)		2080	91	4550	78
	D7		1300	57	4230	72
	D8		890	39	2700	46
	D9		2260	99	5050	86
	D10		2260	99	5130	88
	D11		2270	99	5230	89

Table 2 :

 2 Impact of the JTE length and corner radius in 1.7kV JBS diode avalanche voltage

		design	D2	D3	D4	D5
	JTE length	150μm	100μm	50μm	35μm	150μm
	Corner radius	150μm	150μm	150μm	150μm	100μm
	Avalanche voltage	> 2000V	> 2000V	1450V	1320V	1910V

Acknowledgments: Autors want to acknowledge Dr. V. Soler for his contribution to this work. This work was partially supported by the DiamMOS project funded by the Spanish Government (TEC2017-86347-C2-2-R), cofounded by the EU-ERDF (FEDER), and by the Agència de Gestió d'Ajuts Universitaris i Recerca Funds under Grant 2017-SGR-1384.