This chapter introduces networks and some of their recurrent properties observed in real-world settings. Graphs are then de ned as the mathematical representation of networks and three probabilistic graph models (namely, the Erdős-Rényi (ER), the stochastic block model (SBM) and the degree corrected stochastic block model (DCSBM) models) are introduced. These models are useful to generate random graphs reproducing some of the properties typically observed in real-world networks and will be key to de ne our main results in the remainder.

.

A B S T R A C T

Categorization, i.e. the ability to assign the same labels to objects sharing similar properties, is one of the main task in machine learning. In recent times, the ever increasing amount of data at our disposal gives us unprecedented possibilities to devise sophisticated and statistically signi cant categorization methods but it also requires a considerable e ort in designing scalable and e cient algorithms, capable to properly deal with these datasets.

Spectral clustering (SC) is one of the most popular techniques to categorize the items of a dataset that can be represented as a graph. This is a class of unsupervised algorithms in which the "best" partition does not require the help of additional information to be determined and is instead obtained by exploiting the dependencies between the dataset's items. In SC algorithms, the information concerning the class structure of the input dataset is determined by the eigenvectors of a suited matrix. The intuitions and results justifying SC are at the crossroads between several elds such as statistics, random matrix theory, computer science, network science, signal processing, statistical physics and have so far mostly been treated independently.

In this manuscript, we study the challenging (but relevant) sparse setting, in which only few entries of the matrix representation of the input dataset are non-zero. We focus in particular on the applications of SC for community detection (in both static and dynamical graphs) and for the sparsi cation of kernel matrices for time-e cient clustering of high dimensional vectors. We build on the recent advances in statistical physics for SC to propose improved algorithms that provably outperform the existing methods for both synthetic as well as real clustering tasks. Moreover, we propose a simple framework that gives a uni ed view of some of the most in uential stateof-the-art methods for SC. The existing algorithms from the literature can often be considered as corner cases of our proposed methods that constitute instead an "optimum" capable of self-adapting to the hardness of the clustering problem. We further detail extensively how to e ciently implement our proposed algorithms for practical SC tasks.

R E S U M É

La catégorisation, c'est-à-dire la capacité à attribuer les mêmes étiquettes à des objets partageant des propriétés similaires, est l'une des principales tâches de l'apprentissage automatique. Ces dernières années, la quantité toujours croissante de données à notre disposition nous o re la possibilité sans précédent de concevoir des méthodes de catégorisation sophistiquées et statistiquement signi catives, mais elle exige également un e ort considérable pour concevoir des algorithmes évolutifs et e caces, capables de traiter correctement ces ensembles de données. Le clustering spectral (SC) est l'une des techniques les plus populaires pour catégoriser les éléments d'un ensemble de données qui peut être représenté sous la forme d'un graphe. Il s'agit d'une classe d'algorithmes non supervisés pour lesquels la "meilleure" partition ne nécessite pas l'aide d'informations supplémentaires et est plutôt obtenue en exploitant les dépendances entre les éléments du jeu de données. Dans les algorithmes SC, l'information concernant la structure des données d'entrée est obtenue grâce aux vecteurs propres d'une matrice appropriée. Les intuitions et les résultats justi ant le SC sont à la croisée des chemins de plusieurs domaines tels que les statistiques, la théorie des matrices aléatoires, l'informatique, la science des réseaux, le traitement du signal, la physique statistique et ont jusqu'à présent été traités de manière indépendante.

Dans ce manuscrit, nous étudions le cadre di cile (mais pertinent) des matrices parcimonieuses, dans lesquelles seules quelques entrées de la représentation matricielle sont di érentes de zéro. Nous nous concentrons en particulier sur les applications du SC pour la détection de communautés (à la fois statiques et dynamiques) et pour la sparsi cation des matrices à noyau pour le clustering de vecteurs en grande dimension. Nous nous appuyons pour cela sur les avancées récentes de la physique statistique pour le SC a n de proposer des algorithmes améliorés qui surpassent, preuve à l'appui, les méthodes existantes pour les tâches de classi cation autant sur des données synthétiques que sur des données réelles. De plus, nous proposons un cadre simple qui donne une vue uni ée de certaines des méthodes les plus in uentes qui forment l'état de l'art pour le SC. Les algorithmes existants de la littérature peuvent souvent être considérés comme des cas extrêmes des méthodes que nous proposons qui constituent plutôt un "optimum" capable de s'adapter à la di culté du problème de classi cation. Nous détaillons également une implémentation e cace des algorithmes que nous proposons pour les tâches pratiques de • The empty set is denoted with ∅.

• I x is the indicator function equal to 1 if the condition x is veri ed and is 0 otherwise.

• 1 n is the all-ones vector of size n.

• When two quantities a n , b n admit a nite limit lim n→∞ a n b n = c, independent of n, then we write a n ∼ b n , or, equivalently, a n = O n (b n ). Conversely, if lim n→∞ a n b n = 0, then we write a n = o n (b n ).

• δ ij is the Kronecker symbol, equal to 1 is i = j and 0 otherwise.

Linear algebra

• Matrices are denoted in standard font capital letters (M). The transpose of a matrix is M T . The i-th row and column of a matrix M are denoted with M i,• and M •,i , respectively. M ij corresponds to the entry of M at row i and column j.

• Column vectors are denoted in small-case bold font (v). Row vectors are denoted with v T . Vectors elements are denoted as v i .

• Scalars are denoted in small-case standard font (a, v i ).

• I n is the diagonal identity matrix of size n.

• M = diag(v) is the diagonal degree matrix so that M ij = δ ij v i .

• tr(•) is the trace of a matrix.

• det(•) is the determinant of a matrix.

• λ i (•) indicates a generic eigenvalue of a matrix.

• For Hermitian matrices, λ ↑ i (•) (resp. λ ↓ i (•)) is the i-th smallest (resp. largest) eigenvalue.

• For non Hermitian matrices λ ↑ R,i (•) (resp. λ ↓ R,i (•)) is the i-th eigenvalue with smallest (res. largest) real part. Similarly, λ

↑ |•| i (•) (resp. λ ↓ |•| i (•))
is the i-th smallest (resp. largest) eigenvalue in modulus.

• Λ(M) is the set of all the eigenvalues of M.

xii I N T R O D U C T I O N

Categorization, i.e. the ability to divide objects into groups, is deeply rooted in human intelligence and takes a prominent role in learning. One could even say that the criterion followed to categorize objects precisely provides a definition of the abstract concept related to the categories themselves [Hul ].

Whether consciously or not, human intelligence allows each one of us to perform two basic operations, sketched in the toy representation of Figure , to accomplish the task of categorizing objects. The rst one requires to identify the informative features of the input data, i.e. those that are necessary and su cient to attribute a category label to the input data. For example, From humans... considering apples and pears, discriminative features may be the taste, the texture, while the weight of the fruit would generally be non-informative. This step is called feature extraction and provides a representation of the object in a parametric space, so that similar objects in the real world (the apples or pears) should correspond to nearby points in the parametric space.

The second step consists in attributing a label to the input, according to its position in the feature space. Depending on the problem considered, this can be done in two main ways. One is classi cation: the task is assigning a single given item to one of the possible available groups; the other is clustering: in this case, the goal is to divide into groups several input items, according to their proximity as shown in the right plot of Figure .   Machine learning aims at reproducing with computers the human ability of nding patterns in data [Bis ]. Machines have a decisive disadvantage with respect to humans in accomplishing complex tasks like pattern recog- nition as they are bounded to a simple representation of inputs in the form of arrays of bits and do not have a complex experience of reality that the ve senses give to humans. Nevertheless the steps of feature extraction and ...to machines successive clustering or classi cation are the fundamental building blocks of several machine learning algorithms [MRT ].

Classi cation is typically achieved with supervised learning, which is a very intuitive (in principle) way of intending learning and is nowadays most commonly deployed. A large dataset of items equipped with their groundtruth labels is used to train an algorithm to generate a rule predicting the labels. Once the training has been completed, the rule can be used to classify newly submitted inputs. To this class of algorithms belong neural net-Supervised learning works that were born with the aim of reproducing the learning process that happens in a human brain [MP ] and constitute nowadays the state-of-theart in most elds in terms of performance. This approach has however its drawbacks: the learning step may be computationally expensive, while the creation of a suited training set may be time and storage consuming.

Related to the problem of clustering instead, unsupervised learning techniques are generally preferred in cases where the machine is not provided with any additional information beyond the data to be clustered. The lack

Unsupervised learning

of "external aid" of course hampers the performances of unsupervised learning as compared to supervised learning, but the inconveniences introduced by the learning step are here bypassed. Unsupervised learning algorithms in fact pro t from the ability of machines to perform hard computing tasks that is precisely where they overcome humans, being capable of processing enormous amount of data in a short time. The advantage of dealing with large datasets is indeed to exploit statistically relevant relations between its elements and to nd patterns with no additional external information.

Even though supervised learning algorithms are nowadays very popular due to their ability to obtain super-human performances, serious concerns should be raised about their requirements of several labelled samples and of computational intensive training [Usa+ ]. The ever increasing size of input datasets, in fact, cannot (and probably should not) be accompanied by an equally fast growth of computational e ort. In this perspective, unsupervised techniques play a fundamental role in modern machine learning.

Clustering has several applications in di erent elds and is, generally speaking, an ill-de ned problem [XW ; RM ; Sch ; WK ]. One of the most popular classes of clustering algorithms, which is at the core of this manuscript, is spectral clustering (SC) whose building blocks are described as follows. The rst step of SC consists in providing the input dataset with a graph representation encoding the pairwise relations between its items. A more extensive introduction to graphs is dedicated in Chapter . For visualization purposes, a graph with two clusters is shown in the left plot of Figure .

Graph representation

The following step of SC is then the feature extraction, which consists in mapping each item (or node in graph language) into a small dimensional vector. This is also known as node embedding [CZC ; NS ], whose representation is given in the right plot of Figure that should be compared to

Feature extraction

Figure . An appropriate embedding is such that "similar" nodes should end up close in the embedded space. In SC, the nodes are embedded exploiting the eigenvectors of a suited graph matrix representation. One of the main advantages of SC with respect to other embedding methods [GF ; CZC ; NS ] relies in its solid theoretical foundations and explainability.

The nal step consists in performing a low dimensional clustering on the input embedded nodes, i.e. in determining boundaries in the embedded space to separate the points. Several algorithms exist to accomplish this task and Clustering can be divided in two groups: those in which partitions are attributed solving an optimization problem such as k-means [Mac+ ], k-medoids [KR ],

expectation maximization [DLR ]; those in which boundaries between clusters are drawn where the density of points is minimal, as shown in Figure such as DBSCAN [Est+ ] and OPTICS [Ank+ ].

The main steps of a typical SC algorithm are summarized in Algorithm .

Here, it has to be noted that the dimension of the embedding is set to k, the number of clusters. This is a customary choice, but not a stringent requirement. In fact, on a more general level, Algorithm is written for pedagogical reasons and not all SC algorithms follow exactly its structure.

In order to justify why Algorithm can be used for clustering, we provide an example for k = 2 clusters, based on the work of [Fie ] that can be considered the rst work on SC. The mathematical and intuitive description the ground-truth labels. The histograms represent the densities of points along the two directions. The dash-dotted black lines correspond to the minima of the histograms and allow one to cluster the input data into k = 3 groups using a DBSCAN-type algorithm.

is willingly left to a low level. More precise results and considerations are at the core of the main body of the manuscript.

A representation of an A: a white dot indicates A ij = 0, while a coloured dot

A ij = 1.
Let us consider a problem in which the relations between items are encoded by binary variables A ij ∈ {0, 1} with A ij = 0 indicating that i and j do not interact and, on the opposite A ij = 1 that they do. This is the simplest possible setting to describe pairwise interactions and the matrix A ∈ {0, 1} n×n is the adjacency matrix of an unweighted graph, that will be more formally de ned in Chapter . Letting D = diag(A1 n ) be the diagonal degree matrix, where 1 n is the all-ones vector of size n, we introduce the graph Laplacian matrix L = D -A. For any vector f ∈ R n we can write

f T L f = n ∑ i,j=1 f i d i δ ij -A ij f j = n ∑ i=1 f 2 i d i - n ∑ i,j=1 A ij f i f j = n ∑ i,j=1 A ij f 2 i -f i f j = 1 2 n ∑ i,j=1 A ij f i -f j 2 , ( )
where δ ij denotes the Kronecker delta. Any pair of interacting nodes (A ij = 1) contributes to this sum with a weight which increases with the distance between f i and f j . Now, supposing f is a node embedding vector, ( f if j ) 2 should be small if i and j are "similar" and large otherwise. Consequently f T L f can be seen as an embedding cost function. In fact, assuming that nodes are more likely to interact if they are similar, to large values of f T L f correspond vectors in which interacting nodes have very di erent embeddings. Minimizing f T L f under the constraint f ⊥ 1 n (which is a trivial embedding) allows one to map "similar" nodes to "nearby" points. Imposing the normalization f 2 = 1, we de ne the embedding vector f * as Algorithm : Spectral clustering Input : Dataset with n items, k number of clusters Output : ∈ {1, . . . , k} n label assignment begin De ne suited matrix representation of the dataset M ∈ R n×n ; Stack the k largest (or smallest) eigenvalues of M in the columns of X ∈ R n×k (Feature extraction); Estimate community labels with a small dimensional clustering algorithm performed on the rows of X (Clustering) ; return end

f * = arg min f ∈R n , f ⊥1 n f T L f .
Since 1 n is the eigenvector of L associated to its smallest eigenvalue (this is easily seen from Equation ()), f * is the eigenvector associated to the second smallest eigenvalue of L, known as Fiedler eigenvector, and it justi es SC in the form of Algorithm for the choice M = L. Following similar arguments, some classical SC algorithms are based on the adoption of the normalized Laplacian matrices L sym = D -1/2 AD -1/2 , L rw = D -1 A and the adjacency matrix A [VL ; LR+ ], with possible extensions also to the weighted case in which A ∈ {0, 1} n×n is replaced by W ∈ R n×n .

'

The eld of SC is very interdisciplinary and, in order to describe at best where the work presented in this manuscript locates in the existing literature, a wide perspective on the problem must be kept.

First of all, as we mentioned earlier, one of the main interests of SC lies in its solid theoretical foundation. Powerful tools based on random matrix theory (RMT) allow one to provide precise results on the performances and applicability of Algorithm . Typically, however, two important hypotheses have to be formulated: the size n of the matrix M must go to in nity and the number of non-zero entries (say m) must go to in nity faster than n

The computational complexity of SC [CBG+ ; AC , for instance]. While the former simply consists in considering the very interesting setting of big datasets, from a practical perspective, the latter hypothesis raises a problem from the point of view of the scalability of Algorithm . In fact, the numerical cost of computing the k leading eigenvectors of a matrix with m non-zero entries grows as O(mk 2 ) [Saa ]. Let us consider the case m = n 2 , i.e. all the entries of the matrix M are non-zero, for a small value of k such as k = 2. The complexity is in this case O(n 2 k 2 ) which makes the cost of Algorithm prohibitive for a standard laptop when n > 10 5 approximately.

The problem of the computational complexity of SC is well known in the literature and led to several sampling strategies that aim at improving the e ciency of Algorithm [TL ]. The simplest procedure simply consists in making the matrix M sparser, setting some of its entries to zero. This improves signi cantly the e ciency of Algorithm . Consider in fact a sampling strategy that keeps on average 10 non-zero entries in each row of M: the computational limit of the laptop considered earlier is now obtained for n = 10 9 which allows one to deal with datasets of considerable sizes.

This sampling procedure, however, comes at a cost. Besides the fact that by "throwing away" some measurements (hence information), the output SC and sparsity of Algorithm may naturally be less accurate, sparsity has long known to be the "Achilles' hill" of SC. In fact, for great levels of sparsi cation, while there exist algorithms di erent from SC that can nd reasonable clusters, Algorithm typically performs very poorly for the standard choices of the matrix M, such as the graph Laplacian L = D -A introduced earlier.

The problem of sparsity is not only related to sampling procedures. A natural application of SC is to perform node embeddings on social networks in order to divide the nodes into communities. Real social networks are often sparse by construction [Bar ] (each node interacts typically with a very small fraction of the total) and, as a consequence, Algorithm may perform poorly in practical community detection (CD) tasks. All in all, these are severe limitations of SC that make it unsuited for a large set of inputs (such as the aforementioned social networks) and that allow for good performances only at the price of a great computational cost.

A breakthrough work on SC in sparse graphs was however provided in [Krz+ ], "redeeming" the use of SC techniques of very sparse graphs, specifically in the context of CD. The work of [Krz+ ], together with [Dec+ ; SKZ ], is based on very deep but non-rigorous intuitions and methods borrowed from statistical physics that renewed the interest of the scienti c community in the topic and led, in the following years, to formalize most of their

Statistical physics

gave powerful insights on SC in sparse graphs results in a mathematically rigorous way. As far as SC is concerned (and in particular CD), the main contribution of [Krz+ ; SKZ ] consists in proposing "new" matrices M for Algorithm to replace the "classical" choices when performing clustering in sparse graphs. The Bethe-Hessian matrix plays a fundamental role in this manuscript, being the choice for M in all our proposed algorithms. In the unweighted case, its de nition reads, for a scalar r ≥ 1

The Bethe-Hessian matrix

H r = (r 2 -1)I n + D -rA,
where I n is the identity matrix of size n. We now attempt to describe in simple terms the fundamental role played by the parameter r, adopting also in this case a willingly hand-waving tone and leaving rigorous details to the following chapters.

The "statistical-physics" approach is deeply related to Bayesian inference in which, given the observation of A, each label con guration = {1, . . . , k} n has a probability to be the "good" label assignment given by

P( |A) = 1 Z exp -ath 1 r H( ) ,
where H( ) is a cost function of the label assignment and Z a normalization constant. For k = 2 classes, the eigenvector associated to the second smallest eigenvalue of H r can be used to obtain an approximation of the vector of marginal node probabilities {P i ( i )} i=1,...,n . It is now fundamental to understand the role played by r. For r → 1, only the con guration minimizing the cost function H( ) matters and the problem of computing the node marginals simply reduces to nding the optimum of H( ). For larger values of r, instead, each con guration contributes to the function P( |A) and prevents the marginals from being potentially determined by meaningless local minima of H( ) introduced by noise: the higher the noise, the larger r should be. In simple words, therefore, tuning r in H r allows one to avoid over-tting.

This concludes the brief introduction to the many facets of SC considered in this manuscript which is a metaphorical dance between powerful physics intuitions, rigorous mathematical results and practical questions of applicability of our proposed algorithms to real-world problems.

In order to provide the reader with the necessary tools needed to best understand our contributions, Part I introduces the technical results exploited in the remainder. In particular, Chapter provides an introduction to graphs, together with their de nitions and properties. Then, Chapter gives the fundamental tools of statistical physics on which our main ndings rely. Finally, Chapter introduces some important mathematical results on the spectral properties of two matrices of fundamental importance in this manuscript: the non-backtracking and the aforementioned Bethe-Hessian matrices.

Part II is then dedicated to the presentation of our original contributions to SC for CD and is opened by Chapter that provides an extensive introduction to the problem.

Here we take the "torch" of our predecessors and address the problem of how overly simplistic assumptions may result in poor algorithmic performances. Speci cally, in [SKZ ], the authors proposed a very powerful spectral algorithm for CD in very sparse graphs, based on the Bethe-Hessian matrix H r . Their theoretical results are formulated under the assumption that the matrix A is generated from the stochastic block model (SBM), according to which each node of the graph has approximately the same number of connections. Real social networks (in which CD is of utmost importance), however, are known to be typically heterogeneous, in the sense that the number of connections each node has varies a lot across the network. As a consequence of this overly simplifying assumption, the algorithm of [SKZ ] (that prescribes a precise choice of r) often performs poorly compared to classical SC algorithms (see Table . ).

Our earliest works are presented in Chapter and explain how we improved over [SKZ ], considering a more realistic generative model, called degree corrected stochastic block model (DCSBM) which indeed allows one to keep the graph heterogeneity into account. The rst of these works is [DC ] LD, Romain Couillet: Community detection in sparse realistic graphs: Improving the Bethe-Hessian in ICASSP IEEE I C A , S S P ( ) in which we proposed (for k = 2 classes of equal size) an alternative choice of r from the one of [SKZ ], able to provide high performance clustering on heterogeneous graphs. The proposed parametrization r, however, requires the knowledge of some of the parameters of the generative model of the graph that cannot be given for granted.

With our next contribution [DCT ] LD, Romain Couillet, Nicolas Tremblay: Revisiting the Bethe-Hessian: Improved Community Detection in Sparse Heterogeneous Graphs in N IPS A N I P S ( )

we answered some of the main open questions of [DC ] . First, we extended our results to the setting of k ≥ 2 classes of arbitrary size, showing that a sequence of properly chosen parameters r must be selected to get an e cient SC algorithm on heterogeneous graphs. Secondly, we showed that these parameters r can be estimated in an unsupervised way. We did so unveiling an unforeseen property of the eigenvalues of the non-backtracking matrix B used in [Krz+ ] and exploiting the known relation between B and H r determined by the Ihara-Bass formula [Ter ]. Finally, backed by several arguments and numerical simulations, we claimed our proposed parametrization to be optimal, in the sense that no other parametrization is expected to perform better on DCSBM-generated graphs.

Having a practical way to estimate the optimal values of r in an unsupervised way further allowed us to devise and test a practical algorithm for CD. The results con rm that the improved parametrization achieves a systematic higher performance with respect to [SKZ ] on all tested datasets.

In the SC literature, however, not only the contributions of the physics community led to very e cient algorithms on sparse graphs. In fact, a notable and independent line of work, with an approach rooted in statistics, proposed the use of new regularized matrices for SC. In this introduction we only mention one these matrices, L sym τ = D -1/2 τ AD -1/2 τ , proposed in [QR ] with D τ = D + τ I n for τ equal to the graph average degree. This is de facto the state-of-the-art of SC for CD on real graphs. In Chapter we present our work related to L sym τ , started in [ DCT b] LD, Romain Couillet, Nicolas Tremblay: Optimal Laplacian regularization for sparse spectral community detection in ICASSP IEEE I C A , S S P ( ) in which we showed that there exists a clear relation between the Bethe-Hessian matrix H r and the regularized Laplacian L sym τ . Furthermore, based on our results in [DC ; DCT ] we showed that the choice of τ of [QR ] is suboptimal. The optimal Laplacian regularization is obtained with a set of τ closely related to the r's to be adopted for H r and that can be estimated from the graph in an unsupervised fashion. Matter of factly, some of the results presented in Chapters , are taken from [DCT a] LD, Romain Couillet, Nicolas Tremblay: A uni ed framework for spectral clustering in sparse graphs in J in which we summarized our results in a uni ed framework making formal statements of our conjectures and providing rigorous proofs to part of them. A further big contribution of [DCT a] is at the centre of Chapter and concerns the applicability of our proposed algorithms on practical real graphs as well as their e cient implementation. Here we propose Algorithm . for CD and conduct a systematic study on real datasets showing the e ciency of our proposed method for practical tasks. On top of this we also released an e cient implementation in Julia language of Algorithm . for CD, called CoDeBetHe.jl (Community Detection with the Bethe Hessian).

Chapter nally closes part II starting from the results posed in [DCT a] to provide a uni ed view of several SC methods that have so far been treated independently. In this chapter, in fact, we clearly show that our proposed algorithm is adaptive to the hardness of the clustering task and that common state-of-the-art methods in both the sparse and dense regime can be understood as limiting (suboptimal) cases of Algorithm . . These results pose solid methodological foundations that led us to work on the extensions of our works to more general settings than CD, treated in Part III. The rst problem we considered concerns dynamical community detection (DCD), which enriches the description by keeping the dynamical nature of real-world graphs into account. In [DCT a] LD, Romain Couillet, Nicolas Tremblay: Community detection in sparse timeevolving graphs with a dynamical Bethe-Hessian in N IPS A N I P S ( )

we considered a sequence of graph snapshots at di erent times with the assumption that community labels may change across time, while having a positive correlation. We developed a novel SC algorithm based on a dynamical Bethe-Hessian matrix which is capable of exploiting the information coming from these correlations and that largely outperforms the state-ofthe-art competitors. The results of [DCT a] are treated in Chapter and the corresponding algorithm is also part of the CoDeBetHe.jl package.

Finally, the work of [ DCT b] LD, Romain Couillet, Nicolas Tremblay: Nishimori meets Bethe: a spectral method for node classi cation in sparse weighted graphs in J S M : T E is presented in Chapter and treats the extension of our works to the setting of matrix sparsi cation to improve the e ciency of SC, while keeping high performances. Testing our proposed algorithm on the prototypical problem in machine learning of clustering a set of images of dogs and cats, we showed that almost perfect accuracy can be achieved even for dramatic levels of sparsi cation, making our algorithm suited for clustering very large datasets. The main result of [DCT b] , however, is deeply related to statistical physics, describing an explicit relation between the Bethe approximation (from which the Bethe-Hessian matrix is derived) and the Nishimori temperature [Nis ], of fundamental importance in the physics of spin-glasses and in Bayes-optimal inference [Iba ].

Together, these works o er a new vision and robust adaptive algorithms, linking statistical physics and more conventional SC methods and able to handle unsupervised clustering in sparse, heterogeneous and dynamical graphs.

We now proceed with the technical chapters of the manuscript that introduce the necessary tools to fully describe our contributions.

T E C H N I C A L T O O L S

This part is devoted to the introduction of the technical tools needed to describe the original work presented in this manuscript. Notably, it is composed of three chapters. The rst one is dedicated to the properties of graphs, the second one to statistical physics methods adapted to sparse graphs and, nally, the last one to the study of the spectra of the non-backtracking and Bethe-Hessian matrices. node, while each black line represents an edge. Right: major highways in the USA, taken from [Bar ].

ically complicated object such as a network, into a set of small dimensional vectors, preserving at least some of the network's properties.

For this reason, to fully address our main contributions and to settle the "vocabulary" of the following chapters, we must rst proceed with a description of networks. We do so following an ideal path leading from observation to theory. In particular, Section . can be seen as the observation step in which networks are introduced as physical objects and their main characteristics as well as their possible representations are discussed. Subsequently, Section . takes the role of a formalization step, de ning graphs as the mathematical representation of networks and laying out the main mathematical de nitions. The third step is the model creation, necessary for a thorough analytical study of graphs. This is the content of the concluding Section . that introduces simple models -of central importance in the remainderto generate random graphs, together with their main properties and relate these random models to the real networks discussed in Section . . . . .

To best understand networks, we now proceed to give some practical examples of real-world systems that can be represented as networks, together with their main representations.

We start considering the simplest family of networks which are undirected and unweighted. The toy representation in the left plot of Figure . falls precisely in this class in which the relation between two nodes is solely encoded by the existence of an edge connecting them. The term unweighted stands Undirected and unweighted networks indeed for the fact that each edge does not bring any additional information beyond its existence. The term undirected, instead, signi es that a symmetry is assumed in the interaction of the two nodes, hence that if node i interacts with node j, than node j interacts with node i. indicates a node, while each black arrow represents an edge together with the direction in which the edge is pointing. Right: a toy example of a multiplexed network with two layers indicated by di erent colours.

The representation of a complex system of, for example, people, with an unweighted and undirected network might come at the cost of a signi cant simpli cation and the question of what kind of interaction should be encoded by edges has no simple answer [Zwe ]. Nevertheless, a lot of systems can be modelled with unweighted and undirected networks. Some examples are given by social networks, the world wide web and biological networks [Sco ; WF+ ; New ; AB ]. The remainder of the manuscript focuses mainly on this class of networks in which information is only encoded in the edge con guration, hence in how nodes are connected among themselves.

A richer representation that goes beyond the binary information of whether or not two nodes interact is obtained with weighted networks [Bar+ ]. In this case, each edge carries additional information on the type of relation between the nodes, in terms of, e.g., intensity of a nity. In some networks weights are easily de ned, for example when considering spatial networks [Bar ], like the one shown in the right plot of Figure . . There, each node Weighted networks represents a city, each road an edge and a weight can be attributed to each edge as function, for example, of the length of the road itself. Another example is the network Slashdot Zoo [KLB ] in which users express a positive or negative endorsement with respect to other users leading to a signed network with positive and negative weights.

A limitation of both classes of networks just introduced is that they assume commutativity in the relationships between items, i.e. that the interaction between a and b is the same as the one between b and a. If this hypothe-Directed networks sis is not veri ed, then it is more appropriate to recur to directed networks in which each edge encodes an orientation of the relationship between nodes, as shown in the left plot of Figure . . For reference, one could consider a network in which an edge (ab) goes from node a to node b if node a sent an email to node b [NFB ]. It appears natural that the edge (ba) has a di er-ent meaning from (ab), justifying the need of a directed network to describe relationships in this case. Other examples of directed networks are for example citation networks [TS ] for which the edge (ab) is a citation of author b from author a, or the links on the webpages on the internet [Bar ]. A natural consequence of adding directionality to the edges is to be capable of grasping the causality of the relations and to describe how information ows on the network.

The last network representation we consider takes into account that in the wide range of systems that can be modelled by networks, it may occur that the relationship between the nodes of a network is not measurable only in terms of a single parameter. Suppose, for instance, one wants to determine the a nity between people. Many factors should be kept into account like, for example, age, geographical provenance, spoken language, social extraction, political orientations and so on. The a nity between items is not necessarily determined by a simple combination of the a nities at di erent levels, but should rather take a vector form, allowing for the possibility to Multiplexed and temporal networks have a large a nity according to some parameters but not to others. The use of multiplexed networks [Kiv+ ] (or multilayer networks) is most suited to this scenario: in this case, multiple representations of the same network are given, encoding information at di erent levels as in the example shown in the right plot of Figure . . For reference, a practical example of a real-world multiplexed network is the one showing parental and economical relations between the families of Florence during Renaissance [BP ] or by transportation networks in which each layer corresponds to a di erent line, such as bus, tram, underground lines [Kiv+ ].

As a particular class of multiplexed networks one can consider temporal networks [HS ] in which the di erent representations of the network simply indicate its evolution across time. Most real-world networks are indeed dynamical and some examples are found in human proximity networks, biological networks, communication networks among the others. The study of temporal networks will be at the centre of Chapter .

After this short introduction on the di erent types of networks and some examples of systems that can be modelled by networks, we now proceed with the characterization of some properties that occur in several real-world networks with a speci c focus to the unweighted and undirected case.

. . As we detailed in the previous paragraph, networks can represent fundamentally di erent systems hence the emergence of some universal behavior is not an obvious fact. It appears instead that many real-world networks share some peculiarities, regardless of the underlying system. We now proceed to present some of these properties, while keeping in mind that they should not be intended as laws, but rather, very recurrent observations. The rst aspect that typically characterizes real-world networks is the socalled small world e ect [WS ]. This property is veri ed when, considering a network composed by n items, the average distance (i.e. the number of interposing nodes) between two arbitrary nodes grows as log(n) [Bar ]

(or

Small world e ect even slower). This e ect is observed in sociology with the fact that humans are on average "six handshakes away" from one-another [Mil ; Gua ]. The small world e ect is observed way beyond social networks, for example in the World Wide Web [Bar ]. As a consequence of this e ect, information ows rapidly on networks and an emerging behavior can be observed.

Another peculiar aspect characterizing real-world networks is the degree distribution. It has been observed in a vast number of networks that the number of connections each node has (we call this number the degree) is far from

The degree distribution a homogeneous distribution [BA ] (see Figure . ). Typically, this distribution is, instead, a power law. In this case we talk about scale free networks. When dealing with a broad degree distribution, most nodes have very few connections while very few are highly connected. These nodes play a prominent role due to their high connectivity and are called hubs.

Remaining on the properties of the degree distribution, in real-world networks the average degree is typically very small compared to its size. This phenomenon goes under the name of sparsity. In others words, out of all the possible connections that can take place, only a very small fraction are typi-Sparsity cally present [Bar ]. Consequently, for a large network, the average degree can be considered as roughly independent of the size of the network itself.

The nal property we wish to introduce concerns the clustering e ect. This consists in the formation of small groups of highly connected items as a consequence of the fact that, in social networks, connections are formed on

The clustering e ect a homophilic principle, i.e. they are created between items sharing similar properties [LM+ ; MSLC ]. Due to the weak transitivity of homophily [BH ], highly connected communities emerge in networks.

These four properties represent some fundamental characteristics of real world networks, considered in the remainder. The clustering e ect and the consequent emergence of a community structure is, in particular, what primarily justi es the task of node clustering. Sparsity and heterogeneity of the degree distribution are instead challenging features to deal with when performing node clustering.

With this introduction on the main properties of real-world networks at hand, we proceed to de ne graphs as the mathematical representation of networks and laying out some formal de nitions. . . .

To formally characterize graphs as the mathematical representation of networks, only single-layered graphs are considered: multiplexed and temporal graphs can be seen as an extension of single-layer graphs and are not formally introduced here. Further discussions on temporal graphs will follow in Chapter . Let us rst formally introduce the de nition of a graph. The set E d denotes the ensemble of all the directed edges of G(V, E d ). Each element e ∈ E d can be equivalently written as (ij), representing a directed edge from i to j. The inverse of (ij) is (ji) and is denoted by e -1 . For an undirected graph, if e ∈ E d then e -1 ∈ E d . The set of undirected edges is denoted with E (with |E d | = 2|E |) and an undirected graph with G(V, E ).

De nition . (Graph

). A graph G(V, E d ) is a tuple (V, E d ) with V the set of A directed edge (ij)
In the sequel we consider only graphs in which no self-edge exists, hence verifying for all i ∈ V, (ii) / ∈ E .

If G(V, E d ) is an unweighted graph, then ω e = 1 for all e. Note that we allow ω e ∈ R: this is not a customary choice as often the weight is required to be strictly positive. We consider this more general case however, because it is particularly handy for our introduction to statistical physics in Chapter , as well as our work concerning sparsi cation of kernel matrices

In red the node i, in blue its neighbours, in green the remaining nodes i for spectral clustering presented in Chapter . In any case, the following de nitions do not involve the weights but only the structure of G(V, E d ).

We now introduce the concept of adjacency on a graph.

De nition . (Adjacent nodes and edges

). Given a graph G(V, E d ), j ∈ V is said to be an adjacent node of i ∈ V if (ij) ∈ E d . Two directed edges (ij), (kl) are said to be adjacent if j = k.
The set of all adjacent nodes of i ∈ V forms the neighbourhood of i.

De nition . (Neighbourhood and degree). The incoming neighbourhood of a node i is de ned as If a node has no neighbours, we say it is isolated. On an undirected graph this corresponds to the condition d i = 0. Neighbours are nodes that can be reached in one step. Exploiting this relation, we now introduce paths.

∂ in i = {j ∈ V : (ji) ∈ E d }, while the outgoing neighbourhood is ∂ out i = {j ∈ V : (ij) ∈ E d }.
In blue, a simple path connecting i and j i j De nition . (Paths). A path between i and j is an ordered sequence of adjacent edges starting from i and ending in j,

P ij = {(ik), (k•), . . . , (•l), (lj)}. The length of the path is |P ij |. A path is simple if no edge appears twice.
Note that on undirected graphs, all paths can be followed in reversed order and, if a path exists between two nodes, then they are said to be connected.

De nition . (Connected nodes). On an undirected graph, if there exists a path P ij from i to j (or equivalently from j to i), then i and j are said to be connected. The connected components of G(V, E ) are the disjoint sets of all nodes that are pairwise connected. If there is a unique connected component equal to V, then G(V, E ) is said to be a connected graph. This de nition exploits the fact that connectivity is commutative on undirected graphs. On the opposite, this is not the case for directed graphs in which there may exist a path from i to j, but not from j to i. We talk in this case about weak connectivity if there is a path P ij or P ji and of strong connectivity if both P ij and P ji exist.

The last de nition provided is that of trees that play a crucial role in the following chapters. In order to de ne trees, cycles need to be introduced: a simple path that starts and ends from the same node, is called a cycle.

De nition . (Tree).

An undirected graph G(V, E ) is said to be a tree if it is connected and it does not contain any cycle.

A tree

The particular interest in trees in this manuscript is a consequence of their structural relation with conditional independence [WJ ]. Speci cally, suppose that a random variable x i is associated to each node i ∈ V and that the dependences between these variables are determined by the edge structure. As a consequence of the fact that a tree is a connected graph, for any pair of nodes, i, j ∈ V, there exists a path connecting them and x i and x j are not independent. On a tree, however, this path is unique and if any node of the path connecting i to j (supposing they are not neighbours) is removed, then i and j are no longer connected and x i and x j become independent. This structural property of tree allows one to say that for any k ∈ P ij , the variable x i depends on x j only through x k , hence x i and x j are independent, conditionally to the value of x k . This fact makes it particularly easy to deal with dependences between random variables on tree nodes and will be further exploited in the remainder. Now that we have introduced the main de nitions concerning graphs, we turn our attention to their matrix representation.

0 1 2 3 4 A =          0 1 0 0 1 1 0 1 0 1 0 1 0 1 0 0 0 1 0 1 1 1 0 1 0         
. .

Let us introduce two fundamental matrices that can be used to represent unweighted graphs. These are, namely, the adjacency and the non-backtracking matrices and will be of central importance in the remainder.

The adjacency matrix

The adjacency matrix (denoted as A) is the most straightforward representation of a graph. In words, for each pair of nodes i, j, the entry A ij indicates whether the two nodes are adjacent or not. De nition . formally introduces the adjacency matrix.

De nition . (Adjacency matrix).

The adjacency matrix of a graph G(V, E d ) with n nodes is denoted with A ∈ {0, 1} n×n with entries

∀ i, j ∈ V A ij = I (ij)∈E d ,
where the notation I x denotes the indicator function, equal to one if condition x is veri ed and zero otherwise. For an undirected graph, A is symmetric.

A visualization of A and the corresponding graph is given in Figure . . For a given graph, the matrix A is uniquely de ned and vice-versa. For this reason we say A is a representation of G(V, E d ).

The adjacency matrix is not the only possible matrix representation of a graph: an alternative, less conventional, representation (of fundamental importance in the following) is provided by the non-backtracking matrix that is de ned on the set of edges (instead of nodes) of the graph.

The non-backtracking matrix

The non-backtracking matrix, denoted with B and also known as Hashimoto edge operator [Has ], can be seen as the directed adjacency matrix of a graph G (V , E d ) constructed as follows: each directed edge in E d is mapped Highlight on two adjacent edges that do not form a backtrack to a node in V and an edge is drawn in E d between the nodes corresponding to e 1 , e 2 ∈ E d if e 1 and e 2 are adjacent and e 1 = e -1 2 , i.e if e 2 is not the same edge as e 1 with reversed orientation. The reason of the name nonbacktracking comes from the fact that edges in G (V , E d ) correspond to paths of length 2 in G(V, E d ) without a backtrack, i.e. paths that cross the same edge in the two directions. The matrix B is de ned as follows:

De nition . (Non-backtracking matrix). For a graph G(V, E d ), the nonbacktracking matrix B ∈ {0,

1} |E d |×|E d | is de ned as ∀ (ij), (kl) ∈ E d B (ij)(kl) = δ jk (1 -δ il ).
Note that on a undirected graph, A is symmetric, while B is not.

Remark . . In De nition . the entries of the matrix B are in one-to-one correspondence with the edges of a directed graph G(V, E d ). If the input graph G(V, E ) is undirected, the non-backtracking matrix is still de ned on the set of its directed edges

E d with |E d | = 2|E | in which the edges (ij) and (ji) are distinguished.
With the main de nitions laid out, we now move to the description of some generative models of random graphs of utmost importance in the remainder of the manuscript. We in particular introduce the Erdős-Rényi model as the simplest random graph generative model, together with its variations adapted to generate random graphs with a community structure.

. In order to get a deeper grasp, make theoretical predictions and have a more profound understanding over classes of graphs (and not single realizations) a reductionist approach is needed. Random generative model, depending on few, highly interpretable parameters, are introduced to create graphs that resemble -at least under some aspects -real-world graphs. For a given generative model, then, the probability of an event on the graph (like, e.g., "All models are wrong some are useful" George Box the existence of a cycle of a given length) can be studied. From now on, unless otherwise speci ed, we considered undirected graphs.

The reductionist approach has its strengths and weaknesses. Having a well de ned probabilistic model to generate G(V, E ) introduces good mathematical control over di erent graph-related quantities, particularly in the asymptotic regime of large graphs (with n → ∞). On the down side, of course, simplifying the description of complicated objects into simple generative models leaves necessarily some relevant features behind. For this reason, in the following sections, when introducing the generative models under analysis, we will try to emphasize which property of typical real graphs are caught by the considered generative model and which ones are not.

Many random graphs models have been introduced in the literature. Some relevant examples that will not be discussed since they go beyond the scope of this introduction, are the small world model of [WS ], the preferential attachment model of [AB ] and the Dorogovtsev-Mendes model [DM ]. The remainder will focus on Erdős-Rényi random graphs, together with its extensions to generate graphs with a community structure: the (degree-corrected) stochastic block model.

. .

Out of all the models of random graphs, the ER model [ER ] is the simplest and most studied. The ER graphs constitute a class of unweighted and undirected graphs. The essence of the ER model is to generate the edges of G(V, E ) independently at random with equal probability. Formally, recalling the bijection between G(V, E ) and the adjacency matrix associated to it, we de ne the ER model as follows:

De nition . (Erdős-Rényi graph). Let c ∈ R + be a positive scalar. The entries of the adjacency matrix A ji = A ij of a random Erdős-Rényi graph are set to 1 independently at random with probability c n and to 0 otherwise.

From a straightforward calculation, one obtains for n 1 that c is the expected average degree of the graph. The ER random graphs show some relevant properties, especially in the asymptotic regime of n → ∞ that will be assumed in the following. The rst of these properties concerns the per- The notation

a n = O n (b n ) is equivalent to lim n→∞ b n /a n = κ, with κ independent of n.
The percolation threshold has applications in solid state physics, epidemiology and technological networks among others [KNZ ].

We now take into consideration the degree distribution of ER random graphs [BB , Chapter ].

P(d i = k) = n -1 k p k (1 -p) n-1-k ,
hence in the large n limit, the expected average degree is equal to c.

ii) If c/log(n) → ∞ as n → ∞, then, with high probability

max i∈V |d i -c| = o n (c).
iii) For n → ∞, with high probability,

min d i > 0 ⇐⇒ c > log (n).
The notation

a n = o n (b n ) is equivalent to lim n→∞ a n /b n = 0.
From Property . , it turns out that for a su ciently high average degree c, the degree distribution is almost regular, hence far from the properties of degree heterogeneity studied in real networks. As a matter of fact, for small values of c, even if the degree distribution is not almost regular, it is far from being a broad scale free distribution. As a consequence, the degree distribution of ER random graphs is not suited to describe real-world graphs. Property . further states the existence of a connectivity transition: below the critical threshold, G(V, E ) has isolated nodes, while above the threshold it is connected with high probability.

Among the properties typically observed in real world networks, we recall the small world e ect according to which there exist a short path (compared to n) connecting any two nodes in G(V, E ). This property holds with high probability also in ER graphs [AB , for example], if G(V, E ) is connected.

Small world e ect in ER graphs

This means that ER random graphs are a suitable candidate to model smallworld e ect of real-world graphs.

Property . (Average distance between nodes of ER).

Let G(V, E ) be an Erdős-Rényi random graph as in De nition . , with c > log (n). The average distance between two arbitrarily chosen nodes is proportional to log(n)/log(c).

For 1 < c < log (n), the presence of disconnected nodes is such that the distance between two arbitrarily chosen nodes diverges but Property . can be extended to this case considering only the giant component of G(V, E ).

With these properties of the degree distribution being laid out, we take the occasion to formally introduce the concept of sparse and dense graphs in

The asymptotic de nition of sparse and dense graphs the asymptotic regime. An ER graph is said to be sparse if, in the asymptotic limit of large n, its average degree stays nite, i.e. if c = O n (1). Conversely, if the expected average degree goes to in nity, the graph is said to be dense.

It is worth making further considerations on the asymptotic de nition of sparsity and relate it with the intuitive one introduced in Section . . In that context, sparse indicated a network with an average degree much smaller than its size n, while here a graph is sparse if its average degree stays nite as n goes to in nity. The two ideas are somewhat related, but they are not coincident. From a practical viewpoint, all networks are nite and one could

On the asymptotic de nition of sparsity vs the network-related concept of sparsity question the e cacy of a de nition that only holds in the asymptotic regime. How to state for nite n whether a network is sparse or not?

The concept of sparsity (and of density) in the asymptotic limit is much more related to random matrix theory (RMT) than it is to network theory. It has nonetheless to be noted that also the asymptotic de nition of sparsity is not unique, with some authors (see e.g. [BGBK+ ]) that would de ne sparse a graph in which c = o n (log (n)). In agreement with some relevant works on CD on sparse graphs [Dec+ ; Krz+ ; BLM ] and the fact that RMT naturally comes into play in our work, we adopt the asymptotic de nition of sparsity, imposing c = O n (1). We will explain however that this choice is not compromising, showing that our proposed algorithms do not rely on the speci c de nition of sparsity.

To conclude this section concerning the main properties of ER random graphs, we introduce the locally tree-like structure that characterizes the Locally tree-like structure sparse regime. Let us denote with G i (V, E ) a rooted graph, i.e. a graph in which a particular node i ∈ V (the root) is speci ed. Denote with B i (t) the ball of radius t around the node i, i.e. the sub-graph made by the set of all nodes that can be reached from i in at most t steps and the corresponding edges. If the law of B i (t) under uniformly random sampling of the root admits a limit L, then we call it local weak limit [Sal ]. In words, L is the asymptotic local distribution of G(V, E ) as seen from a random vertex. As a consequence of this property, a sparse ER graph locally looks like a tree and hence, with high probability, there are no cycles of nite size. Figure . displays an example of a Poisson GW tree, rooted at i.

The tree-like approximation of sparse graphs will be of fundamental use in the following but it raises an important question when compared to realworld graphs. We argued earlier that real networks are sparse in the sense that the average degree is very small compared to n and that it is convenient to introduce an asymptotic concept of sparsity in prevision to the applications to RMT. As a "side e ect" however, the resulting ER graph turns out to Real networks are sparse but not tree-like be locally tree-like. This property is not necessarily met in real networks in which, on the opposite, many short cycles may be present. We will be oblivious of this major di erence for most of the follow up discussions in which the tree-like assumption will be heavily exploited. This question will reappear in Chapter in which we will consider the validity of our theoretical approach (based on the tree-like approximation) on real graphs.

The main feature captured by the ER random model is the small world effect which is of major importance in complex networks. As we mentioned already in Section . , the focus of our work is devoted to the problem of node classi cation hence on how the nodes of a graph can be divided into groups, according to the preferential attachment described by the edge conguration . The ER model completely misses this point, since all nodes are "equal" and no reasonable partition can be made. It can rather be seen as good null model: any algorithm should not nd a community structure on Note that in the sparse regime, for n → ∞ the Binomial distribution convergences towards a Poisson distribution. A di erent possible way to encode information concerning the nodes' features, is by assigning a weight to each edge. This case is, for the moment, not considered and its analysis is deferred to Chapter a ER random graph. We now proceed introducing two variations to the ER model that describe random graphs with a community structure.

. .

As a consequence of the homophilic nature of interactions on social graphs, typically real networks are structured in communities [GN ]. A more extensive discussion on communities and the problem of community detection will be treated in Part II. As an example, the plot a of Figure . displays a

The community structure of real graphs real network with a community structure: the node colour code identi es the communities and clearly evidences that connections within communities are more likely than connections across communities. This can be equivalently visualized in terms of the matrix A, as shown in the plot b of Figure . , in which the basis of the adjacency matrix is chosen so that nodes in the same community are close to one another.

The complete homogeneity of ER random graphs makes them unsuited to describe this self-organization behavior and a more involved model is necessary. We introduce a variation of the ER model, which goes under the name of stochastic block model (SBM) [HLL ]. The stochastic block model let C ∈ R k×k be a symmetric matrix with positive elements. The entries of the matrix A ij = A ji are set to one independently at random with probability

P(A ij = 1) = min C i , j n , 1 ,
and are equal to zero otherwise.

The matrix C represents the class a nity matrix and imposes di erent average degrees between nodes in the same community and nodes in different communities, hence reproducing the class structure. The vector π = (π 1 , . . . , π k ) determines the class sizes. Let V a = {i :∈ V : i = a}, then it is easy to see that the expectation of the class size is E[|V a |] = nπ a . Plot c of Figure . shows a realization of a graph generated from the SBM, while the plot d displays the corresponding adjacency matrix.

The SBM inherits many of the properties of the ER model: the small world e ect, the degree distribution and the convergence to a Poisson GW tree in the sparse regime. In particular, we now de ne a GW tree with labels: for each node with label a, a Poisson random number with mean C ab π b of sons will be assigned to class b [MNS , for istance]. For example, in the case of two communities of equal size (π = 1 2 /2), the expected number of neighbours of a node in class a is C aa nodes in class a, and C ab nodes in class b, with C aa > C ab . A toy representation of this process is shown in Figure . . As we just mentioned, the SBM, like the ER, has a very regular degree distribution, not adapted to model real-world networks. To x this limitation, the degree corrected stochastic block model (DCSBM) [KN ] is introduced which, in addition to generating a class structure also allows one to have an arbitrary node degree distribution.

De nition . (Degree corrected stochastic block model). Let ∈ {1, . . . , k} n be the class label vector, where k is the number of classes and P( i = a) = π a and let C ∈ R k×k be a symmetric matrix with positive elements.

The degree corrected stochastic block model

Let θ ∈ Θ = [θ min , θ max ] be a random variable that encodes the intrinsic node connectivity, distributed according to ν,

satisfying Θ dν(θ) = 1 (nor- malization), E[θ] = Θ θ dν(θ) = 1, E[θ 2 ] = Θ θ 2 dν(θ) ≡ Φ = O n (1).
For each node, θ i is drawn independently at random from the distribution ν.

The entries of the matrix A ij = A ji are set to one independently at random with probability

P(A ij = 1) = min θ i θ j C i , j n , 1 ,
and are equal to zero otherwise.

From a simple computation, one can see that the expected average degree of node i is proportional to θ i , i.e. E[d i ] ∝ θ i . Consequently the vector θ = (θ 1 , . . . , θ n ) can be used to produce any degree distribution on the graph.

For the DCSBM we established in [DCT a] the percolation threshold in Percolation threshold in the DCSBM the particular case in which the expected average degree is the same for all classes. The result is formally stated as follows:

Theorem . (Percolation threshold in DCSBM, [DCT a]). Consider a graph G(V, E ) generated according to the DCSBM procedure as in De nition . . Let Π = diag(π 1 , . . . , π k ) and assume that the constant c is the largest eigenvalue of CΠ with eigenvector 1 k . Then, for all large n, with high probability, G(V, E ) has a giant component if and only if cΦ > 1.

From a simple calculation, one obtains that c is the expected average degree of G(V, E ) and Theorem . can be straightforwardly related to Property . . The proof of Theorem . is provided in Appendix A.

To conclude this Section, let us recapitulate the main properties of the DCSBM. The DCSBM captures the small world e ect typically observed in real networks and is suited to produce a graph with an arbitrary degree distribution (hence even a broad, power law distribution). The typical organization into communities of real networks is also correctly described by the DCSBM. This model however inherits, in its sparse version for C ab = O n (1), a locally tree-like structure from the ER and the SBM. As we mentioned already, this property will be heavily exploited in the remainder and the legitimacy of ignoring this discrepancy will be further discussed in Chapter . This concludes our introduction to graphs, leaving place to the next chapter on statistical physics. In particular, we will introduce the Ising model on a graph, in which to each node is associated a random variable s i called spin.

This time the size of each labelled progeny is a Poisson with mean C ab π b Φ [GLM ].

T H E I S I N G M O D E L O N S P A R S E G R A P H S Abstract

This chapter provides an introduction to statistical physics, with a particular attention to the Ising model describing interacting spins placed in correspondence to the nodes of a graph. This model is studied with the help of two closely related approximations (namely belief propagation (BP) and the Bethe approximation) that are asymptotically exact on locally tree-like graphs such as the sparse realizations of the degree corrected stochastic block model (DCSBM). The tools introduced in this chapter will be extensively exploited in Parts II, III. Statistical physics plays an important role in shaping the technical parts of this manuscript in terms of methods, approximations and intuitions. In fact, statistical physics has signi cantly contributed to the analysis and design of new (and old) algorithms for machine learning and, more in general, for statistical inference [Car+ ; Iba ; ZK ; OS ; MM , for instance]. Speci cally, parts II and III show how Ising-like models [Isi ] on graphs can be exploited to interpret and improve e cient algorithms for spectral clustering (SC). The role of this chapter is hence to provide the reader a basic description of the methods, approximations and interpretations of Ising-like models on graphs.

. Statistical physics has its origin back in the second half of 19 th century when a lot of e ort was conveyed by the scienti c community to give a mechan-ical interpretation to thermal phenomena [Bru a]. The laws of thermodynamics describe the behavior of (almost in nitely) many bodies (particles, molecules...) from a coarse grained perspective, with only few parameters such as the pressure and the temperature. These laws rely on axioms that do not invoke any mechanical principle, using instead the concept of heat. Although not formally de ned, back in the 19 th century, there was a general agreement that heat could be intended as a form of movement [ Bru b]. As a historically relevant example [Max ], let us consider the classical free gas problem to provide a practical case-study. With Newton laws it is in principle possible, given a model describing the gas molecules interactions, to compute the trajectories of each molecule across time. In practice, it turns out that this is more of a mental experiment than other, since the typical number of molecules inside an average size room is of order 10 23 : for this magnitude of the system, any analytical or numerical solution of New-

The example of the free gas ton's equations is unapproachable, even with powerful modern computers.

To the impossibility of solving such a large system of equations, one needs to add that of measuring the initial conditions of the system (i.e. positions and velocities of all 10 23 particles) needed to solve Newton's equations.

A mechanical study of the free gas seems consequently unapproachable. According to thermodynamics instead, only few parameters are su cient to describe the free gas: the pressure, temperature, volume and number of molecules [Cla ]. This suggests that the information encoded by the trajectories is redundant and way too rich: many di erent gas con gurations lead to the same measurable macroscopic quantities. This evidence required a new perspective relating the microscopic (where Newton's law are de ned) to the macroscopic (where thermodynamics lays) worlds. It was Boltzmann who gave the main contributions in the outbreak of the probabilistic interpretation of thermodynamic [Gib ]. In this picture,

From trajectories

to probability distributions the system as a whole is considered from a statistical viewpoint, and the goal is determining the probability distribution (rather than the exact values) of the position and velocity of molecules. This way, only few, measurable parameters are su cient to describe the system, relating the microscopic scale interpretation to macroscopic thermodynamics observables [Par ].

Let us now formalize the probabilistic approach introduced by Boltzmann, rst de ning the phase space. Let n be the number of items composing the system and let d f be the number of degrees of freedom each item has. In the free gas example, n is the number of molecules and d f = 6 are the three coordinates of space and the three coordinates of velocity. Each gas con guration is encoded by the vector

f = { f 1 1 , . . . , f 1 d f , . . . , f n 1 , . . . , f n d f } containing all the
coordinates of all degrees of freedom of each item. The space Ω in which f lives is called the phase space. A con guration of the system corresponds to a point in Ω and its evolution to a trajectory in Ω. Supposing to be at thermal equilibrium, each con guration f ∈ Ω is visited with the following probability

µ β ( f ) = e -βH( f ) Z β . ( . )
This is the so-called Boltzmann distribution. The parameter β = (κ B T) -1

The Boltzmann distribution is the inverse temperature, where κ B is the Boltzmann constant and T the temperature of the system. The function H( f ) is called Hamiltonian and corresponds to the energetic cost of each con guration f . Finally Z β is the normalization constant that, assuming that the variables f live in a discrete space, is written as follows

Z β = ∑ f ∈Ω e -βH( f ) .
Averages over the Boltzmann distribution ( . ) are denoted with • β . Taking a closer look at Equation ( . ), each con guration f ∈ Ω is visited with a probability that depends on its energetic cost, favouring small energetic con gurations and its expression gives a microscopic interpretation of the system. Three facts should be underlined at this point:

• all con gurations with nite energy can be visited with non-vanishing probability;

• the temperature determines the rate of exploration: for large temperatures (small β) high energy con gurations are more likely to be visited;

• each observable quantity is a function of the system con guration. Now that the main concepts have been laid out, we de ne and study the Ising model [Isi ] which was developed to provide a microscopic interpretation to spontaneous magnetization and is one of the most studied and well understood problems in statistical physics.

. Some materials, called ferromagnetic, share the peculiar property of having a spontaneous magnetic behaviour in absence of an external magnetic eld when taken below a critical, material dependent, temperature T c , also known

The thermal equilibrium corresponds to a condition of the system such that the observables are not evolving with time. Note that β has the dimensionality of an inverse energy, but for simplicity we refer to it as inverse temperature. This assumption is made for simplicity as all the results can be reformulated in the continuous limit using integrals instead of sums. In the following all considered con gurations spaces are indeed going to be discrete, justifying the reason of this choice. as Curie temperature [CG ]. Above T c , instead, no magnetic property is observed. This behavior de nes a phase transition between the ferromagnetic phase at T < T c in which the magnetic behavior is observed and the paramagnetic phase at T > T c in which no magnetic behaviour exists. An experimental plot of this phase transition is shown in Figure .

The ambition of the Ising model is to provide a microscopic interpretation of the origin of spontaneous magnetization. The idea is to suppose that a ferromagnetic material is composed by a large number of interacting dipoles, each one with its own magnetization, called spins, s i ∈ {±1}. The magnetization of the material is given by the sum of all these spins: if they are randomly arranged, then no magnetization appears, while it does otherwise. Referring to Section . , in this case the feature vector is f ≡ s and the phase space is Ω = {±1} n . The summation is denoted ∑ s∈Ω ≡ ∑ s for convenience. The Hamiltonian, appearing in the Boltzmann distribution ( . ), reads for the Ising model on an undirected graph G(V, E )

The Ising Hamiltonian

H(s) = -∑ (ij)∈E J ij s i s j -∑ i∈V h i s i , ( . )
where the term J ij models the interaction between the spins s i and s j and h i is a local magnetic eld applied to the spin s i . Equation ( .) is actually the most generic form of the Hamiltonian: the problem originally solved by Ising considers a one dimensional spin chains with:

Four spins of a d Ising chain

h i = h ∀ i ( . ) J ij = J δ j,i+1 + δ j,i-1 , ( . )
for J > 0. According to Equations ( . , . ) there is an energetic gain for neighbouring spins to be aligned rewarding them with a negative energy

The expression "Ising Hamiltonian" for arbitrary couplings and elds is a slight abuse of language that is made to indicate that the spins are s i ∈ {±}.

contribution, while penalizing neighbouring spins with opposite orientation with a positive energy contribution. The external eld h polarizes all spins towards the same direction. The lowest energy con guration of the Hamiltonian ( . ) (called the ground state) under the constraint ( . ) is hence obtained for s = 1 n , corresponding to the ferromagnetic con guration in which all spins are aligned.

To determine whether or not this model predicts spontaneous magnetization, it is necessary to study the explicit expression of the magnetization m = s β . Sadly for Ising, in absence of external magnetic eld (h = 0), no magnetization appears in one dimension. The model can however explain spontaneous magnetization in dimension greater or equal than two. To obtain the analytical expression of m it is necessary to compute the dimensionless free energy F β , de ned as follows.

De nition . (Dimensionless free energy). Let Z β be the normalization constant appearing in Equation ( . ). The dimensionless free energy is log (Z β ) is a moment generating function of the Boltzmann distribution

F β = -log Z β .
The term dimensionless comes from the fact that F β , so de ned does not have the physical dimensionality of an energy. The magnetization is obtained taking a derivative of F β with respect to the external eld:

m i = s i β = 1 Z β ∑ s s i e -βH(s) = 1 βZ β ∂ h i ∑ s e -βH(s) = 1 βZ β ∂ h i Z β = 1 β ∂ h i log Z β = - 1 β ∂ h i F β .
So, computing analytically the free energy corresponds to the main goal and it is this function that determines the state of the system. The energy F β can alternatively be written in a form known from classical thermodynamics. Denoting the average energy with U = ∑ s H(s)µ β (s) and the entropy with S = -κ B ∑ s µ β (s)log µ β (s), then, from a straightforward calculation

F β = β(U -TS).
At low T, the con gurations with small energy are favoured (see also Equation ( . )), while as the temperature increases, the entropic term gains more weight and high energy con gurations are more likely to be observed. The temperature balances these two terms causing a phase transition between the ordered ferromagnetic con guration at low T to a disordered paramagnetic con guration at high T. The whole problem of computing m and consequently also T c hence boils down to nding an explicit expression to Z β . Unfortunately, this problem can be solved for only few choices of J, such as (for instance) those corresponding to the -dimensional Ising chain [Isi ], the -dimensional square lattice [Ons ], the fully connected Curie-Weiss model [KPW ], the Bethe lattice [Tho ]. When more involved geometries intervene and J corresponds to the (weighted) adjacency matrix on an arbitrary graph G(V, E ), the exact computation of the partition function Z β becomes intractable [DGM ] and approximate methods must be adopted to obtain the result.

The interest in studying the Ising model for more complex geometries and interactions comes from the fact that it represents the perhaps most elementary way to model pairwise interactions. Its applicability is consequently not bounded to the explication of spontaneous magnetization, but has found several applications in a variety of elds ranging from optimization problems, computer science, biology, sociology and economics [CFL ; SN ]. For this reason, we consider in the remainder the problem of how to determine whether or not the system is in the paramagnetic phase in the case in which J is a weighted adjacency matrix of a graph. The focus will be on two methods, namely the Bethe approximation and the BP algorithm, to study the Ising model on graphs.

From now on we assume that J is the weighted adjacency matrix of a graph G(V, E ) (as sketched in Figure . ) with h i = 0 for all i.

In the case of a ferromagnetic Ising Hamiltonian in which J ij = J > 0 for all (ij) ∈ E , only two phases exist: the paramagnetic and the ferromagnetic. In this case, determining the stability condition of the paramagnetic phase is equivalent to determining the Curie temperature. For other choices of J, the paramagnetic phase always exists, but the ferromagnetic phase does not necessarily appear and other phases may emerge, instead. This case will be further considered in Section . and in Chapter .

. In this section we introduce two approximate methods to study Ising-like models on tree-like weighted graphs. Both approximations are introduced in a discrete phase space more general than Ω = {±1} n , but deeper considerations on the model and methods on how to determine the emergence of the paramagnetic phase are made for Ising variables.

The two approximations here considered are the cavity method, also known in the computer science literature under the name of belief propagation (BP) [MPV ; Pea ; YFW+ ] and the variational Bethe approximation [Bet ; Pei ]. The two methods are adapted to sparse, locally tree-like graphs and introduce the non-backtracking and Bethe-Hessian matrices, of central importance to Parts II and III, providing them with a physical interpretation.

. . Belief propagation is an iterative algorithm used to compute the marginal probabilities of a factorized probability distribution, such as µ β (s). Lemma . in particular is the main result on which BP algorithm builds, providing an exact expression of the node and edge marginals of a probability distribution de ned on a tree [WJ , for instance].

Lemma . (Belief propagation on a tree). Let G(V, E ) be a tree and let µ(s) be a probability distribution de ned on G(V, E ) satisfying

µ(s) = ∏ (ij)∈E φ ij (s i s j ), ( . )
with s ∈ {1, . . . , k} n . Then the edge marginal µ ij (s i , s j ) = ∑ s\s i ,s j µ(s) and the node marginal µ i (s i ) = ∑ s\s i µ(s) can be written in the following form:

µ i (s i ) = ∏ k∈∂i η ki (s i ) ( . ) µ ij (s i s j ) = φ ij (s i s j ) ∏ k∈∂i\j η ki (s i ) ∏ l∈∂j\i η lj (s j ). ( . )
The quantities η ij (s j ) are de ned on the set of directed edges E d and are called messages.

From Equation ( . , . ), exploiting µ i (s i ) = ∑ s j µ ij (s i , s j ), it is obtained that the messages have to satisfy the following xed point equation Message passing

η ji (s i ) = ∑ s j φ ij (s i , s j ) ∏ l∈∂j\i η lj (s j ).
( . )

Solving Equation ( . ) and then injecting the result in Equation ( . , . ), one obtains the node and edge marginal distributions. Of course, Lemma . is relevant for determining the magnetization of the Ising model on a tree, since the Boltzmann distribution µ β (s) can precisely be written in the form of Equation ( . ). Lemma . however only applies to trees and not to graphs with cycles for which loopy BP should be instead considered. Before detailing the extension of BP to the case of graphs with cycles (hence that are not trees) we sketch here the proof of Lemma . since it is very pedagogical and helpful to understand the essence of BP algorithm.

Proof of Lemma . . Consider an directed edge (ij) ∈ E d and let E ij be dened as follows

E ij = {(kl) ∈ E : ∃ P NB (ij),(kl) ∧ P NB (ij),(lk) },
where P NB e,e is a non-backtracking path from for e to e . By convention, if no such path exists, then P NB e,e = ∅, while (ij) ∈ E ij . In words, E ij is the set of all undirected edges that can be reached from i only passing through j. As a consequence of the fact that on a tree there exists a unique path connecting any two nodes, the two following properties are veri ed:

∀ i ∈ V, E = k∈∂i E ik ; ( . ) ∀ (ij) ∈ E , E = {(ij)} ∪ k∈∂i\j E ik reached from (ji) ∪ l∈∂j\i E jl reached from (ij) . ( . ) Furthermore, note that ∀ j = k, E ij ∩ E ik = ∅.
A pictorial representation of the de nition of E ij is given in Figure . . Exploiting Equation ( . ), µ(s) can then be written as:

µ(s) = ∏ (ab)∈E φ ab (s a , s b ) = ∏ k∈∂i ∏ (ab)∈E ik φ ab (s a , s b ) ≡ ∏ k∈∂i ψ ik (s V ik ),
where V ik is the set of nodes connected by edges in E ik (i included). The node marginal can then be written in the following form

µ i (s i ) = ∑ s\s i µ(s) = ∑ s\s i ∏ k∈∂i ψ ik (s V ik ) = ∏ k∈∂i ∑ s V ik \s i ψ ik (s V ik ). Denoting η ji (s i ) = ∑ s V ik \s i ψ ik (s V ik )
, we obtain the rst equation of Lemma . . Note that η ji (s i ) indeed only depends on s i since the sum is run over all spins s V ik , except s i . Proceeding in a similar way, the expression of the edge marginal is obtained from Equation ( . ).

The essence of the proof of Lemma . relies on the conditional independence of the node variables on trees that we already discussed in Chapter . More speci cally, to obtain Equation ( . ), one could imagine to remove the node i, obtaining d i (the degree of i) disconnected sub-graphs in which variables are independent and hence factorize. Similarly Equation ( . ) is obtained removing the nodes i and j from the graph.

The proof of Lemma . hence heavily exploits the conditional independence and the fact that for any pair of nodes, only a single path connecting them exists, since there are no cycles. But what happens when cycles are present? It was stated in Chapter that sparse ER, SBM, DCSBM graphs are lo-BP on graphs with cycles cally tree-like, i.e. with high probability they contain cycles but of no nite size. Consequently, removing a node i from G(V, E ) one obtains d i subsets of nodes that are not independent (since they are not disconnected) but that are only weakly dependent. This intuition is at the basis of loopy BP that consists in approximating the marginals in the following way:

η ji (s i ) = Z i Z ji ∑ s j φ ij (s i , s j ) ∏ l∈∂j\i η lj (s j ). ( . ) µ i (s i ) ≈ 1 Z i ∏ k∈∂i η ki (s i ) ( . ) µ ij (s i , s j ) ≈ 1 Z ij φ ij (s i , s j ) ∏ k∈∂i\j η ki (s i ) ∏ l∈∂j\i η lj (s j ), ( . )
where Z i , Z ij are the normalization constant and, we recall, the ≈ sign comes from the fact that G(V, E ) is not a tree.

The application of the BP algorithm of graphs with cycles is known to give, in general, very good performances [MWJ ], but even more interestingly, it has been shown that on locally tree-like graphs (such as sparse ER) the estimation of the marginals is asymptotically exact [DM+ b]. To con rm this result, Figure . shows the average magnetization m = 1 n 1 T m as a function of the inverse temperature β for a ferromagnetic Ising model with J = JA, with A the adjacency matrix of a ER sparse graph (De nitions . and . ). The prediction of BP is compared to the direct computation of m achieved by Monte Carlo method [Has ], evidencing an almost perfect agreement.

To conclude this section, we wish to show how the stability of the paramagnetic phase can be estimated from BP on a Ising model (for which we recall that s i ∈ {±1}) and how BP equations are related to the non-backtracking High temperature expansion of BP matrix introduced in De nition . . To do so, we consider the so-called paramagnetic xed point of Equation ( . ), corresponding to the case in which the messages are constant, i.e., independent of s. This corresponds to the paramagnetic phase, since, from Equation ( . ) also the node marginal is independent of s, hence m = 0 n . Letting η ji (s i ) = ∑ s j φ ij (s i , s j )(1 + ij s j ), one obtains, to rst order in

λ ↓ |•| 1 (BU β c ,J ) = 1,
where λ

↓ |•| 1 (•)
denotes the eigenvalue of a matrix with largest modulus. When β < β c , then the paramagnetic phase is stable, since iterating Equation ( . ), the perturbation vanishes, while it is not for β > β c . The spectral properties of BU β are hence fundamental to determine the value of β c and are considered in Chapter .

We now consider the second method adapted to sparse tree-like graphs, namely the Bethe approximation.

. . An alternative method to estimate the averages on factorized probability distribution on sparse graphs relies on a variational approach. We recall that the ultimate goal we have is to obtain an analytic expression of the free energy F β a task that is, in general, not feasible. The basic idea of variational approaches is to substitute the expression of µ β (s) which makes the problem intractable, with another distribution p q (s) that can be treated analytically. The vector q represents a set of parameters on which p q (s) depends and q should be chosen so to obtain the best approximation of F β , given the expression of p q . Explicitly this is done introducing the variational free energy Fβ (q) as in Equation ( . ):

Fβ (q) = ∑ s p q (s) βH(s) + log p q (s) . ( . )
When can immediately see that, for any distribution p q (s), Fβ (q) ≥ F β : in

The variational free energy fact, exploiting βH(s) = F βlog µ(s), one obtains

Fβ (q) = F β + ∑ s p q (s) log p q (s) µ β (s) = F β + D KL (p q µ β ) ≥ F β ,
where D KL (• •) is the Kullback-Leibler divergence which is always greater or equal to zero [Cov ]. Finding the minimum of Fβ (q) with respect to q is the same as minimizing the Kullback-Leibler divergence, which is notoriously a measure of resemblance of two probability distributions. arg min q Fβ (q) = arg min q D KL (p q µ β ).

The rationale behind the Bethe approximation [Bet ; Pei ] is well explained by the following corollary of Lemma . . Corollary . (Factorization of µ(s) on a tree). Under the same hypothesis of Lemma . , the probability distribution µ(s) can be written as

µ(s) = ∏ (ij)∈E µ ij (s i , s j ) ∏ i∈V [µ i (s i )] 1-d i ,
where d i is the degree of node i.

The proof of Corollary . is straightforward from Lemma . . The Bethe approximation hence uses as p q (s) a distribution that factorizes as in Corollary . . Once again, this approximation is exact on trees and it is expected to be accurate in sparse graphs. Explicitly, we obtain: ) where mi = ∑ s p Bethe m, χ (s)s i , while χij = ∑ s p Bethe m, χ (s)s i s j . from this expression the variational Bethe free energy reads

p Bethe m, χ (s) = ∏ (ij)∈E 1 + mi s i + mj s j + χij s i s j 4 ∏ i∈V 1 + mi s i 2 1-d i , ( .
FBethe β ( m, χ) = -∑ (ij)∈E βJ ij χij + ∑ (ij)∈E ∑ s i s j f 1 + mi s i + mj s j + χij s i s j 4 + ∑ i∈V (1 -d i ) ∑ s i f 1 + mi s i 2 , ( .
)
where f (x) = x log(x). To nd the "stable" con gurations of m, χ it is then necessary to minimize the variational free energy with respect to its arguments. Interestingly, the xed points of BP are minima of the Bethe free energy [MPV ; WF ; Hes+ ] drawing a tight relation between the two methods that one could already guess from the relation between Corollary . and Lemma . . This relation is well explained using the Bethe free energy to determine the stability condition of the paramagnetic phase, introducing a protagonist of this manuscript: the Bethe-Hessian matrix. To accomplish this task, we must verify under what conditions m = 0 n is a minimum of the variational free energy, so we rst impose the gradient of FBethe β ( m, χ) to be equal to zero in m = 0 n .

0 = ∇ m FBethe β ( m, χ) m=0 n 0 = ∇ χ FBethe β ( m, χ) m=0 n .
The rst condition is always veri ed, while the second imposes χij = th(βJ ij ).

To determine whether or not m = 0 n is a minimum of the free energy, we need to compute the Hessian matrix in m = 0 n that reads [WF ] The weighted Bethe-Hessian matrix

H β,J ij ≡ ∂ 2 FBethe β ( m, χ) ∂ mi ∂ mj m=0 n = δ ij 1 + ∑ k∈∂i th 2 (βJ ik ) 1 -th 2 (βJ ik ) - th(βJ ij ) 1 -th 2 (βJ ij ) . ( .
)
The notation m is used to stress the fact that m is the exact value of s β , while m is the one obtained from the Bethe approximation.

< c

Imag( )

< c = c
Real( ) The presence of negative eigenvalues in the spectrum of H β,J implies that m = 0 n is not the global minimum, hence that the paramagnetic phase is not the stable con guration. Note that, for β su ciently small, m = 0 n is indeed a minimum of the Bethe free energy: in fact H β=0,J = I n . In order to determine the condition under which H β,J displays negative eigenvalues, it is useful to introduce the following de nition of the generalized Bethe-Hessian of which H β,J constitutes a particular case.

Imag( ) = c
De nition . (Generalized Bethe-Hessian matrix). Given a graph G(V, E ), a vector ω ∈ R |E d | so that ω e = ω e -1 and a parameter r ∈ C \ {±ω e } e∈E d , the generalized Bethe-Hessian matrix H ω (r) ∈ C n×n is de ned as

(H ω (r)) ij = δ ij 1 + ∑ k∈∂i ω 2 ik r 2 -ω 2 ik - rω ij r 2 -ω 2 ij .
The matrix H β,J then simply corresponds to the choice r = 1 and ω ij = th(βJ ij ). The matrix H ω (r) has an important relation with the weighted nonbacktracking matrix introduced in Section . . , as stated by the following theorem [WF ; SMM ]:

Theorem . (Watanabe-Fukumizu formula). Let B be the non-backtracking matrix of a graph G(V, E ) and let ω e ∈ R be a weight associated to each edge in E d , with ω e = ω e -1 . Denote with U ω ∈ R |E d | the diagonal matrix with entries U ω,ee = δ e,e ω e . Letting H ω (r) be the generalized Bethe-Hessian matrix of De nition . , the following relation holds for all r ∈

C \ {±ω ij } (ij)∈E d det rI |E d | -BU ω = det [H ω (r)] ∏ (ij)∈E r 2 -ω 2 ij .
This theorem is a generalized form of the Ihara-Bass formula [Ter ] which is obtained for ω ij = 1 for all (ij). Exploiting Theorem . , then one obtains the following relation

λ ↓ |•| 1 (BU β c ,J ) = 1 =⇒ det[H β c ,J ] = 0,
where we recall that β c is the prediction of transition temperature according to BP. The two approximations hence give the same prediction of the transition temperature, since β c is the smallest β for which det[H β,J ] = 0. This behavior is schematized in Figure . in which the spectra of the matrices B and H β,J are shown for two di erent values of β.

To conclude, in this chapter we introduced the basics of statistical physics and of Ising-like models, detailing two methods of fundamental importance for the analysis of the Ising model on sparse graphs. The study of the stability of the paramagnetic phase allowed us to naturally introduce two deeply related matrices that are the non-backtracking and the Bethe-Hessian. We showed that the spectral properties of these matrices are fundamental to draw conclusion on the transition temperatures and a more detailed study of these properties is the object of the next chapter. In this chapter the spectral properties (i.e. the behavior concerning the set of eigenvalues and eigenvectors) of the non-backtracking and Bethe-Hessian matrices introduced in the previous chapters are studied, with a particular attention to the case of large random graphs generated from the degree corrected stochastic block model (DCSBM). The concepts and results introduced in this chapter form a bridge between Chapter in which these matrices were given a physical interpretation and the subsequent chapters of Part II and III in which they will be exploited to analyze and propose algorithms for spectral clustering (SC). The objective of this chapter is hence twofold: on one side it is to give the rationale behind SC from a random matrix theory (RMT) perspective; on the other side it is to introduce the main existing results from the literature of RMT that will be exploited in the remainder.

. Let us rst of all lay out some de nitions concerning matrix eigenvalues with a particular focus to large random matrices. In the following, λ i (M) denotes an arbitrary eigenvalue of M. Hermitian matrices have only real eigenvalues that, consequently, can be ordered [Bha ]. We denote in particular with λ ↑ i (M) the i-th smallest eigenvalue of M and with λ ↓ i (M) the i-th largest. For non-Hermitian matrices, eigenvalues may have a non-zero imaginary part and λ ↑/↓ |•| i (M) denotes the i-th smallest/largest eigenvalue in modulus. We now de ne the empirical spectral measure μM (λ) of a matrix [CD ].

De nition . (Empirical spectral distribution). Given a square matrix M ∈ C n×n the empirical spectral measure of its eigenvalues is de ned as

μM (λ) = 1 n n ∑ i=1 I λ=λ i (M) ,
where we recall that I x is the indicator function. In the particular case in which M is a Hermitian matrix, hence λ i (M) ∈ R for all i, μM (λ) is the histogram of the eigenvalues of M. The de nition of the empirical spectral measure allows us to introduce the fundamental concept of isolated and bulk eigenvalues on large random matrices.

De nition . (Bulk and isolated eigenvalues). Let M ∈ C n×n be a random matrix so that, for all large n with high probability, the empirical spectral measure of M converges to a limit measure, i.e. μM (λ) -→ n→∞ µ(λ).

Denoting with supp(µ) the support of the limiting measure and with λ max = max

λ∈supp(µ)
|λ|, all eigenvalues λ i (M) that satisfy

min λ∈supp(µ) λ -λ i (M) λ max = o n (1)
are said to be bulk eigenvalues, while all others are isolated eigenvalues.

Looking at De nition . , a non vanishing value of μM (λ) implies, in the large n limit, that many eigenvalues of M are almost degenerate. When an eigenvalue of M is isolated instead, it is found at a macroscopic distance from all others and it does not contribute to μM (λ). This de nition is of SC is based on the existence of isolated eigenvalues crucial importance for the remainder, because SC on large random matrices relies on the exploitation of the isolated eigenvalues of a properly chosen matrix. In fact, in simple words, the eigenvectors associated to the isolated eigenvalues of M are related to the expectation of M, while the bulk eigenvalues to its noisy part, as it will be shown in the remainder of this chapter in which this statement will be made more rigorous. Figure . shows a toy representation of the bulk and isolated eigenvalues of a matrix, together with the corresponding empirical and limiting spectral measures. We now get to the detailed study of some spectral properties, relevant to the results introduced in the next chapters, of the non-backtracking and Bethe-Hessian matrices.

. First of all, let us recall the de nition of the non-backtracking matrix B on a graph G(V, E ).

The matrix B ∈ R |E d |×|E d | is de ned on the set of directed edges of G(V, E ) and reads ∀ (ij), (kl) ∈ E d , B (ij),(kl) = δ jk (1 -δ il ).
( . )

The non-backtracking matrix hence is non-symmetric also on undirected graphs and consequently its eigenvalues are not necessarily real. In Chapter we also introduced the weighted version of the non-backtracking matrix, obtained from the linearization of belief propagation (BP) equations, that can be written as BU ω , where

U ω = diag(ω) ∈ R |E d |×|E d | and ω ∈ R |E d |
is a vector encoding the weight of each directed edge of G(V, E ).

We rst initiate our introduction of the spectral properties of the nonbacktracking matrix, limiting ourselves to the unweighted case, to then make further considerations to the more general weighted case.

. .

The adoption of the non-backtracking for spectral clustering (SC) was proposed in [Krz+ ] in which the authors pioneered a method to conjecture some of the spectral properties of B on large sparse stochastic block model (SBM) (De nition . ) random graphs with k = 2 communities of equal size. We report here the claim formulated in [Krz+ ], together with the heuristic method to derive it that will also be adopted in the subsequent chapters.

Claim

λ ↓ |•| 1 (B) = c + o n (1) λ ↓ |•| 2 (B) = c in -c out 2 + o n (1).
All other eigenvalues satisfy |λ

↓ |•| i≥3 (B)| ≤ √ c + o n (1).
The assumption (c inc out )/2 > √ c is technical and is needed to ensure that λ

↓ |•|
2 (B) is isolated. We will come back to this hypothesis in Chapter , explaining its deep meaning and relation to information theory. For the moment, however, we treat it as a technical requirement needed to obtain two isolated eigenvalues in the spectrum of B (see Figure . ) and proceed detailing the intuitions leading to the conjecture of [Krz+ ].

A realization of a sparse SBM locally converges to a tree for which random variables de ned on the nodes of the graph are conditionally independent [GLM ]. To exploit this property, we express the probability distribution of the label of a neighbour of an arbitrarily chosen node i, conditionally to i using Bayes theorem , Bayes theorem P(A|B) = P(B|A)P(B) P(A)

P( j | i , A ij = 1) = P(A ij = 1| i , j ) ∑ j P(A ij = 1| i , j ) = C i , j 2c . ( . )
So, for each node j in the neighbourhood of i, i = j with probability c in /(c in + c out ) and i = j with probability c out /(c in + c out ). Moreover,

A node in class a has C ab /2 neighbours in class b on average i the variables j for j ∈ ∂i are asymptotically independent conditionally to i due to the locally tree-like structure of G(V, E ). These facts are at the base of Claim . whose validity is argued in tree steps: (i) de ne a guess eigenvector of B; (ii) show that in expectation (over Equation ( . )) it is an eigenvector of B; (iii) determine under what conditions the variance of the guess eigenvector vanishes, hence guaranteeing the goodness of the approximation of guess eigenvector with its expectation.

First of all, the guess eigenvector g (r) of B is de ned , with r an integer guaranteeing that, for any node i ∈ V, the sub-graph formed by all nodes that are at most r steps away from i is tree-like.

The guess eigenvector ) where {(jk) : d(jk, ωx) = r} is the set of directed edges (jk) such that the shortest non-backtracking path connecting (jk) to (wx) is of length r and

g (r) ij = 1 λ r ∑ (ωx) : d(jk,ωx)=r k =i σ x , ( .
σ x = 1 if x = 1 and σ x = -1 if x = 2.
The sum in Equation ( . ) runs over an exponentially large (in r) number of terms hence, for r su ciently large, one expects good concentration properties of g (r) . From a simple calculation, the following identity is veri ed "Almost" eigenvalue equation

Bg (r) = λg (r+1) , ( . )
which means that g (r) is "almost" an eigenvector of B, provided that the dependence on r is small. Furthermore, λ represents the eigenvalue of this approximate eigenvector equation. Studying the expectation of g (r) , we determine the value of λ so that g (r) depends on r only marginally. For convenience, g (r) is rewritten in the following form:

g (r) ij = 1 λ r ∑ (qω) : d(jk,qω)=r-1 k =i ∑ x∈∂ω\q σ x .
We now take the expectation of the inner term conditionally to σ ω :

E   ∑ x∈∂ω\q σ x   = c in -c out 2 σ ω .
This gives a recursive formula for E[g (r) ij ], in fact

The expectation of the guess eigenvector

E[g (r) ij ] = 1 λ r ∑ (qω) : d(jk,qω)=r-1 k =i c in -c out 2 σ ω = c in -c out 2λ E[g (r-1) ij ] = ... = c in -c out 2λ r σ j ,
which leads to the natural choice λ = (c inc out )/2 that should correspond to an approximate eigenvalue of B as is Equation ( . ). To determine whether or not g (r) is well approximated by its expectation, its variance has to be studied. Let us rst take under analysis the second moment, following a similar procedure to the one adopted in [Mos ] for trees.

The variance of the guess eigenvector

E g (r) ij 2 = 1 λ 2r ∑ (ωx) : d(jk,ωx)=r k =i ∑ (vy) : d(jk,vy)=r k =i E[σ x σ y ] = 1 λ 2r ∑ (ωx) : d(jk,ωx)=r k =i     1 + ∑ (vy) : d(jk,vy)=r k =i,(vy) =(ωx) E[σ x σ y ]     .
Taking the second term under analysis:

∑ (vy) : d(jk,vy)=r k =i,(vy) =(ωx) E[σ x σ y ] ≈ ∑ (vy) : d(jk,vy)=r k =i,(vy) =(ωx) E[σ x ]E[σ y ] ≈ ∑ (vy) : d(jk,vy)=r k =i E[σ x ]E[σ y ],
where we exploited that for x = y σ x is asymptotically independent of σ y and that the number of paths of length r from (jk) is exponentially large in r: for r su ciently large (say r ∼ log(n)) removing the path (jk) → (ωx) is only a minor approximation. One then obtains

V g (r) ji ≈ 1 λ 2r ∑ (ωx) : d(jk,ωx)=r 1 = c λ 2 r ,
hence the variance vanishes whenever λ = (c inc out )/2 > √ c. Comparing this result with Claim . , this is precisely the hypothesis that guarantees λ

↓ |•|
2 (B) to be an isolated eigenvalue of the non-backtracking matrix. Repeating the same procedure, but letting σ i = 1 for all i, one gets that

λ ↓ |•| i (B) = c + o n (1)
as stated in Claim . . This short proof shows that the eigenvalue λ

↓ |•|
2 (B) is associated to an eigenvector that depends on the class structure of G(V, E ) so long that it is an isolated eigenvalue. When

λ ↓ |•| 2 (B) ≤ √
c, instead, this approximation is no longer valid and this eigenvalue is lost among the bulk eigenvalues of B.

The claim of [Krz+ ] attracted a lot of attention from the mathematical community and in [BLM ] Claim . was formally proved and extended to the case of k ≥ 2 communities of arbitrary size. Although formally more rigorous, the idea underlying the proof of [BLM ] follows the steps described above, based on the local convergence of G(V, E ) to a tree. To enunciate the main theorem of [BLM ] let us introduce the following assumption that is formulated under the more general DCSBM hypothesis.

Assumption . . Let C ∈ R k×k be the class a nity matrix and π be the expected class size vector as in De nition . , with Π = diag(π). Further let θ be the vector encoding the degree heterogeneity in the DCSBM and denote

Φ = E[θ 2 i ].
The matrices C, Π satisfy the following hypothesis:

The eigenvalues of CΠ are real, in fact,

for 1 ≤ p ≤ k CΠv p = λ p v CΠ C 1/2 ω p v p = λ p C 1/2 ω p v p C 1/2 ΠC 1/2
Hermitian

ω p = λ p ∈R ω p • CΠ1 k = c1 k , for some positive c, • λ p (CΠ)Φ > √ cΦ for all 1 ≤ p ≤ k.
Let us get deeper into the details of this assumption.

. First, recall from De nition . that all entries of C and π are positive, hence all entries of CΠ are positive as well. This implies that there is a non-null probability of connection between any two classes. As a consequence, one may apply the Perron-Frobenius theorem [JH , Theorems . . , . . ] on CΠ: its eigenvalue of largest modulus, denoted with c, is positive, simple and its corresponding eigenvector is the only one with all positive entries. By de nition λ ↓ 1 (CΠ) = c. . CΠ1 k = c1 k : this assumption imposes that the Perron-Frobenius eigenvector of CΠ is the vector of all ones, 1 k . From De nition . , one can show that this assumption implies that E[d], the expected average degree, is equal to c. This importantly ensures that the average degree does not depend on the class; in fact, the expected degree of each node i belonging to class p equals θ i (CΠ1 k ) p = θ i c, independently of p.

. The last condition ensures that exactly k isolated eigenvalues appear in the spectrum of B.

With this assumption at hand, we are in position to formally state the main result of [BLM ] Theorem . (Spectrum of B on sparse SBM graphs). Let G(V, E ) be a graph generated from the SBM (θ = 1 n ) and satisfying Assumption . for c = O n (1), then the following relations are satis ed with high probability:

For two communities of equal size

λ ↓ |•| 1 (CΠ) = c λ ↓ |•| 2 (CΠ) = c in -c out 2 ∀ 1 ≤ i ≤ k, λ ↓ |•| i (B) = λ ↓ |•| i (CΠ) + o n (1), ∀ i ≥ k + 1, |λ ↓ |•| i (B)| ≤ √ c + o n (1).
Both the results of [Krz+ ] and [BLM ] only consider the SBM, in which a homogeneous degree distribution is assumed. The extension to the case of DCSBM-generated graphs was considered in [GLM a] for the case of k = 2 communities of equal size.

Theorem . (Spectrum of B on 2-class sparse DCSBM graphs). Let G(V, E ) be a graph generated from the DCSBM and satisfying Assumption . for k = 2 and π ∝ 1 2 and c = O n (1), then the following relations are satis ed: Note that, Theorem . holds for two classes of equal sizes, but we believe, motivated by a vast empirical analysis shown in Figure . and Part II, that Theorem . has a straightforward generalization in the case only Assumption . is veri ed, so for k ≥ 2 classes of arbitrary size.

∀ 1 ≤ i ≤ k, λ ↓ |•| i (B) = λ ↓ |•| i (CΠ)Φ + o n (1), ∀ i ≥ k + 1, |λ ↓ |•| i (B)| ≤ √ cΦ + o n (1).
With this result we conclude the section on the spectral properties of B on sparse DCSBM graphs and brie y move to the dense setting.

. .

The main focus of this manuscript is devoted to the sparse regime in which the graph average degree c = O n (1). As it has been previously discussed, however, the asymptotic concept of sparsity is theory-related and its validity is debatable on real, nite size graphs. Even if all intuitions motivating the non-backtracking matrix as well as the analysis of Section . . seem to have as a strict requirement that G(V, E ) has a locally tree-like structure, it appears that the spectral behavior of B on dense DCSBM graphs is very much alike the one obtained in sparse graphs. For this reason, the nonbacktracking matrix constitutes a natural bridge between the two regimes as it will be more extensively commented in Chapter .

From Theorem . , it appears that all eigenvalues r of B di erent from ±1 have to satisfy the following relation on an arbitrary graph G(V, E ).

Quadratic eigenvalue problem

det (r 2 -1)I n + D -rA = 0.

( . )

Here and in the following, the notation θ i ∼ [U (a, b)] γ indicates the following procedure: (i) n random numbers {r i } i=1,...,n are independently drawn from a uniform distribution in a, b;

(ii) each of these number are raised to the power γ; (iii) The solution of Equation ( .) is known as quadratic eigenvalue problem [TM ] and allows one to relate the eigenvalues of B to those of the smaller (unweighted) Bethe-Hessian matrix H r = (r 2 -1)I n + D -rA. Now, let us rst consider a graph G(V, E ) generated from the SBM with average degree c going to in nity faster than log(n). Among the properties of G(V, E ) enumerated in Chapter , we recall here that its degree sequence is almost regular, i.e. that D -cI n = o n (c). It can be shown [WW ] that the values of r solving Equation ( . ) can be approximated by r solving

θ i = nr i / ∑ i r i , so that E[θ i ] = 1.
det[(r 2 -1 + c)I n -rA] = 0.
( . )

To obtain Equation ( . ) from Equation ( . ), D has been replaced by cI n . The solution of Equation ( . ) is easier to analyze, since it trivially relates r On regular graphs, the eigenvalues of B are directly related to the ones of A to the eigenvalues of A:

r = λ i (A) ± λ 2 i (A) -4(c -1) 2 . ( . )
Considering the simplest case of two classes of equal size (as in Claim . ) the eigenvalues of A satisfy the following identities with high probability (for instance [CZ , Proposition . ])

λ ↓ |•| 1 (A) = c + o n (c) λ ↓ |•| 2 (A) = c in -c out 2 + o n (c) |λ ↓ |•| i≥2 (A)| ≤ 2 √ c + o n ( √ c).
Theorem . (Spectrum of B on 2-class dense SBM graphs). Let G(V, E ) be a graph generated from the SBM and satisfying Assumption . for k = 2, π ∝ 1 2 and c/log(n) → ∞, then the following relations are satis ed with high probability

∀ 1 ≤ i ≤ k, λ ↓ |•| i (B) = λ ↓ |•| i (CΠ) + o n (c), ∀ i ≥ k + 1, |λ ↓ |•| i (B)| ≤ √ c + o n ( √ c).
Once again, the enunciation of Theorem . uses a notation that can be easily generalized to k > 2. Interestingly, however, the empirical evidence suggests that Theorem . can also be generalized to the DCSBM setting, including Φ = 1 as in Theorem . . We give a visual con rmation of this claim in Figure . . With this evidence, let us summarize the two main messages to retain from this section:

• The proofs to obtain the spectral properties of B in the sparse regime heavily exploit its tree-like structure. In the dense regime in which the tree-like hypothesis is not veri ed, however, the spectral behavior of B is essentially unchanged, hence resilient to the de nition of sparsity.

• The proof of Theorem . requires as a hard constraint the degree concentration that characterizes dense SBM-generated graphs. The empirical evidence however suggests that this is a technical hypothesis and that Theorem . can be extended also to the DCSBM.

This discussion concludes the properties of sparse DCSBM generated graphs. We now proceed considering the weighted case.

. . Recently, in [SM ] the spectrum of B was studied under very generic hypothesis on weighted graphs. Let us recall their main theorem. In the following the notation ' • ' is used to indicate the Hadamard entry-wise product.

Theorem . (Spectrum of BU ω on weighted random graphs). Let P be a symmetric random matrix with entries with 0 ≤ P ij ≤ 1 and max(P ij ) = O n (1/n). The adjacency matrix of G(V, E ) is generated setting its entries to one independently (up to symmetry) at random with probability

P(A ij = 1) = P ij .
Let W be a symmetric random matrix with independent entries

W ij = ω ij and denote with ω = {ω e } e∈E d , U ω = diag(ω) and L = max(|W ij |). Further let ρ = λ ↓ |•|
non-backtracking matrix B ω = BU ω veri es the following identities with high probability

∀ 1 ≤ i ≤ r 0 , λ ↓ |•| i (B ω ) = λ ↓ |•| i (P • E[W]) + o n (1) |λ ↓ |•| i≥r 0 +1 (B ω )| ≤ max( √ ρ, L).
Although the formulation of Theorem . is quite involved, the result it conveys is simple: the leading eigenvalues of B ω are determined by the expectation of the weighted adjacency matrix P • E[W], while the bulk eigenvalues are con ned on a circle in the complex plane and are determined by the variance of the weighted adjacency matrix, P • E[W • W]. This theorem can be seen as a generalization of Theorem . and . that are obtained for

W = 11 T n and P ij = θ i θ j C i , j
n . With this result, we conclude the properties of the non-backtracking matrix that and brie y summarize some of the (few) known facts concerning the closely related Bethe-Hessian matrix.

. Unlike the non-backtracking matrices, few rigorous results exist on the spectrum of the Bethe-Hessian matrix, most of which exploit its relation with the non-backtracking matrix as is Theorem . . First of all, let us recall the (generalized) Bethe-Hessian matrix of De nition .

(H ω (r)

) ij = δ ij 1 + ∑ k∈∂i ω 2 ik r 2 -ω 2 ik - rωij r 2 -ω 2 ij
This section gives two important claims on the spectrum of H ω (r): (i) the existence of isolated eigenvalues for DCSBM-generated graphs under Assumption . and (ii) the position of the smallest bulk eigenvalue of H ω (r) for a particular choice of r.

The circle theorem implies for r ∈ R λ ↑ 1 (H r ) ≥ r 2 -1 -(r -1)d max r r 2 1 (r 1)d max dmax 1
Let us rst consider the unweighted case in which H ω (r) simply reads

H r = (r 2 -1)I n + D -rA.
To understand the origin of isolated eigenvalues in the spectrum of H r , it is enough to note that for a su ciently large r ∈ R, thanks to Gershgorin circle theorem [HJ ] H r 0, i.e. all its eigenvalues are positive, as already commented along Chapter , when studying the stability of the paramagnetic phase of the Ising model. For all r > λ In red, the theoretical curve of the bulk eigenvalue limiting distribution as in [SKZ ].

↓ |•| 1 (B),
Considering r = λ

↓ |•|
1 (B) ± for a small , it turns out that, again, all eigenvalues of H r are non-zero, hence the smallest eigenvalue is isolated. Generalizing this reasoning to the following eigenvalues, from an empirical point of view, there is a one-to-one correspondence between the p-th largest (real) eigenvalue of B and the p-th smallest of H r as stated in the following claim.

1 ≤ p ≤ k : λ ↑ p H λ ↓ |•| p (B) = 0.
Note that this is not a trivial consequence of Theorem . which only implies that there exists q such that λ

↑ q (H λ ↓ |•| p (B)
) = 0. This claim instead goes deeper, showing an explicit bijection between the largest (isolated) eigenvalues of B and the smallest (also isolated) of H r , setting p = q.

Claim . concerns only the isolated eigenvalues of B and H r . The bulk eigenvalues distribution of the Bethe-Hessian matrix was instead studied in [SKZ ] with the cavity method [Rog+ ] for SBM graphs. The result of [SKZ ] allows one to obtain µ(λ) solving a xed point equation. The result is shown in Figure . . The method of [SKZ ] can be easily generalized to the DCSBM setting, leading in particular the following claim.

Claim . (Limiting spectral distribution of

H r ). Let G(V, E ) be a realization of the DCSBM verifying Assumption . for c = O n (1). Let ρ = λ ↓ |•| 1 (B)
, then the following relation holds with high probability:

λ ↑ k+1 H √ ρ = 0 + , .
i.e. , for r equal to the radius of the bulk of B, the bulk of H r is asymptotically close to zero (but positive).

The relation between the eigenvalues of B ω and those of H ω (x) (the mapping between isolated eigenvalues and the relation the bulk edges) holds as well in the weighted regime, as evidenced by extensive simulations that will be more deeply discussed in Part III and Claim . , . can be generalized to this setting. The theoretical results on the spectrum of H ω (r) are very limited comparatively to those obtained for B ω . Nevertheless, Theorem . together with Claim . and . makes it possible to infer some relevant properties of the spectrum of H ω as a non-rigorous "corollary" to the theorems characterizing the non-backtracking matrix spectrum.

The relation between the eigenvectors of H r and B

Finally, let us conclude this section on the spectral properties of the Bethe-Hessian matrix, showing that not only its eigenvalues, but also its eigenvectors are related to those of B.

Proposition . (Eigenvectors of B ω and H ω (r)). Let B ω be the weighted non-backtracking matrix of a graph G(V, E ) and H ω (r) the corresponding generalized Bethe-Hessian matrix. Consider an eigenvector g of B ω with eigenvalue λ so that det[H ω (λ)] = 0 and λ = |ω ij | for all (ij) ∈ E .

Let ψ(g) ∈ R n be de ned as

ψ i (g) = ∑ j∈∂i g ij ω ij .
Then H λ ψ(g) = 0, i.e. ψ(g) is an eigenvector of H λ with zero eigenvalue.

The proof of Proposition . allows one to prove part of Theorem . but its main purpose is to explicitly show the relation between the eigenvectors of the two matrices.

Proof. From the de nition of ψ(g) and of B ω we can rewrite

ψ i (g) = ∑ k∈∂i ω ik g ik = ∑ k∈∂i\j ω ik g ik + ω ij g ij = (B ω g) ji + ω ij g ij .
Inverting the role of i and j, the following system is obtained

λ ω ij ω ij λ g ji g ij = ψ i (g) ψ j (g)
.

Solving this system, g ij can be written as a function of ψ i (g) and ψ j (g):

g ij = 1 λ 2 -ω 2 ij λψ j (g) -ω ij ψ i (g) .
Reinjecting this relation in the de nition of ψ i (g) and exploiting the definition of H ω (r), one nally concludes the proof.

The proof of Proposition . concludes the study of the spectral properties of the Bethe-Hessian matrix. In the next section we relate the results obtained so far with those introduced in Chapter in which the study of the spectral properties of B ω was motivated by the Bethe approximation of Ising-like models on sparse graphs.

. Chapter showed that the leading eigenvalue of the non-backtracking matrix plays a crucial role in determining the stability of the paramagnetic phase in the Ising-like models. In fact, letting β c satisfy λ

↓ |•| 1 (BU β c ,J ) = 1, ( . )
according to the Bethe approximation, the paramagnetic phase is stable only if β < β c . Based on this result, we list some important facts.

Percolation threshold on sparse DCSBM

In Chapter we de ned the problem of percolation on random graphs as the critical condition (on the parameters of the generative model of the graph) needed to ensure, with high probability, the existence of a giant component in G(V, E ). In Theorem . we proved in particular that on a DCSBMgenerated graph with expected degree independent of the class label, the existence of a giant component in G(V, E ) appears if and only if cΦ > 1.

Let us detail some consequences of this fact

• Referring to Assumption . , we immediately notice that the condi-

tion λ p (CΠ) > √ cΦ implies cΦ > 1, since λ ↓ 1 (CΠ) = c.
Consequently, all theorems and conjectures formulated in this chapter hold for graphs with a giant component.

• According to Theorem . , for a DCSBM-generated graph satisfying Assumption . , the leading eigenvalue of the non-backtracking matrix is asymptotically close to cΦ. The presence of an eigenvalue greater than one in the spectrum of B hence indicates the presence of a giant component. The relation between percolation and the spectrum of B was conjectured already in [KNZ ] in which the authors consider an arbitrary given graph G(V, E ) in which each edge is kept with probability p and determine the critical p to observe a giant component.

• The absence of an eigenvalue of the non-backtracking matrix greater than one makes it impossible to de ne β c . Consider for simplicity an unweighted graph, for which Equation ( . ) can be rewritten as

λ ↓ |•| 1 (B)th(β c J) = 1.
Since th(β c J) < 1 and λ

↓ |•|
1 (B) < 1 by hypothesis, β c cannot be de ned. This implies that no spontaneous magnetization can be observed is there is no giant component in G(V, E ), as shown in [Leo+ ]. Furthermore, if cΦ > 1, the transition temperature reads

β c = 1 J ath 1 cΦ .

Isolated eigenvectors of H r

Let us now consider a very important relation between the Bethe free energy (Equation ( .)) and the eigenvectors of the Bethe-Hessian and nonbacktracking matrices, referring, once again, to the unweighted case for simplicity. Since B is a non-negative matrix, the eigenvector associated to its largest eigenvalue (the one de ning β c as in Equation ( .)), has entries with the same sign. Note also that in Section . . the guess eigenvector the eigenvalue is chosen with all positive entries. Consequently, exploiting Claim . and Proposition . it turns out that all the entries of the eigenvector associ-

ated to λ ↑ 1 (H λ ↓ |•| 1 (B)
) have the same sign. Recalling the derivation and physical interpretation of the Bethe-Hessian matrix we detailed in Chapter , the eigenvectors of H r are pointing towards the directions in which minima of the Bethe free energy may appear. The spectral properties of B hence not only give an expression to the transition temperature β c but also predict that the stable phase is the ferromagnetic one, in which all spins are aligned.

This observation is more general and can be extended to all eigenvectors of H r (as well as to H β,J ) with a negative eigenvalue. This fundamental intuition will be at the core of our proposed algorithms of Part II and III.

Phase transitions in weighted graphs

In Chapter we showed how the spectral properties of B ω determine the instability of the paramagnetic phase according to the Bethe approximation. Yet, even if a phase transition is predicted, no other conclusion was made on what happens in general when β > β c , i.e. when the system is no longer in the paramagnetic phase. This is because this question depends generally on the matrix J. In the particular case in which J = JA with J > 0 only two phases exist: the paramagnetic and the ferromagnetic. If the system is not in one phase, then it is in the other. This is well explained by the fact that all entries of the eigenvector associate with the smallest eigenvalue of H r are positive, as we said in the previous paragraph. This property is however a direct consequence of the fact that J has only positive entries. What happens in the more general case in which J ij may be negative?

In this case, we should, more precisely, refer to Equation ( . ) as spin glass rather than Ising Hamiltonian [MPV ]. The study of spin glasses is a vast eld to which we have not the ambition to give an introduction here. We borrow however an example from the random bond Ising model to relate this physical problem to the spectrum of B ω .

Let G(V, E ) be a realization of a Erdős-Rényi random graph and de ne the following Hamiltonian on G(V, E ).

H(s) = -∑ (ij)∈E J ij s i s j ,
in which for all (ij) ∈ E , the J ij are independent random variable distributed according to pδ(J -1) + (1p)δ(J + 1), for 1/2 ≤ p ≤ 1. Let ω = {th(βJ e )} e∈E d . From Theorem . we obtain in this case that

λ ↓ |•| 1 (B ω ) = max c(2p -1), √ c th(β).
The transition temperature determining the instability of the paramagnetic phase hence occurs at

β c = ath 1 max(c(2p -1), √ c) .
According to the value of p we have two di erent scenarios, dictated by whether or not c(2p -1) > √ c. This is precisely the condition that determines the appearance of one isolated eigenvalue in the spectrum of B ω , according to Theorem . . Speci cally, for large p (the case p = 1 reduces to the ferromagnetic Ising Hamiltonian considered in the previous paragraph) the largest eigenvalue of B ω is isolated and the entries of its eigenvector are positive in expectation and when β = β c there is a transition from the paramagnetic to the ferromagnetic phase. For c(2p -1) < √ c instead, the largest eigenvalue of B ω belongs to the bulk and the resulting phase is called spinglass in which the global magnetization is null, but local order is observed. This discussion closes Part I which was dedicated to the presentation of the main technical tools needed to introduce the original contributions of this manuscript. We now proceed with the rst part of our contributions, focused on the problem of community detection in sparse graphs with a heterogeneous degree distribution.

C O M M U N I T Y D E T E C T I O N

This part is devoted to the presentation of our contributions to spectral clustering for community detection. First, Chapter provides an overview to the basics of community detection, with a particular attention to the problem of inference in graphs generated from the degree corrected stochastic block model. Spectral clustering is then reviewed as an e cient class of algorithms to perform community detection. Chapter details our main contribution to the eld, which is stated in the form of a claim which naturally leads to an e cient algorithm for community detection. Successively, Chapter draws an explicit relation between the results of Chapter and some works based on the regularized Laplacian matrix, answering some open questions related to this popular method for spectral clustering. The problem of how to e ciently translate our theoretical results into a practical algorithm is thoroughly treated in Chapter . Finally, based on our theoretical results, Chapter serves as a conclusion to Part II, with a critical and retrospective look to spectral clustering for community detection.

of-the-art SC algorithms for CD is provided, underlining the weaknesses of each of the existing approaches that either are not suited to perform well in the presence of a broad degree distribution or of a low average degree, two properties that typically characterize real-world graphs. Part I gave an introduction to graph theory, statistical physics and random matrix theory. These tools are the fundamental building blocks needed to describe our contributions. This chapter is a bridge between Part I and the remainder in which our main results are presented.

In Chapter some properties of real-world graphs have been introduced, recalling, among others, the recurrent presence of a community structure [LM+ ; MSLC ] according to which nodes in the same class are more densely connected than nodes in di erent classes. The problem of community detection (CD) can be informally de ned as the task of determining a non-overlapping node partition of a graph, unveiling its underlying community structure. This de nition is somewhat tautological given its dependence on the concept of communities for which there exist no shared consensus in the scienti c literature [FH ]. Its clear requirement is that each node has to be assigned to a unique class. More general, and possibly more realistic, denitions in which each node may belong to more than one class exist [XKS ] but will not be considered in the remainder. Some of the earliest approaches [Sco ] to CD relied on the concept of cliques, i.e. sub-graphs of G(V, E ) in which all nodes are connected among themselves. This represents the extreme case in which nodes in the same A graph with k = 2 communities community are maximally connected. Communities, however, can be dened even under less stringent conditions. Among the several strategies to perform CD [For ; GN ; New a; FH ] in this chapter we consider two in particular. In the rst, the class structure is determined solving an optimization problem. This approach is very popular in the CD literature and inspired some of the earliest works on spectral clustering (SC) [VL ]. The second method, instead, considers CD as an inference problem from the degree corrected stochastic block model (DCSBM). In Section . we brie y introduce the key ideas of some of the most commonly deployed de nitions of CD as an optimization problem, evidencing, in particular, the major weaknesses of this approach. Subsequently, in Section . , we review the main results concerning inference in the sparse DCSBM and discuss how this approach is capable to overcome the limitations of the optimization strategy. Finally, Section . serves as a review to state-of-the-art methods of SC for CD. A particular attention will be devoted to how each of the considered algorithm deals (or not) with sparsity, i.e. low average degree and heterogeneity in the degree distribution. We recall, from Chapter , that these are two very recurrent properties that characterize real graphs and they generally constitute a big challenge for SC.

. De ning communities as the solution of an optimization problem consists in identifying a quality function assessing how satisfactory a given class partition is on a graph. Such function should depend on the partition and the graph G(V, E ), in the form of its adjacency matrix A. Some of the earliest choices for the quality function rely on graph cuts, such as the ratio cut (RCut) [HK ], de ned in Equation ( . ) and the normalized cut (NCut) [SM ], de ned in Equation ( . ).

Q RCut A ( ) = 1 2 k ∑ p=1 1 |V p | ∑ i∈V p ∑ j/ ∈V p A ij ( . ) Q NCut A ( ) = 1 2 k ∑ p=1 ∑ i∈V p ∑ j/ ∈V p A ij ∑ i∈V p ∑ j∈V A ij , ( .
)
where V p = {i ∈ V : i = p} is the set of all nodes with label p.

Graph cuts

According to these cost functions, high values of Q • A are obtained when there are several edges connecting nodes in di erent communities. The goal is hence to nd that minimizes Q • A , under the constraint = 1 n , which prevents all nodes from being assigned to the same cluster.

Another quality function that is very popular in modern CD is the modularity, introduced in [NG ] to evaluate the quality of a community partition:

Modularity Q Mod A ( ) = 1 4|E | ∑ i,j∈V A ij - d i d j 2|E | δ( i , j ), ( .
)
where we recall that d = A1 n indicates the degree vector. The modularity attributes a large score to con gurations in which nodes in the same community are connected by a greater than expected number of edges. In fact, for a xed degree sequence, d i d j /2|E | is the probability that nodes i and j are connected if edges were placed at random. The modularity has subsequently been exploited to de ne CD algorithms in which the label assignment is obtained maximizing the Q Mod A ( ) and it is one of the most widely adopted strategies to CD [Rad+ ; New b; CNM ; Blo+ ; TWVE ].

De ning communities according to a score function may seem a particularly good strategy for two main reasons. The rst is that the de nition of communities does not make any assumption on the matrix A and is hence adapted to any graph. The second is that it gives a common sense de nition of communities, expressing a property of a good class assignment. This approach has, however, strong algorithmic limitations.

• Optimization problems such as NCut, RCut and the modularity maximization are NP hard and only approximate solutions can be obtained by e cient algorithms [FH ; VL ].

The pitfalls of the optimization approach to CD In other words, the modularity compares the realization of the matrix A with a typical realization of a null model, called con guration model. Or better, does not make any explicit assumption on the matrix A. In the following we will reconsider this assertion and argue that, matter of factly, the optimization approach makes some implicit assumptions of A. Although this has not been proved but only conjectured, it is believed that NP-hard optimization problems do not admit a polynomial time solution, i.e. , for a problem of size n, an exponential in n number of operations is required to obtain the solution [VL ].

• There exists a large number of structurally di erent con gurations having values of Q • A value very close to the maximum [GDMC ].

Running multiple times an approximate algorithm looking for the optimum of Q •

A may output similar results in terms of the score, but corresponding to rather di erent label assignments.

• The authors of [FB ] showed that, even in the presence of well dened clusters (such as cliques), optimizing the modularity score over the number of clusters k, small communities may be joined together, causing the so-called resolution limit. This is a consequence of the fact

The e ect of the resolution limit is well evidenced by ring of cliques graph: here the colour is the assignment obtained maximizing the modularity that the values of Q Mod A are not directly comparable for di erent values of k. Albeit the authors of [FB ] only consider the modularity, they claim that similar e ects are expected to be seen also when optimizing other cost functions, such as

Q RCut A or Q NCut A .
To circumvent this problem, in [AFG ] it was introduced a generalized modularity, depending on a positive regularizer γ:

Q GMod A ( ; γ) = 1 4|E | ∑ i,j∈V A ij -γ d i d j 2|E | δ( i , j ). ( . )
Tuning the value γ it is possible to identify communities at di erent length scales, but this requires an ad hoc solution, depending, in general, on the underlying graph.

• From the optimization perspective, communities can be de ned even on graphs with no community structure, such as Erdős-Rényi (ER) random graphs. This is not only a philosophical problem, related to the fact that a good CD algorithm should be capable of detecting whether or not communities are present on the graph. In fact, considering for instance the modularity, one expects that on ER graphs, any partition satis es Q Mod A ≈ 0. It has however been shown in [GSPA ] that high modularity partitions can be found on ER graphs, evidencing that the modularity maximization may lead to over-tting.

These problems altogether are severe limitations of the optimization approach to CD and justify the adoption of a di erent strategy, based on inference on the DCSBM. We will show in the next section how Bayesian inference is able to overcome the aforementioned limitations of optimization and how it is able to motivate their origin. . . . Let us recall that, according to De nition . , the DCSBM generates the adjacency matrix of G(V, E ) setting to one its entries independently (up to symmetry) at random with probability

P(A ij = 1| i , j ) = min θ i θ j C i , j n , 1 . ( . )
We recall that C ∈ R k×k is the class a nity matrix, generating the community structure of G(V, E ), while the vector θ ∈ R n creates an arbitrary degree distribution with the expected degree of note i being proportional to θ i . Furthermore, the vector π ∈ R k is de ned so that π p is the expected fraction of nodes with label p and we denote with Π = diag(π). We work under Assumption . , according to which 1 k is an eigenvector of CΠ with eigenvalue c, representing the (class-independent) expected average degree.

Given the generative model of Equation ( . ) the labels are indisputably de ned by the vector and the number of classes by k. The goal of CD given a DCSBM-generated graph is hence to infer . Let us de ne the overlap to characterize more formally CD on DCSBM graphs.

Two permutations of the same solution

De nition . (Overlap). Let , ˆ ∈ {1, . . . , k} n be two vectors and P the set of all permutations of . The overlap between and ˆ is de ned as

Q ov ( , ˆ ) = 1 1 -1 k max p∈P 1 n n ∑ i=1 δ( ˆ i , p( i )) - 1 k .
In words, the term

1 n ∑ n i=1 δ( ˆ i , p( i ))
represents the fraction of correctly labelled nodes; the term 1/k, instead is the probability that δ( ˆ i , p( i )) = 1 if the entries of ˆ were randomly assigned. For all , the function Q Ov ( , ˆ ) is between 0 (random label assignment) and 1 (perfect label assignment). Note that, since we are taking the maximum over all label permutations, Q ov can never be negative.

The overlap being de ned, CD can be formulated as the task of nding a partition ˆ so that Q ov ( , ˆ ) > 0 [Abb ]. In words this means to infer better than random guess.

Remark . (Detection in sparse graphs). Note that on sparse DCSBM-generated graphs, with high probability, min i (d i ) = 0, i.e. there are isolated nodes that cannot be classi ed better than random guess [MNS ]. For this reason, when considering the sparse setting, it is most appropriate to talk about detection and not exact recovery that is structurally unfeasible.

An example of a disassortative graph

The class structure of G(V, E ) is simply determined by the matrix C, imposing that C aa > C ab for b = a. This results in an adjacency matrix in which fewer edges connect nodes in di erent classes, as required by the intuitive de nition of communities. Note however that, even though it is a typical requirement that the diagonal elements of C are greater than the odiagonal ones, the DCSBM can also be used to describe disassortative community structures. In this case groups of nodes are identi ed as a class because they repel (rather than attract) each other. As a concrete example one may think of the vertices of a graph as the words contained in a text with edges if two words appear next to each other: on this graph, adjectives and nouns represent two disassortative communities [NG ]. In the case of two classes, this corresponds to imposing C aa < C ab . While the de nition of a disassortative community structure with the DCSBM is possible, in the remainder we will mainly focus on the assortative case.

Coming to the details of CD as an inference problem, the probability distribution of the labels given the graph observation A can be retrieved with a Bayesian approach. In particular The prior P( ) encodes pre-existing information on the class label assignment. We suppose nothing is known about the community structure and consequently the uniform -independent prior is chosen. The likelihood is obtained from Equation . , while the evidence is an -independent normalization constant. For the DCSBM, assuming θ i θ j C i , j /n < 1 for all i, j, the posterior distribution reads:

P( |A) ∝ ∏ (ij)∈E θ i θ j C i , j n • ∏ (ij)/ ∈E 1 -θ i θ j C i , j n ∝ exp    ∑ (ij)∈E log C i , j + ∑ (ij)/ ∈E log 1 -θ i θ j C i , j n    . ( . )
Obtaining the marginal node probability from Equation ( . ), one can assign to each node the label maximizing the node marginal [Pei ]. A possible way to accomplish this task is to sample from the distribution ( . ) using Monte Carlo Markov chains [Pei a]. The Bayes optimal procedure is, however, typically quite expensive from the computational viewpoint. Luckily, for sparse graphs, the asymptotically exact expression of P i ( i |A) can be e ciently obtained using the belief propagation (BP) algorithm introduced in Chapter . The node marginals are obtained solving the following system BP for CD of equations [Dec+ , Eq. , , ]:

P i ( i |A) = π i Z i e -h i ∏ j∈∂i ∑ j C i , j η ij ( j ) + o n (1) ( . ) η ji ( i ) = 1 Z ji π i e -h i ∏ k∈∂i\j ∑ k C k , i η ik ( k ) ( . )
Note that the eld h i comes from the contribution of all the "non edges" appearing in Equation ( . ) and this expression speci cally holds in the sparse SBM.

h i = 1 n ∑ k∈V ∑ k C i , k P( k |A). ( . )
Note that Equations ( . , . , . ) are obtained for the stochastic block model (SBM) (which we recall corresponds to the choice θ = 1 n ) but the result can be easily generalized to the DCSBM. Using BP propagation it is also possible to iteratively learn the parameters C and π that cannot be supposed to be known in advance.

Remark . (The estimation of k). In [Dec+ ] the authors explain how to estimate the number of communities k if not known in advance. The procedure consists in running BP multiple times for di erent values of k and keeping the one minimizing the Bethe free energy. As shown in [Pei ], however, when the underlying graph is very far from a realization of the SBM -like a ring of cliques -the estimate of k may become inaccurate, su ering from a resolution limit.

The computational complexity of BP for CD scales as

O(|E |k 2 ) [Pei ], making BP computationally e cient in the sparse regime in which c = O n (1) (or, equivalently, |E | = O n (n)).
The main interest in BP, however, comes from the fact that it is asymptomatically exact in the sparse regime. Studying the convergence of BP allowed the authors of [Dec+ ] to formulate some deep conjectures on the theoretical feasibility of inference in the sparse SBM, claiming that if BP is not capable of obtaining a non-zero overlap partition, then no other e cient algorithm can.

. . One of the most striking results of [Dec+ ] concerns the conjecture of the existence of a detectability threshold when inferring the labels. Consider the case of k communities of equal size with

C = c out 1 k 1 T k + (c in -c out )I k . The expected average degree is c = (c in + c out (k -1))/k. We de ne α as: α = c -c out √ c . ( . )
The authors of [Dec+ ] conjecture that for α > α c = 1 there exist ecient algorithms that are capable to perform CD, obtaining a non-zero overlap. In particular, α > α c is the condition needed by BP to converge towards an informative xed point. What happens when α < α c is slightly more involved. In particular, for any k, BP is not (asymptotically) capable of achiev-

Algorithmic and information theoretic thresholds

ing an overlap greater than zero if a random initialization is chosen. It is supposed that no other e cient algorithm (i.e. with computational complexity polynomial in n) can achieve this task. Hence, α = α c is the algorithmic threshold of CD for the SBM. Yet, under proper conditions, it is observed that Note that the existence of a detectability threshold had been previously conjectured in [RL ] in which the authors, however, overestimate the size of the undetectable region. Picture adapted from [Moo , Figure ].

for an initialization su ciently close to the informative xed point, BP obtains a positive overlap even for α < α c . When BP cannot obtain a positive overlap even when initialized to the ground-truth assignment, we are at the information theoretic threshold below which there is no way to distinguish a SBM from an Erdős-Rényi (ER) random graph.

In the case of k = 2, 3 communities it is claimed that the algorithmic and information theoretic thresholds coincide, hence that for α < α c no algorithm can (asymptotically) obtain a positive overlap . For k ≥ 4 on disassortative graphs and k ≥ 5 on assortative ones, multiple transitions exist:

• α < α d : if BP is initialized to the ground truth assignment, it converges to the factorized xed point P i ( i ) = 1/k: detection is unfeasible. • α d < α < α h : if BP is
initialized to the ground truth assignment, it converges to a solution with positive overlap. However, there is no way to distinguish this solution from any other: detection is unfeasible.

• α h < α < α c : if BP is initialized to the ground truth assignment, it leads to a partition with positive overlap which is distinguishable from all others. Detection is feasible, but it requires an exponential time.

Figure . summarizes the phase transitions. Since its appearance, the conjectures of [Dec+ ] conveyed a lot of attention and several authors contributed to give them a rigorous proof. In particular, [Mas ; MNS ] focused on the SBM for k = 2 communities, independently showing that detection can be e ciently obtained when α > α c . In the same setting [MNS ] instead proved the negative part of the conjecture, i.e. that detection is impossible when α < α c . Moreover, [MNS ] proved that a modi ed version of BP maximizes the fraction of nodes labelled correctly for some α > α c . The extension of the possibility part for k > 2 was proved in [AS ], while in [CO+ ] the conjecture was proved for disassortative graphs. Finally, the results of [Dec+ ] have been extended to the k = 2 class setting on DCSBMgenerated graphs. In this case α should be re-de ned as α = (c

-c out ) Φ c , where we recall that Φ = E[θ 2 i ].
In [GLM ] it was shown that detectability is impossible if α < α c = 1, while the positive part was proved in [GLM a]. . . :

To conclude this section, let us relate the model-based and optimizationbased approaches.

As it was described in Section . , de ning communities as the solution to an optimization problem makes a requirement on what a good class partition should be like, with no hypothesis on the underlying graph. This makes it a seemingly good way of performing CD on arbitrary graphs. On the opposite, designing an algorithm for CD inspired from DCSBM gives a good mathematical control and, in some cases, information theoretic guarantees. Nevertheless, the model-based approach relies on some assumptions that are not necessarily veri ed on arbitrary graphs: recall for instance the locally tree-like structure of sparse DCSBM graphs.

This may lead to thinking of the inference approach as a mere mathematical exercise. This section on the opposite argues that the model-based approach should be generally preferred to the optimization one. In fact, the latter actually relies on some implicit hypothesis on the matrix A and its limitations can be clearly interpreted from a Bayesian perspective.

To simplify the discussion, let us consider the k = 2 class DCSBM. In this case, letting

σ i = 1 if i = 1 and σ i = -1 if i = 2,
the posterior probability P( |A) ≡ P(σ|A) of Equation ( . ) can be rewritten as

P(σ|A) = 1 Z exp    ∑ (ij)∈E βσ i σ j -∑ i∈V h i (σ)σ i    ≡ e -βH(σ) Z , ( .
)
where β = 1 2 log c in c out and h i (σ) is a local magnetic eld due to nonedges that prevents all nodes from ending up in the same class. Equation ( . ) precisely corresponds to the Boltzmann distribution for the Ising Hamiltonian with local elds, depending on the con guration σ. The Bayesian approach is equivalent to nding the magnetization m = s β , associated to the Hamiltonian H(σ). Finding the ground state of H(σ), i.e. the con guration σ corresponding to its minimum, instead is equivalent to nding the maximum of the generalized modularity Q GMod A ( ; γ), for a proper value of γ, as shown in [New ].

This observation puts us in position to make two very important remarks. The most questionable assumption of the DCSBM is that of generating edges independently at random. In terms of log-likelihood, this translates into a sum over all graph edges, as shown in Equation ( . ). This sum is the same appearing in

Q GMod A , Q Mod A , Q RCut A and Q NCut A
that can be associated to a generative model (di erent from the DCSBM) in which the edges of G(V, E ) are also generated independently at random, evidencing how the functions Q •

A rely on some "silent" assumptions on the matrix A.

Furthermore, from a statistical physics perspective, the functions Q • A (eventually taken with a negative sign) can generally be considered as Hamiltonians, i.e. cost functions associated to a given label con guration. In all cases (also for the DCSBM), the Hamiltonian is what de nes communities from a microscopic perspective. Stating which de nition is the best is a di cult task that we are not going to investigate. However it should be remarked that what is essentially di erent in the optimization and inference approaches is how to retrieve the communities from the Hamiltonian: in one case they are obtained from the marginals of the Boltzmann distribution, in the other from the ground state energy.

At this point the natural question to ask is "what is best?". Finding a de nitive answer to this puzzle is beyond the scope of this manuscript. The answer is likely to be problem-dependent [Pei ] and, although we do not claim to provide universal arguments, we give two reasons why, at least in our setting, the Bayesian approach should be preferred.

• The Hamiltonian, or equivalently the cost Q •

A is what de nes the concept of communities. The optimization approach, however, only takes into account the minimum of the Hamiltonian, disregarding the rest of its pro le. Consider two functions Q • A , one being convex and the other having multiple minima with similar values of Q • A ( ). These two settings are clearly di erent: in the rst the label assignment is uniquely de ned by minimum of Q • A , whereas, in the second, several con gurations could be considered as almost equally good community structures. Taking only the minimum of the Hamiltonian means to disregard all other con gurations that may have, instead, a potentially great importance. The Bayesian approach does not consider exclusively the minimum of Q •

A , but the whole energy landscape, giving a generally richer description of the problem.

• The solution of the optimization problem does not involve the condence that one can have in the solution. To get more practical, consider a set of realizations of the DCSBM with the same ground-truth labels.

Running an optimization algorithm on each of these graphs indepen-

A node partition of an ER random graph with Q Mod ≈ 0.25 dently will likely lead to di erent partitions, especially as the value of α decreases. Using the Bayes rule to determine the node marginals, instead, gives in expectation very similar results of the marginals, making this method resilient to noise. This can be well understood by taking a closer look at Equation ( . ). For hard detection problems, β → 0, leading the system into the paramagnetic phase in which no clustering can be made. The Bayes approach hence correctly predicts the absence of communities in Erdős-Rényi (ER) random graphs. As the problem gets easier, instead, β increases and the marginals will progressively tend to one. In the extreme case of β → ∞ (which is the easiest possible scenario), the magnetizations σ β corresponds Recall that α is the parameter determining the hardness of CD in the DCSBM.

to the ground state con guration. The optimization approach hence boils down to the (overly optimistic) assumption that the ground-truth label vector corresponds to the minimum of the Hamiltonian, completely disregarding the contribution of the noise. This leads, in general, to over-tting [ZK ], explaining why the modularity maximization leads to high scores on completely random graphs. This concludes our introduction to the DCSBM and more generally to CD. We now proceed with a short review of some of the most in uential methods for SC for CD, including both those inspired by the optimization and the Bayesian approaches. The literature on SC is incredibly vast and the following review considers a limited sample of all the existing contributions. The main criterion motivating the inclusion of a particular algorithm in the following section concerns whether or not it is equipped with a deep theoretical and algorithmic explanation.

. :

Spectral clustering is a popular method to perform CD and has deep relations with both the optimization and the Bayesian approaches. Algorithm . describes the typical steps of SC for CD. Not all SC algorithms can be written in the form of Algorithm . that has the sole purpose of providing the reader a practical reference.

Taking a closer look at Algorithm . , rst note that k is required as an input. It has been extensively discussed that determining a good value for k is one of the big challenges of CD and SC is not an exception. 1) . end spect to the class structure of the graph. The arguments motivating and interpreting the existence of such eigenvectors are treated in the following sections and ultimately characterize each SC algorithm. Note that, from a practical viewpoint, the fact that these informative eigenvectors are in largest or smallest position is of fundamental importance, otherwise one would need to compute all eigenvectors and choose the most informative ones, with a very time consuming routine. However, as it will be made clear in the following, having k informative eigenvectors in dominant position cannot be given for granted, especially on sparse graphs.

(t+1) i ← arg min a=1,...,k X i,• -µ (t) a 2 for 1 ≤ i ≤ n; t ← t + 1 end return (t-
The choice of the matrix M is perhaps the most fundamental task to design a SC algorithm. The small dimensional embedding X obtained from M must be a suited input to the nal small dimensional clustering step. Coherently with the majority of the SC literature [CZC ], we systematically perform this step with the k-means algorithm, which we now summarize.

Given the input matrix X, k-means assigns the labels ˆ as the solution to an optimization problem formulated as follows:

The k-means algorithm

ˆ = arg min k ∑ a=1 ∑ i∈V a X i,• -µ a 2 ( . ) µ a = 1 |V a | ∑ i∈V a X i,• , ( . )
where V a = {i : i = a} is the set of all points with attributed label equal to a and X i,• is the i-th row of X. The term µ a is the average of all the rows with label a and is the centre of the cluster. Equation ( . ) attributes label a to node i if µ a is the closest centre to X i,• . Note that, although not denoted explicitly, the term µ a depends on . This in is an NP-hard optimization problem for which an approximate solution is typically searched using Lloyd algorithm [Llo ], detailed in Algorithm . . The labels are initialized (for example with a random assignment or with some more sophisticated technique [AV ]) then the centres and the labels are iteratively updated, assigning each node to the closest centre. The complexity of Lloyd algorithm scales as O(nkd emb ) [SMR ].

With Lloyd's algorithm, there is no guarantee that the optimum of Equation ( . ) is reached and the output will depend, in general, on the initialization (step of Algorithm . ). To cure this aw, k-means is commonly run multiple times and the solution reaching the lowest cost in terms of Equation ( . ) is kept.

Furthermore, note that the k-means objective function only involves the distances from the centres of each clusters. As a consequence, in the context of SC for CD, a good input for k-means is so that the distance between the points in the embedded space ideally depends only on the node's label. As we will comment in the following, however, the presence of a heterogeneous degree distribution must be accounted for, because it typically takes a (deleterious) role in determining the embedding.

With the basics on k-means being established, we now turn to a detailed description and justi cation of some state-of-the-art SC algorithms for CD.

. .

The rst contribution to SC for CD dates back to [Fie ] in which, for k = 2 communities, it was proposed to reconstruct communities using the eigenvector corresponding to the second smallest eigenvalue of the combinatorial graph Laplacian matrix L = D -A. It was then shown [VL , for instance] that this eigenvector provides a relaxed solution of the RCut problem. In particular, let f be de ned as follows

( f ) i =      |V 2 | |V 1 | if i = 1 |V 1 | |V 2 | if i = 2. ( . )
Then, it can be shown [VL ] that SC provides a relaxation of the graph-cut optimization problems

Q RCut A ( ) ∝ f T (D -A) f . ( . )
The relaxation consists in optimizing Equation ( . ) considering f ∈ R n , instead of having binary entries as in Equation ( . ). Then the labels can be estimated taking the sign of u, the eigenvector associated to the smallest eigenvalue of L = D -A, subject to the constraint u ⊥ 1 n . This condition is imposed to prevent all nodes from being set in the same class. Luckily, 1 n is an eigenvector of L, precisely the one associated to its smallest eigenvalue, hence u is the eigenvector associated to λ ↑ 2 (L), i.e. to the second smallest eigenvalue. The result can be generalized to k > 2. Referring to Algorithm . , one has M = L, the informative eigenvectors correspond to the smallest eigenvalues and step is not performed.

X , 1 X , 2 Class 1 Class 2 X , 1
Figure . : Scatter plot of the rst vs second column of the matrix X containing the leading eigenvectors of L sym = D -1/2 AD -1/2 . The colour code indicates the ground-truth labels, while the sizes of the points are proportional to the node degree. Left: homogeneous degree distribution. Right:

heterogeneous degree distribution with θ i ∼ [U (3, 10)] 3 . For both plots n = 5 000, k = 2.
Remark . (Multiplicity of the zero eigenvalue). It can be shown [VM ] that on a connected graph, λ ↑ 2 (L) (also known as Fiedler eigenvalue) is strictly positive. On a disconnected graph, instead, the multiplicity of the 0 eigenvalue equals the number of connected components of G(V, E ), n cc and the corresponding eigenvectors can be written as v (1≤a≤n cc ) i = I i∈V a , where V a are the nodes belonging to the a-th connected component. Running Algorithm . with M = L and k = n cc returns a label assignment in which communities are identi ed by the connected components of the graph.

A similar result justi es the exploitation of the normalized graph Laplacian matrices (L rw = D -1 A and L sym = D -1/2 AD -1/2 ) that provide a relaxed solution of the NCut problem [VL ]. In [SM ], M = L rw , the informative eigenvectors correspond to the k largest eigenvalues and step is again not performed. Similarly to Remark . , the multiplicity of the eigenvalue 1 is equal to the number of connected components in G(V, E ), in fact

(D -A)v = 0 =⇒ D -1 Av = v.
The matrix L rw is also called random walk Laplacian because its entry

L rw ij = 1 d i A ij
is the probability that a random walker transitions from node i to node j. The choice M = L sym is instead proposed in [NJW a] in which the informative eigenvectors correspond to the k largest eigenvalues and step is performed. Note that the matrices L rw and L sym have the same set of eigenvalues and, the eigenvectors are also trivially related, in fact

D -1 Ax = λx D -1 A D -1/2 y x = λ D -1/2 y x D -1/2 AD -1/2 y = λy.
Strictly speaking, only the eigenvectors of L rw and not those of L sym provide a relaxed solution to the NCut problem. In several contexts, however, the use of L sym is preferred because it is Hermitian.

From a graph-cut point of view, all three Laplacian matrices L, L sym , L rw are able to create a node embedding which depends on the community labels and that thus justi es Algorithm . for the aforementioned choices of M. We recall, however, that a good embedding for k-means should also be independent from the degree distribution. Figure . compares the embedding obtained with the matrix L sym in the case of a homogeneous degree

The e ect played by the degrees in determining the small dimensional node embedding distribution (on the left) and of a heterogeneous degree distribution (on the right). The size of each dot is proportional to the degree of the corresponding node, clearly evidencing that nodes with a small degree tend to be mapped to nearby points, potentially hampering the performance of k-means. It is for this reason that step of Algorithm . is performed in [NJW a], prior to k-means, so to obtain an embedding unspoiled by the degrees. This is not needed instead for the embedding obtained from L rw that is naturally independent from the degree distribution [MRD ].

Concluding, both the algorithms of [NJW a] (based on L sym ) and of [SM ] (based on L rw ) consider and solve the problem induced by the presence of a heterogeneous degree distribution in the graph. On the opposite, the algorithm of [Fie ] (based on L) does not and its performance is consequently hampered on graphs with an arbitrary degree distribution.

Another classical choice of matrix M is in [New ], which is based on a relaxation of the modularity maximization problem. In this case M is the modularity matrix M mod = A -dd T 2|E | . For k = 2 communities, the algorithm of [New ] nds the labels according to the sign of the leading eigenvector of M mod , while for k > 2 the procedure is slightly more involved and it deviates from the general structure of Algorithm . . The matrix M mod is a rank one perturbation to the adjacency matrix, whose leading eigenvector Relaxation of the modularity maximization is strongly correlated (in the su ciently dense regime) to the degree vector:

(Ad) i = ∑ j∈∂i d j = d i c + O( d i ).
( . )

In the spectrum of M mod , the leading eigenvector of A (corresponding to a ferromagnetic con guration in which all entries have the same sign) corresponds to an eigenvalue close to zero. All other eigenvectors, being approximately orthogonal to d, are almost unperturbed. Consequently, the leading eigenvector of M mod is strongly related to the eigenvector associated to the second largest eigenvalue of A.

The choice M = A is thoroughly studied in [LR+ ] in which SC is performed on the k leading eigenvectors of the adjacency matrix. To keep into account for the degree heterogeneity, a similar approach is considered in [Jin+ ] for DCSBM-generated graphs. In this case, a modi ed version of step in Algorithm . is performed, normalizing each row of X by X i,1 .

Many results justifying the earlier or more sophisticated SC algorithms exist in the dense regime [LR+ ; CLV ; RCY+ ; MRD ; AC ; GLM b], also in the case for graphs with broad degree distributions. These are motivated by good concentration properties of the spectra of these matrices that can be predicted and studied with random matrix theory (RMT). However, as anticipated, real networks tend instead to be sparse and the aforementioned SC algorithms often perform poorly for practical applications. The origin of the problem if well understood comparing the two plots in Figure . . In the top plot, the spectrum of L sym (which we recall is the same as L rw ) is shown for a dense SBM-generated graph, clearly evidencing that the two largest eigenvalues are isolated. The bottom plot shows the spectrum of the same matrix for a sparse SBM with c = O n (1). In this case, the informative eigenvalue of L sym is "swallowed" in the bulk of uninformative eigenvalues and running Algorithm . for M = L rw does not lead to a good community partition. The spreading of the bulk is not limited to the Laplacian matrices, but characterizes also the adjacency matrix. In [BGBK+ ; BGBK+ ]

it was

Spectrum of A for c = O n (1) compared to the limiting distribution in the dense regime in blue

2 1 0 1 2 / c 0.0 0.5 1.0 1.5 A c ( )
recently shown for the SBM and c = o n (log (n)) that the spectral measure of A does not tend to a limit distribution and that the largest eigenvalue correspond to the nodes with highest degree, and not to the community structure.

It is important to underline that the spreading of the bulk is not only related to the asymptotic de nition of sparsity. As a practical example, we consider the political blogs network [AG ] representing relations between weblogs of USA politicians during elections. This network is particularly interesting as a benchmark for CD because a ground-truth label assignment can be de ned according to whether the politician is liberal or democrat. Running the algorithms of [SM ] or [NJW a] fails completely in recovering the ground truth assignment, since the vector x 2 is completely uninformative, as shown in the left plot of Figure . . It was however noticed in [JY+ ] that the eigenvector x 3 (associated to the third largest eigenvalue) is instead strongly correlated to the ground truth labels, as shown in the right plot of Figure . . This is a practical example in which the informative eigenvector is swallowed in the uninformative bulk. In this particular case it is still easy to nd but in more general settings, losing track of its Note that this network is undirected but in Figure . we consider a symmetrized version of the adjacency matrix. precise position is a practical obstacle for SC. Besides, losing the isolation of informative eigenvalues, the associated eigenvectors tend to merge with the eigenvectors associated to close-by (non-informative) eigenvalues.

. .

Finding e cient algorithms for SC in sparse graphs has been a challenge to which the scienti c community has devoted a lot of attention in the latest years. One prominent line of research studied the positive e ect that regularization has on SC. In particular, it was observed that using D τ = D + τI n and A τ = A + τ1 n 1 T n in place of D and A, much better concentration properties are achieved in sparse graphs and informative eigenvalues are restored to their dominant position [Ami+ ; JY ; LR+ ; LLV ; LLV ]. A particularly interesting method in this direction is proposed in [QR ] which uses

Regularized SC the regularized symmetric reduced Laplacian matrix L sym τ = D -1/2 τ AD -1/2 τ ,
for the heuristic choice τ = d, the average degree. In terms of Algorithm . , the algorithm proposed in [QR ] sets M = L sym τ , searches for the k eigenvectors corresponding to the k largest eigenvalues of M, and then performs the normalization step on the rows of the resulting matrix X to cope with an arbitrary node degree distribution. In [ZR ] it was shown that also the algorithm of [QR ] can be understood as a relaxation of an optimization problem, called CoreCut.

Generally speaking, the aforementioned works show that regularization improves the concentration properties of random matrices with low average degree. In all cases, however, the authors consider a moderately sparse regime in which the average degree goes (asymptotically) to in nity at a slow rate. The main theorems of these papers hence do not hold in the truly sparse regime in which c = O n (1). Besides, theory often suggests that large values of the regularizer τ should be preferred to small ones but small values of τ seem to perform better in practice [JY ; QR ]. In any case, regard-less the lack of full theoretical justi cation, the algorithm of [QR ] can be considered by all means as state-of-the-art in terms of performance, dealing with both sparsity (meant in this case as low average degree) and the broad degree distribution that typically characterizes real graphs.

The earliest contributions to the theory of SC in truly sparse graphs came from statistical physics intuitions. In particular, the authors of [Krz+ ] proposed an algorithm based on the non-backtracking matrix (De nition . ) for detection in the SBM. We recall from Chapter that the matrix B is obtained from the linearization of BP equations that provide, in this case, the asymptotically exact solution to inference in the SBM. The algorithm of [Krz+ ] Statistical physics methods for CD cannot be expressed exactly in terms of Algorithm . , since the size of B is |E d | and not n. However, the p-th column of X ∈ R n×k is obtained from the vector ψ(g p ), where g p is the eigenvector of B associated to the p-th largest eigenvalue and ψ(g p ) i = ∑ k∈∂i g p,ik . From Claim . , the authors of [Krz+ ] conjectured that, if α > α c (Equation ( . )), i.e. if there exists a polynomial-time algorithm capable of performing detection, then the vectors {ψ(g p )} p=2,...,k are correlated to the community structure and running k-means on the rows of X gives a label assignment with positive overlap. In order to show more clearly the origin of this claim, let us consider the case k = 2. According to Claim . , the eigenvector of B associated to its second largest eigenvalue is correlated to

The spectrum of B 2 0 2 4 Real( ) Imag( )
the class structure, with E[g 2,ij ] = σ j and so is ψ(g 2 ). The corresponding eigenvalue is λ

↓ |•| 2 (B) = c in -c out 2 + o n (1), while the radius of the bulk is √ c + o n (1)
. It is easy to check that the informative eigenvalue is isolated (λ

↓ |•| 2 (B) > √ c) whenever α > α c
. This result is of fundamental importance because it de nes a spectral algorithm in which the algorithmic and the detectability thresholds coincide. The spectral algorithm of [Krz+ ] was the rst (and still one of the few) obtaining this very powerful result. As we mentioned already, the conjectures of [Krz+ ] have later been formally proved [Mas ] and extended also to the DCSBM setting [GLM ] in which the nonbacktracking reaches as well the information-theoretic threshold.

A closely related algorithm is the one proposed in [SKZ ] which instead uses the eigenvectors attached to the smallest eigenvalues of the Bethe-Hessian matrix H r . Recalling Proposition . , the vector ψ(g p ) used in [Krz+ ] satis es the following relation:

H λ ↓ |•| p (B) ψ(g p ) = 0,
i.e. it is an eigenvector with zero eigenvalue of the Bethe-Hessian matrix for r = λ

↓ |•| p (B).
The matrix H r has some important numerical advantage with respect to B: it is smaller in size and it is Hermitian. However, translating the algorithm of [Krz+ ] in terms of the Bethe-Hessian requires to compute the k leading eigenvalues of B, annulling the advantage of using H r . In [SKZ ] Note that for g p=1 is the Perron-Frobenius eigenvector which is irrelevant for CD. the authors determined a unique value of r = λ

↓ |•| 1 (B)
, guaranteeing that the k smallest eigenvalues of H r are isolated for α > α c . As a by product, the authors also showed that the number of negative eigenvalues of H r for

r = λ ↓ |•| 1 (B)
is an estimator of the number of classes for the SBM. This is a very powerful result, because the algorithm of [SKZ ], unlike all others mentioned so far, does not need k as an input.

Remark . . Note that λ ) does not necessarily need to be calculated directly, but it can be estimated easily from the degree sequence d as follows, exploiting Theorem .

↓ |•| 1 (B
λ ↓ |•| 1 (B) = ∑ i∈V d 2 i ∑ i∈V d i .
Interestingly, the authors of [SKZ ] observed (with no further explanation) that SC based on their proposed Bethe-Hessian matrix performs slightly better than the non-backtracking matrix of [Krz+ ], especially for k > 2.

All these results are powerful as they propose algorithms capable of reaching the information-theoretic threshold, but they also have inherent weaknesses as they only guarantee a positive correlation of their output classication with the underlying true structure. Speci cally, these positive correlations do not imply that the classi cation performance is maximal. In particular, even in [GLM ] where spectral clustering on the DCSBM is studied, the problem of eigenvector pollution due to degree heterogeneity is not discussed and a fortiori not corrected. . In the state-of-the-art methods we presented, while sparsity is not properly accounted for in [Fie ; SM ; NJW a; New ; LR+ ; Jin+ ], the line of research based on B (and consequently H r ) [Krz+ ; SKZ ; BLM ; GLM ] does account for sparsity and provides methods to reach the detectability threshold; yet, all they guarantee is the possibility to obtain a positive, possibly suboptimal, correlation between the algorithm output and the underlying community structure. As for the works on regularization [Ami+ ; JY ; LR+ ; LLV ; LLV ; QR ], they only establish theoretical results of perfect community recovery far from the (most interesting) detection threshold.

In Chapter we show that a proper parametrization of the Bethe-Hessian matrix allows one to solve both the issues of sparsity and heterogeneity at once on DCSBM-generated graphs. We propose a simple SC scheme which provably performs non-trivial clustering as soon as α > α c and is robust to degree heterogeneity, as it retrieves eigenvectors not infected by the node degrees. Our central result comes in the form of a claim which is supported by three parallel non-rigorous but convincing arguments. The proposed algorithm is extensively tested on synthetic DCSBM graphs and our simulations evidence that it outperforms standard competing spectral techniques.

In this chapter we further provide a natural explanation to the observed higher performances of the Bethe-Hessian of [SKZ ] with respect to the non-backtracking of [Krz+ ].

Successively, Chapter studies SC based on the matrix L sym τ and shows why a small value of τ is preferable, drawing a connection to our results of Chapter . We further show for which value of τ the leading eigenvectors of L sym τ allow for non-trivial community reconstruction as soon as α > α c , addressing a further unanswered question of [QR ].

Chapter then considers the problem of e ciently translating our theoretical results of Chapters , into a practical algorithm which can be applied on arbitrary graphs. The proposed algorithm is extensively tested on real networks (for which the performance is measured in terms of modularity and DCSBM log-likelihood) and, in all cases, it outperforms or it is on par with the standard competing spectral techniques.

Finally, Chapter closes Part II, providing a new vision to SC and in particular a compelling new connection between our proposed approach and all aforementioned standard spectral methods, so far treated independently. In particular, we show that our proposed algorithm for CD represents a "bridge" between the aforementioned competing methods and is capable to self-adapt to di erent levels of sparsity as well of hardness of the clustering problem. Chapter showed that in spectral clustering (SC) theory it is challenging to address simultaneously sparsity and heterogeneity in the degree distribution that typically characterize real-world graphs. Consequently, many commonly adopted spectral algorithms have poor performances on real datasets because either one of these two problems is not kept into account. This chapter shows that the eigenvectors corresponding to the smallest eigenvalues of the unweighted Bethe-Hessian matrix H r , de ned as

The Bethe-Hessian matrix

H r = (r 2 -1)I n + D -rA, ( . )
can be exploited for a set of properly chosen values of r to e ciently perform SC in sparse graphs generated from the degree corrected stochastic block model (DCSBM) of De nition . . This proposed parametrization naturally accounts for the hardness of the detection problem and we claim it to be optimal in the sense that no other parametrization (including the one of [SKZ ]) achieves higher performance, at least for DCSBM graphs. Moreover, for k classes of equal size and C = c out 1 k 1 T k + (c inc out )I k , it obtains a positive overlap as soon as theoretically possible, achieving the DCSBM detectability threshold.

To the best our knowledge, there is no equivalent method which is explicitly adaptive to the hardness of the detection problem as the one described in this chapter. Moreover, the set of the optimal values of r can be directly and e ciently computed for any graph (hence not only for the DCSBM).

The central contribution of this chapter is a claim describing the shape of the eigenvectors attached to the smallest eigenvalues of the set of Bethe-Hessian matrices obtained for the optimal r values. This result comes in the form of a claim because, to the best of our knowledge, the mathematical tools to formally prove it are still lacking. The claim, however, is backed by several arguments as well as by numerous numerical simulations. For simplicity, we informally summarize here its content.

To each k-class graph, community detection is performed by rst associating a set of k Bethe-Hessian matrices {H ζ p } p=1,...,k , with ζ p de ned so that the p-th smallest eigenvalue of H ζ p is equal to zero. We then extract from each matrix the eigenvector x p attached to the zero eigenvalue (H ζ p x p = 0). The eigenvectors {x p } p=1,...,k are stacked in the columns of the matrix X Informal statement of the main result de ned in Algorithm . and used to produce the small dimensional node embedding on which the k-means algorithm is applied. Claim . justi es the relevance of the above procedure as an e cient community detection (CD) method on DCSBM graphs, stating that:

• The largest value of ζ p that can be de ned is ζ k . We show that, as a consequence, the (k + 1)-th smallest eigenvalue of H r is always positive for all r and the maximal number of negative eigenvalues of H r as function of r is precisely equal to k. This allows one to build an estimator for the number of classes.

• The same hypotheses must be veri ed to guarantee that the k dominant eigenvectors of the non-backtracking matrix (De nition . ) and the set of vectors {x p } p=1,...,k bring information on the class structure. This, for k classes of equal size and C = c out 1 k 1 T k + (c inc out )I k , implies that the vectors {x p } p=1,...,k can be used to reconstruct communities as soon as theoretically possible, i.e. when α > α c .

• Unlike what was observed in Figure . where the eigenvectors of L sym = D -1/2 AD -1/2 (but similar comments may be made for A, D -A, H r for a suboptimal choice of r) were tainted by the degree heterogeneity, the entries of {x p } p=1,...,k are not; consequently they provide a suited input for k-means.

The properties of {x p } allow us to de ne an e cient algorithm for CD in sparse graphs with a heterogeneous degree distribution, solving at once the two main challenges discussed in Chapter . We now proceed with the formal statement of Claim . , together with a clear de nition of the hypotheses under which it is formulated. . . .

Let G(V, E ) be a k class DCSBM generated graph as in De nition . . Recall that C ∈ R k×k is the class a nity matrix, while π ∈ R k is the vector determining the expected class sizes, with 0 < π p < 1 the probability of an arbitrary node to belong to class p. In order to consider an asymptotically non-trivial community detection setting, we work under Assumption . , which we recall here for convenience.

Assumption . . Let C ∈ R k×k be the class a nity matrix and π be the expected class size vector as in De nition . , with Π = diag(π). Further let θ be the vector encoding the degree heterogeneity in the DCSBM and denote

Φ = E[θ 2 i ].
The matrices C, Π satisfy the following hypothesis:

• CΠ1 k = c1 k , for some positive c, • λ p (CΠ)Φ > √ cΦ for all 1 ≤ p ≤ k.
In the particular case in which π =

1 k /k and C = c out 1 k 1 T k + (c in - c out )I k , it is easily veri ed that λ p (CΠ)Φ > √ cΦ implies α > α c = 1, i.
e. we assume to be above the detectability threshold. In a more interesting setting in which communities have arbitrary sizes, Assumption . consists in assuming that the number of isolated eigenvalues of the non-backtracking matrix is equal to k. Note that, as we mentioned already in Chapter , Assumption . implies that, with high probability, the graph G(V, E ) has a giant component, in agreement with Theorem . .

Remark . (Negative eigenvalues of CΠ).

It must be noted that Assumption . implies λ p (CΠ) > 0 for all 1 ≤ p ≤ k. However, the results of Claim . remain valid if one replaces the second point of Assumption . by |λ p (CΠ)Φ| > √ cΦ, that is, λ ↓ p (CΠ) for p ≥ 2 may be of arbitrary sign and only its modulus may be lower-bounded; λ ↓ 1 (CΠ) = c, being the Perron-Frobenius eigenvalue, is necessarily positive in any case.

As the notations to describe this more general case are more cumbersome, we prefer here to focus on the simpler case where CΠ 0. Further comments to the more general setting in which CΠ may have negative eigenvalues will be made in the following. We now proceed providing the rigorous statement of our result.

. .

H r

The central result of this chapter revolves around the characterization under Assumption . of a particular set of values r = ζ p which must be carefully selected to best operate on H r . These values are de ned as follows.

De nition . (ζ p ). Consider an arbitrary graph G(V, E ) composed of n cc connected components. Let G (j) (V (j) , E (j) ) be the subgraph reduced to the j-th connected component, and H an integer between 1 and n. We de ne, if it exists, ζ (j)

p as:

ζ (j) p =min r≥1 {r : λ ↑ p (H (j) r ) = 0}. ( . )
In words,

ζ (j)
p is, if it exists, the smallest value of the parameter r ≥ 1 such that the p-th smallest eigenvalue of H (j) r is null.

Note that, even though we focus in the remainder on DCSBM-generated graphs, the ζ p 's are well de ned on arbitrary graphs. Here we list a few important properties of these values:

. At r = 1, H (j) 1 = D (j) -A (j)
is the combinatorial Laplacian of the subgraph associated to the j-th connected component. As this subgraph is by de nition connected, it is well known [CG ] q for q > p do not exist.

. Empirically, on as many connected graphs as we could think of, we have observed that the function λ ↑ p (H r ) for r ≥ 1 either never crosses zero (in which case ζ p does not exist), crosses zero exactly twice and is convex between these two crossings (in which case ζ p is the lowest of the two values), or, in very symmetric cases, touches zero exactly once without crossing it (in which case ζ p is that value).

With the de nition of ζ p being laid out, we are now in position to state our main result.

Claim . . Let G(V, E ) be generated according to De nition . , that is, a DCSBM with k classes. Let D and A be its degree and adjacency matrices, and H r = (r 2 -1)I n + D -rA, for r ∈ R, its Bethe-Hessian matrix. Provided that Assumption . is satis ed for c = O n (1), we have with high probability for all large n that:

• There is only one connected component j for which ζ

ζ p =min r>1 {r : λ ↑ p (H r ) = 0}.
( . )

• The largest p for which ζ p exists is equal to k, the number of underlying communities of the DCSBM. One has

1 = ζ 1 < ζ 2 ≤ . . . ≤ ζ k ≤ √ cΦ.
More precisely:

∀ p s.t. 1 ≤ p ≤ k, ζ p = c λ ↓ p (CΠ) + o n (1). ( . ) • For 2 ≤ p ≤ k and ζ p ≤ r ≤ √ cΦ, the p smallest eigenvalues of H r are isolated.
In particular, zero is an isolated eigenvalue of H ζ p . Its corresponding eigenvector x p is correlated to the community structure and the entries of x p are in expectation (over realizations of A) independent of the degrees of the graph.

In simple terms, Claim . predicts that, in a graph of k communities, SC can be successfully performed by successively retrieving the eigenvector associated to the null eigenvalue of each of the matrices H ζ 2 , . . . , H ζ k . In the speci c case of k classes of equal size with C aa = c in and C ab = c out if a = b, Claim . states that for 2 ≤ p ≤ k the ζ's are degenerate. As we will see next, this value of r di ers from the choice r = λ [SKZ ], unless α = α c , i.e. , exactly at the detectability threshold.

↓ |•| 1 (B) = √ cΦ advo- cated by
Figure . provides a visual representation of the typical spectrum of H r for r not too far from a ζ p : the eigenvalue of H r closest to zero is clearly isolated and the eigenvector associated to this eigenvalue is strongly aligned to the community structure. In Figure . , for a typical realization of a DCSBM, λ ↑ p (H r ) is represented as a function of r for p = 1, 2, 3 and 4 in solid lines

This statement allows to de ne ζ p with respect to the Bethe-Hessian matrix of the whole graph, instead of the Bethe-Hessian matrix of its giant component. The fact that Equation ( . ) is veri ed with high probability is not evident. In fact, considering all the disconnected components at once might change the ordering of the eigenvalues. More details are to be found in Appendix B. Precisely, we will see that λ ↑ k+1 (H r ) > 0 for all r > 1 with high probability, so that ζ k+1 is not de ned; in fact it was interestingly shown in [SKZ ] and we will verify that λ The non-obvious parts of the claim are (i) of course that such ζ p 's do exist up to p = k, (ii) that they are concentrated around a deterministic value depending on the statistics of the model, (iii) importantly, that zero is indeed an isolated eigenvalue in the spectrum of H ζ p , (iv) that the associated eigenvector is informative for the underlying community structure and that is not infected by the degrees of the graph.

↑ k+1 (H √ cΦ ) ↓ 0 + as n → ∞,
As we already mentioned, the statement of Claim . is formulated as a claim in the sense that, while e orts have been made to rigorously prove parts of the result [CZ ], the mathematical tools required to fully prove Claim . , to the best of our knowledge, do not exist yet. Instead, the remainder of this section will propose three complementary supporting elements, arising from non-rigorous but convincing approximations, in particular borrowing arguments from the eld of statistical physics. Speci cally, we will successively show • under Section . . that the vectors x p , solution of H ζ p x p = 0, are correlated with the community labels.

• under Section . . , that the informative null eigenvalue of H ζ p (associated to x p ) is located in p-th smallest position and is isolated. This result is algorithmically crucial to determine ζ p itself.

• under Section . . , that the entry x p,i is essentially independent of d i , the degree of node i; precisely, it is strictly independent of d i on average (over random allocation of the labels) and only depends on d i through a noise term of order 1/ √ d i otherwise.

• under Section . . we brie y recap the proof of [CZ ] that considers the moderately sparse regime. We will in particular focus on both the main intuitions and weaknesses of the proof.

. . .

We rst proceed to our agenda by arguing that the eigenvectors x p of H ζ p are correlated with the community labels. Consider the belief propagation (BP) equations ( . , . , . ). These are the asymptotically Bayes optimal solution for inference in the sparse DCSBM. As we saw in Chapter for the Ising model, these equations admit a trivial xed point, for

η ij ( i ) = π i , ∀ (ij) ∈ E d .
It is a trivial solution because it leads to a random node partition. However, the non-backtracking matrix naturally comes into play linearising Equations ( . , . , . ) around this trivial xed point.

(B) R, 2 (B) R, 3 (B) R, 4 (B) c 2 (C ) c 2 (C ) 1 c
θ i ∼ [U (3, 10)] 3 , c = 6.
In particular, let

η ji ( i ) = π i + b ji ( i ) for small b • (•). Letting b T = [b T 1 , . . . , b T k ] ∈ R |E d |k , with b p ∈ R |E d | containing the entries of b ij (p)
, we obtain as in [Krz+ ]:

Linearization of the asymptotically Bayes optimal BP equations

(T ⊗ B)b = b + o n (1), ( .
)
where ⊗ is the Kronecker product and T = ΠC c -Π1 k 1 T k . From Equation ( . ), our interest is in the eigenvalues of (T ⊗ B) equal (or close) to one. From the properties of the Kronecker product, the set of eigenvalues of (T

⊗ B) is Λ(T ⊗ B) = {λ i (B)λ j (T) : 1 ≤ i ≤ |E d |, 1 ≤ j ≤ k}.
This induces a one-to-one relation between the k -1 non-zero eigenvalues of T and the k -1 eigenvalues of B, that need to satisfy the relation λ q (T)λ t (B) = 1 for some t, q. From the expression of T, it comes that the pth largest eigenvalue of T satis es λ

↓ p (T) = 1 c λ ↓ p+1 (CΠ) for 1 ≤ p ≤ k -1 (see Appendix B.
for details) so that, for 2 ≤ p ≤ k, there must exist

ζ p ∈ Λ(B) such that: ζ p = c λ ↓ p (CΠ) + o n (1). ( . )
To these eigenvalues correspond k -

1 exact eigenvectors g p ∈ R |E d | of B satisfying Bg p = ζ p g p (with g p = b p + o n (1)
) that are naturally "informative" as they are associated to structural eigenvalues of T. Also, from a BP standpoint, these eigenvectors are small deviations from the uninformative xed point, so must point towards informative directions.

Note that λ ↑ Note importantly here that Equation ( . ) de nes ζ p as a real eigenvalue of the matrix B. So far, this de nition needs not correspond to the claimed de nition of ζ p as introduced in Claim . : Section . . will show that these two de nitions are indeed equivalent. In addition, by Assumption . , √ cΦ, these are the only informative eigenvalues of B. Speci cally, the real eigenvalues of B inside the bulk are divided between (i) the (non-informative) eigenvalues -1, 0, 1, and (ii) the

λ ↓ 1 (CΠ) = c
k eigenvalues λ = ζ p ≈ c λ ↓ p (CΠ) ≤ √ cΦ just described.
Having identi ed the k -1 informative eigenvectors g 2≤p≤k ∈ R |E d | of B, we now recall that the embedding vectors the form the column of X ∈ R n×k appearing in Algorithm . are obtained from ψ(g p ) i = ∑ k∈∂i g p,ki .

The vector ψ(g p ) further satis es

The eigenvector attached to the null eigenvalue of

H ζ p is informative H ζ p ψ(g p ) = 0,
hence ψ(g p ) ≡ x p . Note that, for ζ 1 = 1 we have H 1 = D -A and x 1 is the vector 1 n , which is irrelevant to reconstruct communities.

Let us speci cally focus on the particular case of k classes of equal size with the usual assumption C ab = c in if a = b and c out otherwise. In this case e cient algorithms for CD exist only if α > α c . Under this condition, B has two informative eigenvalues ≈ (cc out )Φ and ≈ c c-c out on either side of the disk (bulk) of radius √ cΦ. As the detection problem becomes harder, the two eigenvalues get closer together until they (asymptotically) meet right at the detectability transition α = α c where (cc out )Φ = √ cΦ. Further increasing the detection di culty (that is, for α < α c ), the two eigenvalues become complex conjugate. This behavior is shown in Figure . (right panel) for k = 2. Summarizing, we have for

2 ≤ p ≤ k The ζ p 's are isolated eigenvalues of B if α > α c (c -c out )Φ = c c -c out = √ cΦ, when α = α c (c -c out )Φ > √ cΦ > c c -c out > 1, when α > α c .
The fact that the leading eigenvalues of largest amplitude remain isolated down to the detectability threshold is the strongest argument in favor of the algorithm proposed by [Krz+ ]. The authors in [Krz+ ] however ignored the e ect on the corresponding eigenvalue inside the bulk of B which, from our present discussion, is similar. This behavior can be extended to more than two classes with arbitrary sizes for which, however, the hypotheses of the conjecture of [Dec+ ] do not hold.

Let us summarize these results in the form of the following argument.

Argument . . Under Assumption . for all large n with high probability, the non-backtracking matrix B of a graph G(V, E ) generated from a k-class DCSBM has k -1 isolated real eigenvalues inside the disk of radius √ cΦ (bulk). These eigenvalues are found at positions

1 < ζ p = c λ ↓ p (CΠ) + o n (1) for 2 ≤ p ≤ k.
Besides, the eigenvectors g p of B associated with these eigenvalues ζ p bring information about the community structure, which can be extracted through the vectors x p ∈ R n , de ned as:

x p,i = ∑ j∈∂i g p,ij , Bg p = ζ p g p ;
or equivalently satisfying

H ζ p x p = 0.
Element . provides a rst statement of Claim . , according to which the class information should be retrieved from the vector x p solution to

H ζ p x p = 0, and that ζ p = c/λ ↓ p (CΠ) + o n (1)
. This argument however does not specify the location (in the ordered list of the eigenvalues) of the null eigenvalue of H ζ p to which the eigenvector x p corresponds nor the structure of the eigenvector x p , and in particular its dependence on the degrees of the graph. These aspects are covered in the subsequent sections.

. . G

In this section, through a statistical physics mapping between Bayesian inference and an Ising model de ned on the graph G(V, E ), we aim to informally justify why the smallest eigenvalues of the matrix H r correspond to "informative states" of the system and why the zero eigenvalue of H ζ p (associated by de nition with the informative eigenvectors x p ) should be isolated and in the p-th smallest position of the spectrum of

H ζ p .
The smallest eigenvalues of H r are informative.

In this section we provide a motivation on why the eigenvectors associated to the smallest eigenvalues of H r should be the informative ones. For simplicity, let us rst consider the case of k = 2 classes of equal size. We write the Ising Hamiltonian on the graph G(V, E ), with communities:

H(s) = -∑ (ij)∈E ath 1 r s i s j . ( . )
The term βJ has here been replaced by ath 1 r for convenience. We now study the magnetization of this Hamiltonian with the (asymptotically exact) Bethe approximation detailed in Chapter . Let us recall the basics on the Bethe-Hessian matrix. The quantities m i = s i β and χ ij = s i s j β cannot be computed analytically on G(V, E ), but are well approximated by The appearance of new local minima can be obtained from the spectral properties of the Bethe-Hessian matrix H r that corresponds to the Hessian of FBethe

β ( m, χ) at m = 0 n . ∂ 2 FBethe β ( m, χ) ∂ mi ∂ mi = (r 2 -1)I n + D -rA r 2 -1 ≡ H r r 2 -1 .
The Hessian in m = 0 n is therefore trivially related to H r . Negative eigenvalues in the spectrum of H r correspond to the appearance of new local minima in FBethe β ( m, χ) that can be approximated by the corresponding eigenvectors. Recalling once again that for large r, all eigenvalues of H r are positive, then the eigenvalues that may become negative (and are hence associated to a non-trivial magnetization) are the smallest ones. In Chapter we already considered the paramagnetic-ferromagnetic phase transition in which there is one negative eigenvalue in the spectrum of H r with the corresponding eigenvector having all positive entries. When G(V, E ) is a realization of the DCSBM with k = 2, however, also the second smallest eigenvalue of H r may become negative: this is a consequence of Theorems . (describing the spectrum of B) and . (describing the relation between the eigenvalues of B and H r ). This eigenvalue corresponds to a further (local) minimum of the Bethe free energy which is induced by the community structure of G(V, E ). Let us summarize some important facts:

• A local minimum correlated to the class structure appears in FBethe β ( m, χ) because the con guration in which s i = 1 if i = 1 and s i = -1 if i = 2 is close to a local minimum of H(s): for two classes of equal size, minimizing H(s) with s ⊥ 1 n is equivalent to minimize the RCut.

• This informative minimum corresponds to the second smallest eigenvalue of H r , because the ferromagnetic con guration has always a lower energy and it corresponds to the global minimum of FBethe β ( m, χ)

The eigenvector associated to the second smallest eigenvalue of H r is informative at su ciently low temperature.

• The interpretation of the eigenvectors of H r is rigorous for large values of r (i.e. , at the paramagnetic point), but a similar behavior is expected to be observed also for other values of r. In particular, the informative eigenvector generally depends on r but it is still correlated to the class structure and found in second smallest position.

This relation between H r and the Ising Hamiltonian allows us to make a further comment on the choice r = ζ p . Recall from Equation ( . ) that exact Bayesian inference can be performed for k = 2 studying the magnetization of the Hamiltonian

H(s) = -∑ (ij)∈E 1 2 log c in c out s i s j -∑ i∈V h i (s)s i , ( . )
where we recall that h i (s) is a local eld that prevents all node from being in the same class. From a straightforward calculation, we get

1 2 log c in c out = ath c in -c out c in + c out = ath 1 ζ 2 + o n (1) .
Relating Equation ( . ) to Equation ( . ), the choice r = ζ 2 corresponds to setting at the Bayes optimal inference temperature. Yet, it should not be forgotten that in Equation ( . ) the local magnetic eld does not appear. The consequence of neglecting this eld is that the informative eigenvector of H r does not correspond to the global (hence stable) minimum of FBethe β ( m, χ), but to a metastable local minimum. Discarding the eld term is analytically convenient, but it creates an uninformative eigenvector in smallest position.

Let us now conclude this discussion, brie y commenting the case k > 2. It appears evident that the mapping between the Ising Hamiltonian and the Bayes optimal solution is, in this case, not straightforward. This is because labels, unlike spins, may take k > 2 di erent values. Nonetheless, a Ising Hamiltonian on G(V, E ) with k > 2 is still useful for CD because in this case k -1 local minima appear in the function FBethe Zero is an isolated eigenvalue of H ζ p .

This result follows from the observation, reported in Figure . , that the kth smallest eigenvalues λ ↑ 1 (H r ), . . . , λ ↑ k (H r ) of H r successively equal zero in this order as r increases. Starting from r = 1, H r = H 1 = D -A for which we know that λ ↑ j (H 1 ) = 0 for 1 ≤ j ≤ n cc with n cc the number of connected components and λ ↑ i (H 1 ) ≥ 0, i > n cc . Increasing r beyond r = 1, the successive smallest eigenvalues of H r rst all increase and remain equal to the left edge of the bulk before escaping, each in turn (and in the order ) towards zero until it reaches asymptotically zero in the limit where r = √ cΦ [SKZ ], before increasing again away for r > √ cΦ. These last ndings may be summarized as follows.

λ ↑ 2 (H r ), . . . , λ ↑ k (H r )),
Argument . . Given a graph G(V, E ) with k classes, generated from the DCSBM, the p smallest eigenvalues of the matrix H r are isolated for ζ p ≤ r ≤ √ cΦ, for all 2 ≤ p ≤ k. The entries of the p smallest eigenvectors are correlated with the class labels. Besides, in the speci c case where r = ζ p , the p-th smallest eigenvalue of H ζ p is equal to zero.

Argument . answers the problem of locating the informative eigenvalueeigenvector pair (0, x p ) of H ζ p introduced in Argument . , and of justifying why it is isolated. The last step, developed in the next section, consists in understanding the structural content of the informative eigenvector x p .

. . From a purely algebraic standpoint, the Bethe-Hessian matrix H r may be seen as a regularized combinatorial graph Laplacian. In particular, we now show that r = ζ p is the good parametrization for which the informative eigenvectors are resilient to degree heterogeneity. Our argument exploits the local convergence of G(V, E ) to a Galton-Watson (GW) tree, that has been extensively commented in . . .

The labelled GW tree i

Fixing A (and thus the degrees d i ), we consider the probability distribution of node label i , conditionally to the label of one of its neighbours j :

P( i | j , A ij = 1) = P( i , j |A ij = 1) P( j |A ij = 1) = dθ i dθ j P( i , j , θ i θ j |A ij = 1) P( j ) = dθ i dθ j P(A ij = 1|θ i , θ j , i , j )P( i )P( j )P(θ i )P(θ j ) Zπ j = π i C i , j c = (ΠC) i , j c = (CΠ) j , i c . ( .
)
Let v p be the eigenvector associated to the p-th largest eigenvalue of CΠ and let us de ne u p ∈ R n with u p,i = v p, i the n-dimensional class-wise expansion of v p : this vector u p is inherently random as the class allocations 1 , . . . , n are here considered random. Then, for n large, under the limiting tree approximation with conditionally independent o springs, we take an expectation over the random allocation of the labels di erent from i (denoted with \i ), with A and i known: ) where the approximation follows from the fact that conditional independence of the neighbors of a given node only holds asymptotically. As a consequence of Equation ( . ), we nd that

E \i [(Au p ) i |A, i ] = ∑ j∈∂i E \i [u p,j |A, i ] = ∑ j∈∂i E \i [v p, j |A, i ] ≈ ∑ j∈∂i ∑ j P( j | i , A ij = 1)v p, j = 1 c ∑ j∈∂i ∑ j (CΠ) i , j v p, j = 1 c ∑ j∈∂i (CΠv p ) i = d i c λ ↓ p (CΠ)v p, i = d i λ ↓ p (CΠ) c u p,i ( .
E \i [H r u p |A, i ] ≈ (r 2 -1)I n + D 1 -r λ ↓ p (CΠ) c u p , ( . )
In order for this equation to be an approximate eigenvector equation for arbitrary degrees in D, the right hand-side term proportional to D must vanish. That is, one must select

r = c λ ↓ p (CΠ) ≈ ζ p
(this last approximation having been introduced and discussed in Section . . ). This result implies that the eigenvectors x p de ned as H ζ p x p = 0, for 1 ≤ p ≤ k, correspond to a noisy version of u p which is not a ected on average over the class allocation by the degree distribution. The approximation

The expectation of x p is independent of the degree distribution holds beyond the average for every typical realization of the class allocation to rst order in d i . In particular, denoting ξ p the "noise" vector satisfying x p = u p + ξ p , and thus H ζ p (u p + ξ p ) = 0, we get

Var \i [(H ζ p u p ) i |A, i ] = Var \i [(H ζ p ξ p ) i |A, i ] = Var \i [(ζ 2 p -1 + d i )ξ p,i -ζ p (Aξ p,i )|A, i ] ≈ O d i (d 2 i )Var \i [ξ p,i |A, i ] + O d i (d i )Var \i [ξ p,i |A, i ] ≈ O d i (d 2 i )Var \i [ξ p,i |A, i ]. ( . )
In this derivation we used the fact that the random variables ξ p,i and [Aξ p ] i are essentially independent and that the variance of the sum of the d i asymptotically independent variables ξ p,1 , . . . , ξ p,n grows linearly with d i . Now, proceeding as in Equation ( . ), we can compute

Var \i [(H ζ p u p ) i |A, i ] from a direct calculation of E \i [(H ζ p u p ) 2 i |A, i ] and E 2 \i [(H ζ p u p ) i |A, i ],
and we obtain (to leading order in

d i ) Var \i [(H ζ p u p ) i |A, i ] = O d i (d i ).
Combining with Equation ( . ), we get that the variance scales approximately as d -1

i . The vector x p can therefore be written as the sum of the deterministic information and a noise with amplitude inversely proportional to the square root of the degree, consistently predicting that nodes with higher degrees are easier to classify.

These results are summarized under the form of our last argument.

Argument . . The eigenvector x p (1 ≤ p ≤ k), solution to H ζ p x p = 0, is a noisy version of the vector u p , de ned as u p,i = v p, i , where CΠv p = λ ↓ p (CΠ)v p . The noise for entry i scales approximately as 1/ √ d i for d i suciently large and is zero on average. Consequently, the entries of x p do not, to rst order, depend on the degree distribution but only on the labels.

Figure . con rms our argument: a k = 2 class DCSBM graph is considered and the informative eigenvector x 2 is plot for r = ζ 2 and r = √ cΦ (as in [SKZ ]) with the marker size of the i-th entry being proportional to 1/ √ d i . We see that for a suboptimal choice of r, all nodes with a small degree are embedded to nearby points. For r = ζ 2 instead, nodes with a small degree are more likely to be misclassi ed, but they are not embedded to nearby points. Moreover, the histograms of the eigenvector entries further con rm this observation. For the optimal regularization, two distinct clouds density peaks appear, while for r = √ cΦ most eigenvector entries (in the example we chose a power law degree distribution) are close to zero.

The following remark summarizes the major consequence of this result. histograms, one sees that, for r = √ cΦ, at 0 there is a density peak, while at r = ζ 2 the density is minimal. For this reason, a slight deviation in the location of y leads to the misclassi cation of several nodes in the earlier case, but not in the latter. We can say that the optimal choice of r produces an embedding that is more robust to the k-means step. For pictorial purposes, Figure . limits to the case k = 2, but the same observation holds for k > 2 classes of arbitrary size in which the negative e ect on k-means of a low quality embedding is even more exacerbated.

Element . combined with Elements . , . completes our justi cation of Claim . . In the next section we give a further argument related to the existence and location of the ζ p 's that holds for moderately dense graphs generated from the stochastic block model (SBM).

. . ζ p

This section brie y provides the main result of [CZ ] in which the authors attempted to prove parts of Claim . . They focused in particular on the existence of isolated eigenvalues inside the bulk of B in the case of k = 2 communities of equal size. For simplicity, the authors of [CZ ] considered a su ciently dense regime with log(n)/c = o n (1) under the SBM assumption. As already discussed in Section . . this setting is particularly convenient due to the degree concentration: D -cI n = o n (c). This allows one to relate the eigenvalues of B and A. In particular, for r ∈ C \ {±1} an arbitrary eigenvalue of B, there exists r = r + o n (r) satisfying

In the dense SBM setting, the eigenvalues of B are trivially related to

those of A r = λ i (A) ± λ 2 i (A) -4(c -1) 2 ,
as it was already showed in Equation ( . ). For any λ i (A) > 4(c -1) one simply obtains that r has two solutions (denoted with r± ) that satisfy

r+ = λ i (A) + o n (c) r-= c r+ + o n (1).
In particular, for the SBM and

1 ≤ i ≤ k, one has that λ ↓ i (A) = λ ↓ i (CΠ) + o n (c
), con rming Claim . and naturally justifying the mapping between inner and outer isolated eigenvalues of B. The main technical contribution of [CZ ] is to prove that r = r + o n (r) and will not be further commented here. Albeit the proof [CZ ] provides a very natural explanation to the existence of isolated eigenvalues in the bulk of B, it strongly relies on the degree concentration assumption which does not hold in the sparse regime and, above all, for the DCSBM. As we commented already in Section . . , this is likely to be only a technical hypothesis since the empirical evidence suggests that the behavior of the isolated eigenvalues of B is essentially preserved in the presence of an arbitrary degree distribution.

. In this section, Claim . is translated into Algorithm . for SC and tested on synthetic graphs. In Algorithm . , the values of ζ 2≤p≤k are successively estimated and the eigenvector corresponding to the zero eigenvalue of H ζ p is stored in the columns of X. This matrix determines the small dimensional node embedding to on which k-means is applied.

Remark . (Value of k). Note that Algorithm . requires k as an input. If not known in advance, k can be estimated counting the number of negative eigenvalues of H √ cΦ , according to [SKZ ]. Further comments on how to e ciently implement this estimation will follow in Chapter . problem. Blue sharp diamonds are the result given by Algorithm . , red diamonds the algorithm using the Bethe-Hessian of [SKZ ], green squares are for the non-backtracking of [Krz+ ], yellow pentagons the adjacency matrix as in [LR+ ], purple stars the random walk Laplacian, black dots are the algorithm of [QR ], the pink triangles are the Bayes optimal solutions obtained with BP [Dec+ ]. For both graphs n = 50 000,

θ i ∼ [U (3, 10)] 4 , π ∝ 1 k , c = 5, C = c out 1 k 1 T k + (c in -c out )I k . Averages over 10 samples.
k = 2 classes of equal size are considered. When α is large enough, small values of r lead to better partitions than large values of r that are more affected by degree heterogeneity. However, for r small, the informative eigenvector is not necessarily corresponding to the second smallest eigenvalue, leading to a meaningless partition. On the contrary, larger values of r show isolated eigenvectors also in the "hard regime". We recall that r = ζ 2 is an α-dependent parameter: for α → α c , ζ 2 is "large enough" so that the informative eigenvalue is isolated, while for α α c it is "small enough" to give good partitions. Also the value of r = (c inc out )Φ/2 is α-dependent and it corresponds to clustering with B as indicated in [Krz+ ]. While it gives good partitions very close to the transition, this choice of r seems largely sub-optimal for easier tasks.

From a direct computation, Figure . hence suggests that the choice r = ζ p is indeed optimal, in the sense that no other value of r achieves a better performance on DCSBM-generated graphs.

. This chapter introduced the main theoretical result of this manuscript, formulated in the form of a claim under the DCSBM assumption. Several independent arguments as well as numerical tests foster the correctness of Claim . . Summarizing, the key contributions of this chapter can be listed as follows.

• We show the existence of a parametrization of the Bethe-Hessian matrix that allows one to produce a node embedding resilient to the presence of a broad degree distribution. This embedding is obtained with- out any preprocessing on the matrix X appearing in Algorithm . and naturally discards the "noise" introduced by the degrees.

• The proposed parametrization is deeply related to the parameters of generative model and with the Bayes optimal inference solution. With Algorithm . it is possible to estimate the optimal parametrization in an e cient and completely unsupervised fashion.

• We claim, backed by several arguments and simulations, that the proposed parametrization is optimal, in the sense that no other parametrization of H r leads asymptotically to better performances on graphs generated from the DCSBM.

All the results above have been thoroughly studied for an assortative community structure, in which each node has the largest probability to get connected to nodes in its own class. As suggested in Remark . , however, all the results can be formulated in a disassortative setting. For instance, for k = 2 communities, let c in < c out . The corresponding graph has more edges connecting nodes in opposite communities that those connecting nodes in the same community and is, therefore, disassortative. The second eigenvalue of CΠ is (c inc out )/2 < 0 and consequently ζ 2 < 0.

Disassortative graphs

Recalling the relation between CD and the Ising model, a negative value of ζ 2 consists in placing an antiferromagnetic interaction J < 0, favouring con gurations in which neighbouring nodes have misaligned spins. In this case, the ferromagnetic con guration 1 n would not correspond to the global minimum of the Bethe free energy and the informative eigenvector would be associated to the smallest (and not second smallest) eigenvalue of H r<0 . More general cases can be designed in which both positive and negative eigenvalues appear in the spectrum of CΠ. While the mathematical result is well de ned, the interpretation of the negative eigenvalues of CΠ is not straightforward. As an example, the right frame of Figure . displays a network designed from the SBM with k = 4, Π ∝ I k and C de ned as:

C = c in c out c out c in ⊗ c out c in c in c out ( . )
with '⊗' the Kronecker product. This graph has a hierarchical structure: it can be divided into two assortative communities, each of them composed of two disassortative communities. In Figure . the position of each node is assigned according to its true label, while the color is assigned according to the output of k-means on the vectors x p solution to H ζ p x p = 0. We can observe that our algorithm performs well also in this particular case.

The results of this chapter introduce some relevant questions that will be answered in the following. In particular, Figure . shows that the algorithm of [QR ] performs very competitively with Algorithm . . The theory justifying the algorithm of [QR ], however, does not cover neither the case of sparse graphs with c = O n (1), nor it provides any comment on the possibility of this algorithm to perform non trivial clustering down to the theoretical detectability threshold α c . The high performance achieved by [QR ], however, requires a deeper understanding of the role of regularization in SC.

Chapter brings new conclusions in this direction working on an explicit relation between the Bethe-Hessian and regularized Laplacian matrices. We will show, in particular, that the regularized Laplacian matrix of [QR ] can indeed be exploited to achieve non-trivial clustering as soon as α > α c in the sparse regime c = O n (1), yet for a slightly di erent value of the regularizer τ with respect to the one proposed in [QR ]. Chapter showed that a conveniently parametrization of the Bethe-Hessian matrix allows one to de ne a spectral clustering (SC) algorithm that deals at once with the sparsity and heterogeneity in the degree distribution that typically characterize real-world graphs. Figure . evidences that the e ciency of our proposed Algorithm . is unbeaten by all competing spectral algorithms on degree corrected stochastic block model (DCSBM)-generated graphs. From this simulation, it however also appears evident that the algorithm of [QR ] obtains very similar performances.

In the literature [QR ; Ami+ ; JY ; LR+ ; LLV ], regularized spectral clustering has been considered for its ability to improve the concentration properties of random matrices related to graphs with low average degree. A clear representation of the positive e ect of regularization is given in with τ = c. For both plots n = 5 000,

θ i ∼ [U (3, 7)] 3 , k = 2, c = 5, c out = 2.
[QR ], left some relevant questions unanswered on the role of regularization in SC that we list as follows:

• The theoretical results of [QR ] together with the proposed choice τ = d (the average degree) are consistent only in the moderately sparse regime in which the expected average degree goes to in nity faster than log(n). Nevertheless, the proposed algorithm is very ecient also for c = O n (1), as shown in Figure . .

Open questions regarding regularized SC

• There exists no theoretical guarantee or conjecture of whether or not the regularization allows one to achieve detectability down to the theoretical threshold.

• In the literature, the characterization of the parameter τ was never properly addressed and its assignment was left to a heuristic choice.

Both in [QR ] and [JY ] the results provided by the authors seem to suggest a large value of τ, but it is observed experimentally that smaller values of τ give better partitions. In the end, the authors in [QR ] settle on τ = d and do not further address the problem of identifying the optimal parametrization.

Working on an explicit relation between the Bethe-Hessian and the regularized Laplacian matrices, in this chapter we provide a natural explanation to regularization. We further give a precise result on the optimal (hardness dependent) regularization as well as on the value of τ needed to achieve detectability down to the theoretical threshold. As it is for Claim . , also in this case the optimal regularization can computed in an unsupervised way. We further naturally explain why small regularizations should be preferred to large ones, answering the three questions raised above. These results create a bridge between statistical physics and regularization techniques.

. . As we just mentioned, our main result revolves around a relation between H r and L sym τ . Letting L rw τ = D -1 τ A, the mapping unfolds from the following basic remark: for ζ p > 1 (as de ned in Equation ( . )),

D τ = D + τI n [(ζ 2 p -1)I n + D -ζ p A]x p = 0 ( . ) ⇔ [D + (ζ 2 p -1)I n ] -1 Ax p = 1 ζ p x p (
∈ R + be such that ζ 2 p -1 ≤ τ ≤ cΦ -1.
Then, under Assumption . and c = O n (1), for all large n with high probability, the p largest eigenvalues of the matrix L rw τ (and equivalently of L sym τ ) are isolated. In particular,

λ ↓ p (L rw ζ 2 p -1 ) = 1 ζ p .
For the sake of clarity, we recall that Assumption . is the same under which Claim . was formulated. Note that Claim . is not an obvious consequence of the equivalence between Equation ( . ) and Equation ( . ) as it is not clear that the eigenvalue 1/ζ p of L rw p -1 for p = 3. For this simulation, n = 5 000, k = 3,

θ i ∼ [U (3, 10)] 3 , c = 8, π = [0.5, 0.2, 0.3] and C 13 = 1.2, C 13 = 2.7, C 32 = 2.4.
of L rw cΦ-1 (i.e., L rw r 2 -1 for r = √ cΦ) must be isolated. The heuristic regularization τ = d proposed in [QR ], while suboptimal according to the claim, is somewhat meaningful as τ must indeed grow with d ≈ c.

As a corollary to Claim . , exploiting Claim . , the matrices L rw/sym τ be exploited to obtain an alternative method to estimate the number of communities, if unknown.

Remark . (Estimation of the number of classes). As a direct consequence of Claim . and Claim . , we have that, for all large n with high probability

k = i : λ i (L rw τ )| τ=cΦ-1 ≥ 1 √ cΦ .
We now proceed to detail the technical arguments in support of Claim . . .

To support Claim . , we rely on three intermediary lemmas of which we give formal proof. Our main result is in a form of a claim and not of a theorem because it relies on Claim . which is not formally proved, but in the following we show that if Claim . is true, then Claim . holds as well.

. .

Lemma . . Let D and A be the degree and adjacency matrices of an arbitrary graph of size n. Let r > 1 and p an integer between 2 and n.

If λ ↑ p (D -rA) < 0 then λ ↓ p L rw -λ ↑ p (D-rA) = 1 r
and both eigenvalues λ ↑ p (D -rA) and 1/r share the same eigenvector. . The main result of Lemma . is to state that this eigenvalue is the p-th largest creating an explicit mapping between the eigenvalues of D -rA (hence of H r ) and L rw τ for a proper value of τ. Note that the condition λ ↑ p (D -rA) < 0 could be loosened, but we do not need a stronger result in the following.

Before proceeding with the proof of Lemma . , for the sake of clarity, we enunciate Courant-Fischer theorem that will be of fundamental use.

Theorem . . [Courant-Fischer, see for instance Bha ] Let M ∈ C n×n be a Hermitian matrix and U a vector subspace of C n . Then,

λ ↑ p (M) = min U:dim(U)=p max z∈U,z =0 z T Mz z T z λ ↑ p (M) = max U:dim(U)=n-p+1 min z∈U,z =0 z T Mz z T z .
We now proceed with the proof of Lemma . .

Proof of Lemma . . Let r > 1. For simplicity of notation, let us denote

λ p (r) ≡ λ ↑ p (D -rA). De ne the matrix M r as M r = -λ p (r)I n + D -rA = D -λ p (r) -rA
where we recall the notation r) , we then have

D τ = D + τI n . Note that one has λ ↑ p (M r ) = 0. Letting Mr ≡ D -1/2 -λ p (r) M r D -1/2 -λ p (
L sym -λ p (r) = 1 r I n -Mr .
Recalling that L rw τ and L sym τ share the same spectrum, one has in particular:

λ ↓ p (L rw -λ p (r) ) = λ ↓ p (L sym -λ p (r) ) = 1 r λ ↓ p I n -Mr = 1 r 1 -λ ↑ p ( Mr ) .
Thus, proving the lemma amounts to proving that λ ↑ p ( Mr ) = 0, which we do now by rst proving that λ ↑ p ( Mr ) ≤ 0 and then that λ ↑ p ( Mr ) ≥ 0. Denote by X = (x 1 | . . . |x p ) ∈ R n×p the matrix concatenating the eigenvectors associated to the p smallest eigenvalues of M r . One has:

∀ 1 ≤ q ≤ p, M r x q = λ ↑ q (M r )x q and λ ↑ q (M r ) ≤ λ ↑ p (M r ) = 0.
Now de ne the vector space V as V = span D 1/2 -λ p (r) X . Since λ p (r) is strictly negative by hypothesis, D -λ p (r) 0 (is positive de nite), which in turn implies that dim(

V) = p. By de nition of V, ∀ z ∈ V, ∃ u ∈ R p : z = D 1/2
-λ p (r) Xu. From the Courant-Fischer theorem, we can write:

λ ↑ p ( Mr ) = min U:dim(U)=p max z∈U,z =0 z T Mr z z T z ≤ max z∈V,z =0 z T Mr z z T z ≤ max u∈R p ,u =0 u T X T D 1/2 -λ p (r) Mr D 1/2 -λ p (r) Xu u T X T D -λ p (r) Xu i.e.: λ ↑ p ( Mr ) ≤ max u∈R p ,u =0 u T X T M r Xu u T X T D -λ p (r) Xu = max u∈R p ,u =0 ∑ p q=1 (x T q u) 2 λ ↑ q (M r ) u T X T D -λ p (r) Xu ≤ 0
where for the last step we exploited λ ↑ q (M r ) ≤ 0 and D -λ p (r) 0. We thus conclude that λ ↑ p ( Mr ) ≤ 0. To prove that the equality holds, we exploit the second relation of the Courant-Fischer theorem. We de ne X = (x p | . . . |x n ) ∈ R n×(n-p+1) the matrix concatenating the eigenvectors associated to the np + 1 largest eigenvalues of M r . For q ≥ p, λ ↑ q (M r ) ≥ 0. We further de ne

W = span D 1/2 -λ p (r)
X , satisfying dim(W) = np + 1. We can write:

λ ↑ p ( Mr ) = max U: dim(U)=n-p+1 min z∈U,z =0 z T Mr z z T z ≥ min z∈W,z =0 z T Mr z z T z i.e.: λ ↑ p ( Mr ) ≥ min u∈R n-p+1 ,u =0 u T XT M r Xu u T XT D -λ p (r) Xu = min u∈R p ,u =0 ∑ n q=p (x T q u) 2 λ ↑ q (M r ) u T XT D -λ p (r) Xu ≥ 0.
As a consequence, λ ↑ p ( Mr ) ≥ 0. Combining both inequalities, we obtain that λ ↑ p ( Mr ) = 0. The fact that the eigenvectors are shared comes from the following. Let x p (r) be the eigenvector of D -rA associated to λ p (r), i.e., [D -rA]x p (r) = λ p (r)x p (r). This can be re-written as L rw -λ p (r) x p (r) = 1 r x p (r).

The rst Lemma introduces a formal mapping between the p-th smallest eigenvalue of D -rA and the p-th largest eigenvalue of L rw τ for the choice τ = -λ ↑ p (D -rA) Claim . however gives a result on the spectrum of L rw τ for ζ 2 p -1 ≤ τ ≤ cΦ -1, hence, unlike Lemma . , it is not limited to a single value of τ. The next lemma takes care of generalizing the result considering the behavior of τ = λ ↑ p (D -rA) as a function of r.

Lemma . . Let D and A be the degree and adjacency matrices of an arbitrary graph. Let p be an integer ≥ 2. Suppose that there exists r p > 1 such that:

(i) ∀ r ≥ r p the eigenvalue λ ↑ p (D -rA) is simple; (ii) λ ↑ p (D -r p A) < 0. Then: ∀ r ≥ r p , ∂ r λ ↑ p (D -rA) < 0.
In words, once λ ↑ p (D -rA) becomes negative, it is strictly decreasing.

Proof of Lemma . . Once again, let us write λ p (r) = λ ↑ p (D -rA) to lighten notations. Let p, r p be an integer ≥ 2 and a scalar > 1 (if they exist) such that λ p (r p ) < 0. Let r ≥ r p . As λ p (r) is an eigenvalue, ∃ x p (r) with ||x p (r)|| 2 = 1 such that:

[D -rA]x p (r) = λ p (r)x p (r), which implies in particular that x p (r) T Ax p (r) = 1 r x p (r) T D -λ p (r) x p (r). As we suppose λ p (r) to be simple, we can apply the eigenvalue perturbation theorem [GLO , for instance]:

∂ r λ p (r) = -x T p (r)Ax p (r) = - 1 r x p (r) T D -λ p (r) x p (r). ( . )
In Equation ( . ), D -λ p (r p ) 0 (as λ p (r p ) < 0 by hypothesis), consequently ∂ r λ p (r) r=r p < 0. We now want to show that for all r > r p , ∂ r λ p (r) < 0.

We proceed with a proof by contradiction.

Suppose that there exists a value r > r p such that ∂ r λ p (r) r=r ≥ 0. From Equation ( . ) it follows that a necessary (but not su cient) condition to be veri ed is that λ p (r )≥0. From a continuity argument on the function λ p (r), and as λ p (r p ) < 0 and λ p (r ) ≥ 0 with r > r p , there exists r ∈ (r p , r ) such that ∂ r λ p (r) r=r > 0 and λ p (r ) < 0. Invoking once again Equation ( . ), no such r can exist, invalidating the hypothesis we made by absurd. We thus conclude that

∀ r ≥ r p , ∂ r λ p (r) < 0, ( . )
nishing the proof.

The eigenvalue simplicity assumption of this lemma is technical and enables us to properly de ne and manipulate the derivative of the p-th smallest eigenvalue. In case of multiplicity, the tools involved are more complicated [GLO ] and not included here. The simplicity assumption is however veri ed with high probability on the DCSBM-generated graphs considered in Claim . , hence justifying our assumption.

With Lemma . we established a relation between τ in the regularized Laplacian and r in the Bethe-Hessian, showing that to larger values or r should correspond larger values of τ. We now enunciate our nal supporting lemma which states that isolated eigenvalues of H r are mapped to isolated eigenvalues of L rw τ while bulk eigenvalues are mapped to bulk eigenvalues.

Lemma . . Let G(V, E ) be a graph with k classes generated from the sparse DCSBM and let D and A be the corresponding degree and adjacency matrices respectively. Consider an integer p between 2 and k and let ζ p be de ned as per Equation ( . ) and r ≥ ζ p . If λ ↑ p (D -rA) < 0 is isolated with associated eigenvector x p , then x p is an eigenvector of L rw -λ ↑ p (D-rA)

, whose corresponding eigenvalue is also isolated. Otherwise, for 1 ≤ p ≤ n and r > 1, if

λ ↑ p (D -rA) is a bulk eigenvalue of D -rA, then the corresponding eigen- value of L rw -λ ↑ p (D-rA)
is also a bulk eigenvalue.

(r) = λ ↑ p (D -rA) to lighten notations. Note that λ ↑ p (H ζ p ) = 0, ∀ 2 ≤ p ≤ k and λ ↑ k+1 (H ζ k+1 ) = o n (1)
, as shown in [SKZ ]. We thus have λ p (ζ p ) < 0, ∀ 2 ≤ p ≤ k + 1 and it is isolated by hypothesis. We can apply Lemma . , for r p = ζ p : for all r ≥ ζ p , ∂ r λ p (r) < 0 and λ p (r) < 0. Consequently, there exists a unique value of r ≥ r, satisfying λ p+1 (r ) = λ p (r). From Lemma . , we know that:

• as λ p (r) < 0: x p (r), the eigenvector associated to λ p (r), is also the eigenvector of L rw -λ p (r) associated to its p-th largest eigenvalue, 1/r.

• as λ p+1 (r ) < 0: x p+1 (r ), the eigenvector of Dr A associated to λ p+1 (r ), is also the eigenvector of L rw -λ p+1 (r ) = L rw -λ p (r) associated to its (p + 1)-th largest eigenvalue, 1/r . Thus, the p-th (resp. (p + 1)-th) largest eigenvalue of L rw -λ p (r) is 1/r (resp. 1/r ). Consider r ≥ ζ p . By hypothesis, λ p (r) is an isolated eigenvalue, that is, we can write

O n (1) = λ p+1 (r) -λ p (r) = λ p+1 (r) -λ p+1 (r ) = r r dx ∂ x λ p (x) = κ(r -r )
for some constant κ = O n (1) independent of n, representing the average value of ∂ x λ p (x) on the integration interval. The constant κ = O n (1) because for any r, r = O n (1), we have that λ p (r), λ p (r ) = O n (1). The eigengap for L rw -λ p (r) is 1/r -1/r = (rr)/(rr ), thus

O n (1) = O n λ p+1 (r) -λ p (r) = O n (r -r ) = O n 1 r - 1 r = O n (1).
So, if λ p (r) is isolated, the eigenvalue 1/r of L rw τ is isolated as well. On the other hand, if λ p (r) is in the bulk, then ∃ q : |λ p (r)λ q (r)| = o n (1). By an argument of continuity on λ p (r), one can analogously de ne an r satisfying λ p (r) = λ q (r ), concluding that in this case |rr | = o n (1) and so eigenvalues in the bulk are mapped into eigenvalues in the bulk.

With this proof we conclude this technical section in which we enunciated and proved three lemmas that formally relate the eigenvalues of H r with those of L rw τ . We now proceed with an informal proof of Claim . .

. . .

Based on Lemmas . -. and Claim . we now proceed to the justi cation of Claim . . Once again, we adopt the notation λ ↑ p (D -rA) = λ p (r). and it is isolated. Further exploiting Lemma . , letting τ(r) = -λ ↑ p (D -rA), we have ∂ r τ(r) > 0 for all r in the interval ζ p ≤ r ≤ √ cΦ. The function τ(r) is thus bijective and increasing on this interval. Since the eigenvalue λ

↑ p (D -rA) is isolated for all ζ p ≤ r ≤ √ cΦ, the corresponding eigenvalue of L rw τ(r)
equal to 1/r is isolated for

ζ 2 p -1 = -λ p (ζ p ) ≤ τ ≤ -λ p ( √ cΦ). Note now that, for 1 ≤ p ≤ k, λ ↑ p (D - √ cΦA) ≤ λ ↑ k+1 (D - √ cΦA) = cΦ -1. This implies that for ζ 2 p -1 ≤ τ ≤ cΦ -1, the top p eigenvalues of L rw
τ are certainly isolated. For the particular case where r = ζ p , by de nition, -λ ↑ p (D -

ζ p A) = ζ 2
p -1 > 0 and the result is straightforwardly obtained by applying Lemma . , concluding our argument.

With the justi cation of our main result being settled, we are now in position to close this chapter with some concluding remarks.

. In this chapter we showed that there is an explicit relation between the Bethe-Hessian and the regularized Laplacian matrices. Building on this relation, we generalized our Claim . to L sym τ . In the end of Chapter three questions concerning SC with the regularized Laplacian matrix were left open. The answers to these questions are the main contribution of this chapter and are summarized as follows.

• In all the works related to t regularization [QR ; Ami+ ; JY ; LR+ ;

LLV ] the approach chosen to study SC consists in providing a bound to M -E[M] , where M is the matrix under study and E[M] its expectation. If on the one hand regularization provides good results for moderately dense graphs, on the other hand it is not su cient to give tight bounds in the regime c = O n (1). In this chapter we used a completely di erent method that justi es the role of regularization starting from the inner isolated eigenvalues of non-backtracking matrix, studied in Chapter . Our analysis, although not formally rigorous, holds in the sparse regime c = O n (1). • After considering the sparse regime c = O n (1), the second natural question concerns whether or not the matrix L sym τ exhibits isolated eigenvalues down to the detectability threshold and, if so, for what τ. Claim . gives a precise result stating that indeed the detectability threshold can achieved specifying the interval of values of τ that can accomplish this task.

• An evidence that appeared from the earlier works on regularization is that small values of τ perform better in practice than large ones, even though no clear explanation was provided. We showed that the optimal Laplacian regularization ranges from 0 (for easy problems) to cΦ -1 for hard ones, naturally justifying the adoption of small regularizations. Figure . provides further evidence showing for di erent τ the performance obtained from Algorithm . setting M = L rw τ and con rming that the proposed choice of τ is optimal.

As a nal remark, we recall that also in this case Claim . can be extended to the setting in which the matrix CΠ potentially has negative eigenvalues. All comments made in Chapter remain valid also in this case, with the di erence that the regularizer τ = ζ 2 p -1 > 0 even if ζ p < 0. The "disassortative" eigenvalues will be associated to negative eigenvalues of

L rw τ , satisfying ζ p L rw ζ 2 p -1 = 1.
Chapter , have been devoted to the formulation of two claims under the DCSBM assumption. The resulting Algorithm . has been introduced as a meta-algorithm for user readability. In the next chapter we consider the problem of how Algorithm . can be e ciently implemented on real graphs that go beyond the DCSBM assumption. Further considerations will be made on the computational complexity and run time of the resulting algorithm.

A L G O R I T H M I C I M P L E M E N T AT I O N Abstract

This chapter considers how to implement Algorithm . to obtain an e cient spectral clustering (SC) method for community detection (CD) on arbitrary graphs. In particular, as a main contribution we provide an e cient estimator of the number of classes k, based on Claim . of [SKZ ] and subsequently show how to retrieve the values {ζ p } p=1,...,k . The proposed algorithm is then extensively tested on several real-world graphs and its performance is shown to typically outperform the one achieved by the competing state-of-the-art SC techniques. Chapter showed that a sequence of carefully parametrized Bethe-Hessian matrices {H ζ p } p=1,...,k can be exploited to perform SC in sparse graphs with a heterogeneous degree distribution. All the results concerning the values {ζ p } p=1,...,k and the corresponding embedding eigenvectors {x p } p=1,...,k are based on the assumption that the graph G(V, E ) is a realization of a sparse DCSBM with n → ∞ nodes. Real-world graphs, however, force us to seri-From DCSBM to real-world graphs ously face three main problems. The rst is that the setting n 1, while it interestingly corresponds to big datasets, it cannot be given for granted. Consequently, an e cient and practical algorithm for CD cannot rely on asymptotic results. Related to this, the second problem lies in the fact that, as we already discussed in Chapter , for nite n there is not a clear boundary between the sparse and the dense settings and our proposed algorithm must be well de ned (and e cient) for any average degree. Finally, short cycles are often observed in real social graphs [HL ]: this fact may put in question our theoretical results based on the Bethe approximation, which is appropriate for tree-like graphs.

It has however to be noted that, while Chapter only considers the DCSBM setting to justify Algorithm . , the de nition of {ζ p } p=1,...,k ensures their existence for any input graph and it does not rely on the n 1, nor the sparsity, nor the DCSBM assumptions. In fact, based on several empirical tests, we observed that the following properties of the non-backtracking matrix (De nition . ) seem to hold in general:

Spectrum of B for the football dataset [GN ] • All complex eigenvalues come in pairs of complex conjugates and most of them are bounded by a circle on the complex plane of radius

r radius ≈ λ ↓ |•| 1 (B).
• The number of real eigenvalues of B, di erent from {-1, 0, 1, λ

↓ |•|
1 (B)}, is even. Half of the eigenvalues are larger in modulus than r radius , while the other half lies between 1 and r radius . All of them are isolated.

Given the connection between the non-backtracking and the Bethe Hessian matrices, from these two points, we can claim that the steps of Algorithm . are all well de ned on arbitrary graphs. By all means, we can consequently say that the set of ζ p 's are a property of the graph for which we have a clear interpretation for the DCSBM. Based on these observations, the goal of this chapter is to design e cient routines to accomplish the steps of Algorithm . on an arbitrary input graph.

Among the questions that have to be explicitly answered in the following, there is the one concerning the estimation of k, which is necessary to de ned the set {ζ p } p=1,...,k . Building on the result of [SKZ ], we develop an e cient algorithm (especially for large values of k) to estimate k, which we recall is a typically hard task to accomplish when performing SC.

Our ndings result in Algorithm . which is tested on some real graphs benchmarks comparing its e ciency against the competing spectral methods. The results are reported in Table . . . In this section we describe in detail how to e ciently implement the subroutines of our proposed algorithm that we summarize in Algorithm . which is the implementation adopted in the CoDeBetHe.jl package.

Before getting into the details of the steps of Algorithm . , let us recall that a further speci city of real graphs is their possibility to be inherently made of disjoint components. We have instead so far worked with the assumption that G(V, E ) has a giant component and that the communities are return k, ˆ . end setting c out = 0). In this case, one can perform a two-stage clustering. First the connected components are detected, looking into the eigenvectors with zero eigenvalue of H ζ 1 = D -A. Afterwards, each connected component is treated independently. We will thus consider, without loss of generality, that the real graphs which we will consider are connected.

. . k

The problem of estimating the number of communities in an unsupervised manner is in general non-trivial. Methods based on the non-backtracking rather than Bethe-Hessian matrix have been studied and exploited to eciently recover communities regardless of the generative model [LL ].

A possible way to de ne an estimator of k is as the largest number of negative eigenvalues of H r as a function of r, or, more succinctly,

k = max r≥1 |i : λ i (H r ) < 0| .
This estimator is well de ned on arbitrary graphs and has a deep interpretation rooted in statistical physics. However, a direct implementation of this estimator is ine cient because it requires to compute several times the eigenvalues of H r for di erent values of r. A crucial result of [SKZ ] is to determine for the SBM (but the result can be easily generalized to the DCSBM) the value of r for which the number of negative eigenvalues of H r is maximal and is precisely equal to r = λ

↓ |•| 1 (B) = √
cΦ. This value of r can be eciently estimated from the degree sequence, since cΦ ≈ ∑ i∈V d 2 i / ∑ i∈V d i .

With extensive simulations, we observed however that on general graphs cΦ is generally not a good approximation for λ

↓ |•| 1 (B).
k = i : λ i   H λ ↓ |•| 1 (B)   < 0 .
As we showed in Chapter , an estimator of k can be designed also on the regularized Laplacian matrix as detailed in the following remark.

Remark . (Estimating k from L sym τ ). Remark . states that k can also be estimated using the regularized Laplacian matrix L sym τ as follows

k = i : λ i L sym λ ↓ |•| 1 (B)-1 > 1 λ ↓ |•| 1 (B)
.

The main advantage of this estimator with respect to the one based on H r is that one needs to compute the eigenvalues with largest rather than smallest algebraic value that, generally, can be done more e ciently.

Moreover, suppose that one wants to include also disassortative communities in the estimation of k. Referring to DCSBM this is the case in which CΠ may have also negative eigenvalues. The estimators of k for the Bethe-Hessian and the regularized Laplacian can be written, respectively, as

k = i : λ i   H λ ↓ |•| 1 (B)   < 0 + i : λ i   H -λ ↓ |•| 1 (B)   < 0 k = i : λ i L sym λ ↓ |•| 1 (B)-1 > 1 λ ↓ |•| 1 (B)
, evidencing that L sym τ allows one to estimate k using a single value of τ, while H r requires two values of r. In the remainder, the "disassortative contribution" is however not considered and consequently the two estimators are essentially equivalent. For notation convenience, we will refer to the one based on H r , but all the results shown below can be generalized to L sym τ .

Let us now consider the problem of counting the number of negative eigenvalues of H √ ρ , where ρ = λ

↓ |•|
1 (B) for convenience. A naïve implementation simply consists in computing the p smallest eigenvalues H √ ρ and check if they are all negative. If the condition is met, then p is increased to p + 1 and the procedure is iterated until the rst positive eigenvalue of H √ ρ is found. This implementation is however non e cient, especially for large k, since it requires to compute several times the same eigenvalues. The complexity of this method scales as O(nc k3 ). A recent workaround strategy [RM ; Tre+ a; Tre+ b] based on random projections and polynomial approximations allows us to de ne an e cient routine that initializes p to a value close to k, which is then computed in a few iterations.

Random projections

Denote with Y ∈ R n× k the matrix containing in its columns the eigenvectors associated to the negative eigenvalues of H √ ρ . Note that, since H √ ρ is Hermitian, Y T Y = I k. Let f ∈ R n be a random vector with zero mean and E[ f f T ] = I n . An estimator of k unfolds from the following relations

k = Tr(I k) = Tr(Y T Y) = Tr(YY T ) = Tr YY T E[ f f T ] I n = E Tr(YY T f f T ) = E Tr( f T YY T f ) ∈R = E f T YY T f = E f T Y Y T Y I k Y T f = E YY T f 2 .
( . ) Equation ( . ) shows that k can be estimated taking the expectation of the square norm of YY T f . At this stage, this procedure is however pointless, since the matrix Y (whose size is determined by k) needs to be computed. We now proceed showing that an approximate method can be devised to estimate k without directly computing Y.

Polynomial approximation

Let us introduce the door function Θ ab (x) = I x∈ [a,b] . The number of negative eigenvalues of H √ ρ is equal to the number of eigenvalues of

H √ ρ = H √ ρ -λ ↑ 1 (H √ ρ )I n in the interval [0, -λ ↑ 1 (H √ ρ )]. Letting Λ √ ρ ∈ R n×n be the diagonal matrix containing the eigenvalues of H √ ρ , we denote with Θ ab (Λ √ ρ ) ij = Θ ab Λ √ ρ ij . Denoting for simplicity λ ↑ 1 (H √ ρ ) = λ min , the following relation holds YY T = Y Θ 0,-λ min (Λ √ ρ ) Y T ≡ Θ 0,-λ min (H √ ρ ).
Obtaining the expression of Θ 0,-λ min (H r ) is still unpractical because all eigenvalues must be computed. An approximation of YY T ≈ Ỹ ỸT can however be e ciently obtained replacing Θ 0,-λ min (x) with its polynomial approximation Θ0,-λ min (x) of order Q. Denoting with c 0≤q≤Q the coe cients of the polynomial, in fact

Ỹ ỸT = Θ0,-λ min (H √ ρ ) = Q ∑ q=0 c q H √ ρ q ,
then, exploiting the fact that H √ ρ and H √ ρ share the same eigenvectors, YY T ≈ Ỹ ỸT . This allows us to de ne an estimator of k as follows

k ≈ 1 N N ∑ i=1 Q ∑ q=0 c q H √ ρ q f i 2 , ( .
)
where N denotes the number of random vectors generated to compute the empirical mean. This estimator of k is obtained in O(cnNQ) operations. Due to the good concentration properties of their norm, if f i are Gaussian random vectors, a very small value of N (e.g. N = 4) is su cient to obtain a good estimator of the expectation. On the other hand, the value of Q must be carefully chosen to guarantee a good approximation of the step function. In practice, it is convenient to increase Q until the variation in the estimation of k remains below a given threshold. To give an order of magnitude, Q is, for the typical problems it was tested on, of order of 100. The fact that the estimator of k based on random projection has a k-independent complexity, makes it particularly suited for large values of k.

The strategy to e ciently compute the number of negative eigenvalues is then to give it a rst rough approximation as per Equation ( . ) and then proceed with the naïve method detailed before to compute this quantity exactly. Empirically, it is convenient to compute an initial estimate of k for k ≈ 5 or greater. For smaller values, the naïve direct computation might be more e cient, but both algorithms provide a fast solution in this case.

The results so far presented allow us to detail Algorithm . to e ciently estimate the number of communities on an arbitrary graph. For simplicity, we indicate here Q as an input of the algorithm. Neglecting the dependence The choice of which polynomial approximation to consider is not straightforward. One possible choice is to use Chebychev polynomials as they have a guarantee on the in nite norm of the approximation error. However, they tend to create Gibbs oscillation around sharp cuto s of the function to approximate. As the function we wish to approximate here is a step function, it is customary to choose Jackson-Chebychev polynomials which explicitly dampen these unwanted oscillations [Jay+ ]. Note that, while k ∈ N, the estimator in Equation ( .) is de ned in R.

Algorithm . : estimate_number_of_classes

Input : Adjancency matrix of undirected connected graph G(V, E ), Q: order of the polynomial approximation, N: number of random vectors

Output : Estimated number of commnities k begin

ρ ← λ ↓ |•| 1 (B ), with B as in Equation ( . ); H √ ρ ← H √ ρ -λ ↑ 1 (H √ ρ )I n ; {c q } 0≤q≤Q ← Jackson-Chebishev coe cients of the polynomial approximation of Θ 0,-λ ↑ 1 (H √ ρ )
; Generate N Gaussian random vectors with zero mean and idendity covariance matrix f 1≤i≤N ∈ R n ;

k init = ceil 1 N ∑ N i=1 ∑ Q q=0 c q H √ ρ q f i 2 ; Compute λ ↑ i (H √ ρ ) for 1 ≤ i ≤ k init ; k ← arg max i λ i (H √ ρ ) < 0 for 1 ≤ i ≤ k init ; while λ ↑ k (H √ ρ ) < 0 do k ← k + 1; Compute λ ↑ k (H √ ρ ) < 0; end return k -1; end
on Q and N, the complexity of Algorithm . scales in the worst case scenario in which k init is very far from k as O(nc k3 ). However, if k init is close to k, Algorithm . converges in O(nc k2 ) operations. To give a practical measure of computational times we tested Algorithm . on a laptop for di erent values of k on graphs generated from the stochastic block model (SBM) with n = 10 5 , c = 8, c out = 4/k. The value of k was computed for k = 2, 8, 30 in 8, 8, 10 seconds.

We now move to the analysis of the second subroutine, dedicated to estimate the values of ζ p .

. . { ζ p }

A crucial step of Algorithm . is to compute the vector ζ ∈ R k, containing the optimal values of r in which the eigenvalues of H r should be computed.

In the remainder, for simplicity, we will denote k = k. From Claim . , a simple relation holds between the values ζ p and the isolated eigenvalues of B for the DCSBM, in fact

The laptop's RAM is . Gb with Intel Core i -U CPU @ . GHz x .

ζ p = c λ ↓ p (CΠ) + o n (1) = λ ↓ |•| 1 (B) λ ↓ |•| p (B) + o n (1). ( . )
This estimation of ζ p via B , de ned in Equation ( .) is computationally e cient but we observed that it can be quite inaccurate for graphs not generated from the DCSBM. Conversely, a naïve line search for

ζ p ∈ (1, √ ρ) (where ρ = λ ↓ |•| 1 (B)) satisfying λ ↑ p (H ζ p ) = 0 is computationally ine cient.
We propose here a faster method, motivated by a Courant-Fischer theorem argument (Theorem . ). In a few words, starting from an initial guess r 0 , we devise an iterative sequence r 0 , r 1 , .

λ ↑ p (H r ) = min U:dim(U)=p max z∈U,z =0 z T H r z z T z ≤max u∈R p u T X T r t H r X r t u u T u = λ ↓ 1 (X T r t H r X r t )
i.e.:

λ ↑ p (H r ) ≤ λ ↓ 1 (X T r t [(r 2 -r t 2 )I n -(r -r t )A + H r t ]X r t ) ≤ (r 2 -r t 2 ) + λ ↓ 1 (Λ r t -(r -r t )X T r t AX r t ).
We can further simplify the earlier expression by exploiting the identity:

Λ r t = X T r t H r t X r t = (r t 2 -1)I p + X T r t DX r t -r t X T r t AX r t .
We thus obtain

λ ↑ p (H r ) ≤ 1 r t (r -r t )(1 + r r t ) + λ ↓ 1 (r t -r )X T r t DX r t + r Λ r t f r t (r )
.

( . )

We now study the function f r t (r ) for r t ∈ (ζ p , √ ρ) and de ne r t+1 ∈ (ζ p , r t ) as the solution (if it exists) to f r t (r t+1 ) = 0.

( . )

The idea is to iteratively approach ζ p from the right and substitute r t ← r t+1 . If r t+1 ∈ (ζ p , r t ) as de ned above exists, then {r t } t≥0 is a lower-bounded decreasing sequence: it thus converges to a limit r ∞ (potentially di erent from The two dashed black lines are the two roots of f r t (r ), the smaller of which is r t+1 . For this simulation n = 50 000,

k = 2, c = 5, c out = 2.5, r = √ cΦ, θ i ∼ [U (3, 10)] 3 , π ∝ 1 k . ζ p ).
Exploiting [RB , Theorem in I. ], denoting with F, G two Hermitian matrices and γ a scalar, λ ↓ 1 (F + γG) is a convex function of γ. As a consequence, the function f r t (r ) is convex and has either no root or two roots. Since lim |r |→∞ f r t (r ) = +∞, it is enough to nd a value of r for which f r t (r ) < 0 to prove that this function has two roots. With a straightforward computation, one can verify that f r t (r t ) < 0, so f r t (r ) has two roots, satisfying r t+1 < r t and r + > r t . By construction, since λ ↑ p (H r ) is negative in the considered interval and λ ↑ p (H r ) ≤ f r t (r )/r t , if f r t (r t+1 ) = 0, then ζ p ≤ r t+1 . Consequently, f r t (r t+1 ) = 0 has a unique solution satisfying ζ p ≤ r t+1 < r t and the algorithm converges to r ∞ = lim t→∞ r t ≥ ζ p . We are left to prove that r ∞ = ζ p . By convergence of r t , we have r t+1r t = o t (1). Plugging this relation solution into Equations ( . , . ), we obtain

λ ↓ 1 (r t+1 Λ r t+1 ) = r t+1 λ ↑ p (H r t+1 ) = o t (1). Since λ ↑ p (H r ) = 0 has a unique solution (r = ζ p ) in the interval r ∈ (1, √ ρ), we obtain r t = ζ p + o t (1) and so r ∞ = ζ p .
The initial value of r can be chosen as r 0 = ζ p+1 (setting ζ k+1 = √ ρ) that certainly falls in the right interval for all the ζ p .

Summarizing, the algorithm builds on two parts: (i) the computation of a p × p matrix obtained from the eigenvectors of H r t with computational cost of O(ncp 2 ), (ii) a subsequent line-search using this matrix, with computational cost O(p 2 ). The advantage of this method is that the line search (which requires many iterations) is computationally cheap, while the most expensive part of the algorithm needs to be performed much fewer times

Algorithm . : compute_ζ

Input : adjacency matrix of a connected, undirected graph G, number of classes k Algorithm . summarizes our proposed routine to obtain the vector ζ. Note that, although the subroutine compute_ζ only outputs the vector ζ, it can also be used to directly compute the informative eigenvectors {x p }. Figure . provides a typical iteration of Algorithm . , while Figure . shows the output of the computation of ζ and con rms the accuracy of the proposed algorithm. On the top line the algorithm is tested on a network created from the DCSBM, for which λ

Output : Vector ζ = {ζ 1 , . . . , ζ k } begin r ← λ ↓ |•| 1 (B ), where B is de ned in Equation ( . ) ; p ← k; while p > 1 do repeat Compute Λ r = diag λ ↑ 1 (H r ), . . . , λ ↑ p (H r ) and X r ∈ R n×p , with H r X r = X r Λ r ; Γ r ← X T r DX r ∈ R p×p ; r * ← r : λ ↓ 1 (r Λ r + (r -r )Γ r ) = (r -r )(1 + rr ), with line-search on r ∈ (1, r); r ← r * until convergence; δ ← multiplicity of λ ↑ p (H r ) = 0; Set ζ p , ζ p-1 , • • • , ζ p-δ+1 to r; p ← p -δ;
↓ |•| 1 (B)/λ ↓ |•| p (B) is a valid estimator for ζ p . The horizontal line indicates √ ρ =
√ cΦ which is the upper bound of ζ p . In our simulations, the a nity matrix C ∈ R k×k is generated randomly (see the caption of Figure . ), for the rst two points we obtained λ ↓ 3 (CΠ) < √ cΦ, invalidating Assumption . . In this case we see that c/λ ↓ p (CΠ) > √ cΦ and the corresponding estimated value of ζ p saturates at √ cΦ. On the contrary, whenever Assumption . is veri ed, the estimate of ζ p is correct.

In the bottom line we compare the two methods to estimate ζ p on the three real networks that clearly show that these two methods are di erent on graphs not generated from the DCSBM. Given that Subroutine . provides a direct computation of values of ζ p , it should be preferred.

↓ |•| 1 (B)/λ ↓ |•| p (B)
(green squares) vs the direct computation of ζ p as the output of Subroutine . (red diamonds) on the three real networks taken from [LK ]. The value of k is estimated according to Algorithm . . For illustrative purposes, we tested the execution time of Algorithm . for a stochastic block model (SBM) network with k = 4 classes of equal size, n = 10 5 , c in = 26.20 on the diagonal elements of C and c out = 4.60 for the o diagonal elements. The values of ζ p were computed to machine precision in approximately 5 seconds on a laptop using our CoDeBetHe.jl Julia implementation. The complexity of this subroutine scales linearly with n, and thus can be applied to large networks but cubically with respect to k (and not quadratically as usual in the spectral clustering context) decreasing the computational e ciency when a rather large number of classes is present.

We now proceed with the last technical step of Algorithm . , which concerns the projection of the embedded points on a unitary hypersphere.

. .

The study performed so far identi es the presence of k informative eigenvalues and describes the content of the associated k eigenvectors X ≡ [x 1 , . . . , x k ] ∈ R n×k with x p the eigenvector associated to the p-th smallest eigenvalue of H ζ p (in particular, the vectors x p 's need not be orthogonal). The rows X 1,• , . . . , X n,• ∈ R k of the matrix X form a k-dimensional feature for every node, which are used in a last small-dimensional clustering step, usually employing the k-means algorithm. The fact that k-means is particularly e cient when the low-dimensional clusters are quite "isotropic" strongly motivates the need for the entries of the vectors x p not to be a ected by the node degrees (which would otherwise spread the clusters unevenly).

The embedding obtained by Algorithm . on the graph polbooks with (bottom) and without (top) the projection step Yet, to further tackle residual degree dependence, a classical method, prior to k-means, consists in normalizing all vectors X i,• to X i,• = 1 (this is Step of Algorithm . ) . This method is motivated by the assumption that the degree dependence in each X ij is separable from the label dependence, a fact that is veri ed in su ciently dense DCSBM networks [Jin+ ] and to some extent also in sparser graphs [QR ]. Besides, under this normalization, the k-means algorithm is restricted to the unitary hypersphere, improving its convergence to the genuine solution, especially for large values of k.

As such, while our proposed algorithm naturally discards degree dependence in the entries of X under the DCSBM setting, the reality of practical networks may disrupt this expected behavior and the projection of the vectors X i,• on the unit hyper-sphere both alleviates this deleterious e ect and further improves the convergence of k-means. We thus adopt this normalization step in our nal Algorithm . and will con rm its practical gains when clustering real graphs.

. . . Computational complexity

The total theoretical complexity of Algorithm . is dominated by Algorithm . and . that both run in O(nck 3 ), as the k-means step costs only O(nk 2 ). This is to compare with the usual complexity of spectral clustering algorithms on sparse graphs that are in O(nck 2 ) [see for instance TL ]. This additional cost comes with a better classi cation performance in many real-world graphs, as discussed in the next section. To give an order of magnitude of computation times, running Algorithm . on a SBM with n = 10 5 (resp., 10 6 ), k = 4, c in = 26.20 on the diagonal elements of C and c out = 4.60 on the o diagonal terms takes approximately 15 (resp., 450) seconds if k is not known and 6 (resp., 100) seconds if k is known a priori.

. In this section we compare the performance of Algorithm . versus competing spectral methods on real-world networks [LK ; Zac ; Lus+ ; GN ; AG ]. Table . shows the performance of di erent SC algorithms on realworld networks. Measuring the quality of an inferred partition ˆ is in general not straightforward since communities are not uniquely de ned. We propose two di erent scores to accomplish this task. One is the modularity Note that the measure of the modularity is meaningful on assortative or disassortative networks but not on "hybrid" networks for which a more involved description would be needed. [NG ], Q Mod A ( ˆ ), de ned in Equation ( .), where we recall that high values of Q Mod A correspond to good quality partitions. Alternatively, the partition quality is evaluated in terms of (normalized) posterior negative loglikelihood of the DCSBM, Q likelihood A :

Q likelihood A ( ˆ ) = - 1 2|E |   ∑ (ij)∈E log   θi θj Ĉˆ i , ˆ j n   + ∑ (ij)/ ∈E log   1 -θi θj Ĉˆ i , ˆ j n     , where θi = d i / d and Ĉab = ∑ i : ˆ i =a ∑ j : ˆ j =b A ij / ∑ i : ˆ i =a ∑ j : ˆ j =b θ i θ j .
Good quality clustering correspond, in this case, to low values Q likelihood A .

The results are compared for di erent clustering algorithms on 15 realworld networks of increasing size. For all networks, the number of communities, when not available, is estimated with Algorithm . and then the same value is used for all competing techniques (which in general do not provide their own dedicated estimator of k). The underlined numbers in the k column indicate instead that k is known. Furthermore, for all networks, community detection is performed only on the largest connected component of the graph and n, c, Φ, k refer to the characteristics of this dominant connected component. Given the embedding X, 5 iterations of k-means are run and the best partition ˆ (in terms of k-means is kept) and the scores

Q Mod A , Q likelihood A are computed.
To keep the stochastic nature of k-means into account, this step is repeated for 8 times and The rst score column of Table . indicates the output of the meta-algorithm Algorithm . for which ζ p is estimated from λ

↓ |•| 1 (B)/λ ↓ |•|
p (B). The second score column provides the output of Algorithm . in which the last step of projection on the hypersphere is not performed. Comparing these two columns, it is clear that Algorithm . generally provides much better partitions both in terms of

Q Mod A and Q likelihood A , as a consequence of λ ↓ |•| 1 (B)/λ ↓ |•|
p (B) being an inappropriate estimator for ζ p in general. The following columns display the results scores obtained by Algorithm . with the projection step (highlighted in cyan), by clustering based on the leading eigenvectors of A [LR+ ], by the Bethe-Hessian as per [SKZ ], by the nonbacktracking matrix as per [Krz+ ], by the random walk Laplacian as per [SM ] and by the symmetric normalized Laplacian of [QR ].

The algorithm of [SM ] based on L rw provides in certain cases very competitive partitions (e.g., in the Wikipedia dataset) but is quite unreliable as it may dramatically fail in others (GNutella P P and Polblogs). Algorithm . without the normalization step provides systematically good partitions, all comparable to those of L sym τ . This is an evidence that Algorithm . e ectively produces a node embedding which is signi cantly resilient to degree heterogeneity. Finally, the improved version of Algorithm . including the projection step further improves the quality of the partition on most datasets, providing on all datasets but Wikipedia the highest reported modularity and lowest, or second lowest measured value of Q likelihood To test Algorithm . we computed the vector ζ up to machine error precision, that is, the convergence stopping criterion in Algorithm . is met when r's update is below machine precision. Notably, due to the non-linearity of the k-means step, larger errors in the estimate of the values of ζ lead, within a certain range, to the same partition. The same classi cation precision can therefore be reached in fewer iterations, thus faster, if needed.

Mail

Let us mention that, comparing the columns "Alg . wp " and "H √ cΦ ", it appears evident that Algorithm . indeed achieves good quality embeddings on real-world graphs. In addition, it appeared from our simulations that the projection step on the unitary hypersphere of the rows of X signi cantly helps (both in terms of quality of the partitioning and of the variance of the score) the convergence of k-means at a low computational cost.

As a side comment, although formally not strictly comparable on even grounds, we evaluated the performances reached by Algorithm . against the popular greedy Louvain method [Blo+ ] using its scikit-network implementation [Ped+ ]. The Louvain method comes with an estimate of the number of communities and relies on a di erent notion of communities than the one we used. Speci cally, it is a hierarchical algorithm which looks for a partition maximizing the modularity, while Algorithm . relies on the DCSBM assumption. Figure . shows that Algorithm . largely outperforms the Louvain method on a DCSBM graphs, both in terms of overlap (left) and modularity (right), except below α c . There, the Louvain method reaches higher modularity, but this is likely an incidental artifact of the modularity optimization constraint [GSPA ].

The situation of real networks is less straightforward. We indeed observed that when the number of communities estimated by both algorithms is similar, they both produce node partitions with a similar modularity. This is however no longer the case when the estimated k are more distinct (e.g., on GNutella P P). While we recommend our estimation method of k due to its interpretability (that we discussed all along the chapter), we acknowledge that the Louvain algorithm often provides very competitive outcomes, at a smaller computational cost but at the expense of any theoretical guarantee. Similar considerations can be made in relation to the more recently proposed Leiden method [TWVE ].

. This chapter considered the problem of providing an e cient and practical implementation to our proposed Algorithm . , itself inspired from Claim . . We kept the focus on two main problems: on the one hand de ning fast routines to accomplish the steps on Algorithm . , on the other hand to design an algorithm which does not heavily rely on the DCSBM assumption and that can perform CD on arbitrary graphs.

In particular, one of the main contributions is the design of Algorithm . , needed to estimate k. We discussed at length in the previous chapters that determining the number of communities on a graph in an unsupervised way is, in general, a very challenging task. Algorithm . is directly inspired from the results of [SKZ ], but what is truly novel is its e cient implementation.

The result of our analysis leads to Algorithm . which has been tested on several real-world graphs, con rming that its ability to perform CD on arbitrary graphs. Given the high clustering performance achieved by Algorithm . , as we can also claim that the intuitions encoded in Claim . for the DCSBM are, to some extent, also veri ed on arbitrary graphs.

The results reported in Table . and Figures . , . put us in position to take a step back and look at our results from a di erent angle. We can observe that, while the results of Algorithm . and [QR ] are always the best, the other competing spectral algorithm lead to very good or very poor partitions, according to the setting considered. In the next chapter we show that a uni ed framework can be devised to interpret the aforementioned SC algorithms, correctly predicting when and why each algorithm fails or succeeds at nding a high quality node partition. degree corrected stochastic block model (DCSBM)-generated graphs and a careful implementation capable of performing CD on arbitrary graphs that do not verify the DCSBM assumption, leading to Algorithm . . We here close our analysis of SC for CD with a high level look over the problem, based on the results obtained and detailed in the previous chapters.

The remainder of the chapter is structured as follows. First we consider the algorithms of [LR+ ; Jin+ ; New ]which are based on the use of the eigenvectors of the adjacency (or of the closely related modularity) matrix. Similarly to the Bethe-Hessian matrix, we argue that a mapping with statistical physics is capable to motivate also these algorithms and naturally justi es the superiority of Bethe-Hessian-based approach.

Laplacian-based algorithms are then considered, explaining how Algorithm . provides an "optimum" between the classical Laplacian matrices [Fie ; SM ; NJW b] and the regularized ones [SKZ ; QR ].

These results allow us to claim that Algorithm . solves the crucial problem of the ambiguous de nition of sparsity, being adapted to operate in both the dense and the sparse regimes.

. In this section we relate Algorithm . with the SC algorithms of [LR+ ; Jin+ ] that, referring to Algorithm . , impose the choice M = A and [New ], which instead uses a rank one perturbation of the adjacency matrix, M = A -dd T 2|E | , showing that these algorithms unfold naturally following the procedure used to derive the Bethe-Hessian matrix, but for a di erent variational approximation.

Recall from Chapter that the Bethe approximation for the Ising model is a variational approximation that is particularly suited when dealing with sparse graphs. A simpler, alternative variational method is represented by the naïve mean eld (NMF) approximation, also known in the inference literature as naïve Bayes [Ris+ ].

Let us consider an Ising-like Hamiltonian without external elds (Equation ( . )) de ned on the graph G(V, E ):

The Ising Hamiltonian

H(s) = -∑ (ij)∈E J ij s i s j .
We recall from Chapter that the two fundamental steps of variational approximation are: (i) de ning a parametric free energy

The variational free energy

Fβ (q) = ∑ s p q (s)(βH(s) + log p q (s)),

where p q (s) is a probability distribution whose expression de nes the variational approximation and (ii) nd the set of parameters q minimizing Fβ (q). In the Bethe approximation, p q (s) is the expression of the Boltzmann distribution ( . ) on trees, making it adapted to sparse graphs. The NMF approximation is instead obtained assuming all spins to be independent, i.e.

The NMF approximation

p MF m (s) = ∏ i∈V 1 + mi s i 2 . ( .
)
The distribution p MF m simply factorizes over all nodes and mi is the expectation of s i according to Equation ( . ). We then can write the corresponding variational free energy as follows: . Yellow stars correspond to SC as per [LR+ ], red squares as per [Jin+ ], green triangles as per [New ] and the blue diamonds are the result of Algorithm . . Left: dense regime, c = 200. Right: sparse regime, c = 8. Averages are taken over 10 samples.

FMF β ( m) = -∑ (ij)∈E βJ ij mi mj + ∑ i∈V ∑ s i 1 + mi s i 2 log 1 + mi s i 2 .
Following the same procedure as in Chapter to obtain the Bethe-Hessian matrix, we now derive the Hessian matrix of FMF β ( m) at the paramagnetic point m = 0 n , which we denote with H MF β .

H MF β ij = ∂ 2 FMF β ( m) ∂ i mi ∂ mj m=0 n = (I n -βJ) ij .
Also in this case, the smallest eigenvalues of H MF β and their corresponding

The NMF Hessian at the paramagnetic point eigenvectors bring information on the local minima of the free energy and, in the context of CD, can be exploited to determine the community structure of G(V, E ). The smallest eigenvalues of H MF β are however trivially related to the largest eigenvalues of J and the two matrices share the same eigenvectors. Unlike the Bethe-Hessian matrix, however, in this case the temperature β plays no role in determining the eigenvectors of H MF β and, consequently, it is pointless to tune it to an optimal value. The relation with SC comes naturally into play when imposing J = A that corresponds precisely to the choice made to derive the Bethe-Hessian matrix H r . This choice naturally leads to a NMF version of SC based on the eigenvectors of H MF β associated to its smallest eigenvalues, the smallest of which corresponds to the ferromagnetic con guration (it is the Perron-Frobenius eigenvalue of A), while the subsequent eigenvectors are correlated with the community structure. From these elements, the relation with [LR+ ; Jin+ ] unfolds naturally.

If instead one chooses J = A -dd T 2|E | , the NMF approximation leads to the algorithm of [New ] (at least in the case of k = 2 communities) in which the ferromagnetic con guration is energetically penalized and clustering should be performed on the eigenvector associated to the smallest (resp. largest) eigenvalue of M MF β (resp. A -dd T 2|E | ). It has to be noted however that (i) unlike the adjacency matrix, the modularity matrix is dense, hence the eigenvectors are computed in a less e cient way; (ii) on su ciently dense graphs, the Perron-Frobenius eigenvector of A is strongly aligned with the degree vector. Given that A is Hermitian, all its eigenvectors (expect the Scatter plot of the leading eigenvector of A (x 1 ) vs d for a dense DCSBM graph

x 1 d leading one) are essentially orthogonal to d and are, therefore, a close approximation of the eigenvectors of A -dd T 2|E | , as well. For these reasons the algorithm of [New ] and of [LR+ ] are expected to have a very similar performance (as con rmed by Figure . ), with the rst being computationally faster. The SCORE algorithm of [Jin+ ] is then an e cient improvement of [LR+ ] which allows one to keep he degree heterogeneity into account. The performances of [New ; LR+ ; LR+ ] are compared with the output of Algorithm . in Figure . for k = 2 class DCSBM graphs with high average degree on the left and with low average degree on the right.

With these results at hand, we can a rm that the algorithms of [New ; LR+ ; Jin+ ] can be directly compared with Algorithm . since they can be obtained following the same procedure, yet for a di erent variational approximation. The nal step is to claim that the Bethe approximation is overall more accurate than NMF [MWJ ] on random graphs. Intuitively, this can be understood saying that NMF can be seen as a rst order approximation (only the rst moment of s is kept into account), while the Bethe approximation is a second order approximation (that considers also the spins correlations).

More speci cally, the NMF is appropriate for dense graphs [OS ], being in fact asymptotically exact on the fully connected Curie-Weiss model [KPW ] in which J = J1 n 1 T n . It performs instead poorly on graphs with a low average

The NMF approximation is appropriate on dense but not on sparse graphs degree, as a consequence of the spectral behavior of A in the sparse regime.

On the opposite, the Bethe approximation (hence H r ) is adapted to deal with sparse graphs. One may wonder, however, if it is still a good option to study dense graphs. It can be shown [OS ] that in the dense regime, the Bethe free energy can be approximated with the so-called TAP free energy [TAP ] which di ers from the NMF free energy due to existence of a correction, called

Onsager reaction term [Ons ].

The appropriateness of the Bethe approximation on dense random graphs can further be understood from a random matrix theory (RMT) perspective. We showed in fact in Section . . that there exists a trivial relation between the spectral properties of the non-backtracking and adjacency matrices on dense SBM-generated graphs and further observed empirically that this relation holds also in the DCSBM case.

For this reason, we claim that SC based on the Bethe-Hessian matrix H r is expected to provide better performances than SC based of the adjacency This is formally proved for the DCSBM in [Jin+ ], but intuitively it holds in general as a consequence of d Recall that the fundamental task of a variational approach is to approximate the free energy (De nition . ). This problem is well de ned, independently of CD and for any graph. The claim of a class of SC algorithms being uniformly superior to another one is, certainly, very hard to motivate, also recalling the di culty to assess the performance of CD on practical tasks. The word "expected" plays here the fundamental role to underline that the assertion has not to be considered as a strict rule (counterexamples in which SC based on A performs better than H r may be potentially found) but rather as a general trend. matrix A. Moreover, unlike A, the matrix H r seems to be a suitable candidate to perform SC for any average degree, solving the crucial problem introduced by the ambiguity of the de nition of sparsity. For this reason, H r allows one to bridge the dense and sparse regimes.

. This section draws some connections between the Laplacian-based algorithms of [Fie ; SM ; NJW b; QR ], including also the Bethe-Hessian of [SKZ ]. We here refer to SC based on L sym/rw and L sym/rw τ under the same terms since from a spectral viewpoint, these pairs of matrices are closely related. Referring to the "classical" normalized Laplacian matrices, the only di erence between [SM ] and [NJW b] resides in step of Algorithm . . Similar comments can be made also for [QR ].

Let us rst consider the graph Laplacian matrix L = D -A of [Fie ]. The Bethe-Hessian matrix can be considered as a regularized version of L. The central result of Claim . revolves around the exploitation of a series of Bethe-Hessian matrices {H ζ p } p=1,...,k whose expression is reported in Equation ( . ) for convenience

H r = (r 2 -1)I n + D -rA.
( . )

For the DCSBM we claimed that, under proper hypotheses, the optimal regularization to extract information from the p-th eigenvector of H r is r =

ζ p = λ ↓ 1 (CΠ) λ ↓ p (CΠ)
+ o n (1), where we recall that (CΠ) ab is the average number of neighbours in class b of a node in class a. For easy detection problems, the o -diagonal terms of CΠ simply go to zero and, due to Assumption . according to which CΠ1 k = c1 k , the eigenvalues of CΠ become degenerate, hence, for all 1 ≤ p ≤ k, ζ p → 1 and H ζ p → D -A. For this reason, we can interpret the combinatorial graph Laplacian matrix as the trivial clustering limit of the optimal Bethe-Hessian matrix. We further recall from Remark . that the multiplicity of the zero eigenvalue of L = H r=1 equals the number of connected components of a given graph. Identifying communities with the connected components corresponds to the extreme case in

The combinatorial graph Laplacian matrix is the trivial clustering limit of the optimal Bethe-Hessian which the number of connections between nodes in the same community is larger than the connections across communities (which is equal to zero) and is the simplest setting in which CD can be performed. From the de nition of ζ (j)

p in Equation ( . ), the choice r = 1 unfolds naturally to detect the connected components of a graph.

From a statistical physics viewpoint, the choice r = 1 (irrespective of the actual value of ζ p ) corresponds to the zero temperature limit in which the free energy F = βU -S → βU neglects the contribution of the entropy. For this reason, when ζ p is far from 1, imposing r = 1 (i.e. , using the graph Recall that β is the inverse temperature and that βJ = ath(r -1 ). . Green dots correspond to SC as per [Fie ], light-blue diamonds as per [SM ] and red dots as per [Jin+ ]. Averages are taken over 10 samples.

Laplacian matrix) is equivalent to assuming a negligible contribution from the noise in G(V, E ) and hence leads to over-tting. In fact, the typical problem of SC based on L is that single nodes are isolated into one community, due to the existence of uninformative local minima in the energy spectrum.

Similar remarks can be made concerning the setting of disassortative communities. As it was commented in Section . , in this case ζ p < 0 and the trivial limit corresponds to ζ p = -1 for which H ζ p = D + A, also known

Disassortative and bipartite graphs

as signless Laplacian [CRS ]. It is known that this matrix is a useful tool to study bipartite graphs that correspond the "extreme disassortative setting" in which connections exist only between nodes in di erent classes.

If on the one hand D -A corresponds to the trivial clustering limit, the Bethe-Hessian H √ cΦ of [SKZ ] is the opposite regime, considering a worstcase scenario. In fact, the condition ζ p = √ cΦ corresponds (for k classes of equal size) to the detectability threshold, below which it is conjectured to be impossible to recover communities with an e cient algorithm [Dec+ ]. The choice r = √ cΦ allowed the authors of [SKZ ] to design an algorithm

The existing sparse SC algorithms are a "worst case" limit of the optimal Bethe-Hessian capable of making non-trivial clustering as soon as theoretically possible but that is generally suboptimal, as extensively commented in the previous chapters. Figure . clearly summarizes the behavior of the regularizer r: small values of r lead to better partitions, provided that they guarantee the existence of isolated informative eigenvalues in the spectrum of H r .

Let us now brie y consider instead the regularized Laplacian matrix

L sym τ = D -1/2 τ AD -1/2 τ ,
where D τ = D + τI n . In Chapter we showed that the optimal regularization consists in the choice τ = ζ 2 p -1. In the trivial regime, this leads to τ → 0, for which L sym τ → L sym (a similar relation holds of course for L rw τ and L rw ). Very similar comments to the ones made for D -A can be done also for the normalized Laplacian matrices that, indeed, are obtained, like L, from the relaxation of an optimization problem. Similarly to [SKZ ], the regularized Laplacian of [QR ] considers instead an "almost" worst case scenario. In order to guarantee detectability down to the threshold with L sym τ as in [SKZ ], one must pick τ = cΦ -1, while [QR ] proposes heuristically τ = c which is close to the parametrization which guarantees a non-trivial result also in the hardest regime.

With these observations being laid out, we conclude this section with two fundamental take-home messages. The Laplacian matrices (with the exception of the Bethe-Hessian of [SKZ ]) have been typically studied on dense Take home messages graphs because the mathematical tools to address this regime are generally more powerful and better known than those in the sparse regime. Unlike the matrix A that is derived from a variational approximation that is unsuited for the sparse regime, however, the poor performances of the Laplacian matrices are rather a consequence of an overly optimistic choice of the parametrization (r or τ) which corresponds to assuming the clustering problem to have a trivial solution. To con rm this claim, Figure . compares the overlaps a function of α = (cc out )/ √ c, obtained by the algorithms of [Fie ; SM ; LR+ ] for dense DCSBM graphs with k = 2 communities of equal size. This plot evidences that the algorithmic threshold of [LR+ ] (based of A) achieves smaller values of α than the one of [Fie ; SM ]. Concluding, while Section . argues that H r is a good candidate to bridge the dense and the sparse regimes, Section . exhibits that Algorithm .

nds the optimal between the trivial clustering regime (assumed de facto by [Fie ; SM ; NJW b]) and the hard one considered in [SKZ ; QR ].

. In this chapter we provided a critical overview of the main methods for SC for CD taken under consideration in this manuscript and showed how they can all be understood in a simple uni ed framework that fosters the adoption of Algorithm . for CD. The aforementioned methods were mostly developed in parallel by di erent communities and motivated with di erent lines of arguments. The existence of a uni ed view (yet, we acknowledge, somewhat biased by the statistical physics perspective) is an important contribution to relate and interpret the existing results. Notably, we discovered that the hyper-parameters involved in these algorithms must be smartly selected, this selection being a function of the di culty of the clustering problem: nding these hyper-parameters is key to the clustering performance.

The necessity to adapt the choice of the graph representation matrix to the hardness of the problem at hand is a profound observation, likely not restricted to CD. In light of this observation, it may appear that some common optimization problems, beyond clustering, have similarly been devised in the past to perform well in easy scenarios, but would require a task-hardness related update to perform better in harder settings.

In the next chapters, we provide practical examples of how the results of Part II can be generalized to more involved scenarios. In particular, in Chapter we consider the problem of dynamical CD that realistically keeps into account for the fact that real-world systems modelled by networks typically evolve with time. Moreover, Chapter treats our extension to the case of weighted graphs, with the speci c application to the sparsi cation of kernel matrices used to obtain cost e cient data clustering spectral algorithms. The dynamical setting is of utmost importance in modern network science, since most systems modelled by graphs indeed have a non-trivial dependence on time. Keeping the graph dynamical component into account allows us to e ciently exploit information coming from di erent times, consequently improving the clustering quality. As a main result, we develop a novel spectral algorithm for dynamic are typically snapshots or averages of the system's interactions occurred in a given time interval. To give a practical example, consider the karate network [Zac ] representing the friendships between 34 members of a karate club. The most signi cant interactions have been established studying the club members relationships in the a period ranging from 1970 to 1972: the resulting graph representation encodes therefore an average of the members' interactions over the considered time window.

G E N E R

Like in the case of the karate network, most systems that can be modelled with graphs have an intimate dynamical component and a richer description can be provided with the use of dynamical graphs [HS ]. Examples of these instances include social groups in which interactions are determined by human proximity [Cat+ ; EP ] or face-to-face interactions [Bar+ ; BC ], biological cells in which edges model gene or protein interactions [PSS ; Leb+ ] and transportation networks [PS ; Bor+ ; Ban+ ], representing how di erent locations are connected among themselves.

The inclusion of an additional degree of freedom representing time makes it particularly challenging to agree on the most basic concepts concerning temporal graphs and no general consensus exists on several de nitions and nomenclature [Hol ]. In the remainder we consider the following de nition A snapshot temporal graph (picture taken from [Muc+ ])

of temporal graph, sometimes referred to as snapshot graph [RC ].

De nition . (Snapshot graph).

A snapshot graph is a temporal graph representation, composed by an ordered sequence of T ∈ N graph snapshots {G 1 (V 1 , E 1 ), . . . , G T (V T , E T )} in which (ij) ∈ E t indicates that nodes i and j are connected at time t. To a temporal graph is associated a sequence of adjacency matrices {A (1) , . . . A (T) }.

A snapshot graph is, by all means, a multiplexed graph (introduced in Chapter ) in which each level corresponds to a snapshot. Note that in De nition . time is discrete and for each t, G t (V t , E t ) is a static graph.

With the basic notions concerning dynamical graphs being laid out, we take into account how the problem of CD can be formulated in its dynamical version, allowing us to extend our results from Part II to this setting.

. The problem of dynamical community detection (DCD) can be informally dened as the task of performing CD on a dynamical graph, assigning a class label to each node as a function of time. The underlying assumption motivating DCD is that, as the graph evolves, also the communities change their structure. As a practical example one can think of a citation network in which, for instance, an edge (ij) ∈ E t means that author i cited author j in year t. The community structure in this graph is determined by the authors' research interests which may change over time, thus altering the community structure. A practical representation of this process is shown in Figure .  (taken from [RB ]) in which changes in the community structure are evidenced with the help of an alluvial diagram.

As a consequence of the communities being expected to evolve smoothly across time [RC ; CR ], high correlations are observed in the community labels at di erent times. A major advantage of DCD with respect to CD is the possibility to exploit such correlations, hence using information that may come from past (or future) realizations of the graph to determine the class structure at a given time.

In order to formally address DCD, it is necessary to clearly frame the problem from a mathematical perspective. In Part II we insisted on how providing a de nition to CD is a generally hard task and, in the dynamical case, the problem of de ning communities is even more exacerbated by the complexity that rules the dynamical evolution of the graph. As a consequence, even the most expert authors in the eld [HS ; RC ; CR ] have struggled to provide a uni ed view on the problem of DCD that, in general, may have very di erent de nitions, strategies as well as objectives. For this reason, we do not attempt in this chapter to provide a (even incomplete) review of DCD: given the diversity of the di erent approaches, in many cases, results cannot be compared on even grounds. On the opposite, in the remainder we will speci cally consider the problem of label inference from the dynamical degree corrected stochastic block model (DDCSBM) [Xu ] which is a generalization of the degree corrected stochastic block model (DCSBM) studied in Part II. Like in the static case, this model has the important advantage to provide a great mathematical control, motivating the reason of our choice.

For the sake of precision, we should denote (ij) ∈ E d,t since the natural representation of this system is obtained with a directed network. . .

In this section we introduce the DDCSBM together with its conjectured detectability threshold. First of all, let us settle some notation. We denote with T ∈ N the number of snapshots and we let {G t (V t , E t )} t=1,...,T be a sequence of unweighted and undirected graphs, each with n nodes. At time step t, E t and V t denote the set of edges and nodes, respectively, which form G t (V t , E t ), with V t ∩ V t = ∅, for t = t: each node has T copies, each copy being a di erent object. We denote with i t , for 1 ≤ i ≤ n and 1 ≤ t ≤ T, a node in V t . We call A (t) ∈ {0, 1} n×n the symmetric adjacency matrix of G t (V t , E t ) and D (t) its associated degree matrix. We now detail the generative model for {G t (V t , E t )} t=1,...,T .

First of all the label dynamics must be speci ed. Let i t ∈ {1, . . . , k} be the label of node i t , i.e. the label of node i at time t and let π ∈ R k be the (time independent) vector encoding the expected fraction of nodes belonging to each class. The labels are updated across time as per De nition . .

De nition . (Label dynamics).

Given three positive integers k, n, T and a vector π ∈ R k satisfying 1 T k π = 1, the vector ∈ {1, . . . , k} nT is de ned as follows. First, the entries { i t=1 } i=1,...,n are initialized by assigning random values with probability proportional to the class sizes: P( i 1 = a) = π a . The remaining entries labels are then obtained for 2 ≤ t ≤ T according to

The label dynamics is a Markov process

i t =    i t-1 w.p. η a w.p. (1 -η)π a , a ∈ {1, . . . , k}, ( . )
i.e., the label of node i t is maintained with probability η and otherwise reassigned at random with probability 1η.

. Note that, according to De nition . , a proportion of the reassigned nodes from time t will be a ected the same labels at time t + 1. It is easy to verify from De nition . that the expected sizes of the clusters are constant across time, in fact:

E [|V t,a |] n = 1 n E η|V t-1,a | + k ∑ b=1 |V t-1,b |(1 -η)π a = η E [|V t-1,a |] n + (1 -η)π a . (
dν(θ) = 1 (normalization), E[θ] = Θ θ dν(θ) = 1, E[θ 2 ] = Θ θ 2 dν(θ) ≡ Φ = O n (1)
. For each node, θ i is drawn independently at random from the distribution ν.

The entries of the (symmetric) adjacency matrix A (t) of G t (V t , E t ) are set to one independently and independently across time, with probability:

P(A (t) ij = 1) = min θ i θ j C i t , j t n , 1
and are equal to zero otherwise.

In words, given the label vector , the DDCSBM generates the elements of {G t (V t , E t )} t=1,...,T independently at random from the static DCSBM with the class labels that change across time. This model makes strong assumptions on the dynamics, keeping xed the matrix C, the number and size of the communities, as well as the parameter η, but it is however capable to generate a snapshot graph with labels correlated across time.

Before proceeding any further, let us make a remark of the independence assumption of each graph across time.

Remark . (Graph independence across time).

In the DDCSBM the instances G t (V t , E t ) are generated independently across time. Let us translate this assumption in simpler terms. In a practical setting, we can de ne a time scale, say τ, which determines the speed at which the graph evolves. Assuming that the snapshots of the graph are independent means to assume that τ 1, i.e.

that the typical time needed for the graph edges to rearrange is much smaller than the distance between two successive snapshots. This assumption is hence adequate when the snapshots are su ciently far in time.

Related to Remark . , it has to be noted that a second time-scale can be de ned, describing the speed at which the community structure evolves. This time scale depends on η and goes to in nity for η = 1 (the community labels are frozen) and to zero for η → 0 (the communities are completely recon gured between one snapshot and the successive one). The fundamental hypothesis introduced by the DDCSBM consists in assuming that the graph evolves at a much faster rate than the communities, so that, while τ → 0 (due to the independence assumption), η = O(1).

Our objective is to infer the vector ∈ {1, . . . , k} nT , hence to assign to each node at each time the appropriate community label. Like in the static case, also for DDCSBM there exists a detectability threshold below which inference is not asymptotically feasible. In particular, in [Gha+ ] the authors retraced the steps made by [Dec+ ] in the static regime, deriving the (asymptotically exact) belief propagation (BP) equations to perform inference in the dynamical stochastic block model (DSBM) (in which θ = 1 n ) to then determine the instability condition of BP. In the setting of k classes of equal size with

C = c out 1 k 1 T k + (c in -c out )I k , c = 1 k C1
k and de ning α as in the static case, i.e. α = (cc out )/ √ c, the authors of [Gha+ ] conjectured that, for

T → ∞, DCD is feasible if α > α c (∞, η), where α c (∞, η) is the unique value of ᾱ > 0 for which the largest eigenvalue of M ∞ (ᾱ, η) = ᾱ2 2η 2 ᾱ2 η 2 ( . )
is equal to one. In [Gha+ ] the authors further gave clues to generalize their result for T nite but did not provide an explicit expression to α c (T, η).

In Section . we extend the conjecture of [Gha+ ], providing the explicit expression to α c (T, η) and generalizing it to the DDCSBM setting.

With this fundamental property of the DDCSBM being settled, let us now present the main contributions and challenges considered in this chapter.

. .

Our main contribution consists in the design of a new, e cient algorithm to perform SC in dynamical graphs generated from the DDCSBM. Albeit SC in dynamical graphs is much less studied than it is on static graphs, recently it has gained more and more interest from the scienti c community and SC algorithms have also been designed to operate in the dynamic regime. The main goal of DCD is to exploit the correlation of the labels across time in order to improve the clustering performance.

One of the most common approaches is given by evolutionary spectral clustering [Chi+ ; Qin+ ; Chi+ ; Liu+ ; XKHI ] in which the community assignment is de ned for all t, optimizing Q ev t , de ned for all t ≥ 2 as:

Q ev t ( (t) , (t-1) ) = hQ snap t ( (t) ) + (1 -h)Q smooth t,t-1 ( (t) , (t-1) ), ( . )
where h is a parameter between 0 and 1, Q snap t and Q smooth t,t-1 are two quality functions evaluating the goodness of a partition on a single snapshot and imposing a smooth change between (t) = { i t } i=1,...,n and (t-1) , respectively. More practically, Q snap t can be, for instance, the modularity, the RCut, or some other cost function introduced in Section . ; an example of Q smooth t,t-1 , instead, is the overlap which corresponds to large values if (t) and (t-1) are similar, while it tends to zero if the community labels are completely rearranged. Tuning the value of h, one can enforce higher (or smaller) levels of smoothness in the label evolution.

Spectral clustering allows one to obtain a relaxed solution of the optimization of Equation ( . ). Beyond all the weaknesses of the optimization approach that we extensively discussed in Part II, it has to be noted that: (i) the function Q ev t does not contain any information from all t > t (it only depends on t and t -1); (ii) the resulting relaxation does not, in general, allow one to achieve the detectability threshold and are not adapted for sparse graphs; (iii) smoothness is often included in a very heuristic way, consequently leading to suboptimal algorithms.

A di erent, but somewhat related approach consists in performing SC using as an input a modi ed adjacency matrix obtained by averaging the last realizations of the adjacency matrix, with possibly a forgetting factor giving less weight to older times [PZ+ ]. For instance, we can write

Ã(t) = (1 -h) Ã(t-1) + hA (t) .
Also in this case, clustering at time t does not exploit information from instances appearing after t. This method has however been studied in the sparse regime c = O n (1) by [KV ] in which the authors showed that for Averaging the adjacency matrix simply corresponds to the choice h = 1 in

Ã(t) = (1 -h) Ã(t-1) + hA (t)
η → 0 (the classes are supposed to be essentially frozen) and T su ciently large, one obtains good clustering performances in the sparse regime, simply running Algorithm . on the matrix M = 1 T ∑ T t=1 A (t) . While this result is powerful, because it formally addresses the sparse regime, it essentially treats DCD as a perturbation to static CD and does not cover the more interesting regime in which communities can evolve rapidly with time.

To the best of our knowledge, the work of [Gha+ ] provides the only existing spectral algorithm properly treating both sparsity and small label persistence. In the spirit of [Krz+ ], the proposed method arises from a linearization of the BP, which is capable of obtaining non-trivial partitions as soon as theoretically possible. However, their resulting dynamical nonbacktracking matrix depends on an a priori unknown parameter, so the algorithm is practically inapplicable.

As an answer to these limitations, we propose a new spectral algorithm adapted to the sparse regime, which is able to detect communities even under little (or no) persistence in the community labels and which bene ts from persistence to improve classi cation performance over a static algorithm run independently at each time-step. Speci cally,

• We introduce a dynamical Bethe-Hessian matrix which, for k = 2 classes of equal size, retrieves non-trivial communities as soon as theoretically possible for any T, η. As a by-product, we o er new results on the spectrum of the dynamical non-backtracking of [Gha+ ].

• We provide an algorithm applicable to any graph with k ≥ 2 communities of arbitrary sizes. The resulting Algorithm . is part of the CoDeBetHe.jl package.

. T

In this section we formulate our generalization for nite T of the claim of [Gha+ ] concerning the detectability threshold in the DSBM, further considering its extension to the DDCSBM setting. For convenience, we assume in the remainder the case of k = 2 classes of equal sizes, leaving to Section . the discussion of how our results can be straightforwardly generalized to the k > 2 setting.

In [Gha+ ] the authors introduce a threshold α c (T, η), however not explicitly de ne, below which (α < α c (T, η)) community detection is not feasible. We go here beyond [Gha+ ] by providing an explicit value for α c (T, η) for all nite T as in the following claim, whose details are treated in Appendix C.

Claim . (Detectability threshold in the DDCSBM). Let {G t (V t , E t )} t=1,...,T be a sequence of graphs generated from a DDCSBM with k = 2 classes of equal size as per De nition . with

C ab = c in if a = b and c out otherwise. Let α = (c -c out ) √ Φ/c
, then the labels of the DDCSBM can be inferred with an e cient algorithm only if α > α c (T, η), where α c (T, η) is the only positive ᾱ for which the largest eigenvalue of M T (ᾱ, η) is equal to one, where

M T (ᾱ, η) =           M d M + 0 . . . 0 M -M d . . . . . . 0 0 M - . . . M + 0 . . . . . . . . . M d M + 0 0 . . . M -M d           , with                  M d = 0 0 0 η 2 ᾱ2 η 2 0 0 0 , M + = 0 0 0 0 0 0 0 ᾱ2 η 2 , M -= η 2 ᾱ2 0 0 0 0 0 0 0 .
The algorithm a priori requires that η be known; otherwise, η can be estimated through cross-validation. Precisely, quoting the authors, this is as far as α c (T, η) is de ned: "We can compute the corresponding nite-time threshold for a xed T by diagonalizing a (3T -2)-dimensional matrix, where we have a branching process with states corresponding to moving along spatial, forward-temporal, or backward-temporal edges at each time step". The expression of α c (T, η) can be computed analytically for T = 2, 3, 4 and T → ∞:

α c (T = 2, η) = 1 + η 2 -1 2 α c (T = 3, η) = √ 2 2 + η 4 + η 2 8 + η 4 -1 2 α c (T = 4, η) = √ 2 2 + η 2 + η 6 + η η 8 + 2η 4 + 8η 2 + 5 -1 2 α c (∞, η) = 1 + η 2 1 -η 2 -1 2 .
For other values of T, α c (T, η) is best evaluated numerically.

The value of α c (T, η) is shown as a function of η for di erent T in Figure . . Let us make some basic observations on the values of α c (T, η). For High label correlation and many graph observations allow one to solve harder CD problems all T: (i) if η = 0 (no correlation among the labels), one recovers α c = 1, the transition's position in the static DCSBM [GLM ], as expected; (ii) if η = 1,

α c = 1/ √
T, the static threshold obtained by averaging the adjacency matrix over its T independent and identically distributed realizations. This result should be related to [KV ] that indeed suggests to used the averaged adjacency matrix to perform SC when η → 1. We also numerically con rm that for all T, α c (T, η) is a decreasing function of η: higher label persistence allows to solve harder problems. . This section develops a "dynamical" Bethe-Hessian matrix associated to the graph sequence {G t (V t , E t )} t=1,...,T , for which we show there exists at least one eigenvector (recall that k = 2 classes so far) strongly aligned to the community labels if α > α c (T, η), thereby allowing us to design a CD algorithm that achieves the detectability threshold. The eigenvectors containing information can be as many as T, but only one of them is guaranteed to exist when α > α c (T, η) and it can alone reconstruct communities.

Before introducing our main result, it is convenient to introduce the aggregate graph G(V, E ), de ned as follows.

De nition . (Aggregate graph). Letting {G t } t=1...T be a sequence of graphs generated as per De nition . ,

G = G(V, E ) is the graph with V = ∪ T t=1 V t and E = ∪ T t=1 E t ∪ ∪ T-1 t=1 ∪ n i=1 (i t , i t+1
) . The adjacency and degree matrices of G(V, E ) are denoted with A, D ∈ N nT×nT , respectively. In other words, the graphs G t (V t , E t ) are joined adding extra edges between the nodes i t and their temporal neighbours i t±1 . . . As in Chapter , our approach exploits a statistical physics analogy between the modelling of spontaneous magnetization of spins with ferromagnetic interaction and the modelling of communities of nodes in sparse graphs. We de ne the following Hamiltonian on the graph G(V, E ).

H ξ,h (s) = -   T ∑ t=1 ∑ (i t ,j t )∈E t ath(ξ) s i t s j t + T-1 ∑ t=1 ∑ i t ∈V t ath(h) s i t s i t+1   ( . )
Note that a very similar cost function has been used in [Muc+ ] to nd communities in dynamical graphs as the solution of an optimization problem. Here, the coupling constants ξ, h ∈ [0, 1) modulate the interaction among nodes at time t and between the same node at time instants t and t + 1, respectively and appear inside inverse hyperbolic tangents for notational ease. Intuitively, the spin vector s can be mapped to the class a liation vector

σ ∈ {±1} nT , in which σ i t = 1 if i t = 1 and σ i t = -1 otherwise.
The rst term in the main parenthesis of ( . ) favors con gurations in which neighboring nodes have the same label, while the second term favors con gurations in which the label is kept across successive time instants, enforcing persistence in the community evolution.

The mesoscale structure of G(V, E ) determines the appearance of some local minima in H ξ,h (s) like, for instance, the con guration s = σ. A sketch of such minima of H ξ,h (s) is provided for T = 2 in Figure . . The lowest energy state corresponds to s = 1 nT : this is the non-informative ferromagnetic con guration. Similarly, mode of Figure . groups together nodes in the same community and is equally useless for reconstruction. On the opposite, modes and of Figure . divide the nodes according to the class structure of G(V, E ) and can be used for community reconstruction. In general, for k classes and T > 2 time frames, kT such local minima arise, mixing together time and class clusters. Note importantly that mode always has a lower energy than mode and mode a lower energy than mode . However, the ordering of energies of modes and is in general not a priori known. We will further comment on this remark which has important consequences for the subsequent analysis as well as for the design of our proposed community detection algorithm.

The eigenvectors corresponding to the smallest eigenvalues of the Bethe-Hessian matrix (De nition . ) associated to the Hamiltonian of Equation ( . ) are strongly correlated with the con gurations appearing in Figure . and can therefore be used to perform DCD. From the results described in Section . . , the dynamical Bethe-Hessian matrix H ξ,h ∈ R nT×nT associated to the Hamiltonian of Equation ( .) is de ned as follows

The dynamical Bethe-Hessian matrix

H ξ,h i t ,j t =      ξ 2 D (t) -ξ A (t) 1-ξ 2 + 1+h 2 (φ t -1) 1-h 2 I n ij if t = t -h 1-h 2 I n ij if t = t ± 1, ( . )
where φ t = 1 if t = 1 or t = T and φ t = 2 otherwise. The aforementioned lack of a precise knowledge of the relative position of the informative modes in the energy spectrum of the Hamiltonian hampers the identi cation of the position of the corresponding informative eigenvectors of H ξ,h , which is of major importance to design a SC algorithm based on H ξ,h .

. .

We thus now turn to our main result centred on the question of appropriately choosing a pair (ξ, h) which ensures non-trivial community detection with H ξ,h as soon as α > α c (T, η).

In order to determine the spectral properties of H ξ,h , we study the eigenvalues of the weighted non-backtracking matrix B ξ,h , to then exploit the relation between the two spectra as per Theorem . and Claim . . Recalling the de nition of the weighted non-backtracking matrix used in Section . . , the expression of B ξ,h on the aggregate graph G(V, E ) reads

The dynamical non-backtracking matrix

∀(ij), (kl) ∈ E d , B ξ,h (ij)(kl) = δ jk (1 -δ il ) ω kl , ( . )
where E d is the set of directed edges of the aggregate graph G(V, E ), ω kl = ξ is (kl) is a spatial edge and ω kl = h if it is a temporal edge. The choice of a proper parametrization (ξ, h) to perform non-trivial DCD with the dynamical Bethe-Hessian matrix comes from the following central claim describing the spectrum of B ξ,h , whose detailed derivation is provided in Appendix C. .

Claim . (Spectrum of B ξ,h on sparse DDCSBM graphs). Let G(V, E ) be a graph generated as per De nition . with k = 2 communities, C a=b = c in , C a =b = c out and γ = (c inc out )/(c in + c out ). Recalling the de nition of M T (•, •) from Claim . , then, with high probability, in the n → ∞ limit, the spectrum of B ξ,h can be described as follows:

• the complex eigenvalues forming the bulk of B ξ,h are bounded by a disk in the complex plane of radius L ξ,h = λ

↓ |•| 1 (M T ( cΦξ 2 , h));
• all the eigenvalues of B ξ,h of magnitude larger than L ξ,h are isolated and are asymptotically close to one of the eigenvalues of either M T (

√ cΦξ, √ h)
(in which case the eigenvectors are not correlated to the community labels) or M T ( √ cΦξγ, ηh) (in which case the eigenvectors are correlated to the community labels).

Figure . con rms numerically Claim . . We choose to compute only the 150 eigenvalues with largest real part to keep a reasonable computational time, while having a large value of n. From Claim . , the position of the isolated and bulk eigenvalues of B ξ,h is de ned for all (ξ, h). Note that two families of isolated eigenvectors appear in the spectrum of B ξ,h : one is the informative family (related to the con gurations , of Figure . ) and one is the uninformative family (related to con gurations , of Figure . ). The ordering of the eigenvalues belonging to these two families is a priori unknown and depends on ξ, h. Let us make a remark on an interesting behavior of the isolated eigenvalues of B ξ,h .

Remark . (Complex isolated eigenvalues in the spectrum of B ξ,h ). Note that the isolated eigenvalues of B ξ,h are asymptotically close to the eigenvalues of the matrix M T (•, •). This matrix is real and non-negative, but it is not symmetric. Consequently, its leading eigenvalue is certainly real (due to Perron-Frobenius theorem), while the subsequent eigenvalues are potentially complex. Although we cannot o er a clear interpretation for the complex nature of some of these isolated eigenvalues, our study is experimentally veri ed to hold also in this case as shown in the left plot of Figure . . Based on Claim . , we now provide two choices of the parametrization (ξ, h) that allow one to devise a SC algorithm based on the Bethe-Hessian matrix to perform DCD as soon as α > α c (T, η).

The optimal Bethe-Hessian parametrization

The rst proposed parametrization that allows one to perform non-trivial DCD as soon as α > α c (T, η) consists in choosing (ξ, h) = (γ, η). Recalling the results of Chapter , we note that, in this setting,

γ = ζ -1 2 + o n (1), the
The optimal parametrization is obtained from the relaxation of the asymptotically Bayes optimal BP equations optimal parametrization of the static Bethe-Hessian matrix. The fact that γ is (asymptotically) the inverse of ζ 2 (and not ζ 2 ), is just due to a notational convenience: in fact in Chapter , r ≥ 1, while here ξ ≤ 1.

First of all we want to show that the matrix B γ,η can be used for community reconstruction. This is a straightforward consequence of Claim . : letting ξ = γ and h = η, we obtain that the leading informative eigenvalue is equal to λ info,1 = λ

↓ |•| 1 (M T (α, η)), while the radius of the bulk is equal to L λ,η = λ ↓ |•| 1 (M T (α, η)) = λ info,1 . By de nition, if α > α c (T, η), then λ info,1 > 1, therefore λ info,1 > L λ,η . So for all α > α c (T, η), λ info,1 is an 2 0 2 4
Real part

= 0. For both simulations, T = 2, η = 0.4, c = 6, c out = 1, θ = 1 n , n = 2 000.
isolated eigenvalue in the spectrum of B λ,η and it contains information on the community labels.

The claim of optimality of the choice (γ, η) comes from the fact that, like in the static case, this is the natural parametrization obtained linearising the Bayes optimal BP equations. Let H be de ned as follows:

H = 1+η 2 1-η 2 1+η 2 1-η 2
Letting a, b ∈ {1, 2}, the xed point BP equations take the form [Gha+ , Equations , , ]

m j t ,i t (a) = e -f t (a) Z j t ,i t ∑ b H ab m i t ,i t+1 (b) ∑ b H ab m i t ,i t-1 (b) ∏ l t ∈∂i t \j t ∑ b C ab m i t ,l t (b) m i t+1 ,i t (a) = e -f t (a) Z i t+1 ,i t ∑ b H ab m i t ,i t-1 (b) ∏ l t ∈∂i t ∑ b C ab m i t ,l t (b)
where

f t (a) = 1 n ∑ j∈V t ∑ b C ab m i t ,j t (b).
Expanding the above messages around the trivial xed point

m j t ,i t (1) = 1/2 + i t ,j t , m i t ,i t±1 (2) = 1/2 -i t ,i t±1 , we obtain j t ,i t = η( i t ,i t-1 + i t ,i t+1 ) + γ ∑ t ∈∂i t \j t i t , t i t+1 ,i t = η i t ,i t-1 + γ ∑ t ∈∂i t i t , t .
These equations can be rewritten in synthetic form introducing exploiting the non-backtracking matrix

B γ,η = + h.o.t
In agreement with our empirical observations, we predict that the informative eigenvalue λ info,1 of B λ,η has a twin companion eigenvalue asymptotically close to one. Recalling that if α > α c (T, η), then the radius of the bulk of B γ,η is L γ,η > 1, we conclude that this is an eigenvalue inside the bulk of B γ,η and isolated down to the detectability threshold, like in the static case. The left plot of Figure . con rms this observation, evidencing that, indeed, the matrix B γ,η has such unitary eigenvalue inside its bulk. Applying Theorem . it is easily shown that the dynamical Bethe-Hessian matrix H γ,η has a zero eigenvalue as shown in the right plot of Figure . . We summarize and formalize these results in the following claim.

Claim . (Spectral clustering with H γ,η ). Let G(V, E ) be an aggregate graph obtained as in De nition . . As n → ∞, with high probability, the complex bulk eigenvalues of B γ,η are asymptotically bounded by a circle in the complex plane of radius L γ,η = λ

↓ |•| 1 (M T (α, η)). Besides, if α > α c (T, η), then 1 < L λ,η , 1 is an isolated eigenvalue of B λ,η
and 0 is an isolated eigenvalue of H λ,η , and the corresponding eigenvectors for both matrices are correlated to the vector of community labels.

The matrix H λ,η hence represents a natural extension of the optimal Bethe-Hessian discussed in Chapter for static CD. While from a theoretical perspective Claim . may seem a su cient result, from a practical viewpoint a fundamental piece of information is lacking: the location of the zero eigenvalue in the ordered set of eigenvalues of H γ,η . We explained extensively in Part II the importance of actually being able to locate the informative eigenvalues of a matrix to design a SC algorithm and one of the big results of Claim . concerns indeed the position of the informative eigenvalues of the Bethe-Hessian. Unlike Claim . , however, Claim . does not specify the location of the informative eigenvalues. This is because, as we anticipated already, the eigenvalues of informative and uninformative families described in Claim . can potentially be in a shu ed order, making it unpractical (if not impossible) to locate the informative eigenvalues without a prior knowledge of the values (γ, η). One may object that (γ, η) must be known to build H γ,η . It has to be recalled, however, that (γ, η) are generally not known in advance and have to be estimated. The lack of knowledge of the position and value of the informative eigenvalues, however, makes it impossible to design an algorithm like . to estimate (γ, η) and perform DCD.

Before proceeding with the second possible parametrization that overcomes this problem, let us formulate a further remark on the matrix B γ,η , relating our results to those of [Gha+ ].

Remark . (Comparison with the spectral method of [Gha+ ]). In [Gha+ ], the authors propose a SC method based on the matrix B γ,η , obtained linearising the BP equations around their trivial xed point as shown above. As we mentioned already in Section . , this matrix needs to know the parameter γ in order to be de ned and the related SC algorithm cannot be straightforwardly implemented since, as we just commented, γ is not easy to estimate.

= 0. For both simulations, T = 2, η = 0.4, c = 6, c out = 1, θ = 1 n , n = 2 000.
On top of this, in [Gha+ ] the authors prescribed, for a general setting with k communities, to perform SC on the k dominant eigenvectors of B γ,η . Claim . however clearly evidences (in the case k = 2) that the k largest eigenvalues are not necessarily informative. Consequently, using the k dominant eigenvectors to perform SC may lead to poor performance, as evidenced in Figure . .

Let us now show that an alternative parametrization (ξ, h) equally allows one to perform DCD as soon as theoretically possible.

An alternative parametrization

The second parametrization (ξ, h) to achieve detectability down to the threshold is inspired from the Bethe-Hessian of [SKZ ] that de ned, for the static Recall that in [SKZ ], SC is performed using H r for r = √ cΦ stochastic block model (SBM), a parametrization independent of c inc out to perform SC. This parametrization consists in choosing (ξ, h) = (γ d , η), where γ d = α c (T, η)/ √ cΦ. Let us rst con rm that B γ d ,η indeed has isolated informative eigenvalues down to the detectability threshold.

Exploiting the result of Claim . , the matrix B γ d ,η has an eigenvector correlated to the class labels equal to λ info,1 = λ

↓ |•| 1 (M T ( √ cΦγγ d , η)). First note that, by de nition, cΦγ 2 d = α c (T, η), while cΦγ 2 = α. For α > α c (T, η), then γ > γ d , and, consequently √ cΦγγ d > α c (T, η).
We then conclude that λ info,1 > 1. From Claim . , we further have that the radius of the bulk spectrum of B γ d ,η is equal to L γ d ,η = 1. As such, the informative eigenvalue λ info,1 of B γ d ,η exists as soon as α > α c (T, η).

We now exploit Claim . , that allows us to conclude that, since L γ d ,η = 1, then the smallest eigenvalue of the bulk (i.e., its left-edge) of H γ d ,η is asymptotically close to zero and all the eigenvectors associated to the isolated eigenvalues of B γ d ,η . Among these, there is at least one (the one associated to λ info,1 ) that is informative.

These results are visually supported by Figure . and summarized with the following claim.

Claim . (Spectral clustering with H

γ d ,η ). Let γ d = α c (T,η) √ cΦ
. As n → ∞, the following facts are veri ed with high probability • the complex eigenvalues forming the bulk spectrum of B λ d ,η are asymptotically bounded within a disk in the complex plane of radius L γ d ,η = 1:

• the smallest eigenvalues of the bulk spectrum of H λ d ,η tend to 0 + ;

• the number of isolated negative eigenvalues of H λ d ,η is equal to the number of real isolated eigenvalues of B λ d ,η greater than 1.

In particular, if α > α c (T, η), at least one of the isolated real eigenvalues of B λ d ,η larger than 1 and one of the negative isolated eigenvalues of H λ d ,η are informative in the sense that their associated eigenvectors are correlated to the vector of community labels.

Claim . indicates that, if α > α c (T, η), certainly there is one informative eigenvector (more precisely, mode of Figure . ) which is associated with one of the few isolated negative eigenvalues of H γ d ,η . Other informative eigenvectors (e.g. mode of Figure . ) may be associated to negative eigenvalues of H γ d ,η , but their existence is not guaranteed. The algorithmic advantage of Claim . with respect to Claim . is that, albeit the exact location of the informative eigenvectors is still unknown, for sure they are attached to negative eigenvalues of H γ d ,η . We empirically con rm that using all the eigenvectors associated with the isolated negative eigenvalues (instead of only the desired informative eigenvector with unknown location) to form a low dimensional vector embedding of the nodes is redundant but it does not severely compromise the performance of the nal k-means step. The choice ξ = γ d and h = η therefore almost immediately induces an explicit algorithm applicable to arbitrary networks and which, as later discussed in Section . , straightforwardly extends to graphs with k > 2 communities. Before detailing the algorithmic implementation of a SC algorithm based on H γ d ,η , let us formulate the following remark.

Remark . (Comparison with the static Bethe-Hessian of [SKZ ]). Recalling that the parameter γ behaves as the inverse of r appearing in the static Bethe-Hessian H r , it is easily shown that the parametrization ξ = γ d is a generalization to the dynamical case of the choice r = √ cΦ proposed in [SKZ ] for CD which is recovered for T = 1 (or equivalently for η = 0) for which we have that α c (T, η) = 1. While γ d allows one to devise a well de ned SC algorithm, like in the static case, the choice ξ = γ d is suboptimal and does not properly keep into account for the degree heterogeneity of the graph.

With these results at hand, we are now in position to consider how Claim . can be translated into a practical algorithm that will then be tested on synthetic graphs generated from the DDCSBM.

Algorithm . : Community detection in sparse dynamical graphs

Input : Adjacency matrices {A (t) } t=1,...,T of the undirected graphs {G t } t=1,...,T ; label persistence, η; number of clusters k. Output : Estimated label vector ˆ ∈ {1, . . . , k} nT begin Compute:

d (t) i ← ∑ n j=1 A (t) ij , c ← 1 nT ∑ T t=1 ∑ n i=1 d (t) i , Φ ← 1 nTc 2 ∑ T t=1 ∑ n i=1 d (t) i 2 ; Compute: α c (T, η) as in Claim . , γ d ← α c (T,η) √ cΦ ; Stack the κ eigenvectors of H γ d ,η with negative eigenvalues in the columns of X ∈ R nT×κ ; Normalize the rows of X i,• ← X i,• / X i,• ; for t = 1 : T do
Estimate the community labels { ˆ i t } i=1,...n using k-class k-means on the rows {X i t } i=1,...,n end return ˆ ∈ {1, . . . , k} nT . end . . .

We have previously summarized the main ideas behind a dynamical version of spectral clustering based on H γ d ,η . These form the core of Algorithm . . Yet, in order to devise a practical algorithm, applicable to a broad range of dynamical graphs, some aspects that go beyond the DDCSBM assumption should be taken into account.

So far, we dealt with k = 2 equal-size communities for which the DDCSBM threshold is well de ned. Real networks may of course have multiple classes with asymmetrical sizes. As in the static case, we argue that, under Assumption . , the left edge of the bulk spectrum of H γ d ,η is still asymptotically close to zero and that some of the eigenvectors associated with the isolated negative eigenvalues carry information for community reconstruction. The value k is, in practice, also likely unknown. This also does not a ect the idea of the algorithm which exploits all eigenvectors associated to the negative eigenvalues of H γ d ,η , without the need of knowing k. The very choice of k is only required by k-means in the last step of SC and may be performed using o -the-shelf k-means compliant tools, e.g., the silhouettes method [Rou ].

In passing, while α c (T, η) is well de ned regardless of k, for k > 2 classes of arbitrary size its value no longer corresponds to the position of a detectability threshold. A further important remark is that η is an input of Algorithm . . If unknown, as it would in general be, one may choose an arbitrary h ∈ [0, 1) and ξ = α c (T, h), to then perform spectral clustering on H ξ,h : the leftmost edge of the bulk spectrum of H ξ,h is asymptotically close to zero for all h and consequently Algorithm . can be used in the same form. However, for a mismatched h, the detectability threshold now occurs beyond the optimal α c (T, η). Close to the transition, this mismatch would give rise to fewer informative isolated negative eigenvalues than expected, resulting in a poor quality label assignment. As a workaround, one may browse through a discrete set of values for h and extract the h maximizing some quality measure, such as the resulting clustering modularity.

We further underline that in step 5, Algorithm . performs the hypersphere normalization step. Although not fully justi ed from a theoretical standpoint, the empirical evidence showed in Part II that this step helps to signi cantly improve the e ciency of the k-means step, correcting to some extent the deleterious contribution of an arbitrary degree distribution.

From a computational point of view, the bottleneck of Algorithm . is to compute the embedding X. The number of negative eigenvalues κ is not a priori known and only suspected to be in the interval {k, . . . , kT}. The strategy based on polynomial approximation and random projections detailed in Section . . is here particularly adapted, allows us to estimate

The number κ of negative eigenvalues to be computed, generally depends on T κ with O(cnT) operations and to compute X with O(ncTk 2 ) operations in the best case scenario and O(ncT 4 k 3 ) operation in the worst case. To give an order of magnitude, a simulation of Algorithm . for n = 10 5 , T = 5 (resp., n = 5 000, T = 100), k = 2, c = 6, η = 0.5, Φ = 1.6, α = 2α c (T, η) takes on average approximately 1 minute (resp., 40 minutes) with its CoDeBetHe.jl implementation. , BP [Gha+ ], the dynamic adjacency matrix of [KV ] (dyn A), the dynamical non-backtracking of [Gha+ ] (dyn B and dyn B opt), the static Bethe-Hessian of [DCT ] (static BH) and the dynamical Louvain algorithm of [Muc+ ] (dyn Louvain); n = 5 000, T = 4, c = 6, η = 0.7, θ = 1 n ; averaged over 20 samples (3 for BP). For all plots, k = 2. Right: overlap comparison of Algorithm . , the static Bethe-Hessian of [DCT a] (StaticBH) and the reconstruction obtained using the eigenvector with zero eigenvalue of H γ,η (opt. BHdyn). For this simulation n = 25 000, k = 2, θ = 1 n , c = 6, T = 4, η = 0.9.

Averages are taken over 10 samples.

. . versus the adjacency averaging method of [KV ] (which we recall assumes

η = 1 -o n (1)
). The overlap is only considered at t = T so to compare Algorithm . on even grounds with [KV ] which only outputs one partition (rather than one for every t). The theoretical detectability threshold line α = α c (T, η) visually con rms the ability of Algorithm . to assign Algorithm . performs DCD for any value of η non trivial class labels as soon as theoretically possible, as opposed to the method of [KV ] which severely fails at small values of η.

The left plot of Figure . then compares the average overlap performance of Algorithm . against competing methods, for varying detection complexities α/α c (T, η). Algorithm . is outperformed only by the BP algorithm, but has a sensibly reduced computational cost. As we underlined already with Remark . , for the non-backtracking method of [Gha+ ] ("dyn B"), the authors suggest to use (as we did here) the eigenvector associated to the second largest eigenvalue of B γ,η , which, as H γ,η , may also have informative and uninformative eigenvalues in reversed order. The curve "dyn B opt" shows the performance obtained using all the isolated eigenvectors of B γ,η and it con rms, in agreement with Claim . , that B γ,η can indeed

The codes used to obtain the BP performance displayed in Figure . are courtesy of Amir Ghasemian, author of [Gha+ ]. : Modularity as a function of time for Algorithm . for η = 0.55, the dynamic adjacency matrix of [KV ] (dyn A), the dynamic Louvain algorithm [Muc+ ] (dyn Louvain) and the static Bethe-Hessian of [DCT ] (static BH). The graph {G t } t=1,...,T is obtained from the Primary school network [GBC ; Ste+ ] dataset, as in Section . . . For Algorithm . , [DCT ] and [KV ], k = 10 is imposed. make non-trivial community reconstruction for all α > α c (T, η). Note that, as in the static case [Krz+ ; SKZ ], B γ,η is outperformed by H γ d ,η which, additionally, is symmetric and smaller in size, is well de ned regardless of γ and is, therefore, a more suitable candidate for CD.

Interestingly, for large values of α, Algorithm . is slightly outperformed by the static Bethe-Hessian of [DCT ], independently run at each time-step. As discussed at the end of Section . , the choice ξ = γ d is sub-optimal compared to the optimal (but out-of-reach in practice) choice ξ = γ, the di erence becoming more visible as α increases away from α c . Supposing one has access to an oracle for γ, running Algorithm . on H γ,η outperforms systematically the static Bethe-Hessian, as shown in the right plot of Figure . . From a dynamical viewpoint, also, the large α regime is of least importance as a static algorithm can, alone, output a perfect reconstruction.

For completeness, Appendix C. provides further numerical performance comparison tests for di erent values of η, Φ, for k > 2, for larger values of n and T, and for graphs with clusters of di erent average sizes.

. . This section shows the results of our experiments on the Primary school network [GBC ; Ste+ ] of the SocioPatterns project. The dataset contains a temporal series of contacts between children and teachers of ten classes of a primary school. For each time 1 ≤ t ≤ T, G t is obtained considering all interactions from time t to time t + 15 min, starting from t 1 = : am until t T = pm for T = 34. Figure . compares the modularity as a function of time for di erent clustering techniques. We empirically observe that, for this dataset, multiple values of η give similar results: this is not surprising because the clusters are here well delineated and we are in the (less interesting) easy detection regime. The value η = 0.55 is considered as an input of Algorithm . , because it approximately matches the value of η estimated from the inferred label vector ˆ (see Equation ( . )).

Figure . shows that Algorithm . performs better than [KV ; Muc+ ] at all times, with a drastic gain during the lunch break, in which the community structure is harder to delineate. As compared to the static Bethe-Hessian, Algorithm . is slightly outperformed only on some times during the lunch break, while for other times it bene ts from the label correlation. De ning a unique, time independent η certainly hampers the performance on this speci c dataset in which a large η is expected during the lesson times, while a small η may be more appropriate during the lunch break.

. In this chapter we tailored a novel spectral algorithm for CD on dynamical graphs. The proposed Algorithm . is capable of reconstructing communities as soon as theoretically possible, thereby largely outperforming state-ofthe-art competing spectral approaches (especially when classes have a shortterm persistence) while only marginally under-performing the optimal but more intensive BP algorithm.

A delicate feature of Algorithm . concerns the estimation of the classpersistence parameter η, if not available. We hinted in Section . at a greedy line-search solution which is however computationally ine cient and lacks of a sound theoretical support. This needs be addressed for Algorithm . to be more self-contained and applicable to real networks.

Beyond this technical detail, the present analysis only scratches the surface of dynamical community detection: the problem in itself is vast and many degrees of freedom have not been here accounted for. The label persistence η and community strength matrix C (and thus the parameter γ in a symmetric two-class setting) are likely to evolve with time as well. We empirically observed that Algorithm . naturally extends to this setting, each temporal block of the matrix H (•,•) now using its corresponding γ (t) d and η t . Yet, while Algorithm . seems resilient to a more advanced dynamical framework, it requires a much more involved theoretical description.

Nonetheless, this chapter showed that the results of Part II can be generalized to more involved settings and that the steps needed to devise an e cient SC algorithms are well delineated and can be retraced for more general clustering problems than CD on static, undirected graphs.

Spectral clustering (SC) techniques can be adopted as a practical correlation clustering algorithm, taking high-dimensional feature vectors [BBC ] as an input. This is done by rst de ning a kernel matrix representation that encodes the pairwise similarity between the elements of the input dataset and then extracting few relevant eigenvectors that provide a small dimensional representation of the initial dataset.

This problem has concrete applications to image, sound, or sentence classi cation [Lan+ ] but, as we mentioned in the Introduction chapter of this manuscript, one of the main pitfalls of SC to categorize high dimensional vectors is its computational complexity and sampling techniques are often adopted to decrease the computational cost of SC [TL ].

In this chapter we consider a simple generative model to study the correlation clustering problem on kernel matrices with very few non-zero entries, allowing one to drastically reduce the computational complexity of SC. We then show the explicit relation between the considered generative model and the random bond Ising model (RBIM) in statistical physics. Our main result is a conjecture relating the smallest eigenvalue of the Bethe-Hessian matrix and the Nishimori temperature [Nis ] appearing in the physics of spin glasses. This relation allows us to devise an e ective algorithm for the sparsi ed (hence cost e cient) version of SC. . . .

Let {z i } i=1,...,n be an n-vector dataset with z i ∈ R d feat . These vectors represent discriminating features of some two-class data (say images) to be clustered in a fully unsupervised manner. In typical modern machine learning, d feat is of the order of a few thousands for images and a few hundreds for natural language text representations, and it is not rare to try and classify up to millions of data vectors z i . The most elementary way to classify the vectors {z i } i=1,...,n is simply to directly run k-means algorithm in the d featdimensional feature space. K-means is however known to fail for large d feat [KKZ ] due to the fact that distances tend to concentrate in high dimensionality, a phenomenon that goes under the name of curse of dimensionality [Bel ]. The top line of Figure . provides a visualization of this phenomenon. To obtain this plot, we consider a label vector ∈ {1, 2} n and draw n Gaussian vectors {z i } i=1,...,n with covariance matrix I d feat and mean µ i . On the left we show a scatter plot between two arbitrary components of the vectors {z i } i=1,...,n , denoting with a di erent colour code the entries related to either class. This plot clearly evidences the e ect of the curse of dimensionality. Furthermore, letting Z ∈ R n×d feat be the matrix containing in its rows the vectors {z i } i=1,...,n , on the right of the rst row we plot the covariance matrix M Z = 1 d feat ZZ T which just appears as a white square because all its entries are approximately equal.

A classical workaround is to perform a dimensionality reduction of the input vectors, embedding them in a lower dimensional space. A good embedding has the role to map "similar" vectors in high dimensions to points Pairwise measurements can be encoded with weighted edges of a graph that are close in the embedded space. The analogy with SC for CD is clear: in that context, "similar" nodes correspond to nodes in the same community.

The most popular technique exploits a spectral approach [Ham+ ; Sau+ ; Gho ]: one starts by de ning a kernel matrix K({z}) ∈ R n×n , the entry K ij ({z}) of which evaluates some a nity metric between z i and z j ; a collection of eigenvectors x 1 , . . . , x k for k typically equal (or close) to the number of classes are extracted and stacked in the columns of the matrix X ∈ R n×k ; the rows of X form the small dimensional embedding on which k-means can be safely run. A popular a nity function is merely the correlation K({z}) = 1 d feat ZZ T , which is the choice adopted for kernel principal

Note the the entry ij of the covariance matrix is

(M Z ) ij = 1 d feat z T i z j = 1 2d feat [ z i 2 + z i 2 -z i -z j 2 ] = µ + 1 2d feat z i -z j 2 + o d feat (1)
and is therefore directly related to the euclidean distance between the points in R d feat .

:

component analysis [Pea ], but other choices exist, such as the heat kernel K ij ({z}) = exp(z iz j 2 /2ν 2 ) for some ν > 0, for instance. A visual example of how dimensionality reduction allows one to cluster high dimensional vectors is shown in the bottom row of Figure . . For large dimensional datasets, the computational complexity of building K({z}) may become prohibitive: for d feat , n beyond a few thousands, the O(d feat n 2 ) cost of building K({z}) added to the (at least) O(n 2 ) cost of computing the leading eigenvectors, makes SC hardly achievable on a modern home computer. To drastically decrease the computational complexity one may proceed to a two-level sparsi cation as recently proposed in [Zar+ ; CCB ]: by randomly discarding elements of the d feat -dimensional features Sparse kernel matrices improve the computational complexity of SC z i and by randomly dropping a number of evaluations of the correlations K ij ({z}). This operation of course impedes the clustering performance, but, as surprisingly proved in [Zar+ ; CCB ] under a "still rather dense graph" regime, the performance loss is negligible for a wide range of sparsity levels. To this end, let S ∈ {0, 1} n×p and H ∈ {0, 1} n×n (symmetric) be Bernoulli masks with parameters √ κ/d feat and c/n, respectively. The resulting sparsi ed kernel matrix then becomes J = K({ z}) • H, where zi,l = z i S i,l .

( . )

i.e., each entry of each of the feature vectors z i is kept only with probability √ κ/d feat , while each measurement K ij ({ z}) is only performed with probability c/n. The computational complexity to build J is thus scaled down to O(κcn). As just mentioned, in [Zar+ ; CCB ] the authors still have to consider (for mathematical convenience) a rather dense regime in which c grows su ciently fast with n and in practice do not solve the computational complexity problem.

Our goal is to devise a SC algorithm that is capable to operate in the sparse regime (c = O n (1)), with computational complexity linear in n. To do so, we observe that the matrix J of Equation ( . ) can be interpreted as a weighted adjacency matrix of a graph. In fact, every ij so that Jij = 0 corresponds to a missing measurement or, in graph language, to a missing edge. On the opposite, when Jij = 0, then it represents the weight of the edge (ij). This allows us to immediately rephrase the problem of sparsi cation of kernel matrices in the graph-theoretic language adopted in the previous chapters.

Following the same methodological pattern adopted so far, we now propose a generative model for J on which our theoretical results will be based. First, we introduce an intermediate matrix J, generated as follows.

De nition . (Generative model of J). Let G(V, E ) be a realization of a Erdős-Rényi graph of size n (De nition . ) with expected average degree c. Further let J ∈ R n×n be a weighted adjacency matrix on G(V, E ) and distributed according to the following generative model. For all edges (ij) ∈ E

The choice of c is not a coincidence: H will enforce an average number of measurements per node (in graph-theory language, a degree) of c to the resulting graph.

the entries J ij are generated independently (up to symmetry), for some β N > 0, according to the distribution P P(x) = p 0 (|x|) e β N x , ( . )

where p 0 (•) is an arbitrary non-negative function satisfying the normalization condition

∞ -∞ dx p 0 (|x|)e β N x = 1. If (ij) / ∈ E , then J ij = 0.
J J Note that, although not very customary when dealing with weighted graphs, according to De nition . , J ij may be negative. We called J an "intermediate" matrix because it is helpful to de ne a generative model for J. Suppose that the vector ∈ {1, 2} n assigns a label to each node in V. De ne σ ∈ {±1} n so that σ i = 1 if i = 1 and σ i = -1 otherwise. Let J be generated as per De nition . , then the matrix J is obtained as follows

J = J • σσ T .
( . )

We now detail the main ingredients motivating the considered generative model for the matrix J.

• The expectation of J: from De nition . , combined with ( . ), the expectation of a non-zero entry of J reads

E[ Jij ] = E[J ij ]σ i σ j . The expectation E[J ij ]
is the same for all (ij) ∈ E and it is positive, in fact

E[x] = ∞ -∞ dx p 0 (|x|)e β N x x = ∞ 0 dx p 0 (|x|) 2sh(β N x) x >0
, consequently, we immediately see that sign of E[ Jij ] is positive for i, j in the same class and it is negative otherwise. This is a reasonable requirement that the matrix J de ned in Equation ( . ) should satisfy and it implies that, on average, K ij ({z}) is larger when i = j . This is readily veri ed in the Gaussian example considered in Figure . for

µ 1 = -µ 2 in which, letting K ij ({z}) = 1 d feat z T i z j , then E[ Jij ] ∝ σ i σ j .
It must be noted that the matrix J of Equation ( . ) must satisfy J1 n = 0 n to be reasonably approximated by J as in Equation ( . ). This can however be easily obtained shifting all the input vectors { zi } i=1,...,n = {z if } i=1,...,n , where f ∈ R d feat is an i-independent vector. Further note that, since the function p 0 (|x|) needs not to be speci ed, the considered generative model includes a large number of random matrices satisfying the condition E[ Jij ] = σ i σ j .

• The noise in J: the parameter β N (that is the main focus of the remainder of the chapter) controls the noise in the realization of J. In fact, considering the limit β N → 0, we get P(x) → p 0 (|x|) and the sign of J ij (and consequently of Jij ) is essentially random. On the opposite, when β N → ∞, J ij > 0 with high probability and consequently sign( Jij ) = σ i σ j . Tuning β N between these two extremes, one determines the likelihood that sign(J ij ) = σ i σ j : the smaller this likelihood is, the more relevant is the contribution of the noise, allowing the considered generative model to recreate hard detection tasks.

• The independence hypothesis: a major concern comparing the matrix J de ned as per Equation ( . ) and ( .) is certainly the independence of the entries assumed in De nition . . In fact, by construction, the entries K ij ({z}), K ik ({z}) and K jk ({z}) are far from being independent. Independence however asymptotically holds between the non-zero entries of J for a su cient level of sparsi cation as a consequence of the absence of short cycles in the underlying Erdős-Rényi graph. In the example we gave, if (ij), (ik) ∈ E , then P[(jk) ∈ E ] = o n (1), hence the dependence of K jk ({z}) on K ij ({z}), K ik ({z}) can be neglected.

For these reasons, we claim that devising a SC algorithm to infer the vector σ from a realization of J allows one to cluster the input vectors {z i } i=1,...,n into their genuine classes. We now formalize the problem from a Bayesian perspective, describing its clear relation with statistical physics.

. .

Let G(V, E ) be the realization of an Erdős-Rényi graph whose nodes are divided in two non-overlapping classes, labelled via the vector σ ∈ {±1} n and let J ∈ R n×n be the matrix generated according to Equation ( . ). Then the following relation holds:

P( J|σ) = ∏ (ij)∈E p 0 (| Jij |)e β N Jij σ i σ j .
In order to infer the vector σ from a realization of J we may express the posterior probability distribution as follows:

P(σ| J) = P( J|σ)P(σ)

P( J) = 1 Z β N , J exp    ∑ (ij)∈E β N Jij σ i σ j    . ( . )
This is the fundamental relation of Bayes optimal inference. Note that computing the marginals of P(σ| J) is equivalent to determine the magnetization of an Ising model on J at β N (see Chapter ), later referred to as Nishimori temperature [Nis ].

The relation between β N and the Ising model has a long history [Nis ] in the physics of disordered systems [BY ] that raised the scienti c interest even before its relation with Bayes inference was unveiled [Iba ]. To explain the importance of the Nishimori temperature in statistical physics, let us introduce the random bond Ising model (RBIM). Let H J (s) be the Ising Hamiltonian (Equation ( . )) associated to a weighted graph J generated according to De nition . and denote with µ(s) the Boltzmann distribution:

The graph G(V, E ) is thus xed and not a random variable.

H J (s) = -∑ (ij)∈E J ij s i s j , ( . ) µ β (s) = 1 Z β,J e -βH J (s) . ( . )
Equations ( . , . ) de ne the RBIM in which, we recall, β N is a parameter of the generative model of J. Nishimori noticed in [Nis ] that at β = β N , i.e., when the temperature of the system coincides with the Nishimori temperature, the exact expression of E H J (s) β can be computed with elementary mathematical tools, where • β denotes the averaging over the Boltzmann distribution ( . ) while E[•] is the averaging over the distribution of Equation ( . ). It has also been shown [NS ; ZK , for instance] that the RBIM at the Nishimori temperature is either in the ferromagnetic con guration (in which s i β > 0 for all i) or in the paramagnetic con guration (for which s i β = 0 for all i). In particular, the system is never in the spin-glass phase under which local order of s appears despite there being no global magnetization. These relevant properties drew research attention to this particular temperature [Geo+ ; GRL ; TPV ] since [Nis ].

Relating the Bayes optimal solution ( . ) with the RBIM ( . ) is trivially done introducing a gauge transformation. If fact, let si = s i σ i and recall that Jij = J ij σ i σ j , then

H J (s) = -∑ (ij)∈E J ij s i s j = -∑ (ij)∈E Jij σ i σ j si σ i sj σ j = -∑ (ij)∈E Jij si sj ,
where we exploited σ 2 i = 1 for all i. This allows one to create an explicit mapping between the posterior distribution of σ and the distribution of s and to use the powerful tools and theoretical results borrowed from statistical physics to devise an e cient algorithm to perform clustering. In the Bayes optimal inference, however, the value of β N must be known, an assumption that for unsupervised clustering tasks is often unrealistic. Earlier works have resorted to studying the problem of mismatched inference, i.e. inference performed for β = β N [ZK ].

. . :

Our main result consists in going beyond mismatched inference by providing an e cient estimate of the Nishimori temperature, β N . To this end, we rst draw an explicit relation between β N and the smallest eigenvalue of the (weighted) Bethe-Hessian matrix; this relation holds under the previously introduced setting, so in particular for a matrix J supported over a (possibly sparse) Erdős-Rényi graph G(V, E ). Besides, we observe and argue that, although the Bethe approximation is particularly adapted to sparse (locally tree-like) graphs G(V, E ), the Nishimori-Bethe relation holds for any degree of sparsity (that is, even when G(V, E ) does not behave locally as a tree).

The main consequences of the Nishimori-Bethe relation, and our main contributions, consist in • the design of a new e cient spectral algorithm which estimates β N with asymptotically perfect accuracy; the algorithm is based on an iterative fast search of a well-parametrized Bethe-Hessian matrix with the smallest amplitude eigenvalue close to zero;

• a new spectral algorithm to approximately solve the Bayesian node classi cation inference problem of Equation ( . ), which outperforms commonly deployed state-of-the-art alternatives. We in particular claim that this spectral algorithm is capable of performing non trivial inference as soon as the Bayes optimal solution can;

• although we claim that these algorithms are still valid under dense graphs G(V, E ), they are speci cally adapted to the sparse regime where c = O n (1); this practically allows for small computational and memory storage costs when performing SC on large datasets; we speci cally support this fact by a concrete application to the classication of 40 000 high resolution images using our proposed sparse but extremely e cient spectral algorithm.

Given the trivial relation between the Bayes optimal inference problem and studying the magnetization of the RBIM at β = β N , we chose, for simplicity, to work with the matrix J rather than J. Several relations will however be made to relate the physics and inference results.

Let us now proceed with some relevant properties of the Nishimori temperature in the RBIM the will be of utmost importance in the remainder. . . . Consider a realization of a Erdős-Rényi graph G(V, E ) with expected average degree c. Let J ∈ R n×n be a weighted adjacency matrix associated to G(V, E ) and generated as per De nition . . We want to study the phase diagram, i.e. the behavior of s β , for s a random variable distributed according to Equations ( . , . ). Figure . displays this phase diagram for the ±J model which corresponds to the particular choice of p 0 (|x|) leading to for β N = 0 to 1 for β N → ∞. The y axis represents T, the inverse of β. The dashed green line is the inverse of β F , the dash dotted blue line is the inverse of β SG and the solid red line is the inverse of β N . but qualitatively similar diagram can be obtained for di erent choices of p 0 (|x|), like the Edwards-Anderson model, for instance [EA ] in which

P(x) = pδ(x -J 0 ) + (1 -p)δ(x + J 0 ), for p ∈ [1/2, 1], J 0 ∈ R + ,
P(x) = 1 √ 2πν 2 exp - (x -J 0 ) 2 2ν 2 .
To best understand the phase diagram in Figure . , rst consider the role played by the two parameters β and β N . For increasing values of β N , there is a larger probability for each edge J ij to carry a positive weight and the minimum of H J (s) is achieved for s ≈ 1 n . For small values of β N , instead, multiple local minima appear. Concerning β, instead, for small values, the Boltzmann distribution (Equation ( . )) tends towards a uniform distribution, while, for large values, the con gurations with small energy H J (s) have a larger probability. Consequently, for large β and β N the average con guration of s tends to align towards 1 n : this corresponds to the ferromagnetic con guration. Conversely, for small values of β N , several edges carry a negative weight, introducing frustration in the system that is found in the spin-glass phase, for which local order of the spins may be observed ( 1 n ∑ i s i 2 β = 0), but globally the magnetization is null ( 1 n ∑ i s i β = 0). Finally, at large values of β, the system is in the paramagnetic phase, for which the spins are randomly aligned and s β = 0 n .

In the particular case where G(V, E ) is a sparse Erdős-Rényi random graph, the Bethe approximation ( . . ) predicts the asymptotically exact position of the transitions between the three phases: the paramagnetic-ferromagnetic transition occurs at β = β F and the paramagnetic-spin glass transition occurs at β = β SG , also known as the de Almeida-Thouless transition [Tho ]. We recall from Section . that the values of β F (resp. β SG ) is determined imposing the leading eigenvalue (resp. the radius of the bulk) of : the weighted non-backtracking with weights equal to th(βJ e ) equals one. Exploiting Theorem . [SM ], this leads to:

c • E[th(β F J ij )] := 1 ( . ) c • E[th 2 (β SG J ij )] := 1. ( . )
Given these premises, we now discuss some relevant properties valid on the Nishimori line, i.e. when β = β N .

. .

First of all, let us introduce the quenched internal energy density, de ned as u(β

) := 1 n E[ H J (s) β ],
where we recall that • β denotes an average taken over the Boltzmann distribution ( . ) and E[•] is the average over the distribution of Equation ( . ). It was shown in [Nis ] that u(β N ) takes a particularly simple expression:

Exact expression of the quenched internal energy at the Nishimori temperature

u(β N ) = 1 n E[ H J (s) β N ] = - 1 n ∑ (ij)∈E E J ij s i s j β N = - 1 n ∑ (ij)∈E E[J ij th(β N J ij )].
The rst two equalities are true by de nition. The elegance of the result of [Nis ] lies in the last relation that identi es -inside the expectation E[•] -the term s i s j β N with th(β N J ij ). We recall from Section . . , for β su ciently small, the system is in the paramagnetic phase s β = 0 and, under the Bethe approximation, the relation s i s j β = th(βJ ij ) is veri ed for any underlying β N . This informally introduces a relation between the Bethe free energy at the paramagnetic point and the Nishimori temperature, which is at the centre of Claim . . We introduce the following property of the probability distribution of Equation ( . ). This relation will be of fundamental use in the following and, in passing, allows us to rewrite u(β N ) as in [Nis ].

Property . . Let f (x) be an arbitrary odd function. Then

E[ f (x) • th(β N x)] = E[ f (x)].
The proof is easily obtained by straightforward calculation. As a consequence of Property . , the quenched internal energy density at the Nishimori temperature takes the simple expression:

u(β N ) = - 1 n ∑ (ij)∈E E[J ij ] = - d 2 • E[J ij ],
where d denotes the average node degree in the graph G(V, E ).

.

β N
Secondly, we recall a well celebrated property of the Nishimori temperature, which states the absence of replica symmetry breaking on the Nishimori line [NS ; ZK ] or, equivalently, that the RBIM at the Nishimori temperature is never in the spin glass phase. This result can be visually understood in Figure . by noticing that the Nishimori temperature is either in the

E[th 2 (β N J)] = E[th(β N J)]
paramagnetic or ferromagnetic phase. Moreover, exploiting Property . and the de nitions of β F , β SG in Equations ( . , . ), on an Erdős-Rényi graph one nds that β SG = β N ⇐⇒ β F = β N . Consequently, there exists a tricritical point where

β F = β SG = β N .
Recalling the connection with statistical inference problems, such as inferring σ in Equation ( . ), rst note that β N is the Bayes optimal inference temperature in the sense that there exists no other β that can asymptotically achieve better inference performance and, therefore, if inference cannot be performed at β = β N , then it is theoretically infeasible. This occurs when the marginals of Equation ( . ) asymptotically give equal probabilities for each σ i to take either values ±1. In terms of the phase diagram, this corresponds to being in the paramagnetic phase, so that β N < β F . In order for non-trivial reconstruction to be possible, the condition β N < β SG < β F Detectability threshold must be imposed and it corresponds to the detectability threshold of this problem [Saa+ ]. When the condition is met, the system is in the informative con guration in which each spin gets oriented towards its planted value σ i . This being said, replacing (or, erroneously estimating) β N by β = β N in Equation ( . ), it may occur that, even though inference is theoretically possible (as β N < β SG < β F ), the estimated labels σ for the mismatched β are not aligned with the ground truth σ. This never happens at β = β N for which inference is achieved as soon as theoretically possible.

With this short introduction on the Nishimori temperature at hand, we now present our main result which relates β N to the spectrum of the nonbacktracking and Bethe-Hessian matrices of the underlying graph G(V, E ).

(ij) ∈ E a weight ω ij = ω ji ∈ (-1, 1) is assigned indepen- dently at random. Further assume that E[ω 2 ij ]/E[ω ij ] ≥ 1 and E[ω 2 ij ]/E 2 [ω ij ] < c.
Then, with high probability, the spectrum of B ω can be described as follows:

• there exist only two real eigenvalues in the spectrum of B ω with modulus greater or equal to one:

λ 1 = cE[ω] + o n (c) , λ -1 = E[ω 2 ] E[ω] + o n (1). ( . )
The eigenvalue λ 1 is the largest in modulus in the spectrum of B ω ;

• all eigenvalues with non-zero imaginary part have a modulus bounded by

L ω = cE[ω 2 ] + o n ( √ c).
Note that Claim . does not make the assumption that c → ∞ as n → ∞, nor that c = O n (1). Extensive simulations indeed concur in suggesting that the claim holds in both dense and sparse graph regimes. The claim is thus stated for any average degree, so long that the underlying graph is of the Erdős-Rényi type. In detail, the technical condition E[ω 2 ij ]/E 2 [ω ij ] < c is set to enforce that the leading eigenvalue λ 1 is greater than the radius of the bulk spectrum (hence that it is isolated) and that λ -1 is smaller than the radius of the bulk: a transition occurs at E[ω 2 ij ]/E 2 [ω ij ] = c where both eigenvalues coincide: λ 1 = λ -1 . This inequality condition will thus ensure, when it comes to statistical inference, that non-trivial σ ∈ {±1} n con gurations can be theoretically recovered (i.e., that the inference problem is feasible). As a practical support to Claim . , Figure . displays the spectrum of the matrix B ω in both moderately dense (c ∼ log 2 (n)) and sparse (c = O n (1)) regimes.

The fundamental corollary of Claim . is that, in the case of present interest where ω e = th(βJ e ), from Equation ( . ), the inner eigenvalue

λ -1 of B ω is equal to λ -1 = E[th 2 (βJ)] E[th(βJ)] + o n (1).
Exploiting Property . , it follows immediately that, at β = β N ,

At β = β N the inner isolated eigenvalue of B equals 1 λ -1 = β=β N 1 + o n (1).
Tuning the value of β until λ -1 = 1 thus provides a method to estimate β N . The question on how to e ciently exploit this essential remark from an algorithmic standpoint will be further discussed in Section . . . Before pushing further our main line of deductions, let us provide some arguments in support of Claim . , treating separately the dense and sparse regimes.

. .

Dense graphs

We rst consider a dense graph regime, i.e., when the average degree c goes to in nity faster than log(n). This argument is inspired from the proof provided in [CZ ] for unweighted dense graphs with a community structure. The proof of [CZ ] can be straightforwardly adapted to the binary case in which ω e ∈ {±ω}, but does not unfold so directly for ω e ∈ R.

We recall that the main advantage of the dense regime follows from the fact that the degree distribution of G(V, E ) is almost regular and the Erdős-Rényi graph is close to a c-regular graph, the analysis of which is easier to handle. This makes it possible to relate the eigenvalues of B ω to those of W ∈ R n×n , de ned as W ij = ω ij if (ij) ∈ E and zero otherwise. The idea is to create a sequence of matrices M(g) ∈ R 2n×2n (one for each eigenvector g of B ω ), in the spirit of a proof proposed by Bass of the celebrated Ihara-Bass formula [HST ], and to show that all the eigenvalues of M(g) can be On dense Erdős-Rényi graphs

D -cI n = o n (c)
approximated, in the large n limit, by the eigenvalues of a common matrix M 0 independent of g, so long that g is an eigenvector corresponding to an eigenvalue λ of B ω for which |λ| ≥ 1. It is the precise study of the spectrum of the limiting M 0 which induces the results of Claim . . More speci cally, let g ∈ C |E d | be an eigenvector of B ω with eigenvalue λ, satisfying |λ| ≥ 1 and let ω ∈ R |E d | be the vector containing the weights of the edges of G(V, E ) (and recall that ω ij = ω ji ). Then de ne the vectors

ψ(g), ψ(g) ∈ C n as ψ i (g) = ∑ j∈∂i ω ij g ij , ψi (g) = ∑ j∈∂i ω 2 ij g ji ( . )
and F(g) ∈ C n×n be any matrix satisfying

F(g)ψ(g) i = ∑ j∈∂i ω 3 ij g ij . ( . )
We now wish to relate the quantities ψ(g), ψ(g), F(g) to the eigenvalues of B ω . In particular,

λψ i (g) = ψ i (B ω g) = ∑ j∈∂i ω ij ∑ (k ) δ jk (1 -δ i )ω kl g k = ∑ j∈∂i ω ij ∑ l∈∂j ω j g j -ω ji g ji = [Wψ(g)] i -ψi (g)
and, similarly,

λ ψi (g) = ψi (B ω g) = ∑ j∈∂i ω 2 ij ∑ (k ) δ ik (1 -δ j )ω k g k = ∑ j∈∂i ω 2 ij ∑ ∈∂i ω i g i -ω ij g ij = [D W ψ(g)] i -[F(g)ψ(g)] i ,
where D W ∈ R n×n is the diagonal matrix with [D W ] ii = ∑ j∈∂i ω 2 ij . Thus, the eigenvalue λ is also an eigenvalue of the matrix

M(g) = W -I n D W -F(g) 0 . ( . )
The main di culty of the analysis is of course introduced by the matrix F(g) which is di erent for each eigenvector of B ω associated to |λ| > 1. In . β N the binary case in which W ij ∈ {±ω} for all (ij) ∈ E , this term simpli es: combining Equations ( . ) and ( . ), we get F(g)ψ = ω 2 ψ and F(g) thus simpli es for all g into F(g) = ω 2 I n ; this allows for a straightforward adaptation of the proof of [CZ ]. The non-binary case is, however, more involved, but the term D W -F(g) ψ is still dominated by D W :

F is negligible with respect to D W [F(g)ψ(g)] i ψ i (g) = ∑ j∈∂i ω 3 ij g ij ∑ j∈∂i ω ij g ij = o n (c).
For the last equality, we exploited the fact that ω ij and ω 3 ij are both bounded in (-1, 1) and have the same sign. Consequently, F(g) can be regarded as a small perturbation of D W . Further exploiting the concentration of the degrees, one may thus write

D W -F(g) -cE[ω 2 ]I n = o n (c).
The eigenvalues of M(g) can therefore be approximated by those of

M 0 = W -I n cE[ω 2 ]I n 0 . ( . )
The spectrum of M 0 is trivially related to the spectrum of W. Letting {λ i (W)} i=1,...,n be the eigenvalues of W and {λ 0,i } i=±1,...,±n those of M 0 , by the block determinant formula [Sil , Section ], it comes that

λ 0,±i = λ i (W) ± λ i (W) 2 -4cE[ω 2 ] 2 .
In particular, it unfolds that

λ 2 i (W) ≥ 4cE[ω 2 ] =⇒ λ 0,-i = cE[ω 2 ] λ 0,i ≡ L 2 ω λ 0,i ( .
)

λ 2 i (W) < 4cE[ω 2 ] =⇒ |λ 0,±i | = cE[ω 2 ] ≡ L ω . ( . )
Applying successively Wigner's semi-circle theorem [Wig ] and Bauer-Fike's theorem [BF ], we thus have that

λ 1 (W) = cE[ω] + o n ( √ c) ( . ) |λ i (W)| i≥2 ≤ cE[ω 2 ] + o n ( √ c). ( . )
Combining ( . , . ) with ( . , . ), along with the fact that the eigenvalues {λ 0,±i } i=1,...,n are a close approximation to the eigenvalues of B ω with modulus greater than one, we obtain the formulation of Claim . . .) with g, one of the eigenvectors of B ω attached to a complex eigenvalue. The expression of F(g) used in this simulation can be found in Appendix A of [DCT b] Right: Spectrum of M 0 , de ned in Equation ( . ). The graph considered is the same for the two matrices, with n = 1 500, c = log 2 (n). The matrix W = th(βJ), with β = 1 and the entries J ij are i.i.d. normal variable with J 0 = 1 and ν = 3. The red dotted line is at cE[th(βJ)], the green dash-dotted line is the position of E[th 2 (βJ)]/E[th(βJ)], while the orange solid line is the circle in the complex plane of radius cE[th 2 (βJ)].

This technical argument provides important intuitions on the spectrum of B ω : (i) the leading eigenvalue of B ω (the largest in modulus) is determined by the expectation of the entries of W; (ii) the radius of the bulk of B ω is determined by the expectation of the squared entries of W; (iii) all eigenvalues of B ω with modulus greater than one come in pairs: they are complex conjugates if their imaginary part is non-zero or harmonic conjugate if they are real. This last observation justi es the existence of a real isolated eigenvalue inside the bulk of B ω .

As a downside, the setting considered in this section is, somehow, too simplistic. The analysis allowed us to neglect the term F(g), which plays the role of the Onsager reaction term [MPV ] which does not appear in the naïve mean-eld approximation but plays a crucial role in the Bethe approximation. The fact that F(g) can be neglected thus indicates that the regime under consideration is somehow too simple, the spectral behavior of B ω being fully determined by W. Consequently, we next discuss the far more interesting sparse regime in which the reaction term plays a fundamental role. In the sparse regime, the structure of the spectrum of the matrix B ω is essentially preserved, as well as the fact that all its eigenvalues all come in real harmonic or complex conjugate pairs.

Sparse graphs

We recall that the Bethe approximation is asymptotically exact on locally tree-like graphs, such as sparse Erdős-Rényi random graphs. In this case, the spectrum of W is no longer formed by an isolated, real eigenvalue and a . β N bulk of eigenvalues close to each other that follow the semi-circle law, as it happens in the dense regime discussed in the previous paragraph.

The non-backtracking matrix B ω , instead, essentially preserves the same spectral structure as in the dense regime, in which the bulk eigenvalues are bounded by a circle in the complex plane, as shown in Figure . and conrmed by Theorem . of [SM ] in which, however, the authors do not mention the existence of inner real eigenvalues in the spectrum of B ω . Yet, the position of the leading eigenvalue of B ω and the radius of its bulk spectrum are the same as in the dense graph case. We then conjecture, supported by extensive simulations, that also the inner isolated eigenvalue has the same position as in the dense regime, given by the square radius of the bulk, divided by the leading eigenvalue of B ω .

Returning to the implications of Claim . of immediate interest, recall that the claim makes it possible to relate the Nishimori temperature to the speci c eigenvalue λ -1 of the matrix B ω . From a numerical standpoint though, λ -1 is not easily accessible since λ -1 is smaller in modulus than most of the complex eigenvalues of B ω , while not being the smallest in modulus (see Figure . ), one needs to compute all the bulk eigenvalues of B ω in order to access λ -1 : this comes at an impractical computational cost of O n ((cn) 3 ) with state of the art methods. We next show that, as a consequence of Claim . , the weighted Bethe-Hessian matrix (De nition . ) can be e ciently used to estimate β N in the RBIM with a computational cost scaling as O n (cn).

. . β N

This section elaborates on our nal relation between the Bethe-Hessian matrix and the Nishimori temperature, as well as on how the respective spectra can be related to the phase diagram of Figure . . First of all, let us recall the explicit expression of the weighted Bethe-Hessian H β,J that will be considered in the following

The weighted Bethe-Hessian matrix

H β,J ij = δ ij 1 + ∑ k∈∂i th 2 (βJ ik ) 1 -th 2 (βJ ik ) - th(βJ ij ) 1 -th 2 (βJ ij ) . ( . )
We recall from Theorem . that whenever λ = 1 is an eigenvalue of B ω for ω e = th(βJ e ), then det[H β,J ] = 0. In order to best understand how H β,J can be used to e ciently estimate β N , let us consider the behavior of its eigenvalues as a function of β. First of all, at su ciently high temperature (small β), for all β N the system is in the paramagnetic phase, for which m = 0 n is a minimum of the Bethe free energy. Consequently H β,J is positive de nite, while all eigenvalues of B ω are smaller than one. Consider now β N to be su ciently large, so that the system undergoes to a paramagnetic-ferromagnetic phase transition (see Figure . ). For β = β F de ned as cE[th(β F J)] = 1, the leading eigenvalue of B ω is (asymptotically) equal to 1 and one of the eigenvalues of H β F ,J is equal to zero. This eigenvalue is necessarily the smallest. For all matrices, the same graph was used with n = 1 000, c = 10. The weights of the edges are ω ij = th(βJ ij ) for the di erent values of β just described and the J ij are drawn independently from a Gaussian distribution N (J 0 , ν), with J 0 = 1 and ν = 1.5. The orange lines vertical line is at x = 1, while the black vertical line in the second row is at x = 0.

For small values of β N , the system undergoes the paramagnetic-spin glass phase transition (see Figure . ) at the temperature β = β SG de ned so that cE[th 2 (β SG J)] = 1. Here, the radius of the bulk of B ω is (asymptotically) equal to one and the bulk of H β SG ,J is (asymptotically) equal to zero.

Finally, further decreasing the temperature, at β = β N de ned by the relation E[th 2 (β N J)] = E[th(β N J)], the eigenvalue λ -1 is equal to one and the smallest eigenvalue of H β N ,J reaches zero for the second time. In Figure . we show the spectra of the matrices B ω and H β,J at β < β F , β = β F , β SG , β N that con rm the relation between the spectra of these two matrices and the phase diagram.

With these fundamental observations at hand, we now show how one can e ciently estimate β N , exploiting the smallest eigenvalue of H β,J . The proposed estimator of β N reads An e cient estimate for

β N βN = max β β : λ ↑ 1 (H β,J ) = 0 ,
Under this de nition, not only does βN provide a consistent estimate of β N for J distributed as De nition . , but it also provides the "best guess" of an hypothetically corresponding β N for matrices J which would follow a di erent distribution. Indeed, βN has the advantage of always being de ned, even for arbitrary matrices J, while having a clear interpretation for the The weights of the non-zero entries of J are distributed i.i.d. according to N (J 0 , ν 2 ) for J 0 ranging from J 0 = 0.5 to J 0 = 4 and ν = 3.5. Averages are taken over ten samples. Right: behavior of the two smallest eigenvalues of H β,J as a function of β. The solid line indicates the smallest eigenvalue, while the dotted line is the second smallest. The vertical lines are set at β F < β SG < β N . For this simulation, n = 30 000 and c = 10. The weights of the matrix J are distributed i.i.d. according to a N (J 0 , ν 2 ) with J 0 = 1 and ν = 1.5.

class of matrices that fall under De nition . . This de nition is particularly reminiscent of Algorithm . for CD.

The equation λ ↑ 1 (H β,J ) = 0 has two solutions for β ≈ β F and β ≈ β N with β N > β F , justifying our proposed estimator. As a visual support, the right plot of Figure . shows λ ↑ 1,2 (H β,J ) as a function of β. The basic idea of the proposed algorithm to compute βN precisely consists in starting from β = β SG to then nd the value of β > β SG for which λ ↑ 1 (H β,J ) = 0. Following this argument, we propose an iterative algorithm based on Courant-Fischer theorem (similar to Algorithm . ) to compute βN . The output of Algorithm . is depicted in the left display of Figure . . Note in particular that, as long as β SG < β N , i.e., as long as E[th 2 (βJ ij )]/E 2 [th(βJ ij )] < c, the value of βN is a good estimate of β N . When the condition is instead not met, βN simply coincides with β SG . The numerical advantage of exploiting the Bethe-Hessian matrix is decisive. First H β,J is symmetric and of size n × n regardless of the average node degree. Most importantly, the only eigenvalue of H β,J that needs be computed is the one of smallest amplitude, so that βN can be estimated at a O n (nc) operations.

With these theoretical results laid out, we now move back to our initial problem of performing correlation clustering on high dimensional vectors and see how Claim . has a practical interest to accomplish this task. From this trivial computation, it also appears that y is expected to be strongly related to the vector σ. In fact, exploiting Property . of the Bethe-Hessian matrix associated to J, we can write at β = β N :

E H β N ,J = I n + E th(βJ ij ) 1 -th 2 (βJ ij ) (D -A) .
From a straightforward calculation (see Proposition of [VL ]), the vector 1 n is the eigenvector of E[H β N ,J ] associated to its eigenvalue of smallest amplitude. As a consequence, from the relation between x and y (or equivalently between J and J), the vector σ is the eigenvector of E[H β N , J ] associated with its eigenvalue of smallest amplitude. Consequently, the eigenvector with zero eigenvalue of H β N , J is a close approximation of σ.

This conclusion immediately translates into Algorithm . , a numerical method to infer the genuine node classi cation σ. The performance of Algorithm . is tested on synthetic datasets and evaluated against two commonly deployed SC techniques that we here brie y review.

A very classical spectral clustering method in weighted graphs exploits the weighted Laplacian matrix L = D -J, where D = diag(| J|1 n ). As shown in [VL ; Kun+ ], the eigenvector attached to the smallest eigenvalue of L provides a (discrete to continuous) relaxed solution of the NPhard optimization signed ratio-cut graph clustering problem which attempts

Competing spectral methods

to maximize the number of edges with positive weights connecting nodes in the same community, while minimizing the number of edges with negative weights connecting nodes in opposite communities. The relation of this method with the SC algorithm of [Fie ] for CD should be evident and similar comments to the ones made in Chapter are valid in this context as well.

The second method we are to consider instead performs SC on the leading eigenvectors of the matrix J. This is a very popular strategy to perform SC in Note that in the context of correlation clustering and for the choice K = 1 d emb ZZ T , this is nothing but the principal component analysis. weighted graphs and, as we explained in Chapter , it can be interpreted as a naïve mean eld (NMF) (hence suboptimal) version of Algorithm . . Also in this case, the comments made in Chapter are valid also in this setting.

Figure . shows that, while at su ciently large degree, Algorithm . and NMF perform essentially equally (as expected), the same cannot be said for a low average degree where the NMF fails severely. Furthermore, also in the weighted case, the Laplacian matrix displays an algorithmic phase transition which is well beyond the one achieved by Algorithm . and it is hence unsuited for SC in di cult settings. Moreover, note that Algorithm . achieves a positive overlap as soon as β N > β SG , i.e. it reaches the theoretical detectability threshold and performs competitively with the Bayes optimal solution obtained by belief propagation (BP) algorithm. Due to its computational complexity, for c = 50, instead of BP we run the TAP algorithm [OS ] that is a convenient approximation of BP to adopt in the presence of a large average degree.

Remark . . (The "spin glass" Bethe-Hessian) The use of the Bethe-Hessian matrix for SC from sparse pairwise measurements was already proposed in [Saa+ ] that studied a similar framework to the one considered in this chapter. In [Saa+ ] the temperature β = β SG is proposed to perform non-trivial clustering as soon as theoretically possible, with a clear reference to the work of [SKZ ] for community detection (CD).

The value β SG , unlike β N , can be easily estimated from the matrix J solving cE[th 2 (β SG Jij σ i σ j )] = cE[th 2 (β SG Jij )] = 1. However, unlike the algorithm of [Saa+ ], Algorithm . represents an optimal relaxation of the Bayes optimal solution and it is expected to yield better performances.

The main di erence between the spin glass Bethe-Hessian and the Nishimori Bethe-Hessian, however, is observed when the underlying graph has a heterogeneous degree distribution (see Figure of [DCT b]) and, in the presence of a Erdős-Rényi graph, the performances of both algorithms are observed to be similar, with a slight computational advantage for H β SG , J . . . We complete the chapter by a robustness test of our proposed algorithm under a real-world machine learning classi cation problem. Speci cally, we consider a sparse (and thus cost-e cient) version of the problem of correlation clustering such as met in image classi cation and show how Algorithm . can be adopted to accomplish this task with higher performance than with competing spectral methods of the literature.

Given the input vectors stored in the columns of the matrix Z ∈ R n×d feat , we recall from Equation ( . ) that the J is obtained applying a i.i.d. symmetric Bernoulli mask with parameter c/n to the matrix K = 1 d feat Z ZT , where Z ∈ R n×d feat is obtain discarding random elements of Z, keeping them with probability √ κ/d feat . We thus practically tested Algorithm . against the naïve mean eld approach which in this setting happens to coincide with the algorithm proposed in [Zar+ ] and against the weighted Laplacian matrix.

As a telling modern data classi cation context, we chose to cluster two classes of high-resolution realistic images randomly produced by generative adversarial networks (GANs) [BDS ]; the interest of using GAN images rather than real images lies in that they can be produced "on-the-y" and in arbitrary numbers. Speci cally, we considered n = 40 000 images divided into two groups of equal size, representing collie dogs and tabby cats. A representative example of the input images generated by the GAN is given in Figure . . For each of these images we extracted discriminating features using an o -the-shelf convolutional neural network (VGG) which produces p = 512-dimensional feature vectors z i . We then measured the overlap performance as a function of the average node degree c of the ensuing graph and for di erent values of κ. The results are reported in Figure . which strikingly evidences that Algorithm . can achieve almost perfect recon-The p = 512 gure is on the low-hand of typical image vector representations: this number may rise to 4k or even to 20k when much more than two classes of images are to be classi ed. struction already for c = 5 when the feature vectors z i are not sparsi ed (κ = p): so, in clearer terms, out of the 40k × 40k = 1.6 • 10 9 correlations needed to evaluate the full K({z}) matrix, only ≈ 6 × 40k = 2.4 • 10 5 is enough to achieve almost optimal performance, thus corresponding to a striking 10 4 -fold gain in complexity for a rather marginal performance loss.

. The central contribution of this chapter is of a theoretical nature and aims at introducing an explicit relation between the Bethe-Hessian matrix and the Nishimori temperature. Yet, beyond this statistical physics endeavour, the result nds direct applications to Bayesian inference. Speci cally, one may anticipate an important impact in more involved applications than those considered in this article, such as in restricted Boltzmann machines (RBM) whose goal is to learn a generative model from a set of examples [AHS ]: the Bethe approximation has recently been adopted to study the RBM from a Bayesian perspective [HT ] so that one may envision that the explicit relation between the Bethe free energy and the Nishimori temperature would lead to a better understanding of state-of-the-art algorithms.

On the side of complexity reduction, exploiting high levels of sparsi cation of data measurements, we showed that our proposed algorithm is capable of accomplishing high quality unsupervised classi cation on very large datasets. This result is all the more fundamental that future machine learning data treatment will call for increasingly larger datasets which cannot be possibly manually labelled and for which unsupervised approaches must be adopted. As a downside though, the generative model we considered for the data a nity (kernel) matrix takes the strong assumption that its entries are drawn from the same probability distribution and only the average (and not the variance, or the distribution itself) embeds information on the node labels. This setting might be too simplistic on generic real data that would require more realistic probability distributions for the generative model of the kernel matrix to be considered.

Possibly most importantly, we worked here under the assumptions that the edges maintained in the sparsi ed graph are drawn independently at random. When dealing with actual kernel matrices, this cost-e cient measure is quite suboptimal: in [LCM ], a more e cient sparsi cation procedure is used which maintains the entries of K({z}) of largest amplitude. In [LCM ], this comes at the cost of computing all the entries of K({z}) but, surely, a more e cient nearest neighbors-type procedure could be implemented as a good performance-complexity compromise [ML ]. Yet, in this setting, although stronger sparsity levels can surely be achieved for the same performance, the key independence property of the entries of K({z}) which we exploited here can no longer be assumed, so that one needs to carefully handle the hard problem of dependencies.

C O N C L U S I O N

In this manuscript we investigated spectral clustering (SC) techniques adapted (but not necessarily limited) to sparse graphs with applications to community detection (CD) on both static and dynamical graphs, as well as correlation clustering of high dimensional vectors. The le rouge connecting all our main results revolves around the claim of the existence of isolated informative eigenvalues inside the bulk of the non-backtracking matrix B. Their location is deeply related with the meta-parameters of the generative model of B and the corresponding eigenvectors can be used to obtain a relaxation of the Bayes optimal solution. Furthermore, exploiting its relation with the non-backtracking matrix, we provided cost e cient SC algorithms based on the Bethe-Hessian matrix, extensively tested on real and synthetic datasets.

A profound result of our proposed algorithms relies in their ability to "selfadapt" to the hardness of the clustering problem, bridging existing results of the literature which considered either hard or trivial, either sparse or dense settings. Supported by extensive numerical simulations, we showed that our proposed algorithms often outperform the state-of-the-art "non-adaptive" competing ones.

If on the one hand our work strongly motivates the adoption of wellparametrized Bethe-Hessian matrix for SC, on the other hand it also poses several questions, both on an algorithmic as well as theoretical levels.

This manuscript gives a great deal of attention to the problem of designing e cient as well as robust algorithms that can be run on arbitrary input graphs. The empirical experience led us to spot some practical limitations that were not foreseen when we designed our algorithms in the rst place. This is particularly the case of Algorithm . for CD in static graphs.

We recall that Algorithm . has to compute a sequence of values {ζ p } p=1,...,k satisfying λ ↑ p (H ζ p ) = 0. To do so, we designed an algorithm that starts from a value r ≥ ζ k to then estimate successively the largest (ζ k ) and arriving to the smallest (ζ 1 ). The proposed Algorithm . computes the ζ p exploiting a relation derived from the Courant-Fischer theorem and does not make any assumption on the function λ ↑ p (H r ). The practical limitations of this implementation are brie y listed as follows.

. Suppose that for the input graph G(V, E ) , ζ k ≈ 1 and the average degree is su ciently large. This corresponds to an easy clustering setting (due to ζ k ≈ 1 hypothesis) in which the most commonly adopted SC would likely output a very accurate result. In this case, Algorithm . has to start from a rather large value r (due to the large average degree) and potentially needs several iterations to converge. Consequently, although the performance of Algorithm . for this easy task is still expected to be very high, its computational time will be signi cantly higher than the one of competing algorithms.

. The value r = λ

↓ |•| 1 (B) (adopted in Algorithm .
) is only presumed to be a good approximation of

r = arg max r |{i : λ i (H r ) < 0}| .
Algorithm . requires r as an input and the mismatch between its actual and estimated values may potentially lead to discard informative eigenvectors.

. The number of communities k in a real social graph may grow with its size due to the typical hierarchical structure of complex systems [Sim ]. Consequently, on large graphs, very large k may be found. To provide a practical example, the LiveJournal social network has n = 34 681 189 nodes and k = 287 512 communities [YL ]. The computational complexity of Algorithm . scales as O(nck 3 ), making it unpractical for a large k like the one of the LiveJournal dataset.

We envision that all three problems listed above can be easily solved designing an algorithm that estimates the ζ p 's from the smallest (ζ 1 = 1) to the largest. This strategy obviously solves the problem raised in ( ). Also point ( ) could be easily addressed, since k can be estimated as the largest p for which ζ p is de ned and has not to be speci ed before computing the ζ p 's. Concerning point ( ), instead, one can envision to adopt a strategy according to which, rst the graph is partitioned in k 1 sub-graphs (computing ζ 1 , . . . , ζ k 1 ), then on each sub-graph so obtained the process is re-iterated at lower scales until no communities can be detected. For a hierarchical structure, it is reasonable to assume k 1 ∼ log(k), hence solving also problem ( ). Unfortunately, we still did not manage to de ne an algorithm such as Algorithm . in which the ζ p 's are successively computed from the smallest one, without making any assumption on λ ↑ p (H r ). There lies one of the main improvements of our implementation.

Concerning our proposed algorithms for dynamic CD, on top of the comments detailed in Section . , the main technical issue is certainly related to point ( ) of the list above: our result is generally well posed for graphs generated from the dynamical degree corrected stochastic block model (DDCSBM), It is worth to remark that, for this particular example, the O(nk 2 ) complexity of typical SC algorithms may also be prohibitive. Furthermore, making an embedding in ≈ 10 5 dimensions is unsuited for k-means, due to the curse of dimensionality. but for general graphs, the value of k may be potentially underestimated. The main weakness of Algorithm . for correlation clustering, instead, resides in point ( ): the entries of H β,J grow exponentially with β and in the easy clustering regime (β N → ∞) this causes convergence problems. Unlike CD, however, this is not solely a matter of speed of convergence, but also of numerical stability. This could be improved adopting a method like the one proposed in [CD ] to compute the eigenvalues and eigenvectors of matrices with di erent orders of magnitude.

All the results of this manuscript concerned only a speci c one of the two steps of SC, that is generating a signi cant small dimensional node embedding. The whole point of SC, however, is to assign labels and the second step of SC has been systematically performed with the use of k-means algorithm. The rst question that naturally arises is whether or not this is an optimal choice. We believe that the answer is unlikely to be positive and that more e cient algorithms can be considered in place of k-means.

One of the main alternatives to k-means is provided by density based clustering algorithms such as [Est+ ; Ank+ ]. These methods are likely more robust than k-means to a asymmetrical embedding such as the one that may appear in Algorithm . for general graphs. It has to be noted, however, that we experimentally veri ed that these methods (that typically do not require k as an input) fail to achieve the detectability threshold on synthetic graphs because, close to the transition, there is no region in space in which the density of points is su ciently low to draw a boundary.

A theoretically more solid alternative (yet less exible to the implementation on arbitrary graphs) consists in studying the probability distribution of the points in the embedding space and run expectation maximization algorithm to perform clustering [DLR ]. Actually, k-means consists in solving exactly this problem, assuming that the probability distributions are isotropic Gaussian with the same variance. It has to be noted, however, that to properly address expectation maximization for the problem considered, one needs to have a full theoretical understanding of the eigenvectors of the Bethe-Hessian matrix. To the best of our knowledge, the tools to rigorously address this question in the sparse regime are still lacking.

On a more pragmatical side, Chapters , evidenced the striking e ciency of the projection of the embedding vectors on a unitary hypersphere prior to the k-means step. This technique is likely to be suboptimal (at least for DCSBM-generated graphs), but we acknowledge that it provides very good results at a moderate computational cost. This evidence, however, needs a thorough theoretical justi cation, that, in the sparse regime, is still lacking. To motivate this assertion, we recall that the projection step is justi ed when the entries of the embedding matrix X ∈ R n×k can be written as

X i,• = [ f i g (1) i , f i g (2) i , . . . , f i g (k) i ].
In practice, the entries of X i,• are decomposed in two contributions: one that remains constant along the row but that is di erent for each node, f i ; one that takes a di erent value for each column of X, but depends on i only through its label i and is therefore capable to group together nodes into communities. Normalizing X i,• = 1, we see that the i-dependent component of X i,• is removed, producing a high quality embedding which depends solely on the node labels. An alternative strategy to preprocess X however exists and it consists in de ning X as follows X

•,i = X •,i X •,1 . ( )
Although these two normalization procedures may seem very similar, in practice they perform very di erently. In particular, the left plot of Figure shows the performance of clustering on the leading eigenvectors A on dense DCSBM graphs and for the two normalization strategies. Note that the normalization obtained from Equation ( ) precisely corresponds to the SCORE algorithm of [Jin+ ]. Figure shows that the two methods perform equally well, thus fully justifying the projection step in this case. On the right, the same plot is proposed in the sparse regime for L sym τ = D -1/2 τ AD -1/2 τ for τ equal the average degree. In this case, adopting the projection on the hypersphere as a normalization corresponds to the algorithm of [QR ], which we extensively commented in Chapter . In this case, the two plots are in evident disagreement, showing that the high performance obtained from the projection step goes beyond its theoretical justi cation. Also in this case, to formally address this question, one would need to have clear results on the eigenvectors of the considered matrix to then study the convergence of k-means on the obtained embedding.

To conclude, our results address the problem of designing a proper node embedding with a SC algorithm. Nonetheless, SC requires also to be capable of extracting the information from the embedding and considering the problem as a whole certainly deserves further investigations.

The work presented in this manuscript raises some important questions also from a theoretical standpoint. Perhaps the most relevant result concerns an explicit relation between the spectral properties of the Bethe-Hessian matrix and the Nishimori temperature as detailed in Chapter . It has to be noted that, although the mapping between the Ising model and Bayesian inference was fully rigorous only in Chapter , the results of both dynamic and static CD are closely related.

From a statistical physics perspective, a question that certainly deserves further exploration concerns the deep meaning and implications of the Bethe-Nishimori relation. From our conjecture, it appears that the paramagnetic point m = 0 n is not a minimum of the Bethe free energy for β F < β < β N , but it is a local minimum for β > β N , with the Nishimori temperature denoting the transition between these two conditions. At low temperature, m = 0 n corresponds to a local minimum of the free energy hence to a conguration that is metastable. This result should be related to existing experimental and numerical observations of the random bond Ising model (RBIM).

On top of this, the Nishimori-Bethe relation is formulated only for Erdős-Rényi (ER) random graphs and for Ising variables. The properties of the Nishimori temperature can however be extended to a larger class of models than the RBIM [Geo+ ] and to graphs not generated from the ER model. The natural line of exploration in this direction is hence to study the relation between the Bethe free energy and the Nishimori temperature for di erent classes of graphs and Hamiltonians.

Moreover, beyond the interest in statistical physics of our results, we also unveiled some previously unseen behavior of the non-backtracking matrix eigenvalues, predicting the existence of isolated eigenvalues inside its bulk. From an algebraic standpoint, the emergence of these eigenvalues is well understood in quasi-regular graphs [CZ ], but we believe that this explanation does not capture the depth and complexity of the problem. Further mathematical tools would then be needed to formally address this question and study these eigenvalues and their related eigenvectors.

Let us now look ahead, proposing some further lines of explorations and applications of our results.

Concerning the problem of CD we detailed extensively in Chapters , that our proposed Algorithm . can e ciently be implemented on arbitrary graphs, but its backbone is deeply motivated by the DCSBM which assumes that the edges of the graph are generated independently at random. In typical realizations of social graphs, however, the entries of the adjacency matrix are highly correlated making indeed questionable the DCSBM assumption in practice. In order to keep into account these additional dependences induced by the presence of short cycles, in [KCN ] the authors recently proposed an improved version of belief propagation (BP) algorithm, adapted to operate in the presence of short cycles, further de ning the related non-backtracking matrix. Other "higher order approximations" exist to properly deal with the non tree-like structure of a graph. To accomplish this task, a valid alternative to the Bethe is the cluster variational method [Pel ].

The capability of properly treating dependencies is not constrained to CD. In fact, in the context of correlation clustering, we discussed that the i.i.d. sparsi cation strategy is suboptimal in terms of performance, indeed because it makes the entries of the kernel matrix asymptotically independent. More e cient sampling techniques may be designed achieving a good level of sparsity, but keeping the correlations among the entries of the matrix. The adoption of proper variational approximations to deal at once with dependencies and sparsity is certainly key to design e cient algorithms.

Most of our results revolve around the very popular (but somewhat simplistic) problem of CD. In the chapters of Part III we however showed how the intuitions and methods developed for CD can be easily extended to more involved settings. This unveils on the one hand the power of the method and on the other hand the depth of Claim . , which is formulated for the DCSBM, but that conveys an intuition that goes well beyond its assumptions.

Most crucially, based on this observation and the discussions of Chapter , we envision that the "Bethe-Hessian" paradigm should be systematically considered as an alternative to the classical Laplacian and "weighted adjacency" matrices that are commonly adopted for several machine learning applications. In fact, we showed that the most popular techniques for SC can be interpreted as limiting cases of our proposed algorithms. The huge advantage of our algorithms is that they constitute a bridge between these extremes and consequently perform better at practical SC problems.

With this in mind, we close this manuscript with a question that invites the practitioner to re ect any time the eigenvectors of a weighted adjacency or Laplacian matrix are exploited in one of the countless applications, whether it is possible to de ne a more appropriate graph matrix representation introducing a proper temperature parameter or a better variational approximation to obtain improved performances. En ce qui concerne le problème du regroupement, les techniques non supervisées sont généralement préférées dans les cas où la machine ne dispose d'aucune information supplémentaire en dehors des données à regrouper. L'absence d'"aide extérieure" entrave bien sûr les performances de l'apprentissage non supervisé par rapport à l'apprentissage supervisé, mais les inconvénients introduits par l'étape d'apprentissage sont ici contournés. Les algorithmes d'apprentissage non supervisé pro tent en fait de la capacité des machines de traiter d'énormes quantités de données en peu de temps qui leur donnent la possibilité d'exploiter e cacement les relations entre ses éléments.

Bien que les algorithmes d'apprentissage supervisé soient aujourd'hui très populaires en raison de leur capacité à obtenir des performances surhumaines, de sérieuses inquiétudes doivent être soulevées quant à leurs exigences en matière d'échantillons étiquetés et d'entraînement intensif en terme de temps calcul [Usa+ ]. En e et, l'augmentation constante de la taille des ensembles de données d'entrée ne peut (et probablement ne doit) pas s'accompagner d'une croissance aussi rapide de l'e ort de calcul. Dans cette perspective, les techniques non supervisées jouent un rôle fondamental dans l'apprentissage automatique moderne.

Le clustering a plusieurs applications dans di érents domaines et est, en général, un problème mal dé ni [XW ; RM ; Sch ; WK ]. L'une des classes les plus populaires d'algorithmes de clustering, qui est au coeur du travail présenté dans ce manuscrit, est composée des méthodes spectrales dont les éléments constitutifs sont décrits comme suit.

La première étape consiste à fournir une représentation sous forme de graphe aux données d'entrée. Ensuite, l'extraction de caractéristiques consiste à mettre en correspondance chaque élément (ou noeud du graphe) avec un vecteur de petite dimension [CZC ; NS ]. Cela doit être fait de manière à ce que des noeuds "similaires" soient projetés sur des vecteurs proches. Dans le clustering spectral, la correspondance entre les noeuds et les vecteurs est obtenue en exploitant les vecteurs propres d'une représentation matricielle de graphe adaptée, M ∈ R n×n . L'un des principaux avantages du clustering spectral par rapport à d'autres méthodes repose sur ses solides fondements théoriques et son caractère explicable.

L'étape nale consiste à déterminer des frontières dans l'espace en petite dimension pour séparer les points. L'algorithme le plus couramment adopté pour accomplir cette tâche est k-means [Mac+ ], mais d'autres solutions telles que k-medoids [KR ], expectation maximization [DLR ] DBSCAN [Est+ ] et OPTICS [Ank+ ] sont également des alternatives valables.

Sans doute, la tâche la plus importante dans les méthodes spectrales est d'identi er une représentation matricielle du graphe M telle que ses vecteurs propres soient capables de produire une bonne projection des noeuds dans l'espace de petite dimension. Certains des choix les plus courants de M sont une combinaison des matrices degré et adjacence du graphe, que nous allons maintenant dé nir.

Nous désignons par A ∈ {0, 1} n×n la matrice d'adjacence, dé nie de telle sorte que A ij = 1 si i et j sont connectés et est égale à zéro sinon. La matrice de degré D ∈ R n×n est une matrice diagonale telle que D ij = δ ij (A1 n ) i .

Certains choix populaires de M sont la matrice d'adjacence A elle-même, la matrice Laplacienne D -A et les matrices Laplacienne normalisée L sym = D -1/2 AD -1/2 , L rw = D -1 A [VL ; LR+ ], avec des extensions possibles également au cas pondéré dans lequel A est remplacé par W ∈ R n×n .

'

Les méthodes spectrales sont très interdisciplinaire et, a n de décrire au mieux où se situe le travail présenté dans ce manuscrit dans la littérature existante, il faut garder une large perspective sur le problème. Tout d'abord, comme nous l'avons mentionné précédemment, l'un des principaux intérêts des méthodes spectrales réside dans leur solide base théorique. De puissants outils basés sur la théorie des matrices aléatoires permettent de fournir des résultats précis sur les performances et l'applicabilité d'un algorithme spectral. Typiquement, cependant, deux hypothèses importantes doivent être formulées: la taille n de la matrice M doit aller à l'in ni et le nombre d'entrées non nulles (que l'on note m) doit aller à l'in ni plus vite que n; [CBG+ ; AC , par exemple]. Alors que la première hypothèse consiste simplement à considérer le cadre très intéressant des grands bases de données, d'un point de vue pratique, la seconde hypothèse pose un problème du point de vue du passage à l'échelle des algorithmes de clustering spectral. En e et, le coût numérique du calcul des k vecteurs propres principaux d'une matrice à m entrées non nulles croît comme O(mk 2 ) [Saa ]. Considérons le cas m = n 2 , c'est-à-dire que toutes les entrées de la matrice M sont non nulles, pour une petite valeur de k telle que k = 2. La complexité est dans ce cas O(n 2 k 2 ) qui est prohibitif pour un ordinateur portable standard lorsque n > 10 5 environ.

Le problème de la complexité de calcul du clustering spectral est bien connu dans la littérature et a conduit à plusieurs stratégies d'échantillonnage visant à améliorer l'e cacité du clustering spectral [TL ]. La procédure la plus simple consiste simplement à rendre la matrice M plus parcimonieuse, en mettant certaines de ses entrées à zéro, ce qui améliore signi cativement la vitesse de calcul de l'algorithme. Considérons en e et une stratégie d'échantillonnage qui conserve en moyenne 10 entrées non nulles dans chaque ligne de M: la limite de calcul de l'ordinateur portable considéré précédemment est maintenant obtenue pour n = 10 9 ce qui permet de traiter des bases de données de taille considérable.

Cette procédure d'échantillonnage a cependant un coût. Au-delà du fait qu'en "jetant" certaines entrées (donc de l'information), les performances de clustering peuvent naturellement être plus faibles, la parcimonie est connue depuis longtemps pour être le point faible des méthodes spectrales. En fait, pour de grands niveaux de parcimonie, les choix standards de M ont typiquement des performances faibles par rapport aux algorithmes non-spectraux.

Le problème de la parcimonie n'est pas seulement lié aux procédures de échantillonnage. Une application naturelle du regroupement dans les graphes consiste à diviser les noeuds d'un réseaux social en communautés. Les réseaux sociaux réels sont souvent parcimonieux par construction [Bar ] (chaque noeud interagit généralement avec une très petite fraction du total) et, par conséquent, le regroupement spectral peut être peu performant dans les tâches pratiques de détection de communautés. Dans l'ensemble, il s'agit de limitations sévères des méthodes spectrales qui les rendent inadaptées à un large ensemble des bases des données (comme les réseaux sociaux susmentionnés) et qui ne permettent de bonnes performances qu'au prix d'un coût de calcul élevé.

Une percée dans le domaine des méthodes spectrales dans les graphes parcimonieux a toutefois été réalisée par [Krz+ ] spéci quement dans le contexte de la détection de communautés. Le travail de [Krz+ ], ainsi que de [Dec+ ; SKZ ], est basé sur des intuitions et des méthodes très profondes mais non rigoureuses empruntées à la physique statistique qui ont renouvelé l'intérêt de la communauté scienti que pour le sujet et ont conduit, dans les années suivantes, à formaliser la plupart de leurs résultats de manière mathématiquement rigoureuse. En ce qui concerne la détection de communautés, la principale contribution de [Krz+ ; SKZ ] consiste à proposer des matrices alternatives M, telles que les matrices hessienne de Bethe et de non-backtracking, pour remplacer les choix "classiques".

La hessienne de Bethe joue un rôle fondamental dans ce manuscrit, étant le choix pour M dans tous nos algorithmes proposés. Dans le cas non pondéré, sa dé nition est la suivante: pour un scalaire r ≥ 1 H r = (r 2 -1)I n + D -rA, où I n est la matrice identité de taille n. La contribution fondamentale de ce manuscrit est d'argumenter que M = H r pour une valeur bien choisie de r constitue un choix puissant pour le regroupement avec des méthodes spectrales dans les graphes parcimonieux qui atteint des performances élevées dans les problèmes du monde réel.

Nous allons maintenant détailler nos principaux résultats.

A n de fournir au lecteur les outils nécessaires pour comprendre au mieux nos contributions, la partie I introduit les résultats techniques exploités dans le reste. En particulier, le chapitre fournit une introduction aux graphes, ainsi que leurs dé nitions et propriétés. Ensuite, le chapitre présente les outils fondamentaux de la physique statistique sur lesquels reposent nos principaux résultats. En n, le chapitre présente des résultats mathématiques importants sur les propriétés spectrales de deux matrices fondamentales dans ce manuscrit: la matrice de non-backtracking et la hessienne de Bethe susmentionnées.

La partie II est ensuite consacrée à la présentation de nos contributions originales dans le domaine des méthodes spectrales pour la détection de communautés et est ouverte par le chapitre qui fournit une introduction approfondie au problème.

Nous reprenons ici le " ambeau" de nos prédécesseurs et abordons le problème de la manière dont des hypothèses trop simplistes peuvent entraîner de mauvaises performances algorithmiques. Plus précisément, dans [SKZ ], les auteurs ont proposé un algorithme spectral très puissant pour la détection de communautés dans les graphes très parcimoniuex, basé sur la matrice hessienne de Bethe H r . Leurs résultats théoriques sont formulés sous l'hypothèse que la matrice A est générée à partir du modèle stochastique par blocs, selon lequel chaque noeud du graphe a approximativement le même nombre de connections. Cependant, les réseaux sociaux réels (dans lesquels la détection des communautés est de la plus haute importance) sont connus pour être typiquement hétérogènes, dans le sens où le nombre de connections de chaque noeud varie beaucoup à travers le réseau. En conséquence de cette hypothèse trop simpliste, l'algorithme de [SKZ ] (qui prescrit un choix précis de r) a souvent des performances médiocres par rapport aux algorithmes classiques de clustering spectral.

Nos premiers travaux sont présentés au chapitre et expliquent comment nous avons amélioré [SKZ ], en considérant un modèle génératif plus réaliste, appelé modèle stochastique par blocs de degrés corrigés (DCSBM) qui permet en e et de tenir compte de l'hétérogénéité du graphe. Le premier de ces travaux est [DC ] LD, Romain Couillet: Community detection in sparse realistic graphs: Improving the Bethe-Hessian in ICASSP IEEE I C A , S S P ( ) dans lequel nous avons proposé (pour k = 2 classes de taille égale) un choix alternatif de r à celui de [SKZ ], capable de fournir un clustering performant sur des graphes hétérogènes. La paramétrisation r proposée nécessite cependant la connaissance de certains des paramètres du modèle génératif du graphe qui ne peuvent être donnés pour acquis.

Avec notre contribution suivante [DCT ] LD, Romain Couillet, Nicolas Tremblay: Revisiting the Bethe-Hessian: Improved Community Detection in Sparse Heterogeneous Graphs in N IPS A N I P S ( ) nous avons répondu à certaines des principales questions posé en [DC ] . Premièrement, nous avons étendu nos résultats au contexte de k ≥ 2 classes de tailles arbitraires, en montrant qu'une séquence de paramètres r correctement choisis doit être sélectionnée pour obtenir un algorithme spectral ecace sur des graphes hétérogènes. Deuxièmement, nous avons montré que ces paramètres r peuvent être estimés de manière non supervisée. Pour ce faire, nous avons dévoilé une propriété imprévue des valeurs propres de la matrice de non-backtracking B utilisée dans [Krz+ ] et exploité la relation connue entre B et H r donnée par la formule d'Ihara-Bass [Ter ]. En n, à l'aide de plusieurs arguments et de simulations numériques, nous a rmons que la paramétrisation que nous proposons est optimale, en ce sens qu'aucune autre paramétrisation ne devrait être plus performante sur les graphes générés par le DCSBM.

Le fait de disposer d'un moyen pratique d'estimer les valeurs optimales de r de manière non supervisée nous a permis de concevoir et de tester un algorithme pratique de détection de communautés. Les résultats conrment que la paramétrisation améliorée atteint une performance systématiquement plus élevée par rapport à [SKZ ] sur tous les ensembles de données testés.

Dans la littérature, les contributions de la communauté des physiciens ne sont pas les seules à avoir conduit à des algorithmes spectrales très e caces sur les graphes parcimonieux. En fait, une ligne de travail notable et indépendante, avec une approche ancrée dans les statistiques a proposé l'utilisation de nouvelles matrices régularisées. Dans ce resumé nous ne mentionnerons que l'une de ces matrices, L En outre, sur la base de nos résultats dans [DC ; DCT ] , nous avons montré que le choix de τ dans [QR ] est sous-optimal. La régularisation optimale du Laplacien est obtenue avec un ensemble de τ étroitement lié aux r à adopter pour H r et qui peut être estimé à partir du graphe de manière non supervisée. D'ailleurs, certains des résultats présentés dans les chapitres , sont tirés de [DCT a] LD, Romain Couillet, Nicolas Tremblay: A uni ed framework for spectral clustering in sparse graphs in J dans lequel nous avons résumé nos résultats dans un cadre uni é en faisant des déclarations formelles de nos conjectures et en fournissant des preuves rigoureuses pour une partie d'entre elles. Une autre contribution importante de [DCT a] est au centre du chapitre et concerne l'applicabilité de nos algorithmes proposés sur des graphes réels pratiques ainsi que leur mise en oeuvre e cace. Nous proposons ici l'Algorithme . pour la détection de communautés et menons une étude systématique sur des ensembles de données réelles montrant l'e cacité de notre méthode pour des tâches pratiques. De plus, nous avons également publié une implémentation e cace en langage Julia de l'Algorithme . pour la détection de communautés, appelée CoDeBetHe.jl. (Community Detection with the Bethe Hessian).

Le chapitre clôt nalement la partie II en partant des résultats posés dans [DCT a] pour fournir une vue uni ée de plusieurs méthodes de clustering spectral qui ont jusqu'à présent été traitées indépendamment. Dans ce chapitre, en fait, nous montrons clairement que l'algorithme que nous proposons est adaptatif à la dureté de la tâche de clustering et que les méthodes courantes de l'état de l'art dans les régimes parcimonieux et denses peuvent être comprises comme des cas limites (sous-optimales) de l'Algorithme . . Ces résultats posent des bases méthodologiques solides qui nous ont conduit à travailler sur les extensions de nos travaux à des contextes plus généraux que la détection de communautés, traitées dans la partie III. Le premier problème que nous avons considéré concerne la detection des communautés dans des graphes dynamiques (DCD), qui enrichit la description en tenant compte de la nature dynamique des graphes du monde réel. Dans [DCT a] En n, les travaux de [ DCT b] LD, Romain Couillet, Nicolas Tremblay: Nishimori meets Bethe: a spectral method for node classi cation in sparse weighted graphs in J S M : T E sont présentés dans le chapitre et traitent de l'extension de nos travaux au cadre de la sparsi cation matricielle pour améliorer l'e cacité des méthodes spectrales, tout en conservant des performances élevées. En testant l'algorithme que nous proposons sur un problème prototypique de l'apprentissage automatique d'images de chiens et de chats, nous avons montré qu'une précision presque parfaite peut être obtenue même pour des niveaux de sparsi cation très élevés, ce qui rend notre algorithme adapté au regroupement de très grands ensembles de données. Le résultat principal de [ DCT b] , cependant, est profondément lié à la physique statistique, décrivant une relation explicite entre l'approximation de Bethe (à partir de laquelle la matrice hessien de Bethe est dérivée) et la température de Nishimori. [Nis ], d'une importance fondamentale en physique des verres de spin et en inférence optimale bayésienne [Iba ]. In this appendix we prove Theorem . which provides a necessary and sucient condition for the existence of a giant component in a multi-class (with unequal sizes) DCSBM. This theorem is an application of [BJR , Theorem . ] to the present DCSBM setting. For convenience, we recall here the formulation of Theorem . . Theorem A. (Percolation threshold in DCSBM, [DCT a]). Consider a graph G(V, E ) generated according to the DCSBM procedure as in De nition . . Let Π = diag(π 1 , . . . , π k ) and assume that the constant c is the largest eigenvalue of CΠ with eigenvector 1 k . Then, for all large n, with high probability, G(V, E ) has a giant component if and only if cΦ > 1.

Proof. In our proof we will use [BJR , Theorem . ] which requires to write the graph G(V, E ) as a graphon. Let us de ne the "node" variable x = θv ∈ S, with probability distribution µ(x), where • θ ∈ Θ is a scalar that distributed according to ν.

• v ∈ {e 1 , . . . , e k } with e p ∈ R k satisfying e p, = δ p ; v is a k-dimensional random vector with law P(v = e p ) = π p ; that is, v encodes the class to which node x belongs.

• The probability density of x is denoted by µ(x) and is de ned on S = ∪ k =1 S where S = [θ min e , θ max e ]. The measure µ(x) is equal to π ν(θ) if x ∈ S and it is indeed normalized:

S dµ(x) = k ∑ =1 π Θ dν(θ) = k ∑ =1 π = 1.
Given two such nodes, we next de ne the graphical kernel κ(x, y) = x T Cy, satisfying the three conditions of [BJR , De nition . ]. With these notations at hand, the generative procedure of the DCSBM under consideration is equivalent to: drawing n independent values {x i } i=1,...,n from µ(•) and then generating the edges independently according to p ij = min κ(x i ,x j ) n

, 1 ; we thus fall precisely under Bollobas' framework.

In order to use the core argument of [BJR , Theorem . ], we still need to de ne the linear operator T κ , an operator on f : R k → R, endowed with the norm f 2 2 = S f 2 (x) dµ(x), as:

∀ x ∈ S, (T k f )(x) = S κ(x, y) f (y) dµ(y)

The square 2-norm of this operator, T κ 2 , reads

T κ 2 2 = sup f 2 ≤1 S
(T κ f ) 2 (x) dµ(x).

(A. )

According to [BJR , Theorem . ], with high probability, a giant component appears in G(V, E ) if and only if T κ > 1. We thus are left to evaluating T κ for the kernel x T Cy: we will show that it equals cΦ, rst by showing that T κ ≤ cΦ and then nding a particular function f for which the bound is attained.

Let f be a function from R k to R. We introduce its associated vector ω f ∈ R k as follows.

(T κ f )(x) = where ∀ , ω f , = Θ θ f (θe ) dν(θ). From Equation (A. ), one thus has:

S (T κ f ) 2 (x)dµ(x) = S ω T f ΠCxx T CΠω f dµ(x) = Φ ω T f ΠC k ∑ =1
π e e T CΠω f

= Φ ω T f ΠCΠCΠω f . (A. )
Injecting the result of Equation (A. ) into Equation (A. ) we obtain

T κ 2 2 = Φ sup f 2 ≤1 ω T f ΠCΠCΠω f = Φ sup f 2 ≤1 ω T f ΠCΠCΠω f ω T f Πω f • ω T f Πω f ≤ Φ sup f 2 ≤1 ω T f ΠCΠCΠω f ω T f Πω f • sup f 2 ≤1 ω T f Πω f (A. )
Analyzing the rst element, we can write for v f = Π 1/2 ω f :

sup f 2 ≤1 ω T f ΠCΠCΠω f ω T f Πω f = sup f 2 ≤1 v T f Π 1/2 CΠCΠ 1/2 v f v T f v f ≤ sup v∈R k v T Π 1/2 CΠCΠ 1/2 v v T v
i.e.:

sup

f 2 ≤1 ω T f ΠCΠCΠω f ω T f Πω f ≤ λ ↓ 1 (Π 1/2 CΠCΠ 1/2 ) = λ ↓ 1 (CΠ) 2 = c 2 ,
as, by hypothesis, c is the Perron eigenvalue of CΠ, and as such is larger than the modulus of any other eigenvalue: λ ↓ 1 (CΠ) 2 is indeed c 2 . We can therefore rewrite Equation (A. ) as

T κ 2 2 ≤ c 2 Φ sup f 2 ≤1 ω T f Πω f .
Analyzing the right-hand side term, we have

ω T f Πω f = k ∑ =1 π ω 2 f , = k ∑ =1 π Θ θ f (θe ) dν(θ)
2 which, by Cauchy-Schwartz's inequality is bounded as

Θ θ f (θe ) dν(θ) 2 ≤ Θ θ 2 dν(θ) Θ f 2 (θe ) dν(θ) ≤ Φ,
where in the last step we used the fact that the norm of f is less than or equal to one. We thus obtain that sup . ) We are thus left to showing that there exists a function f for which the bound is reached. Let us consider f (x) = x / √ Φ. It is easy to check that this function has unit norm

f 2 ≤1 ω T f Πω f ≤ Φ and conclude that T κ 2 2 ≤ (cΦ) 2 . ( A
f 2 2 = S x √ Φ 2 dµ(x) = 1 Φ S θ 2 e 2 dµ(x) = 1 Φ S θ 2 dµ(x) = 1.
Furthermore, observe that ω f = √ Φ1 k . Then, we have

T κ 2 2 = Φ sup f 2 ≤1 ω T f ΠCΠCΠω f ≥ Φω T f ΠCΠCΠω f ≥ Φ 2 1 T k Π(CΠ) 2 1 k = (cΦ) 2 (A. )
as, by hypothesis, we suppose that CΠ1 k = c1 k .

Combining Equations (A. ) and (A. ), we nd that T κ = cΦ so that, from [BJR , Theorem . ], we conclude that the percolation transition is at cΦ = 1 and a giant component exists if and only if cΦ > 1.

Proof of Corollary B. . A connected graph with |E | = |V | can be obtained by adding an edge between any two nodes of a particular tree de ned on V. The graph G thus contains only one cycle. We now apply Lemma B. "backwards", i.e., removing nodes with unitary degree (called leaves) from G(V, E ) without a ecting the non-zero eigenvalues of B. By iteratively removing all leaves, the graph G(V, E ) reduces to a cycle. We are thus left to prove that λ . T

We here want to show that λ ↓ p (T) = 1 c λ ↓ p+1 (CΠ) for 1 ≤ p < k. Consider the following equivalent identities for the matrix T = ΠC c -Π1 k 1 T k : In this appendix we provide the supporting arguments to Claim . .

ΠC c -Π1 k 1 T k a p = λ ↓ p (T)a p CΠ c -1 k 1 T k Π b p = λ ↓ p (
Recall the De nition . of the aggregate graph G. As a consequence of the sparsity of each G t , the graph G, obtained by connecting together the same node at successive times is locally tree-like, i.e. the local structure of G around a node v ∈ V is the same as that of a Galton-Watson tree T (v) [DM+ a], rooted at v, designed according to the following procedure:

. Let v ∈ {1, 2} be the label of v . Next generate its progeny by creating d s spatial children (i.e., nodes which live at the same time as v), where d s is a Bernoulli random variable with mean cΦ, and two temporal children (i.e., nodes which are the projection of v at neighbouring times) . For each spatial child w, assign the label w = v with probability c in /(c in + c out ) and w = 3v otherwise; the temporal children keep the same label as v with probability (1 + η)/2 and change it with probability (1η)/2

. Each node thus created further generates its own set of o spring, with the only di erence that the temporal children only bear one extra temporal child, while spatial children bear two.

In the limit n, T → ∞, for any arbitrary v ∈ V, the probability distribution around a neighbourhood of v reachable in a nite number of steps is asymptotically the same as T (v), the Galton-Watson tree rooted at v. The local tree-like structure is preserved for nite T (and n → ∞) but the boundary conditions imposed by t = 1 and t = T must be accounted for.

Taking inspiration from the results of [JM+ ] on robust reconstruction on trees, the authors of [Gha+ ] conjectured a generalization for a multitype branching process, such as just described to construct T (v). The conjecture of [Gha+ ] (which we adapted to the DDCSBM, rede ning α, follow-ing the work of [GLM ]) states that, for T → ∞, community detection is possible if and only if the largest eigenvalue of

M ∞ (α, η) = α 2 2η 2 α 2 η 2 (C. )
is greater than one. Taking a closer look at M ∞ , one sees that rst row is given by the contribution of spatial children, while the second one from temporal children. For nite T, at each time instant, one can identify three types of edges: spatial edges (connecting nodes in G t to nodes in G t ), forward temporal edges (connecting nodes in G t to nodes in G t+1 ) and backwards temporal edges (connecting nodes in G t to nodes in G t-1 ). We construct a matrix MT (α, η) ∈ R 3T×3T identifying the rows and the columns as {(backwards temporal) t , (spatial) t , (forward temporal) t } t=1,...,T .

A (backwards temporal) t edge goes from a node in V t to a node in V t-1 that has, on average, cΦ spatial children with label correlation equal to γ = (c inc out )/(c in + c out ) and one backwards temporal child, with label correlation equal to η. Similarly (spatial) t goes from a node in V t to a node in V t having cΦ temporal children and, one forward and one backwards temporal children; nally, (forward temporal) t goes from V t to V t+1 with one forward temporal child and cΦ spatial children. The entry i, j of MT (α, η) is then set equal to the number of o -springs of type j of a node reached by an edge of type i, multiplied by the square label correlation. As forward temporal edges do not exist for t = T and backwards temporal edges do not exist for t = 1, the matrix MT (α, η) ∈ R 3T×3T the vector g (r) to be an approximate eigenvector of B ξ,h . By the de nition of g (r) , we nd that (B ξ,h g (r) ) ij = λ info,1 g (r+1) ij .

(C. )

We now analyze this expression exploiting the tree-like approximation elaborated in Appendix C. . Resuming from this approximation, the expectation of g (r)

ij may be written under the following form:

E[g (r) ij ] = 1 λ r info,1 cΦγξχ (r-1) s + φ i ηhχ (r-1) t σ j . (C. )
Here the rst addend is the contribution of the spatial children of j which are on average cΦ in number, and for each of them the weight of the connecting edge is equal to ξ while the correlation between the labels γ = E[σ j σ k ]. Each spatial child being at a distance r -1 from the target edges -themselves at a distance r from (jk) -contributes to the sum through a term which we denoted χ (r-1) s > 0. Similarly, the second addend is the contribution of the temporal children which are φ i = 2 in number if (ij) is a spatial edge or φ i = 1 if (ij) is a temporal edge; their own contribution is denoted χ (r-1) t > 0. The correlation of the labels of temporal children is equal to η and the weight of the edges is equal to h. Importantly note that, as a consequence of γ, ξ, η, h being assumed to be all positive, both χ where M ∞ (•, •) is the matrix introduced in Equation ( . ). For simplicity we will denote it as M ∞ . For, say, r ∼ log(n), χ (r)

s/t ≈ λ ↓ |•| 1 (M ∞ ) r v s/t
, where v = (v s , v t ) is the eigenvector associated to the eigenvalue of M ∞ of largest amplitude. Equation (C. ) can therefore be further approximated as

E[g (r) ij ] = λ ↓ |•| 1 (M ∞ ) λ info,1 r (cΦλξv s + φ i ηhv t ) σ j + o λ ↓ |•| 1 (M ∞ ) λ info,1 r .
This expression naturally leads to the choice λ info,1 = λ

↓ |•| 1 (M ∞ ) for which E[g (r)
ij ] is independent of r, thus turning Equation (C. ) into an approximate eigenvector equation and λ info,1 into a close approximation of one of the real eigenvalues of B ξ,h .

We now extend this result to the case of nite T, and bring further conclusion on all the eigenvalues of B ξ,h belonging to the informative family.

The case of nite T

As we discussed already along Appendix C. , the case of nite T introduces further di culties due to the time-boundaries t = 1 and t = T. This being accounted for, when analyzing the contribution of each edge, not only we have to distinguish between spatial and temporal edges, but also to specify the time at which the edge lives. More precisely, suppose that j ∈ V t for 1 ≤ t ≤ T. We can rewrite Equation (C. ) as f ,t are respectively the contributions to the of a spatial, a backwards temporal and a forward temporal child of a node j ∈ V t . The relation between all the χ's directly unfolds from the branching process at nite T that we already discussed in Appendix C. . More precisely, let where M T (•, •) is the matrix appearing in Claim . . Following the argument we just detailed for T → ∞, we then get that the largest eigenvalue of the informative family is asymptotically close to λ info,1 = λ

E[g (r) ij ] = 1 λ info,
χ (r) = {χ
↓ |•| 1 M T √ cΦλξ, ηh .
This analysis also allows us to describe the subsequent eigenvalues λ info,i≥2 belonging to the informative family that have a smaller modulus. These modes are metastable con gurations of the branching process as in con guration of Figure . . In these modes, nodes belonging to di erent communities are still distinguished (hence the reason why these modes are informative), but the class identi cation σ x may be reversed across time. This results in a state in which neighbours are more likely to change label than to keep it, hence they have negative label correlation and lead to negative values of χ. This means to relax the constraint χ > 0 and thus no longer looking for the leading eigenvalue of M T . From this intuition we argue that the subsequent informative eigenvalues of B ξ,h coincide with the subsequent eigenvalues of M T √ cΦγξ, ηh which, as we already commented in the main text, are not necessarily real.

We now proceed extending our arguments to the uninformative family of isolated eigenvalues of B ξ,h . . . As in the static case, not all stable con gurations of the branching process of Appendix C. are informative. In particular, two nodes of G might be considered to belong to the same community only because they live at the same time. Based on the technique detailed in Section C. . , we now describe the position of the eigenvalues forming the uninformative family. Although these eigenvalues are not informative, the awareness of their presence is crucial if one has to avoid to mistakenly use one of these for community reconstruction.

We proceed again by studying the largest of these eigenvalues (which is also the largest eigenvalue of B ξ,h ), to then extended the result to all the others. Let us denote {λ uninfo,i } i=1,...,T this second set of (trivial and noninformative (C. )

According to this expression, we set σ x = 1 for all nodes and thus the correlation between σ x and σ y is always unitary. Following the argument developed to determine the value of λ info,1 , we then obtain

λ uninfo,1 = λ ↓ |•| 1 M T cΦξ, √ h . (C. )
As in Section C. . , this eigenvalue is necessarily real and the subsequent eigenvalues of the uninformative family are given by the subsequent eigenvalues of M T √ cΦξ, √ h and can be complex. We underline once again that the ordering of {λ info,i } i≥1 and {λ uninfo,i } i≥1 is not known a priori. So far we determined the position of the isolated eigenvalues under the assumption that the expectation of the approximate eigenvectors are signi cant. In order to know when this analysis holds, we have to study the variance of the entries of the approximate eigenvectors and see under what conditions it vanishes. This analysis will also allow us to determine the value of the radius of the bulk of B ξ,h .

. . B ξ ,h

To begin with, we investigate under which conditions the approximate eigenvector Equations (C. , C. ) hold. We then proceed with a study of the variance of g (r) ij ). When the variance vanishes, the eigenvector is well approximated by its expectation and we conjecture it is isolated. On the contrary, when the variance diverges it is because it gets asymptotically close to the bulk of uninformative eigenvalues and is no longer isolated.

Let us rst consider the eigenvector attached to λ info,1 : E g , where we exploited the fact that the paths (jk → wl) and (jk → vl) are asymptotically independent and that the number of paths leading to nodes a distance r from (jk) is exponentially large in r, unlike the number of paths leading to (vy) from (jk). We thus obtain that the variance V[g (r) ij ] of g (r) ij grows as

V g (r) ij = O   λ ↓ |•| 1 M T ( cΦξ 2 , h) λ 2 info,1   r . (C. )
As a consequence, the variance of g (r)

ij vanishes if and only if

λ info,1 > λ ↓ |•| 1 M T ( cΦξ 2 , h) .
Considering now the problem of evaluating the variance for all the {λ info,i } i≥1 and {λ uninfo,i } i≥1 , note that, the variance is only determined by the rst addend of Equation (C. ). This term does not depend on the con guration σ and is, therefore, the same for all the isolated eigenvectors. Consequently, for all the isolated eigenvectors, the variance vanishes if the corresponding eigenvalue is greater than L ξ,h = λ

↓ |•| 1
M T ( cΦξ 2 , h) , which is precisely the radius of the bulk of B ξ,h , since an informative eigenvalueeigenvector pair (λ info,i , g i ), (resp. (λ uninfo,i , b i )), for B ξ,h can only exist provided that λ info,i (resp. λ uninfo,i ) is greater than L ξ,h , hence concluding the justi cation of Claim . . . This section compares numerically the performance of Algorithm . against the main spectral methods commented along Chapter . In Figure C. the algorithms are tested for a di erent number of classes, value of η and degree distribution. For k > 2 a symmetric setting with classes of equal size and C ab = c out for all a = b is considered, so that the spectral algorithm of [Gha+ ] is still well de ned. Figure C. indeed con rms that Algorithm . (i) bene ts from high label persistence η; (ii) systematically outperforms the two considered competing dynamical sparse spectral algorithms [KV ], [Gha+ ]; (iii) is capable of handling an arbitrary degree distribution.

To compare the performance of Algorithm . and the static Bethe-Hessian of [DCT ], the case of small and large values of α should be treated separately. Close to the transition, Algorithm . improves over the static Bethe-Hessian and this gets more evident as η increases: the joint solution of the problem at all times allows one to improve the clustering performance in the hard detection regime. For large values of α, instead, there seems to exist α * (η) beyond which regularity only marginally improves the detection performance and Algorithm . performs equally (or slightly worse) than the static algorithm of [DCT ]. Here, Algorithm . su ers the suboptimal choices commented in Section . made to obtain a practical algorithm achieving non-trivial reconstruction when close to α c (T, η). On the opposite, the static Bethe-Hessian of [DCT ] is explicitly designed to optimally perform community detection for all values of α and any degree distribution, thereby justifying the two curves for large values of α.

A last remark concerns the capability of Algorithm . to recover communities of unequal sizes. Figure C. shows the accuracy of reconstruction of two communities of di erent size, as a function of the size of the smallest cluster over the size of the biggest. In order to obtain comparable results for di erent values of the ratio of the sizes of the two clusters, the following strategy is adopted: let Π ∈ R 2×2 be the diagonal matrix de ned so that Π ii is the fraction of nodes belonging to class i (Tr(Π) = 1), CΠ1 2 = c1 n . For a given ratio Π 11 /Π 22 , the matrix C is constructed so to let the leading eigenvalue of CΠ equal to c, and the second eigenvalue (which controls the hardness of CD) equal to a xed value. The overlap (averaged over time) is then evaluated independently over the large and small class, to keep this measure meaningful: in the case |V small | |V large |, assigning all nodes to the same cluster would output a large overlap.

Figure

  Figure : A toy sketch of the steps of feature extraction and clustering. The blue arrows indicate the feature extraction step, while the black circles group together nearby points in the feature space.

Figure :

 : Figure :Left: a graph with two communities, highlighted with a di erent colour code. Right: a dimensional node embedding of the graph on the left.

Figure :

 : Figure : Three clouds of points in a 2-dimensional space. The colours indicatethe ground-truth labels. The histograms represent the densities of points along the two directions. The dash-dotted black lines correspond to the minima of the histograms and allow one to cluster the input data into k = 3 groups using a DBSCAN-type algorithm.

Figure . :

 . Figure . : Left: an example of a graph with n = 12 nodes. Each red dot indicates a node, while each black line represents an edge. Right: major highways in the USA, taken from [Bar ].

Figure . :

 . Figure . : Left: an example of a directed graph with n = 12 nodes. Each red dot indicates a node, while each black arrow represents an edge together with the direction in which the edge is pointing. Right: a toy example of a multiplexed network with two layers indicated by di erent colours.

Figure . :

 . Figure . : Empirical degree distribution P(k). A: actor collaboration graph. B: The World Wide Web. C: Power grid network. Picture taken from [BA ].

  or vertices) and E d the set of edges connecting the nodes. For all e ∈ E d a weight ω e ∈ R can be associated. The set of weights is W = {ω e } e∈E d .

Figure . :

 . Figure . : Left: an undirected graph G(V, E ) with n = 5 nodes. Right: the adjacency matrix A corresponding to G(V, E ).

  Figure . : Size of the largest connected component divided by n as function of cfor di erent ER-generated random graphs. The black vertical line is at c = 1. For this simulation, n = 10 6 and no averaging has been taken.

  Figure . : a: the dolphin network [Lus ] with the colour code indicating class membership. b: the adjacency matrix of a. c: a realization of the SBM (De nition . ) with k = 4 communities with colour code indicating the class membership. d: the adjacency matrix of c.

  i

Figure

  Figure . : A toy example of a Poisson GW tree with a two class community structure. The color code indicates the class label assignment.

.

  The Boltzmann distribution . . . . . . . . . . . . . . . . . . . . The Ising model . . . . . . . . . . . . . . . . . . . . . . . . . . Tree-like approximations . . . . . . . . . . . . . . . . . . . . . . Belief propagation . . . . . . . . . . . . . . . . . . . . . The Bethe approximation . . . . . . . . . . . . . . .

Figure . :

 . Figure . : Saturation magnetization of iron (Fe), cobalt (Co) and nickel (Ni) as a function of the temperature. Picture taken from [CG ].

Figure

  Figure . : A toy model of the Ising model on a graph. The arrows indicate the spins orientation.

Figure . :

 . Figure . : Sketch of a tree. The node i in red, while in green, blue and orange the edges and nodes E ix , V ix with x = j, k, l, respectively.

  Figure . : Average magnetization as a function of βJ for a Ising model on a graph in which J = JA and A is the adjacency matrix of a ER graph with n = 30 000 and c = 5. The red curve is the value of m obtained from BP, while the black dots using a Monte Carlo Markov chain simulation. The vertical line is the theoretical prediction of the phase transition.

Figure . :

 . Figure . : Left: histogram of the eigenvalues of the matrix H β,J (with a zoom on the smallest) for β < β c (on top) and β = β c on the bottom. The vertical line is at λ = 0. Right: scatter plot of the real and imaginary part of the eigenvalues of BU β,J , highlighting in red the largest for β < β c (on top) and β = β c on the bottom. The vertical line is at Re(λ) = 1. For this simulation, J = A, where A is the adjacency matrix of an ER random graph with n = 3 500 and c = 5.

Figure . :

 . Figure . : Spectrum of the adjacency matrix of a ER random graph with n = 800 and c = 50, averaged over 10 realizations showing the empirical (in red) and limiting (in blue) distributions.

Figure . :

 . Figure . : Scatter plot in blue of the real vs. imaginary part of the eigenvalues of B of Equation ( . ). For this simulation, n = 3 500, k = 2, c = 5, θ = 1 n , c out = 0.5. The orange lines are the theoretical prediction of Claim . .

Figure . :

 . Figure . : Scatter plot in blue of the real vs. imaginary part of the eigenvalues of B of Equation ( . ). For this simulation, n = 3 500, k = 3, c = 5, θ i ∼ [U (3, 10)] 3 . The orange lines are the theoretical prediction corresponding to λ ↓ |•| 1≤i≤k (CΠ)Φ (vertical lines) and √ cΦ (circle).

Figure . :

 . Figure . : Scatter plot in blue of the real vs. imaginary part of the eigenvalues of B of Equation ( . ). For this simulation, n = 3 500, k = 3, c = 80, θ i ∼ [U (3, 10)] 3 . The orange lines are the theoretical prediction corresponding to λ ↓ |•| 1≤i≤k (CΠ)Φ (vertical lines) and √ cΦ (circle).

Figure . :

 . Figure . : Histogram of the eigenvalues of the Bethe-Hessian matrix H r in greenfor r = √ c on a SBM random graph with expected average degree c = 5, k = 3 and n = 5 000. In red, the theoretical curve of the bulk eigenvalue limiting distribution as in[SKZ ].

  and B Claim . (Isolated eigenvalues of B and H r ). Let B and H r be the nonbacktracking and Bethe-Hessian matrices of a realization of a k-class DCSBM satisfying Assumption . . Then, the k leading eigenvalues of B satisfy

.

  Community detection as an optimization problem . . . . . . . Inference in the DCSBM . . . . . . . . . . . . . . . . . . . . . . Basic properties . . . . . . . . . . . . . . . . . . . . . . . Detectability threshold . . . . . . . . . . . . . . . . . . . Optimization vs model based: a statistical physics perspective . . . . . . . . . . . . . . . . . . . . . . . . . . Spectral clustering: related works . . . . . . . . . . . . . . . . . Classical methods . . . . . . . . . . . . . . . . . . . . . . Spectral clustering in sparse graphs . . . . . . . . . . . Contributions . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure . :

 . Figure . : Schematic representation of the phase transition in the sparse SBM. Picture adapted from [Moo , Figure ].

  Figure . : Empirical eigenvalue distribution of L sym on SBM-generated graphs with k = 2. Top: dense regime, c in = 0.06n, c out = 0.02n. Bottom: sparse regime, c in = 6, c out = 2. For both simulations n = 5 000.

Figure

  Figure . : Eigenvectors of L rw = D -1 A on the (symmetrized) political blogs network [AG ]. Left: eigenvector associated to the second largest eigenvalue; Right: eigenvector associated to the third largest eigenvalue. The background color indicates the ground-truth labels.

r

  its associated Bethe-Hessian matrix. Let p be The ζ p 's are well de ned on an arbitrary graph G(V, E )

  p≥2 exists: it is the giant component. In the following, abusing notation, we simply write ζ p instead of ζ (j) p to refer to the ζ's associated to this giant component. One has ζ 1 = 1 and, if it exists, ζ p≥2 veri es

Figure . :

 . Figure . : Left: spectrum of the matrix B on the complex plane. The blue dots represent the uninformative eigenvalues, the orange diamonds the real eigenvalues outside the bulk and the maroon squares the real eigenvalues inside the bulk. The lines indicate the theoretical values. For this network n = 5 000, k = 3, θ i ∼ [U (3, 10)] 3 , c = 8. The o -diagonal elements of C are distributed according to C p>q ∼ N (c out , c out /k) with c out = 4 and the diagonal elements are then xed to have CΠ1 k = c1 k . The element of π are distributed as π i ∼ N (1/k, 1/2k). The entries are then rescaled sot that Tr(Π) = 1. Right: position of the rst four real eigenvalues of B as a function of α. For this simulation n = 5 000, k = 2, θ i ∼ [U (3, 10)] 3 , c = 6.

Figure . :

 . Figure . : Spectrum of the matrix H r for r = 3/2 √ cΦ. On the x axis the eigenvalues, on the y axis the respective histogram. In green the bulk of uninformative eigenvalues and in red the two isolated eigenvalues -zoomed to simplify the readability -. The two drawings give a sketchy representation of the ferromagnetic ground state (all black spins) and of the informative state (half black and half white). For this network, n = 5 000, k = 2, π = 1 k /2, c in = 8, c out = 3, r = 6.4, θ i ∼ [U (3, 5)] 4 .

βFigure . :

 . Figure . : Behavior of λ ↑ p (H r ) as a function of r for 1 ≤ p ≤ k in solid lines, and for p = k + 1 in dotted line. The vertical lines are the theoretical values of c/λ ↓ p (CΠ)(dashed dotted line), of √ cΦ (dotted line) and of λ ↓ p (CΠ)Φ (dashed lines). For this simulation, n = 50 000, c = 6, c out = 2, k = 4, C p>q ∼ N (c out , 2c out /k) and π ∝ 1 k .

Figure

  Figure . : Informative eigenvector x 2 of the Bethe-Hessian matrix for r = √ cΦ (top) and r = ζ 2 (bottom). On the right, the corresponding histogram. For this plot, n = 10 000, k= 2, π = 1 2 /2, c in = 14, c out = 2, θ i ∼ [U (3, 10)] 6 . Only the giant component of G(V, E ) is considered.The background colours refer to the ground-truth labels, while the markersize is proportional to 1/ √ d i , where d i is the degree of node i.

Figure . :

 . Figure . : Overlap comparison as a function of the hardness α of the detectionproblem. Blue sharp diamonds are the result given by Algorithm . , red diamonds the algorithm using the Bethe-Hessian of[SKZ ], green squares are for the non-backtracking of[Krz+ ], yellow pentagons the adjacency matrix as in[LR+ ], purple stars the random walk Laplacian, black dots are the algorithm of[QR ], the pink triangles are the Bayes optimal solutions obtained with BP[Dec+ ]. For both graphs n = 50 000,θ i ∼ [U (3, 10)] 4 , π ∝ 1 k , c = 5, C = c out 1 k 1 T k + (c inc out )I k .Averages over 10 samples.

  Figure . : Overlap comparison as a function of α, using the second smallest eigenvector of H r , for di erent values of r. In colour code the values of r ranging from r = 1 (blue) to r = cΦ (yellow). The red squares indicate r = (c inc out )Φ/2, that is equivalent to clustering with the matrix B [Krz+ ],the purple hexagons represent the Bethe-Hessian of [SKZ ], the green diamonds are the proposed Algorithm . and the blue crosses are the graph Laplacian (r = 1). For these simulations, n = 20 000, c = 8, θ i ∼ [U (3, 10)] 4 . Average over 10 realizations.

Figure . :

 . Figure . : Plot of a SBM graph with n = 5 000, k = 4 communities and C as in Equation ( . ). The colours are attributed according to the output of Algorithm . , adapted to operate with negative values of r, while the nodes positions are attributed according to the ground truth labels.

.

  Main result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Supporting arguments . . . . . . . . . . . . . . . . . . . . . . . . Technical Lemmas . . . . . . . . . . . . . . . . . . . . . Informal proof of Claim . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure . :

 . Figure . : Top: Spectrum of L sym , Bottom: spectrum of L sym τ

  . ) inducing a natural mapping between the Bethe-Hessian matrix at r = ζ p and the regularized random walk Laplacian at τ = ζ 2 p -1. In particular, the vector x p solution of Equation ( . ) coincides with the eigenvector of L rw ζ 2 p -1 associated with the eigenvalue 1/ζ p , thoroughly studied in Chapter . Note that L rw τ and L sym τ have the same eigenvalues and the eigenvectors are trivially related. Our main result states that the regularization τ = ζ 2 p -1 guarantees that the top p eigenvalues of L rw/sym τ are isolated down to the detectability threshold, i.e. as soon as α > α c . We formulate this result in the form of a claim. Claim . . Consider the graph G(V, E ), generated from a k-class DCSBM. Let ζ p for 2 ≤ p ≤ k be de ned as per Equation ( . ) with ζ 1 = 1 and τ

ζ 2 p

 2 -1 corresponds to the p-th largest and that it remains isolated (as is zero in the spectrum of H ζ p ). Figure . gives a visual representation of the spectrum of ζ p L sym ζ 2 p -1 , showing that, indeed, its p-th largest eigenvalue is asymptotically close to one. The eigenvector x p , associated to the p-th largest isolated eigenvalue of L rw τ is also the solution to H ζ p x p = 0 (from Equation ( . )) and its properties have been extensively discussed in Chapter . Intuitively, by picking τ away from ζ 2 p -1, the entries of the p-th eigenvector are likely to be more polluted by the degrees of the network. This suggests that large values of τ are likely to lead to sub-optimal partitions. Claim . further asserts that, for any τ in the interval [ζ 2 p -1, cΦ -1], the p dominant eigenvalues of L rw τ are isolated. Since ζ p ≤ ζ k ≤ √ cΦ, regardless of the hardness of the detection problem, the k largest eigenvalues 1

Figure . :

 . Figure . : Spectrum of ζ p L sym ζ 2

.

  The fact that the matrix L rw -λ ↑ p (D-rA) has an eigenvalue equal to 1/r comes easily by construction of L rw -λ ↑ p (D-rA)

  Figure . : Clustering on the eigenvector associated to the second largest eigenvalue of L rw τ . The circles indicate values of τ ∈ [1, c 2 ]: the colour code and the marker size distinguish the values of τ. The squares are obtained for τ = c, while the orange line (with pentagons) is obtained for τ = ζ 2 2 -1. Superimposed to this line, the diamonds are the output of Algorithm . . For this simulation n = 50 000, k = 2, c = 10, θ i ∼ [U (3, 15)] 5 , π = 1 2 . Averages are taken over 5 realizations.

  Figure . : Plot of the f r t (r ) (in purple). The blue continuous line indicates the value of r t , the red dashed dotted line the theoretical value of ζ p = c/λ ↓ p (CΠ) for the DCSBM. The two dashed black lines are the two roots of f r t (r ), the smaller of which is r t+1 . For this simulation n = 50 000, k = 2, c = 5, c out = 2.5, r = √ cΦ, θ i ∼ [U (3, 10)] 3 , π ∝ 1 k .

  end return ζ = {ζ 1 , . . . , ζ k }; end with respect to the greedy line-search to obtain the same accuracy. The total complexity of the algorithm needed to compute the vector ζ = (ζ 1 , . . . , ζ k ) T , scales as O(nck 3 ).

  Figure . : (Top: DCSBM setting) Exact value of c/λ ↓ p (CΠ) (p = 2 on the left and p = 3 on the right) in blue, compared to the estimate of ζ p obtained using the dominant eigenvalues of B (green) and its direct computation using Algorithm . (red), for problems of di erent hardness. For this simulation, n = 50 000, k = 3, θ i ∼ [U (3, 10)] 3 , c = 10. For each point π i ∼ N (1/k, 1/2k) and the o -diagonal elements of C are distributed as ∼ N (c out , c out /k) and c out : 0.1 → 7.5. (Bottom: real networks) Estimate of the values of ζ p as function of p, computed as λ

  Figure . : Comparison of the overlap (left) and modularity (right) resulting from the label estimation of Algorithm . (blue diamonds) and Louvain algorithm (orange crosses) applied on the giant component of G, as function of the detection hardness. For this simulation, n = 50 000, k = 2, π ∝ 1 2 , c = 8. Averages are taken over 10 samples.

.

  Adjacency-based algorithms . . . . . . . . . . . . . . . . . . . Laplacian-based algorithms . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . This chapter serves as a conclusion to Part II which studied SC for community detection (CD), proposing a novel algorithm. The two main contributions considered so far concern a theoretical analysis of Algorithm . on

Figure . :

 . Figure . : Overlap comparison as a function of the hardness of the detection of DCSBM-generated graphs with n = 10 000, c = 200, k = 2, θ i ∼ [U (3, 10)] 4. Yellow stars correspond to SC as per[LR+ ], red squares as per[Jin+ ], green triangles as per[New ] and the blue diamonds are the result of Algorithm . . Left: dense regime, c = 200. Right: sparse regime, c = 8. Averages are taken over 10 samples.

  Figure . : Overlap comparison as a function of the hardness of the detection of DCSBM-generated graphs with n = 10 000, c = 200, k = 2, θ i ∼ [U (3, 10)] 4. Green dots correspond to SC as per[Fie ], light-blue diamonds as per[SM ] and red dots as per[Jin+ ]. Averages are taken over 10 samples.

  A L I Z AT I O N S This part of the manuscript presents the extensions of our works on spectral clustering for community detection, presented in Part II. First, in Chapter we consider the setting of dynamical community detection on graphs generated from the dynamical degree corrected stochastic block model, proposing a novel spectral algorithm. Then, in Chapter we treat the problem of i.i.d. sparsi cation of kernel matrices for spectral clustering. I N D Y N A M I C A L G R A P H S Abstract This chapter extends the results of Part II considering the problem of community detection (CD) in dynamical graphs in which the community structure evolves over time.

Figure . :

 . Figure . : Alluvial diagram mapping change in scienti c research in the years from 2001 to 2007. Each block corresponds to a eld and its height is proportional to the number of citations occurred in that eld. Source: doi.org/ . /journal.pone. .g .

Figure . :

 . Figure . : Alluvial diagram representing the dynamics of label vector obtained according to De nition . . In this example k = 4, T = 4. The rows (and the colour code) corresponds to the four classes, while the columns to the time snapshots.

  Figure . : Position of α c (T, η) as a function of η (x axis) and T (color code). The black and red solid lines correspond to the analytical values of α c (T, η), for T = 1 and T = ∞, respectively. The dashed dotted lines are the position of α c (T, η) computed numerically, and the thick solid pale lines are the analytical values of α c (T, η) for T ∈ {2, 3, 4}.

Figure . :

 . Figure . : Three successive snapshots of the a aggregate graph G(V, E ). Classes are emphasized by node colors and can evolve with time. Network edges, that change over time, are indicated in solid lines, while "temporal edges" in dashed lines connect each graph to its temporal neighbors. Nodes of a common time step are circled in orange.

Figure

  Figure . gives a visual representation of the graph G(V, E ).

Figure . :

 . Figure . : Sketch of the 4 stable modes for two communities and T = 2. In black we indicate the direction s i > 0, in white s i < 0. The red edges correspond to the frustrated edges connecting spins with opposite direction.

  Figure . : The eigenvalues of B ξ,h with largest real part, for n = 10 000, T = 5, η = 0.4, c = 10, c out = 4, θ i ∼ [U (3, 10)] 3 . Left: ξ = 0.2, h = 0.9. Right: ξ = 0.4, h = 0.7. The orange and black dashed lines are the theoretical positions of the eigenvalues forming the informative and uninformative families, respectively. The thickest of these lines correspond to the leading eigenvalue of the respective family. The imaginary eigenvalues are represented with circles in the complex plane. The solid black line is a part of the circle of radius L ξ,h .

Figure . :

 . Figure . : Left: spectrum in the complex plane of B γ,η The vertical line is at 1, while the circle has radius L γ,η . Right: histogram of the eigenvalues of H γ,η . The vertical line is at λ = 0. For both simulations, T = 2, η = 0.4, c = 6, c out = 1, θ = 1 n , n = 2 000.

Figure . :

 . Figure . : Left: spectrum in the complex plane of B γ d ,η The vertical line is at 1 and the circle has radius L γ d ,η = 1 In red, the 4 isolated eigenvalues of B γ d ,η are evidenced. Right: histogram of the eigenvalues of H γ d ,η . The vertical line is at λ = 0. For both simulations, T = 2, η = 0.4, c = 6, c out = 1, θ = 1 n , n = 2 000.

Figure

  Figure . : Overlap comparison at t = T for Algorithm . (on the left) vs. [KV ](on the right), in color gradient, for various detectability hardness levels α (x-axis) and label persistence η (y-axis); n = 10 000, T = 5, c = 10, θ = 1 n ; averaged over 4 samples.

Figure . :

 . Figure .: Left: mean overlap across all values of t, as a function of α, for Algorithm . , BP[Gha+ ], the dynamic adjacency matrix of[KV ] (dyn A), the dynamical non-backtracking of[Gha+ ] (dyn B and dyn B opt), the static Bethe-Hessian of[DCT ] (static BH) and the dynamical Louvain algorithm of[Muc+ ] (dyn Louvain); n = 5 000, T = 4, c = 6, η = 0.7, θ = 1 n ; averaged over 20 samples (3 for BP). For all plots, k = 2. Right: overlap comparison of Algorithm . , the static Bethe-Hessian of [DCT a] (StaticBH) and the reconstruction obtained using the eigenvector with zero eigenvalue of H γ,η (opt. BHdyn). For this simulation n = 25 000, k = 2, θ = 1 n , c = 6, T = 4, η = 0.9.Averages are taken over 10 samples.

Figure . compares the

  Figure . compares the overlap as a function of α and η for Algorithm . versus the adjacency averaging method of[KV ] (which we recall assumes

  Figure .: Modularity as a function of time for Algorithm . for η = 0.55, the dynamic adjacency matrix of[KV ] (dyn A), the dynamic Louvain algorithm[Muc+ ] (dyn Louvain) and the static Bethe-Hessian of[DCT ] (static BH). The graph {G t } t=1,...,T is obtained from the Primary school network[GBC ; Ste+ ] dataset, as in Section . . . For Algorithm . ,[DCT ] and[KV ], k = 10 is imposed.

  Figure . : Example of dimensionality reduction using the covariance matrix M Z = 1 d feat ZZ T . Two Gaussian isotropic blobs of n = 4 000 vectors in d feat = 3 000 with di erent means and identity covariance are considered. Top: on the left a scatter plot of two arbitrary chosen features in which the colour code shows the original label assignment; on the left the corresponding covariance matrix. Bottom: on the left the characteristic features after dimensionality reduction using the two leading eigenvectors of M Z ; on the right the corresponding covariance matrix.

Figure . :

 . Figure . : Phase diagram of the RBIM for J ij ∈ {-1, 1}. The x axis goes from 1 2

Figure . :

 . Figure . : Spectrum of the matrix B ω in the complex plane. The entries J ij are generated independently according to N (J 0 , ν 2 ). The edge weights are de ned asω ij = th(βJ ij ). Left: dense regime, n = 250, c = 2 log 2 (n), J 0 = 1, ν = 4, β = 1. Right: sparse regime, n = 3 000, c = 5, J 0 = 1, ν = 1, β = 10.For both plots, the dashed red line corresponds to cE[th(βJ)], the dash-dotted green line to E[th 2 (βJ)]/E[th(βJ)], while the orange continuous line is the circle in the complex plane centered at the origin and of radius cE[th 2 (βJ)].

Figure . :

 . Figure . : Left: spectrum of the matrix M(g) de ned in Equation (.) with g, one of the eigenvectors of B ω attached to a complex eigenvalue. The expression of F(g) used in this simulation can be found in Appendix A of[DCT b] Right: Spectrum of M 0 , de ned in Equation ( . ). The graph considered is the same for the two matrices, with n = 1 500, c = log 2 (n). The matrix W = th(βJ), with β = 1 and the entries J ij are i.i.d. normal variable with J 0 = 1 and ν = 3. The red dotted line is at cE[th(βJ)], the green dash-dotted line is the position of E[th 2 (βJ)]/E[th(βJ)], while the orange solid line is the circle in the

:Figure . :

 . Figure . : First row: spectrum of the matrix B ω in the complex plane for different values of β; Second row: histogram of the eigenvalues of H β,J (zoomed in on the smallest eigenvalues) for di erent values of β. First column: β = 0.5β F , paramagnetic phase; Second column: β = β F paramagnetic-ferromagnetic transition; Third column: β = β SG paramagnetic-spin glass phase transition; Fourth column: β = β N , Nishimori temperature.For all matrices, the same graph was used with n = 1 000, c = 10. The weights of the edges are ω ij = th(βJ ij ) for the di erent values of β just described and the J ij are drawn independently from a Gaussian distribution N (J 0 , ν), with J 0 = 1 and ν = 1.5. The orange lines vertical line is at x = 1, while the black vertical line in the second row is at x = 0.

Figure . :

 . Figure . : Left: computation of βN for di erent values of β N . The blue dots represent the ratio between βN , computed with Algorithm . and the analytical value of β N . The purple diamonds are the value of β SG /β N , while the orange line is at y = 1. For these plots, n = 10 000 and c = 5.The weights of the non-zero entries of J are distributed i.i.d. according to N (J 0 , ν 2 ) for J 0 ranging from J 0 = 0.5 to J 0 = 4 and ν = 3.5. Averages are taken over ten samples. Right: behavior of the two smallest eigenvalues of H β,J as a function of β. The solid line indicates the smallest eigenvalue, while the dotted line is the second smallest. The vertical lines are set at β F < β SG < β N . For this simulation, n = 30 000 and c = 10. The weights of the matrix J are distributed i.i.d. according to a N (J 0 , ν 2 ) with J 0 = 1 and ν = 1.5.

Figure . :

 . Figure.: Overlap performance as a function of β N /β SG and three di erent values of c. For β N < β SG inference is asymptotically unfeasible. Two classes of equal size are considered and the entries of J are generated independently according to a Gaussian with mean J 0 σ i σ j . In the examples, n = 30 000 and averages are over 10 samples.

Figure . :

 . Figure . : Left plot: example of random generations of the input GAN images. Right plots: overlap classi cation performance of 40 000 GAN images, as a function of the expected average underlying graph degree c. Here, we consider K ij ({z}) = 1 d feat z T i z j and we take either κ = d feat : all features of the images are kept, or κ = 20: on average, only √ κ/d feat features (d feat = 512) are used. Averages are taken over 10 realizations.

Figure :

 : Figure : Overlap performance as a function of the detection problem on DCSBMgenerated graphs comparing the normalization technique adopted prior to the k-means step. The green squares ("projection") are the normalization of the rows of X, while the green dots ("division") the division of the entries of the columns of X by the entries of its rst column. Left: overlap obtained by Algorithm . for M = A, n = 10 000, k = 2, c = 200, θ i ∼ [U (3, 10)] 4 . Right: overlap obtained by Algorithm . for M = L sym τ and τ equal to the average degree, n = 25 000, k = 2, c = 5, θ i ∼ [U (3, 10)] 4 . Both simulations are averaged over 10 realizations.

  [QR ] avec D τ = D + τI n pour τ égal au degré moyen du graphe. Cet algorithme est de facto l'état de l'art pour la détection de communautés avec des méthodes spectrales sur des graphes réels. Dans le chapitre , nous présentons nos travaux relatifs à L sym τ , commencés dans [DCT b] LD, Romain Couillet, Nicolas Tremblay: Optimal Laplacian regularization for sparse spectral community detection in ICASSP IEEE I montré qu'il existe une relation claire entre la matrice hessienne de Bethe H r et le Laplacien régularisé L sym τ .

LD,

  Romain Couillet, Nicolas Tremblay: Community detection in sparse timeevolving graphs with a dynamical Bethe-Hessian in une séquence d'instantanés de graphes à di érents moments avec l'hypothèse que les étiquettes des communautés peuvent changer dans le temps, tout en ayant une corrélation positive. Nous avons développé un nouvel algorithme spectral basé sur la matrice hessienne de Bethe dynamique, capable d'exploiter l'information provenant de ces corrélations et qui surpasse largement les concurrents de l'état de l'art. Les résultats de[DCT a] sont traités dans le chapitre et l'algorithme correspondant fait également partie du paquet CoDeBetHe.jl.

  Ensemble, ces travaux o rent une nouvelle vision et des algorithmes adaptatifs robustes, reliant la physique statistique et les méthodes spectrales les plus conventionnelles dans des graphes parcimonieux, hétérogènes et dynamiques. A P P E N D I C E S A D C S B M P E R C O L AT I O N T H R E S H O L D

S

  κ(x, y) f (y) dµ(y) = S x T Cy f (y) dµ(y) θe ) dν(θ) ≡ x T CΠω f , (A. )

  = 1 on a cycle. It is straightforward to check that on any d-regular graph, the vector 1 |E d | is the Perron-Frobenius eigenvector of B with eigenvalue equal to d -1. A cycle is a d-regular graph with d = 2, hence the result.

  T)b p ; b p = Π -1 a p Since CΠ1 = c1 k and 1 T k Π1 k = 1, then b k = 1 k . By introducing d p = Π 1/2 b p , we can write: Π 1/2 CΠ 1/2 c -Π 1/2 b k b T k Π 1/2 d p = λ ↓ p (T)d p Π 1/2 CΠ 1/2 c d k d T k d p = λ ↓ p (T)d pGiven that the d k are eigenvectors of a symmetric matrix, then d T p d q = δ pq , so, for p < kΠ 1/2 CΠ 1/2 c d p = λ ↓ p (T)d p CΠ c b p = λ ↓ p (T)b pFrom this last equation the result follows directly. C S P E C T R A L C L U S T E R I N G I N D Y N A M I C A L G R A P H S . T

t

  are positive as well. By recurrence, the values of χ

  (r)f ,t } t=1,...,T , then the following relation holds:χ (r) = M T cΦγξ, ηh χ (r-1) , (C. )

  ) eigenvalues. The approximate Perron-Frobenius eigenvector b ∈ R |E d | can be written as b : d(jk,wx)=r k =i W (jk)→(wx) .

  (jk)→(wx) + ∑ (vy) : d(jk,vy)=r (vy) =(wx),k =i σ x σ y W (jk)→(wx) W (jk)→(vy) cΦξ 2 , h) , this addend of diverges, and so does the variance of g (r)ij : in this case, g(r) cannot be an approximate eigenvector of B ξ,h . Consider next the second addend of Equation (C. ): : d(jk,vy)=r(vy) =(wx),k =i σ x σ y W (jk)→(wx) W (jk)→(vy) : d(jk,wx)=r l =i ∑ (vy) : d(jk,vy)=r (vy) =(wx),k =i E[σ j σ x W (jk)→(wx) • σ j σ y W (jk)→(vy) ] ≈ 1 λ info,1 ∑ (wx) : d(jk,wx)=r l =i E[σ j σ x W (jk)→(wx) ] ∑(vy) : d(jk,vy)=r (vy) =(wx),k =i E[σ j σ y W (jk)→(vy) ]

Figure

  Figure C. : Overlap comparison of Algorithm . , the dynamic adjacency matrix of [KV ] (Dyn A), the dynamic non-backtracking of [Gha+ ] (Dyn B) and the static Bethe-Hessian of [DCT ] (Static BH). The title of each row ant column indicates the values of η, k, Φ considered. For Φ = 1 a power law degree distribution is adopted. The vertical line indicates the position of α/α c (T, η) = 1. For all simulations: c = 6, n = 25 000, T = 4. Averages are taken over 10 samples.

  Figure C. : Overlap averaged over time achieved by Algorithm . on graphs with two communities of di erent size, as a function of the ratio of the size of the two communities. For this simulation n = 10 000, T = 5, c = 6, θ = 1 n , η = 0.7. The second largest eigenvalue of CΠ is xed to λ ↓ 2 (CΠ) = 4. Averages over 15 samples.

  SC. . Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . From networks to graphs . . . . . . . . . . . . . . . . . . . . . Random graphs . . . . . . . . . . . . . . . . . . . . . . . . . Main result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Supporting arguments . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . Numerical results on real graphs . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Adjacency-based algorithms . . . . . . . . . . . . . . . . . . . Laplacian-based algorithms . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . III . Community detection in dynamical graphs . . . . . . . . . . . Detectability threshold for nite T . . . . . . . . . . . . . . . . Main result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Algorithm and performance comparison . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . .

	C O N T E N T S
	Abstract	vi
	Resumé	viii
	I
		vii

. The Boltzmann distribution . . . . . . . . . . . . . . . . . . . . The Ising model . . . . . . . . . . . . . . . . . . . . . . . . . . Tree-like approximations . . . . . . . . . . . . . . . . . . . . . Basic de nitions . . . . . . . . . . . . . . . . . . . . . . . . . . The non-backtracking matrix . . . . . . . . . . . . . . . . . . . The Bethe-Hessian matrix . . . . . . . . . . . . . . . . . . . . Relations with the Ising model . . . . . . . . . . . . . . . . . II . Community detection as an optimization problem . . . . . . . Inference in the DCSBM . . . . . . . . . . . . . . . . . . . . . Spectral clustering: related works . . . . . . . . . . . . . . . . Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . Main result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Supporting arguments . . . . . . . . . . . . . . . . . . . . . . . Performance comparison . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Implementation details . . . . . . . . . . . . . . . . . . . . . ix . : . Correlation clustering . . . . . . . . . . . . . . . . . . . . . . . Basic properties of the random bond Ising model . . . . . . . . A relation between β N and the Bethe free energy . . . . . . . Algorithm and performance comparison . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . IV . De nition of ζ on the sparse DCSBM . . . . . . . . . . . . . . The eigenvalues of T . . . . . . . . . . . . . . . . . . . . . . . Detectability threshold at nite T . . . . . . . . . . . . . . . . Spectrum of B ξ,h . . . . . . . . . . . . . . . . . . . . . . . . . . Performance comparison . . . . . . . . . . . . . . . . . . . . x A C R O N Y M S SC spectral clustering CD community detection DCD dynamical community detection RMT random matrix theory ER Erdős-Rényi GW Galton-Watson SBM stochastic block model DSBM dynamical stochastic block model DCSBM degree corrected stochastic block model DDCSBM dynamical degree corrected stochastic block model NMF naïve mean eld BP belief propagation RBIM random bond Ising model xi S Y M B O L S Miscellaneous • The cardinality of set A is denoted with |A|.

  The size of these two neighbourhoods are called incoming degree, d in i = |∂ in i|, and outgoing degree, d out i = |∂ out i|. For undirected graphs ∂ in i = ∂ out i ≡ ∂i and the degree is de ned as d i = |∂i|.

  For a more formal statement of the concept of local weak convergence, refer to[Sal ; DM+ a]. For a sparse ER random graph, with c = O Poisson GW tree. In red the root i, in blue the rst generation of nodes, in green the second.Property . (Convergence to Poisson Galton-Watson tree of ER).A sparse ER random graph rooted at i with c = O n (1) converges locally to a Poisson GW tree so obtained: consider the node i as the root and generate d i neighbours (called sons), where d i is a Poisson random variable with parameter c and iteratively repeat the operation for each son, creating the following generations.

n (1), the law of a uniformly randomly rooted ER converges locally to the Poisson Galton-Watson (GW) distribution with parameter c.

. i Figure . : A toy example of a

  ∈ R |E d |×|E d | has entries U β,J e,e = δ e,e th(βJ e ) and B is the non-backtracking matrix of De nition . . The critical (inverse) temperature β c determining the instability of the paramagnetic phase is

	(•)
	ij = ∑	th(βJ jl ) jl + h.o.t.,
	l∈∂j\i
	where h.o.t. denotes the higher order terms in • . This equation can be more
	succinctly written as
	BU β,J = + h.o.t.,	( . )
	where U β,J

  . (Spectrum of B on 2-class sparse SBM). Let G(V, E ) be a realization of a 2-class SBM as in De nition . with π 1 = π 2 (the two classes have asymptotically equal size) and C i , j = c in if i = j and equals c out otherwise. Denote with c = (c in + c out )/2 = O n (1) the expected average degree and suppose (c inc out )/2 >

	√	c. Then, for all large n, with high probability, the
	two largest eigenvalues of B in modulus are real, isolated and satisfy

  Find the k largest (or smallest) eigenvalues of M and stack the corresponding eigenvectors in the columns of a matrix X ∈ R n×k ; (Optional) Normalize the rows of X ; Estimate community labels ˆ as the output of k-means performed on the points in R k de ned by the rows of X ; Lloyd algorithm for k-means[Llo ] Input : Input data matrix X ∈ R n×d emb ; k: number of clusters Output : Label assignment ˆ ∈ {1, . . . , k} n begin Initialize the vector ; Set t ← 0 and (t) ← ;

	Algorithm . : while Convergence is not met do
	µ	(t)
	return ˆ .
	end

"Algorithm 0" of SC Secondly, Algorithm . relies on the fact that some of the eigenvectors of an appropriate graph matrix representation M are informative with re-Algorithm . : Typical SC algorithm on a graph with k classes Input : G(V, E ), k Output : Estimated label community vector ˆ . begin Choose the matrix M ∈ R n×n , a suited graph representation;

a as in Equation ( . ) for 1 ≤ a ≤ k;

  but the limit is never reached. and p = 5 in dotted line. In this instance, ζ 5 does not exist (and all subsequent ζ p 's do not exist either), ζ 1 = 1, and ζ 2,3,4 exist and lie in (1, 2).

  has unit multiplicity, so that ζ p > 1 + o n (1) for all p > 1. Along with Equation ( . ), we thus have 1 + o n (1) < ζ p ≤ √ cΦ + o n (1), and thus ζ p is asymptotically con ned inside the bulk of radius √ cΦ + o n (1) Figure . con rms that inside the disk of radius

	The ζ p 's are
	eigenvalues of B	of B.

  the bulk of non-informative eigenvalues. At their point of escape, they successively shift until they cross zero: this is where the successive values ζ 2 ≤ ... ≤ ζ k are de ned. This in particu-

	ζ p is an isolated
	eigenvalue of B and
	0 is an isolated
	eigenvalue of H ζ

p lar implies that the p-th smallest eigenvalue of H ζ p coincides with (i) the null eigenvalue of H ζ p , as well as with (ii) its largest isolated (so the last informative) eigenvalue. This allows us to rede ne ζ p as in Equation ( . ) of Claim . . Note in passing that, letting r further increase beyond ζ k , the left edge of the non-informative bulk of H r progressively shifts back (from positive values

  Proof of Lemma . . Consider the values {ζ p } 2≤p≤k as de ned in Eq.( . ). As G(V, E ) is generated from a DCSBM we know from Claim . that these k -1 values of ζ p exist with high probability. De ne ζ k+1 = √ cΦ and x p an integer such that 2 ≤ p ≤ k. Let us write once more λ p

  The rst part of Claim . asserts that for ζ 2 p -1 ≤ τ ≤ cΦ -1 the p largest eigenvalues of the matrix L rw τ are isolated. According to Claim . , for ζ p ≤ r ≤ √ cΦ the eigenvalue λ p (r) is isolated. Thanks to Lemma . , since λ p (ζ p ) < 0 we have: ∂ r λ p (r) < 0 for all r in the interval ζ p ≤ r ≤ -rA) < 0 for all r in the interval ζ p ≤ r ≤ √ cΦ. The hypotheses of Lemma . and Lemma . are therefore veri ed. We can then assert that the eigenvalue 1/r is the p-th largest of the matrix L rw

	√	cΦ,
	implying λ ↑
	-λ ↑ p (D-rA)

p (D

  Spectral CD in sparse and heterogeneous graphs Input : Adjancency matrix of undirected, connected graph G(V, E ) Output : Estimated number of communities k, estimated label community vector ˆ . •,p ← x p ∈ R n , where x p is the eigenvector with eigenvalue λ ↑ p (H ζ p ) = 0; end Normalize the rows of X i,• ← X i,• / X i,• ; Estimate community labels ˆ as output of k-class k-means on the rows of X;

	begin
	k ← estimate_number_of_classes (Algorithm . );
	ζ ← compute_ζ (Algorithm . );
	Initialize X ∈ R n× k;
	for p = 1 : k do
	X
	Each connected
	component of the
	graph is treated
	independently

contained within the giant component. In practice, communities may live in disconnected subgraphs (in a two-class DCSBM, this would correspond to Algorithm . :

  For this reason, it is best to compute directly the leading eigenvalue. This can be done e ciently, because all eigenvalues of B ∈ {0, 1} |E d |×|E d | di erent from ±1 are also eigenvalues of the smaller matrix B ∈ R 2n×2n [Krz+ ; CZ , for instance], This leads us to the following estimator of k, adopted in the remainder

	B =	A I n -D 0 I n	( . )
	so all computation involving the eigenvalues of B can be performed on the
	matrix B . Computing the p largest eigenvalues of a matrix with m non-
	zero elements costs O(mp 2 ) with state-of-the-art methods such as restarted
	Arnoldi methods [Saa ], hence λ	↓ |•| 1 (B) can be e ciently computed in O(cn).
	Estimator of the
	number of
	communities

  . . , such that r t → ζ p . The convergence is guaranteed when setting r 0 = ζ p+1 , under the convention ζ k+1 = √ ρ. The values of ζ p are then estimated from the largest, ζ k , to the smallest, ζ 2 . Let us now get to the mathematical details of this subroutine. We denote by X T r t = (x T 1 (r t ), . . . , x T p (r t )) T ∈ R n×p where x p (r t ) is the eigenvector of H r t corresponding to the p-th smallest eigenvalue, λ ↑ p (H r t ), while Λ r t = diag λ ↑ 1 (H r t ), . . . , λ ↑ p (H r t ) . For another value r = r t , and applying the Courant-Fischer theorem, we can write

  Table . reports the results ofQ Mod A , Q likelihood

A in terms of mean ± standard deviation.

Table . :

 . Log-likelihood and modularity comparison on real networks[LK ; Zac ; Lus+ ; GN ; AG ]. When the standard deviation is below 0.01 is not indicated.

	Vip	HepPh	Wikipedia	GNutella P P	Politicians	Power grid	GrQc	Facebook	Tv	Polblogs
							.			.	.			.			.
	.	.	.			.					.			,
	.	.	.	.			.	.	.					.
	. ± .	. ± .	. ± .	. ± .	.	.	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	.	.	. ± .	. ± .
			. ± .	. ± .	. ± .	. ± .	.	.			.	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	.	.
			.	. ± .	.	.	.	.			.	. ± .	.	. ± .	.	. ± .	.	. ± .	.	.
	. ± .	. ± .	. ± .	. ± .	.	. ± .	. ± .	.	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	.	.	.	. ± .
	. ± .	. ± .	. ± .	. ± .	.	.	.	.	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	.	.	.	. ± .
	. ± .	. ± .	. ± .	. ± .	. ± .	.	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	.	.	. ± .	. ± .
	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	.	.	. ± .	. ± .	.	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	. ± .	.	.	. ± .	. ± .
	. ± .	. ± .	. ± .	. ± .	.	.	.	.	. ± .	. ± .	. ± .	. ± .	.	. ± .	. ± .	. ± .	. ± .	. ± .	.	.	.	. ± .

. ± . . ± . . ± . . ± . . ± . . ± . . ± . . ± . . ± . . ± . . ± . . ± .

  CD, studied on graphs generated from the dynamical degree corrected stochastic block model (DDCSBM).

	In Part II the problem of community detection (CD) with spectral cluster-
	ing (SC) techniques has been deeply investigated on static graphs. The word
	"static" here simply refers to the absence of the notion of time. Taking a step
	back and referring to Chapter , we must recall that graphs have a scienti c
	interest for their ability to represent and model real world systems of inter-
	acting agents that are rarely static. As a consequence, graph representations

. Community detection in dynamical graphs . . . . . . . . . . . . The dynamical degree corrected stochastic block model . . Related works and contributions . . . . . . . . . . . . Detectability threshold for nite T . . . . . . . . . . . . . . . . Main result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The dynamical Bethe-Hessian matrix . . . . . . . . . . . Community detectability with the dynamic Bethe-Hessian . . . . . . . . . . . . . . . . . . . . . . . . . . Algorithm and performance comparison . . . . . . . . . . . . . Algorithm implementation . . . . . . . . . . . . . . . . . Performance comparison on synthetic datasets . . . . . Test on Sociopatterns Primary school . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  . ) where V t,a is the set of nodes in V t with label a. At time t = 1, by hypothesis, E[|V 1,a |] = nπ a . Plugging this relation in Equation ( . ), one obtains with a recursive argument that for all t, E[|V t,a |] = nπ a . Figure.shows the alluvial graphs of a typical realization of for k, T = 4. Given the generative model of the label vector, we now provide the de nition of the DDCSBM.De nition . (Dynamical degree corrected stochastic block model). Let T ∈ N be the number of instances of snapshots graph with n nodes. Denote with ∈ {1, . . . , k} nT the class label vector, generated according to De nition . , where k is the number of classes. Further let C ∈ R k×k be a symmetric matrix with positive elements. and θ ∈ Θ = [θ min , θ max ] be a ran-

	The dynamical
	degree corrected
	stochastic block
	model

dom variable that encodes the intrinsic node connectivity, distributed according to ν, satisfying Θ

  Algorithm . : The Nishimori-Bethe relation for sparse SC Input : Weighted adjacency matrix of a graph J ∈ R n×n , precision error ∈ R. Value of βN ∈ R + , estimated label vector σ ∈ {-1, 1} n . begin Shift the non-zero Jij as: Jij ← Jij -1 2|E | 1 T n J1 n ; Compute βN ← Compute_ βN (Algorithm . ) ; Compute H βN , J (Equation ( . )) ; Compute x ← the eigenvector associated to λ ↑ 1 (H βN , J ) ; Estimate σ as the output of 2-class k-means on the entries of x ;

	end
	return β t , σ.

Output :

  On pourrait même dire que le critère suivi pour catégoriser les objets fournit précisément une dé nition du concept abstrait lié aux catégories elles-mêmes[Hul ].Consciemment ou non, l'intelligence humaine permet à chacun d'entre nous d'e ectuer deux opérations de base. La première est d'identi er les caractéristiques informatives des données d'entrée, c'est-à-dire celles qui sont nécessaires et su santes pour leur attribuer une étiquette. Ceci est fait en fournissant une représentation de l'objet dans un espace paramétrique, de sorte que des objets similaires dans le monde réel correspondent à des points proches dans l'espace paramétrique. Cette étape est connue sous le nom d'extraction de caractéristiques. La deuxième étape consiste à attribuer une étiquette à chaque élément de la base des données, en fonction de sa position dans l'espace des caractéristiques. Selon le problème considéré, cela peut être fait principalement de deux manières. L'une est la classi cation: il s'agit d'attribuer chaque élément à l'un des groupes possibles; l'autre est le regroupement: dans ce cas, l'objectif est de diviser en groupes plusieurs éléments, en fonction de leur proximité.L'intérêt porté à l'extraction de caractéristiques et à l'étape suivante de clustering ou classi cation va au-delà de l'apprentissage humain et est à la base de nombreux algorithmes d'apprentissage automatique[MRT ].La classi cation est généralement réalisée à l'aide d'algorithmes supervisés, qui constituent une manière très intuitive (en principe) d'envisager l'apprentissage et sont aujourd'hui le plus déployés. Un grand ensemble de données dont les étiquettes sont connues est utilisé pour entraîner un algorithme à générer une règle capable de prédire les étiquettes. Une fois l'apprentissage terminé, la règle peut être utilisée pour classer des nouvelles données. Les réseaux neuronaux font partie de cette classe d'algorithmes. Ils sont nés pour reproduire le processus d'apprentissage qui se produit dans le cerveau humain et constituent aujourd'hui l'état de l'art dans la plupart des domaines en termes de performances. Cette approche présente toutefois des inconvénients: l'étape d'apprentissage peut être coûteuse en termes de temps de calcul, tandis que la création d'une base des données avec leurs étiquettes peut prendre beaucoup de temps et de capacité de stockage.

	P U B L I C AT I O N S
	. LD, Romain Couillet: Community detection in sparse realistic graphs:
	Improving the Bethe-Hessian in ICASSP IEEE I		C
		A	, S	S	P	(	)
	La catégorisation, c'est-à-dire la capacité de regrouper des objets, est pro-. LD, Romain Couillet, Nicolas Tremblay: Classi cation spectrale par la fondément ancrée dans l'intelligence humaine et joue un rôle prépondérant laplacienne déformée dans des graphes réalistes in XXVII GRETSI ( ) dans l'apprentissage.
	. LD, Romain Couillet, Nicolas Tremblay: Revisiting the Bethe-Hessian:
	Improved Community Detection in Sparse Heterogeneous Graphs in N	IPS
	A	N	I	P	S		(	)
	. LD, Romain Couillet, Nicolas Tremblay: Optimal Laplacian regulariza-
	tion for sparse spectral community detection in ICASSP IEEE I
		C	A	, S	S	P
	(	)
	. LD, Romain Couillet, Nicolas Tremblay: A uni ed framework for spec-
	tral clustering in sparse graphs accepted to J
	. LD, Romain Couillet, Nicolas Tremblay: Community detection in sparse
	time-evolving graphs with a dynamical Bethe-Hessian in N	IPS A
		N	I	P	S	(	)
	. LD, Romain Couillet, Nicolas Tremblay: Nishimori meets Bethe: a spec-
	tral method for node classi cation in sparse weighted graphs in J
	S	M	: T		E

The non-backtracking matrix of[Krz+ ] and the Bethe-Hessian of[SKZ ] were known already, but were not used forCD.

We consider k = 2 for simplicity, but we will show in the remainder that H r can be used for SC in the presence of k ≥ 2 classes.

If A as a homophilic relation with B and B with C, then it is likely that A has a homophilic relation with C. The expression weak transitivity is used to denote that the relationship is only likely to be homophilic, while transitivity would lead to an implication.

(P • E[W • W]) and r 0 be the number of eigenvalues of P • E[W] that are greater than max( √ ρ, L). The spectrum of the weighted

(B) is the largest r for which det[H r ] = 0. At r = λ ↓ |•| 1 (B),one eigenvalue of H r is equal to zero and it is necessarily the smallest one.

(T) = 0.

We are thus left to showing why speci cally the p-th smallest eigenvalue of H r for the particular choice of r = ζ p is of utmost importance, and why it corresponds to the null isolated eigenvalue of H ζ p .

The codes for the SBM are taken from the rst author of[Dec+ ] personal webpage and are adapted to the DCSBM setting modifying the eld term appearing in Equations ( . , . , . ).

In order to design their dynamical non-backtracking matrix, the average number of connections among nodes in the same and across communities must be known.

Note that α 2 = cΦγ, where γ = (c inc out )/(c in + c out ) is the label covariance of neighbouring nodes.

Algorithm . : Community Detection with the Bethe-Hessian

Input : adjacency matrix of undirected graph G(V, E ), number of classes k

Output : Estimated label community vector ˆ . begin Estimate c = 1 n ∑ i d i and Φ = 1 nc ∑ i d 2 i ; for p = 2 : k do ζ p ← r such that λ ↑ p (H r ) = 0 on 1 < r ≤ √ cΦ; X •,p ← x p such that H ζ p x p = 0; end Estimate community labels ˆ from the node embedding X = [X •,2 , . . . , X •,k ]; return ˆ . end Generally speaking, Algorithm . can be considered as a meta-algorithm that is written for user readability. Deeper considerations on how to eciently implement it on arbitrary graphs (hence beyond the DCSBM) assumption will be dedicated in Chapter .

In Figure . we compare the overlap performance (de ned in Equation .), achieved by Algorithm . versus competing methods for synthetic DCSBM graphs as a function of the hardness of the problem α. On the left hand-side of Figure . is depicted the case k = 2, while on the right k = 5. In both cases detection is asymptotically feasible only if α > α c .

In both cases, Algorithm . outperforms the competing spectral algorithms and has a very close performance to the Bayes optimal solution obtained with the BP algorithm of [Dec+ ]. As α → α c , Algorithm . , the algorithm of [SKZ ] based on H √ cΦ and the one of [Krz+ ] based on B give essentially the same result, which con rms that they are indeed equivalent at the phase transition. For easier detection problems though, except for the algorithm of [QR ] which is only slightly less accurate, the performance of all methods is largely improved by Algorithm . . Interestingly, note that

, giving an intuition on why the Bethe-Hessian method of [SKZ ] performs better than the non-backtracking approach of [Krz+ ]. As for the standard spectral clustering algorithm which exploits the dominant eigenvectors of A [LR+ ], it can only perform non trivial community reconstruction for α far beyond the threshold α = α c . The algorithm of [SM ], based on the random walk Laplacian L rw , is here incapable of making any non-trivial reconstruction for the considered set of parameters, suggesting that its dominant eigenvectors are not informative.

To further con rm our theoretical results, Figure . depicts the overlap of Algorithm . with M = H r for di erent values of r.

In Section . . we showed that the problem of Bayesian inference in a two class setting given a matrix J as in Equation ( . ) can be easily mapped to the RBIM that we extensively studied in Section . . This mapping is induced by a simple gauge transformation, that enables the use of Algorithm . to estimate β N directly from J. Recall that Jij = J ij σ i σ j , where σ i ∈ {±1} encodes the label information of node i. Let x ∈ R n be an eigenvector of H β, J with eigenvalue λ and let y have entries y i = x i σ i . Then

so that λ is an eigenvalue of H β,J with eigenvector y. Consequently, the smallest eigenvalue of H β N , J is asymptotically close to zero and Algorithm . can indeed be used to estimate β N .

O P T I M A L B E T H E -H E S S I A N

.

ζ

In this appendix we show that for a graph generated from the sparse DCSBM, the two following de nitions of ζ

p (where j is the giant component) are indeed equivalent with high probability for p ≥ 2 if ζ (j) p exists:

We will further discuss that, with high probability, ζ

p > 1 exists only for the giant component. This appendix is structured as follows. First we enunciate two lemmas and one corollary to state that when an arbitrary connected graph has at most one cycle, the spectrum of its associated non-backtracking matrix B does not have real eigenvalues that are larger than one in modulus.

We then proceed by arguing that with high probability, for large DCSBM graphs, the small connected components do not have more than one cycle, implying that all the real eigenvalues of B that are larger than one in modulus come from the giant component. Finally we show that this last statement implies that the two de nitions of ζ Let G(V, E ) be a tree. Then, all the eigenvalues of its associated non-backtracking matrix are equal to zero.

From Lemma B. we conclude that non-zero eigenvalues of B imply the existence of cycles in G(V, E ) which must exist to de ne the ζ p 's. The second lemma considers the e ect of adding a node to an arbitrary graph G(V, E ).

Lemma B. (Spectrum of G plus a node). Let G be a graph obtained by adding one node and one edge to the graph G, i.e. for i / ∈ V and an arbitrary j ∈ V, V = V ∪ {i} and E = E ∪ (ij). Then, all the non-zero eigenvalues of B(G ) are equal to the non-zero eigenvalues of B(G), where B(G) is the non-backtracking matrix of G.

ζ Applying recursively Lemma B. one easily obtains that, attaching a tree to a node of G(V, E ), all the non-zero eigenvalues of B are left unchanged. In simple words, the tree part of a graph is "invisible" to the spectrum of B and contributes to it with null eigenvalues. The last property is a Corollary of Lemma B. and shows that cycles lead to non-trivial eigenvalues in the spectrum of B. With the results of the former two Lemmas, we proceed to argue that all the eigenvalues of B that are larger than one in modulus come from the giant component. Let Λ(B) be the set of all eigenvalues of B. Since the graph is disconnected, Λ(B) = n cc j=1 Λ B (j) , i.e., each connected component contributes independently to the eigenvalues of B. The expected size of the small connected components grows as log(n) [BJR , Thm. . ]. We claim that this implies that the probability that a small connected component contains two or more cycles is o n (1), thus tending to zero as n grows to in nity. Applying Lemma B. (for zero cycles) and Corollary B. (for one cycle), we conclude that, with high probability all the real eigenvalues of B larger than one in modulus will come from the giant component. From Theorem . , for all the eigenvalues λ = ±1,

p≥2 we gave in Equation ( . ), we have that all the ζ (j) p > 1 are in the spectrum of B. We may conclude that, if ∃ j such that ζ (j) p≥2 > 1, then with high probability j is the giant component. We now conclude showing that this statement implies the equivalence between (B. ) and (B. ).

Let p > 1, then certainly ζ (j)

p > 1, if it exists. We denote with j the giant component and we want to prove that for r, p ≥ 2

) = 0 and so it is enough to show that p = q. We proceed with a proof by contradiction. Suppose that p = q, then there ex-

Applying Gershgorin circle theorem [JH ], it is easy to show that for r > d max -1, the matrix H r is positive de nite. Consequently, there exists r ≥ ζ

r ) = 0. From the Ihara-Bass formula, r is thus in Λ(B), so there is a real eigenvalue of B larger than one not coming from the giant component. This is in contradiction with what stated above, thus concluding our argument.

We now provide a proof of Lemma B. , B. and Corollary B. .

The notation d(ij, k ) = m indicates that there exists a non-backtracking path from the edge (ij) to the edge (k ) of length m. From a straightforward calculation one obtains

For all trees, for any two edges we have that d(ij, k ) ≤ n -1, so there is a value of m c ≤ n -1, which represents the maximal distance between any two directed edges, such that, for all vectors g,

This relation comes from the fact that, by de nition of m c , no two edges are at a distance equal to m c + 1, so {(k : d(ij, k ) = m c + 1} = ∅ for any edge (ij). Now, let us consider g to be an eigenvector of B, such that Bg = λg. Then we can write

Thus concluding that any eigenvector g of B is associated to eigenvalue zero. Note that in other words, this means that B is nilpotent.

Proof of Lemma B. . Let i be the newly added node and j the node in V to which i is attached. The matrix B(G ) can be written by adding to the matrix B(G) two rows and two columns corresponding to the directed edges (ij) and (ji). We introduce the notation

Its element-wise de nition reads for all (yx) ∈ E d as I (•j) yx = δ xj . Similarly, we de ne

The matrix B(G ) can be written as:

We now look for the non-zero eigenvalues of λ of B(G ). Consider λ ∈ C * . Using a block matrix determinantal formula one has:

A non-zero eigenvalue of B(G ) (cancelling the determinant) is thus necessarily also an eigenvalue of B(G), ending the proof.

Note that, since the rst and the last rows of MT (α, η) only have zero entries, MT (α, η) has the same non-zero eigenvalues as M T (α, η) of Claim . . This also implies that M T (α, η) shares the non-zero eigenvalues of a matrix of size (3T -2) × (3T -2) as conjectured in [Gha+ ].

.

B ξ ,h

In this appendix we provide support to Claim . , studying the spectrum of B ξ,h with a method that can be seen as a generalization of [Krz+ ], introduced in Section . . . We recall that this method consists in identifying a guess eigenvector of B ξ,h and determine the conditions under which its expectation is expected to be a good approximation of one the eigenvectors of the dynamical non-backtracking matrix. To do so, we rst determine the position of the eigenvalues belonging to the informative family (starting from the largest) and then of the uninformative family. Secondly, we analyze the variance of the expression of the expected eigenvector and see under what condition the expectation is meaningful. With this result we nally determine the value of L ξ,h (the radius of the bulk of B ξ,h ).

. .

In this section we determine the position of the informative eigenvalues of B ξ,h with modulus larger than L ξ,h . To do so, we rst study the largest of them in the limiting case T → ∞, to then extend our ndings for nite T to all other eigenvalues.

The limiting case of T → ∞ where {(jk) : d(jk, wx) = r} is the set of directed edges (jk) such that the shortest directed non-backtracking path connecting (jk) to (wx) is of length r, and where W (jk)→(wx) is the "total weight" of this shortest path de ned as the product of each edge weight ω ij , i.e,

The quantity σ x ∈ {±1} takes its value according to the label of node x. The value of λ info,1 appearing in Equation (C. ) will be chosen in order to enforce