Renaud Marlet

Pascal Monasse

Bertrand Neveu

Chaohui Wang

Nikos Komodakis

Guillaume Obozinski

Mathieu Aubry

Vincent Lepetit

Thomas Belos

Sophie Cohen-Bodénès

Francois Darmon

Martin De

La Gorce

Théo Deprelle

Rahima Djahel

Yuming Du

Benjamin Dubois

Laura Fernandez

Thibault Groueix

Mateusz Kozinski

Timothée Lacroix

Loïc Landrieu

PhD thesis Pierre-Alain Langlois

Zhe Liu

Francisco Vitor

Suzano Massa

Tom Monnier

Xuchong Qiu

Simon Roburin

Yohann Salaun

Othman Sbai

Martin Simonovsky

Raphael Sulzer

Maria Vakalopoulou

Andreas Damianou

Pablo G Moreno

Sungsoo Ahn

Akash Kumar

Aaron Klein

Kurt Cutajar

Alexandra Gessner

Borja Balle

Javier Gonzalez

Vicky Schneider

Cliff Mccollum

Neil Lawrence

Brigitte Mondou

Sylvie Cach

Anna Zaverio

Mariam Sidibe

etc. Jihed Hadroug

Introduction to Probabilistic Graphical Models

The completion of this thesis would not have been possible without the support and nurturing of an exhaustive

v Abstract

Probability theory offers a mathematical framework for quantifying the uncertainty about the model and the data, forms an underpinning of many machine learning systems. Comparing to deterministic models, the prediction of a probabilistic model takes the form of a probability distribution over all possible values. However, it may render the prediction computationally intractable. To overcome the computational issue, variational techniques are often employed, where variational bounds are used to approximate the original objectives, casting the original problem as an optimization problem.

The primary goal of this thesis is to explore better variational formulations for probabilistic graphical models and deep neural networks in terms of computational efficiency or sample efficiency.

For learning discrete graphical models, we propose a new variational inference algorithm for the dual problem of the maximum likelihood estimation, inspired by the recent advances of stochastic variance reduction algorithms. Roughly speaking, we obtain a concave optimization over a convex polytope, where each variable, corresponding to a clique in the graph, is subject to the simplex constraint; besides, the local consistency among cliques induces a set of equality constraints. We thus propose an augmented Lagrangian formulation to solve this optimization problem. Our algorithm, which can be interpreted as an inexact gradient descent for the outer loop, requires only a fixed number of inner block-coordinate updates to obtain a sufficiently good estimate of the gradient for the Lagrange multipliers. We prove that the proposed algorithm enjoys a linear convergence in both the dual and the primal, and compare it with state-of-the-art algorithms on three applications.

Probability theory has a natural application to neural networks if we consider the hidden activations and/or the network parameters as random variables. This type of model is generally known as the Bayesian neural network, and which is typically trained via the minimum description length (MDL) principle, corresponds to formulate supervised learning as lossless compression. We propose an alternative learning formulation based on the rate-distortion theory for lossy compression. In a nutshell, we introduce a mutual information regularizer to control the amount of information stored in the parameters with respect to any minibatch in the train-set, and derive a variational upper bound on the mutual information yielding an EM-like algorithm. Comparing to MDL with stochastic gradient descent, our algorithm is more stable and can potentially achieve a better testing accuracy.

The same formulation can be applied to meta-learning, where the EM-like algorithm can be derived from an empirical Bayes formulation and motivated vi from applying a hierarchical Bayes model to the dataset constituting a two-level hierarchy. Recall that the idea behind meta-learning is learning to generalize across tasks. The key difference is that we now restrain the mutual information between the parameters of the model and the dataset associated with a particular task, for which we take a variational upper bound and obtain a sum of local KL divergences between the variational posterior and the true posterior of each task. We derive a novel amortized variational inference that couples all the variational posteriors into a meta-model, which consists of a synthetic gradient network and an initialization network. Moreover, the combination of local KL divergences and synthetic gradient network allows backpropagating information from unlabeled data, thereby attaining better variational posteriors. We further provide a theoretical analysis to justify our formulation in terms of the generalization performance and demonstrate its superior performance on few-shot classification benchmarks.

The second part of this thesis is dedicated to the memory issue of overparameterized neural networks. To improve memory efficiency, we propose two ideas briefly sketched as follows. The first idea, which is inspired by the information bottleneck principle, makes use of the teacher-student framework, where an overparameterized teacher network is used as additional supervision for training the thinner student network. The supervision is imposed by maximizing the mutual information between selected pairs of activations, while the intractable mutual information is approximated by a variational lower bound. Another idea comes from an empirical observation that over-parameterized neural networks are robust to symmetry constraints on network parameters, which is embarrassingly simple, has a negligible effect on training and testing accuracy. Both ideas have been extensively tested on standard benchmarks and compared with state-of-the-art methods.

vii Résumé L'intelligence artificielle (IA), suggérée par son nom, est un type d'intelligence distincte de l'intelligence naturelle (IN) appartient à l'homme et à l'animal. Cependant, l'intelligence est elle-même une boîte de mystères. Nous ne sommes pas près de le comprendre de si tôt. Au fil des ans, les chercheurs en IA ont tenté de rendre un système d'IA impossible à distinguer de celui de son frère IN. Le test de Turing, conçu pour évaluer les systèmes d'IA proposés, incarne cette passion. À mesure que la recherche sur l'IA progresse, la définition de l'IA évolue également avec le temps. Le terme IA a été utilisé à l'origine pour désigner les systèmes informatiques imitant IN. Dans son utilisation moderne, IA se réfère également à l'ensemble du domaine qui étudie les machines intelligentes. En tant que domaine scientifique, IA est née à la conférence de Dartmouth en 1956 et est devenue un domaine interdisciplinaire associant informatique, statistiques, neurosciences, physique, etc. [START_REF] Jordan | Artificial intelligence: The revolution hasn't happened yet[END_REF] a précisé que l'essentiel de ce que l'on appelle l'IA aujourd'hui est ce qu'on appelle l'apprentissage automatique (ML) depuis plusieurs décennies. ML est un terme moins mystérieux, car il s'agit d'un domaine qui se concentre davantage sur les aspects techniques des systèmes d'intelligence artificielle.

Bien que la construction de systèmes d'intelligence artificielle ne nécessite pas d'ingénierie inverse, s'inspirer de IN peut apporter des idées prometteuses. La recherche en IA a depuis longtemps mis l'accent sur l'étude et la reproduction partielle ou totale du mécanisme de l'IN. Les méthodes induites sont parfois appelées méthodes inspirées du cerveau. Des exemples de ce type incluent divers modèles informatiques, appelés réseaux de neurones artificiels [START_REF] David E Rumelhart | Learning representations by back-propagating errors[END_REF][START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF][START_REF] Maass | Networks of spiking neurons: the third generation of neural network models[END_REF], qui imitent de manière lâche les neurones biologiques. Cette famille de méthodes et de modèles est maintenant largement développée et largement reconnue comme l'apprentissage profond [START_REF] Lecun | Deep learning[END_REF].

Comme [START_REF] Russell | Artificial Intelligence: A Modern Approach[END_REF] l'ont expliqué, des «avions artificiels» ont eu lieu lorsque les frères Wright et d'autres ont cessé d'imiter les oiseaux. Une autre philosophie consiste à ne pas imiter IN, qui découle de formulations mathématiques de machines à penser basées sur des concepts tels que la logique et la rationalité. Ces formulations peuvent remonter au syllogisme Aristote et à l'algèbre de Boole. Les exemples les plus représentatifs de ce type sont les méthodes logiques, telles que la logique propositionnelle logique du et la premier ordre. Contrairement aux réseaux de neurones artificiels, les modèles logiques sont plus intuitifs et transparents du point de vue humaine. Ils sont capables d'intégrer la connaissance du domaine sous la forme de formules logiques et de mener un raisonnement mathématiquement correct d'une manière compréhensible pour l'homme. Au viii milieu des années 1965, la recherche sur l'IA sur ce type de méthodes était perçue comme le paradigme dominant [START_REF] Haugeland | Artificial intelligence: The very idea[END_REF]), qui présente de nombreuses applications remarquables, telles que le «general problem solver» de [START_REF] Newell | Gps, a program that simulates human thought[END_REF] et le «advice taker» de [START_REF] Mccarthy | Programs with common sense[END_REF]. Ces réalisations font partie des raisons pour lesquelles IA a attiré une telle attention dans les années soixante.

Cependant, les modèles logiques classiques souffrent du soi-disant problème de qualification [START_REF] Russell | Artificial Intelligence: A Modern Approach[END_REF] en raison de leur nature déductive. Pour traiter les exceptions dans le monde réel, une méthode doit prendre en compte l'incertitude, qu'elle soit due à des observations partielles, à un environnement non déterministe ou à une combinaison des deux. Il est maintenant bien connu que la théorie des probabilités offre un cadre mathématiquement justifié pour quantifier l'incertitude. En fait, la théorie des probabilités est une généralisation naturelle de la logique (Jaynes 1996) sans faire référence au «hasard» ou à la «variable aléatoire» contrairement à l'introduction classique de la théorie des probabilités par [START_REF] Kolmogorov | Foundations of the Theory of Probability[END_REF]. Cette ligne de recherche remonte à Laplace (1820), [START_REF] Maynard | A treatise on probability[END_REF], [START_REF] Jeffreys | Theory of Probability[END_REF]. [START_REF] Richard | Probability, frequency and reasonable expectation[END_REF] a donné une justification mathématique basée sur quelques axiomes.

Les axes de recherche susmentionnés reposent apparemment sur différentes philosophies et ont développé deux branches de l'IA, qui, en raison de leurs caractéristiques principales, sont parfois qualifiées d'IA basée sur les données et d'IA basée sur la connaissance. Intuitivement, le premier s'appuie sur des approximateurs de fonctions générales (inspirés du cerveau) pour approximer les mappages d'entrée-sortie sous-jacents des données, tandis que le second intègre la connaissance humaine dans les modèles, indépendamment de tout ensemble de données particulier. Certes, il n'existe pas de méthode purement axée sur les données ou sur les connaissances. Même les réseaux de neurones ne peuvent pas généraliser sans biais inductifs.

La théorie des probabilités offre un cadre élégant pour intégrer ces deux types de méthodes en ce sens que les connaissances de haut niveau et les informations de bas niveau sont modélisées sous forme de variables aléatoires avec des distributions explicites ou implicites. Un exemple à cet égard est le modèle graphique probabiliste (MGP) [START_REF] Pearl | Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference[END_REF][START_REF] Koller | Probabilistic graphical models: principles and techniques[END_REF], qui prend en charge de manière inhérente le raisonnement probabiliste tout en possédant une représentation de la connaissance de graphe inspirée par des réseaux de neurones. Contrairement aux connexions dans les réseaux de neurones, la structure des graphes dans un MGP est souvent liée à une relation réelle ou à une abstraction réelle, ce qui permet une inférence et un raisonnement efficace. Les exemples classiques de MGP incluent les modèles de Markov cachés [START_REF] Leont | Conditional markov processes[END_REF], l'allocation de Dirichlet latente [START_REF] David M Blei | Latent dirichlet allocation[END_REF], les réseaux bayésiens [START_REF] Pearl | Bayesian networks: A model of self-activated memory for evidential reasoning[END_REF], etc. Il existe également des exemples ressemblant à des méthodes logiques ou à des réseaux de neurones. Par exemple, le réseau logique de Markov [START_REF] Richardson | Markov logic networks[END_REF] est une MGP dans laquelle des variables aléatoires sont associées à des propositions logiques; Les réseaux de croyances profondes [START_REF] Hinton | Deep belief networks[END_REF]) et les machines de Boltzmann restreintes profondes [START_REF] Salakhutdinov | Deep boltzmann machines[END_REF]) sont des MGP dotés de structures graphiques en couches similaires à celles des réseaux de neurones ix profonds.

Comme chaque épée possède deux tranchants, le coût à payer pour la rigueur mathématique de la quantification de l'incertitude est la calculabilité ou l'évolutivité. équipés du cadre de probabilité, nous avons également introduit de nombreuses quantités insolubles, telles que la valeur attendue, le dénominateur de la fonction de densité de probabilité et la vraisemblance marginale, qui impliquent toutes le calcul d'intégrales insolubles. Pour approcher de telles quantités, il existe deux idées principales: la chaîne de Markov (MCMC) [START_REF] Metropolis | Equation of state calculations by fast computing machines[END_REF][START_REF] Hastings | Monte carlo sampling methods using markov chains and their applications[END_REF][START_REF] Geman | Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images[END_REF]) et l'inférence variationnelle (IV) [START_REF] Peterson | A mean field theory learning algorithm for neural networks[END_REF][START_REF] Michael I Jordan | An introduction to variational methods for graphical models[END_REF], Blei et al. 2017). MCMC est basé sur une simulation de Monte Carlo: pour tirer des échantillons de la distribution de probabilité cible, il construit une chaîne de Markov telle que la distribution stationnaire de la chaîne de Markov soit la distribution de probabilité cible; ces échantillons sont ensuite utilisés pour approximer les intégrales avec des sommes déductibles. Bien que, par construction, MCMC produise des estimations non biaisées par la loi de grand nombre, elle souffre d'une lente convergence. L'inférence variationnelle, au contraire, sacrifie l'exactitude pour la rapidité et pose l'inférence approximative comme un problème d'optimisation, dans lequel une famille de distributions plus simples (c'est-à-dire, des distributions de propositions) est d'abord posée, puis le membre le plus proche de cette famille de la distribution cible est choisie en fonction d'une mesure de divergence (par exemple, la divergence de Kullback-Leibler). Comparé à MCMC, bien que présentant un biais irrémédiable, IV a tendance à être plus rapide et plus évolutif pour des données volumineuses.

A propos de cette thèse

Dans cette thèse, je me concentrerai principalement sur l'inférence variationnelle et les modèles probabilistes. En particulier, je couvrirai plusieurs projets auxquels j'ai participé au cours de ma thèse concernant l'amélioration de l'efficacité des systèmes IA / ML avec des techniques variationnelles. La thèse inclut deux parties. Dans la première partie, l'efficacité des modèles probabilistes graphiques est étudiée. Dans la deuxième partie, plusieurs problèmes d'apprentissage profond sont examinés, qui sont liés à l'efficacité énergétique ou à l'efficacité des échantillons. Je montrerai brièvement dans les paragraphes suivants pour chaque partie les principaux problèmes que nous avons rencontrés et les solutions que nous avons proposées pour un apprentissage automatique plus efficace.

Modèles graphiques probabilistes

Le premier problème abordé dans cette thèse est l'efficacité de calcul des MGP. Comme dans de nombreux modèles probabilistes, les MGP souffrent d'un problème de calcul intraitable. Plus précisément, l'probabiliste inférence et l'estimation paramètre de vraisemblance maximale du MGP sont en général insolubles en raison de la fonction de partition: pour calculer la valeur de la fonction ou son gradient impliquent une intégrale sur tout l'espace échantillon, ce qui est onéreux en terme x de calcul, même dans le cas discret, la taille de l'espace échantillon augmente de manière exponentielle avec l'augmentation du nombre de noeuds. En tant que tel, l'inférence variationnelle est devenue une bête de somme pour les MGP. C'est aussi parce que l'inférence variationnelle est un cadre général plutôt qu'un algorithme particulier. En choisissant différentes distributions de proposition, l'inférence variationnelle peut prendre différentes formes. Compte tenu des idées de la mécanique stochastique [START_REF] Parisi | Statistical field theory[END_REF], la formulation variationnelle de champ moyen et de Bethe sont les exemples classiques largement utilisés dans les MGP [START_REF] Wainwright | Graphical models, exponential families, and variational inference[END_REF]. En nous concentrant sur la formulation variationnelle convexifiée de Bethe (Wainwright et al. 2005b, Globerson and Jaakkola 2007b[START_REF] Meshi | Convexifying the bethe free energy[END_REF], nous identifions que son dual problème est fortement concave mais non lisse, où le non-lissé provient des contraintes induites par couplage de marginalisation entre les variables duales. Nous proposons un nouvel algorithme inspiré du développement algorithmique de l'optimisation convexe pour la minimisation du risque empirique [START_REF] Nicolas L Roux | A stochastic gradient method with an exponential convergence rate for finite training sets[END_REF][START_REF] Johnson | Accelerating stochastic gradient descent using predictive variance reduction[END_REF], Defazio et al. 2014a[START_REF] Shalev | Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization[END_REF]. En particulier, nous étendons l'algorithme stochastique à dual coordonnée (SDCA) [START_REF] Shalev | Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization[END_REF]) et la méthode de multiplicateur lagrangien augmenté (ADMM) [START_REF] Boyd | Distributed optimization and statistical learning via the alternating direction method of multipliers[END_REF]) pour le problème dual. Comme l'analyse classique de convergence d'ADMM ne s'applique pas aux problèmes convexes incluant plus de trois blocs de variables (Chen et al. 2016a), nous effectuons une nouvelle analyse de convergence pour notre algorithme basée sur l'analyse par rapport à des gradients inexacts pour les problèmes de minmax. Nous montrons également que la séquence de solutions primales et la séquence de solutions dual générées par notre algorithme bénéficient d'une convergence linéaire, ce qui est confirmé empiriquement par trois expériences représentatives.

Apprentissage profond

Dans la deuxième partie de la thèse, nous nous concentrons sur apprentissage profond. Pour les réseaux de neurones profonds, le problème de l'apprentissage est généralement un problème d'optimisation non convexe à grande échelle, dans lequel ni la valeur objective ni le gradient ne peuvent être calculés efficacement. Rappelez-vous qu'un problème d'optimisation non convexe est considéré comme extrêmement difficile car le problème peut avoir plusieurs minima locaux et points de selle. Cependant, lorsqu'un réseau de neurones est sur-paramétré, par exemple en augmentant la profondeur et / ou la largeur du réseau, un algorithme d'optimisation simple tel que la descente de gradient stochastique (SGD) [START_REF] Robbins | A stochastic approximation method[END_REF] ou ses variantes telles que AdaGrad [START_REF] Duchi | Adaptive subgradient methods for online learning and stochastic optimization[END_REF], ADAM (Kingma and Ba 2014) peuvent trouver des optima globaux dans le cas de la classification d'images supervisées [START_REF] Zhang | Understanding deep learning requires rethinking generalization[END_REF]. De plus, les résultats empiriques de [START_REF] Hinton | Improving neural networks by preventing co-adaptation of feature detectors[END_REF], He et al. (2016a), Zagoruyko and Komodakis (2016c) montrent que les réseaux sur-paramétrisés sont moins enclins à overfitting et atteignent souvent de meilleures performances de généralisation. Ce phénomène a été étudié par de nombreux chercheurs théoriques. Depuis, quelques explications ont été publiées, par exemple la conjecture des minima plats de [START_REF] Shirish Keskar | On large-batch training for deep learning: Generalization gap and sharp minima[END_REF]), Wu xi et al. (2017), les idées de PAC-Bayes de Dziugaite and Roy (2017), [START_REF] Zhou | Compressibility and generalization in large-scale deep learning[END_REF], [START_REF] Valle Pérez | Deep learning generalizes because the parameter-function map is biased towards simple functions[END_REF] et la perspective de la théorie de l'information de Tishby and Zaslavsky (2015), [START_REF] Achille | Emergence of invariance and disentangling in deep representations[END_REF].

Bien que les réseaux sur-paramétrés semblent fonctionner bien dans la pratique, le sur-paramétrage pose deux problèmes: l'inefficacité mémoire / énergie et l'inefficacité de l'échantillon. Le premier problème devrait être attribué au nombre considérable de paramètres. Notez qu'en raison de la nature hautement parallélisable des réseaux de neurones, la sur-paramétrisation peut ne pas poser de problème en termes de vitesse. Cependant, le nombre de paramètres à stocker dans la mémoire et le nombre d'opérations à effectuer lors des passes en avant et en arrière posent un problème sérieux lors de l'application de l'apprentissage profond à des domaines plus vastes avec des ressources limitées. Le deuxième problème est plus fondamental car il est notoirement connu que les réseaux de neurones ont faim de données. Des résultats à la fois théoriques et empiriques (Cybenko 1989b[START_REF] Raghu | On the expressive power of deep neural networks[END_REF], Zhang et al. 2016) montrent que les réseaux de neurones sont capables d'exprimer un large éventail de fonctions. Selon la théorie de l'apprentissage classique [START_REF] Leslie | A theory of the learnable[END_REF], cela implique que la complexité du modèle peut être extrêmement élevée, de même que la complexité de l'échantillon. En d'autres termes, pour obtenir une petite erreur de test, nous aurons besoin d'un grand nombre de données de formation. Néanmoins, dans de nombreux cas, la taille des données de formation ne répond pas à cette condition en raison de problèmes de distribution de données à grande échelle ou de problèmes de confidentialité. Ces deux problèmes ne sont évidemment pas isolés. Notre objectif ultime est de former un petit réseau sur un petit jeu de données, tout en offrant des performances comparables. À cette fin, plusieurs ingrédients du système d'apprentissage profond devraient être révisés.

Le premier ingrédient que nous étudions est la régularisation, qui consiste à remodeler le problème d'apprentissage afin d'éviter le overfitting. En apprentissage profond, les régulateurs traditionnels tels que le weight decay (c.-à-d. La pénalité de L2) sont moins efficaces que les dropout [START_REF] Srivastava | Dropout: a simple way to prevent neural networks from overfitting[END_REF]) et la batch normalisation (Ioffe and Szegedy 2015a), qui ont tous deux une interprétation bayésienne [START_REF] Diederik P Kingma | Variational dropout and the local reparameterization trick[END_REF][START_REF] Teye | Bayesian uncertainty estimation for batch normalized deep networks[END_REF]. En fait, de nombreuses techniques de régularisation correspondent à l'imposition de connaissances préalables d'un point de vue bayésien. Cependant, il est difficile de savoir quelle forme de connaissance préalable convient à un apprentissage profond. À la lumière de minimum description length (MDL) interprétation des réseaux de neurones (Rissanen 1978, Hinton and[START_REF] Hinton | Keeping the neural networks simple by minimizing the description length of the weights[END_REF], qui définit l'apprentissage comme une compression sans perte, nous empruntons une autre voie pour introduire l'antérieur comme terme de régularisation. compression avec perte des données. Plus précisément, nous prenons l'information mutuelle entre l'échantillon bootstrap et les poids neural-net comme terme de régularisation. L'algorithme proposé ressemble à l'algorithme classique de Blahut-Arimoto pour la compression avec perte [START_REF] Blahut | Computation of channel capacity and rate-distortion functions[END_REF][START_REF] Arimoto | An algorithm for computing the capacity of arbitrary discrete memoryless channels[END_REF], dans lequel nous calculons une variation postérieure locale pour chaque échantillon bootstrap, puis nous l'utilisons pour mettre à jour la composition postérieure agrégée. Contrairement aux méthodes MDL (Hinton and xii [START_REF] Hinton | Keeping the neural networks simple by minimizing the description length of the weights[END_REF][START_REF] Graves | Practical variational inference for neural networks[END_REF][START_REF] Blundell | Weight uncertainty in neural networks[END_REF], nous utilisons l'agrégation des postérieurs pour calculer la distribution prédictive plutôt que de prendre la variation postérieure. De manière empirique, notre méthode présente certaines propriétés intéressantes par rapport aux méthodes conventionnelles MDL. L'efficacité de l'échantillon peut être obtenue dans une configuration multi-tâche spéciale, dans laquelle les tâches partagent certaines caractéristiques communes. Le problème est appelé méta-apprentissage et est généralement résolu par des mécanismes d'apprentissage [START_REF] Zhang | Understanding deep learning requires rethinking generalization[END_REF][START_REF] Ravi | Optimization as a model for few-shot learning[END_REF][START_REF] Finn | Model-agnostic meta-learning for fast adaptation of deep networks[END_REF][START_REF] Snell | Prototypical networks for few-shot learning[END_REF][START_REF] Mishra | A simple neural attentive meta-learner[END_REF], Gidaris and Komodakis 2018). étant donné que chaque tâche contient ses propres jeux de données, l'ensemble des données fournies dans le méta-apprentissage est une hiérarchie à deux niveaux de jeux de données. Nous considérons donc une bayésienne hiérarchique formulation comprenant deux niveaux de variables latentes, les variables latentes du premier niveau correspondant aux paramètres de chaque tâche et la variable latente du second niveau correspondant à l'hyper-paramètre partagé entre les tâches. Nous optons pour approximer la partie postérieure de l'hyper-paramètre par une estimation ponctuelle, ce qui conduit à la méthode dite de Bayes empirique. Il se trouve que le régularisateur d'informations mutuelles susmentionné apparaît dans la ELBO de la log-vraisemblance marginale, mais cette fois nous obtenons une information mutuelle entre les pondérations et les données associées à une tâche. Notez que, dans Bayes empiriques, nous traitons les poids individuels comme des échantillons de la même variable aléatoire. Enfin, nous déduisons une inférence variationnelle amortie qui couple toutes les postérieures variationnelles locales (une pour chaque échantillon du paramètre) dans un méta-modèle, qui consiste en un réseau de gradient synthétique et un réseau d'initialisation. Nos résultats empiriques sur le repère mini-ImageNet pour la classification épisodique en few-shot surpassent de manière significative les méthodes de pointe antérieures.

Un excellent travail proposé par [START_REF] Ba | Bach. Breaking the curse of dimensionality with convex neural networks[END_REF], [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF] s'est avéré utile pour améliorer à la fois l'efficacité énergétique et l'efficacité d'échantillonnage de l'apprentissage profond (Yim et al. 2017, Zagoruyko andKomodakis 2016b), ce qui est réalisé en forçant le réseau de neurones (c'est-à-dire l'étudiant) à imiter un réseau de neurones d'enseignants pré-formés. Ce travail est important car nous pouvons adapter le réseau d'étudiants à une taille suffisamment petite sans sacrifier la performance [START_REF] Furlanello | Born again neural networks[END_REF]. Il est à noter que, sans transfert de connaissances, la formation des petits réseaux a tendance à être underfitting et que l'optimisation pourrait être beaucoup plus difficile que celle de réseaux sur-paramétrés. De plus, à condition que le réseau d'enseignants soit formé sur un jeu de données source à grande échelle (par exemple, ImageNet), le transfert de connaissances enseignant-élève offre un cadre naturel pour l'apprentissage par transfert. Cela étant dit, les travaux existants sur le transfert de connaissances enseignant-élève exigent que le réseau d'élèves soit suffisamment similaire au réseau d'enseignants. Cependant, aucune justification mathématique n'est donnée pour expliquer le mécanisme derrière le cadre enseignant-élève. Inspirés par l'interprétation de information bottleneck de l'apprentissage profond [START_REF] Tishby | Deep learning and the information bottleneck principle[END_REF], où l'apprentissage est présenté comme une optimisation xiii minimisant l'information mutuelle entre la représentation latente et l'entrée, tout en maximisant l'information mutuelle entre la représentation latente et l'étiquette, nous proposons de maximiser l'information mutuelle entre la représentation de l'enseignant et celle de l'élève. Pour incorporer ce terme d'information mutuelle négatif en tant que perte supplémentaire, nous utilisons une borne inférieure variationnelle avec une hypothèse gaussienne. Nous comparons notre méthode avec les méthodes existantes de transfert de connaissances pour les tâches de distillation et de transfert de connaissances et montrons que notre méthode surperforme systématiquement les méthodes existantes. Nous démontrons en outre les avantages de notre méthode sur le transfert de connaissances entre architectures de réseaux hétérogènes en transférant les connaissances d'un réseau de neurones à convolution (CNN) à un perceptron multicouche (MLP) sur CIFAR-10.

Le dernier travail de cette thèse était dû à une découverte empirique: les réseaux de neurones profonds sont robustes aux contraintes de symétrie imposées aux poids. Nous proposons plusieures techniques de symétrisation et montrons empiriquement que ces techniques sont très simples et universellement applicables à presque tous les réseaux sur-paramétrés sans nécessiter de réglage précis en post-traitement et que, en fonction de la taille du réseau, la symétrie ne peut avoir que peu effet négatif sur le réseau au terme de précision. Par exemple, notre ResNet-101 symétrisé a presque 25% moins de paramètres que sa version d'origine, et introduit une perte de précision de seulement 0,2% sur ImageNet. En tant qu'application, il peut être utilisé pour réduire les besoins en mémoire des réseaux de neurones avec un coût de calcul négligeable au moment de la formation.

Introduction

Artificial intelligence (AI), suggested by its name, is a type of intelligence distincted from natural intelligence (NI) appearing in humans and animals. However, intelligence is itself a chamber of mysteries. We are not close to understanding it any time soon. Over the years, AI researchers attempted to make an AI system indistinguishable from that of its NI sibling. The Turing test, which is designed to evaluate proposed AI systems, epitomizes this passion. As AI research advances, the definition of AI is also evolving over time. The term AI was originally used to refer to the computer systems mimicking NI. In its modern usage, AI also refers to as the whole field that studies intelligent machines. As a scientific field, AI was born at the Dartmouth conference in 1956 and has become an interdisciplinary field involving computer science, statistics, neuroscience, physics and so on. [START_REF] Jordan | Artificial intelligence: The revolution hasn't happened yet[END_REF] clarified that most of what is being called AI today is what has been called machine learning (ML) for the past several decades. ML is a less mysterious term as it is a field that focuses more on the mathematical and engineering aspects of AI systems.

While building an AI system does not necessarily require reverse engineering, drawing inspirations from NI can bring promising ideas. AI research has a long history of emphasis on studying and reproducing partially or entirely the mechanism of NI. The induced methods are sometimes called brain-inspired methods. Examples of this kind include a variety of computational models, called artificial neural networks [START_REF] David E Rumelhart | Learning representations by back-propagating errors[END_REF][START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF][START_REF] Maass | Networks of spiking neurons: the third generation of neural network models[END_REF], which loosely imitate biological neurons. This family of methods and models is now largely developed and widely recognized as deep learning [START_REF] Lecun | Deep learning[END_REF].

As argued by [START_REF] Russell | Artificial Intelligence: A Modern Approach[END_REF], "artificial flights" were made when the Wright brothers and others stopped imitating birds. A different philosophy is to avoid mimicking NI, which stems from mathematical formulations of thinking machines based on concepts such as logic and rationality. These formulations can date back to Aristotle's syllogism and Boolean algebra. The most representative examples of this kind are logical methods, such as propositional logic and first-order logic. Unlike artificial neural networks, logical models are more intuitive and transparent from a human perspective. They are able to incorporate domain knowledge in the form of logical formulas and conduct mathematically sound reasoning in a human-understandable way. In the mid-1965s, AI research on this kind of methods was perceived as the dominant paradigm [START_REF] Haugeland | Artificial intelligence: The very idea[END_REF]), which has many remarkable applications, such as the general problem solver by [START_REF] Newell | Gps, a program that simulates human thought[END_REF] and the advice taker by [START_REF] Mccarthy | Programs with common sense[END_REF]. These achievements were part of the reasons why AI drew so much attention in the 1960s.

However, classical logical models suffer from the so-called qualification problem [START_REF] Russell | Artificial Intelligence: A Modern Approach[END_REF] due to their deductive nature. To handle exceptions in the real world, a method must take into account uncertainty, whether due to partial observations, non-deterministic environments, or a combination of both. It is now well known that probability theory offers a mathematically justified framework for quantifying uncertainty. In fact, probability theory is a natural generalization of logic (Jaynes 1996) without making reference to "chances" or "random variables" in contrast to the classical introduction of probability theory by [START_REF] Kolmogorov | Foundations of the Theory of Probability[END_REF]. This line of research dates back to Laplace (1820), [START_REF] Maynard | A treatise on probability[END_REF], [START_REF] Jeffreys | Theory of Probability[END_REF]. A mathematical justification was given by [START_REF] Richard | Probability, frequency and reasonable expectation[END_REF] based on a few axioms.

The aforementioned lines of research are apparently based on different philosophies and have developed two branches of AI, which, due to their main characteristics, are sometimes labeled as data-driven AI and knowledge-driven AI respectively. Intuitively, the former relies on general (brain-inspired) function approximators to approximate the underlying input-output mappings of the data, while the latter incorporates human knowledge into the models independent of any particular dataset. Certainly, there is no pure data-driven or knowledge-driven method. Even neural networks cannot generalize without inductive biases.

Probability theory offers an elegant framework to integrate these two kinds of methods in the sense that both high-level knowledge and low-level information are modeled as random variables with either known or implicit distributions. An example in this regard is the probabilistic graphical model (PGM) [START_REF] Pearl | Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference[END_REF][START_REF] Koller | Probabilistic graphical models: principles and techniques[END_REF], which inherently supports probabilistic reasoning while possesses a graph knowledge representation inspired by neural networks. Unlike the connections in neural networks, the graph structure in a PGM is often linked to a real-life relationship or its abstraction in the real world, which in turn enables efficient inference and reasoning. Classical examples of PGMs include hidden Markov models [START_REF] Leont | Conditional markov processes[END_REF], latent Dirichlet allocation [START_REF] David M Blei | Latent dirichlet allocation[END_REF], Bayesian networks [START_REF] Pearl | Bayesian networks: A model of self-activated memory for evidential reasoning[END_REF] etc. There are also examples that resemble logical methods or neural networks. For instance, Markov logic network [START_REF] Richardson | Markov logic networks[END_REF]) is a PGM in which random variables are associated with logical propositions; deep belief networks [START_REF] Hinton | Deep belief networks[END_REF] and deep restricted Boltzmann machines [START_REF] Salakhutdinov | Deep boltzmann machines[END_REF] are PGMs with layered graph structures which are similar to that of deep neural networks.

As every sword has two edges, the price to pay for the mathematical rigorousness of uncertainty quantification is computability or scalability. Equipped with the probability framework, we also introduced many intractable quantities, such as the expected value, the denominator of a probability density function, and the marginal likelihood alike, which all involve computing intractable integrals. To approximate such quantities, there are two mainstream ideas: Markov Chain Monte Carlo (MCMC) [START_REF] Metropolis | Equation of state calculations by fast computing machines[END_REF][START_REF] Hastings | Monte carlo sampling methods using markov chains and their applications[END_REF][START_REF] Geman | Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images[END_REF] and variational inference (VI) [START_REF] Peterson | A mean field theory learning algorithm for neural networks[END_REF][START_REF] Michael I Jordan | An introduction to variational methods for graphical models[END_REF], Blei et al. 2017). MCMC is based on Monte Carlo simulation: to draw iid samples from the target probability distribution, it constructs a Markov chain such that the stationary distribution of the Markov chain is the target probability distribution; these samples are then used to approximate the integrals with tractable sums. Although by construction, MCMC produces unbiased estimates by the strong law of large numbers, it suffers from a slow convergence to the stationary. Variational inference, on the other hand, sacrifices exactness for speed, and casts the approximate inference as an optimization problem, in which a family of simpler distributions (i.e., proposal distributions) is first posited, and then the closest member of that family to the target distribution is chosen with respect to some divergence measure (e.g., the Kullback-Leibler divergence). Compared with MCMC, although being irredeemably biased, VI tends to be faster and more scalable to large data.

About this thesis

In this thesis, I will mainly focus on variational inference and probabilistic models. In particular, I will cover several projects that I have participated in during my PhD about improving the efficiency of AI/ML systems with variational techniques. The thesis consists of two parts. In the first part, the computational efficiency of probabilistic graphical models is studied. In the second part, several problems in deep learning are investigated, which are related to either energy efficiency or sample efficiency. I will briefly show in the next paragraphs the main problems we have encountered and the solutions we have proposed towards more efficient machine learning.

Probabilistic graphical models

The first problem considered in this thesis is the computational efficiency of PGMs. As in many probabilistic models, PGMs suffer from computational intractability issue. Specifically, probabilistic inference and maximum likelihood parameter estimation of PGMs are in general intractable due to the partition function: to compute the function value or its gradient involve an integral over the entire sample space, which is computationally expensive since, even for the discrete case, the size of the sample space grows exponentially as the number of nodes increasing. As such, variational inference has become a workhorse for PGMs. This is also because variational inference is a general framework rather than a particular algorithm. By choosing different proposal distributions, variational inference can take different forms. Along with the insights from stochastic mechanics [START_REF] Parisi | Statistical field theory[END_REF], the mean-field variational formulation and the Bethe variational formulation are the conventional examples widely used in PGMs [START_REF] Wainwright | Graphical models, exponential families, and variational inference[END_REF]. Focusing on the convexified Bethe variational formulation (Wainwright et al. 2005b, Globerson and Jaakkola 2007b[START_REF] Meshi | Convexifying the bethe free energy[END_REF], we identify that its dual problem is strongly concave but non-smooth, where the non-smoothness comes from the constraints induced by marginalization coupling between the dual variables. We propose a new algorithm inspired by the algorithmic development on convex optimization for empirical risk minimization [START_REF] Nicolas L Roux | A stochastic gradient method with an exponential convergence rate for finite training sets[END_REF][START_REF] Johnson | Accelerating stochastic gradient descent using predictive variance reduction[END_REF], Defazio et al. 2014a[START_REF] Shalev | Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization[END_REF]. In particular, we extend the stochastic dual coordinate ascent (SDCA) algorithm [START_REF] Shalev | Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization[END_REF] and the augmented Lagrangian method of multipliers (ADMM) [START_REF] Boyd | Distributed optimization and statistical learning via the alternating direction method of multipliers[END_REF] for the dual problem. Since the classical convergence analysis of ADMM does not hold for convex problems involving more than three blocks of variables (Chen et al. 2016a), we conduct a new convergence analysis for our algorithm based on the analysis of inexact gradients for min-max problems. We also show that both the sequence of primal solutions and the sequence of dual solutions generated by our algorithm enjoy a linear convergence, which is confirmed empirically in three representative experiments of PGMs.

Deep learning

In the second part of the thesis, we focus on deep learning. For deep neural networks, the learning problem is generally a large-scale nonconvex optimization problem, in which neither the objective value nor the gradient can be computed efficiently. Recall that a nonconvex optimization problem is considered extremely difficult since the problem may have many local minima and saddle points. However, when a neural network is over-parameterized, for example, by increasing the depth and/or the width of the network, a simple optimization algorithm like stochastic gradient descent (SGD) [START_REF] Robbins | A stochastic approximation method[END_REF] or its variants such as AdaGrad [START_REF] Duchi | Adaptive subgradient methods for online learning and stochastic optimization[END_REF], ADAM (Kingma and Ba 2014) can find global optima in the case of supervised image classification [START_REF] Zhang | Understanding deep learning requires rethinking generalization[END_REF]. Moreover, empirical results by [START_REF] Hinton | Improving neural networks by preventing co-adaptation of feature detectors[END_REF], He et al. (2016a), Zagoruyko and Komodakis (2016c) show that over-parameterized networks are less prone to overfitting, and often achieve a better generalization performance. This phenomenon has been studied by many theoretical researchers. A few explanations have been published since then, for example, the flat minima conjecture by [START_REF] Shirish Keskar | On large-batch training for deep learning: Generalization gap and sharp minima[END_REF], [START_REF] Wu | Towards understanding generalization of deep learning: Perspective of loss landscapes[END_REF], the PAC-Bayes insights by Dziugaite and Roy (2017), [START_REF] Zhou | Compressibility and generalization in large-scale deep learning[END_REF][START_REF] Valle Pérez | Deep learning generalizes because the parameter-function map is biased towards simple functions[END_REF] and the information theory perspective by [START_REF] Tishby | Deep learning and the information bottleneck principle[END_REF], [START_REF] Achille | Emergence of invariance and disentangling in deep representations[END_REF].

Although it seems over-parameterized networks work well in practice, there are two issues brought by the over-parameterization: the memory/energy inefficiency and the sample inefficiency. The first issue should be attributed to the tremendous number of parameters. Note that due to the highly parallelizable nature of neural networks, over-parameterization may not be a problem in terms of speed. However, the number of parameters to be stored in the memory and the number of operations to be carried out during the forward and backward passes do cause a serious problem when applying deep learning to broader domains with resource constraints. The second issue is more fundamental since neural networks are notoriously known to be data-hungry. Both theoretical and empirical results (Cybenko 1989b[START_REF] Raghu | On the expressive power of deep neural networks[END_REF], Zhang et al. 2016) show that neural networks are capable of expressing a diverse range of functions. According to the classical learning theory [START_REF] Leslie | A theory of the learnable[END_REF], this implies that the model complexity can be extremely high, and so does the sample complexity 1 . In other words, to achieve a small testing error, we will need a large number of training data. Nevertheless, in many cases, the size of the training data does not meet this condition due to either the heavy-tailed data distribution or privacy concerns. These two issues are obviously not isolated. Our ultimate goal is to have a small network trained on a small dataset which still yields comparable performance. To this end, multiple ingredients of the deep learning system should be revised.

The first ingredient we investigate is regularization, which is a process of reshaping the learning problem in order to prevent overfitting. In deep learning, traditional regularizers such as weight decay (i.e., the L2 penalty) are less effective compared to dropout [START_REF] Srivastava | Dropout: a simple way to prevent neural networks from overfitting[END_REF]) and batch normalization (Ioffe and Szegedy 2015a), both of which have a Bayesian interpretation [START_REF] Diederik P Kingma | Variational dropout and the local reparameterization trick[END_REF][START_REF] Teye | Bayesian uncertainty estimation for batch normalized deep networks[END_REF]). In fact, many regularization techniques correspond to imposing prior knowledge from a Bayesian perspective. However, what form of prior knowledge is suitable for deep learning is not clear. In light of the minimum description length (MDL) [START_REF] Rissanen | Modeling by shortest data description[END_REF]) interpretation of neural networks [START_REF] Hinton | Keeping the neural networks simple by minimizing the description length of the weights[END_REF], which formulates the learning as lossless compression, we take an alternative path to introduce the prior as a regularization term for the lossy compression of the data. Specifically, we take the mutual information between the bootstrap sample and the neural-net weights as the regularization term. The proposed algorithm resembles the classical Blahut-Arimoto algorithm for lossy compression [START_REF] Blahut | Computation of channel capacity and rate-distortion functions[END_REF][START_REF] Arimoto | An algorithm for computing the capacity of arbitrary discrete memoryless channels[END_REF], in which we compute a local variational posterior for each bootstrap sample and then use it to update the aggregated posterior. Unlike MDL methods [START_REF] Hinton | Keeping the neural networks simple by minimizing the description length of the weights[END_REF][START_REF] Graves | Practical variational inference for neural networks[END_REF][START_REF] Blundell | Weight uncertainty in neural networks[END_REF], we use the aggregated posterior to compute the predictive distribution rather than taking the variational posterior. Empirically, our method exhibits some attractive properties over the conventional MDL methods.

The sample efficiency can be achieved in a special multi-task setting where tasks share some common characteristics. The problem is referred to as meta-learning and usually solved by learning-to-learn schemes [START_REF] Zhang | Understanding deep learning requires rethinking generalization[END_REF][START_REF] Ravi | Optimization as a model for few-shot learning[END_REF][START_REF] Finn | Model-agnostic meta-learning for fast adaptation of deep networks[END_REF][START_REF] Snell | Prototypical networks for few-shot learning[END_REF][START_REF] Mishra | A simple neural attentive meta-learner[END_REF], Gidaris and Komodakis 2018). Since each task contains its own datasets, the overall data provided in meta-learning is a two-level hierarchy of datasets. We thus consider a hierarchical Bayes formulation consisting of two levels of latent variables, where the latent variables in the first level are the parameters of each task, and the latent variable in the second level is the hyperparameter shared across tasks. We opt to approximate the posterior of the hyperparameter by a point estimate, which leads to the so-called empirical Bayes. It turns out that the aforementioned mutual information regularizer appears in the evidence lower bound of the marginal loglikelihood, but this time we obtain the mutual information between the weights and the data associated with some task. Note that, in empirical Bayes, we treat individual weights as iid samples of the same random variable. Finally, we derive an amortized variational inference that couples all local variational posteriors (one for each sample of the parameter) into a meta-model, which consists of a synthetic gradient network and an initialization network. Our empirical results on the mini-ImageNet benchmark for episodic few-shot classification significantly outperform previous state-of-the-art methods.

A seminal work proposed by [START_REF] Ba | Bach. Breaking the curse of dimensionality with convex neural networks[END_REF], [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF] has been shown to be useful for improving both energy efficiency and sample efficiency of deep learning (Yim et al. 2017, Zagoruyko andKomodakis 2016b), which is achieved by forcing the neural network (i.e., the student) to mimic a pretrained teacher neural network. This work is important because we can tailor the student network to be sufficiently small without sacrificing the performance [START_REF] Furlanello | Born again neural networks[END_REF]. Note that, without knowledge transfer, the training of small networks tends to underfitting and the optimization could be much more difficult than that of over-parameterized networks. Moreover, provided that the teacher network is trained on a large-scale source dataset (e.g., ImageNet), the teacher-student knowledge transfer offers a natural framework for transfer learning. That being said, existing works require the student network to be sufficiently similar to the teacher network. Yet no mathematical justification is given to explain the mechanism behind the knowledge transfer. Inspired by the information bottleneck interpretation of deep learning [START_REF] Tishby | Deep learning and the information bottleneck principle[END_REF], where the learning is cast as an optimization that minimizes the mutual information between the latent representation and the input while maximizing the mutual information between the latent representation and the label, we propose to maximize the mutual information between the teacher's representation and the student's representation.

To incorporate this negative mutual information term as an additional loss, we employ a variational lower bound with the Gaussian assumption. We compare our method with existing knowledge transfer methods on both knowledge distillation and transfer learning tasks and show that our method consistently outperforms existing methods. We further demonstrate the strength of our method on knowledge transfer across heterogeneous network architectures by transferring knowledge from a convolutional neural network (CNN) to a multi-layer perceptron (MLP) on CIFAR-10.

The last work of this thesis was due to an empirical finding -deep neural networks are robust to symmetry constraints imposed on the weights. We propose several symmetrization techniques and show empirically that these techniques are very simple and universally applicable to almost any over-parameterized network without requiring fine-tuning as post-processing, and, depending on network size, symmetry can have little or no negative effect on network accuracy. For instance, our symmetrized ResNet-101 has almost 25% fewer parameters than its original version with only 0.2% accuracy loss on ImageNet. As an application, it can be used to reduce the memory requirement of neural networks with additionally a negligible computational cost at training time.

Organization of the thesis

As we have mentioned, The thesis will be divided into two parts. The first part contains the topics in probabilistic graphical models covered by Chapter 2 and Chapter 3. The second part includes the topics in deep learning covered by Chapter 4, Chapter 5, Chapter 6, Chapter 7 and Chapter 8. The detailed organization of this thesis is sketched as follows. Chapter 1 covers the prerequisite mathematics for the presentation of the following chapters. It includes a basic introduction to information theory and a basic introduction to the mathematical concepts used by the convergence analysis for convex optimization. Chapter 2 gives a general introduction to probabilistic graphical models, including common models, variational inference and the formulations for parameter estimation. Chapter 3 is based on the following publication: Shell X. Hu, Pablo G. Moreno, Xi Shen, Yang Xiao, Guillaume Obozinski, Neil D. Lawrence, Andreas Damianou. "Empirical Bayes Transductive Meta-Learning with Synthetic Gradients." NeurIPS Workshop on Meta-Learning, 2019. (Long version submitted to ICLR 2020) Chapter 7 is based on the following publication:

Shell X.

Information Theory Abstract

In this chapter, we cover several basic definitions and theorems in convex optimization and information theory, which will be used in the subsequent chapters. It is however safe to skip this chapter.

Background: Convex Optimization and Information Theory

Convex Optimization

Definition 1.1.1 (Convexity). A function f (x) is convex if dom(f ) is a convex set and αf (x) + (1 -α)f (y) ≥ f αx + (1 -α)y , ∀x, y ∈ dom(f ), α ∈ [0, 1].
If the above inequality holds only for ">", we say f (x) is strictly convex.

Lemma 1.1.1 (First order condition). A differentiable function f (x) is convex iff f (y) ≥ f (x) + ∇f (x), y -x , ∀x, y ∈ dom(f ).
Proof. By the definition of convexity,

f (y) ≥ f (x) + f αx + (1 -α)y -f (x) α . (1.1) Let g(α) = f αx + (1 -α)y , then f (x) = g(0)
and f (y) = g (1). By taking limits on both sides, we can rewrite the above inequality as

f (y) ≥ f (x) + lim α→0 g(α) -g(0) α (1.2) = f (x) + ∇g(0) (1.3) = f (x) + ∇f (x), y -x .
(1.4)

Note that ∇g(α) = ∇f (x + α(y -x)), y -x .

Corollary 1.1.1. A differentiable function f (x) is convex iff ∇f (x) -∇f (y), x -y ≥ 0, ∀x, y ∈ dom(f ).
Proof. By the Lemma 1.1.1,

f (y) ≥ f (x) + ∇f (x), y -x (1.5) f (x) ≥ f (y) + ∇f (y), x -y . (1.6)
The result follows by combining the above inequalities, which cancels out f (x) and f (y) on both sides.

2. f (y) ≤ f (x) + ∇f (x), y -x + L 2 x -y 2 , ∀x, y ∈ dom(f ). 3. αf (x) + (1 -α)f (y) ≤ f αx + (1 -α)y + α(1-α)L 2 x -y 2 , ∀x, y ∈ dom(f ). 4. L 2 x 2 -f (x) is convex. 5. If f is twice differentiable, ∇ 2 f (x) LI, ∀x ∈ dom(f ). 6. ∇f (x) -∇f (y), x -y ≥ 1 L ∇f (x) -∇f (y) 2 2 , ∀x, y ∈ dom(f ). Proof. We first prove 1 ⇔ 4. Let g(x) := L 2 x 2 -f (x), then ∇g(x) = Lx -∇f (x).
Since by Corollary 1.1.1 and Cauchy-Shwartz inequality, we have

∇f (x) -∇f (y), x -y ≤ ∇f (x) -∇f (y) x -y (1.8) ≤ L x -y 2 , (1.9)
where the last inequality follows from the fact that ∇f (x) is Lipschitz continuous. By rearranging terms, we obtain

(Lx -∇f (x)) -(Ly -∇f (y)), x -y ≥ 0. (1.10)
Again, applying Corollary 1.1.1 on g(x), we conclude that g(x) is convex, which can be used to show the equivalence 1 ⇔ 3 by expanding

g(αx + (1 -α)y) ≤ αg(x) + (1 -α)g(y).
(1.11)

Next, it is straightforward to show the equivalences 4 ⇔ 2 and 4 ⇔ 5, since .14) To show the equivalence 1 ⇔ 6, we first introduce .18) which indicates ∇f x (z) is Lipschitz continuous. Similarly, ∇f y (z) can also be shown to be Lipschitz. Then, by the statement 2, we have

g(x) is convex ⇔ ∇ 2 g(x) = LI -∇ 2 f (x) 0 if f is twice differentiable (1.12) ⇔ g(y) -g(x) ≥ ∇g(x), y -x (1.13) ⇔ f (y) ≤ f (x) + ∇f (x), y -x + L 2 x -y 2 . ( 1
f x (z) = f (z) -∇f (x), z (1.15) f y (z) = f (z) -∇f (y), z . (1.16) Note that ∇f x (z) = ∇f (z) -∇f (x). Therefore, ∇f x (z) -∇f x (z ) = ∇f (z) -∇f (z ) (1.17) ≤ L z -z , ( 1
f x (y) ≤ f x (z) + ∇f x (z), y -z + L 2 z -y 2 .
(1.19)

Background: Convex Optimization and Information Theory

Minimizing both sides with respect to y yields

min y f x (y) ≤ min y f x (z) + ∇f x (z), y -z + L 2 z -y 2 (1.20) = f x (z) - 1 2L ∇f x (z) 2 . (1.21) Since f x (z) is convex in z, it follows that z * = arg min z f x (z) = x ⇔ ∇f x (z) = 0.
The above inequality can be rewritten as

1 2L ∇f x (z) 2 = 1 2L ∇f (z) -∇f (x) 2 ≤ f x (z) -f x (x) (1.22) = f (z) -f (x) + ∇f (x), x -z . (1.23) Let z = y, we have 1 2L ∇f (y) -∇f (x) 2 ≤ f (y) -f (x) + ∇f (x), x -y . (1.24)
Similarly, we obtain a symmetric inequality from f y (z):

1 2L ∇f (x) -∇f (y) 2 ≤ f (x) -f (y) + ∇f (y), y -x . (1.25)
Merging these two inequalities, we show that .26) which concludes the proof for 1 ⇔ 6.

1 L ∇f (z) -∇f (y) 2 ≤ ∇f (x) -∇f (y), x -y , ( 1
In addition to the statements in Theorem 1.1.2, we have also derived two intermediate results, i.e., eq.(1.9) and eq.( 1.24), from the proof, which may also be of interests.

For many convex optimization problems, the Lipschitz continuity is an essential assumption in order to prove the convergence of a proposed algorithm. In addition to that, if we further assume the strong convexity condition holds, there exists algorithms that guarantee a linear convergence. .27) for some strong-convexity constant µ > 0.

Definition 1.1.3 (Strong convexity). A differential function f (x) is strongly convex if f (y) ≥ f (x) + ∇f (x), y -x + µ 2 x -y 2 ∀x, y ∈ dom(f ), ( 1
Without loss of generality, we used the differentiability in the definition of strong convexity. However, this is not required since one could replace the gradients with sub-gradients.

Information Theory 13 Information Theory

We will encounter the Jensen's inequality many times in this thesis, which forms the underpinning of variational inference.

Lemma 1.2.1 (Jensen's inequality). If f is a concave function and X ∈ dom(f ) is a random variable, then E[f (X)] ≤ f E[X] .
(1.28)

Moreover, if f is strictly concave, then E[f (X)] = f E[X] ⇔ X = constant. (1.29) Proof. Since f is concave, for any x, y ∈ dom(f ), f (x) ≤ f (y) + ∇f (y) (x -y).
(1.30)

Choosing y = E[X] and taking expectations over both sides, we have

E[f (X)] ≤ f (y) + ∇f (y) (E[X] -y) (1.31) = f (E[X]) + ∇f (E[X]) (E[X] -E[X]) (1.32) = f (E[X]).
(1.33)

If f is strictly concave and X is a random variable, then eq.(1.30) holds only for "<", and

E[f (X)] < f (E[X]). Hence, E[f (X)] = f (E[X]) holds only when X is a constant. In this case, X ≡ E[X].
The mathematical concept of entropy or Shannon entropy was first introduced by [START_REF] Elwood | A mathematical theory of communication[END_REF]. Intuitively, if one attempts to encode a random variable X and assigns a code length to it inversely proportional to p(X), then the code of X = x will take log 2 1 p(x) bits. Thus, the entropy can be interpreted as the average code length in the binary format. The formal definition reads as follows: Definition 1.2.1 (Entropy). Let X ∈ X be a discrete random variable with distribution p(X). The quantity

H(X) = E X∼p(X) log 2 1 p(x) = x∈X p(x) log 2 1 p(x) (1.34)
is called the (Shannon) entropy of X. H(X) is also written as H(p(X)) to emphasize that it is a functional of the distribution p.

Note that it is valid to use logarithm with respect to other bases, since the definition of the entropy differs only by a multiplicative constant. The definition of entropy also holds when X = (X 1 , . . . , X n ) as a vector of random variables, for which p(X 1 , . . . , X n ) is a joint distribution accordingly, and H(X 1 , . . . , X n ) is called the joint entropy. When considering interactions between two random variables, we need an extension of the entropy induced by the conditional distribution.

Background: Convex Optimization and Information Theory Definition 1.2.2 (Conditional entropy). Let X, Y be two discrete random variables with the joint distribution p(X, Y ). The quantity

H(X|Y ) = E y∼p(Y ) H p(X|Y = y) (1.35) = E x,y∼p(X,Y ) 1 log p(x|y) (1.36)
is called the conditional entropy.

We have only defined the entropy for discrete random variables. The continuous counterpart of entropy is called the differential entropy. Definition 1.2.3 (Differential entropy). Let X be a continuous random variable with probability density function f (x). The quantity

h(X) = X f (x) log 1 f (x) dx (1.37)
is called the differential entropy of X.

It turns out that the definition of entropy for both discrete and continuous cases can be unified using Lebesgue integral:

h(X) = - X f log f dµ, (1.38)
where µ is the Lebesgue measure. In particular, for the discrete case, µ is the counting measure on X . In both cases, f is the Radon-Nikodym derivative of the probability measure/distribution P induced by X, namely, f = dP dµ , then

h(X) = - X dP dµ log dP dµ dµ = - X log dP dµ dP.
(1.39)

A related quantity is the Kullback-Leibler (KL) divergence, which is also known as the relative entropy. Definition 1.2.4 (Kullback-Leibler divergence). Let P , Q be two probability measures over X . If P is absolutely continuous with respect to Q, that is, Q(X) = 0 implies P (X) = 0, then the Radon-Nikodym derivative dP dQ exists and the quantity

D KL (P Q) = X dP dQ log dP dQ dQ = X log dP dQ dP (1.40)
is called the Kullback-Leibler divergence.

Note that the difference between the entropy and the KL divergence is that the underlying measure µ is replaced by another probability measure Q and the negative sign is dropped in the definition of KL divergence. For continuous random variables, the Radon-Nikodym derivative dP dQ is computed as the density ratio dP/dµ dQ/dµ . In practice, the entropy and the KL divergence have very distinct usages, although their definitions are similar. The entropy is usually used as an uncertainty "measure", while the KL divergence is considered as a "distance" between two probability distributions. A highly related quantity to both quantities is the mutual information.

1.3 Rate-distortion theory 15 Definition 1.2.5 (Mutual information). Let X, Y be two random variables with joint distribution P XY and marginal distributions P X and P Y . The mutual information of X and Y is defined by

I(X; Y ) = D KL (P XY P X P Y ).
(1.41)

By the definition of mutual information, I(X; Y ) ≥ 0 and I(X; Y ) = 0 if and only if X |= Y . It is also symmetric: I(X; Y ) = I(Y ; X). Thus, I(X; Y ) is widely used to measure the statistical dependence between X and Y . In other words, it measures the information about one random variable provided by another random variable. There are a few other useful properties that connects the mutual information to the entropy. Lemma 1.2.2. Let X, Y be two random variables with joint distribution P XY and marginal distributions P X and P Y . The mutual information of X and Y can be decomposed as

I(X; Y ) = h(X) + h(Y ) -h(X, Y ) (1.42) = h(X) -h(X|Y ) (1.43) = h(Y ) -h(Y |X).
(

1.44)

There is an analogy between the aforementioned information quantities and the true measures over sets. Namely, the joint entropy, conditional entropy and mutual information can be considered as the measure of a set union, set difference and set intersection respectively.

Rate-distortion theory

The rate-distortion theory was invoked to define the "goodness" of a representation of an information source (such as the weights of a neural network). The rate is referred to the length of the representation and the distortion is introduced to measure the quality of the representation. In other words, rate-distortion theory is concerned with the average amount of information about the information source that must be preserved by any lossy data compression scheme such that the reproduction can be subsequently generated from the compressed data with average distortion less than or equal to some specified error.

The readers are invited to follow a detailed description of the rate-distortion theory at Chapter 10 of [START_REF] Thomas | Elements of information theory[END_REF]. This section will only cover the minimal material.

Let us first introduce some notations. Denote by X ∈ X the input random variable, enc(X) the code of X produced by an encoder enc(•) and X ∈ X the estimate of X produced by a decoder dec(•).

Definition 1.3.1. A distortion function is a mapping d : X × X → R + . Definition 1.3.2.
The distortion associated with an encoder enc : X → {1, . . . , K} and a decoder dec : {1, . . . , K} → X is defined as d(x, dec(enc(x))). There is a monotonic trade-off between the rate of the code and the expected distortion. Essentially, the larger the rate, the smaller is the achievable distortion. This trade-off is characterized by the celebrated rate-distortion theorem of Shannon and Kolmogorov. .46) In general, R(D) is difficult to calculate directly since the mutual information is intractable. However, since the mutual information has a nice variational characterization, we are able to compute R(D) approximately by an iterative procedure. We will first introduce the related properties of I(X; X) and R(D), and then derive the variational characterization of R(D). Lemma 1.3.1. The mutual information I(X; X) is functional of p(x) and p(x|x), thus it can be equivalently rewriten as I(X; X) = I(p(x), p(x|x)). Moreover, the mutual information is convex in p(x|x) and concave in p(x).

D := E x∼p(x)
R(D) = min p(x|x) : x,x p(x)p(x|x)d(x,x)≤D I(X; X) with (1.45) I(X; X) := x,x p(x)p(x|x) log p(x|x) p(x) and p(x) = x p(x)p(x|x). ( 1
Proof. First, we decompose

I(X; X) = H( X) -H( X|X), = H(p(x)) -H(p(x|x), p(x)).
Given that both H(p(x|x), p(x)) and p(x) = x p(x|x)p(x) are linear in p(x), and H(p(x)) is a concave function in p(x), it immediately follows that I(X; X) = I(p(x), p(x|x)) and it is concave in p(x).

To prove the convexity wrt p(x|x), we rewrite

I(X; X) = E p(x) D KL (p(x|x) p(x)).
Since the KL divergence is convex in both arguments, fixing p(x), we have I(X; X) is convex in p(x|x).

Proposition 1.3.2. For a rate-distortion function R(D) wrt a distribution p(X), we have the following properties

1. R(D) is nonincreasing in D for all D ≥ 0. 2. R(D) is convex in D for all D ≥ 0. 3. R(D) is continuous in D for all D ≥ 0. 4. R(D) ≤ H(p), ∀D ≥ 0.
Proof. Having defined R(D) as the solution of a constrained minimization, we have 1. The constraint set will be enlarged if D increases. Thus, the minimum can only remain unchanged or smaller, which implies that R(D) is nonincreasing.

2. Consider two rate-distortion pairs (R 0 , D 0 ) and (R 1 , D 1 ), which are achieved at q 0 (x|x) and q 1 (x|x) respectively. Now define

q α (x|x) = αq 0 (x|x) + (1 - α)q 1 (x|x). Then, D α = αD 0 + (1 -α)D 1 . By Lemma 1.3.1, we have R(D α ) ≤ I(p, q α ) ≤ αI(p, q 0 ) + (1 -α)I(p, q 1 ), = αR(D 0 ) + (1 -α)R(D 1 ), = αR 0 + (1 -α)R 1 ,
which implies the convexity.

Assume by contradiction

that R(D) is discontinous at D α such that R(D α ) = lim D→D - α R(D) > lim D→D + α R(D).
Then, there exists two end points D 0 ∈ (0, D α ) and D 1 ∈ (D α , +∞) such that αR(D 0 ) + (1 -α)R(D 1 ) < R(D α ) for some α ∈ (0, 1), which is a contradiction to the convexity of R(D).

This is due to

I(X; X) = H(X) -H(X| X) ≤ H(X). Then, R(D) ≤ min p(x|x) : x,x p(x)p(x|x)d(x,x)≤D H(X) = H(X), since H(X) is independent of p(x|x).
Since R(D) is convex, for any β ≤ 0, there exists a point on the curve of R(D) for 0 ≤ D ≤ D max , such that the slope of a tangent to the curve of R(D) at that point is equal to -β. Denote such a point by (D s , R(D s )), which is not necessarily unique. Then the tangent line intersects with the R-axis at (0, R(D s ) -s • D s ), where s = -β. See To further simplify the minimization of the RHS of (1.49), we resort to the following variational characterization of the mutual information.

Lemma 1.3.2 [START_REF] Thomas | Elements of information theory[END_REF] (Theorem 10.8.1)). The mutual information has a variational form:

I(X; Y ) = min m(y) D KL (p(x, y) p(x)m(y)),
where the argmin is attained at m * (y) = p(y) = x p(x, y).

By applying Lemma 1.3.2, the optimization in (1.49) (1.51) where M is the set of valid instances of m(x).

becomes min p(x|x)∈P x min m(x)∈M I(p(x|x), m(x)) + β • D(p(x|x)),
For the above characterization, [START_REF] Arimoto | An algorithm for computing the capacity of arbitrary discrete memoryless channels[END_REF], [START_REF] Blahut | Computation of channel capacity and rate-distortion functions[END_REF] provide a convergent iterative algorithm for computing p * (x|x) and m * (x) given a fixed β. Proposition 1.3.3 (Blahut-Arimoto Algorithm). Let M := {m | m(x) > 0, ∀x}. Provided an initial p t (x) at t = 0. At iteration t > 0, the Blahut-Arimoto algorithm takes the following steps: βd(x,x) x m t (x )e -βd(x,x) ,

p t (x|x) = m t (x)e -
(1.52)

m t+1 (x) = x p(x)p t (x|x). (1.53)
Moreover, the algorithm converges to the global minimum.

Proof. The proof can be found in [START_REF] Blahut | Computation of channel capacity and rate-distortion functions[END_REF] or Cover and Thomas (2012).

Introduction

Most of problems in machine learning amount to learn the mapping between inputoutput pairs of the form (x, y) given an iid sample of input-output observations. In the case when Y is high-dimensional and represents a space of structured objects, the problem is known as structured prediction. It includes applications such as semantic image segmentation, where Y = {1, . . . , m} k for an image with k pixels (each takes m possible values); human pose estimation, where Y = R k for k body joints; dependency parsing, where Y is the set of spanning trees, and so on. Accordingly, the input space X may also be structured, but the structure among the input is usually implicitly captured by the feature representation.

Structured prediction is computationally expensive even it has already saved a large amount of computation compared with its unstructured counterpart. For example, a discrete output space, e.g, Y = N k , contains an exponential number of elements and grows very quickly when k increases. Therefore, the operations involving the whole output space is generally intractable, e.g., computing the mean E[Y ] is NP-hard. A probabilistic graphical model (PGM) is a special family of distributions, where conditional independence (CI) assumptions among the domain are made based on human knowledge. The CI assumptions enable a factorization in the distribution, which reduces a global computation to local computations within the factors. Another key idea of PGMs is that the CI assumption can be linked to separation in a graph by associating each node a random variable. This is where the name of PGM comes from. Adding conditional independences will basically reduce the treewidth1 of the graph, and thus make the statistical inference (approximately) solvable in polynomial time.

Although the CI assumptions yield computational benefits, one would ask whether these assumptions are realistic? A probabilistic graphical model is sometimes referred to as a knowledge-driven method in constrast with data-driven methods such as deep learning. It is knowledge-driven since we derive a model from high-level human knowledge to hypothesize the observed data. Thus, the model itself is, at least partially, interpretable. For example, a time series is naturally modeled as a chain graph in the context of PGMs. This amounts to introducing the first-order Markovian assumption. For real problems, such as weather forecasting, this assumption leads to a probabilistic model based only on the observation of yesterday. This is certainly not the best choice, but it is intuitive from a human perspective. In general, PGM offers a practical framework to embed a wide range of independence assumptions, which is not easy for data-driven methods.

In the following, I will present the key ingredients of PGMs. For a sake of clarity the presentation will concentrate on variational inference.

Models

Let X := (X 1 , . . . , X n ) denote a random vector, where each element X i is a random variable taking value in the space X i . A PGM with respect X includes a probability distribution p(x) over X and a graph G = (V, E), such that each node in V is associated with a random variable and the edges in E encode the relationships/dependencies between X i 's. Following [START_REF] Wainwright | Graphical models, exponential families, and variational inference[END_REF], I treat p(x) as the Radon-Nikodym derivative of the probability measure with respect to the base measure (which is counting measure in the discrete case and Lebesgue measure in the continuous case).

Factorization

Given G, we are able to incorporate the CI assumptions into the definition of p(x) through the factorization according to G. Specifically, p(x) is expressed as a product of nonnegative local functions (called potential functions), each of which literally involves only a small set of variables forming a factor. In the directed case, a factor denotes the parent-child neighborhood. In the undirected case, a factor is a clique, which is a fully connected subgraph, such as a node or an edge. Note that the set of factors is not necessarily the set of maximal cliques, i.e., the set of cliques that are not properly contained within any other clique. Therefore, the factorization according to G may not be unique. From a modeling perspective, a model of p(x) is more powerful by including higher-order cliques, since it requires fewer CI assumptions, but at the same time the computational overload is increased.

Denote by A the set of factors. For a ∈ A, we have a potential function ψ a : X a → R, where X a := × i∈a X i . We say that p(x) factorizes according to G if

p(x) = 1 Z a∈A ψ a (x a ), with Z = x∈X a∈A ψ a (x a ), (2.1)
where Z is called the partition function with respect to ψ, serving as a normalizing constant. For undirected graphs, the corresponding PGMs are called Markov random fields (MRFs) and p(x) is called the Gibbs distribution. For DAGs, the corresponding PGMs are called Bayesian networks (BNs) and p(x) takes a special form:

p(x) = i∈V ψ i (x i , pa(x i )), (2.2)
where ψ i ≥ 0 and ∀i, ∀pa(x i ), x i ψ i (x i , pa(x i )) = 1. In this case, we always have Z = 1, since the potential function ψ i (x i , pa(x i )) is exactly the conditional probability distribution p(x i |pa(x i )). Note that, without the factorization, the time complexity for computing Z is O(|X |), where |X | returns the cardinality of the space X . For PGMs, the time complexity reduces to an exponential function in the treewidth, which is much smaller than O(|X |).

Introduction to Probabilistic Graphical Models

A slightly more general graphical representation is called the factor graph [START_REF] Frank R Kschischang | Factor graphs and the sum-product algorithm[END_REF], which is a bipartite graph G = (V, E, A) including the set of variable nodes V , the set of factor nodes A, and the set of edges between a factor and a variable. An example of the factor graph is shown in Figure 2.1. The factor graph is particularly useful in understanding the belief propagation algorithms for probabilistic inference [START_REF] Frank R Kschischang | Factor graphs and the sum-product algorithm[END_REF]).

Equivalence between conditional independence and factorization

The key feature of PGMs is the connection between the graph and the factorization of the probability distribution. The connection is built upon the CI assumptions. In fact, the graph itself does not specify the conditional independences. The connectivity is a proxy to reason the dependencies between random variables since each random variable is associated with a node. In analogy, verifying a conditional independence, S ⊥ ⊥ T | E, is equivalent to checking if S is separated from T by E, where S, T, E are three disjoint sets of nodes. The concept of separation is not uniquely defined. For a undirected graph, the definition of separation is straightforward:

S is separated from T given E, if E intersect with every path between S and T .

For a directed acyclic graph (DAG), the d-separation [START_REF] Koller | Probabilistic graphical models: principles and techniques[END_REF]) is used, for which I adopt an informal definition from [START_REF] Murphy | Machine Learning: A Probabilistic Perspective[END_REF]: S and T are d-separated given E, if any undirected path between S and T satisfies one of the following conditions:

1. The path contains an indirect causal-effect structure, A → M → B or an indirect evidential-effect structure A ← M ← B, where M ∈ E.

2. The path contains a common-causal structure, A ← M → B, where M ∈ E.

3. The path contains a V-structure, A → M ← B, where neither M nor its descendants is in E. Now, we are ready to define the two types of CI in PGMs induced by the factorization and the graph respectively:

• Independences in distribution:

I(p) := {(A ⊥ ⊥ B | E) p(A, B|E) = p(A|E)p(B|E)}.
• Independences in graph:

I(G) := {(A ⊥ ⊥ B | E) A is separate/d-separated from B given E}.
We say G is an I-Map of p(x) if I(G) ⊆ I(p), which means that G does not contain any local independence that does not hold in p(x). Conversely, p(x) may have additional independences that are not reflected in G. As a special case, if G is fully connected, then it is an I-Map for any distribution since it does not assert any conditional independence assumption.

There is a fundamental connection between CI assumptions and the factorization. Koller et al. (2009, Theorem 3.1, 3.2, 4.1, 4.2) have shown this connection for the directed case and the undirected case respectively. The results can be summarized as follows.

1. If p(x) factorizes according to G, then G is an I-Map of p(x).

2. If G is an I-Map of p(x), then p(x) factorizes according to G.

For the second statement, we need an additional assumption for undirected graphs: p(x) is positive, which is also known as the Hammersley-Clifford theorem (Koller et al. 2009, Theorem 4.2).

Is I(G) = I(p)?

In addition to the above equivalence, one may be interested in whether the statement I(G) ⊇ I(p) also holds true given that G is an I-Map of p(x) and p(x) factorizes over G. If so, the set of conditional independences encoded in G is equivalent to that encoded in p(x). Unfortunately, this desired property is not true in general as shown by a counterexample in Koller et al. (2009, Example 3.3), where the CI assertion A ⊥ ⊥ B is not refected in the graph A → B, even if p(A, B) = p(A)p(B|A) does factorize according to the graph.

If p(x) factorizes according to G, p(x) is specified by the set of potential functions {ψ a } a∈A by eq. (2.1). The CI assertions outside I(G) can then be viewed as the set of constraints on the potential functions, which can be written as polynomial equalities about the potential functions (Koller et al. 2009, Exercise 3.13). A basic property of polynomial functions is that it is either identically zero or non-zero almost everywhere. Hence, the set of potential functions that satisfy the CI outside I(G) has measure zero. Hence, for almost all instances of p(x) that factorize according to G, we have I(G) = I(p). For a more rigorous analysis, please refer to Koller et al. (2009, Section 3.3.2). Back to the counterexample aforementioned, the CI outside I(A → B) is given by p(B|A) = p(B). Then, the set {p(A, B) | p(B|A) = p(B)} has measure zero. This makes sense since I(A → B) = I(p) except for a special case when p(B|A) = p(B).

Introduction to Probabilistic Graphical Models

Comparison between BNs and MRFs

To finalize this section, I make a comparison between the Bayesian networks and the Markov random fields. There is no concise conclusion on which model is more powerful. In fact, they introduce relatively different CI assumptions. For example, MRFs cannot precisely represent the CIs induced by the v-structures, while BNs cannot precisely represent the CIs induced by cyclic structures (Murphy 2012, Section 19.2.3). A special case is the chordal graph, which can be perfectly modeled by either BNs or MRFs. In addition, we say a marginalization is closed for p(x) if p(x) factorizes in G implies p(x V \i ) factorizes in the induced graph by removing node i and connecting all neighbors of i. This however only holds for the leafs in the case of BNs. A comparison between Bayesian networks and Markov random fields is shown in Table 2

Inference

Given a probability distribution p(x) defined by a PGM, our focus will be solving one or more of the following probabilistic query problems, which are generically referred to as inference problems:

1. Computing the density function p(x) via eq.( 2.1): one needs to estimate the value of the partition function Z.

2. Computing the marginal distribution p(x A ) over a particular subset A ⊂ V of nodes.

3. Computing the conditional distribution p(x A |x B ) for two disjoint subsets A, B.

Computing the mode of p(x).

The first three problems are called probabilistic inference or marginal inference, since they all involve computing summations of the form x S a∈A ψ a (x a ), specifically, the quantities read as 1. p(x) = a∈A ψa (xa) x a∈A ψa(x a ) , the density function. ψa(xa) x a∈A ψa(x a ) , the marginal distribution.

Inference 27

p(x

A ) = x\x A a∈A

p(x

A |x B ) = a∈A ψa(xa)
x \x B a∈A ψa(x a ) , the conditional marginal distribution. On the other hand, the problem of computing the mode is relatively easier, which by definition can be expressed as arg max

x p(x) = arg max x ψ(x).
(2.3)

Computing the mode is sometimes known as decoding or maximum a posteriori (MAP) inference or most probable explanation (MPE) inference.

Although these inference problems are NP-hard in the worst case, exact inference is possible for certain special cases, such as when the graph is a tree (i.e., an undirected graph without cycles or a directed graph in which each node has a single parent) or when the (loopy) graph whose tree decomposition satisfies the running intersection property, meaning that any subset of nodes containing a given variable forms a connected component. For a tree-structured PGM, the sum-product belief propagation algorithm yields exact marginals for each clique. A variant called max-product belief propagation can be used for solving the MAP inference problem. From an algorithmic point of view, sum-product belief propagation is a form of nonserial dynamic programming [START_REF] Bertele | Nonserial dynamic programming[END_REF] while max-product belief propagation is a standard dynamic programming algorithm with the optimal substructure induced by the tree-structured graph. These two algorithms have a computational complexity that scales only linearly in the number of nodes. The same algorithms can also be extended to the tree decomposition case with running intersection property, which is known as the junction tree algorithm (see for example Koller et al. (2009, Chapter 10) or Wainwright (2008, Chapter 2) for a complete description), and the computational complexity also depends on the treewidth of the graph.

However, there are many PGMs, e.g., the grid-structured graphs for semantic image segmentation, for which the treewidth is infeasibly large. Therefore, the exact inference may not be practical any more. Many smart ideas for approximate inference have been proposed, including variational methods [START_REF] Michael I Jordan | An introduction to variational methods for graphical models[END_REF] and Monte Carlo methods [START_REF] Robert | Monte Carlo statistical methods[END_REF]. The former casts the inference as an optimization by minimizing some divergence (e.g., the Kullback-Leibler divergence) between an approximate distribution and the intractable true distribution. The approximate distribution is then used as a proxy for computing the quantities of interest for probabilistic queries (such as the mean and the mode of the population). The latter is based on a simple idea: generating samples from a proposal distribution, which are then processed to compute any quantity of interest. The most popular Monte Carlo method is called Markov chain Monte Carlo (MCMC), including implementations such as the Gibbs sampling and the Metropolis-Hastings algorithm [START_REF] Metropolis | Equation of state calculations by fast computing machines[END_REF], which is guaranteed to work in high-dimensional spaces. By generating sufficient samples, the approximate quantities are able to achieve any level of accuracy. That being said, both the Introduction to Probabilistic Graphical Models bias and the variance of MCMC estimate approaches zero as the Markov chain runs longer and longer. On the other hand, variational methods are considered if the error due to the variance is higher than that of the bias within a short amount of time. This is usually the case for the inference and learning problems of PGMs. Because of that variational methods are preferred over Monte Carlo methods. Moreover, comparing to MCMC, variational inference is easier to scale to large data. I will therefore only cover variational inference in this chapter.

Marginal inference

If exact inference for a PGM is computationally intractable, we can alternatively search for a tractable PGM that approximates the original PGM as close as possible. For instance, if we remove all other conditional independences except for those contained in a spanning tree of the graph, we obtain an approximate PGM with tractable inference. This strategy is critical since, under a certain time budget, approximate inference may be the only afordable choice for large-scale machine learning problems.

The key idea behind the variational view of inference is that we perform inference on an approximate distribution that remains the closest to the original PGM in terms of some distance measures, such as KL divergence, Jensen-Shannon divergence, Wasserstein distance and so on.

Exponential family formulation for inference

Consider a PGM= (p(x), G) with p(x) = 1 Z a∈A ψ a (x a ). It turns out that a special parameterization of ψ a (x a ) offers a different interpretation of the inference. Specifically,

ψ a (x a ; θ a ) = exp θ a , φ a (x a ) (2.4)
with both θ a and φ a (x a ) in R da (d a ≥ 1 is a hyperparameter). As an example, φ i (x i ) is usually provided as a |X i |-dimensional one-hot vector, denoted by e x i , with 1 for the x i -th element and 0 elsewhere; and φ a (x a ) = ⊗ i∈c φ i (x i ) is the result of a series of tensor products. Then, we can interpret θ a = (log ψ a (x a )) xa∈Xa to be the vector of log-potentials. In this chapter, we assume φ a (x a ) is not learnable. However, this assumption can be relaxed in the case of inference, since we can always absorb φ a into θ a and let ψ a (x a ) = exp( θ a , e xa ). With this parameterization of ψ, we obtain an exponential family form for p(x) as .5) where θ := (θ a ) a∈A is the natural parameter; φ(x) := (φ a (x a )) a∈A is called the sufficient statistics; and F (θ) = log Z is the log-partition function in which we explicited the dependency on θ. Note that exponential family is indeed a strict subset of the distributions that factorize according to G, among which it achieves the maximum entropy.

p θ (x) = exp θ, φ(x) -F (θ) = exp a∈A θ a , φ a (x a ) -F (θ) , ( 2
The different interpretation of the inference is that the variational inference for exponential family with Kullback-Leibler (KL) divergence is equivalent to approximating the log-partition function. To give a formal justification, we will first go through some properties of the log-partition function F (θ).

Lemma 2.3.1 (Wainwright 2008, Proposition 3.1). The log-partition function associated with any regular exponential family satisfies 1. ∂F (θ) ∂θa = E p θ φ a (X a ) for all a ∈ A.

H(q) = X q(x) log q(x)ν(dx). (2.6)
Proof. Recall that the Legendre-Fenchel transformation of F (θ) is

F * (q) = sup θ∈dom(F ) [ X θ, φ(x) q(x)ν(dx) -F (θ)]. (2.7)
Taking derive with respect to θ yields the condition

q(x) = exp θ, φ(x) X exp θ, φ(x) ν(dx)
.

(2.8) Thus, log q(x) = θ, φ(x) -F (θ).

(2.9)

Using the above equality and the fact that X q(x)ν(dx) = 1, eq.( 2.7) can be rewritten as .11) It concludes the proof.

F * (q) = sup θ∈dom(F ) [ X q(x) log q(x) + F (θ) ν(dx) -F (θ)] (2.10) = X q(x) log q(x)ν(dx). ( 2

Computing F (θ) as a KL minimization

We show in the following theorem that the KL minimization is related to the Legendre-Fenchel transformation of F (θ).

Introduction to Probabilistic Graphical Models

Theorem 2.3.1. For p θ in exponential family defined in eq. (2.5), minimizing D KL (q p θ ) over a convex set P is equivalent to finding an optimal solution to the problem

max q∈P θ, E q [φ(X)] + H(q) .
(2.12)

If p θ ∈ P, the above optimal objective is exactly equal to F * * (θ) ≡ F (θ).

Proof. Substituting the definition of p θ (x), we observe that

D KL (q p θ ) = F (θ) -E q log exp θ, φ(X) q(X) (2.13) = F (θ) -E q [ θ, φ(X) ] -H(q) ≥ 0. (2.14)
Since F (θ) does not depend on q(x), we have that

max q∈P E q [ θ, φ(X) ] + H(q) ⇔ min q∈P D KL q p θ . (2.15)
Since the KL divergence is strictly convex in q for θ fixed, if p θ (x) ∈ P, we have min q∈P D KL (q p) = 0, and the argmin is attained at q * (x) = p θ (x), yielding .16) Note that H(q) = -F * (q). Since F (θ) is convex in θ by Lemma 2.3.1, we have F (θ) = F * * (θ).

F (θ) = E q * [ θ, φ(X) ] + H(q * ). ( 2
Theorem 2.3.1 offers a variational formulation for computing F (θ), which is the key idea that casts the marginal inference as an optimization problem. More importantly, it derives a dual representation for F (θ) with the dual variable q ∈ P. Denote by ∆ is the probability simplex. It is easy to see that if P = ∆, Theorem 2.3.1 holds since p θ ∈ ∆, that is,

F (θ) = max q∈∆ θ, E q [φ(X)] + H(q) .
(2.17)

Note that q is indeed a functional corresponding to a primal functional θ ≡ log ψ, which means that the same result holds even if we did not use the linear parameterization θ(x) = log ψ(x) = θ, φ(x) . Since now we specify the primal variable θ = (θ a ) a∈A to be a vector with the same dimensionality as that of φ(x), we expect the dual variable to be a vector as well rather than a functional.

Denote by µ the dual variable, which is also known as the mean parameter of the exponential family. In fact, the primal and the dual is coupled by the so-called moment matching condition: (2.18) which is given by the Fenchel-Young's inequality when the equality holds. If we were using functional primal and dual, namely, θ and q respectively, the above condition became

µ a = ∂F (θ) ∂θ a = E p θ [φ a (X a )], ∀a ∈ A,
q(x) = ∂F (θ) ∂θ(x) = p θ (x). (2.19)
Note that, for all θ ∈ Θ := {θ | F (θ) < +∞}, there is an induced dual variable µ by the moment matching condition. This induces a space of µ, called marginal polytope, and denoted by

M := {(µ a ) a∈A | ∃θ ∈ Θ s.t. E p θ [φ a (X a )] = µ a , ∀a ∈ A}.
(2.20)

With the same reasoning, we can define the "functional" version of the marginal polytope to be .21) Due to the coupling between the primal and the dual, in eq.( 2.17), the space of q can actually be Q, which is a subset of ∆. Thus, by the definition of Q, we have

Q := {q ∈ ∆ | ∃θ ∈ Θ s.t. q(x) = p θ (x), ∀x ∈ X }. ( 2
F (θ) = max η∈Θ θ, E pη [φ(X)] + H(p η ) .
(2.22)

Similarly, we have, in the next theorem, a dual representation for F (θ) in terms of the marginal polytope.

Theorem 2.3.2 (Wainwright 2008, Theorem 3.4). For any µ belonging to the interior of M (denote by M • ), there exists θ ∈ Θ := {θ | F (θ) < +∞}, such that the moment matching condition holds, namely,

µ a = E p θ [φ a (X a )], ∀a ∈ A. (2.23)
For any natural parameter θ ∈ Θ satisfying the above condition, the Fenchel conjugate of F (θ) is defined by

F * (µ) =        -H(p θ ) if µ ∈ M • lim ν∈M • →µ F * (ν) if µ ∈ M \ M • +∞ o.w.
(2.24)

Hence, the log-partition function F (θ) has a variational representation in terms of Fenchel duality:

F (θ) = max µ∈M θ, µ -F * (µ) . (2.25)
Intuitively, the variational representation of F (θ) in eq.( 2.25) is itself a maximization over the marginal polytope. It in general does not reduce the computational complexity of the inference problem, since both M and F * (µ) are inherently intractable. In particular, M has an exponential number of linear constraints; F * (µ) depends on the negative entropy of p θ (x), but which in general lacks an explicit form to compute in terms of µ. Furthermore, since F is not strictly convex, unless p(x) belongs to a minimal exponential family, meaning that there does not exist a nonzero θ such that θ, φ(x) = constant for any x, θ(µ) is in general not uniquely determined. All these entanglements render difficulty to formulate the Fenchel conjugate of F (θ) explicitly.

There are special cases where the variational representation of F (θ) is amenable, such as PGMs induced by degenerate graphs (no edges) and tree-structured graphs, in which cases both M and F * take special forms. The ideas, such as mean-field variational inference and Bethe variational inference, rely on extending the special forms of M and F * to general PGMs, which are categorized as approximate marginal inference.

Mean-field inference

The idea of mean-field approximation first appeared in statistical field theory [START_REF] Parisi | Statistical field theory[END_REF]. The mean-field variational inference has been sucessfully applied to Bayesian inference and other probabilistic reasonings (Blei et al. 2017). In the context of PGMs, the mean-field approximation arises from ignoring certain types of dependencies between random variables. • Suppose that T = (V, E T , A T ) is a subgraph of the original graph G, such that T is acyclic, in other words,

E T ⊂ E, A T = V ∪ E T ⊂ A.
• Denote by Θ T the set of natural parameters induced by T , namely,

Θ T = {θ ∈ Θ | θ a = 0 if a ∈ A T }.
The mean-field variational inference for such a PGM picks an approximate distribution q(x) by solving min q∈Q mf D KL (q p θ ), (2.26) where

Q mf := {q ∈ ∆ | ∃θ ∈ Θ T s.t. q(x) = p θ (x)
, ∀x ∈ X } and ∆ is the probability simplex. This is equivalent to approximating F (θ) by

F mf (θ) := max µ∈M mf θ, µ -F * mf (µ) with (2.27) M mf := {(µ a ) a∈A | ∃θ ∈ Θ T s.t. E p θ [φ a (X a )] = µ a , ∀a ∈ A}, (2.28) F * mf (µ) := - ij∈E T H(q ij ) + i∈V (d(i) -1)H(q i ), (2.29)
where q i , q ij satisfies E qa [φ a (X a )] = µ a for all i, ij ∈ A T . Moreover, F mf (θ) ≤ F (θ).

Inference 33

Proof. By Theorem 2.3.1, we have min

q∈Q mf D KL (q p θ ) ⇔ max q∈Q mf θ, E q [φ(X)] + H(q) . (2.30)
By the definition of Q mf , there must exists a η ∈ Θ T , such that q = p η . Since T is acyclic, there exists a junction tree representation of p η (x) (Wainwright 2008, eq.(2.12)):

p η (x) = ij∈E T q ij (x ij ) i∈V q i (x i ) d(i)-1 , (2.31)
where q a (x a ) is the marginal distribution of x a , i.e., q a (x a ) = p η (x a ), for all i, ij ∈ A T , and d(i) is the number of edges in E T connected to node i. Hence,

H(q) = H(p η ) = ij∈E T H(q ij ) - i∈V (d(i) -1)H(q i ).
(2.32) By Theorem 2.3.2, since η also belongs to Θ, there exists a µ ∈ M • such that E q [φ a (X a )] = µ a , ∀a ∈ A. Note that the mapping defined by this equation between q and µ is one-to-one, since q ≡ p η belongs to a minimal exponential family. This established a change of variable for the RHS optimization in eq.( 2.30):

max µ,q θ, µ + H(q) (2.33) s.t. µ ∈ M • ∩ {µ | E q [φ(X)] = µ} and q ∈ Q mf .
(2.34)

Since q can be seen as a function of µ, the equivalence follows by absorbing the above constraints in eq.( 2.34) into a single constraint encoded by M mf , leaving a single variable µ.

The mean-field approximation F mf (θ) yields a lower bound of the true value of F (θ) simply because M mf is a subset of M.

Note that the variational problem in eq.( 2.27) involves solving equations E qa [φ a (X a )] = µ a for all a ∈ A T , which is in general nontrivial. For a special case where X a is discrete, A = V ∪ E and φ a (x a ) is an one-hot encoded tensor such that φ i (x i ) = e x i and φ ij (x ij ) = φ i (x i )φ j (x j ) , we have that

E qa [φ a (X a )] = µ a ⇔ q a = µ a .
In this case, the mean parameter µ a (x a ) is indeed the marginal probability of x a . If in addition, we choose T = (V, E T = ∅, A T = V ), then

F * mf (µ) = - i∈V H(µ i ),
and the optimization in eq.( 2.27) can be rewritten as

max µ θ, µ + i∈V H(µ i ) (2.35) s.t. ∀a ∈ A : µ a ∈ ∆ and ∀ij ∈ E : µ ij = µ i µ j . (2.36)
This is a concave optimization over a non-convex set, which is typically optimized by a coordinate ascent scheme. For the optimization in eq. (2.35), the update rule is

µ i (x i ) ∝ exp θ i , φ i (x i ) exp j∈neighbors(i) θ ij , φ i (x i )µ j .
(2.37)

Loopy belief propagation

Although the belief propagation algorithm was originally designed for acyclic graphs, it was found by [START_REF] Pearl | Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference[END_REF] that the same propagation rules can be extended to general graphs with loops, resulting in an iterative algorithm called loopy belief propagation (LBP). For a discrete PGM with density p(x)

= 1 Z i∈V ψ i (x i ) ij∈E ψ ij (x i , x j
), LBP involves the following updates:

m i→j (x j ) = x i ψ i (x i )ψ ij (x i , x j ) k∈neighbors(i)\j m k→i (x i ), (2.38) μi (x i ) ∝ ψ i (x i ) j∈neighbors(i) m j→i (x i ), (2.39)
where m i→j ∈ R |X j | is the message sending from node i to node j; and μi is an estimate of the marginal probability of node i. At each iteration of LBP, each node will send a message to each of its neighbors. The iterations continue until (µ i ) i∈V converges. Note that LBP does not guarantee the convergence of μ. Thus, in practice, a heuristic called damping is used to facilitate the convergence of LBP, which uses damped messages of the form mi→j (x j ) ← λm i→j (x j ) + (1 -λ) mi→j (x j ) (2.40) rather than the original messages in the iterations. For a more general PGM with higher order factors, we usually apply LBP on its factor graph. Please refer to the monograph by Nowozin et al. (2011, Section 3.2) for a full description. [START_REF] Yedidia | Understanding belief propagation and its generalizations[END_REF] shows that LBP amounts to solving the fixed-point equations induced by the KKT conditions of the Bethe approximation of F (θ). Thus, if LBP converges, it converges to a fixed point of the KKT system.

To formally introduce the Bethe approximation of F (θ), the local consistency polytope should be introduced first, which is defined by

L := {(τ a ) a∈A | ∀a ∈ A : τ a ∈ ∆, ∀i ∈ a, x i ∈ X i : xa\x i τ a (x a ) = τ i (x i )}. (2.41)
Intuitively, in the context of PGMs, the local consistency polytope contains all pseudo marginals that are locally consistent. They are pseudo simply because the real marginals not only satisfy these constraints but also many more marginalization constraints. On the other hand, for an exponential family, the marginal polytope contains real marginals if φ a (x a ) = e xa , ∀a ∈ A, x a ∈ X a . The relationship between these two polytopes is characterized by the following lemma.

Lemma 2.3.3 (Wainwright 2008, Proposition 4.1). For any PGM = p θ (x), G = (V, E, A) such that p θ (x) ∝ exp a∈A θ a , e xa , the marginal polytope is a subset of the local consistency polytope, that is, M ⊆ L. If, in addition, G is acyclic, M = L.

The Bethe approximation of F (θ) is based on the observation that L can be used to approximate M, which is specified as follows.

F Bethe (θ) := max µ∈L θ, µ -F * Bethe (µ) with (2.42) F * Bethe (µ) := - a∈A H(µ a ) + i∈V d(i)H(µ i ), (2.43)
where d(i) is the number of factor nodes connected to i in the factor graph (note that V ⊂ A). Comparing to eq.( 2.25), it involves two kinds of approximation:

• The marginal polytope M is replaced by the local consistency polytope L.

• The Fenchel conjugate of F (θ) is approximated by

F * Bethe (µ) = -H(q), where q ∈ Q Bethe := q | ∃τ ∈ L s.t. q(x) = a∈A τa(xa) i∈V τ i (x i ) d(i) , ∀x ∈ X .
Unlike F mf (θ) lower bounds F (θ), the Bethe approximation F Bethe does not yield a bound, since L is an outer bound of M, which only produces a variational relaxation to the original maximization problem.

The Bethe variational inference has many variants based on different entropy approximations:

• Based on the insight that the Bethe entropy approximation is induced from trees, the Kikuchi variational inference proposed by [START_REF] Yedidia | Dual decomposed learning with factorwise oracle for structural SVM of large output domain[END_REF] (also known as the generalized belief propagation) strengthens the approximation by exploiting the more general junction trees to derive the local consistencies as well as the entropy approximation.

• The Bethe objective turns out to be non-concave although the local consistency polytope is a convex set. Consequently, the Bethe variational inference may converge to a local optima depending on the initial conditions. Therefore, a line of research focused on concave entropy approximation has been proposed. For example, the tree-reweighted (TRW) entropy approximation proposed by Wainwright et al. (2005b) replaces the Bethe entropy approximation by a convex combination of tree based entropies. In general, for an entropy approximation of the form

H(µ) := a∈A c a H(x a ), (2.44)
Heskes (2006) shows that H(µ) is concave if (c a ) a∈A (also known as counting numbers) can be reparameterized by (α a ≥ 0) a∈A and (α a,i ≥ 0) a∈A,i∈a such that ∀i ∈ V :

c i = α i - a i α a,i , (2.45) ∀a ∈ A : c a = α a + i∈a α a,i .
(2.46) [START_REF] London | The benefits of learning with strongly convex approximate inference[END_REF] shows that if in addition α a is strictly positive for all a belongs to A, H(µ) is strongly concave. However, the Bethe entropy approximation often outperforms the TRW entropy approximation [START_REF] Meshi | Convexifying the bethe free energy[END_REF], which means that convexifying the entropy approximation may have unexpected negative impacts. One solution proposed by [START_REF] Meshi | Convexifying the bethe free energy[END_REF] is that the counting numbers should be tuned to be as close as possible to that of the Bethe counterpart, while they satisfy the sufficient conditions given by [START_REF] Heskes | Convexity arguments for efficient minimization of the bethe and kikuchi free energies[END_REF].

To conclude, the loopy belief propagation is indeed an iterative algorithm to solve the KKT system of the Bethe approximation of the log-partition function. It explains why LBP does not always converge in practice even with the damping trick. Moreover, by viewing it this way, many existing algorithms from (convex) optimization can be applied here with a global convergence guarantee. For example, the Frank-Wolfe algorithm has been applied to TRW and other convexified Bethe formulations [START_REF] Belanger | Marginal inference in mrfs using frank-wolfe[END_REF][START_REF] Rahul | Barrier frank-wolfe for marginal inference[END_REF], where each subproblem amounts to solving a MAP inference (with an off-the-shelf MAP solver). Alternatively, one can employ a stochastic subspace descent [START_REF] Richtárik | Stochastic reformulations of linear systems: algorithms and convergence theory[END_REF] with each subspace corresponding to a spanning tree, where each subproblem amounts to solving a marginal inference for the subspace only (which can be solved exactly by the sum-product belief propagation for trees). In the extreme case when the subspace reduces to a single factor, the algorithm is known as the block-coordinate proximal gradient method (Shalev-Shwartz andZhang 2016, Beck and[START_REF] Beck | On the convergence of block coordinate descent type methods[END_REF]. Note that, unlike LBP, all these algorithms have a convergence guarantee to a stationary point. For strongly concave entropy approximations, these algorithms can even be shown to be linearly convergent2 .

MAP inference

In general, the MAP inference is considered easier than marginal inference, since it can be seen from eq.( 2.3) that the partition function is not involved in the MAP inference. For a PGM with exponential family density, we will see in the following theorem that the difference between the MAP inference and the marginal inference is quite small. Theorem 2.3.4 (Wainwright 2008, Theorem 8.1).

For any PGM = p θ (x), G = (V, E, A) such that p θ (x) ∝ exp a∈A θ a , e xa .
The MAP inference has the following alternative formulations:

max x∈X a∈A θ a , e xa = max µ∈M θ, µ = lim β→+∞ F (βθ) β (2.47)
The first implication from eq.( 2.47) is that the only difference between marginal inference and MAP inference for exponential family is the entropy term. Then, it is natural to hypothesize that anything works for marginal inference may also work for MAP inference. Indeed, many algorithms for MAP inference rely on bounding the marginal polytope by the local consistency polytope. The resulting formulation is called linear programming relaxation (LPR):

max µ∈L θ, µ ≡ a∈A θ a , µ a (x a ) , (2.48)
which is lower bounded by max µ∈M θ, µ . The following proposition describes what we should expect from LPR.

Proposition 2.3.5 (Wainwright 2008, Proposition 8.3). The extrem points of L and M are related as follows:

1. All extreme points of M are of the form (e xa ) a∈A given that x ∈ X , and each one is also an extreme point of L.

For any graph with cycles, L also includes additional extreme points with fractional elements that lie strictly outside M.

If we obtained a fractional solution µ by LPR, an additional rounding scheme is needed to obtain the MAP solution (e.g., the heuristic suggested by [START_REF] Ravikumar | Quadratic programming relaxations for metric labeling and markov random field map estimation[END_REF]).

The LPR for MAP inference can be viewed as a counterpart of the Bethe approximation of F (θ) for marginal inference. However, the equivalence established in eq.( 2.47) does not hold for the Bethe case. That is,

max µ∈L θ, µ = lim β→+∞ F Bethe (βθ) β . (2.49)
This is because the exchange of the "limit" and the "max" is possible only if F * Bethe is convex, which is obviously not the case for Bethe approximation. Consequently, the max-product LBP does not solve the LPR problem unless the graph is treestructured (Wainwright 2008, Section 8.4.2). However, the TRW approximation does not have such a restriction, thus the TRW version of max-product LBP does find a fixed point of LPR's KKT system (Wainwright et al. 2005a).

Since LPR is indeed a linear programming over the local consistency polytope, several authors proposed alternative algorithms for either the primal or the dual problem. Due to the entanglement in L (in particular, the marginalization Introduction to Probabilistic Graphical Models constraints), only a few works optimize the LPR problem in the primal directly. [START_REF] Yanover | Linear programming relaxations and belief propagation-an empirical study[END_REF] show that TRW max-product LBP is significantly faster than general-purpose linear programming solvers. [START_REF] Ravikumar | Message-passing for graph-structured linear programs: Proximal methods and rounding schemes[END_REF] present a double-looped proximal point algorithm for LPR, where each inner loop involves a Bregman projection onto L being equivalent to performing a TRW sum-product LBP.

On the other hand, the dual of LPR, with the dual variable δ := (δ ai ) a∈A,i∈a , has a nice decomposition property in the primal variable µ:

min δ a∈A max xa∈Xa θ a , e xa + b a δ ba , e xa - i∈a δ ai , e x i .
(2.50)

Several authors proposed efficient algorithms for solving the dual problem, which are identified by the same name called dual decomposition. Specifically, [START_REF] Komodakis | MRF optimization via dual decomposition: Message-passing revisited[END_REF] applied subgradient descent on δ achieving a O( 12 ) time complexity to obtain a -accurate solution; [START_REF] Werner | A linear programming approach to max-sum problem: A review[END_REF], Globerson and Jaakkola (2007a), [START_REF] Sontag | Tree block coordinate descent for map in graphical models[END_REF] applied block-coordinate descent on δ by choosing different blocks. Please refer to [START_REF] Sontag | Introduction to dual composition for inference[END_REF] for a survey on dual decomposition. The problem with the block-coordinate scheme is that the objective function of LPR is non-smooth, which may get stuck at a non-stationary point even when the objective is convex. To overcome this limitation, Nesterov's smoothing [START_REF] Yu | Smooth minimization of non-smooth functions[END_REF] has been applied to the dual of LPR [START_REF] Willsky | Convex relaxation methods for graphical models: Lagrangian and maximum entropy approaches[END_REF][START_REF] Hazan | Norm-product belief propagation: Primal-dual message-passing for approximate inference[END_REF][START_REF] Savchynskyy | A study of Nesterov's scheme for Lagrangian decomposition and MAP labeling[END_REF]. Note that this is equivalent to computing

F (βθ) β
with β > 0. In this case, [START_REF] Meshi | Convergence rate analysis of MAP coordinate minimization algorithms[END_REF] has proved a O( 1 β ) time complexity for block-coordinate methods. By further tweaking the primal of LPR to be ρ-strongly concave, Meshi et al. (2015a) has shown that a linear convergence rate (i.e. a O 1 βρ log 1 time complexity) can be achieved. In addition, for a discrete PGM, the MAP inference can be cast as a combinatorial optimization. For example, for a binary MRF with submodular3 potentials, the MAP inference can be solved exactly by GraphCuts [START_REF] Boykov | Fast approximate energy minimization via graph cuts[END_REF]. Please refer to [START_REF] Kappes | A comparative study of modern inference techniques for discrete energy minimization problems[END_REF] for other discrete minimization techniques.

Learning

A PGM is specified by the pair (p(x), G). The notion of learning in PGM is usually referred to as learning the parameters of p(x) rather than learning the conditional independences represented by G. To avoid ambiguity, the former is called parameter estimation, and the latter is called structure learning. To narrow down the discussion, in this section, I consider only the frequentist setting of parameter estimation and assume that the structure is fixed. I will begin with a special case of parameter estimation for exponential family and then continue on the more general structured output learning.

Maximum likelihood estimation of exponential family

Recall that a PGM with the exponential family form is naturally parameterized by its natural parameters. Consider the independent and identically distributed (iid) setting for parameter estimation, where the dataset D := {x i } n i=1 consists of iid data points. The maximum likelihood estimation (MLE) is achieved by maximizing the log-likelihood with respect to the natural parameter, which is an instance of empirical risk minimization with the objective function

L(θ; D) := - 1 n n i=1 log p θ (x i ) (2.51) = -θ, 1 n n i=1 φ(x i ) + F (θ). (2.52) Denote by μ = 1 n n i=1 φ(x i ) the empirical mean parameter. If μ ∈ M • , there exists a θ μ induced by μ such that μ = E p θ μ [φ(x)]
(2.53) by Lemma 2.3.2. Thus, minimizing L(θ; D) amounts to computing the entropy since

H(p θ μ ) = max θ∈Θ θ, μ -F (θ) (2.54) = θ μ, μ -F (θ μ).
(2.55)

In practice, we find θ μ by gradient descent with the gradient .56) These highlight the connection between MLE and maximum entropy principle. The optimality condition of MLE is exactly the moment matching condition.

∇L(θ; D) = -μ + E p θ [φ(x)] = -μ + µ. ( 2

Structured output learning

As we have mentioned in the beginning of this chapter, the structured prediction is a general learning problem for estimating the mapping between the high-dimensional input x ∈ X and output y ∈ Y. In many cases, we are only interested in the structure (i.e. conditional independences) within the output y rather than the structure within the input x, since the latter is often hard to understand from a human perspective. Thus, for structured prediction, we estimate the parameters of the conditional distribution p(y|x) rather than the joint distribution p(x, y). The associated PGM is known as conditional random field (CRF) [START_REF] Lafferty | Conditional random fields: Probabilistic models for segmenting and labeling sequence data[END_REF]).

It should be noted that CRF is not the only model for structured prediction. Since our focus is graphical model, we assume the structured output space decomposes according to the graph given by the CRF, that is,

Y := Y 1 × • • • × Y n .

Introduction to Probabilistic Graphical Models

The potential functions of p(y|x) are usually parameterized as a form of exp θ a (x, y a ) , and the density of CRF is given by

p θ (y|x) = 1 Z(x, θ) exp a∈A θ a (x, y a ) , (2.57)
where Z(x, θ) = Y exp a∈A θ a (x, y a ) ν(dy). To simplify the notations, I will denote θ(x, y) := a∈A θ a (x, y a ). Note that θ a (x, y a ) has no restriction in its form of parameterization. However, most of existing algorithms are designed for the following linear parameterization:

θ a (x, y a ) = θ a , φ a (x, y a ) (2.58)
with θ a ∈ R da the actual parameter to be learned for each factor a and φ a (x, y a ) the feature representation with respect to the input x and the label y a . I will stick with this linear parameterization in the following, although this may not be necessary. To reduce the number of parameters, several papers, e.g., [START_REF] Pletscher | Spanning tree approximations for conditional random fields[END_REF], also considered parameter sharing between factors, which can be hard coded into the parameterization of θ(x, y).

The relationship between CRF and MRF is sometimes called the discriminativegenerative pair (Ng and Jordan 2002), since one could instead define a MRF with density p(x, y) using exactly the same set of potentials for p(y|x). One example is logistic regression v.s naive Bayes. Both are special PGMs with null graph.

To learn a CRF from a dataset D := {(x i , y i )} n i=1 , the maximum likelihood estimation is always a good choice. The objective of MLE reads as

1 n log p θ (y i |x i ) = 1 n n i=1 θ(x i , y i ) -log y∈Y exp θ(x i , y) . (2.59)
It is characterized by a difference of two terms, where the second term is exactly the log-partition function-the main computational difficulty as discussed in the previous sections.

Alternatively, if we know how to evaluate the prediction ŷ with respect to the ground truth y, namely, the task-specific loss (y, ŷ) is provided, then we can directly minimize the expected task-specific loss empirically:

min 1 n n i=1 (y i , ŷi ). ( 2

.60)

This formulation implicitly depends the prediction ŷ, which is usually chosen to be the result of the MAP inference ŷ = arg max y∈Y θ(x, y).

(2.61) Thus, from a computational point of view, the direct loss minimization is more economical, as the log-partition function is not involved.

In case ŷ → (y, ŷ) is non-differentiable and the y → arg max y∈Y θ(x, y) is discontinuous, the max-margin Markov network [START_REF] Taskar | Max-margin markov networks[END_REF] can be used to remove the arg max from the learning objective. The idea was further extended by [START_REF] Tsochantaridis | Large margin methods for structured and interdependent output variables[END_REF], [START_REF] Joachims | Cutting-plane training of structural svms[END_REF] leading to following two convex upper bounds for (y, ŷ):

Margin rescaling : (y, ŷ) ≤ (y, ŷ) + θ(x, ŷ) -θ(x, y) (2.62) ≤ max z (y, z) + θ(x, z) -θ(x, y) (2.63) Slack rescaling : (y, ŷ) ≤ (y, ŷ) • 1 + θ(x, ŷ) -θ(x, y) (2.64) ≤ max z (y, z) • 1 + θ(x, z) -θ(x, y) (2.65)
The original max-margin training in [START_REF] Taskar | Max-margin markov networks[END_REF][START_REF] Tsochantaridis | Large margin methods for structured and interdependent output variables[END_REF] was in fact motivated differently from the above bounds. The key idea is as follows. If we can obtain zero prediction error on D, then the following inequalities hold:

∀i ∈ {1, . . . , n}, y ∈ Y : θ(x i , y i ) -θ(x i , y) ≥ γ, ( 2.66)
where γ := min i θ(x i , y i ) -max y∈Y θ(x i , y) ≥ 0 defines the margin. Since both the direction and the magnitude of θ have an impact on the margin, we thus would like to fix the norm of θ and maximize the margin γ, which yields the following optimization problem max γ,θ :

θ 2 =1 s.t. ∀i ∈ {1, . . . , n}, y ∈ Y : θ(x i , y i ) -θ(x i , y) ≥ γ.
(2.67)

The above problem can be equivalently expressed as a convex quadratic programming of the form

min θ 1 2 θ 2 2 s.t. ∀i ∈ {1, . . . , n}, y ∈ Y : θ(x i , y i ) -θ(x i , y) ≥ 1. (2.68)
In case the zero prediction error cannot be obtained, a slack variable ξ i can be introduced for each data point:

min θ,ξ≥0 λ 2 θ 2 2 + 1 n n i=1 ξ i (2.69) s.t. ∀i ∈ {1, . . . , n}, y ∈ Y : θ(x i , y i ) -θ(x i , y) ≥ 1 -ξ i .
(2.70)

The margin rescaling and slack rescaling are then introduced naturally since we do not want to punish all errors equally. Specifically, the constraints become

Margin rescaling : θ(x i , y i ) -θ(x i , y) ≥ (y i , y) -ξ i (2.71) Slack rescaling : θ(x i , y i ) -θ(x i , y) ≥ 1 - ξ i (y i , y) .
(2.72) I will only focus on the margin-rescaling bound since it is the most commonly used bound in the literature. An in-depth discussion on the slack-rescaling bound can be found in the paper by [START_REF] Choi | Fast and scalable structural svm with slack rescaling[END_REF]. Since the RHS of eq.( 2.63) can be interpreted as a natural generalization of the hinge loss in support vector machine (SVM) (Hazan et al. 2010), the method proposed by [START_REF] Tsochantaridis | Large margin methods for structured and interdependent output variables[END_REF] is usually called structured output SVM (SSVM). The objective of the margin-rescaled SSVM reads as

1 n n i=1 max z (y i , z) + θ(x i , z) -θ(x i , y i ) + λ 2 θ 2 2 .
(2.73)

As a special case when ≡ 0, we retrieve the structured perceptron [START_REF] Collins | Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms[END_REF].

One may notice that, in eq.( 2.73), there exists n max operations in the form of

max z (y i , z) + θ(x i , z) , (2.74)
which is called loss augmented MAP inference. Ideally, we would like the taskspecific loss to be decomposable according to the factors, then eq.( 2.74) reduce to normal MAP inference, since the task-specific loss can be folded into the original potentials to form the new potentials. There are a few non-decomposable exceptions, such as the intersection-over-union (IoU) loss (Blaschko andLampert 2008, Ranjbar et al. 2013) and the area-under-the-curve (AUC) loss [START_REF] Rosenfeld | Learning structured models with the auc loss and its generalizations[END_REF], where efficient subroutines for the loss-augmented inference (Blaschko and Lampert 2008) or proper surrogates of the loss (Yu and Blaschko 2018) are available.

In general, the SSVM training is more costly than MLE training. The difference is that the MLE involves computing the softmax functions (i.e., "log-sum-exp") while the SSVM involves only computing the max functions. Recall that this difference also appears between the MAP inference and the marginal inference.

In the following, I will review several representative algorithms for SSVM, some of which have been implemented in open-source projects such as SVMStruct [START_REF] Joachims | Cutting-plane training of structural svms[END_REF] and Shogun Machine Learning Toolbox [START_REF] Sonnenburg | shogun-toolbox/shogun: Shogun 6.1[END_REF]. [START_REF] Joachims | Cutting-plane training of structural svms[END_REF] emphasize that eq.( 2.69) with margin rescaling can be viewed as a quadratic programming (QP) with n|Y| constraints. This version of SSVM is referred to as the n-slack margin-rescaled SSVM. Since |Y| has an exponential number of elements, it is too cumbersome for standard QP solvers to handle so many constraints. Akin to support vectors in the plain SVM, we expect only a small fraction of the constraints to be active.

Cutting plane algorithm

The cutting plane algorithm proposed by [START_REF] James E Kelley | The cutting-plane method for solving convex programs[END_REF] is designed exactly for addressing this case. It maintains a working set W i for each data point i, which is initialized as an empty set. For each inner iteration, a constraint is added for each data point if the corresponding loss-augmented margin exceeds the current slack variable by more than . Then, in the outer loop, the QP problem is solved with respect to the current constraints stored in the working set. The cutting plane algorithm for SSVM is summarized as follows.

1: Initialize a working set W i for each data point; ξ = 0. 2: for all t = 1, . . . , T do 3:

for all i = 1, . . . , n do 4: ŷi = arg max z (y i , z) + θ(x i , z) . 5: if (y i , ŷi ) + θ(x i , ŷi ) -θ(x i , y i ) > ξ i + then 6: W i = W i ∪ {ŷ i }. 7:
Update θ, ξ by solving (2.69) with constraints only in W i for all i. end for 10: end for If the loss-augmented MAP inference can be solved efficiently, the outer loop is guaranteed to converge quickly. The following theorem shows that only a polynomial number of constraints needed for any desired precision .

Theorem 2.4.1 (Tsochantaridis et al. 2005, Theorem 18).

Assuming that θ(x, y) = θ, φ(x, y) . Let R := max i max y∈Y φ(x i , y i ) -φ(x i , y) , ¯ := max i max y∈Y (y i , y),
and for any > 0, the cutting plane algorithm for (2.69) terminates after adding at most max 2n ¯ , 8 ¯ R2 λ 2 constraints to the working sets.

A variant called one-slack SSVM is also considered by [START_REF] Joachims | Cutting-plane training of structural svms[END_REF], which involves solving n loss augmented inference per iteration, but in total it requires fewer iterations.

Stochastic subgradient descent

Note that the cutting plane algorithm still keeps a double-loop structure in the algorithm. The key computational gain in the cutting plane algorithm is the working set, which enables each inner loop to operate a subset of data points in contrast with a standard subgradient descent who needs to accumulate subgradients from all data points. To go beyond the idea of working set, one can consider a stochastic subgradient descent (SSD) algorithm. In fact, [START_REF] Shalev | Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization[END_REF] sucessfully applied SSD to SVM showing a sublinear convergence speed. SSD has also been applied to SSVM as a baseline in Lacoste-Julien et al. (2013b).

The algorithm is actually much simpler than the cutting plane algorithm. It also works pretty well in practice. Consider a linear parameterization θ(x, y) = θ, φ(x, y) , where θ ∈ R d . By Danskin's theorem for minimax problems, the subgradient of (2.73) is

1 n n i=1 φ(x i , ŷi ) -φ(x i , y i ), (2.75)
where ŷi is the result of loss augmented inference on data point i. Then, the SSD for SSVM takes the form of

θ (t+1) = θ (t) - η t n φ(x i , ŷi ) -φ(x i , y i ) (2.76)
with the step size η t at iteration t satisfying

∞ t=1 η t = ∞, ∞ t=1 η 2 t < ∞. (2.77)
This is called Robbins-Monro conditions for convergence guarantee.

Block-coordinate Frank-Wolfe algorithm

It is a common practice in the SVM literature to optimize the dual problem rather than the primal one, since the dual variable at optimum is much sparser than the primal variable (thanks to the hinge loss). By using this property, one can make the classifier much more memory efficient, especially when the number of data points is significantly less than the number of parameters. In case θ(x, y) is convex in θ, the primal objective is also convex in θ, then maximizing the dual problem is equivalent to minimizing the primal problem.

To simplify the discussion, I assume that θ(x, y) = θ, φ(x, y) . In other words, θ(x, y) is linear in θ. With this assumption, the dual problem is derived in the following proposition.

Proposition 2.4.2. Denote by α ∈ R n|Y| := (α i (y)) i∈ [n],y∈Y the vector of dual variables, the dual problem of (2.73) is given by

max α : ∀i, α i ∈∆ D(α) := - λ 2 Aα 2 + b, α (2.

78)

where the matrix

A ∈ R d×n|Y| := 1 λn φ(x i , y i ) -φ(x i , y) i∈[n],y∈Y and the vector b ∈ R n|Y| := 1 n (y i , y) i∈[n],y∈Y
.

Proof. The convex optimization of (2.73) can be rewritten as

min θ max (z i ) i∈[n]∈Y n i∈[n] -λA i θ + b i , e z i + λ 2 θ 2 , (2.79) where (A i ) i∈[n] ≡ A and (b i ) i∈[n] ≡ b.
Since the strong duality holds, switching the order of the min and the max yields an equivalent problem. Note that minimizing over θ is now an unconstrained convex problem, which has a closed form solution:

θ * (z) = i∈[n] A i e z i .
Substituting this back to the objective yields the dual problem in eq.( 2.78).

The Frank-Wolfe algorithm was originally designed by [START_REF] Frank | An algorithm for quadratic programming[END_REF] for convex optimization problems of the form min x∈C f (x) over a convex set C, where the convex function f is assumed to be continuously differentiable. At each iteration, let x (k) be the current solution, the Frank-Wolfe algorithm updates along the descent direction x (k) -s * , where s * is obtained by minimizing the linearization of f at x (k) over C, namley, s * = arg min x∈C s, ∇f (x (k) .

The following proposition shows that the Frank-Wolfe algorithm is a good choice for solving (2.78).

Proposition 2.4.3. Let s * = arg max s : ∀i, s i ∈∆ s, ∇D(α) . Then, s * ≡ (z * i ) i∈[n] , where z * i = arg max y∈Y (y i , y) + Aα, φ(x i , y) .
Proof. First, we identify that arg max (2.78), which consists of the following steps:

y∈Y (y i , y) + Aα, φ(x i , y) = arg max y∈Y -λA i Aα + b i ,
1: Initialize α (0) i ∈ ∆ for all i.
2: for all k = 1, . . . , K do

3:

Pick i at random in {1, . . . , n}.

4:

Let z * i = arg max y∈Y (y i , y) + Aα (k) , φ(x i , y) .

5:

Let γ = 2n k+2n .

6:

Update α

(k+1) i = (1 -γ)α (k) i + γe z * i . 7: end for
The above algorithm can be made more practical in several aspects. By using the representer theorem, one can compute a candidate value for the primal variable:

θ = Aα = n i=1 A i α i = 1 λn n i=1 y∈Y α i (y) φ(x i , y i ) -φ(x i , y) , (2.81)
which is important since maintaining α may not always be possible even it is a sparse vector. Instead, we can keep n local copies of θ:

w i = A i α i for all i ∈ [n], such that w (0) i = 0 and θ (t) = n i=1 w (t)
i . line 6 is then equivalent to the primal update

w (t+1) i = (1 -γ)w (t) i + γ λn φ(x i , y i ) -φ(x i , y) . (2.82)
Besides, since D(α) is quadratic, the value of γ can be chosen by line search in a closed form. See Lacoste-Julien et al. (2013b, Algorithm 4) for the version of BCFW with line search.

Dual decomposition for SSVM

Previous algorithms introduced in this section are designed for general structured output learning. In the case of PGMs, they did not attempt to leverage the particular structure encoded in the model. Instead, they have to invoke a subroutine for lossed augmented MAP inference at each iteration, which has shown to be NP-hard in Section 2.3. The question of whether approximate inference can be used during learning has been studied by [START_REF] Kulesza | Structured learning with approximate inference[END_REF], [START_REF] Finley | Training structural SVMs when exact inference is intractable[END_REF]. A counterexample given by [START_REF] Kulesza | Structured learning with approximate inference[END_REF] shows that not all approximations yield convergent SSVM learning. However, the approximate inference based on linear programming relaxation (LPR) has empirically and theoretically demonstrated good performance [START_REF] Finley | Training structural SVMs when exact inference is intractable[END_REF] as long as the same inference scheme is used in training and testing. Further theoretical analysis for LPR based loss augmented inference have been conducted by [START_REF] Kulesza | Structured learning with approximate inference[END_REF], [START_REF] Meshi | Train and test tightness of lp relaxations in structured prediction[END_REF]. In particular, [START_REF] Kulesza | Structured learning with approximate inference[END_REF] gave a PAC-Bayes bound and [START_REF] Meshi | Train and test tightness of lp relaxations in structured prediction[END_REF] provided a theoretical explanation to the observation that linear programming relaxations are often tight in practice.

Recall that LPR can be efficiently solved by dual decomposition based algorithms [START_REF] Komodakis | MRF optimization via dual decomposition: Message-passing revisited[END_REF][START_REF] Werner | A linear programming approach to max-sum problem: A review[END_REF], Globerson and Jaakkola 2007a[START_REF] Sontag | Tree block coordinate descent for map in graphical models[END_REF]. It turns out that the same technique can be used to blend the inference and learning in the sense that the inference part and the learning part are updated in an alternating fashion. This basically summarized the idea behind the work of [START_REF] Meshi | Learning efficiently with approximate inference via dual losses[END_REF], [START_REF] Hazan | A primal-dual message-passing algorithm for approximated large scale structured prediction[END_REF], Komodakis (2011b). Specifically, for the PGM = p θ (y|x), G = (V, E, A) , the SSVM learning objective in eq.( 2.73) is upper bounded by

1 n n i=1 max µ∈L a∈A -λA ia θ a + b ia , µ a + λ 2 θ 2 2 , (2.83)
where A ia and b ia are defined similarly as A and b introduced in Proposition 2.4.2:

A ia ∈ R da×|Ya| := 1 λn φ a (x i , y ia ) -φ a (x i , y a ) ya∈Ya , b ia ∈ R |Ya| := 1 n a (y ia , y a ) ya∈Ya .
It should be noticed that the task-specific loss (y i , y) is assumed to be decomposable according to G. This allow us to take advantage of the underlying structure of the problem.

To derive the dual problem, I associate each equality constraint in L a Lagrangian multiplier δ (i) as , and let

δ (i) := (δ (i) as ) a∈A,s∈a , η (i) (θ) := (-λA ia θ a + b ia ) a∈A .
Note that a superscript is added to distinguish different data points. The dual objective of (2.83) reads as

1 n n i=1 min δ (i) a∈A max ya∈Ya η (i) (θ), e ya + c a δ ca , e ya - s∈a δ as , e ys + λ 2 θ 2 2 , (2.84)
which is naturally an upper bound of (2.83), thus it is also an upper bound of the original SSVM objective.

The above objective function only involves δ (i) and θ. We can compute an approximate stochastic gradient with respect to θ by taking a few block-coordinate descent steps on δ (i) . Note that the coordinate descent updates on δ (i) can be derived in closed form, but the detailed form depends on the block sampling strategy. A comparison on different strategies can be found in [START_REF] Sontag | Introduction to dual composition for inference[END_REF].

To sum up, the dual decomposition learning algorithm takes the following steps:

1: Initialize δ = 0, θ = 0. 2: for all t = 1, . . . , T do 3:
Pick i at random in {1, . . . , n}.

4:

Perform a few (usually ≤ 10) block-coordinate updates on δ (i) .

5:

Perform a stochastic subgradient descent on θ, where the stochastic gradient is estimated based on the updated δ (i) . 6: end for A convergence proof for the above algorithm can be found in [START_REF] Meshi | Learning efficiently with approximate inference via dual losses[END_REF].

Conclusion

In this chapter, I covered the basic ingredients on the recent development of approximate inference and learning for probabilistic graphical models. The main difficulty comes from the computation over the exponentially large output space. In inference and learning, this difficulty is reflected in computing the value or the gradient of the log-partition function. More efficient algorithms will allow the current formulations to scale up. Besides, most of existing methods focused on convex PGMs. There are a few exceptions, such as the structured prediction energy networks [START_REF] Belanger | Structured prediction energy networks[END_REF]. It remains an open question whether the current developed techniques are applicable to non-convex PGMs.

Introduction

Learning in graphical models has historically relied on the computation of the (sub)gradient of the log-likelihood w.r.t. to the canonical parameters, which requires to solve a MAP or probabilistic inference problem at each iteration. This approach is slow given that the inference problem is itself computationally expensive. The difficulty of inference and learning in graphical models is related to the fact that the log-partition function is in general intractable.

Recent progress on the optimization problems whose objective is a large finite sum of convex terms has shown that they could be optimized very efficiently by stochastic algorithms that sample one term at a time [START_REF] Nicolas L Roux | A stochastic gradient method with an exponential convergence rate for finite training sets[END_REF][START_REF] Shalev | Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization[END_REF], Defazio et al. 2014b). It turns out that the dual objective of the maximum likelihood estimation of CRF (a.k.a. the maximum entropy principle) decomposes additively over all cliques if a decomposable entropy surrogate is used. Even though this dual formulation has a potential to take advantage of stochastic algorithms, and can be optimized without resorting to solve a global inference on the entire graph per iteration, all dual parameters (i.e. mean parameters) are coupled by the marginal polytope constraints, which are in general intractable. Even its most commonly used relaxation, namely the local consistency polytope, is itself in practice difficult to optimize over. Recently, Meshi et al. (2015ba) proposed to replace the marginalization constraints, which are part of the local consistency polytope, by quadratic penalty terms. The relaxed problem has then only separable constraints over the cliques that makes it possible to use efficient block coordinate optimization schemes.

Following these ideas, we consider a dual formulation for CRF learning in which the marginalization constraints are replaced by an augmented Lagrangian term, and the intractable Shannon entropy is replaced by a quadratic surrogate so that stochastic dual coordinate ascent (SDCA) can be used to optimize over the mean parameters, with similar guarantees as in [START_REF] Shalev | Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization[END_REF]. We finally show that by periodically updating the Lagrangian multipliers as we are optimizing the relaxed dual, we can gradually enforce the marginalization constraints, while retaining global linear convergence. In terms of the primal problem associated with the Lagrange multipliers, our algorithm is an inexact gradient descent algorithm using stochastic approximation of the multiplier gradients.

Our paper is organized as follows. We review CRF learning in Section 3.3. A dual augmented Lagrangian formulation is presented in Section 3.4. The proposed algorithm is presented in Section 3.5, followed by its convergence analysis in Section 3.6. Finally, we present experiments on three applications in Section 3.7 (Most notations used in the paper can be found in Appendix 3.F).

Related Work

Due to the independent interest of inference problem in discrete graphical models, in particular in computer vision, a significant amount of work has been devoted to

CRF Learning

51

develop efficient approximate inference algorithms [START_REF] Komodakis | MRF optimization via dual decomposition: Message-passing revisited[END_REF][START_REF] Sontag | Tightening LP relaxations for MAP using message passing[END_REF][START_REF] Savchynskyy | A study of Nesterov's scheme for Lagrangian decomposition and MAP labeling[END_REF][START_REF] André | AD3: Alternating directions dual decomposition for MAP inference in graphical models[END_REF]. However, the learning problem is not necessarily easier (can even fail to converge) with an approximate inference approach as the subroutine [START_REF] Kulesza | Structured learning with approximate inference[END_REF].

There is a large body of research on efficient algorithms for structured learning. For the max-margin formulation, the fastest algorithms to date rely on block coordinate Frank-Wolfe updates (Lacoste-Julien et al. 2013a, Meshi et al. 2015b, Tang et al. 2016). Using dual decomposition in the inner inference problem, [START_REF] Meshi | Learning efficiently with approximate inference via dual losses[END_REF], [START_REF] Hazan | A primal-dual message-passing algorithm for approximated large scale structured prediction[END_REF], Komodakis (2011a) proposed to solve the classical saddle-point formulation for structured learning problem with algorithms that alternate between message passing and model parameter updates. Going further Meshi et al. (2015b), Yen et al. (2016) work on a purely dual formulation to enable clique-wise updates. For maximum likelihood learning, exponentiated gradient and its block variants can be applied [START_REF] Collins | Exponentiated gradient algorithms for conditional random fields and max-margin Markov networks[END_REF]. Other recent work have relied on incremental algorithms [START_REF] Schmidt | Non-uniform stochastic average gradient method for training conditional random fields[END_REF] and the fact that the Gauss-Southwell rule can be applied efficiently for coordinate descent in some forms of graphical models [START_REF] Nutini | Coordinate descent converges faster with the Gauss-Southwell rule than random selection[END_REF]. Another interesting work by Le Priol et al. ( 2018) runs SDCA for CRF learning (but still uses an expensive marginalization oracle).

The BCMM algorithm of [START_REF] Hong | A block successive upper bound minimization method of multipliers for linearly constrained convex optimization[END_REF] which uses stochastic block coordinate updates inside ADMM inspired our approach. But our algorithm performs multiple passes over all blocks before updating the multiplier; and we prove stronger convergence rates.

We list related structured learning methods with their main characteristics in Table 1 in Appendix 3.B. 5. Yen et al. (2016) is the most similar work to ours: the proposed algorithm constructs greedily an (initially sparse) working set of cliques, which is incremented at each epoch, while we perform stochastic updates on all cliques and possibly several passes over the data between each update of all Lagrange multipliers. Also, our work is leveraging the connection with SDCA, and we prove both linear convergence in the primal and the dual whereas Yen et al. ( 2016) prove only linear convergence in the dual.

It is noteworthy that the work of Gidel et al. ( 2018), which was published at the same time as this work at AISTATS 2018, studies a related algorithmic framework based on the Frank-Wolfe algorithm for the same type of convex optimization problems. The main difference is that they take Frank-Wolfe updates for the inner loop with a specific analysis to Frank-Wolfe. Besides, they consider applications beyond the scope of probabilistic graphical models.

CRF Learning

A discrete conditional random field (CRF) is a family of conditional distributions over a vector of discrete random variables Y := (Y 1 , . . . , Y m ) given the observation X. The form of the CRF is assumed to be a product of local functions (a.k.a. factors or clique functions) that each depends on only a small number of random variables (i.e. a clique). If there exists multiple cliques that share the same local function, then we group cliques by clique types. Specifically, let w τ ∈ R dτ be the parameter vector associated with the clique type τ ∈ T , where T is the set of clique types. Let C denote the set of all cliques, and C τ the set of cliques of type τ ∈ T . Note that each clique c has a unique clique type, which we denote by τ c . With these notations the density function of the CRF can be written as

p(y|x; w) := 1 Z(x, w) τ ∈T c∈Cτ exp w τ , φ c (x, y c ) ,
where w = (w τ ) τ ∈T ; we denoted Z(x, w) the partition function and φ c (x, y c ) ∈ R dτ c the feature map for clique c. Since all random variables are discrete, we use a one-hot vector

y i ∈ Y i := {u ∈ {0, 1} k i : u 1 = 1} to represent the value of Y s .
Here k i is the cardinality of Y i . For a clique c, the value for the corresponding random variables is

y c = ⊗ i∈c y i ∈ Y c := i∈c Y i ,
where ⊗ (resp. ) denotes the tensor product of vectors (resp. of spaces). Similarly, y ∈ Y is of the form y = ⊗ i∈V y i . W.l.o.g., we consider in the paper only cliques of size at most 2, that is C = V ∪ E, with V and E respectively the set of nodes and of edges of the graph; the framework generalizes easily to higher-order cliques. Notations used in the paper are listed in Appendix 3.F.

CRF as exponential family

Given a sample (x (n) , y (n) ), for each clique c, let

η (n) c (w) := [ w τc , φ c (x (n) , y c ) : y c ∈ Y c ]; then a natural parameter for the exponential family form of the conditional distribution p(y | x (n) ) is η (n) (w) := [η (n) c (w) : c ∈ C].
The associated sufficient statistics is T (y) := [y c : c ∈ C], and η (n) (w), T (y) = c η (n) c (w), y c . With these notations, p(y | x (n) ) has the exponential family form:

p y | η (n) (w) = exp η (n) (w), T (y) -F η (n) (w) ,
where F (η) := log y exp η, T (y) = log Z(x (n) , w) is the log-partition function.

Given i.i.d. samples {(x (n) , y (n) )} 1≤n≤N , the maximum likelihood estimator for w is computed by the maximizing n log p(y (n) | x (n) ; w). Using the exponential family representation, we can rewrite this problem in two equivalent forms:

max w N n=1 η (n) (w), T (y (n) ) -F η (n) (w) , and min w N n=1 F θ (n) (w) with θ (n) (w) another natural parameter obtained via the affine transformation θ (n) (w) = η (n) (w) -η (n) (w), T (y (n) ) 1. Alternatively, by defining Ψ (n) as a sparse block matrix with |T | × |C| blocks, whose (τ c , c)-th block is the matrix Ψ (n) c ∈ R dτ c ×kc with Ψ (n) c = [φ c (x (n) , y c ) -φ c (x (n) , y (n) c ) : y c ∈ Y c ],
3.4 Relaxed Formulations

53

we have θ

(n) c (w) = Ψ (n) c
w τc and θ (n) (w) = Ψ (n) w. W.l.o.g., we assume N = 1 and drop the superscript (n) from now on, since one may view N graphs as a single large graph with several connected components.

Regularized maximum likelihood estimation with a regularization constant λ > 0 is thus formulated as

min w F θ(w) + λ 2 w 2 2 . (3.1)
In order to extend this formulation to cover as well max-margin learning (i.e., structured SVMs), we consider the loss-augmented CRF learning introduced by Pletscher et al. ( 2010) and [START_REF] Hazan | A primal-dual message-passing algorithm for approximated large scale structured prediction[END_REF], which leads to a slightly generalized formulation:

min w γF 1 γ θ (w) + λ 2 w 2 2 , (3.2)
where θ (w) := θ(w) + is then the natural parameter, with = [ c (y c , y c ) : y c ∈ Y c ] : c ∈ C the user-defined loss and γ ∈ (0, +∞) the temperature hyperparameter. For a derivation for the loss-augmented CRF see Appendix 3.A.

It is well known that the cost of gradient descent to optimize either eq.( 3.1) or eq.( 3

.2) (for γ > 0) is prohibitive since ∇ wτ F (θ(w)) = c∈Cτ Ψ c E θ [Y c
] involves an expectation over the exponentially large space Y. To exploit the underlying structure of the function F it is useful to work on the dual problem. Indeed, since F is convex, it has a variational representation based on conjugate duality:

F (θ) = max µ µ, θ -F * (µ),
where F * is the Fenchel conjugate of F , and the dual variable µ called the mean parameter is defined by µ = (µ c ) c∈C with µ c = E θ [Y c ]. The set of valid mean parameters form the so called marginal polytope M, which is defined as the convex hull of {T (y) : y ∈ Y}. Moreover, if let H Shannon (µ) denote the Shannon entropy of a CRF with mean parameter µ, it is a classical result (Wainwright 2008, Thm 3.4) that

F * (µ) = -H Shannon (µ) + ι M (µ),
where ι M (µ) equal to 0 if µ ∈ M and +∞ otherwise.

Relaxed Formulations

In this section, we derive general relaxed dual, primal and corresponding saddlepoint formulations for the CRF learning problem: first, we use the classical local polytope relaxation (Sec. 3.4.1). Second, we further relax the marginalization constraints via an augmented Lagrangian (Sec. 3.4.2). Third, we propose a surrogate for the entropy, which is decomposable, and retains good properties even when the aforementioned constraints are relaxed (Sec. 3.4.3). The resulting formulation is convex and is amenable to fast optimization algorithm that are presented in Section 3.5.

Classical local polytope relaxation

Both M and H Shannon (µ) are in general intractable due to the exponentially large structured-output space Y and they are typically replaced by decomposable surrogates.

It is common to relax M to the local consistency polytope [START_REF] Wainwright | Graphical models, exponential families, and variational inference[END_REF] L := µ ∈ I :

y j ∈Y j µ ij (y i , y j ) = µ i (y i ), ∀{i, j} ∈ E, ∀y i ,
where I denotes the Cartesian product of simplex constraints on each clique. Note that L ⊇ M, since any set of true marginals must satisfy the simplex constraints and the marginalization constraints, but not vice versa. Equivalently, if we define A i = I k i ⊗ 1 k i , the equality constraints can be written in a matrix form as µ i -A i µ ij = 0 for all {i, j} ∈ E. Combining all equations, we have Aµ = 0, where A is a |E| × |C| block matrix (see Appendix 3.F). So, we have equivalently

L = I ∩ {µ : Aµ = 0}.
Since H Shannon is also intractable for graphs with large tree-width, we will use an approximation H Approx which will be constructed so as to be defined and concave on the whole set I. We propose several entropy approximations suited to our needs in Section 3.

F I (θ ) := max µ µ, θ -F * I (µ), F L (θ ) := max µ µ, θ -F * L (µ),
where

F * I (µ) := -H Approx (µ) + ι I (µ) and F * L (µ) := F * I (µ) + ι {Aµ=0} . Replacing F with F L in eq.(3.
2) yields the relaxed primal

P (w) := γF L 1 γ θ (w) + λ 2 w 2 2 . (3.3)
The corresponding dual objective function is given by

D(µ) := µ, -γF * L (µ) - 1 2λ Ψµ 2 2 . (3.4)
See Appendix 3.B.1 for a derivation.

A dual augmented Lagrangian

It is difficult to optimize D(µ), since the optimization requires some form of projection onto L, which can be shown to be equivalent to perform graph-wise marginal inference [START_REF] Collins | Exponentiated gradient algorithms for conditional random fields and max-margin Markov networks[END_REF]). The difficulty is due to the coupling equality constraint Aµ = 0. Meshi et al. (2015b) proposed to relax ι {Aµ=0} by a quadratic term 1 2ρ Aµ 2 2 , which corresponds to employ the penalty method [START_REF] Bertsekas | The method of multipliers for equality constraints[END_REF]. They argue that it is not crucial to enforce exact Aµ = 0 in learning, since the relaxed problem works well in practice and enables an efficient optimization with only clique-wise updates. However, the penalty method is known to have issues associated with the choice of ρ: unless we use a carefully designed scheduling to update ρ, for a reasonably small ρ, the algorithm will be slow; on the other hand, using a large fixed value of ρ degrades the problem to independent logistic regression problems, and, thereby, leads to suboptimal solutions.

Instead, we propose to solve problem eq.( 3.4) as a saddle problem of the form max µ min ξ D ρ (µ, ξ) where D ρ is the augmented Lagrangian

D ρ (µ, ξ) := , µ -γF * I (µ) + ξ, Aµ - 1 2ρ Aµ 2 2 + 1 2λ Ψµ 2 2 , (3.5)
with ξ is the Lagrangian multiplier and ρ > 0.

Using duality again, we can derive an associated relaxed primal objective

Pρ (w, δ, ξ) := γF I θ (w) + A δ γ + λ 2 w 2 2 + ρ 2 δ -ξ 2 2 ,
so that min (w,δ) Pρ (w, δ, ξ) is a primal problem associated with the dual problem max µ D ρ (µ, ξ).

Strong duality between these two problems yields a representer theorem

w = - 1 λ Ψµ , δ = ξ - 1 ρ Aµ (3.6)
which provides a duality gap gap(w, δ, µ, ξ)

:= Pρ (w, δ, ξ) -D ρ (µ, ξ)
for the convergence of the maximization of D ρ (µ, ξ) with respect to µ. Moreover, it is easy to check that min ξ,δ Pρ (w, δ, ξ) = P (w) because min δ F I (θ(w) + A δ) = F L (θ(w)) for any w (see Appendix 3.B.2). This shows that w defined in eq. (3.6) is also an optimum of the original primal problem min w P (w). As a consequence, if a sequence µ t converges to µ then the corresponding w t = -1 λ Ψµ t converges to a solution of eq.(3.2). For more details, see Appendix 3.B.

Gini entropy surrogate

We seek a concave entropy surrogate H Approx that decomposes additively on the cliques. Since the constraint Aµ = 0 is relaxed, we need a surrogate well defined on the whole set I. The Bethe entropy [START_REF] Yedidia | Dual decomposed learning with factorwise oracle for structural SVM of large output domain[END_REF]) is generally non-concave. Its concave counterparts, such as the tree-reweighted entropy (Wainwright et al. 2005b) or the region-based entropy [START_REF] Yedidia | Dual decomposed learning with factorwise oracle for structural SVM of large output domain[END_REF][START_REF] London | The benefits of learning with strongly convex approximate inference[END_REF], are only concave on the local consistency polytope, but non-concave on I.

Moreover, a generic difficulty with these entropies is that they do not have Lipschitz gradients, which prevents the direct application of proximal methods with usual quadratic proximity terms. We thus propose a coarse but convenient entropy surrogate of the form:

H Approx (µ) = c∈C h c (µ c ) with h c (µ c ) := (1 -µ c 2 2 ).
Another surrogate with the same separable form is the second-order Taylor expansion of the oriented tree-reweighted entropy (OTRW, Globerson and Jaakkola 2007b) around the uniform distribution. This surrogate is also concave on I (although not strongly concave) and smooth. Preliminary experiments however did not show that using this more sophisticated entropy improved the results. See Appendix 3.C for more details.

Algorithm

Given the form of the entropy surrogate proposed, D ρ decomposes as a sum of convex separable terms over the block associated to cliques plus a smooth term:

D ρ (µ, ξ) = - c∈C f * c (µ c ) -r(µ) with (3.7) f * c (µ c ) := -γh c (µ c ) + ι c (µ c ) r(µ) := -A ξ + , µ + 1 2λ Ψµ 2 + 1 2ρ Aµ 2 ,
where c := {µ c ∈ R dc + | µ c 1 = 1} is the canonical simplex. It can thus be maximized efficiently by a block-coordinate proximal scheme, such as the proximal stochastic dual coordinate descent (SDCA, Shalev-Shwartz and Zhang 2016), which has linear convergence guarantees both in the primal and the dual.

To solve min ξ max µ D ρ (µ, ξ) we thus propose an algorithm similar to the block coordinate method of multipliers (BCMM) of [START_REF] Hong | A block successive upper bound minimization method of multipliers for linearly constrained convex optimization[END_REF]: perform dual stochastic block coordinate ascent (SDCA) on the variables µ c to partially maximize D ρ (µ, ξ) in µ and regularly take a gradient descent step in ξ. Our algorithm, is an inexact dual augmented Lagrangian (IDAL) method, in the sense that it is an inexact gradient descent algorithm on the function ξ → d(ξ) := max µ D ρ (µ, ξ). To be precise, if at epoch t, ξ takes the value ξ t and μt-1 is the value of µ from the previous epoch, Algorithm 2 takes T in stochastic block-coordinate proximal gradient steps on µ to obtain μt . Denoting L c the Lispchitz constant of r w.r.t. µ c , µ c is then updated by a partial gradient step, and an application of the proximal operator of

μt = A(μ t-1 , T in , t) 5:
Stop if G t ≤ and Aμ t 2 ≤ 6:

ξ t+1 = ξ t -1 L d Aμ t 7: end for 8: Output: μTex , ξ Tex Algorithm 2 SDCA version of A(µ, T in , t) 1: µ t,0 = µ 2: for s = 1, . . . , T in do 3:
Draw a clique c uniformly at random 4:

µ t,s c = Prox 1 Lc f * c µ t,s-1 c -1 Lc ∇ µc r(µ t,s-1 ) 5: µ t,s -c = µ t,

Convergence Analysis

In this section, we study the convergence rate of our algorithm. First, we show that if we use an iterative and linearly convergent algorithm A to approximately solve min µ D ρ (µ, ξ), and if we use warm starts, that is, following the notations of the previous section, we use μt-1 as the initial value to solve min µ D ρ (µ, ξ t ), then running A for a fixed number of iterations is sufficient to guarantee global linear convergence in the primal and in the dual. We show that SDCA or simple block-coordinate proximal gradient descent are applicable as the algorithm A.

Conditions for global linear convergence

To study the convergence, we consider:

• μt := µ (ξ t ) = argmax µ D ρ (µ, ξ t ).
• µ t,s , the value of µ after s inner steps at epoch t.

• μt := µ t,T in the value of µ at the end of epoch t.

• D ρ -suboptimality: ∆ s t := D ρ (μ t , ξ t ) -D ρ (µ t,s , ξ t ), with at the end of each epoch ∆t := ∆ T in t = ∆ 0 t+1 . • d-suboptimality: Γ t := d(ξ t ) -d(ξ ).
Theorem 3.6.1 (Linear convergence of the outer iteration). Let A be an algorithm that approximately solves max µ D ρ (µ, ξ t ) in the sense that

∃β ∈ (0, 1), E[ ∆t ] ≤ β E[∆ 0 t ].
Then, ∃κ ∈ (0, 1) characterizing d(ξ) and C > 0, such that, if λ max (β) is the largest eigenvalue of the matrix

M (β) = 6β 3β 1 1 -κ ,
then after T ex iterations of Algorithm 1 we have

E[ ∆Tex ] E[Γ Tex ] ≤ C λ max (β) Tex E[ ∆0 ] E[Γ 0 ] .
The proof is deferred to Appendix 3.D as well as the proofs of the following corollaries.

The constant κ in the theorem is of the form κ = τ L d with L d the Lipschitz constant of d(ξ) and τ a restricted strong convexity constant for d(ξ) obtained by [START_REF] Hong | On the linear convergence of the alternating direction method of multipliers[END_REF]

(β) < 1, we have (1 -π) T in ≤ β, then E[ ∆t ] and E[Γ t ] converge linearly to 0.
Note that linear convergence of the expectations implies that ∆ t and Γ t converge linearly to 0 almost surely, as a classical consequence of Markov's inequality and the Borel-Cantelli lemma. We will show in the next section that when A is SDCA it is linearly convergent.

Note that the convergence of the gaps ∆ t and Γ t imply the linear convergence for the augmented Lagrangian formulation, in the following sense:

Corollary 3.6.3.Let D ∞ (µ) := , µ -γF * I (µ)-1 2λ Ψµ 2 2 , so that we have D(µ) = D ∞ (µ) -ι {Aµ=0} . If ∆ t and Γ t converge linearly to 0, then |D ∞ (μ t ) -D ∞ (µ )| and Aμ t 2
2 both converge to 0 linearly.

Furthermore, if A is linearly convergent as in Corollary 3.6.2, the algorithm is linearly convergent in terms of the total number of inner steps (for SDCA this is the total number of clique updates) performed by algorithms A throughout: Corollary 3.6.4 (Total number of inner updates). With the notations of the previous corollary, for any β ∈ (0, 1) such that λ max (β) < 1, it is possible to obtain

E[ ∆t ] ≤ and E[Γ t ] ≤ with a total number of inner iterations T tot := T in T ex such that T tot ≥ log(β) log λ max (β) log(1 -π) log( ).

Convergence Analysis 59

We show in Appendix 3.D.5 that to have λ max (β) < 1 we should have β = ακ with α < 1 3(1+2κ) . To reason in terms of rate, if the rate of convergence is r then we should have T tot ≥ log ( ) log (1-r) . So identifying the rate of convergence of the algorithm yields r = 1-exp log(1-π) log(λmax(β)) log (β) . If α and κ are not too large, we can get a simplified expression for the rate, characterized as follows.

Corollary 3.6.5 (Convergence rate). If κ < 1 2 and α = 1 12 , if T in ≥ log(ακ) log(1-π) , then, there exist a constant C > 0 such that after a total of t T in + s inner updates, we have

E[∆ s t + Γ t ] ≤ C 1 - κπ 2 log(12/κ) t T in +s .

Convergence results with SDCA

Given the structure of D ρ , if the functions f * c in eq.( 6.11) are strongly convex, a good candidate for A is stochastic dual coordinate ascent (SDCA). Indeed, the results of [START_REF] Shalev | Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization[END_REF] show that Moreover SDCA allows us to bound the duality gap by the increase of D ρ , which yields linear convergence in the primal.

Proposition 3.6.7. Let ŵt = w(μ t ). If A is SDCA, then E[P ( ŵt ) -P (w )] ≤ 1 π E[ ∆t ] + E[Γ t ].
For the sake of the natural surrogates for the entropy (like the Gini-OTRW entropy proposed in Appendix 3.C), individual functions f * c are not strongly convex, although -D ρ is strongly convex, because the entropy surrogate is strongly concave on L and the term Aµ 2 is strongly convex on Ker(A) ⊥ . In that case another decomposition is relevant

: if σ is the strong convexity constant of -D ρ , then let f * c (µ c ) = ι c (µ c ) + σ µ c 2 2 and r(µ) = -H Approx (µ) + r(µ) -σ µ c 2 2 . We again have D ρ (µ) = -c∈C f * c (µ c ) -r(µ)
, with f * c strongly convex and r convex and smooth. SDCA and its theory are here applicable again and guarantees that Proposition 3.6.6 and following hold. However, for the convergence in the primal a slightly different argument is needed. Proposition 3.6.8 (Linear convergence in the primal). Let w t,s = w(µ t,s ). If A is a linearly convergent algorithm and the function µ → -H approx + 1 2ρ Aµ 2 2 is strongly convex, then P (w t,s ) -P (w ) converges to 0 linearly.

Discussion

Optimization with inexact gradients [START_REF] Devolder | First-order methods of smooth convex optimization with inexact oracle[END_REF]) and inexact proximal operators [START_REF] Schmidt | Convergence rates of inexact proximal-gradient methods for convex optimization[END_REF]) have been shown to yield the same convergence rate as their exact counterparts, provided that errors decrease at a certain rate. Linear convergence of an inexact augmented Lagrangian method in which both inner and outer optimizations use Nesterov's accelerated gradient descent is shown in [START_REF] Lan | Iteration-complexity of first-order augmented Lagrangian methods for convex programming[END_REF]. We use the same ideas, except that we leverage the large finite sum structure of the dual problem to use randomized algorithms. The use of warm-start is also similar to its use in the meta-algorithm proposed by [START_REF] Sonnenburg | shogun-toolbox/shogun: Shogun 6.1[END_REF], who use inexact gradient descent on the Moreau-Yosida regularization of a non-smooth objective. In our context, this approach would actually be applicable by working on P ρ (w, ξ) instead of working in the dual. An investigation in this direction is of interest but beyond the scope of this paper.

Experiments

We evaluate our algorithm IDAL on three different CRF models including 1) a simulated Gaussian mixture Potts model with grid graph and two clique types (nodes and edges); 2) a semantic segmentation model with planar graph and two clique types (nodes and edges); 3) a multi-label classification model with fully-connected graph and unique clique type for all cliques.

We compare with algorithms using only clique-wise oracles for solving min ξ max µ D ρ (µ, ξ), namely, the soft-constrained block-coordinate Frank-Wolfe algorithm (SoftBCFW) by Meshi et al. (2015b) and the greedy direction method of multipliers (GDMM) algorithm by Yen et al. (2016). Note that SoftBCFW in fact solves only the special case max µ D ρ (µ, ξ ≡ 0), thus it will converge to a different point than IDAL. In addition, we include a third baseline for the special case using SDCA (referred as SoftSDCA). Since SoftBCFW and GDMM have been shown outperforming other baselines such as Lacoste-Julien et al. (2013a), [START_REF] Meshi | Learning efficiently with approximate inference via dual losses[END_REF] and Hazan and Urtasun (2010), we will not make an extensive comparison for all these algorithms.

Setup

Gaussian mixture Potts models

This is an extension of the Potts model given observations, whose conditional density function is defined via Bayes' rule p(y|x) ∝ p(x|y)p(y), with p(y) a Potts distribution associated with a grid graph and parameterized by w binary ∈ R k 2 , and with p(x|y) = i p(x i |y i ) assumed to factorize into independent conditional Gaussian distributions with canonical parameters w unary ∈ R 2k , i.e., p(x i |y i ) ∝ exp( w unary (y i ), [x i , x 2 i ] ). We consider a 10 × 10 grid graph with node cardinality k = 5. To generate the data, we first draw the label y from p(y), and then the observation x i is generated from the conditional Gaussian p(x i |y i ) for each node. The simulated dataset contains 100 samples and is equally divided for training and testing.

Semantic image segmentation

We consider a typical CRF model used in computer vision for labeling image pixels with semantic classes. The graph is built upon clustering pixels into superpixels. Each superpixel defines a node. Two superpixels with a shared boundary define an edge. The CRF model takes the form p(y|x) ∝ exp i w unary ψ i (x, y i ) + i,j w binary ψ ij (x, y i , y j ) , where ψ i (x, y i ) measure the intra-cluster compatibility

62

Inexact Dual Augmented Lagrangian Method for CRF Learning within the superpixel i, and ψ ij (x, y i , y j ) measure the inter-cluster compatibility between superpixels i and j. We conduct the experiment on the MSRC-21 dataset introduced by [START_REF] Shotton | Textonboost: Joint appearance, shape and context modeling for multi-class object recognition and segmentation[END_REF], which has 21 classes, 335 training images and 256 testing images.

Multi-label classification

The task for this problem is assigning each input vector a set of binary target labels.

It is natural to model the inter-label dependencies by CRFs that treat each label as a node in a fully connected label graph. Following [START_REF] Finley | Training structural SVMs when exact inference is intractable[END_REF], we define the CRF density function as

p(y|x) ∝ exp( i w i φ i (x, y i )+ i,j w ij φ ij (y i , y j )),
where the feature maps are specified as φ i (x, y i ) = y i ⊗ x for each node and φ ij (y i , y j ) = y i ⊗ y j for each edge. We conduct the experiments on the Yeast dataset2 , which contains 1500 training samples and 917 testing samples. Each sample has 14 labels and 103 attributes.

Hyperparameters

In theory, T in could be very large depending on the choice of α and the condition number π. We find that in practice only a relatively small T in is needed. We empirically choose T in = 1 2 |C|. We set the number of outer iterations T ex = 3000 and the stopping threshold = 10 -3 . The ranges of λ is pre-defined as {10, 1.0, 0.01, 0.001} and the range of γ is {100.0, 10.0, 1.0, 0.001}. For each experiment, we choose the best λ and γ in terms of the validation accuracy and a reasonable running time (not all experiments finished in 3000 outer iterations). We set ρ = 1.0 or ρ = 0.1 as in Meshi et al. (2015b).

Results

To compare IDAL with GDMM, we use the criterion P ρ ( ŵt , δt , ξ t ) -D ρ (μ t , ξ t ) + P ρ ( ŵt , δt , ξ t ) -D ρ (μ Tex , ξ Tex ), which is an upper bound of the theoretical quantity ∆t + Γ t that we analyzed. To compare IDAL with SoftBCFW, since ξ = 0 for SoftBCFW, we use the criterion D ρ (μ t , ξ ), in which ξ is obtained from running IDAL to convergence. Besides, we also use the criteria Aμ t 2 (it measures the convergence of d(ξ), since ∇d(ξ t ) Aμ t ) and the testing accuracy, which are applicable for all three algorithms. The results are shown in Figure 3.1.

There are several interesting points that we can say based on the results: 1) by tightening the marginalization constraints Aµ = 0, it does help to gain a better testing accuracy (e.g., IDAL gains small improvements over SoftBCFW); 2) based on the curves of D ρ (µ, ξ ), we can see that it is key to approach µ by first obtaining ξ , which again shows the importance of enforcing exactness of the local consistency polytope; 3) IDAL is shown to be a faster algorithm than GDMM.

One possible reason is that GDMM is in fact an active-set algorithm, which means the number of updated cliques at very beginning is insufficient comparing to IDAL. Based on our analysis, we have shown that the quality of the approximate gradient Aμ t depends on T in . Therefore, it is very likely that GDMM suffers from a slow convergence because of the poor gradients.

Conclusion

We proposed a relaxed dual augmented Lagrangian formulation for CRF learning, in which, thanks to dual decomposition, SDCA can be used to partially optimize over mean parameters in order to yield a sufficiently good approximation of the multiplier gradient. Our theoretical analysis shows that if warm-starts are leveraged and multiplier gradients are approximated with a linearly convergent algorithm, global linear convergence can be obtained. If SDCA is used, linear convergence is obtained both in the primal and for the convergence of the dual Lagrangian method.

Comparing to other baselines such as GDMM and SoftBCFW, our algorithm is faster in terms of the distance to the optimal objective function value (i.e. ∆t + Γ t ) and the feasibility of the constraints Aµ 2 2 . It would be of interest to investigate the use of the same dual augmented Lagrangian formulation for both inference and learning, since according to [START_REF] Wainwright | Estimating the wrong graphical model: Benefits in the computationlimited setting[END_REF], this should improve the performance.

In future work, we intend to investigate applications to other problems in machine learning, the use of Nesterov acceleration or quasi-Newton methods for multiplier updates, or the connection to other approaches based on Moreau-Yosida regularization.

3.A Loss-Augmented CRF 65 Appendix

3.A Loss-Augmented CRF

In order to extend our learning formulation so as to encompass as well max-margin structured learning (i.e., structured SVM) in additional to maximum likelihood learning, we show in this section that our formulation can be generalized to cover the loss-augmented CRF learning introduced by Pletscher et al. ( 2010) and [START_REF] Hazan | A primal-dual message-passing algorithm for approximated large scale structured prediction[END_REF].

The loss-augmented CRF p γ (y | y , x) is an extension of the standard CRF with additional user-defined loss functions c (y c , y c ) for all cliques and an extra temperature hyperparameter γ ∈ (0, +∞). We introduce a modified natural parameter η (w) := η( w (3.8)

A justification for the form of the loss-augmented CRF is based on a rationale that distinguishes the label to predict y (which is essentially true unknown label) from the label provided by the annotation y . The assumption made is then that, given y c , the annotation y c is independent of x and y c for c = c. This entails that p(y, y |x) = p γ (y | y , x) ∝ p(y | y )p γ (y | x), which yields the above form for p γ (y | y , x; w) by Bayes' rule for p(y | y ) ∝ exp( c∈C c (y c , y c )).

For learning, we use a rescaled maximum likelihood objective (i.e., multiplied by γ) of the form

min w γF 1 γ θ (w) + λ 2 w 2 2 , (3.9)
with which we can see γ only affects the entropy term in the variational representation of F , thus it plays a role to determine the learning regime. When γ → 0, we retrieve a max-margin formulation for structured output learning, since the corresponding variational problem based on Fenchel duality is min

w max µ∈M µ, θ + λ 2 w 2 2 . (3.10)
Note that this is identical to the linear programming relaxation of the structured SVM formulation studied by [START_REF] Meshi | Learning efficiently with approximate inference via dual losses[END_REF].

It is also possible to retrieve the maximum likelihood regime by making a change of variable: w = w/γ. Then, eq.(3.9) becomes

min w F θ(w ) + 1 γ + λγ 2 w 2 2 . (3.11)
Increasing γ decreases the effect of the loss term and simultaneously increases the effect of the regularization. The maximum likelihood regime is thus retrieved by letting γ → +∞ and λ → 0.

3.B Derivations of dual, and relaxed primal and dual objectives

In this section, we derive the dual objective D(µ) of P (w). Given the augmented Lagrangian D ρ (µ, ξ), we first introduce a relaxed primal Pρ (w, δ, ξ) involving a new primal variable δ whose components can be interpreted as messages exchanged between cliques in the context of marginal inference via message-passing algorithms.

The partial minimization with respect to δ then yields the corresponding primal of D ρ (µ, ξ) with respect to µ for a fixed ξ: P ρ (w, ξ) := min δ Pρ (w, δ, ξ), which can be interpreted as a Moreau-Yoshida smoothing of the original objective P ρ (w).

3.B.1 Derivation of the dual objective D(µ)

Given that θ (w) = Ψ w + and introducing the Fenchel conjugate of F L , we have

P (w) = γF L 1 γ θ (w) + λ 2 w 2 2 = max µ∈L Ψ w + , µ -γF * L (µ) + λ 2 w 2 2 .
Given that the local polytope constraints are defined by linear inequalities, weak Slater constraint qualification are satisfied, so that strong duality holds and an equivalent dual problem in µ is obtained by switching the order of min w and max µ :

D(µ) = , µ -γF * L (µ) + min w Ψ w, µ + λ 2 w 2 2 = , µ -γF * L (µ) -λ max w - 1 λ Ψµ, w - 1 2 w 2 2 = , µ -γF * L (µ) - 1 2λ Ψµ 2 2 .

3.B.2 Derivation of an extended primal

P ρ (w, ξ) := min δ Pρ (w, δ, ξ) := γF I θ(w) + A δ γ + λ 2 w 2 2 + ρ 2 δ -ξ 2 .

3.B Derivations of dual, and relaxed primal and dual objectives 67

Proof. Clearly, we have D(µ) = min ξ D ρ (µ, ξ). For a fixed value of ξ, consider the Lagrangian

L ρ,ξ (µ, ν, ν , w, δ) = , µ -γF * I (µ)- 1 2λ ν 2 - 1 2ρ ν 2 + ξ, ν + w, Ψµ-ν + δ, Aµ-ν ;
Then clearly min w,δ L ρ,ξ (µ, ν, ν ; w, δ) = D ρ (µ, ξ). We compute the associated primal as

Pρ (w, δ, ξ) = max µ,ν,ν L ρ,ξ (µ, ν, ν , w, δ) = max u µ, + Ψ w + A δ -γF * I (µ) + max ν ν, -w - 1 2λ ν 2 + max ν ν , ξ -δ - 1 2ρ ν 2 ,
which yields the desired form of P ρ (w, ξ) = min δ Pρ (w, δ, ξ) upon expliciting Fenchel conjugates.

Proposition 3.B.2. The following equalities hold:

min δ F I 1 γ (θ(w) + A δ) = F L 1 γ θ(w) and min ξ,δ Pρ (w, δ, ξ) = P (w). Proof. Notice that min δ F I 1 γ (θ(w) + A δ) = min δ max µ 1 γ θ(w) + A δ, µ + H Approx (µ) -ι I (µ) = max µ 1 γ θ(w), µ + H Approx (µ) -ι I (µ) -ι {Aµ=0} = F L 1 γ θ(w)
, where the second equality follows by exchanging minimization and maximization (strong duality holds by Slater's conditions) and minimizing with respect to δ.

To show that min ξ,δ Pρ (w, δ, ξ) = P (w), it is easy to minimize over ξ first, which cancels out the term ρ 2 δ -ξ 2 by setting ξ = δ. Then, δ only appears in F I and the result follows from the first result.

3.B.3 Interpretation as Moreau-Yosida smoothing

To understand the structure of P ρ (w, ξ), we shall look at Pρ (w, δ, ξ). One may be interested in where does δ comes from? In fact, forming the Lagrangian of min w P (w) with Lagrangian multiplier δ corresponding to the marginalization constraint Aµ = 0, we see that

L(w, δ, µ) := θ (w), µ -γF * I (µ) + λ 2 w 2 2 + δ, Aµ .
Recall that the Moreau-Yosida regularization of a function f is defined as the infimal convolution

M ρf (x) = min z f (z) + ρ 2 z -x 2 .
Both P ρ (w, ξ) and D ρ (µ, ξ) have a nice interpretation in terms of the Lagrangian L and Moreau-Yosida regularization. Note that the Moreau-Yosida regularization admits the same optimum as the original function, and that it is smooth even when the original function is not. It is furthermore γρ γ+ρ -strongly convex if the original function is γ-strongly convex. Proposition 3.B.3. P ρ (w, ξ) and D ρ (µ, ξ) are respectively the Moreau-Yosida regularizations of L µ : w, δ → max µ L(w, δ, µ) and L w : µ, δ → min w L(w, δ, µ) about δ. that is

P ρ (w, ξ) = M ρL µ (w, ξ) = min δ max µ L(w, δ, µ) + ρ 2 δ -ξ 2 2 D ρ (µ, ξ) = M ρL w (µ, ξ) = min δ min w L(w, δ, µ) + ρ 2 δ -ξ 2 2 .
Proof.

For P ρ (w, ξ), note that max µ L(w, δ, µ) ≡ γF I θ(w)+A δ γ + λ 2 w 2 2 . The equivalent form is immediately derived from Proposition 3.B.1. For D ρ (µ, ξ), note that min w L(w, δ, µ) ≡ θ, µ -γF * I (µ) -1 2λ Ψµ 2 + δ, Aµ , and min δ δ, Aµ + ρ 2 δ -ξ 2 ≡ ξ, Aµ -1 2ρ Aµ 2 2 .
Thus, the equivalence holds.

Note that the penalty formulation corresponds to a special case of Pρ (w, δ, ξ) and D ρ (µ, ξ) with ξ = 0. It introduces an additional term ρ 2 δ 2 , thus making the primal strongly convex with respect to δ and the dual smoother in µ. This effect is similar to that of using Moreau-Yosida smoothing. However, the additional term ρ 2 δ 2 will never vanish, so Aµ = 0 will never be satisfied. The more Aµ = 0 is violated, the less the structure of CRF will be perserved.

3.B.4 Duality gaps and representer theorem

Besides, if we define gap(w, δ, µ, ξ) := Pρ (w, δ, ξ) -D ρ (µ, ξ) as an upper-bound estimate of the duality gap

P ρ (w, ξ) -D ρ (µ, ξ), specifically gap(w, δ, µ, ξ) = γF I 1 γ θ (w) + A δ + γF * I (µ) -θ (w) + A δ, µ + λ 2 w 2 + 1 2λ Ψµ 2 --w, Ψµ + ρ 2 ξ -δ 2 + 1 2ρ Aµ 2 -ξ -δ, Aµ ,
we can see that the recovered w and δ by the optimality condition make the 2nd and 3rd term of gap(w, δ, µ, ξ) disappear. We will see later this is important in designing the algorithm to solve max µ D ρ (µ, ξ).

Finally, we give a rough picture of all the quantities that we introduced in this section, which can be easily derived from Proposition 3.B.3.

D(µ) ≤ D ρ (µ, ξ) ≤ P ρ (w, ξ); D ρ (µ) ≤ P (w) ≤ P ρ (w, ξ) ≤ Pρ (w, δ, ξ); max µ min ξ D ρ (µ, ξ) ≤ min w P (w),
with equalities hold for the saddle point (µ , w , ξ ). Moreover, the first-order optimality conditions are given as

w = - 1 λ Ψµ , δ = ξ - 1 ρ Aµ (3.12) Proof. By constructions, D(µ) = min ξ D ρ (µ, ξ) ≤ D ρ (µ, ξ) and P ρ (w, ξ) = min δ Pρ (w, δ, ξ) ≤ Pρ (w, δ, ξ).
Other inequalities are the consequences of Proposition 3.B.3 and the min-max inequality. Since the strong duality holds (Slater conditions satisfied and the problem is convex), we know that the equalities will hold at the saddle point.

Given the saddle point (µ , w , ξ ), to derive w , δ from µ , we know that w , δ = arg min w,δ Pρ (w, δ, ξ ). The result follows after computing ∇ w Pρ (w, δ, ξ ) = 0 and ∇ δ Pρ (w, δ, ξ ) = 0. So our strategy for CRF learning is min ξ max µ D ρ (µ, ξ), since we know that D ρ (µ , ξ ) ≡ L(w , δ , µ ) ≡ P (w ).

Since we work on the space of µ and ξ, to compute the primal objectives or the duality gap, we can use the mapping specified by the optimality condition eq.(3.12). More precisely, we define

w(µ t,s ) = - 1 λ Ψµ t,s , δ(µ t,s , ξ t ) = ξ t - 1 ρ Aµ t,s ,
which is equivalent to the representer theorem. The above condition is also useful to recover intermediate w t,s from µ t,s , which allows us to test on the validation set or decide if we should stop the learning earlier.

3.B.5 Comparison with State-of-the-Art Structured Learning Methods

A number of recent works for CRF learning can be viewed as optimizing formulations which are exactly or fairly close to one of P (w), D(µ), P ρ (w, δ), Pρ (w, δ, ξ) or D ρ (µ, ξ). In the following table, we compare these approaches, in terms of the optimization formulation, the convergence rate (respectively in the primal or in the dual), and the inference oracle used for computing the gradients (or blockwise gradients).

3.C Gini Oriented Tree-Reweighted Entropy

The Bethe entropy [START_REF] Yedidia | Dual decomposed learning with factorwise oracle for structural SVM of large output domain[END_REF]) is generally non-concave. Its concave counterparts, such as the tree-reweighted entropy (Wainwright et al. 2005b) or the region-based entropy [START_REF] Yedidia | Dual decomposed learning with factorwise oracle for structural SVM of large output domain[END_REF][START_REF] London | The benefits of learning with strongly convex approximate inference[END_REF], are only concave on the local consistency polytope, but non-concave on I \ L (i.e., when Aµ = 0). Indeed, the Bethe entropy and its concave variants are of the form

H Bethe (µ) = i∈V c i H i (µ i ) + {i,j}∈E c ij H ij (µ ij )
, where c i and c ij are counting numbers. Even when H Bethe is concave on L, some of the c i or c ij can be negative.

The construction of the oriented tree-reweighted entropy stems from the expression of the entropy of a directed tree as the sum of of the entropy of the root and the conditional entropies of the variable at each node given their parent variable. Precisely, for an oriented tree T with the root i 0 , the joint entropy can be computed as

H T (Y ) := H(Y i 0 ) + j→i∈T H(Y i | Y j ).
(3.13)

On a general graph, if T is a (directed) spanning tree of the graph, then

H T (Y ) := H(Y i 0 ) + t→i∈T H(Y i | Y j ) ≥ H(Y i 0 ) + m k=1 H(Y i k | Y i k-1 , . . . , Y i 0 ) =: H Shannon (Y ). (3.14)
Thus, for any probability distribution over the set of valid directed spanning trees, in which tree T has probability ρ T , the inequality above entails that

H Shannon (Y ) ≤ T ρ T H T (Y ) =: H OTRW (Y )
, where ρ T ≥ 0 and T ρ T = 1.

H T (Y ) is concave since it is a sum of concave functions, and so is H OTRW (Y ) (who is a convex combination of H T (Y )). To see that, we need to prove the following fact.

Remark 3.C.1 (Concavity of the conditional entropy). The conditional entropy

H(Y j | Y i ) is in fact a function of µ ij , namely H(Y j | Y i ) = H(µ ij ) -H(A i µ ij ). Moreover, H(Y j | Y i ) is a concave function of µ ij . Proof. By definition, H(Y j | Y i ) = y j ,y i µ ij (y j , y i ) log y j µ ij (y j , y i ) µ ij (y j , y i ) = H(µ ij ) -H(A i µ ij ).

3.C Gini Oriented Tree-Reweighted Entropy 71

To show H(Y j | Y i ) is concave, we compute its Hessian:

∂ 2 H(Y j | Y i ) ∂µ 2 ij = -diag 1 µ ij + A diag 1 Aµ A = -diag 1 µ ij + diag { 1 μi (y i ) 11 } k i y i =1
where μi = A i µ ij , and denotes entrywise division. Let's focus on the i-th block of the negative Hessian. To show that the i-th block is positive semidefinite, that is, that

diag 1 µ ij (y i , y j ) 1≤y j ≤k j - 1 μi (y i ) 11 0, (3.15)
we can use the Schur complement condition for positive semidefiniteness. Let U = μi (y i ). Since μi (y i ) 0,

L -B U -1 B 0 iff U B B L =   μi (y i ) 1 1 diag { 1 µ ij (y i ,y j ) } 1≤y j ≤k j   0.
We also have L = diag

1 µ ij (y i ,y j ) k j y j =1 0, then U B B L 0 iff U -BL -1 B = μi (y i ) -1 diag {µ ij (y i , y j )} k j y j =1 1 = μi (y i ) -μi (y i ) 0.
Because the last inequality holds, we know eq.( 3.15) must be true, which implies that the Hessian of

H(Y j | Y i ) is negative semidefinite, thus H(Y j | Y i ) is concave.
Note that H OTRW (µ) is concave on the entire set I, unlike many Bethe entropy variants who are only concave in the local consistency polytope.

We define -→ E the directed edge set by expanding each edge from E with two directed edges, ρ i and ρ tij respectively as the probabilities of i (as the root) and i → t appearing in an oriented spanning tree when the latter is drawn with probability ρ T . Then the oriented tree-reweighted entropy takes the form (3.16) where

H OTRW (µ) := {i,j}∈E ρ j|i H e (µ ij ) -H i (A i µ ij ) + ρ i|j H e (µ ij ) -H j (A j µ ij ) + i∈V ρ i H i (µ i ),
H i (µ i ) = -y i µ i (y i ) log µ i (y i ), H e (µ ij ) = -y i ,y j µ i (y i , y j ) log µ i (y i , y j )
and ρ i , ρ i|j , ρ j|i are node/edge appearance probabilities in [0, 1]. H OTRW is concave, since H i is concave and it can be checked that so is

µ ij → H e (µ ij ) -H i (A i µ ij )
(although not strongly concave). It is easy to precompute the appearance probabilities ρ i and ρ tij via a variant of the directed matrix-tree theorem. See [START_REF] Koo | Structured prediction models via the matrix-tree theorem[END_REF] for more details.

A generic difficulty with entropies, is that H i and H e do not have Lipschitz gradients, which prevents the direct application of proximal methods with usual quadratic proximity terms. We thus propose to replace H i and H e by their second-order Taylor approximation around the uniform distribution. This yields a surrogate of the form

H GTRW (µ) := {i,j}∈E ε k i ρ j|i A i µ ij 2 + k j ρ i|j A j µ ij 2 -k i k j (ρ tij + ρ tji ) µ ij 2 + i∈V k i ρ i (1 -µ i 2 ), (3.17)
where ε = 1. Since this function is not strongly convex w.r.t. µ ij because k j I k i -A i A i has a non-trivial kernel, so we also consider variants with ε < 1 and denote them H GTRW,ε . We call this approximation the Gini OTRW entropy, since it is consistent with the definition of Gini conditional entropy of [START_REF] Furuichi | Information theoretical properties of Tsallis entropies[END_REF].

3.D Proof of Theorem 3.6.1 and associated Corollaries

To prove Theorem 3.6.1, we first need to show d(ξ) is a smooth function, and then we build up the associated lemmas which will be used in the proof of Theorem 3.6.1. Finally, in the end of this section, we prove Corollary 3.6.2 and Corollary 3.6.3 as the results to show the linear convergence in the dual and in the primal. Proof. By definition we have

3.D.1 Smoothness of d(ξ)

D ρ (µ, 0) = -µ, + γF * I (µ) + 1 2λ Ψµ 2 + 1 2ρ Aµ 2 .
We then have

d(ξ) = max µ D ρ (µ, ξ) = max µ µ, A ξ -D ρ (µ, 0) so that if J(µ) := D ρ (µ, 0), then d(ξ) = J * (A ξ)
A ξ 1 + -γs 1 - 1 λ Ψ Ψµ 1 - 1 ρ A Aµ 1 = 0 A ξ 2 + -γs 2 - 1 λ Ψ Ψµ 2 - 1 ρ A Aµ 2 = 0
s 1 -s 2 , µ 1 -µ 2 ≥ 0,
which together with the equations above yields

A (ξ 1 -ξ 2 ) - 1 λ Ψ Ψ(µ 1 -µ 2 ) - 1 ρ A A(µ 1 -µ 2 ), µ 1 -µ 2 ≥ 0.
Hence,

ξ 1 -ξ 2 , A(µ 1 -µ 2 ) ≥ 1 λ Ψ(µ 1 -µ 2 ) 2 + 1 ρ A(µ 1 -µ 2 ) 2 ≥ 1 ρ A(µ 1 -µ 2 ) 2 .
Now substituting ∇d(ξ 1 ) -∇d(ξ 2 ) = A(µ 1 -µ 2 ) into the above inequality and using the Cauchy-Schwarz inequality yields

∇d(ξ 1 ) -∇d(ξ 2 ) ≤ ρ ξ 1 -ξ 2 .
That completes the proof.

3.D.2 Associated lemmas for Theorem 3.6.1

We first quantify in the next two lemmas how much D(µ, ξ t ) should be minimized in µ to provide a sufficiently accurate approximate gradient that it guarantees descent on d.

Lemma 3.D.2 (Error on the gradient). Denote μt := µ (ξ t ) = argmin µ D(µ, ξ t ); g t := ∇d(ξ t ) = Aµ (ξ t ) and ĝt := Aμ t . Let ∆t := D ρ (μ t , ξ t ) -D ρ (μ t , ξ t ). We have

1 2L d ĝt -g t 2 ≤ ∆t ,
∈ ξ -∂d * (y (ξ)) ⇒ ξ ∈ ∂d * (y (ξ)).
The strong convexity of d * (y) implies that, for all y,

d * (y) -d * (y (ξ)) -ξ, y -y (ξ) ≥ 1 2L d y -y (ξ) 2 .
But for any µ, we have

D ρ (µ, ξ) = Aµ, ξ -D ρ (µ, 0) ≤ Aµ, ξ -d * (Aµ)
, and, for µ (ξ), this inequality is an equality, since we have D ρ (µ (ξ), ξ) = y (ξ), ξd * (y (ξ)) and y (ξ) = Aµ (ξ). As a consequence, setting y = Aµ, we have

D ρ (µ (ξ), ξ) -D ρ (µ, ξ) ≥ 1 2L d Aµ -Aµ (ξ) 2
by definition of D ρ (µ, ξ). We conclude the proof by substituting µ with μt and ξ with ξ t .

Inexact Dual Augmented Lagrangian Method for CRF Learning Lemma 3.D.3 (Guaranteed decrease on d). If we take inexact gradient on ξ with a fixed step size 1

L d , namely ξ t+1 = ξ t -1 L d ĝt , then d(ξ t ) -d(ξ t+1 ) ≥ τ L d Γ t -∆t , (3.18)
where τ ∈ (0,

L d ) satisfying 1 2τ g t 2 ≥ Γ t . Proof. Since d(ξ) is L d -smooth, we have d(ξ t+1 ) -d(ξ t ) ≤ ∇d(ξ t ), ξ t+1 -ξ t + L d 2 ξ t+1 -ξ t 2
Using the gradient step and ∇d(ξ t ) = g t , the above inequality can be simplified as

d(ξ t+1 ) -d(ξ t ) ≤ g t , -1/L d ĝt + L d 2 1/L d ĝt 2 = 1 2L d ĝt -g t 2 -g t 2 . (3.19)
We notice that the error bound given by the Lemma 2.3 of Hong and Luo (2017) holds for d(ξ). Specifically,

∃τ > 0, such that ∇d(ξ) ≥ τ ξ -ξ . Since d(ξ) is L d -smooth and ∇d(ξ ) = 0, we have d(ξ) -d(ξ ) ≤ L d 2 ξ -ξ 2 ≤ L d 2τ ∇d(ξ) 2 , which implies 1 2τ g t 2 ≥ Γ t ,
where τ = τ L d . By using eq. (3.19) and the above inequality on g t 2 , we obtain

d(ξ t ) -d(ξ t+1 ) ≥ 1 2L d g t 2 -ĝt -g t 2 ≥ τ L d Γ t -∆t .
Since for each value of ξ t the value and gradient of d(ξ t ) need to be computed approximately by minimizing the augmented Lagrangian D ρ (•, ξ t ), and since the difference between two consecutive strongly convex objectives is

D ρ (µ, ξ t ) -D ρ (µ, ξ t-1 ) = ξ t-1 -ξ t ,
Aµ , which is a function that converges to zero when if the sequence {ξ t } t converges, a warm-restart strategy using μt as the initial point to the subproblem max µ D ρ (µ, ξ t+1 ) is beneficial, as characterized by the following lemma. Lemma 3.D. 4 (Dual gap at warm start). Denote ∆ 0 t+1 := D ρ (μ t+1 , ξ t+1 ) -D ρ (µ t+1,0 , ξ t+1 ). If we let µ t+1,0 = μt , then ). The initial gap of µ at iteration t can then be decomposed as

∆ 0 t+1 ≤ (4 + 2 ω ) ∆t + (1 + 2ω)Γ t , ∀ω > 0. ( 3
∆ 0 t+1 = D ρ (μ t+1 , ξ t+1 ) -D ρ (µ t+1,0 , ξ t+1 ) + d(ξ t ) -d(ξ t ) -D ρ (μ t , ξ t ) + D ρ (μ t , ξ t ) = d(ξ t ) -D ρ (μ t , ξ t ) + D ρ (μ t , ξ t ) -D ρ (µ t+1,0 , ξ t+1 ) + D ρ (μ t+1 , ξ t+1 ) -d(ξ t ) = D ρ (μ t , ξ t ) -D ρ (μ t , ξ t ) + D ρ (μ t , ξ t ) -D ρ (μ t , ξ t+1 ) + d(ξ t+1 ) -d(ξ t ) = ∆t + 1 L d ĝt 2 + d(ξ t+1 ) -d(ξ t )
Again, we used the gradient step ξ t+1 = ξ t -1 L d ĝt , and recall that Aμ t = ĝt . Now, we can bound the term ĝt 2 from above using the fact that

1 L d ĝt 2 = 1 L d g t 2 + 2 g t , ĝt -g t + ĝt -g t 2 ≤ 1 L d (1 + ω) g t 2 + (1 + 1/ω) ĝt -g t 2 ,
where the last inequality stems from the Cauchy-Schwarz inequality g t , ĝtg t ≤ g t ĝt -g t and the fact that for any any a, b ∈ R and ω > 0, we have 2ab ≤ ωa

3.D.3 Proof of Theorem 3.6.1

Combining Lemma 3.D.3 and 3.D.4, we now show that IDAL enjoys a linear convergence rate if we take a fixed number of inner iterations to estimate the gradient.

Theorem 3.6.1 (Linear convergence of the outer iteration). Let A be an algorithm that approximately solves max µ D ρ (µ, ξ t ) in the sense that

∃β ∈ (0, 1), E[ ∆t ] ≤ β E[∆ 0 t ].
Then, ∃κ ∈ (0, 1) characterizing d(ξ) and C > 0, such that, if λ max (β) is the largest eigenvalue of the matrix

M (β) = 6β 3β 1 1 -κ ,
Inexact Dual Augmented Lagrangian Method for CRF Learning then after T ex iterations of Algorithm 1 we have

E[ ∆Tex ] E[Γ Tex ] ≤ C λ max (β) Tex E[ ∆0 ] E[Γ 0 ] . Proof. Note that Γ t+1 -Γ t = d(ξ t+1 ) -d(ξ t )
. By using Lemma 3.D.3, we have an upper bound on Γ t+1 in terms of Γ t and ∆t , namely

Γ t+1 ≤ ∆t + (1 -κ) Γ t with κ = τ L d . (3.22)
On the other hand, we can also derive an upper bound on ∆t+1 in terms of Γ t and ∆t . To achieve that, we relate the inner problem with Γ t by running the steps on

µ until E[ ∆t+1 ] ≤ (1 -π) T in E[∆ 0 t+1 ] ≤ β E[∆ 0 t+1 ], which means T in ≥ log β log(1-π)
. By Lemma 3.D.4, we have

E[ ∆t+1 ] ≤ β E[∆ 0 t+1 ] ≤ β 4 + 2 ω E[ ∆t ] + β(1 + 2ω) E[Γ t ]. (3.23)
Combining eq.( 3.23) and eq.( 3.22), and taking expectations on both sides, we get

E[ ∆t+1 ] E[Γ t+1 ] ≤ M E[ ∆t ] E[Γ t ] (3.24)
Since by definition, all the elements of M are positive, we can telescope a sequence of matrix multiplications to get

E[ ∆Tex ] E[Γ Tex ] ≤ M E[ ∆Tex-1 ] E[Γ Tex-1 ] ≤ • • • ≤ M Tex E[ ∆0 ] E[Γ 0 ] (3.25)
Assuming the eigen decomposition of M takes the form M = P DP -1 , then M t = P D t P -1 . Applying norms on both sides of the vector inequality, we have .26) Note that C = P op P -1 op is a constant.

E[ ∆Tex ] E[Γ Tex ] ≤ P op λ max (β) Tex P -1 op E[ ∆0 ] E[Γ 0 ] . ( 3

3.D.4 Proofs of

λ max (β) < 1, we have (1 -π) T in ≤ β, then E[ ∆t ] and E[Γ t ] converge linearly to 0.
Proof. If the algorithm A for the inner loop is linearly convergent, it implies that, after T in iterations,

E[ ∆t ] ≤ (1 -π) T in E[∆ 0 t ] ≤ β E[∆ 0 t ]
. Then, by Theorem 3.6.1, we have

E[ ∆t ] E[Γ t ] ≤ C λ max (β) t E[ ∆0 ] E[Γ 0 ] . As t → +∞, both E[∆ 0 t ]
Aµ = 0 satisfy 1 2 Aμ t 2 2 ≤ 2L d Γ t + 2 σ A 2 op ∆t . Furthermore, if we let D ∞ (µ) := , µ -γF * I (µ) -1 2λ Ψµ 2 2
, so that we have D(µ) = D ∞ (µ) -ι {Aµ=0} , then (given that µ t ∈ I throughout the algorithm) the gap between the smooth part of the objective in μt and at the optimum can be bounded as follows

|D ∞ (μ t )-D ∞ (µ )| ≤ B 2L d Γ t +B A op √ σ 2 ∆t + 1+2 L d ρ Γ t + 1+2 A 2 op ρσ ∆t .
Finally, if Γ t and ∆t converge to 0 linearly then both the residuals Aμ t

(ξ t ) 2 2 ≤ 2L d (d(ξ t ) -d(ξ )) = 2L d Γ t .
But since ∇d(ξ t ) = Aμ t , and using the strong convexity of µ → D ρ (µ, ξ), we have

1 2 Aμ t 2 2 ≤ Aμ t 2 2 + A 2 op μt -μt 2 2 ≤ 2L d Γ t + 2 A 2 op σ ∆t .
For the second inequality, by definition of D ∞ , we have

D ρ (µ, ξ) := D ∞ (µ) + ξ, Aµ - 1 2ρ Aµ 2 2 , and D ∞ (µ ) = D(µ ).
But then we have

|D ∞ (μ t ) -D ∞ (µ )| = |D ∞ (μ t ) -D ρ (μ t , ξ t )| + |D ρ (μ t , ξ t ) -D ρ (μ t , ξ t )| + |D ρ (μ t , ξ t ) -D ∞ (µ )| ≤ | ξ t , Aμ t | + 1 2ρ Aμ t 2 2 + ∆t + Γ t ,
and

| ξ t , Aμ t |+ 1 2ρ Aμ t 2 2 ≤ B Aμ t +B A op μt -μ t 2 + 1 ρ ( Aμ t 2 2 + A 2 op μt -μ t 2 2 ),
which yields the result using the same inequalities as before.

Finally, to show the implications of linear convergence, by Lemma 2.3 of Hong and Luo (2017), there exists τ > 0 such that ∇d(ξ) ≥ τ ξ -ξ . So that, since ∇d(ξ t ) 2 2 ≤ 2L d Γ t , we have that if the sequence (Γ t ) t∈N is bounded then so is (ξ t ) t∈N . Letting B be a bound on ξ t the previous statements shows the results.

3.D.5 Proofs of

T tot ≥ log(β) log λ max (β) log(1 -π) log( ).
Proof. To guarantee that (1 -π) T in < β requires that T in ≥ log (1-π) log(β) and to guaranteed that λ max (β) Tex < requires similarly that T ex ≥ log( ) log(λ(β)) . Taking the product of these inequalities yields the result.

Corollary 3.6.5 (Convergence rate). If κ < 1 2 and α = 1 12 , if T in ≥ log(ακ) log(1-π) , then, there exist a constant C > 0 such that after a total of t T in + s inner updates, we have

E[∆ s t + Γ t ] ≤ C 1 - κπ 2 log(12/κ) t T in +s .
Proof. Using solving the quadratic formula for the largest eigenvalue of a two-bytwo matrix yields

λ max (β) = (1 -κ + 6β) + (1 -κ -6β) 2 + 12β.
It is immediate to verify that λ max (β) < 1 if and only if β < 1 3 κ 1+2κ . This shows that we need to choose β = ακ with α < 1 3(1+2κ) . So in particular, if α < 1 9 , then the previous inequality is satisfied for any 0 < κ < 1.

Moreover, if κ ≤ 1 2 and α < 1 6 , we have λ max (β) = λ max (ακ) < 1 -κ(1 -6α). Indeed, letting x = 3β, and α = 3α, we have

2λ max (β) = (1 -κ + 2x) + (1 -κ -2x) 2 + 4x = 1 -κ + 2x + (1 -κ) 2 + 4xκ + 4x 2 = 1 -κ + 2α κ + (1 -κ) 2 + 4α κ 2 + 4α 2 κ 2 ≤ 1 -κ + 2α κ + (1 -κ) 2 + 4α κ(1 -κ) + 4α 2 κ 2 ≤ 2 (1 -κ + 6ακ). Setting α = 1 12 , given that the rate r is r = 1 -exp log(1-π) log(λmax(β)) log(β)
, we have

r ≥ 1 -(1 -π) log(1-κ 2 ) log( κ 12 ) ≥ log(1 -κ 2 ) log( κ 12 ) π ≥ κ -2 log( κ 12 ) π,
where, for the second and the third inequality, we used the fact that log(1 -z) ≥ z respectively for z = π and for z = -κ 2 .

3.E Convergence results with SDCA 79

3.E Convergence results with SDCA

In this section, we specify the detailed form of D ρ (µ, ξ), and show how to apply the proof scheme of [START_REF] Shalev | Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization[END_REF] to SDCA for the maximization of D ρ (µ, ξ) w.r.t. µ in order to prove Proposition 1. We first write a fully decomposed expression of D ρ (µ, ξ). We have:

D ρ (µ, ξ) = c∈C c , µ c -f * c (µ c ) - 1 2λ τ ∈T c∈Cτ Ψ c µ c 2 - 1 2ρ e∈E i∈e µ i -A i µ e 2 + e∈E i∈e ξ ei , µ i -A i µ e , ( 3.27)
where

-f * c (µ c ) = γh c (µ c ) -ι c (µ c
). We assume here that the entropy surrogate used is such that h c is σ c -strongly concave w.r.t. µ c .

In particular this corresponds to two possible choices:

• The naive Gini entropy, for which

h c (µ c ) = (1 -µ c 2 2
).

• The Gini-OTRW entropy (see Appendix 3.C) for which, given positive numbers ρ i , ρ i|j and ρ j|i for all nodes and edges, we have

-h i (µ i ) = ρ i k i (1 -µ i 2 2 ) for i ∈ V -h ij (µ ij ) = h i|j (µ ij ) + h j|i (µ ij ) for {i, j} ∈ E with h i|j (µ ij ) = k i ρ i|j (ε A j µ ij 2 2 -k j µ ij 2 2 ) for ε < 1 which is σ c -strongly concave in µ c with σ i = 2k i ρ i if i ∈ V else σ {i,j} = 2(1 -ε)k i k j (ρ tij + ρ tji ).
(For ε = 1, the surrogate is not strongly concave, and a modification of the decomposition into a separable terms and a smooth term must be used to leverage strong convexity: see the discussion in Section 6.2 after Proposition 2).

The proof of convergence for SDCA is based on showing that the expected increase in dual objective provides an upper bound on a measure of duality gap. For the problem, we are considering the gap of interest is gap(w, δ, µ, ξ) := Pρ (w, δ, ξ)-D ρ (µ, ξ), which is an upper bound on the duality gap P ρ (w, ξ) -D ρ (µ, ξ). It can be decomposed as follows:

gap(w, δ, µ, ξ) = γF I ( + Ψ w + A δ) + γF * I (µ) -+ Ψ w + A δ, µ + λ 2 w 2 + 1 2λ Ψµ 2 --w, Ψµ + ρ 2 ξ -δ 2 + 1 2ρ Aµ 2 -ξ -δ, Aµ = c∈C f * c 1 γ θc (w, δ) + f * c (µ c ) -θc (w, δ), µ c + τ ∈T λ 2 w τ 2 + 1 2λ c∈Cτ Ψ c µ c 2 --w τ , c∈Cτ Ψ c µ c + e∈E i∈e ρ 2 ξ ei -δ ei 2 + 1 2ρ µ i -A i µ i 2 -ξ ei -δ ei , µ i -A i µ i , (3.28)
where θc is defined by

θc (w, δ) :=        i + Ψ i w τ i + e s δ ei for c = i ∈ V, e + Ψ e w τe - i∈e A i δ ei for c = e ∈ E. (3.29)
We now proceed to characterize the progress of the algorithm at each iteration, and to that end, we introduce appropriate notations. In particular, since ξ is fixed during the algorithm, we drop the dependance on ξ in different functions: Denote the objective of the subproblem w.r.t. clique c as

D ρ,c (µ c , µ s -c ) := -f * c (µ c ) -r(µ c , µ s -c ), (3.30)
with r defined by

r(µ c , µ s -c ) := 1 2λ b∈Cτ c \{c} Ψ b µ s b + Ψ c µ c 2 +            e c 1 2ρ µ i -A i µ s e 2 -µ i , e i ξ ei + i , c = i ∈ V, i∈e 1 2ρ µ s i -A i µ e 2 -µ e , - i∈e A i ξ ei + e , c = e ∈ E.
It is straightforward to show that r is convex and smooth with cliquewise smoothness constants

L i = 1 λ eig max (Ψ i Ψ i ) + |{e : e i}| ρ , i ∈ V L e = 1 λ eig max (Ψ e Ψ e ) + 1 ρ i∈e eig max (A i A i ), e ∈ E.
The proof of convergence hinges on the following key lemma.

3.E Convergence results with SDCA 81 Lemma 3.E.1. Taking one of the following updates on µ c with µ -c fixed:

• µ s+1 c = arg max µc D ρ,c (µ c , µ s -c ). • or, if u ∈ ∂f c ( θc (w s , δ s )), where f c is the conjugate function of f * c , solve α = arg max α∈[0,1] D ρ,c (µ s c + α(u -µ s c ); µ s ), and set µ s+1 c = µ s c + α(u -µ s c ).
Then, with π = min c∈C σc |C|(σc+Lc) , the following inequality holds

E c [D ρ (µ s+1 , ξ) -D ρ (µ s , ξ)] ≥ π E c [ Pρ (w s , δ s , ξ) -D ρ (µ s , ξ)], ∀ξ,
where w s , δ s are updated to maintain the optimality conditions:

w s = - 1 λ Ψµ s , δ s = ξ - 1 ρ Aµ s .
Proof. Consider the following quantity:

Dρ,c (µ c ; µ s ) := -f * c (µ c ) -r(µ s ) -∇ µc r(µ s ), µ c -µ s c - L c 2 µ c -µ s c 2 .
We have Dρ,c (

µ c ; µ s ) ≤ D ρ,c (µ c ; µ s -c ), since µ c → r(µ c , µ s -c ) is L c -smooth. First, for the update µ s+1 c = arg max µc D ρ,c (µ c , µ s -c
), we have that, for any direction u -µ s c and any step size α ∈ [0, 1]

D ρ (µ s+1 , ξ) -D ρ (µ s , ξ) = D ρ,c (µ s+1 c , µ s -c ) -D ρ,c (µ s c , µ s -c ) ≥ D ρ,c (µ s c + α(u -µ s c ), µ s -c ) -D ρ,c (µ s c , µ s -c ) ≥ Dρ,c (µ s c + α(u -µ s c ); µ s ) -D ρ,c (µ s c , µ s -c ). (3.31)
Showing the desired inequality for the second form of update thus implies the inequality for the first type of update. Expliciting Dρ,c (µ

s c + α(u -µ s c ); µ s ), we have Dρ,c (µ s c + α(u -µ s c ); µ s ) = -f * c (µ s c + α(u -µ s c )) (3.32) -r(µ s ) -∇ µc r(µ s ), α(u -µ s c ) - α 2 L c 2 u -µ s c 2 .
Since f * c (u) assumed σ c -strongly convex, we have

f * c (µ s c + α(u -µ s c )) ≤ αf * c (u) + (1 -α)f * c (µ s c ) - σ c 2 α(1 -α) u -µ s
+ α(u -µ s c ); µ s ) ≥ -α f * c (u) -f * c (µ s c ) + ∇ µc r(µ s ), u -µ s c (3.34) -f * c (µ s c ) -r(µ s ) + σ c 2 α(1 -α) - α 2 L c 2 u -µ s c 2 .

Inexact Dual Augmented Lagrangian Method for CRF Learning

Now, if we choose u ∈ ∂f c -∇ µc r(µ s ) , by Fenchel conjugacy, it follows that

f c -∇ µc r(µ s ) = -f * c (u) -∇ µc r(µ s ), u .
One can easily see that θc (w s , δ s ) = -∇ µc r(µ s ) by maintaining the optimality conditions

∀c ∈ C : w s τc = - 1 λ b∈Cτ Ψ b µ s b , ∀e ∈ E, i ∈ e : δ s ei = ξ ei - 1 ρ (µ s i -A i µ s e ).
Thus, we can further simplify eq.( 3.34) as

Dρ,c (µ s c + α(u -µ s c ); µ s ) ≥ α f c ( θc (w s , δ s )) + f * c (µ s c ) -θc (w s , δ s ), µ s c + D ρ,c (µ s c , µ s -c ) + σ c 2 α(1 -α) - α 2 L c 2 u -µ s c 2 ≥ D ρ,c (µ s c , µ s -c ) + α f c ( θc (w s , δ s )) + f * c (µ s c ) -θc (w s , δ s ), µ s c , (3.35) provided that σc 2 α(1 -α) -α 2 Lc 2 ≥ 0, that is, 0 ≤ α ≤ σc σc+Lc . The key observation is that gap(w, δ, µ, ξ) = c∈C f c ( θc (w, δ)) + f * c (µ c ) -θc (w, δ), µ c (3.36)
if we maintain the optimality conditions. By using eq.( 3.36) and taking expectation E c w.r.t. a uniform random choice of the clique c on both sides of eq.( 3.35), we guarantee that, for α ∈ [0, σc σc+Lc ],

E c α |C| gap(w s , δ s , µ s , ξ) ≤ E c Dρ,c (µ s c + α(u -µ s c ); µ s ) -D ρ,c (µ s c , µ s -c ) ≤ E c [D ρ (µ s+1 , ξ) -D ρ (µ s , ξ)].
So, we can choose the maximum value σc σc+Lc for α. It follows that

E c [D ρ (µ s+1 , ξ) -D ρ (µ s , ξ)] ≥ min c∈C σ c |C|(σ c + L c ) E c [ gap(w s , δ s , µ s , ξ)],
as desired.

We can now prove Proposition 3.6.6.

3.E Convergence results with SDCA 83

Proof. Denote ∆ s t := D ρ (μ t , ξ t ) -D ρ (µ t,s , ξ t ). Since we update µ t,s to µ t,s+1 using SDCA, according to Lemma 3.E.1, we have

E c [∆ s t -∆ s+1 t ] = E c [D ρ (µ t,s+1 , ξ t ) -D ρ (µ t,s , ξ t )] ≥ π E c [ Pρ (w(µ t,s ), δ(µ t,s , ξ t ), ξ t ) -D ρ (µ t,s , ξ t )] ≥ π E c [D ρ (μ t , ξ t ) -D ρ (µ t,s , ξ t )] = πE c [∆ s t ],
and π = min c∈C σc |C|(σc+Lc) . The above inequality implies that

E c [∆ s+1 t ] ≤ (1 -π) E c [∆ s t ] ≤ (1 -π) s+1 E c [∆ 0 t ].
The result follows if we set T in = s + 1.

3.E.1 Proof of Propositions 3.6.7 and 3.6.8

Proposition 3.6.7. Let ŵt = w(μ t ). If A is SDCA, then

E[P ( ŵt ) -P (w )] ≤ 1 π E[ ∆t ] + E[Γ t ].
Proof. Recall that P (w ) = D(µ ) = D ρ (µ , ξ ) by Corollary 3.B.4.

P (w t,s ) -P (w ) = P (w t,s ) -D ρ (μ t , ξ t ) + D ρ (μ t , ξ t ) -P (w ) = P (w t,s ) -D ρ (μ t , ξ t ) + D ρ (μ t , ξ t ) -D ρ (µ , ξ ) ≤ P (w t,s , δ t,s , ξ t ) -D ρ (μ t , ξ t ) + d(ξ t ) -d(ξ ) ≤ P (w t,s , δ t,s , ξ t ) -D ρ (µ t,s , ξ t ) + d(ξ t ) -d(ξ ) = gap(w t,s , δ t,s , µ t,s , ξ t ) + Γ t
If A is SDCA, by Lemma 3.E.1, we have

E[P (w t,s ) -P (w )] = E[ gap(w t,s , δ t,s , µ t,s , ξ t ) + Γ t ] ≤ E 1 π (∆ s t -∆ s+1 t ) + Γ t ≤ 1 π E[∆ s t ] + E[Γ t ].
Given that ∆t = ∆ T in t , the result follows by setting s = T in .

Proposition 3.6.8 (Linear convergence in the primal). Let w t,s = w(µ t,s ). If A is a linearly convergent algorithm and the function µ → -H approx + 1 2ρ Aµ 2 2 is strongly convex, then P (w t,s ) -P (w ) converges to 0 linearly. Proof. Note that, if σ is the strong convexity constant of D ρ w.r.t. µ, then given that P ρ (w, ξ) = min δ Pρ (w, δ, ξ) with

Pρ (w, δ, ξ) = γF I θ(w) + A δ γ + ρ 2 δ -ξ 2 + λ 2 w 2 2 ,
Inexact Dual Augmented Lagrangian Method for CRF Learning we also have

P ρ (w, ξ) = max µ µ, Ψ w + γH approx (µ) + ξ, Aµ - 1 2ρ Aµ 2 2 + λ 2 w 2 2 ,
which shows that w → P ρ (w, ξ) is a function with Lipschitz gradient as the sum of w → λ 2 w 2 2 and of the Fenchel conjugate of a strongly convex function. Let L P be its Lipschitz smoothness constant and note that it does not depend on the value of ξ. We thus have

P ρ (w t,s , ξ t ) -P ρ ( wt , ξ t ) ≤ L P w t,s -wt 2 2 .
Then given the representer theorem, and by strong convexity of µ → D ρ (µ, ξ) we have

w t,s -wt 2 2 = Ψ(µ t,s -μt ) 2 2 ≤ 1 σ Ψ 2 op (D ρ (µ t,s , ξ t ) -D ρ (μ t , ξ t ))
So that, since P (w t,s ) ≤ P ρ (w t,s , ξ t ) and P (w ) = P ρ (w , ξ ), we have

P (w t,s )-P (w ) ≤ P ρ (w t,s , ξ t )-P ρ ( wt , ξ t )+P ρ ( wt , ξ t )-P ρ (w , ξ ) ≤ L P σ Ψ 2 op ∆ s t +Γ t .
Finally, global linear convergence in the primal also follows from the linear convergence of ∆t and Γ t .

3.F Notation summary

Given the number of notations in the main paper, we summarize some of them in Tables 3.F.1,3.F.2 and 3.F.3. The block matrices Ψ and A are schematically drawn below to illustrate their structure.

Ψ =     c . . . τc • • • Ψ c     A =     i ij . . . . . . ij • • • I k i -A i    
3.F Notation summary 85

Y i = S k S k := {u ∈ {0, 1} k | u 1 = 1} Y c i∈c k i Y c := × i∈c Y i Y i∈V k i Y := × i∈V Y i T the set of clique types C τ
the set of cliques of type τ M the marginal polytope L the local polytope

I I := i∈V ∆ k i × e∈E ∆ ke
Y i can take k c N k c := |Y c | = i∈c k i τ N the
(n) c R dτ c ×kc Ψ (n) c := φ c (x (n) , y c ) -φ c (x (n) , y (n) c ) yc∈Yc Ψ (n) R τ dτ × c kc see the drawing (n) c R kc (n) c := c (y (n) c , y c ) yc∈Yc (n) R c kc (n) := [ (n) c ] c∈C θ (n) c (w) R kc θ (n) c (w) := Ψ (n) c w τc + (n) c θ (n) (w) R c kc θ (n) (w) := [θ (n) c (w)] c∈C , the natural parameter F R c kc → R the log partition function of θ T (y) R c kc the sufficient statistics µ c
R kc the mean parameter associated with the clique c µ R c kc the mean parameter

F * R c kc → R the Fenchel conjugate of F ι C R c kc → {0, +∞} the indicator function of set C λ R + the coefficient of the regularizer A i R k i ×ke
the matrix encoding the marginalization constraint for i in e. A R e i∈e k i × c kc see the matrix form the strong convexity constant of

µ c → -H Approx (µ) L c the Lipschitz constant of µ c → 1 λ Ψ w(µ) + 1 ρ A δ(µ, ξ t )

Introduction

Deep learning has made remarkable progress on pushing the edge of machine learning tasks such as achieving nearly human level image classification (Krizhevsky et al. 2012b[START_REF] Sun | Deep learning face representation from predicting 10,000 classes[END_REF], defeating world champions in strategy games [START_REF] Silver | Mastering the game of go without human knowledge[END_REF], OpenAI 2017) and outscoring top humans in reading and comprehension tests (Alibaba 2018).

At the heart of deep learning, the key assumption is that these mappings can be well approximated by multilayered artificial neural networks, also known as deep neural networks (DNNs), which have been proven to be universal function approximators (Cybenko 1989a). Training DNNs with backpropagation (BP) dates back at least to 1980s [START_REF] David E Rumelhart | Learning representations by back-propagating errors[END_REF]). An extensive review on the history of deep learning can be found in [START_REF] Schmidhuber | Deep learning in neural networks: An overview[END_REF].

The recent success of deep learning should be attributed to the development of general-purpose processing on graphics processing units (GPGPU), stochastic gradient algorithms, and open-source software frameworks. With stochastic gradient descent (SGD) [START_REF] Robbins | A stochastic approximation method[END_REF] or its variants such as ADAM (Kingma and Ba 2014), modern deep neural networks (DNNs) (Krizhevsky et al. 2012b, Szegedy et al. 2015b, Simonyan and Zisserman 2014a, He et al. 2016a) are capable of training on potentially infinitely large datasets and achieve good performance on a wide range of problems. The DNN training with backpropagation from the outputs to the inputs is also called end-to-end training. In other words, the learning of the data representation and the final predictor are carried out simultaneously. However, there are still many mysteries within this end-to-end framework. They have recently drawn much attention in the community. For example,

• The optimization "puzzle": given that the learning problems of DNNs are highly non-convex, it is not clear why SGD with a proper random initialization converges to a global minima (near zero training error) with high probability [START_REF] Kawaguchi | Deep learning without poor local minima[END_REF], Wu et al. 2017). Moreover, it has been empirically observed that over-parameterized DNNs, namely, DNNs possessing hundreds of millions of parameters, are easier to train [START_REF] Hinton | Improving neural networks by preventing co-adaptation of feature detectors[END_REF], Denil et al. 2013a).

• The generalization "puzzle": classical learning theory suggests that overparameterized models will overfit the training data. Surprisingly, empirical observations [START_REF] Zhang | Understanding deep learning requires rethinking generalization[END_REF], Neyshabur et al. 2018[START_REF] Sagun | Empirical analysis of the hessian of over-parametrized neural networks[END_REF] find that, in the case of DNNs, increasing the model size often leads to better generalization performance even without explicit regularization. DNNs are able to achieve zero training error even with random labels [START_REF] Zhang | Understanding deep learning requires rethinking generalization[END_REF] implying that the complexity of DNNs is extremely high.

For the optimization puzzle, there are several existing works identify that the non-convex problem of DNNs has geometrically elegant structure to facilitate the optimization. [START_REF] Kawaguchi | Deep learning without poor local minima[END_REF] show that, for deep linear neural networks (namely without non-linear activations), every local minima is a global minima, and every critical point that is not global is a saddle point. This result can be extended to non-linear cases with extra conditions, such as the independent activation assumption [START_REF] Kawaguchi | Deep learning without poor local minima[END_REF]) and the "one layer has more neurons than training points" assumption [START_REF] Nguyen | Optimization landscape and expressivity of deep cnns[END_REF]. Although these assumptions are not always realistic (exceptions exist, e.g., [START_REF] Nguyen | Optimization landscape and expressivity of deep cnns[END_REF]'s assumption holds on VGGNet (Simonyan and Zisserman 2014a)), they do shed light on why SGD works so well. In reality, bad local minima exists. But for over-parameterized DNNs, [START_REF] Soudry | Exponentially vanishing sub-optimal local minima in multilayer neural networks[END_REF] show that, under mild assumptions, the volume of the basin containing bad local minima is exponentially vanishing comparing to that of the basin containing global minima as the number of data points increases. Similar results have been reported by different authors [START_REF] Soudry | No bad local minima: Data independent training error guarantees for multilayer neural networks[END_REF][START_REF] Benjamin | Global optimality in neural network training[END_REF], Wu et al. 2018a[START_REF] Boulch | Sharesnet: reducing residual network parameter number by sharing weights[END_REF] towards understanding the loss lanscape of DNNs. Thus, it is very likely that the optimization puzzle is due to the special structure of DNNs, and the over-parameterization induced by increasing the depth and/or the width of a neural network does not only achieve a more powerful model but also eliminate bad local minima [START_REF] Lu | Depth creates no bad local minima[END_REF]Kawaguchi 2017, Li et al. 2017b).

The second puzzle reveals a more interesting property of DNNs, especially of the over-parameterized ones. However, it does not entails that any DNN with a large number of parameters will generalize. Consider the extreme case where the number of hidden units goes to infinity, as considered in convex neural networks [START_REF] Bengio | Convex neural networks[END_REF], Bach 2017). Just training the top layer, which is a convex problem, will result in zero training error, as the randomly initialized hidden layer has all possible features (Soudry andHoffer 2017, Neyshabur et al. 2018). Obviously, such an extreme case will have serious overfitting issues. On the other hand, empirically designed DNNs often have good generalization although being heavily over-parameterized. For example, the largest wide residual network considered by Zagoruyko and Komodakis (2016c) has 500× more parameters than the size of CIFAR-10 (Krizhevsky 2009a). VGGNet has even 138M parameters comparing to 1.2M images in ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF], which is already considered as one of largest scale datasets. Thus, there must exist some implicit regularizations that are common in the network architectures or in the optimizers. Many existing works attempt to explain this phenomenon by, for example, the flat minima conjecture (Hochreiter and Schmidhuber 1997b[START_REF] Shirish Keskar | On large-batch training for deep learning: Generalization gap and sharp minima[END_REF], Wu et al. 2017), the PAC-Bayes theory (Dziugaite and Roy 2017, [START_REF] Zhou | Compressibility and generalization in large-scale deep learning[END_REF][START_REF] Valle Pérez | Deep learning generalizes because the parameter-function map is biased towards simple functions[END_REF]) and the information theory perspective [START_REF] Hinton | Keeping the neural networks simple by minimizing the description length of the weights[END_REF][START_REF] Tishby | Deep learning and the information bottleneck principle[END_REF][START_REF] Achille | Emergence of invariance and disentangling in deep representations[END_REF]. So far, the generalization puzzle is still one of the hottest topics in deep learning.

Although over-parameterization is important from the modeling and the optimization point of view, it is not necessarily a desired property from the computation and the deployment aspects. It is not surprising that there is a huge redundancy in DNNs. One could prune out even 99% parameters without hurting the model performance (Han et al. 2015a). Therefore, a natural question emerges: can we design a compact DNN which inherits the geometry and generalization properties of its over-parameterized counterparts while only contains minimal parameters/weights to be learned?

To answer this question, I first study the common DNN architectures in Section 4.2, the potential issues of DNNs in Section 4.3, and then review existing model compression approaches in Section 4.4. It turns out that the generalization is closely related to the compressibility of the model, which is formally characterized by different theories, such as the minimum description length (MDL) principle (Rissanen 1978, Hinton and[START_REF] Hinton | Keeping the neural networks simple by minimizing the description length of the weights[END_REF] and the information bottleneck principle [START_REF] Tishby | The information bottleneck method[END_REF][START_REF] Tishby | Deep learning and the information bottleneck principle[END_REF], for which I briefly review in Section 4.5. Finally, in Section 4.6, I discuss the knowledge distillation method, which offers a new but probably more straightforward way to mitigate over-parameterization.

Deep Network Architectures

A neural network is a special composite function of the input signal in the sense that the functions are connected in a way resembling the network of biological neurons. A function in the neural network is sometimes referred as a (hidden) layer. Obviously the input and the output are the first and the last layers respectively.

The computation from the input to the output is called the forward pass. Accordingly, the backward pass occurs when we try to compute the derivatives with respect to the layers as well as the parameters associated to the functions, which is also known as the backpropagation-an application of chain rule.

The first neural network, called the threshold logic, was proposed by [START_REF] Warren | A logical calculus of the ideas immanent in nervous activity[END_REF]. Back to the time when backpropagation firstly gained recognition [START_REF] Werbos | Beyond regression:" new tools for prediction and analysis in the behavioral sciences[END_REF][START_REF] David E Rumelhart | Learning representations by back-propagating errors[END_REF]), the multilayer perceptron (MLP) was one of the first neural networks considered in supervised learning (Cybenko 1989a), which includes alternatively linear functions and non-linear functions such as sigmoid, tanh, rectified linear unit (ReLU) and so on.

For specific purposes, different types of architecture are proposed. The most common type is called feedforward neural network, such as the aforementioned MLP. On the other hand, if there exists a function in the network that the previous outputs from itself are used as inputs, the neural network is called recurrent neural network [START_REF] David E Rumelhart | Learning representations by back-propagating errors[END_REF]), which has an advantage in modeling arbitrarily long sequences.

The recent development of neural networks is mainly focused on convolutional neural networks (CNNs) [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]) due to their big success in image classification (Krizhevsky et al. 2012b). CNNs were designed for images, inspired by the structure of visual cortex, but they have been extended to other kinds of data (Oord et al. 2016). Convolutional neural networks make use of the spatial information in the input signals to enable weight sharing, which makes it invariant to translations of the input. Besides, weight sharing significantly reduces the number of parameters to be learned compared to fully connected neural networks (e.g. MLPs) with the same input-output dimensions. We list several commonly used CNN architectures in the following, and summarize their characteristics in terms of the number of parameters and their computational costs (measured by FLOPS) in Table 1.

• The LeNet proposed by LeCun et al. ( 1998) is one of the first convolutional neural networks. The architecture starts with two convolutional layers, each of which followed by a subsampling layer, to build up a task specific representation from the input. Then, there are two fully connected (FC) layers taking the representation as input to output the final classification scores/probabilities.

• AlexNet (Krizhevsky et al. 2012b) can be seen as a deeper and wider extension of the LeNet, which leads to a significant improvement in the accuracy of image classification. It was considered as the "return" of deep learning. There are other key differences between AlexNet and LeNet. Rather than using saturated nonlinear activations, such as sigmoid function or tanh function, AlexNet uses the rectified linear units (ReLU) to mitigate the vanishing gradient problem [START_REF] Bengio | Learning long-term dependencies with gradient descent is difficult[END_REF]). Besides, it also replaces averagepooling operations with max-pooling operations in subsampling layers in order to avoid the blur effects. Apart from the architecture differences, it is one of the first examples showing that dropout [START_REF] Hinton | Improving neural networks by preventing co-adaptation of feature detectors[END_REF]) can be used after linear or convolutional layers to avoid overfitting in overparameterized networks. AlexNet has 61 millions parameters across five convolutional layers and three fully connected layers. The spatial resolution is reduced three times before the full connected layers.

• VGGNet (Simonyan and Zisserman 2014a) is an even larger variant of AlexNet with 13 convolutional layers and 5 max-pooling layers. It is the first neural network using small convolution filters (e.g. 3 × 3 in spatial dimensions), which is based on an empirical observation that large receptive fields 1 can be preserved by stacking multiple small convolutions. However, VGGNet has 138 millions of parameters across 13 convolutional layers and 3 fully connected layers in order to achieve a deep architecture, which makes it one of the largest architectures in the literature.

• The network in network (NIN) (Lin et al. 2013a) starts a trend of fully convolutional neural networks (Springenberg et al. 2014, Szegedy et al. 2015b, He et al. 2016a). It is easy to verify that a fully connected layer with input dimension C • H • W and output dimension K is equivalent to a convolutional layer including K convolutional kernels of shape C × H × W . Thus, fully connected layers can be implemented by convolutions unless there are computational concerns. A big advantage is that fully convolutional CNNs do not require inputs to have uniform spatial dimensions. If the input tensor is C × H × W , we get a K × 1 × 1 output tensor. When the width or the height of the input tensor is larger, the output tensor has spatial dimensions larger than 1. To obtain a K × 1 × 1 output tensor, the global average-pooling over the spatial dimensions is introduced. NIN sheds light on the special case of convolution when H = W = 1 (i.e. 1 × 1 convolution) can be seen as applying MLP on each bin of the input tensor whose shape is C × 1 × 1. This insight has been very influential on recent neural network architectures.

• GoogLeNet (Szegedy et al. 2015b) was one of the first attempts to reduce the computational overhead without sacrificing the performance. The inception module (a.k.a. the bottleneck layer) was first introduced in GoogLeNet.

Inspired by NIN, GoogLeNet uses 1 × 1 convolutions to reduce 4 times the number of input channels before going through an computationally expensive convolutional layer. This saved a large number of operations while maintaining the performance. There are several extensions of GoogLeNet, such as Inception V2 (Ioffe and Szegedy 2015b) and V3 [START_REF] Szegedy | Rethinking the inception architecture for computer vision[END_REF].

Apart from the architectural differences, the batch normalization (BN) is introduced as an alternative regularizer to weight decay, dropout etc. BN accelerates the training by reducing internal covariate shifts. It also alleviates the problems of vanishing/exploding gradients.

• ResNet (He et al. 2016a) and Komodakis (2016c) studied the trade-off between increasing the number of layers and increasing the number of parameters per layer, and proposed a variant of ResNet called Wide ResNet.

For a more complete discussion on neural network architectures, we refer the reader to the survey by [START_REF] Bulten | An analysis of deep neural network models for practical applications[END_REF]. We show in Figure 1 a summary picture taken from their paper.

Memory and Energy Issues with Over-Parameterized DNNs

From Figure 1 and Table 1, we can see that all commonly used deep neural networks contain millions of parameters, and require gigas of FLOPS for a single forward pass. This situation has been mitigated in recent architectures by replacing fully connected layers with convolutional layers and by using convolutions with small kernels. In general, fully connected layers introduce more parameters but require less FLOPS comparing to convolutional layers. However, none of the designs of neural networks can avoid over-parameterization completely, even with a good balance between the number of parameters and the FLOPS. Inevitably, over-parameterization leads to inefficiency in memory and energy. Moreover, the resulting model is hard to interpretable. Applications such as embedded systems, internet of things and mobile phones are examples where computers have limited processors, memory and batteries. The goal of such resource limited applications is quite different from that of achieving human-level performance. Deploying deep neural networks on embedded systems [START_REF] Stylianos | Toolflows for mapping convolutional neural networks on fpgas: A survey and future directions[END_REF]) has become a recent trend towards intelligent mobile computing. However, it is infeasible to design an architecture for each embedded system from scratch.

One valid solution is to adaptively compress the universal model. Specifically, according to the memory and computation requirements of a given application, we reduce the storage of network parameters or eliminate less important parameters. This is generally called model compression in deep learning.

Energy efficiency is another desideratum from an implementation perspective. Regarding this perspective, Max Welling shared his opinion in his keynote talk "Intelligence per kilowatthour" at ICML 2018 (Welling 2018):

In the 19th century the world was revolutionized because we could transform energy into useful work. The 21st century is revolutionized due to our ability to transform information (or data) into useful tools. Driven by Moore's law and the exponential growth of data, artificial intelligence is permeating every aspect of our lives. But intelligence is not for free, it costs energy, and therefore money. Evolution has faced this problem for millions of years and made brains about a 100× more energy efficient than modern hardware (or, as in the case of the sea-squirt, decided that it should eat its brain once is was no longer necessary). I will argue that energy will soon be one of the determining factors in AI. Either companies will find it too expensive to run energy hungry ML tools (such as deep learning) to power their AI engines, or the heat dissipation in edge devices will be too high to be safe. The next battleground in AI might well be a race for the most energy efficient combination of hardware and algorithms.

We reuse an interesting plot in Figure 2 from Welling (2018) illustrating the development of deep neural networks and the corresponding energy consumptions. With such a rapid growth, by 2025, the hardware can host a neural network with as many parameters as neurons in the human brain. However, digital computers are not comparable with its biological counterpart in terms of energy efficiency. Moreover, artificial neural networks usually only target on a single task. Without model compression, we are simply wasting the resource of the earth.

Model Compression Approaches

Before discussing what is the best way to adaptively compress a neural network, one may ask: is it possible to compress a pretrained network without sacrificing its performance significantly? The answer is very likely, since it can be seen from Table 1 and Figure 1 that the performance of a neural network is not fully determined by the number of parameters. For example, ResNet-18 has only 5.6M parameters comparing to VGGNet which has 138M parameters. However, their top-1 accuracy and FLOPS are almost identical. Denil et al. (2013a) shows that there is significant redundancy in the parameterization and that up to 95% parameters can be predicted from a small portion of learned parameters. That is, parameters are not equally important. [START_REF] Li | Measuring the intrinsic dimension of objective landscapes[END_REF], [START_REF] Giryes | Deep neural networks with random gaussian weights: A universal classification strategy?[END_REF] argue that the key to learn a good representation of the data is the architecture itself. With a good network architecture, a large portion of parameters can be even random values. All of these observations shed light on why model compression is possible. More intuitions and explanations will be discussed in Section 4.5.

We now provide a brief survey for model compression, classifying these approaches in terms of their main characteristics.

Low-rank approximation

Not surprisingly, there are a large amount of weights close to zeros in a learned neural network. Thus, the sparsity may be the first property to be explored in order to reduce the model size. Given a pretrained neural network, the weights are presented as matrices or tensors, Denton et al. (2014a), Jaderberg et al. (2014a) explored the idea of low-rank approximation as a postprocessing step to optimize the weight storage. Specifically, the original weight matrices/tensors are sparsely reconstructed through a matrix/tensor decomposition. For example, Denton et al. (2014a) perform a singular value decomposition for each convolutional layer, and fine-tune the upper layers until the prediction performance is restored. They achieve a speedup in convolutional layers by a factor of 2 to 3 times and a reduction in parameters in fully connected layers by a factor of 5 to 10 times.

Weight pruning

Since not all weights are equally important, some weights can be pruned according to some "saliency" criterion, which has been explored in the 90s by LeCun et al. (1990a), [START_REF] Hassibi | Optimal brain surgeon and general network pruning[END_REF]. Standard critera include magnitude based thresholding, that is, |w| < threshold, and the minimal decrease in training loss (LeCun et al. 1990a[START_REF] Hassibi | Optimal brain surgeon and general network pruning[END_REF]): min q min δw ∆loss(δw) | δw q + w q = 0 .

Weight pruning can also be done in an iterative way by gradually building a mask of important weights (Han et al. 2015b). Retraining intervenes at each iteration to update the masked weights. Alternatively, the mask can be a part of the optimization. This was an idea proposed by [START_REF] Guo | Dynamic network surgery for efficient dnns[END_REF]. They iteratively solve min

w,T L(w T ),
where the mask T is a function of w; denotes element-wise product.

It is also possible to prune out a whole convolutional filter. Liu et al. ( 2017) associate a scaling factor (reused from a batch normalization layer) to each channel in the convolutional layers. Then, a sparsity inducing regularization is imposed on these scaling factors during training to automatically identify unimportant channels. The channels with small scaling factor values will be pruned, followed by finetuning to achieve comparable (or even higher) accuracy as a normally trained full network.

Weight sharing

Weight sharing plays an important role in convolutional neural networks. This idea has been further extended to design even more efficient convolutional modules that directly build up efficient CNNs rather than compressing a pretrained one.

Flattened CNNs (Jin et al. 2014) allows for an acceleration of the feedforward pass by replacing each C × H × W convolution with C × 1 × 1, 1 × H × 1, and 1 × 1 × W convolutions, which can be seen as applying SVD on the original weights, but this is trained in an end-to-end manner rather than through post-processing.

SqueezeNet (Iandola et al. 2016a) achieves AlexNet-level accuracy with 50 times fewer parameters. It employs strategies such as replacing 3 × 3 filters by 1 × 1 filters; decreasing the number of input channels for 3 × 3 filters; downsampling late in the network so that convolution layers have large activation maps.

Other efficient CNNs including MobileNet (Howard et al. 2017a), ResNext [START_REF] Xie | Aggregated residual transformations for deep neural networks[END_REF] and ShuffleNet [START_REF] Ma | Shufflenet V2: practical guidelines for efficient CNN architecture design[END_REF], which are built upon efficient CNN modules, such as grouped convolution, depthwise convolution, channel shuffle and so on.

Note that all the aforementioned efficient CNNs can be combined with other compression methods with a minor loss of accuracy.

Weight generated by a compact neural network

As mentioned by Denil et al. (2013a), it is possible to accurately predict the remaining weights from only a few weights. This is due to the fact that weights in learned networks tend to be structured. Inspired by this observation, a line of research consists of several different ways to form a weight generator to achieve model compression or relevant goals, which itself is a small network (i.e. a meta model), but it generates the actual weights for a larger network for the main task.

• The fast-weight network was proposed by [START_REF] Schmidhuber | Learning to control fast-weight memories: An alternative to dynamic recurrent networks[END_REF] for MLPs, in which one network produces context-dependent weight changes for a second network. A subsequent work by [START_REF] Gomez | Evolving modular fast-weight networks for control[END_REF] demonstrated practical applications for the fast-weight network in the context of artificial control, where a generator network is learned by evolution algorithms.

• The HyperNetworks (Ha et al. 2017a) was proposed to allow non-shared weights for RNNs, since the weight sharing with recurrence was the main issue that causes gradient vanishing and exploding problems. The hypernetwork generates weights adaptively according to the layer-embedding vectors.

• The deep fried convnets [START_REF] Yang | Deep fried convnets[END_REF]) uses a special weight generator, reparameterizing a fully connected layer with d inputs and n outputs by the so called adaptive fastfood transform, which is a generalization of the fastfood transform for approximating kernels. It reduces the storage and the computation costs from O(nd) to O(n) and O(n log d) respectively.

• The dynamic filter networks by [START_REF] Jia | Dynamic filter networks[END_REF] was proposed in the context of video prediction and stereo vision. It depends on two related inputs. One input is used to predict sample specific and location specific information, forming the weights (i.e. filters) of the convolution, which will be applied on another input.

• The Bayesian neural networks [START_REF] Blundell | Weight uncertainty in neural networks[END_REF][START_REF] Lacoste | Deep prior[END_REF] can also be seen as a particular example of the weight generation, since it actually learns a distribution of the weights rather than a point estimate. In this case, the weight generator is exactly the distribution, generating weights amounts to sampling weights.

Depending on how small the meta network is, the actual weights to be learned is much less than learning the main network directly. It is certainly not the best method to yield the best compression rate, since the size of the meta network is not adaptive to different scenarios. Comparing to post-processing methods, such as low-rank approximation and weight pruning, this method is able to train all the modules in an end-to-end manner.

It is worth mentioning that the weight generating idea also has a wide application in meta learning [START_REF] Gordon | Decision-theoretic meta-learning: Versatile and efficient amortization of few-shot learning[END_REF], Garnelo et al. 2018a[START_REF] Bertinetto | Learning feed-forward one-shot learners[END_REF], Gidaris and Komodakis 2018).

Quantization

It has been recognized back in the 90s that the weights and activations of the neural networks do not take values in the whole real axis, thus they are not necessarily stored in the floating point format [START_REF] Fiesler | Weight discretization paradigm for optical neural networks[END_REF][START_REF] Balzer | Weight quantization in boltzmann machines[END_REF]. Quantization is the process of constraining an input to take value from a discrete set, given the input originally ranges over a large set of values. In the case of deep learning, quantization has been applied to weights, activations and gradients through rounding [START_REF] Courbariaux | Binaryconnect: Training deep neural networks with binary weights during propagations[END_REF], Wu et al. 2018b), vector quantization [START_REF] Gong | Compressing deep convolutional networks using vector quantization[END_REF], Chen et al. 2015) or optimization with discrete-set constraints (Rastegari et al. 2016, Carreira-Perpinán andIdelbayev 2017), which is aimed at achieving the same level of accuracy as that of their full-precision counterparts.

In particular, the quantization can be applied either in the postprocessing phase to coarsen the pretrained weights or in an end-to-end manner to directly obtain discrete weights and/or activations. The former is easier to implement. For example, [START_REF] Gong | Compressing deep convolutional networks using vector quantization[END_REF] implemented vector quantization for the fully connected layers, which was able to reduce the memory up to 24 times while keeping the drop of the top-5 accuracy within 1%. Han et al. (2015a) used a similar vector quantization to achieve the state-of-the-art model compression in accordance with magnitude-based weight pruning and entropy encoding in an iterative manner.

The end-to-end training of quantized neural networks with gradient based methods is more challenging due to the presence of discrete weights. It turns out that passing through a quantization layer is somehow equivalent to sampling from a discrete distribution. The latter has been widely studied in deep neural networks [START_REF] Hinton | Improving neural networks by preventing co-adaptation of feature detectors[END_REF][START_REF] Diederik P Kingma | Variational dropout and the local reparameterization trick[END_REF]. As an example, BinaryConnect [START_REF] Courbariaux | Binaryconnect: Training deep neural networks with binary weights during propagations[END_REF] uses quantization in the forward pass and treats it as an identity function in the backward pass. This idea is also known as the straight through gradient estimator (Bengio et al. 2013). Although it is a biased estimator, it has been shown to be an useful trick in practice. The same idea can also be applied to activations leading to fully quantized neural networks (Courbariaux et al. 2016[START_REF] Hubara | Quantized neural networks: Training neural networks with low precision weights and activations[END_REF].

Apart from being memory efficient, the quantization also enables energy efficient computations. For example, the dot product between two binary vectors amounts to counting the number of 1's in the logical conjunction of these two vectors. The bitwise operations can be carried out using arithmetic logic, which are much less costly than floating point operations.

Towards Understanding Generalization via Compression

In previous sections, we have seen that deep neural networks are typically designed to be over-parameterized, which renders potential limitations in applications due to memory and/or energy constraints. What is more critical is that a model with thousands of parameters are considered to have high complexity. This is a bad news in light of the classical learning theory, since the classical generalization bounds are based on measures of model complexity. However, deep learning has impressive generalization performance for a wide range of problems. This indicates that there must be some form of inductive bias or implicit regularization along with the network architectures or with the stochastic optimizers, which facilitates the learning converging to good solutions with high probability.

We have also reviewed model compression methods for deep learning, many of which are able to reduce a large number of parameters without sacrificing the generalization. This means that the compression does not ruin the recipe of deep learning. Han et al. (2015a) show that the compressed model can sometimes be even better than the original model. Therefore, we may interpret the compression as a particular regularization if we incorporate it during the training. It turns out that the connection between compression and generalization has already been exploited in different contexts. From a data communication point of view, the idea of understanding generalization through compression has been employed by [START_REF] Hinton | Keeping the neural networks simple by minimizing the description length of the weights[END_REF] via the minimum description length (MDL) theory [START_REF] Rissanen | Modeling by shortest data description[END_REF]. From an optimization point of view, the flat minima conjecture (Hochreiter and Schmidhuber 1997b) associates the model compressibility with the geometry of the optimization landscape. More recently, compression scheme has been embedded into the probably approximately correct (PAC) learning framework to achieve PAC bounds [START_REF] Arora | Stronger generalization bounds for deep nets via a compression approach[END_REF]) and PAC-Bayes bounds (Dziugaite and Roy 2017, [START_REF] Zhou | Compressibility and generalization in large-scale deep learning[END_REF][START_REF] Valle Pérez | Deep learning generalizes because the parameter-function map is biased towards simple functions[END_REF]. Based on information theory, [START_REF] Tishby | Deep learning and the information bottleneck principle[END_REF] established the informationtheoretical foundation for representation learning, where they interpret SGD as an implicit regularizer. Although being agnostic to the learning problem, it guides the learning biases towards achieving compressed representations.

The generalization puzzle

Due to the success of deep learning, many researchers attempt to answer the following question: why does deep learning generalizes so well even without explicit regularizations? The classical intuitive explanation (especially for CNNs) is that deep learning builds up a hierarchical feature representation in analogy to the primary visual cortex of the mammalian brain; the generalization comes from this bio-inspired "inductive bias", which leads to a special hypothesis space.

A recent empirical result shown by [START_REF] Zhang | Understanding deep learning requires rethinking generalization[END_REF] challenges this explanation, for which completely different opinions2 have been expressed. Some of the reviewers found these results not surprising at all, but others viewed them as thought-provoking.

To test the capacity of common DNNs, [START_REF] Zhang | Understanding deep learning requires rethinking generalization[END_REF] conduct randomization tests on CIFAR-10 (Krizhevsky 2009a) and ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF] with three network architectures: Inception V3 [START_REF] Szegedy | Rethinking the inception architecture for computer vision[END_REF], AlexNet (Krizhevsky et al. 2012b) and a 2-layer MLP with 512 hidden units, which means either the labels or the images are corrupted via the following ways:

• Random labels: all the labels are replaced with random ones.

• Shuffled pixels: a random permutation of the pixels is chosen and then the same permutation is applied to all the images in both the training and testing set. • Random pixels: a different random permutation is applied to each image independently.

• Gaussian: A Gaussian distribution (with matching mean and variance to the original image dataset) is used to generate random pixels for each image.

The result on CIFAR-10 is shown in Figure 3(a). In the case where all labels are random, the inputs and the labels are independent. However, DNNs are still able to achieve almost zero training errors, although the convergence time is much slower. Of course, DNNs overfit the train-set but this is considered unusual since DNNs rarely overfit real datasets. An immediate implication from this experiment is that DNNs have high capacity. That is, the hypothesis space of DNNs admits high complex solutions. In practice, DNNs generalize well on real datasets. For example, Neyshabur et al. (2018) show that (in Figure 3(b)), on CIFAR10 with ResNet18, training with increasing number of parameters leads to a decrease in the testing error. In this case, both the training and testing errors converge to fix values, which suggests that being over-parameterized does not affect the generalization capability. This is not a standalone experiment, but an observation practically verified by a large number of network architectures [START_REF] Novak | Sensitivity and generalization in neural networks: an empirical study[END_REF]. Similarly, [START_REF] Wu | Towards understanding generalization of deep learning: Perspective of loss landscapes[END_REF] argue that the reason why DNNs do not overfit in general is because learning tends to converge to a low complexity solution. They empirically verify that (in Figure 3(c)) over-parameterized MLPs do not overfit on a three-order polynomial regression with only 5 data points. Although there are many different complex solutions, learned DNNs favor simple solutions that agree with the train-set regardless of the number of layers.

To summarize, we leanrned several conclusions from these experiments:

• DNNs have high capacity to fit almost any dataset.

Towards Understanding Generalization via Compression

101

• DNNs do not overfit on real datasets as the number of parameters increasing, which is possibly due to implicit regularization or simply because adding more parameters does not necessarily increase the model complexity.

• DNNs tend to converge to simple solutions.

In order to understand the implications from these conclusions, we may turn to the classical framework in the learning theory-probably approximately correct (PAC) learning.

Denote by R(f

) := E x,y [ (f (x), y)] the risk of f ∈ H wrt a loss function (•, •), and R(f ) := 1 n n i=1 (f (x i ), y i ) is the empirical risk,
where H is the hypothesis space. The PAC learning framework states that, with high probability, the generalization error bound for any f ∈ H takes the form of

R(f ) -R(f ) ≤ O complexity(H) √ n , (4.1)
where complexity(H) is a complexity measure of the hypothesis space H, such as the VC dimension [START_REF] Blumer | Learnability and the vapnik-chervonenkis dimension[END_REF] or the Rademacher complexity [START_REF] Koltchinskii | Rademacher penalties and structural risk minimization[END_REF]. [START_REF] Harvey | Nearly-tight vc-dimension bounds for piecewise linear neural networks[END_REF] give a near-tight estimate of the VC dimension for DNNs with ReLU activations:

VC-dimension(H) = O LE log(E) (4.2)
with L the number of layers and E the number of edges/links in the network.

Combining the conclusions we learned from the last section, one may notice that the classical PAC bounds can hardly explain the generalization of deep learning.

Several possible reasons are listed as below.

• The VC-dimension is dependent of the number of parameters. However, this leads to a loose generalization bound, cannot explain the actual behaviors. Moreover, when the VC-dimension is much larger than the number of data points in the train-set, we can no longer identify a hypothesis with good generalization, since the hypothesis space can shatter a set that is larger than the train-set. In other words, there always exists two hypotheses predict exactly the same in the train-set, but they disagree on the test-set.

• The PAC learnability holds for the entire hypothesis space and for any distribution of the data. Thus, the universal statement also holds for the hypothesis with the highest complexity. In practice, the obtained solutions are in fact much simpler making the theoretical results less informative.

Given this situation, there are many attempts recently towards solving this puzzle. Among these attempts, the flat minima conjecture (Hochreiter and Schmidhuber 1997b[START_REF] Shirish Keskar | On large-batch training for deep learning: Generalization gap and sharp minima[END_REF], Wu et al. 2017) is one of the well established conjectures based on the insights of the special geometry of the loss landscape. From a stochastic perspective, the PAC-Bayes explanations (Dziugaite and Roy 2017, [START_REF] Zhou | Compressibility and generalization in large-scale deep learning[END_REF][START_REF] Valle Pérez | Deep learning generalizes because the parameter-function map is biased towards simple functions[END_REF]) and the information theoretical explanations [START_REF] Hinton | Keeping the neural networks simple by minimizing the description length of the weights[END_REF][START_REF] Tishby | Deep learning and the information bottleneck principle[END_REF][START_REF] Achille | Emergence of invariance and disentangling in deep representations[END_REF]) also shed light on the generalization puzzle. In fact, all of these conjectures and explanations are either directly linked to model compression or derived from the compressibility.

Sharpness: the bridge between compressibility and generalization

Keskar et al. ( 2016) observes that there is a degradation in the performance when using SGD with large batch size. They empirically find that large-batch SGD tends to converge to sharp minima, which are minima in which Hessians have large spectral norms. The sharpness is a widely used geometrical property in optimization. For example, it is used by Łojasiewicz (1993), [START_REF] Kurdyka | On gradients of functions definable in o-minimal structures[END_REF] to define the Kurdyka-Łojasiewicz condition. In deep learning, Hochreiter and Schmidhuber (1997b) has associated sharpness with generalization in terms of the minimum description length (MDL) principle (see Section 4.5.3): sharp minima correspond to weights which have to be specified with high precisions, while flat minima on the other hand only need to be determined with low precisions. In the terminlogy of MDL, the weights with low precision need fewer bits to describe (i.e. are of low complexity), thus, they have better generalization. Intuitively, low-precision weights are more robust to noise and dataset/domain shifts. As an example (Figure 4), if there is a small shift between the training function and the testing function, who generate the training data and the testing data respectively, a sharp minimum on the training function may be a point with high value on the testing function. This is exactly a case of overfitting. The sharpness has more implications. [START_REF] Wu | Towards understanding generalization of deep learning: Perspective of loss landscapes[END_REF] show that flat minima imply smoother classifiers. Here, smoothness is in terms of the sensitivity to input changes. Thus, it also entails low complexity. Theorem 4.5.1 [START_REF] Wu | Towards understanding generalization of deep learning: Perspective of loss landscapes[END_REF], Corollary 1). For 2-layer neural networks of

f (x) := K k=1 a k σ(w k x + b k ) s.t. w F ≤ W (4.3)
with W a positive constant and σ(•) the ReLU activation function, denote by R(f ) the empirical risk wrt f (x), then

2E ∇ x f (x) 2 ≤ ∇ 2 c R(f ) 2 F + W, (4.4)
for f (x) trained on a large enough dataset.

This theorem shows that if we train a 2-layer ReLU DNN to convergence, and the minimum is flat (i.e. ∇ 2 c R(f ) 2 F is small), then the returned DNN has low complexity. A similar implication has been demonstrated by [START_REF] Achille | Emergence of invariance and disentangling in deep representations[END_REF], where they show that the mutual information between a flat minimum and the true labels is low. On the contrary, the loss landscape of the learning with the random labels does not contain flat minima, since the mutual information between any solution and the random labels is predictably high.

The sharpness based on the magnitude of the eigenvalues of the Hessian is too expensive for large DNNs. [START_REF] Shirish Keskar | On large-batch training for deep learning: Generalization gap and sharp minima[END_REF] propose an alternative definition based on the sensitivity wrt weights. The definition is simplified by [START_REF] Dinh | Sharp minima can generalize for deep nets[END_REF]:

max w ∈B(w, ) R(f w ) -R(f w ) 1 + R(f w ) max w ∈B(w, )
R(f w ) -R(f w ), (4.5) where B(w, ) is an Euclidean ball centered at the minimum w with radius . The similar equal holds since R(f w ) is near zero at the minimum w. However, both definitions of sharpness are sensitive to reparameterizations, as pointed out by [START_REF] Dinh | Sharp minima can generalize for deep nets[END_REF]. One can reparameterize the neural networks without changing the inputs and outputs while making the sharp minima to flat minima. Thus, sharpness alone cannot be a local measure to predict generalization.

Denote by R(f ) the risk with respect to the scoring function f . Specifying q(w) = N (µ, σ 2 I), with probability 1 -δ over the training set, the PAC-Bayes generalization bound in Theorem 4.5.2 can be rewriten as

E q(w) [R(f w )] ≤ E q(w) [ R(f w )] + O D KL (q π) n , (4.6) ≤ R(f µ ) + max w∈B(µ,σ) [ R(f w )] -R(f µ ) sharpness +O D KL (q π) n . (4.7)
This result was first observed by [START_REF] Behnam Neyshabur | Exploring generalization in deep learning[END_REF]. They also verified empirically that the generalization depends on both the sharpness and the KL divergence between the posterior q(w) and the prior π(w).

MDL: a lossless-compression-induced supervised learning framework

The minimum description length (MDL) principle [START_REF] Rissanen | Modeling by shortest data description[END_REF]) is a model selection framework. It is based on the insights that the goal of statistical inference may be cast as trying to find regularity in the data, and the regularity may be identified as the ability to compress [START_REF] Grunwald | A tutorial introduction to the minimum description length principle[END_REF]. MDL is related to the Occam's razor. The latter only suggests that among all the hypotheses that are compatible with the evidence, the best hypothesis is the simplest one. MDL embodies Occam's razor by viewing machine learning as lossless data compression, where the objective is a trade-off between the data fitting (i.e. the compatibility with the evidence) and the model complexity. This is also known as the two-part codes in the literature. [START_REF] Hinton | Keeping the neural networks simple by minimizing the description length of the weights[END_REF] utilizes the MDL as a bridge to explicitly link the regularizations of DNN training with the lossless data compression. They consider a communication game associated with the supervised learning framework. Suppose that a sender observes both the inputs x := {x i } n i=1 and the corresponding labels y := {y i } n i=1 , while a receiver only observes the inputs. Denote by D := (x, y). A further required assumption is that the data are iid sampled. In order to send y, rather than sending the raw data, they communicate in terms of a decoder (i.e. the model), which is known a priori by both sides with the form of p(y|x, w), where w is the message (i.e. the parameter). Note that x, y, w are assumed to be integer-valued for the sake of simplicity in the communication game. The steps of the game are shown as follows.

1. The sender encodes D to a message ŵ and compress ŵ losslessly to a bitstream using entropy coding (e.g. Huffman coding) according to a prior distribution p(w), which is agreed a priori by both sides.

2. The sender transmits the bitstream of ŵ to the receiver. The code length is log p( ŵ).

3. The receiver decodes the bitstream and recovers ŵ given access to p(w). The receiver now obtain the model p(y|x, ŵ).

4. The sender transmits every y i via a lossless compression using entropy coding according to p(y i |x i , ŵ). Each has a code lengthlog p(y i |x i , ŵ).

5. The receiver reconstructs y i with the decoder p(y i |x i , ŵ).

The only issue now is what is a good criterion for encoding y in the first step? From a data transmission viewpoint, a good criterion should be linked with the communication cost. That is, the code length should be as short as possible. This idea leads to an optimization of the form min w -n i=1 log p(y i |x i , w) -log p(w), (4.8) which computes the optimal code length for the communication game. By defining p(w), for example, as N (0, λ 2 I), we obtain exactly the 2 regularized maximum loglikelihood estimation with the regularization coefficient λ > 0, which encourages w to take values in the 2 ball. The argmin according to eq.( 4.8), namely ŵ, is also known as the maximum a posterior estimate in statistics. Thus, we may call the above communication game MAP coding.

Alternatively, instead of picking the MAP code ŵ, we may sample w from an auxilary distribution q(w|D), which is constructed by the sender given D. Let w denote the sample drawn from q(w|D), which carries auxilary information to be transmitted between the sender and the receiver. According to the MAP coding, the code length would be

- n i=1 log p(y i |x i , w) -log p( w).
(4.9)

Having sent y, the receiver now has all the ingredients to recover q(w|D) by the same algorithm used by the sender. Then, the receiver can decode the auxilary message according to q(w|D). If the transmission of the auxilary message was conducted separately, the extra costlog q( w|D) has to be paid. However, since the main message and the auxilary message are sent via a single communication, we have savedlog q( w|D) bits back. In other words, the actual communication cost in total is

- n i=1
log p(y i |x i , w) -log p( w) + log q( w|D). (4.10)

Now, taking into account the sampling, the only way to optimize the total communication cost in average is to manipulate the auxilary distribution q(w|D), such that the expected eq.( 4.10) is minimized. In other words, we choose q(w|D) to be argmin q -n i=1 E q(w) [log p(y i |x i , w)] + D KL (q(w) p(w)) . (4.11) This version of the communication game is referred to as bits-back coding, which was originally proposed by [START_REF] Hinton | Keeping the neural networks simple by minimizing the description length of the weights[END_REF]. Comparing to the MAP coding, the bits-back coding in average saves H q(w|D) bits. Note that the "bits-back" argument is just a way to motivate that sending a distribution of the weights is much more efficient than sending a point estimate in terms of the communication cost. The auxilary task does not necessarily exist as a real task.

The auxilary distribution q(w|D) in fact have different names in different contexts.

In Bayesian inference, it is called the variational posterior, while in PAC-Bayesian theory, it is called the Gibbs predictor.

Intuitively, the analogy between lossless data transmission and supervised learning is based on the insight that the data encoder and the supervised learner are both mechanisms to abstract information. If the data transmission favors simple and precise encoding, so does the supervised learning. Thus, MDL is a good principle to align compression with generalization guiding the learning towards low complexity solutions. As can be seen from ( 4.11), very noisy weights can be communicated very cheaply, but they also cause extra variance in the data fitting term, making it more expensive to communicate.

The objective function in eq.( 4.11) is variously known as the free energy in thermodynamics or the negative evidence lower bound (ELBO) in variational inference. In the context of approximate Bayesian inference [START_REF] Graves | Practical variational inference for neural networks[END_REF][START_REF] Blundell | Weight uncertainty in neural networks[END_REF], it can be derived from

θ * = arg min θ D KL q(w|θ) p(w|D) (4.12) = arg min θ dwq(w|θ) log q(w|θ) p(w)p(D|w) (4.13) = arg min θ -E q(w|θ) [log p(D|w)] + D KL (q(w|θ) p(w)) (4.14)
In practice, the data fitting term is intractable due to the expectation over w, which is a high-dimensional vector. So, as a solution, Monte Carlo integration can be applied, but the optimization will suffer from high-variance gradients. Alternatively, if q(w|θ) is parameterized as a distribution in the location-scale family, the reparameterization trick (Kingma and Welling 2013) can be used. It employs a change of variables: w = f (φ, ), where φ is the variational parameter; ∼ N (0, I) is a noise variable. Then, the first term in eq.( 4.14) can be rewriten as -E p( ) [log p(D|f (φ, ))]. The same reparameterization trick can be applied to the KL term in eq. (4.14). But that is not necessary, as KL divergence sometimes has closed form (e.g. for Gaussians). By applying this trick, we obtain unbiased stochastic gradients wrt the variational parameter φ, transforming it to an optimization problem that is solvable by stochastic gradient descent (SGD).

Variational Bayes with sparsity inducing prior explicitly align model compression with generalization. By imposing sparsity inducing priors, such the mixture of Gaussians (Ullrich et al. 2017a) and continuous relaxations of the spike-and-slab distribution [START_REF] London | The benefits of learning with strongly convex approximate inference[END_REF], the learned weights are easier to be pruned or quantized afterwards, since they are explicitly guided towards sparse or clustered structures while targeting on good performance on the training data. In practice, even with a standard Gaussian prior, variational Bayes provides a natural way to prune weights by examining the signal-to-noise-ratio (SNR) |µ| σ [START_REF] Graves | Practical variational inference for neural networks[END_REF][START_REF] Blundell | Weight uncertainty in neural networks[END_REF].

So far we have reasoned the relationship between compression and generalization through the MDL intuition with the noisy weights. There is in fact a more straightforward justification from the PAC-Bayesian learning theory. Moreover, one can easily identify the connection between MDL and PAC-Bayesian bounds.

Let us first introduce some notations. The risk associated with a scoring function f w (x, y) and a posterior q(w) is given by R(q, f ) = E w∼q(w) E (x,y)∼p * (x,y) [f w (x, y)], (4.15) where p * is the underlying data distribution, which is unknown and independent of any model. Let (x, y) ∈ (X × Y) n be an iid sample of size n drawn from the data distribution. The empirical risk is defined by

R(q, f ) = E w∼q(w) 1 n n i=1 [f w (y i , x i )]. (4.16)
The PAC Bayesian theory [START_REF] David | Pac-bayesian stochastic model selection[END_REF] provides a probably approximately correct bound on the generalization error, which is a data-driven bound that are computed on the sample (x, y), and holds uniformly over all encoder q(w) which is a valid posterior distribution.

Theorem 4.5. 2 (McAllester (2003)). For a sample (x, y) of size n drawn iid from the data distribution, with probability at least 1 -δ over the choice of (x, y), we have, for all valid posterior q(w), scoring function f w (x, y) and prior π(w),

R(q, f ) ≤ R(q, f ) + D KL (q π) -log δ + log n + 2 2n -1 . (4.17)
The quality of the generalization bound depends on the number of data points and the KL divergence between q(w) and π(w). Moreover, it is easy to see that the right hand side of eq.( 4.17) resembles the objective function of variational Bayes, if we consider a specical case of the scoring function: x,y ) .

f w (x, y) = -log p(y|x, w) -log y e -fw(
(4.18)

In fact, [START_REF] Germain | Pacbayesian theory meets bayesian inference[END_REF] has identified that minimizing the PAC-Bayesian bound is equivalent to maximizing the Bayesian marginal likelihood.

In order to directly relate generalization error with model compression, one could define a prior reflecting the model size. For example, [START_REF] Zhou | Compressibility and generalization in large-scale deep learning[END_REF] considered a prior

π(w) = 1 Z u(|w|)2 -|w| with Z = w u(|w|)2 -|w| (4.19)
based on the model size |w|, where u(•) is the uniform distribution, and a posterior q(w) being the delta distribution with a point mass at w. The KL divergence is then upper bounded by

D KL (q π) ≤ |w| log 2 -log(u(|w|)) (4.20)
due to the fact that Z ≤ 1. By plugging eq.( 4.20) into Theorem 4.5.2, we see clearly that the generalization bound depends on the model size.

Information bottleneck: a lossy-compressioninduced supervised learning framework

MDL is an analogy between lossless data compression and supervised learning. However, since the data usually contains noise, it is more appropriate to use a lossy data compression scheme. [START_REF] Tishby | Deep learning and the information bottleneck principle[END_REF] take a first attempt on interpreting the DNN training with SGD optimizer in terms of a lossy data compression scheme-the information bottleneck methods. It becomes one of the hottest topics in theoretical deep learning, although it also brought a controversial discussion [START_REF] Michael | Unsupervised pre-training across image domains improves lung tissue classification[END_REF]).

Information bottleneck

The information bottleneck method [START_REF] Tishby | The information bottleneck method[END_REF] extends the rate-distortion framework for supervised learning or other problems involving finding the best tradeoff between preserving the relevance wrt a random variable Y and removing the irrelevance wrt another random variable X. This is done by searching a "bottleneck", such that the information that X contains about Y is squeezed through the bottleneck. The key underlying assumption is that Y must not be independent of X, namely, I(X; Y ) > 0. In the case of supervised learning, it is common that Y is a deterministic function of X. Thus,

I(X; Y ) = H(Y ) -H(Y |X) = H(Y ) > 0.
Specifically, the goal is to create a summary of X, the bottleneck, denoted by X, such that it forms a graphical model X ← X ← Y , and the distortion is defined in terms of X and Y . The proposed objective leads to an abstract optimization problem min I( X; X) -βI( X; Y ), (4.21) which is a generalization of the standard rate-distortion tradeoff. I( X; Y ) can be seen as a "correct" measure of the distortion for this setting. To see this, we rewrite Comparing to (1.51), we see that

-I( X; Y ) = -E p(x,
D(p(x|x), m(x), p(y|x)) = E p(x|x)m(x) D KL p(y|x) p(y|x) (4.27)
with an extended parameter p(y|x) ∈ P y , where P y is the set of valid p(y|x).

To solve (4.26), assuming that we are access to p(x) and p(y|x), a Blahut-Arimoto-like algorithm is proposed by [START_REF] Tishby | The information bottleneck method[END_REF].

Towards Understanding Generalization via Compression 109

Proposition 4.5.3 [START_REF] Tishby | The information bottleneck method[END_REF], Theorem 5). The minimization in (4.26) is performed by the convergent alternating iterations. Denoting by t the iteration step, we have

p t+1 (x|x) = m t (x) exp -βD(p t (x|x), m t (x), p t (y|x)) x m t (x ) exp -βD(p t (x |x), m t (x ), p t (y|x )) (4.28) m t+1 (x) = x p(x)p t+1 (x|x) (4.29) p t+1 (y|x) = x p(y|x)p(x) p t+1 (x|x) m t+1 (x) (4.30)
Note that the above alternating minimization algorithm is not in general a realistic algorithm, since we do not access to either p(x) or p(y|x) in most problems, such as the real supervised learning problems. However, the rate distortion theory and information bottleneck still provide an interesting new way to understand the generalization in machine learning. The neural network is a CNN with ReLU activations and 5 hidden layers, which is trained on the CIFAR-10 dataset with SGD for 10 4 epochs. Each trajectory is associated with a hidden layer, and is represented by a sequence of points calculated at each epoch. The color is chosen to be directly linked to epoch. There is a clear transition between two phases, which can be seen for all trajectories. Due to the data processing inequality, lower layers always have higher mutual information. Source: the figure is taken from the comment of Naftali Tishby in the discussion at https://openreview.net/forum?id=ry_WPG-A-. [START_REF] Tishby | Deep learning and the information bottleneck principle[END_REF] propose a hypothesis to explain the phenomenon that overfitting is not observed in highly over-parameterized neural networks: the training of deep neural networks is implicitly regularized by SGD, which biases the learned representation of the data towards minimal sufficient statistics. This phenomenon is verified in a small-scale experiment, where they observe that there exists a two-phase transition during the learning. Denote by X, Y, T the input, the output and the representation respectively. In the first phase, the SGD has a fast convergence rate which is related to gain a sufficiency of the representation, namely, achieving high I(T ; Y ). Later, the SGD enters an asymptotic phase which is related to the compression of the representation, namely, achieving small I(T ; X). An example of the evolving dynamics for all layers is shown in Figure 5. Thus, the conjecture is that SGD implicitly optimizes an objective which works similarly as that of the information bottleneck. Recent works on interpreting SGD as variational inference by [START_REF] Chaudhari | Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks[END_REF], [START_REF] Mandt | Stochastic gradient descent as approximate bayesian inference[END_REF] share some interesting thoughts related to this conjecture.

Information bottleneck insights on generalization of DNNs

Other than focusing on the noise introduced by SGD, Achille and Soatto ( 2018), [START_REF] Dai | Compressing neural networks using the variational information bottleneck[END_REF] relate information bottleneck with dropout [START_REF] Hinton | Improving neural networks by preventing co-adaptation of feature detectors[END_REF] through the local reparameterization trick [START_REF] Diederik P Kingma | Variational dropout and the local reparameterization trick[END_REF]. Specifically, denote by Z i the activation of the ith layer, the information bottleneck principle reads as min L i=1

β i I(Z i ; Z i-1 ) -I(Z i ; Y ), (4.31)
where β i >= 0 is the coefficient to balance the minimality and sufficiency. Due to the intractable mutual information, a variational lower bound of the mutual information is used, leading to the following surrogate minimization:

min L {p(z i |z i-1 )} L i=1 , {q(z i )} L i=1 , q(y|z L ) (4.32) L := L i=1 β i E z i-1 ∼p(z i-1 ) [D KL (p(z i |z i-1 ) q(z i ))] -E (x,y)∼p * (x,y) E z L ∼p(z L |x) [log q(y|z L )],
where {q(z i )} i , q(y|z L ) are variational distributions; p(z i |z i-1 ) comes from the decomposition of the joint distribution p(z i , z i-1 ) = p(z i |z i-1 )p(z i-1 ). As a specical case,

z 0 = x. So, p(z L |x) = z\z L i p(z i |z i-1
). It can be shown that if we specify p(z i |z i-1 ) as a conditional Gaussian via

z i = (z i-1 i ) • w i with i ∼ N (0, I), (4.33)
which is known as the Gaussian dropout, we obtain an equivalent way to inject Gaussian noise to the weights. It is interesting to point out that, by maximizing the lower bound of the information bottleneck, i.e., (4.32), also known as the variational information bottleneck (Alemi et al. 2016), we are in fact doing exactly the same thing that MDL and variational Bayes are asking to. The equivalence is shown by the local reparameterization trick, that is, instead of introducing noise to the weights, it is more efficient to propagate noise to activations directly. This will also reduces the variance of the stochastic gradients [START_REF] Diederik P Kingma | Variational dropout and the local reparameterization trick[END_REF]. We have seen that, empirically, the combination of SGD and over-parameterized DNNs is so far the fastest and most reliable way to achieve good performance in various machine learning tasks. On the other hand, we have seen in Section 4.4 that model compression methods, such as pruning and quantization, are applied either in an end-to-end fashion or by a post-processing scheme, which do not degrade the original models. However, model compression methods are unavoidably limited by the initial network architectures, which are not designed for energy efficient scenarios. As a result, compressed networks often cannot be tailored to fit mobile and embedded devices. Moreover, compression does not necessarily reduce the running time memory even the networks can be stored more efficiently. Another big issue is that over-parameterized models are often trained with huge data. Many tasks in real life are not able to be solved in that way, since gathering sufficiently large data is not always possible. Now, a natural question is can we make use of the nice properties of the over-parameterization in deep learning if we only have limited data?

Transfer learning [START_REF] Pletscher | Entropy and margin maximization for structured output learning[END_REF][START_REF] Caruana | Learning many related tasks at the same time with backpropagation[END_REF]) is targeted on this problem focusing on recognizing and applying relevant knowledge learned from previous tasks to new tasks. There are many methods can be used for transfer learning. The common approach is called finetuning, which amounts to training a source network on the source task and then copy the weights of the source network to initialize the target network. It is noticed that the top layers are more specific to the task itself while the lower layers are more generic [START_REF] Yosinski | How transferable are features in deep neural networks? In Advances in neural information processing systems[END_REF]. So, we usually froze low layers while finetuning, meaning that they do not change during the learning on the target task. A slightly more sophisticated idea is to regularize the distance between the weights of the source network and the weights of the target network. This idea has been successfully used by passive-aggressive algorithm for online learning [START_REF] Crammer | Online passive-aggressive algorithms[END_REF], adaptive SVM for computer vision (Yang et al. 2007, Aytar and[START_REF] Aytar | Tabula rasa: Model transfer for object category detection[END_REF], and more recently, elastic weight consolidation for overcoming catastrophic forgetting in continual learning [START_REF] Kirkpatrick | Overcoming catastrophic forgetting in neural networks[END_REF].

It turns out that both model tailoring and transfer learning can be cast as a knowledge transfer problem. In the former case, we transfer knowledge between two DNNs using the same dataset. In a slightly different scenario, namely, transfer learning, we perform the same transfer but acrossing two similar datasets.

The teacher-student framework emerged as a solution to the knowledge transfer problems, which was firstly proposed by [START_REF] Ba | Bach. Breaking the curse of dimensionality with convex neural networks[END_REF] as a model compression method. Within the same dataset, the framework is also known as knowledge distillation due to [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF]. [START_REF] Ba | Bach. Breaking the curse of dimensionality with convex neural networks[END_REF] focus on the regularization aspect of knowledge distillation. They argue that a shallow neural network can be as good as a deep neural network with the same number of parameters. Although shallow neural networks are more difficult to train, with the distillation loss as a regularization, they perform equally well. Hinton et al. (2015) pay more attention on distilling knowledge from an ensemble of models (e.g. by dropout) into a single model as well as improving the training of an ensemble of specialist neural networks.

Concretely, we aim to train a student network (usually smaller) to mimic a pretrained teacher network (usually over-parameterized and trained with a sufficiently large dataset). The framework was originally implemented by matching the predictions of the two networks. For a classification problem, this means matching the final outputs (i.e. logits) of the two networks either via 2 distance on the log scale [START_REF] Ba | Bach. Breaking the curse of dimensionality with convex neural networks[END_REF] or by the cross entropy loss [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF]. Recently, several different implementations of the teacher-student framework have been proposed for convolutional neural networks. Moreover, rather than matching only the logits or the softmax layers, it has been shown that knowledge distillation can be applied to intermediate layers [START_REF] Romero | Fitnets: Hints for thin deep nets[END_REF]. Since the activation dimensions are mismatched between the teacher and the student, Zagoruyko and Komodakis (2016b) propose to extract statistics and then match those statistics. For example, matching the averages of activation maps can be interpreted as matching attention maps. Similarly, [START_REF] Romero | Fitnets: Hints for thin deep nets[END_REF] solve the mismatch issue by performing a linear regression on the student activation to match the teacher activation. An illustrative comparison between [START_REF] Romero | Fitnets: Hints for thin deep nets[END_REF] and Zagoruyko and Komodakis (2016b) is shown in Figure 6 and 7.

Conclusion

In this chapter, we have reviewed the over-parameterization design and its implications in deep learning. In particular, we have shown that over-parameterized DNNs can be compressed without significant performance drop, and have discussed the relationship between model generalization and compression. Finally, we have mentioned the teacher-student framework as a general framework to distil knowledge from over-parameterized DNNs. The graphical model of Bayesian supervised learning. Given a dataset, the supervised learning can be viewed as an autoencoding of the labels, where w is the shared latent variable.

Supervised learning via lossy compression

Given a sample S := X × Y with inputs X := {(x i )} n i=1 and labels Y := {(y i )} n i=1 , consider a sender who would like to send Y to a receiver. Instead of transmitting the raw labels, the sender may compress the data with an encoder q(w|S) and send the encoder as well as the inputs. The receiver can then reconstruct the labels by combining the encoder and a predefined decoder p(y | x, w):

ŷi ∼ q(y | x i , S) := p(y | x i , w)q(w|S)dw, ( 5.1)
where ŷi the reconstructed label for input x i . Note that q(y | x i , S) resembles the predictive distribution in the Bayesian terminology, whose graphical model is shown in Figure 1. This data-transmission view motivates a new learning objective for Bayesian models based on information theory. The Bayesian inference can be seen as a special case where we use the true posterior p(w|S) as the encoder. However, the encoder q(w|S) is not necessarily equal to the posterior p(w|S). It comes from an empirical view of the joint distribution p(S, w):

p(S, w) = q(w|S)p * (S) with p * (S) = n i=1 p * (x i , y i ), (5.2)
where we denote the "true" distribution of data as p * (x, y), which is unknown and model independent. The data-transmission analogy of Bayesian inference offers us a direct way to relate data compression with model generalization. To see this, we analyze the limits of the encoder q(w|S). If we force w to memorize everything in S including the noise, e.g. to have an identity map, we have to pay a high price for the communication cost. However, the encoder may not be even useful for another sample since it is too specific to a particular sample S. On the other hand, if w is independent of S, then the receiver has no way to decode the message no matter how powerful the decoder is.

The trade-off between the compression and the reconstruction can be formulated as a rate-distortion trade-off (Cover and Thomas 2012). The compression rate is measured by the mutual information I(w; S) 1 . The distortion function, denoted by 5.1 Supervised learning via lossy compression 115 d(w, S), measures the quality of the reconstruction, can be naturally defined as the negative log-likelihood. This argument is in line with the Minimum Description Length principle under lossless compression, which says that the best model among all equally good models is the one that leads to the best compression of the data. However, the proposed learning formulation is based on lossy compression. Specifically, the rate-distortion trade-off is expressed as the following optimization problem:

min q(w|•)∈Q I(w; S) ≡ I(q(w|S)) s.t. E p * (S) E q(w|S) d(w, S) ≤ D (5.3)
I(q(w|S)) := E p * (S) E q(w|S) log q(w|S) q(w) , (5.4)

d(w, S) := - n i=1 log p(y i |x i , w) = -log p(S|w) + constant, (5.5)
where Q is the set of properly normalized pdfs. A similar rate-distortion analysis for unsupervised representation learning has been conducted by Alemi et al. (2017). Note that q(w) = S p * (S)q(w|S) is the aggregated posterior (Makhzani et al. 2015, Tomczak andWelling 2017). This coupling term renders optimization difficult. We show in the following lemma how to convert eq.( 5.3) to an equivalent but more convenient problem. By applying Lemma 5.1.1 to eq.( 5.3), we rewrite the rate-distortion trade-off as:

min m(w) min q(w|•)∈Q I(q(w|S), m(w)) s.t. E p * (S) E q(w|S) d(w, S) ≤ D (5.6) I(q(w|S), m(w)) := E p * (S) E q(w|S) log q(w|S) m(w) .
(5.7)

Using the Lagrange dual function we convert eq. 5.6 into an unconstrained optimization problem:

F (β) := min m(w) min q(w|•)∈Q I(q(w|S), m(w)) + β E p * (S) E q(w|S) d(w, S) -D .
(5.8)

Interestingly, we can see a connection with the maximum entropy principle (Jaynes 1957), where m(w) term relates to the base measure of the entropy of q(w). minimize I(w; S), since the decoder p(y|x, w) only depends on x and w, and I( Ŝ, S) ≤ I(w; S) by data processing inequality.

One may also see the connection to the empirical Bayes (Robbins 1985, Kucukelbir and[START_REF] Kucukelbir | Population empirical bayes[END_REF], since eq.( 5.6) involves optimizing the "posterior" q(w|S) and the "prior" m(w) at the same time. This perspective links information theory and (empirical) Bayes at a model level rather than at an inference level. It is worth mentioning that [START_REF] Achille | Emergence of invariance and disentangling in deep representations[END_REF] was the first arriving the objective in eq.( 5.3). However, their derivation was quite different and did not use the aforementioned variational upper bound for the mutual information.

Approximate Blahut-Arimoto Algorithm

The direct optimization of eq.( 5.8) is cumbersome, since we need to parameterize a high dimensional mapping q(w|S). Hence, we resort to the classical Blahut-Arimoto algorithm [START_REF] Arimoto | An algorithm for computing the capacity of arbitrary discrete memoryless channels[END_REF][START_REF] Blahut | Computation of channel capacity and rate-distortion functions[END_REF] for computing F (β), which is an alternating minimization using the following fixed point equations:

q(w|S) = m(w) exp(-β d(w, S)) m(v) exp(-β d(v, S))dv (5.9) m(w) = S p * (S)q(w|S). (5.10)
The iterative process defined by the Blahut-Arimoto algorithm optimizes q(w|S) and m(w) at the same time which, in theory, can lead to edge cases where m(w) "chases" q(w|S). However, in practice, this approach has proven to be effective, as shown in the empirical Bayes literature (Robbins 1985, Kucukelbir and[START_REF] Kucukelbir | Population empirical bayes[END_REF] which is relevant due to our treatment of m(w). In section 3, we will show these practical benefits in our scenario. As future work, we plan to address these theoretical questions exploiting the new uncovered links between distortion-rate theory and empirical Bayes.

We now continue with our inference method by making two approximations to the above equations:

1. We use a variational approximation q(w|θ) for q(w|S) by solving

θ(S) = arg min θ D KL (q(w|θ) q(w|S)) (5.11) = arg min θ D KL (q(w|θ) m(w)) + β E q(w|θ) d(w, S) .
(5.12)

Following [START_REF] Blundell | Weight uncertainty in neural networks[END_REF], we parameterize q(w|θ) as a Gaussian distribution.

2. We approximate m(w) S p * (S)q(w|θ(S))

1 K K k=1 q(w|θ(B k )) =: m(w), where B k is a bootstrap sample of size n b drawn from the empirical distribu- tion p S (x, y) = 1 n n i=1 δ(x i = x)δ(y i = y). Note that we only resample data to create B k if n b > n.
We call the resulting approach β-BNN. The detailed algorithm is shown in Algorithm 3. Note that the step for updating q(w|θ) resembles the ELBO derived by [START_REF] Blundell | Weight uncertainty in neural networks[END_REF] for vanilla BNNs, except that the coefficient β is now formally introduced, and instead of setting m(w) to be N (0, I), m(w) is approximated by a mixture of variational posteriors. It is clear that n b and K determine how close we follow the classical Blahut-Arimoto steps. However, we do not want to take very large n b and K, since both steps are approximated. Inspired by this argument, we also consider an online version, where m(w) is updated whenever a new variational posterior is produced.

Algorithm 3 Approximate Blahut-Arimoto Algorithm 1: Input: S (dataset), β (coefficient), K (# mixture components), n b (size of a bootstrap sample). 2: Initialize: Θ = {θ (0) k = (0, I)} K k=1 ; m(w) = 1 K θ∈Θ q(w|θ). 3: for all t = 1, . . . , T do 4:
Draw K bootstrap samples {B k } K k=1 of size n b from p S (x, y).

5:

for all k = 1, . . . , K do 6:

θ (t)
k ← n b SGD steps on the loss of (5.12) initialized at θ (t-1) k

.

7: end for 12: end for 13: Output: Θ.

Θ = Θ ∪ {θ (t) k } \ {θ (t-1) k }. 8: if do online update or k = K then 9: m(w) = 1 K θ∈Θ q(w|θ).

Experiments

We test β-BNN, online β-BNN and vanilla BNN [START_REF] Blundell | Weight uncertainty in neural networks[END_REF] on the colorful MNIST dataset (Bulten 2017), where each image is converted to RGB space and blended with a random background. We also test a fixed-prior β-BNN, which is a special case: m(w) ≡ N (0, I).

For this experiment, T = 100 is sufficient to converge; q(w|θ) and m(w) are specified in Section 5.2; p(y|x, w) is implemented by a multilayer perceptron (Linear400-ReLU-Linear400-ReLU-Linear10-Softmax); θ

(t)
k is obtained by running SGD over B k once with batch size 128 and learning rate 10 -3 . The comparison is shown in Table 1, where the prediction is arg max y p(y|x, E[w]). We allocate 10000 points as the validation set, and choose β -1 from 10 candidates ranging uniformly from 10 -11 to 10 -2 . We set K = 5 for the comparison as the performance only increases marginally for K ≥ 5.

We empirically observe that fixed-prior BNNs suffer from slow convergence due to very large KL terms (about 10 6 ). Thus, we need to choose very small β (about 10 -9 ) to compensate, which will not be scalable for much larger networks. The convergence comparison is shown in Figure 2, where we can see that online β-BNN converges to a better local minimum and enjoys a faster convergence.

Discussion

The proposed algorithm is in fact quite similar to the training algorithm of vanilla BNNs, except that we update the "prior" every few iterations. We however did not fully take advantage of the stochasticity in eq. (5.8), which allows us to explicitly take the generalization into account. Consider two iid samples S and T drawn from the data distribution p * (S). We may take the following stochastic objective E q(w|θ(S)) log q(w|θ(S)) m(w) + β/2 E q(w|θ(S)) d(w, S) + E q(w|θ(S)) d(w, T ) (5.13) as the loss function for each iteration, that is, we use S to update q(w|•), attaining θ(S), but take into account the performance of another sample T . We leave this extention to our future work.

Introduction

While supervised learning of deep neural networks can achieve or even surpass human-level performance [START_REF] He | Delving deep into rectifiers: Surpassing human-level performance on imagenet classification[END_REF][START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF], they can hardly extrapolate the learned knowledge beyond the domain where the supervision is provided. The problem of solving rapidly a new task after learning several other similar tasks is called meta-learning [START_REF] Schmidhuber | Evolutionary principles in self-referential learning[END_REF][START_REF] Bengio | Learning a synaptic learning rule[END_REF][START_REF] Thrun | Learning to learn: Introduction and overview[END_REF]; typically, the data is presented in a two-level hierarchy such that each data point at the higher level is itself a dataset associated with a task, and the goal is to learn a meta-model that generalizes across tasks. In this paper, we mainly focus on few-shot learning [START_REF] Zhang | Understanding deep learning requires rethinking generalization[END_REF], an instance of meta-learning problems, where a task t consists of a query set d t := {(x t,i , y t,i )} n i=1 serving as the test-set of the task and a support set d l t :={(x l t,i ,y l t,i )} n l i=1 serving as the train-set. In meta-testing1 , one is given the support set and the inputs of the query set x t := {x t,i } n i=1 , and asked to predict the labels y t := {y t,i } n i=1 . In meta-training, y t is provided as the ground truth. The setup of few-shot learning is summarized in Table 1.

Support set

Query set

d l t := {(x l t,i , y l t,i )} n l i=1 x t := {x t,i } n i=1 y t = {y t,i } n i=1

Meta-training Meta-testing

Table 1: The setup of few-shot learning. If task t is used for meta-testing, y t is not given to the model.

A important distinction to make is whether a task is solved in a transductive or inductive manner, that is, whether x t is used. The inductive setting is what was originally proposed by [START_REF] Zhang | Understanding deep learning requires rethinking generalization[END_REF], in which only d l t is used to generate a model. The transductive setting, as an alternative, has the advantage of being able to see partial or all points in x t before making predictions. In fact, [START_REF] Nichol | On first-order meta-learning algorithms[END_REF] notice that most of the existing meta-learning methods involve transduction unintentionally since they use x t implicitly via the batch normalization (Ioffe and Szegedy 2015b). Explicit transduction is less explored in meta-learning, the exception is [START_REF] Liu | Learning to propagate labels: Transductive propagation network for few-shot learning[END_REF], who adapted the idea of label propagation [START_REF] Zhu | Semi-supervised learning using gaussian fields and harmonic functions[END_REF] from graph-based semi-supervised learning methods. We take a totally different path that meta-learn the "gradient" descent on x t without using y t .

Due to the hierarchical structure of the data, it is natural to formulate metalearning by a hierarchical Bayes (HB) model [START_REF] John | Some history of the hierarchical bayesian methodology[END_REF][START_REF] Berger | Statistical Decision Theory and Bayesian Analysis[END_REF], or alternatively, an empirical Bayes (EB) model (Robbins 1985, Kucukelbir and[START_REF] Kucukelbir | Population empirical bayes[END_REF]. The difference is that the latter restricts the learning of meta-parameters to point estimates. In this paper, we focus on the EB model, as it largely simplifies the training and testing without losing the strength of the HB formulation.

The idea of using HB or EB for meta-learning is not new: Amit and Meir (2018) derive an objective similar to that of HB using PAC-Bayesian analysis; [START_REF] Grant | Recasting gradient-based meta-learning as hierarchical bayes[END_REF] show that MAML [START_REF] Finn | Model-agnostic meta-learning for fast adaptation of deep networks[END_REF]) can be understood as a EB method; [START_REF] Ravi | Amortized bayesian meta-learning[END_REF] consider a HB extension to MAML and compute posteriors via amortized variational inference. However, unlike our proposal, these methods do not make full use of the unlabeled data in query set. Roughly speaking, they construct the variational posterior as a function of the labeled set d l t without taking advantage of the unlabeled set x t . The situation is similar in gradient based meta-learning methods [START_REF] Finn | Model-agnostic meta-learning for fast adaptation of deep networks[END_REF][START_REF] Ravi | Optimization as a model for few-shot learning[END_REF], Li et al. 2017c[START_REF] Nichol | On first-order meta-learning algorithms[END_REF][START_REF] Flennerhag | Transferring knowledge across learning processes[END_REF]) and many other meta-learning methods [START_REF] Zhang | Understanding deep learning requires rethinking generalization[END_REF][START_REF] Snell | Prototypical networks for few-shot learning[END_REF], Gidaris and Komodakis 2018), where the mechanisms used to generate the task-specific parameters rely on groundtruth labels, thus, there is no place for the unlabeled set to contribute. We argue that this is a suboptimal choice, which may lead to overfitting when the labeled set is small and hinder the possibility of zero-shot learning (when the labeled set is not provided).

In this paper, we propose to use synthetic gradient [START_REF] Jaderberg | Decoupled neural interfaces using synthetic gradients[END_REF]) to enable transduction, such that the variational posterior is implemented as a function of the labeled set d l t and the unlabeled set x t . The synthetic gradient is produced by chaining the output of a gradient network into auto-differentiation, which yields a surrogate of the inaccessible true gradient. The optimization process is similar to the inner gradient descent in MAML, but it iterates on the unlabeled x t rather than on the labeled d l t , since it does not rely on y t to compute the true gradient. The labeled set for generating the model for an unseen task is now optional, which is only used to compute the initialization of model weights in our case. In summary, our main contributions are the following:

1. In section 6.2 and section 6.3, we develop a novel empirical Bayes formulation with transduction for meta-learning. To perform amortized variational inference, we propose a parameterization for the variational posterior based on synthetic gradient descent, which incoporates the contextual information from all the inputs of the query set.

2. In section 6.4, we show in theory that a transductive variational posterior yields better generalization performance. The generalization analysis is done through the connection between empirical Bayes formulation and a multitask extension of the information bottleneck principle. In light of this, we name our method synthetic information bottleneck (SIB). A comparison between MAML eq.( 6.6) and our method (SIB) eq.( 6.10) is shown in (b) and (c). MAML is an inductive method since, for a task t, it first constructs the variational posterior (with parameter θ K ) as a function of the support set d l t , and then test on the unlabeled x t ; while SIB uses a better variational posterior as a function of both d l t and x t : it starts from an initialization θ 0 t (d l t ) generated using d l t , and then yields θ K t by running K synthetic gradient steps on x t .

Meta-learning with transductive inference

The goal of meta-learning is to train a meta-model on a collection of tasks, such that it works well on another disjoint collection of tasks. Suppose that we are given a collection of N tasks for training. The associated data is denoted by D := {d t := (x t , y t )} N t=1 . In the case of few-shot learning, we are given in addition a support set d l t in each task. In this section, we revisit the classical empirical Bayes model for meta-learning. Then, we propose to use a transductive scheme in the variational inference by implementing the variational posterior as a function of x t .

Empirical Bayes model

Due to the hierarchical structure among data, it is natural to consider a hierarchical Bayes model with the marginal likelihood

p f (D) = ψ p f (D|ψ)p(ψ) = ψ N t=1 wt p f (d t |w t )p(w t |ψ) p(ψ). (6.1)
The generative process is illustrated in Figure 1 (a, in red arrows): first, a metaparameter ψ (i.e., hyper-parameter) is sampled from the hyper-prior p(ψ); then, for each task, a task-specific parameter w t is sampled from the prior p(w t |ψ); finally, the dataset is drawn from the likelihood p f (d t |w t ). Without loss of generality, we assume the log-likelihood model factorizes as

log p f (d t |w t ) = n i=1 log p f (y t,i |x t,i , w t ) + log p(x t,i |w t ), = n i=1 - 1 n t ŷt,i (f (x t,i ), w t ), y t,i + constant. (6.2)

Meta-learning with transductive inference 123

It is the setting advocated by [START_REF] Minka | Discriminative models, not discriminative training[END_REF], in which the generative model p(x t,i |w t ) can be safely ignored since it is irrelevant to the prediction of y t . To simplify the presentation, we still keep the notation p f (d t |w t ) for the likelihood of the task t and use t to specify the discriminative model, which is also referred to as the task-specific loss, e.g., the cross entropy loss. The first argument in t is the prediction, denoted by ŷt,i = ŷt,i (f (x t,i ), w t ), which depends on the feature representation f (x t,i ) and the task-specific weight w t . Note that rather than following a fully Bayesian approach, we leave some random variables to be estimated in a frequentist way, e.g., f is a meta-parameter of the likelihood model shared by all tasks, for which we use a point estimate. As such, the posterior inference about these variables will be largely simplified. For the same reason, we derive the empirical Bayes (Robbins 1985, Kucukelbir and[START_REF] Kucukelbir | Population empirical bayes[END_REF] by taking a point estimate on ψ. The marginal likelihood now reads as

p ψ,f (D) = N t=1 wt p f (d t |w t )p ψ (w t ). (6.3)
We highlight the meta-parameters as subscripts of the corresponding distributions to distinguish from random variables. Indeed, we are not the first to formulate meta-learning as empirical Bayes. The overall model formulation is essentially the same as the ones considered by [START_REF] Amit | Meta-learning by adjusting priors based on extended pac-bayes theory[END_REF], [START_REF] Grant | Recasting gradient-based meta-learning as hierarchical bayes[END_REF], [START_REF] Ravi | Amortized bayesian meta-learning[END_REF]. Our contribution lies in the variational inference for empirical Bayes.

Amortized inference with transduction

As in standard probabilistic modeling, we derive an evidence lower bound (ELBO) on the log version of eq.( 6.3) by introducing a variational distribution q θt (w t ) for each task with parameter θ t :

log p ψ,f (D) ≥ N t=1 E wt∼q θ t log p f (d t |w t ) -D KL q θt (w t ) p ψ (w t ) . ( 6.4)
The variational inference amounts to maximizing the ELBO with respect to θ 1 , . . . , θ N , which together with the maximum likelihood estimation of the metaparameters, we have the following optimization problem:

min ψ,f min θ 1 ,...,θ N 1 N N t=1 E wt∼q θ t -log p f (d t |w t ) + D KL q θt (w t ) p ψ (w t ) . (6.5)
However, the optimization in eq.( 6.5), as N increases, becomes more and more expensive in terms of the memory footprint and the computational cost. We therefore wish to bypass this heavy optimization and to take advantage of the fact that individual KL terms indeed share the same structure. To this end, instead of introducing N different variational distributions, we consider a parameterized family of distributions in the form of q φ(•) , which is defined implicitly by a deep neural network φ taking as input either d l t or d l t plus x t , that is, q φ(d l t ) or q φ(d l t ,xt) .

Note that we cannot use entire d t , since we do not have access to y t during meta-testing. This amortization technique was first introduced in the case of variational autoencoders (Kingma and [START_REF] Diederik | Auto-encoding variational bayes[END_REF][START_REF] Jimenez Rezende | Stochastic backpropagation and approximate inference in deep generative models[END_REF], and has been extended to Bayesian inference in the case of neural processes (Garnelo et al. 2018b). Since d l t and x t are disjoint, the inference scheme is inductive for a variational posterior q φ(d l t ) . As an example, MAML [START_REF] Finn | Model-agnostic meta-learning for fast adaptation of deep networks[END_REF]) takes q φ(d l t ) as the Dirac delta distribution, where φ(d l t ) = θ K t , is the K-th iterate of the stochastic gradient descent .6) In this work, we consider a transductive inference scheme with variational posterior q φ(d l t ,xt) . The inference process is shown in Figure 1(a, in green arrows). Replacing each q θt in eq.( 6.5) by q φ(d l t ,xt) , the optimization problem becomes min

θ k+1 t = θ k t + η ∇ θ E wt∼q θ k t log p(d l t |w t ) with θ 0 t = φ, a learnable initialization. ( 6
ψ,f min φ 1 N N t=1 E wt∼q φ(d l t ,x t ) -log p f (d t |w t ) + D KL q φ(d l t ,xt) (w t ) p ψ (w t ) . (6.7)
In a nutshell, the meta-model to be optimized includes the feature network f , the hyper-parameter ψ from the empirical Bayes formulation and the amortization network φ from the variational inference.

Unrolling exact inference with synthetic gradients

It is however non-trivial to design a proper network architecture for φ(d l t , x t ), since d l t and x t are both sets. The strategy adopted by neural processes (Garnelo et al. 2018b) is to aggregate the information from all individual examples via an averaging function. However, as pointed out by Kim et al. (2019), such a strategy tends to underfit x t because the aggregation does not necessarily attain the most relevant information for identifying the task-specific parameter. Extensions, such as attentive neural process (Kim et al. 2019) and set transformer (?), may alleviate this issue but come at a price of O(n 2 ) time complexity. We instead design φ(d l t , x t ) to mimic the exact inference arg min θt D KL (q θt (w t ) p ψ,f (w t |d t )) by parameterizing the optimization process with respect to θ t . More specifically, consider the gradient descent on θ t with step size η:

θ k+1 t = θ k t -η ∇ θt D KL q θ k t (w) p ψ,f (w | d t ) . ( 6.8)
We would like to unroll this optimization dynamics up to the K-th step such that θ K t = φ(d l t , x t ) while make sure that θ K t is a good approximation to the optimum θ t , which consists of parameterizing

∇ θt D KL (q θt (w t ) p ψ,f (w t |d t )).
By doing so, θ K t becomes a function of φ, ψ and x t2 , we therefore realize q φ(d l t ,xt)

as q θ K t . For (a), we opt to either let θ 0 t = λ to be a global data-independent initialization as in MAML [START_REF] Finn | Model-agnostic meta-learning for fast adaptation of deep networks[END_REF] or let θ 0 t = λ(d l t ) with a few supervisions from the support set, where λ can be implemented by a permutation invariant network described in Gidaris and Komodakis (2018). In the second case, the features of the support set will be first averaged in terms of their labels and then scaled by a learnable vector of the same size.

For (b), the fundamental reason that we parameterize the gradient is because we do not have access to y t during the meta-testing phase, although we are able to follow eq.( 6.8) in meta-training to obtain q θ t (w t ) ∝ p f (d t |w t )p ψ (w t ). To make a consistent parameterization in both meta-training and meta-testing, we thus do not touch y t when constructing the variational posterior. Recall that the true gradient decomposes as

∇ θt D KL q θt p ψ,f = E 1 n n i=1 ∂ t (ŷ t,i , y t,i ) ∂ ŷt,i ∂ ŷt,i ∂w t ∂w t (θ t , ) ∂θ t + ∇ θt D KL q θt p ψ (6.9)
under a reparameterization w t = w t (θ t , ) with ∼ p( ), where all the terms can be computed without y t except for ∂ t ∂ ŷt,i . Thus, we introduce a deep neural network ξ(ŷ t,i ) to synthesize it. The idea of synthetic gradients was originally proposed by [START_REF] Jaderberg | Decoupled neural interfaces using synthetic gradients[END_REF] to parallelize the back-propagation. Here, the purpose of ξ(ŷ t,i ) is to update θ t regardless of the groundtruth labels, which is slightly different from its original purpose. Besides, we do not introduce an additional loss between ξ(ŷ t,i ) and ∂ t ∂ ŷt,i since ξ(ŷ t,i ) will be driven by the objective in eq.(6.7). As an intermediate computation, the synthetic gradient is not necessarily a good approximation to the true gradient.

To sum up, we have derived a particular implementation of φ(d l t , x t ) by parameterizing the exact inference update, namely eq.( 6.8), without using the labels of the query set, where the meta-model φ includes an initialization network λ and a synthetic gradient network ξ, such that φ(x t ) = θ K t , the K-th iterate of the following update:

θ k+1 t = θ k t -η E 1 n n i=1 ξ(ŷ t,i ) ∂ ŷt,i ∂w t ∂w t (θ k t , ) ∂θ t + ∇ θt D KL q θ k t p ψ . (6.10)
The overall algorithm is depicted in Algorithm 4. We also make a side-by-side comparison with MAML shown in Figure 1. Rather than viewing eq.( 6.10) as

EB Transductive Meta-Learning with Synthetic Gradients

Classifier forward SGD an optimization process, it may be more precise to think of it as a part of the computation graph created in the forward-propagation. The computation graph of the amortized inference is shown in Figure 2, As an extension, if we were deciding to estimate the feature network f in a Bayesian manner, we would have to compute higher-order gradients as in the case of MAML. This is inpractical from a computational point of view and needs technical simplifications [START_REF] Nichol | On first-order meta-learning algorithms[END_REF]. By introducing a series of synthetic gradient networks in a way similar to [START_REF] Jaderberg | Decoupled neural interfaces using synthetic gradients[END_REF], the computation will be decoupled into computations within each layer, and thus becomes more feasible. We see this as a potential advantage of our method and leave this to our future work3 .

θ 0 SGD SGD θ 1 θ K x t f(x t ) f x l t f(x l t ) init y l t θ f(x) . detach() ŷ grad Classifier backward ∇ θ D KL ξ( ŷ ) ≈ ∂ℓ ∂ ŷ ξ λ Classifier forward ŷ t Loss y t KL ℓ t D KL (q θ K∥p ψ ) + -ELBO

Synthetic gradient module

Generalization analysis of empirical Bayes via the connection to information bottleneck

The learning of empirical Bayes (EB) models follows the frequentist's approach, therefore, we can use frequentist's tool to analyze the model. In this section, we study the generalization ability of the empirical Bayes model through its connection to a variant of the information bottleneck principle Achille and Soatto (2017), [START_REF] Tishby | The information bottleneck method[END_REF].

Generalization analysis of empirical Bayes via the connection to information bottleneck 127

Algorithm 4 Variational inference with synthetic gradients for empirical Bayes 1: Input: the dataset D; the step size η; the number of inner iterations K; pretrained f . 2: Initialize the meta-models ψ, and φ = (λ, ξ). Sample a task t and the associated query set d t (plus optionally the support set d l t ).

5:

Compute the initialization θ 0 t = λ or θ 0 t = λ(d l t ).

6:

for k = 1, . . . , K do 7:
Compute θ k t via eq.( 6.10).

8:

end for 9:

Compute w t = w t (θ K t , ) with ∼ p( ).

10:

Update ψ ← ψ -η ∇ ψ D KL (q θ K t (ψ) p ψ ).
11:

Update φ ← φ -η ∇ φ D KL (q φ(xt) p f • p ψ ).
12:

Optionally, update f ← f + η ∇ f log p f (d t |w t ). 13: end while

Abstract form of empirical Bayes

From eq.( 6.3), we see that the empirical Bayes model implies a simpler joint distribution since log p ψ,f (w 1 , . . . , w N , D) = N t=1 log p f (d t |w t ) + log p ψ (w t ), (6.11) which is equal to the log-density of N iid samples drawn from the joint distribution p(w, d, t) ≡ p ψ,f (w, d, t) = p f (d|w, t)p ψ,f (w)p(t) 4 (6.12) up to a constant if we introduce a random variable to represent the task and assume p(t) is an uniform distribution. We thus see that this joint distribution embodies the generative process of empirical Bayes. Correspondingly, there is another graphical model of the joint distribution characterizes the inference process of the empirical Bayes:

q(w, d, t) ≡ q φ (w, d, t) = q φ (w|d, t)q(d|t)q(t), (6.13) where q φ (w|d, t) is the abstract form of the variational posterior with amortization, includes both the inductive form and the transductive form. The coupling between p(w, d, t) and q(w, d, t) is due to p(t) ≡ q(t) as we only have access to tasks through sampling.

We are interested in the case that the number of tasks N → ∞, such as the few-shot learning paradigm proposed by [START_REF] Zhang | Understanding deep learning requires rethinking generalization[END_REF], in which the objective of eq.( 6.7) can be rewritten in an abstract form of E q(t) E q(d|t) E q(w|d,t) -log p(d|w, t) + D KL q(w|d, t) p(w) . (6.14) In fact, optimizing this objective is the same as optimizing eq.( 6.7) from a stochastic gradient descent point of view.

The learning of empirical Bayes with amortized variational inference can be understood as a variational EM in the sense that the E-step amounts to aligning q(w|d, t) with p(w|d, t) while the M-step amounts to adjusting the likelihood p(d|w, t) and the prior p(w).

Connection to information bottleneck

The following theorem shows the connection between eq.( 6.14) and the information bottleneck principle. Theorem 6.A.1. Given distributions q(w|d, t), q(d|t), q(t), p(w) and p(d|w, t), we have eq.( 6.14) ≥ I q (w; d|t) + H q (d|w, t), (6.20) where I q (w; d|t) := D KL q(w, d|t) q(w|t)q(d|t) is the conditional mutual information and H q (w|d, t) := E q(w,d,t) [-log q(w|d, t)] is the conditional entropy. The equality holds when ∀t : D KL (q(w|t) p(w)) = 0 and D KL (q(d|w, t) p(d|w, t)) = 0.

In fact, the lower bound on eq.( 6.14) is an extention of the information bottleneck principle (Achille and Soatto 2017) under the multi-task setting, which, together with the synthetic gradient based variational posterior, inspire the name synthetic information bottleneck of our method. The tightness of the lower bound depends on both the parameterizations of p f (d|w, t) and p ψ (w) as well as the optimization of eq. (6.14). It thus can be understood as how well we can align the inference process with the generative process. From an inference process point of view, for a given q(w|d, t), the optimal likelihood and prior have been determined. In theory, we only need to find the optimal q(w|d, t) using the information bottleneck in eq.( 6.20). However, in practice, minimizing eq.( 6.14) is more straightforward.

Generalization of empirical Bayes meta-learning

The connection to information bottleneck suggests that we can eliminate p(d|w, t) and p(w) from the generalization analysis of empirical Bayes meta-learning and define the generalization error by q(w, d, t) only. To this end, we first identify the empirical risk for a single task t with respect to particular weights w and dataset d as

L t (w, d) := 1 n n i=1 t (ŷ i (f (x i ), w), y i ). (6.15)
The true risk for task t with respect to w is then the expected empirical risk E d∼q(d|t) L t (w, d). Now, we define the generalization error with respect to q(w, d, t)

Experiments 129

as the average of the difference between the true risk and the empirical risk over all possible t, d, w:

gen(q) := E q(t)q(d|t)q(w|d,t) E d∼q(d|t) L t (w, d) -L t (w, d)
= E q(t)q(d|t)q(w|t) L t (w, d) -E q(t)q(d|t)q(w|d,t) L t (w, d), (6.16) where q(w|t) is the aggregated posterior of task t.

Next, we extend the result from [START_REF] Xu | Information-theoretic analysis of generalization capability of learning algorithms[END_REF] and derive a datadependent upper bound for gen(q) using mutual information. Theorem 6.A.2. Denote by z = (x, y).

If t (ŷ i (f (x i ), w), y i ) ≡ t (w, z i ) is σ-subgaussian under q(w|t)q(z|t), then L t (w, d) is σ/
√ n-subgaussian under q(w|t)q(d|t) due to the iid assumption, and gen(q) ≤ 2σ 2 n I q (w; d|t). (6.31) Plugging this back to Theorem 6.A.1, we obtain a different interpretation for the empirical Bayes ELBO. Corollary 6.4.1. If t is chosen to be the negative log-likelihood, minimizing the population objective of empirical Bayes meta-learning amounts to minimizing both the expected generalization error and the expected empirical risk: eq.( 6.14) ≥ n 2σ 2 gen(q) 2 + E q(t)q(d|t)q(w|d,t) L t (w, d).

(6.17)

The Corollary 6.4.1 suggests that eq.( 6.14) amounts to minimizing a regularized empirical risk minimization. In general, there is a tradeoff between the generalization error and the empirical risk controlled by the coefficient n 2σ 2 , where n = |d| is the cardinality of d. If n is small, then we are in the overfitting regime. This is the case of the inductive inference with variational posterior q(w|d l , t), where the support set d l is fairly small by the definition of few-shot learning. On the other hand, if we were following the transductive setting, we expect to achieve a small generalization error since the implemented variational posterior is a better approximation to q(w|d, t). However, keeping increasing n will potentially over-regularize the model and thus yield negative effect. An empirical study on varying n can be found in Table 6.C.1 in Appendix 6.C.

Experiments

In this section, we first validate our method on few-shot learning, and then on zero-shot learning. Note that many meta-learning methods, such as MAML, cannot do zero-shot learning since they rely on the support set.

Few-shot classification

We compare SIB with state-of-the-art methods on few-shot classification problems. Our code is available at https://github.com/hushell/sib_meta_learn.

Datasets

We choose standard benchmarks of few-shot classification for this experiment.

Evaluation metrics

In few-shot classification, a task (aka episode) t consists of a query set d t and a support set d l t . When we say the task t is k-way-n l -shot we mean that d l t is formed by first sampling k classes from a pool of classes; then, for each sampled class, n l examples are drawn and a new label taken from {0, . . . , k -1} is assigned to these examples. By default, each query set contains 15k examples. The goal of this problem is to predict the labels of the query set, which are provided as ground truth during training. The evaluation is the average classification accuracy over tasks.

Network architectures

Following Gidaris and Komodakis (2018), [START_REF] Qiao | Few-shot image recognition by predicting parameters from activations[END_REF], [START_REF] Gidaris | Boosting few-shot visual learning with self-supervision[END_REF], we implement f by a 4-layer convolutional network (Conv-4-64 or Conv-4-128 5 ) or a WideResNet (WRN-28-10) (Zagoruyko and Komodakis 2016c). We pretrain the feature network f (•) on the 64 training classes for a stardard 64-way classification. We reuse the feature averaging network proposed by Gidaris and Komodakis (2018) as our initialization network λ(•), which basically averages the feature vectors of all data points from the same class and then scales each feature dimension differently by a learned coefficient. For the synthetic gradient network ξ(•), we implement a three-layer MLP with hidden-layer size 8k. Finally, for the predictor ŷij (•, w i ), we adopt the cosine-similarity based classifier advocated by [START_REF] Chen | A closer look at few-shot classification[END_REF] and Gidaris and Komodakis (2018).

Training details

We run SGD with batch size 8 for 40000 steps, where the learning rate is fixed to 10 -3 . During training, we freeze the feature network. To select the best hyperparameters, we sample 1000 tasks from the validation classes and reuse them at each training epoch.

Comparison to state-of-the-art meta-learning methods

In Table 2, we show a comparison between the state-of-the-art approaches and several variants of our method (varying K or f (•)). Apart from SIB, TPN [START_REF] Liu | Learning to propagate labels: Transductive propagation network for few-shot learning[END_REF] and CTM [START_REF] Li | Finding Task-Relevant Features for Few-Shot Learning by Category Traversal[END_REF]) are also transductive methods.

First of all, comparing SIB (K = 3) to SIB (K = 0), we observe a clear improvement, which suggests that, by taking a few synthetic gradient steps, we do obtain a better variational posterior as promised. For 1-shot learning, SIB (when K = 3 or K = 5) significantly outperforms previous methods on both MiniImageNet and CIFAR-FS. For 5-shot learning, the results are also comparable. It should be noted that the performance boost is consistently observed with different feature networks, which suggests that SIB is a general method for few-shot learning.

However, we also observe a potential limitation of SIB: when the support set is relatively large, e.g., 5-shot, with a good feature network like WRN-28-10, the initialization θ 0 t may already be close to some local minimum, making the updates later less important.

For 5-shot learning, SIB is sligtly worse than CTM [START_REF] Li | Finding Task-Relevant Features for Few-Shot Learning by Category Traversal[END_REF]) and/or [START_REF] Gidaris | Boosting few-shot visual learning with self-supervision[END_REF]. CMT [START_REF] Li | Finding Task-Relevant Features for Few-Shot Learning by Category Traversal[END_REF]) can be seen as an alternative way to incorporate transduction -it measures the similarity between a query example and the support set while making use of intra-and inter-class relationships. Gidaris 2019) uses in addition the self-supervision as an auxilary loss to learn a richer and more transferable feature model. Both ideas are complementary to SIB. We leave these extensions to our future work.

Zero-shot regression: spinning lines

Since our variational posterior relies only on x t , SIB is also applicable to zero-shot problems (i.e., no support set available). We first look at a toy multi-task problem, where I(w t ; d t ) is tractable.

Denote by D train := {d t } N t=1 the train set, which consists of datasets of size n: d = {(x i , y i )} n i=1 . We construct a dataset d by firstly sampling iid Gaussian random variables as inputs: x i ∼ N (µ, σ 2 ). Then, we generate the weight for each dataset by calculating the mean of the inputs and shifting with a Gaussian random variable w : w = 1 n i x i + w , w ∼ N (µ w , σ 2 w ). The output for x i is y i = w •x i . We decide ahead of time the hyperparameters µ, σ, µ w , σ w for generating x i and y i . Recall that a weighted sum of iid Gaussian random variables is still a Gaussian random variable. Specifically, if w = i c i x i and

x i ∼ N (µ i , σ 2 i ), then w ∼ N ( i c i µ i , i c 2 i σ 2 i ). Therefore, we have p(w) = N (µ + µ w , 1 n σ 2 + σ 2 w
). On the other hand, if we are given a dataset d of size n, the only uncertainty about w comes from w , that is, we should consider x i as a constant given d. Therefore, the posterior p(w|d

) = N ( 1 n n i=1 x i + µ w , σ 2 w )
. We use a simple implementation for SIB: The variational posterior is realized by

q θ K t (w) = N (θ K t , σ w ), θ k+1 t = θ k t -10 -3 n i=1
x i ξ(θ k t x i ), and θ 0 t = λ ∈ R; (6.18) t is a mean squared error, implies that p(y|x, w) = N (wx, 1); p ψ (w) is a Gaussian distribution with parameters ψ ∈ R 2 ; The synthetic gradient network ξ is a three-layer MLP with hidden size 8. In the experiment, we sample 240 tasks respectively for both D train and D test . We learn SIB and BNN on D train for 150 epochs using the ADAM optimizer (Kingma and Ba 2014), with learning rate 10 -3 and batch size 8. Other hyperparameters are specified as follows: n = 32, K = 3, µ = 0, σ = 1, µ w = 1, σ w = 0.1. The results are shown in Figure 3.

On the left, both D KL (q θ K t (w t ) p(w t |d t )) and D KL (p ψ (w) p(w)) are close to zero indicating the success of the learning. More interestingly, in the middle, we see that θ 0 t , θ 1 t , . . . , θ 4 t evolves gradually towards the ground truth, which suggests that the synthetic gradient network is able to identify the descent direction after meta-learning.

Zero-shot classification: unsupervised multi-source domain adaptation

A more interesting zero-shot multi-task problem is unsupervised domain adaptation. We consider the case where there exists multiple source domains and a unlabeled target domain. In this case, we treat each minibatch as a task. This makes sense because the difference in statistics between two minibatches are much larger than in the traditional supervised learning. The experimental setup is similar to few-shot classification described in Section 6.5.1, except that we do not have a support set and the class labels between two tasks are the same. Hence, it is possible to explore the relationship between class labels and self-supervised labels to implement the initialization network λ without resorting to support set. We reuse the same model implementation for SIB as described in Section 6.5.1. The only difference is the initialization network. Denote by z t := {z t,i } n i=1 the set of self-supervised labels of task t, the initialization network λ is implemented as follows: (6.19) where λ6 is a global initialization similar to the one used by MAML; L t is the self-supervised loss, ẑt is the set of predictions of the self-supervised labels. One may argue that θ 0 t = λ would be a simpler solution. However, it is insufficient since the gap between two domains can be very large. The initial solution yielded by eq. (6.19) is more dynamic in the sense that θ 0 t is adapted taking into account the information from x t .

θ 0 t = λ -η ∇ θ L t ẑt ŷt (f (x t ), w t (θ, )), f (x t ) , z t ,
In terms of experiments, we test SIB on the PACS dataset (Li et al. 2017a), which has 7 object categories and 4 domains (Photo, Art Paintings, Cartoon and Sketches), and compare with state-of-the-art algorithms for unsupervised 134 EB Transductive Meta-Learning with Synthetic Gradients domain adaptation. We follow the standard experimental setting [START_REF] Carlucci | Domain generalization by solving jigsaw puzzles[END_REF], where the feature network is implemented by ResNet-18. We assign a selfsupervised label z t,i to image i by rotating the image by a predicted degree. This idea was originally proposed by Gidaris et al. (2018) for representation learning and adopted by [START_REF] Xu | Self-supervised domain adaptation for computer vision tasks[END_REF] for domain adaptation. The training is done by running ADAM for 60 epochs with learning rate 10 -4 . We take each domain in turns as the target domain. The results are shown in Table 3. It can be seen that SIB-Rot (K = 3) improves upon the baseline SIB-Rot (K = 0) for zero-shot classification, which also outperforms state-of-the-art methods when the baseline is comparable.

Conclusion

We have presented an empirical Bayesian framework for meta-learning. To enable an efficient variational inference, we followed the amortized inference paradigm, and proposed to use a transductive scheme for constructing the variational posterior. To implement the transductive inference, we make use of two neural networks: a synthetic gradient network and an initialization network, which together enables a synthetic gradient descent on the unlabeled data to generate the parameters of the amortized variational posterior dynamically. We have studied the theoretical properties of the proposed framework and shown that it yields performance boost on MiniImageNet and CIFAR-FS for few-shot classification. Lemma 6.A.1. If, for all t, f t (X, Y ) is σ-subgaussain under P X ⊗ P Y , then (6.26) Proof. The proof is adapted from the proof of Xu (2016, Lemma 4.2).

E P (T ) E P (X,Y |T ) f T (X, Y ) -E P (X|T )P (Y |T ) f T (X, Y ) ≤ 2σ 2 I(X; Y |T ).
LHS ≤ E P (T ) E P (X,Y |T ) f T (X, Y ) -E P (X|T )P (Y |T ) f T (X,
(f (x i ), w), y i ) ≡ t (w, z i )
is σ-subgaussian under q(w|t)q(z|t), then L t (w, d) is σ/ √ n-subgaussian under q(w|t)q(d|t) due to the iid assumption, and gen(q) ≤ 2σ 2 n I q (w; d|t). (6.31) Proof. First, if t (ŷ(f (x), w), y) is σ-subgaussian under q(w|t)q(z|t), by definition, E q(w|t)q(z|t) exp(λ t (w, z)) ≤ exp(λE q(w|t)q(z|t) t (w, z)) exp(λ 2 σ 2 /2) (6.32)

It is straightforward to show L t (w, d) is σ/ √ n-subgaussian since E q(w|t)q(d|t) exp(λL t (w, d)) = n i=1 E w,z i exp( λ n t (w, z i )) (6.33) ≤ n i=1 exp λ n E w,z i t (w, z i ) + λ 2 σ 2 2n 2 (6.34) = exp λE w,z t (w, z) exp( λ 2 σ 2 2n ) (6.35) = exp λE q(w|t)q(d|t) L t (w, d) exp( λ 2 (σ/ √ n) 2 2
). ( 6.36) By Lemma 6.A.1, we have gen(q) = E q(t) E q(w|d,t)q(d|t) L t (w, d) -E q(w|t)q(d|t) L t (w, d) (6.37) ≤ 2σ 2 n I(w; d|t) (6.38) as desired.

6.B Importance of synthetic gradients 137

6.B Importance of synthetic gradients

To further verify the effectiveness of the synthetic gradient descent, we implement an inductive version of SIB inspired by MAML, where the initialization θ 0 t is generated exactly the same way as SIB using λ(d l t ), but it then follows the iterations in eq. (6.6) as in MAML rather than follows the iterations in eq.( 6.10) as in standard SIB.

We conduct an experiment on CIFAR-FS using Conv-4-64 feature network. The results are shown in Table 6.B.1. It can be seen that there is no improvement over SIB (K = 0) suggesting that the inductive approach is insufficient. (5-way) and 15 queries in each category. We report the results with using different learning rate η as well as different number of updates K. Note that K = 0 is the performance only using the pre-trained feature.

6.C Varying the size of the query set

We notice that changing the size of d t (i.e., n) during training does make a difference on testing. The results are shown in Table 6

Introduction

Deep neural networks (DNNs) play important roles in various computer vision tasks, e.g., depth estimation [START_REF] Eigen | Depth map prediction from a single image using a multi-scale deep network[END_REF], pose estimation [START_REF] Toshev | Deeppose: Human pose estimation via deep neural networks[END_REF], optical flow [START_REF] Dosovitskiy | Flownet: Learning optical flow with convolutional networks[END_REF], object classification (He et al. 2016c), detection [START_REF] Girshick | Fast r-cnn[END_REF], and segmentation (Simonyan and Zisserman 2014b).

A typical DNN approach for a computer vision task is to train a sophisticated end-to-end neural network with a large amount of labeled data. Such an approach often delivers state-of-the-art performance if a sufficient amount of data is available. However, in many cases, it is impossible to gather sufficiently large data to train a DNN. For example, in many medical image applications (Schlegl et al. 2014), the amount of available data is constrained by the number of patients of a particular disease.

A popular approach for handling such lack of data is transfer learning [START_REF] Sinno Jialin Pan | A survey on transfer learning[END_REF], where the goal is to transfer knowledge from the source task to facilitate learning on the target task. Typically, one considers the source task to be generic with a larger amount of available data that contains useful knowledge for learning the target task, e.g., knowledge from natural image classification [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF]) is likely to be useful for fine-grained bird classification (Welinder et al. 2010). [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF] proposed the teacher-student framework for transferring such knowledge between DNNs being trained on the source and target tasks respectively. The high-level idea is to introduce an additional regularization for the DNN being trained on the target task, i.e., the student network, which allows learning the knowledge existing in the DNN that was pre-trained on the source task, i.e., the teacher network. While the framework was originally designed for knowledge transfer between DNNs on the same dataset, recent works (Yim et al. 2017, Zagoruyko andKomodakis 2016b) started exploiting its potential for more general transfer learning tasks, i.e., when the source data and the target data are different.

Many knowledge transfer methods have been proposed with various intuitions. Hinton et al. [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF] and [START_REF] Ba | Bach. Breaking the curse of dimensionality with convex neural networks[END_REF] propose to match the final layers of the teacher and the student network, as the outputs from the final layer of the teacher network provide more information than raw labels. [START_REF] Romero | Fitnets: Hints for thin deep nets[END_REF] proposes to match intermediate layers of the student network to the corresponding layers of the teacher network. Recent works [START_REF] Chen | Coupled end-to-end transfer learning with generalized fisher information[END_REF][START_REF] Huang | Like what you like: Knowledge distill via neuron selectivity transfer[END_REF][START_REF] Yim | A gift from knowledge distillation: Fast optimization, network minimization and transfer learning[END_REF], Zagoruyko and Komodakis 2016b) relax the regularization of matching the entire layer by matching carefully designed features/statistics extracted from intermediate layers of the teacher and the student networks, e.g., attention maps (Zagoruyko and Komodakis 2016b) and maximum mean discrepancy [START_REF] Huang | Like what you like: Knowledge distill via neuron selectivity transfer[END_REF].

Evidently, there is no commonly agreed theory behind knowledge transfer. This causes difficulty in understanding empirical results and in developing new methods in a more principled way. In this paper, we propose variational information distillation (VID) as an attempt towards this direction in which we formulate the knowledge transfer as maximization of the mutual information between the teacher and the student networks. This framework proposes an actionable objective for knowledge transfer and allows us to quantify the amount of information that is transferred from a teacher network to a student network. Since the mutual information is computationally intractable, we employ a variational information maximization (Agakov 2004) scheme to maximize the variational lower bound instead. See Figure 1 for the conceptual diagram of the proposed knowledge transfer method. We further show that several existing knowledge transfer methods (Li andHoiem 2017, Romero et al. 2014) can be derived as specific implementations of our framework by choosing different forms of the variational lower bound. We empirically validate our VID framework, which significantly outperforms existing methods. We observe the gap is especially large in the cases of small data and heterogeneous architectures.

In summary, the overall contributions of our paper are as follows:

• We propose variation information distillation, a principled knowledge transfer framework based on maximizing mutual information between two networks based on the variational information maximization technique.

• We demonstrate that VID generalizes several existing knowledge transfer method. In addition, our implementation of the framework empirically outperforms the state-of-the-art knowledge transfer methods on various knowledge transfer experiments, including knowledge transfer between (heterogeneous) DNNs on the same dataset or on different datasets.

• In particular, we demonstrate that heterogeneous knowledge transfer between convolutional neural networks (CNN) and multilayer perceptrons (MLP) is possible on CIFAR-10. Our method yields a student MLP that significantly outperforms the best-reported MLPs [START_REF] Lin | Network in network[END_REF][START_REF] Urban | Do deep convolutional nets really need to be deep and convolutional?[END_REF]) in the literature.

Variational information distillation (VID)

In this section, we describe VID as a general framework for the knowledge transfer in the teacher-student framework. Specifically, consider training a student neural network on a target task, given another teacher neural network pre-trained on a similar (or related) source task. Note that the source task and the target task can be the same. The underlying assumption is that the layers in the teacher network have been trained to represent certain attributes of given inputs that exist in both the source task and the target task. For a successful knowledge transfer, the student network must learn how to incorporate the knowledge of such attributes to its own learning.

From a perspective of information theory, knowledge transfer can be expressed as retaining high mutual information between the layers of the teacher and the student networks. More specifically, consider an input random variable x drawn from the target data distribution p(x) and K pairs of layers R = {(T (k) , S (k) )} K k=1 , where each pair (T (k) , S (k) ) is selected from the teacher network and the student network respectively. Feedforwarding the input x through the networks induces K pairs of random variables {(t (k) , s (k) )} K k=1 which indicate activations of the selected layers, e.g., t (k) = T (k) (x). The mutual information between the pair of random variables (t, s) is defined by:

I(t; s) = H(t) -H(t|s)
= -E t [log p(t)] + E t,s [log p(t|s)], (7.1)

where the entropy H(t) and the conditional entropy H(t|s) were derived from the joint distribution p(t, s). Note that the joint distribution p(t, s) is a result of aggregation over the layers with input x sampled from the empirical distribution p(x). Intuitively, the definition of I(t; s) can be understood as a reduction in uncertainty about the knowledge of the teacher encoded in its layer t when the the student layer s is known.

We now define the following loss function which aims to learn a student network for the target task while encouraging high mutual information with the teacher network:

L = L S - K k=1 λ k I(t (k) , s (k) ), (7.2)
where L S is the task-specific loss function for the target task and λ k > 0 is the hyper-parameter introduced for regularization of the mutual information. Equation eq.( 7.2) needs to be minimized with respect to the parameters of the student network. However, the minimization is hard since the computation of the exact mutual information is intractable. We instead propose a variational lower bound for each mutual information term I(t; s), in which we define a variational distribution q(t|s) that approximates p(t|s): [log q(t|s)], (7.3) where the expectations are over the distribution p(t, s) and the last inequality is due to the non-negativity of the Kullback-Leiber divergence D KL (•). Such a technique is known as the variational information maximization [START_REF] Barber | The im algorithm: a variational approach to information maximization[END_REF]. Finally, we obtain VID by applying the variational information maximization to each mutual information term I(t (k) , s (k) ) in eq.( 7.2), leading to a minimization of the following loss function:

I(t; s) = H(t) -H(t|s) = H(t) + E t,s [log p(t|s)] = H(t) + E t,s [log q(t|s)] + E s [D KL (p(t|s)||q(t|s))] ≥ H(t) + E t,s
L = L S - K k=1 λ k E t (k) ,s (k) [log q(t (k) |s (k) )]. (7.4)
The objective L is jointly minimized over the parameters of the student network and the variational distribution q(t|s). Note that the entropy term H(t) has been removed from the equation eq.( 7.3) since it is constant with respect to the parameters to be optimized. Alternatively, one could interpret the objective eq.(7.4) as jointly training the student network for the target task and maximization of the conditional likelihood to fit the activations of the selected layers from the teacher network. By doing so, the student network obtains the "compressed" knowledge required for recovering activations of the selected layers in the teacher network.

Algorithm formulation

We further specify our framework by choosing a form made for the variational distribution q(t|s). In general, we employ a Gaussian distribution with heteroscedastic mean µ(•) and homoscedastic variance σ as the variational distribution q(t|s), i.e., the mean µ(•) is a function of s and the standard deviation σ is not. Next, the parameterization of µ(•) and σ is further specified by the type of layer corresponding to t. When t corresponds to intermediate layer of the teacher network with spatial dimensions indicating channel, height and width respectively, i.e., t ∈ R C×H×W , our choice of variational distribution is expressed as follows:

log q(t|s) = -

C c=1 H h=1 W w=1 log q(t c,h,w |s) (7.5) = C c=1 H h=1 W w=1 log σ c + (t c,h,w -µ c,h,w (s)) 2 2σ 2 c + constant,
where t c,h,w denote scalar components of t indexed by (c, h, w). Further, µ c,h,w represents the output of a single unit from the neural network µ(•) consisting of convolutional layers and the variance is ensured to be positive using the softplus function, i.e., σ 2 c = log(1 + exp(α c )) + where α c ∈ R being the parameter to be optimized and > 0 is minimum variance introduced for numerical stability. Typically, one can choose s from the student network with similar hierarchy and spatial dimension as t. When spatial dimension of two layers are equal, 1 × 1 convolutional layers are typically used for efficient parameterization of µ(•). Otherwise, convolution or transposed convolution with larger kernel size could be used to match the spatial dimensions.

We additionally consider the case when the layer t = T (logit) (x) ∈ R N corresponds to the logit layer of the teacher network. Here, our choice of the variational distribution is expressed as follows:

-log q(t|s) = - N n=1 log q(t n |s) (7.6) = N n=1 log σ n + (t n -µ n (s)) 2 2σ 2 n + constant,
where t n indicates the n-th entry of the vector t, µ n represents the output of a single unit of neural network µ(•) and σ n is parameterized by softplus function to enforce positivity. For this case, the corresponding layer s in the student network is the penultimate layer S (pen) instead of the logit layer in order to match the hierarchy of two layers without being too restrictive on the output of the student network. Furthermore, we found that using a simple linear transformation for the parameterization of the mean function was sufficient in practice, i.e., µ(s) = Ws for some weight matrix W.

The aforementioned implementations turned out to perform satisfactorily during the experiments. We also considered using heteroscedastic variance σ(•), but it gave unstable training with ignorable improvements. Other types of parameterizations such as a heavy-tailed distribution or the mixture density network (Bishop 1994) could be used to gain additional performance, but we leave it for future exploration.

See Figure 2 for an illustration of the training VID using the implementation based on equation eq.(7.5). Here, we display the change in the evaluated log-likelihood of the variational distribution aggregated over channels, i.e., log q(t h,w |s) = c log q(t c,h,w |s), given input x (Figure 2(a)) throughout the VID training process. One observes that the student network is trained gradually for the variational distribution to estimate the density of the intermediate layer from the teacher network (Figure 2(b), 2(c) and 2(d)). As a comparison, we also optimize the variational distribution for the student network trained without knowledge transfer, (Figure 2(e)). For this case, we observe that this particular instance of the variational distribution fails to obtain high log-likelihoods, indicating low mutual information between the teacher and the student networks. Interestingly, the parts that correspond to the background achieve higher magnitudes compared to that of the foreground in general. Our explanation is that the output of layers corresponding to the background that mostly corresponds to zero activations (Figure 2(f)) and contains less information, being a relatively easier target for maximizing the log-likelihood of the variational distribution.

Connections to existing works

The infomax principle

We first describe the relationship between our framework and the celebrated infomax principle [START_REF] Linsker | An application of the principle of maximum information preservation to linear systems[END_REF]) applied to representation learning [START_REF] Vincent | Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion[END_REF], stating that "good representation" is likely to contain much information in the corresponding input. Especially, such a principle has been successfully applied to semi-supervised learning for neural networks by maximizing the mutual information between the input and output of the intermediate layer as a regularization to learning the target task, e.g., learning to reconstruct input based on autoencoders [START_REF] Rasmus | Semi-supervised learning with ladder networks[END_REF]. Interestingly, our framework can be viewed similarly as an instance of semi-supervised learning with modification of the infomax principle: layers of the teacher network contain important information for the target task, and a good representation of the student network is likely to retain much of their information. One recovers the traditional semi-supervised learning based on the infomax principle when we set t (k) = x in the equation eq.(7.2).

Generalizing mean squared error matching

Next, we explain how existing knowledge transfer methods based on mean squared error matching can be seen as a specific instance of the proposed framework.

In general, the methods will be induced from the equation eq.(7.4) by making a specific choice of the layers R = {(T (k) , S (k) )} K k=1 for knowledge transfer and parameterization of heteroscedastic mean µ(•) in the variational distribution:

-log q(t|s) = N n=1 (t n -µ n (s)) 2 2 + constant. (7.7)
Note that the equation eq.(7.7) corresponds to a Gaussian distribution with unit variance over every dimension of the layer in the teacher network. [START_REF] Ba | Bach. Breaking the curse of dimensionality with convex neural networks[END_REF] showed that knowledge can be transferred between the teacher and the student networks that were designed for the same task, by matching the output of logit layers T (logit) , S (logit) from the teacher and the student networks with respect to mean squared error. Such a formulation is induced from the equation eq.(7.7) by letting R = {(T (logit) , S (logit) )}, and µ(s) = s in the equation eq.(7.7). This was later extended for knowledge transfer between the teacher and the student networks designed for different tasks by [START_REF] Li | Learning without forgetting[END_REF], through adding an additional linear layer on top of the penultimate layer S (pen) in the student network to matching with logit layer T (logit) in the teacher network. This is induced similarly from the equation eq.(7.7) by letting R = {(T (logit) , S (pen) )}, and µ(•) being a linear transformation, i.e., µ(s) = Ws. Next, Romero et al. [START_REF] Romero | Fitnets: Hints for thin deep nets[END_REF] proposed a knowledge transfer loss for minimizing the mean squared error between intermediate layers from the teacher and the student networks, with additional convolutional layer introduced for adapting different dimension size between each pair of matched layers. This is recovered from the regularization term in the equation eq.(7.7) by choosing layers for the knowledge transfer to be intermediate layers of the teacher and the student networks, and µ(•) being a linear transformation corresponding to a single 1 × 1 convolutional layer.

In general, the described methods are similar to our implementation of the framework as they all use Gaussian distribution as the variational distribution. However, our method differs mainly in two ways: (a) allowing to use more flexible nonlinear functions for heteroscedastic mean and (b) modeling different variance for each dimension in the variational distribution. This allows transferring mutual information in a more flexible manner without wasting much capacity of the model. Especially, modeling unit variance for all dimensions of the layer t in the teacher network could be highly restrictive for the student network. To illustrate, the layer of the teacher network might include an activation t n that contains information irrelevant to the task of the student network, yet requires much capacity for regression of µ n (s) to t n . This would raise over-regularization issues, i.e., wasting the majority of the student network's capacity on trying to fit such a unit. Instead, modeling high homoscedastic variance σ n for such dimension make its contribution ignorable to the overall loss, allowing one to "filter" out such unit in an efficient way.

Comparison with feature matching

Besides the knowledge transfer methods based on mean squared error matching, several works [START_REF] Chen | Coupled end-to-end transfer learning with generalized fisher information[END_REF][START_REF] Huang | Like what you like: Knowledge distill via neuron selectivity transfer[END_REF][START_REF] Yim | A gift from knowledge distillation: Fast optimization, network minimization and transfer learning[END_REF], Zagoruyko and Komodakis 2016b) have proposed to indirectly match the handcrafted features extracted from intermediate layers. More specifically, Zagoruyko and Komodakis (2016b) proposed to match the "attention maps" generated from activations from the layers. [START_REF] Huang | Like what you like: Knowledge distill via neuron selectivity transfer[END_REF] later generalized the attention map to the maximum mean discrepancy of the activations. [START_REF] Yim | A gift from knowledge distillation: Fast optimization, network minimization and transfer learning[END_REF] proposed to match the feature called Flow of Solution Procedure (FSP) defined by the Gram matrix of layers adjacent in the same network. [START_REF] Chen | Coupled end-to-end transfer learning with generalized fisher information[END_REF] considered matching the reconstructed input image from the intermediate layers of the teacher and the student networks. These methods could be seen as smartly avoiding the aforementioned over-regularization issue by filtering out information in the teacher network using expert knowledge. However, such methods potentially lead to suboptimal results when the feature extraction method is not apt for the particular knowledge transfer task and may discard important information from the layer of the teacher network in an irreversible way.

Experiments

We demonstrate the performance of the proposed knowledge transfer framework by comparing VID to state-of-the-art knowledge transfer methods on image classification. We apply VID to two different locations: (a) VID between intermediate layers of the teacher and the student network (VID-I) and (b) VID between the logit layer of the teacher network and the penultimate layer of the student network (VID-LP). For comparison, we consider the following knowledge transfer methods: the original knowledge distillation (KD) [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF], learning without forgetting (LwF) [START_REF] Li | Learning without forgetting[END_REF], hint based transfer (FitNet) (Zagoruyko and Komodakis 2016b), activation-based attention transfer (AT) Zagoruyko and Komodakis (2016b) and polynomial kernel-based neural selectivity transfer (NST) [START_REF] Huang | Like what you like: Knowledge distill via neuron selectivity transfer[END_REF]. Note that we consider FitNet as a regularization for training the student network (Zagoruyko and Komodakis 2016b) instead of a stagewise training procedure as first proposed in [START_REF] Romero | Fitnets: Hints for thin deep nets[END_REF]. We compare knowledge transfer methods for knowledge transfer between same and different datasets, which is commonly referred to as the knowledge distillation and transfer learning tasks respectively.

In all the experiments, we select the same pairs of intermediate layers for knowledge transfer based on VID-I, FitNet, AT and NST. Similarly, the same pairs of layers for knowledge transfer are used for LwF and VID-LP. All the hyper-parameters of both our and existing methods are chosen according to the performance on a validation set, which is 20% of the training set. We carefully pick the set of candidate values of hyper-parameters such that all the values proposed in the original works are included. The presented performances are the average of three repeated runs. More details about experiments are included in the supplementary material. The implementation of the algorithm will be made publicly available shortly.

Knowledge distillation

We first compare knowledge transfer methods on the traditional knowledge distillation task, where a student network is trained on the same task as the teacher network. By distilling the knowledge from a large teacher network into a small student network, we can speed up the computation for prediction. We further investigate two problems for this task: whether we can benefit from knowledge transfer in the small data regime and how much performance we lose by reducing the size of the student network? Note that we do not evaluate the performance of VID-LP and LwF as they are designed for transfer learning. When applied, KD, VID-LP and LwF delivered similar performance.

Reducing training data

Knowledge transfer can be a computationally expensive task. Given a pre-trained teacher network on the whole training data set, we explore the possibility of using a small portion of the training set for knowledge transfer. We demonstrate the effect of a reduced training set by applying knowledge distillation on CIFAR-10 (Krizhevsky 2009b) with four different sizes of training data. We employ wide residual networks (WRN) (Krizhevsky 2009b) for the teacher network (WRN-40-2) and the student network (WRN-16-1), where the teacher network is pre-trained on the whole training set of CIFAR-10. Knowledge distillation is applied to four different sizes of training set: 5000 (the full size), 1000, 500, 100 data points per class.

We compare VID-I with KD, FitNet, AT and NST. We also provide performances of the teacher network (Teacher) and the student network trained without any knowledge transfer (Student) as baselines. We choose four pairs of inter- mediate layers similarly to (Zagoruyko and Komodakis 2016b), each of which is located at the end of a group of residual blocks. We implemented VID-I using two 1 × 1 convolutional layers with hidden channel size as twice of the output channel size. The results are shown in Table 1. Our method, VID-I, outperforms other knowledge transfer methods consistently across all regimes. The performance gap increases as the size of dataset get smaller, e.g., VID-I only drops 10.26% of accuracy even when 100 data points per each class are provided to the student network. There is a 31.88% drop without knowledge transfer and a 15.52% drop for the best baseline, i.e., KD + AT.

Varying the size of the student network

The size of the student network gives a trade-off between the speed and the performance in knowledge transfer. We evaluate the performance of knowledge transfer methods on different sizes of the student network. The teacher network (WRN-40-2) is pre-trained on the whole training set of CIFAR-100. A student network with four choices of size, i.e., WRN-40-2, WRN-16-2, WRN-40-1, WRN-16-1, is trained on the whole training set of CIFAR-100. We compare our VID-I with KD, FitNet, AT and NST along with the Teacher and Student baselines. The choices of intermediate layers are the same as the previous experiment.

The results are shown in in Table 1. As also noticed by [START_REF] Furlanello | Born again neural networks[END_REF], the student network with the same size as the teacher network outperforms the teacher network with all the knowledge transfer methods. One observes that VID-I consistently outperforms FitNet, AT and NST, which correspond to the same choice of layers for knowledge transfer. It also outperforms KD except for the case when the structure of the student network is identical to that of the teacher network, i.e.,  where two methods can be combined to yield the best performance. to the student network (ResNet-18/VGG-9) for the MIT-67 dataset with varying number of data points per class (denoted by M ). We use M ≈ M avg to denote the setting where the number of data points per class is non-uniform and M avg in average.

Transfer learning

We evaluate knowledge transfer methods on transfer learning. The teacher network is a residual network (ResNet-34) (He et al. 2016c) pre-trained on the ImageNet dataset [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF]. We apply transfer learning to improve the performance of two separate image classification tasks. The first task is a finegrained bird species classification based on the CUB-200-2011 dataset (Welinder et al. 2010), which contains 11,788 images in total for 200 bird species. The second task is an indoor scene classification based on the MIT-67 dataset [START_REF] Quattoni | Recognizing indoor scenes[END_REF], which contains 15,620 images for 67 classes of indoor scenes. For both tasks, there are a relatively few numbers of images per class, which can significantly benefit from knowledge transfer from the ImageNet classification task. To evaluate the performance at various levels of data scarcity, we subsample both We evaluate the knowledge transfer methods in two scenarios: a smaller student network of the same architecture (ResNet-18) and different architecture (VGG-9) (Simonyan and Zisserman 2014b). We compare our VID-I and VID-LP with LwF, FitNet, AT and NST. We evaluate the performance of the student network without transfer learning (Student) as a baseline. For the teacher and the student network with ResNet architecture, we choose the outputs of the third and fourth groups of residual blocks (from the input) as the intermediate layers for knowledge transfer. In the case of the VGG-9 student network, we choose the fourth and fifth max-pooling layers as the intermediate layers for knowledge transfer, which corresponds to the same spatial dimension as the intermediate layers selected from the teacher network. For applying VID-I to the ResNet-18 student network, we as mentioned in Section 7.3.1. We use a teacher network with convolutional layers (WRN-40-2) pre-trained on CIFAR-10. We use a MLP with five fully connected hidden layers as the student network, constructed by stacking one linear layer, three bottleneck linear layers and one linear layer in sequence. Each is followed by a non-linearity activation in between. Here, the bottleneck layer indicates a composition of two linear layers without non-linearity that is introduced to speed up learning by reducing the number of parameters. All the hidden layers have the same h number of units and the bottleneck linear layer is composed of two linear layers with a size of h × h 4 and h 4 × h. The knowledge transfer between intermediate layers is defined between the outputs of four residual groups of the teacher network and the outputs of the first four fully connected layers of the student network. We compare VID-I with KD and FitNet since these knowledge transfer methods do not rely on spatial structures. For the same reason, AT and NST are not applicable to multilayer perceptrons. VID-I is implemented with multiple transposed convolutional layers without non-linearities. Specifically, the inputs for the variational distributions, i.e., the hidden layers of the MLP is treated as a tensor with 1 × 1 spatial dimensions. Single transposed convolutional layer with a 4 × 4 kernel, unit stride and zero padding is followed by multiple transposed convolutional layers with a 4 × 4 kernel, two strides, and single padding in order to match the spatial dimension of the corresponding layer of the teacher network for knowledge transfer. More details on implementations of the student network and the auxiliary distribution are in the supplementary material.

The results are shown in Table 5. Both FitNet and VID-I improve the performance comparing the baseline of directly training the intermediate layers of the student network. VID-I significantly outperforms FitNet on MLPs with different sizes. Furthermore, MLP-4096 outperforms the the state-of-the-art performance with MLP reported by [START_REF] Lin | Network in network[END_REF] (78.62%) and [START_REF] Urban | Do deep convolutional nets really need to be deep and convolutional?[END_REF] (74.32%) significantly. More importantly, our method bridges the performance gap between CNN (84.6% using one convolutional layer [START_REF] Urban | Do deep convolutional nets really need to be deep and convolutional?[END_REF])) and MLP shown in previous works.

Conclusion

In this work, we proposed the VID framework for effective knowledge transfer by maximizing the variational lower bound of the mutual information between two neural networks. The implementation of our algorithm is based on Gaussian observation models and is empirically shown to outperform other benchmarks in the distillation and transfer learning tasks. Using more flexible recognition models, e.g., [START_REF] Diederik P Kingma | Improved variational inference with inverse autoregressive flow[END_REF], for accurate maximization of mutual information and alternative estimation of mutual information, e.g., [START_REF] Belghazi | Mine: mutual information neural estimation[END_REF], are of future interest. Despite our principled approach, there exist many unanswered questions in knowledge transfer, such as which attributes of the teacher network should be chosen as a target for transfer learning? which layers of the student network should we transfer the knowledge into? We hope to be able to answer

Introduction

For a long time neural networks had a capacity problem: making them have too many parameters for a limited amount of data would dramatically affect their generalization capabilities. Thus, several regularization techniques were developed, for example, early stopping [START_REF] Weigend | Predicting the future: A connectionist approach[END_REF] and L 2 regularization. The advent of batch normalization (Ioffe and Szegedy 2015a), skip-connections (Hochreiter and Schmidhuber 1997a, Srivastava et al. 2015, He et al. 2016b), and overall architecture search helped to mitigate this problem very recently, so that increasing capacity no longer hurts the accuracy, but even before that it was known that "unimportant" connections between neurons can be removed, resulting in a network with significantly fewer parameters and little or no drop in performance. Optimal brain damage (LeCun et al. 1990b) proposed a second order method to prune such connections. Soft weight sharing [START_REF] Nowlan | Simplifying neural networks by soft weightsharing[END_REF]Hinton 1992, Ullrich et al. 2017b) can be used to very effectively reduce the number of parameters in a trained network. Denil et al. (2013b) showed that it is possible to learn part of filters and predict the rest. In fact, parameter sharing is one of the most important features of convolutional neural networks, where sharing is built into convolutional layer, and was thoroughly explored. Recurrent neural networks also heavily rely on sharing parameters over multiple time steps. It is also the key element in training siamese and triplet networks.

More recently, trained on massive amounts of data image recognition neural networks were quickly increasing in the number of parameters, starting from AlexNet (Krizhevsky et al. 2012a) and VGG [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]. Network-In-Network (Lin et al. 2013b) proposed to stack MLPs which share local receptive fields, removing the need of massive fully-connected layers, and reducing the number of parameters to achieve the same accuracy as AlexNet by several times. This technique was further adopted by Szegedy et al. (2015a) in their Inception architectures. Downsampling and upsampling can also be seen as a way of parameter sharing in SqueezeNet (Iandola et al. 2016b). More recently, Highway (Srivastava et al. 2015) and later ResNet (He et al. 2016b) proposed to add skip-connections, which allowed training of very deep networks with improved accuracy. It was later shown by Wide ResNet (Zagoruyko and Komodakis 2016a) that the number of parameters was key to their accuracy, and depth was complementary. Either very deep, or very wide residual networks could be trained with massive number of parameters, without suffering from decreased accuracy. After that, parameter sharing exploration continued on ResNet architectures. For example, [START_REF] Boulch | Sharesnet: reducing residual network parameter number by sharing weights[END_REF] proposed to share some portion of convolutional filters in each group of residual block, for example every second convolution, achieving relatively small performance loss.

Not all parameter sharing approaches brought both performance improvement and reduction in parameters, and there appears to be some trade-off between parameter sharing and computational efficiency. As an extreme case of parameter sharing with significant performance loss, HyperNetworks (Ha et al. 2017b), proposed to have another network to generate filters. Among methods improving computational efficiency, a good example is MobileNet (Howard et al. 2017b), which suggested to reparameterize each 3 × 3 convolution as a pair of depth-wise convolution and 1 × 1 convolution, can also be seen as a way of sharing parameters. Their work was later extended to grouped 1 × 1 convolution by ShuffleNet (Zhang et al. 2018), which further reduced the complexity of residual block.

Another approach to reducing number of parameters is post processing, when a trained networks is modified, either with or without additional fine tuning. A number of low rank approximation by tensor decomposition approaches were proposed, such as works of Jaderberg et al. (2014b) and [START_REF] Lebedev | Speeding-up convolutional neural networks using fine-tuned cp-decomposition[END_REF]. For example, Denton et al. (2014b) used low-rank decomposition of the weight matrices to reduce the effective number of parameters in the network. Such approaches tend to be less applicable as more parameter-effective architectures are being invented. A very effective way of reducing parameters in a trained network is deep compression proposed by [START_REF] Han | Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding[END_REF], with a reduction of parameters of dozens of times, achieved by pruning, weight sharing, quantization and compression. Weight sharing locations are determined by weight values in a trained network, similar in value neurons share the same weight. Common approach today is to train a massive network, and reduce it later. This can even improve results over single-time trained network [START_REF] Han | DSD: regularizing deep neural networks with dense-sparse-dense training flow[END_REF].

Recurrent neural networks are known to be significantly overparameterized as well. For example, Kim and Rush (2016) show that it is possible to reduce neural machine translation model size by 13× with an insignificant BLEU score drop by doing teacher-student knowledge distillation. Several works [START_REF] Merity | Pointer sentinel mixture models[END_REF][START_REF] Inan | Tying word vectors and word classifiers: A loss framework for language modeling[END_REF], Kim et al. 2016) focus on reducing number of parameters in language modeling tasks.

All of the above suggest that architectures and methods used for training deep neural networks are suboptimal, and could be significantly improved by organizing parameter sharing from the start. It would be interesting to learn it, and a few attempts were made, e.g. in DCT space with hashing (Chen et al. 2016b) and FFT space [START_REF] Mathieu | Fast training of convolutional networks through ffts[END_REF]. Also, automatic architecture search approaches, such as [START_REF] Zoph | Learning transferable architectures for scalable image recognition[END_REF], do not handle sharing.

Despite so much work on reducing neural network parameters, surprisingly, very little attention has been given to weight symmetry. To the best of our knowledge, it has not been used in the context of parameter sharing so far. We propose a number of possible approaches to enforce symmetry on weights from scratch, i.e. not using a pretrained network. Imposed correctly, such symmetry can significantly reduce the number of parameters and computational complexity by sacrificing little or no accuracy, both at train and test time. Post-processing methods such as deep compression could be applied later. Also, our parameterizations are simple to implement in any modern framework with automatic differentiation. Besides, specialized routines for symmetric matrix multiplication and convolution could be used in both training and testing [START_REF] Goto | High-performance implementation of the level-3 blas[END_REF], Nath et al. 2011[START_REF] Igual | Level-3 blas on a GPU : Picking the low hanging fruit[END_REF].

We believe that our findings are surprising and uncover interesting properties of deep neural networks, which could led to further advances in understanding and efficiency. Our contributions are summarized below.

• We propose an effective use of symmetry for model compression for the first time;

• Explore various ways of imposing symmetry constraints;

• Experimentally show that symmetry can successfully be imposed in various architectures and datasets, with no or little loss in accuracy;

• We show that our symmetric parameterization is generic and can be applied to both image classification and natural language processing tasks;

Symmetric reparameterizations

In this section we introduce several ideas to impose axial symmetry on the weights of convolutional / fully-connected layers, which prunes out a large fraction of redundant parameters and gains computational speed-up for both training and testing. Throughout this section we consider all operations applied to matrices, which can be easily extended to multidimensional tensors by block symmetry.

Motivation

Real symmetric matrices have only real eigenvalues, while rotation matrices whose rotation angles are positive have at least one complex eigenvalue. Thus, restricting the weight matrix to be symmetric for some layers is equivalent to forcing these layers to learn non-rotational transformations. In a deep neural network, this restriction, which can be considered as an inductive bias, should not hurt the overall performance, since different layers are encouraged to focus on particular transformations, and then a good representation of the input is built up by compositing all of them. On the other hand, introducing symmetric weights for network compression have multiple benefits:

• Symmetric matrix multiplications could deliver potential speedup over generic BLAS routines.

• Training from scratch: it avoids additional fine tuning compared with postprocessing methods (Jaderberg et al. 2014b[START_REF] Lebedev | Speeding-up convolutional neural networks using fine-tuned cp-decomposition[END_REF], Han et al. 2016).

• Easy to combine with other compression methods (e.g. with ShaResNet (Boulch 2017)).

Soft constraints

We first try to see if it is possible to enforce symmetry constraints in a soft manner during training, which has the advantage that training procedure remains very similar to standard supervised training. We start by adding a penalty term to the training loss, which leads to a soft constraint on W :

L(W ) + ρ vec(W ) -vec(W ) p , ( 8.1)
where L(W ) is a task-specific loss with respect to W ; ρ is a hyper-parameter to control the slackness of the constraints W = W ; vec(W ) is an operator that vectorizes the matrix into a column vector. For the norm of the penalty term, we use either p = 2 or p = 1. However, L 1 -norm turns out to be slightly more effective given that it promotes sparsity, which would possibly result in a larger number of exactly symmetric weights. Due to the soft constraints, the resulting matrix is not guaranteed to be symmetric and so at test time we use only the upper triangular part.

Hard constraints

Alternatively, we can make use of a specific parameterization by a linear operator T : W → Ŵ for some layer in the neural network that explicitly encode symmetry, where W is the actual weight and Ŵ is the constructed weight for that layer. We are interested in the case where W has fewer elements than Ŵ , which enables a reduction in weights while keeping exactly the same network architecture as if it is fully-parameterized. Note that we choose T to be nonparametric, thus the only additional burden of training is to forward and backward propagate through T ; while in testing, since Ŵ is symmetric, we only need to store the upper triangular of the learned Ŵ , which saves almost half of the space.

To this end, we propose different instantiations of T (W ) in terms of different ways to construct symmetric matrices.

Triangular parameterization

One of the simplest way to impose hard symmetry constraint is to define the linear operator T as a sum of the upper triangular matrix of W ∈ R N ×N , its transpose W and the diagonal vector

v ∈ R N : Ŵ = T (triu(W ), v) := diag(v) + triu(W ) + triu(W ) , ( 8.2)
where triu(W ) returns the upper triangular part of the matrix W . Note that we use the full matrix W in the above expression just to simplify the notation. In fact, the triangular parameterization stores only the upper triangular of W and the main diagonal v, while elements of the lower triangular will never be touched. Thus, the number of parameters in this parameterization is 1 2 N (N + 1), reduced by almost 2 times compared to full parameterization. Note that (triu(W ), v) should be initialized from the same distribution as if W is learned directly. Due to strong sharing, the gradient with respect to W will be twice higher in magnitude, so the learning rate needs to be adjusted accordingly. a LDL parameterization with additional assumptions1 on Ŵ . Specifically, the reparameterization T with actual weights L and D is given by Ŵ = T (L, D) := LDL , (8.5) where L, D are restricted to be unit lower triangular matrix and diagonal matrix respectively.

N-way symmetry parameterization

Inspired by triangular parameterization, which can be viewed as an axial symmetry about the main diagonal, we consider a more general N-way parameterization with respect to multiple axes of symmetry, where N denotes the number of repeated parts. Thus, previously introduced parameterizations are 2-way symmetries.

Given V as the actual weight, which is considered in general to be smaller than the required weight matrix W , we create a symmetrized version of W by a composition of linear transformations

Ŵ = T 1 • T 2 • . . . • T k (V ), (8.6)
where T j (•) is a basic linear operator which does one of the following things: translation, reflection, rotation, tessellation etc. In fact, the form of N-way symmetry is quite flexible. We consider only the cases where symmetry leads to efficient computations in testing. In this work, we focus on two N-way parameterizations: blocking and triangulizing. The details are listed as follows.

• Blocking: Given V as a M √ N × M √ N matrix, N -way blocking can be obtained by a series of reflections (denoted by reflect axis (•)) to include mirrors.

• Triangulizing: Given V as an isosceles right triangle with area M 2 N , Ŵ is obtained by a series of reflections about different axes.

We demonstrate several examples of N-way symmetries in Figure 1.

Combining with other methods

It is not surprising that hard-constrained symmetry parameterization can be viewed as a special weight sharing method. Nevertheless, symmetry parameterization is capable of taking advantage of special matrix computation routines to speed up both training and testing, which is not the case for unstructured weight sharing methods such as duplicating randomly picked elements to form Ŵ .

Symmetry parameterization can also be complementary to other parameter reduction or weight sharing methods. For example, we force weight sharing not only between residual blocks within the same stage [START_REF] Boulch | Sharesnet: reducing residual network parameter number by sharing weights[END_REF]) but also within the same residual block using symmetry parameterizations. We show in Table 3 that triangular parameterization can be combined with ResNeXt [START_REF] Xie | Aggregated residual transformations for deep neural networks[END_REF] and MobileNet (Howard et al. 2017b), which have already been designed to take advantage of weight sharing. Besides, post-processing methods [START_REF] Han | Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding[END_REF]) can be applied on the trained symmetric weights with fine tuning, thus further reducing the number of parameters. We show experiments on combing channelwise symmetry with other parameter reduction methods (e.g. ShaResNet [START_REF] Boulch | Sharesnet: reducing residual network parameter number by sharing weights[END_REF])) in Section 8.4.1.

Implementations of block symmetry

The proposed symmetry parameterization is a generic parameter reduction method, which can be easily adapted to various network architectures. For example, convolutional / linear layers in feedforward networks (such as VGG [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], ResNeXt [START_REF] Xie | Aggregated residual transformations for deep neural networks[END_REF], MobileNet (Howard et al. 2017b) etc.) and in recurrent networks (such as LSTM (Hochreiter and Schmidhuber 1997a), GRU [START_REF] Cho | On the properties of neural machine translation: Encoder-decoder approaches[END_REF] etc.) can be symmetrized. In the case of grouped convolution in ResNeXt, block symmetry can be imposed on each group kernel: for instance, suppose that filters are of shape g × N × N × k × k, where g is the number of groups, we can impose symmetry on dimensions of N × N . However, symmetry is not directly applicable to DenseNet [START_REF] Yedidia | Dual decomposed learning with factorwise oracle for structural SVM of large output domain[END_REF], as the number of filters grows with every layer. We conduct some experiments along this direction in Section 8.4.1.

Imposing symmetry in convolutional neural networks

We denote by (W , W ) the whole set of parameters of the convolutional neural network, where W is the subset of parameters to be symmetrized, and W is the subset of free parameters. To be more specific, we assume there are totally L convolutional layers being reparameterized to equip symmetric parameters. That is, W := {W l } L l=1 with W l satisfying certain symmetric properties: W l ∈ R No×N i ×K h ×Kw is constructed so that the number of input channels is equal to the number of outputs channels (i.e. N i = N o = N ) and the spatial domain is a square (i.e. K h = K w = K). We propose to impose symmetry on slices of W l , namely, on the slice W l i , which is a square matrix. Depending on which direction to slice the tensor, we have channel-wise symmetry and spatial symmetry. In general, we can write W l := {W l i } i∈I . For channel-wise symmetry, W l i is a N × N symmetric matrix, and I := (k h , k w ) | k h , k w ∈ {1, . . . , K} ; For spatial symmetry, W l i is a K × K symmetric matrix, and I := (k i , k o ) | k i , k o ∈ {1, . . . , N } . Since these two symmetries are not exclusive, we can indeed impose both at the same time.

In theory, both channel-wise and spatial symmetries will reduce the freedom of layers, and their ability to approximate functions. Enforcing them to a shallow network can be problematic, since it may significantly reduce the expressive power of the network. Deep highway and residual networks, on the other hand, are more robust to symmetric weights, since they have many layers that are capable to make relatively small changes and iteratively improve the representation of the input [START_REF] Greff | Highway and residual networks learn unrolled iterative estimation[END_REF]). In addition, spatial symmetry can be further motivated by the success of scattering networks [START_REF] Bruna | Invariant scattering convolution networks[END_REF], whose filters are fixed as wavelets and constructed to enjoy certain symmetric properties.

We show in experiments that the proposed symmetries can be applied to several modern deep neural networks without suffering a significant drop in both training and testing accuracy.

Imposing symmetry in recurrent neural networks

We chose perhaps the most popular RNN variant, LSTM, which is known to be overparameterized, to experiment with symmetry. In a LSTM cell, the weight matrices between the hidden unit and gates (input, forget, output) as well as the weight matrix between the hidden unit and itself are square, so it is immediately valid to apply aforementioned symmetry parameterizations. This reduces about 25% parameters from the standard LSTM. We show in Section 8.4.3 that the symmetrized LSTM works as well as the standard LSTM in language modeling.

Experiments

This section is composed as follows. We start with CIFAR experiments, where we first test various symmetry parameterizations on wide residual networks (WRN) by Zagoruyko and Komodakis (2016a). After determining which parameterizations work best, we determine in which layers symmetry can be applied. We then test it with WRN of different widths and depths to determine the best configuration in terms of parameter reduction, computational complexity and simplicity. We also apply the proposed symmetrization to other architectures, and show that the conclusions drawn from CIFAR are able to transfer to larger datasets (ImageNet-1K dataset). Finally, we apply symmetrization to language modeling tasks.

We emphasize that our goal here is not to show state-of-the-art accuracy, but to show that very simple symmetry constraints can be used to significantly reduce Spatial symmetry parameterizations do not work as well as channel-wise symmetry. This is expected since the size of spatial dimensions is much smaller compared with channel dimensions (i.e. N i = N o > k h = k w ). Thus, the constraints imposed on spatial dimensions are much harsher making the learning much more difficult.

Based on the aforementioned analysis, we choose triangular parameterization (in terms of validation accuracy, parameter reduction and simplicity both at train and test time) as the main method to conduct all experiments further in this section.

Wide Residual Networks with symmetry

In this section we compare symmetrized WRN to its wider and thinner counterparts, and discuss the choice of network architecture. We use the terms conv0, conv1 and conv2 to refer to the first, the second and the third convolutional layers respectively (basic blocks have only conv0 and conv1 ). For our experiments we choose the bottleneck and constrain the mid-bottleneck 3 × 3 conv1 convolution to be symmetric, keeping 1 × 1 conv0 and conv2 unconstrained. This choice is because the approximating power of the residual block is least reduced, as real eigenvalues of conv1 are rotated by the surrounding convolutions, resulting in overall rich parameterization.

We present results for WRN-40-1-bottleneck and WRN-40-2-bottleneck trained on CIFAR in Table 2. We also train thinner networks with the same number of parameters to compare with their symmetric variants. On both datasets symmetric parameterization compares favorably to both wider and thinner non-symmetric counterparts in terms of accuracy and number of parameters. We further illustrate this in Fig. 2a, where we show training and validation accuracy of WRN with respect to various depths and widths. WRN has a lower accuracy in shallower networks when the training accuracy does not reach 100%, that is, the network struggles to fit into training data. In such cases symmetry constraints damage both training and validation accuracy significantly. Here we also notice that symmetry constraints cause much smaller accuracy drop in networks which are able to fit into training data perfectly, having almost 100% training accuracy. We hereafter refer to such networks as overparameterized, and we should note that overparameterization should not be confused with overfitting, that is, overparameterized network do not suffer from poor generalization.

Other architectures

In this section we show that triangular symmetry works as well on other architectures and larger networks. We pick a residual network variant, ResNeXt, which reduces the number of parameters and computational complexity in ResNet by using grouped 3 × 3 convolution in bottleneck block. For large networks we use WRN-28-10, both basic and bottleneck variants, and a simple feedforward VGG. We also apply symmetry to MobileNets, a popular architecture for mobile devices. Even though being feedforward, it compares favorably to smaller residual networks such as ResNet-18 and ResNet-34, with significant reduction in parameters needed to achieve the same accuracy.

Results are presented in Table 3. We put triangular symmetry constraint on the second layer in each residual block of WRN-basic-28-10, and on 3 × 3 convolutional layers in WRN-bottleneck-28-10. In both cases, there are no drops in accuracy, as expected in overparameterized networks which easily achieve 100% training accuracy. Triangular parameterization works well even with ResNeXt, which has much less parameters in 3 × 3 layers. That is also the case for VGG, which, in contrast to others, does not have 1 × 1 or depth-wise convolutions between layers with symmetry. We believe the fact that there is no big reduction in accuracy is due to the existence of redundant parameters. In MobileNet, we parameterize all square 1× layers, and observe relatively small drop in accuracy.

Overall, we observe that overparameterized networks can easily benefit from symmetry parameterizations in terms of the number of parameters and potential speedup from more efficient implementation. It is surprising to see that triangular parameterization works well even with ResNeXt and MobileNet, which have already been designed to enjoy weight sharing. Among all these experiments, VGG with triangular parameterization reduces almost half of the parameters (i.e. 9.2 million), yet the accuracy still remains almost the same.

Importance of batch normalization

One might notice that batch normalization Ioffe and Szegedy (2015a) could potentially be a symmetry-breaking component when combined with symmetric convolutional or linear layers, as it can be viewed as an affine transform of each feature plane, or a diagonal fully connected layer. We, however, successfully apply symmetric parameterization to networks without batch normalization.

To test the influence of batch normalization on symmetric parameterization, we trained Network-In-Network on CIFAR with and without batch normalization, convergence plots are presented on Fig. 3. As can be seen, the difference in accuracies is very similar in both cases. If batch normalization played a significant role in improving symmetric parameterization, Network-In-Network without batch normalization (left) and triangular symmetry would have a higher accuracy drop. We use learning rate of 0.1 to train the networks with batch normalization, and of 0.01 without. Also, for triangular parameterization without batch normalization we reduce learning rate on upper triangular part by 2 to compensate for gradient magnitude increase due to sharing.

Combining symmetric parameterization with ShaResNet

We discussed the possibility of combining symmetry parameterizations with other parameter reduction methods in Section 8.2.4. Here, we conduct an experiment that combines 2-way channelwise symmetry with ShaResNet Boulch (2017), which is a weight sharing method that all conv1 of residual blocks (bottleneck block) within a group (i.e. layers between two dimensionality reduction convolutions) share the same weights. The results are shown in Table 4. It can be seen that the combination further reduces the number of parameters, while the testing performance is only slightly affected. In particular, with the bottleneck architecture, triangular symmetry pluses ShaResNet reduce about 33% of parameters and the performance drop is less than 1%.

N-way symmetries

As discussed in Section 8.2.3, it is possible to push forward the triangular parameterization to a more general N-way triangular parameterization. In addition to triangulizing and blocking, we also consider a chunking implementation (a naive N-way weight sharing: given V as a M × M N matrix. We define by tile N × (V ) a transforming function to tile/copy V N times to construct Ŵ = tile N × (V )) as an example to show that carefully designed N-way symmetries yield better performance than a naive N-way weight sharing. Recall that standard triangular parameterization is equivalent to 2-way triangulizing. In this section, we test our proposals including chunking, blocking and triangulizing to achieve N-way channelwise symmetry. For chunking, we examine the cases of N = 2, 4, . . . , 64. If the number of channels in a convolutional layer is less than N, we simply set N equal to the number of channels. For blocking and triangulizing, it is non-intuitive to come up with N-way symmetry for N > 4, so we only test the cases of N = 4, 16 and N = 2, 4, 8 respectively.

In our experiments, as shown in Fig. 4, chunking causes the steepest linear decrease in validation accuracy. Triangular parameterizations work fine even in the case of 8-way symmetry, which reduces almost 3 8 percentage of parameters from 2-way symmetry while only suffer about 1% decrease in accuracy.

ImageNet experiments

In this section we present ImageNet results for networks with triangular symmetry and without, to check if our conclusions from CIFAR transfer to larger dataset. We start with results for relatively small networks, MobileNet and ResNet-18. In MobileNet we impose triangular symmetry on all square 1 × 1 convolutional layers. ResNet-18 has basic block architecture and doesn't have enough parameters to fit into training data well, so we impose symmetry on every second 3 × 3 convolutional layer in block. Both MobileNet and ResNet-18 are relatively shallow networks and have 28% less parameters with triangular symmetry, so, as expected, the drop in accuracy is significant, see Table 5. Still, constrained MobileNet has only 3M parameters and achieves almost the same accuracy with ResNet-18.

As for large networks, we trained ResNet-50 and ResNet-101 with bottleneck block architecture and triangular symmetry on all 3 × 3 layers. As on CIFAR, drop in accuracy is much smaller for these: a reduction of 23% parameters (which would correspond to approximately ResNet-40 for ResNet-50 and ResNet-77 for ResNet-101) causes only 0.5% accuracy drop compared to unconstrained ResNet-50, and even smaller for ResNet-101, which is about 0.2%. We show convergence curves for both ResNet-50 and its symmetric variant on Fig. 2b.

All networks were trained in the same conditions and with the same hyperparameters. We used large mini-batch training approach as proposed in [START_REF] Goyal | Accurate, large minibatch SGD: training imagenet in 1 hour[END_REF] on 8 GeForce 1080Ti GPUs, scaling learning rate proportionally to mini-batch size. Also, we do not regularize batch normalization and depth-wise convolution parameters in MobileNet. Surprisingly, our MobileNet and ResNet baselines outperform the original networks proposed in Howard et al. (2017b) and He et al. (2016b). We plan to make our code and networks available for download online.

Language modeling

We test symmetry on a common language modeling Penn-Tree-Bank [START_REF] Marcus | Building a large annotated corpus of english: The penn treebank[END_REF]) dataset. Our experimental setup reflects that of [START_REF] Zaremba | Recurrent neural network regularization[END_REF] 2 . We use a network with 2-layer LSTM and dropout, and test out a medium size configuration with 650 neurons (dropout 0.5), and a larger model with 1500 neurons (dropout 0.65). We try to symmetrize weights corresponding to input and hidden and gates separately and altogether, as well for each gate separately. Surprisingly, we find that adding symmetry on all hidden gates does not hurt, and even obtain slightly lower perplexity for both medium and large models. The results are presented in Table 6. We use the average parameterization 1 2 (W + W ) as we notice it gives slightly better results. Also, there is no L 2 -regularization applied during training, so the original W weights converge to be almost symmetric. There is also a large variation in final validation perplexity, so we train each network 5 times with different random seed and report mean±std results for all models.

For a fair comparison we also trained thinner networks with 565 and 1300 hidden neurons for medium and large networks correspondingly, so that the number of parameters is approximately equal to those of hidden-symmetrized networks. Symmetrized versions of large networks compare favourably to these networks.

We also include experimental results on a larger wikitext-2 [START_REF] Merity | Pointer sentinel mixture models[END_REF] dataset in Table 6, which is about 2 times larger than Penn-Tree-Bank, as well experiments with symmetry on each gate separately. We apply averaging symmetry to hidden gates of LSTM and compare to thinner networks with comparable number of parameters. In this case symmetry slightly hurts perplexity of medium model, and more significantly large model. This might be due to different dropout regularization in the networks (we use the same dropout rates as in Penn-Tree-Bank3 , which may not be the best choices for wikitext-2).

Conclusion

In this paper, we have shown that, quite surprisingly, deep neural network weights can be successfully parameterized to be symmetric without suffering a significant loss in accuracy. The proposed symmetry parameterizations could lead to potentially significant improvements for a wide range of mobile applications in terms of computational efficiency (dedicated routines for symmetric convolutions and matrix multiplication can be applied) and storage efficiency (memory requirements for storing network weights are dramatically reduced).

For future work, it would be interesting to compare with other structural weight matrices (e.g., [START_REF] Liang Zhao | Theoretical properties for neural networks with weight matrices of low displacement rank[END_REF])) with computational benefits and understand what kinds of inductive bias are implied. It should also be noted that our universal approximation analysis holds only for approximating single univariate continuous functions. It remains an open question what theoretical guarantees a symmetric deep neural network can provide for more general multivariate functions.

Conclusion

The main thread of this thesis has been devoted to the efficiency of machine learning methods. However, the emphases are different for different parts. For probabilistic graphical models, we focused on the computational efficiency, for which we have investigated linearly convergent algorithms with the presence of equality constraints. For deep neural networks, we considered several efficiency issues due to the over-parameterization. Throughout the chapters, our proposed solutions mainly arise from variational inference. Because of its flexibility, we also heavily explored the underlying structure of particular problems.

In Chapter 2, we reviewed the inference and learning problems of discrete graphical models, which are NP-hard for general graphs. The computation of learning is even heavier since each gradient step of learning involves a full inference on the entire graph. Therefore, a strong incentive as discussed in Chapter 3 is that we would like to learn discrete graphical models with block-coordinate approximate inference, where blocks correspond to cliques in the graph. We devoted a large fraction of Chapter 3 to derive this algorithm. In particular, we exploited the fact that strong duality holds for the regularized maximum likelihood estimation.

The dual problem has a nice structure since 1) the primal solution can be directly retrieved from the dual solution via the representer theorem and 2) the objective can be decomposed into a non-smooth (and strongly concave) but separable part, a smooth but non-separable part, and an equality constraint. We employ an ADMM-like scheme to handle the equality constraint resulting in a min-max problem. It introduces additionally a linear term and a quadratic term absorbed by the non-smooth part and the smooth part respectively. Our contributions lie in both the algorithmic aspect and the convergence analysis. The proposed algorithm is inspired by inexact gradient descent. We control the number of inner maximization steps (by a linearly convergent subroutine) to yield a linear rate convergence for the outer minimization. Our theoretical analysis focuses on the trade-off between the inner exactness and the outer decrease, characterizing the global convergence in terms of the total number of inner iterations, which also implies a linear convergence in the primal.

The second topic of the thesis is dedicated to Bayesian deep learning. Bayesian approaches on neural networks treat the parameters or weights as random variables. We reviewed the relationship between Bayesian neural networks and compression in Chapter 4, as well as the recent understanding of the generalization in deep learning via compression. Following the lossy compression scheme, namely, the information bottleneck principle, we proposed, in Chapter 5 and Chapter 6, to use the mutual information between the weights and the dataset as a regularizer for Bayesian supervised learning and Bayesian meta-learning. We used a variational upper bound on the mutual information, reformulated the learning problem as an alternating minimization (aka bi-level optimization). Although this formulation was not derived from Bayes' rule, the optimization variables resemble the prior and the posterior in classical Bayesian models. It can also be viewed as an empirical Bayes approach, as discussed in Chapter 6. Apart from the insights on the connections between the empirical Bayes and the information bottleneck, our main contributions are the proposed new algorithms.

For supervised learning, there is only one instance of the dataset, to introduce a mutual information regularizer, we reinterpreted the dataset as a bootstrap sample drawn from the empirical data distribution, thus it boils down to compute the mutual information between the weights and a bootstrap sample (or equivalently a minibatch). The developed algorithm, as shown in Chapter 5, is referred to as β-BNN, since the inner step resembles the SGD step for a standard BNN with the KL term weighted by β, while the outer step maintains the aggregated posterior (i.e., the prior) as a running average of previous variational posteriors.

For meta-learning, the main challenge is how to implement a meta-model capable of gathering domain knowledge rapidly. Inspired by the aforementioned alternating minimization, we proposed to parameterize the inner minimization with respect to the variational posterior as a part of the meta-model, which consists of a synthetic gradient network and an initialization network. In particular, the synthetic gradient is the key to enable the transductive inference towards a better chracterization of the inner optimization dynamics. To support this proposal, we first showed in theory that, under this setting, the generalization error of the variational posterior can be largely reduced by transduction. Besides, we demonstrated that our algorithm significantly outperforms previous state-of-the-art methods on Mini-ImageNet and CIFAR-FS.

In the last two chapters, namely, Chapter 7 and Chapter 8, we presented our works about model compression for over-parameterized neural networks. The compression is done by either making a compact student network (which would be guided by a pretrained teacher network) or taking advantage of the symmetry in the weights. We proposed a new framework in Chapter 7 for the knowledge transfer from the teacher network to the student network by maximizing a variational lower bound of the mutual information between the activations of these two networks. This idea can also be used to transfer learning when the teacher is pretrained on another task. On the other hand, we conducted multiple experiments in Chapter 8 on image classification and language modeling and verified that enforcing weight symmetry is feasible in these tasks.

Future work: integrating data-driven methods and knowledge-driven methods

Coming back to the discussion in the introduction, a good restarting point is a recent blog (Sutton 2019) posted by Richard Sutton who suggests that "a bitter lesson" we all should learn from the competition between data-driven AI and knowledge-driven AI: "1) AI researchers have often tried to build knowledge into their agents, 2) this always helps in the short term, and is personally satisfying to the researcher, but 3) in the long run it plateaus and even inhibits further progress, and 4) breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning."

A few weeks later, Max Welling responds with a relatively neutral opinion on Twitter [START_REF] Welling | Do we still need models or just more data and compute[END_REF]:

"When you have sufficient data, you do not need to impose a lot of human generated inductive bias on your model. You can let the data speak. However, when you do not have sufficient data available you will need to use human-knowledge to fill the gaps." This is not the first debate between the supporters of data-driven methods and knowledge-driven methods. A thought-provoking summary was published by Marvin Minsky about thirty years ago in his article titled "Logical vs. Analogical or Symbolic vs. Connectionist or Neat vs. Scruffy" (Minsky 1991). His insights are of a fundamental and look-ahead character. One of the insights he shared is that since each method has its own virtues and deficiencies, an integrated system is needed to exploit the advantage of both:

"In favor of the top-down side, research in artificial intelligence has told us a little -but only a little -about how to solve problems by using methods that resemble reasoning. If we understood more about this, perhaps we could more easily work down toward finding out how brain cells do such things. In favor of the bottom-up approach, the brain sciences have told us something -but again, only a little -about the workings of brain cells and their connections. [. . .] I'll argue that the solution lies somewhere between these two extremes, and our problem will be to find out how to build a suitable bridge."

Minsky did not simply suggest that we should combine logical methods or probabilistic graphical models with deep neural networks. What he meant "an integrated system" is beyond a naive combination. He described some properties of the integrated system: "To solve typical real-world commonsense problems, a mind must have at least several different kinds of knowledge. First, we need to represent goals: what is the problem to be solved. Then the system must also possess adequate knowledge about the domain or context in which that problem occurs. Finally, the system must know what kinds of reasoning are applicable in that area. Superimposed on all of this, our systems must have management schemes that can operate different representations and procedures in parallel, so that when any particular method breaks down or gets stuck, the system can quickly shift over to analogous operations in other realms that may be able to continue the work."

This description, however, did not provide a specific solution and obviously, none of the solutions we have had possess all of these properties. My personal take is the following. An AI system of this kind must be reconfigurable such that

• it is able to automatically set up different goals (i.e., losses or rewards) for different scenarios;

• it is able to unify the feature (or knowledge) representation for the information learned from the data or embedded through human prior knowledge.

• it embodies a general reasoning mechanism on top of the feature representations regardless of the particular problem to be solved.

• it considers robustness as equally important as its main goals.

Indeed, none of these desiderata is completely new, however, they have not been put all together to build a functional system. Finding a principle way to integrate different pieces is an interesting direction for future work.

On the other hand, an AI system, as argued by Lawrence (2019), will never replace NI systems since the evolution, not to mention its time span, is unique. As such, blending knowledge-driven methods with data-driven methods or the other way around is arguably the most straightforward solution to build an AI system comparable to its NI counterpart, which is like to stand on the shoulder of the evolution giant. However, there are many foreseeable challenges towards a reasonable implementation of such blending, among which the main technical issue is how to unify the representations learned from different sources, which have to be consistent when they refer to the same thing. Besides, it will be more efficient to employ different paradigms for learning different functionalities. For example, if we are trying to implement a football-playing AI, the data-driven methods will be more effective in learning ball touches, dribbles and so on from practical games, while the knowledge-driven methods will be more effective in learning gestures, tactics, formations and so on.

Last but not least, it will still be relevant to improve variational techniques since they are the key building blocks to realize robustness and uncertainty. For future work, one important topic is to achieve more efficient algorithms for variational inference. The current state-of-the-art algorithms are based on stochastic gradient descent, thus they inherit the limitations of gradient methods -it is not a global optimization algorithm and its convergence suffers from the oscillation near a stationary point. To overcome bad local minima, which occurs in general variational inference, one promising idea is to treat the optimization as an optimal control problem. If we learn to solve this problem, taking gradient as a part of the state information, and keeping a good balance of exploration and exploitation, we may find a better local minimum with much faster convergence rate.

3 .Definition 1 . 3 . 5 .

 3135 The rate associated with an encoder enc : X → {1, . . . , K} is defined as log K. Definition 1.3.4. A (R, D) pair is achievable for a distribution p(X) and distortion function d(•, •), if there exists an encoder enc : X → {1, . . . , e R } and a decoder dec : {1, . . . , e R } → X such that E x∼p(x)d(x, dec(enc(x))) ≤ D. The closure of the set of achievable pairs is called the rate-distortion region. The rate-distortion function R(D) is the infimum of rates R such that (R, D) is achievable.

Figure 1 .

 1 1 for an example. On the other hand, given an autoencoder induced by p(x|x) ∈ P x, where P x := {p(x|x) ≥ 0 | x p(x|x) = 1}, we rewrite the rate and the distortion respectively as I(p(x|x)) := I(X; X), (1.47) D(p(x|x)) := x,x p(x)p(x|x)d(x, x). (1.48)

Figure 1 . 1 :

 11 Figure 1.1: The rate-distortion plane.

Figure 2 . 1 :

 21 Figure 2.1: A factor graph example. Circles represent variable nodes. Boxes represent factor nodes. Note that there are only edges between circles and boxes.

Theorem 2 . 3 . 3 .

 233 Consider the following PGM = p θ (x), G = (V, E, A) : • p θ (x) = exp θ, φ(x) -F (θ) belonging to a minimal exponential family, and θ ∈ Θ := {θ | F (θ) < +∞};

4 . 3 .

 43 Definition 3.4.1. Let F I and F L be the counterparts of F obtained by relaxing M to I and L respectively, which, in other words, are the Fenchel conjugates of F * I and F * L when these are defined by H Approx :

Proposition 3 . 6 . 6 .

 366 If A is SDCA, let |C| be the total number of cliques, σ c the strong convexity constant of f * c , and L c the Lipschitz constant of µ c → r(µ), then A is linearly convergent with rate π = min c∈C σc |C|(σc+Lc) .

Figure 3 . 1 :

 31 Figure 3.1: The comparison between IDAL and other baselines. For the choices of hyperparameters in terms of accuracy and speed, we set (λ = 10, ρ = 1, γ = 1) for Gaussian mixture Potts, (λ = 10, ρ = 1, γ = 10) for semantic segmentation and (λ = 1, ρ = 0.1, γ = 1) for multi-label classification. The x-axis is running time in seconds.

  ) + (similarly we have θ ) that includes the loss term = [ c (y c , y c ) : y c ∈ Y c ] : c ∈ C . The density function of the loss-augmented CRF then takes the form p γ (y | y , x; w) = exp η (w)/γ, T (y) -F (η (w)/γ) .

  Pρ (w, δ, ξ) Proposition 3.B.1. For a fixed ξ, the primal objective function of D ρ (µ, ξ) takes the form

4 .

 4 The relations between D, D ρ , P and P ρ could be summarized as

Lemma 3 .

 3 D.1. d(ξ) is convex and L d -smooth, where L d ≤ ρ.

  and d is a convex function by Fenchel conjugacy. For any ξ 1 and ξ 2 , denote by µ 1 and µ 2 the minimizers of D ρ (•, ξ 1 ) and D ρ (•, ξ 2 ) respectively. By convexity of d(ξ) and the definition of subgradient, there exists s 1 ∈ ∂F * I (µ 1 ) and s 2 ∈ ∂F * I (µ 2 ) such that

  where L d is the smoothness constant of d.Proof. Let d * (y) = max ξ ξ, y -d(ξ). Then, it can easily be checked by using the definition of d and exchanging the order of maximization and minimization that d * (y) = min µ D ρ (µ, 0) + ι {Aµ=y} , Since d is convex, we have d(ξ) = max y ξ, y -d * (y), so that if y (ξ) is a maximizer for fixed ξ we have 0

  By definition, we have D ρ (μ t+1 , ξ t+1 ) = D ρ (µ t+1, * , ξ t+1 ) = d(ξ t+1

  2 2 and the gap in objective value |D ∞ (μ t ) -D ∞ (µ )| converge to 0 linearly. Proof. For the first inequality, by Lemma 3.D.1 we know that d is an L d -smooth function. It is then a standard result (see e.g. Nesterov 2013, Thm 2.1.5) that we therefore have ∇d

4 (

 4 Total number of inner updates). With the notations of the previous corollary, for any β ∈ (0, 1) such that λ max (β) < 1, it is possible to obtain E[ ∆t ] ≤ and E[Γ t ] ≤ with a total number of inner iterations T tot := T in T ex such that

Proposition 3 . 6 . 6 .

 366 If A is SDCA, let |C| be the total number of cliques, σ c the strong convexity constant of f * c , and L c the Lipschitz constant of µ c → r(µ), then A is linearly convergent with rate π = min c∈C σc |C|(σc+Lc) .

  type of a clique τ c N the type of clique c w τ R dτ the parameter shared by all cliques with type τ w R τ dτ w := [w τ ] τ ∈T φ c (x, y c ) R dτ c the feature vector associated with the clique c given Y c = y c Z(x, w) R + * the partition function of p(y|x; w) c (y (n) c , y c ) R + the user defined loss function associated with the clique c γ (0, +∞) the temperature parameter of the loss-augmented CRF Ψ

Figure 1 :

 1 Figure 1: Left: The comparison of top-1 accuracy on the ImageNet dataset. Right: Top-1 accuracy vs. giga floating point operations per second. The top-1 accuracy is calculated on the ImageNet dataset with a single forward pass. The number of parameters for a network is demonstrated by the size of the disk, which is ranged from 5M to 155M . Source: Figure 1 & 2 in Canziani et al. (2016).

Figure 2 :

 2 Figure 2: Energy consumption of deep neural networks. We will soon reach the capacity of the human brain. Source: Welling (2018).

Figure 3 :

 3 Figure 3: (a) Fitting random labels and random pixels on CIFAR10. Training losses of various experiments decrease to near zero. Source: Figure 1 in Zhang et al. (2016). (b) Experiments with ResNet18 architectures of different sizes on CIFAR-10. Even when after network is large enough to completely fit the training data(reference line), the test error continues to decrease for larger networks. Source: Figure 1 in Neyshabur et al. (2018). (c) The data (5 points) is generated from y = x 3 -2x 2 + 1 + N (0, 0.1). MLP do not overfit even with 12 layers which is a network containing 18000 parameters. As a comparison, the dashed line shows an overfitted solution. Source: Figure1in[START_REF] Wu | Towards understanding generalization of deep learning: Perspective of loss landscapes[END_REF].

Figure 4 :

 4 Figure 4: An intuitive example shows that flat minima generalizes better. Source: Figure 1 in Keskar et al. (2016).

  x|x), m(x)) + βE p(x|x)m(x) D KL p(y|x) p(y|x) .

Figure 5 :

 5 Figure 5: The trajectories of the points (I(T ; X), I(T ; Y )) at different epochs from different layers.The neural network is a CNN with ReLU activations and 5 hidden layers, which is trained on the CIFAR-10 dataset with SGD for 10 4 epochs. Each trajectory is associated with a hidden layer, and is represented by a sequence of points calculated at each epoch. The color is chosen to be directly linked to epoch. There is a clear transition between two phases, which can be seen for all trajectories. Due to the data processing inequality, lower layers always have higher mutual information. Source: the figure is taken from the comment of Naftali Tishby in the discussion at https://openreview.net/forum?id=ry_WPG-A-.

Figure 6 :

 6 Figure 6: FitNet by Romero et al. (2014).

Figure 7 :

 7 Figure 7: Attention transfer by Zagoruyko and Komodakis (2016b).

Figure 1 :

 1 Figure 1:The graphical model of Bayesian supervised learning. Given a dataset, the supervised learning can be viewed as an autoencoding of the labels, where w is the shared latent variable.

Lemma 5 . 1 . 1

 511 (10.8.1 (Cover and Thomas 2012)). The mutual information has a variational form: I(X; Y ) = min m(y) D KL (p(x, y) p(x)m(y)), where m * (y) = p(y) = p(x, y)dx.

Table 1 :Figure 2 :

 12 Figure 2: Testing accuracy over training epochs.

Figure 1 :

 1 Figure 1: (a) The generative and inference processes of the empirical Bayes model are depicted in solid and dashed arrows respectively, where the meta-parameters are denoted by dashed circles due to the point estimates. A comparison between MAML eq.(6.6) and our method (SIB) eq.(6.10) is shown in (b) and (c). MAML is an inductive method since, for a task t, it first constructs the variational posterior (with parameter θ K ) as a function of the support set d l t , and then test on the unlabeled x t ; while SIB uses a better variational posterior as a function of both d l t and x t : it starts from an initialization θ 0 t (d l t ) generated using d l t , and then yields θ K t by running K synthetic gradient steps on x t .

Figure 2 :

 2 Figure 2:The computation graph to compute the negative ELBO, where the input and output of the synthetic gradient module are highlighted in red. The detach() is used to stop the back-propagation down to the feature network. Note that we do not include every computation for simplicity.

3 :

 3 while not converged do 4:

Figure 1 :

 1 Figure 1: Conceptual diagram of the proposed knowledge transfer method. The student network efficiently learns the target task by minimizing the cross-entropy (CE) loss while retaining high mutual information (MI) with the teacher network. The mutual information is maximized by learning to estimate the distribution of the activations in the teacher network, provoking the transfer of knowledge.

Figure 2 :

 2 Figure 2: Plots for the heat maps corresponding to the variational distribution evaluated for spatial dimensions of the intermediate layer in the teacher network, i.e., log q(t h,w |s) = c log q(t c,h,w |s). Each figure corresponds to (a) original input image, (b, c, d) log-likelihood log q(t h,w |s) that was normalized and interpolated to fit the spatial dimension of the input image (red pixels correspond to high probability), (d) log-likelihood of variational distribution optimized for the student network trained without any knowledge transfer applied and (f) magnitude of the layer t averaged for each spatial dimensions.

  datasets into three different sizes (50, 25, 10 per class for MIT-67 and 20, 10, 5 per class for CUB-200-2011) and compare the knowledge transfer methods.

Figure 1 :

 1 Figure 1: N-way parameterizations. (a) Original 4 × 4 weight matrix. (b) 4-way blocking: V is the bottom-right block; Ŵ = reflect -(reflect | (V )). (c) 4-way triangulizing: V is the top triangle; Ŵ = reflect / (reflect \ (V )). (d) 8-way triangulizing: V is the top-left triangle; Ŵ = reflect / (reflect \ (reflect | (V ))).

  Convergence curves of top-1 (top lines) and top-5 (bottom lines) validation errors of ResNet-50 and its triangularly symmetrized variant on ImageNet.

Figure 2 :

 2 Figure 2: Image classification results for ResNet with symmetric filters.

Figure 3 :

 3 Figure 3: Combining symmetric parameterization without batch normalization (left) and with (right), Network-In-Network. Training accuracy in shown by dashed lines, validation -solid. Accuracy drop is similar in both cases.

Figure 4 :

 4 Figure 4: N-way sharing (chunking) v.s. N-way symmetries (blocking, triangulizing) on CIFAR-10 with WRN-16-1-bottleneck. x-axis represents N. y-axis represents the accuracy.

  Hu and Guillaume Obozinski. "SDCA-Powered Inexact Dual Augmented Lagrangian Method for Fast CRF Learning." International Conference on Artificial Intelligence and Statistics (AISTATS), 2018.

	Chapter 4 gives a general introduction to deep neural networks with an emphasis
	on network compression and the over-parameterization phenomenon of neural
	networks.
	Chapter 5 is based on the following paper:
	Shell X. Hu, Pablo G. Moreno, Andreas Damianou, Neil D. Lawrence.
	"β-BNN: A Rate-Distortion Perspective on Bayesian Neural Networks."
	NeurIPS Workshop on Bayesian Deep Learning, 2018
	Chapter 6 is based on the following paper:

Background: Convex Optimization and

	Sungsoo Ahn, Shell X. Hu, Andreas Damianou, Neil D. Lawrence,
	Zhenwen Dai. "Variational Information Distillation for Knowledge
	Transfer." IEEE Conference on Computer Vision and Pattern Recogni-
	tion (CVPR), 2019.
	Chapter 8 is based on the following publication:
	Shell X. Hu, Sergey Zagoruyko, Nikos Komodakis. "Exploring Weight
	Symmetry in Deep Neural Networks." Computer Vision and Image
	Understanding (CVIU), 187, p.102786, 2019.

Chapter 1

1 Convex Optimization 11

Definition 1.1.2 (Lipschitz continuous gradient). A differentiable function is said to have Lipschitz continuous gradient if there exists a constant

L > 0 such that ∇f (x) -∇f (y) ≤ L x -y , ∀x, y ∈ dom(f ), (1.7)

where L is called the Lipschitz constant of f . Theorem 1.1.2. The following statements are equivalent:

1. A differentiable convex function f (x) has Lipschitz continuous gradient.

1.

  Theorem 1.3.1 (Rate-distortion theorem, Cover and Thomas (2012) (Theorem 10.2.1)). The rate-distortion function for an iid random variable X with distribution p(x) and a bounded distortion function d(•, •) is given by

  .1.

		Bayesian networks	Markov random fields
		n
	Factorization	p(x) =	p(x i |x pa(i) )	p(x) = 1 Z	ψ a (x a )
		i=1		a∈A
	Independence	d-separation	Separation
	Marginalization	Not closed in general,	Closed
		only when marginalizing leaf nodes

Table 2 .1: Comparison

 2 of some properties of Bayesian networks and Markov random fields.

1

  Lc f * c . Then, by Danskin's theorem 1 , applied to equation eq.(3.5), we have that Aμ t is an approximate gradient of d(ξ t ), and so, Algorithm 1 updates ξ with ξ t+1 = ξ t -1 L

d Aμ t , where L d is the Lispchitz constant of d(ξ). As for the stopping criteria, 1 Bertsekas (See e.g. 1999, Prop. B.25) 3.6 Convergence Analysis 57 Algorithm 1 IDAL scheme 1: Input: T in , T ex , 2: Initialize: μ0 c = 1 kc 1 for all c ∈ C and ξ 1 = 0 3: for t = 1, . . . , T ex do 4:

  (seeLemma 3.D.3 in Appendix 3.D.2). If A is a linearly convergent algorithm with rate π and if it runs for T in iterations, such that, for some β : λ max

	Corollary 3.6.2.

Table 3 .

 3 B.1: The Comparison of Structured Learning Methods

	Method	Learning Regime Primal/Dual	Convergence Inference Oracle
	Meshi et al. (2010)	SSVM	Primal (w, δ)	Sublinear	graphwise MAP (inexact)
	Hazan and Urtasun (2010)	LossAugCRF	Primal (w, δ)	Sublinear	graphwise marginal (inexact)
	Lacoste-Julien et al. (2013a) SSVM	Dual (µ)	Sublinear	graphwise MAP
	Schmidt et al. (2015)	CRF	Primal (w)	Linear	graphwise marginal
	Le Priol et al. (2018)	CRF	Dual (µ)	Linear	graphwise marginal
	Tang et al. (2016)	CRF	Dual (µ)	Sublinear	graphwise MAP
	Meshi et al. (2015b)	SSVM (soft)	Dual (µ, ξ = 0) Sublinear	cliquewise MAP
	Yen et al. (2016)	SSVM	Dual (µ, ξ)	Linear	cliquewise MAP
	IDAL	LossAugCRF	Dual (µ, ξ)	Linear	cliquewise marginal

Corollary 3.6.2 and Corollary 3.6.3 Corollary

3.6.2. If

A is a linearly convergent algorithm with rate π and if it runs for T in iterations, such that, for some β :

  and E[∆ 0 t ] converge to zero linearly.We restate Corollary 3.6.3 in a more detailed way.(Corollary 3.6.3). Let σ denote the strong convexity constant of µ → D ρ (µ, ξ) and L d the smoothness constant of d. Assume that ( ξ t 2 ) t∈N is almost surely bounded by a constant B. Then the squared residuals to the constraint

	Corollary 3.D.1

Table 3 .

 3 F.1: Notations for sets

	Notation Dimension Description
	C	the set of cliques
	E	the set of edges
	V	the set of nodes

Table 3 .

 3 F.2: Notations for variables, parameters and functions

	Notation	Domain	Description
	k i	N	the number of values that

Table 3 .

 3 F.3: Notations smoothness, strong convexity constant and related quantities

	Notation Description
	L d	the Lipschitz constant of ∇d(ξ)
	τ	the constant of PL inequality for d(ξ)
	σ c

4.2 Deep Network Architectures 91

	CNN	#parameters size(M) Conv% FC% FLOPS(G) Conv% FC% computation
	LeNet-5	0.431	6.0	94.0	0.0066	87.8	12.2
	AlexNet	61	3.8	96.2	0.72	91.9	8.1
	VGGNet	138	6.3	93.7	2.6	96.3	3.7
	NIN	7.6	100	0	1.1	100	0
	GoogLeNet	6.9	85.1	14.9	1.6	99.9	0.1
	ResNet-18	5.6	100	0	1.8	100	0
	ResNet-50	12.2	100	0	3.8	100	0
	ResNet-101	21.2	100	0	7.6	100	0

Table 1 :

 1 The computation and size of common convolutional neural networks.

  inherits the best elements of NIN and GoogLeNet, which is now one of the most commonly used architectures.

	The key ingredient
	introduced in this architecture is the skip connection, which further alleviates
	the issues of vanishing/exploding gradients. Moreover, it is shown by Orhan
	and Pitkow (2017) that the skip connections eliminate saddle points thus
	the training of ResNet can potentially converge to a better local minima.
	As a result, He et al. (2016a) demonstrated that there is no big technical
	problems on training a neural network with a thousand layers. Zagoruyko

1

Informally, it is the region in the image that involves in the computation of a neuron.

Table 2 :

 2 Average classification accuracies (with 95% confidence intervals) on the test-set of MiniImageNet and CIFAR-FS. For evaluation, we sample 2000 and 5000 episodes respectively for MiniImageNet and CIFAR-FS and test three different architectures as the feature extractor:

	Conv-4-64, Conv-4-128 and WRN-28-10. We train SIB with learning rate 0.001 and try different
	numbers of synthetic gradient steps K.

  Each benchmark is composed of disjoint training, validation and testing classes.

	MiniImageNet is proposed by Vinyals et al. (2016), which contains 100 classes,
	split into 64 training classes, 16 validation classes and 20 testing classes; each
	image is of size 84×84. CIFAR-FS is proposed by Bertinetto et al. (2018), which
	is created by dividing the original CIFAR-100 into 64 training classes, 16 validation
	classes and 20 testing classes; each image is of size 32×32.

  Left: the mean-square errors on D test , E t D KL (q θ K t (w t ) p(w t |d t )), D KL (p ψ (w) p(w))and the estimate ofI(w; d) ≈ E t D KL (q θ K t (w t ) p ψ (w t )). Middle: the predicted y's by y = θ k t x for k = 0, . . . , 4. Right: the predictions of SIB.

	Prediction comparison
	2.0
	GT of task 1
	SIB predictions
	GT of task 2
	SIB predictions
	GT of task 3
	SIB predictions
	y
	Figure 3: et al. (

Table 3 :

 3 Multi-source domain adaptation results on PACS with ResNet-18 features. Three domains are used as the source domains keeping the fourth one as target.

Table 6 .B.1: Average

 6 5-way classification accuracies (with 95% confidence intervals) withConv-4-64 on the test set of CIFAR-FS. For each test, we sample 5000 episodes containing 5 categories

			inductive SIB		SIB
			Training on 1-shot	Training on 1-shot	Training on 5-shot
			Testing on	Testing on	Testing on
	K	η	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot
	0	-	59.7±0.5% 75.5±0.4% 59.2±0.5% 75.4±0.4% 59.2±0.5% 75.4±0.4%
	1 1e-1 59.8±0.5% 71.2±0.4% 65.3±0.6% 75.8±0.4% 64.5±0.6% 77.3±0.4%
	3 1e-1 59.6±0.5% 75.9±0.4% 65.0±0.6% 75.0±0.4% 64.0±0.6% 77.0±0.4%
	5 1e-1 59.9±0.5% 74.9±0.4% 66.0±0.6% 76.3±0.4% 64.0±0.5% 76.8±0.4%
	1 1e-2 59.7±0.5% 75.5±0.4% 67.8±0.6% 74.3±0.4% 63.6±0.6% 77.3±0.4%
	3 1e-2 59.5±0.5% 75.8±0.4% 68.6±0.6% 77.4±0.4% 67.8±0.6% 78.5±0.4%
	5 1e-2 59.8±0.5% 75.7±0.4% 67.4±0.6% 72.6±0.6% 67.7±0.7% 77.7±0.4%
	1 1e-3 59.5±0.5% 75.6±0.4% 66.2±0.6% 75.7±0.4% 64.6±0.6% 78.1±0.4%
	3 1e-3 59.9±0.5% 75.9±0.4% 68.7±0.6% 77.1±0.4% 66.8±0.6% 78.4±0.4%
	5 1e-3 59.4±0.5% 75.7±0.4% 69.1±0.6% 76.7±0.4% 66.7±0.6% 78.5±0.4%
	1 1e-4 58.8±0.5% 75.5±0.4% 59.0±0.5% 75.7±0.4% 59.3±0.5% 75.7±0.4%
	3 1e-4 59.4±0.5% 75.9±0.4% 58.9±0.5% 75.6±0.4% 59.3±0.5% 75.9±0.4%
	5 1e-4 59.3±0.5% 75.3±0.4% 60.1±0.5% 76.0±0.4% 60.5±0.5% 76.4±0.4%

Table 6 .

 6 .C.1. C.1: Average classification accuracies on the validation set and the test set of Mini-ImageNet with backbone Conv-4-128. We modify the number of query images, i.e., n, for each episode to study the effect on generalization.

	n	5-way, 5-shot Validation	Test	5-way, 1-shot Validation	Test
	3	77.97 ± 0.34%	75.91 ± 0.66%	63.60 ± 0.52%	61.32 ± 1.02%
	5	78.14 ± 0.35%	76.01 ± 0.66%	64.67 ± 0.55%	62.50 ± 1.02%
	10 78.30 ± 0.35% 76.22 ± 0.66% 65.34 ± 0.56% 63.22 ± 1.04%
	15	77.53 ± 0.35%	75.43 ± 0.67%	65.14 ± 0.55%	62.59 ± 1.02%
	30	76.21 ± 0.35%	74.04 ± 0.67%	63.37 ± 0.53%	60.96 ± 0.98%
	45	75.65 ± 0.36%	73.27 ± 0.66%	62.08 ± 0.51%	59.59 ± 0.93%

Table 1 :

 1 Experimental results (test accuracy) of knowledge distillation on the CIFAR-10 dataset from teacher network (WRN-40-2) to student network (WRN-16-1) with varying number of data points per class (denoted by M ).

	M	5000	1000	500	100
	Teacher	94.26	-	-	-
	Student	90.72 84.67 79.63 58.84
	KD	91.27 86.11 82.23 64.24
	FitNet	90.64 84.78 80.73 68.90
	AT	91.60 87.26 84.94 73.40
	NST	91.16 86.55 82.61 64.53
	VID-I	91.85 89.73 88.09 81.59
	KD + AT	91.81 87.34 85.01 76.29
	KD + VID-I	91.7	88.59 86.53 78.48

.11 73.69 72.16 67.19Table 2 :

 2 Experimental results (test accuracy) of knowledge distillation on the CIFAR-100 dataset from the teacher network (WRN-40-2) to the student networks (WRN-d-w) with varying factor of depth d and width w.

	(d, w)	(40,2) (16, 2) (40, 1) (16, 1)
	Teacher	74.16	-	-	-
	Student	74.34	70.42	68.79	65.46
	KD	75.80	72.87	70.99	66.03
	FitNet	74.29	70.89	68.66	65.38
	AT	74.76	71.06	69.85	65.31
	NST	74.81	71.19	68.00	64.95
	VID-I	75.25	73.31	71.51	66.32
	KD + AT	75.86	73.13	71.4	67.07
	KD + VID-I 76

Table 3 :

 3 Experimental results (test accuracy) of transfer learning from the teacher network

Table 4 :

 4 Experimental results (test accuracy) of transfer learning from the teacher network (ResNet-34) to the student network (ResNet-18/VGG-9) for the CUB-200-2011 dataset with varying number of data points per class (denoted by M ). We use M ≈ M avg to denote the setting where the number of data points per class is non-uniform and M avg in average.

Table 2 :

 2 CIFAR test error (median of 5 runs) of triangular channel-wise parameterization on WRN-40 with bottleneck layers. conv0 and conv1 refer to the first and the second convolutional layers respectively.

	base network	symmetry location width #params CIFAR-10 CIFAR-100
		none	1	0.59M	6.12	26.86
	WRN-40-1-bottleneck	none	0.875	0.45M	6.29	28.36
		conv1	1	0.45M	6.24	27.66
		none	2.0	2.34M	4.95	22.51
	WRN-40-2-bottleneck	none	1.75	1.79M	5.12	23.18
		conv1	2.0	1.76M	4.96	22.98

  WRN of various depth and width with bottleneck blocks and triangular symmetry. Dash (solid) lines denote train (val) accuracy respectively (medians over 5 runs).

		100
		98
	train/test accuracies, %	90 92 94 96	0.22M 0.86M 0.17M 0.67M 0.50M 0.40M	0.50M 1.97M 0.38M 1.49M	WRN-bottleneck, width=1	1.52M 6.02M 1.14M 4.49M
		88			WRN-bottleneck, width=2
			0.13M		WRN-bottleneck-symm, width=1
		86	0.10M		WRN-bottleneck-symm, width=2
			20	40	60	80	100
				network depth
	(a)

Table 3 :

 3 Triangular symmetry applied to various architectures on CIFAR. Triangular channelwise symmetry is imposed on every second convolution in residual blocks in WRN and ResNeXt, and in all square convolutions in MobileNet and VGG. Median test accuracy of 5 runs is reported.

Table 4 :

 4 Combining ShaResNet Boulch (2017) with 2-way channelwise symmetry. Test errors (mean/std/median over 5 runs) are compared for different symmetry parameterizations on CIFAR-10 using WRN-16-1-bottleneck with symmetric conv1.

	network	share symmetry #params mean std median
		none	0.219M	8.40 0.24	8.49
		none	0.171M	8.64 0.15	8.63
	WRN-16-1-bottleneck	triangular 0.172M	8.76 0.19	8.84
		triangular 0.147M	9.49 0.34	9.36
		average	0.171M	9.35 0.23	9.25

Table 6 :

 6 Language modeling perplexity (lower is better) on Penn-Tree-Bank and wikitext-2 datasets. Only hidden gate weights are symmetrized with triangular parameterization. We count only parameters in RNN, skipping encoder and decoder. Mean±std results over 5 runs are reported.

	Model	symmetry #parameters	Penn-Tree-Bank validation test	wikitext-2 validation test
	LSTM-650	6.8M	85.38 ± 0.29 81.49 ± 0.04 99.59 ± 0.17 94.26 ± 0.21
	LSTM-565	5.1M	85.48 ± 0.45 81.47 ± 0.28 100.60 ± 0.28 94.99 ± 0.24
	LSTM-650	5.1M	83.73 ± 0.42 79.73 ± 0.23 100.81 ± 0.45 95.43 ± 0.37
	LSTM-1500	36M	81.90 ± 0.54 77.95 ± 0.41 95.59 ± 0.34 90.92 ± 0.20
	LSTM-1300	27M	80.71 ± 0.14 77.42 ± 0.18 96.08 ± 0.26 90.77 ± 0.21
	LSTM-1500	27M	79.66 ± 0.41 75.69 ± 0.32 97.13 ± 0.77 91.97 ± 0.85

This argument may be wrong since, as mentioned by[START_REF] Vapnik | Measuring the vc-dimension of a learning machine[END_REF], the notion of model complexity should depend on a subset of functions which are reachable from the initialization.

Informally, a tree decomposition of a graph is a grouping of the nodes in the graph in tree structure. The width of a tree decomposition is the size of the largest group minus one, and the treewidth of a graph is the smallest width among all tree decompositions.

Since the entropy term does not have a Lipschitz gradient on the boundary of L, the proof of the linear convergence has to follow the stochastic mirror descent framework[START_REF] Beck | Mirror descent and nonlinear projected subgradient methods for convex optimization[END_REF][START_REF] Bolte | First order methods beyond convexity and lipschitz gradient continuity with applications to quadratic inverse problems[END_REF][START_REF] Zhang | On the convergence rate of stochastic mirror descent for nonsmooth nonconvex optimization[END_REF]) using properties of Bregman divergence. This is relatively straightforward, but to the best of my knowledge there is no published work dedicated to this.

We say that f : 2 S → R is submodular if for any A ⊂ B ⊂ S and x ∈ S, we havef (A ∪ {x}) -f (A) ≥ f (B ∪ {x}) -f (B).

http://sourceforge.net/projects/mulan/files/datasets/yeast.rar

https://openreview.net/forum?id=Sy8gdB9xx

Note that the rate is the minimum I( Ŝ, S) over the output Ŝ of the autoencoder. We

To distinguish from testing and training within a task, meta-testing and meta-training are referred to as testing and training over tasks.

In section 6.5, we verify our proposal empirically. Our experimental results demonstrate that our method significantly outperforms the state-of-the-art meta-learning methods on few-shot classification benchmarks under the one-shot setting.

θ K t is also dependent of f . We deliberately remove this dependency to simplify the update of f .

We do not insist on Bayesian estimation of the feature network because most Bayesian versions of CNNs underperform their deterministic counterparts.

Conv-4-64 consists of 4 convolutional blocks each implemented with a 3 × 3 convolutional layer followed by BatchNorm + ReLU + 2 × 2 max-pooling units. All blocks of Conv-4-64 have

feature channels. Conv-4-128 has 64 feature channels in the first two blocks and 128 feature channels in the last two blocks.

λ is overloaded to be both the network and its parameters.

If Ŵ is symmetric and it factorizes as Ŵ = LDU = U DL , then by uniqueness, it follows that Ŵ = LDL . However, Ŵ has a LU decomposition if Ŵ satisfies a particular rank condition studied by[START_REF] Okunev | Necessary and sufficient conditions for existence of the lu factorization of an arbitrary matrix[END_REF]. Thus, we in fact assumes Ŵ is better conditioned.

https://github.com/pytorch/examples/tree/master/word_language_model

Suggested by https://github.com/pytorch/examples/tree/master/word_language_model

Acknowledgments

Chapter

SDCA-Powered Inexact Dual Augmented Lagrangian Method for Fast CRF Learning Abstract

We propose an efficient dual augmented Lagrangian formulation to learn conditional random fields (CRF). Our algorithm, which can be interpreted as an inexact gradient descent algorithm on the multiplier, does not require to perform global inference iteratively, and requires only a fixed number of stochastic clique-wise updates at each epoch to obtain a sufficiently good estimate of the gradient w.r.t. the Lagrange multipliers. We prove that the proposed algorithm enjoys global linear convergence for both the primal and the dual objectives. Our experiments show that the proposed algorithm outperforms state-of-the-art baselines in terms of speed of convergence.

Chapter

4

A Survey on Over-Parameterization in Deep Learning: Compression and Generalization Abstract Deep neural networks are commonly over-parameterized, namely, the parameterization is redundant, and the number of parameters can be even larger than the number of training points. Over-parameterization has shown to be an interesting property for both optimization and generalization in the sense that it not only makes the optimization easier but also assists the learning to pursue a better generalization. In this chapter, we will review representative neural network architectures, state-of-the-art network compresssion techniques and the connection between the compression and the generalization of over-parametrized models. In addition, we will cover a related problem -knowledge distillation, which enables a relaxation of the training setting of over-parameterized models with an application on small-data-small-network training. The material of this chapter is based on a literature review up to November 2018. Since the field of deep learning theory is evolving sufficiently fast, some opinions and arguments may not be valid anymore.

Abstract

We propose an alternative training framework for Bayesian neural networks (BNNs), which is motivated by viewing the Bayesian model for supervised learning as an autoencoder for data transmission. Then, a natural objective can be invoked from the rate-distortion theory. Specifically, we end up minimizing the mutual information between the weights and the dataset with a constraint that the negative log-likelihood is smaller than a certain value. The classical Blahut-Arimoto algorithm for solving this kind of optimization is infeasible due to the intractable expectations over the weights and the dataset, so we develop a new approximation to the steps of the Blahut-Arimoto algorithm. Our method exhibits some attractive properties over the conventional KL-regularized training of BNNs with fixed Gaussian prior: firstly, improved stability during optimization; secondly, a more flexible prior which can be understood from an empirical Bayes viewpoint.

Chapter

Empirical Bayes Transductive Meta-Learning with Synthetic Gradients Abstract

We propose a meta-learning approach that learns from multiple tasks in a transductive setting, by leveraging the unlabeled query set in addition to the support set to generate a more powerful model for each task. To develop our framework, we revisit the empirical Bayes formulation for multi-task learning. The evidence lower bound of the marginal log-likelihood of empirical Bayes decomposes as a sum of local KL divergences between the variational posterior and the true posterior on the query set of each task. We derive a novel amortized variational inference that couples all the variational posteriors via a meta-model, which consists of a synthetic gradient network and an initialization network. Each variational posterior is derived from synthetic gradient descent to approximate the true posterior on the query set, although where we do not have access to the true gradient. Our results on the Mini-ImageNet and CIFAR-FS benchmarks for episodic few-shot classification outperform previous state-of-the-art methods. Besides, we conduct two zero-shot learning experiments to further explore the potential of the synthetic gradient.

Appendix Appendix

6.A Proofs

Theorem 6.A.1. Given distributions q(w|d, t), q(d|t), q(t), p(w) and p(d|w, t), we have eq.( 6.14) ≥ I q (w; d|t) + H q (d|w, t), (6.20) where I q (w; d|t) := D KL q(w, d|t) q(w|t)q(d|t) is the conditional mutual information and H q (w|d, t) := E q(w,d,t) [-log q(w|d, t)] is the conditional entropy. The equality holds when ∀t : D KL (q(w|t) p(w)) = 0 and D KL (q(d|w, t) p(d|w, t)) = 0.

Proof. Denote by q(w|t) := E q(d|t) q(w|d, t)q(d|t) the aggregated posterior of task t. eq.( 6.14) can be decomposed as

The inequality is because D KL (q(w|t) p(w)) ≥ 0 for all t's. Besides, we used the notation H q,p , which is the conditional cross entropy.Recall that D KL q(d|w, t) p(d|w, t) = -H q (d|w, t) + H q,p (d|w, t) ≥ 0. We attain the lower bound as desired if this inequality is applied.

The following lemma and theorem show the connection between I q (w; d|t) and the generalization error. We first extend Xu (2016, Lemma 4.2 The results are shown in Table 3 and4. The knowledge transfer from ResNet-34 to VGG-9 gives a very similar performance to the transfer from ResNet-34 to ResNet-18 for all the knowledge transfer methods. This shows that knowledge transfer methods are robust against small architecture changes. Our methods outperform other knowledge transfer methods in all regions of comparison. Both VID-I and VID-LP outperforms baselines that correspond to the same choice of layers for knowledge transfer. For the MIT-67 dataset, we observe that our algorithm outperforms even the finetuning method, which requires pre-training of the student network on the source task.

Knowledge transfer from CNN to MLP

The transfer learning experiments show the robustness of the knowledge transfer method against small architecture changes. This leads to an interesting question: whether a knowledge transfer method can work between two completely different network architectures. A solution to this question can open a new direction of knowledge transfer and potentially offer solutions to many problems, e.g., speeding up prediction of recurrent neural networks (RNNs) by transferring knowledge from a RNN to a CNN, speeding up prediction of CNN on CPU or low-energy device by transferring knowledge from a CNN to a multi-layer perceptron (MLP). In this paper, we evaluate the performance of knowledge transfer from CNN to MLP on CIFAR-10.

There is a well-known performance gap between CNN and MLP on CIFAR-10 ( [START_REF] Lin | Network in network[END_REF][START_REF] Urban | Do deep convolutional nets really need to be deep and convolutional?[END_REF]. The state-of-the-art performance on CIFAR-10 with MLP is 78.62% with initialization from auto-encoders by [START_REF] Lin | Network in network[END_REF] and 74.32% using knowledge distillation by [START_REF] Urban | Do deep convolutional nets really need to be deep and convolutional?[END_REF]. [START_REF] Urban | Do deep convolutional nets really need to be deep and convolutional?[END_REF] also trained a single convolutional layer achieving the performance of 84.6% using knowledge distillation.

We apply the knowledge transfer methods in the knowledge distillation setting

Average parameterization

Let us also consider a redundant, but more straightforward formulation of the reparameterization T , in which we keep N × N matrix W as the actual weight, and define T as a sum of W and its transpose W divided by two:

Although this is a redundant parameterization to obtain a symmetric matrix, we wanted to explore its performance given that it is known that overparameterized networks are often easier to optimize / train (e.g., directly training compact networks is much more challenging compared to first training overparameterized ones and then properly pruning their parameters). In this case, the learning rate remains the same since the gradient has been scaled by definition. As in triangular parameterization, W is initialized from the same distribution as basic non-symmetric parameterization. The number of parameters is the same as nonsymmetric parameterization at training time, but 1 2 N (N + 1) at testing time due to the fact that Ŵ is symmetric.

Eigen parameterization

In addition, we consider a more generic eigen parameterization, inspired by the fact that any symmetric matrix has an eigen decomposition. Given a matrix V ∈ R N ×R and a vector λ ∈ R R as actual weights, T is defined by Ŵ = T (V , λ) := V diag(λ)V . (8.4) Ideally, V has to be an orthogonal matrix, but we relax this constraint due to a heavy computation of performing projected stochastic gradient descent. We still initialize V and λ from an eigen decomposition of the initial full weight matrix.

The number of parameters in such relaxed parameterization is N (R + 1) at training time, and 1 2 N (N + 1) at testing time. We empirically choose R = N/2 as it yields similar performance as the case of R = N .

Note that a similar approach was suggested by Denil et al. (2013b), where Ŵ is parameterized by matrices U and V with columns of U forming a dictionary of basis functions.

LDL parameterization

There is a close relationship between the eigen decomposition of a matrix and its LDL decomposition. Recall that the LDL decomposition factorizes a matrix as a product of an unit lower triangular matrix L (meaning that all elements on the diagonal are 1's), a diagonal matrix D and the transpose of L. The advantage of using LDL decomposition over eigen decomposition is that it is much easier to maintain a valid decomposition of Ŵ during training. We thus also consider Exploring Weight Symmetry in Deep Neural Networks the number of parameters in various network architectures.

There are two common ResNet variants are considered as baselines in the following experiments: basic blocks and bottleneck blocks (He et al. 2016b). Basic blocks have two 3 × 3 convolutional layers and a parallel residual connection; bottleneck block is a combination of 1 × 1, 3 × 3 and 1 × 1 convolutional layers. Similar to WRN, we refer to WRN-n-k-blocktype as the network of depth n, width k (number of channels multiplier), and blocktype meaning either basic or bottleneck.

Code for all our experiments is available at https://github.com/hushell/ deep-symmetry.

CIFAR experiments

The results of various experiments on CIFAR-10/100 datasets are presented below.

Symmetry parameterizations

We first compare various symmetry parameterizations on CIFAR-10 with WRN-16-1-bottleneck, which is a relatively small network enabling us to perform quick experiments. The median validation errors over 5 runs are reported in Table 1 All symmetry parameterizations have certain drop in performance comparing to the baseline. We observe that channel-wise triangular / average parameterizations have the lowest drop. Eigen parameterization does not work well in this experiment, which is possibly a consequence of V is not forced to be an orthogonal matrix. On the other hand, LDL parameterization as an alternative attains a better result. We also test soft channel-wise L 1 -norm (see eq. 8.1), where the number of parameters remains the same in training, and reduced at test time by using upper triangular weights only. The slackness is controlled by the coefficient ρ. For a large ρ, the soft-constrained symmetrization is slightly better than hard-constrained symmetrizations.